DevHeads.net

Non Blocking write in apache

Hi All,

I am using proxy_http to proxy my request to tomcat from apache. But in
non-blocking call it is not working properly(timeout issue). I am using the
default timeout i:e 60 sec.

When writev returns -1 with errorno==4101, it should wait and then again
try for write.
while ((rv == -1) && (errno == EAGAIN || errno == EWOULDBLOCK)
&& (sock->timeout > 0)) _____ code snippet from
sendrecv.c

In debugging I have checked that sock->timeout is equal to 0. Therefore
this condition is failed and request failed.

When default timeout is 60sec then why does sock->timeout becomes 0 ??

On further debugging, I came to know the reqtimeout_eor function is setting
timeout to 0.

static apr_status_t reqtimeout_eor(ap_filter_t *f, apr_bucket_brigade *bb)
{
if (!APR_BRIGADE_EMPTY(bb) && AP_BUCKET_IS_EOR(APR_BRIGADE_LAST(bb))) {
reqtimeout_con_cfg *ccfg = f->ctx;
ccfg->timeout_at = 0;
}
return ap_pass_brigade(f->next, bb);
}

I am not able to understand why this function is setting to 0.

Can somebody help me to understand the issue ?

Thanks
Hemant

Comments

Re: Non Blocking write in apache

By Hemant Chaudhary at 09/12/2018 - 05:48

Hi All,

Actually it is setting sock->timeout to 0 in writev_nonblocking() in
core_filters.c.
arv = apr_socket_timeout_set(s, 0);

Why does the default timeout is changed and set to 0 ??
Thanks
Hemant

On Wed, Sep 12, 2018 at 4:03 PM Hemant Chaudhary <

Re: Re: Non Blocking write in apache

By Yann Ylavic at 09/12/2018 - 10:07

On Wed, Sep 12, 2018 at 12:48 PM Hemant Chaudhary
<hemantdude. ... at gmail dot com> wrote:
Because writev_nonblocking() is really non-blocking and should return
EAGAIN/EWOULDBLOCK when things start to block.
But writev_nonblocking() is a local function, called (indirectly) by
the core output filter which deals with EAGAIN appropriately.

Why is it an issue for mod_proxy_http? EAGAIN remains local to the
core filtering, but however mod_proxy_http always passes a FLUSH
bucket at the end of the body, so everything should be fine at the
core filter level...

Regards,
Yann.

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/14/2018 - 01:08

Hi Yann

I have tried to upload a file of 26.5mb on apache tomcat, which was
successfull.
I tried to proxy request to tomcat with same file. But get following error.
Attaching the error_log.

Can you please help me to understand where I am going wrong ?

Thanks
Hemant

Re: Re: Non Blocking write in apache

By Yann Ylavic at 09/14/2018 - 05:25

Hi Hermant,

On Fri, Sep 14, 2018 at 8:08 AM Hemant Chaudhary
<hemantdude. ... at gmail dot com> wrote:
I don't run Windows, sorry I'm no expert there, more questions than
answers below...
So apr_poll() returns APR_EINVAL, right?

What could be the reasons for poll() in cygwin to return EINVAL?
I don't know how cygwin implements poll(), but for instance I see that
WSAPoll() can return:
WSAEINVAL An invalid parameter was passed. This error is returned if
the fdarray parameter contains a NULL pointer. This error is also
returned if invalid flags were specified in the events member of any
of the WSAPOLLFD structures pointed to by the fdarray parameter when
requesting socket status. This error is also returned if none of the
sockets specified in the fd member of any of the WSAPOLLFD structures
pointed to by the fdarray parameter were valid.

What's the value of "timeout" for apr_poll() in send_brigade_blocking()?
Could you debug apr_poll() and see if something like the above happens
to the parameters passed to poll()?

Regards,
Yann.

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/14/2018 - 06:52

Hi Yann,
Please don't get confuse with Cygwin(Windows). I am running apache on
NonStop(Tandem). I have put breakpoint at apr_poll() but it is not going
there.

Few Events of writing :
1) apache writes 779 bytes to tomcat from apr_socket_sendv() function and
writev functions returns success.
2) apache writes 24604 bytes to tomcat from apr_socket_sendv() function but
writev function returns -1 with EWOULDBLOCK and it has timeout=0(because
set in writev_nonblocking())
3) apache again tries to write 32796 to tomcat from apr_socket_sendv()
function but writev function returns -1 with EWOULDBLOCK and it has
timeout=0(because set in writev_nonblocking()).
4) apache again tries to write 40988 to tomcat from apr_socket_sendv()
function but writev function returns -1 with EWOULDBLOCK and it has
timeout=0(because set in writev_nonblocking()).
5) apache again tries to write 49180 to tomcat from apr_socket_sendv()
function but writev function returns -1 with EWOULDBLOCK and it has
timeout=0(because set in writev_nonblocking()).
6) After this I am getting 4022 error.

What is happening here that after 24604 bytes in step 2, apache is 8192
bytes in buffer till step 5 and then giving 4022 error.
All timeouts has default value.

I am wondering that why is it not going into select() if writev is
returning -1 with EWOULDBLOCK.
Why is it failing after 49180 bytes. Does max size for this buffer is 49K??

I am able to upload same file if I am using blocking call.
Please let me know if you need more information.

Thanks
Hemant

Re: Re: Non Blocking write in apache

By Yann Ylavic at 09/16/2018 - 06:18

Hi Hermant,

On Fri, Sep 14, 2018 at 1:53 PM Hemant Chaudhary
<hemantdude. ... at gmail dot com> wrote:
Well, isn't the error_log attached to the previous message relevant
(be it cygwin or not)?
Btw, LogLevel trace6 would help here.

OK, all those EWOULDBLOCK make the core output filter to bufferize the
"pending" data for the next time it's called.

Here the 64K bufferized limit/threshold is reached
(non_file_bytes_in_brigade >= THRESHOLD_MAX_BUFFER in
ap_core_output_filter()), so the core output filter can't continue
nonblocking anymore and enters send_brigade_blocking(), which does
send_brigade_nonblocking() + apr_poll() until the 64K buffer is sent
completely.

However it seems that on your system the first call to
send_brigade_nonblocking() fails (since apr_poll() is never called),
possibly a limit on apr_socket_sendv() (i.e. writev() syscall), either
more than 64K total bytes or nvec > 4?

I think you need to debug/gdb in send_brigade_nonblocking() for this
last call and determine where and why send_brigade_nonblocking() =>
writev_nonblocking() => apr_socket_send() => writev(vec, nvec) fails.
Maybe a unit test (a simple standalone main() program) can determine
the limits for writev() on your system, so that the right values for
THRESHOLD_MAX_BUFFER and MAX_IOVEC_TO_WRITE (in
"server/core_filters.c") can be tuned for NonStop(Tandem) ...

Hope that helps,
Yann.

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/17/2018 - 05:31

Hi Yann,

Thanks for your explanation. I got the issue.
Actually NonStop has maximum limit of 52KB only for writev. Therefore I was
getting 4022 after 49KB. Because after 49KB, apache is adding 8K more bytes
which leads to 56KB. Therefore, writev was failing.

To solve this, I have changed THRESHOLD_MAX_BUFFER in core_filter.c to
50000. But again I am seeing that after 49KB, apache is adding 8192 Bytes
in it, which leads to failure of writev again.

Do I need to change in more places or the above fix should work.
Thanks
Hemant

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/17/2018 - 06:51

Hi,

in the apache error_log with trace6, it is trying to flush because it
reached thresold_max_buffer. But at the same time, it is adding data in
buffer which increases it size to to more than 52KB and fails on NonStop.
How should I stop apache so that it should go to select function till 52KB
success.

Attaching error_log with new change.

Thanks
Hemant

On Mon, Sep 17, 2018 at 4:01 PM Hemant Chaudhary <

Re: Re: Non Blocking write in apache

By Yann Ylavic at 09/17/2018 - 08:27

Hi,

On Mon, Sep 17, 2018 at 1:51 PM Hemant Chaudhary
<hemantdude. ... at gmail dot com> wrote:
Yes, THRESHOLD_MAX_BUFFER is the limit *above* which the buffer is
flushed, but not the maximum number of bytes flushed...

The simpler would be to take a 10K margin, like defining
THRESHOLD_MAX_BUFFER to 42K.
Otherwise, you need a patch like the one attached (untested)...

Regards,
Yann.

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/17/2018 - 10:46

If it will flush data then at what condition it will go to apr_poll
function ??

What I understand is that if buffer is filled with 52KB data and writev is
not ready then it should go into apr_poll().

Now the problem may exist with 42KB also.

I hope I am explained my issue.

Thanks
Hemant

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/18/2018 - 03:48

Hi Yann,

42Kb works for me. Thanks for your support.

Regards,
Hemant

On Mon, Sep 17, 2018 at 9:16 PM Hemant Chaudhary <

Re: Re: Non Blocking write in apache

By Hemant Chaudhary at 09/26/2018 - 05:22

Hi All,

I have a limitation for writev and readv to 52KB.
I was getting error for writev, therefore I changed THRESHOLD_MAX_BUFFER to
42KB. After that I was not getting any error.
Now in GET request of large file, I am getting the same 4022 error. This
may be because of readv function.

Is there any buffer/place where I can restrict readv to 42KB only,so that
it will not cross 52KB.

Thanks
Hemant

On Tue, Sep 18, 2018 at 2:18 PM Hemant Chaudhary <

Re: Re: Non Blocking write in apache

By Yann Ylavic at 09/17/2018 - 08:40

On Mon, Sep 17, 2018 at 3:27 PM Yann Ylavic <ylavic. ... at gmail dot com> wrote:
Sorry, it did not compile, so here is v2.
With this patch, you could compile httpd with "-D
MAX_BYTES_TO_WRITEV=52000" for instance.