DevHeads.net

Apache 2.4 (Apache Benchmarking) - Too many open files (24)

Hi,
Below is the issue encountered when i am trying to use apache 2.4 to
validate performance using the benchmarking tool(ab tool).

ab is invoked with 2000/1000 concurrent requests and subjected to 80000 cgi
requests. The module enabled is mod_cgid. Occurs with worker/event mpm
enabled

MPM configuration is:
<IfModule mpm_worker_module>
ServerLimit 80
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
MaxRequestWorkers 2000
MaxConnectionsPerChild 0
</IfModule>

================================================
2000 concurrent requests

Benchmarking x.x.x.x (be patient)
socket: Too many open files (24)

OR

1000 concurrent requests

Benchmarking x.x.x.x (be patient)
Completed 8000 requests
apr_socket_recv: Connection timed out (110)
Total of 8972 requests completed
================================================

This occurs on RHEL as well as on HP-UX.
Please help if anyone has encountered the same problem.

Thanks,
RS

Comments

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By =?UTF-8?Q?Rub=C... at 09/22/2016 - 10:10

25 * 2000 = 50.000 threads. May be you have to setup ulimit in your OS.
B.R.

2016-09-22 6:32 GMT+02:00 Rashmi Srinivasan < ... at gmail dot com>
:

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Yann Ylavic at 09/22/2016 - 01:40

Hi,

On Thu, Sep 22, 2016 at 6:32 AM, Rashmi Srinivasan
< ... at gmail dot com> wrote:
You should probably change the 'ulimit -n' on the host running ab,
something like:

$ ulimit -n 4096 # once for the whole terminal/shell session
$ ab ... # as many times as needed ;)

Regards,
Yann.

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/22/2016 - 01:57

Thank you so much, I will check.
Would need help on the timeout error as well:

Benchmarking x.x.x.x (be patient)
Completed 8000 requests
apr_socket_recv: Connection timed out (110)
Total of 8972 requests completed

Thanks,
RS

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Yann Ylavic at 09/22/2016 - 05:17

On Thu, Sep 22, 2016 at 8:57 AM, Rashmi Srinivasan
< ... at gmail dot com> wrote:
This can be related to some limits reached on the server side (cgi
connection/execution time, MaxRequestWorkers, ...).
Any error message in httpd?

Regards,
Yann.

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/22/2016 - 10:04

This is my mpm conf:

<IfModule mpm_event_module>
ServerLimit 80
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
MaxRequestWorkers 2000
MaxConnectionsPerChild 0
</IfModule>

The issue occurs

Access log has the status:

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/22/2016 - 10:20

Hi,

This is my mpm conf:

<IfModule mpm_event_module>
ServerLimit 80
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250
ThreadsPerChild 25
MaxRequestWorkers 2000
MaxConnectionsPerChild 0
</IfModule>

The issue occurs even upon increasing the serverlimit and MaxRequestWorkers.

Access log has the following status:
[22/Sep/2016:20:17:21 +0530] "GET /cgi-bin/script.cgi HTTP/1.0" 200 16

Please check and let me know.

Thanks ,
RS

On Thu, Sep 22, 2016 at 8:34 PM, Rashmi Srinivasan <

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By =?UTF-8?Q?Rub=C... at 09/23/2016 - 05:34

Are you testing from de same host that runs httpd?
Could you paste paste the output of ulimit -a , ie:

[apache@----------- ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30504
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited

******* open files (-n) 1024*

pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

In rhel default seems to be 1024.

2016-09-22 17:20 GMT+02:00 Rashmi Srinivasan < ... at gmail dot com

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Yann Ylavic at 09/22/2016 - 11:24

On Thu, Sep 22, 2016 at 5:20 PM, Rashmi Srinivasan
< ... at gmail dot com> wrote:
What's in the *error* log (path from the ErrorLog directive)?
What about resources (CPU, load average, ...) on the httpd machine
during the stress?

Regards,
Yann.

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/22/2016 - 14:44

Hi,
Thanks for checking this,

I wanted to check the below:'
Stress testing of apache 2.2 works very well with the default mpm
worker onfiguration
However, apache 2.4 doesnt work at all with the default
configuration, gives the error:
2000 concurrent requests
Benchmarking x.x.x.x (be patient)
socket: Too many open files (24)

After changing the configuration, as described in the previous email, it
starts processing them:
Lots of timeouts are occurring.
Error in the error_lo is:
"AH00046: child process still did not exit,
sending a SIGKILL"
(2)No such file or directory: [client x.x.x.c]
AH01256: connect #7 to cgi daemon failed, sleeping before retry
11)Resource temporarily unavailable: AH01252:
couldn't create child process: 11:
I also noticed that the CPU utilzation reached 99.9%
Which is when apache terminates

What changes are to be done in this case.
Please advise

Thanks,
RS

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/23/2016 - 03:13

Is there a minimum requirement for apache 2.4 to run these tests?
I have tried on different machines with different configurations with the
same outcome.
Please advise.

Thanks,
RS

On Fri, Sep 23, 2016 at 1:14 AM, Rashmi Srinivasan <

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/23/2016 - 04:55

Hi,
Do you feel this is the issue with some configuration or would this be an
issue with apache itself.
Has anybody encountered a performance issue of this kind.
Please comment

Thanks,
RS

On Fri, Sep 23, 2016 at 1:43 PM, Rashmi Srinivasan <

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By =?UTF-8?Q?Rub=C... at 09/23/2016 - 09:55

I think your problem is not Apache, but your OS.
I think Apache 2.2 and 4.4 uses diferent mpm mode (my memory is poor).
I think you OS is limiting the number of Opens Files that can Manage.
(safety reasons)
Please, can you print you ulimit output, as I advise on a previous mail?

Thanks.

2016-09-23 11:55 GMT+02:00 Rashmi Srinivasan < ... at gmail dot com

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Stormy at 09/23/2016 - 14:32

At 12:18 AM 9/24/2016 +0530, Rashmi Srinivasan wrote:
Pure guesswork, but this looks low -- on a 32bit system, try 2^15; on
64bits, 2^16 ... or more ...

P.

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/23/2016 - 14:52

Thanks
I will check this.

I have another concern related to performance.
on RHEL machine, running ab for 20000 requests with 100 concurrent requests
takes almost 46 seconds, while this was taking just 2 seconds in 2.2
Please can you tell if this is the case and how can this be improved?

Thanks

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By Rashmi Srinivasan at 09/23/2016 - 13:48

Hi,

This is the output of ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15944
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Thanks,

On Fri, Sep 23, 2016 at 8:25 PM, Rubén Toribio Aldeguer < ... at riu dot com>
wrote:

Re: Apache 2.4 (Apache Benchmarking) - Too many op

By =?UTF-8?Q?Rub=C... at 09/26/2016 - 02:53

I think that is your problem: OS user for httpd or ab only can open
(sumuntaneousy) 1024 files.
open files (-n) 1024

Try to set to 10000 for example. Read "man ulimit" to set how to do it. I
dont remenber sinstax.

2016-09-23 20:48 GMT+02:00 Rashmi Srinivasan < ... at gmail dot com