DevHeads.net

httpd processes are too many created from Apache 2.2.31 on Linux Server.

Hello.

I'm using Apache 2.2.31 on Linux Server.

I've faced an issue that Memory Utilization is over 80% on Linux Server.

When I checked /var/log/messages on Linux Server, I found out that httpd
processes were too many created on Linux server.

Jan 6 18:39:00 <server name> kernel: [621588.393289] Out of memory: Kill
process 4577 (httpd) score 16 or sacrifice child
Jan 6 18:39:00 <server name> kernel: [621588.402442] Killed process 4577
(httpd) total-vm:709776kB, anon-rss:124196kB, file-rss:0kB

When I checked Apache error logs, I found out below errors from Apache
error logs at the same timeframes.

[Sat Jan 06 18:29:17 2018] [error] [client IP] PHP Warning: ocilogon():
OCIEnvNlsCreate() failed. There is something wrong with your system -
please check that LD_LIBRARY_PATH includes the directory with Oracle
Instant Client libraries in <PHP File Path> on line 42, referer: <Site URL>
[Sat Jan 06 18:29:17 2018] [error] [client IP] PHP Warning: ocilogon():
Error while trying to retrieve text for error ORA-01804\n in <PHP File
Path> on line 42, referer: <Site URL>
[Sat Jan 06 18:29:17 2018] [error] [client IP] PHP Warning: ocilogon():
OCIEnvNlsCreate() failed. There is something wrong with your system -
please check that LD_LIBRARY_PATH includes the directory with Oracle
Instant Client libraries in <PHP File Path> on line 42, referer: <Site URL>
[Sat Jan 06 18:29:17 2018] [error] [client IP] PHP Warning: ocilogon():
Error while trying to retrieve text for error ORA-01804\n in <PHP File
Path> on line 42, referer: <Site URL>
[Sat Jan 06 18:29:17 2018] [error] [client IP] PHP Warning: ociparse()
expects parameter 1 to be resource, boolean given in <PHP File Path> on
line 103, referer: <Site URL>

Are these errors related to be created too many httpd processes on Linux
Server?

Can please give me any advice for this?

Here is Apache information.

Server version: Apache/2.2.31 (Unix)
Server built: Jul 19 2016 00:11:31
Server's Module Magic Number: 20051115:40
Server loaded: APR 1.5.1, APR-Util 1.4.1
Compiled using: APR 1.5.1, APR-Util 1.4.1
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Thank you in advance.

Thanks,

Junyoung

Comments

Re: httpd processes are too many created from Apac

By Eric Covener at 01/11/2018 - 10:24

On Wed, Jan 10, 2018 at 11:14 PM, 갈준영 < ... at mz dot co.kr> wrote:
How many were created? What makes you say it was too many vs. an
expected number that just used too much memory?

You're using prefork, which is going to create lots of processes and
use the most memory.

Re: httpd processes are too many created from Apac

By =?UTF-8?B?6rCI7... at 01/11/2018 - 21:47

Hello. Eric.

Thank you for your reply.

How many were created?
-> I counted httpd processes in /var/log/messages and its count was about
520 at issue timeframes.

What makes you say it was too many vs. an expected number that just used
too much memory?
-> It was too many processes and each process used about 1~2% memory at
issue timeframes.

I thought Apache configuration would need to be tuned proper. How can I
tune Apache configuration proper?

If I make change from prefork to worker, may there be any issue on Apache
after this change? and what do I beware of this change?

Can you give me any advice for these?

</var/log/messages>
Jan 6 18:38:44 <server name> kernel: [621540.383077] [ pid ] uid tgid
total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 6 18:38:44 <server name> kernel: [621540.393878] [ 1614] 0 1614
2841 603 12 3 0 -1000 udevd
Jan 6 18:38:44 <server name> kernel: [621540.404792] [ 1722] 0 1722
2833 477 11 3 0 -1000 udevd
Jan 6 18:38:44 <server name> kernel: [621540.415425] [ 1723] 0 1723
2833 477 11 3 0 -1000 udevd
Jan 6 18:38:44 <server name> kernel: [621540.426185] [ 2191] 0 2191
2340 441 9 3 0 0 dhclient
Jan 6 18:38:44 <server name> kernel: [621540.437389] [ 2243] 0 2243
28018 133 25 3 0 -1000 auditd
Jan 6 18:38:44 <server name> kernel: [621540.448060] [ 2264] 0 2264
60908 1038 21 4 0 0 rsyslogd
Jan 6 18:38:44 <server name> kernel: [621540.458624] [ 2278] 0 2278
3428 363 11 3 0 0 irqbalance
Jan 6 18:38:44 <server name> kernel: [621540.469936] [ 2287] 0 2287
1095 145 8 3 0 0 rngd
Jan 6 18:38:44 <server name> kernel: [621540.480736] [ 2305] 32 2305
8823 427 21 3 0 0 rpcbind
Jan 6 18:38:44 <server name> kernel: [621540.491693] [ 2326] 29 2326
9965 626 24 3 0 0 rpc.statd
Jan 6 18:38:44 <server name> kernel: [621540.502835] [ 2357] 81 2357
5448 357 15 3 0 0 dbus-daemon
Jan 6 18:38:44 <server name> kernel: [621540.514226] [ 2486] 0 2486
19460 353 40 4 0 -1000 sshd
Jan 6 18:38:44 <server name> kernel: [621540.525033] [ 2508] 38 2508
7322 207 20 3 0 0 ntpd
Jan 6 18:38:44 <server name> kernel: [621540.535788] [ 2521] 0 2521
12028 130 26 3 0 0 vsftpd
Jan 6 18:38:44 <server name> kernel: [621540.546415] [ 2541] 0 2541
21728 769 43 4 0 0 sendmail
Jan 6 18:38:44 <server name> kernel: [621540.556995] [ 2550] 51 2550
19593 720 40 3 0 0 sendmail
Jan 6 18:38:44 <server name> kernel: [621540.567918] [ 2562] 0 2562
3700 335 10 3 0 0 wconmonitor
Jan 6 18:38:44 <server name> kernel: [621540.579131] [ 2617] 0 2617
29879 538 16 3 0 0 crond
Jan 6 18:38:44 <server name> kernel: [621540.589888] [ 2633] 0 2633
1078 277 8 3 0 0 atopacctd
Jan 6 18:38:44 <server name> kernel: [621540.600984] [ 2648] 0 2648
4267 39 13 3 0 0 atd
Jan 6 18:38:44 <server name> kernel: [621540.611585] [ 2754] 498 2754
20543 620 41 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.623080] [ 2756] 498 2756
20543 678 41 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.634615] [ 2757] 498 2757
20543 506 43 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.645846] [ 2758] 498 2758
20543 506 43 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.656888] [ 2759] 498 2759
20543 506 43 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.668342] [ 2760] 498 2760
20576 518 43 3 0 0 zabbix_agentd
Jan 6 18:38:44 <server name> kernel: [621540.679932] [ 2786] 0 2786
128804 28352 217 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.690757] [ 2798] 0 2798
1615 372 9 3 0 0 agetty
Jan 6 18:38:44 <server name> kernel: [621540.701739] [ 2799] 0 2799
1078 376 8 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.712903] [ 2802] 0 2802
1078 376 8 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.723993] [ 2806] 0 2806
1078 367 8 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.735089] [ 2808] 0 2808
1078 359 8 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.745947] [ 2810] 0 2810
1078 374 8 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.756583] [ 2812] 0 2812
1078 349 7 3 0 0 mingetty
Jan 6 18:38:44 <server name> kernel: [621540.767447] [17525] 48 17525
128803 27331 207 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.778347] [17526] 48 17526
128803 27331 207 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.789195] [17527] 48 17527
128803 27331 207 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.800018] [28635] 48 28635
161601 29887 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.810782] [28727] 0 28727
23675 4135 23 3 0 0 atop
Jan 6 18:38:44 <server name> kernel: [621540.821510] [31957] 48 31957
161427 29861 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.832301] [32136] 48 32136
162129 30496 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.842856] [32351] 48 32351
161820 30191 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.853243] [32492] 48 32492
161612 29890 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.863194] [32540] 48 32540
161607 29887 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.873523] [32696] 48 32696
161605 29901 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.884215] [32726] 48 32726
162002 30282 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.894821] [ 435] 48 435
161677 29985 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.905643] [ 505] 48 505
161832 30173 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.916016] [ 516] 48 516
162188 30518 246 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.926681] [ 549] 48 549
161675 29954 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.937464] [ 618] 48 618
161867 30238 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.947836] [ 707] 48 707
161589 29976 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.958002] [ 760] 48 760
161597 29905 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.968780] [ 770] 48 770
161619 29899 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.979526] [ 776] 48 776
161708 30095 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621540.990366] [ 789] 48 789
161839 30118 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.001229] [ 1056] 48 1056
161607 29894 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.012371] [ 1071] 48 1071
161899 30270 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.023406] [ 1191] 48 1191
162258 30540 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.034299] [ 1335] 48 1335
161667 29946 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.044761] [ 1494] 48 1494
161601 29904 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.090253] [ 4263] 48 4263
161550 29746 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.100782] [ 4267] 48 4267
161371 29642 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.111509] [ 4273] 48 4273
161335 29615 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.122256] [ 4276] 48 4276
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.133027] [ 4279] 48 4279
161397 29657 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.143564] [ 4282] 48 4282
161419 29642 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.154038] [ 4285] 48 4285
161445 29624 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.164362] [ 4288] 48 4288
161221 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.175126] [ 4289] 48 4289
161376 29631 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.185916] [ 4294] 48 4294
161396 29677 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.196684] [ 4300] 48 4300
161412 29689 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.207430] [ 4303] 48 4303
161419 29646 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.218248] [ 4306] 48 4306
161483 29703 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.229085] [ 4309] 48 4309
161329 29614 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.239750] [ 4312] 48 4312
161353 29623 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.250430] [ 4315] 48 4315
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.260565] [ 4318] 48 4318
161385 29564 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.271319] [ 4319] 48 4319
161386 29565 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.282094] [ 4322] 48 4322
161386 29614 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.292890] [ 4323] 48 4323
160770 25725 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.303624] [ 4325] 48 4325
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.314481] [ 4328] 48 4328
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.325490] [ 4331] 48 4331
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.336011] [ 4336] 48 4336
161353 29623 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.346820] [ 4345] 48 4345
161360 29639 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.357155] [ 4348] 48 4348
161289 29565 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.367769] [ 4351] 48 4351
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.378520] [ 4354] 48 4354
161353 29628 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.389290] [ 4374] 48 4374
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.400039] [ 4377] 48 4377
161353 29628 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.410831] [ 4380] 48 4380
161362 29639 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.421616] [ 4383] 48 4383
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.432120] [ 4389] 48 4389
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.442784] [ 4392] 48 4392
161222 29555 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.453382] [ 4398] 48 4398
177469 29487 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.463626] [ 4401] 48 4401
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.474448] [ 4404] 48 4404
161402 29576 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.485198] [ 4407] 48 4407
161334 29629 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.496001] [ 4413] 48 4413
161431 29701 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.506753] [ 4422] 48 4422
161359 29650 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.517600] [ 4428] 48 4428
177332 29851 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.528310] [ 4431] 48 4431
161386 29564 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.538791] [ 4434] 48 4434
161314 29605 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.549524] [ 4438] 48 4438
161060 29558 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.559663] [ 4441] 48 4441
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.570450] [ 4444] 48 4444
161330 29607 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.581236] [ 4447] 48 4447
161406 29693 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.591973] [ 4452] 48 4452
161353 29623 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.602741] [ 4455] 48 4455
161327 29603 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.613585] [ 4458] 48 4458
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.624334] [ 4461] 48 4461
161355 29630 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.634811] [ 4464] 48 4464
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.645623] [ 4467] 48 4467
161224 29503 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.655954] [ 4470] 48 4470
161349 29627 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.666484] [ 4473] 48 4473
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.677318] [ 4476] 48 4476
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.688074] [ 4479] 48 4479
177444 31052 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.698849] [ 4482] 48 4482
161365 29641 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.709633] [ 4485] 48 4485
161334 29604 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.720397] [ 4488] 48 4488
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.730978] [ 4491] 48 4491
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.741701] [ 4494] 48 4494
176028 29233 228 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.752335] [ 4497] 48 4497
161353 29623 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.763100] [ 4498] 48 4498
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.773886] [ 4499] 48 4499
161320 29651 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.784702] [ 4503] 48 4503
177332 29851 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.795535] [ 4504] 48 4504
161410 29636 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.806372] [ 4515] 48 4515
161419 29646 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.817215] [ 4518] 48 4518
161222 29561 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.827915] [ 4522] 48 4522
161222 29525 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.838243] [ 4528] 48 4528
161361 29660 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.848534] [ 4529] 48 4529
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.858556] [ 4530] 48 4530
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.869178] [ 4535] 48 4535
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.879932] [ 4536] 48 4536
161224 29503 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.890716] [ 4545] 48 4545
161295 29487 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.901520] [ 4548] 48 4548
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.912053] [ 4551] 48 4551
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.922776] [ 4552] 48 4552
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.933328] [ 4553] 48 4553
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.944076] [ 4554] 48 4554
161352 29923 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.954530] [ 4563] 48 4563
161222 29523 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.964901] [ 4566] 48 4566
161222 29574 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.975709] [ 4569] 48 4569
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.986499] [ 4576] 48 4576
161309 29580 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621541.997327] [ 4577] 48 4577
177444 31049 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.008084] [ 4582] 48 4582
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.018822] [ 4590] 48 4590
161817 29998 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.029516] [ 4591] 48 4591
161331 29612 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.039761] [ 4596] 48 4596
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.085486] [ 4597] 48 4597
161353 29625 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.096625] [ 4603] 48 4603
161289 29615 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.107685] [ 4606] 48 4606
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.118524] [ 4614] 48 4614
161060 29433 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.129195] [ 4615] 48 4615
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.139774] [ 4620] 48 4620
161526 29796 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.150528] [ 4623] 48 4623
161329 29613 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.160696] [ 4629] 48 4629
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.171430] [ 4632] 48 4632
161224 29491 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.182212] [ 4635] 48 4635
161332 29602 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.192973] [ 4636] 48 4636
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.203689] [ 4638] 48 4638
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.214576] [ 4639] 48 4639
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.225337] [ 4640] 48 4640
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.235788] [ 4641] 48 4641
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.246567] [ 4654] 48 4654
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.256810] [ 4659] 48 4659
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.267408] [ 4662] 48 4662
129572 27924 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.278308] [ 4665] 48 4665
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.289029] [ 4668] 48 4668
175855 29969 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.299751] [ 4671] 48 4671
161347 29618 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.310549] [ 4677] 48 4677
161338 29626 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.321310] [ 4680] 48 4680
161060 29703 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.331867] [ 4683] 48 4683
161224 29503 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.342021] [ 4686] 48 4686
161329 29610 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.357248] [ 4689] 48 4689
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.367677] [ 4692] 48 4692
161349 29620 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.378440] [ 4695] 48 4695
161332 29608 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.389332] [ 4698] 48 4698
129570 27992 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.400211] [ 4707] 48 4707
161224 29507 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.411094] [ 4716] 48 4716
161298 29690 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.422004] [ 4719] 48 4719
161329 29605 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.432676] [ 4725] 48 4725
161222 29498 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.443498] [ 4728] 48 4728
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.454114] [ 4731] 48 4731
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.464831] [ 4734] 48 4734
161222 29499 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.475633] [ 4737] 48 4737
175508 28954 227 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.486452] [ 4740] 48 4740
161224 29697 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.497231] [ 4743] 48 4743
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.507987] [ 4746] 48 4746
161288 29558 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.518798] [ 4749] 48 4749
161222 29500 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.529441] [ 4752] 48 4752
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.540054] [ 4755] 48 4755
161420 29596 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.550680] [ 4761] 48 4761
161346 29632 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.561431] [ 4764] 48 4764
177444 31126 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.572196] [ 4767] 48 4767
161324 29602 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.582956] [ 4770] 48 4770
161060 29356 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.593727] [ 4771] 48 4771
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.604540] [ 4776] 48 4776
161061 29353 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.615317] [ 4779] 48 4779
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.626071] [ 4782] 48 4782
129570 27963 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.636580] [ 4785] 48 4785
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.647286] [ 4788] 48 4788
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.657552] [ 4791] 48 4791
161289 29571 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.668251] [ 4792] 48 4792
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.678960] [ 4793] 48 4793
177137 28952 225 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.689752] [ 4800] 48 4800
161353 29577 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.700501] [ 4803] 48 4803
177712 31470 246 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.711210] [ 4804] 48 4804
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.722005] [ 4805] 48 4805
161419 29642 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.732569] [ 4808] 48 4808
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.743272] [ 4813] 48 4813
161221 29487 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.753809] [ 4816] 48 4816
161364 29634 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.764091] [ 4817] 48 4817
175863 29709 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.774858] [ 4818] 48 4818
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.785632] [ 4823] 48 4823
161340 29612 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.796470] [ 4830] 48 4830
161463 29733 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.807271] [ 4831] 48 4831
161233 29499 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.818052] [ 4834] 48 4834
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.828523] [ 4837] 48 4837
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.839054] [ 4838] 48 4838
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.849788] [ 4848] 48 4848
161222 29500 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.859783] [ 4851] 48 4851
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.870473] [ 4854] 48 4854
161062 29575 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.881251] [ 4863] 48 4863
161191 29585 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.891986] [ 4866] 48 4866
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.902777] [ 4870] 48 4870
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.913287] [ 4875] 48 4875
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.924092] [ 4876] 48 4876
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.934624] [ 4887] 48 4887
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.945383] [ 4890] 48 4890
161320 29559 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.955765] [ 4891] 48 4891
161360 29602 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.966280] [ 4896] 48 4896
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.977081] [ 4897] 48 4897
161471 29901 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.987848] [ 4902] 48 4902
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621542.998659] [ 4905] 48 4905
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.009481] [ 4908] 48 4908
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.020278] [ 4909] 48 4909
177444 29538 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.030861] [ 4911] 48 4911
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.041494] [ 4912] 48 4912
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.087228] [ 4917] 48 4917
161224 29541 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.098062] [ 4918] 48 4918
161344 29545 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.108863] [ 4929] 48 4929
161376 29631 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.119606] [ 4932] 48 4932
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.130273] [ 4938] 48 4938
161380 29555 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.140899] [ 4941] 48 4941
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.151557] [ 4944] 48 4944
161294 29485 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.162335] [ 4947] 48 4947
161222 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.173105] [ 4950] 48 4950
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.183890] [ 4953] 48 4953
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.194661] [ 4959] 48 4959
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.205444] [ 4962] 48 4962
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.216238] [ 4965] 48 4965
179412 30217 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.226961] [ 4968] 48 4968
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.237465] [ 4974] 48 4974
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.248187] [ 4977] 48 4977
161222 29497 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.258327] [ 4980] 48 4980
161060 29355 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.269135] [ 4983] 48 4983
161391 29660 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.279925] [ 4986] 48 4986
161362 29632 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.290767] [ 4989] 48 4989
161306 29560 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.301550] [ 4992] 48 4992
161411 29664 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.312329] [ 4998] 48 4998
161062 29665 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.323072] [ 5004] 48 5004
161060 29355 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.333586] [ 5007] 48 5007
161331 29601 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.344291] [ 5010] 48 5010
161087 29454 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.354803] [ 5013] 48 5013
161221 29484 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.365365] [ 5016] 48 5016
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.376186] [ 5019] 48 5019
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.386977] [ 5022] 48 5022
161386 29559 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.397756] [ 5025] 48 5025
161060 29731 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.408543] [ 5028] 48 5028
161222 29528 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.419324] [ 5031] 48 5031
161060 29684 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.430013] [ 5034] 48 5034
161328 29566 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.440694] [ 5037] 48 5037
161286 29565 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.451318] [ 5040] 48 5040
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.462117] [ 5043] 48 5043
161222 29502 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.472888] [ 5046] 48 5046
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.483744] [ 5049] 48 5049
161333 29611 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.494521] [ 5052] 48 5052
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.505326] [ 5055] 48 5055
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.516128] [ 5058] 48 5058
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.526887] [ 5064] 48 5064
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.537398] [ 5067] 48 5067
129570 28267 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.548179] [ 5068] 48 5068
161221 29484 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.558347] [ 5069] 48 5069
161329 29617 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.569087] [ 5070] 48 5070
161359 29630 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.579857] [ 5071] 48 5071
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.590719] [ 5072] 48 5072
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.602799] [ 5088] 48 5088
177588 30315 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.614750] [ 5091] 48 5091
161224 29491 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.627213] [ 5094] 48 5094
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.639951] [ 5097] 48 5097
161224 29493 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.651901] [ 5098] 48 5098
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.664065] [ 5099] 48 5099
129631 28085 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.676319] [ 5102] 48 5102
161223 29504 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.688560] [ 5107] 48 5107
161295 29487 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.700552] [ 5110] 48 5110
161221 29502 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.712660] [ 5115] 48 5115
161061 29552 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.724931] [ 5116] 48 5116
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.737067] [ 5121] 48 5121
161151 29368 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.748309] [ 5124] 48 5124
161075 29557 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.758539] [ 5127] 48 5127
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.769179] [ 5128] 48 5128
161060 29571 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.780048] [ 5133] 48 5133
175251 29553 229 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.790784] [ 5134] 48 5134
161224 29503 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.801434] [ 5137] 48 5137
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.812174] [ 5142] 48 5142
129570 28292 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.822741] [ 5145] 48 5145
161060 29771 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.833289] [ 5146] 48 5146
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.843988] [ 5147] 48 5147
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.854470] [ 5148] 48 5148
161089 29454 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.864534] [ 5151] 48 5151
161224 29503 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.875254] [ 5156] 48 5156
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.886025] [ 5161] 48 5161
129572 28076 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.896797] [ 5162] 48 5162
161321 29597 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.907561] [ 5167] 48 5167
161224 29491 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.918151] [ 5168] 48 5168
129572 27924 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.928853] [ 5169] 48 5169
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.939493] [ 5179] 48 5179
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.950479] [ 5182] 48 5182
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.961555] [ 5187] 48 5187
161338 29616 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.972556] [ 5188] 48 5188
129570 28364 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.983757] [ 5193] 48 5193
161222 29547 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621543.993672] [ 5196] 48 5196
161329 29605 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.003463] [ 5197] 48 5197
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.014253] [ 5202] 48 5202
161239 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.024692] [ 5205] 48 5205
161108 29399 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.035147] [ 5208] 48 5208
161060 29651 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.045835] [ 5211] 48 5211
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.056111] [ 5214] 48 5214
161321 29577 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.102294] [ 5215] 48 5215
161532 30191 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.113406] [ 5216] 48 5216
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.127806] [ 5217] 48 5217
129570 28102 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.140984] [ 5220] 48 5220
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.153140] [ 5232] 48 5232
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.167621] [ 5238] 48 5238
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.179822] [ 5240] 48 5240
161299 29490 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.193194] [ 5241] 48 5241
161222 29499 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.207648] [ 5244] 48 5244
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.218804] [ 5247] 48 5247
161329 29629 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.232718] [ 5248] 48 5248
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.246928] [ 5257] 48 5257
161386 29612 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.256995] [ 5260] 48 5260
161289 29478 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.267365] [ 5263] 48 5263
160280 28594 238 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.277366] [ 5272] 48 5272
161321 29596 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.288330] [ 5273] 48 5273
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.298253] [ 5280] 48 5280
161060 29479 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.308743] [ 5283] 48 5283
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.318979] [ 5284] 48 5284
161419 29646 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.328837] [ 5289] 48 5289
177332 30053 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.338930] [ 5292] 48 5292
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.349449] [ 5295] 48 5295
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.361897] [ 5298] 48 5298
161060 29418 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.378639] [ 5301] 48 5301
161062 29470 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.389092] [ 5304] 48 5304
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.399667] [ 5307] 48 5307
177444 29543 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.410208] [ 5310] 48 5310
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.421014] [ 5316] 48 5316
161060 29505 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.431582] [ 5319] 48 5319
160430 28910 240 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.442282] [ 5322] 48 5322
175252 29853 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.453097] [ 5325] 48 5325
161247 29525 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.463389] [ 5326] 48 5326
161290 29480 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.474353] [ 5327] 48 5327
161060 29859 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.485262] [ 5332] 48 5332
161359 29649 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.496086] [ 5335] 48 5335
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.506867] [ 5340] 48 5340
161388 29563 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.519186] [ 5343] 48 5343
161388 29562 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.529646] [ 5346] 48 5346
161060 29572 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.540270] [ 5349] 48 5349
161386 29559 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.552211] [ 5358] 48 5358
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.562610] [ 5361] 48 5361
161387 29611 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.573398] [ 5364] 48 5364
161290 29480 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.584568] [ 5367] 48 5367
161419 29642 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.595567] [ 5370] 48 5370
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.606494] [ 5373] 48 5373
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.617714] [ 5379] 48 5379
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.628580] [ 5382] 48 5382
161385 29559 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.639264] [ 5385] 48 5385
161290 29518 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.650303] [ 5388] 48 5388
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.661147] [ 5394] 48 5394
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.671906] [ 5397] 48 5397
161223 29487 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.683048] [ 5400] 48 5400
161294 29516 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.693603] [ 5403] 48 5403
161290 29480 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.704084] [ 5406] 48 5406
160434 28775 240 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.714572] [ 5412] 48 5412
161452 29718 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.725039] [ 5415] 48 5415
129570 28042 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.735487] [ 5421] 48 5421
161332 29517 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.746272] [ 5424] 48 5424
161060 29440 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.756612] [ 5427] 48 5427
161455 29716 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.767207] [ 5430] 48 5430
177334 29852 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.777975] [ 5433] 48 5433
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.788723] [ 5439] 48 5439
161060 29442 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.799325] [ 5442] 48 5442
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.810116] [ 5445] 48 5445
161223 29487 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.820993] [ 5448] 48 5448
129572 27924 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.831720] [ 5452] 48 5452
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.842684] [ 5453] 48 5453
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.853277] [ 5458] 48 5458
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.863254] [ 5466] 48 5466
176635 28912 238 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.873388] [ 5469] 48 5469
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.884209] [ 5472] 48 5472
161222 29501 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.894947] [ 5473] 48 5473
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.905716] [ 5481] 48 5481
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.916175] [ 5482] 48 5482
178045 30449 231 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.926941] [ 5483] 48 5483
161328 29566 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.937481] [ 5490] 48 5490
161338 29500 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.948264] [ 5493] 48 5493
161224 29491 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.958476] [ 5496] 48 5496
176030 30081 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.969213] [ 5499] 48 5499
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.980016] [ 5502] 48 5502
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621544.990755] [ 5503] 48 5503
161296 29486 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.001593] [ 5504] 48 5504
161297 29489 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.012468] [ 5505] 48 5505
161074 29395 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.023231] [ 5514] 48 5514
176825 30572 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.033806] [ 5517] 48 5517
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.044731] [ 5520] 48 5520
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.055119] [ 5521] 48 5521
161295 29487 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.065580] [ 5522] 48 5522
161298 29489 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.076374] [ 5523] 48 5523
176888 29354 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.087350] [ 5528] 48 5528
161060 29568 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.133321] [ 5529] 48 5529
161295 29486 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.144373] [ 5532] 48 5532
161151 29363 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.154764] [ 5541] 48 5541
161300 29596 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.165163] [ 5547] 48 5547
176179 29339 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.175902] [ 5550] 48 5550
161408 29584 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.186696] [ 5553] 48 5553
176132 30023 245 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.197491] [ 5556] 48 5556
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.208297] [ 5557] 48 5557
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.219078] [ 5558] 48 5558
161386 29560 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.229780] [ 5562] 48 5562
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.240478] [ 5569] 48 5569
161060 29502 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.251154] [ 5572] 48 5572
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.261983] [ 5573] 48 5573
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.272848] [ 5583] 48 5583
161306 29588 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.283614] [ 5584] 48 5584
161306 29558 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.294444] [ 5589] 48 5589
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.305239] [ 5590] 48 5590
161060 29366 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.315989] [ 5595] 48 5595
161060 29360 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.326721] [ 5598] 48 5598
161222 29489 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.337250] [ 5599] 48 5599
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.347981] [ 5600] 48 5600
161296 29557 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.358125] [ 5607] 48 5607
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.368868] [ 5610] 48 5610
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.379878] [ 5611] 48 5611
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.390301] [ 5616] 48 5616
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.401216] [ 5617] 48 5617
175508 28954 227 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.412020] [ 5618] 48 5618
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.422795] [ 5621] 48 5621
176971 30604 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.433363] [ 5624] 48 5624
161060 29429 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.443452] [ 5625] 48 5625
161223 29487 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.454017] [ 5634] 48 5634
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.464428] [ 5637] 48 5637
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.475146] [ 5643] 48 5643
161292 29482 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.485970] [ 5644] 48 5644
161062 29354 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.496894] [ 5649] 48 5649
161328 29568 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.507605] [ 5652] 48 5652
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.518467] [ 5653] 48 5653
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.529199] [ 5656] 48 5656
161328 29580 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.539783] [ 5657] 48 5657
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.550488] [ 5658] 48 5658
161460 29671 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.560659] [ 5659] 48 5659
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.571398] [ 5660] 48 5660
175770 30342 231 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.582206] [ 5667] 48 5667
176756 28909 239 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.593012] [ 5668] 48 5668
161060 29415 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.603758] [ 5669] 48 5669
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.614528] [ 5674] 48 5674
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.625270] [ 5684] 48 5684
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.635771] [ 5690] 48 5690
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.646528] [ 5694] 48 5694
161060 29461 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.656806] [ 5697] 48 5697
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.667355] [ 5700] 48 5700
161221 29484 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.678111] [ 5703] 48 5703
129570 27922 211 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.688911] [ 5706] 48 5706
161221 29484 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.699551] [ 5709] 48 5709
176825 30573 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.710198] [ 5712] 48 5712
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.721152] [ 5714] 48 5714
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.731915] [ 5717] 48 5717
175850 29943 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.742780] [ 5722] 48 5722
160529 29276 228 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.753422] [ 5727] 48 5727
175522 29432 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.764137] [ 5730] 48 5730
175917 29641 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.774826] [ 5733] 48 5733
174871 29605 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.785634] [ 5735] 48 5735
145502 28035 212 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.796377] [ 5738] 48 5738
175721 29598 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.807107] [ 5741] 48 5741
175522 29357 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.817869] [ 5746] 48 5746
175472 29349 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.828555] [ 5749] 48 5749
145826 28043 213 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.839096] [ 5750] 48 5750
161360 29583 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.849865] [ 5751] 48 5751
175657 29348 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.859861] [ 5752] 48 5752
161295 29487 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.870515] [ 5768] 48 5768
175657 29327 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.881291] [ 5769] 48 5769
175522 29360 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.892028] [ 5770] 48 5770
175911 29181 229 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.902750] [ 5771] 48 5771
175657 29322 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.913542] [ 5772] 48 5772
145502 28269 212 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.924320] [ 5773] 48 5773
161295 29486 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.934802] [ 5774] 48 5774
175721 29410 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.945583] [ 5775] 48 5775
175657 29340 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.955878] [ 5796] 48 5796
175388 29323 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.966395] [ 5798] 48 5798
175472 29407 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.977224] [ 5799] 48 5799
175472 29363 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.987952] [ 5800] 48 5800
177588 30082 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621545.998729] [ 5801] 48 5801
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.009488] [ 5802] 48 5802
175472 29372 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.020211] [ 5803] 48 5803
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.030795] [ 5804] 48 5804
177238 30190 231 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.041515] [ 5805] 48 5805
175522 29628 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.052148] [ 5806] 48 5806
175786 29899 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.062856] [ 5807] 48 5807
175472 29367 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.073603] [ 5809] 48 5809
145762 28439 222 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.084382] [ 5814] 48 5814
175992 29610 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.130499] [ 5817] 48 5817
175472 29635 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.141160] [ 5822] 48 5822
175472 29382 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.151772] [ 5850] 48 5850
175786 29320 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.162549] [ 5851] 48 5851
175786 29644 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.173313] [ 5852] 48 5852
145631 28087 213 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.184052] [ 5853] 48 5853
174871 29184 229 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.194832] [ 5854] 48 5854
145566 28063 213 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.205644] [ 5855] 48 5855
176679 28956 227 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.216473] [ 5856] 48 5856
175590 29346 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.227188] [ 5857] 48 5857
179149 29994 230 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.237683] [ 5878] 48 5878
177003 28955 226 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.248413] [ 5880] 48 5880
175981 29334 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.258555] [ 5882] 48 5882
161060 29391 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.269313] [ 5887] 48 5887
175472 29372 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.280064] [ 5888] 48 5888
175472 29386 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.290945] [ 5889] 48 5889
175522 29356 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.301698] [ 5890] 48 5890
175836 28928 229 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.312715] [ 5891] 48 5891
175522 29326 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.323742] [ 5898] 48 5898
175590 29519 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.335186] [ 5908] 48 5908
175472 29774 244 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.346638] [ 5909] 48 5909
175472 29608 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.357692] [ 5910] 48 5910
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.368590] [ 5911] 48 5911
175472 29357 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.379277] [ 5912] 48 5912
175472 29358 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.392711] [ 5913] 48 5913
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.409658] [ 5914] 48 5914
175472 29332 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.420617] [ 5915] 48 5915
145502 27989 212 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.431406] [ 5916] 48 5916
145317 28077 213 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.442081] [ 5917] 48 5917
175472 29358 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.452837] [ 5940] 48 5940
128804 27332 206 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.463220] [ 5941] 48 5941
175472 29650 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.474373] [ 5942] 48 5942
175391 29317 229 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.485218] [ 5944] 48 5944
175522 29361 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.496045] [ 5946] 48 5946
175522 29407 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.506891] [ 5947] 48 5947
175472 29375 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.517863] [ 5949] 48 5949
161060 29352 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.528717] [ 5950] 48 5950
145367 27986 213 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.539483] [ 5955] 48 5955
175582 29658 243 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.550184] [ 6031] 48 6031
128848 27338 210 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.560395] [ 6070] 48 6070
161192 29548 241 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.571215] [ 6073] 48 6073
175434 28232 240 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.582140] [ 6081] 48 6081
175436 29372 242 4 0 0 httpd
Jan 6 18:38:44 <server name> kernel: [621546.592989] Out of memory: Kill
process 4803 (httpd) score 16 or sacrifice child
Jan 6 18:38:44 <server name> kernel: [621546.602544] Killed process 4803
(httpd) total-vm:710848kB, anon-rss:125016kB, file-rss:864kB
Jan 6 18:38:44 <server name> kernel: [621546.617821] httpd invoked
oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Jan 6 18:38:44 <server name> kernel: [621546.627838] httpd cpuset=/
mems_allowed=0
Jan 6 18:38:44 <server name> kernel: [621546.633552] CPU: 2 PID: 4600
Comm: httpd Tainted: G E 4.4.16-27.56.amzn1.x86_64 #1
Jan 6 18:38:44 <server name> kernel: [621546.637539] Hardware name: Xen
HVM domU, BIOS 4.2.amazon 08/24/2006
Jan 6 18:38:44 <server name> kernel: [621546.637539] 0000000000000000
ffff8800eb817a70 ffffffff812c8f7f ffff8800eb817c58
Jan 6 18:38:44 <server name> kernel: [621546.637539] 0000000000000000
ffff8800eb817b00 ffffffff811ce19d ffff8801effdeb30
Jan 6 18:38:44 <server name> kernel: [621546.637539] ffffffff810b2ce1
0000000000000000 0000000000000010 ffffffff817d0c59
Jan 6 18:38:44 <server name> kernel: [621546.637539] Call Trace:
Jan 6 18:38:44 <server name> kernel: [621546.637539] [<ffffffff812c8f7f>]
dump_stack+0x63/0x84
Jan 6 18:38:44 <server name> kernel: [621546.687443] [<ffffffff811ce19d>]
dump_header+0x5e/0x1d8
Jan 6 18:38:44 <server name> kernel: [621546.687443] [<ffffffff810b2ce1>]
? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Jan 6 18:38:44 <server name> kernel: [621546.687443] [<ffffffff81163885>]
oom_kill_process+0x205/0x3d0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff81163ee1>]
out_of_memory+0x431/0x480
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff81168fae>]
__alloc_pages_nodemask+0x91e/0xa60
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff811acd18>]
alloc_pages_current+0x88/0x120
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff81160184>]
__page_cache_alloc+0xb4/0xc0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff811624c8>]
filemap_fault+0x188/0x3e0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffffa0122d16>]
ext4_filemap_fault+0x36/0x50 [ext4]
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff81189ead>]
__do_fault+0x3d/0x70
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff8118e2e7>]
handle_mm_fault+0xf27/0x1870
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff810cd19a>]
? hrtimer_try_to_cancel+0x1a/0x110
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff814dcc41>]
? do_nanosleep+0x91/0xf0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff8105ea33>]
__do_page_fault+0x183/0x3f0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff814dcc0a>]
? do_nanosleep+0x5a/0xf0
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff8105ecc2>]
do_page_fault+0x22/0x30
Jan 6 18:38:44 <server name> kernel: [621546.715834] [<ffffffff814dfc58>]
page_fault+0x28/0x30
Jan 6 18:38:44 <server name> kernel: [621546.815582] Mem-Info:
Jan 6 18:38:44 <server name> kernel: [621546.818934] active_anon:1704405
inactive_anon:1380 isolated_anon:0
Jan 6 18:38:44 <server name> kernel: [621546.818934] active_file:25
inactive_file:29 isolated_file:0
Jan 6 18:38:44 <server name> kernel: [621546.818934] unevictable:4133
dirty:0 writeback:0 unstable:0
Jan 6 18:38:44 <server name> kernel: [621546.818934]
slab_reclaimable:3463 slab_unreclaimable:31406
Jan 6 18:38:44 <server name> kernel: [621546.818934] mapped:2363
shmem:1487 pagetables:124645 bounce:0
Jan 6 18:38:44 <server name> kernel: [621546.818934] free:24488
free_pcp:793 free_cma:0
Jan 6 18:38:44 <server name> kernel: [621546.858492] Node 0 DMA
free:15904kB min:140kB low:172kB high:208kB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 6 18:38:44 <server name> kernel: [621546.905297] lowmem_reserve[]: 0
3728 7446 7446
Jan 6 18:38:44 <server name> kernel: [621546.911785] Node 0 DMA32
free:48424kB min:33772kB low:42212kB high:50656kB active_anon:3414356kB
inactive_anon:4936kB active_file:56kB inactive_file:52kB unevictable:5864kB
isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3840164kB
mlocked:5864kB dirty:0kB writeback:0kB mapped:7240kB shmem:5328kB
slab_reclaimable:8600kB slab_unreclaimable:64688kB kernel_stack:13936kB
pagetables:258600kB unstable:0kB bounce:0kB free_pcp:1652kB local_pcp:740kB
free_cma:0kB writeback_tmp:0kB pages_scanned:1037020 all_unreclaimable? yes
Jan 6 18:38:44 <server name> kernel: [621546.965981] lowmem_reserve[]: 0 0
3717 3717
Jan 6 18:38:44 <server name> kernel: [621546.972390] Node 0 Normal
free:33624kB min:33668kB low:42084kB high:50500kB active_anon:3403264kB
inactive_anon:584kB active_file:60kB inactive_file:48kB unevictable:10668kB
isolated(anon):0kB isolated(file):0kB present:3932160kB managed:3806484kB
mlocked:10668kB dirty:0kB writeback:0kB mapped:2212kB shmem:620kB
slab_reclaimable:5252kB slab_unreclaimable:60936kB kernel_stack:12752kB
pagetables:239980kB unstable:0kB bounce:0kB free_pcp:1520kB local_pcp:596kB
free_cma:0kB writeback_tmp:0kB pages_scanned:1047808 all_unreclaimable? yes
Jan 6 18:38:44 <server name> kernel: [621547.027641] lowmem_reserve[]: 0 0
0 0
Jan 6 18:38:44 <server name> kernel: [621547.033532] Node 0 DMA: 0*4kB
0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB
(U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Jan 6 18:38:44 <server name> kernel: [621547.052434] Node 0 DMA32: 354*4kB
(UE) 824*8kB (UME) 608*16kB (UME) 243*32kB (UE) 118*64kB (UE) 46*128kB (UE)
7*256kB (UME) 5*512kB (ME) 5*1024kB (M) 0*2048kB 0*4096kB = 48424kB
Jan 6 18:38:44 <server name> kernel: [621547.075549] Node 0 Normal:
194*4kB (UE) 692*8kB (UME) 527*16kB (UE) 180*32kB (UME) 57*64kB (UE)
18*128kB (UME) 16*256kB (UME) 6*512kB (M) 0*1024kB 0*2048kB 0*4096kB =
33624kB
Jan 6 18:38:44 <server name> kernel: [621547.098792] Node 0
hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 6 18:38:44 <server name> kernel: [621547.109457] 2436 total pagecache
pages
Jan 6 18:38:44 <server name> kernel: [621547.114400] 0 pages in swap cache
Jan 6 18:38:44 <server name> kernel: [621547.118905] Swap cache stats: add
0, delete 0, find 0/0
Jan 6 18:38:44 <server name> kernel: [621547.125423] Free swap = 0kB
Jan 6 18:38:44 <server name> kernel: [621547.129464] Total swap = 0kB
Jan 6 18:38:44 <server name> kernel: [621547.133515] 1965981 pages RAM
Jan 6 18:38:44 <server name> kernel: [621547.137595] 0 pages
HighMem/MovableOnly
Jan 6 18:38:44 <server name> kernel: [621547.142490] 50343 pages reserved

<Server Specification>
CPU: 4 vCPU.
Memory: 7.5 GiB.
Network Bandwidth: 750Mbps.

<Prefork Configuration>
##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 20
MinSpareServers 15
MaxSpareServers 45
ServerLimit 20000
MaxClients 20000
MaxRequestsPerChild 4000
</IfModule>

Thanks,

Junyoung

2018-01-11 23:24 GMT+09:00 Eric Covener < ... at gmail dot com>:

Re: httpd processes are too many created from Apac

By Daniel at 01/13/2018 - 22:14

You cant switch from prefork to something else until you get rid of
mod_php, which is not thread-safe, so if you switch to worker chances
are you might have other issues.

Worth checking "ulimit -u" but it seems your problem is hitting memory
limit like the kernel messages say.

I would think it is time to spread the load among more than one server
or give this one more resources.

The recommendation here though is to upgrade to Apache httpd 2.4, and
move the php processing to php's own dedicated daemon which is called
php-fpm, that way apache httpd can use the prefered mpm event instead
of prefork.

You can check more about it here:
<a href="https://wiki.apache.org/httpd/PHP-FPM" title="https://wiki.apache.org/httpd/PHP-FPM">https://wiki.apache.org/httpd/PHP-FPM</a>

If your php application is too demanding php-fpm will allow you to
host it into another machine with more resources, or thinking further,
setup a balanacer in httpd and spread the php-fpm load among several
machines

2018-01-12 2:47 GMT+01:00 갈준영 < ... at mz dot co.kr>: