apache server reached MaxClients setting consider raising the MaxClients setting

Question

I am running centos 5 5 with 768mb ram  i keep getting server reached MaxClients setting  consider raising the MaxClients setting in the logs also apache runs really slow  when i look at cacti graphs it shows the server is not even using all the resources   here is the current configuration   lt IfModule prefork c gt  StartServers       8 MinSpareServers    5 MaxSpareServers    10 ServerLimit        1024 MaxClients         768 MaxRequestsPerChild  4000  lt  IfModule gt    lt IfModule worker c gt  StartServers         2 MaxClients         150 MinSpareThreads     25 MaxSpareThreads     75  ThreadsPerChild     25 MaxRequestsPerChild  0  lt  IfModule gt     free -m              total       used       free     shared    buffers     cached Mem            768        352        415          0          0         37 -   buffers cache         315        452 Swap             0          0          0    top - 11 03 54 up 41 days  11 53   1 user   load average  0 05  0 03  0 00 Tasks   35 total    1 running   34 sleeping    0 stopped    0 zombie Cpu s    0 0 us   0 0 sy   0 0 ni  99 7 id   0 0 wa   0 0 hi   0 0 si   0 3 st Mem     786432k total    389744k used    396688k free         0k buffers Swap         0k total         0k used         0k free     38284k cached   I have tried the following but the server responds very slowly   lt IfModule worker c gt   StartServers         2  MaxClients         150  MinSpareThreads     25  MaxSpareThreads     75  ThreadsPerChild     25  MaxRequestsPerChild  0  StartServers    20 MaxClients      1024 ServerLimit     1024 MinSpareThreads 128 MaxSpareThreads 768 ThreadsPerChild 64 MaxRequestsPerChild 0  lt  IfModule gt   free -m              total       used       free     shared    buffers     cached Mem            768        324        443          0          0         37 -   buffers cache         286        481 Swap             0          0          0      regilero  I have updated to    lt IfModule prefork c gt    StartServers       12   MinSpareServers    12   MaxSpareServers    12   MaxClients         50   MaxRequestsPerChild  300  lt  IfModule gt    using top i see  Tasks   36 total    1 running   35 sleeping    0 stopped    0 zombie Cpu s    0 0 us   0 3 sy   0 0 ni  99 7 id   0 0 wa   0 0 hi   0 0 si   0 0 st Mem     786432k total    613180k used    173252k free         0k buffers Swap         0k total         0k used         0k free     76488k cached    PID USER      PR  NI  VIRT  RES  SHR S  CPU  MEM    TIME   COMMAND                                                                                                                                                                              1 root      20   0 10364   92   60 S  0 0  0 0   1 09 53 init                                                                                                                                                                                 2 root      20   0     0    0    0 S  0 0  0 0   0 00 00 kthreadd 808                                                                                                                                                                         3 root      20   0     0    0    0 S  0 0  0 0   0 00 00 khelper 808                                                                                                                                                                        124 root      16  -4 12620    8    4 S  0 0  0 0   0 00 00 udevd                                                                                                                                                                              533 root      20   0 95504 5692  228 S  0 0  0 7   4 02 94 memcached                                                                                                                                                                          546 root      20   0  5924  332  276 S  0 0  0 0   6 54 51 syslogd                                                                                                                                                                            557 root      20   0  101m 1456  868 S  0 0  0 2  13 18 64 snmpd                                                                                                                                                                              570 root      20   0 62640  316  208 S  0 0  0 0   2 39 56 sshd                                                                                                                                                                               579 root      20   0 21656   24   20 S  0 0  0 0   0 00 00 xinetd                                                                                                                                                                             589 root      20   0 12072   12    8 S  0 0  0 0   0 00 05 mysqld safe                                                                                                                                                                        940 mysql     20   0  559m 164m 3832 S  0 3 21 5 209 33 88 mysqld                                                                                                                                                                            1015 root      20   0 20880  200  132 S  0 0  0 0   0 10 48 crond                                                                                                                                                                             1023 root      20   0 46748    4    0 S  0 0  0 0   0 00 00 saslauthd                                                                                                                                                                         1024 root      20   0 46748    4    0 S  0 0  0 0   0 00 00 saslauthd                                                                                                                                                                         3605 root      20   0 62832 2168  636 S  0 0  0 3   0 02 58 sendmail                                                                                                                                                                          3613 smmsp     20   0 57712 1648  504 S  0 0  0 2   0 00 01 sendmail                                                                                                                                                                         17610 root      20   0 85932 3312 2600 S  0 0  0 4   0 00 02 sshd                                                                                                                                                                             17612 mcmap     20   0 86072 1760 1012 S  0 0  0 2   0 00 17 sshd                                                                                                                                                                             17613 mcmap     20   0 12076 1656 1292 S  0 0  0 2   0 00 01 bash                                                                                                                                                                             17637 root      20   0 45052 1432 1120 S  0 0  0 2   0 00 00 su                                                                                                                                                                               17638 root      20   0 12180 1800 1324 S  0 0  0 2   0 00 08 bash                                                                                                                                                                             17740 root      20   0  246m 9264 4516 S  0 0  1 2   0 00 19 httpd                                                                                                                                                                            18264 apache    20   0  282m  43m 4940 S  0 0  5 7   0 00 56 httpd                                                                                                                                                                            18514 apache    20   0  279m  40m 4832 S  0 0  5 3   0 01 47 httpd                                                                                                                                                                            18518 apache    20   0  273m  36m 4396 S  0 0  4 7   0 00 45 httpd                                                                                                                                                                            18528 apache    20   0  251m  13m 3660 S  0 0  1 8   0 00 41 httpd                                                                                                                                                                            18529 apache    20   0  278m  40m 4340 S  0 0  5 3   0 00 99 httpd                                                                                                                                                                            18530 apache    20   0  278m  40m 4268 S  0 0  5 3   0 00 67 httpd                                                                                                                                                                            18548 apache    20   0  272m  33m 3516 S  0 0  4 4   0 00 28 httpd                                                                                                                                                                            18552 apache    20   0  280m  42m 3684 S  0 0  5 5   0 00 48 httpd                                                                                                                                                                            18553 apache    20   0  271m  33m 3768 S  0 0  4 3   0 00 45 httpd                                                                                                                                                                            18555 apache    20   0  274m  36m 3672 S  0 0  4 7   0 00 58 httpd                                                                                                                                                                            18572 apache    20   0  247m 9020 2856 S  0 0  1 1   0 00 01 httpd                                                                                                                                                                            18578 apache    20   0  280m  42m 3684 S  0 0  5 6   0 00 76 httpd                                                                                                                                                                            18589 apache    20   0  246m 5452  676 S  0 0  0 7   0 00 00 httpd                                                                                                                                                                            18588 root      20   0 12624 1216  932 R  0 0  0 2   0 00 06   free -m              total       used       free     shared    buffers     cached Mem            768        578        189          0          0         74 -   buffers cache         504        263 Swap             0          0          0    Just added current picture of cacti result last 4 hours  busy periods are monday tuesday  So i will wait till next week to see further results of the config change  but it looks like an improvement as before i only had max 10 threads available  Looking at this do you think i can make more improvment   free -m              total       used       free     shared    buffers     cached Mem            768        619        148          0          0         49 -   buffers cache         570        197 Swap             0          0          0   NEW TEST  On a 2GB Ram VPS box i have now set prefork to  StartServers      20 MinSpareServers   20 MaxSpareServers   20 ServerLimit  256 MaxClients   256 MaxRequestsPerChild  4000   today morning my memcache server died from  Nov 20 09 28 40 vps22899094 kernel  Out of memory  Kill process 12517  memcached  score 81 or sacrifice child Nov 20 09 28 40 vps22899094 kernel  Killed process 12517  UID 497   memcached  total-vm 565252kB  anon-rss 42940kB  file-rss 44kB   What should the optimal values be to set in apache     etc sysconfig memcached  PORT  11211  USER  memcached  MAXCONN  1024  CACHESIZE  1024  OPTIONS  -l 127 0 0 1     etc my cnf   mysqld  datadir  var lib mysql socket  var lib mysql mysql sock user mysql   Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links 0  bind-address 127 0 0 1   script thread concurrency 2 query cache size   16M query cache type 1 query cache limit 5M    MyISAM    key-buffer-size                  32M  myisam-recover                   FORCE BACKUP    SAFETY    max-allowed-packet               16M  max-connect-errors               1000000    CACHES AND LIMITS   tmp-table-size                   32M max-heap-table-size              32M  query-cache-type                 0  query-cache-size                 0 max-connections                  50 thread-cache-size                16  open-files-limit                 65535  table-definition-cache           1024  table-open-cache                 2048    INNODB    innodb-flush-method              O DIRECT  innodb-log-files-in-group        2  innodb-log-file-size             5M  innodb-flush-log-at-trx-commit   1  innodb-file-per-table            1  innodb-buffer-pool-size          921M   LOGGING   log-error                         var log mysqld log log-queries-not-using-indexes    1 slow-query-log                   1 slow-query-log-file               var log mysqld-slow log   mysqld safe  log-error  var log mysqld log pid-file  var run mysqld mysqld pid

User · Answer

Here s an approach that could resolve your problem  and if not would help with troubleshooting    Create a second Apache virtual server identical to the current one Send all  normal  user traffic to the original virtual server Send special or long-running traffic to the new virtual server   Special or long-running traffic could be report-generation  maintenance ops or anything else you don t expect to complete in  lt  lt 1 second   This can happen serving APIs  not just web pages   If your resource utilization is low but you still exceed MaxClients  the most likely answer is you have new connections arriving faster than they can be serviced   Putting any slow operations on a second virtual server will help prove if this is the case   Use the Apache access logs to quantify the effect

User · Answer

I recommend to use bellow formula suggested on Apache   MaxClients    total RAM - RAM for OS - RAM for external programs     RAM per httpd process   Find my script here which is running on Rhel 6 7  you can made change according to your OS      bin bash  echo  HostName  hostname     Formula  MaxClients    RAM - size all other processes   size apache process  total httpd processes size  ps -ylC httpd --sort rss   awk    sum     9   END   print sum      echo  total httpd processes size  total httpd processes size  total http processes count  ps -ylC httpd --sort rss   wc -l  echo  total http processes count  total http processes count  AVG httpd process size   expr  total httpd processes size    total http processes count  echo  AVG httpd process size  AVG httpd process size  total httpd process size MB   expr  AVG httpd process size   1024  echo  total httpd process size MB  total httpd process size MB  total pttpd used size   expr  total httpd processes size   1024  echo  total pttpd used size  total pttpd used size  total RAM size  free -m  grep Mem  awk   print  2    echo  total RAM size  total RAM size  total used size  free -m  grep Mem  awk   print  3    echo  total used size  total used size  size all other processes   expr  total used size -  total pttpd used size  echo  size all other processes  size all other processes  remaining memory     total RAM size -  size all other processes   echo  remaining memory  remaining memory  MaxClients      total RAM size -  size all other processes     total httpd process size MB   echo  MaxClients  MaxClients  exit

User · Answer

When you use Apache with mod php apache is enforced in prefork mode  and not worker  As  even if php5 is known to support multi-thread  it is also known that some php5 libraries are not behaving very well in multithreaded environments  so you would have a locale call on one thread altering locale on other php threads  for example    So  if php is not running in cgi way like with  php-fpm you have mod php inside apache and apache in prefork mode  On your tests you have simply commented the prefork settings and increased the worker settings  what you now have is default values for prefork settings and some altered values for the shared ones    StartServers       20 MinSpareServers    5 MaxSpareServers    10 MaxClients         1024 MaxRequestsPerChild  0   This means you ask apache to start with 20 process  but you tell it that  if there is more than 10 process doing nothing it should reduce this number of children  to stay between 5 and 10 process available  The increase decrease speed of apache is 1 per minute  So soon you will fall back to the classical situation where you have a fairly low number of free available apache processes  average 2   The average is low because usually you have something like 5 available process  but as soon as the traffic grows they re all used  so there s no process available as apache is very slow in creating new forks  This is certainly increased by the fact your PHP requests seems to be quite long  they do not finish early and the apache forks are not released soon enough to treat another request    See on the last graphic the small amount of green before the red peak  If you could graph this on a 1 minute basis instead of 5 minutes you would see that this green amount was not big enough to take the incoming traffic without any error message   Now you set 1024 MaxClients  I guess the cacti graph are not taken after this configuration modification  because with such modification  when no more process are available  apache would continue to fork new children  with a limit of 1024 busy children  Take something like 20MB of RAM per child  or maybe you have a big memory limit in PHP and allows something like 64MB or 256MB and theses PHP requests are really using more RAM   maybe a DB server    your server is now slowing down because you have only 768MB of RAM  Maybe when apache is trying to initiate the first 20 children you already reach the available RAM limit   So  a classical way of handling that is to check the amount of memory used by an apache fork  make some top commands while it is running   then find how many parallel request you can handle with this amount of RAM  that mean parallel apache children in prefork mode   Let s say it s 12  for example  Put this number in apache mpm settings this way    lt IfModule prefork c gt    StartServers       12   MinSpareServers    12   MaxSpareServers    12   MaxClients         12   MaxRequestsPerChild  300  lt  IfModule gt    That means you do not move the number of fork while traffic increase or decrease  because you always want to use all the RAM and be ready for traffic peaks  The 300 means you recyclate each fork after 300 requests  it s better than 0  it means you will not have potential memory leaks issues  MaxClients is set to 12 25 or 50 which is more than 12 to handle the ListenBacklog queue  which can enqueue some requests  you may take a bigger queue  but you would get some timeouts maybe  removed this strange sentende  I can t remember why I said that  if more than 12 requests are incoming the next one will be pushed in the Backlog queue  but you should set MaxClient to your targeted number of processes    And yes  that means you cannot handle more than 12 parallel requests   If you want to handle more requests    buy some more RAM try to use apache in worker mode  but remove mod php and use php as a parallel daemon with his own pooler settings  this is called php-fpm   connect it with fastcgi  Note that you will certainly need to buy some RAM to allow a big number of parallel php-fpm process  but maybe less than with mod php Reduce the time spent in your php process  From your cacti graphs you have to potential problems  a real traffic peak around 11 25-11 30 or some php code getting very slow  Fast requests will reduce the number of parallel requests    If your problem is really traffic peaks  solutions could be available with caches  like a proxy-cache server  If the problem is a random slowness in PHP then    it s an application problem  do you do some HTTP query to another site from PHP  for example   And finally  as stated by  Jan Vlcinsky you could try nginx  where php will only be available as php-fpm  If you cannot buy RAM and must handle a big traffic that s definitively desserve a test   Update  About internal dummy connections  if it s your problem  but maybe not    Check this link and this previous answer  This is  normal   but if you do not have a simple virtualhost theses requests are maybe hitting your main heavy application  generating slow http queries and preventing regular users to acces your apache processes  They are generated on graceful reload or children managment    If you do not have a simple basic  It works  default Virtualhost prevent theses requests on your application by some rewrites     RewriteCond   HTTP USER AGENT     internal  dummy  connection     NC    RewriteRule    -  F L    Update   Having only one Virtualhost does not protect you from internal dummy connections  it is worst  you are sure now that theses connections are made on your unique Virtualhost  So you should really avoid side effects on your application by using the rewrite rules   Reading your cacti graphics  it seems your apache is not in prefork mode bug in worker mode  Run httpd -l or apache2 -l on debian  and check if you have worker c or prefork c  If you are in worker mode you may encounter some PHP problems in your application  but you should check the worker settings  here is an example    lt IfModule worker c gt    StartServers           3   MaxClients           500   MinSpareThreads       75   MaxSpareThreads      250    ThreadsPerChild       25   MaxRequestsPerChild  300  lt  IfModule gt    You start 3 processes  each containing 25 threads  so 3 25 75 parallel requests available by default   you allow 75 threads doing nothing  as soon as one thread is used a new process is forked  adding 25 more threads  And when you have more than 250 threads doing nothing  10 processes  some process are killed  You must adjust theses settings with your memory  Here you allow 500 parallel process  that s 20 process of 25 threads   Your usage is maybe more     lt IfModule worker c gt    StartServers           2   MaxClients           250   MinSpareThreads       50   MaxSpareThreads      150    ThreadsPerChild       25   MaxRequestsPerChild  300  lt  IfModule gt

User · Answer

Did you consider using nginx  or other event based web server  instead of apache   nginx shall allow higher number of connections and consume much less resources  as it is event based and does not create separate process per connection   Anyway  you will need some processes  doing real work  like WSGI servers or so  and if they stay on the same server as the front end web server  you only shift the performance problem to a bit different place   Latest apache version shall allow similar solution  configure it in event based manner   but this is not my area of expertise

[performance] apache server reached MaxClients setting, consider raising the MaxClients setting

Examples related to performance

Examples related to apache

Examples related to thread-safety

Examples related to cacti