[performance] apache server reached MaxClients setting, consider raising the MaxClients setting

When you use Apache with mod_php apache is enforced in prefork mode, and not worker. As, even if php5 is known to support multi-thread, it is also known that some php5 libraries are not behaving very well in multithreaded environments (so you would have a locale call on one thread altering locale on other php threads, for example).

So, if php is not running in cgi way like with php-fpm you have mod_php inside apache and apache in prefork mode. On your tests you have simply commented the prefork settings and increased the worker settings, what you now have is default values for prefork settings and some altered values for the shared ones :

StartServers       20
MinSpareServers    5
MaxSpareServers    10
MaxClients         1024
MaxRequestsPerChild  0

This means you ask apache to start with 20 process, but you tell it that, if there is more than 10 process doing nothing it should reduce this number of children, to stay between 5 and 10 process available. The increase/decrease speed of apache is 1 per minute. So soon you will fall back to the classical situation where you have a fairly low number of free available apache processes (average 2). The average is low because usually you have something like 5 available process, but as soon as the traffic grows they're all used, so there's no process available as apache is very slow in creating new forks. This is certainly increased by the fact your PHP requests seems to be quite long, they do not finish early and the apache forks are not released soon enough to treat another request.

See on the last graphic the small amount of green before the red peak? If you could graph this on a 1 minute basis instead of 5 minutes you would see that this green amount was not big enough to take the incoming traffic without any error message.

Now you set 1024 MaxClients. I guess the cacti graph are not taken after this configuration modification, because with such modification, when no more process are available, apache would continue to fork new children, with a limit of 1024 busy children. Take something like 20MB of RAM per child (or maybe you have a big memory_limit in PHP and allows something like 64MB or 256MB and theses PHP requests are really using more RAM), maybe a DB server... your server is now slowing down because you have only 768MB of RAM. Maybe when apache is trying to initiate the first 20 children you already reach the available RAM limit.

So. a classical way of handling that is to check the amount of memory used by an apache fork (make some top commands while it is running), then find how many parallel request you can handle with this amount of RAM (that mean parallel apache children in prefork mode). Let's say it's 12, for example. Put this number in apache mpm settings this way:

<IfModule prefork.c>
  StartServers       12
  MinSpareServers    12
  MaxSpareServers    12
  MaxClients         12
  MaxRequestsPerChild  300
</IfModule>

That means you do not move the number of fork while traffic increase or decrease, because you always want to use all the RAM and be ready for traffic peaks. The 300 means you recyclate each fork after 300 requests, it's better than 0, it means you will not have potential memory leaks issues. MaxClients is set to 12 25 or 50 which is more than 12 to handle the ListenBacklog queue, which can enqueue some requests, you may take a bigger queue, but you would get some timeouts maybe (removed this strange sentende, I can't remember why I said that, if more than 12 requests are incoming the next one will be pushed in the Backlog queue, but you should set MaxClient to your targeted number of processes).

And yes, that means you cannot handle more than 12 parallel requests.

If you want to handle more requests:

  • buy some more RAM
  • try to use apache in worker mode, but remove mod_php and use php as a parallel daemon with his own pooler settings (this is called php-fpm), connect it with fastcgi. Note that you will certainly need to buy some RAM to allow a big number of parallel php-fpm process, but maybe less than with mod_php
  • Reduce the time spent in your php process. From your cacti graphs you have to potential problems: a real traffic peak around 11:25-11:30 or some php code getting very slow. Fast requests will reduce the number of parallel requests.

If your problem is really traffic peaks, solutions could be available with caches, like a proxy-cache server. If the problem is a random slowness in PHP then... it's an application problem, do you do some HTTP query to another site from PHP, for example?

And finally, as stated by @Jan Vlcinsky you could try nginx, where php will only be available as php-fpm. If you cannot buy RAM and must handle a big traffic that's definitively desserve a test.

Update: About internal dummy connections (if it's your problem, but maybe not).

Check this link and this previous answer. This is 'normal', but if you do not have a simple virtualhost theses requests are maybe hitting your main heavy application, generating slow http queries and preventing regular users to acces your apache processes. They are generated on graceful reload or children managment.

If you do not have a simple basic "It works" default Virtualhost prevent theses requests on your application by some rewrites:

  RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
  RewriteRule .* - [F,L]

Update:

Having only one Virtualhost does not protect you from internal dummy connections, it is worst, you are sure now that theses connections are made on your unique Virtualhost. So you should really avoid side effects on your application by using the rewrite rules.

Reading your cacti graphics, it seems your apache is not in prefork mode bug in worker mode. Run httpd -l or apache2 -l on debian, and check if you have worker.c or prefork.c. If you are in worker mode you may encounter some PHP problems in your application, but you should check the worker settings, here is an example:

<IfModule worker.c>
  StartServers           3
  MaxClients           500
  MinSpareThreads       75
  MaxSpareThreads      250 
  ThreadsPerChild       25
  MaxRequestsPerChild  300
</IfModule>

You start 3 processes, each containing 25 threads (so 3*25=75 parallel requests available by default), you allow 75 threads doing nothing, as soon as one thread is used a new process is forked, adding 25 more threads. And when you have more than 250 threads doing nothing (10 processes) some process are killed. You must adjust theses settings with your memory. Here you allow 500 parallel process (that's 20 process of 25 threads). Your usage is maybe more:

<IfModule worker.c>
  StartServers           2
  MaxClients           250
  MinSpareThreads       50
  MaxSpareThreads      150 
  ThreadsPerChild       25
  MaxRequestsPerChild  300
</IfModule>

Examples related to performance

Why is 2 * (i * i) faster than 2 * i * i in Java? What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism? How to check if a key exists in Json Object and get its value Why does C++ code for testing the Collatz conjecture run faster than hand-written assembly? Most efficient way to map function over numpy array The most efficient way to remove first N elements in a list? Fastest way to get the first n elements of a List into an Array Why is "1000000000000000 in range(1000000000000001)" so fast in Python 3? pandas loc vs. iloc vs. at vs. iat? Android Recyclerview vs ListView with Viewholder

Examples related to apache

Enable PHP Apache2 Switch php versions on commandline ubuntu 16.04 Laravel: PDOException: could not find driver How to deploy a React App on Apache web server Apache POI error loading XSSFWorkbook class How to enable directory listing in apache web server Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details How to enable php7 module in apache? java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient The program can't start because api-ms-win-crt-runtime-l1-1-0.dll is missing while starting Apache server on my computer

Examples related to thread-safety

Are these methods thread safe? apache server reached MaxClients setting, consider raising the MaxClients setting How can I make a JUnit test wait? How to stop a thread created by implementing runnable interface? What Makes a Method Thread-safe? What are the rules? Android - Best and safe way to stop thread Why is Java's SimpleDateFormat not thread-safe? What is thread Safe in java? How does lock work exactly? Thread-safe List<T> property

Examples related to cacti

apache server reached MaxClients setting, consider raising the MaxClients setting