This just started happening three weeks or so ago. The content of my website hasn't changed, it's just a phpBB forum using MySQL as a back end.

Nothing has changed in well over a year but recently, every two days or so, the server just shuts down and cannot be accessed at all, I have to notify my service provider to physically restart the machine.

It seems to be tied to these SIGTERM errors I find in the logs. Problem is I have no idea how to fix these kinds of things or find the root cause as my skills in this area are lacking.

Anyone have any ideas what could be going on?

Apache/2.2.3 (CentOS)
20051115
Linux 2.6.18-028stab057.4 #1 SMP Fri Aug 1 10:47:59 MSD 2008 x86_64

From the error log:

[Sun Nov 01 15:18:53 2009]开发者_如何学运维 [notice] caught SIGTERM, shutting down

Not entirely sure that's what's causing me my problems but it doesn't sound like a good thing.

I can post more log information if needed

[Sun Nov 01 05:07:17 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 05:07:17 2009] [notice] Digest: done
[Sun Nov 01 05:07:19 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 05:07:19 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 05:07:19 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 05:07:19 2009] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Sun Nov 01 05:07:19 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 15:16:11 2009] [error] [client 127.0.0.1] Invalid method in request get
[Sun Nov 01 15:18:53 2009] [notice] caught SIGTERM, shutting down
[Sun Nov 01 15:19:27 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 01 15:19:27 2009] [notice] Digest: done
[Sun Nov 01 15:19:27 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Sun Nov 01 15:19:27 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Sun Nov 01 15:19:27 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Sun Nov 01 15:19:27 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 01 16:42:31 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 16:59:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:37:42 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:54 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:55 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 17:44:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:14:45 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 18:40:30 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:10:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:35:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 20:43:46 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:25:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:26:03 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 21:56:57 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:21:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:16 2009] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Sun Nov 01 22:25:17 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:18 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:20 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:21 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:25:22 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:46:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:52:56 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:07 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 22:56:58 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:00:41 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:09:59 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:23:49 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:27:14 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:36:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Sun Nov 01 23:54:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:44 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:12:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:13:23 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:39:06 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:52:50 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 00:53:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:39 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:13:40 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:23:01 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:43:47 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:45:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:46:27 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 01:47:25 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:23:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:37 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:25:38 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 02:52:51 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:29 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 03:49:48 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:35:33 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 04:46:01 2009] [error] [client 66.96.201.63] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 04:46:02 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/README
[Mon Nov 02 05:01:06 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde
[Mon Nov 02 05:01:07 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde2
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde3
[Mon Nov 02 05:01:08 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/horde-3.0.9
[Mon Nov 02 05:01:09 2009] [error] [client 58.246.73.74] File does not exist: /var/www/vhosts/default/htdocs/Horde
[Mon Nov 02 05:01:09 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:10 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 05:01:11 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/
[Mon Nov 02 09:17:58 2009] [notice] caught SIGTERM, shutting down
[Mon Nov 02 09:18:58 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Digest: generating secret for digest authentication ...
[Mon Nov 02 09:18:58 2009] [notice] Digest: done
[Mon Nov 02 09:18:58 2009] [notice] mod_python: Creating 4 session mutexes based on 10 max processes and 0 max threads.
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] RSA server certificate CommonName (CN) `plesk' does NOT match server name!?
[Mon Nov 02 09:18:58 2009] [warn] Init: SSL server IP/port conflict: default-64-6-237-192:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:83) vs. webmail:443 (/etc/httpd/conf.d/zz010_psa_httpd.conf:159)
[Mon Nov 02 09:18:58 2009] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!
[Mon Nov 02 09:18:58 2009] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Mon Nov 02 09:32:34 2009] [error] [client ::1] Directory index forbidden by Options directive: /var/www/html/

SIGTERM is used to restart Apache (provided that it's setup in init to auto-restart): http://httpd.apache.org/docs/2.2/stopping.html

The entries you see in the logs are almost certainly there because your provider used SIGTERM for that purpose. If it's truly crashing, not even serving static content, then that sounds like some sort of a thread/connection exhaustion issue. Perhaps a DoS that holds connections open?

Should definitely be something for your provider to investigate.

I had mysterious SIGTERM shutdowns in our L.A.M.P. server, and it turned out to be an error in a custom PHP module, which was caused by mismatched versions. It was found by looking in the apache access/error logs at the time of malfunction. Don't forget to turn error logging on.

Have you asked your provider to investigate? I assume this is not a dedicated server,

On the face of it, this seems like a security exception and somone is trying to exploit it / or there is a process running at a set time which is causing this, can you think of anything that runs on the server every 2 days? Logging tools?

SIGTERM is the signal sent to a process to request its termination. The symbolic constant for SIGTERM is defined in the header file signal.h. Symbolic signal names are used because signal numbers can vary across platforms, however on the vast majority of systems, SIGTERM is signal #15.

from this page:

I found this info:

The mod_fastcgi process manager isn't particularly patient though (there's room for improvement here) and since it has to shutdown too, sends a SIGTERM to all of the FastCGI applications it is responsible for. Apache will restart the process manager and it will restart its managed applications (as if the server was just started). SIGTERM is, well, SIGTERM - your application should exit quickly.

What this implies to me is that if Database I/O, or some other part of the CGI script, fails to respond in a timely fashion (ie getting slower with data-volume growth), that mod_fastcgi is killing the script......is that how other people interpret these docs or what am I missing..

Apache is not running

It could also be something as simple as Apache not being configured to start automatically on boot. Assuming you are on a Red Hat-like system such as CentOS or Fedora, the chkconfig –list command will show you which services are set to start for each runlevel. You should see a line like

    httpd           0:off   1:off   2:on    3:on    4:on    5:on    6:off

If instead it says "off" all the way across, you can activate it with chkconfig httpd on. OR you can start apache manually from your panel.

Try to upgrade and install new packages

sudo apt-get update && sudo apt-get upgrade -y

During writing to a large Excel file with python and checking the memory in another terminal using free -m command, when the memory had reached 0, the console output was

Killed

Looking up the logs of the server, I found that the error was that same as yours:

Received SIGTERM, shutting down

Therefore, in my case, it was a memory problem. I did not know that writing to Excel takes so much RAM, see Why does writing to a workbook of a few MB with Python's openpyxl module eat Gigabytes of RAM?.

try to disable the rewrite module in ubuntu using sudo a2dismod rewrite. This will perhaps stop your apache server to crash.