Why is this crashing?

I have a web server that suddenly started crashing for no reason. I hadn’t changed any software, it didn’t get hacked, it didn’t run out of memory. It just suddenly decided to crash three times in a 24 hour period… and then again two days later.

I never did figure out why, although I narrowed it down to an issue with php-fpm. However, while troubleshooting, I discovered something interesting: I could bring the server back up simply by restarting the fpm daemon. Linux was fine, my web app was fine, even the nginx web server daemon was fine. All I needed to do was restart fpm.

If only there was a robot that could watch the server for you and automatically restart fpm when needed. If only someone had a nice writeup about it. If only you could read it.

Well guess what? There is.

I have.

You can.

Monit: our savior

Our hero is a tiny program called monit. It runs as a daemon on your server and watches what you tell it to watch. It comes with built in settings to watch apache, mysql, memcache, and many others. It’s free and open source (of course), but they also offer a paid cloud service so you can monitor all your monit daemons on all your servers from a central dashboard. There’s also an app for that. On an Ubuntu server, you can install it as easily as typing sudo apt-get install monit.

Once you have, there’s a little bit of configuration. For each service you want to watch, you need to add a profile to the /etc/monit/conf.d directory.

Just one wrinkle: monit doesn’t come with a profile to monitor fpm. That’s okay; we’ll make our own.

Roll up your sleeves

Monit profile files follow a really simple format:

	

		
check process <name> with <way of checking process>
  group permissions
  start program = <something>
  stop program = <something>
  if failed <thing we’re watching>
    additional conditions
    then restart
  if 3 restarts within 5 cycles then timeout
  depends on <something it depends on>

	
	

This is all pretty easy to understand. It’s even easier with an example:

	

		
check process php5-fpm with pidfile /var/run/php5-fpm.pid
  group root     #change accordingly
  start program = "/etc/init.d/php5-fpm restart"
  stop program  = "/etc/init.d/php5-fpm stop"
  if failed host localhost port 80 protocol http
     and request '/'
     with timeout 20 seconds for 5 cycles
     then restart
  ## If the restarts attempts fail then alert.
  if 3 restarts within 5 cycles then timeout
  depends on nginx

	
	

If you install monit using your package manager (apt or yum), then it should automatically start when your server does, and it will automatically watch anything you put in /etc/monit/conf.d. It logs the results in /var/log/monit.log in case you want to see what it’s been up to. You can also tell it to email you if it catches something (but you need to set up access to an smtp server).

Bottom line

It will automatically restart your server, but it won’t solve your underlying server problems for you. However, if you have a brittle environment that is difficult or impossible to completely fix, monit can make sure your site will stay up. This will literally save me from losing sleep.

Check it out for yourself here: https://mmonit.com/monit/

And for further reading: https://www.digitalocean.com/community/tutorials/lemp-stack-monitoring-with-monit-on-ubuntu-14-04