I’ve noticed that occasionally the server completely stop responding, disconnects all clients, and generates no more logs (see this post for an example of what this looks like). Checking in supervisorctl, the server status will still be RUNNING.
As a solution while we work to figure out the underlying cause, I’ve written a simple shell script that monitors the logging on the server and restarts supervisord if it notices that nothing was logged since the last time the script ran. The shell script is run via cron every 5 minutes, but we could increase the interval if desired. Since I implemented this on my server I haven’t noticed any connection issues or hangups, but I’ll update this post when I learn more.
The script can be found here. To install, run crontab -e under a user with the proper permission to access the necessary log files and add the following line. Root works, but it’s probably safer to not do that.
*/5 * * * * /bin/bash /home/pi/evolver/server_monitor.sh >> /home/pi/evolver/server_monitor.log
This will run the script every 5 minutes. If it needs to restart the server, you will see a line in the log file saying so.
Any feedback is welcome! Ideally we figure out whats going on that is causing the connection issues but this should help in the meantime.