We've been working on a software solution to run on embedded devices for the last several months, and recently found ourselves needing to solve the problem of how to run some of our software at determined times in the device lifecycle. In this post, we'll explore the solutions that we explored, why we didn't choose any of those solutions, and what we ended up with that has worked well.
Our goal is deceptively simple - we need to ensure that our software runs when a device is powered on. I say deceptively simple because we need to ensure that other supporting services are running, the Linux community at large has many competing solutions to this problem, and reliable documentation has been hard to find. In our specific case, we have a collection of services that need to run on boot, comprised of bash scripts and ruby processes. Our software requires an active network connection and a local redis server. Additionally, our deployment device is a Raspberry Pi running Rasbian Linux, so our solution must work there.
We researched a few different solutions, outlined below. Ultimately we used Systemd.
System V (init.d) Script
Research how to run a bash script on boot in Linux, and the answer will most likely be creating a script in the
/etc/init.d/ folder. The script itself is a bash script, which ends up being a lot of boilerplate code and very easy to get wrong. This is widely supported, although deprecated on most Linux distributions.
- Already installed
- Widely supported across Linux distros
- Lots of boilerplate code
- Difficult to read and write
- No formal concept of load order to init scripts
- Deprecated in favor of newer tools such as Upstart, Runit, or Systemd
- Have to handle daemonization ourselves
The above is copied from this repo.
When starting a long-running process (daemon), it was up to us to write handle logging, PID file creation, signal handling and more. When writing a ruby daemon, we wrapper our script using the daemons gem, but this still required extra effort and the creation of a "control" script. Additionally, System V doesn't provide a way to monitor and ensure that a daemon is kept alive, meaning if our script fails it would not be restarted.
After passing on System V, Monit seems like a much better choice. It has a relatively easy configuration file syntax, can start our processes on boot, and can even ensure that the process is kept alive - if the process fails, monit can restart it automatically.
Monit itself is a daemon that can be installed via a package manager ( in our case on Ubuntu and Rasbian) and is launched on boot. The monit process can be configured fairly easily to monitor and start scripts, although there are a few drawbacks that made monit our less than ideal choice.
- Monit is launched on boot via System V
- Easy configuration sytnax
- Can restart a failed process automatically if desired
- Can kill and restart a process based on parameters such as CPU usage and memory consumption
- Does provide a formal way to declare that a monit service depends upon another monit service
- Requires installation, such as apt-get on Raspian and Ubuntu
- Still requires us to handle daemonization ourselves, with logging, PID files, and signal handling
- Monit only provides a "sanitized" shell to launch processes, meaning a useless
$PATHand more needs manipulating
- Monit is a "polling" process, meaning a process that dies can be dead until Monit checks again
Ultimately we didn't go with Monit because we needed to ensure that one of our processes was running always - and there will always be a delay with Monit polling for statuses of each process at an interval.
We did not do a lot of research into Runit, simply because Upstart and Systemd were what was available on Ubuntu. It really only popped on our radar because of Mike Perham of Sidekiq fame.
- Not available on Ubuntu or Raspian
Initially Upstart looked like the perfect solution, as it is the designated init system in Ubuntu 14.0 and has a clean syntax. However, the Linux community is attempting to converge on a standard, and with the release of Ubuntu 15.0 Systemd and not Upstart, is the default init system. Combined with the fact that Raspian comes with Systemd and not Upstart, and the choice was made for us.
- Not available on Raspbian
- Deprecated on Ubuntu 15.0 release
Finally - Systemd
The final solution we found involved using Systemd to manage our processes on boot and keep them alive reliabily.
- Installed and the default service manager on Ubuntu and Raspian
- Simple configuration syntax
- Can immediately restart a failed process
- No need to daemonize our script!
- Simple settings for user, working directory, and output
- Integrated with syslog
- Allows for "pre" and "post" run scripts before executing the process
- Documentation didn't make it clear that your service configuration file could exist anywhere on the filesystem.
In the end, Systemd provded to provide the exact functionality we required, without requiring another dependency such as Monit to manage our services.