Though it may seem like alphabet soup at first, initramfs is the solution. Short for initialize RAM filesystem, this lets us load a stripped-down rootfs into memory before loading the real one.
The nerves_initramfs bears close resemblance to the U-boot scripting we saw above, and, for practical purposes, it does pretty much the same thing: verify our firmware. It takes a different approach to get there, but the TL;DR is that it lets us revert to our known good partition if we need to.
Since Elixir is built on top of Erlang, we have access to built-in features like heart. Essentially, this module lets us ‘take a pulse’ at predefined intervals to make sure that everything is running smoothly. We can ask heart to check up on internet connectivity, an active link to NervesHub, or external sources like sensors or MQTT service.
In an article titled You Gotta Have Heart, Doug Selph explains that “Heart runs in the BEAM as an optional feature, and when active will trigger a restart if it detects an application crash.” By implementing a watchdog system, a heart callback will either give the ok to keep running, or it will automatically reboot the system.
You can check out the nerves_heart repository for all the juicy details about how we implement this feature in Nerves.
Our final tactic for creating reliable embedded systems is Shoehorn, a shim that ensures our virtual machine (VM) will always pass initialization. Let’s take a step back to look at this advantage.
Let’s say our application has some critical dependencies that provide networking, secure shell (SSH) access, or a link to NervesHub. If something goes wrong, these tools let us access the device to fix the problem. The trouble, however, is that when our application crashes, then these dependencies crash with it. This leaves us stuck in quite the jam: our broken app also broke the tools for fixing it.
By using Shoehorn, however, we’re able to configure our initialize sequence to separate the dependencies’ startup processes from the app itself. That way, even if the app crashes, those dependencies keep running. Therefore, we can still access our device because we still have networking, SSH, and NervesHub.
All of these device hardening strategies ultimately boil down to a central design philosophy. In his talk, Jon Carstens implored us to always keep this mantra at the front of our mind:
“ALWAYS BE ABLE TO UPDATE THE DEVICE!”
Troubleshooting and repair can be hard, but they’re a lot less frustrating than not being able to access the device at all. That’s why all these techniques give us doorways to go through, even in the worst of times.
Eager to learn more about how Very makes reliable embedded devices on the Nerves platform? Check out our Nerves development services.