Those who are interested in prototyping and building IoT devices often come to the point, ‘Okay what are the next steps to bring my product into the market? Perhaps sell a few on Tindie and gather more feedback.‘
This is huge, BUT what might stop you from moving forward? IoT Security.
A lot of effort is required to get the Arduino application secure using the native security functions of the ESP32 microcontroller. The security functions force you to rewrite the application in the ESP-IDF or you do it the hacky way by attaching the Arduino code into a native ESP-IDF secured application. Furthermore, securing the application is only one part. You still have to think about a few other things, for example, the product lifecycle, how to update the application on the device in the field, and how to manage the certificates on the device, since they need to be updated from time to time. Additionally, when working with Arduino and ESP-32, there is a broader selection of libraries. Getting these to work is also a little challenging when securing the device.
Starting the chapter on IoT Security, the entire implementation from scratch took me a minimum of 50% of the entire product development process. Since it was such a big effort, I want to share my learnings and hopefully save you some time.
For a better overview, here is a brief description of the System Architecture of the used hardware and software components, and protocols.
Backend:
- rabbitMQ
- Apache2.0 as a firmware server for OTA updates
Protocol:
- MQTT – TLS Encrypted (here using self-signed certificates)
Device:
- esp32 lolin dev-kit – 4MB flash.
IDE:
- Arduino and PlatformIO for application development & ESP IDF for secure flashing
Security+ Concept
From this perspective, it’s more a security+ standpoint. The + comes from further consideration about what if the device locks itself out of the secure infrastructure due to a faulty OTA update or a certificate becoming invalid over time.
1. Communication implies first encrypted communication, second authentication, and third checking for integrity. As explained, we are going to use mutual authentication through x.509 certificates to not only authenticate the backend server, but also the IoT device itself. Find further reading about general Transport – Layer – Security here.
2. Storing the certificates securely, so that no one can read them from the flash or an application running on the uC itself.
2.1. Flash encryption: In case firmware updates are not expected or updates are done manually, embed the certificates in the firmware, encrypt the firmware binary, and flash it on the device. More reading about how to do this are in the upcoming chapters.
2.2. Store the certificates in an encrypted partition of the ESP32, the NVS using OTA updates.
3. Secure OTA updates in the optimal case where the device draws firmware updates from an HTTPs firmware server.
4. Utilization of secure boot – meaning you sign your app with a key, and the uC will only execute a signed application. Following, the uC can’t be abused in a sense of downloading the wrong application and starting to monitor your private network. (Sadly, not covered in the upcoming example).
5. Fallback – in case something goes wrong. Say the certificates are not valid anymore, and the device can not connect to the backend. Implement a second secure connection to a different backend server. In this case, it might be only password-protected, yet nevertheless encrypted via TLS (not mutual). Here you can publish new certificates to the devices. This is especially good since you can only send certificates to specific devices. This might be one of many fallbacks as to why some IoT Infrastructure providers developed their own mechanisms. Considering this factor is important.
Secure Communication
As already mentioned above, we use mutual TLS. When we understand the concept of TLS, which does not seem feasible to cover in this article, we can talk about the implementation on the uC side. The implementation within the Arduino Framework took a little while, since it was hard to find working libraries and good documentation. For this reason, I want to share this amazing library and example with you, where a TLS-secured MQTT connection was implemented. In addition, the examples include a script to create self-signed certificates, which you use on your backend server and on the uC.
Once you have this up and running – congrats! You’ve made the first step to a safer IoT world.
Firmware / Flash Encryption
Next – Why? Simply because it is just too easy to read out the flash with all valuable data in plain text. If you want to test for yourself how a flash containing an app looks from the inside, use this command to read out the flash of the ESP32. The same would work with a different tool set for other uC as well.
Brief explanation: available baud rates go from 115200 to 576000. From experience, if you download the firmware too quickly, an error can occur, and you have to start again. In the read flash section, you paste the addresses. Since it’s a μC with 4MB flash, we read the address from 0x0 to 0x40000 (0x just means you gonna read a hex number) — following the entire flash, which then contains, the bootloader, physical interface storage (PHY), where the wifi data and firmware are stored. If you are new to partition tables and storage types, I encourage you to read this article or the ESP32 documentation.
Once done, you can open the trick.bin file in your preferred IDE, I am using VS-Code with this hex reader plugin.
Hello key, hello wifi password, hello every string which is saved on your uC you can read out with ease.
Since this was so easy, let’s start to secure the device by using the out-of-the-box security functions by Espressif.
Prepare Arduino Code in PlatformIO
What’s the important part here, is knowing the offset of each partition for later encryption and flashing. What we can’t set in platformIO, is the offset for the partition table and bootloader, these are set within platformIO itself. You can change them by changing some source files but it’s not required, because you can just match them within the ESP-IDF. For your reference, the default offsets in platformIO:
- Bootloader: 0x1000
- Partion Table: 0x8000
- Firmware:0x10000
In my example, I am using two OTA partitions, generating an encrypted nvs (non-versatile-storage) for the TLS certificates.
Setting up Your Encryption Workspace
After successfully creating your Arduino firmware in a separate project, let’s create a separate encryption workspace that runs the ESP-IDF. Follow these steps to install the IDF. Once done, copy the hello_world example from the IDF into your project folder. Here you can manage flash encryption certificates and some shell files which automate the encryption and flashing process of your firmware on your ready-to-ship secure ESP32. Create a flash encryption key per unique device in the workspace:
1. Generate flash encryption Key for the esp32
2. Burn the key on your ready-to-ship esp32
If the key is not burned and the device is started after enabling flash encryption, the ESP32 will generate a random key that the software cannot access or modify. This is what we don’t want!
3. Set the Bootloader log verbosity to (no output) — this will decrease the size of the bootloader, so you can enable flash encryption without changing the partition-table offset in platformIO and in the IDF.
4. Enable Flash Encryption (for testing the procedure, set the usage mode to develop only)
5. Double-check the offset of the partition table to address 0x8000
Flash and Execute Your Arduino Code Onto the ESP32
1. Build your Arduino code in your project workspace
2. from .pio/build/lolin32/ — copy the firmware.bin and the partitions.bin into your ESP-IDF encryption workspace.
3. build your normal application with: idf.py build
4. Next, we can encrypt all binaries we need to flash our esp-32 with these lines of code. Finally, we have a mix of encrypted bootloader files orignal from the hello_world example containing the enabled flash encryption config and the encrypted partition table and firmware from the Arduino application.
5. Flashing the ESP32
SUCCESS! Your application should be running on your uC. You can check if the firmware on your UC is now fully encrypted and working.
Encrypted NVS
The encrypted NVS is the perfect space to store keys and certificates because they will remain here despite the OTA updates and can be updated separately. Furthermore, the Arduino EEPROM library adopts the NVS functions of the ESP32.
To make use of the encrypted NVS, firmware encryption has to be enabled and the according flags in the partition table have to be set to encrypted (ref. screenshot portion table above). Once the device is starting up, it automatically creates an NVS key to encrypt all data handling with the NVS.
After this is successfully set up, we can store all keys and certificates on the device. To do so, we use our developed updating mechanism to also update outdated keys. In my example, I connect to password-protected TLS secured broker, where I upload the new keys. To save resources, I created another vhost MQTT broker in rabbitmq.
OTA Updates
After setting a good base by storing all sensitive data in a safe compartment, the encrypted NVS, the OTA updates are pretty straightforward since it hopefully does not carry any sensitive data with them.
Luckily, Arduino ESP32 just started to support HTTPs OTA updates, which means, that you can be sure the device will download the right firmware. For OTA updates, this library works very sufficiently. As a note, the firmware binary will get encrypted as soon it is on the device.
Some Last Words
This approach will save you time at developing a safe and fast IoT protoype based on Arduinio and a ESP32. Getting all ESP32 out-of-the-box safety features running with Arduino code was a trial and error journey, since it is not meant to be used like this. However, maybe it will be in the future. In the end, I am happy to share how to implement basic features which are running and tested, which are mainly the flash and NVS encryption together with the OTA update. For sure, this article does not cover all the little pitfalls you might get into when implementing on your own.
Anyway, a secure software implementation is only as good as good hardware. Following the use of the latest hardware revision (older versions of the esp32 get exploited), it is possible to read out the certificates for flash encryption and secure boot by voltage glitching and manipulating the r/w rules of the storage.
As an outlook, the here-described IoT security measurements for a secure IoT prototype have been only part of the effort. A lot of time also went into the right setup of the backend & frontend application.
Thanks for reading! I hope this brief overview is valuable for your upcoming IoT prototype projects!
