Build, test and release · Build

What it is built on

The platform, the frameworks and the libraries under the firmware, which versions are pinned and why, how they depend on one another, and where the fragile parts are.

The firmware sits on one platform (pioarduino's, which brings the toolchain, Arduino-ESP32 and ESP-IDF), seven libraries pulled in by PlatformIO, and two libraries kept in the repository. Everything that goes into the firmware is pinned in platformio.ini: the platform's release pins the framework and the toolchain in turn, so a fresh checkout builds the firmware from the same sources as the last release. Two libraries are fetched unpinned and not built in: IRremote, which M5Cardputer asks for, and Unity, for the tests. PlatformIO itself is installed at its latest.

What the firmware is built on top to bottom; our code also calls ESP-IDF directly where Arduino has no call for it our code src/: the Apps, the Services, the platform code · lib/: our libraries, most tested on a PC pulled in (lib_deps, pinned) M5Cardputer · M5Unified · M5GFX · RadioLib WireGuard · LibSSH-ESP32 · libsodium kept in lib/ MeshCore, unchanged SD, one file changed Arduino-ESP32 3.3.12 WiFi · Network · NetworkClientSecure · BLE · Preferences · FS · SPI ESP-IDF 5.5.5 rebuilt with our settings (custom_sdkconfig) FreeRTOS · lwIP · mbedTLS 3.6.6 · NimBLE · Wi-Fi · HTTP server · FatFs · libsodium · NVS · OTA ESP32-S3 (Cardputer ADV) · SX1262 LoRa radio and GNSS (Cap LoRa-1262) pioarduino platform 55.03.312: Arduino-ESP32, ESP-IDF and the Xtensa GCC 14.2 toolchain
The layers, top to bottom. Our code mostly goes through Arduino, and calls ESP-IDF directly where Arduino has nothing for it.

The platform

platform = https://github.com/pioarduino/platform-espressif32/releases/download/55.03.312/platform-espressif32.zip
board = m5stack-stamps3
framework = arduino

pioarduino is the community's continuation of PlatformIO's Espressif platform: PlatformIO's own stopped at Arduino-ESP32 2.x. Release 55.03.312 brings:

  • Arduino-ESP32 3.3.12, the layer most of our code talks to: Arduino.h, WiFi and Network (clients and servers), NetworkClientSecure (TLS over TCP), BLEDevice and BLEServer (on NimBLE), Preferences, FS, and SPI (the bus the card and the radio share). Its SD library is replaced by our copy (below).
  • ESP-IDF 5.5.5, under Arduino. The firmware calls it directly where Arduino has nothing, among others: FreeRTOS tasks and semaphores, the heap (how much is free, and its largest block), esp_ota_ops and esp_partition (the update slots), esp_core_dump (crash reports), lwIP's sockets, DNS and interfaces (the network tools), SNTP and esp_wifi (the clock, the country, the Wi-Fi scanner), the HTTP server (Web Share), FatFs (formatting and listing the card), mbedTLS 3.6.6 (certificates, and the updates' signatures), Espressif's libsodium component (Ed25519, see below), NimBLE's host (Bluetooth) and NVS.
  • The Xtensa GCC 14.2 toolchain, esptool, CMake and Ninja.

The framework is rebuilt with our settings

Arduino-ESP32 normally comes with ESP-IDF already compiled, with Espressif's settings. Because platformio.ini has a custom_sdkconfig section, pioarduino recompiles ESP-IDF's libraries with ours (its "hybrid compile", ADR 0006). The first build after a fresh checkout, or after any change to custom_sdkconfig, does it and takes several minutes; later builds take about 15 seconds (Build, test and release). Much of the memory work sits in those settings:

  • TLS buffers taken for each record as it comes or goes, at most 16 KB in and 4 KB out, and the CA chain and the configuration freed once a connection is up (ADR 0006).
  • Smaller stacks for the esp_timer and FreeRTOS timer tasks, and less code in IRAM, the internal RAM the heap doesn't get (milestone M6).
  • Wi-Fi's send buffers taken when sending rather than held from the start, and fewer receive buffers.
  • NimBLE trimmed to a companion's needs, its controller run from flash (ADR 0012).

Neither ESP-IDF's settings tool nor Arduino's prebuilt link step follows every setting, so three things hold it together:

  • scripts/check_sdkconfig.py fails the build when a custom_sdkconfig setting isn't what the framework's libraries were built with: the settings tool drops a setting whose dependencies aren't met, without a word. Its list of known exceptions holds two settings, with the reason they resolve otherwise.
  • scripts/ble_controller_flash.py changes Arduino's link lists so that Bluetooth's controller really runs from flash: it links the controller's flash library instead of libbtdm_app, and drops the ROM's Bluetooth linker script. If either change finds nothing to change, the build fails.
  • Twelve -Wl,--wrap=mbedtls_ssl_* flags in build_flags: ESP-IDF's own build adds them when the TLS buffers are dynamic, Arduino's doesn't, and without them the dynamic code is compiled but never called.

One habit follows from where the platform lives here. pioarduino reinstalls Arduino's libraries before a rebuild only in a folder whose path has .platformio in it, and the Docker volume is mounted at /pio: so each rebuild starts from the previous build's settings, not from Arduino's, and a line deleted from custom_sdkconfig keeps its old value. To change a setting, write its new value out; never just delete the line.

The libraries pulled in

LibraryPinned atWhat forNeeds
M5Cardputer1.1.1The board: starting it, the keyboardM5Unified, M5GFX, IRremote (fetched, not built in)
M5Unified0.2.26The board's start-up, the speaker, the IMU, the internal I2C bus (the radio cap's expander)M5GFX
M5GFX0.2.33The screen's driver and drawing; our widget kit is on it (ADR 0002)—
RadioLib7.8.1The SX1262 LoRa radio: the scanner and the meshArduino's SPI
esphome/wireguard0.4.8The WireGuard VPNlwIP, mbedTLS, libsodium (X25519)
LibSSH-ESP325.10.0The SSH and SCP clientsmbedTLS, lwIP, libsodium
esphome/libsodium1.10021.11X25519, for WireGuard's, SSH's and MeshCore's key exchanges, and libsodium's random numbers—

The pins on the M5 libraries hold back a known break. M5Cardputer accepts any version of M5Unified and M5GFX, and 0.3.0 of both draws the screen turned a quarter, so they stay at what the releases up to v0.25.0 were built with. The pin on libsodium is older than the fix below: 1.10021.12 defined crypto_sign_ed25519_open, as LibSSH's own Ed25519 did, and the two didn't link. LibSSH's copy is no longer built, so the pin may no longer be needed; raise it only with a clean build.

Two libsodiums

The firmware links two libsodiums, and the linker takes each function from whichever has it: esphome/libsodium (above) supplies X25519 and the random numbers, and Espressif's libsodium component, prebuilt with the framework, supplies Ed25519 (SSH keys, MeshCore's signatures). LibSSH carries its own copies of curve25519 and Ed25519 under the same names; scripts/libssh_filter.py leaves them out of the build, and LibSSH uses libsodium's, which are the same functions with the same key formats.

The libraries kept in the repository

  • lib/meshcore_upstream: MeshCore's protocol core, MIT-licensed, copied unchanged at a pinned commit (1.10.0, 81dbae5; ADR 0011). Upstream builds it with three more libraries (rweather/Crypto, orlp/ed25519, densaugeo/base64); its port/ folder gives the same names on what the firmware already has (our SHA-256 and AES-128, and libsodium), plus a small base64 decoder of its own. Only the protocol is used, not upstream's boards, radios or companion protocol.
  • lib/SD: Arduino-ESP32's SD library (Apache-2.0), which ours replaces by having the same name. One file is changed, sd_diskio.cpp: it gives the card the extra byte it needs to signal that it is busy, so a write that worked is no longer taken for a failure, and it records why a write failed (in sd_fault.h, added; ADR 0007).

The other folders of lib/ are our own code (the Gemini client, IRC, the mesh, updates, the widget kit and more). lib/meshtastic is our reading of Meshtastic's packet header, presets and default channel, for the LoRa scanner and the console's packet list: the mesh speaks MeshCore.

The tests and the tools

  • The host tests (native environment): Unity, built with the container's GCC. test/arduino_shim stands in for Arduino.h and Stream.h, which MeshCore's files include, and Debian's libsodium (libsodium-dev) is linked in where the device builds it from lib_deps. native-coverage builds the same tests with gcov and reports with gcovr.
  • Docker: the roro9stack-build image is python:3.12-slim with GCC (build-essential), PlatformIO, gcovr, git and libsodium. The platform, the toolchain and the rebuilt framework are cached in the roro9stack-pio volume, so building and testing the firmware needs nothing but Docker (Build, test and release). CI installs the same set itself in a plain python:3.12-slim.
  • CI runs on Gitea Actions. On a pull request: the checks, the host tests with their coverage, then the firmware. On a push to main: the checks, the host tests and the coverage badge. On a tag: the tests, then the firmware built clean and signed and published as a release (ADR 0008). A change to the site or the documents only runs the site's workflow.
  • This site is built by Zola 0.22.0, pinned by its checksum in CI. site/tools/gen_dev_docs.py writes its generated pages from the repository's documents, src/main.cpp and the key tables; they are committed, and CI fails when one is out of date. site/tools/check_site.py checks the built site.

Where it is fragile

  1. The rebuilt framework. Much of the memory work sits on pioarduino's hybrid compile, a community project, and it has bitten more than once: settings the settings tool dropped without a word (the dynamic TLS buffers, while DTLS was on), settings that stayed after their line was deleted (now all written out), legacy names in the generated defaults that put Arduino's value back (now given under both names), and link flags that ignore a setting (the Bluetooth script and the TLS wraps).
  2. ESP-IDF 5.5's dynamic TLS buffers. A server that splits one handshake message over several small records overflows the heap and restarts the device, and so can anyone between the device and a server, such as a hostile Wi-Fi network. ESP-IDF fixed it on its master branch; no release branch has it yet. The owner accepted the risk (ADR 0006). The three servers found doing it are refused by name (lib/net/src/tls_blocklist.cpp), which keeps honest servers from restarting the device but does nothing against an attacker. A second flaw, a record header split over two TCP segments, drops the connection (an open ESP-IDF pull request). When a pioarduino release brings an ESP-IDF with the fix, the list can go.
  3. The M5 libraries. M5Cardputer leaves their versions open, and they move fast: 0.3.0 broke the screen. Raise their pins on purpose, and look at the screen.
  4. MeshCore. It is copied unchanged, so updating it is a copy, not a merge: copy the files its README lists from the new commit, read upstream's changes, check that port/ still covers what the files include, and run the host tests; test_meshcore_upstream holds the port to upstream's own results.