Decisions · ADR 0006

The framework is rebuilt with our own SDK settings, for smaller TLS buffers

Arduino-ESP32 ships its ESP-IDF libraries prebuilt, with one sdkconfig for every ESP32-S3 board. Its TLS settings give every connection a 16 KB receive buffer and a 16 KB send buffer for its whole life. On a device with no PSRAM…

Arduino-ESP32 ships its ESP-IDF libraries prebuilt, with one sdkconfig for every ESP32-S3 board. Its TLS settings give every connection a 16 KB receive buffer and a 16 KB send buffer for its whole life. On a device with no PSRAM and about 340 KB of RAM, an IRC connection over TLS left a 12.6 KB low in M2, against a 40 KB floor.

Those settings are compiled into the libraries, so changing them means rebuilding them. pioarduino supports this as a "hybrid compile": custom_sdkconfig in platformio.ini lists the settings, and the build regenerates the framework's libraries from ESP-IDF (the same 5.5.5 the prebuilt ones come from) before building the app. We set:

  • MBEDTLS_ASYMMETRIC_CONTENT_LEN, with 16 KB to receive (servers send full TLS records) and 4 KB to send (IRC lines are short): 12 KB less per connection.
  • MBEDTLS_DYNAMIC_BUFFER, DYNAMIC_FREE_CONFIG_DATA, DYNAMIC_FREE_CA_CERT: buffers allocated when needed, and handshake-only data (the CA chain) freed once connected. Asked for from the start, but in the firmware only since 2026-11-20: see Later below.

The rebuild also follows the board definition instead of the generic one: PSRAM support is off (the Cardputer ADV has none) and the flash size is 8 MB.

Consequences

  • With IRC connected over TLS, a Debug Build has 78 KB free and a 59 KB low (it was 31 KB and 12.6 KB), and 46 KB at the lowest under the heaviest combined load measured (IRC, two refused installs, a 1.6 MB put and get).
  • The first build after a fresh checkout, or after changing custom_sdkconfig, takes about 4 minutes instead of 45 s: it downloads ESP-IDF into the PlatformIO volume and compiles it. Later builds reuse it.
  • Everything the firmware depends on was checked in the regenerated sdkconfig: app rollback, core dumps to flash (ELF), the 5 s task watchdog, FreeRTOS run-time stats, the certificate bundle. Not the TLS settings themselves, as it turned out (Later): since 2026-11-19 the build checks every one (scripts/check_sdkconfig.py).
  • The project now owns its partition table (default_8MB.csv, identical to the framework's), which the hybrid build requires. Changing it would break updates over the air: the app slots must stay where they are.
  • Generated files (sdkconfig.*, managed_components/, .dummy/) are ignored by git.
  • A TLS server that sends records over 16 KB would still fail, as before; one that needs us to send records over 4 KB would now fail. Neither happens with IRC.

Later (2026-11-19)

The architecture re-check found that the three dynamic-buffer settings were never in the firmware. ESP-IDF's Kconfig allows MBEDTLS_DYNAMIC_BUFFER only without DTLS, and Arduino's base configuration has DTLS on, so the rebuild dropped all three without a word; and even with them, Arduino's prebuilt link flags lack the --wrap=mbedtls_ssl_* that route mbedTLS through ESP-IDF's dynamic code. Every connection since this rebuild has had static buffers, 16.3 KB to receive and 4.3 KB to send (16,717 and 4,429 bytes), for its whole life. The figures above are true, and were measured on those. Of the rebuild's own step (54 → 78 KB free, 34 → 59 KB low, M2.md), 12 KB is the smaller send buffer; the rest was not attributed (likely the board's settings, such as PSRAM support off). The earlier steps of M2 (stacks, mDNS) account for the remainder.

Built for a test with DTLS off and the wraps added (never merged), dynamic buffers kept the low during a TLS handshake 24 KB higher (57.3 KB against 33.7 KB, from 93 KB free; an update check 55.2 against 28.0 KB), with the same heap at rest. But a server sending its certificate in 512-byte records (openssl s_server -max_send_frag 512) corrupted the heap during the handshake, and the device restarted (a panic in the heap allocator, under mbedtls_ssl_parse_certificate, called through the dynamic wrapper): mbedTLS 3.6 reassembles a handshake message split across records into a buffer the dynamic code sized for the first record only. The static build handles the same server. Any TLS server can do it, and the Gemini App connects to any, so dynamic buffers stay out until ESP-IDF fixes this; the three settings are gone from platformio.ini. (Reversed the next day: below.)

The same check found that the rebuild does not start from Arduino's base configuration: pioarduino's reinstall does nothing in this project's PlatformIO volume (its path has no .platformio), so each rebuild starts from the last one's resolved configuration, and a setting removed from custom_sdkconfig stays in the libraries.

Later (2026-11-20): dynamic buffers, the crash accepted

The owner chose the 24 KB over the crash: this is a hobby device. The dynamic buffers are in the firmware, with DTLS off (nothing here uses it: only NetworkClientSecure links mbedTLS's TLS, over TCP) and the twelve --wrap flags of ESP-IDF's mbedtls CMakeLists.txt in build_flags.

How often it would happen: a scan made TLS 1.2 handshakes, as the device does (its TLS 1.3 is off), and looked for a handshake message split over records. None of the device's own servers did (git.twis.la, Libera, the four capsules on the Gemini start page), nor 16 IRC networks, nor 57 large websites and APIs; 3 of the 2,808 Gemini capsules that answered, of Kennedy's known hosts, did, all in 512-byte records. Those three are refused by name, by the Gemini App and the tls command (lib/net/src/tls_blocklist.cpp), and a new one found goes on that list.

What is accepted: a server that splits its handshake overflows the heap and restarts the device. The bytes written past the buffer are the server's, and they arrive before its certificate is checked, so whoever sits between the device and a server (a hostile Wi-Fi network) can do it to any connection, the update check's included. A second known flaw drops a connection instead: a record header split over two TCP segments on a non-blocking socket, as Arduino's are, is lost (esp-idf PR #19410, open).

ESP-IDF fixed the first on its master branch on 2026-09-23 (commits c21756db34, sizing the buffer for the whole handshake message, 833142f655, refusing any read past the buffer, and cd23f4cbb5, an alert followed by a record), after mbedTLS 4: not in release/v5.5 or release/v6.0. When the framework brings them, the list can go.

The free-heap thresholds before a TLS connection (80 KB for an update check, 70 for tls, 60 for IRC, 55 for a Gemini fetch) stay as they were: an open connection now takes a block back for each record it receives, up to 16,749 bytes at any moment, so the memory a lower threshold would hand out is not reliably free. What changed with them: each also wants one free block of 17 KB (lib/net/src/tls_room.h), since a full record must fit in one piece; Gemini checks again before each redirect, a new handshake; an update check or install makes IRC step aside whenever it is connected, and the daily check waits while it is, by IRC's state rather than the heap (Q172), and counts its task's stack before it starts; HTTPS requests, update probe with them, refuse the blocked hosts too.

This page is generated from docs/adr/0006-framework-rebuilt-for-smaller-tls-buffers.md in the repository. To change it, change that file and run site/tools/gen_dev_docs.py.