Debug Builds and the Debug Console · Console

The Debug Console

Connect to a Debug Build over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do.

Connect

export RORO_OTA_HOST=10.39.39.12   # or pass -H <ip>

scripts/rdbg.py            # interactive: the backlog, then live lines; type commands
scripts/rdbg.py info       # one command, and its reply
scripts/rdbg.py -b tasks   # the same, with the backlog shown first

scripts/rdbg.py is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It reads the token from ~/.config/roro9stack/debug-token and talks to TCP 2323 on the device's address. In interactive mode, Ctrl+D or quit leaves. Piped input works too, but rdbg.py leaves as soon as its input ends, before the replies arrive: keep the input open for a few seconds, as in (printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py. For one command, the one-shot form above is simpler.

The device listens only while Wi-Fi is connected, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.

What you get

On connecting, in order:

  1. a banner: roro9stack v0.11.0-3-g12006bd+debug debug console. 'help' lists the commands. Backlog follows.
  2. the backlog: the last 4 KB of console output, oldest first, boot messages included (a ring buffer in RAM);
  3. then every new line, live: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too);
  4. the replies to your commands, in the same stream, each preceded by the > command line when it runs.

A line status: heap <free> min <lowest> appears every 10 seconds in the stream: a free-memory trace you get without asking.

> net
net: IRC            in          0  out          0
net: Gemini         in          0  out          0
net: Debug Console  in       1038  out     205115
net: Updates        in       4788  out        204

If you are not reading fast enough (a slow link), the device says so in the stream instead of stalling: [... 312 bytes lost: the console ran faster than the network].

The protocol

It is a plain line protocol, easy to speak from anything. This is what rdbg.py does, and all it needs:

StepDetail
ConnectTCP 2323. One client at a time: a second one waits until the first leaves
AuthenticateSend the token and \n within 10 seconds. The comparison takes the same time whatever you send
RefusedAfter about a second the device sends denied\n and hangs up, and prints debug: refused a client from <ip> on its own console. No quick retries
CommandsOne line each, up to 240 bytes. The device queues at most 8; past that, debug: busy, command dropped
Leavequit or exit closes the connection
Binary commandsget, put, screenshot, coredump get: a text header line, then raw bytes (see files and screens)

A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about.

How commands run

your PC rdbg.py token first Debug Console task token check, one client ring → socket, live lines → command queue answers these itself: get · put · screenshot coredump get · reset (they work with the loop stuck) main loop runCommand(): the same commands as USB serial console 4 KB ring + USB serial, if there's room ESP-IDF logs (tee) storage task SD card jobs ls rm install get put TCP 2323 queue read printf jobs
The console task only queues command lines. The main loop runs them with the same code as USB serial commands. Binary commands are answered by the console task itself, so they work when the main loop is stuck.
  • Text commands run on the main loop, exactly like serial ones: they touch the Apps and the Services from the one task allowed to. The main loop prints > command as it starts, and the reply follows in the stream.
  • Binary commands run on the console's own task: get, put, screenshot, coredump get and reset. A failed put closes the connection, so the rest of the file is never read as commands.
  • That split is the point: with the main loop stuck (an infinite loop, a deadlock), text commands queue forever, but reset still restarts the device at once, coredump get still reads the dump, and get still reads files. A loop stuck for 5 seconds is a panic with a core dump anyway: see Crashes and Safe Mode.
  • Card work stays on the storage task: ls, rm, cp and install from the main loop, get and put from the console task, all as jobs the storage task runs, so the card is only ever touched from one place.
  • Writes to the console never wait for USB: a host attached to the USB port but not reading used to stall the main loop for up to two seconds per line.

Security

  • The token is checked before anything else, and a wrong one costs a second.
  • Anyone on the same network with the token can read the console, press keys and restart the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
  • The stream is plain text: fine on a home network, not across the internet. Do not forward port 2323.
  • Release builds have no console at all. Nothing listens.

The decision is ADR 0004.

Without rdbg.py

Anything that can open a TCP connection works. The token and each command are just lines:

import socket
s = socket.create_connection(("10.39.39.12", 2323))
s.sendall(b"<token>\n")      # then read the banner line
s.sendall(b"info\n")         # read until the stream goes quiet

scripts/rdbg.py adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.