It said "No"

TL;DR

• The last post ended with three things as "next". All three are in: a help key (v0.13.0), a shell on the device (v0.14.0), notes of any size (v0.15.0).
• Fn+h lists the keys of the screen you're on, and every hint line is gone. The same lists make the key tables on this website.
• CI went from over seven minutes to about one for a pull request. It had been rebuilding the whole framework at every run because of one file that isn't in git.
• The Shell runs the firmware's commands on the device: Tab completes every word and paths on the card, rm behaves like Unix's and asks first, * and ? work.
• The editor opens any file. A 1.2 MB note uses the same 17.5 KB as a 62-byte one, saves in 4 KB pieces, and is rewritten in 2.6 s when you leave it. A power cut at any byte leaves the note or the last save, never something in between.
• This site publishes itself when a change is merged, through an SSH key that can do exactly one thing.
• A test script sent the word "No" to an IRC channel. That one can't be fixed, only prevented.
• 507 host tests, 39 more than last time.

The cast

Fn+hthe help key, on every screen

Lists the keys that work where you are. Every screen had a line at the bottom doing that, differently and never completely. Those lines are gone.

The Shellan App, since v0.14.0

The commands I had been typing from a PC over Wi-Fi, on the device's own keyboard. It took more than one try to decide what it should show, and one crash to decide where its commands run.

The test scripttypes keys over the Debug Console

Tireless, exact, and with no idea what is on the screen. It typed the right letters. The App under them had changed.

The window8 KB of a note, around the cursor

All of a note that is in memory. The rest stays on the card, described by a short list. It moves when the cursor nears its edge, and nobody is meant to notice.

<note>.editthe side file

Where a long note's changes wait, four kilobytes at a time, until the note is left and rewritten. Also what a power cut leaves behind, on purpose.

One key instead of a hint line on every screen

Every screen had a line at the bottom: Enter open d delete r rename. Each was written by hand, each was different, and none had room for everything. The screen is 240 pixels wide.

So: Fn+h, everywhere, and ? wherever you aren't typing text. It opens a panel over the App, titled with where you are, listing that screen's keys and then the ones that work everywhere. Any other key closes it.

The hint lines went, all of them, with one exception: the first-start Setup keeps its own, because someone in their first minute doesn't know the help key exists. It tells them on its first and last screens.

The lists started as code inside each App. They are now data in one file, 52 small tables, and the same script that builds the developer docs reads that file and writes the key tables in the user guide. CI fails if the site's copy is out of date, so the guide can't list a key the firmware doesn't have.

The framework that was rebuilt every time

A pull request took over seven minutes to check, and a release thirteen and a half. For a firmware that builds in 77 seconds on my machine.

Where the time went, for one pull request:

roro9stack rebuilds the Arduino framework with its own settings, for smaller TLS buffers. The rebuilt libraries were sitting in the runner's cache the whole time. But the build system decides whether they still match by reading a file in the project folder, and that file is generated: it isn't in git. Every fresh checkout had no such file, so every run concluded the libraries were stale and rebuilt them. 260 seconds, each time, to produce what was already there.

The fix is to keep that file with the libraries it describes. Two more things came out of looking:

• The version was a -D flag on every compiler command line. Every commit changes the version, so every commit recompiled every file, on my machine too, and no cache could ever have helped. It's now one generated header that one file includes.
• A release built the firmware twice: once to check it, once to sign it.

With a build cache for pull requests on top: 27 seconds for the firmware with one file changed, and about a minute for the whole run on the real runner. A release takes under three minutes, and still compiles its own sources from nothing: no published file contains an object built for another commit.

A shell, and what it should show

The Debug Console's commands are the tool I use most, and they needed a PC. The Shell is an App that runs them on the device.

The first version was an afternoon's work and wrong in three ways.

It showed too much. The firmware prints all the time: IRC connecting, a packet heard, whatever a PC on the USB port is asking for. Version one showed everything printed in the ten seconds after a command, on the theory that the reply would be in there somewhere. It was, among everything else. The fix was to stop guessing: the console now knows who each line is for. A command run from the Shell prints as the Shell's, and so does an answer that arrives a second later from another task, which notes who asked. Ctrl+b shows everything, for when that's what you want.

rm was dangerous in a new way. Over the console, rm <folder> had always removed the folder and everything in it, which is fine for a script and less fine for a thumb on a small keyboard. It's now Unix's: a folder needs -r, and in the Shell it asks unless you say -f.

Tab did one word. It now follows the firmware's own help text, word by word: lora st becomes lora status, gnss track lists start stop. The words are read from the help text as it is written, so a new command completes without anyone maintaining a table. Past the command, it completes paths on the SD card. And * and ? work in file names.

One more addition, small and my favourite: an App's name with a capital letter opens it. Notes, Irc, Storage. Every command is lowercase, so the capital is the whole syntax.

It said "No"

The Shell's first version ran each command from inside the key handler. I test the UI by sending key presses over the Debug Console, so the call chain was: the main loop, a remote command, a key, the App manager, the Shell, the command interpreter a second time, the file command, and printf under all of it. The main loop has under 2 KB of stack to spare. rm on a folder went past it.

The device crashed, and restarted, as it should. It came back up in the Launcher.

My test script didn't know. It had a list of keys to send and it sent them. Its next Enter, meant for the Shell, landed in the Launcher and opened the first App in the list. That is IRC, which connected, as it's configured to, and joined its channels.

A few lines later the script reached its test of the capital-letter feature: type No, press Tab to complete it to Notes, press Enter. Tab completes nothing in IRC. Enter sends.

One word, to a channel of real people, from my nick. Not harmful, not explainable either, and not something any commit can take back.

Two things changed that afternoon:

• The Shell hands its line to the main loop, which runs it at the same depth as any console command. The crash is gone, and the crash report had decoded to exactly that chain of calls.
• info reports the App in front, and the test helper checks it before every line it types. A script that survives a restart is typing somewhere else, and now it stops.

The rule I'd had since the day before was "take a screenshot before any key that deletes something". It was the right rule for the wrong failure. Typing is also an action.

A megabyte in 17 KB

The Notes editor held the whole note in memory and stopped at 16 KB. That was a limit for the first version only; F1's notes say so in bold.

The device has no spare memory to throw at this, so the design is the old one from editors that ran on less: the note is the file on the card, plus one window in memory. The window is about 8 KB around the cursor. Everything else is a list of pieces: "bytes 0 to 40,000 of the file", "then 9,000 bytes of what was typed". When the cursor nears the window's edge, the window is written away if it changed, and the next one is loaded.

What makes it usable is what it writes, and when:

Memory with a note open is 17.5 KB, for a note of 62 bytes or of 1.2 MB. Going to the end of the megabyte takes as long as any other key.

The part that has to be right

An editor that loses text is worse than no editor, and this one now has a side file, a temporary file and the note itself, any of which can be half written when the power goes. So the rewrite ends with a mark: once the complete new file is on the card, one small write to the side file says "done". Before that mark, the old note and its side file are the truth. After it, the new file is, and whatever was interrupted is finished the next time the note is opened.

That is a claim, and it's the kind I don't trust until something has tried to break it. Two tests do:

• A power cut at every 997th byte of two saves and a rewrite. After each, the note has to be one of exactly three texts, the one that was reported as saved has to be there, and no stray file may be left.
• 36,000 random keys on six notes, typing, deleting, moving, jumping, saving and cutting the power, compared with a plain string after every key.

They pass. But the first run had three failures, and they're worth a line each, because only one of them was the editor's:

1. I had worked out by hand where the cursor should be after a recovery, and got it wrong by four.
2. I had assumed windows would break between groups of three characters in my test text. They break between characters, which is all they promise.
3. A file replaced by a shorter one lost its pending edits without a word. The design says they are set aside as .edit.lost and the editor tells you. The code checked the pieces against the new file's length first, found them out of range, concluded there was nothing valid to keep, and deleted them. A real bug, in exactly the path that exists to never delete typed text.

Two of three were the test being wrong. The third is why the tests exist.

What I had promised, and what I measured

I'd said the rewrite would take about two and a half seconds a megabyte. The first build took 3.5 to 4.5 seconds for 1.2 MB. It was copying in 2 KB blocks. With 4 KB blocks it takes 2.6, about 450 KB a second, which is what this card gives a plain copy.

A site that publishes itself

Until this morning, publishing this site meant logging into the web server and running a script, by hand, after every merge.

Now CI does it, after a merge and after a release. The interesting part is what the key in CI is allowed to do, which is one thing:

from="<the runner>",restrict,command="/path/to/rororefresh.sh" ssh-ed25519 AAAA… roro9stack-ci

My first plan had the command as a secret in CI, next to the key. With a forced command there is nothing to keep secret: CI connects and sends no command at all, and a stolen key can refresh the website and do nothing else. The server's address, the user, the key and the server's host key are secrets; the repository is public and none of them is in it.

Checked against a throwaway SSH server before the real one: asking for id; cat /etc/passwd runs the refresh. No terminal. scp copies nothing. A different host key stops the run.

On its first real run, the live page changed fifteen seconds after the run started. This post got here that way.

The device was right

A pattern from the day, three times over.

The keys that vanished. Twice, the first key my script sent after a quiet minute did nothing. The F1 notes describe that exact bug, fixed. I wrote it up as a possible regression. It isn't: a key that wakes a dark screen only wakes it, same as on the real keyboard. That's documented, on a page of this site, which I wrote.

The letters in the wrong place. In the editor test I typed two letters at the top of a note, moved 400 lines down, typed four more, fetched the file and compared it with what I expected. It differed: liMID ne 000400 where I expected MID line 000400. The file matched the screen exactly. Moving down keeps the cursor's column, as it should, and I had typed two letters first.

The "No". The device did what it was sent. Every key arrived, in order.

Three times the instrument was right and the reading was wrong. The remote key command now takes ctrl-, alt- and shift-, by the way, because checking the editor's jump to the end of a note needed Ctrl, and "the remote key command can't send that" had been in the not-checked list of every milestone since the editor existed.

What I didn't check

• The real keyboard, for Fn+h, ?, Ctrl+b and the Alt scroll. Everything was driven by remote keys.
• The power button's path in the editor, which writes the side file only, and the screen turning off.
• Memory with IRC connected and a long note open. After the morning, I didn't connect IRC.
• A file of tens of megabytes.
• Renaming or deleting a note from the Storage App leaves its side file behind. The Notes App handles both.

By the numbers

Where it stands

The last post promised three things and this one delivers them, which would be a tidy story if a script of mine hadn't said "No" to a room of strangers halfway through. The device did nothing wrong all day. It crashed where I had written a crash, restarted as designed, and typed what it was sent.