OneVariable Newsletter - 2021-12-29

This article was originally featured on the OneVariable Applied Research newsletter. If you'd like to get early access to these reports, you can sign up for the newsletter here. Newsletter emails are sent every Wednesday.

Articles are posted on my blog one week after appearing on the newsletter.

Abstract and Results

It has been start-and-stop this week, with a little bit of down time for celebration with Christmas Nachos, and struggling with some hardware drivers, but bootloading over USB now works!

As the bootloader already uses the Anachro network protocol over USB, this means that supporting the bootloader over the RS-485 network should be relatively straight forward to implement.

Now that a debugger is no longer a hard requirement, working with multiple devices at once becomes MUCH easier!

Happy little progress bars

See the animated progress on Asciinema.

Bootloading over USB

Getting the bootloader working was my main focus this week, which included:

• Fixing and de-macro'ing the network stack implementation
• Writing the bootloader network protocol
• Implementing the async boot client
• Implementing the boot server
• Debugging the QSPI issues

Next up is getting RS-485 re-integrated, and making sure everything works as expected. This means I need to assemble some more boards!

Background Information

This week, I really struggled with the QSPI hardware on the nRF52. The network protocol work went surprisingly smoothly, but I was beating my head against a wall to figure out why the bootloader code was able to interact with the QSPI flash successfully, but I kept getting corrupted data in the application code.

I tried a LOT of things, ranging from lowering the interface speed, to adding retries, to adding all kinds of verification steps. However, the issue was still elusive.

Finally, I broke out my logic analyzer, and resorted to sniffing the full QSPI connection, as well as using some of the pins from the expansion header to output additional context information.

After tediously reviewing the signals in the logic analyzer, I realized that my corruption was that the QSPI hardware was always sending three garbage bytes before the rest of the firmware image! This was confusing, but I quickly remembered that the QSPI hardware requires you to send and receive 32-bit (word) aligned data, and after some quick debugging additions, I realized that I wasn't (always) upholding that guarantee.

The bootloader had gotten lucky and the source buffer WAS 4-byte aligned, but no such luck in the application code.

A seven line fix to a three day problem

Powerbus Mini under close observation

This was due to the fact that I was sending data directly from the deserialized network packet buffers, which meant that the firmware data might be at an odd offset, in one case starting at memory address 0x20003FB3.

Rather than causing some sort of error or crash, the hardware was instead copying data from address 0x20003FB0, the next lowest 4-byte aligned address, which included three other bytes from the network message and omitted three necessary bytes from the end.

The fix for this was simple: Define a special 4-byte aligned data structure, and copy the network data there first before issuing a QSPI command. With this in place, everything started working perfectly.

This was frustrating, but an important reminder to implement ALL of the robustness checks (like checking for pointer alignment) early on in the driver development process, which could have caught this small error much faster.

With this bug conquered, I'm back to making smooth progress.

Suggestions for Future Research

The path forward is pretty clear at this point, I need to:

• Assemble 2-3 more Powerbus Mini boards, so my first board has some friends to talk to
• Re-integrate the RS-485 based communication
• Test the bootloader over the network

I'll also probably want to implement:

• Sending logs over the network
• A logging server for decoding the logs
• Some kind of non-volatile settings
• Start integrating the A4 scripting engine

Not much else to say, I'm excited to be unblocked and to make more progress.

A quick request!

I'd super appreciate it if you could share this newsletter with other folks who are interested in hardware, tooling, or embedded Rust.

Sending it along in your favorite Discord or Slack channels, on Twitter or other social media, or just forwarding the email would help me out a lot!

If they are interested in subscribing (or if you had this forwarded to you), they can subscribe to the newsletter here! I also post these newsletters a week later on my blog, which has an RSS feed you can also subscribe to.

Thanks again for subscribing and following along :)