Developing Reliable Embedded Devices

A recurring theme in my last three jobs has been the automation of hardware and hardware software integration testing. During long software development cycles (years), it is critical that features that are considered "done" or "stable" do not unexpectedly regress. During short software development cycles (months), it is critical to never spend time where it isn't necessary. This is true for Safety Critical devices (like airplanes and gas detectors), as well as Consumer Products and IoT devices.

Typically, I have seen the following behavior too often:

• Hardware is developed, and either a hand-built or small batch production is made
• Hardware engineers test that the device or subsystem turns on without letting out the "magic smoke"
• Firmware engineers are brought in to make special software to validate the hardware (blink lights, send serial data, etc)
• Software developers work for months and months, occasionally doing a manual test of only the feature they are working on
• At some point a full test is done by engineers or test technicians
• If something is not working, there is now a mad scramble to figure out what broke, and where. Sometimes this could be anywhere in 5 engineers' work across 3-6 months. Good luck.

Not all development is like this, and there are a few things that can greatly reduce schedule slips and unexpected engineering effort, at the cost of very little extra planning:

System Planning

1. Make an electrical block diagram BEFORE designing anything. This will help hardware, software, and testing
2. Use state diagrams wherever possible. This is harder for large systems, but invaluable for rapid embedded development. Finite State Machines and Hierarchical State Machines can greatly simplify development, and replace or reduce Requirements Documents

Hardware Development

1. Iterative hardware development. Don't design a complete system and expect it to work the first time
2. Make development hardware first. Big and segmented PCBs are okay!
3. Use PCBs instead of breadboards whenever possible. You never know when a wire is going to come loose

Software Development

1. First step should always be a testing firmware. This should be separated from the regular firmware to reduce complexity, and can serve as your rough draft
2. Second step should be a software test environment (unit tests, CI). Its not necessary for your testing firmware, but once you start writing business and application logic, that needs to be tested
3. Write down what the software should do, from a high level. This could be a state diagram, some requirements, or even just some bullet points that you can expand later
4. Document TODOs. Not everything has to work at first, but it should be easy to find what is still missing or hard-coded.
5. Keep the Bus Factor low. At least two people should know how each part works, either by developing together, reviewing each others work, or testing each others work.

In the future, I plan to expand on these, and explain how these can work together with Automated Hardware Integration Testing, typically called ATE (automated test equipment), HIL (hardware in the loop)

Interested in more? Let me know on twitter @bitshiftmask, or send me an email!