How do I test firmware code for my embedded system that also controls hardware peripherals?

"How do I test" tends to be the wrong question, since the immediate follow-up question to that is always "what are your requirements?". It can't really be answered without knowing the requirements.

Regarding testing in general:

Tests are often divided in two categories: black box testing vs white box testing. Black box testing is external tests that treats the DUT as a whole unit and tests how it fulfills things like environment requirements, EMC and so on. Whereas "white box"/"unit" testing tests internal components of the DUT separately, one at a time, to see if they function as expected. You need both kinds of tests and some black box testing could also be left behind after development in the form of production tests.

And naturally you would preferably have ways to test the hardware as well as the software. Designing the hardware with testing in mind is recommended, especially for advanced products. You might need to add test points for pogo pin test rigs etc. I know you are specifically asking about firmware, but on embedded systems hardware and software go hand in hand. If you for example expect 5V idle voltage on the UART pins and test this in a test rig and get 0V, is the problem bad soldering, a broken IC or some firmware bug? It could be any of these and as far as the test goes, the cause doesn't matter, as long as the fault is caught.

The most important thing about any embedded testing in general is that it ought to be carried out on the specific hardware. Partially because of the MCU ISA which is very different from some generic PC, partially because there is always hardware present which is highly relevant to the test. This means that embedded firmware cannot be meaningfully unit tested without the given hardware and therefore various "test suites" from the PC world are more or less utter garbage, to be instantly dismissed.

One way to go about "white box" testing is the buzzword "TDD" (Test-driven Development") where you write unit tests for the firmware at the same time as you write the firmware. That is, the tests are integrated in the driver source and activated/deactivated with some #define switch.

The main advantages of TDD is that the firmware gets written so that it is easily testable, and that you write the tests at the same time as you have everything about the unit still fresh in mind. Disadvantages is that it clutters down the code and can add artificial limits to the product such as increased need for memory or timing.

In safety-related products, the general rule is that no code which isn't actually executed in the live product is allowed to be present. So TDD with the tests integrated in the source is unacceptable.

One may then keep the tests entirely separated from the firmware and personally I like to handle that through version control. I create a separate "test repo" which is different from the normal firmware repo, but contains a copy of the source. It can still be TDD but you keep the sources separated.

In safety-critical firmware, there is an ideal design like this:

• All source code in the project must exist to fulfill a specific project requirement.
• There may exist no source code in the project which cannot be traced to a requirement.
• All tests must exist to see if the specific piece of firmware fulfills the requirement it was written to fulfill.

This means that the requirement specification, the source code and the tests get directly connected to each other and all of them may need to be continuously updated during development. For example if there is a need code for which no requirement exists, then perhaps the specification needs an update. If you come up with a great stress test, then maybe that one should be mentioned in the requirements as something that the DUT must handle and if it doesn't, then maybe the firmware needs an update too.

For non-critical firmware the above is utopia and rarely something you manage 100%, but it could be the goal to strive towards. In particular it is important that the specification remains a live document, or otherwise it might turn outdated or partially obsolete as the product is developed.

The classic mistake is that there is one person in charge of the specification, typically a project manager (who may not even be an engineer!) who writes it all down in the beginning of the project and then the specification just sits there statically like some dead weight document. In reality there are always details which you only find out when you start to implement the product and the requirements likely need to be updated accordingly. (And that's without even mentioning confused customers causing significant requirements creep.)

Another similar classic mistake is to develop the whole product then dump it on some test engineer who is then expected to come up with some sensible tests. That's just wrong - testing ought to have been considered much earlier and so the job of the test engineer shouldn't be to come up with tests, but rather to question if the tests that the developer has proposed are sound, to flesh them out further and then carry them out. Also, if it is impossible for the test engineer to do nothing but black box testing, then how do we know if appropriate white box testing was even carried out? Typically there is always some white box testing done during development, but it is not necessarily documented anywhere. I can drop way too many RL anecdotes about this, it is something that goes wrong very often.

There's research indicating that the most common errors in safety-related firmware isn't plain bugs, but failure to adapt the product to the intended environment. That is: incomplete requirements missing out use-cases or environmental considerations. And this would be why good engineers are those who are potty-trained all the way from back in school to always think outside the box, to always question the specification. And the contrary: those who just do as they are told without any critical thinking are not true engineers.

Now for your specific project, in order to come up with sensible tests, you always have to start with the requirements. Set the requirements and then make tests for them. Some relevant ones for UART might be:

• Baudrate accuracy tolerance. How much may the project deviate from the specified baud rate?

• Real-time throughput. How often does data arrive and how much data over time is the DUT supposed to handle?

• UART overflow and framing error situations. How should the DUT handle these? There are lots of nasty test cases you can apply here, like feeding the UART a plain square wave with the same baud rated as the DUT. Or force-feed the DUT repeated data as rapidly as possible.

• Data corruption on UART protocols. What is acceptable in terms of 1-bit errors, 2-bit errors, burst errors?

  (You can write a simulator on the PC injecting such errors into the packets, at least as far as the payload is concerned. But in the real world, bit errors may as well also hit the start/stop bits.)

• Grounding. How exactly is the signal ground integrated and where is it connected to supply and/or chassis grounds? What will happen if it goes missing? What happens if we inject noise on the shield/chassis? Things like that.

• EMC. You'll have external requirements from some standard about EMC (susceptibility and radiated) and these will dictate how the UART signals need to be designed. Do you need hardware filtering, differential signals (RS485), shielded cables, ESD protection etc etc.

posted about 1 month ago

CC BY-SA 4.0

1mo ago

Lundin‭

1951 reputation 37 73 323 89

If the main function is logic and activation of on/off signals, and the number of output combinations and their timing needs are not prohibitive, The minimal test with actual mcu hardware, actual relays, and indicator lights to make visible the relay output. You could also use an oscilloscope with digital inputs, or any gpio device to record the outputs in more detail. You'd have to arrange whatever inputs needed to exercise all your code.

Big agreement with Lundin's answer about doing not only black-box design verification testing (ie as opposed to production testing, where you take for granted the design is robust if executed to plan), but also subsystem verification testing, since it's just so common that black-box-only misses failmodes that could've been elicited at subsystem level. Beyond that you're entering into the world of DFMEA, which will make your designs better but also tremendously laborious.

Design verification of the electronics seems to take a different pattern from that of the firmware. In both cases, besides the actual function of the device, the real challenge lies in demonstrating robustness, which can be idiosyncratic in both firmware and hardware, but in different ways.

Implicit functions of hardware, IME, seem to be dominated by the pattern "withstands X Y Z". For an industrial setting, there are also common weak points related field wired hook ups (ie when your test equipment gets integrated into an industrial automation system on a production line, which commonly have field-wired terminal strip type construction in the control boxes). Special attention should be paid to these miswire scenarios, including multiple failures (ie wires exchanged, line of connections shifted down on a terminal strip).

When faced with this in the context of "operational qual" for a factory test equipment of a product which has with 24V "high speed" plc signaling, I took the time to generate all possible combinations of miswiring in its 8p connector (in that case, the miswiring originating from production errors in the internal wire harness of the DUT), identify the stress cases, reduce them for symmetry (ie since there were multiple output lines of identical design), which resulted in qualitatively unique stress-cases and non-stress DUT-test-fail-cases that numbered slightly more than the number of connections, something like 10 vs 8p connector in that particular case. We then constructed physical specimens of the "bad" DUT's with each of those miswirings, to demonstrate that the test equipment design survived them, caught the failure, and kept going normally -- and this was repeated for several specimens of the test equipment, each vs all specimens of the "stress-case-generating" DUT, as well as non-stress specimens of "good DUT" and "bad DUT" representing the known test-fail-cases of the DUT (in that case not reduced for symmetry, since it was an operational qualification as well as a design verification - but do reduce if multiple DUT-production-failmodes result in the same production-test-failure, because the purpose of the OQ was to show the test-equipment function, which is catching the DUT-test-fail-case, regardless of which DUT-production-failmode caused it.

To back up a little bit, the take-home of this might be that you should be clear (ie we're talking spreadsheets here) about:

• what are the functions of the DUT, and of the test-equipment
• distinguish between design-failmode and production-failmode, of both the DUT and test-equipment
• aggregate DUT-production-failmodes into DUT-test-fail-cases
• have a systematic process to identify design-failmodes connected to implicit functions, which are most often functions which can be verbalized in the form "withstands X Y Z"
• special attention to "withstands defective DUT stress-cases" and "withstands miswire in the context of the test-equipment being integrated into field-wired industrial automation"

Returning to the firmware, you can apply the same principles, but attempting a generative approach with spreadsheets (systematically identifying failmodes), which is a huge chore but at least doable for the hardware, gets out of hand entirely for nontrivial firmware projects. The possibility space includes all the known weaknesses of the C language, for starters. The "CERT" guidelines can be starting point there.

Previously mentioned difference in emphasis in implicit requirements between firmware vs hardware. For firmware, it's not as much dominated by the "withstands external-condition-X" pattern. Although there's some of that too, e.g. improper string input to the UART. But you also have things like not crashing (test: clear-on-reset variable and watchdog timer), not having unbounded memory use (test: stack paint and guard, best-practice: no heap, possible practice: periodic resets), not corrupting your data in a million ways that C lets you do, not using uninitialized data, etc etc etc.

But those above are all one-offs and and I haven't yet worked on anything with a comprehensive approach, since for better or worse the quality people I've encountered have all been too scared to touch the software and firmware side of things, and basically left the geeks to their own devices.

Other helpful mentalities to reduce the combinations of the unpredictable in firmware, are the principles of functional decomposition (helps testability), encapsulation (reduces cross-interactions), and actual functional programming (guarantees testability in principle) even if you can't realize it across the board due to the nature of the platform/language/stateful system. But it should be possible to structure the code so that like 90%+ is written in a style that facilitates verification and behaves consistently when parts of the system not directly connected to it are changing.

Also since you mentioned it, mechanical relays can introduce electrical transients which if you're just starting out in designing this stuff, they can stress both the relay's drive circuit, and the relay contact themselves can be overstressed by certain loads and are vulnerable to the contacts spot-welding themselves closed-circuit ... but that's its own subject.

posted about 1 month ago

CC BY-SA 4.0

1mo ago

Pete W‭

450 reputation 9 13 60 203

The first thing to do is to create a test program on the PC. Actually, that should have been created along with implementation of the command set in the microcontroller.

My usual workflow is to set up the command processor in the micro, create a template test program on the PC, and start the firmware doc file. Then to add each new command:

1. Think out the command, then add its description (which includes the protocol) to the firmware doc file. This then becomes the reference for what the command should do and what the interface at each end of the communication line should look like.
2. Add the command to the list of commands, with opcode, to the firmware.
3. Add the command routine to the firmware, and of course any supporting facilities it may need in the firmware.
4. Build the firmware, which also creates the include file of command and response opcode mnemonics in whatever language you are using for the PC test program.
5. Add a suitable command to the PC test program. Note that commands the user enters into the test program are distinct from the binary commands the PC sends to the micro. In your case, I might add two commands to the test program called ON and OFF. Each would be followed by a number for the particular relay to be switched on or off. When the test program gets these commands, it emits whatever commands are required to the micro to get it to perform the specified action.
6. Carefully test the remote system by controlling it from the test program, making sure what you expect to happen actually happens. Try to break it. Try all the corner cases you can think of. Send it out of range values. Make sure everything is handled correctly.

Having a test program that can cause specific commands to be sent to the micro is also very useful in debugging. You can set a breakpoint in the firmware at the command in question, then see why it doesn't get executed properly. Set the breakpoint after the command routine has gotten all parameters from the UART, but before it does anything with that data. Once you single step, the UART won't work anymore.

My test programs are basically command interpreters. When run, they prompt the user for a command to enter. I always have the three commands "?", "HELP", and "Q", at minimum, in addition to commands that cause communication with the firmware. "?" and "HELP" do the same thing, which is to list the available commands with brief descriptions, one line each. "Q" quits the test program.

posted about 1 month ago

CC BY-SA 4.0

Olin Lathrop‭

9528 reputation 19 423 1122 516