Error handling in embedded systems
How do I handle errors in my embedded system? I have learned that it is a de facto standard for driver functions to return an error code that reveals if the function fulfilled its main purpose or if it encountered an error. If we take a look at the switch-case construct in Lundin's state machine for a coffee machine, we see that a special function error_handler() is called and takes the error code as input argument if the result is COFFEE_NO_WATER.
#include <stdio.h>
int main()
{
static coffee_state_t state = COFFEE_SCAN_BUTTONS;
coffee_result_t result;
for(;;)
{
result = state_machine[state]();
state = evaluate_result(state, result);
}
}
static coffee_state_t evaluate_result (coffee_state_t current_state,
coffee_result_t result)
{
// stay in the same state as default behavior:
coffee_state_t next_state = current_state;
switch(current_state)
{
case COFFEE_MIX_WATER:
{
if(result == COFFEE_NO_WATER)
{
next_state = COFFEE_MAINTENANCE;
error_handler(COFFEE_NO_WATER);
}
break;
}
}
return next_state;
}
static void error_handler (coffee_result_t error_code)
{
// do something meaningful here based on error code
(void)error_code;
}
My question is, what is error_handler() supposed to do? What does its innards look like? Is its purpose to make visible to the user/programmer which errors has happened? Or is its purpose something else?
To this end, I might add that I have understood that there are at least two kinds of errors - "soft errors" and "hard errors". A soft error could be that an opcode is received correctly via UART, but it is an operation that our microcontroller program does not recognize/support. Since the UART connection is still intact we can send a message to the other end (which might be a PC) stating what went wrong, giving a clear indication that an error happened.
A hard error could be that the UART connection disconnects or that baud rates are not correctly configured. In such instances, how should the error be handled?
2 answers
You are accessing this answer with a direct link, so it's being shown above all other answers regardless of its score. You can return to the normal view.
In real world microcontroller systems there is very little "error handling". There are two types of things that can go wrong, with each handled differently.
Expected conditions that prevent normal operation
These are things like a bad checksum on a command packet, the resistance between the electrodes is too low, a consumable has been depleted, etc. Each of these should have been considered in the original design and the response thought out.
In the case of a bad checksum, you usually ignore the whole packet as if it never happened. On some systems you might keep an error counter if there is a way to eventually get that counter to a user or a service technician. You might also detect how often it happens and declare an error condition when it happens too often. Again, that only makes sense if there is a means to alert the user to the problem. You might blink a red LED, for example. However, in many cases the only available answer is to ignore the command. If your system is a black box, then it doesn't respond when you send it garbage.
If shorted or depleted electrodes, for example, are expected to occur in the field, then some user interface for that should be designed into the system. Often just lighting an LED or blinking one with a specific pattern is good enough.
Actual failures
You can never "handle" all failures. For example, if the power supply breaks, the processor might get fried or not run at all. In the majority of cases, a broken unit acts broken. The user either replaces it or calls field service to have it fixed or replaced. You can't design for all contingencies.
There are an infinite number of possible failures, and detecting and handling each one adds cost. Of course the more critical the operation of the system is, the more cost is worth spending on failure detection and handling.
For a mass-market high-volume product that most customers buy on price, like a toy for example, any extra failure detection is probably not worth it. If it fails, the user will just toss it.
At the other end of the spectrum when safety is critical, you can spend multiple times the cost of the basic unit to reduce failures, detect them when they occur, and try to fail in the most safe possible way. In such cases there are usually standards that the customer requires the product to be designed to. The answer then is to do what the standard says.
Summary
Except for safety-critical systems, you detect and handle the few error conditions the system was designed for, and then just do the best you can with anything else. Bad checksum? Ignore the packet. Voltage too low? Stop operating. Voltage too high? You'll never know because the microcontroller is toast.
0 comment threads
The core idea in that design is that the program shouldn't have error handling all over the place, and it should also not make important state decisions all over the place. Those two things should both be centralized somewhere and they are often related. How to design the internals is highly project-specific.
In the linked example I had moved the decision maker outside the error handler, but it could as well be integrated into one function. All of it depends on how complex the application is, how many states and errors there are etc.
An error handler typically needs to do 3 things:
-
Evaluate errors reported.
Is this error critical or something that can be ignored, or maybe somewhere in between? If the error handler is the only code "weighing" the errors, then we do not need to rewrite large amounts of code when regarding one kind of error as more/less serious than previously. -
Take action accordingly.
Is there something that needs to be stopped or aborted? Should the program revert to a safe state? Are there functions for that, which resets variables etc? Or in some cases the proper action might be the reset and reboot the MCU itself.Since the error handler sits at the top of the application tiers, it will have access to most drivers and modules etc.
-
Report the error somewhere.
Maybe post it on some bus or display, maybe log it in memory, maybe lit some LED etc etc.
In a system where you also have a state machine of some sort, you can integrate change of state with the actions that might be taken.
It might be wise/necessary to also let the error handler know where the error is coming from, especially when the same kind of error can occur in many different places.
For example if you have a safe state, then the only place where you can revert from it is inside the error handler. Then there is only one place in the code where errors are evaluated, and only one place where the state changes happen.
Take the alternative design: if there are 10+ places where errors are reported and state changes happen in those 10+ places, then maintaining the code becomes burdensome. If we decide that "error x" isn't critical, we would have to track down all those places reporting "error x" and rewrite how the program reacts to that error.
Having error handling all over the place also tends to clutter down the code significantly and distract a lot from what the code is actually doing when it functions as it should.
Not to mention the most serious problem with that design: the program changes states and execution path from all over the place - aka spaghetti/"stateghetti" programming. Similarly, it is also spaghetti programming to have a big collection of various global status flags that are changed from all over the place, and which the program acts upon all over the place. It becomes very hard to follow the program flow and as complexity grows, the program flow can even start to take unpredicted ways.
One metric sometimes used when speaking of bad code is "cyclomatic complexity", which is basically the number of paths any given code can take given certain circumstances. The more it increases, the harder the program flow becomes to follow, let alone to prove that all possible execution paths are actually taken, or that all use-cases that the program should cover are actually covered. Complexity always leads to more bugs too.
If we can instead split up decision-making, program flow, error handling and the actual algorithms in different places, everything becomes so much maintainable than if all of that is merged together in one big unmaintainable blob.
One sign of code smell is where every driver ends up with some function named "handler", which... handles stuff... often in vague ways. Could be error handling, could be error reporting, could be state changes, could be calls to unrelated parts of the program. When the programmer who wrote the code can't easily summarize what a certain function does, then that's a bad sign. That function should most likely be removed and replaced with centralized code.
0 comment threads
1 comment thread