Hardware debugging: from first board to intermittent faults

A board that will not start and one that resets once a day look like different problems. They are tackled with the same method, which Keysight sums up in three steps in its notes on infrequent faults: confirm that the fault exists, isolate it from what works, and gather clues (when it appears, how often, under what conditions) until you find the cause.

Before applying power

In the article on seven checks before your first PCB batch we recommended writing a first power-up plan. In every board bring-up we do, it is the first thing we prepare. This is how to carry it out.

The first thing to look at is the power supplies. With the board unpowered, measure the resistance to ground at the input and on each rail, and compare it with a known-good board if you have one. Some rails read low at first and rise as the capacitors charge. If the input stays close to 0 Ω, investigate before applying power. It is the same check recommended by PCB manufacturers' guides such as JLCPCB's.

A good practice is to put ferrite beads on the supply of each block. They do more than help with electromagnetic compatibility: they also let you disconnect a block during the first power-up and bring the board up in parts.

The bench supply must be current-limited. The value depends on the design and on what you want to check at each step. If you do not know the consumption, start low and work up. The sign that all is well is that the supply stays in constant-voltage mode and the current settles where expected. If the voltage drops and the current hits the limit, disconnect. Then measure each converter and regulator output before connecting what it feeds: an output that is too high can damage the low-voltage circuits downstream.

Measuring without being fooled by the probe

Before blaming the board, rule out the measurement. The probe and its ground lead are part of the circuit you are measuring.

The ground lead

The probe's ground lead is an inductance. Together with the probe's capacitance it forms a resonant circuit that adds a damped oscillation to fast edges. The longer the lead, the worse it gets. A classic example, which Tektronix shows in its probe guide: the same pulse measured with the probe's short ground and with an extended ground lead, so as not to have to move it. The second measurement shows ringing that does not exist on the board. The advice is to always use the shortest ground possible, and not to replace the tip adapter with a few centimetres of soldered wire.

Bandwidth and loading

Tektronix gives two rules of thumb. To measure amplitudes accurately, the oscilloscope's bandwidth should be about five times the frequency of the signal. To measure edges, the combined rise time of probe and oscilloscope should be three to five times faster than that of the pulse, and it can be estimated as 0.35 divided by the bandwidth.

The probe also loads the circuit, mainly through its capacitance. For most measurements a 10X, 10 MΩ probe with 20 pF or less is enough. A 1X probe has considerably more capacitance, and fast digital signals may need an active probe. Compensate the probe every time you connect it or change the adapter.

Converter ripple

To measure the output ripple of a switching supply, TI advises resting the probe tip on the test point and the probe's own ground barrel on the ground point, with no lead. A ground lead forms a large loop that picks up noise.

Firmware as a witness

The microcontroller knows things about the board that the oscilloscope cannot see. On the first board, the test firmware should include a few debugging aids:

Debug access. On Arm® microcontrollers, SWD needs only two lines, data and clock, and gives access to all the debug features. It has to be reachable with the board assembled.
The cause of the last reset. Microcontrollers store it in a register: reset pin, watchdog, software reset, CPU lockup... On Nordic parts, if none is flagged, the reset came from power-on or a brown-out. Read it and store it at start-up, together with a reset counter. That way a reset nobody saw leaves a trace.
A log that does not halt the CPU. Techniques such as SEGGER's RTT leave messages in a RAM buffer that the debug probe reads while the program keeps running. According to SEGGER, without affecting real-time behaviour.
Commands to switch each block on separately. Together with the ferrite beads, they make it possible to isolate which part of the board causes odd consumption or a reset.

There is one clue that is almost welcome when it shows up: the fault disappears when you stop at a breakpoint or connect the debugger. It has happened to us, and it is good news, because it points to a timing problem. That is where a log that does not halt the CPU is most valuable.

Cornering the intermittent fault

A fault that appears once a day is not fixed by staring at it. First you have to be able to trigger it or, at least, capture it when it appears. It is a regular part of our test and validation work, including on products already on the market.

Log before changing anything

Note every occurrence: time, temperature, what the equipment was doing, how it was powered. The reset cause and reset counter from the firmware help a lot here. The pattern may show up in the log before it shows up on the oscilloscope.

Capture it with the oscilloscope

To wait for a rare event, oscilloscopes have advanced triggers, such as zone triggering on the screen, and segmented memory, which according to Keysight stores only the active stretches around each trigger, with a timestamp. That way you capture many events without running out of memory, and the time between them is already a clue.

Force it with temperature and supply voltage

If the fault depends on temperature, you can hunt for the area. Cooling small spots with freeze spray while the fault is present can make it disappear, which points to the component or solder joint. Spray manufacturers describe it as a thermal shock that opens cold or brittle joints. With gentle heat and hot-cold cycles, thermal faults tend to appear during the transition rather than at steady state. Freeze spray causes condensation: do not power a board that has got wet.

Also try the supply voltage at the bottom of its range. Two usual suspects:

Ceramic capacitors that have lost capacitance. Murata explains that X5R, X6S and X7R types lose capacitance relative to their nominal value when a DC voltage is applied, and C0G types do not. Decoupling that looks sufficient on the schematic may not be at the real voltage. Each part's curve is in its datasheet or the manufacturer's tools.
Voltage dips. If a current peak pulls a rail below the microcontroller's threshold, it resets on brown-out. The reset cause stored by the firmware gives it away.

What stays on record

Take notes during bring-up: what was measured, what was changed on the board and what is left for the next revision. In the prototype phase, that notebook is what turns a half-working board into the change list for the next iteration.

If you have a board that fails now and then and you do not know where to start, we can help you corner the fault.

From first power-up to validation:Prototyping and bring-up Test and validation