The 7-Year Bug That Took 3 Minutes to Fix
A story about persistence, a bored forklift driver, and why some bugs only show up in the real world.
The 7-Year Bug That Took 3 Minutes to Fix
I’ve been an engineer for over 20 years. I’ve debugged race conditions at 3am, traced memory leaks through million-line codebases, and stared at oscilloscopes until the waveforms haunted my dreams.
But nothing prepared me for the bug that took 7 years to find and 3 minutes to fix.
The Setup
At ENA Tecnologia, I worked on a 180 Hz railway AC generator inverter. This was railway safety equipment — the kind that keeps trains running and passengers alive. There’s no “it mostly works” in this world.
The device had redundant electronics that monitored certain conditions. One key safety check: if there’s no commander signal, there should be no output. If something looked wrong, the device would short-circuit its own power supply to blow its own fuse. Kill itself to protect the system. That was the safety strategy, and it was the right one.
Except sometimes it triggered when it shouldn’t.
The Problem
Every few months, we’d get a report: “The fuse blew again.”
We knew it was the safety mechanism triggering incorrectly. But here’s the thing — we could never see it happen. We could never reproduce it.
I tried everything:
- Different input conditions
- Different load configurations
- Temperature variations
- Timing edge cases
- Left it running idle for days
Nothing. The bug was a ghost.
But the reports kept coming. So every three months or so, I’d pull out the documentation, review the schematics, try something new, and fail again. Then I’d put it back in the drawer until the next report.
This went on for seven years.
The Breakthrough
Then one day, the factory was dying. The company was running out of projects, running out of work. People were idle, waiting to see if the doors would close for good.
One of our lab technicians was bored. And when lab technicians get bored, interesting things happen.
This guy decided he was going to do something different. He grabbed the forklift, went to the warehouse, and came back with an entire spool of cable — about 400 meters of it. He connected the whole thing to the device in the lab.
BAM.
The safety mechanism triggered instantly.
He called me right away: “Hey Pedro, I reproduced it!”
I practically ran down to the lab.
The Fix
With the bug finally reproducible, I could see exactly what was happening. The long cable created enough inductance that it was coupling noise into the safety detection circuit. The redundant electronics saw this noise as a fault condition and did exactly what they were designed to do — disconnect.
The fix? A larger capacitor to filter the noise to ground.
Three minutes of soldering.
Seven years of searching.
The device has worked flawlessly ever since.
What I Learned
1. Real-world conditions are different from lab conditions.
In the lab, we used standard test cables — maybe 5 or 10 meters. In the field, 400 meters of cable is completely normal. But who’s going to drag a full spool into the lab? It’s just cable, right?
2. Sometimes the fix is trivial. Finding it is the hard part.
A capacitor. That’s all it was. The engineering was simple. The detective work took 7 years.
3. Boredom can be productive.
If that technician hadn’t been bored, if the factory had been busy, we might still be chasing that ghost. Sometimes the breakthrough comes from someone doing something “stupid” that nobody else thought to try.
4. Persistence matters, but so does luck.
I didn’t give up on this bug for 7 years. But I also couldn’t have solved it without that forklift and that spool of cable. In engineering, you need both.
5. Most people aren’t trained to think past 5 minutes.
A manager at that company taught me something that stuck with me: some problems take one second to solve — they’re obvious. Some take 10 seconds, 30 seconds, maybe 5 minutes of real concentration. And then there are problems that take an hour. A day. A week. Months. Years.
Most people are trained to think up to about 5 minutes. If they don’t have a solution by then, they assume it’s impossible. But the really complex, high-reward problems? They don’t yield to 5 minutes of thought. They require persistent thinking — coming back to the same problem over days, months, years.
If you’re only ever giving problems 5 minutes before giving up, you’re never going to do anything powerful. You need to be a stubborn motherfucker. That’s what it takes to solve the real hard stuff.
This is the kind of problem I love — the “impossible” ones that everyone’s given up on. If you’ve got a ghost in your system, let’s talk.