Your Celery dashboard has never woken you up

Think about the last time you found out a Celery queue was backed up. Really picture it. Odds are it wasn't your monitoring that told you. It was a customer, or a coworker in Slack asking why their export hadn't shown up, or you noticing by accident. Then you opened the dashboard, and there it was, obvious in hindsight: the throughput line flat on the floor for the last ninety minutes.

The dashboard had the data the whole time. It just had no way to tell you, because a dashboard only works when a human is looking at it, and nobody looks at a dashboard at 2am.

That's the core problem with treating a dashboard as monitoring. A dashboard is pull: you have to decide to go look. An alert is push: it comes and finds you. Async failures are, almost by definition, the ones that happen while no one is watching. A synchronous request that breaks throws an error in someone's face right away. A background job that stops running fails in the dark, and the only question that matters is how long it stays dark before something drags you to it.

Dashboards are especially weak at the slow silent failure. One worker out of eight stops consuming, and your aggregate throughput drops maybe twelve percent, which reads as ordinary jitter on a graph. A backlog grows at a hundred tasks a minute, and for the first twenty minutes it's a gentle upward slope nobody would glance at twice, and then it's a wall. By the time the shape on the chart looks alarming, you're already inside the incident. Graphs are built to make trends legible over hours. The outages that hurt tend to announce themselves in minutes.

And then there's the honest reason dashboards miss things: you stop looking at them. Everyone does. You build a gorgeous board the week you launch the service, you check it obsessively for a while, and then it turns into wallpaper. Six months later half the panels are quietly broken and no one has noticed, because no one has looked.

So here's the opinion, and I'll stand behind it. A dashboard you have to remember to check is not monitoring. It's decoration with good production values. Monitoring is the thing that interrupts you when your belief that everything is fine turns out to be wrong. If it can't interrupt you, it isn't doing the job, however many panels it has.

Which means the number actually worth optimizing isn't on the dashboard at all. It's the time between "something broke" and "a human's phone buzzed." Speed to alert. Everything else, the graphs and the ninety days of retention, is forensics: genuinely useful once you already know there's a problem and you're trying to understand it, useless for finding out in the first place.

None of this means burn the dashboards. They earn their keep for capacity planning, for the post-incident reconstruction of what actually happened, for the slow seasonal drift that no single alert would ever catch. That's real value. It's forensic value. The mistake is putting a forensic tool on the front line and calling it defense.

That gap, between the moment something breaks and the moment you find out, is the entire reason Kanari exists. But the exercise below is worth doing whether or not you ever install a thing.

Answer one question honestly. Over the last few months, who noticed your async problems first, your tooling or a person? If the honest answer is "usually a person," your dashboard is doing forensics and you have no defense. Fixing that doesn't start with a nicer graph. It starts with writing down the three or four conditions that should drag you out of whatever you're doing (a worker that's stopped consuming, an oldest task that's been waiting past the latency you actually promised, a backlog draining slower than it fills) and wiring each one to something that pushes. Today, before the next 2am.