Your Celery queue depth alert is lying to you

Your alert fires. celery queue depth > 1000. Quick: is that an incident?

You can't answer, and neither can anyone else, from that number alone. A thousand tasks that each take three milliseconds drain before you finish reading the alert. A thousand tasks stacked behind one wedged job that's been running for ten minutes is a full outage in progress. Same number, opposite worlds, and depth can't tell them apart.

We alert on queue depth because it's the easiest thing to measure. LLEN on the list, compare to a threshold, done. But easy to measure and worth measuring are different properties, and depth is the Celery metric that's easiest to graph and easiest to be wrong about.

Depth only turns into latency when you divide by throughput. That's the whole trick, and it's Little's Law in work clothes: the time a task waits is the number ahead of it divided by how fast you drain. A queue 1,000 deep draining at 500 per second is a two-second wait. The same 1,000 draining at 5 per second is over three minutes. The depth is identical. The experience your users get is not. A depth threshold is really a latency threshold with the throughput term deleted, and you're quietly betting throughput never changes. It always changes.

It breaks in the other direction too, and that direction is worse because it's the silent one. A single worker ghosts and stops consuming. Depth barely moves, because the other workers keep draining. But the oldest task in that queue is now sitting there aging, and aging, and no count-based alarm will ever fire, because the count looks fine. Depth is blind to exactly the failure you most want to catch.

The signal that doesn't lie is the age of the oldest waiting task. Time in queue. It answers the question depth only gestures at: how long has the task that's been waiting longest actually been waiting? That maps straight onto whatever SLA you promised, it needs zero knowledge of your throughput to mean something, and it climbs the instant something stalls even while the count stays flat. If you get to alert on one number, make it that one.

The other pair worth watching is arrival rate against drain rate, because that tells you where you're heading instead of where you are. If tasks arrive faster than you clear them, the backlog grows without bound and no amount of depth-watching will save you. The honest ETA to empty is the current depth divided by the gap between the two rates, and if drain sits at or below arrival, that ETA is infinite. Depth is a snapshot. The two rates are the trajectory.

There's one more reason to distrust the number. Celery workers prefetch. With worker_prefetch_multiplier at its default, each worker reserves a batch of tasks beyond the ones it's actually running, and on the Redis transport those reserved messages leave the visible queue and sit in an unacked limbo. So your LLEN can read low while a pile of tasks is reserved-but-not-running inside a worker that may or may not be healthy. The count you're alerting on isn't even the whole count.

I'll say it plainly: queue depth is the most-alerted, least-useful metric in a Celery stack. It feels like monitoring because it's a number that goes up when things are bad. But it also goes up when things are fine, and it stays flat through some of the worst failures, and a signal that's loud during health and quiet during failure is worse than no signal at all.

If you want to stop being lied to, the fix is cheap. The tasks already carry the information. Peek at the oldest message in the queue, read the timestamp it was enqueued with, and alert when that age crosses the latency you actually promised someone, instead of when a count crosses a round number you picked because it looked big.

Measuring that oldest-task age continuously, per queue, is a good chunk of what Kanari does. You can also wire a crude version of it yourself this afternoon, and you probably should.

Either way: depth can stay on the graph for context. Just take it off the pager.