Laravel queue jobs not processing: wrong connection

If your jobs table is growing while the queue worker sits there reporting healthy, the two processes are reading and writing different queues. Laravel lets you set the connection in two places, and the one in the worker's command line silently wins over the one in your environment file.

Ours disagreed for nineteen days. It cost 55,470 orphaned jobs, every transactional email in that window, and — the part I find hardest to defend — nobody noticed, because from every angle the system looked fine.

What we saw

We were auditing a production database before an unrelated migration when a count came back wrong:

SELECT COUNT(*) FROM jobs;
-- 56588

Fifty-six thousand pending jobs. The oldest was nineteen days old. The newest was thirty seconds old and there was a fresh one every thirty seconds, forever.

The queue worker container had been up for weeks. Its logs showed a clean supervisord boot and nothing else — no errors, no warnings, no processed jobs. Every health check passed. The application served traffic normally. Customers were completing orders.

The arithmetic told us where to look before the configuration did. One job class accounted for 55,470 of the rows, and the scheduler dispatched it every thirty seconds:

2 per minute × 60 × 24 = 2,880 per day
2,880 × 19 days ≈ 54,720

That is not a job failing and retrying. That is a job being enqueued perfectly and never once being read, since the day the scheduler entry went live.

The two places a queue connection is set

The environment file said one thing:

QUEUE_CONNECTION=database

The worker's container definition said another, hardcoded months earlier and never revisited:

SUPERVISOR_PHP_COMMAND: "php /var/www/html/artisan queue:work redis
  --queue=high,default,low --sleep=3 --tries=3 --max-time=3600"

That first argument to queue:work is the connection name, and it overrides QUEUE_CONNECTION completely. The behaviour is documented and genuinely useful — it is how you run separate workers against separate backends. It is dangerous only because it is set in a different file, in a different repository concern, from the value it overrides.

So the application dispatched jobs into MySQL. The worker polled Redis, found nothing, slept three seconds, and polled again. It did that several million times without complaint.

Why nothing logged an error

This is the part worth internalising, because it generalises far beyond queues.

Nothing failed. Dispatch wrote a row and returned success — that is what dispatch does. The worker blocked on an empty list and returned success — that is what polling an empty queue does. An error requires some component to attempt something impossible, and neither component ever attempted anything impossible. Each half was working correctly. The system was broken only in the relationship between them, and nothing in the stack is responsible for that relationship.

Every monitor we had was pointed at a component. Container health: passing. Error rate: zero. Database: fine. Not one of them was pointed at the contract.

Why the damage was smaller than it should have been

The dominant job class was an outbox recovery job — a safety net that re-publishes events whose delivery was not confirmed after the transaction committed. So we checked what it would have had to recover:

SELECT status, COUNT(*) FROM outbox_messages GROUP BY status;
-- sent  1668

Every row sent. Zero pending. The primary publish path had a perfect record for the entire window, which is why nineteen days of a dead safety net produced no visible symptom.

We got away with it. Read that sentence as the accusation it is: we did not detect the failure, we were rescued by the fact that the thing it protected never needed protecting. Had the primary path faltered once during those nineteen days, the recovery mechanism would have been sitting in a MySQL table watching it happen.

The rest of the backlog was less lucky. Several hundred registration notifications and booking-status emails were in there. Those never sent, and nobody filed a ticket — which tells you something uncomfortable about how much of that mail anyone was reading.

Clearing it: why we did not just release the backlog

The instinct on finding 56,000 stuck jobs is to point a worker at them and let them drain. We deliberately did not.

Those jobs were nineteen days stale. Releasing them would have delivered hundreds of "welcome, you've registered" emails to people who registered three weeks ago, and status updates for rides that finished long before. Delivering a stale message is a worse outcome than never delivering it, and unlike the silent failure, customers would definitely have noticed that one.

So: group by class, decide per class, back up, then discard.

mysqldump --single-transaction app_prod jobs failed_jobs | gzip > jobs-backup.sql.gz
TRUNCATE TABLE jobs;

The backup makes the decision reversible for the cost of a few megabytes. Take it even when you are confident, because the confidence is about the job classes you identified, not the ones you skimmed past.

Then the fix, which was one line — pointing the application at the connection the worker had been watching all along. Jobs began clearing in two to four milliseconds each.

The same bug is not equally dangerous everywhere

Here is the detail that changes how you should weight this. We fixed the mismatch as part of moving the queue onto Redis, and that move altered the failure's blast radius entirely.

In MySQL, 56,000 orphaned jobs were a large table on a disk with hundreds of gigabytes free. Genuinely harmless — which is precisely why it survived nineteen days.

On Redis, the identical bug consumes memory, and queue entries carry no TTL. They sit there until a worker takes them. On a shared box without swap, unbounded memory growth does not politely degrade; it reaches a limit and something gets killed, and the process the kernel selects is chosen by size rather than by blame — frequently your database rather than the cache that caused it.

Same misconfiguration, same silence, radically different consequence. Moving a queue to a faster substrate also moves it to a less forgiving one. If you are making that migration, fix your queue-depth monitoring first, not afterwards.

You cannot alert on a metric that does not exist

The obvious follow-up is an alert on queue depth. We went to add one and found the gap was one level deeper than expected.

The Redis exporter reports totals — memory, client count, keys per database — but it does not publish the length of any individual list unless you name it explicitly:

REDIS_EXPORTER_CHECK_KEYS: "queues:*"

Until that line existed there was no queue-depth metric in Prometheus at all. A dashboard would have shown a healthy Redis for all nineteen days, because every metric it displayed was genuinely healthy. An absent metric and a good metric look identical on a graph.

The alert we wrote was wrong, and testing caught it

A healthy queue drains in milliseconds, so any depth that survives a long window means nothing is consuming it. That was the rule:

min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

We tested it by planting a synthetic backlog. It went pending, correctly. Then we deleted the key — and it stayed pending.

min_over_time keeps returning samples from its whole window after a key disappears. So any burst that got scraped once would fire this five minutes later and hold it for half an hour: exactly the false-positive noise that teaches a team to ignore alerts. Pairing it with a check for a currently-present sample fixes it, because an emptied Laravel queue deletes its Redis list and the series simply stops:

redis_key_size{key=~"queues:.*"} > 10
  and
min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

Re-tested: pending with a backlog, inactive the moment it drains. An alert is a piece of production code, and an untested one is likelier to erode trust than to protect anything.

What to check on your own system

Two commands, worth running now rather than during an incident. What the application believes:

php artisan tinker --execute="echo config('queue.default');"

And what the worker is actually executing — read the process, not the config file that you believe produced it:

docker compose exec app_queue ps aux | grep queue:work

If those two disagree, you have this bug, and your logs will not tell you. Then confirm something is genuinely draining, rather than that nothing has arrived: a depth of zero and a broken consumer look the same from outside.

The principle

Health checks verify components. This failure lived in the space between two healthy components, where nothing was looking, and the only honest signal available was a number that nobody was collecting.

For any handoff between two processes, monitor the queue between them rather than the processes themselves. Depth over time is the cheapest true statement you can make about a distributed system: it goes up when the producer outruns the consumer, and it stays up when the consumer is gone. Neither of those facts is visible from either end alone.