How Slaygent Processes 10,000 Companies in 5 Minutes

Slaygent is an agentic research engine for company data. Give it a domain and a question, and it searches the company's website, follows the most relevant pages, renders JavaScript when necessary, and uses an LLM to return a written report. A single report usually takes seconds. The harder problem is doing that for an entire spreadsheet at once.

Our first architecture needed roughly three hours to process 10,000 companies. The system described here brings the same workload to a final state in about five minutes. That improvement did not come from making one research task 36 times faster. It came from changing how thousands of tasks are scheduled, executed, and recovered in parallel.

The result is a queue-driven engine in which no machine owns a research task from beginning to end. During destructive load tests we kill workers while paid work is in flight; unfinished steps return to the queue, another worker continues from the last completed step, and the run finishes without restarting the research chain. This article explains the architecture behind that behavior and the production problems that shaped it.

Why async, why queues

A research task is a loop, not a straight line:

• Scrape the homepage
• Ask a model what's missing and where to look next
• Scrape the pages it points to; escalate the ones that fail to a real browser
• Ask the model again
• Repeat until the model can write the report

Each round costs money (paid LLM calls) and time (a user is waiting), and most of it is spent waiting on external things: a slow website, a browser render, a model generating tokens. That puts a hard floor under single-task latency - you can tune the pipeline forever and it barely moves. Meanwhile, a user rarely sends one URL; they send a spreadsheet of them. The only lever left is parallelism: run everything asynchronously, many tasks at once, and get throughput from concurrency instead of speed.

Parallelism is what creates the durability problem. At load there are hundreds of paid tasks in flight at any instant, and the slowest remain active for minutes. Deployments, autoscaler scale-downs, and out-of-memory terminations can interrupt the workers carrying them. In a design where one worker owns a job from first step to last, each interruption restarts the complete research chain and repeats its paid LLM calls. Slaygent instead treats worker loss as a normal recovery path for unfinished steps.

The usual options

• Retry the whole job on redelivery. Fine when jobs are cheap and few are in flight - here it's the double-billing problem above.
• Split into many small independent jobs. Works for resizing ten thousand images; breaks the moment steps depend on each other. Research steps do - each model call decides what gets scraped next. Something has to remember the plan, and if that something is a process, its memory dies with it.
• Rewrite as an explicit state machine. Works, but turns a readable procedure ("scrape, then triage, then analyze") into a pile of handlers, so every future change costs double.
• Adopt a workflow engine. Temporal and similar tools solve this properly with durable execution. If you already run one, use it. We didn't want a second distributed system for one problem.

What we wanted was the technique inside those engines, without the engine.

A task is just a checklist

In an ordinary worker, "where are we in this job" lives in a call stack and local variables - invisible to other machines and gone the instant the process dies. Our engine keeps that state in Redis: every task, every step, every intermediate result is shared state any machine can read. A worker holds nothing durable - it reads state, does one bounded piece of work, writes state back.

The mechanism:

• Every expensive step (a scrape, a render, an LLM call) gets a deterministic ID derived from the task and the step's exact inputs
• Before doing a step, the pipeline checks Redis: is the result already there?
• If yes - use it and move on, instantly
• If no - send a request to the fleet that handles that kind of work, then stop. The pipeline exits completely and the worker moves on to other tasks
• When the step finishes, the task gets requeued and some worker - any worker - runs the pipeline again from the beginning

This sounds wasteful but isn't: finished steps resolve instantly from cache, so a replay costs milliseconds and lands exactly on the next missing step. The task is just a checklist - any worker can pick it up, see what's done, and continue.

Two rules keep replays correct:

• Durable effects are idempotent. Queues can redeliver messages after a crash. Step results use a first-writer-wins commit, while credit charges and refunds use task-level idempotency keys. External work remains at-least-once: if a worker dies after a provider responds but before the result is committed, that call can be repeated, but its durable result and the user's credit transition are not duplicated.
• Replay depends on deterministic control flow. No random sampling, wall-clock branching, or unordered iteration feeds step inputs. Nondeterministic work runs inside a step, where its output is cached, or is pinned into the task inputs at submission. We enforce this in code review as a correctness rule.

Deploys require compatibility discipline because an in-flight task can replay under newer code. A cached result is reused when its deterministic step ID remains unchanged; a change that must invalidate an old result therefore has to change the step's identity explicitly. We keep changes compatible with in-flight tasks where possible and use task-level idempotency keys for credit transitions. This limits repeated work, but it does not provide the workflow-versioning guarantees of a dedicated engine such as Temporal.

An orchestrator that owns nothing

A fleet of pipeline workers runs the replay loop, and that is the whole orchestrator. Because it holds nothing between steps, no replica is ever in charge of a task, and any of them can be replaced at any moment.

Around it, the engine is event-driven end to end. Specialized fleets - scraping, browser rendering, LLM calls - consume requests, and finishing one piece of work is itself the event that triggers the next: the browser fleet writes its result to the step's address and requeues the task for whichever orchestrator replica is free. No session stickiness, no handoff protocol.

If a worker dies mid-step:

• Every claimed step carries a short lease the worker renews while it works
• A periodic sweep re-issues steps whose lease has lapsed - a live lease blocks the sweep, so it only takes work from dead workers, never slow ones
• Steps that keep failing land in a dead-letter queue for a human to inspect, with the user's credits refunded automatically

The same recovery path applies when a replica is terminated during scale-down.

Scaling decisions use workload pressure rather than CPU utilization alone. Worker fleets measure Redis stream lag, pending messages, and active leases; API and browser capacity also use request and submission velocity so replicas can start before downstream queues fill. Each fleet scales independently because its resource profile is different: browser rendering is memory-intensive, LLM calling is network-bound, and direct HTTP scraping is comparatively cheap. Every fleet also keeps a small warm floor to avoid cold-start latency at the beginning of a burst.

The numbers

We validated throughput with a load test of 10,000 URLs, each submitted as a complete research task as fast as the API would accept it.

What that turned into:

• ~16,000 page scrapes, following the model's requests for more pages
• ~4,300 browser renders - the one-in-four that hit JavaScript walls
• ~24,000 LLM calls, two to three per task
• More than four downstream operations per submitted URL

The results:

• All 10,000 tasks reached a final state in about five minutes - over 30 tasks per second sustained
• Median ~10 seconds from submission to finished report; the tail stretched into minutes for the hardest sites
• Roughly 300 tasks active at any moment, continuously exercising the worker-recovery paths
• The API served tens of thousands of requests with zero errors at ~15 ms median - it only accepts work and answers polls; everything heavy is behind the queue
• Every work queue drained by the end; failures (almost entirely sites that could not be scraped under load) were terminal and auto-refunded, with no tasks left indefinitely processing

In a separate destructive test, we terminated general workers during research and browser replicas during active renders. Expired leases caused unfinished steps to be reissued, and the run still reached terminal states. Coordination overhead remained small: Redis, holding every queue, lease, and step result, stayed below one-third of a CPU core throughout the run.

Problems that only show up in production

Production load exposed three issues that led directly to configuration or implementation changes.

1. 502 errors during deploys

A trickle of 502s that only happened around deploys. The platform's router counted an API replica as alive the moment its container started - but for the first seconds, the process is still importing code and opening connections; nothing is listening on the port yet. Requests routed there bounced as 502s, and since they never reached the application, the application logs were clean: users saw errors while every service reported healthy.

The fix: a health endpoint, plus routing traffic to a replica only after it answers. A replica joins the pool when it can prove it's serving, not when its container exists. Deploys stopped showing up in the error rate at all.

2. Handling SIGTERM correctly

When the platform retires a replica, it sends SIGTERM and a short grace period before the kill. Our first behavior in that window was wrong twice: the dying replica kept claiming fresh work right up to the end, and it had claimed ahead - consumers read in batches, so it held a pile of claimed-but-not-started messages. In a consumer-group queue, claimed messages belong to the claimer; other replicas can't take them. So on every deploy, that list froze until the sweep's timeout - tasks that normally finish in ten seconds sat stuck for minutes. Technically correct, bad in practice.

The fix has three parts:

• Stop claiming instantly on SIGTERM - the signal flips a flag the consume loop checks before every read
• Never claim more than you can run - no more messages than free execution slots, because anything beyond that is a message you might strand
• Drain and exit - finish what's actually in flight; if the grace period runs out first, the replica just becomes an ordinary dead worker and the sweep handles it

Workers drain on SIGTERM when the grace period permits; lease recovery handles forced termination.

3. Congestion and timeouts

Under load, slow URLs piled up in the browser fleet: every render slot held by a page that had been loading for a minute or more, healthy pages queued behind them. The first instinct was more architecture - a fast lane and a slow lane, so slow pages could only block each other. Then we looked at the data:

• p99 of renders that ever succeeded: ~22 seconds
• The pages holding the slots almost never rendered at all - even minutes of waiting returned nothing

So instead of adding another fleet, we changed the render timeout from 45s to 25s and the scrape timeout from 15s to 5s. The measured p99 for successful scrapes was under five seconds, while the successful browser-render p99 was about 22 seconds. The new limits nearly doubled slot turnover on the same hardware and removed the queue buildup.

The retry data also supported one attempt on a fresh browser worker: one page that had stalled for four minutes rendered in six seconds after reassignment. Separately, the direct HTTP layer classifies dead domains, refused connections, and hard bot walls before browser escalation, keeping roughly a third of those failures out of the browser queue.

When you should not do this

• Jobs are cheap and volume is low - just retry the whole job.
• The pipeline changes constantly across a large team - replay demands determinism from every future contributor; a workflow engine that enforces the rules mechanically may be the better buy.
• You don't have per-step tracing - you'll want it from day one, because work that hops between machines can't be debugged by tailing one log.

For a long-running pipeline with Redis already available, the minimum mechanism consists of:

• Deterministic step identities
• Replay against cached results
• Idempotent durable effects
• Leases, so dead workers get noticed

The engine runs in production at slaygent.co. Deployments and autoscaling do not restart active research chains: completed steps remain durable in Redis, and expired leases return unfinished work to the queue for recovery.