97% of frontier-LLM quality at up to 1,000x lower cost

Tens of millions of texts. Thirty minutes. Under ten dollars. Ninety-seven per cent of frontier-model quality.

Not every problem needs a model that can talk. Some just need one that can decide, millions of times a day, quickly and cheaply.

Large language models are brilliant at understanding text. At high volume, they are also slow and expensive. When a workload grows to tens of millions of texts every day, the question stops being "which model is smartest?" and becomes "what does it cost, and can it finish before tomorrow's batch arrives?"

This is the story of a pipeline we built for exactly that situation: tens of millions of text pieces a day, about 400 words each, processed in roughly 30 minutes for under $10 a day on a pod of GPUs. The core is a fine-tuned ModernBERT model, with an LLM in the loop as teacher and fallback. Compared with sending everything to an LLM, it is around 100x faster, about 100x cheaper than the cheapest small LLMs and up to 1,000x cheaper than frontier models.

The problem: LLMs at this volume

At around 400 words per text, each piece is roughly 500 to 550 tokens. Twenty million texts a day is therefore about 10 billion input tokens a day, before counting any output.

Two limits make an LLM-only design impractical at that scale:

Cost. Even the cheapest small LLMs cost around $0.10 to $0.15 per million input tokens. Ten billion tokens is $1,000 to $1,500 a day on the cheapest models, and $10,000 or more a day on mid-tier and frontier models. Every day, indefinitely.

Rate limits. LLM APIs cap how many tokens you can send per minute, typically in the single-digit to low double-digit millions for a large account. Ten billion tokens at 5 million tokens per minute takes about 33 hours. The daily batch would never catch up.

	LLM only	Hybrid ModernBERT pipeline
Daily volume	~10 billion tokens	~10 billion tokens
Time to process	17 to 80+ hours, set by the rate limit	~30 minutes
Daily cost	~$1,000 (cheapest models) to $10,000+ (frontier)	Under $10
Bottleneck	API rate limits	GPU throughput, which scales horizontally

The idea: let the LLM teach, not do the bulk work

Most high-volume text tasks (classifying, tagging, routing, filtering, scoring) don't need a model that can write poetry. They need one narrow skill, applied consistently, millions of times. That is exactly what a small encoder model does well.

So we split the work:

The LLM is the teacher. It labels a representative sample of the data, following a written rubric.
ModernBERT is the workhorse. A compact encoder model is fine-tuned on those labels and processes the full daily volume.
The LLM is the safety net. Texts where ModernBERT is not confident are routed back to the LLM, using the API capacity the bulk work no longer needs.

We chose ModernBERT because it is fast, handles long inputs (up to 8,192 tokens, so 400-word texts are never truncated), and is efficient on modern GPUs thanks to techniques such as unpadding and alternating local and global attention.

The industry is reaching the same conclusion

In September 2026, TypeSafe AI released Jev, which it calls a "System One" model: instead of generating text, it returns typed values with probabilities and confidence scores, for fast decisions inside software. TypeSafe says this makes it up to 100x faster and 100x cheaper than conventional LLMs for suitable tasks.

It's a strong signal of where production AI is heading: for decisions at volume, the answer is a fast model that outputs a label and a confidence score, not a paragraph. The pipeline in this article applies the same principle with open models you can run today on your own GPUs, in your own cloud or region, trained on your own data and labels. For a fixed, high-volume task, that specialisation is what takes the savings from 100x towards 1,000x.

Step 1: getting the most out of the LLM rate limit

The LLM does two jobs: labelling training data, and handling the uncertain cases in production. In both, the rate limit is the scarce resource, so we treat it like one.

A rate-limit-aware scheduler. Requests are paced against both the tokens-per-minute and requests-per-minute limits, so the pipeline runs at close to 100% of the allowed throughput instead of bursting, hitting errors and backing off.
High concurrency with back-pressure. Many requests are in flight at once, with automatic retries and exponential backoff when the provider pushes back.
Right-sized requests. Prompts are compact, outputs are constrained to a short structured format, and several texts can share one request where the task allows it, which cuts overhead per text.
Batch APIs where latency allows. Labelling for training doesn't need instant answers, so it can use the providers' cheaper batch endpoints.
Deduplication and caching. Identical or near-identical texts are labelled once.

The result: every token of API capacity goes into useful work, and none of it is spent on the texts a small model can handle alone.

Step 2: training the ModernBERT model

A distilled model can only be as good as the labels it learns from, so most of the effort goes into the data, not the training loop:

Representative sampling. The labelled sample covers every category, source and edge case we see in production, not just the most common ones.
A written rubric. The LLM labels against explicit definitions and examples, which makes its labels consistent and auditable.
Human review of disagreements. Where the LLM is inconsistent, or two prompts disagree, people review the cases and the rubric is tightened.
A held-out golden set. A fixed evaluation set, reviewed by people, measures the fine-tuned model against both the LLM and human judgement before anything goes to production.

Fine-tuning itself is fast: a ModernBERT model trains on a single GPU in hours, not days, which makes it cheap to retrain whenever the data or the rubric changes.

Step 3: a GPU pod that finishes in 30 minutes

Throughput comes from keeping every GPU busy:

Data-parallel sharding. The daily batch is split into shards and spread across a pod of GPUs, each running its own copy of the model. Adding GPUs shortens the run almost linearly.
Length-aware batching. Texts are grouped by length and packed together, so the GPUs don't waste compute on padding.
Reduced precision. Inference runs in reduced precision, which substantially increases throughput.
Streaming I/O. Reading, tokenising and writing results overlap with GPU work, so the GPUs never wait for data.
Short-lived capacity. The pod exists only for the length of the run, then shuts down. You pay for about half an hour of GPU time a day, not for idle machines.

This is why the cost lands under $10 a day: a pod of GPUs for roughly 30 minutes, doing nothing but matrix multiplications on real text.

Step 4: confidence routing, the "hybrid" part

A small model will sometimes be unsure. Instead of accepting those guesses, the pipeline checks the model's confidence for every text:

Confident predictions (the vast majority) are accepted directly.
Uncertain predictions go to the LLM, which now has plenty of spare rate limit because it isn't doing the bulk work.
Every LLM decision on an uncertain case is saved as a new training example. The next retraining learns from exactly the cases the model found hard, so the share sent to the LLM shrinks over time.

The confidence threshold is a business dial: lower it for maximum savings, raise it for maximum quality. Either way, the cost of the LLM fallback stays small because it only sees a small fraction of the traffic.

Step 5: keeping it accurate in production

Distilled models don't fail loudly; they drift quietly as the incoming text changes. We guard against that with:

Continuous sampling. A small random sample of each day's texts is also labelled by the LLM, and agreement is tracked over time.
Alerts on drift. Falling agreement, or a shift in the distribution of predicted labels, triggers a review.
Scheduled retraining. New labels, including the hard cases from the fallback, feed a regular retraining cycle, validated against the golden set before release.

Quality: 72 versus 74

Speed and cost only matter if the answers are good enough. We evaluated both approaches on the same refined, masked and anonymised sample of production texts: personal and sensitive details were removed, and the labels were reviewed and corrected by people, so neither model could benefit from memorised data and the reference answers were trustworthy.

Metric	Proprietary frontier LLM	Our hybrid ModernBERT pipeline
F1 score	74	72

In other words, the pipeline delivers about 97% of a frontier model's quality (72 against 74) at a small fraction of the cost and time. For a daily process over tens of millions of texts, a two-point gap is a very good trade for a 100x to 1,000x reduction in cost, especially because the uncertain cases, where most of the remaining errors sit, can still be routed to the LLM.

Results

Metric	LLM only	Hybrid ModernBERT pipeline
Processing time for the daily batch	17 to 80+ hours	~30 minutes (around 100x faster)
Daily compute cost	~$1,000 to $10,000+	Under $10 (about 100x to 1,000x cheaper)
Quality (F1 on the evaluation sample)	74 (frontier model)	72
Scales with volume by	Negotiating higher rate limits	Adding GPUs to the pod
LLM rate limit used for	Everything	Labelling and uncertain cases only

Comparisons use public list prices for LLM APIs and typical rate limits for large accounts; exact savings depend on the model you would otherwise use, your volume and your contract.

Lessons learned

Start with the LLM, then distil. An LLM prototype tells you whether the task is solvable and gives you the labels to train with. Don't skip it; just don't stay there.
The rubric is the product. Most quality gains came from clearer label definitions, not from model tweaks.
Rate limits are a design constraint, not an afterthought. Plan how you will use every token per minute before you write the first prompt.
Measure against people, not only against the LLM. The LLM is a strong teacher, but it is not ground truth.
Keep the LLM in the loop. A hybrid design gets most of the savings of a small model and most of the quality of a large one.

When this approach fits, and when it doesn't

It fits when you have high volume, a well-defined task and a stable set of outputs: classification, tagging, routing, moderation, extraction of fixed fields, scoring or de-duplication.

It fits less well when every text needs open-ended reasoning or free-form generation, or when volume is low enough that an LLM's cost doesn't matter. In those cases, better prompts, caching or a smaller LLM are usually the right first step.

Frequently asked questions

Is a fine-tuned ModernBERT model as accurate as an LLM?

On a narrow, well-defined task it can come very close, because it learns directly from LLM and human labels for that one task. Routing uncertain cases back to the LLM closes most of the remaining gap.

How much labelled data does distillation need?

It depends on the number of classes and how subtle they are. Typically tens of thousands of good labels are enough to start, and the confidence-routing loop keeps adding the hardest examples over time.

Why not just raise the LLM rate limit?

Higher limits reduce the time problem but not the cost problem: at billions of tokens a day, even the cheapest LLMs cost hundreds to thousands of dollars daily, compared with under $10 for a GPU pod running a small model.

Can this run in our own cloud or region?

Yes. The model is small and self-hosted, so data can stay in your cloud account or on dedicated in-region GPUs, with only the uncertain cases (or none at all) sent to an external LLM.

How we can help

We design and build distillation pipelines like this one: from the LLM prototype and labelling rubric to the fine-tuned model, the GPU inference pipeline and the monitoring around it, in your cloud or on sovereign in-region GPUs. See the approach on our research page: LLM distillation up to 1000x cheaper.

The takeaway: use the smartest model to teach, and the fastest model to work. Keep the smart one on call for the hard cases.

Processing millions of texts with an LLM today? Book a consultation and we'll estimate what a hybrid pipeline would save.