Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline.
Tim Koopmans recently discussed what typically goes wrong with real-time AI at scale, sharing hard-fought lessons learned while building an ML-based financial trading app — and how to avoid falling into these traps yourself, including the practices and infrastructure choices that can help.
Why AI Performance Fails at Scale
You Can’t Dig Yourself Out of Tail Latency
All too often, latency looks fine in testing, then a P99 spike surfaces under real concurrent load. For example, as Tim’s app approached ~740K operations per second, its P99 latency skyrocketed to 3 seconds.
“I kept blaming the model for being slow, but it turns out the model was fine,” Tim explained. “It was just that the feature lookups were killing me.” Each inference call was doing just a handful of reads, but those reads queued up behind writes under load. The average latencies seemed fine, but that P99 tail latency was unacceptable.
“Tail latency isn’t a bug that you can fix, it’s a property of your architecture.”
Once you hit highly concurrent write throughput, you get lock contention — and that impacts tail latencies. At this point, retries, bigger caches, and connection pool tuning don’t help. As Tim put it, “Tail latency isn’t a bug that you can fix, it’s a property of your architecture. For example, if your storage engine is producing GC pauses at exactly the wrong moment, you’re going to cop a latency spike, no matter what.”
The culprit in Tim’s app was Postgres under pressure: “It’s not a slow database, but it was just a database being asked to do too much in this particular case.”
Stale Features Kill Accuracy
If you notice a mysterious accuracy drop that the model itself can’t explain, feature freshness might be the problem.
For Tim, this issue was particularly frustrating. User profile staleness was blowing past a five-minute SLA target by hours, vector embeddings were going stale, and offline evaluation metrics looked fine the entire time. As Tim put it, “You have this maddening situation where offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”
“Offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”
When the model was in production, it started making calls that didn’t track. After spending considerable time debugging the model, it turned out to be fine. The problem was that the model was making decisions based on old data — garbage in, garbage out.
Vector Indexes Need Maintenance
No matter what vector database vendors imply, “set it and forget it” isn’t a realistic strategy for embeddings. Every re-embedding pass rots the index a little more, whether you notice it happening or not.
Tim hit this too. He was re-embedding content every time he improved the model, and the index quality rotted a bit more with every pass. At one point, the recall rate — the share of true best-matches an approximate search actually finds — dropped to a dismal 42%, while query latency ballooned at the same time. He explained, “HNSW graphs degrade as they take on mutations. The nasty thing is you don’t really realize that until you realize your results are tainted.”
He advised treating a vector index like any other database index — with the same care and attention. That means:
- Monitor recall accuracy and results returned
- Plan for partial builds or batch builds
- Understand that changing your similarity function, search parameters, or embedding model means rebuilding the graph from scratch
Resource Contention: Keep Workloads Separated
Another common problem is resource contention — for example, training and serving fighting over the same hardware. With everything running on the same infrastructure, GPU, RAM, and CPU all compete for resources. Many people don’t realize that vector search is a CPU cost, not a memory cost, since you’re traversing a graph rather than just storing vectors.
The fix follows standard distributed systems thinking: separate your write path from your read path, and separate training from serving where possible. These are sound engineering principles that prevent one workload from degrading another.
Retraining Is Inevitable
Retraining isn’t optional, and it isn’t free. Every model swap requires transition time.
There’s a window where the old model is still serving stale predictions and the new one hasn’t warmed up yet. For the database, this can mean new access patterns, cache misses, cold reads, or request queues building up. When data drifts or user behavior changes, the model trained three months ago is already getting worse — that’s the signal it’s time to retrain.
Plan for it in advance. One practical approach is blue-green deployments, canaries, and running old and new models in parallel under different names, with the actual cutover handled at the application layer rather than all at once. At a scale like Tripadvisor — with 100 million ML models — the process becomes considerably more complex, but the principle holds.
Avoiding the Doom Loop with a High-Performance Database
These problems tend to build on each other and snowball. Latency causes staleness, staleness degrades accuracy, degraded accuracy triggers retraining, retraining causes contention, and contention makes latency worse again — a compounding “doom loop.”
Here are some approaches for breaking that cycle.
Monitor, Monitor, Monitor
Be obsessive about monitoring. Watch freshness and backlog in particular, because a growing backlog is what eventually drives up tail latency. Watch index health, since that’s where recall rots. And load test beyond steady state, because you can’t predict when a confluence of factors will cause usage to surge.
Isolate Your Workloads
Isolating workloads addresses two problems at once: the write storms that cause tail latency, and training and serving sharing the same infrastructure. A database that handles concurrent writes well and isolates workloads properly can absorb both.
For example, with ScyllaDB, the write path is lock-free and multi-writer. Every node takes writes in an active-active fashion, and no row gets locked in the process. As a result, a burst of concurrent writes doesn’t back up into a queue the way it would on a database built around single-writer assumptions.
A practice called workload prioritization controls how workloads compete for system resources, ensuring latency-sensitive queries remain fast even when other heavy workloads run on the same cluster. A retraining job or backfill won’t steal resources from live inference serving.
Separate Vector Indexing
To address the vector index problem, keep the index separate rather than bolting it onto the same process as the core database. With ScyllaDB Vector Search, writes land in the core database first, and the index is built from that data asynchronously as its own service.
If the index can’t keep up with the write rate — whether from a re-embedding pass or a full rebuild after a similarity function change — it falls behind, but it never misses a write and the core database is not impacted. Even if the vector store goes down, the embeddings still persist in the core database. Because ANN queries are CPU-heavy, keeping them on a separate service means they’re not competing with writes for the same CPU cycles.
Under billion vector benchmarks, that separated architecture held P99 latency under 10 milliseconds at a concurrency of 300, handling approximately 150,000 ANN queries per second at a moderate recall target. Higher recall will introduce latency and throughput tradeoffs, so always test this in advance to understand how performance varies for your specific workload.
Absorb Write Spikes at the Architecture Level
The underlying lesson across all of these challenges is the same: the architecture needs to be designed to absorb shocks rather than amplify them. Write spikes, retraining cycles, index rebuilds, and model swaps are all predictable events. Building a pipeline that degrades gracefully under each of them — rather than cascading into a doom loop — is the difference between a system that works in production and one that only works in development.
