Why Finding Bugs in LLM Serving Engines Is Hard [testing-llm-serving-engines]

    This project did not end as well as I hoped. Here is a look back at what we tried, why backend bugs kept slipping past our tests, and a few thoughts at the end.

    Before working on serving engines, I tested ML framework APIs in PyTorch and TensorFlow (FlashFuzz, Centaur). We found plenty of bugs that way, but after a while their practical impact on LLMs felt limited: operator and kernel correctness is an obvious target, with major labs and many academic groups already watching it closely alongside us.

    One layer up, almost nobody tests serving systems like vLLM and SGLang end to end, even though companies everywhere run them in production. Getting individual operators right does not mean the whole system works when wired together. Serving engines coordinate request scheduling, memory management, custom kernels, and multi-GPU communication, and when something goes wrong in efficiency or correctness, the impact is much larger.

    With that in mind, I spent the past six months hunting for bugs in vLLM and SGLang. How it played out was far from what I pictured at the start: most of the over 40 bugs we ended up collecting were in the frontend, such as tool-call parsers, API request handling, config validation, and weight loading. For the backend components we actually wanted to test (scheduler, KV cache, kernels, and multi-GPU communication), we tried several approaches and mostly turned up crashes and hangs under unusual configurations. Silent errors and silent data corruption (SDC) were very hard to catch.

    1. Checking External Outputs Misses Bugs (Configuration Testing and Numerical Comparison)

    We started with configuration testing: using static analysis to extract valid launch arguments from the engine, combining them into configuration pairs that differed by a single field, and running them on fixed prompts to check for crashes, hangs, or large differences in perplexity.

    Early on we ran about 5,500 configurations on SGLang and many rounds on vLLM, then gave up after a month. First, the configuration space was huge and we had no good way to prioritize which combinations to run. Second, argument fuzzing was nothing new as an idea, and the bugs it found were few and shallow. More importantly, we realized that in a serving system, deciding whether an output is right (the oracle problem) is much harder than generating inputs.

    A serving engine’s output naturally varies: even with temperature set to 0, changing the batch size, attention backend, or hardware can produce different tokens, so checking output text and perplexity is not enough:

    • Output text shows nothing unusual: A study of 929 bugs across five serving engines found that over 35% do not manifest as crashes. In sglang#17839, for example, the KV cache address pointers were completely wrong, yet gsm8k accuracy still reached 75%. When I tried replacing 30% of Qwen2.5-7B’s KV cache with tokens from unrelated prompts, accuracy only dropped from 87% to 50%; it took zeroing those entries out to push accuracy down to 5%, and real bugs rarely zero out whole blocks of data. Attention is already built to ignore irrelevant context, so it quietly ignores part of the damage from infrastructure bugs along with it.
    • Perplexity gives no signal: we used perplexity anomalies as an oracle for three months and did not find a single bug.

    Since text outputs gave us no signal, we looked lower down for finer-grained numerical signals:

    • Comparing logprob distributions: we used the fp32 implementation in HF Transformers as a reference (the same idea as KL divergence). It turned out that vLLM’s bf16 path was closer to fp32 than HF’s own bf16 path, and nothing stood out beneath the numerical noise. The only bug we caught was in the reference implementation itself: transformers forgot to multiply by weight_scale when loading FP8 compressed-tensors checkpoints, which we reported and got fixed (transformers#47145).
    • Comparing hidden states layer by layer: intentionally injected perturbations could be traced to the exact layer, but any perturbation smaller than 10% of that layer’s output was swamped by bf16 rounding noise.

    2. Checking Internal State When Python Cannot See the GPU (PBT and Runtime Verification)

    Around that time I was reading Designing Data-Intensive Applications and felt that checking internal invariants (Property-Based Testing / Runtime Verification) was a better fit for a stateful system like this: regardless of what the model generates, certain properties have to hold.

    I started by mining properties from the bug-fix history. Past KV cache bugs turned out to be a small fraction of the total, whereas frontend parsers broke much more often across versions and models. Following that lead, we used PBT to find quite a few frontend issues. Still, the frontend has small state, deterministic behavior, and clear correctness criteria, and everything reproduces on CPU. None of those bugs were deep, and the more of them we found, the less I liked them, because they had little to do with scheduling, memory management, kernels, or multi-GPU communication.

    To test the backend, we used PyMOP, a tool from my advisor’s group, to attach monitors to internal engine methods and wrote 33 specs for vLLM’s scheduler, KV cache manager, and block pool. The KV cache specs focused on block lifecycle and liveness: refcount cannot be smaller than the number of requests referencing a block, blocks cannot be double-freed, an in-use block cannot appear in the free pool, and a block cannot be evicted while its refcount is non-zero.

    RV worked well when replaying known bugs. Out of 8 known bugs we tried, 6 finished running and our monitors caught 5. The best example was SGLang’s PR #25015, which fixed an EAGLE draft position that was off by 1. The final generated text was still right, so looking at the output alone would never have caught it, but the monitor flagged all 78 buggy draft forward passes.

    Using it to find new bugs was a different story. We ran hundreds of configurations overnight, covering roughly 500 million events, without a single violation. Those KV cache checks ran for months and found nothing beyond known regressions.

    What we learned is that the Python layer of the KV cache is very thin, and doing RV in Python hits several dead ends:

    1. Process isolation and concurrency timing: vLLM V1 puts the scheduler in a separate EngineCore process while the monitor only attaches to the parent process. Without VLLM_ENABLE_V1_MULTIPROCESSING=0, tests finish normally while receiving zero events and raising no errors. SGLang also spawns its scheduler in a separate process, and observing it in-process requires changing how the server starts, which alters the concurrency we want to test. With the overlap scheduler enabled, tensors on the GPU may be modified asynchronously; reading them from Python forces a synchronization and can still observe half-updated values, causing false violations.
    2. The state is not in Python: we evaluated our specs against 55 historical scheduler and KV cache bugs and found they could deterministically catch only 10. Among the 31 bugs they missed, 11 happened before hooks took effect (config validation, initialization, process startup), and 8 lived in workers, Triton kernels, or CUDA events where Python hooks cannot reach.
    3. CUDA graphs are the biggest hurdle: production deployments keep CUDA graphs enabled, and during graph replay not a single line of Python runs, leaving Python-level monitors completely blind. Several of the worst silent bugs we collected happened on the replay path (such as SGLang #26880 and #27338). We only caught PR #25015 earlier because we passed --disable-cuda-graph to force the draft loop to run step by step in eager mode. Inserting checks during graph capture does not work either: even a single synchronization like torch.equal aborts capture with cudaErrorStreamCaptureInvalidated. Our first baseline run crashed the server that way, and we ended up skipping checks at 22 call sites during capture.

    To check invariants under CUDA graphs, the check itself has to be a GPU kernel captured into the graph alongside the model.

    When we talked to the SGLang authors later, we learned they had already built something called KV canary: it implements the checks as GPU kernels, patches them into model.forward, and captures and replays them inside the decode and EAGLE draft CUDA graphs. The host computes which slots to check outside the graph and writes them into a fixed device buffer; any violation is written to a device ring buffer and copied back to the host asynchronously (PyTorch’s built-in async assert avoids host synchronization, but once it fires, it corrupts the entire CUDA context).

    We tried it out. When we injected an offset error into req_to_token, the canary caught it; across 4 real configurations, however, with the longest run covering 1.28 million tokens, it never fired once. Instead, we found that when topk > 1, the canary’s own buffer size failed to account for EAGLE’s topk, crashing graph capture in the same way as #27338.

    Still, we did not come away empty-handed. Because I had a Framework Desktop (AMD GPU) and my advisor’s Mac Studio on hand, I caught several bugs in MLX and Rocm-attention, which showed the method itself can work.

    3. Why I Decided to Stop: Triggering Cost and Hardware Barriers

    Beyond observability, our conversation with the SGLang authors confirmed that we were running into the same walls:

    1. Reading data changes timing: the Python layer is thin, and the real reads and writes happen inside kernels. Pulling data off the GPU is hard, and the act of pulling it changes execution timing (SGLang’s memory-pool debug check synchronizes the GPU on every allocation, and vLLM’s NaN counter copies data back to the CPU at every step, so both are off by default in upstream CI; disabling CUDA graphs to run in eager mode slows decode down by about 10% to 20%). A friend of mine recently worked out how to extract data from CUDA graphs in real time with zero overhead; here is an early plug for their work.
    2. Buggy states are expensive to reach: many backend bugs only surface when a specific model architecture is combined with speculative decoding and CUDA graphs, and then hit by request preemption or a particular scheduling interleaving. Unlike verification, testing has to keep running the system to reach the buggy state, and in vLLM and SGLang just reaching those states is already expensive.
    3. Hardware barriers: some races only appear under specific hardware topologies and multi-GPU interconnects. For example, we found vllm#55208 by reading the code, and reproducing it requires two engine instances running over NIXL, which we cannot run locally at all. Judging from Red Hat’s talk at the vLLM summit, standard industry practice is mostly to throw expensive multi-GPU nodes at the problem, keep different models and benchmarks running continuously, and watch for metric drops. Given our earlier experiment where replacing 30% of the KV cache still left accuracy at 50%, that testing setup leaves plenty of room for improvement.

    After that conversation, I felt that pushing further with our hardware budget and a black-box or gray-box testing setup had very low return on effort, so I decided to wrap it up and move on to writing papers and starting the next project.

    Takeaways

    Looking back on the past six months, I am left with a couple of thoughts:

    1. Complex backend state and concurrency may be better suited to verification than testing.In systems with complex runtime environments and subtle race conditions, running tests does not always work well. Moving RV onto the GPU is one thing that can be done in the future (someone at UMD is already working on this); if I were choosing tools again to continue working on the backend, though, I would switch to verification. With model checking, for instance, we can model the scheduler, KV cache manager, or cross-node transfer protocols directly and let the checker exhaustively explore execution orderings. That avoids needing GPUs or specific multi-GPU hardware topologies, and it does not rely on finding a signal in the model’s output. Each of the 1,032 properties we mined earlier comes with a TLA+-style draft, which makes a natural starting point for modeling. Some people have already gone further down this path: Vosti wrote a formal specification for deterministic inference and used Verus to prove that scheduling and prefix-sharing paged KV caches do not change the output logits; along the way, their tests also found that vLLM’s batch-invariant mode and SGLang’s deterministic mode still produce inconsistent logits under certain execution changes.
    2. If we still want to do testing, we should craft inputs that hit backend scheduling boundaries.Looking back, testing is not useless here; the inputs we generated before (whether random launch configurations or ordinary prompts) simply had a hard time hitting the pain points of the backend state machine. A new idea I had recently is that much of the deep backend scheduling logic is driven by specific model outputs or token states. Take the EOS token: when it appears at a valid boundary (such as under concurrent requests, at a prefix-caching hit boundary, or at a specific position inside a speculative decoding draft), it immediately triggers a chain of request completion accounting, KV cache block reclamation, batch reorganization, and even preemption. Instead of blindly fuzzing parameters from the outside, we can design input sequences that legitimately trigger these control-flow transitions to stress-test the backend scheduler directly. That may be a direction worth trying next for serving-engine testing.