Shadow Traffic: comparing two RPC providers before you switch

You have a benchmark open in another tab — a provider's own numbers, or a comparison someone forwarded — and no way to check it against the traffic your application actually sends. The question underneath it is narrower than the benchmark answers: for the calls you make, at the blocks you ask for, would a second endpoint have returned the same thing?

You can answer that without a load test. Mirror a copy of your real read requests to a second endpoint, keep the primary authoritative, and compare what comes back. Most of the work happens before the first mirrored request, in deciding what counts as a difference.

Why someone else's benchmark cannot answer your question

A benchmark measures the benchmarker's traffic: their method mix, their region, their time window, their client. A feature list measures what a provider chose to publish. Neither describes your workload, and neither tells you what happens when your indexer asks for a log range at a block height your backfill has already passed. A number you cannot reproduce is a number you cannot defend in a planning meeting, which is the only place it has to survive.

Shadow traffic closes that gap. You mirror a copy of your production read requests to a second endpoint, keep the primary authoritative so nothing downstream depends on the copy, and record both responses. Failover is a runtime decision about which endpoint to use; shadow traffic is the measurement that tells you whether that decision is safe.

What we want out of it is not a score but a count: of the requests you mirrored, how many came back different, and what kind of difference each one was. Once the method has produced a shortlist, compare RPC providers is the next stop.

Design the diff before you run it

The failure mode is not a bad comparison. It is a comparison you cannot perform. You mirror traffic for a week, collect two piles of responses, and find you did not record the block height, or the params, or which endpoint answered — so the two piles cannot be lined up. All of it is decided before the first mirrored request.

Pick the unit of comparison

The unit is the request, not the response: the method plus a hash of the params. Two responses that look identical may be answering different questions, and two that look different may be answering the same one.

Record, per request: method, params hash, block tag or block number, timestamp, which endpoint answered, the HTTP status, and the body. The status matters because JSON-RPC is transport agnostic — the concepts work over sockets, over HTTP, or in-process [1] — so the HTTP layer and the JSON-RPC layer are separate things to record. The body matters because a response carries either a result or an error, never both [2], and a diff that stores only results silently drops every failure.

Write down what counts as a difference, per method

Decide this before you look at any result. Once you have seen a difference you will find a reason it does not matter, and the reason will be plausible.

The three classes are a design decision, not a standard. What is standardised is narrower and more useful: the error code is fixed by spec while the message text is not. Method not found is code -32601 [3], and our own error pages record three different strings for that one code [4]:

• the method debug_traceTransaction does not exist/is not available
• the method trace_block does not exist/is not available
• Method not found

A diff keyed on message text reports three differences where there is one.

Gas estimates are compared with a tolerance because they are not guaranteed identical across nodes: if the caller leaves the fee fields out, the client picks its own values for the simulation, and the spec calls those implementation-defined [5]. eth_getTransactionCount with the pending tag is not comparable at all; record it and leave it out of the count.

Choose the window and the traffic mix

Mirror a representative slice of production, not a synthetic loop and not 100 percent of traffic. At full rate you are measuring the candidate's rate limiter, not its node: 429 is the specified HTTP expression of a rate limit [6], and the spec explicitly declines to define how the server identifies the user or counts requests [7]. Whatever limit you hit is a property of the plan you are on, not of the node behind it.

State the sampling rule in the report — every Nth request, or a percentage by method — so the denominator means something. Run for at least one full traffic cycle that includes your peak, and longer if your method mix varies by day. A comparison run at 3am answers a question nobody asked.

What to mirror, and what to leave out

Mirror reads. Leave writes out.

A broadcast is not a request you can diff the way you diff a read: the response tells you the node accepted the transaction, not what the chain did with it, and the two endpoints are not answering the same question. The write path needs its own deliberate test — a small number of transactions you control, run against the candidate with its own pass criteria — and this method does not cover it.

Mirroring a read is safe in a way that mirroring a write is not: the copy has no effect on state, so a wrong answer from the candidate costs you a log line.

Three more exclusions. Anything with a private key or a signature in the params never leaves your system twice. Anything you are not permitted to send stays out. And keep the mirrored volume inside the candidate's published limits, because a run that trips a rate limit measures the limit, not the endpoint.

Before you point traffic at the candidate, check that it answers at all: the endpoint tester sends a single request and shows you the response, which is enough to catch a wrong URL, a missing key, or a chain the endpoint does not serve.

What a difference actually means

A difference is not a verdict. It is a class, and the class tells you what to do next.

Pin the block, or you are diffing clocks

The single decision that removes most false differences: compare at a fixed block height, or at a tag that resolves to one, never at the chain tip. Two nodes asked at different instants will disagree about the present and agree completely about the past.

The finalized tag is specified as the most recent crypto-economically secure block, one that only manual, community-coordinated intervention can re-org; safe is the most recent block safe from re-orgs under honest majority and certain synchronicity assumptions [8]. They are different guarantees, and a candidate that resolves one and not the other is telling you something. An unresolvable safe or finalized tag is answered the way an unknown block is [9], so an unresolvable tag is a spec-permitted difference class rather than a bug.

Pinning also separates a real difference from a timing artefact. A latest-block request that lands on a node a block or two behind can come back header not found on a load-balanced endpoint [10]; that is a node that answers but is behind the tip, not a candidate that cannot serve the call.

Four kinds of difference

Different answers at the same pinned height are the ones that matter, and they have three sourced sub-cases. A method the candidate does not implement returns -32601 [3]. State the primary serves and the candidate has pruned comes back as missing trie node <hash> (path ) state … [11], which the error page attributes to state at a past block being pruned [12]; the same condition has at least three wordings, including state at block #2 is pruned and height 1 is not available, lowest height is … [13]. A log range the primary accepts and the candidate rejects is a per-node cap, not a data difference: the figure in the message is that node's own cap, and two nodes behind one endpoint can disagree about it [14].

Different errors with the same outcome are why you classify by code and retryability rather than by string equality. One condition arrives under several wordings [13]; one code arrives under several wordings [4]. There is also no specified code for a pruned-state read — 4444, "Pruned history unavailable", is specified for transaction and receipt lookups only [15] — so the code you see for that class is whatever the node chose.

What the method cannot tell you

Four limits, stated plainly. It cannot say which provider is right when both are internally consistent; that needs a third source or the chain itself. It cannot measure behaviour under your peak unless you mirror at peak. It cannot tell you what happens when your primary fails, which is a different test with a different design. And it cannot price anything: a shadow run produces differences, not invoices.

Reporting the result without over-claiming

Report rates with denominators. "N of M requests differed, of which K were the stale-head class" is a finding; "the candidate looked worse" is not. Keep the denominator visible in every sentence that carries a number, because the first question in the meeting will be what M was.

Separate the two outcomes that get merged by accident. "The candidate cannot serve this" is a blocker: a method it does not implement, state it has pruned, a range it rejects. "The candidate served it differently" is a decision: something to accept, work around, or take back to the provider. The first list is short and urgent; the second is longer and negotiable.

The one number worth carrying into a planning meeting is the count of requests your application would have failed on, by class — the actual requests, from your actual traffic, not a percentage of a synthetic total.

Do not convert the run into a latency ranking. A mirrored copy shares your network path and your client, so it cannot rank providers on speed.

Where this leaves you

An empty diff means the candidate is not worse on the paths you exercise. It does not mean it is better, and it does not cover the paths you did not mirror.

A non-empty diff is a list of specific things to fix or accept before cutover: a method to route elsewhere, a block range to chunk, a tag to stop using. That list is the input to the cutover and rollback piece that follows this one, which is where the decision gets made and reversed if it has to be.

Test an endpoint

Sources

1. JSON-RPC 2.0 Specification — transport agnosticism — jsonrpc.org (JSON-RPC Working Group), 2010-01-01
2. JSON-RPC 2.0 Specification — Response object and error member — jsonrpc.org (JSON-RPC Working Group), 2010-01-01
3. JSON-RPC 2.0 Specification — error code table — jsonrpc.org (JSON-RPC Working Group), 2010-01-01
4. SwiftNodes /errors — method-does-not-exist-or-is-not-available, "The exact messages, and where we saw them" — SwiftNodes, 2026-09-24
5. execution-apis/src/eth/execute.yaml — eth_estimateGas description — ethereum/execution-apis (GitHub), 2021-06-16
6. RFC 6585 — Additional HTTP Status Codes, Section 4 "429 Too Many Requests" — IETF (RFC 6585, Standards Track), 2012-04-01
7. RFC 6585 — Section 4, scope limit on 429 — IETF (RFC 6585, Standards Track), 2012-04-01
8. execution-apis/src/schemas/block.yaml — BlockTag schema description — ethereum/execution-apis (GitHub), 2021-06-16
9. execution-apis/src/schemas/block.yaml — BlockTag, unresolvable tag — ethereum/execution-apis (GitHub), 2021-06-16
10. header not found — SwiftNodes, 2026-09-24
11. SwiftNodes /errors — missing-trie-node, "The exact messages, and where we saw them" — SwiftNodes, 2026-09-23
12. missing trie node (state at a past block is pruned) — SwiftNodes, 2026-09-23
13. JSON-RPC error messages, explained (SwiftNodes /errors index) — SwiftNodes, 2026-09-24
14. exceed maximum block range: 50000 (and every other eth_getLogs range error) — SwiftNodes, 2026-09-24
15. execution-apis/src/eth/transaction.yaml — eth_getTransactionReceipt, "Pruned history unavailable" — ethereum/execution-apis (GitHub), 2021-06-16

This article was researched, written and edited by AI under human supervision, and checked against Rankwrite's public editorial rubric.