On this page
Running validators means spending a lot of time on performance problems, and for Ethereum, head vote accuracy accounts for most of them.
Our validator got a head vote wrong. Did the block arrive at our node too late, or was there never a block in that slot at all?
These are entirely different problems. The first one we can fix by improving our propagation paths. The second one means the proposer missed their slot, and there's nothing we can do about it. But the single number that performance dashboards like Rated report, something like "89% accuracy," mixes the two together.
To separate the causes, you first need to know exactly what your validator voted at each slot. We set out to check that, ended up much deeper than expected, and eventually recomputed the consensus spec's shuffle ourselves. It had to work on the regular beacon nodes we already operate, not depend on an archive node or an external data provider.
On-chain attestations carry no names
Open an attestation recorded on chain and it looks like this.
There's no validator index anywhere. Participation is expressed only as a bitfield. If our validator sat in seat 202 of that slot's committee, then bit 202 of that committee's segment tells you "voted" or "didn't."
So to read your own bit, you first need the committee roster for that slot. Without the roster, the bitfield is just a meaningless pile of binary.
The usual way to get the roster is to ask the node directly.
But most of the validator nodes we run were brought up with checkpoint sync and prune by default. Beyond roughly the last hour, everything comes back 404. Committee rosters derive from beacon state, and those states are already gone.
The standard fix is restoring states with --reconstruct-historic-states. Since hierarchical state diffs landed, this isn't as expensive as it used to be; per the Lighthouse docs, a default-setting archive runs in the 400 GiB range on mainnet. It still costs a resync, reconstruction time, and a bigger disk budget on a production node. Not a price worth paying just to see your validators' voting history. And above all, we wanted answers right now, from nodes already running.
Committee rosters can be recomputed
This calls for a change of perspective. A committee roster was never stored data in the first place. It's a computed result determined by exactly two inputs:
- The epoch's RANDAO mix (the shuffle seed)
- The active validator set at that point
And both of these still survive on a pruned node.
RANDAO is kept in a ring buffer. The beacon state's randao_mixes holds EPOCHS_PER_HISTORICAL_VECTOR entries, which is 65,536 epochs of history, roughly nine and a half months. A seed from a few days ago is comfortably in range, and a standard API call pulls it straight from our node.
The active set is recoverable from the current registry. Every validator record carries its activation_epoch and exit_epoch, so the active set at any past epoch falls out of today's registry with a one-line filter.
Because Ethereum's shuffle is deterministic, feeding the same inputs into the same rules produces the same result again.
Inverting the shuffle
One wall remains: Ethereum has over a million active validators (on Hoodi; mainnet is around 880,000), and running compute_committee per spec means shuffling the entire set. That's 90 hashes per index, so about 100 million per epoch, and multiplied by thousands of epochs it's beyond what a script can handle.
But this shuffle (the swap-or-not construction) has a convenient property: it inverts. Run the rounds in reverse order and you get the inverse permutation.
We don't need the full roster. We need one thing: where our validator landed after the shuffle. So instead of shuffling a million entries forward, we run just ours backward, once.
Feed in the validator's rank within the active set, call it once in the inverse direction (90 hashes), and out comes its post-shuffle global position. Divide by committee boundaries and the slot, committee number, and seat within the committee are all determined.
For a dozen or so validators across thousands of epochs, that's about 1.5 million hashes total. It computes in under two seconds.
Two things that bit us
That's the idea. Implementation brought a few problems.
First, the RANDAO request argument. The spec's get_seed reads randao_mixes[epoch + 65536 - 2]. Put that number into the API as-is and you get a 400 back, because the node validates against the current epoch. Since the index wraps modulo the ring buffer length, requesting epoch - 2 points at the same cell.
Then there was a hidden serial bottleneck. We had parallelized block fetches, but the job stayed slow. The remaining cost was somewhere unexpected: the header lookups that check whether each slot belongs to the canonical chain were running twice per slot, all serially. Prefilling that stage in parallel too cut the same job from 13 minutes to 92 seconds.
Verifying the results
A computation that starts from theory needs an answer key. So we checked it two ways, with different failure modes.
First, against the live API. For the current epoch the node still serves committee rosters, so we derived the same epoch through both paths and compared. Slot, committee number, seat within committee, committee size: all matched. As a bonus, the active validator count derived from the registry exactly equaled the sum of API committee sizes, which validates not just the shuffle but the active-set filter too.
Second, against an independent dataset. ethPandaOps' xatu publishes attestation data with validator indices already resolved. We picked a date where that data exists and compared every (validator, slot) row on inclusion distance, head correctness, and final classification. All 280 of 280 rows matched.
What the tool is for
The immediate win is losing an external dependency. We had been relying on ethPandaOps' public dataset, and in the middle of a P2P experiment, of all times, collection for that network stopped. Since this tool made self-reconstruction possible, we trust the numbers pulled from our own nodes and use external data only for cross-checking.
It also means that if you run a beacon node, you already hold every ingredient needed for per-validator attestation analysis. No archive node, no third-party API, just the beacon node you already have.
We use this to break validator performance down by cause. Separating failures that propagation work can fix from failures it can't shows exactly where investment actually moves performance. A single accuracy percentage never told us the specific cause, and that had made these problems hard to act on.
Open source: beacon-attestation-forensics
Our implementation is published as beacon-attestation-forensics. It uses only the Python standard library. The verify command described in this post (the live API comparison) ships with it, so don't take the results on faith: run the verification yourself.