Three Gates Against Ethereum Slashing

The scariest thing about running validators isn't getting hacked. It's slashing.

"The node looked dead, so we started a new one. But the old one was alive the whole time, just not being detected."

The moment the same key signs from two places, you're a slashing target. The protocol doesn't ask whether it was a mistake or malice. Two contradictory signatures, and that's the end of it. And if many other validators were slashed around the same time, the correlation penalty can take most of your balance with it.

Causing this kind of incident doesn't take spectacular incompetence, either. An alert at 3 a.m., ambiguous kubectl get pods output, and a guess that "it's probably dead." That's about all it takes.

So you have to start from the premise that people make mistakes. Here's how the three gates FP Validated has placed around its Ethereum validators (before deployment, at startup, at signing time) each work, including what they stop and what they can't.

Why one defense isn't enough

A common mistake is stacking the same kind of defense twice. Put a confirmation step in the deployment script, then add a checklist before anyone runs the script. Both rest on the same assumption: a person will follow the procedure. When that assumption breaks, both layers collapse at once. That's not redundancy. It's just one thick layer.

Real redundancy means each layer fails for a different reason. We split our gates by point in time.

The first gate defends at the moment a change lands in the repository, in an IaC (Infrastructure as Code) environment. The second trusts what the network shows right now, and the third trusts the signing records it has kept itself. Punch a hole in one layer, and the other two still judge independently.

Gate 1 - Before deployment: declare key ownership in code

The most common cause of incidents is not knowing exactly which node is running what. This first gate targets that ambiguity itself. In our deployment repository, every configuration file that runs validators carries a block like this.

Which keys, on which network, owned by which entity, in which state. All of it written in a file, not in someone's memory. CI collects these blocks across the whole repository and checks them.

The most important rule is a single one: if more than one entity declares the same public key as active on the same network, CI fails.

Open a PR like that by mistake and it won't merge. The most direct cause of double signing gets filtered out at the merge step.

Migrations as a state machine

Say you're moving a node from A to B. The usual way looks like this:

1. Merge a PR that adds B
2. Merge a PR that removes A

Between step 1 and step 2, there can be a window where both are active. And beyond the time gap itself, human error flips the order or forgets step 2 entirely. It happens.

So we use three states. active is the entity currently signing (one per key). staged is the entity prepared to go next: configuration complete, but not signing. retired has given up ownership.

A migration happens inside a single PR, with A: active → retired and B: staged → active together. The two changes are bound atomically, so a state where both sides are active can never exist in the repository. Having more than one staged candidate is blocked separately.

Tying declarations to actual deployment

Changing the state without changing the actual workload would mean nothing. So CI also catches mismatches between state and deployment parameters. state: active with validator replicas not equal to 1 fails; not active with replicas not equal to 0 also fails. In other words, a configuration marked retired while its pod keeps running can't enter the repository.

Keeping names and contents in sync

There's one more case we've actually run into: keeping a key set's name while swapping one of the public keys inside it. Everything that references the name (secret names, Vault paths) silently starts pointing at different contents.

So we force the key set name to end with a hash of its public key list. In the example above, the trailing 9f2c1d4b7e08 in lido-mainnet-set-a-9f2c1d4b7e08 is exactly that: the first 12 characters of the SHA-256 of the sorted public key list.

Change the contents and the hash changes; change the hash and the name has to change; leave the name unchanged and CI blocks it. The same hash must also appear in keystore secret names and Vault paths, which makes "different key set, same secret reference" structurally impossible.

The reverse direction is covered too. Because CI checks that the public keys configured on the validator exactly match the declared list, a key that's actually attached but not declared in code fails the check, and that validator doesn't start. Every signing key is a declared key, and an undeclared key cannot sign.

Gate 2 - At startup: ask the network directly

When a validator client starts, it doesn't begin signing right away. It first watches the network quietly for about two epochs (roughly 13 minutes). If it sees traces of its own keys already active somewhere during that window, it refuses to start signing and shuts itself down. It's looking for a doppelganger, its own double.

The trade is 13 minutes of attestation rewards forfeited at every startup, in exchange for catching the "my key is running somewhere I don't know about" scenario. Given that a single slashing erases years of rewards, it's not a trade you have to think about.

Most Ethereum clients ship their own doppelganger detection, and on mainnet we treat it as a mandatory setting.

The limits of doppelganger detection

First, it only works within the observation window. If the same key comes up elsewhere after signing has already started, it won't be detected. It's strictly a startup-time defense.

It also can't be enabled in every configuration. DVT clients like SSV have doppelganger detection now, but they didn't in the early days. Depending on the client and deployment shape, the feature may be missing, or enabling it may create other operational constraints.

Why we don't do automatic failover

In ordinary systems, spinning up another instance automatically on failure is common sense. For validators, that's a slashing trigger.

Nodes that look dead but are actually alive are more common than you'd think. A health check endpoint that stops responding while everything else runs. A Kubernetes control plane issue that drops detection. A process that looks frozen but keeps producing signatures. None of these are rare.

So signing workloads get no automatic failover. It's blocked at the deployment manifest level, and no one can bypass it on their own judgment.

Instead, every migration follows the same sequence:

1. Secret gate: Prepare the secrets the new location needs, while cutting off the old entity's access paths to the keys. Before taking the pod down, cut off its signing material first.
2. Hard stop: Delete the old workload and wait until its pods are completely gone. Before starting, check whether a new pod already exists at the destination, and abort immediately if it does. Even after the termination wait, rescan the entire cluster for leftover pods, and don't proceed if a single one remains.
3. Verified start: Only after a person confirms the previous step ended clean does the new deployment go out.

It's slow, and it always requires human hands. We do it anyway, because a downtime penalty and a double-sign slashing are not comparable losses.

Gate 3 - At signing time: dangerous signatures never get made

The validator client doesn't hold the keys

In our setup, the validator client doesn't hold signing keys at all. A remote signer called Web3Signer manages them, and the client only sends requests: "please sign this." The separation itself is already a defense. Every signature funnels through a single path, and that's where you can put a check.

Every request is checked against history

Web3Signer records everything it has ever signed in a database and checks each new request against it. Asked to sign a block for a slot it already signed? Refused. An attestation that contradicts a past vote? Refused. Whether the configuration is wrong or two processes are running, a slashable signature never gets created in the first place.

The first two gates stop that situation from forming. This gate makes sure that even if it forms, no output comes of it.

The DB is the weak point

The catch: this defense only works while the history is in the DB. Attach an empty database and Web3Signer knows no past; it will pass anything. So this DB is an asset that must outlive the application, and it can't be treated like an ordinary stateless workload. We hold ourselves to four rules:

1. Decouple the DB and secret lifecycles from the app. Deleting deployment resources leaves the DB volume and credentials intact. This blocks the scenario where redeploying an app or cleaning up a namespace takes the slashing history with it.
2. Credentials live in Vault and are injected via External Secrets. Plaintext never enters a manifest.
3. The DB port opens only to the signer. NetworkPolicy allows access to 5432 only from Web3Signer, so no other workload can touch the history, accidentally or otherwise.
4. Missing required settings block deployment entirely. Render-time guards in the chart fail template rendering if required values like the network name, DB credential keys, or persistent volume paths are empty. Bad configuration dies before it ever reaches the cluster. Mutually exclusive options enabled together are caught here too.

Any single gate can be breached. Stacking them is what matters

The three conditions in the right column don't overlap. That's the result of spreading the layers across different grounds, so that an event which disables one layer doesn't disable the others with it. Even so, we can't claim this is airtight. The job is keeping a written record of how each layer can be breached, and swapping a layer out when those conditions start to overlap.