Managing Infrastructure Access and Keys Reliably

Validator operations tend to run into two recurring problems: who can get into the cluster, and where the signing keys are kept.

Access permissions and signing keys look like separate problems, but the incidents I have seen usually had the same root cause: someone made a temporary exception under pressure. An account was granted admin privileges because a node went down in the middle of the night. A key was handed over as a file because a deployment was urgent. Exceptions created this way are often neither recorded nor revoked.

At FP Validated, permissions and keys are defined as files and records in the source of truth, and the actual cluster state is managed so that it does not drift from those definitions.

This article covers the following:

1. How Kubernetes permissions are split into files, and what applies those files to the cluster
2. How human accounts and workload accounts are treated differently
3. How validator signing can be performed without delivering the signing key into the cluster, and what alternatives are used when that is not possible
4. What changed after two real incidents

1. Permissions and keys are divided into four areas

In production, permissions and keys are managed in four separate areas. Each area owns different resources, and the failure mode is different.

2. Permissions live in the repository

Role-Based Access Control (RBAC) is built into Kubernetes. The problem begins when permissions start being created with commands such as kubectl create clusterrolebinding. At that point, who created the permission, when, and why exists only inside the cluster, while the review history exists nowhere.

Moving permission definitions into repository files gives every change a review trail, makes it possible to roll back to an earlier commit, and ensures that manual changes in the cluster are reverted to the repository state on the next synchronization.

2.1 Permission files are kept in one place

The management cluster's permission definitions are collected in one location in the repository.

The file that applies this directory to the cluster is 10-rbac.yaml.

With selfHeal: true, if someone edits a role directly in the cluster, Argo CD restores it to the repository state. Permissions granted manually during an emergency disappear on the next synchronization. prune: false is a safeguard in the opposite direction. Even if a file is deleted accidentally, the permissions that have already been applied are not removed all at once.

2.2 Human roles are split into three levels

If permissions are divided only into "administrator" and "everyone else," every investigation eventually turns into a request for administrator access. Human-facing roles are therefore defined in advance by purpose.

There is exactly one write verb in the read-only role: port-forwarding.

create on pods/portforward is required for incident investigation. If an internal metrics endpoint is not exposed through a dashboard, an operator needs this verb to forward it locally. Because this is a write verb inside a read-only role, the decision is documented in the file and reviewed.

security-auditor can read RBAC objects themselves. To audit who has which permissions, the auditor must be able to see bindings. Because that visibility can also give an attacker a map of the permission structure, it is separated into an audit-only role.

2.3 Humans enter through groups, and accounts are generated

If a kubeconfig is created and distributed for every person, revocation becomes difficult. Instead, groups in the identity provider, Authentik, are treated as the single source of truth, and a synchronization Job creates the corresponding cluster-side accounts and bindings.

Authentik groups carry attributes like these.

The Job scans active users, reads those attributes, and reconciles the following for each cluster:

1. Check that the fp-operator-access namespace exists.
2. Create a per-user ServiceAccount named fp-operator-<username>.
3. Create either an fp-operator-<username>-cluster-admin or readonly binding. If the target role has changed, delete and recreate the binding because Kubernetes does not allow roleRef to be modified.
4. Delete managed bindings that are no longer present in the desired state.

Step 4 handles revocation. Remove someone from the Authentik group, and the binding disappears from the cluster on the next run. Deletion is limited to objects selected by management labels, so the Job does not touch bindings it did not create.

Three guards are applied.

All three guards stop the entire run instead of merely skipping the affected user. If permission synchronization partially succeeds, the cluster state cannot be trusted. Bindings carrying the fpvalidated.com/break-glass=true label are excluded from deletion. There must still be a way into the cluster if the identity provider itself is down.

The Job has both an apply mode and an output-only mode. Because permission changes can be difficult to reverse, the desired state is printed and reviewed before it is applied.

2.4 Argo CD is restricted in what it can create

Argo CD is the component that actually creates resources in the cluster. Without restrictions, anyone who can commit to the repository effectively becomes a cluster administrator.

An Argo CD AppProject restricts which resource kinds an application may create and where it may deploy them. The projects are divided into five groups.

The allowlist for the validator project, where chain workloads live, is the narrowest.

Chain nodes only need to live inside their namespaces. Namespace-scoped resources are therefore allowed, while cluster-scoped resources are limited to namespaces and volumes. If a chain chart accidentally attempts to create a ClusterRole, synchronization is blocked.

2.5 The wallet-ops deployment was actually blocked

This configuration caused the wallet operations dashboard, wallet-ops, to fail deployment once. The chart was trying to create a ClusterRole.

The permission itself was minimal. It could only create a token validation request. That makes it tempting to add ClusterRole to the validator project's allowlist.

We did not do that. The project is shared by 56 chain applications, including Lido and horcrux validator signers. Adding one line would allow all 56 applications to create cluster-scoped permission objects. Instead of widening the allowlist, we decided to create a separate lz-dashboard project scoped only to this repository and namespace. Until that project exists, the two chart objects remain in a synchronization-failed state.

2.6 Workload permissions stop at the namespace boundary

Across the entire chain repository, there is not a single ClusterRole. When a workload needs Kubernetes API permissions, it gets a dedicated ServiceAccount and a namespace-scoped Role.

For example, the CronJob that periodically restarts a node receives only this much access.

That is the minimum required to restart a Deployment in its own namespace. ReplicaSet read access is included because kubectl rollout status reads ReplicaSets, and the reason is documented in a comment. Even if this CronJob's token leaks, it cannot inspect pods belonging to another chain.

Authorization is checked separately for pod-to-pod calls as well. When the dashboard calls the backend with a token issued by the Kubernetes API server, the backend sends that token back to the Kubernetes API for verification.

The token is a projected token with a fixed audience and a one-hour lifetime. A token issued for a different purpose is rejected if its audience does not match. Requests from human operators do not use this path; they are validated using tokens issued by Authentik.

3. Keys are stored differently depending on their purpose

Validator operations involve signing keys, treasury wallet keys, cluster credentials, and node identity keys. If all of them are kept in one vault, anyone who can open that vault can access everything.

3.1 Vaults are separated by what they contain

Vaults are separated inside 1Password as well. This separation defines the blast radius of an incident.

Operational keys and treasury keys are separated because they have different owners and duties. Someone responsible for restoring a node does not need access to the reward wallet key. The recovery-only vault is not opened during normal operations.

Secret values are never printed to the screen. Only item names and success or failure are checked. Configuration templates contain op:// references instead of actual values.

SSH keys are also created per person. Shared server private keys are eliminated, and only public keys are added to the server's authorized_keys. A script handles registration when a new server is added, and does not add a key again if it is already registered.

3.2 Keys are delivered from Vault into the cluster

Inside the cluster, External Secrets Operator (ESO) retrieves keys from Vault and creates Kubernetes Secrets. Connection information is defined in one ClusterSecretStore per cluster.

A chain chart references that store and declares only the keys it needs. For a Cosmos-based validator, three objects are created: the node key, signing key, and signing state.

deletionPolicy: Retain protects against accidental deletion. Even if the chart is removed or synchronization drifts, the signing-key Secret remains. If a validator-key Secret disappears, the node cannot sign, and depending on the chain, it may be penalized.

The signing state file, priv_validator_state.json, is delivered through an ExternalSecret as well. This file records the last height that was signed and prevents double-signing. Restoring only the key while omitting the state can result in signing the same height twice.

3.3 What changed after two incidents

The current configuration is the result of two real incidents.

First: a static token quietly expired.

A workload cluster's ClusterSecretStore used a Vault token stored in a Secret. ESO does not renew this token. When the token expired, the store changed to InvalidProviderConfig, and secret synchronization stopped across multiple chains at once. Existing Secrets remained, so the nodes continued running, which is why the failure was detected late.

The fix was to remove the token altogether. With Vault's Kubernetes authentication, Vault sends the pod's ServiceAccount token back to the Kubernetes API server for verification and then issues a short-lived token. There is no long-lived token for a person to manage.

bound_service_account_names and bound_service_account_namespaces provide the restriction. Only the specified accounts in the specified namespaces may use the role. If another pod tries to request the same role using its own token, the request is rejected. This Job is a PostSync hook, so the configuration is re-applied even after Vault is rebuilt.

ESO inside the management cluster already uses this authentication method. The path used by ESO in other clusters to access this Vault is still being migrated. Vault's pod must be able to send TokenReview requests to the remote cluster API server at 10.50.8.68:16443, so the network path and a non-expiring review account are being prepared first. The migration is being tested on one chain before being expanded.

Second: an internal certificate renewed automatically and disconnected the stores.

The internal root Certificate Authority (CA) was configured to renew automatically every 90 days. The certificate management tool's default behavior was to generate a new key pair during renewal. Once renewal happened, three stores in workload clusters that still held copies of the old CA could no longer trust the server certificate.

The fix had two stages. First, a bundle containing both the old and new CA was deployed to restore connectivity. Then the renewal policy was changed.

Keeping the key means the public key stays the same across renewal, so previously deployed copies remain valid. The transition was completed without downtime. Verification showed three healthy stores, 48 synchronized ExternalSecrets, and zero restarts.

Early in the investigation, the wrong Secret was identified as the cause because an orphaned Secret with a similar name was still present. The investigation only moved in the right direction after checking the actual name referenced by the ClusterSecretStore manifest. The orphaned Secret was deleted after the incident was resolved.

4. Signing is performed without downloading the key

So far, this has been about moving keys safely. There is another option: do not move them at all. Whenever a signature is needed, send a signing request to the place where the key resides. Three approaches are used depending on the target.

Security increases as you move down the table, but so does operational cost. If the chain client supports remote signing, KMS is used. If not, threshold signing or local signing is used.

4.1 Threshold signing requires enough shares to produce a signature

Threshold signing splits a key into multiple shares, gives each share to a different signer, and requires a defined number of shares before a signature can be produced. A single leaked share cannot produce a valid signature.

The Stable chain validator operates with a 2-of-3 scheme. Each signer retrieves and mounts its own share and an inter-signer communication key from Vault.

To prevent double-signing, the deployment strategy is fixed to Recreate. With the default strategy of starting a new pod before terminating the old one, two pods can briefly coexist and sign twice with the same share.

4.2 KMS signing keeps the key out of the pod

The Decentralized Verifier Network (DVN) signing service does not hold the key. Its configuration contains only the key-store location and the expected address.

address identifies the owner of the signature that this signer is supposed to produce. There is no private-key field. Each signing operation follows the configured remote-signing path.

Because the key is never delivered to the pod, compromising the pod does not give an attacker a private key to steal. The remaining risk is abuse of signing authority, so three defenses are applied.

Signature-value verification. The address is recovered from the signature returned by the key store and compared with the expected address. If it does not match, the signature is discarded. Azure, GCP, and AWS return responses in different formats, but all three paths pass through the same verification function. Adding a new backend cannot bypass this check.

Signing ledger. Every signing operation is recorded in a file, and an identical request is rejected if it appears again. The ledger is a file that can only be safely written by one process, so increasing the replica count would break this protection. Production is therefore fixed to one replica with autoscaling disabled. If replicas are increased for performance, a shared ledger must be implemented first.

Fail closed. If signer identities do not match or the same signer responds more than once, no signature is produced. When the state is ambiguous, the default behavior is not to sign.

The supported backends were deliberately reduced to three: Azure Key Vault, GCP KMS, and AWS KMS. The reasoning is the same: every additional signing path introduces another path that must be verified. When the OpenBao path was removed, not only the code but also chart configuration, documentation, and integration tests were removed. A repository-wide name search was used to confirm that no remnants remained.

Gasolina was also migrated from Vault-based mnemonic signing to KMS signing. Initially, the mnemonic was retrieved from Vault and mounted as a read-only file.

The configuration file now contains only information identifying which key in which key store should be used. Even if that information leaks, signing authority does not come with it. Calling the signing operation still requires a separate cloud identity.

5. Both permissions and keys are operated from declared state

Permissions and keys were explained separately, but in both cases the authoritative state comes from declarations in the repository and Vault rather than from whatever happens to exist in the cluster.

Anything created directly in the cluster is not the source of truth. A binding created with kubectl is removed on the next synchronization, and a manually inserted Secret is overwritten by the ExternalSecret.

6. Adopt it in this order

The practical adoption order is as follows.

1. Move permission definitions into the repository and enable selfHeal. Permissions granted manually are reverted on the next synchronization.
2. Split AppProjects and narrow the allowlist for cluster-scoped resources. This becomes much harder after the number of applications grows.
3. Eliminate static tokens. Human-managed tokens and certificates eventually expire. Replace them with workload-identity-based authentication.
4. Separate vaults by purpose. Keeping operational keys, treasury keys, and recovery-only material separate reduces the blast radius of an incident.
5. Move signing to remote services. KMS and threshold-signing support varies by chain, so this is handled last.