Why Do We Use Kubernetes to Operate Blockchain Nodes?

The first question we hear when introducing Kubernetes for blockchain node operations is, “Why Kubernetes at all?” Docker or systemd is sufficient for running just a few nodes. However the challenge begins not with the absolute number of nodes, but as combinations of chain architectures, node roles, and recovery conditions begin to multiply. The same failure or version change can require different placement constraints, data lifecycles, and recovery sequences. At that point, operations shift from simply running processes to deciding which state can be rebuilt, which state must be preserved, and when changes should occur.

We choose Kubernetes so these decisions would not remain buried in per-server procedures or individual operators’ memories. We turn common operating rules into executable workload definitions, while expressing differences between chains and roles as values and policies. Adding nodes, recovering from failures, and performing upgrades then follow the same pattern: apply those definitions to the environment and verify the result.

Adopting Kubernetes does not make block processing faster or chain data inherently safer. It requires us to operate additional controller, network, and storage layers, and automated recovery can introduce new risks for validators. This article explains why we accept those risks and continue to use Kubernetes, based on our actual architecture and operating practices.

Unifying chain and role-specific operations and recovery under one policy

The most direct approach is to retain separate installation guides and dedicated runbooks for each chain. This is convenient when bringing up new nodes quickly. But every worker replacement or common security change requires us to reinterpret placement, data attachment, networking, and access control through each chain’s procedures. As the number of nodes grows, the number of runbooks grows even faster, and similar changes can produce different outcomes depending on who performs when and what.

We also choose not to impose the same workload structure and recovery rules on every node. Optimism RPC nodes combine an execution client and a rollup node. Canton brings PostgreSQL, a participant, a validator, and a management UI together as one system. Initia validators also include the Horcrux signing domain. Each has a different composition and set of components. Abstracting them all into a single generic scope may reduce the number of deployment files, but it also hides the actual failure boundaries and recovery completion criteria behind templates.

So instead of standardizing every implementation, we standardized the criteria behind operating decisions. Each workload declares its replica count and placement constraints, data lifecycle, RPC and P2P endpoints, and the location and transfer conditions of signing authorities. Kubernetes uses those conditions to place processes and connect the required data and networks, while chain and role-specific policies determine the appropriate recovery path.

For example, the same worker failure leads to different priorities. An RPC node should rebuild its runtime and return behind its existing Service endpoint when other nodes whose local data ownership matters must reconnect its existing path or volume. A validator must confirm that the previous signing authority has ended and that the handoff is complete before starting a new process. Rather than applying one automatic recovery action to every node, we translate the same failure into a state transition appropriate for each role.

We reuse policy delivery and operational records through a management AKS cluster, while P2P execution paths, chain data, and signing workloads remain in a separate execution cluster. Within the execution cluster, data paths and credentials are separated by roles, although the control plane and parts of the network failure domain are still shared. Our direction is to distribute critical signing workloads further across separate workers and failure domains, creating a clearer boundary between the scope of reusable policies and the failure domains of chain execution.

This standardization introduces its own risks. A faulty common policy can propagate across multiple chains, and chain-specific safety conditions can disappear behind ordinary options. We therefore limit standardization to repeatable placement and connectivity principles, while keeping protocol-specific state and signing conditions in chain-specific policies.

When adding a new chain, we no longer need to rebuild the entire operating model. We can combine proven placement, data, network, and signing patterns, then add only the chain-specific conditions. Desired-state management is the starting point for applying this model repeatedly across real clusters.

Moving from per-server tasks to desired-state operations

The simplest way to reduce per-server work is to bundle installation commands into shell scripts or automation tools. This can standardize execution order, but the intended state of each server still has to be verified separately afterward. If an emergency fix remains on only one server or is skipped, the documentation and the actual environment diverge. We therefore made the state that must be maintained—not the commands used to reach it but the unit of operation.

In our current implementation, recurring workload structures live in Helm charts, while values shared by a chain and network are separated from node-specific values into distinct values files. An Argo CD Application connects the selected chart, values files, and target cluster as one deployment unit.

When multiple processes form a single node role, as with Optimism RPC execution client and rollup node, they are defined together from the same network values while each workload retains its own replica count, Service, and data path. Canton combines a broader set of components. A single Argo CD Application brings together our common internal chart and values files with upstream PostgreSQL, participant, and validator charts as multiple deployment sources. Each chart retains its own lifecycle, while we can review the versions and configuration that form one environment in a single diff. Instead of logging into servers to coordinate component installation order, we change the desired state of the application as a whole.

A Git change represents the next desired state shown by Argo CD. For active validator Applications, we disable automatic synchronization and set Prune=false. After an operator reviews the rendered diff and target and explicitly starts synchronization, Kubernetes controllers maintain the applied images, configuration, replica counts, and placement. This separates automation that maintains desired state from the decision to promote that state into the execution environment.

The common chart does not choose one workload controller for every case. We use StatefulSet when a stable binding between identity and volume must be preserved, as with Sui validators and use Deployment with Recreate when a single process only needs to be replaced on a designated data path, as with Walrus storage nodes. Rather than using StatefulSet for every workload that has state, we choose the controller based on the relationship that must survive Pod replacement.

A defect in a common chart can still spread across multiple chains. For configuration changes that require a restart, we record a checksum in a Pod template annotation so that Kubernetes creates a new Pod template version. Structurally invalid value combinations fail during rendering. Configurations whose meaning becomes unclear under a common schema are moved into dedicated templates, while protocol transitions that require human judgment remain outside automated reconciliation.

As a result, adding nodes, changing configuration, and recreating workloads after failures no longer require different commands on each server. They follow the same path of applying reviewed definitions. This makes the execution environment reproducible, but it cannot recreate several terabytes of chain data instantly owned by a new Pod. The next challenge is bringing different storage types into the same recovery workflow.

Bringing different storage types into one fast recovery workflow

Placing all chain data on dynamically provisioned network PVCs would simplify the configuration. If a worker fails, the same volume could be attached to another worker, while snapshots and capacity changes could be managed through one API. For nodes that continuously read and write several terabytes of state, however, disk latency and throughput affect synchronization speed more than reattachment convenience. Moving data already accumulated on bare-metal local disks into a common storage layer would change not only cost, but also failure points and performance characteristics.

We therefore did not standardize on a single storage technology. We choose among hostPath, static local PVs, and PVCs according to the characteristics of the chain data, while leaving the data reconnection and worker placement rules in Kubernetes definitions. For rebuildable RPC data, we favor local NVMe performance with snapshot or resynchronization paths. For long-lived accumulated data, we prioritize preserving ownership of the existing disk.

The Walrus storage node shows this decision most directly. Because it handles a large volume of data, we mount the host data path into the container and use node affinity to pin the workload to the worker that owns the disk. Using DirectoryOrCreate would start the workload faster by creating an empty directory automatically, but an incorrect mount path could make a new node with no data appear healthy. We use Directory so the deployment fails immediately rather than beginning an incorrect resynchronization. Deployment with Recreate replaces the process, while changing the worker that owns the data remains an explicit operator decision.

Sui validators also use local disks, but we represent them as a 4 TiB static PV and PVC. The PV binds the physical path to its worker through node placement constraints, while the Retain reclaim policy prevents the underlying data from being deleted automatically even if the PVC is deleted and the PV is released. For Walrus, the simplicity of mounting the path directly mattered more. For Sui, tracking the StatefulSet identity and volume ownership through the Kubernetes API was more useful. The implementations differ because the relationships that must survive recovery differ, even when both workloads use local storage.

Both approaches follow the same recovery sequence: reapply the workload definition → select the worker that owns the data → reconnect the existing path or PVC → start the process. We do not need to recreate mount commands and startup sequences on each server, and we retain the performance of local NVMe. If network storage later becomes a better fit for a particular chain, we can select a PVC-backed path in the same chart without redesigning the entire workload.

The trade-off is that a node using local storage treats the worker and disk as a single failure domain. Kubernetes does not automatically move the data to another server. Rather than hide this limitation behind automatic recovery, we use directory validation, the Retain policy, and placement constraints to block an invalid startup. We switch to a chain-specific restore or resynchronization procedure only when the disk itself is lost. This preserves local-disk performance while making the safe reconnection conditions for each storage type repeatable under a common recovery policy.

Operating internal RPC endpoints separately from P2P identity

The RPC endpoints we operate are not exposed directly to external users. They are consumed by services inside the cluster and by chain-integration workloads. We therefore do not need to assign them public addresses or configure an external Ingress. Internal consumers still need a stable address when Pods are replaced. P2P, by contrast, must remain reachable from external peers, so we preserve the node key, advertised address, and port separately.

We place a ClusterIP Service in front of each RPC node and label it with the network name and RPC target role. When a Pod is recreated with a different IP, the EndpointSlice points to the new Pod while internal workloads continue using the existing Service DNS name. When adding an independent RPC node, we register its Service DNS name in the provider configuration instead of registering the Pod IP directly. Role labels also serve as the inventory key for querying and managing RPC targets by network.

The RPC provider layer used by the DVN follows the same principle. Gasolina separates internal RPC endpoints in the relevant namespace, dedicated hostNetwork RPC endpoints, and an external HTTPS fallback into distinct egress rules. The external HTTPS route is not an endpoint that exposes our RPC service. It is an external provider fallback used by internal workloads when needed. When the internal RPC configuration changes, we can review the allowed destinations and fallback routes together in the workload definition.

Avalanche RPC treats internal RPC and P2P differently even when they run on the same node. HTTP and WebSocket routes are available only to internal workloads through a ClusterIP Service. The staking port used by external peers, 9651, is bound through hostPort, and the public IP to advertise is configured explicitly. Internal consumers continue using the same Service address after a Pod replacement, while external peers reconnect through the existing P2P endpoint.

hostPort and hostNetwork preserve external endpoints directly, but they also constrain worker placement and introduce port-collision and node-firewall considerations. We therefore limit them to P2P and selected host-bound endpoints rather than using them as a general RPC exposure mechanism. For workloads such as Walrus that require hostNetwork, only the external storage HTTPS endpoint uses the host network. In-cluster access and metrics collection use a separate ClusterIP Service. We apply NetworkPolicy to the Pod network based on role, namespace, and port, while host-network paths are managed through node firewalls and explicit placement constraints.

With this design, internal RPC remains available through the same Service address regardless of Pod placement, while externally exposed paths are limited to endpoints that require them, such as P2P. The Service hides changing Pod IPs, while the P2P configuration preserves the node key, advertised address, and worker placement.

Defining validator replacement policies for each chain’s signing model

Kubernetes is designed to restore the desired replica count and restart failed Pods on other workers. For validators, we had to constrain this default behavior. Scaling replicas or automatically failing over instances that share the same validator identity and key could create two active signers while attempting to improve availability. Our operating criterion is therefore not the number of Pods, but whether exactly one logical signing authority exists at any given time.

On Initia, the validator does not hold the complete signing key directly. Instead, it submits signing requests to a 2-of-3 Horcrux signer set. Each signer runs as an independent workload and mounts only its own key share. Replacing the validator Pod does not require moving the complete key or signing state, and signing authority remains within the Horcrux threshold. However, splitting the signers into three workloads is not the same as distributing them across physical failure domains. The logical separation should also place the signers on independent workers and across separate failure domains.

This architecture is not created simply by running three signer Pods. The chain ID, cosigner addresses, and threshold must match across all three signers, and each Horcrux signer must connect to the designated validator’s remote-signing endpoint. We define these relationships in Helm values and reject conflicts at render time, such as setting chainNodes alongside the existing validator address or passing an empty chainNodes value. Invalid values are caught before Horcrux starts by moving this cross-validation into the template.

Within Ethereum, the workload design changes depending on where the signing state is owned. In the classic configuration, we replace a single-replica validator client using a Deployment with Recreate, while actual signing requests are sent to a separate Web3Signer. Because the keys and slashing-protection database live outside the client Pod, the client remains a replaceable execution layer. We also enable doppelganger protection to check for activity from the previous signer immediately after a restart. SSV, by contrast, does not call a single external signer. Multiple operators jointly perform validator duties with distributed key shares. Operators with their own identities and data paths run as a StatefulSet.

Recreate reduces overlap between the old and new Pods during a normal deployment, but it does not provide fencing for the previous execution environment. A worker being NotReady does not mean that the validator process on it has terminated. If the old Pod is force-deleted on a worker that has only lost connectivity to the control plane and a new instance starts with the same key, conflicting signatures may be produced when the old worker recovers. We therefore do not use automatic failover for active validators based only on worker state. We apply the change only after confirming that the previous signing authority has stopped and that its state has been handed over.

This choice can delay recovery and lead to missed duties or missed blocks during some failures. We consider irreversible loss from conflicting signatures a greater risk than a temporary reduction in availability. We do not use Kubernetes simply to bring validators back as quickly as possible. We use it to encode chain-specific signing boundaries in deployment policy and to keep automated steps and operator-confirmed handoff points within the same operational workflow.

Operating chain upgrades as verifiable changes

Once deployment definitions are consolidated in Git, the natural next step is to enable automatic synchronization and rolling updates. We stopped that automation for active validators. A Git change could immediately replace a signing workload or start a new instance before the previous one has stopped. If a binary changes the data format, reverting only the image tag may not restore the previous state. Controlling when a change begins and the conditions under which it stops mattered more than deployment speed.

Our upgrades therefore begin in Git but do not run automatically. After changing an image or binary, checksum, or runtime configuration, we review the actual diff and target in Argo CD, and an operator explicitly initiates synchronization. Kubernetes executes the approved sequence of changes, while our operating procedure determines when to replace an active validator and when to expand the rollout. For an RPC node pool, we apply the change to one instance first and verify connectivity and data compatibility before proceeding. A validator is replaced only after it passes the signing-transition conditions described earlier.

Choosing not to copy the entire chain database for every upgrade was also deliberate. Recovery of multi-terabyte datasets is handled by the local-disk, snapshot, or resynchronization procedures described earlier. The backup created during the upgrade sync wave preserves only node identity and operational state—small in size, but difficult to reconstruct. Rather than lengthening the deployment window by protecting everything in one step, we separated backup and recovery paths according to the nature of the state.

Once the backup Job completes, the validator is replaced using Recreate, while the three Horcrux signers continue running as separate applications. When the new validator’s remote-signing endpoint becomes available, the Horcrux signers reconnect to the validator Service. The change is not complete until we verify the running binary, block progression, and the resumption of signing. Here too, Recreate handles only the normal replacement order. Preventing double signing during a worker partition is handled separately according to the signing model. Horcrux and SSV depend on consistent quorum and signing state, while Web3Signer depends on shared slashing-protection state. With local signing, we must also confirm that the previous process has stopped or that its access to the key has been revoked.

Rollback decisions also distinguish between binaries and data. If the data format is unchanged, we can reapply the previous image and configuration. If a migration has occurred, the rollback path requires restoring an earlier snapshot or resynchronizing the node. Unless this distinction is made before deployment, the “rollback button” will fail at the moment it is needed most.

This process is slower than automated deployment. In return, the approved version, pre-change backup, single-instance replacement, and post-change verification remain connected as one change unit. Our goal in using Kubernetes for upgrades is not to maximize automation. It is to encode both the scope that can be automated safely and the points where the process must stop for verification.

Applying this change model across multiple chains requires a clear boundary between the management plane and the execution plane.

Separating the management plane from node execution

Running GitOps and access-management tools in the same cluster as chain workloads makes configuration and state inspection simpler. It also places failures or policy errors in those management tools within the same failure domain as the P2P and RPC execution paths. We decided that separating operational control from chain execution was more important than management convenience. What we centralized was not the node runtime, but the operational path through which changes are approved and applied.

Our management AKS currently hosts SSO and access policies, Argo CD, secret integration, and management logs, while a separate execution cluster handles the P2P and RPC workloads and chain data. Argo CD in AKS applies the desired state to the OVH cluster, but P2P and RPC traffic does not pass through the management plane. If the management plane fails, new deployments and centralized visibility become limited, while nodes that are already running continue participating in their chains and serving requests. Conversely, a disk or network failure in the execution cluster does not take the approval records and management systems down with it.

This architecture does not eliminate the risk associated with centralized management accounts. Argo CD and management credentials can affect deployment state across multiple chains, so a compromise can have a wide blast radius. To reduce that exposure, AppProject separates source repositories, target clusters, and allowed resource scopes. We do not enable automatic synchronization or automatic deletion for high-impact workloads such as validators. Secrets are not stored in Git. They are delivered as Secrets only to the namespaces that require them. Separating the management plane does not remove risk. It prevents change authority and execution failures from spreading across every area at once.

Management controls are reused centrally, while the blast radii of chain execution, data paths, and credentials remain separated. SOC 2 is where we connect the records showing that these boundaries and change procedures operated as intended.

Connecting operational records to evidence for SOC 2 controls

SOC 2 does not require the use of Kubernetes. What matters is demonstrating that the controls defined by the organization also operate in practice. Kubernetes and GitOps are useful because they let us connect the desired state recorded in Git, the changes applied by Argo CD, and the state running in Kubernetes into a single operational trail.

Consider the upgrade described earlier. It leaves behind the change request and approver, the applied commit, the operator-initiated Argo CD synchronization, the result of the pre-change backup, and the binary that actually ran. Adding block progression and the resumption of signing lets us confirm that the approved change was deployed and that the validator resumed its role. Instead of creating separate evidence after the fact, the operating process itself becomes the source of evidence.

The existence of Git records or Kubernetes logs alone is not sufficient. We must also define whether the required events are actually collected, whether they are retained externally so they remain available after a cluster failure or compromise, and whether recovery completion is verified at the service and chain levels rather than by a Pod’s Running state. This article covers only the broad structure through which access, change, and recovery controls are recorded along the same operational path. In the next article, we plan to turn this flow into concrete SOC 2 operating controls. Using real operational examples, we will examine periodic access reviews, Kubernetes API audit policy and external retention, methods for linking Git, Argo CD, and runtime state through a single change identifier, recovery testing and RTO and RPO measurement, and the considerations and evidence required for each risk assessment.

Closing

Looking back, adopting Kubernetes was not simply a matter of changing the runtime environment for our nodes. It was a process of revisiting the operational experience that had accumulated differently across chains and deciding what should become a shared standard and what should remain a chain-specific judgment. We’ve learned that standardization is not about eliminating differences, but about making different requirements explainable and manageable through the same set of criteria.

Our view of automation changed as well. What mattered more than automating additional tasks was whether changes and failures occurred within a predictable process. Our approach to operational automation is to encode recurring decisions in code and policy while routing high-risk decisions through explicit verification.

Going forward, we want infrastructure in which operational complexity does not grow in direct proportion to the number of chains. We will continue improving our shared policies and management systems while more clearly separating each chain’s execution environment and risk boundary. We also want deployment and recovery, access control, and operational records to form one coherent operating process rather than remain scattered across separate tasks.

If you are considering Kubernetes for blockchain node operations, start by examining how often the same operational decisions recur rather than focusing on features or trends. If each node requires solving the same problems again, or if operational knowledge remains confined to the experience of particular individuals, it may be time to move those decisions into the system. Our goal is not a larger cluster, but infrastructure that lets us operate more chains with the same level of confidence.