Upgrade Procedures
Work in Progress -- This page is under active development. Detailed step-by-step instructions for each upgrade path will be added as they are finalized.
Topics to Cover
Upgrade Methods
Cosmovisor-based Upgrades -- Automated binary upgrades triggered by on-chain governance proposals. Cosmovisor monitors the chain for upgrade plans and automatically swaps the binary at the specified block height.
Binary Replacement Upgrades -- Manual upgrade procedure involving stopping the node, replacing the
lithodbinary, and restarting. Suitable for non-governance upgrades or emergency patches.Governance Proposal Upgrades -- End-to-end workflow for proposing, voting on, and executing chain upgrades through the on-chain governance module.
Rollback Procedures -- Steps to revert to a previous binary version and chain state in the event of a failed upgrade.
Environment Promotion
All changes follow a strict promotion path through environments before reaching mainnet:
Devnet --> Staging --> MainnetEach environment serves as a validation gate. Changes must pass all tests, health checks, and approval requirements at each stage before promotion to the next.
Mainnet Deployment Gate
The mainnet deployment process follows a controlled, auditable pipeline:
Git Tag -- A release is tagged in the source repository (e.g.,
v1.2.0).CI Build -- The tagged commit triggers a CI pipeline that builds, tests, and signs the release artifacts.
Approval -- A designated CODEOWNER must explicitly approve the deployment.
ArgoCD Sync -- Upon approval, ArgoCD synchronizes the new version to the mainnet environment.
Rollback Procedures
Application Rollback (current flow)
Note: This page previously documented an ArgoCD-based rollback. The ArgoCD/Kubernetes deployment was never wired into Makalu — production runs on EC2 + Docker Compose. Updated 2026-05-12 to match reality.
Automatic --
.github/workflows/deploy-simple.yamlhas arollbackjob that runs onfailure()and restores the previousdocker-compose.yaml+.envsnapshot taken at deploy start. No operator intervention needed when CI itself detects deploy failure.Manual -- Operators can re-run a previous green deploy by triggering
deploy-simple.yamlviaworkflow_dispatchand pinning thetaginput to the prior successful image SHA. The post-deploy SHA-verify step confirms the rollback landed.
Database Rollback
Point-in-Time Recovery (PITR) -- Database state can be restored to any point in time within the retention window. This is critical for recovering from data corruption or unintended state changes introduced by a faulty upgrade.
Deployment Windows
Deployments are restricted to defined maintenance windows to minimize risk and ensure adequate staffing for monitoring:
Devnet
Any day
Any time
Staging
Monday -- Thursday
09:00 -- 17:00
Mainnet
Tuesday -- Thursday
14:00 -- 16:00
Freeze Periods
The following freeze periods restrict mainnet deployments to reduce risk during high-impact windows:
Major events
No mainnet deployments during major network events, governance votes, or external market events that could amplify the impact of any issues.
Minimum interval
A minimum of 48 hours must pass between successive mainnet deployments to allow adequate observation of each change.
Holiday freeze
No mainnet deployments between December 20 and January 5 (inclusive) to account for reduced staffing during the holiday period.
Last updated