Monitor the user journey end to end
Infrastructure uptime alone does not prove that a Web3 product works. Define synthetic journeys for wallet connection, quote or state loading, transaction preparation, signature, submission, confirmation, indexing, and final UI update. Monitor each stage with identifiers that let operators follow one user action across frontend, API, worker, RPC, chain, and indexer.
Separate chain truth from derived application state. Dashboards should show block or slot lag, RPC latency and error rates, indexer checkpoints, queue depth, event-processing failures, database freshness, and the last successfully reconciled state.
Use chain-specific finality and freshness
Confirmation semantics differ across EVM networks, Solana, Sui, and other ecosystems. Define when the product may show submitted, confirmed, finalized, or settled, and record which status drives irreversible business actions. Alerts should account for normal block-time variation without masking real stalls.
RPC providers can return inconsistent or stale data. Compare critical reads where justified, track provider-specific errors, and maintain a tested failover path. Provider switching must preserve chain ID, commitment or finality settings, archive requirements, rate limits, and websocket subscriptions.
Observe contracts and privileged changes
Track upgrades, ownership changes, role grants and revocations, pauses, fee changes, treasury movements, oracle updates, bridge configuration, and other privileged events. Alert context should include chain, contract, transaction, actor, previous and new value, expected change window, and links to the approved release record.
Business invariants can reveal failures that infrastructure metrics miss. Monitor supply, reserves, shares, reward liabilities, queue totals, reconciliation differences, and unusual permission or transfer patterns according to the product model.
Write actionable incident runbooks
Each alert needs an owner, severity, trigger, validation query, immediate containment options, communication path, escalation target, recovery steps, and evidence to retain. Distinguish provider outage, indexing lag, UI regression, contract failure, key compromise, data corruption, and expected maintenance because their safe responses differ.
Runbooks should specify who may pause services or contracts and what approval is required. Recovery should include verification that chain and derived state agree before normal operation resumes.
Test operations before launch
Exercise RPC failover, websocket loss, delayed events, duplicate events, reorgs or forks where applicable, queue backlog, database restore, expired credentials, failed deployments, and alert escalation. Release evidence should include dashboard links, alert tests, synthetic journey results, backup proof, access ownership, and a dated handoff record.