This chapter explains how observability answers, at the same time, whether a service is alive, whether business functions are available, whether data is fresh, and whether an anomaly is converging. The Root platform, Registrar nodes, Discovery nodes, content distribution platform, off-chain trust indexer, database, and official website backend each produce operational and business evidence. The website may use an old snapshot to improve the user experience, but an old snapshot cannot replace the actual backend health status. Metrics must include their source, time, status, and latency; unknown values must not be treated as zero.

Observational evidence is divided into four layers: raw facts defined by service interfaces or source code, operational metrics aggregated by the backend, display data used by official website pages, and smoke tests and alert records produced by operational procedures. When recording a metric, at least source, observedAt, dataStatus, and lag should be retained together so that “the interface has no data,” “the data is zero,” and “an old snapshot is being displayed” can be distinguished. A display result on the official website only shows that the frontend obtained a response at a certain time; it cannot replace independent verification of a node, database, or off-chain trust indexer.

19.1 Service Health and Readiness Model

The checking sequence should progress from “can it respond?” to “can it complete the business operation?” First verify the process and /health, then verify readiness and the database connection, and finally check the dependencies among the Root platform, Registrar nodes, Discovery nodes, content distribution platform, and off-chain trust indexer. A 200 response from an interface can only serve as evidence for that checkpoint. A service may be marked as business-available only after dependency checks, data status, and registration/discovery smoke tests have all passed. If summary continues to fail, the backend should remain in a failed state even if the page can still display an old snapshot.

Liveness only proves that the process can respond; readiness should also check necessary dependencies and configuration; business smoke tests verify real paths such as registration, publication, and discovery. A health check should record the target, response code, duration, dependency status, and time. A connectable database does not mean that the indexer has caught up with upstream events, and a responsive summary interface does not mean that the statistics are complete. Recovery confirmation requires at least consecutive successful health checks together with the corresponding business smoke tests.

19.2 Registration Metrics

Registration metrics should establish a traceable chain around each resource registration. It is recommended that requestId be bound to the registration request, resourceDid and the resource version or content digest be bound to the registration fact, and the publication sequence, distribution batch, and index version be bound to subsequent stages. Client-side repeated clicks or network retries may create multiple request records, but must not cause the same resource version to be counted as multiple successful registrations. When the VC is empty, the returned result should be recorded truthfully; an empty value must not be interpreted as a valid credential. Periodic statistics should be calculated in the aggregation layer using a unified time zone; the frontend must not accumulate counts based on page refreshes.

Registration metrics should distinguish request receipt, validation rejection, successful registration, VC return, Root platform acceptance, publication completion, and discovery visibility. Deduplication should use a request or resource-version correlation identifier rather than performing rough deduplication by IP address. Production statistics should exclude smoke tests that are explicitly marked as such. TODAY, THIS WEEK, and THIS MONTH must use a unified time zone and half-open time intervals; periodic values must not be confused with cumulative values because of caching or degradation.

19.3 Publication and Propagation Metrics

Propagation metrics use the resource DID, version, content digest, and publication sequence throughout all stages, and record the event, receipt, verification, distribution, and indexing times separately. End-to-end latency is the last queryable time minus the initial acceptance time; it must not be replaced by the duration of one individual segment. Retries and duplicate batches should be counted idempotently. A stalled cursor, sequence gap, or offline node should enter backlog or failure metrics.

The propagation chain should at least answer “at which stage did the resource stop?”: the Registrar node accepted it but the Root platform has not published it; the Root platform published it but content distribution has not completed; distribution completed but the Discovery node has not indexed it; or indexing completed but the query conditions did not match. The following correlation can be used for verification, rather than treating the resource count on a page as a substitute for propagation facts:

resourceDid + version
  -> publicationSequence
  -> distributionBatch / contentDigest
  -> discoveryIndexVersion / indexedAt
  -> queryRequestId / visibleAt

Retries of the same resource version should reuse an idempotency key or be marked as retries. If a record exists for only one stage, the operational result should explicitly state “accepted,” “propagating,” or “not yet queryable,” rather than simply reporting success.

19.4 Discovery Query Metrics

Discovery metrics distinguish exact DID queries, structured-condition queries, and semantic queries, and record request duration, result count, empty results, rejections, timeouts, and server errors. Relevance is not the same as trustworthiness. Results still need status, credential, publication proof, and freshness checks. Error responses retain correlation identifiers so that a request from the official website can be traced to Discovery node logs.

Query metrics should record the query mode, a digest of the normalized input, request identifier, service node, hit count, duration, response status, and data status. The original task description of a semantic query may be used for troubleshooting and quality analysis, but logs and statistics should not retain unnecessary personal or sensitive content; a digest, length, and hash can be used instead of the full text. An empty result must be evaluated together with whether the request was valid, whether the index is fresh, and whether the resource is within the node’s coverage; it cannot be directly classified as a Discovery node failure. After an exact DID query hits, the resource status, publication evidence, and version must still be checked.

19.5 Index Freshness and Synchronization Metrics

Synchronization status = upstream latest sequence - local cursor + latest successful time + latest error
Uncertain freshness -> mark as unknown; do not fill it with 0

Index rebuilding, incremental synchronization, synchronization pauses, and synchronization failures should use different status values. Operations personnel can then determine whether there are no new events, processing is delayed, or the synchronization path is interrupted. Synchronization observation includes the current event cursor, upstream latest sequence, lag distance, latest successful time, latest failure reason, index update time, and pending item count. A continuously increasing latest_sequence only shows that new observable events exist upstream; it does not by itself prove that this node has completed indexing. When the lag distance is unknown, display unknown rather than zero. Index rebuilding and incremental synchronization should be marked separately.

Freshness assessment should observe the upstream latest sequence, the locally processed cursor, the latest successful synchronization time, and the latest error together. If the latest_sequence of the off-chain trust indexer continues to increase, this only proves that observable upstream events still exist; the local cursor must also be checked for progress, along with any sequence or digest gaps and whether the Discovery node has completed the corresponding index. If there is no upstream sequence, the interface fails, or the node has not yet produced data, the status should be unknown or unavailable; it must not be disguised as a lag distance of zero.

19.6 Governance and Authorization Metrics

Governance metrics record the event source, on-chain location, event type, index cursor, node status changes, and latest synchronization time. Authorization eligibility, resource control, and publication results are observed separately. The indexing of a governance event does not mean that all resource statuses have been updated, and a normal node does not mean that every authorization artifact is valid. Verification failures should retain their evidence and reasons. Governance observation should also retain the original event digest and separately record the receipt, decoding, projection, and external-serving times of the off-chain trust indexer. Resource control is proven by the resource’s own signature and DID document; node authorization is proven by the governance path. These must remain separate in metrics and failure analysis.

19.7 Network-Level and Node-Level Statistics

The Network page may display both network-level aggregates and node-level details, but the aggregation source, update time, and missing nodes must be traceable. All periodic statistics use the same time zone, boundaries, and deduplication rules. When an old snapshot is used, it carries the snapshot time and degradation marker. Aggregated values must not simply add cumulative values with different definitions.

All windows use the same time zone and half-open interval [start, end). Missing data, request failure, and an actual zero value must be encoded separately; missing values must not be filled with zero.

19.7.1 Network-Level Metrics

Network-level metrics describe nodes and resources within the statistical scope, including node count, registration, publication, discovery, governance synchronization, and network status. Each metric should state its inclusion conditions, deduplication method, source interface, and update time. If a node lacks data, the aggregate result should state its coverage and missing data; the node must not be counted as zero merely because it did not report.

19.7.2 Node-Level Metrics

Node-level metrics record health status, authorization status, resource processing volume, synchronization cursor, index freshness, and latest error by node DID or stable node identifier. Offline, unauthorized, not-yet-synchronized, and statistics-interface failure states use different values, avoiding the concealment of real conditions with zeros or empty strings.

19.7.3 Registration, Publication, and Discovery Metrics

Registration, publication, and discovery metrics respectively represent acceptance by a Registrar node, trusted publication by the Root platform, and index visibility at a Discovery node. They must not be merged into one “success count.” The same resource version is correlated by DID, version, and publication sequence. Retries, duplicate indexing, and old snapshots must not create artificial growth.

19.7.4 Metric Time Windows

Time windows use a unified time zone and explicit boundaries. Periodic values use fixed start and end times, while cumulative values are calculated from the defined statistical start point. Interfaces, the official website Network page, and backend reports must use the same definition to avoid abnormal periodic values caused by differences in server time zones, request times, or cache generation times.

19.7.5 Missing Data and Degraded Display

Missing metrics, interface failures, data that has not yet been produced, and old-snapshot display are marked separately. A page may display old data with its time, but must show the degraded state. The backend continues to record the actual failure. Missing data must not be converted into zero, and a snapshot must not be treated as a real-time health conclusion.

19.8 Visibility of On-Chain Governance Events

Governance-event display should distinguish the on-chain event time, receipt and indexing time of the off-chain trust indexer, the time when the official website interface was generated, and the time when the page displayed the event. The event sequence or on-chain location is used for ordering and traceability; bilingual descriptions are display-layer mappings and cannot replace the original event. When the chain or indexing service is temporarily unavailable, the page should show an unavailable or delayed state.

19.9 Logs, Traces, and Correlation Identifiers

Cross-service records should at least correlate the request identifier, resource DID, version, publication sequence, event cursor, and node identifier, and should record the time, level, result, and error digest. Logs must not write private keys or sensitive request content. Error information exposed by public interfaces should be sufficient to locate a problem without revealing internal configuration.

Correlation fields at the interface layer should correspond to the protocol types: node-to-node requests use requestId, requestNonce, bodyHash, and upstreamAuth; resource propagation uses publicationCursor, batch sequence, and resource package version; discovery queries use the request identifier, Discovery node DID, candidate count, retrieval backend, and duration; the off-chain trust indexer should also record the last processed checkpoint, governance sequence, and synchronization time. These fields should be organized as structured logs rather than merely concatenated into a free-form message.

{
  "requestId": "req-example",
  "resourceDid": "did:oan:SKDM:example-resource",
  "packageVersion": "1.0.0",
  "publicationCursor": 42,
  "nodeDid": "did:oan:INDS:example-node",
  "operation": "discovery_query",
  "result": "completed",
  "elapsedMs": 18
}

The example only shows public or low-sensitivity fields that can be used to correlate a processing flow. Production logs must still filter resource descriptions, endpoint credentials, signature material, and internal error stacks according to the deployment policy. requestId supports cross-service tracing, but is not an authorization credential and cannot replace resource control proof or the governance-event sequence.

19.10 Snapshot, Cache, and Degraded-Mode Behavior

Snapshot replacement must satisfy four conditions: “complete response, structural validation, source update, and time update.” A partial response cannot replace a complete old snapshot. An old snapshot is a user-experience mechanism, not health proof. The frontend may first display a browser-side snapshot and refresh in the background. After success, it replaces the snapshot with complete new data and updates the source and time. After failure, it keeps the snapshot but displays the degraded state; continued failure still triggers a backend alert. A loading state is displayed only on a cache-free first visit; loading must not permanently cover data that is already available.

A snapshot record should contain at least the data source, generation time, schema version, and data status, for example:

{
  "source": "official-discovery-summary",
  "capturedAt": "2026-09-10T10:00:00Z",
  "dataStatus": "stale",
  "schemaVersion": 1,
  "coverage": "official-nodes",
  "payload": { "resourceCount": 12 }
}

Here, resourceCount: 12 is a business value from one particular snapshot and does not represent the current real-time count. The frontend may continue to display it when a refresh fails, but the backend must still record the refresh failure and its duration. When the interface recovers, only a complete response that passes structural validation may replace the snapshot.

19.10.1 Loading an Old Snapshot

When the page opens, it first reads an available browser-side snapshot and marks its generation time and source. If snapshot parsing fails, the schema version is incompatible, or the content integrity check fails, the snapshot should be discarded and the page should enter the first-load flow; damaged data must not be rendered as normal statistics.

19.10.2 Background Data Refresh

After displaying an old snapshot, the frontend requests the latest interface in the background without blocking the user’s view of the existing content. The request should have timeout and failure handling. The refresh must not change the resource details or query results currently being viewed unless the new response passes complete structural validation.

19.10.3 New Data Replacement

Only a complete and structurally valid new response may replace the old snapshot, while the cache time, source, and data status are updated together. If the new response is partially missing, retain the usable old data and mark the degraded state; an empty response or zero value must not replace valid content.

19.10.4 Long-Term Failure and Alerts

Consecutive failures, long-term absence of data, and a synchronization cursor that has not advanced for an extended period should enter backend alerting. Record the interface, first failure time, latest successful time, and evidence of the error. Continuing to display a snapshot must not suppress an alert. Recovery must be confirmed by consecutive successes and a business check.

19.10.5 First Load Without a Cache

When no snapshot exists, display the first-load state and explicitly finish it after success, failure, or timeout. A failure state should explain that the data is temporarily unavailable. It must not display loading text indefinitely, and it must not fabricate a zero value or an empty health conclusion.

19.11 Backup, Recovery, and Disaster Recovery

Backups cover node identity and authorization material, databases, resource packages, publication queues, synchronization cursors, configuration versions, and website deployment artifacts. Private keys and credentials are encrypted separately and access to them is restricted. After recovery, first verify integrity, sequence continuity, and key correspondence; then start write tasks, and confirm recovery through health checks, registration/discovery smoke tests, and index catch-up. The recovery order should first restore configuration and node identity, then the database and its migrations, resource packages, and publication queue. Next verify the synchronization cursor, governance authorization status, and website build artifacts, and finally restore external writes. The recovery process must prevent old queue items from being published repeatedly and prevent cursor rollback from causing duplicate projections. The service may first start in read-only mode or with writes paused; open writes only after checking resourceDid, version, publication sequence, and authorization status. A passing /health only means that the process has recovered; it cannot by itself prove that historical indexing, governance projection, and incomplete publication tasks have been restored.

19.12 Deployment Smoke Tests and Functional Verification

Minimum smoke-test loop:

1. Record the deployment commit, artifact digest, and environment configuration.
2. Check service health, database connection, and index cursor.
3. Use an isolated identity to complete registration and an expected-failure request.
4. Check Root platform publication, distribution, and exact/semantic discovery queries.
5. Save responses, logs, the timeline, and the cleanup result for test resources. Smoke tests use isolated test identities and test resources, and save the submission time, correlation identifiers, responses, logs, and cleanup results. Successful registration cannot replace discovery verification. Root platform publication, content distribution, index convergence, and database status must also be checked. If any key step fails, the response is uncertain, or production data remains uncleared, the deployment must not be declared successful. Smoke-test results should form a reviewable timeline rather than output only a “passed” string. Each step records the target address or service role, commit or artifact digest, request identifier, response status, duration, key response fields, and side-effect cleanup result. Registration acceptance, Root platform publication, and a discovery query hit are assessed separately. When writing a real resource to a production environment is inappropriate, use a controlled failure request to verify validation logic and an existing isolated test resource to verify the discovery path. If any key step is uncertain, the deployment status should remain pending verification.

19.12.1 Registration Function Smoke Test

Use an isolated identity and a controlled resource to verify request validation, control signatures, Registrar node acceptance, duplicate submission, and error responses. If the interface returns a registration VC, check its structure and its association with the resource DID, version, or digest. Clean up the test data after the test and save the evidence.

19.12.2 Discovery Function Smoke Test

Use a published isolated test resource to execute an exact DID query and a semantic query, and check the resource DID, version, status, source, publication evidence, and index time. If the result is empty, an old version is returned, or the interface fails, use the publication record and synchronization cursor to determine whether this is propagation delay or a functional failure.

19.12.3 Node Health Check

Check the liveness, readiness, and dependency status of the Root platform, Registrar nodes, Discovery nodes, content distribution platform, off-chain trust indexer, and official website backend one by one. Record the response code, duration, version, and check time; do not check only whether the process exists.

19.12.4 Database and Indexer Check

Check the database connection, migration status, critical records, and indexer cursor. Verify the relationship and continuous change between the upstream latest sequence, local cursor, and latest_sequence, and distinguish the absence of new events from a stopped synchronization.

19.12.5 Deployment Handling after Failure

When a key smoke test fails, stop publication confirmation and retain the logs, commit, artifact hash, and test-resource information, then handle the issue through the repair or rollback process. Only after a retry succeeds, data side effects have been cleaned up, and evidence is complete may an acceptance result be issued again.

19.13 Data Retention, Deletion, and Privacy Operations

Public metrics return only necessary aggregate data. Access statistics, request correlation identifiers, and operational logs should be controlled by backend permissions. Browser-side local identity private keys, backup contents, and sensitive material submitted by users must not be written to access statistics. Deletion operations should explain the different retention rules for online data, backup copies, and immutable audit records.

19.14 Operational Alerts and Long-Term Failure Detection

An alert record should contain at least the service, interface, first failure time, latest successful time, consecutive failure count, error digest, impact scope, and current status.

Use a stable fingerprint to deduplicate the same failure. A recovery alert must have both consecutive successful health checks and the corresponding business smoke test; it must not rely only on restarting the process. Long-term failure detection covers summary, governance synchronization, publication queue, and index-cursor stagnation. Alerts record the service, interface, consecutive failure count, first failure time, latest successful time, error digest, and impact scope, and deduplicate the same failure. Recovery requires consecutive success and business verification. When the website continues to display an old snapshot, backend alerts must not be suppressed. Periodic checks such as the summary interface should retain two facts: “the latest success” and “the latest failure,” rather than only retaining the current status. An alert event should at least correlate the service role, interface, time window, consecutive failure count, latest successful sequence or time, error digest, and impact scope. Short-lived network jitter for the same interface may be merged into one failure fingerprint, while continued failure should be escalated. During recovery, first confirm consecutive interface success, then confirm that the data time or cursor has advanced again, and finally execute the affected registration/discovery business smoke test. Continuing to display an old snapshot in the browser must not pause or clear this alert.

19.15 Public Service Status and Maintenance Communication

Public status information should state the affected service, start time, current status, data delay, and next update, without exposing keys, internal addresses, or user data. During maintenance, read-only queries or old snapshots may be retained, but the time point and unavailable functions should be marked. A recovery announcement should be based on actual checks and business smoke tests.

References