DTL: eth_getBlockByNumber("finalized") permanently stuck at old block — TypeError in TransportDB._getFullBlock

[infoboy27]

Summary

Our Metis Andromeda mainnet replica node has had eth_getBlockByNumber("finalized", false) stuck at block 16392433 (produced 2024-04-10) for a very long time, while latest/safe advance normally and the node is otherwise fully synced. We only recently started monitoring the finalized tag specifically, so we can't say exactly when it first got stuck — but it's clearly not recent.

Environment

• dtl: was v0.2.2-1, upgraded to v0.2.6 to test — same behavior on both
• l2geth: was v0.3.7, upgraded to v0.3.9 alongside the dtl upgrade
• Node type: replica (not verifier)
• DATA_TRANSPORT_LAYER__SYNC_L1_BATCH=false, FP_BLOCK=22482000, L1_BEACON_ENDPOINT set to our own beacon (confirmed healthy, sync_distance=0)

Confirmed NOT a network-wide issue

https://andromeda.metis.io (public RPC) currently returns finalized=23120778 against latest=23122723 — normal, healthy lag. So this is isolated to our specific deployment/database, not an Andromeda-mainnet-wide condition.

Root cause (as far as we can tell)

The DTL's L1 ingestion loop is permanently stuck at latestBatch.index: 42764 / highestSyncedL1BatchIndex: 42764 (batch submitted at L1 block 19626685, prevTotalElements: 16382433, size: 10001 → last element = block 16392433, matching exactly what finalized is stuck at).

Every polling cycle also logs a recurring error from a different code path:

TypeError: Cannot read properties of null (reading 'transactions')
    at TransportDB._getFullBlock (packages/data-transport-layer/src/db/transport-db.ts:634:37)
    at TransportDB.getLatestBlock (packages/data-transport-layer/src/db/transport-db.ts:394:17)
    at <anonymous> (packages/data-transport-layer/src/services/server/service.ts:639:21)

getLatestBlock() calls _getLatestEntry(TRANSPORT_DB_KEYS.BLOCK), which is apparently returning null, and _getFullBlock(null) then crashes on null.transactions. This happens on every GET /block/latest/1088?backend=l1 request. We're not 100% sure this HTTP-triggered error is the actual cause of the batch-index freeze (vs. an unrelated symptom of the same underlying missing/corrupted local DB record), but it's the only concrete error we can see, and it reproduces identically on both v0.2.2-1 and v0.2.6.

We tried:

• Restarting the dtl container — no change, error recurs immediately.
• Upgrading dtl v0.2.2-1→v0.2.6 and l2geth v0.3.7→v0.3.9 — no change, identical error/stuck state on the new version.

Questions

1. Is this a known local-database corruption pattern? Is there a lighter-weight repair than a full resync (e.g., clearing/rebuilding just the "latest BLOCK" index key)?
2. Is a full DTL resync from L1_START_HEIGHT (13625200) the only path, and if so, roughly how long should that take against a normal L1 RPC (~12M blocks of L1 history to replay)?
3. Is DATA_TRANSPORT_LAYER__MINIO_ENABLED (pointing at memoapi.metis.io:6083, bucket metis-1088-prod) supposed to accelerate historical resync? We couldn't find where it's actually consumed in the current develop branch's l1-ingestion service — is it still functional / still recommended for replica nodes?

Happy to share our full DTL logs / DB if that helps debug.

[infoboy27]

Update — found the actual root cause and fix

Following up on this — the TypeError/stuck-batch symptom above was real but turned out to be a secondary effect of a config issue, not database corruption. Root cause:

1. DATA_TRANSPORT_LAYER__SYNC_L1_BATCH was set to false in our env — this effectively left the StateBatchAppended side of ingestion inert, so finalized (which depends on state batches, not just the sequencer-batch inbox) could never advance.
2. After setting SYNC_L1_BATCH=true, the DTL then failed to decode a later StateBatchAppended event with data signature does not match function appendStateBatchByChainId — some batches submitted after the blob-DA rollout appear to have a different calldata shape than this decoder expects, and instead of skipping the undecodable event, the ingestion loop stalled on the same L1 range indefinitely.
3. We also found l2geth was silently still running v0.3.7 in production despite docker-compose already specifying v0.3.9 (never redeployed after a prior config edit).

Fix applied:

• Force-recreated l2geth on v0.3.9.
• Set DATA_TRANSPORT_LAYER__SYNC_L1_BATCH=true and added DATA_TRANSPORT_LAYER__BATCH_INBOX_BLOB_SENDER.
• Built a locally-patched dtl image (based on v0.2.7-dev1) that skips a StateBatchAppended event when decoding throws INVALID_ARGUMENT, instead of stalling the checkpoint on it. This was the load-bearing change — SYNC_L1_BATCH=true alone just moved the stall to the first undecodable event.
• Adjusted the dtl healthcheck (curl :7878 → pgrep on the dtl process) since the HTTP healthcheck was timing out and flapping the container during the backfill catch-up.

Result: finalized is now advancing again (was permanently stuck since ~2024-04-10, confirmed via the block's own timestamp — much longer than we initially realized), currently backfilling through L1 history at a steady rate.

If this decode gap for newer StateBatchAppended calldata shapes is a known issue, a proper upstream fix (rather than our skip-on-error patch) would obviously be preferable — happy to share the exact failing transaction/calldata if useful.