Update fails silently on Zimacube 1: slot B permanently marked bad, system always falls back to A

[SeeDborg]

Slot B is marked bad by the bootloader. This explains why updates appear to succeed in the UI while the system keeps rebooting into A.

Steps taken to diagnose/fix

1. Checked journals around update attempts:
   journalctl --list-boots + journalctl -b <offset> -p err on the short boots corresponding to update attempts show no rauc/grub-editenv/mnt-boot errors, no useful trace at all. Whatever fails on slot B fails before persistent logging starts.
2. Forced boot to B:
   rauc status mark-active kernel.1 + reboot. System took over 10 minutes to come back, ended up on A again with B still marked bad. No confirmation it even attempted B.
3. Full manual reinstall via SSH:
   sudo rauc install zimaos-x86_64-1.7.0.raucb. Completed successfully end to end (boot.0, kernel.1, rootfs.1 all rewritten, "Installing done").
4. Rebooted after the clean reinstall: still stuck on A, still doesn't boot into B.

Analysis

This rules out bundle/package corruption: slot B receives a fully clean, freshly-written image and still fails to boot or gets reverted. This points to something at the bootloader/GRUB level rather than the update package itself.

The end symptom (slot B permanently marked bad, silent fallback to A) resembles what's described in #478 (mnt-boot.mount / grub-editenv failures during early boot), though I can't confirm it's the same root cause since my logs don't show those specific errors, the failure happens too early to be captured.

One detail that might be relevant: I also see nvme nvme1: Device not ready; aborting reset, CSTS=0x1 in the journal on normal boots on A too, so there may be an NVMe timing/reset issue on this hardware interacting badly with the A/B boot-attempt window.

What I'd need

I don't currently have physical console access to this unit, so I can't see what happens on screen during a B boot attempt (that's the next thing I'll try to arrange). Has anyone looked into what causes a slot to get marked bad here? Happy to provide more logs/output on request.

[SeeDborg]

Resolved: root cause identified

Got physical console access and found the actual failure. Slot B was hitting a kernel panic during boot, not a silent bootloader fallback:

lsblk: /dev/nvme0n1p5
/dev/nvme1n1p5: not a block device
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00002000

Hardware: IceWhale Technology CO,.LTD ADL-N/ADL-N, BIOS 5.27 05/06/2024

Root cause

A second NVMe SSD in the front bay was enumerating too slowly/inconsistently at boot. I'd already seen nvme nvme1: Device not ready; aborting reset, CSTS=0x1 in journal logs on normal boots on slot A, it just happened to squeak by in time there. On slot B, the init script's lsblk/root-mount step ran before nvme1n1p5 existed, and since init can't recover from a missing root, the kernel panicked instead of retrying.

Fix

Removed that front-bay NVMe SSD. Update completed successfully right after.

Suggestion for the team

This seems like it could bite anyone with a slower-enumerating NVMe in the front bay (or any secondary NVMe with marginal timing), since the current init flow doesn't appear to retry or wait on the root device before giving up. A retry/wait loop before the kill init panic path would likely prevent this class of failure, and might also explain some of the other "stuck update" reports floating around (e.g. #478), even if the underlying trigger differs case to case.

Closing this out, but happy to provide more detail if useful for a permanent fix upstream.