Add state machine for image switch (milestone 4c) by alicefr · Pull Request #50 · bootc-dev/bootc-operator

alicefr · 2026-06-03T09:50:25Z

Implements the daemon-side state machine that detects image mismatches between spec.desiredImage and the booted image,
stages updates via bootc switch in a background goroutine, and triggers reboot when desiredImageState == Booted.

Add Switch(ctx, image, apply) to the Executor interface
Rewrite the reconciler with async staging, cancel-on-spec-change, and single-patch-per-reconcile
Background goroutine signals completion via source.Channel, letting the reconcile loop handle all status updates through
one path
Apply operations (reboot) are never cancelled — the daemon restarts after reboot and reconciles fresh

alicefr · 2026-06-03T09:51:07Z

@jlebon I skipped the e2e test since we still need to define how to test the upgrade. The state machine is only checked by the env tests for now

alicefr · 2026-06-11T13:50:14Z

There are still some issue with this PR... investigating.

alicefr · 2026-06-11T15:14:58Z

I think we should prioritize #57 and implement at least the happy path test for the reboot

jlebon · 2026-06-17T02:56:38Z

I haven't re-reviewed past the 2nd commit yet (tests).

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

jlebon

Full review this time.

This is getting quite close! Very cool to see it all tested in the e2e test.

Rewrite the reconciler to detect image mismatches between spec.desiredImage and the booted image, stage via bootc switch in a background goroutine. Once, it finished to staged the image, the termination of the goroutine triggers once more the reconciliation loop which will detect that the system requires a reboot. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Replace raw JSON bytes with a bootc.Status struct in the test fake. Status() serializes the struct via json.Marshal, and Stage() auto-mutates the status (staging sets Staged). Reboot() records the call for test assertions. Add newBootcStatus() and newBootEntry() helpers to build test state without verbose JSON constants. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Replace the bootcStatusFull JSON constant with newBootcStatus() struct construction. Tighten the error assertion to match the exact error chain. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Add envtest cases for the daemon reconciler state machine: - TestStagingTriggered: image mismatch triggers bootc stage - TestStagingError: stage failure sets Degraded condition - TestAlreadyStaged: skip stage when image already staged - TestRebootingSet: reboot triggered when desiredImageState is Booted - TestRollback: restage when desired image changes - TestCancelInflightStage: spec change cancels in-flight stage Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

The TestUpdateReboot verifies that the upgrades to the new image is successfully performed. It starts by patching the desiredImage to a new one. After the reboot, the node should be Idle with the new booted image and schedulable, proving that the uncordon was successful. Additionally, the test verifies that the image is the one we built for upgrades by checking the existance of the file /usr/share/update-marker. We don't assert on intermediate states (Staging, Rebooting) because they are too transient to catch reliably with polling — the daemon restarts after the reboot so the Rebooting window is brief and a spurious reconcile during shutdown can overwrite it before the next poll. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

The daemon runs bootc switch which downloads the images. This operations requires consumes additional memory limits, otherwise it gets OOM killed. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

TestUpdateReboot fails because the node briefly enters a Degraded state during the reboot window, but the controller only logged the state name, not the daemon's error message. The daemon restarts after the reboot so its previous logs are lost. Log the Degraded condition's Message field so the next failure reveals the cause. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

alicefr · 2026-06-19T13:28:17Z

There has been some flaky test runs. I have removed the check for the rebooting reason because apparently the bootcnode is sometimes set to degraded if during reboot the bootc status doesn't report correctly the status. It was hard to detect it because the daemon restarts and we loose the logs while the controller was reporting only degraded without the reason and the message.

Although, the code is correct, even if the bootcnode is set to degraded for some time, it then recovers since the test failure was reporting healthy condition. Example of a run can be found here.

The e2e test verifies now that the end state is the desired one and we successfully managed to upgrade without checking intermediate conditions.

The test testcontrollermembership fails with the daemon not able pulling from the registry. This has already been reported in bink in: bootc-dev/bink#59 and fixed by: bootc-dev/bink#60 Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

jlebon · 2026-06-19T15:50:55Z

There has been some flaky test runs. I have removed the check for the rebooting reason because apparently the bootcnode is sometimes set to degraded if during reboot the bootc status doesn't report correctly the status.

Nice. Yeah this would be fixed by #50 (comment).

jlebon

LGTM, nice work! 🎉

alicefr marked this pull request as draft June 3, 2026 09:52

alicefr force-pushed the milestone-4c branch from ef264c4 to 0456b98 Compare June 3, 2026 10:12

alicefr marked this pull request as ready for review June 3, 2026 11:50

jlebon reviewed Jun 9, 2026

View reviewed changes

alicefr force-pushed the milestone-4c branch 4 times, most recently from 7e1ff20 to 2afd447 Compare June 10, 2026 12:36

alicefr mentioned this pull request Jun 11, 2026

Detect external bootc status changes via fsnotify (milestone 4d) #59

Open

alicefr marked this pull request as draft June 11, 2026 13:49

alicefr force-pushed the milestone-4c branch 2 times, most recently from 56119dc to ba87f0d Compare June 11, 2026 14:21

alicefr marked this pull request as ready for review June 11, 2026 15:14

alicefr force-pushed the milestone-4c branch 2 times, most recently from e4557ce to 413cf0d Compare June 12, 2026 11:47

alicefr mentioned this pull request Jun 12, 2026

Set the target image reference for the bink node #57

Merged

jlebon force-pushed the milestone-4c branch from 413cf0d to abc1f6f Compare June 12, 2026 15:46

alicefr force-pushed the milestone-4c branch 2 times, most recently from 1226849 to ed7b12d Compare June 16, 2026 07:23

jlebon reviewed Jun 17, 2026

View reviewed changes

daemon: add Stage and Reboot methods to bootc executor

45ebf7a

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

alicefr force-pushed the milestone-4c branch 2 times, most recently from e76449d to 7fdea08 Compare June 17, 2026 14:42

jlebon reviewed Jun 17, 2026

View reviewed changes

alicefr force-pushed the milestone-4c branch 2 times, most recently from 9dc4817 to 2a136b0 Compare June 18, 2026 11:53

alicefr force-pushed the milestone-4c branch 3 times, most recently from 9660931 to 75db0f2 Compare June 18, 2026 13:20

jlebon mentioned this pull request Jun 18, 2026

Don't unnecessarily update status conditions if nothing actually changed #66

Open

jlebon reviewed Jun 18, 2026

View reviewed changes

alicefr force-pushed the milestone-4c branch from 75db0f2 to b4939c8 Compare June 19, 2026 11:23

alicefr added 2 commits June 19, 2026 13:24

alicefr force-pushed the milestone-4c branch from b4939c8 to 58ff9fb Compare June 19, 2026 11:33

alicefr added 3 commits June 19, 2026 11:45

e2e: add NodeImageUpdateDigestedPullSpec to e2e env

1ccfeeb

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Alice Frosi <afrosi@redhat.com>

alicefr force-pushed the milestone-4c branch from 58ff9fb to 31a4c84 Compare June 19, 2026 11:46

alicefr added 3 commits June 19, 2026 13:19

alicefr force-pushed the milestone-4c branch from e67b557 to c6bfcfb Compare June 19, 2026 13:22

jlebon approved these changes Jun 19, 2026

View reviewed changes

Comment thread internal/daemon/reconciler_test.go

jlebon merged commit 8f9a5e9 into bootc-dev:main Jun 19, 2026
7 checks passed

Conversation

alicefr commented Jun 3, 2026

Uh oh!

alicefr commented Jun 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alicefr commented Jun 11, 2026

Uh oh!

alicefr commented Jun 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlebon commented Jun 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alicefr commented Jun 19, 2026

Uh oh!

jlebon commented Jun 19, 2026

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants