Skip to content

fix: local-upgrade.sh rebuilds the search index only through core's --rebuild-if-stale - #30

Merged
leandrolerena merged 2 commits into
mainfrom
fix/issue-24
Nov 10, 2026
Merged

leandrolerena merged 2 commits into
mainfrom
fix/issue-24

Conversation

@leandrolerena

@leandrolerena leandrolerena commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Description

.make-scripts/local-upgrade.sh rebuilt the search index a second time after core's post-upgrade migrate hook had already rebuilt it. Since core chart 2.2.1-beta.9 the migrate Job runs core's backend/init.sh, which runs elastic_index --rebuild-if-stale and treats exit 3 as a warning. The script then compared images and schema hashes and ran its own elastic_index --rebuild --force. Every core or Elasticsearch upgrade got two full rebuilds and two search outages. And when Elasticsearch refused documents, the script's rebuild exited 3, which aborted make local-upgrade under set -euo pipefail even though the rebuild had succeeded.

  • After the rollout and ready waits, the script reads elastic_index --help once from the upgraded core pod (.make-scripts/local-upgrade.sh:252). If it lists --rebuild-if-stale, the script runs elastic_index --rebuild-if-stale instead of its own --rebuild --force (:255). That call does nothing when the hook's stamps match the running core and Elasticsearch. It rebuilds only when the hook stamped against the Elasticsearch pod that the StatefulSet rollout then replaced: helm upgrade here has no --wait, so the hook can run before that rollout.
  • For an older core the script keeps the image and schema-hash comparison.
  • Both paths go through run_elastic_index (:155-164). It treats exit 3 as a warning on stderr and passes any other non-zero exit through.
  • --sync-lookups is detected from the same captured help text, so the older-core path runs one exec fewer.
  • If the --help exec fails, the script falls back to the old path.

Look at first

  • .make-scripts/local-upgrade.sh:252-263: on a new core, the upgrade now always ends with one --rebuild-if-stale exec. That exec is cheap when nothing is stale: one stale check and a count of set-aside rows. If another rebuild holds core's advisory lock, it waits for it without a timeout, then re-checks.

Verification beyond CI

CI here only runs enforce-pr-label.

  • A stub harness that fakes kubectl and helm ran the earlier skip version through five cases. Before the fix, two fail: a new core still rebuilt, and exit 3 aborted the script. After the fix, all five pass: an old core rebuilds once when the schema moved, exit 3 is a warning, exit 1 fails, and nothing moved means no rebuild.
  • On the current head (2660301), a stub kubectl around run_elastic_index --rebuild-if-stale under set -euo pipefail: exit 0 continues, exit 3 warns and continues, exit 1 and exit 2 abort with that code. bash -n is clean.
  • shellcheck was not re-run on the current head.

Not verified

  • No real make local-upgrade on a kind cluster across a core or Elasticsearch image change.
  • Not checked on a real cluster that kubectl exec passes exit code 3 through.

Closes #24

A core whose elastic_index lists --rebuild-if-stale rebuilds stale indices in
its post-upgrade migrate hook, so the script no longer rebuilds them a second
time. For an older core the script keeps its image and schema-hash comparison,
and its own rebuild now treats exit 3 (documents set aside) as a warning instead
of aborting make local-upgrade.

Fixes #24
@leandrolerena leandrolerena added the pr-bugfix PR introduces a bugfix label Nov 9, 2026
@leandrolerena leandrolerena self-assigned this Nov 9, 2026
… rollout

helm upgrade has no --wait, so core's migrate hook can check the index
stamps against the Elasticsearch pod the rollout then replaces. Running
the same no-op-when-current check once the rollout is done closes that,
and shares the exit-3 handling with the older-core rebuild.
@leandrolerena leandrolerena changed the title fix: local-upgrade.sh leaves the search index to core's --rebuild-if-stale fix: local-upgrade.sh rebuilds the search index only through core's --rebuild-if-stale Nov 10, 2026
@leandrolerena
leandrolerena merged commit 936b733 into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix PR introduces a bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

local-upgrade.sh rebuilds the search index again after the migrate hook's --rebuild-if-stale

1 participant