Repository navigation
fix: local-upgrade.sh rebuilds the search index only through core's --rebuild-if-stale - #30
Merged
Merged
Conversation
A core whose elastic_index lists --rebuild-if-stale rebuilds stale indices in its post-upgrade migrate hook, so the script no longer rebuilds them a second time. For an older core the script keeps its image and schema-hash comparison, and its own rebuild now treats exit 3 (documents set aside) as a warning instead of aborting make local-upgrade. Fixes #24
… rollout helm upgrade has no --wait, so core's migrate hook can check the index stamps against the Elasticsearch pod the rollout then replaces. Running the same no-op-when-current check once the rollout is done closes that, and shares the exit-3 handling with the older-core rebuild.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
.make-scripts/local-upgrade.shrebuilt the search index a second time after core's post-upgrade migrate hook had already rebuilt it. Since core chart 2.2.1-beta.9 the migrate Job runs core'sbackend/init.sh, which runselastic_index --rebuild-if-staleand treats exit 3 as a warning. The script then compared images and schema hashes and ran its ownelastic_index --rebuild --force. Every core or Elasticsearch upgrade got two full rebuilds and two search outages. And when Elasticsearch refused documents, the script's rebuild exited 3, which abortedmake local-upgradeunderset -euo pipefaileven though the rebuild had succeeded.elastic_index --helponce from the upgraded core pod (.make-scripts/local-upgrade.sh:252). If it lists--rebuild-if-stale, the script runselastic_index --rebuild-if-staleinstead of its own--rebuild --force(:255). That call does nothing when the hook's stamps match the running core and Elasticsearch. It rebuilds only when the hook stamped against the Elasticsearch pod that the StatefulSet rollout then replaced:helm upgradehere has no--wait, so the hook can run before that rollout.run_elastic_index(:155-164). It treats exit 3 as a warning on stderr and passes any other non-zero exit through.--sync-lookupsis detected from the same captured help text, so the older-core path runs one exec fewer.--helpexec fails, the script falls back to the old path.Look at first
.make-scripts/local-upgrade.sh:252-263: on a new core, the upgrade now always ends with one--rebuild-if-staleexec. That exec is cheap when nothing is stale: one stale check and a count of set-aside rows. If another rebuild holds core's advisory lock, it waits for it without a timeout, then re-checks.Verification beyond CI
CI here only runs
enforce-pr-label.kubectlandhelmran the earlier skip version through five cases. Before the fix, two fail: a new core still rebuilt, and exit 3 aborted the script. After the fix, all five pass: an old core rebuilds once when the schema moved, exit 3 is a warning, exit 1 fails, and nothing moved means no rebuild.2660301), a stubkubectlaroundrun_elastic_index --rebuild-if-staleunderset -euo pipefail: exit 0 continues, exit 3 warns and continues, exit 1 and exit 2 abort with that code.bash -nis clean.shellcheckwas not re-run on the current head.Not verified
make local-upgradeon a kind cluster across a core or Elasticsearch image change.kubectl execpasses exit code 3 through.Closes #24