fix(taxonomy): reap stalled runs so taxonomy generation can recover (ENG-1924)

[pandeymangg]

What does this PR do?

A taxonomy run can be orphaned in pending/running if the taxonomy service crashes mid-run or its terminal callback (POST /internal/v1/taxonomy/runs/{id}/failed or the success PUT) is lost. The Hub never transitions the run, so Formbricks Web treats it as "in progress" — it disables Generate and polls GET /runs/{id} every 5s forever, recoverable only by a manual DB edit.

This adds a periodic Hub-side reaper that force-fails runs stalled past a timeout, so polling stops, the failure surfaces, and generation can be retried.

• TaxonomyRepository.FailStuckRuns(ctx, olderThan, message, code) — a single bulk UPDATE marking pending/running runs older than the cutoff (COALESCE(started_at, created_at)) as failed. Idempotent and race-safe: the status IN ('pending','running') filter skips runs that finished on their own between sweeps.
• runTaxonomyRunReaper — a ticker poller mirroring the existing runRiverQueueDepthPoller, launched in the API process only when the taxonomy service is configured. Cross-tenant, idempotent, safe under multiple replicas.
• Config — TAXONOMY_STUCK_RUN_TIMEOUT_SECONDS (default 1800) and TAXONOMY_REAPER_INTERVAL_SECONDS (default 60), via the existing DurationSec pattern.

Decisions flagged for review:

• Reaped runs are failed with error_code = internal_error and the raw message "taxonomy run timed out without completing" (operator-facing; Formbricks Web maps the code to a localized message — ENG-1917). Alternative: service_unavailable (retry-friendlier). Easy to flip.
• ⚠️ The default timeout (30 min) must exceed the longest legitimate run duration, or healthy long-running generations get reaped. Tune to the real max + margin.

No schema/migration changes (uses existing columns). No API changes.

Closes ENG-1924 · Follow-up to ENG-1213 · pairs with the Web-side wedge fix ENG-1917.

How should this be tested?

• Automated: TestTaxonomyRepository_FailStuckRuns (in tests/) creates a stuck running run (aged started_at), a stuck pending run (aged created_at), and a fresh run, then asserts the reaper fails the two stuck ones and leaves the fresh one. Runs via make tests (needs test_db) — executed in CI.
• Manual (end-to-end): with the taxonomy service configured, start a run and stop the taxonomy service mid-run so it's orphaned in running; within TAXONOMY_STUCK_RUN_TIMEOUT_SECONDS + TAXONOMY_REAPER_INTERVAL_SECONDS the reaper marks it failed, the UI stops polling, and Generate re-enables.
• Verified locally: go build ./..., make fmt, make lint (0 issues), go vet ./tests/.

Checklist

Required

• Filled out the "How to test" section in this PR
• Read Repository Guidelines
• Self-reviewed my own code
• Commented on my code in hard-to-understand bits
• Ran make build
• Ran make tests (integration tests in tests/)
• Ran make fmt and make lint; no new warnings
• Removed debug prints / temporary logging
• Merged the latest changes from main onto my branch with git pull origin main
• If database schema changed: added migration in migrations/ with goose annotations and ran make migrate-validate — N/A, no schema change

Appreciated

• If API changed: added or updated OpenAPI spec and ran contract tests (make tests or API contract workflow) — N/A, no API change
• If API behavior changed: added request/response examples or Swagger UI screenshots to this PR — N/A
• Updated docs in docs/ if changes were necessary
• Ran make tests-coverage for meaningful logic changes

🤖 Generated with Claude Code

[coderabbitai]

Walkthrough

Adds configurable timeout and interval settings for taxonomy run reaping. Implements a repository method that marks stale pending or running runs as failed with error and completion metadata. Wires a cancellable periodic reaper into application startup when taxonomy service configuration is present. Adds persistence tests for stale running and pending runs and verifies fresh runs remain active.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is conventional, concise, and accurately summarizes the taxonomy run reaper change.
Description check	✅ Passed	The PR description follows the template and fills the summary, testing, and checklist sections with relevant details.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/api/app.go`:
- Around line 805-809: Update runTaxonomyRunReaper to return immediately when
either interval or timeout is zero or negative, before creating the ticker or
calculating cutoff timestamps. This must safely disable the reaper and prevent
invalid configuration from causing a panic or processing active runs
incorrectly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: ec32e0f0-58ff-4837-b8bb-3e13c65289a0

📥 Commits

Reviewing files that changed from the base of the PR and between f6fba9e and 02ec991.

📒 Files selected for processing (4)

• cmd/api/app.go
• internal/config/config.go
• internal/repository/taxonomy_repository.go
• tests/taxonomy_persistence_test.go

[BhagyaAmarasinghe]

Some comments including AI review

[xernobyl]

Two low-severity notes, neither blocking — the change is correct and CI is green. Security pass (gitleaks + manual injection/PII/egress check) came back clean.

[xernobyl]

Thank you for the changes. LGTM.