fix: Optimize cleanup jobs, eliminate raw ES client usage, add test coverage

[niemyjski]

Summary

Comprehensive optimization of cleanup jobs: eliminate all direct Elasticsearch client usage, fix critical bugs, restore correct loop semantics, and add full integration test coverage.

Critical Bug Fixes

@min_count:2 → @min:2 in GetDuplicateSignaturesAsync

The @min_count syntax is silently ignored by Foundatio Parsers, causing the aggregation to return ALL signature hashes (not just duplicates). This would have caused FixDuplicateStacks to treat every stack as a duplicate on the next run.

FixDuplicateStacks only ran one batch (regression from refactor)

The original code looped until GetDuplicateSignaturesAsync returned empty. The refactor broke this. Restored the while loop with:

• Batch tracking and per-batch logging
• Infinite-loop guard: exits if an entire batch fails to process
• ImmediateConsistency() on CountAsync inside GetDuplicateSignaturesAsync (one refresh per batch, matching original Indices.RefreshAsync call pattern — NOT per item)

ReassignStackAsync data-loss hazard on empty sequence

If sourceStackIds was empty, PatchAllAsync would have no stack filter and would reassign ALL events to the target stack. Added materialization + early return guard.

FixDuplicateStacks event-first ordering

Moved ReassignStackAsync before SaveAsync (soft-delete). If event reassignment fails, duplicate stacks remain visible and no data is lost. Previously: soft-delete first → event reassignment failure → orphan cleanup deletes the events.

GetDistinctFieldValuesAsync afterKey cursor leak

afterKey was only populated when composite.AfterKey != null, never cleared. Added: always clear cursor first, then repopulate. Callers checking afterKey.AfterKey.Count > 0 now correctly detect end-of-pagination.

Architecture (eliminate raw ES client from jobs)

• Refactored CleanupOrphanedDataJob to use repository methods exclusively
• Added GetDistinctFieldValuesAsync using composite aggregation (encapsulated in repository — composite aggregation is not in Foundatio's DSL, so raw client use is justified and documented)
• Added RemoveAllByProjectIds/RemoveAllByOrganizationIds, RemoveAllByStackIdsAsync to IEventRepository
• Added ReassignStackAsync using parameterized Painless script
• Added GetDuplicateSignaturesAsync to IStackRepository

Other Fixes

• Added OperationCanceledException filter before generic catch
• Added lock renewal at page boundaries in CleanupDataJob
• Consolidated two SaveAsync calls into one using spread syntax
• Removed redundant is_deleted:false filter (repository applies soft-delete filter by default)
• Reverted page size to 5 for org/project cleanup (2.5s sleep/item makes larger pages impractical)
• CompositeKeyResult converted from class to record
• Pattern matching throughout (is null/is not null, is [])
• XML documentation on composite aggregation rationale and Painless script safety
• Added :{Message} to error log format strings

Rebased onto main

Resolved merge conflict with #2268 (Deleted counter tests) — preserved all tests from both branches.

Test Coverage (39 job tests + 278 repo tests, all passing)

New job tests (named Method_Given_Expected):

• RunAsync_SuspendedOrganization_SuspendsRelatedTokens
• RunAsync_SoftDeletedOrganization_RemovesAllRelatedData
• RunAsync_SoftDeletedProject_RemovesProjectAndEvents
• RunAsync_SoftDeletedStack_RemovesStackAndEvents
• RunAsync_EventsOutsideRetentionPeriod_RemovesExpiredEvents
• DeleteOrphanedEventsByStack_WithLargeDataset_DeletesAllOrphanedEvents
• CleanupSoftDeletedOrganizations_WithMultiplePages_RemovesAllData
• CleanupSoftDeletedStacks_WithMultiplePages_RemovesAllStacks
• EnforceRetention_WithMultipleOrganizations_RespectsPerOrgRetention
• EnforceRetention_WithEventsOutsideRetention_DeletesOnlyExpiredEvents
• Plus 5 deleted-counter tests from #2268 (merged)

New repository tests:

• GetDistinctStackIds_WithMultipleStacks_ReturnsAllUniqueIds
• GetDistinctStackIds_WithPagination_ReturnsAllIds
• ReassignStack_WithSourceEvents_MovesAllEventsToTarget
• RemoveAllByProjectIds_WithMatchingEvents_RemovesAll
• RemoveAllByOrganizationIds_WithMatchingEvents_RemovesAll
• GetDuplicateSignatures_WithDuplicates_ReturnsSignatures
• GetDuplicateSignatures_WithNoDuplicates_ReturnsEmpty
• GetDuplicateSignatures_WithSoftDeletedStacks_ExcludesThem

[Copilot]

Pull request overview

This PR refactors cleanup jobs to rely on repository abstractions, improves duplicate stack cleanup behavior, and adds integration coverage for cleanup/repository operations.

Changes:

• Reworked orphaned-data and cleanup jobs with lock renewal, cancellation checks, and repository-based deletes/updates.
• Added event repository helpers for bulk deletion, stack reassignment, and distinct-id aggregation.
• Added integration tests for cleanup pagination, retention, duplicate signatures, and event repository operations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Exceptionless.Core/Jobs/CleanupDataJob.cs	Extends lock duration and renews locks during paged cleanup/retention loops.
src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs	Replaces direct Elasticsearch calls with repository methods for orphan cleanup and duplicate-stack fixing.
src/Exceptionless.Core/Repositories/EventRepository.cs	Adds bulk delete, stack reassignment, and composite aggregation helpers.
src/Exceptionless.Core/Repositories/Interfaces/IEventRepository.cs	Exposes new event repository cleanup/query APIs and composite cursor type.
src/Exceptionless.Core/Repositories/StackRepository.cs	Adds duplicate signature aggregation lookup.
src/Exceptionless.Core/Repositories/Interfaces/IStackRepository.cs	Exposes duplicate signature lookup API.
tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs	Adds cleanup pagination and retention integration coverage.
tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs	Adds integration coverage for orphan cleanup and duplicate stack merging.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs	Adds coverage for distinct ids, stack reassignment, and bulk delete helpers.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs	Adds duplicate signature repository coverage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.AppHost	39%	40%	134	❌
Exceptionless.Core	70%	63%	7897	➖
Exceptionless.Insulation	24%	23%	203	❌
Exceptionless.Web	72%	62%	4135	➖
Summary	69% (14171 / 20543)	61% (7305 / 11900)	12369	➖

[niemyjski]

Merged origin/main and addressed the outstanding review feedback in 92fa434.

Summary:

• Removed the accidental packages/ diff from the PR; the final diff against origin/main is back to the intended cleanup/repository/test files.
• Added/kept integration coverage for the new repository methods, duplicate signature lookup, soft-delete filtering, empty ReassignStackAsync safety, orphan cleanup, and cleanup usage accounting.
• Updated duplicate-stack cleanup to loop until no duplicate signatures remain and to renew locks during long-running cleanup paths.
• Documented the composite aggregation cursor path and aligned it with the current Elastic client API.
• Treated missing event indexes as zero matching events during event cleanup, which fixes empty soft-deleted project cleanup.

Verification:

• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 26 passed
• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupOrphanedDataJobTests: 24 passed
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.EventRepositoryTests: 17 passed, 1 skipped performance test
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.StackRepositoryTests: 13 passed

All review threads are resolved.