Skip to content

fix(observability): harden OTLP collector delivery - #5

Draft
afourniernv wants to merge 3 commits into
codex/3196-stack-resource-boundsfrom
codex/3196-stack-delivery-resilience
Draft

afourniernv wants to merge 3 commits into
codex/3196-stack-resource-boundsfrom
codex/3196-stack-delivery-resilience

Conversation

@afourniernv

Copy link
Copy Markdown
Owner

Summary

Make the sandbox trace relay tolerate a collector that is unavailable at gateway startup or restarts later, and surface collector-side partial acceptance instead of treating every successful gRPC call as full delivery. This is the fifth draft in the split OTLP relay stack.

Related Issue

Extends NVIDIA/OpenShell#2641 and the work in NVIDIA/OpenShell#3196. This draft does not close either item.

Changes

  • Configure the collector channel lazily so collector availability does not gate gateway startup
  • Reconnect through the tonic channel when a collector starts or returns after an outage
  • Keep bounded in-flight export admission from the preceding stack layer
  • Distinguish full success from OTLP partial_success
  • Log rejected-span counts and collector diagnostics
  • Bound the collector-recovery test probe with per-attempt and total deadlines
  • Document startup, reconnection, timeout, and partial-success behavior

Testing

  • mise run pre-commit passes
  • cargo test -q -p openshell-server otel_relay
  • cargo test -q -p openshell-server supervisor_otel_export
  • E2E tests added/updated (not applicable to this isolated delivery layer)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Documentation updated

Stack

Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv afourniernv mentioned this pull request Oct 5, 2026
10 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant