Feat(ops-broker): secure in-cluster broker for cluster-level test ops, with assets/restart/chaos suites#114
Open
e-300 wants to merge 3 commits into
Open
Feat(ops-broker): secure in-cluster broker for cluster-level test ops, with assets/restart/chaos suites#114e-300 wants to merge 3 commits into
e-300 wants to merge 3 commits into
Conversation
Signed-off-by: Luis Valdez <luisvaldez2h@gmail.com>
…haos mesh Signed-off-by: Ebad Shahid <ebad@ebad.dev>
… pkg/cluster SDK, and assets + restart test suites Signed-off-by: Ebad Shahid <ebad@ebad.dev>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an in-cluster ops-broker: a small trusted HTTP service (cmd/ops-broker) that exposes a fixed, allowlisted set of cluster operations behind bearer-token auth and tightly scoped RBAC. Tests reach it through the new pkg/cluster SDK rather than touching the cluster directly, keeping credentials in-cluster. On top of this, it adds three env-gated test suites — test/assets (pricing ground-truth), test/restart (restart recovery + Prometheus history survival), and test/chaos (Chaos Mesh fault injection) — addressing issues #88, #78, and #83. It is test/CI infrastructure only, with no product or runtime behavior changes.
Changes:
- New broker service with config/auth/handlers/k8s layers, fixtures, deploy manifests (RBAC, Deployment, Service, placeholder NetworkPolicy), and docs (API contract + run guide).
- New
pkg/clusterSDK plus three broker-driven, env-gated suites (assets/restart/chaos) with READMEs and unit tests using fake clientsets /httptest. - Supporting changes: k8s client-go dependencies in
go.mod/go.sum, a smoke script, and.gitignoreupdates.
Reviewed changes
Copilot reviewed 40 out of 42 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
cmd/ops-broker/*.go |
Broker server: config, bearer-token auth, HTTP handlers, k8s operations (apply/delete config, restart, kill pod, wait-ready, logs, chaos CRs). |
cmd/ops-broker/fixtures/pricing-fixed-v1.yaml |
Custom pricing fixture; unit comment contradicts the test's monthly interpretation. |
pkg/cluster/*.go |
Test-side SDK that calls the broker contract; covered by httptest-based unit tests. |
test/assets/* |
Ground-truth assets suite + pricing helpers/README validating node/disk discovery and cost math. |
test/restart/* |
Restart-recovery suite verifying readiness, no-panic, API 200, and history survival. |
test/chaos/* |
Chaos Mesh fault-injection suite; PassCriteria declared but not enforced, and kill-opencost asserts nothing. |
deploy/ops-broker/* |
RBAC/Deployment/Service/NetworkPolicy + kustomization; default resources include a crash-looping placeholder NetworkPolicy. |
docs/*.md |
Frozen broker API contract and step-by-step run guide. |
scripts/smoke-ops-broker.sh |
Smoke-test script for a deployed broker. |
go.mod / go.sum |
Adds k8s api/apimachinery/client-go (v0.33.13). |
.gitignore |
Ignores broker build artifacts/local token files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+10
to
+12
| # * Units are per HOUR: CPU $/core-hour, RAM $/GiB-hour (1024^3), storage | ||
| # $/GiB-hour. (The GCP sample storage 0.00005479452 == 0.04 / 730h, i.e. an | ||
| # hourly rate, confirms storage is hourly, not monthly.) |
Comment on lines
+13
to
+14
| - deployment.yaml | ||
| - networkpolicy.yaml |
Comment on lines
+18
to
+24
| // PassCriteria defines the expected outcomes for a chaos scenario | ||
| type PassCriteria struct { | ||
| MinSuccessRate float64 // Minimum percentage of requests that should succeed (0-1) | ||
| MaxLatency time.Duration // Maximum acceptable latency | ||
| RequiredErrorCode int // Expected HTTP status code for errors (0 if not checking) | ||
| RequireJSONErrors bool // Must errors be in JSON format | ||
| } |
Comment on lines
+79
to
+81
| Test: func(t *testing.T) { | ||
| testAllocationAPIWithRetries(3, defaultRetryInterval) | ||
| }, |
|
|
||
| ### Required Tools | ||
|
|
||
| - Go 1.19+ |
Comment on lines
+400
to
+405
| func min(a, b int) int { | ||
| if a < b { | ||
| return a | ||
| } | ||
| return b | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an in-cluster ops-broker: a small trusted service that exposes a fixed, allowlisted set of operations over authenticated HTTP, backed by tightly scoped RBAC.
Tests call it through the
pkg/clusterSDK and never touch the cluster directly credentials stay in-cluster, and tests are just untrusted callers pressing pre-approved buttons.Chaos Mesh is the injection engine: the broker creates allowlisted
PodChaos/NetworkChaosresources that Chaos Mesh executes (kill pods, partition/latency between OpenCost and Prometheus).On top of this it adds three broker-driven, env-gated suites —
test/assets(pricing ground-truth),test/restart(restart recovery + Prometheus history survival), andtest/chaos(4 fault scenarios).High Level Architecture
Available Operations
kubectlequivalentconfigmap/secret)kubectl apply -fkubectl delete configmap/secretkubectl rollout restart deploykubectl delete podkubectl rollout statuskubectl get podskubectl logskubectl get nodes -o jsonkubectl get pv,pvcRunning it (high level):
deploy/ops-broker/and create its auth token.go testagainst the broker.Full step-by-step is in
docs/running-broker-test-suites.md; the frozen HTTP APIis in
docs/broker-api-contract.md.Related Issues
#88, #78, #83
User Impact
None — test/CI infrastructure only; no product or runtime behavior changes.
Testing
(incl. the restart history-survival check and all 4 chaos scenarios).
go vet ./...+bats -r test/integrationagainstthe demo) — clean apart from 2 pre-existing, unrelated demo flakes.
httptest.scripts/smoke-ops-broker.sh).rollout-readiness race in the restart wait.