feat: add moss bench command for latency benchmarking (closes #433) - #448
feat: add moss bench command for latency benchmarking (closes #433)#448JITHENTHIRIYA wants to merge 5 commits into
Conversation
|
@JITHENTHIRIYA fix the failing checks. |
Codex reviewThe benchmark command is broadly coherent, but one added test currently passes even when the command under test crashes, so it does not protect the cloud path it claims to cover. |
|
|
||
| async def capture_query(index: str, q: str, opts: Any) -> Any: | ||
| captured.append(opts) | ||
| return mock_client.query.return_value |
There was a problem hiding this comment.
CONSIDER This mock can raise inside the command because mock_client.query has already been replaced with capture_query, so mock_client.query.return_value no longer exists. Since the test also ignores the runner.invoke result, it can still pass after the command exits with an exception. Use a saved mock result and assert success, e.g. mock_result = MagicMock(); async def capture_query(...): captured.append(opts); return mock_result; result = runner.invoke(...); assert result.exit_code == 0, result.output.
|
@HarshaNalluru I have fixed the failing checks. Please review |
Pull Request Checklist
Please ensure that your PR meets the following requirements:
Description
Adds
moss bench, a new top-level CLI command that runs a query workload against an index and reports p50/p95/p99 latency percentiles. Users can supply queries inline(
--query, repeatable) or from a file (--queries-file, one query per line), control the number of timed runs (--runs, default 20) and warmup iterations (--warmup,default 3), and choose between local and cloud query modes. Output is a Rich table per-query plus an overall aggregate in human mode, or structured JSON with
--jsonforscripting and CI.
Fixes #433
Type of Change
Changes
packages/moss-cli/src/moss_cli/commands/bench.py— new command implementation with_percentile(linear interpolation, no extra dependencies) andbench_commandpackages/moss-cli/src/moss_cli/main.py— imports and registersbench_commandasmoss benchpackages/moss-cli/tests/test_bench.py— 15 tests covering_percentileedge cases, CLI validation errors, and happy-path scenarios (no credentials required)packages/moss-cli/README.md— adds### Benchmarksection under Commands and a feature bullet in the Features listExample Output
Loading index my-index locally...
Running 6 warmup + 40 timed iterations across 2 queries...
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Query ┃ p50 (ms) ┃ p95 (ms) ┃ p99 (ms) ┃ min (ms) ┃ max (ms) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ what is semantic search? │ 4.12 │ 5.88 │ 6.74 │ 3.91 │ 7.02 │
│ how does embedding work? │ 3.98 │ 5.21 │ 6.10 │ 3.75 │ 6.55 │
└──────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
Overall (40 runs)
p50: 4.05 ms
p95: 5.60 ms
p99: 6.48 ms
min: 3.75 ms max: 7.02 ms