Skip to content

feat: add moss bench command for latency benchmarking (closes #433) - #448

Open
JITHENTHIRIYA wants to merge 5 commits into
usemoss:mainfrom
JITHENTHIRIYA:feat/moss-bench-command
Open

feat: add moss bench command for latency benchmarking (closes #433)#448
JITHENTHIRIYA wants to merge 5 commits into
usemoss:mainfrom
JITHENTHIRIYA:feat/moss-bench-command

Conversation

@JITHENTHIRIYA

@JITHENTHIRIYA JITHENTHIRIYA commented Jul 20, 2026

Copy link
Copy Markdown

Pull Request Checklist

Please ensure that your PR meets the following requirements:

  • I have read the CONTRIBUTING guide.
  • I have updated the documentation (if applicable).
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

Description

Adds moss bench, a new top-level CLI command that runs a query workload against an index and reports p50/p95/p99 latency percentiles. Users can supply queries inline
(--query, repeatable) or from a file (--queries-file, one query per line), control the number of timed runs (--runs, default 20) and warmup iterations (--warmup,
default 3), and choose between local and cloud query modes. Output is a Rich table per-query plus an overall aggregate in human mode, or structured JSON with --json for
scripting and CI.

Fixes #433

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Changes

  • packages/moss-cli/src/moss_cli/commands/bench.py — new command implementation with _percentile (linear interpolation, no extra dependencies) and bench_command
  • packages/moss-cli/src/moss_cli/main.py — imports and registers bench_command as moss bench
  • packages/moss-cli/tests/test_bench.py — 15 tests covering _percentile edge cases, CLI validation errors, and happy-path scenarios (no credentials required)
  • packages/moss-cli/README.md — adds ### Benchmark section under Commands and a feature bullet in the Features list

Example Output

Loading index my-index locally...
Running 6 warmup + 40 timed iterations across 2 queries...

    Benchmark — my-index

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Query ┃ p50 (ms) ┃ p95 (ms) ┃ p99 (ms) ┃ min (ms) ┃ max (ms) ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ what is semantic search? │ 4.12 │ 5.88 │ 6.74 │ 3.91 │ 7.02 │
│ how does embedding work? │ 3.98 │ 5.21 │ 6.10 │ 3.75 │ 6.55 │
└──────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

Overall (40 runs)
p50: 4.05 ms
p95: 5.60 ms
p99: 6.48 ms
min: 3.75 ms max: 7.02 ms

Review in cubic

@CLAassistant

CLAassistant commented Jul 20, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HarshaNalluru

Copy link
Copy Markdown
Contributor

@JITHENTHIRIYA fix the failing checks.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codex review

The benchmark command is broadly coherent, but one added test currently passes even when the command under test crashes, so it does not protect the cloud path it claims to cover.


async def capture_query(index: str, q: str, opts: Any) -> Any:
captured.append(opts)
return mock_client.query.return_value

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CONSIDER This mock can raise inside the command because mock_client.query has already been replaced with capture_query, so mock_client.query.return_value no longer exists. Since the test also ignores the runner.invoke result, it can still pass after the command exits with an exception. Use a saved mock result and assert success, e.g. mock_result = MagicMock(); async def capture_query(...): captured.append(opts); return mock_result; result = runner.invoke(...); assert result.exit_code == 0, result.output.

@JITHENTHIRIYA

Copy link
Copy Markdown
Author

@HarshaNalluru I have fixed the failing checks. Please review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tooling: add a moss bench command to the CLI

3 participants