Skip to content

Mark add json-comp endpoint#574

Open
joanhey wants to merge 7 commits into
MDA2AV:mainfrom
joanhey:mark-comp
Open

Mark add json-comp endpoint#574
joanhey wants to merge 7 commits into
MDA2AV:mainfrom
joanhey:mark-comp

Conversation

@joanhey

@joanhey joanhey commented Apr 19, 2026

Copy link
Copy Markdown
Contributor

Description


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run all benchmark tests
/benchmark -f <framework> -t <test> Run a specific test
/benchmark -f <framework> --save Run and save results (updates leaderboard on merge)

Always specify -f <framework>. Results are automatically compared against the current leaderboard.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

@joanhey

joanhey commented Apr 19, 2026

Copy link
Copy Markdown
Contributor Author

Is it correct the new [test] baseline TCP fragmentation ??
Because not even nginx pass it.

@MDA2AV

MDA2AV commented Apr 19, 2026

Copy link
Copy Markdown
Owner

because nginx is some hacky implementation on these endpoints

@MDA2AV

MDA2AV commented Apr 20, 2026

Copy link
Copy Markdown
Owner

nginx now passes validation as well as all other frameworks so far, including swoole.

@joanhey

joanhey commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark -f mark

@github-actions

Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: mark | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 2,885,695 6725.0% 153MiB +12.7% +1.3%
baseline 4096 2,811,865 6434.6% 154MiB +9.7% +1.3%
pipelined 512 3,455,851 6621.5% 119MiB +10.4% +2.6%
pipelined 4096 3,693,793 6653.2% 157MiB +15.5% +3.3%
limited-conn 512 2,013,552 5936.8% 126MiB +10.3% +1.6%
limited-conn 4096 2,180,777 6300.4% 137MiB +8.1% +3.0%
json 4096 1,051,596 6436.9% 145MiB +5.4% +4.3%
json-comp 512 746,323 5989.8% 138MiB NEW NEW
json-comp 4096 790,147 6617.1% 153MiB NEW NEW
json-comp 16384 796,011 6339.1% 178MiB NEW NEW
upload 32 589 2153.1% 644MiB -0.3% +1.3%
upload 256 442 5647.4% 2.6GiB -9.4% +4.0%
api-4 256 41,331 254.7% 38MiB +4.3% +2.7%
api-16 1024 124,423 1083.5% 65MiB +0.7% +8.3%
static 1024 286,924 6571.1% 200MiB -0.5% +4.2%
static 4096 285,016 6472.1% 230MiB +0.5% -2.5%
static 6800 278,623 6282.1% 267MiB -1.3% +1.1%
async-db 1024 189,592 2895.5% 138MiB -7.6% +1.5%
Full log
  1453921 requests in 5.10s, 86.32GB read
Requests/sec: 285016.94
Transfer/sec:     16.92GB
[info] CPU 6472.1% | Mem 230MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.56ms   34.56ms 745.26ms   97.78%
    Req/Sec     4.44k   836.11    18.30k    93.93%
  1434361 requests in 5.10s, 85.16GB read
Requests/sec: 281318.19
Transfer/sec:     16.70GB
[info] CPU 6530.4% | Mem 235MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.71ms   36.26ms 881.47ms   97.79%
    Req/Sec     4.44k     0.89k   16.39k    93.53%
  1435704 requests in 5.10s, 85.24GB read
Requests/sec: 281493.70
Transfer/sec:     16.71GB
[info] CPU 6128.9% | Mem 236MiB

=== Best: 285016 req/s (CPU: 6472.1%, Mem: 230MiB) ===
[info] saved results/static/4096/mark.json
httparena-bench-mark
httparena-bench-mark

==============================================
=== mark / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.43ms  125.99ms   1.89s    95.18%
    Req/Sec     4.48k     1.32k   15.75k    87.94%
  1420091 requests in 5.10s, 84.31GB read
Requests/sec: 278382.24
Transfer/sec:     16.53GB
[info] CPU 6274.6% | Mem 260MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    50.48ms  132.45ms   1.81s    95.01%
    Req/Sec     4.46k     1.28k   15.04k    87.09%
  1411973 requests in 5.10s, 83.83GB read
Requests/sec: 276802.20
Transfer/sec:     16.43GB
[info] CPU 6392.6% | Mem 264MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    46.06ms  118.54ms   1.76s    95.47%
    Req/Sec     4.49k     1.25k   16.15k    88.64%
  1420903 requests in 5.10s, 84.36GB read
Requests/sec: 278623.46
Transfer/sec:     16.54GB
[info] CPU 6282.1% | Mem 267MiB

=== Best: 278623 req/s (CPU: 6282.1%, Mem: 267MiB) ===
[info] saved results/static/6800/mark.json
httparena-bench-mark
httparena-bench-mark

==============================================
=== mark / async-db / 1024c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   4.81ms   3.16ms   7.79ms   22.20ms   216.80ms

  1895927 requests in 10.00s, 1895929 responses
  Throughput: 189.55K req/s
  Bandwidth:  721.23MB/s
  Status codes: 2xx=1895929, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1895925 / 1895929 responses (100.0%)
  Reconnects: 75718
  Per-template: 377915,378894,380436,380151,378529
  Per-template-ok: 377915,378894,380436,380151,378529
[info] CPU 2895.5% | Mem 138MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   4.95ms   3.60ms   8.02ms   23.70ms   194.40ms

  1879761 requests in 10.00s, 1879761 responses
  Throughput: 187.94K req/s
  Bandwidth:  715.16MB/s
  Status codes: 2xx=1879761, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1879761 / 1879761 responses (100.0%)
  Reconnects: 75176
  Per-template: 374946,375954,376305,376754,375802
  Per-template-ok: 374946,375954,376305,376754,375802
[info] CPU 2952.4% | Mem 137MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   5.00ms   3.58ms   8.09ms   24.40ms   198.40ms

  1868523 requests in 10.00s, 1868525 responses
  Throughput: 186.81K req/s
  Bandwidth:  710.71MB/s
  Status codes: 2xx=1868525, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1868518 / 1868525 responses (100.0%)
  Reconnects: 74744
  Per-template: 373045,373462,374101,374773,373137
  Per-template-ok: 373045,373462,374101,374773,373137
[info] CPU 2971.9% | Mem 138MiB

=== Best: 189592 req/s (CPU: 2895.5%, Mem: 138MiB) ===
[info] input BW: 12.66MB/s (avg template: 70 bytes)
[info] saved results/async-db/1024/mark.json
httparena-bench-mark
httparena-bench-mark
[info] skip: mark does not subscribe to crud
[info] skip: mark does not subscribe to baseline-h2
[info] skip: mark does not subscribe to static-h2
[info] skip: mark does not subscribe to baseline-h3
[info] skip: mark does not subscribe to static-h3
[info] skip: mark does not subscribe to gateway-64
[info] skip: mark does not subscribe to gateway-h3
[info] skip: mark does not subscribe to production-stack
[info] skip: mark does not subscribe to unary-grpc
[info] skip: mark does not subscribe to unary-grpc-tls
[info] skip: mark does not subscribe to stream-grpc
[info] skip: mark does not subscribe to stream-grpc-tls
[info] skip: mark does not subscribe to echo-ws
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-16-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-4-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/async-db-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
[info] restoring loopback MTU to 65536
[info] restoring CPU governor → performance

@joanhey

joanhey commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark -f mark --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ /benchmark --save cannot start: main has diverged and cannot be auto-merged into this branch. Please merge or rebase main manually, push, and re-run /benchmark --save.

@joanhey

joanhey commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark -f mark --save

@github-actions

Copy link
Copy Markdown
Contributor

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: mark | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 2,802,794 6748.2% 152MiB +9.5% +0.7%
baseline 4096 2,887,122 6687.5% 157MiB +12.6% +3.3%
pipelined 512 3,680,607 6609.0% 120MiB +17.6% +3.4%
pipelined 4096 3,688,724 6441.5% 156MiB +15.4% +2.6%
limited-conn 512 2,000,001 5879.2% 127MiB +9.6% +2.4%
limited-conn 4096 2,194,435 6328.9% 136MiB +8.8% +2.3%
json 4096 1,059,267 6653.5% 144MiB +6.2% +3.6%
json-comp 512 549,662 5976.4% 135MiB NEW NEW
json-comp 4096 578,639 6360.6% 149MiB NEW NEW
json-comp 16384 577,858 6207.5% 178MiB NEW NEW
upload 32 590 2167.4% 694MiB -0.2% +9.1%
upload 256 463 5995.6% 2.4GiB -5.1% -4.0%
api-4 256 35,047 281.9% 39MiB -11.6% +5.4%
api-16 1024 130,964 1138.4% 64MiB +6.0% +6.7%
static 1024 288,784 6505.1% 193MiB +0.2% +0.5%
static 4096 287,265 6548.7% 233MiB +1.3% -1.3%
static 6800 281,790 6213.6% 268MiB -0.2% +1.5%
async-db 1024 187,687 2956.1% 138MiB -8.6% +1.5%
Full log
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.69ms   31.56ms 901.91ms   98.02%
    Req/Sec     4.50k     0.86k   17.79k    93.78%
  1452440 requests in 5.10s, 86.23GB read
Requests/sec: 284761.84
Transfer/sec:     16.91GB
[info] CPU 6355.5% | Mem 229MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.74ms   24.99ms 641.41ms   98.20%
    Req/Sec     4.53k   838.52    17.50k    93.11%
  1465287 requests in 5.10s, 86.99GB read
Requests/sec: 287265.21
Transfer/sec:     17.06GB
[info] CPU 6548.7% | Mem 233MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.58ms   30.27ms 753.50ms   98.02%
    Req/Sec     4.50k   812.61    17.33k    93.19%
  1450783 requests in 5.10s, 86.13GB read
Requests/sec: 284470.55
Transfer/sec:     16.89GB
[info] CPU 6316.6% | Mem 236MiB

=== Best: 287265 req/s (CPU: 6548.7%, Mem: 233MiB) ===
[info] saved results/static/4096/mark.json
httparena-bench-mark
httparena-bench-mark

==============================================
=== mark / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    46.23ms  118.98ms   1.79s    95.37%
    Req/Sec     4.54k     1.32k   21.14k    88.93%
  1436609 requests in 5.10s, 85.29GB read
Requests/sec: 281639.44
Transfer/sec:     16.72GB
[info] CPU 6313.7% | Mem 259MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.96ms  129.43ms   1.81s    95.16%
    Req/Sec     4.53k     1.29k   19.40k    87.66%
  1431510 requests in 5.10s, 84.99GB read
Requests/sec: 280687.85
Transfer/sec:     16.66GB
[info] CPU 6454.2% | Mem 266MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.74ms  128.90ms   1.99s    95.19%
    Req/Sec     4.51k     1.31k   16.82k    87.48%
  1437224 requests in 5.10s, 85.33GB read
  Socket errors: connect 0, read 0, write 0, timeout 3
Requests/sec: 281790.78
Transfer/sec:     16.73GB
[info] CPU 6213.6% | Mem 268MiB

=== Best: 281790 req/s (CPU: 6213.6%, Mem: 268MiB) ===
[info] saved results/static/6800/mark.json
httparena-bench-mark
httparena-bench-mark

==============================================
=== mark / async-db / 1024c (tool=gcannon) ===
==============================================
[info] resetting postgres for a clean per-profile baseline
[info] starting postgres sidecar
httparena-postgres
[info] postgres ready (seeded)
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   5.08ms   3.42ms   8.21ms   23.50ms   251.30ms

  1819087 requests in 10.00s, 1819089 responses
  Throughput: 181.87K req/s
  Bandwidth:  692.04MB/s
  Status codes: 2xx=1819089, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1819086 / 1819089 responses (100.0%)
  Reconnects: 72730
  Per-template: 362540,363801,364586,364789,363370
  Per-template-ok: 362540,363801,364586,364789,363370
[info] CPU 2832.9% | Mem 137MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   5.11ms   3.16ms   8.25ms   22.80ms   295.60ms

  1818176 requests in 10.00s, 1818178 responses
  Throughput: 181.77K req/s
  Bandwidth:  691.71MB/s
  Status codes: 2xx=1818178, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1818174 / 1818178 responses (100.0%)
  Reconnects: 72744
  Per-template: 362514,363380,364630,364248,363402
  Per-template-ok: 362514,363380,364630,364248,363402
[info] CPU 2867.3% | Mem 137MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   4.95ms   3.30ms   7.97ms   24.00ms   269.60ms

  1876872 requests in 10.00s, 1876876 responses
  Throughput: 187.65K req/s
  Bandwidth:  713.83MB/s
  Status codes: 2xx=1876876, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 1876873 / 1876876 responses (100.0%)
  Reconnects: 75099
  Per-template: 374687,375220,375955,376099,374912
  Per-template-ok: 374687,375220,375955,376099,374912
[info] CPU 2956.1% | Mem 138MiB

=== Best: 187687 req/s (CPU: 2956.1%, Mem: 138MiB) ===
[info] input BW: 12.53MB/s (avg template: 70 bytes)
[info] saved results/async-db/1024/mark.json
httparena-bench-mark
httparena-bench-mark
[info] skip: mark does not subscribe to crud
[info] skip: mark does not subscribe to fortunes
[info] skip: mark does not subscribe to baseline-h2
[info] skip: mark does not subscribe to static-h2
[info] skip: mark does not subscribe to baseline-h2c
[info] skip: mark does not subscribe to json-h2c
[info] skip: mark does not subscribe to baseline-h3
[info] skip: mark does not subscribe to static-h3
[info] skip: mark does not subscribe to gateway-64
[info] skip: mark does not subscribe to gateway-h3
[info] skip: mark does not subscribe to production-stack
[info] skip: mark does not subscribe to unary-grpc
[info] skip: mark does not subscribe to unary-grpc-tls
[info] skip: mark does not subscribe to stream-grpc
[info] skip: mark does not subscribe to stream-grpc-tls
[info] skip: mark does not subscribe to echo-ws
[info] skip: mark does not subscribe to echo-ws-pipeline
[info] skip: mark does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/mark.json - 18 new, 18 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
[info] restoring loopback MTU to 65536

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants