Run a command, watch its entire process tree, and report how much CPU,
memory, and disk I/O it used. The companion binary is named tricorder.
Think of it as a more thorough /usr/bin/time -v: it polls the process tree
on an interval, follows children and grandchildren, and writes a single
record summarising peak memory, total bytes read/written, average CPU load,
and total CPU time. Output is a tidy TSV, one-line JSON, or pretty-printed
JSON, suited to spreadsheets, pipelines, and human readers.
Visit us at Fulcrum Genomics to learn more about how we can power your bioinformatics with tricord and beyond.
- Whole-tree accounting — follows forked children and aggregates their resource usage; an exited child's I/O is still counted.
- Three output formats — TSV (Snakemake-compatible prefix, extended with
tricord-specific columns by default), one-line JSON (--format json) for programmatic consumers, or pretty-printed JSON (--format json-pretty) for human readers. Pass--snakemaketo emit the strict 10-column Snakemake schema when downstream tooling pins to it. - Optional one-line summary to stderr (
--summary). - Optional per-tick trace (
--trace <PATH>) — one TSV row per sample, so you can see how memory, I/O, and CPU evolved during the run instead of just the final aggregate. - Optional Markdown export (
--export-markdown <PATH>) — renders the aggregate record as a two-column Markdown table, ready to paste into PR descriptions, issue comments, and design docs. - Clean shutdown — forwards
SIGINT/SIGTERM/SIGHUPto the child's process group so orchestrators can tear runs down without leaking children. - No runtime dependencies — single self-contained Rust binary.
No Python, no
psutil. - Cross-platform — Linux (full column set) and macOS (graceful degradation; see Platform notes).
Requires Rust 1.94.0 or later.
cargo install tricord
# This installs the `tricorder` binary (the crate name is `tricord`,
# the binary name is `tricorder`).
# Or, from source:
git clone https://github.com/fg-labs/tricord.git
cd tricord
cargo build --release
# Binary is at target/release/tricordertricorder --out timing.tsv -- bash -c 'samtools sort -@ 8 big.bam -o sorted.bam'Usage: tricorder [OPTIONS] --out <PATH> -- <CMD>...
Options:
--out <PATH> Output file path
--format <FORMAT> tsv | json | json-pretty [default: tsv]
--interval <SECONDS> Sampling interval [default: 0.5]
--summary Print one-line summary to stderr after the run
--trace <PATH> Also write a per-tick TSV trace to this path
--export-markdown <PATH>
Also write a Markdown table of the aggregate
record to this path
--snakemake Emit only the original Snakemake aggregate schema
(TSV, JSON, and Markdown). Does not affect --trace.
-v, --verbose... Increase log level (-v, -vv, -vvv)
-h, --help Print help
-V, --version Print version
The command to benchmark is everything after the -- separator. No shell
interpretation is done by tricorder itself; if you need shell features (pipes,
quoting), invoke bash -c '...' explicitly.
When used in a pipeline, tricorder releases its own copies of pipe-backed stdin
and stdout after spawning the command. The command keeps the original streams:
closing its stdout lets downstream readers see EOF immediately, and closing its
stdin lets upstream writers detect a broken pipe, even while the command is still
running. Stderr remains available for diagnostics and summaries. Other copies of
the same pipe, such as stderr redirected with 2>&1, can still keep it open.
Use ordinary file paths for benchmark outputs. Pipe-backed /dev/stdout,
/dev/stdin, and equivalent descriptor paths are rejected before the command starts.
Library calls to tricord::run_command leave the caller's standard streams alone.
Use tricord::run_command_releasing_pipes to opt into the CLI behavior. This
permanently redirects the caller's pipe- or socket-backed stdin/stdout to
/dev/null, affecting every thread and subsequent command in that process;
regular files and terminals are unchanged. Rust's stdout buffer is flushed before
redirection. Callers must flush any other buffering layers and stop concurrent
access to these descriptors before opting in.
By default tricorder emits a Snakemake-compatible 10-column prefix followed
by every column it has added on top (tricord-extended schema). Pass
--snakemake to drop the additions and produce exactly the original
Snakemake schema; the prefix's column order and value formatting are
identical in both modes.
s h:m:s max_rss max_vms max_uss max_pss io_in io_out mean_load cpu_time major_page_faults minor_page_faults voluntary_ctx_switches involuntary_ctx_switches peak_n_threads peak_n_procs loadavg_1m_start loadavg_1m_end max_swap page_cache_start page_cache_end
12.3456 0:00:12 101.50 2048.00 95.20 96.00 1.25 0.50 175.00 21.60 42 1234 80 7 13 4 0.50 2.25 64.00 512.00 480.25
| Column | Units | Meaning |
|---|---|---|
s |
seconds (%.4f) |
Wall-clock running time |
h:m:s |
H:MM:SS |
Same value, human-readable |
max_rss |
MiB (%.2f) |
Larger of the sampled simultaneous tree RSS peak and the exit-accounted per-process RSS high-water mark; not an exact tree peak |
max_vms |
MiB | Peak summed virtual memory size |
max_uss |
MiB | Peak summed unique set size |
max_pss |
MiB | Peak summed proportional set size (Linux only — see below) |
io_in |
MiB | Total bytes read from disk by the process tree |
io_out |
MiB | Total bytes written to disk by the process tree |
mean_load |
percent of one core | Average CPU load over the run (e.g. 175 = 1.75 cores) |
cpu_time |
seconds | Total user + system CPU time across the process tree |
major_page_faults |
integer | Total major page faults (pages brought in from backing store) across the tree. tricord-added; omitted under --snakemake. |
minor_page_faults |
integer | Total minor page faults across the tree. tricord-added; omitted under --snakemake. Always - on macOS — see Platform notes. |
voluntary_ctx_switches |
integer | Total voluntary context switches across the tree (processes yielding the CPU on their own — typically waiting on I/O or a sleep). tricord-added; omitted under --snakemake. Always - on macOS. |
involuntary_ctx_switches |
integer | Total involuntary context switches across the tree (processes preempted by the scheduler). tricord-added; omitted under --snakemake. Always - on macOS. |
peak_n_threads |
integer | Peak instantaneous thread count across the tree — max over sampling ticks of the summed per-process thread counts. tricord-added; omitted under --snakemake. |
peak_n_procs |
integer | Peak instantaneous live-process count across the tree — max over sampling ticks of how many processes were observed. Catches "I asked for 16 workers but the tool spawned 200." tricord-added; omitted under --snakemake. |
loadavg_1m_start |
float (%.2f) |
System 1-minute load average sampled just before the child started. Frames the rest of the numbers — a peak cpu_time of 800% on an idle host (loadavg ~1) means something very different from the same peak on a thrashing host. tricord-added; omitted under --snakemake. |
loadavg_1m_end |
float (%.2f) |
System 1-minute load average sampled just after the child exited. Paired with loadavg_1m_start to show whether the run drove the host load up. tricord-added; omitted under --snakemake. |
max_swap |
MiB | Peak summed swap usage across the process tree (max over sampling ticks of VmSwap summed across PIDs). tricord-added; omitted under --snakemake. Always - on macOS — the kernel has no public per-process swap-usage API. |
page_cache_start |
MiB | System page-cache size (Cached in /proc/meminfo) sampled just before the child started. Frames the memory numbers the same way loadavg_1m_start frames CPU numbers: a run slowed by page-cache eviction from other work on the host looks identical to a real regression without it. tricord-added; omitted under --snakemake. On macOS renders - (no Cached equivalent) — see Platform notes. |
page_cache_end |
MiB | System page-cache size sampled just after the child exited. Paired with page_cache_start — a drop is a direct sign the run (or a neighbor) evicted cached pages. tricord-added; omitted under --snakemake. On macOS renders -. |
Missing values render as -. Even when a command exits before the first sampling tick, exit accounting supplies CPU time, mean load, and an RSS high-water mark; unsampled metrics remain - (JSON null), rather than zero. data_collected means either sampling or exit accounting succeeded. Records without either source retain the all-NA resource row.
BenchmarkRecord::peak_n_procs is now Option<u64> in the Rust API: None means no sampling tick succeeded. Existing sampled records retain their numeric JSON value; exit-only records use null. TSV column order and formatting are unchanged, but short runs now report available exit metrics instead of an all-NA row.
Wall time runs from just before spawning the child until its wait returns, excluding sampler shutdown. The sampler is woken on exit instead of waiting for the next interval.
CPU time is the larger of the sampled per-PID cumulative total and wait4's final user + system CPU total. These overlap and are never added. Exit accounting includes the direct child and descendants whose usage propagated through waits, recovering short-lived commands and the final unsampled CPU tail. It does not guarantee complete accounting for orphaned or unwaited descendants; taking the larger total can still miss work unique to each source. RSS similarly takes the larger of the sampled simultaneous tree peak and the exit-accounted per-process high-water mark, which is not an exact simultaneous tree peak. Other memory metrics and I/O remain sampled. Per-tick traces remain sampled and can end below the final aggregate CPU total.
When --trace is given, tricorder also writes a separate TSV with one row
per sampling tick — useful for "did it spike or stay flat?" plots and for
post-mortem on OOM kills. The aggregate --out file is unaffected.
s rss vms uss pss io_in io_out cpu_time n_procs major_page_faults minor_page_faults voluntary_ctx_switches involuntary_ctx_switches n_threads swap
0.5012 102.30 2048.00 95.20 96.00 1.25 0.50 0.75 3 0 850 12 1 5 0.00
1.0027 120.45 2048.00 112.10 113.00 2.50 1.00 1.55 3 2 1200 28 3 7 16.50
1.5042 101.50 2048.00 95.20 96.00 2.50 1.00 2.40 2 0 340 15 0 4 8.25
| Column | Units | Meaning |
|---|---|---|
s |
seconds (%.4f) |
Time since tricorder started |
rss, vms, uss, pss |
MiB (%.2f) |
Instantaneous sum across the live process tree at this tick |
io_in, io_out |
MiB | Cumulative bytes read/written across every PID observed so far (including exited children) |
cpu_time |
seconds | Cumulative user + system CPU time across observed PIDs |
n_procs |
integer | Number of live processes in this tick |
major_page_faults, minor_page_faults |
integer | Page faults that occurred during this tick (per-tick delta, summed across observed PIDs). Minor is always - on macOS. |
voluntary_ctx_switches, involuntary_ctx_switches |
integer | Context switches that occurred during this tick (per-tick delta, summed across observed PIDs). Both are always - on macOS. |
n_threads |
integer | Live thread count across the tree at this tick (sum across PIDs). Instantaneous, not cumulative. |
swap |
MiB | Summed swap usage across the live tree at this tick. Instantaneous, not cumulative. Always - on macOS. |
Memory columns are instantaneous, so they can go up or down between rows;
I/O, CPU, and the page-fault deltas describe activity within the tick — they
can be zero or non-zero per row but won't drop a previously-counted total.
The trace is tricord-native and is not affected by --snakemake.
Default (full) mode includes tricord-added fields:
{"running_time":12.3456,"max_rss":101.5,"max_vms":2048.0,"max_uss":95.2,"max_pss":96.0,"io_in":1.25,"io_out":0.5,"mean_load":175.0,"cpu_time":21.6,"major_page_faults":42,"minor_page_faults":1234,"voluntary_ctx_switches":80,"involuntary_ctx_switches":7,"peak_n_threads":13,"peak_n_procs":4,"loadavg_1m_start":0.50,"loadavg_1m_end":2.25,"max_swap":64.0,"page_cache_start":512.0,"page_cache_end":480.25,"data_collected":true}Under --snakemake the tricord-added keys are absent from the object
(not set to null), so downstream parsers that hard-code the Snakemake key
set see exactly what they would have seen from snakemake.benchmark:
{"running_time":12.3456,"max_rss":101.5,"max_vms":2048.0,"max_uss":95.2,"max_pss":96.0,"io_in":1.25,"io_out":0.5,"mean_load":175.0,"cpu_time":21.6,"data_collected":true}Raw numeric types, null for metrics with no observation: unsupported platform metrics, failed reads, or sampled metrics when no sampling tick succeeded.
When --export-markdown is given, tricorder writes a two-column Markdown
table of the aggregate record to that path, alongside the primary --out
file. Designed for pasting into PR descriptions, issue comments, and design
docs — for the per-tick trace, keep using --trace.
| metric | value |
|:----------|--------:|
| s | 12.3456 |
| h:m:s | 0:00:12 |
| max_rss | 101.50 |
| max_vms | 2048.00 |
| max_uss | 95.20 |
| max_pss | 96.00 |
| io_in | 1.25 |
| io_out | 0.50 |
| mean_load | 175.00 |
| cpu_time | 21.60 |Same column order, value formatting, and - / NA rules as the TSV.
tricord runs on Linux and macOS. The Linux implementation reads
/proc/<pid>/{stat,status,smaps_rollup,io} via the procfs crate; the
macOS implementation uses libproc's proc_pidinfo and proc_pid_rusage
(RUSAGE_INFO_V4).
| Metric | Linux | macOS |
|---|---|---|
max_rss |
/proc/<pid>/status, reconciled with wait4 (KiB converted to MiB) |
proc_taskinfo, reconciled with wait4 (bytes converted to MiB) |
max_vms |
/proc/<pid>/status |
proc_taskinfo |
max_uss |
/proc/<pid>/smaps_rollup (Private_Clean + Private_Dirty) |
proc_pid_rusage::ri_phys_footprint |
max_pss |
/proc/<pid>/smaps_rollup (Pss) |
mirrors max_uss (kernel does not compute PSS — see below) |
io_in, io_out |
/proc/<pid>/io |
proc_pid_rusage::ri_diskio_* |
cpu_time |
/proc/<pid>/stat (utime + stime), reconciled with wait4 |
proc_taskinfo::pti_total_user + pti_total_system, reconciled with wait4 |
major_page_faults |
/proc/<pid>/stat (majflt) |
proc_pid_rusage::ri_pageins |
minor_page_faults |
/proc/<pid>/stat (minflt) |
not exposed — column is - |
voluntary_ctx_switches, involuntary_ctx_switches |
/proc/<pid>/status (voluntary_ctxt_switches, nonvoluntary_ctxt_switches) |
not split by proc_pid_rusage — both columns are - |
peak_n_threads, n_threads |
/proc/<pid>/status (threads) |
proc_taskinfo::pti_threadnum |
loadavg_1m_start, loadavg_1m_end |
/proc/loadavg (first field) |
getloadavg(3) |
max_swap, swap |
/proc/<pid>/status (VmSwap) |
not exposed — both columns are - |
page_cache_start, page_cache_end |
/proc/meminfo (Cached) |
not exposed — both columns are - when resource data was collected. macOS's VM system does not track file-backed page-cache pages as a separate, directly comparable quantity. |
The macOS kernel does not compute proportional set size — there is no
equivalent of Linux's /proc/<pid>/smaps[_rollup]'s Pss: line.
We populate max_pss with the same phys_footprint value used for
max_uss. For benchmarking workloads (a single dominant compute child plus
shared system libraries) the two are typically within a few percent.
If you need real PSS numbers, run on Linux.
tricorder spawns the child in its own process group and installs handlers
for SIGINT, SIGTERM, and SIGHUP. When any of these arrive at tricorder,
they are forwarded to the child's process group. tricorder then waits for
the child to exit, writes the (partial) output, and returns:
- the child's exit code if it exited normally
128 + signumif the child was killed by a signal
Hitting Ctrl-C during a run thus tears the child down cleanly and still produces a record you can inspect for "what was happening when I killed it".
use std::time::Duration;
use tricord::{
run::{run_command, RunOptions},
format::OutputFormat,
SchemaMode,
};
let options = RunOptions {
interval: Duration::from_millis(500),
output_path: std::path::Path::new("/tmp/timing.tsv").into(),
format: OutputFormat::Tsv,
force_summary: false,
trace_path: None,
markdown_path: None,
schema_mode: SchemaMode::Full,
};
let outcome = run_command("samtools", &["sort".into(), "in.bam".into()], &options).unwrap();
println!("exit={} cpu_time={:.2}s", outcome.exit_code(), outcome.record.cpu_time);tricord started life as a Rust port of an in-house Python helper
(bench-cmd.py) that wrapped snakemake.benchmark.benchmarked() so we
could time just the expensive part of a rule — benchmark: measures the
entire shell: block, including any prewarming or staging the rule does
before the work you care about.
The default TSV output is therefore bit-format-compatible with
snakemake.benchmark.write_benchmark_records(extended_fmt=False): identical
columns, identical formatting, drop-in replacement for use inside a rule's
shell: block. But there's nothing Snakemake-specific about tricorder
itself — it's a general-purpose process-tree resource sampler that's just as
happy being invoked by make, a CI script, or by hand at a terminal.
A few values are computed slightly more accurately than Snakemake's sampler:
io_in/io_out: Snakemake reports the latest snapshot's per-process values, summed across alive processes. If a child exits between snapshots its I/O is dropped.tricordkeeps the last-observed cumulative value per PID and sums those at the end, so an exited child's I/O is still counted.cpu_time: same correction — Snakemake takes the latest snapshot's alive-process sum;tricordaccumulates per-PID maxima and takes the larger of that total and the child's exit-accounted CPU time.mean_load: derived from the correctedcpu_time, so it doesn't inherit the first-poll-zero quirk frompsutil.cpu_percent().
For long, mostly-monolithic runs the differences are well under a percent.
MIT — see LICENSE.