tabl

A fast TSV data wrangler in pure C — dplyr/tidyr for pipes.

tabl brings the tidyverse verbs (filter, mutate, select, summarize, arrange, pivot_longer/pivot_wider, joins) to the shell as a single small binary with no R, no Python, and no dependencies beyond zlib. It is built to sit in the middle of a pipeline:

sesame region -r chr1:1-1000000 *.idat \
  | tabl filter 'beta > 0.3' \
  | tabl summarize -g sample_name 'm=mean(beta)' 'n=n()' \
  | cinderplot -x sample_name -y m -o fig.pdf

• Input: TSV, plain or gzipped, from a file or stdin (-). Row 1 is the header unless -H (columns become V1, V2, ...). Lines starting with # are comments — skipped by every verb, and counted on stderr so nothing vanishes quietly — except that a #-prefixed header (#chrom start end, VCF's #CHROM) is understood as the header. --keep-comments reads them as data instead.
• Output: TSV on stdout, ready for the next verb, cinderplot, R, or anything else. Missing values: input accepts "", NA, na, NaN; values tabl computes are written as NA (input cells pass through as-is).
• Fast: single-pass streaming wherever the semantics allow; group-by hashes its keys, so input never needs pre-sorting. On a 5M-row/86MB table a three-aggregate group-by runs ~6x faster than the equivalent awk.

Install

conda install -c zhou-lab tabl

Packages are built for linux-64 and osx-arm64. Intel macOS (osx-64) is not packaged — build from source there, which needs nothing but a compiler and zlib.

Or from source — requires only a C11 compiler and zlib:

make && make install          # PREFIX=/usr/local by default
make test                     # golden tests + awk oracle
make asan                     # address/undefined sanitizers

The verbs

verb	dplyr/tidyr equivalent	notes
filter EXPR	filter()	NA rows drop, like dplyr
select SPEC	select()	names, 1-based indices, ranges a:b, -e drop, -r regex
mutate N=EXPR ...	mutate()	sequential; may replace columns
summarize [-g SPEC] N=AGG ...	group_by() + summarise()	hashed groups, first-seen order; --grouped: O(1) memory
count [SPEC]	count()	-s sorts by count desc
distinct [SPEC]	distinct()	-a keeps all columns
arrange SPEC	arrange()	-col = descending, NAs last, stable
longer -c SPEC	pivot_longer()	-n/-v name the key/value cols; -r regex, --drop-na
wider [-n C -v C]	pivot_wider()	last dup wins, with a warning
join [LEFT] RIGHT	left_join() etc.	--inner, --full; -k a,b or -k left=right
head / tail	slice_head/tail()	-n N
rename NEW=OLD	rename()	header-only rewrite
cols	names()	indices + names, one per line
view	print(tbl)	aligned, right-justifies numeric columns

Aliases: summarise, sort, pivot_longer/gather, pivot_wider/spread work too.

Expressions

filter and mutate take expressions over column names:

tabl filter 'sales > 10 && region == "east"' d.tsv
tabl mutate 'margin = (sales-cost)/sales' 'flag = if_else(margin > 0.5, "hi", "lo")' d.tsv

• Operators: + - * / % ^, comparisons == != < <= > >=, logic && || ! (single & | = work as the doubled forms). Backticks quote odd names: `my col`.
• A cell compares numerically when both sides parse as numbers, as text otherwise — so age > 30 and city == "NY" both do what they look like.
• NA propagates through everything (&&/|| use R's three-valued logic); test it with is_na(x), fill it with coalesce(x, 0).
• Functions: abs sqrt log log2 log10 exp floor ceil round pow min max (row-wise), len substr upper lower cat/paste num str if_else/ifelse is_na coalesce.

summarize adds aggregates, freely composable inside expressions:

tabl summarize -g region,year \
    'n=n()' 'total=sum(sales)' 'avg=mean(sales)' \
    'p90=quantile(sales, 0.9)' 'span=max(sales)-min(sales)' d.tsv

n sum mean median min max sd var first last n_distinct quantile(x,p) — with R semantics: sum mean median min max sd var quantile skip NAs (sd/var are n−1; median/quantile are R's type 7), while n() counts every row, first/last take the row's value NA and all, and n_distinct counts NA as its own level. Groups come out in first-seen order; pipe through tabl arrange to sort them.

Reshaping

# wide -> long: stack the twelve month columns
tabl longer -c jan:dec -n month -v value wide.tsv

# long -> wide: one column per assay metric
tabl wider -n metric -v value long.tsv

longer and wider are exact inverses on tidy data; the test suite round-trips them.

Documentation and examples

• zhou-lab.github.io/tabl — a one-page cheatsheet (also docs/index.html locally).
• examples/ — sample data plus run_all.sh, a runnable walkthrough of every verb (sh examples/run_all.sh).
• tabl --help — colorized cheat sheet.

Prior art: csvtk, tsv-utils, Miller, qsv, dply, and GNU datamash.

License

MIT — see LICENSE.