Skip to content

Add C extension build, test and CI infrastructure - #533

Merged
jeromekelleher merged 5 commits into
tskit-dev:mainfrom
nspope:numba-to-c-ext
Nov 20, 2026
Merged

jeromekelleher merged 5 commits into
tskit-dev:mainfrom
nspope:numba-to-c-ext

Conversation

@nspope

@nspope nspope commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First step in replacing numba with a C extension. Infrastructure is copied from tsinfer.

  • lib/ is a C library skeleton with tskit-style error handling, meson build with strict warnings and CUnit smoke tests
  • _tsdatemodule.c is a minimal _tsdate module with LibraryError
  • setup.py, Makefile, MANIFEST.in, cibuildwheel config, uv cache-keys
  • CI includes C tests (gcc/clang/valgrind) and Python-C coverage jobs; wheels built with the shared tskit-dev build-wheels workflow
  • clang-format hook and codecov components

I'll build this in pieces, and only wire into the tsdate Python package at the end.

@nspope
nspope marked this pull request as ready for review November 18, 2026 23:14
@nspope

nspope commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@jeromekelleher, if you've got a minute could you glance over the infrastructure I've added here?

@jeromekelleher-bot jeromekelleher-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. It follows the tsinfer setup closely, which is what we want. Two things:

1. The release wheel build will fail. The shared build-wheels.yml@v22 workflow installs cibuildwheel with uv sync --locked --only-group wheels. This PR adds cibuildwheel to the packaging group, and there's no wheels group. I checked locally on this branch:

$ uv sync --locked --only-group wheels
error: Group `wheels` is not defined in the project's `dependency-groups` table

wheels.yml only runs on a release or a push to test-publish, so the PR checks don't catch this. We'd only find out when we try to release. The fix is to add a wheels = ["cibuildwheel"] group like tsinfer has, and re-lock. After that it's worth pushing to test-publish once to check the full wheel and publish pipeline end to end. The build-* artifact merge in the publish job is also new.

2. Fewer platforms get a wheel. Until now we've shipped a pure-Python wheel that installs everywhere. From the next release, wheels only cover cp311–cp313, plus x86_64 only on Linux. Anyone on Python 3.14, Linux aarch64 or musl will have to build from the sdist, which needs a compiler. That happens even though the extension isn't used yet. I think we should at least add cp314-*, since tskit already builds cp312–cp315. We should also keep this in mind if we cut a release before the C port is wired in.

@jeromekelleher

Copy link
Copy Markdown
Member

@nspope - in case you're wondering, I'm using claude to do some initial reviews. I'm "reviewing the reviews" so please take this as my input (if in LLMese).

@nspope

nspope commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @jeromekelleher.

  • added a wheels group (as in tsinfer) and re-locked.
  • added cp314 and cp315 (matching tskit main).
  • checking dependency wheels turned up that cibuildwheel's install test would try to build numba (and scipy on win32) from source on platforms where they publish no wheels. So win32 is skipped; macOS x86_64 wheels are built but their install test is skipped, since numba has no Intel macOS wheels after 0.62.1; musl stays skipped (no numba wheels).
  • the macOS and musl exclusions will go away when numba is removed at the end of the port.

I pushed to test-publish to check the full pipeline, the jobs show up in the CI overview in this PR. The macos-14 runner has been retired, so switched to macos-15. Presumably this'll hit tsinfer/tskit too.

@jeromekelleher-bot

Copy link
Copy Markdown

Thanks @nspope, this all looks great, and good to see the full pipeline go through on test-publish. Two small follow-ups:

  • The shared workflow fix is now in v23 (Replace retired macos-14 runner with macos-15 in build-wheels .github#57, which changes the build-wheels default to macos-15). Could you bump all the @v22 references to @v23 (in docs.yml, lint.yml, tests.yml and wheels.yml) and drop the os-list override in wheels.yml?
  • Now that we build wheels for 3.14 and 3.15, could you add the Programming Language :: Python :: 3.14 and 3.15 classifiers to pyproject.toml?

Happy to merge after that.

@nspope

nspope commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Done

@jeromekelleher
jeromekelleher added this pull request to the merge queue Nov 20, 2026
@jeromekelleher

Copy link
Copy Markdown
Member

Great, thanks @nspope!

I think it's OK to do the next step in one go, and just switch from numba to C in one go. I'd use a prompt something like this:

Port the current numba accelerated functions to C, following tskit-dev best practices. Ensure numerical equivalence by first creating oracle versions of each accelerated function in the test suite in plain python, and testing these thoroughly for equivalence against the numba accelerated versions. Think carefully about edge cases and establishing the behaviour of the current implementation around regions of difficulty, and encode that in the test suite. Then create the equivalent C version. Perform some lightweight micobenchmarking of the accelerated functions to compare the C and numba versions, to ensure that there is no significant loss of performance. Once correctness and performance equivalence has been established, drop numba entirely. Do this in a series of commits.

@jeromekelleher
jeromekelleher removed this pull request from the merge queue due to a manual request Nov 20, 2026
@jeromekelleher
jeromekelleher merged commit fbf8d0d into tskit-dev:main Nov 20, 2026
17 checks passed

This branch was successfully deployed

1 active deployment
release — 196fa28f Deployed Nov 19, 2026 by nspope via publish #249
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants