Repository navigation
Conversation
Member
|
Documentation preview available at https://docs.seqan.de/preview/seqan/hibf/338 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #338 +/- ##
=======================================
Coverage 99.79% 99.79%
=======================================
Files 53 53
Lines 1934 1939 +5
Branches 7 7
=======================================
+ Hits 1930 1935 +5
Misses 4 4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
generated via Claude Code Opus 5.5
eseiler
force-pushed
the
feat/python
branch
from
September 28, 2026 12:09
9c77f22 to
23856b1
Compare
|
The input adapter only borrowed Config.input. If the input was reassigned while the library was still requesting values, e.g. by the input itself, the adapter called a freed object and the interpreter crashed. The adapter now owns a reference. All builders destroy it with the GIL held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IBF agents size their buffers by the number of bins. After increase_bin_number_to or try_increase_bin_number_to, an existing agent returned wrong counts and copied the IBF's new number of words into its old buffer, writing out of bounds. All agents are now wrapped in py_agent, which rebuilds an IBF agent when the IBF's number of bins differs from the one the agent was created for. HIBF agents use the same wrapper; an HIBF cannot change, so they are never rebuilt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HIBF(config, layout) passed the layout to the library unchecked. An empty layout crashed the construction. A layout with more user bins than the config built an index that reported user bin ids beyond number_of_user_bins, and counting agents wrote those ids out of bounds. The user bin ids of the layout must now be 0, ..., number_of_user_bins - 1, each exactly once. Otherwise, a ValueError is raised before the construction starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library only asserts that merged sketches have the same number of registers. In a Release build, merging a smaller sketch into a larger one read past the smaller buffer, and the estimate became nan. merge and merge_and_estimate now raise a ValueError instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library allocates and zero-fills 2^num_bits bytes before checking that num_bits is in [5, 32]. HyperLogLog(40) therefore tried to touch 1 TiB before raising. The binding now checks the range first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Methods that modify an IBF or a HyperLogLog sketch are not synchronised. Operations such as batch_membership_for and save release the GIL, so a concurrent increase_bin_number_to can free the memory they are reading. The README and the docstrings now state that such methods must not run while another thread uses the object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config stored the number of user bins inferred from the first sequence it was given. Assigning a longer input afterwards kept the old count, and the index silently left out the extra user bins. A number_of_user_bins of 0 now means "infer from the input" whenever the value is needed: by the property, the builders, to_string, write_layout_file, __eq__ and __repr__. Pickling keeps the 0, so a restored config keeps following its input. validate_and_set_defaults still stores the inferred value, like the other defaults it sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Values were converted with numpy.asarray(..., dtype="uint64"), which truncates floats: 1.9, 2.7 and 3.2 were stored as 1, 2 and 3, and queries were truncated the same way. Arrays, buffers and NumPy scalars must now have an integer dtype. Signed values are still reinterpreted as two's complement. Python integers and iterables of them are converted value by value via __index__: floats raise TypeError, and values outside [0, 2**64) raise OverflowError. That no longer depends on how the installed NumPy version handles out-of-range Python integers. For one million Python ints, this takes 51 ms; the previous numpy.asarray path took 55 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The HIBF agents summed the counts of a split bin's technical bins in uint16_t (membership) or value_t (counting). A query value is counted in its own technical bin and may be a false positive in the others, so the sum can exceed the number of query values. With maximum_fpr = 0.3, a query of 60 000 values summed to 81 266, wrapped, and missed its user bin. The sums now use size_t, and counting agents saturate at the maximum of value_t when storing a count. The counts per technical bin still use the counter type, so they wrap for queries with more values than it can count. The Python bindings now raise ValueError for such queries: at most 65 535 values for membership queries, and at most the maximum of the dtype for counting agents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config, Layout, InterleavedBloomFilter and HierarchicalInterleavedBloomFilter define __eq__ by value, but kept the identity-based default hash. Two equal IBFs therefore ended up as two entries in a set. Python requires equal objects to have equal hashes. A hash over the values would be expensive and most of these objects are mutable, so __hash__ is now None, as for lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
config::operator== is documented to compare all options except input_fn, but it skipped number_of_hash_functions and track_occupancy. Configs that differ only in these compared equal, in C++ and via Config.__eq__ in Python. Configs read from layout files of versions 1 and 2 derive track_occupancy from empty_bin_fraction, as validate_and_set_defaults does, so they still compare equal to the validated configs they were written from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
interleaved_bloom_filter::clear(bin_index) resets the occupancy of the bin, but the overload for a range of bins only cleared the bits. With track_occupancy, the cleared bins kept their old occupancy. In Python, ibf.clear(1) reset the occupancy of bin 1, while ibf.clear([2]) did not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config owns a Python object, but its type had no tp_traverse or tp_clear. A reference cycle through Config.input was therefore never collected. For example, a bound method of an object that stores the config leaked the object and all its data until exit, when nanobind reported the leak. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
counting_agent() returned object, so type checkers could not see bulk_count. It now returns a std::variant of the three counting agents, which the stub shows as a union. timings is typed dict[str, float], and Config.input as a sequence or callable of array-likes, or None. Layout's top_level_max_bin_id got a docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it Config.__repr__ built the complete repr of the input and then cut it to 60 characters. For a list of 20 000 arrays, this took 19 s. The input is now described by its type and its name or length, e.g. <list of length 20000>, which takes 43 us. Floats are formatted with Python's repr, so 0.1 + 0.2 shows as 0.30000000000000004 instead of 0.3, and 1.0 as 1.0 instead of 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
__getstate__ serialised into an ostringstream, which grows by doubling, and then copied the result into a bytes object. Pickling a 256 MiB IBF raised the peak memory by 1280 MiB. The object is now serialised twice: once into a counting stream buffer to determine the size, and once directly into a bytes object of that size. The counting pass does not touch the data, because cereal writes each vector with a single call. The peak memory now rises by 512 MiB: the bytes object and pickle's own copy of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
save() never checked whether closing the file succeeded. Data that was still buffered at that point, which is all of it for small objects, was lost without an error, e.g. on a full disk. Larger objects failed during serialisation, but with cereal's RuntimeError instead of an OSError. save() now writes through a file buffer that records errno of the first failed write, checks close(), and raises an OSError with that errno in both cases. errno cannot be read after cereal's exception, because unwinding may overwrite it; the test saw ENOENT instead of EFBIG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
membership_for and bulk_count held the GIL, so Python threads with one agent each could not query in parallel. They now release it while the library runs. For 400 queries of 5000 values against an HIBF with 1000 user bins, 4 threads with one agent each took 0.04 s instead of 0.18 s. Agents keep their results in buffers, so one agent used by two threads at once would race. Each agent is now leased for the duration of a query; a second thread using it gets a RuntimeError. The lease flag is only accessed while holding the GIL. The filters' own membership_for uses a new agent per call and needs no lease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The library throws std::logic_error for a bin count, bin size or number of hash functions out of range, which nanobind maps to RuntimeError. It does not check empty_bin_fraction at all, although it determines the number of technical bins: 1.0 divides by zero, and a negative fraction yields fewer technical bins than bins. The binding now checks all four and raises ValueError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README claimed that agents avoid reallocating buffers. The HIBF agent still creates a counting agent per IBF on every query, so the README now only calls agents reusable. It also describes that queries release the GIL, that an agent used by two threads at once raises, and that importing the module on a free-threaded build enables the GIL again, which the checks rely on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyproject.toml uses a PEP 639 license expression, license = "BSD-3-Clause". scikit-build-core
0.10.0 and 0.10.7 reject it ("Field project.license has an invalid type"); 0.11.0 accepts it and
builds the bindings, and all Python tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nanobind>=2.0 was never tested. With Clang 23 and libc++ on Python 3.14: * 2.0.0 and 2.9.0 do not compile: nb_types.h and nb_call.h use std::ptrdiff_t and alloca without including their headers. * 2.13.0 and 2.15.0 compile and work, but stubgen does not qualify nested classes. The stubs refer to CountingAgentUInt16 instead of HierarchicalInterleavedBloomFilter.CountingAgentUInt16, which type checkers cannot resolve. * 3.0.1, the first 3.x release, and 3.1.0 build and pass all Python tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bindings are built with -march=native by default, which stays. pip caches wheels built from an sdist or a git URL pinned to a commit (not from local directories). With a cache shared between machines of different CPU generations, e.g. a cluster home directory, a cached wheel fails with an illegal-instruction error. The README now explains this and how to avoid it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The module contains MIT-licensed code: simde, robin_hood and the HyperLogLog implementation. It also contains BSD-3-Clause code from nanobind and cereal. The license expression is now "BSD-3-Clause AND MIT". The wheel now contains their licences in .dist-info/licenses/third_party. nanobind's, cereal's and simde's licence files are copied. The notices of robin_hood and the HyperLogLog code are generated from robin_hood.hpp and LICENSES/MIT.txt. Dependencies found on the system instead of via CPM are not covered; CMake warns about missing licences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README now says that: * Apple's clang cannot build the library, because it lacks OpenMP. CMake stops with "HIBF requires OpenMP"; Homebrew's LLVM works. * Building, also from an sdist, needs network access for CPM unless the dependencies are cached or found on the system. * A wheel links the OpenMP runtime by absolute path and is not relocatable without delocate or auditwheel, and two OpenMP runtimes in one process can conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_setstate_invalid_bytes truncated the pickle itself and accepted any exception. pickle raised UnpicklingError before __setstate__ ran, so the test never checked how invalid state is handled. It now passes truncated state to __setstate__, expects cereal's RuntimeError, and checks that the object stays uninitialised. No test uses pytest.raises(Exception) any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Python CI only built Release, where the library's assertions are disabled. Misuse that the library only asserts against, e.g. merging HyperLogLog sketches of different sizes, therefore passed CI. The test job now runs for both build types with each compiler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All builds of the bindings shared build/python/{wheel_tag}, so switching between Debug and Release
reconfigured the same directory. The build type is now part of the name. CMake keeps the
compiler of an existing build directory, so the README says to delete it after changing CXX.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pip install . builds the Python bindings in build/python/ (see build-dir in pyproject.toml). The repository had no .gitignore, so these files showed up as untracked unless a global ignore covered build/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
<tuple> and <string_view> were included but not used. The include comments of <fstream> and <utility> now list what is used from them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the library's classes, the binding classes now declare all six special members, defaulted where possible. input_adapter holds a mutex and an atomic, so it can neither be copied nor moved. For array_like, dtype_like and input_like, NB_OBJECT_DEFAULT declares the default constructor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
generated via Claude Code Opus 5.5