Skip to content

fix(mem): vendor mimalloc 3.5.3 so free(NULL) before init is safe (supersedes #2562) - #2731

Open
DeusData wants to merge 2 commits into
mainfrom
fix/mimalloc-3.5.3-preinit-free
Open

fix(mem): vendor mimalloc 3.5.3 so free(NULL) before init is safe (supersedes #2562)#2731
DeusData wants to merge 2 commits into
mainfrom
fix/mimalloc-3.5.3-preinit-free

Conversation

@DeusData

@DeusData DeusData commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Supersedes #2562 by @futile, which found and fixed this; it lands as a maintainer change because we vendor third-party code ourselves. Credit and co-authorship below.

The bug

With mimalloc 3.3.2's release build, free(NULL) before the allocator's constructor dereferenced a page map that does not exist yet. glibc 2.44's newlocale() does exactly that during libstdc++ start-up, so every Linux binary built against it died with SIGSEGV before main (#2562 has the full analysis). Upstream fixed it in microsoft/mimalloc@b2226c1 (mimalloc #1341), first released in v3.4.4.

The change

  1. Vendor mimalloc v3.5.3, upstream's latest published release (LICENSE, include/, src/), byte for byte from the tag, then re-apply only the local patches main still needs:
  2. -DMI_PROFILE=0 for the production allocator (MIMALLOC_CFLAGS in Makefile.cbm). mimalloc 3.5.2+ compiles its allocation-sampling profiler in by default, which puts a sample countdown on the allocation path; the product never uses it. The test build keeps upstream's default.
  3. vendored/mimalloc/PATCHES.md: the pin, a re-verification recipe, which check pins each patch and the build flag, and the refresh procedure. THIRD_PARTY.md points at it, the SBOM says 3.5.3, and scripts/vendored-checksums.txt is regenerated (mimalloc lines only).
  4. tests/test_prod_allocator_contract.sh, wired into scripts/test.sh right after Step 5 builds the production binary. The unit suites link mimalloc's debug build without the global override, so they cannot see this class of bug. The contract links three small fixtures against prod_mimalloc.o itself:
    • free(NULL) from an ELF .preinit_array hook, before every constructor (Linux), plus a check that malloc really is mimalloc;
    • the banner's first line is exactly vX.Y.Z with no build clock, the object is the release build, and it has no profiling line (pins MI_PROFILE=0);
    • _mi_prim_decommit sets needs_recommit on macOS and keeps upstream's value elsewhere.

Provenance

  • Upstream microsoft/mimalloc, tag v3.5.3 (annotated tag object deac3dcf5f0614884013d411a5c07e16bec96c9f), commit d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45, tree fbc4e8f72287d127c079ada5ede870b7d63e55f5; published GitHub release of 2026-09-17 on main3. Verified in a fresh bare clone and against the GitHub API.
  • microsoft/mimalloc@b2226c1 is an ancestor of the tag.
  • Before the patches were re-applied, the staged LICENSE blob and the include/ and src/ trees had exactly the tag's object ids (50 files). Afterwards the only differences are the two hunks above; vendored/mimalloc/PATCHES.md has the one-loop recipe to re-check this from a fresh clone.
  • License unchanged (MIT, same LICENSE blob as before).

Verification

  • Linux arm64 (glibc 2.39), gcc 13.3 and clang 18.1, 4 runs each: main's allocator object segfaults in the preinit fixture (exit 139) 4/4 with both compilers; this branch passes all three fixtures 4/4 with both.
  • Each pin fails when its change is reverted: without the recommit patch the decommit fixture reports needs_recommit=0 on macOS; without the comment rewording scripts/security-vendored.sh blocks the upstream line; an object built with MI_PROFILE=1, with MI_DEBUG=2, or with a suffix after the version fails the banner fixture.
  • Memory-related suites under ASan + UBSan on macOS (mem, mem_events, arena, slab_alloc, str_intern, graph_mem, graph_buffer, store_bulk, sqlite_writer, parallel, worker_pool, stack_overflow_a/b/c, system_info, pipeline): 787 passed, no sanitizer or allocator reports.
  • scripts/security-audit.sh, scripts/security-vendored.sh and scripts/security-strings.sh (on the production binary) pass; every --contracts-only step passes except Step 0w (language-count), which fails identically on main since dd357a7 and is untouched here.
  • Linux arm64 production build with gcc (Linux flags, global override on): no warnings, and a full django index runs cleanly with no allocator errors.

Measurements

Apple M5 Pro, macOS 26.6, full index, 2 interleaved reps (main fceb25d with mimalloc 3.3.2 vs this branch):

corpus metric main this PR
TypeScript peak phys_footprint 1.07 / 1.01 GB 1.04 / 1.04 GB
TypeScript peak RSS 1.87 / 1.84 GB 1.85 / 1.86 GB
TypeScript wall 30.6 / 29.1 s 29.3 / 30.0 s
django peak phys_footprint 0.41 / 0.41 GB 0.38 / 0.39 GB
django peak RSS 0.59 / 0.59 GB 0.58 / 0.60 GB
django wall 6.5 / 6.2 s 6.5 / 6.5 s

Without the macOS recommit patch, the same v3.5.3 tree peaks at 2.09 / 2.04 GB phys_footprint on TypeScript, so that patch is still needed.

Graph quality: identical on both corpora (full node and edge hashes, node census per label, edge census per type, no dangling edges, no duplicate label/qualified-name groups; django 68,022 nodes / 322,464 edges, TypeScript 306,239 / 835,984).

Linux arm64 (Colima, glibc 2.39, gcc 13.3), django, 2 interleaved reps: peak RSS 0.67 / 0.68 GB on main vs 0.69 / 0.68 GB on this branch.

Notes for review

  • Besides the vendored tree this touches Makefile.cbm (the MI_PROFILE=0 flag plus two comments), scripts/test.sh (one new step), scripts/vendored-checksums.txt (mimalloc lines only) and scripts/ci/generate-sbom.py (version string).
  • On MinGW, mimalloc 3.4.5+ only takes its CRT-TLS process-attach path when MI_MINGW_UCRT64 is defined (upstream's CMake sets it; our Makefile does not), so our Windows build now also initializes through the constructor-attribute path. The Windows legs and the smoke allocator-ownership check cover this.

Credit

The diagnosis, the upstream pointers, the zone-comment wording and the preinit reproduction are @futile's work from #2562; Felix Rath is Co-authored-by on both commits. #2562 will be closed with thanks once this merges.

@DeusData DeusData mentioned this pull request Oct 20, 2026
Open
2 of 4 tasks
DeusData and others added 2 commits October 21, 2026 00:33
@DeusData @futile
fix(mem): vendor mimalloc 3.5.3 so free(NULL) before init is safe
f9f44b5 
With mimalloc 3.3.2's release build, free(NULL) before the allocator's
constructor dereferenced a page map that does not exist yet. glibc 2.44's
newlocale() does exactly that during libstdc++ start-up, so every Linux
binary died with SIGSEGV before main (reported with a reproduction in
#2562). Upstream fixed it in b2226c19 (mimalloc #1341), first released in
v3.4.4; v3.5.3, upstream's latest published release, contains it.

Vendored from microsoft/mimalloc tag v3.5.3 (tag object deac3dcf, commit
d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45): LICENSE, include/ and src/,
written blob by blob from a bare clone. Before the local patches were
reapplied, the staged LICENSE blob and include/ and src/ trees had the
tag's exact object ids (50 files; src/arena-meta.c is gone upstream).
Afterwards the only differences are two documented hunks:

- src/prim/unix/prim.c: macOS always recommits on decommit, carried from
  63fb7b3 (the phys_footprint fix).
- src/prim/osx/alloc-override-zone.c: one comment reworded, because
  scripts/security-vendored.sh rejects the upstream "fork()" spelling
  even inside a comment. No code change.

The reproducible-banner patch (no __DATE__/__TIME__) is dropped: upstream
removed the build clock in v3.4.5 (mimalloc #1355). The two Makefile
comments that pointed at that patch now say so.

The production object is built with -DMI_PROFILE=0: mimalloc 3.5.2+
compiles its allocation-sampling profiler in by default, with a sample
countdown on the allocation path, and the product never uses it.

vendored/mimalloc/PATCHES.md records the pin, a re-verification recipe,
the check that pins each patch and the refresh procedure; THIRD_PARTY.md
points at it, the SBOM version is 3.5.3 and scripts/vendored-checksums.txt
is regenerated (mimalloc lines only; security-vendored.sh passes).

Measured on Apple M5 Pro (macOS 26.6), full indexes, 2 interleaved reps,
main fceb25d (3.3.2) -> this change: TypeScript peak phys_footprint
1.07/1.01 -> 1.04/1.04 GB, peak RSS 1.87/1.84 -> 1.85/1.86 GB, wall
30.6/29.1 -> 29.3/30.0 s; django footprint 0.41/0.41 -> 0.38/0.39 GB,
RSS 0.59/0.59 -> 0.58/0.60 GB, wall 6.5/6.2 -> 6.5/6.5 s. The graphs are
identical: full node and edge hashes, node and edge census, no dangling
edges (django 68022 nodes / 322464 edges, TypeScript 306239 / 835984).
The same tree without the recommit patch peaks at 2.09/2.04 GB footprint
on TypeScript, so 63fb7b3 still does its job. Linux arm64 (glibc 2.39,
gcc 13.3): both trees build without warnings; django peak RSS 0.67/0.68
GB on main, 0.69/0.68 GB with this change.

Co-authored-by: Felix Rath <felixm.rath@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData @futile
test(mem): pin the production allocator's start-up and local patches
ddc986e 
The unit suites link mimalloc's debug build without the global override,
so they could not see the #2562 start-up crash, and nothing guarded the
vendored allocator's local behaviour: prod_mimalloc.o is compiled with -w
and without CFLAGS_COMMON, so -Wdate-time never reaches the vendored
sources, and the macOS recommit patch had no test at all.

tests/test_prod_allocator_contract.sh links three small fixtures against
$BUILD_DIR/prod_mimalloc.o itself:

1. free(NULL) from an ELF .preinit_array hook, before every constructor
   (Linux). Also asserts malloc returns a mimalloc-owned block, so a
   fixture that did not link the override fails instead of passing.
2. mi_options_print_out's first line is exactly "vX.Y.Z" with no build
   clock, and the object is the release build (debug level 0). Upstream
   dropped the clock in v3.4.5; this keeps a refresh from bringing it back.
   The banner also has no "profiling" line, which pins -DMI_PROFILE=0.
3. _mi_prim_decommit sets needs_recommit on macOS and leaves upstream's
   release value elsewhere (local patch 1 in vendored/mimalloc/PATCHES.md).

scripts/test.sh runs it right after Step 5 builds the production binary,
on every Linux and macOS leg; other platforms skip with the reason.

Linux arm64 (glibc 2.39), gcc 13.3 and clang 18.1, 4 runs each: main's
3.3.2 object segfaults in fixture 1 (exit 139) 4/4 with both compilers;
this branch passes all three fixtures 4/4 with both. macOS arm64:
reverting the recommit patch turns fixture 3 red (needs_recommit=0);
an object with a git-describe suffix in the banner, built with
MI_DEBUG=2, or built with MI_PROFILE=1 turns fixture 2 red.

The preinit reproduction follows the test Felix Rath wrote for #2562.

Co-authored-by: Felix Rath <felixm.rath@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData force-pushed the fix/mimalloc-3.5.3-preinit-free branch from bc51fd1 to ddc986e Compare October 20, 2026 22:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@DeusData