fix: release large muxer write buffers to the OS

[JxnLexn]

Summary

Sustained fMP4 muxing with large, variably sized writes can create severe process-heap retention under real long-lived camera workloads. This pull request moves large transient IOContext::WritePacket() callback buffers out of the regular process heap while preserving their copy and lifetime semantics.

Fixes #328

Allocation path

IOContext::WritePacket() currently copies every FFmpeg write through Napi::Buffer<uint8_t>::Copy(). The affected production workload uses four permanent HKSV prebuffers, 4-second fragments, the default 2 MiB AVIO buffer, and high-resolution H.264 remux input. The resulting large, variably sized writes repeatedly churn the Node/libc heap.

Changes

• Keep the existing regular Node-buffer path for writes below 64 KiB.
• Allocate larger transient write buffers with page-backed storage (mmap on POSIX, VirtualAlloc on Windows).
• Preserve page alignment by passing the allocation length through finalize_hint instead of storing a header in the mapping.
• Report the mapping to V8 with Napi::MemoryManagement::AdjustExternalMemory() and subtract it in the finalizer.
• Release the mapping from the external-buffer finalizer.
• Preserve a safe copy fallback for embedders that disallow external buffers and for allocation failure.
• Handle external-buffer creation failures without leaks or double frees.
• Add direct and thread-safe callback regression tests proving that retained large buffers remain valid after the native write returns.

FFmpeg data is still copied before the callback returns, so the patch does not introduce a zero-copy lifetime hazard.

The previously included FMP4Stream.emitBoxGroup() view optimization has been removed from this PR. It is independent and changes observable backing-buffer/byteOffset semantics.

Production evidence

Affected deployment:

• Ubuntu 24.04.4 x86_64, glibc 2.39
• Node.js 24.18.0
• fragDuration: 4_000_000
• default bufferSize: 2 * 1024 * 1024
• no production MALLOC_* tunables

A run started on August 19, 2026 at 18:36 CEST. Proxmox RRD data shows the container below 1 GiB around 19:30, approximately 1.6 GiB at 20:00, 5.2 GiB at 20:30, 9.4 GiB at 21:00, and at its 10 GiB cgroup limit by 21:30. At 00:16 CEST the HomeKit process alone held 9,360,352 KiB of anonymous/private-dirty RSS. The cgroup had more than 53 million memory.high events and no OOM kill. A preceding run had reached the same limit before restart.

This does not show the 300 MiB warm-up plateau seen in the single-stream Debian/glibc-2.36 reproduction. Short isolated tests against the same real camera source, without HomeKit/HAP queues, also showed lower post-GC growth with MALLOC_MMAP_THRESHOLD_=65536 or a low trim threshold.

Verification

The revised head 76016bb has been verified with:

• npm run build:tests
• npm run build:tsc
• npm run lint
• npm run lint:gyp
• git diff --check
• a complete native Linux x64 addon build and link on Ubuntu 24.04 / glibc 2.39 / Node.js 24.19.0, using the project Jellyfin FFmpeg build inputs and prebuildify --napi --strip --arch=x64 --tag-libc
• focused tests against that freshly built addon proving that retained large direct-callback and thread-safe-callback buffers remain valid after the native write returns

The GitHub workflow for this fork PR still requires maintainer approval before its jobs can start.

[seydx]

Hey, thanks for the PR

The write path really does allocate a fresh copy per call, and at realistic fragment lengths those writes really are large and variably sized, so that part is accurate. I measured it here and got p50 17.7 KiB / p90 78 KiB with 4s fragments, which lines up with what you describe.

Where I get stuck is the conclusion. I tried to reproduce the unbounded growth before merging and I can't. Setup: Debian bookworm, glibc 2.36, node-av 6.2.0-beta.17, a single long-lived FMP4Stream over an endless lavfi source, 4s fragments like HKSV, fragments dropped immediately, 5 minutes per config, RSS sampled after a forced GC.

Default glibc: 284 MiB at t=60, peak 308, and completely flat from t=160 to t=300. With MALLOC_MMAP_THRESHOLD_=65536: peak 313. That one matters because it forces every allocation of 64 KiB and up through mmap, which is exactly what this PR does by hand, and it does not lower the peak at all. With bufferSize: 64 * 1024 I get 9168 write buffers instead of 1140 for the same amount of data, so 8x the number of allocations, and RSS ends up at 302 instead of 303. And after stop() RSS drops from 303 to 181, so the high-water mark does come back, which contradicts the "stopping the stream does not return it" point in the issue.

I also ran a control with no node-av at all, just Buffer.allocUnsafe in the same size distribution. That plateaus at 98 MiB within the first 20 seconds and stays there across 6.2 TiB of allocation churn. Same step-then-plateau shape on macOS. So what I see is an allocator warm-up curve rather than a leak, and I think a 10 minute window extrapolated linearly to "200 MiB/hour" is exactly what that curve looks like before it flattens.

Two things about the patch itself, independent of the above. napi_create_external_buffer does not tell V8 how much memory is behind the buffer, so the GC loses the pressure it currently gets from the V8-owned backing store of Buffer::Copy. We already have a convention for that, see Napi::MemoryManagement::AdjustExternalMemory in frame.cc and packet.cc with the SyncExternalMemory() helper. Without it the mappings can end up living longer than the copies do today, which would work against the goal. The other one is smaller: storing the length in an 8 byte header inside the mapping and returning allocation + sizeof(size_t) means the buffer data is never page aligned. napi_create_external_buffer takes a finalize_hint, so you can pass the length through that and drop the header.

The emitBoxGroup change I'd want to look at separately. Skipping a Buffer.concat per fragment is a real saving, but it makes the emitted fragment a view with a non-zero byteOffset into a larger backing buffer, which is visible to anyone doing new Uint8Array(data.buffer). That deserves its own PR and a note in the changelog rather than riding along with the native change.

I'm not closing this. What would move it forward for me is a longer run, a few hours rather than ten minutes, showing that the curve does not flatten out, plus your exact glibc version and platform, and the fragDuration and bufferSize you're running with. That last part matters: with the default fragDuration of 1 the writes here are p50 0.3 KiB, so the new 64 KiB path would never even trigger. It would also help to know whether MALLOC_ARENA_MAX=2 changes anything in your setup, since a lot of the RSS in my runs is x264's encoder threads and their per-thread arenas, not the write buffers.

[JxnLexn]

Thanks for the detailed reproduction and review. Your three patch-level points are correct, and I updated the branch in 76016bb:

• added symmetric Napi::MemoryManagement::AdjustExternalMemory(+length/-length) accounting;
• passed length through finalize_hint, so the mapped buffer stays page-aligned and no header is stored in the mapping;
• removed the emitBoxGroup() view optimization from this PR entirely. I agree that its observable byteOffset/backing-buffer semantics make it a separate change.

I also rebased onto current main, including the merged pipeline teardown work from #331.

The affected production environment and actual FMP4 settings are:

• Ubuntu 24.04.4 x86_64, glibc 2.39
• Node.js 24.18.0
• four permanent HKSV prebuffer streams
• H.264/AAC remux for supported input codecs
• fragDuration: 4_000_000 (4 seconds)
• default bufferSize: 2 * 1024 * 1024
• no MALLOC_* environment variables

I agree that the earlier 10-minute isolated run alone was not enough to extrapolate indefinitely. The production history now gives a longer window, though: after a restart on August 19 at 18:36 CEST, Proxmox RRD shows the container below 1 GiB around 19:30, ~1.6 GiB at 20:00, ~5.2 GiB at 20:30, ~9.4 GiB at 21:00, and at the 10 GiB cgroup limit by 21:30. It then stays pinned there by reclaim. At 00:16 CEST the HomeKit process itself had 9,360,356 KiB RSS, of which 9,360,352 KiB was anonymous/private dirty, with zero swap. The cgroup had >53 million memory.high events and no OOM kill. A preceding run had independently reached the same limit before restart.

So I agree with avoiding a blanket conclusion from the short synthetic curve, but the real four-stream remux workload does not plateau near 300 MiB. The platform difference may matter as well: production is glibc 2.39, not 2.36.

For allocation isolation, the earlier test was a single FMP4Stream against the same real camera stream, outside HomeKit/HAP/application queues. In short A/B runs, MALLOC_MMAP_THRESHOLD_=65536 and a low trim threshold reduced post-GC anonymous growth materially. I have not set MALLOC_ARENA_MAX=2 in production because that requires restarting the currently memory-saturated service; I will report that separately from a controlled run rather than changing the live process just to complete the matrix.

I also revised #328 and this PR description to separate the multi-hour production non-plateau evidence from the shorter isolated allocator A/B evidence, and removed the claim that the ten-minute run by itself proved an indefinite linear rate.

[JxnLexn]

Additional verification for the revised head 76016bb:

I reconstructed the repository's Linux x64 CI build locally in an isolated worktree using the project's Jellyfin FFmpeg build inputs and the ghcr.io/seydx/jellyfin-ffmpeg/linux64-gpl:latest image.

Environment:

• Ubuntu 24.04 / glibc 2.39
• x86_64
• Node.js 24.19.0
• N-API prebuild produced by prebuildify --napi --strip --arch=x64 --tag-libc

Results:

• the complete native addon compiled and linked successfully;
• src/bindings/io_context.cc compiled as part of that full build;
• npm run build:tests passed;
• npm run build:tsc passed;
• the two focused IOContext lifetime tests passed against the newly built native addon:
  • large direct-callback write buffer remains valid after the native write returns;
  • large thread-safe-callback write buffer remains valid after the native write returns.

This closes the previous verification gap caused by the incomplete restored FFmpeg build tree. The GitHub fork workflow still requires maintainer approval before it can run.

[seydx]

@JxnLexn

Releasing Beta 19

https://github.com/seydx/node-av/actions/runs/32320409935

can you test it pls once its released

[JxnLexn]

Beta 19 is now running in the affected production environment.

Deployment details:

• camera.ui installer state: 2.1.10
• HomeKit plugin: 1.2.7
• Ubuntu 24.04.4 x86_64 / glibc 2.39
• Node.js 24.18.0
• four active HKSV FMP4 prebuffer sessions
• 4-second fragments and the same real camera workload described above

I replaced only node-av and @seydx/node-av-linux-x64, from 6.2.0-beta.18 to 6.2.0-beta.19, with a complete rollback copy retained. The process maps confirm that the beta.19 native addon is loaded.

Before the controlled restart, the beta.18 HomeKit process had been running for about 7h45m and held:

• RSS: 2,230,480 KiB
• anonymous RSS: 2,153,504 KiB
• 27 threads

The beta.19 HomeKit process started at 06:48:36 UTC. Initial observations under all four active prebuffers:

• about 1 minute: 287,600 KiB RSS / 213,852 KiB anonymous
• about 6 minutes: 296,380 KiB RSS / 220,100 KiB anonymous
• one warm-up step around 7 minutes to roughly 340 MiB
• about 15 minutes: 350,012 KiB RSS / 273,604 KiB anonymous

Since that warm-up step, RSS has fluctuated in a narrow roughly 347-350 MiB range rather than rising continuously. Threads stayed at 27 and FDs stayed essentially stable. The container is healthy and there are no node-av/HomeKit errors.

This is only an initial result, not yet evidence of a long-term plateau. I am continuing the same workload and collecting longer-run samples before drawing a conclusion about the two beta.19 packet-release fixes versus the IOContext allocation change in this PR.

[seydx]

@JxnLexn

any news?

[JxnLexn]

Long-run beta.19 result from the affected production workload:

Environment / versions

• camera.ui installer state: 2.1.10
• HomeKit plugin: 1.2.7
• Ubuntu 24.04.4 x86_64 / glibc 2.39
• Node.js 24.18.0
• four permanent HKSV FMP4 prebuffers, H.264/AAC remux
• fragDuration: 4_000_000, default bufferSize: 2 * 1024 * 1024
• both node-av and @seydx/node-av-linux-x64: 6.2.0-beta.19
• the process maps confirm that the beta.19 native addon is loaded

The current HomeKit process started at 2026-08-20 07:13:10 UTC and has now run for more than 13h34m under the same real four-camera workload. Its current and lifetime values are:

• current RSS: 406,208 KiB
• lifetime maximum RSS (VmHWM): 445,824 KiB
• current anonymous RSS: 329,096 KiB
• file RSS: 77,112 KiB
• threads: 27
• FDs: 63
• swap: 0

For comparison, beta.18 had reached 2,230,480 KiB RSS and 2,153,504 KiB anonymous RSS after only about 7h45m. Even beta.19's lifetime maximum RSS is about 80.0% below that beta.18 observation; its current anonymous RSS is about 84.7% below beta.18.

The Docker container remains running and healthy with:

• 0 container restarts
• OOMKilled: false
• cgroup memory.events: high 0, max 0, oom 0, oom_kill 0

There is one separate caveat. The first beta.19 HomeKit process, started at 06:48:36 UTC, exited unexpectedly at 07:13:05 UTC after about 24 minutes and was restarted by the camera.ui plugin supervisor five seconds later. Its last valid sample was only 351,572 KiB RSS / 275,164 KiB anonymous RSS. There was no kernel/cgroup OOM, no Docker restart, and no core dump; camera.ui logged Plugin HomeKit exited and the supervisor restart, but no normal closed line with an exit code or signal. This happened around the end of several short live-view sessions. I cannot attribute that one process exit to node-av from the available evidence, so I am treating it as a separate open observation rather than as part of the former memory-exhaustion failure.

One measurement note for transparency: the CSV sampler's PID selection became invalid after its first two samples because pgrep -f began matching the monitor command itself. I therefore did not use the later CSV rows as HomeKit measurements. The long-run conclusion above is based on the unchanged process start time/PID, the process's kernel-maintained lifetime VmHWM, its current /proc status, the loaded-addon mappings, supervisor logs, and Docker/cgroup counters.

My current conclusion is therefore deliberately scoped: beta.19 has so far eliminated the severe multi-hour RSS growth in this real production workload. The two packet-lifetime fixes included in beta.19 appear sufficient for the original memory-exhaustion symptom here, despite beta.19 not containing the IOContext allocation change from this PR. On this evidence, that additional change may no longer be necessary for this issue. The single early HomeKit-process exit remains worth tracking separately if it recurs.