Repository navigation
Conversation
A store hit now bails to the slow path instead of calling the GC barrier, mirroring into a metatable's MtCache, or growing properties. With no calls left the handler needs no stack frame: setf 6.8 -> 3.9 cyc, 58 -> 32 insts.
A small-integer key inside the array part is now handled without calls, so the handlers need no stack frame; any other key tail-calls the previous body. geti 4.3 -> 3.3 cyc (47 -> 26 insts), seti 8.0 -> 2.9 cyc (80 -> 29 insts); a string key pays the extra hop: hget 6.8 -> 8.0 cyc.
Decimal integers in string.format, tostring, concat and %q go through
format_into (int_format_into) into a stack buffer instead of a format!
String each; %o/%x use a local digit loop. format("%d,%d") 322 -> 166 cyc.
%a/%e/%f/%g now come from a port of lj_strfmt_num.c: an exact decimal bignum written straight into the output, with a rescale fast path for %e/%g below 14 digits. Ties are decided from the double itself, since the limbs past the rounding digit can be truncated (upstream rounds %.0e of 2.5e10 up), and %f of numbers below 2^-500 is zero outright (upstream wraps its limb ring there). %F is rejected, as in Lua, and an unknown conversion's error quotes the whole spec. tostring and concat now use %.14g like LuaJIT instead of 5.5's %.15g/%.17g round-trip. Float tostring 1500 -> 289 cyc.
A bare conversion skips parse_spec, literal runs are copied in one go,
precision zeros are a count instead of a staging buffer, the unpadded
case appends directly, and the output starts at 64 bytes.
format("%d,%d") goes from 166 to 142 cycles/op (LuaJIT -joff: 83).
The buffer lives in a userdata upvalue of format, as io keeps its state,
rather than in the VM. It is taken for the call, so a format re-entered
from __tostring gets a fresh one, and buffers past 64 KiB are freed.
format("%d,%d") goes from 142 to 119 cycles/op, %g and %.3f lose ~16.
A spec is spanned and checked as Lua's getformat/checkformat do, but
packed straight into an SFormat: the conversion's type, subtype and
accepted flags come from one match, and the modifiers are validated only
when there are any. Integers, strings, chars and pointers are written by
ports of lj_strfmt_putfxint/strfmt_putfstrlen/lj_strfmt_putfchar, which
replace FmtSpec and the old padding helpers, and plain %d stays a direct
NumBuffer write.
Matching Lua more closely along the way: a missing argument is reported
before the spec, spec errors come at the conversion's own point (after
the argument for integers and %e/%f/%g, after __tostring for %s), the
span runs to the conversion so '%5-d' and '%5%' are quoted whole, a
spec over 21 characters is 'invalid format (too long)', and %#o
prints one leading zero ('%#.3o' of 8 is 010, '%#.0o' of 0 is 0).
format("%d,%d") goes from 119 to 104 cycles/op (LuaJIT -joff: 83);
particles_bench 2400 frames is 2.14 Gcyc against 2.39 before this
branch and LuaJIT's 1.99.
Gc::new_with_trailing hands the trailing bytes to a callback instead of copying a slice, so a value can be followed by something other than bytes, and trailing_ptr / from_trailing_ptr convert between a Gc and the start of its trailing bytes. new_with_bytes is now a wrapper.
The array part moves from a Vec to a slots cell: a TrailingBytes cell of values that is never traced on its own. The table's trace marks it and traces its values, so the table's barrier covers stores into it, as with JSC's butterflies, and a replaced cell is left for the sweep instead of freed by the table's drop. First step towards tables without drop glue. Constructor arrays are a second GC object until they move into the table's own cell: ctor_arr +15%, ctor_arr10 +20%; geti/ipairs/len/particles within noise.
properties moves from a Vec to a slots cell holding the shape's slots, grown by doubling (from 4) on a property add that finds it full; the IC transition hit stores through push_property. Like the array part, the table traces it and its barrier covers stores. Creating objects costs a second GC object until the slots move into the table's own cell: ctor +18%, vnew +14%, setfield_new +14%; field loads and stores within noise.
GcAlloc hands out GC cells nothing traces and makes deallocation a no-op: the holder marks the cell (RawTable::allocation) when it is traced, and the sweep frees it after. The vendored RawTable loses its Drop impl, so with the array and named parts already in slots cells, TableState has no drop glue left, which is now asserted. dead_keys_are_not_retained counted only GC bytes, which now include the cleared table's own buckets; its baseline is a table as big whose keys own nothing. Benchmarks within noise (pack -3%).
A table cell now carries trailing values: its shape's inline slots, then
the array part a constructor sized, so a constructor's table is one
allocation. Slots past the inline capacity go to a spill cell. The
capacity is part of the shape: one root per size in INLINE_CAPS (0, 2,
4, 8, 16, 32), a template rooted at the size its fields need, and {} at
0 for now.
IC entries hold a SlotLoc resolved at fill time, a byte offset from the
table's lock for an inline slot or into the spill cell, so a hit reads
neither the shape's capacity nor the spill pointer for inline slots.
TableState reaches its own trailing values through
RefLock::trailing_ptr_of, which uses the provenance new_with_trailing
exposes.
Against the branch base: ctor -20%, vnew -17%, ctor_mt -17%, ctor_arr
-26%, ctor_arr10 -15%, particles -8%; field loads and stores within
noise (+1 to +3 instructions per hit). setfield_new (fields added to
{}) is +14% until {} gets a size hint.
A local initialized from a constructor, or from setmetatable(constructor,
mt), gets inline room for the distinct constant names that the following
statements of its block store into it (name.k = v, function name.k,
function name:k), up to a redeclaration of the name, as JSC's
StaticPropertyAnalyzer counts put_by_id into a new object. A rebound
setmetatable only costs the hint.
setfield_new -41%, class_new (setmetatable({}, C) then self.k = v) -35%,
module_tbl -19%; vnew moved by code placement only (equal with
-align-all-functions=6).
The integer, misc and dict-string parts move into one Owned cell, made by the first key that needs a hash part and traced through the table like a slots cell; dict mode is now the shape's is_dict. A table without hash keys pays one word for them instead of three RawTables, and TableState is 64 bytes (from 184). ctor -14%, vnew -17%, pack -10%, ctor_arr -9%, class_new -9%, particles -8%; hash lookups within noise.
rehash_ints counted each array key into its bin through memory, a store-to-load chain per key that took ~60% of a 1M-element append loop. Count non-nil values per power-of-two range instead, as LuaJIT's countarray does, and copy the old part with memcpy past 32 values. Appending 10M values: -31% cycles; 100-value tables: -19%; table.insert: -8%.
…48 bytes A table's metatable is now its shape's MtCache's back-pointer, which the cache leaves untraced: each table marks its metatable in its own trace, so a shape an inline cache holds still doesn't keep a metatable alive. A metatable's own MtCache moves into the aux cell (the hash parts' cell, made on adoption). Store ICs no longer check for a metatable on a hit; instead a site never caches a key MtCache::mirror watches. Header 64 -> 48 bytes, one malloc size class less for every table. ctor -5%, ctor_mt -3%, upd -5%; setmeta (a fresh metatable per object) +13%, as adopting a metatable now makes its aux cell.
nseq becomes an array part in the table's own cell, as a constructor's items do, capped at the largest array part a rehash makes (MAX_ASIZE): our nil is not zero bytes, so every slot is written. nrec picks the root shape whose inline capacity fits it. table.create(100) then 100 stores: -51% cycles; table.create(10): -70%; table.create(0, 4) then 4 fields: -41%.
A generic for gets a fourth hidden slot after the closing value, where TFORCALL keeps its position in a table it walks itself, as LuaJIT keeps its index in the control slot; the loop variables move up by one. The slot isn't counted toward the 200-local limit, so the limit matches lua. TFORCALL recognises the next that pairs returns and ipairs's iterator (kept in State) and steps without a call, taking TFORLOOP's jump too. Steps through the array part and string keys stay in the handler without a stack frame; tfor_next takes integer and other hash keys, tfor_ipairs the integer hash and __index. TableState::next_at walks in next's order by position: the part in bits 29-30, the index below, so positions stay small integers. Per element vs luajit -joff: pairs 73 -> 10.6 cycles (LJ 2.9), ipairs 36 -> 4.6 (LJ 9.9). particles 2400: 1.786 -> 1.647 Gcyc (LJ 2.01).
close_upvalues ran Vec::retain over every open upvalue of the thread whenever any was open, so each CLOSE, including the one at the end of every generic for, paid for all captured locals of every frame. Keep the list sorted by stack slot as Lua keeps openupval: a capture inserts in place, searching from the top, and a CLOSE checks the last entry and pops while it is at or above the level. Empty generic-for loop (next, e): 46.8 -> 36.6 cycles; pairs(e) 69.4 -> 55.9; ipairs(e) 49.0 -> 40.7.
TFORPREP checks the iterator and state, as LuaJIT's ISNEXT does, and marks the position slot: 0 to walk with next, -1 for ipairs, nil to call the iterator. TFORCALL dispatches on the mark alone instead of comparing the iterator each step, and next's first step no longer detours through tfor_next. The state is still checked to be a table, since debug.setlocal can change it; a changed iterator is not followed, as in LuaJIT. Testing the ipairs mark first keeps ipairs from paying a taken branch. vs the previous commit: empty next loop -11%, pairs(dict) -6%, pairs over a 10-field record -8%, ipairs per element -2%, an ipairs loop's setup +5% (the checks moved into TFORPREP).
pairs on a table without __pairs, and ipairs, now write their results straight into the call's slots from their own CALL entries, as math's builtins do, instead of going through the native-call machinery. Other call shapes (a TAILCALL, all results kept, another argument count) and a __pairs still go to the builtins. __pairs joins the metamethod list, so its presence is a bit in the metatable's MtCache, kept current like __tostring's; Symbols::pairs is now mm_pairs. Empty loop with pairs(e): 53.1 -> 36.4 cycles; ipairs(e): 45.7 -> 28.0; pairs over a 10-field record -5% per element.
next_at's positions now have five parts: array, integer keys, named slots, dict-mode string keys, other keys. Only one of the two string parts is ever non-empty (a shape-mode table's string hash part is empty, a dict-mode table's shape has no slots), so a step no longer loads the shape to ask which it is. Five parts take three bits: positions keep 2^28 per part. pairs over a 10-field record: -8% cycles; dict and ipairs unchanged; array part +2.6% (the part match orders its compares differently; testing the array part before the match instead moved the loss to records).
Each frame carries ret, the continuation that takes its results: RETURN pops the frame and jumps to it with the values, their count and the function slot, and the continuation decodes what it needs from the instruction that made the call (LuaJIT Remake style). CALL's lands the results; metamethods and iterators get their own (store to R[A], discard, conditional skip, loop variables), staged above the frame and entered through meta_call; frames the executor makes leave the results for it. The executor hands a call site's results to its continuation through PendingRet. This replaces the Continuation payload, HAS_CONT/PARENT_NON_LUA, num_results, schedule_meta_call and frame_return. A native metamethod that forwards to Lua without a follow-up sequence now just works.
MtCache keeps each metamethod's value next to its presence bit, updated by the same mirror; __close and __name join the cached set. Metamethod lookups (Context::mm_of, binop_metamethod, the __index/__newindex walks) read the cache instead of looking names up; debug builds check it against the metatable. EQ, LT, LE, UNM, BNOT and LEN handle the common operand types inline and leave the rest, metamethods included, to _slow siblings, so their fast paths need no stack frame. A constant-key miss whose cache entry already records the key as absent goes straight to an __index function or __newindex, without refilling the entry (and its barrier) every time.
CLOSE and RETURN take the innermost to-be-closed variable off the list and call its __close like a metamethod; the continuation runs the instruction again for the next one. A RETURN's results stay in place, with their end recorded below the call for a MULTRET RETURN to find again. CloseSequence is gone; meta_call now takes its staging slot in the ip register.
A native can return CallThen: call stack[at] and run a continuation on the results. It then gets a frame of its own (NATIVE, holding the native and its continuation, the call's slot in num_extras), the call returns to it through ret_native, and run_natives drives natives calling natives without leaving the interpreter. Protected calls mark the frame PROTECTED (and HANDLER for xpcall's message handler in the window's first slot); the unwinder hands such a frame the error through native_error, and the next run_thread runs its continuation with it. pcall and xpcall are continuations now instead of ProtectedCall sequences, without an executor round trip or allocation per call.
CALL of a native whose CallThen calls a Lua function that fits pushes both frames and dispatches, and ret_native runs a continuation that returns and jumps to the frame's continuation itself; everything else still goes through run_natives. Stack::insert moves a few values without memmove. pcall(f, i): 87 -> 39 cycles.
The arithmetic and bitwise opcodes (register and immediate forms) share binop_slow, the table reads get_slow, the table writes set_slow, and LT, LE and their immediate forms cmp_slow; each decodes its operands from the opcode, as LuaJIT Remake's generic slow paths do. Each of the ~30 per-opcode slow handlers carried its own copy of the numeric mix, the metamethod lookup and the call staging: the interpreter's code goes from 88 KB to 58 KB (75 KB before the continuation work), handlers with a stack frame from 82 to 42.
coroutine.resume, wrap and yield return unboxed Resume and Yield actions. A resume parks the resumer in a protected native frame and switches the handlers' thread argument to the coroutine: a first resume calls its body with ret_coroutine_end as the continuation, a later one hands the values to the continuation its last yield left. A yield, and the body's return, move the values into the waiting native frame and switch back. Dispatch reaches thread states without borrow guards (Thread::state_mut), reports the thread it ended on, and the executor rebuilds its thread stack from the new resumer links. Contexts that can't switch, and resumers waiting in the executor, still go through it. Also fixes a stale register read in TAILCALL of a native that grew the stack. ResumeSequence and UnwrapResumeSequence are gone.
The driver looped over its NativeState enum, which LLVM kept in memory: each step stored it a byte or word at a time and the next read it back wider, stalling on store forwarding, at entry too. It now loops over a phase tag and scalars, and run_natives unpacks the state at the call site for drive_natives. Small value moves between coroutines skip memcpy.
A local that nothing assigns after its declaration (parameters, local functions, most locals; tracked per variable across the chunk's functions, as LuaJIT's VSTACK_VAR_RW) is captured by value: the closure's slot holds the value itself, so CLOSURE allocates no cell, the frame needs no close on return, and GETUPVAL/GETTABUP read the slot directly. Children are assembled once the chunk is compiled, when every assignment is known; the assembler rewrites the reads of the remaining upvalues to GETUPVAL_REF/GETTABUP_REF/SETTABUP_REF, which go through the cell. A local function's capture of itself takes the new closure. Differences from PUC (intended, as in LuaJIT Remake): debug.setupvalue on a by-value upvalue would change only that closure's copy, upvaluejoin could not share it, upvalueid would differ between closures, and debug.setlocal would not reach closures that captured the local by value. None of these debug functions exist yet. Cycles vs the previous commit: closure capturing a fresh local -42%, fixpoint-fact -35%, richard/queen/ray/deltablue -3%.
The assembler turns RETURN0 and RETURN1 into RETURN in a function that may have something to close at return (a TBC, a generic for, or a local captured into a cell) or has varargs, as luaK_finish does with needclose. Their handlers then skip the frame-flags test and find the function slot right below the registers instead of reading num_extras: 4 instructions fewer per return. Native tail calls, which return from whatever frame made them, go through RETURN's handler. Cycles vs the previous commit: call -3%, 3-argument call -14%, recursive-fib-uv -3%.
ThreadState::call_limit is the value stack's length capped by the frames' capacity: each frame's base is above its caller's, so a call whose window ends below it has room for its frame too, and CALL tests that one bound instead of the stack length and whether the frames are full. Frames only grow where the frame pointer is rebound (op_call_grow, push_lua), never when the value stack does. FrameStack keeps a pointer to the innermost frame instead of a length, so a call or a return stores it without loading and adding first. CALL plus RETURN1 is 9 instructions shorter. Cycles vs the previous commit: recursive-fib-uv -6%, two-result return -3.4%.
The compiler emits a CALL that wants no result or one as CALL_R0 or CALL_R1, whose handlers give the callee ret_call0 or ret_call1 directly instead of loading it from the table by C. C stays for the generic paths. Cycles vs the previous commit: call -4%, queen -4%.
ret_call0/1/2, and ret_call when its count is fixed, no longer store thread.top: nothing reads it before a MULTRET producer sets it. What #43 needed the store for, a top left high by a callee keeping dead slots traced, moves to the GC checks that have nothing pending above the frame (NEWTABLE, CONCAT, CLOSURE): they bound top by the frame's window before leaving for the collector, as LuaJIT's fixtop does. One store fewer per call; call -1%, recursive-fib-uv +3% with the same instructions minus the store (ret_call1 grew past a 32-byte boundary).
Compares are conditional branches that the compiler emits in place of the compare + JMP pair: each jumps when its test holds (J.., JT, JTSET) or when it doesn't (JN.., JF, JFSET), so a taken branch is one dispatch and no k bit is decoded at run time. JNLT a b is not JLE b a: they differ on NaN and in the metamethod they call. The set, each with its negation: JEQ, JLT and JLE on two registers; JEQI, JLTI, JLEI, JGTI and JGEI against a literal (CmpImm, a 15-bit integer and a was-a-float flag, as Lua's isSCnumber, in a new 16-bit field h); JEQS against a string constant, by identity since strings are interned; JT and JF; and JTSET and JFSET, which copy the value they test for `and`/`or`. They replace EQ, LT, LE, TEST, TESTSET and the five immediate compares: 22 opcodes for 10, and no JMP after them. Every branch carries a 32-bit offset in the extension word. Jump lists hold the branches themselves, `not` swaps a branch for its negation, and the slow paths and the comparison metamethods' continuation (one ret_cond) jump by the caller's offset. branch_if! keeps an asm barrier on both outcomes so each ends in its own dispatch: with one shared, collatz and primes2 ran 5-6% slower. EQ's fast path also compares two inline integers, which went through op_eq_slow when unequal. Cycles vs the previous commit: primes2 -24%, queen -25%, recursive-fib-uv -8%, collatz -4%, deltablue flat.
Context::with_buf lends a byte buffer kept in State across uses (taken and put back, so a nested use gets a fresh one; one grown past 64 KiB is dropped), and build_string interns what it holds. CONCAT reserves the operands' length in it once instead of growing a fresh Vec per operand, and tostring of non-strings, table.concat, gsub with a string or table replacement, upper, lower, reverse, char and rep build there too (rep also borrows its separator instead of copying it). format and gsub with a function already had buffers of their own. Cycles vs the previous commit: a .. b .. c -51% (181 -> 87 per expression, LuaJIT 61), a .. i -34%, tostring(i) -19%, table.concat -37%, upper -30%, rep -29%, gsub with a string -17%.
ret_call_n!'s `for i in 0..$n` is `0..0` in ret_call0, which clippy's reversed_empty_ranges rejects as an error.
Objects lived in their own malloc allocations on a linked list, and the sweep walked the list, reading every object and calling free on each dead one. Following LuaJIT 3.0's GC design, they now live in 256 KiB chunks aligned to their size (its arenas; `Arena` is the collector here), whose metadata at the start holds a block and a mark bitmap with a bit per 16-byte cell. Allocation bumps a pointer through a free run and sets the object's block bit; sweeping a chunk is `block & mark`, `block ^ mark` per 64 cells, so dead objects are never read unless they still need dropping. Objects over 32 KiB get a huge chunk of their own. The header loses its `next` pointer and is the tagged vtable alone (8 bytes). Marks move to the bitmap; the header keeps a gray bit (queued for tracing; marked and not gray is black) and a weak bit. Since the sweep no longer visits objects, the ones only a GcWeak reached are collected in a list and settled when marking ends: dropped if unmarked, but their box is kept through the sweep so weak pointers still see them dead, as before. A run is a free block up to the next block start, found by scanning the bitmaps from where the last search stopped. When a sweep begins the current run is retired (its tail becomes a free block) and allocation only uses chunks already swept, sweeping the next one itself if none has room, so nothing allocated mid-sweep is freed by it. Pacing counts whole cells: a chunk's header tracks the bytes allocated in it (added when a run retires) and marked this cycle, which gives the sweep its freed and kept bytes; the object count is gone. Every dead object is still read for its drop function; drop-free types get chunks of their own in a later commit. Cycles vs the previous commit (median of 5 interleaved runs): particles -17%, storage -35%, fixpoint-fact -51%, binary-trees -10%, havlak -11%, json/qt -2%, ray/richard flat; deltablue +45%, all of it in get_slow's table -> shape -> keys loads (same call counts, same number of GC cycles): the long-lived shapes and keys now share cache lines with short-lived objects instead of each other. RSS: havlak 198 -> 281 MB, json 20 -> 25 MB, qt 197 -> 185 MB, others flat.
The sweep read every dead object to call its drop function, a no-op for most types. Objects whose type needs dropping now come from a separate space of chunks, with its own run and search position, and only those chunks drop their white blocks when swept; the rest free them with the bitmap operations alone. Huge objects still check their one object. Drop types are mostly long-lived metadata (shapes and their keys, prototypes, threads), so this also takes them out from between the short-lived objects, which is most of the deltablue loss in the previous commit: get_slow's shape -> keys loads now hit lines full of other shapes. Cycles vs the previous commit: deltablue -20%, havlak -5% (RSS 280 -> 180 MB), particles +2%. Against the commit before the chunk heap (median of 3 interleaved runs): fixpoint-fact -57%, storage -34%, binary-trees -32%, mandel-metatable -26%, cd -21%, havlak -17%, particles -17%, table-sort -7%, json -5%, chameneos -5%, ray/queen/fib/richard flat; deltablue +25%, coroutine-ring +5.5%, table-sort-cmp +3%. RSS: cd 44 -> 33 MB, json 20 -> 16 MB, havlak 209 -> 180 MB, table-sort 231 -> 252 MB, deltablue 215 -> 237 MB.
Marking an object read its header for its size and whether it needs tracing. Types that hold no pointers and have no drop glue (strings, boxed integers, the cells tables keep their slots and hash parts in) now come from a third space of chunks, so a mark finds out it has a leaf from the chunk header and takes the size from the bitmaps: the cells up to the next block or free start, or up to the leaf run's top when the object is the last one in it, whose tail has no bits set. The object is never read. It also moves table aux cells away from their tables, which costs particles a little but leaves denser table chunks; keeping aux cells with the tables and only strings and boxed ints apart measured deltablue +8%, havlak +3%, particles -2.6% against this. Cycles vs the previous commit (median of 3 interleaved runs): deltablue -8%, havlak -5%, mandel-metatable +3.6% and coroutine-ring +2.6% (neither collects; code placement), the rest within +-1.5%.
The backward barrier checked the phase, then whether the parent was black, which with marks in the bitmaps means a header test and a bitmap load, and that grew Gc::write_if_clean enough that ff_resume stopped inlining it. Following LuaJIT 3.0's quad-color scheme, the header's gray bit now means "writes need no barrier": objects are born light gray (gray bit set, unmarked), marking queues them dark gray (marked too), tracing clears it (black), and the first write after that sets it again. The fast path is the gray bit alone. When it is clear, the out-of-line slow path either lightens the object (white, or any object outside marking) or, for a black one while marking, queues it to be traced again as before. Like LuaJIT 2's table barrier it ignores the child. Cycles vs the previous commit (median of 3 interleaved runs): coroutine-ring -4.9%, mandel-metatable -3.9%, havlak -2.2%, storage -2.1%, cd -1.8%, json -1.7%, deltablue +3.2% (every survivor's first write after a sweep takes the slow path), the rest within +-1%.
LuaJIT 3.0's GC design has marking work only on metadata, but marking here still read each object's header when it was found (its size, the weak and needs-trace bits) and wrote its gray bit, so every traced object was touched once when found and again when traversed. Marking now sets the mark bit and takes the size from the bitmaps, as leaves already did: the cells up to the next block or free start, or up to the run top of the object's space when it is the last object of the active run; a huge object's size is in its chunk header. Every object outside a leaf chunk is queued, and traversal skips the few in drop and huge chunks that have nothing to trace. The gray bit is no longer set when an object is queued, so a write to a queued object takes the barrier and queues it again; traversing it twice is the price LuaJIT 3.0 accepts too. Each traversal now counts as tracing work (the old undo of a re-traced object's credit would underflow when the first traversal hasn't happened yet), and an object only a GcWeak reached counts as marked when it is settled rather than when it is reached. Cycles vs the previous commit (median of 3 interleaved runs): fixpoint-fact -5%, deltablue -2% (RSS 220 -> 193 MB), qt -1.9%, particles -1.4%, table-sort +3.7% (RSS 251 -> 276 MB), the rest within +-1.5%.
Gray objects went on one global stack. Following LuaJIT 3.0, each chunk now has its own stack of the cells of its objects waiting to be traced, kept in the metadata in place of the mark bits of the metadata's own cells, with the stack itself allocated on demand. Tracing drains one chunk's stack before taking another from the gray queue, a binary heap of chunks by stack length, largest first and the latest queued among equals. Lengths change, so a waiting chunk (whose stack only grows) is queued again each time its length reaches a power of two, and entries for chunks already drained are skipped when they surface. The chunk being drained is never queued again: its length goes up and down by one, and re-queuing it at each power of two made 1.7M heap entries for 25K chunk switches on storage (binary-trees +33%). Objects queued by a write barrier still go on the separate gray-again queue, traced last. Cycles vs the previous commit (median of 3 interleaved runs): havlak -4.5%, mandel-metatable +3.9% (it barely collects), fixpoint-fact +3.3%, deltablue +1.9%, particles +1.5%, the rest within +-1%. A plain LIFO stack of chunks instead of the heap measured the same within noise.
A huge object got a chunk of its own, with the whole chunk metadata (4 KiB at 256 KiB chunks) for one object. Following LuaJIT 3.0, it is now allocated on its own, aligned to the chunk size, with no metadata in front of it; its size, mark bit and whether it needs tracing or dropping are in a hash table keyed by its address. Offset 0 of a chunk is metadata, never an object, so an aligned object pointer is a huge one: marking tests that bit and takes a cold path for huge objects. Where this differs from the design: - Not rounded up to whole chunks. LuaJIT 3.0's memory manager only hands out chunk-sized memory; the system allocator can use what follows. - No gray bit in the table: every huge object is a GcBox with the gray bit in its own header. - The design doesn't say how gray huge blocks are traced. They wait in a list of their own in the gray queue, traced before the next chunk. - Huge objects are swept all at once when the sweep starts (so those allocated during the sweep are left alone), where a huge chunk was swept as one incremental step. - Huge leaf objects are no longer queued for tracing. Cycles vs the previous commit (median of 3 interleaved runs): table-sort -4.5% (RSS 276 -> 249 MB), coroutine-ring +8.3% with the same instruction count (its hot handlers are byte-identical; op_eq_slow moved to a different alignment), the rest within +-1%.
The collector slept until half as much as survived the last cycle had been allocated (dmm's default sleep factor of 0.5). It now waits for as much again, like LuaJIT's default pause of 200 (lj_gc.c: threshold = estimate * pause / 100). Cycles (median of 3 interleaved runs): havlak -8.1%, storage -7.8%, binary-trees -7.8%, cd -7.6%, particles -7.1%, json -2.9%, table-sort +3.9% (not yet understood), the rest within noise. Peak RSS rises with the extra headroom: cd 36 -> 68 MB, havlak 180 -> 263 MB, table-sort 249 -> 293 MB, deltablue 193 -> 225 MB, json 16 -> 26 MB, binary-trees 36 -> 45 MB.
Marking set the mark bit and also found the object's size, scanning the bitmaps for the next block start (clamped to the space's run top), only so pacing could count marked bytes and each chunk its marked total. LuaJIT 3.0's marking sets the mark bit and nothing else; now so does ours. The sweep counts each chunk's marked cells from the bitmaps instead: an add carries from each black start through the extent bits after it until the next white or free start, across words, so the black blocks' cells are a popcount per word. Debug builds check the count against the objects' sizes. Pacing credited mark work when an object was marked; it now credits it when the object is traced, where its size is read anyway, so objects that are never traced (leaf objects) earn no mark credit, as in LuaJIT 2. That can only make the collector do more work per allocated byte, not less. Cycles vs the previous commit (median of 3 interleaved runs): havlak -3.8%, storage -3.1%, binary-trees -3.0%, particles -2.5%, cd -1.8%, json +1.3%, deltablue +1.1%, the rest within noise. Cycles end sooner, so peak RSS falls: binary-trees 45 -> 40 MB, deltablue 225 -> 207 MB, table-sort 293 -> 271 MB, havlak 248 -> 241 MB.
Free blocks next to each other stayed separate after a sweep; the allocator's run search merged them when it found them. LuaJIT 3.0 merges them in the sweep. The sweep now keeps only the free starts that follow a live block, using the same add-carry as the marked-cell count: a carry from each live start runs through the extent bits and stops at, and so flips, the first free start after it; the carry out of a word carries the state into the next. Where this differs from the design: it describes "coarse" merging, from each word's last block's state (comparing the block and mark words); this merges every pair of neighbouring free blocks, for about the same work. Debug builds check that no two free blocks are neighbours after a sweep. Cycles vs the previous commit (median of 3 interleaved runs): havlak +2.6%, mandel-metatable +5.0% (it barely collects), the rest within +-2%; RSS unchanged (havlak's peak varies 233-271 MB between runs of one build).
Chunks and huge objects came from the system allocator with chunk alignment, which for a huge object left it to the allocator whether the alignment cost memory. Both are now mapped with mmap, as LuaJIT 3.0 requests them from the OS: map the size plus the alignment slack, then unmap the slack before and after, so only the pages of the size stay mapped. Fresh mappings are zeroed, so chunk metadata needs no clearing. Where this differs from the design: - No memory manager in between: every huge object is its own mmap and munmap, and freed mappings are not kept for reuse. - A huge object takes whole pages (16 KiB on Apple silicon), not whole chunks. - Empty chunks are still never unmapped. Cycles vs the previous commit (median of 3 interleaved runs): within -2.8% to +2.2%, no pattern. Peak RSS: table-sort 271 -> 243 MB, table-sort-cmp 102 -> 99 MB, deltablue 207 -> 204 MB, qt 185 -> 182 MB; a script churning 20-140 KiB strings 432 -> 370 MB.
A single-threaded, non-moving Immix after mmtk-core's ImmixSpace and ImmixAllocator: 4 MiB chunks of 32 KiB blocks of 256-byte lines, each chunk's metadata in its first three blocks. A byte per line holds the epoch of the marking that last found it live (0: free), so lines are never cleared before marking; two bits per 16-byte cell hold the mark bit and an AUX bit, which starts an object in drop blocks and flags one over a line in leaf blocks. Allocation bumps through holes, runs of free lines: the next hole in the current block, then a swept block of the same kind with free lines, then a free block, sweeping further before mapping a chunk. Objects over a line that miss the current hole go to a second bump region in free blocks (mmtk's overflow allocation). Plain and leaf allocation writes no metadata, and is charged to the metrics a hole or block at a time, so live bytes count lines held. Traced objects mark their lines exactly when traced, from the size mark_one already reads (mmtk marks at scan time). Leaves are still never read: they mark their first line, and the allocator skips the first free line after a live one in leaf blocks (the Immix paper's conservative marking); a leaf over a line has its AUX bit set and is marked exactly. The sweep drops dead objects in drop blocks, resets unmarked lines, and frees empty blocks to a pool any kind can take, or lists blocks with free lines for their kind; wholly free chunks are unmapped but one. trace's huge and grow paths are cold tail calls, so its common case needs no stack frame. Deviations: conservative marking for leaves, where mmtk is exact everywhere; one LIFO gray stack instead of the per-chunk gray stacks and priority queue; no defragmentation, which needs moving; huge objects are those over 16 KiB, aligned to 4 MiB; mark bits are cleared as each marking starts, as in mmtk's prepare. Every cycle is a full one for now.
Marks now stay set from one collection to the next, so whatever survived a collection is old: a minor collection keeps every mark and the line epoch, traces from the root and the objects written since the last collection, skips everything marked, and promotes what it marks in place. A full one clears the marks and bumps the epoch first, as before. Both are stop-the-world: once started, a collection runs to its end whatever the debt, sweep included. The quad-color barrier's gray bit is mmtk's unlog bit: a black object written outside marking is now turned gray and remembered (mmtk's modbuf) rather than only turned gray, and the remembered objects are the gray stack a minor collection starts with. write_if_clean takes its slow path for them. Huge objects allocated since the last collection are listed, the only ones a minor collection sweeps (mmtk's large object nursery), and the interner lists the strings it made since the last collection, the only ones a minor collection can kill. When to collect follows mmtk's StickyImmix trigger: once the memory in use (blocks taken, huge objects' pages, external allocations; mmtk's reserved pages) passes the heap limit. The collection is full if less than the minimum nursery (2 MiB) was left under the limit when the last one ended, judged against the limit it ran under as StickyImmix::on_pause_end does, and minor otherwise. Each full collection sets the limit by MemBalancer's fallback rule: live + sqrt(live * 4096) pages + the minimum nursery, in OS pages. Lua::collect_all forces a full collection; collect_young forces a minor one for tests. The incremental pacing factors are gone. Deviations: no defragmentation headroom in the limit (mmtk adds 5% for Immix, room to copy into, which we never do); a minimum heap of 4 MiB, my choice, with no maximum; collections start once the debt passes the 64 KiB GC granularity rather than at the first block past the limit. weakly_held_objects_are_freed now compares against a strong table emptied the same way: it measured the weak table's own emptied storage, which only stayed small while collections ran every 64 KiB. Official suite: files.lua:257 fails, as a file only collectgarbage() would close (a no-op here) is no longer collected that early.
Memory in use, for the trigger and for the limit a full collection sets, is now the lines and huge objects allocated and not freed plus external allocations, not mmtk's reserved pages. Those count a block whole while any line of it lives, and survivors spread thin: on particles_bench a full collection left 0.19 MB of objects in 0.31 MB of lines over 208 blocks (6.8 MB), the limit grew from that, and with more room survivors spread over more blocks still. Each full collection now sets the limit to twice what survived it, LuaJIT's pause of 200 and the old sleep factor of 1.0, in place of MemBalancer's fallback rule, whose square-root headroom is several times the live size on small heaps. The minimum heap drops from 4 MiB to 1 MiB. Below about 2 MiB live, less than the minimum nursery is left under the limit, so collections there are full ones, as StickyImmix decides. particles_bench peak RSS, single runs: 35.6 MB before; 9.1 MB with lines and twice what survived at a 4 MiB minimum heap; 5.65 MB at 1 MiB, where it runs 562 full collections of about 24 us each instead of 123 minor and 9 full, for about 2.6% more cycles than at 4 MiB (an instrumented build). 4ed6e80 measured 5.4 MB.
Since e229b54 every collection runs to its end once started, sweep included, as mmtk's Immix sweeps inside the pause, but the machinery for a mutator running between collection steps stayed. It goes now. The sweep is one call at the end of marking: huge objects, then every block in use, found from each chunk's free mask, then wholly free chunks returned to the OS. Gone are the per-block swept flag, the sweep cursor, the allocator sweeping on demand, Phase::Sweep, Stop::AtSweep and CollectionPhase::Sweeping. The collector leaves Mark before settling weak objects and sweeping, so a drop that panics can't leave the mutator running with barriers feeding a marking that won't trace the root again. Marking ran to its end already; the debt-driven API around it goes too. RunUntil, Stop::Full, Arena::mark_debt and Arena::cycle_debt are removed. Arena::collect_debt collects if there is any debt, finish_marking returns the marked arena directly, and MarkedArena::start_sweeping becomes sweep, which ends the collection. do_collection marks in a plain loop rather than dispatching on the phase for every object. Lua::collect_debt goes straight to finalize_and_sweep, which absorbs Lua's private finish_cycle. Docs that described incremental collection are updated: the Arena docs, GcWeak::upgrade and is_dropped, Lua::live_bytes, and GC_GRANULARITY, whose 64 KiB was tuned for incremental slices and hasn't been measured since. Two weak table tests are renamed for what they now exercise: incremental_collection is repeated_collections and mode_change_mid_cycle is mode_change_between_collections.
The GC granularity set how much debt built up between incremental steps. With stop-the-world collections it only let memory in use run that far past the heap limit before a collection started, the same as a bigger limit. Lua's 64 KiB was tuned for incremental slices. The interpreter's check now fires once the room left under the limit is used up, and Lua::collect_debt collects at any positive debt, so the limit is the only trigger, as in mmtk. gc_granularity, set_gc_granularity and GC_GRANULARITY are gone. defer_gc_check, for a host that keeps stepping without collecting, uses a fixed 64 KiB. particles_bench at 2400, 10 interleaved runs: 562 full collections become 614; median cycles 1081M before and 1071M after, within build-to-build noise; peak RSS 5.57 MB before and 5.41 MB after.
A block that took no allocation since the last collection can't lose anything to a minor one: the marks and line epoch stay, so its lines and objects come out of the sweep as they went in. A minor collection now sweeps only the blocks allocation took since the last collection, which take_free_block and pop_recyclable link through BlockInfo::next, idle while a block is allocated in. The recyclable lists now persist across minor collections and get swept young blocks back; a full collection still clears them and sweeps every block. mmtk sweeps every block after a nursery collection. deltablue, two interleaved runs each, against 31ffae4: the sweep of a minor collection drops from about 0.14 ms to 0.08 ms, out of about 3.7 ms per minor; cycles 4413M/4145M before and 4249M/4301M after, within noise; peak RSS 219 MB before and 214-215 MB after, with 9 minor collections instead of 10, likely from the changed reuse order.
meta_call, which every metamethod call goes through, carried its native and __call arms inline, so it set up a 0xb0 stack frame, spilled the handler table and reloaded it before the final dispatch, re-read the function from the slot its stager had just written, and checked the stack twice. It now calls only a Lua function whose window fits call_limit, pushing the frame with push_lua_above as CALL does; the function comes in the instruction register as Value::to_raw. Natives, callable values and calls that need the stack grown go to meta_call_slow, the old body, which now also grows the frames to the stack as op_call_grow does, so the next call fits call_limit and stays in meta_call. An lldb trace of one `v + v` with an __add returning its argument: meta_call goes from 61 to 37 instructions, the whole metamethod path from 205 to 181.
binop_slow now rewrites the site that reached it, once, by what it saw there; the compiler never emits these forms: - a float and an integer: the register form's _NUM opcode (ADD_NUM .. IDIV_NUM), which handles every inline-int/float pair, so `x / n` with a float x and an integer n no longer goes through binop_slow each time; - a metamethod from a table: ARITH_MM (from the left operand), ARITH_MM_R (from the right, the left a number while numbers have no metatable) or ARITH_MMI (immediate forms, constant on either side). These read the metamethod from the metatable's MtCache through a per-opcode index table, stage the call above the frame and jump to meta_call, all without a stack frame. A metamethod form that then sees numbers or another operand pattern goes back to the generic opcode for good (Instruction::with_no_quicken), so a polymorphic site can't flip between forms; a miss only because the stack needs growing keeps it. A _NUM form stays. The original opcode lives in bits the form leaves unused (d for the register forms, c above `flipped` for ARITH_MMI), and Instruction::unquickened gives back the emitted word, which the listing prints. Userdata metamethods and integer-only slow paths (boxed ints, overflow) are not quickened. An lldb trace of one `v + v` with an __add returning its argument: 181 instructions after the previous commit, 104 now (op_arith_mm 41, meta_call 37, RETURN1 8, ret_store_a 18), the same as `v:add(v)` through a quickened SELF. Cycles not measured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some things still need work on async natives:
Still slower than LuaJIT on the LJR suite:
areas that need improvement vs luajit on particles_bench (12000 frames):
bad table marking perf, large heaps that aren't garbage might be slow
NAN normalization when floats enter the VM
https://claude.ai/artifact/CkV3HRXxZjAD7RQNjvcEag
improved opcode quickening
GC TODO: