Skip to content

v0.8.1 streaming API: verified loose ends before release #386

Description

@mario4tier

Review of the v0.8.1 streaming API for consistency, usefulness and test coverage, across C/Rust/Java/C#. Baseline 2ef8a8cfd (post-#384).

80 candidate findings were raised; the 16 rated blocker were then adversarially verified against the shipped libta-lib.so.0.8.1. None survived as a blocker — 1 refuted, 11 should-fix, 4 nice-to-have. The 64 remaining are unverified and listed last; given that verification downgraded 16 of 16, treat them as leads.

Framing: only C has ever shipped, and the C streaming tier is new here — so its shape is still free, and this is the last moment it is.

A. API shape — settle before the streaming tier freezes

  • Composed Update is not atomic. Ruled: documented, not fixed — PR docs(errors): intermediate overflow is undefined, said once (#386) #403. The mechanism is real and all five sites still tear at 7518b3343, measured with a control-paired probe (two handles primed identically, A fed one extra bar that Update rejects, then the same tail to both): KC returns TA_SUCCESS with wrong values for 7,019 bars before re-converging; MACDEXT and STOCHRSI reject 20,000 of 20,000 later bars and never recover; STOCH and STOCHF commit their own ring/extrema/today before the first sub and tear the window. PVO/PPO/APO/BBANDS/STDDEV/KDJ/ADXR are clean — their subs get the raw bar, already vetted by the parent's U3 (KDJ inherits STOCH's tear transitively).
    Two claims above did not survive the re-measurement. (1) "Fix STOCH/STOCHF and KAMA divide by a value their guard never tested: TA_SUCCESS with inf/nan output for well-formed OHLC #390 and 4 of the 6 sites become unreachable" — no: fix(stoch,kama): divide by the value the guard tests, and bound the efficiency ratio (#390) #394 guards the divisor, not the quotient, so %K = (close−lowest)/(highest−lowest)·100 still overflows when the close sits outside the window, and TA-Lib never checks low ≤ close ≤ high; the repro uses a 1e-303-wide window and a close of 1e6, with no input near 1e306. (2) TA_REAL_MAX (3e37) does not bound this — it constrains optIn real parameters only, and the published domain for a bar is U3's "any finite double", so U3 and "a rejection changes nothing at all" are jointly false as published rather than merely out of domain.
    Why documented rather than fixed. Peek-then-commit roughly doubles composed Update cost on five functions for a corner no feed reaches, with nothing to gate it against. Investigate: batch and streaming tiers return different values (not just a different zero) when the input contains NaN #191 already settled the sibling case — overflow of the library's own arithmetic is a property of double, not of an indicator: not flagged, not gated, not specified. TA_ALLOC_ERR settled the form: name the boundary, then stop. So this became one Part 3 entry, Intermediate overflow, beside Allocation failure; the §2.4 paragraph that enumerated the effects is deleted; the U-tier promise is qualified where it is made; and the public pages carry the fact once (api/README.md §4.2 and the streaming error table).
  • TA_StreamAdvance(void *stream) writes through an unchecked void *. It and TA_StreamOutRange are the only two void * parameters in ta_func.h; every other stream call is typed and does diagnose the slip. Passing &s, a double[], or an unrelated struct compiles clean under -Wall -Wextra -Wpedantic -std=c99, returns TA_SUCCESS, and Advance writes 8 bytes into it (measured: caller's handle pointer mutated; an unrelated struct's second field incremented). The write is gated on outRangeCount < TA_MAX_INDEX, so corruption depends on what the second word decodes to. TA_StreamAdvance is brand new in this release — typing it costs nothing now. Tracked in Streaming: type TA_StreamOutRange and TA_StreamAdvance per function (TA_<NAME>_OutRange / TA_<NAME>_Advance) #387 (per-function TA_<NAME>_OutRange / TA_<NAME>_Advance).
  • C's OpenAndFill aliasing guard is pointer-equality only. A partially overlapping output passes it, returns TA_SUCCESS, corrupts the fill, and then poisons the ring that memcpys from the already-clobbered input tail — so every later Update is wrong, with no error anywhere. 201 entries, 36 ring-seed sites, one emission site (c_stream.rs:449). C is the only backend with the hole: C# uses Span<T>.Overlaps, and Java/Rust cannot express the call. Both extents are computable at the guard from historyLen and Lookback — the formula C# already uses. Note the guard is currently undocumented, so nothing published is being broken by tightening it. Documented OK (docs/error-handling-spec.md rule N8, Appendix E — decided C: partial buffer overlap is accepted and silently corrupts output #225)
  • RSI/CMO under Metastock need Lookback + 2. TA_RSI_Open / TA_CMO_Open reject at exactly Lookback + 1, so the documented pre-flight rule is false for them. Worse, TA_RSI_OpenAndFill on that path returns TA_INSUFFICIENT_HISTORY with outReal[0] already written and outBegIdx/outNBElement set to 13/1 — against the contract that on non-success the buffers are untouched and the indices undefined. Batch tier unaffected. Recorded only in docs/streaming-api-design.md:435-440, which is repo-internal. Removing Metastock (Remove Metastock compatibility (TA_Compatibility) from the TA functions #388) retires the +2 entirely; until then this needs the public exception documented, and the OpenAndFill contract violation fixed regardless — that half is independent of compatibility. No fix needed
  • C# SUPERTREND accepts cross-typed aliased outputs (Span<double>/Span<int>, the corpus's only mixed pair) in both tiers and returns silent garbage where C returns TA_BAD_PARAM. C# is not in the v0.8.1 release (IsPackable=false, no NuGet step), so this gates nothing — but it is a real bug and the generator's recorded premise for skipping the check is false. Fix 097b74b
  • C# multi-output value structs document the inverse of their real equality. Core_MACD.cs:641-643 says NaN does not equal NaN; measured True — record-struct equality routes through EqualityComparer<double>.Default, so different NaN payloads also compare equal and collapse in a Dictionary. The same docs cite a "Java Value contract" that does not exist. Fix e638e9e

B. Test coverage

  • No PR gate runs a single streaming value comparison. pr-codegen-gate.yml has five jobs, all compile-only; its own comment at :226-227 says running ta_regtest "stays in the nightly". The whole stream_verify pass is dev-nightly-only. PR Gate focus on code validity. Numerical validity is heavier and are instead done periodically (nightly CI).
  • The Value() vacuity floor is dead in Java and C#. Those servers emit no value_checked/value_legs/value_ok keys, so the floor evaluates 0 > 0 and is skipped. Deleting the Java/C# Value leg tomorrow would be invisible. (C emits 201 of each, Rust 1005/1005/1272.) No fix needed
  • The C server has no short-history reject leg — shortHistoryAccepted is 201 in Rust/Java/C#, 0 in ta_codegen_serve.c. server_gen.rs emits it at three sites, none of them C. Done (PR test(serve): C gains the short-history reject leg the other three have (#386) #391, kevinlincg)
  • Clone has no generator-side gate in any backend, and only a per-function runtime ratchet in C; in Rust/Java/C# its existence rests on one corpus-wide OR bit. The derivation it needs already exists in peek_suite. Fix 2c1394d (C); No fix needed (Rust/Java/C#)
    Two follow-ups raised against these commits were checked by sabotage and do not hold: 102598205's note that nothing verifies TA_MA_Clone's dispatch switch is true of peek_suite only — stream_verify sends one vector per non-default MAType, so re-routing the EMA arm to TA_SMA_Clone fails the run (exit 79, naming optInMAType:1). And the fork/clone leg is not blind to a shallow-shared buffer despite feeding both handles the same bars: a rolling accumulator reads the trailing slot before overwriting it, so sharing the ring in TA_SMA_Clone fails at the first shared bar (exit 87, clone_bad:"the fork left batch"). The one real gap is that this leg exists in the C server only — 0 clone probes in Rust, Java and C#. No live defect there: every stream handle's copy constructor deep-copies every array field (Java 201 classes / 310 fields / 0 shallow; C# the same; Rust derives Clone over Vec<f64>).
  • OpenAndFill's aliasing rejection has no gate in C#, and the four backends' guards have three different strengths. No fix needed
  • Nothing verifies the 1,809 new streaming symbols are exported from the Windows DLL — the exact bug class of CHANGELOG Missing TA_GetVersionString function in Windows DLL #57. Fix df228ca

C. Docs

  • ta-lib.org/api/stream/ still says "Not yet released — planned for v0.8.x" Fix e63a721. The banner is correct today — CHANGELOG.md still reads ## [0.8.1] Not Released Yet — so the item was never a stale-text bug; it is a release-process gap, and only the C page's banner should go at 0.8.1 (the six Rust/Java/C# ones say "Estimated release: Q1 2027" and are true). Nothing in scripts/ or .github/ removes it, the page is hand-written so regen-check cannot see it, and README-DEVS.md's "After a release" has no step for it — so on release day ta-lib.org would tell every visitor the flagship feature of the release is unreleased. pre-release-checks.py now refuses to cut the release while that one page still carries the banner, beside the "200+ indicators" and soname-reconcile assertions that already have this shape.

D. Release process

  • CMake and autotools ship different SONAMEs for the same library, and CMake's changes on every patch release. Fix 7518922 + ABI gate 0e55bb3
  • CPack packaging does not follow the SONAME rule. Fix 69191d1 — the .deb installs into /usr but shipped a ta-lib.pc naming /usr/local, so pkg-config pointed every build at a prefix the package never populates (and at a stale /usr/local/lib/libta-lib.so.0 where one exists). package_deb's dist_test_pass was a literal # TODO ... just pretend — the committed digests record "dist_test_pass": "True" for a .deb nothing ever opened; it now requires the payload to carry the soname ABI.manifest declares. And CPACK_PACKAGE_VERSION_PATCH read TA_LIB_VERSION_BUILD, defined nowhere — set(X ${undefined}) unsets X and CPack's own default refilled it, so it was the right value by accident.
    The rest is Post-release: split the .deb into libta-lib1 + ta-lib-dev so the SONAME actually buys co-installability #401 (post-release: the libta-lib1 + ta-lib-dev split renames released artifacts, so it cannot ship in the release it would rename). Multiarch /usr/lib/<triplet> is not being done — ld.so searches the triplet dirs first, so ours is the shadowed copy rather than the shadower, and differing sonames mean nothing collides either way.
  • The shipped .so exports every non-static symbol. Spun off as The shipped .so exports 1,519 internal symbols: TA_LIB_API is decorative on ELF #400 — 3,968 exported vs 2,449 header-declared, so 1,519 internals are in the ABI surface, the 401 _Internal seams included. No -fvisibility=hidden anywhere, so TA_LIB_API is decorative on ELF. Hiding them is itself an ABI change, hence its own issue rather than a fix here.
  • No abstract/generic streaming tier. TA_FUNC_FLG_STREAM is set on 201 functions and readable via TA_GetFuncInfo, but nothing in ta_abstract.h consumes it — the shipped .so exports 201 each of the nine per-function calls and nothing generic. A wrapper author must hand-write 201 x 9 bindings. Purely additive, closes no door, so it can follow the release. Not planned for v0.8.1
  • docs/v0.8.1-checklist branch — 176 of 201 functions, zero boxes ticked in two weeks. Merge, convert, or drop. this is long term work, not blocking on v0.8.1, renamed to not make it release specific.

Refuted — do not re-raise

  • "feat(streaming): a rejected Update leaves the handle alone; skipping a bar becomes an explicit call #384 must land before v0.8.1" — done, it is the baseline commit 2ef8a8cfd.
  • "Checkpoint-by-re-opening is false for path-dependent functions" — measured false. OBV and EMA-under-Metastock re-open bit-identically; the reproduction retained Lookback+1 bars, which is not what the sentence says to retain.
  • "Composed tear is permanent / silently corrupted forever" — it decays to bit-identical, and needs inputs three orders of magnitude past TA_REAL_MAX. Half-refuted, corrected 2026-09-08. "Silently" is right for KC (7,019 bars of wrong values under TA_SUCCESS) and wrong for MACDEXT and STOCHRSI, which reject every later bar forever. "Permanent" is right for those two. And TA_REAL_MAX is not the bound — see the item in section A.

Unverified backlog

64 further candidates were raised and not adversarially checked. Highest-value, roughly ranked: Rust's two naming conventions on one type (Core::SMA beside Core::sma_open) and its opener tuple both recoverable from the handle; arithmetic_overflow blanket-allowed crate-wide in Rust; C reading process-global candle settings live on every Update while the other three snapshot at Open; Java's Core.class at 2.48 MB / 2230 methods; no shared trait/interface over stream handles in any backend; SupertrendOut's fields named real/integer; markdown links rendering literally in 90 javadoc sites; java_stream_suite's emit ratchet stale at 168 against 201; suites building with an empty HelperRegistry for 70 of 201 functions; STREAM FILL VACUOUS's floor being a corpus-wide sum.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions