[vitormattos]

Context

LibreSign has an intermittent Behat infrastructure failure tracked in #8422.

The visible symptom is usually:

cURL error 52: Empty reply from server

The diagnostic work in this PR proved that the HTTP error is a consequence of a native PHP crash, not the root cause.

What is already proven

We have direct reproductions with PHP 8.3.35 showing:

• exit status 139;
• SIGSEGV;
• a direct core dump;
• the PHP built-in server master process crashing;
• the crash also reproduces with PHP_CLI_SERVER_WORKERS=0;
• Imagick is not required;
• OPcache/JIT are not required by the already reproduced crash;
• the failure has appeared with different database backends;
• the failure survives USE_ZEND_ALLOC=0, so the Zend allocator is not required.

Two independent cores stopped at the same instruction:

zend_objects_store_del+141

The object pointer inspected by GDB does not look like a valid live zend_object. The current evidence is consistent with an object that was freed, overwritten, or otherwise corrupted before zend_objects_store_del() became the visible crash point.

That still does not prove that the original bug is in Zend itself. The corruption may happen earlier in PHP core, a shared extension, or code exercised by the application.

Strongest reproducer

The strongest application-level signal is still the request-signature path.

Failures have repeatedly appeared around:

POST /ocs/v2.php/apps/libresign/api/v1/request-signature

and the original Behat scenario:

tests/integration/features/file/list.feature
Scenario: Return a list with 3 pages

The diagnostic workflow therefore focuses on this path instead of repeatedly running the whole suite without isolation.

Controlled application-path experiments

The focused workload separates:

1. email signer with normal notification;
2. email signer with notify=0;
3. account signer with notify=0;
4. draft request with no signer;
5. repeated validate-uuid calls;
6. the exact Return a list with 3 pages scenario repeated in one long-lived PHP process.

The exact scenario is extracted from the repository feature file instead of being reimplemented manually.

Test-history / contamination experiments

The crash has happened after a long-lived PHP process had already served many scenarios, so previous tests may influence heap layout or object lifetime.

To test this without running the whole suite, the same PHP process now runs these progressively larger sequences:

• file/list.feature only;
• file/envelope.feature + file/list.feature;
• build/sign.feature + file/envelope.feature + file/list.feature;
• account + admin + build/sign + file/envelope + file/list.

This lets us identify whether a specific prelude is needed to reproduce the corruption.

Process-lifetime isolation

The same request-signature workload is tested with:

• many requests through one long-lived php -S process;
• one request per fresh Behat/PHP server process.

If only the long-lived case fails, delayed destruction, accumulated process state, or reuse of previously corrupted memory becomes much more likely.

PHP runtime isolation

The workflow compares:

• PHP 8.3, 8.4 and 8.5;
• OPcache unloaded;
• CLI OPcache enabled with JIT disabled;
• tracing JIT in a separate case;
• default runner extension layout;
• controlled minimal extension layout;
• Imagick added back;
• two groups of additional native extensions;
• Zend allocator;
• glibc allocator with MALLOC_CHECK_ / MALLOC_PERTURB_;
• glibc allocator with tcache disabled.

The effective ini files and loaded module list are recorded for every case. Results must be interpreted from the effective runtime artifact, not from the requested setup alone.

PHP debug + sanitizers

A dedicated job now builds PHP 8.3.35 from php-src with:

• --enable-debug;
• -g3;
• frame pointers preserved;
• AddressSanitizer;
• UndefinedBehaviorSanitizer;
• OPcache disabled;
• only the extensions needed by the focused Nextcloud + MariaDB environment.

The focused request-signature workload then runs with:

USE_ZEND_ALLOC=0

This is the most important new diagnostic path. If the visible SIGSEGV is caused by an earlier invalid read, invalid write, use-after-free, double-free, or undefined behavior, ASan/UBSan should report the first invalid operation closer to its origin instead of only showing the later crash in zend_objects_store_del().

Sanitizer logs are uploaded with the normal core/GDB artifacts.

Valgrind

A smaller focused case runs under Valgrind Memcheck with:

• USE_ZEND_ALLOC=0;
• child tracing;
• origin tracking;
• a non-zero exit code on invalid memory access.

Valgrind is slower than ASan and changes timing significantly, so it is supporting evidence rather than the primary reproducer.

Xdebug trace

Xdebug is intentionally isolated in its own case.

It is not used as proof of the native root cause because Xdebug changes the executor, installs observer hooks, and changes memory layout.

If the crash still reproduces with Xdebug loaded, function tracing gives us a PHP-level timeline immediately before the native failure and can identify the last controller/service path executed.

Crash diagnostics

For a direct core, the artifact records:

• full GDB backtrace;
• registers;
• loaded shared libraries;
• process mappings;
• symbol at the program counter;
• instructions around the fault;
• the value and nearby memory for the object pointer;
• likely zend_object fields;
• eu-stack output when available;
• PHP binary metadata and disassembly of zend_objects_store_del().

For a live hang, the watchdog records:

• process tree and process state;
• /proc/<pid>/status;
• wchan;
• current syscall;
• memory maps;
• file descriptors;
• TCP sockets;
• a short live GDB backtrace;
• a short strace.

Every artifact also records the diagnostic case id, effective PHP runtime and failure classification so a timeout cannot be confused with a SIGSEGV.

Relevant upstream PHP reports

None of the reports found so far exactly matches the LibreSign reproducer.

php-src #10593

zend_objects_store_del sees invalid object-store state. A PHP maintainer points to an object that may already have been freed / freed twice.

This is currently the closest mechanism-level similarity, but it is not a reproducer for LibreSign.

php-src #20818

Verified SIGSEGV in zend_objects_store_del, caused by tracing JIT passing the wrong value type to object destruction.

Our crash already reproduces without tracing JIT, so this specific bug does not explain it.

php-src #9400

Known PHP_CLI_SERVER_WORKERS bug in the built-in server that can hang requests.

It does not report this SIGSEGV, and our crash also reproduces with workers disabled, so it is not sufficient to explain the current failure.

php-src #21995

Memory-layout-sensitive shutdown UAF in PHP 8.4.

It is useful evidence that extension layout can expose latent UAFs, but the affected path/version is different from the current PHP 8.3 request-time crash.

php-src #23355

OPcache/map-pointer corruption involving different FPM pools and pool-local extensions.

The mechanism is interesting, but the reproducer requires FPM + shared OPcache state. Our current crash occurs in CLI server mode with OPcache not required.

php-src #23352 / #23353

Open UAF reports in ext/dom for PHP 8.3/8.4 involving DOMDocument::adoptNode().

There is currently no evidence that the LibreSign request-signature path executes that operation. The sanitizer build is a better discriminator than assuming DOM is responsible.

php-src #22061

Double-free through re-entrant destruction in Zend hash cleanup.

Again, this is a useful mechanism-level comparison because it can produce an already-destroyed object at zend_objects_store_del(), but there is no evidence yet that LibreSign executes the required destructor/re-entrancy pattern.

How to interpret the next run

• ASan/UBSan reports an earlier invalid access:

  • investigate that first operation, not the later Zend crash.

• exact list scenario fails without any prelude:

  • the scenario itself is sufficient; reduce inside it.

• exact list scenario passes but one prelude sequence fails:

  • bisect that prelude to identify the state contamination.

• draft request fails:

  • signer, identify-method and mail handling are not required.

• email notify fails while email no-notify passes repeatedly:

  • event/mail/notification handling becomes a strong discriminator.

• long-lived fails while fresh-process passes repeatedly:

  • process lifetime and delayed destruction become strong discriminators.

• controlled runtime fails:

  • unrelated runner extensions are not required.

• one extension group restores the failure:

  • bisect that group.

• Xdebug case reproduces:

  • use the final function trace as a PHP-level breadcrumb trail, but keep native evidence authoritative.

A passing single run is evidence, not proof. Any discriminator must be repeated before changing production code.

Regular CI

The normal LibreSign Behat workflows remain unchanged.

All crash-specific instrumentation stays isolated in this draft diagnostic workflow.

Out of scope

Do not:

• change LibreSign production behavior based only on the current crash frame;
• add retries to hide the failure;
• treat cURL error 52 as the root cause;
• blame Imagick, OPcache, worker count, DOM, Xdebug, or a database without a controlled discriminator;
• assume zend_objects_store_del() is where the corruption starts.

Refs #8422

[vitormattos]

Fixed by:
#8951