PS-11483: Wrong query results when an internal temporary table is converted to on-disk InnoDB

[jaideepkarande]

Summary

Backports upstream MySQL fix for Bug #116741 (Oracle commit cde9f3eaaf9, ships in MySQL 8.4.11) to the Percona Server 8.4 series, plus the Percona-specific test adjustment required to make the upstream reproducer actually exercise the fixed path on this branch.

• Jira: https://perconadev.atlassian.net/browse/PS-11483
• Customer ticket: CUSTOM-258
• Target: 8.4 (branch base is 8.4.10-10)
• Sibling PR (PXC): PXC-5305: Wrong query results when an internal temporary table is converted to on-disk InnoDB percona-xtradb-cluster#2348

Without the fix, a query can silently return an incorrect result set — no error, no warning — when an internal temporary table is converted from MEMORY to an on-disk InnoDB table mid-execution. Rows that exist are dropped from the result. The only "workaround" is avoiding the conversion, which does not hold in production as data volume grows.

Commits

Commit	Scope
7ab214a Bug#37308710 — Innodb tmp table causes incorrect query results	sql/sql_tmp_table.cc (+7), main.with_recursive_innodb_tmp_table test + result
a8ea488 CUSTOM-258	main.with_recursive_innodb_tmp_table test + result only

Whole-PR diff is exactly three files:

mysql-test/r/with_recursive_innodb_tmp_table.result
mysql-test/t/with_recursive_innodb_tmp_table.test
sql/sql_tmp_table.cc

The only source change is one statement in create_ondisk_from_heap():

/*
  Reading from the in-memory table clears wtable's not-started state, so reset
  it here.
*/
wtable->set_not_started();

Reading rows from the in-memory temporary table advances table state. After conversion the table kept a started state, so subsequent reads from the on-disk temporary table could miss the expected row for CONST access.

Why the second commit is needed (Percona divergence from upstream)

The upstream subtest reaches the conversion path by running the query at tmp_table_size = 102400 (in-memory) and then at tmp_table_size = 1024 (expected to spill to disk).

Percona-only patch PS-8647 in sql/sys_vars.cc silently rewrites any tmp_table_size below 1 MiB up to exactly 1 MiB and pushes a warning. The rewrite is unconditional and independent of internal_tmp_mem_storage_engine, so it also applies when the session uses the MEMORY engine.

Consequence on this branch: the effective tmp_table_size is 1 MiB in both runs. The 99-row derived result (~40-byte reclength) fits comfortably in 1 MiB, create_ondisk_from_heap() is never entered, and the MEMORY→InnoDB transition the backport is meant to exercise never happens. The upstream test therefore failed with two diff hunks: unexpected Tmp_table_size is set below 1MiB warnings on each SET, and Created_tmp_disk_tables reporting 0 where upstream expects 1.

a8ea488 is test-only — the backported source fix stays as-is. It gives the reproducer enough data to overflow the 1-MiB floor PS-8647 imposes:

• col1 widened to VARCHAR(300) CHARACTER SET latin1, generator grown to 4000 rows → derived table ~1.2 MiB
• max_heap_table_size explicitly saved/restored and set to 16 MiB for this section (the preceding with_recursive_wl9248 block left it at 61000, which would cap MEMORY well below tmp_table_size and defeat the size differential)
• tmp_table_size values at or above the PS-8647 threshold: 16 MiB in-memory (Created_tmp_disk_tables = 0) and exactly 1 MiB on-disk (Created_tmp_disk_tables = 1) — both ≥ 1 MiB, so the strict < check never fires and no unexpected warnings appear

The count(distinct d.col1) = 1 assertion — the actual bug being verified — is untouched. A comment block documents the divergence so the next porter does not "fix" the values back to the upstream ones.

Testing

Full MTR (FULL_MTR=yes, --big-test, CI_FS_MTR=yes, WITH_PS_PROTOCOL=yes), x86_64:

Config	Job	Build	OS	Result	Failures
Debug	percona-server-8.4-pipeline-parallel-mtr	#1431	ubuntu:noble	UNSTABLE	1
RelWithDebInfo	percona-server-8.4-pipeline-parallel-mtr	#1432	ubuntu:noble	UNSTABLE	5
Debug + ASAN	percona-server-8.4-ASAN-pipeline-parallel-mtr	#22	ubuntu:resolute	UNSTABLE	69

The target test now passes in every configuration

Build	Result
#1431 Debug	main.with_recursive_innodb_tmp_table w3 [ pass ] 42288
#1432 RelWithDebInfo	main.with_recursive_innodb_tmp_table w8 [ pass ] 3369
#22 ASAN	main.with_recursive_innodb_tmp_table w3 [ pass ] 80485

Baseline diff

Baselines are the most recent runs of the same jobs, #1430 (Debug) and #21 (ASAN) — both on release-8.4.11-11, i.e. a different patch base than this branch (8.4.10-10). Deltas below are interpreted with that in mind.

Debug — #1431 vs #1430

One failure, main.all_persisted_variables, deterministic (failed, then retry-fail). Root cause is a persisted-variable count mismatch, not a behaviour change:

-include/assert.inc [Expect 489 persisted variables in the table.]
+include/assert.inc [Expect 490 persisted variables in the table.]

The 8.4.10-10 base has one more persistable system variable than the test's hard-coded expectation. This PR registers no system variable — its entire diff is three files, none of them sql/sys_vars.{cc,h} or all_persisted_variables.{test,result} — so it cannot move this count. Pre-existing on the branch base; the 8.4.11-11 baseline passes because the count was reconciled there.

The baseline's own failure (the unit_tests ctest roll-up) does not reproduce in #1431.

RelWithDebInfo — #1432

rocksdb_rpl.rpl_rocksdb_row_img_idx_{noblob,full,min}.row, one shutdown_report, and the unit_tests roll-up (also failing in baseline #1430). None touch the temporary-table path. No same-config/same-OS baseline exists for RelWithDebInfo on this job — the nearest, #1425, ran on debian:bullseye with KEYRING_VAULT_MTR=yes.

ASAN — #22 vs #21

69 vs 28. All 28 baseline failures reproduce; the ~41 additional ones are dominated by an environment regression in the ubuntu:resolute image, not by this change. Tests that shell out to external binaries (main.symlink, main.import_symlink, main.mysqld_safe, main.mysql_system_cmd_unix, main.log_errchk, main.temp_table_debug) now fail on LeakSanitizer reports raised against Ubuntu resolute's rust-coreutils, not against mysqld. Sample, from main.temp_table_debug:

==237077==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 ... in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:67
    #1 ... (/usr/lib/cargo/bin/coreutils/ln+0x53f5e6)

ubuntu:resolute is a development release and the image is pulled fresh per build; #21 ran 5 days earlier. The remaining clusters (10 × rocksdb, 2 × rocksdb_stress, 9 × group_replication clone/recovery) are whole-worker environmental/timing failures under instrumentation.

Both #21 and #22 carry the identical pre-existing ASAN DEADLYSIGNAL SEGV signature (pc 0x0, <unknown module>), so that one is not new either.

Caveats / follow-ups

1. Baselines are on 8.4.11-11, this branch is on 8.4.10-10. Notably Fixed tarball name and revision number in build-binary.sh (JEN-318) #22 reports leak signatures absent from Fixed tarball name and revision number in build-binary.sh (JEN-318) #21 (2602 B/2 allocs, 43224 B/68, 33 B/1, 86448 B/136). None are in the temporary-table path and a set_not_started() state reset allocates nothing, but formally classifying them needs an ASAN run on the unmodified 8.4.10-10 base. Recommend firing that before merge; happy to do it on request.
2. main.all_persisted_variables on 8.4.10-10 (489 vs 490) deserves its own ticket — it will fail for anyone testing this base.
3. SLACK_CHANNEL in Percona-Lab/ps-build jenkins/pipeline-parallel-mtr.groovy is interpolated as "#${SLACK_CHANNEL}" against a default that already carries #, producing channel: ##ps-upstream-merges. Slack notifications from these jobs never land. Worth a separate ticket.

🤖 Generated with Claude Code

[inikep]

LGTM, but don't merge to 8.4 trunk (only to the custom build)