Skip to content

aligner: mates that share a file name keep their own converted temp files (#1265) - #1270

Merged
FelixKrueger merged 2 commits into
devfrom
rust/issue-1265-nondir-temp-names
Nov 21, 2026
Merged

FelixKrueger merged 2 commits into
devfrom
rust/issue-1265-nondir-temp-names

Conversation

@FelixKrueger

@FelixKrueger FelixKrueger commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Paired-end runs whose R1 and R2 files have the same name in different folders (-1 a/r.fq -2 b/r.fq) no longer overwrite their converted temp files. Those files now carry the mate number (r.fq_1_C_to_T.fastq) when the two file names match; every other run keeps its temp names, and output is unchanged.

Fixes #1265

Details (AI-assisted)
  • Affected: non-directional runs that write the converted reads as files — HISAT2 (always, also with --multicore), Bowtie 2 with --no_stream_converted or when --temp_dir can't hold named pipes, and all three --combined_index --non_directional paired-end models. Streamed Bowtie 2, Bowtie 2 --multicore, single-end and 5-Base were never affected. Perl names these files the same way.
  • With real Bowtie 2 on 400 simulated non-directional pairs, the same-name --no_stream_converted run used to exit 1 (Chromosome number extraction failed for *); it now gives the same BAM records as the distinct-name run and leaves no temp files.
  • File names are compared ignoring ASCII case (case-only names collide on macOS). A new check stops a run if two converted files of a pair would still share a path.
  • Tests: the new faithful and combined-index CLI tests fail on dev; removing the fix from either call path makes them fail via the new check. fmt, clippy -D warnings and cargo test -p bismark pass locally with and without binseq-input; HISAT2 was not run (not installed locally).
  • Same-named directional and pbat pairs, which already worked, also get the mate number in their temp names; their output does not change.
  • Review follow-ups are in the second commit: the single-pass model checks its two paths before writing, the equal-path check ignores ASCII case and names the pair, the faithful test checks each mate's converted sequence, and the CHANGELOG links this PR. Unicode-case name clashes and leftover temp files on errors are not handled here.
  • The wiki/ temp-file paragraph will follow after merge.

…iles (#1265)

The paired-end converted temp files were named <prefix.>?<file name><kind>.<ext>
with no mate number, so when R1 and R2 have the same file name in different
directories (-1 a/r.fq -2 b/r.fq) and both mates are converted to the same
kind, the second conversion overwrote the first. That is every non-directional
run that writes the converted reads as files: HISAT2 (always files, also with
--multicore, which is one -p N instance), Bowtie 2 with --no_stream_converted
or when no named pipe can be made in --temp_dir, and the three
--combined_index --non_directional paired-end models; the single-pass model
had both writers open on one path. With real Bowtie 2 the faithful run exited 1
("Chromosome number extraction failed for *"); the combined models died on a
read-name desync. Perl names these files the same way.

ConvertOptions::for_pair sets mate_in_name when the pair's file names are equal
(ASCII case-insensitive, since case-only names collide on macOS), and the PE
converters then name the files <file name>_<mate><kind>, e.g.
r.fq_1_C_to_T.fastq, r.fq_1_CT_GA_tagged.fastq. convert_pe_pair applies it for
every convert_pe_kind call site; the tagged model applies it inline. Pairs with
different file names keep today's names, so perl-oracle runs and stderr are
unchanged; same-named directional/pbat pairs, which worked, also get the
infix. converted_name is unchanged, so streamed FIFO names do not change.
reject_shared_converted_path stops a run whose pair would still share a
converted path. process_pe_chunk prints its "Created ... converted version"
lines after the pair's conversions instead of between them.

With real Bowtie 2 on 400 simulated non-directional pairs, the same-name
--no_stream_converted run now gives the same BAM records as the distinct-name
run and leaves no temp files.

Fixes #1265
The single-pass combined model checks its two tagged temp paths before opening
either writer, so a shared path stops the run before anything is written; the
separate check after conversion in process_pe_chunk_combined_nondir_tagged is
gone. The equal-path check is now convert::reject_shared_temp_path: it
compares ignoring ASCII case, matching for_pair, and names the pair ("the
converted reads of <r1> and <r2> share the temp file <path>").

tests: the faithful same-name test logs each aligner input's first header and
sequence and checks that each mate is its own reads, converted.

CHANGELOG links #1270 and says what users saw. One overlong doc line wrapped.
@FelixKrueger
FelixKrueger merged commit 08f57e9 into dev Nov 21, 2026
16 checks passed
@FelixKrueger
FelixKrueger deleted the rust/issue-1265-nondir-temp-names branch November 21, 2026 11:54
FelixKrueger added a commit that referenced this pull request Nov 21, 2026
wiki: paired-end temp names when R1 and R2 share a file name (#1270)
FelixKrueger added a commit that referenced this pull request Nov 21, 2026
…site) (#1274)

From the 3.4.0 pre-release audit (plans/11212026_release-3.4.0/):
- CHANGELOG: #1270 entry describes R2 overwriting R1's converted reads;
  #1264 names the UMI modes it needs; #1272 says streaming is the default
  since 3.2.0 and the hang is under CPU contention.
- rust/README.md: Milestones lines for #1264, #1266, #1270 and a 3.3.0
  backfill for #1261; the #1272 line links its PR; the header no longer
  points at the closed project board.
- Docs site: dedup --umi, --bclconvert and --parallel entries; the
  regular-file rule applies only above --umi_mismatches 0; extractor
  inputs are BAM/SAM/CRAM and colliding output names stop the run; the
  aligner's mate-2 --un/--ambiguous check; a note on the 3.2.0/3.3.0
  streamed-run hang; a typo.
- THIRD-PARTY-NOTICES and a Dockerfile comment drop stale beta/GA text.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant