Conversation
A pread on an open Hadoop stream was serialized twice: `synchronized` on FileInputStream.read(long, ...) in the Java SDK, and fs.File's mutex held across the object storage wait in Pread. Random reads of uncached blocks on one file were therefore limited to one in-flight request per file, which caps HBase gets on a cold cache at the number of HFiles per region server. Java: drop `synchronized` from the two positioned read methods. A ReentrantReadWriteLock keeps close() from freeing the fd under an in-flight pread (read lock around jfs_pread, write lock around jfs_close), so preads overlap each other but never a close. Sequential reads, seek and close keep the stream monitor as before. Go: File.Pread holds the lock only while it checks the size, flushes pending writes and obtains the reader; the vfs.FileReader.Read call runs outside it, as the FUSE path already does. File.Close marks the file closed and waits for in-flight preads before releasing the reader, so a slow read cannot see the closed reader and return EOF, and a read after close fails with EBADF instead of opening a new reader that is never closed. Tests: TestFileConcurrentPread shows four preads on one file reach the object storage at the same time (one before), TestFileCloseWaitsForPread covers the close ordering, and JuiceFileSystemTest gains a check that a pread completes while another thread holds the stream monitor plus concurrent-read and close-during-read regression tests.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7506 +/- ##
===========================================
+ Coverage 22.03% 57.12% +35.08%
===========================================
Files 31 180 +149
Lines 24803 59801 +34998
===========================================
+ Hits 5466 34161 +28695
- Misses 18742 22028 +3286
- Partials 595 3612 +3017 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Every random read went through a sliceReader: fileReader.Read created a slice, started `go s.run()` and parked on the slice's cond until the block arrived, even when the block was already in the local cache. That hand-off is cheap for FUSE but not for the Hadoop SDK, where the caller is an OS thread pinned by a cgo call: each park hands its P to another M and the wake-up goes through a futex, so cache hits get slower the more threads read one file at once. With positioned reads no longer serialized per stream (5488d6a), an HBase workload on a warm cache lost throughput at high concurrency and the process grew extra OS threads under load; this commit brings both back to where they were. fileReader.Read now first tries readCached: when the range lies in one chunk, is not part of a sequential session (readahead stays with the slice readers), and every block is in the local cache, it copies the data on the calling goroutine without a slice, a goroutine or the dataReader lock. A miss anywhere falls back to the regular path unchanged, so cold reads keep their concurrency. The read at offset 0 and reads inside a session's readahead window also take the regular path so sequential detection and readahead behave as before; the fast path still records the session as checkReadahead would. chunk: add CachedReader with ReadCachedAt, implemented by rSlice as the cache-only half of ReadAt (same cache-hit metrics, same removal of a partial block). Readers that do not implement it simply never take the fast path. Tests: TestReadCachedAt covers block boundaries, eviction of one block, reads beyond the slice and a disabled cache. TestReadCached shows a cached read reaching neither the object storage nor a slice reader, and a partly cached read falling back to one.
mwkang
force-pushed
the
feature/hadoop-concurrent-pread
branch
from
September 8, 2026 01:26
f360790 to
229d111
Compare
Drop the short-read check in ReadCachedAt: io.ReaderAt returns a non-nil error whenever n < len(p), and both cache readers (os.File.ReadAt, the page reader) follow it, so the branch could not run. Tests for the paths a cached read leaves the fast path on: a range past the end of the file, offset 0, a closed reader, an inode the metadata engine does not know, a chunk store whose reader has no ReadCachedAt, and a cached block shorter than its key claims (which is removed). A read inside a hole is served as zeros without a slice reader or the object storage; a read spanning the hole and uncached data takes the regular path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
close #7504
Positioned reads on one Hadoop stream were serialized by
synchronizedinFileInputStreamand by thefs.Filemutex held across the object storage wait inFile.Pread, so a file with cache misses had at most one in-flight read.The second commit keeps cache hits cheap once those reads overlap. Every random read used to start a sliceReader goroutine and park until it finished, even for a cached block; for a caller pinned to an OS thread by cgo (the Hadoop SDK) each park is a P hand-off and a futex wake-up, which adds up when several threads read one file.
fileReader.Readnow copies a fully cached range on the calling goroutine and leaves everything else (cache misses, offset 0, sequential sessions with readahead) to the regular path.