Fix export transactions#659
Merged
Merged
Conversation
Before that commit, if you were starting an export of transactions, but stop in the middle, the process was still continuing in EB. Producing unnecessary calls to DB. The request could be stopped by nginx because of a timeout and then the user might then want to re-request, leading to double work in EB.
It was hardcoded to 1 previously because I thought it would break the transactions order if we we batching our transactions, but actually not, the order is preserved (by time). So it ease the load of the DB (batching request, instead of 1 by 1 query for each transaction).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, if a transaction export started but the user canceled it (or the connection dropped), the backend would just keep churning away in the background. This resulted in a ton of zombie DB calls and wasted CPU.
We saw this hit us in prod: EB was a bit slow, Nginx timed out the download, and the user (rightfully) clicked retry multiple times. This created a "stack" of simultaneous exports for the same request, which blew up our RAM.
Changes:
batchSizefrom 1 to 100. Previously, we were doing a DB round-trip for every single transaction, which was super inefficient (because I thought we wouldn't preserve transactions order with batch, which I was wrong). Now we fetch and process in chunks of 100, which significantly reduces the IO overhead and helps the stream keep up with Nginx timeouts.Testing:
I simulated the prod failure by setting a 1s Nginx timeout and forcing a 2s delay between batches in EB. The internal process now correctly catches the disconnect and stops immediately.