Add e2e test for S3 destination processing pipeline#202
Conversation
PuneetPunamiya
commented
May 7, 2026
- Tests the complete flow from source S3 bucket through document processing, chunking, and vector embeddings generation to destination S3 bucket
There was a problem hiding this comment.
Code Review
This pull request introduces a new end-to-end test, TestS3Destination, which validates the unstructured data processing pipeline using S3 for both source and destination. The feedback focuses on improving the robustness and cleanliness of the test code. Key suggestions include avoiding hardcoded SQS URLs by capturing them from the creation response, using t.Fatal for setup failures to prevent invalid test states, removing redundant client initializations, and ensuring the teardown logic comprehensively cleans up all created resources, including the data storage bucket and SQS queue.
0927039 to
57009cf
Compare
57009cf to
ae77fb9
Compare
ae77fb9 to
7ab850d
Compare
|
@PuneetPunamiya , After rebasing you haven't rename the unstructureddataproduct to unstructureddatapipeline at line 942 and 1012 as function name is updated , please take a look of that
|
7ab850d to
98e3b49
Compare
Tests the complete flow from source S3 bucket through document processing, chunking, and vector embeddings generation to destination S3 bucket Signed-off-by: Puneet Punamiya <ppunamiy@redhat.com>
98e3b49 to
dc475d2
Compare
| } | ||
| t.Log("UnstructuredDataPipeline is ready after S3 destination patch") | ||
|
|
||
| if err := apimachinerywait.PollUntilContextTimeout( |
There was a problem hiding this comment.
why this wait is required ?
dc475d2 to
0da2ef9
Compare
Verify destination embeddings are not re-uploaded when a new source file is added, and are re-uploaded when an existing file is overwritten. Signed-off-by: Puneet Punamiya <ppunamiy@redhat.com>
0da2ef9 to
afcd7a3
Compare