feat: add artifacts-based configuration for selective destination syncing by PuneetPunamiya · Pull Request #126 · redhat-data-and-ai/unstructured-data-controller

PuneetPunamiya · 2026-03-30T14:34:12Z

Enables selective syncing of file types with custom artifact paths under
a stages directory structure
Users can configure which file types (converted/chunks/embeddings) to sync
and customize their destination paths
Example usage in UnstructuredDataProduct

spec:
  destinationConfig:
    artifacts:
     - type: stage
       name: documentProcessorConfig
       path: processed-documents
     - type: stage
       name: chunksGeneratorConfig
       path: chunks
     - type: stage
       name: vectorEmbeddingsGeneratorConfig
       path: vector-embeddings

gemini-code-assist

Code Review

This pull request introduces a SyncToDestination configuration across the DocumentProcessor, ChunksGenerator, and VectorEmbeddingsGenerator components, allowing for granular control over which processed files are synchronized to the final destination. The controllers have been updated to conditionally trigger reconciliation and synchronization based on these flags and the presence of unprocessed files. Additionally, the Snowflake destination logic was refactored to handle multiple file types and prevent redundant uploads by comparing metadata. Review feedback suggests improving the robustness of JSON parsing for raw file paths by using minimal anonymous structs and refactoring the metadata comparison logic to reduce code duplication using generics.

piyush-garg · 2026-06-01T03:45:09Z


+func buildDestinationPath(ctx context.Context, unstructuredDataProductCR operatorv1alpha1.UnstructuredDataPipeline, filePaths []string) []unstructured.ArtifactFiles {
+	logger := log.FromContext(ctx)
+	var artifactFileGroups []unstructured.ArtifactFiles


what is artifactFileGroup? is it not containing list of files ?

Here ArtifactFileGroup is one configured artifact i.e.

Files - list of file paths in filestore

Path - destination path

Shall I rename it to something like ArtifactSyncGroup

piyush-garg · 2026-06-01T03:46:28Z


-	// list all files in the filestore for the data product
+	// list files to check if processing is needed
 	filePaths, err := r.fileStore.ListFilesInPath(ctx, dataProductName)


why it is done here ?

piyush-garg · 2026-06-01T03:48:55Z

+		}
+
+		artifactPath := artifact.Path
+		if artifactPath == "" {


why can we check suffix and path together in single loop ?

…cing Enables selective syncing of file types with custom artifact paths under a stages directory structure Users can configure which file types (converted/chunks/embeddings) to sync and customize their destination paths Example usage in UnstructuredDataProduct spec: destinationConfig: artifacts: - type: stage name: documentProcessorConfig path: processed-documents - type: stage name: chunksGeneratorConfig path: chunks - type: stage name: vectorEmbeddingsGeneratorConfig path: vector-embeddings Signed-off-by: Puneet Punamiya <ppunamiy@redhat.com>

Signed-off-by: Puneet Punamiya <ppunamiy@redhat.com>

gshikhar2021 · 2026-06-04T11:36:30Z

+	var extractedPrefix string
+	if idx := strings.LastIndex(filePathInFilestore, "/"); idx != -1 {
+		extractedPrefix = filePathInFilestore[:idx]
+	}
+


here we are extracting out the prefix using last '/'index , what if there is '/' in the file name ??????
can't we take dp name and prefix from CR and create the complete prefix???

PuneetPunamiya had a problem deploying to e2e-tests March 30, 2026 14:34 — with GitHub Actions Failure

gemini-code-assist Bot reviewed Mar 30, 2026

View reviewed changes

Comment thread pkg/unstructured/destination.go Outdated

Comment thread pkg/unstructured/destination.go Outdated

PuneetPunamiya force-pushed the adds-support-for-selective-destination-sync branch from ab1e868 to 973922f Compare March 31, 2026 10:00

PuneetPunamiya had a problem deploying to e2e-tests March 31, 2026 10:00 — with GitHub Actions Failure

PuneetPunamiya force-pushed the adds-support-for-selective-destination-sync branch from 973922f to 3d60885 Compare April 7, 2026 05:32

PuneetPunamiya had a problem deploying to e2e-tests April 7, 2026 05:32 — with GitHub Actions Failure

PuneetPunamiya force-pushed the adds-support-for-selective-destination-sync branch from 3d60885 to 847640c Compare April 7, 2026 05:39

PuneetPunamiya had a problem deploying to e2e-tests April 7, 2026 05:39 — with GitHub Actions Failure

PuneetPunamiya changed the title ~~feat: add syncToDestination flags for selective file syncing~~ feat: add artifacts-based configuration for selective destination syncing Apr 7, 2026

PuneetPunamiya force-pushed the adds-support-for-selective-destination-sync branch from 847640c to e1b8e69 Compare April 7, 2026 06:21

PuneetPunamiya had a problem deploying to e2e-tests April 7, 2026 06:21 — with GitHub Actions Failure

PuneetPunamiya force-pushed the adds-support-for-selective-destination-sync branch from e1b8e69 to ea88c03 Compare April 7, 2026 08:10

PuneetPunamiya had a problem deploying to e2e-tests April 7, 2026 08:10 — with GitHub Actions Failure