Skip to content

feat(AmazonPhotos): add Videos importer - #1513

Merged
vipulkala95 merged 1 commit into
dtinit:masterfrom
vipulkala95:feat/amazon-videos-importer
Aug 21, 2026
Merged

vipulkala95 merged 1 commit into
dtinit:masterfrom
vipulkala95:feat/amazon-videos-importer

Conversation

@vipulkala95

Copy link
Copy Markdown
Collaborator

feat(amazon-photos): add Amazon Videos importer

Summary

Adds AmazonVideosImporter for the VIDEOS vertical, completing Amazon Photos as an import destination for both photos and videos. It reuses the AmazonImportHelper introduced in #1512 for per-job client creation (bounded LRU, endpoints resolved once per job), MD5/streamed download, and typed error classification — so photos and videos share one hardened code path. The importer is wired into AmazonTransferExtension alongside the existing photos importer.

What's changed

  • New AmazonVideosImporter — imports VideosContainerResource, creates albums, and uploads videos via AmazonImportHelper.getOrCreateClient(...). Duplicates (DuplicatesConflictError) are skipped idempotently; storage-quota conditions (InsufficientStorage, NoActiveSubscriptionFound) surface as a terminal DestinationMemoryFullException. Opt-in platform retry via the enableRetrying setting, consistent with the photos importer.
  • AmazonTransferExtension — constructs and returns the videos importer for DataVertical.VIDEOS.
  • New AmazonVideosImporterTest — mirrors the photos importer test suite.

Testing

Unit tests (./gradlew :extensions:data-transfer:portability-data-transfer-amazon:build, Java 11 — green):

  • Retry-executor selection (enabled/disabled/absent), storage-quota → terminal failure, duplicate-skip, upload happy path (uploaded-time mapping + temp cleanup), and per-job client isolation via the shared helper.
  • Line coverage for AmazonVideosImporter: 100%.

End-to-end (local demo server, Google → Amazon, VIDEOS vertical):

  • Simple upload — small videos imported via the single-request upload path. ✅
  • Multipart upload — a large video exercised the multipart path (initiate → upload parts → complete → completion polling with backoff); job ended SUCCESSFULLY_COMPLETED with the client's [MultipartPoll] progress observable in the logs. ✅
  • Idempotent re-run — re-running the same transfer detected all items as already present and skipped duplicates, completing with 0 errors ✅

Add AmazonVideosImporter (VIDEOS vertical) that reuses AmazonImportHelper for per-job client creation and error classification, and wire it into AmazonTransferExtension alongside the photos importer.
@vipulkala95
vipulkala95 merged commit 70ad414 into dtinit:master Aug 21, 2026
7 checks passed
@vipulkala95
vipulkala95 deleted the feat/amazon-videos-importer branch August 21, 2026 10:49
vipulkala95 added a commit that referenced this pull request Sep 9, 2026
### Summary
Adds support for the `MEDIA` data vertical to the Amazon Photos
extension. reusing the shared import path introduced with the
Photos/Videos importers (#1512, #1513).
  
  ### Changes
- **`AmazonMediaImporter`** — imports `MediaContainerResource` (albums +
photos + videos) into Amazon Photos, delegating album creation,
download+MD5, upload, and duplicate/quota handling to
  the shared `AmazonImportHelper`.
- **`UploadItemRequest`** — new immutable parameter object with
`forPhoto`/`forVideo` factories; keeps the per-vertical field mapping in
one place. `AmazonPhotosImporter` and
`AmazonVideosImporter` are refactored to use it, removing duplicated
upload wiring.
- **`AmazonMediaTransmogrificationConfig`** — transmogrification config
for the MEDIA vertical.
- **`AmazonTransferExtension`** — registers the importer for
`DataVertical.MEDIA`.
- **`AmazonOAuthConfig`** — adds import/export scopes for the MEDIA
vertical.
- **`AmazonImportHelper`** — shared upload/album operations extended to
serve the media importer.
  
  ### Testing
- Completed transfer using the vertical successfully for both videos and
photos.
- New `AmazonMediaImporterTest` covers albums+photos+videos
registration, favorited items, duplicate skip, insufficient-storage →
`DestinationMemoryFullException`, retry-executor selection,
  and happy-path uploads with uploaded time.
  - New `AmazonOAuthConfigTest` cases for media import/export scopes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants