Skip to content

feat: dedicated Loom importer that copies videos and defers processing to first view - #3077

Merged
richiemcilroy merged 11 commits into
mainfrom
feat/loom-importer
Oct 24, 2026
Merged

richiemcilroy merged 11 commits into
mainfrom
feat/loom-importer

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Loom imports currently run as full media-server jobs: ffmpeg reads Loom's HLS/DASH segments one at a time, then every video is transcoded and gets thumbnail and GIF work on the shared processing fleet, and bulk imports compete with recordings for slots. This PR adds a dedicated importer that only copies Loom media into Cap storage. Anything costly waits until someone opens the video.

flowchart LR
  W[import-loom-video workflow] -->|Loom URL + upload targets| I[loom-importer<br/>media.cap.so/loom-importer]
  I -->|parallel segment / range GETs| L[(Loom CDN)]
  I -->|stream copy| R{codecs}
  R -->|H.264 / AAC| M[result.mp4 + thumbnail + GIF]
  R -->|VP9 / Opus| O[raw-upload.webm<br/>loomDeferredProcessing: pending]
  M --> H[progress webhook<br/>transcription=on-view]
  O --> H
  S[/s/ or /embed/ first view/] -->|pending original| P[process-video workflow]
Loading

Importer (apps/media-server/src/loom-importer)

  • New entrypoint, same image as the media server: bun run src/loom-importer/index.ts, served under /loom-importer/*, speaking the existing /video/status, /video/import and /video/process/:jobId/status protocol with the same webhook and multipart upload targets.
  • Reuses materializeStreamingInput for HLS and DASH-as-HLS, then picks the highest-bandwidth variant plus its default audio rendition (streaming.ts:selectHlsTracks). Every segment is downloaded in parallel to local files and ffmpeg reads the rewritten local playlist (streaming.ts:prefetchMaterializedHls). Timing is handled exactly as before; only network latency goes away.
  • Progressive MP4 sources use the existing ranged downloadUrlToPathParallel.
  • Output is a stream copy (media.ts:buildRemuxArgs): H.264/AAC becomes a faststart result.mp4; VP9/Opus becomes the original WebM, reported as deferredSourceKey.
  • Every fetch is limited to https://*.loom.com with redirects disabled (fetch.ts:assertAllowedSourceUrl). Uploads only go to presigned targets supplied by the web app.
  • In-memory queue (LOOM_IMPORTER_MAX_JOBS, LOOM_IMPORTER_MAX_QUEUED), global and per-job connection limits, a free-disk guard, graceful drain on SIGTERM, and retried terminal webhooks.
53 min Loom webinar, us-east-1 Media server today Importer
Download (HLS, 779 MB) ~minutes, one segment at a time 3.1 s cached, ~40 s uncached
Make playable full transcode + previews 1.8 s remux
Upload to S3 single PUT 1.3 s multipart

Web

  • LOOM_IMPORTER_URL (optional). When set, Loom downloads in import-loom-video.ts go to the importer. Retries that already have a preserved original keep using the media server, and job reconciliation checks both workers (loom-media-dispatch.ts). When unset, behaviour is unchanged.
  • DASH imports get a raw-upload.webm target (multipart on S3, presigned PUT on Drive). Multipart uploads now carry the right content type (loom-import-upload.ts).
  • Progress webhook: transcription=on-view skips transcription on completion and sets metadata.transcribeOnView, which the stalled-transcription cron honours. A valid deferredSourceKey sets metadata.loomDeferredProcessing (progress/route.ts).
  • The first share or embed view of a pending original claims it once and starts the normal process-video workflow (lib/loom-deferred-processing.ts). Until then the existing raw-preview fallback plays the WebM.
  • POST /api/internal/loom-imports/redrive (media-server secret, max 25 ids) restarts failed or long-stalled Loom imports in place on the same Cap video, mirroring the existing restart path in actions/loom.ts (lib/loom-import-redrive.ts).

Rollout

  1. Run the importer on its own instance behind the cap-media ALB with a /loom-importer/* rule.
  2. Set LOOM_IMPORTER_URL=https://media.cap.so/loom-importer in production.

RetriggerView in GreptileConfidence Score: 5/5

No new blocking issue was found in the changes since the previous review.

Summary

Adds a dedicated Loom importer that copies videos to storage and leaves conversion and transcription until the first view.

  • Loom imports copy video before full processing.
  • Pending WebM videos start full processing on first view.
  • Failed or stalled Loom imports can restart on the same video.

All six supplied earlier threads were unnumbered and were manually resolved by richiemcilroy without explanatory replies. No duplicate comments were posted.

Reviews (6) · Last reviewed commit: "fix: keep deferred Loom claims when a co..." · Reviewed by Greptile

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread apps/web/app/s/[videoId]/page.tsx Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Oct 24, 2026 9:26am UTC

@vercel
vercel Bot temporarily deployed to Preview – cap-status October 24, 2026 09:18 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@vercel
vercel Bot temporarily deployed to Preview – cap-status October 24, 2026 09:22 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 2 potential issues (2 medium).

View 1 other finding in Capy.

Open in Capy

Comment on lines +510 to +513
const originalKey =
worker === "importer" && loomSourceUrl && isDashUrl(loomSourceUrl)
? `${userId}/${videoId}/raw-upload.webm`
: undefined;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Progressive WebM imports have no original upload target

Medium Severity

When Loom returns a progressive VP9/Opus WebM URL, getLoomDownloadUrl accepts it and this workflow now sends it to the importer. However, originalKey is allocated only for .mpd URLs, so originalUpload is absent. runLoomImport probes the downloaded WebM, chooses the WebM container, and throws “This Loom recording needs an original upload target”; subsequent attempts repeat the same failure. The previous media-server path could transcode this input. Supply an original target for progressive WebM sources too, or retain the media-server path for them.

Open in Capy

Comment on lines +881 to +883
const deferredLoomRawFileKey = hasActiveUpload
? null
: getPendingLoomProcessing(metadata, video.owner.id, videoId);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failed deferred starts never reach the reclaim path

Medium Severity

If startVideoProcessingWorkflow rejects the workflow start after the deferred helper inserts videoUploads, it leaves that row in phase error. Both this page and the embed page define hasActiveUpload by the existence of any upload row, including error rows, so this guard prevents getPendingLoomProcessing from running even after its 30-minute reclaim timeout. The stalled-processing cron also excludes error rows. A transient startup failure therefore leaves optimization stuck until manual intervention. Allow failed deferred starts through a reconciliation/retry path that verifies the previous workflow is inactive while preserving uncertain active starts.

Open in Capy

@richiemcilroy
richiemcilroy merged commit 0eee68c into main Oct 24, 2026
45 checks passed

This branch was successfully deployed

1 active and 1 inactive deployments
Preview – cap-web — 3e8cd63a Deployed Oct 24, 2026 by vercel[bot]
Preview – cap-status — 3e8cd63a Deployed Oct 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant