Repository navigation
feat: dedicated Loom importer that copies videos and defers processing to first view - #3077
Conversation
|
@greptileai review |
|
@greptileai review |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
@greptileai review |
|
@greptileai review |
|
@greptileai review |
| const originalKey = | ||
| worker === "importer" && loomSourceUrl && isDashUrl(loomSourceUrl) | ||
| ? `${userId}/${videoId}/raw-upload.webm` | ||
| : undefined; |
There was a problem hiding this comment.
Progressive WebM imports have no original upload target
Medium Severity
When Loom returns a progressive VP9/Opus WebM URL, getLoomDownloadUrl accepts it and this workflow now sends it to the importer. However, originalKey is allocated only for .mpd URLs, so originalUpload is absent. runLoomImport probes the downloaded WebM, chooses the WebM container, and throws “This Loom recording needs an original upload target”; subsequent attempts repeat the same failure. The previous media-server path could transcode this input. Supply an original target for progressive WebM sources too, or retain the media-server path for them.
| const deferredLoomRawFileKey = hasActiveUpload | ||
| ? null | ||
| : getPendingLoomProcessing(metadata, video.owner.id, videoId); |
There was a problem hiding this comment.
Failed deferred starts never reach the reclaim path
Medium Severity
If startVideoProcessingWorkflow rejects the workflow start after the deferred helper inserts videoUploads, it leaves that row in phase error. Both this page and the embed page define hasActiveUpload by the existence of any upload row, including error rows, so this guard prevents getPendingLoomProcessing from running even after its 30-minute reclaim timeout. The stalled-processing cron also excludes error rows. A transient startup failure therefore leaves optimization stuck until manual intervention. Allow failed deferred starts through a reconciliation/retry path that verifies the previous workflow is inactive while preserving uncertain active starts.
Loom imports currently run as full media-server jobs: ffmpeg reads Loom's HLS/DASH segments one at a time, then every video is transcoded and gets thumbnail and GIF work on the shared processing fleet, and bulk imports compete with recordings for slots. This PR adds a dedicated importer that only copies Loom media into Cap storage. Anything costly waits until someone opens the video.
flowchart LR W[import-loom-video workflow] -->|Loom URL + upload targets| I[loom-importer<br/>media.cap.so/loom-importer] I -->|parallel segment / range GETs| L[(Loom CDN)] I -->|stream copy| R{codecs} R -->|H.264 / AAC| M[result.mp4 + thumbnail + GIF] R -->|VP9 / Opus| O[raw-upload.webm<br/>loomDeferredProcessing: pending] M --> H[progress webhook<br/>transcription=on-view] O --> H S[/s/ or /embed/ first view/] -->|pending original| P[process-video workflow]Importer (
apps/media-server/src/loom-importer)bun run src/loom-importer/index.ts, served under/loom-importer/*, speaking the existing/video/status,/video/importand/video/process/:jobId/statusprotocol with the same webhook and multipart upload targets.materializeStreamingInputfor HLS and DASH-as-HLS, then picks the highest-bandwidth variant plus its default audio rendition (streaming.ts:selectHlsTracks). Every segment is downloaded in parallel to local files and ffmpeg reads the rewritten local playlist (streaming.ts:prefetchMaterializedHls). Timing is handled exactly as before; only network latency goes away.downloadUrlToPathParallel.media.ts:buildRemuxArgs): H.264/AAC becomes a faststartresult.mp4; VP9/Opus becomes the original WebM, reported asdeferredSourceKey.https://*.loom.comwith redirects disabled (fetch.ts:assertAllowedSourceUrl). Uploads only go to presigned targets supplied by the web app.LOOM_IMPORTER_MAX_JOBS,LOOM_IMPORTER_MAX_QUEUED), global and per-job connection limits, a free-disk guard, graceful drain on SIGTERM, and retried terminal webhooks.Web
LOOM_IMPORTER_URL(optional). When set, Loom downloads inimport-loom-video.tsgo to the importer. Retries that already have a preserved original keep using the media server, and job reconciliation checks both workers (loom-media-dispatch.ts). When unset, behaviour is unchanged.raw-upload.webmtarget (multipart on S3, presigned PUT on Drive). Multipart uploads now carry the right content type (loom-import-upload.ts).transcription=on-viewskips transcription on completion and setsmetadata.transcribeOnView, which the stalled-transcription cron honours. A validdeferredSourceKeysetsmetadata.loomDeferredProcessing(progress/route.ts).process-videoworkflow (lib/loom-deferred-processing.ts). Until then the existing raw-preview fallback plays the WebM.POST /api/internal/loom-imports/redrive(media-server secret, max 25 ids) restarts failed or long-stalled Loom imports in place on the same Cap video, mirroring the existing restart path inactions/loom.ts(lib/loom-import-redrive.ts).Rollout
cap-mediaALB with a/loom-importer/*rule.LOOM_IMPORTER_URL=https://media.cap.so/loom-importerin production.

Confidence Score: 5/5
No new blocking issue was found in the changes since the previous review.Summary
Adds a dedicated Loom importer that copies videos to storage and leaves conversion and transcription until the first view.
All six supplied earlier threads were unnumbered and were manually resolved by
richiemcilroywithout explanatory replies. No duplicate comments were posted.Reviews (6) · Last reviewed commit: "fix: keep deferred Loom claims when a co..." · Reviewed by Greptile