fix(youtube): distinguish transient 429 from a missing transcript#34
Merged
Conversation
…oses #33) YouTube rate-limits the /api/timedtext content endpoint with HTTP 429 + an anti-bot page (transient, per-IP; normal browsing still returns 200). youtube_transcript_api raises IpBlocked solely on 429, and the sidecar swallowed it, so a temporary block was reported as "_No transcript available._" and cached as the final answer. Classify the fetch outcome (ok|none|blocked|error) in a new dependency-free markitdown-sidecar/yt_transcript.py: - blocked (429): honest "transcript exists but rate-limited, try later" message; flagged via transcript_status -> result.noStore so server.js and lib/mcp.js skip cache.put and refresh keeps the good snapshot. - none: genuinely absent -> unchanged message, still cacheable. Language selection unchanged (first listed transcript). Tests: 761 JS (+5) and 9 stdlib-only Python (markitdown-sidecar/test_youtube.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QzC4vKjJBjuAoD1JzyYAWp
syswave-dev
added a commit
that referenced
this pull request
Jun 22, 2026
PR #34 added markitdown-sidecar/yt_transcript.py (imported by app.py) but the Dockerfile only COPYied app.py + limits.py, so the deployed image crash-looped on `ModuleNotFoundError: No module named 'yt_transcript'`. Add the new module to the COPY layer. Claude-Session: https://claude.ai/code/session_01QzC4vKjJBjuAoD1JzyYAWp Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 22, 2026
syswave-dev
added a commit
that referenced
this pull request
Jun 25, 2026
…metadata on cache hits (#39) Strip 1x1 tracking pixels in cleanDom, add the sciencedaily-lead-image recipe (unwrap #text), and serve the full persisted metadata (og:image/twitter:image, description, author, …) on /api cache hits in both frontmatter and format=json. Batches untagged main commits since v3.1.0: #34 (youtube 429 vs missing transcript), #35 (bundle yt_transcript.py), #37 (X-Transcript-Status). Bumps version to 3.2.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #33.
Problem
YouTube videos that have a transcript were rendered as
_No transcript available._. The cause is a transient per-IP HTTP 429 on the/api/timedtextcontent endpoint (anti-bot rate-limit; normal browsing still returns 200, andapi.list()still sees the transcript).youtube_transcript_apiraisesIpBlockedsolely on 429, and the sidecar swallowed every exception, so a temporary block was indistinguishable from a genuinely absent transcript — and got cached as the final answer.Change
markitdown-sidecar/yt_transcript.py(dependency-free):fetch_snippetsreturns(snippets, status)with statusok | none | blocked | error, classifying a 429 asblocked._fmt_ts/_format_transcriptmoved here verbatim.app.py: honest message onblocked("transcript exists but YouTube rate-limited the request, try again later") instead of_No transcript available._; returnstranscript_status.lib/markitdown-client.js: passestranscriptStatusthrough.lib/web.js: setsresult.noStore = trueonblocked/error.server.js+lib/mcp.js: skipcache.putwhennoStore; refresh paths keep the existing good snapshot instead of overwriting it with a placeholder.none/okcache normally.A genuinely absent transcript keeps the old message and stays cacheable. Language selection is unchanged (first listed transcript: DE for German videos, EN for English videos).
Tests
noStoreon blocked/error, no-noStoreon ok/none, not-cached at the server layer) —node --test.python3 markitdown-sidecar/test_youtube.py.youtube_transcript_api: the reported video now classifies asblocked(previously masked as "none").🤖 Generated with Claude Code