LangChain YouTube Transcript Loader That Survives Production

Every LangChain YouTube transcript tutorial shows the same three lines: YoutubeLoader.from_youtube_url(), load, done. Then you deploy to a cloud box and it silently returns nothing. Below is a loader that returns proper Document objects from a hosted API instead — about 20 lines, plus a summarize chain to prove it works end to end.

Why the built-in YoutubeLoader dies on servers

YoutubeLoader doesn't call an official API. It scrapes YouTube's internal caption endpoints from whatever machine your code runs on. From a residential IP that mostly works. From an AWS, GCP, or Hetzner IP it gets blocked or served empty responses, because YouTube treats datacenter traffic as bot traffic. I wrote up the details in why YouTube transcript scraping breaks on cloud IPs — the short version is that the failure is environmental, so it passes every local test and dies exactly where it matters.

youtube2text.org does the fetching on its side and hands you JSON, which turns your loader into a plain HTTP call.

The loader

import requests
from langchain_core.documents import Document

API_KEY = "yt_your_key_here"

def load_youtube_transcript(url: str, max_chars: int = 150000) -> Document:
    resp = requests.get(
        "https://youtube2text.org/api/transcribe",
        params={"url": url, "maxChars": max_chars},
        headers={"x-api-key": API_KEY},
        timeout=90,
    )
    resp.raise_for_status()
    r = resp.json()["result"]
    return Document(
        page_content=r["content"],
        metadata={
            "videoId": r["videoId"],
            "title": r["title"],
            "pubDate": r["pubDate"],
            "source": url,
        },
    )

The API returns videoId, title, pubDate, content, contentSize, and a truncated flag. Transcript text goes into page_content, everything else into metadata — exactly the shape downstream chains and vector stores expect.

No key yet? Grab the shared demo key:

curl https://youtube2text.org/api/demo-key

It's limited to 5 videos per month per IP, which is enough to test the loader.

A summarize chain

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains.combine_documents import create_stuff_documents_chain

llm = ChatOpenAI(model="gpt-4o-mini")
prompt = ChatPromptTemplate.from_template(
    "Summarize this video transcript in five bullet points:\n\n{context}"
)
chain = create_stuff_documents_chain(llm, prompt)

doc = load_youtube_transcript("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(chain.invoke({"context": [doc]}))

create_stuff_documents_chain stuffs the whole transcript into one prompt, which works because maxChars caps the payload at 150,000 characters by default. For hour-long podcasts on small-context models, pass a lower maxChars to the loader or switch to a map-reduce pattern.

What to handle in real code

Two failure modes are worth catching explicitly. A video with captions disabled returns HTTP 404 with error code TRANSCRIPT_UNAVAILABLE — no retry or library swap fixes that, the text simply doesn't exist. And RATE_LIMIT_EXCEEDED (429) includes retryAfterSeconds in the error JSON, so read it and back off instead of hammering:

if resp.status_code == 429:
    wait = resp.json()["error"].get("retryAfterSeconds", 60)

One honest limitation: content is plain text without timestamps. That's fine for summarization and retrieval, but if you need "jump to 12:34" deep links, you'll have to settle for video-level attribution via the videoId in metadata.

Where to go from here

The same Document objects drop straight into a vector store — I walk through the full pipeline in RAG over an entire YouTube channel. If your stack is LlamaIndex rather than LangChain, the equivalent setup is just as short.

When the demo key's 5 videos stop being enough, get your own key — free tier included, paid plans from $5.99 for 50 videos a month.