LlamaIndex YouTube Transcripts: Query a Set of Videos

A LlamaIndex YouTube transcript pipeline is three steps: fetch transcripts as Document objects, build a VectorStoreIndex, ask questions across all the videos at once. Here's the whole thing in one script, plus the one chunking gotcha that ruins most first attempts.

Fetch transcripts as Documents

I run youtube2text.org, a REST API that returns YouTube transcripts as JSON — no scraping from your own IP, which matters the moment this runs anywhere other than your laptop (see why local scrapers get blocked if you're curious). One GET per video:

import requests
from llama_index.core import Document, VectorStoreIndex

API_KEY = "yt_your_key_here"

VIDEOS = [
    "https://www.youtube.com/watch?v=VIDEO_ID_1",
    "https://www.youtube.com/watch?v=VIDEO_ID_2",
    "https://www.youtube.com/watch?v=VIDEO_ID_3",
]

def fetch_transcript(url: str) -> dict:
    resp = requests.get(
        "https://youtube2text.org/api/transcribe",
        params={"url": url},
        headers={"x-api-key": API_KEY},
        timeout=90,
    )
    resp.raise_for_status()
    return resp.json()["result"]

docs = []
for url in VIDEOS:
    r = fetch_transcript(url)
    docs.append(Document(
        text=r["content"],
        metadata={"videoId": r["videoId"], "title": r["title"]},
    ))

For a key, sign in with Google at youtube2text.org/app/keys, or pull the shared demo key from GET https://youtube2text.org/api/demo-key (5 videos/month per IP — fine for this three-video example, not for a channel).

The chunking gotcha: auto-captions have no punctuation

LlamaIndex's default node parser is SentenceSplitter, which finds chunk boundaries at sentence endings. Manually uploaded captions have those. Auto-generated captions frequently don't — you get forty minutes of lowercase text with no periods, SentenceSplitter finds nothing to split on, and you end up with bloated, awkward chunks.

Token-based splitting sidesteps the problem entirely:

from llama_index.core.node_parser import TokenTextSplitter

splitter = TokenTextSplitter(chunk_size=512, chunk_overlap=64)

512 tokens with 64 overlap has been the sweet spot in my testing: big enough that a chunk carries a complete thought, small enough that retrieval stays precise.

Build the index and query it

index = VectorStoreIndex.from_documents(docs, transformations=[splitter])
query_engine = index.as_query_engine(similarity_top_k=4)

response = query_engine.query("What did they say about pricing changes?")
print(response)

for node in response.source_nodes:
    meta = node.metadata
    print(f'- {meta["title"]} (https://www.youtube.com/watch?v={meta["videoId"]})')

Because every Document carries videoId and title in metadata, each retrieved chunk knows which video it came from — source_nodes gives you working links back to the sources. This uses whatever embedding model and LLM you've configured in Settings; the default OpenAI setup works out of the box if OPENAI_API_KEY is set.

The limitation you should know about

The API returns transcripts as plain text without timestamps, so your citations resolve to a video, not a moment inside it. "This claim came from the Q3 earnings breakdown video" is achievable; "at 14:32" is not. For most Q&A use cases video-level attribution is enough, and honestly, chunk text plus a video link gets a human to the right spot in under a minute.

Also note the two error codes worth handling in the fetch loop: TRANSCRIPT_UNAVAILABLE (404, video has no captions — skip it) and RATE_LIMIT_EXCEEDED (429, comes with retryAfterSeconds in the error body).

Scaling past a handful of videos

Three videos is a demo. For a whole channel you want the RSS feed for video discovery, quota-aware fetching, and a persistent vector store — I cover that pipeline in RAG over a YouTube channel. LangChain user instead? Same loader pattern, different framework.

Free tier covers 5 videos a month; the pro plan does 500 for $9.99, which comfortably indexes most channels. Keys at youtube2text.org/app/keys.