How transcripts and translations are made
The pipeline runs on the Yakopop server as queued jobs, not in the browser. Each stage is swappable: a provider registry lets a stage move to another service without touching the build. This is the real flow in apps/server/src/pipeline/build.ts.
1. Get the words available planned
Available: the creator's or auto-generated YouTube captions, fetched with yt-dlp without downloading the video, or a .vtt, .srt or TTML file the creator uploads. The browser extension instead reads the text tracks the streaming page already loads (DASH and HLS manifests, WebVTT and TTML responses, or video.textTracks).
Planned: speech-to-text for a video with no captions. The hook exists (transcribe stage, an OpenAI-compatible POST …/audio/transcriptions with response_format=verbose_json, reference server tools/stt/server.py running faster-whisper) but the pipeline does not use it yet, and video file upload is listed under "not yet".
Source: apps/server/src/pipeline/captions.ts, providers.ts, docs/plans/extension.md, README.md
2. Cues available
Whatever the source, the result is a list of timed text pieces. The builder groups caption text into chunks of about 200 characters with a per-character time map, so annotated sentences can be timed afterwards. Sound notes in SDH captions are dropped. The extension cuts a track into windows of about five minutes, each cached under a hash of language, target, annotator and text, so an episode is annotated once.
// packages/shared/src/subtitles.ts
{ "start": 0.8, "end": 5.7, "text": "今日は天気がいいですね" }
Source: packages/shared/src/subtitles.ts (Piece, parseSubtitles, splitWindows), pipeline/captions.ts
3. Annotate: words, readings, meanings, grammar available
An annotator turns caption text into sentence-sized lines. Each line is a list of tokens [surface, reading, romanisation, gloss, grammar] and a translation made of chunks that point at token numbers. Three annotators exist:
Claude (a language model with structured JSON output and per-language rules for the reading and romanisation systems; best quality; may be limited to admins on a server). A free local annotator for Japanese: kuromoji for splitting, readings and parts of speech, wanakana for kana and rōmaji, Jisho's public API for meanings, with rougher results and a label saying so. An external annotation service that speaks Yakopop's contract: POST {url} with { lang, target, pieces, options }, answering { lines: [{ start, end, t, en }] }.
The rules the model follows are visible in the source: the tokens must add up to the caption text with nothing invented or dropped, a gloss is one to three words, particles with no equivalent get a role tag (#TOPIC, #SUBJ, #OBJ) instead of a translation, and every translated chunk carries the index of the source word it translates, or -1.
Source: apps/server/src/pipeline/annotate.ts (systemPrompt, LANG_RULES), local-ja.ts, providers.ts
4. Translate available
The annotator writes the translation into the creator's chosen target language as part of annotation, with every chunk linked to a word. Extra output languages travel on the same line as line.tr (a translation and one gloss per token, per language). Where an annotator does not translate on its own (the free local one), a sentence translator is used: a LibreTranslate-compatible service you configure, or MyMemory's free API (about 5,000 characters a day). A track in a dub's language starts from the captions in that language, or from the original captions machine-translated piece by piece, timings kept.
Source: pipeline/providers.ts (translators), build.ts (translatePieces), packages/shared/src/text.ts (translationOf)
5. Time the words available planned
Today: timing is per line, taken from the caption cues (timing: "line"). The words inside a line are spread across it by weight, the length of their romanisation plus a constant, on the client (wordAt, wordTime). That is what lights the words in the demos above. Planned: per-word forced alignment. The speech-to-text server can already return word times (timestamp_granularities[]=word), and the film voice tool uses them.
Source: pipeline/build.ts ("word timing is interpolated on the client for now"), packages/shared/src/text.ts, tools/stt/server.py
6. Save and play available
The track is saved as a new version of that language's lesson, the player fetches it as JSON, and the extension asks the server for windows. A creator can edit lines afterwards in the studio, with revisions and restore.
GET /api/lessons/:id
POST /api/overlay/windows { lang, target, site, pieces } → key
GET /api/overlay/windows/:key → status or lines
GET /api/translation/providers → which stages are configured
GET /api/dict/:lang/:word → dictionary entry for a word card
Source: apps/server/src/routes/{lessons,overlay,system,studio}.ts. The window cache returns lines only for a key the client could compute, so it never hands subtitles to someone who does not hold them.
The data shapes
A line, as the repository defines it (Line, Token, Segment in packages/shared/src/types.ts). This is the first Japanese sample sentence with the times a lesson line would carry. start and end are absent on the shared demo sentences.
{
"start": 0.8, "end": 5.7,
"t": [
["今日", "きょう", "kyō", "today", "noun · time"],
["は", "は", "wa", "#TOPIC", "particle"],
["天気", "てんき", "tenki", "weather", "noun"],
["が", "が", "ga", "#SUBJ", "particle"],
["いい", "いい", "ii", "good", "i-adjective"],
["ですね", "ですね", "desu ne", "isn't it", "copula + particle"]
],
"en": ["The ", ["weather", 2], " is ", ["nice", 4], " ", ["today", 0], ", ", ["isn't it", 5], "?"],
"tr": {
"es": {
"line": [["Hoy", 0], " hace ", ["buen", 4], " ", ["tiempo", 2], ", ", ["¿verdad?", 5]],
"gloss": ["hoy", "#TOPIC", "tiempo", "#SUBJ", "bueno", "¿verdad?"]
}
}
}
A lesson wraps lines with: id, lang, videoLang, target, title, source (YouTube id or upload), captions (manual, auto, uploaded), annotatedBy (claude, hand, local, external), timing (line or word), audio dubs and the sibling subtitles tracks.
What the speech-to-text server answers, when asked for word times:
{ "task": "transcribe", "language": "ja", "duration": 0, "text": "…",
"segments": [{ "id": 0, "start": 0, "end": 0, "text": "…",
"words": [{ "word": "…", "start": 0, "end": 0, "probability": 0 }] }],
"words": [ … ] }
Source: tools/stt/server.py. Numbers are placeholders; the keys are the server's.
Try it yourself
The same flow from the command line, on a machine with the repository running. Only fetch captions for videos you have the rights to use.
npm run build-lesson -w @yakopop/server -- <video url> --lang ja --to en --max-minutes 3
# the lesson is queued as a job; the worker fetches captions, annotates, translates and saves
npm run prewarm-overlay -w @yakopop/server -- path/to/file.srt --lang ja --to en --site demo
Source: README.md "Video lessons (beta)", apps/extension/README.md. Demo 4 above shows the shape of the result with sample data.
