fix: drain YouTube backlog by capping downloads per poll #82

Merged

joezombie merged 2 commits from fix/youtube-watcher-download-backlog into main

2026-06-17 20:17:42 +00:00

joezombie commented

2026-06-17 20:11:47 +00:00

(Migrated from github.com)

Problem

The YouTube watcher fetched playlists fine but downloaded no new episodes for months (btb feed: 1141 playlist videos, ~831 backlog, newest file on disk dated January).

Root cause: ProcessFeedAsync fetched full per-video metadata for the entire not-yet-downloaded backlog (hundreds of sequential GetVideoDetailsAsync calls) before the download loop ran. With a large backlog this took 30–60+ minutes, so the poll never reached the download step before the container was restarted (frequent redeploys), and progress was lost every cycle.

Confirmed on the live deployment: a poll spent 15+ min still in the metadata phase (video 331/1141) and reached the download step 0 times.

Fix

Step 2 now builds PlaylistVideoInfo from titles only — no per-video detail fetch. Safe because SyncPlaylistInfoAsync only needs the title to fuzzy-match files and preserves existing metadata when fields are null; full details are still fetched in Step 4 for videos actually downloaded.
SelectNewVideosToDownload<T> caps downloads to MaxDownloadsPerPoll = 5 per cycle, so each poll completes quickly, persists progress incrementally (MarkVideoDownloadedAsync per video), and survives restarts. The backlog drains across polls.

Tests

New unit tests for the selection/cap logic (cap, oldest-first ordering, exclusion of downloaded, fewer-than-cap).
Exposed internals to Castr.Tests via InternalsVisibleTo.
Full suite: 206 passed, 0 failed; build clean.

Tradeoff

At 5/poll on a 60-min interval the ~831 backlog drains over ~1 week. Raise MaxDownloadsPerPoll or trigger polls manually to catch up faster; the per-poll cap is the resilience mechanism.

🤖 Generated with Claude Code

## Problem The YouTube watcher fetched playlists fine but **downloaded no new episodes for months** (btb feed: 1141 playlist videos, ~831 backlog, newest file on disk dated January). **Root cause:** `ProcessFeedAsync` fetched full per-video metadata for the *entire* not-yet-downloaded backlog (hundreds of sequential `GetVideoDetailsAsync` calls) **before** the download loop ran. With a large backlog this took 30–60+ minutes, so the poll never reached the download step before the container was restarted (frequent redeploys), and progress was lost every cycle. Confirmed on the live deployment: a poll spent 15+ min still in the metadata phase (video 331/1141) and reached the download step 0 times. ## Fix - **Step 2** now builds `PlaylistVideoInfo` from titles only — no per-video detail fetch. Safe because `SyncPlaylistInfoAsync` only needs the title to fuzzy-match files and preserves existing metadata when fields are null; full details are still fetched in Step 4 for videos actually downloaded. - **`SelectNewVideosToDownload<T>`** caps downloads to `MaxDownloadsPerPoll = 5` per cycle, so each poll completes quickly, persists progress incrementally (`MarkVideoDownloadedAsync` per video), and survives restarts. The backlog drains across polls. ## Tests - New unit tests for the selection/cap logic (cap, oldest-first ordering, exclusion of downloaded, fewer-than-cap). - Exposed internals to `Castr.Tests` via `InternalsVisibleTo`. - Full suite: **206 passed, 0 failed**; build clean. ## Tradeoff At 5/poll on a 60-min interval the ~831 backlog drains over ~1 week. Raise `MaxDownloadsPerPoll` or trigger polls manually to catch up faster; the per-poll cap is the resilience mechanism. 🤖 Generated with [Claude Code](https://claude.com/claude-code)