YouTube-Kanäle als Podcasts abonnieren
Eine YouTube-Adresse (Kanal, Playlist, Video, Feed) lässt sich wie ein Podcast abonnieren und in der Suche einfügen. Der Server liest dafür nur die Kanalseite (einmal) und danach den RSS-Feed des Kanals; Ton oder Video holt er nie: Die Adresse einer Folge ist die Videoseite (watch?v=…), und jede App sucht sich den reinen Audio-Stream selbst (docs/youtube.md). Shorts bleiben draußen, Stille-Analyse und ID3-Kapitel gibt es für solche Folgen nicht. Der Player im Browser verweist auf die Apps; die Desktop-App fragt ihren Hauptprozess (yt-dlp) nach dem Stream. Eine Dauer, die der Feed nicht hat, meldet der Player mit dem Fortschritt (progress/save: duration_seconds). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
1d6ddabb7b
commit
cde90bed8e
6 changed files with 504 additions and 8 deletions
46
docs/youtube.md
Normal file
46
docs/youtube.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# YouTube channels as podcasts
|
||||
|
||||
A YouTube channel (or playlist) can be subscribed to like a podcast. Only its
|
||||
sound is played, and **the server never downloads or streams a video or its
|
||||
sound**: each app finds the audio stream itself.
|
||||
|
||||
## Server (diora-web, `podcasts/youtube.py`)
|
||||
|
||||
- `POST /podcasts/feeds/add/` takes a YouTube address as `rss_url`: a channel
|
||||
(`/@name`, `/channel/UC…`, `/c/…`, `/user/…`), a playlist (`?list=…`), a video
|
||||
(subscribes its channel) or the feed itself. The server reads the channel page
|
||||
once (feed address, name, picture) and stores the feed
|
||||
`https://www.youtube.com/feeds/videos.xml?channel_id=UC…` (or `?playlist_id=…`).
|
||||
`GET /podcasts/search/?q=<YouTube address>` answers with that one channel.
|
||||
- From then on the feed is refreshed like any other. It carries the latest 15
|
||||
videos. **Shorts are left out.** There is no duration (`duration_seconds` 0)
|
||||
until a player knows it: `POST /podcasts/progress/save/` takes an optional
|
||||
`duration_seconds` (number, 1 s … 7 days), stored only while the episode has none.
|
||||
- An episode from such a feed has
|
||||
- `audio_url` = `https://www.youtube.com/watch?v=<11-character id>`. This is
|
||||
**the** marker: a client recognises a YouTube episode by this prefix;
|
||||
- `guid` = `yt:video:<id>`, `artwork_url` = the video's thumbnail,
|
||||
`description` = the video description as HTML (links, line breaks; timestamps
|
||||
like `00:00 Intro` work as usual).
|
||||
- Nothing on the server touches the sound of such an episode: no ID3 chapters,
|
||||
no transcripts, and `/podcasts/episodes/<id>/silences/` answers
|
||||
`{"ready": true, "spans": [], "error": "youtube"}` (skip silence is up to the
|
||||
app, as on Android anyway).
|
||||
|
||||
## Apps
|
||||
|
||||
A client that sees a `watch?v=` address resolves it **itself, on the device,
|
||||
to an audio-only stream** (no video track), and plays or downloads that.
|
||||
|
||||
| Client | How |
|
||||
|---------|-----|
|
||||
| Android | NewPipeExtractor: the original audio track, progressive, opus before m4a, ≤ 160 kbit/s; resolved when the stream is opened and once more on a 403; downloads in 10 MB ranges (in one go YouTube throttles to ~50 KB/s) |
|
||||
| Desktop | yt-dlp in the main process (`-f bestaudio`), Electron itself as its JavaScript runtime; yt-dlp is fetched from its GitHub release on first use and kept up to date (`src/main/youtube.js`) |
|
||||
| Browser | can't (YouTube doesn't let a web page read its pages, and a stream address only works for the device that asked for it): the episode is listed but says it plays in the app |
|
||||
|
||||
Stream addresses are bound to the device's IP address and run out after a few
|
||||
hours: resolve shortly before playing, and again when the player reports an
|
||||
error (podcasts-web.js reloads after a media error anyway).
|
||||
|
||||
Progress, queue, played state etc. work as for any episode: they belong to the
|
||||
episode on the server, not to the stream address.
|
||||
194
podcasts/test_youtube.py
Normal file
194
podcasts/test_youtube.py
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
import json
|
||||
from unittest.mock import patch
|
||||
|
||||
import feedparser
|
||||
from django.contrib.auth.models import User
|
||||
from django.test import TestCase
|
||||
|
||||
from . import youtube
|
||||
from .models import PodcastEpisode, PodcastFeed
|
||||
from .views import FeedFetch, _load_chapters
|
||||
|
||||
CHANNEL = 'UC5NOEUbkLheQcaaRldYW5GA'
|
||||
FEATURED = 'UC1_uAIS3r8Vu6JjXWvastJg' # another channel the page shows
|
||||
|
||||
PAGE = (
|
||||
f'<html><script>{{"channelId":"{FEATURED}"}}</script><head><meta property="og:title" content="Tages & Schau">'
|
||||
'<meta property="og:image" content="https://yt3.example/pic.jpg">'
|
||||
f'<link rel="alternate" type="application/rss+xml" title="RSS" href="{youtube.FEED}?channel_id={CHANNEL}">'
|
||||
f'</head><body><script>var x = {{"externalId":"{CHANNEL}"}};</script></body></html>'
|
||||
)
|
||||
|
||||
|
||||
def _entry(vid, title, kind='watch?v=', when='2026-10-14T10:00:00+00:00', text='Hallo\nhttps://example.org/x\n\n00:00 Anfang'):
|
||||
link = f'https://www.youtube.com/{kind}{vid}'
|
||||
return (
|
||||
f'<entry><id>yt:video:{vid}</id><yt:videoId>{vid}</yt:videoId><yt:channelId>{CHANNEL}</yt:channelId>'
|
||||
f'<title>{title}</title><link rel="alternate" href="{link}"/>'
|
||||
f'<author><name>Tages Schau</name></author><published>{when}</published><updated>{when}</updated>'
|
||||
f'<media:group><media:title>{title}</media:title>'
|
||||
f'<media:content url="https://www.youtube.com/v/{vid}?version=3" type="application/x-shockwave-flash" width="640" height="390"/>'
|
||||
f'<media:thumbnail url="https://i1.ytimg.com/vi/{vid}/hqdefault.jpg" width="480" height="360"/>'
|
||||
f'<media:description>{text}</media:description></media:group></entry>'
|
||||
)
|
||||
|
||||
|
||||
FEED_XML = (
|
||||
'<?xml version="1.0" encoding="UTF-8"?>'
|
||||
'<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015" xmlns:media="http://search.yahoo.com/mrss/" '
|
||||
'xmlns="http://www.w3.org/2005/Atom">'
|
||||
f'<link rel="self" href="{youtube.FEED}?channel_id={CHANNEL}"/><id>yt:channel:{CHANNEL[2:]}</id>'
|
||||
f'<yt:channelId>{CHANNEL[2:]}</yt:channelId><title>Tages Schau</title>'
|
||||
f'<link rel="alternate" href="https://www.youtube.com/channel/{CHANNEL}"/>'
|
||||
'<author><name>Tages Schau</name></author><published>2010-01-01T00:00:00+00:00</published>'
|
||||
+ _entry('aaaaaaaaaaa', 'Video eins')
|
||||
+ _entry('bbbbbbbbbbb', 'Ein Short', kind='shorts/')
|
||||
+ _entry('ccccccccccc', 'Video zwei', when='2026-10-13T10:00:00+00:00')
|
||||
+ '</feed>'
|
||||
)
|
||||
|
||||
|
||||
class _Page:
|
||||
def __init__(self, text, status=200):
|
||||
self.status_code = status
|
||||
self.raw = self
|
||||
self._body = text.encode()
|
||||
|
||||
def read(self, n, decode_content=True):
|
||||
return self._body[:n]
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
|
||||
class ResolveTests(TestCase):
|
||||
def resolve(self, url, page=PAGE):
|
||||
seen = []
|
||||
|
||||
def get(u, **kw):
|
||||
seen.append(u)
|
||||
return _Page(page)
|
||||
with patch('podcasts.youtube.requests.get', get):
|
||||
return youtube.resolve(url), seen
|
||||
|
||||
def test_a_channel_page_is_its_own_channel_not_one_it_features(self):
|
||||
page = (f'<script>{{"channelId":"{FEATURED}"}}</script>'
|
||||
f'<link rel="canonical" href="https://www.youtube.com/channel/{CHANNEL}">')
|
||||
found, _ = self.resolve('https://www.youtube.com/@3blue1brown', page)
|
||||
self.assertEqual(found['feed'], f'{youtube.FEED}?channel_id={CHANNEL}')
|
||||
|
||||
def test_a_handle_leads_to_its_channel_feed(self):
|
||||
found, seen = self.resolve('https://www.youtube.com/@tagesschau/videos')
|
||||
self.assertEqual(found, {'feed': f'{youtube.FEED}?channel_id={CHANNEL}', 'title': 'Tages & Schau',
|
||||
'artwork': 'https://yt3.example/pic.jpg'})
|
||||
self.assertEqual(seen, ['https://www.youtube.com/@tagesschau/videos'])
|
||||
|
||||
def test_a_video_leads_to_its_channel_without_the_videos_name(self):
|
||||
page = (f'<meta property="og:title" content="Ein Video"><script>{{"channelId":"{FEATURED}"}};'
|
||||
f'{{"videoDetails":{{"videoId":"aaaaaaaaaaa","channelId":"{CHANNEL}"}}}}</script>')
|
||||
found, seen = self.resolve('https://youtu.be/aaaaaaaaaaa', page)
|
||||
self.assertEqual(found['feed'], f'{youtube.FEED}?channel_id={CHANNEL}')
|
||||
self.assertEqual((found['title'], found['artwork']), ('', ''))
|
||||
self.assertEqual(seen, [youtube.WATCH + 'aaaaaaaaaaa'])
|
||||
|
||||
def test_a_playlist_has_its_own_feed(self):
|
||||
found, _ = self.resolve('https://www.youtube.com/playlist?list=PLabcdefghijk')
|
||||
self.assertEqual(found['feed'], f'{youtube.FEED}?playlist_id=PLabcdefghijk')
|
||||
|
||||
def test_other_youtube_pages_are_no_channel(self):
|
||||
with self.assertRaises(youtube.NotAChannel):
|
||||
youtube.resolve('https://www.youtube.com/feed/subscriptions')
|
||||
|
||||
def test_a_page_without_a_channel_is_none(self):
|
||||
with self.assertRaises(youtube.NotAChannel):
|
||||
self.resolve('https://www.youtube.com/@nobody', '<html>consent</html>')
|
||||
|
||||
|
||||
class YouTubeFeedTests(TestCase):
|
||||
def setUp(self):
|
||||
self.user = User.objects.create_user(username='alice', password='pw12345678')
|
||||
self.client.force_login(self.user)
|
||||
|
||||
def subscribe(self, url):
|
||||
pages = []
|
||||
|
||||
def get(u, **kw):
|
||||
pages.append(u)
|
||||
return _Page(PAGE)
|
||||
fetched = []
|
||||
|
||||
def fetch(u, **kw):
|
||||
fetched.append(u)
|
||||
return FeedFetch(200, feedparser.parse(FEED_XML))
|
||||
with patch('podcasts.youtube.requests.get', get), patch('podcasts.views._fetch_feed', fetch):
|
||||
r = self.client.post('/podcasts/feeds/add/', json.dumps({'rss_url': url, 'title': url}),
|
||||
content_type='application/json')
|
||||
return r, pages, fetched
|
||||
|
||||
def test_subscribing_to_a_channel_reads_its_feed_never_a_video(self):
|
||||
r, pages, fetched = self.subscribe('https://www.youtube.com/@tagesschau')
|
||||
self.assertEqual(r.status_code, 200, r.content)
|
||||
feed = PodcastFeed.objects.get()
|
||||
self.assertEqual(feed.rss_url, f'{youtube.FEED}?channel_id={CHANNEL}')
|
||||
self.assertEqual(feed.title, 'Tages Schau')
|
||||
self.assertEqual(feed.artwork_url, 'https://yt3.example/pic.jpg') # kept: the feed has none
|
||||
self.assertEqual(pages, ['https://www.youtube.com/@tagesschau'])
|
||||
self.assertEqual(fetched, [feed.rss_url])
|
||||
eps = {e.title: e for e in PodcastEpisode.objects.all()}
|
||||
self.assertEqual(set(eps), {'Video eins', 'Video zwei'}) # no Short
|
||||
one = eps['Video eins']
|
||||
self.assertEqual(one.audio_url, 'https://www.youtube.com/watch?v=aaaaaaaaaaa')
|
||||
self.assertEqual(one.guid, 'yt:video:aaaaaaaaaaa')
|
||||
self.assertEqual(one.artwork_url, 'https://i1.ytimg.com/vi/aaaaaaaaaaa/hqdefault.jpg')
|
||||
self.assertEqual(one.description,
|
||||
'<p>Hallo<br><a href="https://example.org/x">https://example.org/x</a></p><p>00:00 Anfang</p>')
|
||||
|
||||
def test_the_same_channel_twice_is_one_subscription(self):
|
||||
self.subscribe('https://www.youtube.com/@tagesschau')
|
||||
r, _, _ = self.subscribe(f'https://www.youtube.com/channel/{CHANNEL}')
|
||||
self.assertFalse(r.json()['created'])
|
||||
self.assertEqual(PodcastFeed.objects.count(), 1)
|
||||
|
||||
def test_a_feed_subscribed_directly_gets_the_channels_picture(self):
|
||||
r, pages, _ = self.subscribe(f'{youtube.FEED}?channel_id={CHANNEL}')
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.assertEqual(PodcastFeed.objects.get().artwork_url, 'https://yt3.example/pic.jpg')
|
||||
self.assertEqual(pages, [f'https://www.youtube.com/channel/{CHANNEL}'])
|
||||
|
||||
def test_no_channel_there_is_an_error(self):
|
||||
with patch('podcasts.youtube.requests.get', lambda u, **kw: _Page('<html></html>')):
|
||||
r = self.client.post('/podcasts/feeds/add/', json.dumps({'rss_url': 'https://www.youtube.com/@x'}),
|
||||
content_type='application/json')
|
||||
self.assertEqual(r.status_code, 400)
|
||||
self.assertFalse(PodcastFeed.objects.exists())
|
||||
|
||||
def test_search_finds_a_pasted_channel(self):
|
||||
with patch('podcasts.youtube.requests.get', lambda u, **kw: _Page(PAGE)):
|
||||
r = self.client.get('/podcasts/search/', {'q': 'https://www.youtube.com/@tagesschau'})
|
||||
self.assertEqual(r.json()['results'][0]['rss_url'], f'{youtube.FEED}?channel_id={CHANNEL}')
|
||||
self.assertEqual(r.json()['results'][0]['title'], 'Tages & Schau')
|
||||
|
||||
def test_the_server_never_fetches_a_videos_sound(self):
|
||||
self.subscribe('https://www.youtube.com/@tagesschau')
|
||||
ep = PodcastEpisode.objects.get(title='Video eins')
|
||||
with patch('podcasts.silence.ensure') as ensure:
|
||||
d = self.client.get(f'/podcasts/episodes/{ep.pk}/silences/').json()
|
||||
ensure.assert_not_called()
|
||||
self.assertEqual((d['ready'], d['spans'], d['error']), (True, [], 'youtube'))
|
||||
with patch('podcasts.chapters.fetch_id3') as id3:
|
||||
self.assertEqual(_load_chapters(ep), [])
|
||||
id3.assert_not_called()
|
||||
|
||||
def test_the_players_duration_is_kept_when_the_feed_has_none(self):
|
||||
self.subscribe('https://www.youtube.com/@tagesschau')
|
||||
ep = PodcastEpisode.objects.get(title='Video eins')
|
||||
save = lambda body: self.client.post('/podcasts/progress/save/', json.dumps({'episode_id': ep.pk, **body}),
|
||||
content_type='application/json')
|
||||
save({'position_seconds': 10, 'duration_seconds': 'x'})
|
||||
save({'position_seconds': 10, 'duration_seconds': 10 ** 9})
|
||||
ep.refresh_from_db()
|
||||
self.assertEqual(ep.duration_seconds, 0)
|
||||
save({'position_seconds': 10, 'duration_seconds': 929.3})
|
||||
save({'position_seconds': 20, 'duration_seconds': 5}) # known now: not changed again
|
||||
ep.refresh_from_db()
|
||||
self.assertEqual(ep.duration_seconds, 929)
|
||||
|
|
@ -28,6 +28,7 @@ from . import chapters as chapters_mod
|
|||
from . import silence as silence_mod
|
||||
from . import feedext
|
||||
from . import transcripts as transcripts_mod
|
||||
from . import youtube as youtube_mod
|
||||
from .models import (
|
||||
EpisodeProgress, EpisodeTranscript, PlayerDevice, PlayerState, PodcastEpisode, PodcastFeed, PodcastQueue,
|
||||
MAX_SPEED, MIN_SPEED, PodcastSettings, counts_as_played,
|
||||
|
|
@ -44,6 +45,7 @@ FEED_WORKERS = 16 # feeds downloaded at the same time
|
|||
REFRESH_WAIT = 40 # seconds a refresh-all (⟳) waits; feeds not done by then wait for the hourly job
|
||||
MAX_DESCRIPTION = 50_000 # characters of shownotes HTML kept per episode
|
||||
MAX_TRANSCRIPTS = 10
|
||||
MAX_DURATION = 7 * 24 * 3600 # seconds: a longer one reported is nonsense
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -537,7 +539,8 @@ def _entry_media_url(entry):
|
|||
return url
|
||||
if not video and (mime.startswith('video/') or path.endswith(VIDEO_EXTENSIONS)):
|
||||
video = url
|
||||
return video
|
||||
# A YouTube feed has no files: the video's page, whose sound the app finds itself.
|
||||
return video or youtube_mod.video_url(entry)
|
||||
|
||||
|
||||
def _apply_feed(feed_obj, fetched, full=False):
|
||||
|
|
@ -569,7 +572,10 @@ def _apply_feed(feed_obj, fetched, full=False):
|
|||
channel.get('itunes_image', {}).get('href', '')
|
||||
or image.get('href', '')
|
||||
or image.get('url', '')
|
||||
or feed_obj.artwork_url # a YouTube feed has none: the channel's, found when subscribing
|
||||
)[:1000]
|
||||
if not feed_obj.artwork_url and youtube_mod.feed_channel_id(channel):
|
||||
feed_obj.artwork_url = youtube_mod.channel_artwork(youtube_mod.feed_channel_id(channel))[:1000]
|
||||
|
||||
feed_obj.last_refreshed_at = now
|
||||
feed_obj.save()
|
||||
|
|
@ -596,7 +602,8 @@ def _apply_feed(feed_obj, fetched, full=False):
|
|||
guid = entry.get('id') or entry.get('guid') or audio_url
|
||||
guid = str(guid)[:1000]
|
||||
title = entry.get('title', 'Untitled')[:500]
|
||||
description = _entry_description(entry)
|
||||
video = youtube_mod.is_video(audio_url)
|
||||
description = youtube_mod.description_html(entry.get('summary')) if video else _entry_description(entry)
|
||||
chapters_url = _entry_chapters_url(entry)
|
||||
psc_chapters = chapters_mod.from_psc(entry)
|
||||
transcripts = _entry_transcripts(entry)
|
||||
|
|
@ -628,7 +635,7 @@ def _apply_feed(feed_obj, fetched, full=False):
|
|||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
artwork_url = entry.get('itunes_image', {}).get('href', '')[:1000]
|
||||
artwork_url = (youtube_mod.entry_artwork(entry) if video else entry.get('itunes_image', {}).get('href', ''))[:1000]
|
||||
|
||||
episode_obj, created = PodcastEpisode.objects.get_or_create(
|
||||
feed=feed_obj,
|
||||
|
|
@ -723,6 +730,15 @@ def podcast_search(request):
|
|||
q = request.GET.get('q', '').strip()
|
||||
if not q:
|
||||
return JsonResponse({'results': []})
|
||||
if youtube_mod.is_youtube(q):
|
||||
try:
|
||||
found = youtube_mod.resolve(q)
|
||||
except youtube_mod.NotAChannel:
|
||||
return JsonResponse({'results': []})
|
||||
return JsonResponse({'results': [{
|
||||
'id': found['feed'], 'title': found['title'] or 'YouTube', 'author': 'YouTube',
|
||||
'artwork_url': found['artwork'], 'rss_url': found['feed'], 'genre': 'YouTube',
|
||||
}]})
|
||||
|
||||
try:
|
||||
url = f'https://itunes.apple.com/search?term={urllib.parse.quote(q)}&media=podcast&limit=20'
|
||||
|
|
@ -830,11 +846,22 @@ def add_feed(request):
|
|||
rss_url = body.get('rss_url', '').strip()
|
||||
if not rss_url:
|
||||
return JsonResponse({'error': 'rss_url required'}, status=400)
|
||||
title = body.get('title', rss_url)
|
||||
artwork = ''
|
||||
if youtube_mod.is_youtube(rss_url) and not youtube_mod.is_feed(rss_url):
|
||||
# A channel, playlist or video address: its channel's feed (the server reads only that).
|
||||
try:
|
||||
found = youtube_mod.resolve(rss_url)
|
||||
except youtube_mod.NotAChannel as e:
|
||||
return JsonResponse({'error': f'No YouTube channel there ({e})'}, status=400)
|
||||
rss_url, artwork = found['feed'], found['artwork']
|
||||
title = found['title'] or title
|
||||
|
||||
feed, created = PodcastFeed.objects.get_or_create(
|
||||
user=request.user,
|
||||
rss_url=rss_url,
|
||||
defaults={'title': body.get('title', rss_url)[:300], **PodcastSettings.new_feed_defaults(request.user)},
|
||||
defaults={'title': title[:300], 'artwork_url': artwork[:1000],
|
||||
**PodcastSettings.new_feed_defaults(request.user)},
|
||||
)
|
||||
|
||||
new_episodes = 0
|
||||
|
|
@ -1101,7 +1128,7 @@ def _load_chapters(episode):
|
|||
chapters = chapters_mod.fetch(url)
|
||||
except chapters_mod.ChaptersInvalid as e:
|
||||
log.info('chapters of episode %s unusable: %s', episode.id, e)
|
||||
if not chapters and episode.audio_url:
|
||||
if not chapters and episode.audio_url and not youtube_mod.is_video(episode.audio_url):
|
||||
try:
|
||||
chapters = chapters_mod.fetch_id3(episode.audio_url)
|
||||
except chapters_mod.ChaptersInvalid as e:
|
||||
|
|
@ -1179,9 +1206,11 @@ def episode_silences(request, pk):
|
|||
on the server ({"ready": false}, ask again later)."""
|
||||
if not request.user.is_authenticated:
|
||||
return JsonResponse({'error': 'authentication required'}, status=401)
|
||||
ep = PodcastEpisode.objects.filter(pk=pk, feed__user=request.user).only('id', 'silences').first()
|
||||
ep = PodcastEpisode.objects.filter(pk=pk, feed__user=request.user).only('id', 'silences', 'audio_url').first()
|
||||
if ep is None:
|
||||
return JsonResponse({'error': 'not found'}, status=404)
|
||||
if youtube_mod.is_video(ep.audio_url): # the server never fetches a video's sound
|
||||
return JsonResponse({'ready': True, 'v': silence_mod.VERSION, 'spans': [], 'error': 'youtube'})
|
||||
if not silence_mod.current(ep.silences): # none yet, or found by older rules
|
||||
silence_mod.ensure(ep.pk)
|
||||
return JsonResponse({'ready': False})
|
||||
|
|
@ -1917,6 +1946,12 @@ def save_progress(request):
|
|||
except PodcastEpisode.DoesNotExist:
|
||||
return JsonResponse({'error': 'not found'}, status=404)
|
||||
|
||||
# A feed without durations (YouTube): the one the player found, once.
|
||||
duration = body.get('duration_seconds')
|
||||
if not episode.duration_seconds and _number(duration) and math.isfinite(duration) and 1 <= duration <= MAX_DURATION:
|
||||
episode.duration_seconds = int(duration)
|
||||
PodcastEpisode.objects.filter(pk=episode.pk, duration_seconds=0).update(duration_seconds=int(duration))
|
||||
|
||||
progress, _ = EpisodeProgress.objects.get_or_create(
|
||||
user=request.user,
|
||||
episode=episode,
|
||||
|
|
|
|||
176
podcasts/youtube.py
Normal file
176
podcasts/youtube.py
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
"""YouTube channels (and playlists) as podcasts.
|
||||
|
||||
The server only reads what YouTube publishes about a channel: its page once,
|
||||
when it's subscribed (feed address, name, picture), and from then on its RSS
|
||||
feed like any podcast's. It never fetches a video or its sound: an episode's
|
||||
address is the video's page (https://www.youtube.com/watch?v=…), and each app
|
||||
finds the audio stream for it itself and plays only that (docs/youtube.md).
|
||||
Hence no skipping of silence and no ID3 chapters found on the server for them.
|
||||
|
||||
Shorts are left out: a minute of sound cut from a vertical video isn't an
|
||||
episode.
|
||||
"""
|
||||
import html
|
||||
import re
|
||||
import urllib.parse
|
||||
|
||||
import requests
|
||||
|
||||
WATCH = 'https://www.youtube.com/watch?v='
|
||||
FEED = 'https://www.youtube.com/feeds/videos.xml'
|
||||
HOSTS = {'youtube.com', 'www.youtube.com', 'm.youtube.com', 'music.youtube.com', 'youtu.be'}
|
||||
TIMEOUT = 15
|
||||
MAX_PAGE = 4 * 1024 * 1024
|
||||
# Without this, a server in the EU gets the cookie consent page instead (SOCS=CAI: declined).
|
||||
COOKIES = {'SOCS': 'CAI'}
|
||||
HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0',
|
||||
'Accept-Language': 'de,en;q=0.8'}
|
||||
|
||||
_CHANNEL_ID = re.compile(r'^UC[\w-]{22}$')
|
||||
_VIDEO_ID = re.compile(r'^[\w-]{11}$')
|
||||
|
||||
|
||||
# Where a page names its own channel, surest first: a channel page in its head, a video's in
|
||||
# its videoDetails.
|
||||
_OWN_CHANNEL = (
|
||||
r'<link rel="canonical" href="https://www\.youtube\.com/channel/(UC[\w-]{22})"',
|
||||
r'type="application/rss\+xml" title="RSS" href="[^"]*channel_id=(UC[\w-]{22})"',
|
||||
r'"externalId":"(UC[\w-]{22})"',
|
||||
r'<meta itemprop="identifier" content="(UC[\w-]{22})"',
|
||||
r'"videoDetails":\{[^{}]*?"channelId":"(UC[\w-]{22})"',
|
||||
)
|
||||
|
||||
|
||||
class NotAChannel(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _host(url):
|
||||
try:
|
||||
return (urllib.parse.urlsplit(url).hostname or '').lower()
|
||||
except ValueError:
|
||||
return ''
|
||||
|
||||
|
||||
def is_youtube(url):
|
||||
"""An address on YouTube (a channel, a video, a playlist, a feed)."""
|
||||
return _host(url) in HOSTS
|
||||
|
||||
|
||||
def is_feed(url):
|
||||
return is_youtube(url) and urllib.parse.urlsplit(url).path == '/feeds/videos.xml'
|
||||
|
||||
|
||||
def is_video(audio_url):
|
||||
"""An episode that is a YouTube video (its sound is found by the app)."""
|
||||
return audio_url.startswith(WATCH)
|
||||
|
||||
|
||||
def channel_feed(channel_id):
|
||||
return f'{FEED}?channel_id={channel_id}'
|
||||
|
||||
|
||||
def video_url(entry):
|
||||
"""The episode address for a YouTube feed entry, '' for a Short (or not a video)."""
|
||||
vid = entry.get('yt_videoid') or ''
|
||||
if not _VIDEO_ID.match(vid) or '/shorts/' in (entry.get('link') or ''):
|
||||
return ''
|
||||
return WATCH + vid
|
||||
|
||||
|
||||
def entry_artwork(entry):
|
||||
thumbs = entry.get('media_thumbnail') or []
|
||||
return (thumbs[0].get('url') or '') if thumbs else ''
|
||||
|
||||
|
||||
_LINK = re.compile(r'(https?://[^\s<>"]+)')
|
||||
|
||||
|
||||
def description_html(text):
|
||||
"""A video description (plain text) as shownotes: links clickable, lines kept."""
|
||||
escaped = html.escape(text or '', quote=False)
|
||||
linked = _LINK.sub(lambda m: f'<a href="{m.group(1)}">{m.group(1)}</a>', escaped)
|
||||
return '<p>' + linked.strip().replace('\n\n', '</p><p>').replace('\n', '<br>') + '</p>' if linked.strip() else ''
|
||||
|
||||
|
||||
def _page(url):
|
||||
r = requests.get(url, headers=HEADERS, cookies=COOKIES, timeout=TIMEOUT, stream=True)
|
||||
try:
|
||||
if r.status_code >= 400:
|
||||
raise NotAChannel(f'HTTP {r.status_code}')
|
||||
body = r.raw.read(MAX_PAGE, decode_content=True)
|
||||
finally:
|
||||
r.close()
|
||||
return body.decode('utf-8', 'replace')
|
||||
|
||||
|
||||
def _meta(page, prop):
|
||||
m = re.search(r'<meta property="%s" content="([^"]*)"' % re.escape(prop), page)
|
||||
return html.unescape(m.group(1)) if m else ''
|
||||
|
||||
|
||||
def resolve(url):
|
||||
"""{'feed', 'title', 'artwork'} for a YouTube address someone wants to subscribe to:
|
||||
a channel (/@name, /channel/…, /c/…, /user/…), a playlist, or a video (its channel).
|
||||
Raises NotAChannel when it leads to none."""
|
||||
parts = urllib.parse.urlsplit(url.strip())
|
||||
query = urllib.parse.parse_qs(parts.query)
|
||||
path = parts.path.rstrip('/')
|
||||
if path == '/feeds/videos.xml':
|
||||
return {'feed': url.strip(), 'title': '', 'artwork': ''}
|
||||
if 'list' in query and path in ('/playlist', '/watch'):
|
||||
playlist = query['list'][0]
|
||||
if not re.match(r'^[\w-]{10,64}$', playlist):
|
||||
raise NotAChannel('playlist id')
|
||||
page = ''
|
||||
try:
|
||||
page = _page(f'https://www.youtube.com/playlist?list={playlist}')
|
||||
except (requests.RequestException, NotAChannel):
|
||||
pass
|
||||
return {'feed': f'{FEED}?playlist_id={playlist}', 'title': _meta(page, 'og:title'),
|
||||
'artwork': _meta(page, 'og:image')}
|
||||
segs = [s for s in path.split('/') if s]
|
||||
if segs[:1] == ['channel'] and len(segs) > 1 and _CHANNEL_ID.match(segs[1]):
|
||||
page_url = f'https://www.youtube.com/channel/{segs[1]}'
|
||||
elif parts.hostname == 'youtu.be' and segs:
|
||||
page_url = WATCH + segs[0]
|
||||
elif segs and (segs[0].startswith('@') or segs[0] in ('c', 'user', 'watch', 'shorts', 'live')):
|
||||
page_url = urllib.parse.urlunsplit(('https', 'www.youtube.com', parts.path, parts.query, ''))
|
||||
else:
|
||||
raise NotAChannel('not a channel address')
|
||||
try:
|
||||
page = _page(page_url)
|
||||
except requests.RequestException as e:
|
||||
raise NotAChannel(type(e).__name__)
|
||||
# The page's own channel, never the first one named (featured channels come earlier).
|
||||
m = None
|
||||
for pattern in _OWN_CHANNEL:
|
||||
m = re.search(pattern, page)
|
||||
if m:
|
||||
break
|
||||
if not m:
|
||||
raise NotAChannel('no channel found')
|
||||
is_channel_page = not (segs and segs[0] in ('watch', 'shorts', 'live')) and parts.hostname != 'youtu.be'
|
||||
return {
|
||||
'feed': channel_feed(m.group(1)),
|
||||
# A video's page is about the video: its channel's name and picture come with the feed.
|
||||
'title': _meta(page, 'og:title') if is_channel_page else '',
|
||||
'artwork': _meta(page, 'og:image') if is_channel_page else '',
|
||||
}
|
||||
|
||||
|
||||
def feed_channel_id(channel):
|
||||
"""The channel a YouTube feed (feedparser's .feed) belongs to, '' for another feed."""
|
||||
for url in (channel.get('link') or '', (channel.get('author_detail') or {}).get('href') or ''):
|
||||
m = re.match(r'^https://www\.youtube\.com/channel/(UC[\w-]{22})$', url)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return ''
|
||||
|
||||
|
||||
def channel_artwork(channel_id):
|
||||
"""The channel's picture (the feed has none), '' if it can't be had."""
|
||||
try:
|
||||
return _meta(_page(f'https://www.youtube.com/channel/{channel_id}'), 'og:image')
|
||||
except (requests.RequestException, NotAChannel):
|
||||
return ''
|
||||
|
|
@ -330,7 +330,10 @@
|
|||
function saveProgress(id, seconds) {
|
||||
forEpisode(state, id, o => { o.position_seconds = seconds; });
|
||||
if (seconds > 0) removeFromInbox(state, new Set([id])); // started: no longer new
|
||||
queueOp('/podcasts/progress/save/', {episode_id: id, position_seconds: seconds}, 'progress:' + id);
|
||||
const body = {episode_id: id, position_seconds: seconds};
|
||||
// A feed without durations (YouTube): the player's, kept by the server if it has none.
|
||||
if (curId() === id && !(cur.episode__duration_seconds > 0) && lastDur > 0) body.duration_seconds = Math.round(lastDur / 1000);
|
||||
queueOp('/podcasts/progress/save/', body, 'progress:' + id);
|
||||
save();
|
||||
// No re-render: progress changes every few seconds while playing.
|
||||
}
|
||||
|
|
@ -708,12 +711,52 @@
|
|||
loaded = true;
|
||||
error = null;
|
||||
buffering = true;
|
||||
if (YOUTUBE.test(cur.episode__audio_url)) { prepareYouTube(); return; }
|
||||
audio.src = cur.episode__audio_url;
|
||||
applyKind();
|
||||
}
|
||||
|
||||
// A YouTube episode's address is the video's page: the server never touches its sound
|
||||
// (diora-web docs/youtube.md). The desktop app finds the audio-only stream on this
|
||||
// computer; a browser can't (YouTube doesn't let a page read it).
|
||||
const YOUTUBE = /^https:\/\/www\.youtube\.com\/watch\?v=[\w-]{11}$/;
|
||||
let youtubeStale = false; // the last stream failed: find it anew
|
||||
let youtubeAsk = 0;
|
||||
|
||||
function prepareYouTube() {
|
||||
const ask = ++youtubeAsk;
|
||||
audio.removeAttribute('src'); // not the stream that failed, while the new one is found
|
||||
try { audio.load(); } catch (e) {}
|
||||
const desk = window.dioraDesktop;
|
||||
const fail = text => {
|
||||
loaded = false;
|
||||
error = 'youtube';
|
||||
wantPlay = false;
|
||||
buffering = false;
|
||||
toast(text);
|
||||
report();
|
||||
syncSoon();
|
||||
};
|
||||
if (!desk || !desk.youtubeAudio) { fail('YouTube-Folgen spielen in der diora-App (Android, Desktop)'); return; }
|
||||
const fresh = youtubeStale;
|
||||
youtubeStale = false;
|
||||
desk.youtubeAudio(cur.episode__audio_url, fresh).then(r => {
|
||||
if (ask !== youtubeAsk || !loaded) return; // moved on meanwhile
|
||||
if (!r || r.error || !r.url) { fail('Diese YouTube-Folge lässt sich gerade nicht abspielen'); return; }
|
||||
if (r.duration > 0) lastDur = r.duration * 1000;
|
||||
audio.src = r.url;
|
||||
applyKind();
|
||||
if (wantPlay) {
|
||||
const p = audio.play();
|
||||
if (p && p.catch) p.catch(() => {});
|
||||
}
|
||||
report();
|
||||
});
|
||||
}
|
||||
|
||||
function unload() {
|
||||
loaded = false;
|
||||
youtubeAsk++;
|
||||
mustReach = null;
|
||||
audio.pause();
|
||||
audio.removeAttribute('src');
|
||||
|
|
@ -764,6 +807,7 @@
|
|||
sync.start = true;
|
||||
applyKind();
|
||||
setSession();
|
||||
if (!audio.getAttribute('src')) { report(); syncSoon(); return; } // a stream still being found
|
||||
const p = audio.play();
|
||||
if (p && p.catch) {
|
||||
p.catch(err => {
|
||||
|
|
@ -935,6 +979,7 @@
|
|||
loaded = false;
|
||||
startAt = at;
|
||||
mustReach = null;
|
||||
youtubeStale = true;
|
||||
if (play && ++failures <= 3) { resume(); return; }
|
||||
error = 'MEDIA_ERR_' + (audio.error ? audio.error.code : 0);
|
||||
wantPlay = false;
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
* diora service worker — caches the app shell for offline use.
|
||||
*/
|
||||
|
||||
const CACHE = 'diora-v111';
|
||||
const CACHE = 'diora-v112';
|
||||
const SHELL = [
|
||||
'/static/css/app.css',
|
||||
'/static/js/app/01-core.js',
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue