Merge branch 'wt-tr-blog' into wt-tr-int
This commit is contained in:
commit
efdc0ffc11
21 changed files with 1114 additions and 21 deletions
|
|
@ -84,6 +84,9 @@ DATABASES = {
|
|||
}
|
||||
}
|
||||
|
||||
# The built-in Whisper instance (OpenAI-compatible, base such as http://whisper:8000/v1);
|
||||
# empty = none (docs/transcription.md).
|
||||
WHISPER_URL = os.environ.get('DIORA_WHISPER_URL', '').strip().rstrip('/')
|
||||
PODCAST_MAX_EPISODES_PER_FEED = int(os.environ.get('PODCAST_MAX_EPISODES_PER_FEED', '200'))
|
||||
|
||||
AUTH_PASSWORD_VALIDATORS = [
|
||||
|
|
|
|||
|
|
@ -29,6 +29,8 @@ kept by the server as a blob it can't read:
|
|||
the key can't be read: the settings ask for it again. Not on diora Reader (e-readers).
|
||||
- Server-side jobs (in the background, without an app open) were left out on purpose: they
|
||||
would need the key on the server. The project Radar has its own server key (docs/projects.md).
|
||||
Podcast articles wait on the server as a wish until a device with the key is open
|
||||
(docs/transcription.md).
|
||||
|
||||
## Building blocks (`static/js/ai.js`, an ES module)
|
||||
|
||||
|
|
@ -53,5 +55,6 @@ kept by the server as a blob it can't read:
|
|||
| Bücher, Zeitschriften | ✦ in the library's head (`static/js/app/55-library-tools.js`) | per book: id, title, author, folders, series, magazine mark; the existing folders | folders to add (existing first; new ones only for ≥ 3 books), series with volume, cleaned title/author where broken, magazine mark |
|
||||
| Podcasts | ✦ in Abos (diora-android `web-native/static/native-podcast.js`, copied to `static/podcasts/`) | per subscription: title, author, folders, level, heard in 12 months, publishing interval, three newest episode titles | folders to add (≤ 2), level Stamm / Strom / Auf Abruf (only with a clear reason) |
|
||||
| Nachrichten, Blogs | ✦ in the feeds' bar (`static/js/feeds.js`; the apps' section bar presses it) | per source: title, address, kind, Ressort or folders, recent titles shown | Ressort for news, folders for blogs, news ↔ blog when clearly wrong |
|
||||
| Podcast-Artikel | ✦ Artikel in an episode (`static/js/podcast-article.js`, docs/transcription.md) | the episode's transcript (feed or Whisper), title, author, shownotes, chapters | an article from the speakers' standpoint with summary and sections, saved to the episode and to Blogs (no review sheet: it is new text, removable) |
|
||||
|
||||
Applied through `_updateBookMeta` (the encrypted meta, as when editing by hand).
|
||||
|
|
|
|||
|
|
@ -182,3 +182,12 @@ Listening without committing: one button plays the Strom like a station.
|
|||
- Web host (`static/js/podcasts-web.js`, `DioraPodcastHost`): `podcastLaterAdd(json episodes)`,
|
||||
`podcastLaterRemove(json ids)`, `podcastListsSwitch(on)`, `podcastStreamSeen()` (keeps the
|
||||
previous time as `state.stream_seen_before` for the marks).
|
||||
|
||||
## Transcripts
|
||||
|
||||
An episode's Transkript tab shows one of two kinds. **From the feed**: `<podcast:transcript>`
|
||||
(SRT, VTT, JSON, Podlove), fetched and parsed on first use and cached in `EpisodeTranscript`
|
||||
(`podcasts/transcripts.py`, `GET /podcasts/episodes/<id>/transcript/`). **Made by Whisper**
|
||||
(since 2026-11): only when the user asks, kept on `TranscriptionJob`; the feed's wins when both
|
||||
exist. From a transcript the device can write an article that is pinned to the episode and shows
|
||||
in Blogs. All of that is in docs/transcription.md.
|
||||
|
|
|
|||
226
docs/transcription.md
Normal file
226
docs/transcription.md
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
# Smarte Transkription: Transkript und Artikel zu einer Folge
|
||||
|
||||
Decided with the user on 2026-11-09. Plan and order of work:
|
||||
diora-android `plans/smarte-transkription.md`.
|
||||
|
||||
## What it is for
|
||||
|
||||
Sometimes the shownotes don't tell whether an episode still has something to teach. Then the
|
||||
user presses a button and, a few hours later at most, gets a readable article written from the
|
||||
speakers' own standpoint. It is meant to let them judge the episode for themselves, so it keeps
|
||||
arguments, reasons and evidence and does not just summarise them. The article sits on the
|
||||
episode and in Blogs.
|
||||
|
||||
Nothing happens on its own. The server never transcribes ahead, by cron or on subscribing; the
|
||||
only way to start a transcription is the `POST` below, sent when the user presses a button.
|
||||
|
||||
## Who does what
|
||||
|
||||
The **server** transcribes. It already downloads and decodes episodes for skipping silence
|
||||
(`podcasts/silence.py`), and browsers and the WebView can't fetch podcast audio (CORS). It cuts
|
||||
the audio at pauses and sends the pieces in parallel to a Whisper instance with an
|
||||
OpenAI-compatible API (`POST /v1/audio/transcriptions`): the built-in one by default (a container
|
||||
beside diora-web, `DIORA_WHISPER_URL`), or any instance the user enters. YouTube episodes are
|
||||
not transcribed: the server never fetches their sound (docs/youtube.md).
|
||||
|
||||
The **device** writes the article with the user's own key (docs/ai.md). The key never reaches the
|
||||
server. It is one LLM call of a few minutes. If diora is open when the transcript is done, it
|
||||
happens right away; otherwise the next time diora is opened on any of the user's devices. The
|
||||
server only remembers that an article was wanted.
|
||||
|
||||
Transcript and article are stored in plain text on the server, like all podcast and feed data.
|
||||
|
||||
## Flow
|
||||
|
||||
1. In the episode the user presses "Transkript erstellen" (the Transkript tab) or
|
||||
"✦ Artikel erstellen" / "✦ Transkribieren und Artikel erstellen". The second sends
|
||||
`article: true`. With a transcript already there (from the feed or an earlier run) only the
|
||||
article is written.
|
||||
2. The server works through the episode (minutes to hours, no GPU on Strato). The UI shows
|
||||
"Transkribiere 7/38 …" while it is open.
|
||||
3. Done: a notice for `GET /feeds/notifications/`. Android's BackgroundSync and the desktop show
|
||||
"Transkript fertig: <Folge>", with "Artikel wird beim Öffnen geschrieben" when one is wanted.
|
||||
Tapping it opens diora at the episode.
|
||||
4. A device with a key (`static/js/podcast-article.js`) picks up the open wishes, reserves one,
|
||||
writes the article and saves it. It then says "Artikel fertig" (on Android as a system
|
||||
notification through the bridge) and opens it on a tap.
|
||||
5. The article is the first tab of the episode ("Artikel") and an item of the favourite blog
|
||||
"Podcast-Artikel", linking back to the episode.
|
||||
|
||||
## API
|
||||
|
||||
**Settings.** `GET /podcasts/transcription/settings/` returns
|
||||
`{engine: "builtin"|"custom", url, model, language, parallel, has_token, builtin_available}`.
|
||||
`POST` takes the same fields and `token` (write-only; `""` removes it). The token is kept
|
||||
encrypted with Fernet like `feeds.SiteLogin.secret`. `language` is `""` for automatic or an ISO
|
||||
code (`de`). `parallel` is 1 to 8, default 3. `builtin_available` is whether `DIORA_WHISPER_URL`
|
||||
is set. `url` is the instance's base, e.g. `https://whisper.example.org/v1`; the server appends
|
||||
`/audio/transcriptions`.
|
||||
|
||||
An empty `model` is sent as `whisper-1`. The built-in URL is read into `settings.WHISPER_URL` from
|
||||
`DIORA_WHISPER_URL`. A custom engine needs a `url`; the token is only sent to custom instances.
|
||||
|
||||
`POST /podcasts/transcription/test/` sends one second of silence to the configured instance and
|
||||
returns `{ok, model, error}`. A custom URL goes through the same SSRF check as chapters
|
||||
(`podcasts/chapters.py` `_open`): no private, loopback or link-local addresses. Only the built-in
|
||||
URL from `DIORA_WHISPER_URL` is exempt.
|
||||
|
||||
**Job.** `POST /podcasts/episodes/<id>/transcribe/` with `{article: true|false}` starts a job, or
|
||||
updates `want_article` on a running one. With a transcript already there it returns
|
||||
`{state: "done"}` straight away (and keeps `want_article`). `GET` on the same address returns
|
||||
`{state, chunks_done, chunks_total, error, want_article, started_at, finished_at}`, with `state`
|
||||
one of `none`, `queued`, `downloading`, `transcribing`, `done`, `error`. `DELETE` cancels a
|
||||
queued or running job. The server runs one job at a time; the others wait as `queued`. A YouTube
|
||||
episode returns 400 `{error: "youtube"}`, an episode over 800 MB 400 `{error: "too large"}`.
|
||||
|
||||
**Transcript.** The result is kept on the job (`TranscriptionJob.data`) in the format of
|
||||
`EpisodeTranscript.data`: `{segments: [{start, end, speaker, text}], source: {type: "whisper",
|
||||
model, language}}`. `speaker` is `""`. It is kept apart from `EpisodeTranscript`, which is a cache
|
||||
of the feed's transcripts and is rewritten when the feed changes them.
|
||||
`GET /podcasts/episodes/<id>/transcript/` returns the feed's transcript when the feed has a
|
||||
usable one, else the Whisper one, else 404 as before. `episode_detail` adds
|
||||
`transcript_origin` (`"feed"`, `"whisper"` or `null`), `has_transcript` is true for both, and it
|
||||
adds `article_item_id` (or `null`) and `transcription` (the job's state as above, or `null`).
|
||||
|
||||
**Article.** `POST /podcasts/episodes/<id>/article/` takes `{title, summary, body_html, model,
|
||||
kind}` and returns `{item_id}`. The server cleans `body_html` with `feeds/fetch.py clean_html`.
|
||||
It creates the user's source "Podcast-Artikel" when missing (`kind` blog, `fav` true,
|
||||
`synthetic` true, `url` `diora://podcast-artikel`) and adds a `feeds.Item` there, or replaces the
|
||||
episode's earlier one. Item fields: the episode's link (else its audio URL) as `url`, the feed's
|
||||
author as `author`, the episode's artwork as `image_url`, now as `published` (so it is on top of
|
||||
the blog stream under "heute"), `words` counted. Saving clears the wish. `DELETE` on the same
|
||||
address removes the article and its item. `GET /feeds/items/<id>/` and `_item_data` carry
|
||||
`episode_id` for such items.
|
||||
`body_html` may contain `href="#t=<seconds>"` links; `clean_html` keeps exactly this fragment form
|
||||
(`podcasts/shownotes.py` `TIME_LINK`) and no other `#` links. Validation: `title` 1 to 300
|
||||
characters, `body_html` non-empty after cleaning and at most 1,000,000 characters, `summary` at most
|
||||
2000, `model` at most 100, `kind` at most 20 (400 otherwise). The item has `content_full` true (no
|
||||
page is ever fetched) and, as the reader's furniture filter would strip repeated "▶ ab" blocks,
|
||||
`GET /feeds/items/<id>/` skips that filter for synthetic sources. The endpoint is
|
||||
`podcasts/article.py`.
|
||||
|
||||
Synthetic sources are skipped by the feed refresh (`refresh_newsfeeds`, `refresh_now`) and by the
|
||||
pruning of old blog items (`feeds/refresh.py`). They can be hidden or have their bell turned off
|
||||
like any blog; they can't be refreshed, renamed to another URL or turned into news. `refresh.refresh()` itself drops them, the OPML export leaves them out, and `_source_data` carries `synthetic`.
|
||||
|
||||
**Open wishes.** `GET /podcasts/transcription/pending/` returns
|
||||
`[{episode_id, title, podcast, duration_seconds}]`: the user's episodes whose transcript is done,
|
||||
that want an article, have none and aren't reserved. `POST
|
||||
/podcasts/episodes/<id>/article/claim/` reserves one for 15 minutes for the calling device
|
||||
(200), or answers 409 when another device holds it, so phone and desktop don't both write it.
|
||||
`DELETE` on the claim releases it (the device gave up, e.g. no key or an error the user saw).
|
||||
|
||||
**Notice.** `GET /feeds/notifications/` additionally takes `t_after` and returns
|
||||
`transcripts: [{job_id, episode_id, title, podcast, want_article}]` (jobs done after that id) and
|
||||
`t_last`. Without `t_after` (a device's first time) the list is empty and only `t_last` comes,
|
||||
like `after`/`last`. Devices that don't know the fields ignore them.
|
||||
|
||||
## Transcribing (`podcasts/transcribe.py`)
|
||||
|
||||
Download with the same limits as `silence.py`, decode with imageio-ffmpeg to 16 kHz mono. Cut
|
||||
into pieces of about eight minutes; each cut moves to the longest pause within ±30 seconds
|
||||
(`silence.py`'s detection), so no word is cut. Pieces go out as FLAC to the built-in instance
|
||||
and as Opus 32 kbit/s to custom ones, to keep uploads small.
|
||||
|
||||
Each piece is sent with `model`, `language` (when set), `response_format=verbose_json` and a
|
||||
`prompt` of podcast title, episode title and author names, so Whisper spells names right. Pieces
|
||||
run in a thread pool of `parallel` workers; each is retried up to three times with a growing
|
||||
pause; progress is written after each. Segment times are shifted by the piece's start; the
|
||||
segments are joined and smoothed with `transcripts.merge`. A job runs in a daemon thread, like
|
||||
`silence.ensure()`; one per server. A job found `downloading` or `transcribing` after a restart
|
||||
is started again.
|
||||
|
||||
## The article (`static/js/podcast-article.js`)
|
||||
|
||||
Input: the transcript as plain text, one line per segment group of about 30 seconds,
|
||||
`[mm:ss] Sprecher: text` (or `[mm:ss] text` without speakers), plus the episode's metadata:
|
||||
podcast title, author, episode title, date, duration, the shownotes as plain text and the
|
||||
chapters. Over about 120,000 tokens (more than about nine hours) the device stops with a clear
|
||||
message for now. A five-hour episode is about 70,000 tokens.
|
||||
|
||||
`DioraAI.ask({system, prompt, schema, effort: "medium", maxTokens: 32000, onProgress, signal})`.
|
||||
`onProgress(words)` reports the words written so far while streaming ("Schreibe Artikel … 2.300
|
||||
Wörter"); `signal` aborts. Both are optional and new in ai.js; the existing callers don't change.
|
||||
|
||||
Schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["title", "summary", "kind", "speakers", "sections"],
|
||||
"properties": {
|
||||
"title": {"type": "string"},
|
||||
"summary": {"type": "string"},
|
||||
"kind": {"type": "string", "enum": ["Interview", "Gespräch", "Monolog", "Diskussion"]},
|
||||
"speakers": {"type": "array", "items": {"type": "string"}},
|
||||
"sections": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["heading", "start", "body_markdown"],
|
||||
"properties": {
|
||||
"heading": {"type": "string"},
|
||||
"start": {"type": "integer"},
|
||||
"body_markdown": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The HTML: the summary as a lead paragraph, then per section an `<h2>`, below it a link
|
||||
"▶ ab 12:34" (`<a href="#t=754">`; `clean_html` keeps fragment links, and feeds.js plays the
|
||||
item's `episode_id` from there when such a link is tapped) and the body. Markdown is limited to paragraphs, emphasis, lists and block
|
||||
quotes, converted on the device; the server cleans the result anyway.
|
||||
|
||||
System prompt (the user's words decide the perspective; keep its substance when changing it):
|
||||
|
||||
```text
|
||||
You turn the transcript of a podcast episode into an article that reads as if the people
|
||||
speaking had written it themselves instead of recording the episode.
|
||||
|
||||
The reader wants to decide whether this episode still has something to teach them. They do not
|
||||
want a summary; they want to understand what is argued here well enough to judge it for
|
||||
themselves. Write for that reader.
|
||||
|
||||
Perspective. Write from the speakers' standpoint and state their positions directly, as their
|
||||
own claims. Never distance yourself from them: no "im Podcast wird die These aufgestellt", "der
|
||||
Sprecher meint", "in dieser Folge geht es um", "die Gäste diskutieren", "laut X". If the episode
|
||||
is one person speaking, write as that person argues, in the third person or impersonal voice,
|
||||
but without reporting verbs. Do not add your own opinion, criticism or fact checks, and do not
|
||||
soften what is said.
|
||||
|
||||
Conversations. Interviews, conversations and discussions are condensed into their core theses;
|
||||
the form of question and answer disappears. Where people hold different positions, keep them
|
||||
apart and name who holds which ("Zuckermann hält dagegen, dass …"), but do not turn it into
|
||||
minutes. Questions from a host matter only where they shape the argument. When the transcript
|
||||
has no speaker names, infer them from the context (introductions, the shownotes, how people
|
||||
address each other); if you can't, do not invent names.
|
||||
|
||||
Substance over summary. Give the arguments with their reasons, examples and evidence, not just
|
||||
the conclusions. Keep figures, dates, quotations that carry an argument (short, in quotation
|
||||
marks, attributed), and every work mentioned (books, articles, studies, films) with author and
|
||||
title. Make clear what is new, distinctive or surprising in the episode and where it restates
|
||||
what is commonly said, by how you weight and order it, without grading it.
|
||||
|
||||
Leave out: advertising and sponsors, appeals for donations, subscriptions or ratings, notes on
|
||||
the podcast itself (schedules, technology, community), greetings, small talk and digressions that
|
||||
lead nowhere.
|
||||
|
||||
Form. Write in the language of the episode. `title`: a title for the article, not the episode
|
||||
title repeated. `summary`: three to six sentences up front that state the central thesis and how
|
||||
the episode gets there. `sections`: as many as the content needs (usually four to ten), each
|
||||
with a heading that says something (not "Einleitung"), `start` = the second in the episode where
|
||||
its content begins (from the [mm:ss] marks), and `body_markdown` in plain paragraphs; lists only
|
||||
where the speakers list things themselves. Follow the order of the episode unless a different
|
||||
order makes the argument clearer. Length: as long as needed for the reader to judge the episode,
|
||||
roughly a tenth of the transcript, at most about 5,000 words.
|
||||
|
||||
The transcript was made by speech recognition: fix obvious mishearings, especially of names,
|
||||
using the metadata, and don't quote garbled passages.
|
||||
```
|
||||
|
||||
Prompt (user turn): the metadata block, then `Transkript:` and the text.
|
||||
|
|
@ -13,7 +13,7 @@ class Command(BaseCommand):
|
|||
parser.add_argument('--source-id', type=int, help='Refresh only this source (even when backing off)')
|
||||
|
||||
def handle(self, *args, **options):
|
||||
qs = Source.objects.select_related('user').order_by('last_refreshed_at')
|
||||
qs = Source.objects.filter(synthetic=False).select_related('user').order_by('last_refreshed_at')
|
||||
if options['source_id']:
|
||||
qs = qs.filter(pk=options['source_id'])
|
||||
else:
|
||||
|
|
|
|||
18
feeds/migrations/0007_transcription.py
Normal file
18
feeds/migrations/0007_transcription.py
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
# Generated by Django 6.1 on 2026-11-09 08:20
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('feeds', '0006_folder_order'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='source',
|
||||
name='synthetic',
|
||||
field=models.BooleanField(default=False),
|
||||
),
|
||||
]
|
||||
|
|
@ -35,6 +35,9 @@ class Source(models.Model):
|
|||
fav = models.BooleanField(default=False)
|
||||
hidden = models.BooleanField(default=False)
|
||||
notify = models.BooleanField(default=False)
|
||||
# Filled by diora itself (the blog "Podcast-Artikel"), not by a feed: never
|
||||
# fetched, pruned, given another URL or turned into news.
|
||||
synthetic = models.BooleanField(default=False)
|
||||
|
||||
# Refresh bookkeeping, as for podcast feeds (see podcasts.views._fetch_feed).
|
||||
last_refreshed_at = models.DateTimeField(null=True, blank=True)
|
||||
|
|
|
|||
|
|
@ -216,7 +216,7 @@ def expire(user, now=None):
|
|||
Item.objects.filter(user=user, gone=True, published__lt=now - timedelta(days=GONE_KEEP_DAYS)).delete()
|
||||
Story.objects.filter(user=user).exclude(items__gone=False).delete()
|
||||
# Blogs: the oldest read articles beyond BLOG_KEEP per blog.
|
||||
for source in Source.objects.filter(user=user, kind=Source.BLOG):
|
||||
for source in Source.objects.filter(user=user, kind=Source.BLOG, synthetic=False):
|
||||
ids = list(source.items.order_by('-published').values_list('id', flat=True)[BLOG_KEEP:])
|
||||
if ids:
|
||||
Item.objects.filter(id__in=ids, starred=False).exclude(read_at=None).delete()
|
||||
|
|
@ -225,7 +225,7 @@ def expire(user, now=None):
|
|||
def refresh(sources, deadline=None):
|
||||
"""Refresh `sources` (downloads in parallel, writes one by one).
|
||||
Returns (refreshed, failed, new items)."""
|
||||
sources = list(sources)
|
||||
sources = [s for s in sources if not s.synthetic] # synthetic sources have no feed to fetch
|
||||
now = timezone.now()
|
||||
keep = {}
|
||||
refreshed = failed = 0
|
||||
|
|
|
|||
|
|
@ -539,3 +539,56 @@ class NewsTextApiTest(TestCase):
|
|||
self.assertEqual(got['related'], [])
|
||||
single = self.client.get(f'/feeds/stories/{self.items[2].story_id}/overview/').json()
|
||||
self.assertNotIn('summary', single)
|
||||
|
||||
|
||||
class SyntheticSourceTest(TestCase):
|
||||
"""A source diora fills itself ("Podcast-Artikel") is never fetched or pruned."""
|
||||
|
||||
def setUp(self):
|
||||
self.u = User.objects.create_user('tester', password='pw12345678')
|
||||
self.client.force_login(self.u)
|
||||
self.synth = Source.objects.create(user=self.u, url='diora://podcast-artikel', title='Podcast-Artikel',
|
||||
kind='blog', fav=True, synthetic=True)
|
||||
self.real = Source.objects.create(user=self.u, url='https://real.example/feed', title='Real', kind='blog')
|
||||
|
||||
def post(self, url, data):
|
||||
return self.client.post(url, json.dumps(data), content_type='application/json')
|
||||
|
||||
def test_refresh_skips_it(self):
|
||||
with mock.patch('feeds.refresh._fetch_feed', return_value=fetched(rss([]), 'https://real.example/feed')) as get:
|
||||
refresh.refresh([self.synth, self.real])
|
||||
self.assertEqual([c.args[0] for c in get.call_args_list], ['https://real.example/feed'])
|
||||
|
||||
def test_refresh_now_and_command_skip_it(self):
|
||||
with mock.patch('feeds.refresh._fetch_feed', return_value=fetched(rss([]), 'https://real.example/feed')) as get:
|
||||
r = self.post('/feeds/refresh/', {})
|
||||
from django.core.management import call_command
|
||||
call_command('refresh_newsfeeds', stdout=mock.MagicMock())
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.assertTrue(get.call_args_list)
|
||||
self.assertTrue(all(c.args[0] == 'https://real.example/feed' for c in get.call_args_list))
|
||||
call_command('refresh_newsfeeds', '--source-id', str(self.synth.id), stdout=mock.MagicMock())
|
||||
|
||||
def test_old_items_are_not_pruned(self):
|
||||
old = timezone.now() - timedelta(days=400)
|
||||
for source in (self.synth, self.real):
|
||||
for n in range(refresh.BLOG_KEEP + 5):
|
||||
Item.objects.create(source=source, user=self.u, kind='blog', guid_hash=f'{source.pk}-{n}',
|
||||
title='x', published=old - timedelta(minutes=n), read_at=old)
|
||||
refresh.expire(self.u)
|
||||
self.assertEqual(self.synth.items.count(), refresh.BLOG_KEEP + 5)
|
||||
self.assertEqual(self.real.items.count(), refresh.BLOG_KEEP)
|
||||
|
||||
def test_cannot_become_news_but_can_be_hidden(self):
|
||||
r = self.post(f'/feeds/sources/{self.synth.id}/', {'kind': 'news'})
|
||||
self.assertEqual(r.status_code, 400)
|
||||
r = self.post(f'/feeds/sources/{self.synth.id}/', {'hidden': True, 'notify': True})
|
||||
self.assertEqual(r.status_code, 200)
|
||||
self.synth.refresh_from_db()
|
||||
self.assertEqual((self.synth.kind, self.synth.hidden, self.synth.notify), ('blog', True, True))
|
||||
self.assertTrue(r.json()['source']['synthetic'])
|
||||
|
||||
def test_not_in_opml(self):
|
||||
body = self.client.get('/feeds/export.opml').content.decode()
|
||||
self.assertIn('real.example', body)
|
||||
self.assertNotIn('diora://', body)
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ def _source_data(s, unread=0, new=0, interval=None):
|
|||
'id': s.id, 'url': s.url, 'title': s.title, 'site_url': s.site_url, 'icon_url': s.icon_url,
|
||||
'kind': s.kind, 'category': s.category, 'folders': s.folders, 'fulltext': s.fulltext,
|
||||
'unread': unread, 'new': new,
|
||||
'fav': s.fav, 'hidden': s.hidden, 'notify': s.notify,
|
||||
'fav': s.fav, 'hidden': s.hidden, 'notify': s.notify, 'synthetic': s.synthetic,
|
||||
# Days between posts (blogs, for "Seltene zuerst"); a blog with one post counts as rare.
|
||||
'interval_days': interval,
|
||||
'last_refreshed_at': _iso(s.last_refreshed_at), 'last_error': s.last_error, 'error_count': s.error_count,
|
||||
|
|
@ -108,6 +108,11 @@ def _item_data(i, content=False):
|
|||
'summary': i.summary, 'image_url': i.image_url, 'read': i.read_at is not None, 'progress': i.progress,
|
||||
'starred': i.starred, 'story': i.story_id,
|
||||
}
|
||||
if i.source.synthetic:
|
||||
# The article of a podcast episode (docs/transcription.md); the lists fetch it with the item.
|
||||
art = getattr(i, 'episode_article', None)
|
||||
if art is not None:
|
||||
d['episode_id'] = art.episode_id
|
||||
if content:
|
||||
d['content'] = i.content
|
||||
d['content_full'] = i.content_full
|
||||
|
|
@ -356,7 +361,7 @@ def add_sources(request):
|
|||
defaults=_place(request.user, kind, x.get('category'), x.get('folders'),
|
||||
title=(x.get('title') or url)[:300]))
|
||||
added += created
|
||||
sources = Source.objects.filter(user=request.user, last_refreshed_at=None)
|
||||
sources = Source.objects.filter(user=request.user, last_refreshed_at=None, synthetic=False)
|
||||
refresh.refresh(sources, deadline=time.monotonic() + REFRESH_WAIT)
|
||||
return JsonResponse({'added': added})
|
||||
|
||||
|
|
@ -386,6 +391,8 @@ def edit_source(request, pk):
|
|||
if body.get('fulltext') in dict(Source.FULLTEXTS):
|
||||
source.fulltext = body['fulltext']
|
||||
new_kind = _kind(body.get('kind'))
|
||||
if source.synthetic and new_kind == Source.NEWS:
|
||||
return JsonResponse({'error': 'Diese Quelle bleibt ein Blog'}, status=400)
|
||||
if new_kind and new_kind != source.kind:
|
||||
# A section becomes a folder and back.
|
||||
if new_kind == Source.BLOG and source.category and not source.folders:
|
||||
|
|
@ -416,7 +423,7 @@ def export_opml(request):
|
|||
if (err := _auth(request)):
|
||||
return err
|
||||
groups = {}
|
||||
for s in Source.objects.filter(user=request.user).order_by('kind', 'category', 'title'):
|
||||
for s in Source.objects.filter(user=request.user, synthetic=False).order_by('kind', 'category', 'title'):
|
||||
kind = 'Nachrichten' if s.kind == Source.NEWS else 'Blogs'
|
||||
place = s.category if s.kind == Source.NEWS else ((s.folders or [''])[0])
|
||||
label = kind + (f' – {place}' if place else '')
|
||||
|
|
@ -440,7 +447,7 @@ def export_opml(request):
|
|||
def refresh_now(request):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
refreshed, failed, new = refresh.refresh(Source.objects.filter(user=request.user),
|
||||
refreshed, failed, new = refresh.refresh(Source.objects.filter(user=request.user, synthetic=False),
|
||||
deadline=time.monotonic() + REFRESH_WAIT)
|
||||
return JsonResponse({'refreshed': refreshed, 'failed': failed, 'new': new})
|
||||
|
||||
|
|
@ -611,7 +618,7 @@ def story_overview(request, pk):
|
|||
def blogs(request):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
qs = Item.objects.filter(user=request.user, kind=Source.BLOG, gone=False).select_related('source')
|
||||
qs = Item.objects.filter(user=request.user, kind=Source.BLOG, gone=False).select_related('source', 'episode_article')
|
||||
if request.GET.get('source', '').isdigit():
|
||||
qs = qs.filter(source_id=int(request.GET['source']))
|
||||
elif 'folder' in request.GET:
|
||||
|
|
@ -640,7 +647,7 @@ def blogs(request):
|
|||
def starred(request):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
items = Item.objects.filter(user=request.user, starred=True).select_related('source').order_by('-published')
|
||||
items = Item.objects.filter(user=request.user, starred=True).select_related('source', 'episode_article').order_by('-published')
|
||||
return JsonResponse({'items': [_item_data(i) for i in items[:200]]})
|
||||
|
||||
|
||||
|
|
@ -649,13 +656,14 @@ def starred(request):
|
|||
def item(request, pk):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
i = Item.objects.filter(user=request.user, pk=pk, gone=False).select_related('source').first()
|
||||
i = Item.objects.filter(user=request.user, pk=pk, gone=False).select_related('source', 'episode_article').first()
|
||||
if i is None:
|
||||
return JsonResponse({'error': 'not found'}, status=404)
|
||||
if refresh.needs_article(i, i.source):
|
||||
refresh.load_article(i)
|
||||
d = _item_data(i, content=True)
|
||||
d['content'] = textproc.html_without_furniture(i.content, _furniture(i.source))
|
||||
if not i.source.synthetic:
|
||||
d['content'] = textproc.html_without_furniture(i.content, _furniture(i.source))
|
||||
return JsonResponse({'item': d})
|
||||
|
||||
|
||||
|
|
@ -770,6 +778,8 @@ def save_login(request):
|
|||
password = body.get('password') or ''
|
||||
if source is None or not username or not password:
|
||||
return JsonResponse({'error': 'Benutzername und Passwort nötig'}, status=400)
|
||||
if source.synthetic:
|
||||
return JsonResponse({'error': 'Diese Quelle hat keine Adresse'}, status=400)
|
||||
try:
|
||||
adapter = logins.check_and_store(request.user, source.url, username, password)
|
||||
except logins.LoginFailed as e:
|
||||
|
|
|
|||
119
podcasts/article.py
Normal file
119
podcasts/article.py
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
"""The article written from an episode's transcript (docs/transcription.md): its text
|
||||
is a feeds.Item in the user's favourite blog "Podcast-Artikel"."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
from django.db import transaction
|
||||
from django.http import JsonResponse
|
||||
from django.utils import timezone
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
from django.views.decorators.http import require_http_methods
|
||||
|
||||
from feeds import fetch
|
||||
from feeds.models import Item, Source
|
||||
|
||||
from .models import EpisodeArticle, PodcastEpisode, TranscriptionJob
|
||||
|
||||
SOURCE_URL = 'diora://podcast-artikel'
|
||||
SOURCE_TITLE = 'Podcast-Artikel'
|
||||
TITLE_MAX = 300
|
||||
SUMMARY_MAX = 2000
|
||||
BODY_MAX = 1_000_000 # characters of body_html
|
||||
MODEL_MAX = 100
|
||||
KIND_MAX = 20
|
||||
|
||||
|
||||
def _auth(request):
|
||||
if not request.user.is_authenticated:
|
||||
return JsonResponse({'error': 'authentication required'}, status=401)
|
||||
return None
|
||||
|
||||
|
||||
def _body(request):
|
||||
try:
|
||||
body = json.loads(request.body or b'{}')
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return None
|
||||
return body if isinstance(body, dict) else None
|
||||
|
||||
|
||||
def article_source(user):
|
||||
"""The user's blog "Podcast-Artikel", made on first use."""
|
||||
source, _ = Source.objects.get_or_create(user=user, url=SOURCE_URL, defaults={
|
||||
'title': SOURCE_TITLE, 'kind': Source.BLOG, 'fav': True, 'synthetic': True})
|
||||
return source
|
||||
|
||||
|
||||
def _text(value, limit):
|
||||
return value.strip() if isinstance(value, str) and len(value.strip()) <= limit else None
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
@require_http_methods(['POST', 'DELETE'])
|
||||
def episode_article(request, pk):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
ep = PodcastEpisode.objects.filter(pk=pk, feed__user=request.user).select_related('feed').first()
|
||||
if ep is None:
|
||||
return JsonResponse({'error': 'not found'}, status=404)
|
||||
if request.method == 'DELETE':
|
||||
art = EpisodeArticle.objects.filter(episode=ep).select_related('item').first()
|
||||
if art is not None:
|
||||
art.item.delete() # the article goes with it (cascade)
|
||||
return JsonResponse({'ok': True})
|
||||
|
||||
body = _body(request)
|
||||
if body is None:
|
||||
return JsonResponse({'error': 'invalid JSON'}, status=400)
|
||||
title = _text(body.get('title'), TITLE_MAX)
|
||||
if not title:
|
||||
return JsonResponse({'error': 'title needed (at most %d characters)' % TITLE_MAX}, status=400)
|
||||
html_in = body.get('body_html')
|
||||
if not isinstance(html_in, str) or not html_in.strip():
|
||||
return JsonResponse({'error': 'body_html needed'}, status=400)
|
||||
if len(html_in) > BODY_MAX:
|
||||
return JsonResponse({'error': 'body_html too large'}, status=400)
|
||||
summary = body.get('summary') or ''
|
||||
if not isinstance(summary, str) or len(summary) > SUMMARY_MAX:
|
||||
return JsonResponse({'error': 'summary too long'}, status=400)
|
||||
model = body.get('model') or ''
|
||||
kind = body.get('kind') or ''
|
||||
if not isinstance(model, str) or len(model) > MODEL_MAX or not isinstance(kind, str) or len(kind) > KIND_MAX:
|
||||
return JsonResponse({'error': 'model or kind too long'}, status=400)
|
||||
|
||||
url = getattr(ep, 'link', '') or ep.audio_url
|
||||
content = fetch.clean_html(html_in, url)
|
||||
if not fetch.plain(content):
|
||||
return JsonResponse({'error': 'body_html empty'}, status=400)
|
||||
fields = {
|
||||
'url': url[:2000],
|
||||
'title': title,
|
||||
'author': (ep.feed.author or '')[:300],
|
||||
'published': timezone.now(),
|
||||
'summary': fetch.plain(summary),
|
||||
'content': content,
|
||||
'content_full': True, # nothing to fetch from a page
|
||||
'words': fetch.word_count(content),
|
||||
'image_url': (ep.artwork_url or ep.feed.artwork_url or '')[:2000],
|
||||
'read_at': None, 'progress': 0, 'progress_at': None,
|
||||
}
|
||||
with transaction.atomic():
|
||||
source = article_source(request.user)
|
||||
art = EpisodeArticle.objects.filter(episode=ep).select_related('item').first()
|
||||
if art is not None:
|
||||
item = art.item
|
||||
for k, v in fields.items():
|
||||
setattr(item, k, v)
|
||||
item.source = source
|
||||
item.save()
|
||||
art.model, art.kind = model, kind
|
||||
art.save(update_fields=['model', 'kind'])
|
||||
else:
|
||||
guid = hashlib.sha256(f'diora-article:{ep.pk}'.encode()).hexdigest()
|
||||
item = Item.objects.create(source=source, user=request.user, kind=Source.BLOG,
|
||||
guid_hash=guid, **fields)
|
||||
EpisodeArticle.objects.create(episode=ep, item=item, model=model, kind=kind)
|
||||
# Saving fulfils the wish and frees the reservation.
|
||||
TranscriptionJob.objects.filter(episode=ep).update(want_article=False, claimed_by='', claimed_until=None)
|
||||
return JsonResponse({'item_id': item.id})
|
||||
|
|
@ -128,25 +128,28 @@ FetchUnavailable = ChaptersUnavailable
|
|||
FetchInvalid = ChaptersInvalid
|
||||
|
||||
|
||||
def _open(url, headers):
|
||||
"""GET `url` with every redirect hop checked; the streamed response of
|
||||
the final hop, which the caller has to close.
|
||||
def _open(url, headers, method='GET', timeout=CHAPTERS_TIMEOUT, **kwargs):
|
||||
"""`method` (GET or POST) `url` with every redirect hop checked; the streamed response of
|
||||
the final hop, which the caller has to close. `kwargs` (data, files) go to
|
||||
requests; a POST only follows 307/308, which keep the method and body.
|
||||
|
||||
Raises ChaptersInvalid for permanent problems and ChaptersUnavailable for
|
||||
ones that may go away (network trouble, server errors).
|
||||
"""
|
||||
redirects = (301, 302, 303, 307, 308) if method == 'GET' else (307, 308)
|
||||
for _ in range(CHAPTERS_MAX_REDIRECTS + 1):
|
||||
try:
|
||||
assert_safe_url(url)
|
||||
except WebDAVInputError as e:
|
||||
raise ChaptersInvalid(str(e))
|
||||
try:
|
||||
r = requests.get(url, timeout=CHAPTERS_TIMEOUT, allow_redirects=False, stream=True, headers=headers)
|
||||
send = requests.get if method == 'GET' else requests.post
|
||||
r = send(url, timeout=timeout, allow_redirects=False, stream=True, headers=headers, **kwargs)
|
||||
except requests.RequestException as e:
|
||||
raise ChaptersUnavailable(e.__class__.__name__)
|
||||
try:
|
||||
_assert_peer_is_safe(r)
|
||||
if r.status_code in (301, 302, 303, 307, 308):
|
||||
if r.status_code in redirects:
|
||||
location = r.headers.get('Location')
|
||||
if not location:
|
||||
raise ChaptersInvalid('redirect without a location')
|
||||
|
|
@ -155,7 +158,7 @@ def _open(url, headers):
|
|||
continue
|
||||
if r.status_code >= 500 or r.status_code == 429:
|
||||
raise ChaptersUnavailable(f'HTTP {r.status_code}')
|
||||
if r.status_code >= 400:
|
||||
if r.status_code >= 400 or r.status_code in (301, 302, 303):
|
||||
raise ChaptersInvalid(f'HTTP {r.status_code}')
|
||||
except WebDAVInputError as e:
|
||||
r.close()
|
||||
|
|
|
|||
59
podcasts/migrations/0027_transcription.py
Normal file
59
podcasts/migrations/0027_transcription.py
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Generated by Django 6.1 on 2026-11-09 08:20
|
||||
|
||||
import django.db.models.deletion
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('feeds', '0007_transcription'),
|
||||
('podcasts', '0026_podcastepisode_sponsor'),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name='EpisodeArticle',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('model', models.CharField(blank=True, max_length=100)),
|
||||
('kind', models.CharField(blank=True, max_length=20)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('episode', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='article', to='podcasts.podcastepisode')),
|
||||
('item', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='episode_article', to='feeds.item')),
|
||||
],
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='TranscriptionJob',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('state', models.CharField(choices=[('none', 'none'), ('queued', 'queued'), ('downloading', 'downloading'), ('transcribing', 'transcribing'), ('done', 'done'), ('error', 'error')], default='none', max_length=15)),
|
||||
('chunks_done', models.PositiveIntegerField(default=0)),
|
||||
('chunks_total', models.PositiveIntegerField(default=0)),
|
||||
('error', models.CharField(blank=True, max_length=500)),
|
||||
('want_article', models.BooleanField(default=False)),
|
||||
('claimed_by', models.CharField(blank=True, max_length=60)),
|
||||
('claimed_until', models.DateTimeField(blank=True, null=True)),
|
||||
('data', models.JSONField(blank=True, null=True)),
|
||||
('created_at', models.DateTimeField(auto_now_add=True)),
|
||||
('started_at', models.DateTimeField(blank=True, null=True)),
|
||||
('finished_at', models.DateTimeField(blank=True, null=True)),
|
||||
('episode', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='transcription_job', to='podcasts.podcastepisode')),
|
||||
],
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name='TranscriptionSettings',
|
||||
fields=[
|
||||
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
|
||||
('engine', models.CharField(choices=[('builtin', 'Eingebaut'), ('custom', 'Eigene Instanz')], default='builtin', max_length=10)),
|
||||
('url', models.CharField(blank=True, max_length=500)),
|
||||
('token_encrypted', models.TextField(blank=True)),
|
||||
('model', models.CharField(blank=True, max_length=100)),
|
||||
('language', models.CharField(blank=True, max_length=10)),
|
||||
('parallel', models.PositiveSmallIntegerField(default=3)),
|
||||
('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='transcription_settings', to=settings.AUTH_USER_MODEL)),
|
||||
],
|
||||
),
|
||||
]
|
||||
|
|
@ -219,6 +219,68 @@ class EpisodeTranscript(models.Model):
|
|||
return f'Transcript of {self.episode}'
|
||||
|
||||
|
||||
class TranscriptionSettings(models.Model):
|
||||
"""Where a user's episodes are transcribed (docs/transcription.md): the
|
||||
built-in Whisper instance (DIORA_WHISPER_URL) or their own."""
|
||||
BUILTIN = 'builtin'
|
||||
CUSTOM = 'custom'
|
||||
ENGINES = [(BUILTIN, 'Eingebaut'), (CUSTOM, 'Eigene Instanz')]
|
||||
|
||||
user = models.OneToOneField(User, on_delete=models.CASCADE, related_name='transcription_settings')
|
||||
engine = models.CharField(max_length=10, choices=ENGINES, default=BUILTIN)
|
||||
# The instance's base, e.g. https://whisper.example.org/v1 (custom only).
|
||||
url = models.CharField(max_length=500, blank=True)
|
||||
# The API token, Fernet-encrypted like feeds.SiteLogin.secret; never handed out.
|
||||
token_encrypted = models.TextField(blank=True)
|
||||
model = models.CharField(max_length=100, blank=True)
|
||||
# '' = automatic, else an ISO code ("de").
|
||||
language = models.CharField(max_length=10, blank=True)
|
||||
parallel = models.PositiveSmallIntegerField(default=3) # 1 to 8 pieces at once
|
||||
|
||||
def __str__(self):
|
||||
return f'Transcription settings of {self.user}'
|
||||
|
||||
|
||||
class TranscriptionJob(models.Model):
|
||||
"""One episode's transcription run, its Whisper result and the wish for an
|
||||
article. Kept apart from EpisodeTranscript, which caches the feed's
|
||||
transcripts and is rewritten when the feed changes them."""
|
||||
NONE, QUEUED, DOWNLOADING, TRANSCRIBING, DONE, ERROR = (
|
||||
'none', 'queued', 'downloading', 'transcribing', 'done', 'error')
|
||||
STATES = [(s, s) for s in (NONE, QUEUED, DOWNLOADING, TRANSCRIBING, DONE, ERROR)]
|
||||
|
||||
episode = models.OneToOneField(PodcastEpisode, on_delete=models.CASCADE, related_name='transcription_job')
|
||||
state = models.CharField(max_length=15, choices=STATES, default=NONE)
|
||||
chunks_done = models.PositiveIntegerField(default=0)
|
||||
chunks_total = models.PositiveIntegerField(default=0)
|
||||
error = models.CharField(max_length=500, blank=True)
|
||||
want_article = models.BooleanField(default=False)
|
||||
# The device writing the article (a short name) and until when it holds the wish.
|
||||
claimed_by = models.CharField(max_length=60, blank=True)
|
||||
claimed_until = models.DateTimeField(null=True, blank=True)
|
||||
# {"segments": [{start, end, speaker, text}], "source": {type: "whisper", model, language}}
|
||||
data = models.JSONField(null=True, blank=True)
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
started_at = models.DateTimeField(null=True, blank=True)
|
||||
finished_at = models.DateTimeField(null=True, blank=True)
|
||||
|
||||
def __str__(self):
|
||||
return f'Transcription of {self.episode} ({self.state})'
|
||||
|
||||
|
||||
class EpisodeArticle(models.Model):
|
||||
"""The article written from an episode's transcript; its text is the
|
||||
feeds.Item (in the blog "Podcast-Artikel")."""
|
||||
episode = models.OneToOneField(PodcastEpisode, on_delete=models.CASCADE, related_name='article')
|
||||
item = models.OneToOneField('feeds.Item', on_delete=models.CASCADE, related_name='episode_article')
|
||||
model = models.CharField(max_length=100, blank=True)
|
||||
kind = models.CharField(max_length=20, blank=True) # Interview, Gespräch, Monolog, Diskussion
|
||||
created_at = models.DateTimeField(auto_now_add=True)
|
||||
|
||||
def __str__(self):
|
||||
return f'Article on {self.episode}'
|
||||
|
||||
|
||||
# An episode counts as played this close to its end (AntennaPod's default),
|
||||
# the same rule as the apps' "als gehört zählen".
|
||||
PLAYED_BEFORE_END_SECONDS = 30
|
||||
|
|
|
|||
|
|
@ -37,6 +37,9 @@ ALLOWED_ATTRS = {
|
|||
URL_ATTRS = {'href', 'src'}
|
||||
LINK_SCHEMES = {'http', 'https', 'mailto'}
|
||||
IMAGE_SCHEMES = {'http', 'https'}
|
||||
# A link to a second in the episode ("#t=754"), as in the article of a transcript
|
||||
# (docs/transcription.md): the only fragment link that is kept.
|
||||
TIME_LINK = re.compile(r'^#t=\d{1,6}$')
|
||||
|
||||
_CONTROL_AND_SPACE = re.compile(r'[\x00-\x20\x7f]+')
|
||||
_SCHEME = re.compile(r'^([a-z][a-z0-9+.\-]*):')
|
||||
|
|
@ -81,7 +84,9 @@ class _Sanitizer(HTMLParser):
|
|||
name = name.lower()
|
||||
if name not in allowed or value is None:
|
||||
continue
|
||||
if name in URL_ATTRS:
|
||||
if tag == 'a' and name == 'href' and TIME_LINK.match(value.strip()):
|
||||
value = value.strip()
|
||||
elif name in URL_ATTRS:
|
||||
value = _safe_url(value, IMAGE_SCHEMES if tag == 'img' else LINK_SCHEMES)
|
||||
if value is None:
|
||||
continue
|
||||
|
|
|
|||
146
podcasts/test_article.py
Normal file
146
podcasts/test_article.py
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
import json
|
||||
|
||||
from django.contrib.auth.models import User
|
||||
from django.test import TestCase
|
||||
from django.utils import timezone
|
||||
|
||||
from feeds.models import Item, Source
|
||||
|
||||
from .models import EpisodeArticle, PodcastEpisode, PodcastFeed, TranscriptionJob
|
||||
|
||||
BODY = ('<p>Zusammenfassung <script>alert(1)</script></p><h2>Teil eins</h2>'
|
||||
'<p><a href="#t=754" onclick="evil()">▶ ab 12:34</a> und <a href="https://example.org/x">Quelle</a></p>')
|
||||
|
||||
|
||||
class ArticleTests(TestCase):
|
||||
def setUp(self):
|
||||
self.user = User.objects.create_user(username='alice', password='pw12345678')
|
||||
self.client.force_login(self.user)
|
||||
self.feed = PodcastFeed.objects.create(user=self.user, rss_url='https://p.example/rss', title='Salon',
|
||||
author='Anna Autorin', artwork_url='https://p.example/feed.jpg')
|
||||
self.ep = PodcastEpisode.objects.create(feed=self.feed, guid='e1', title='Folge 1',
|
||||
audio_url='https://p.example/e1.mp3', artwork_url='https://p.example/e1.jpg')
|
||||
|
||||
def url(self, ep=None):
|
||||
return f'/podcasts/episodes/{(ep or self.ep).id}/article/'
|
||||
|
||||
def post(self, data=None, ep=None):
|
||||
data = {'title': 'Ein Artikel', 'summary': 'Kurz.', 'body_html': BODY, 'model': 'claude', 'kind': 'Interview'} \
|
||||
if data is None else data
|
||||
return self.client.post(self.url(ep), json.dumps(data), content_type='application/json')
|
||||
|
||||
def test_requires_login_and_ownership(self):
|
||||
self.client.logout()
|
||||
self.assertEqual(self.post().status_code, 401)
|
||||
self.assertEqual(self.client.delete(self.url()).status_code, 401)
|
||||
bob = User.objects.create_user(username='bob', password='pw12345678')
|
||||
self.client.force_login(bob)
|
||||
self.assertEqual(self.post().status_code, 404)
|
||||
self.assertEqual(self.client.delete(self.url()).status_code, 404)
|
||||
self.assertFalse(Item.objects.exists())
|
||||
|
||||
def test_create(self):
|
||||
r = self.post()
|
||||
self.assertEqual(r.status_code, 200)
|
||||
item = Item.objects.get(pk=r.json()['item_id'])
|
||||
self.assertEqual(item.user, self.user)
|
||||
self.assertEqual(item.kind, Source.BLOG)
|
||||
self.assertEqual(item.title, 'Ein Artikel')
|
||||
self.assertEqual(item.summary, 'Kurz.')
|
||||
self.assertEqual(item.author, 'Anna Autorin')
|
||||
self.assertEqual(item.image_url, 'https://p.example/e1.jpg')
|
||||
self.assertEqual(item.url, 'https://p.example/e1.mp3')
|
||||
self.assertTrue(item.content_full)
|
||||
self.assertGreater(item.words, 3)
|
||||
self.assertLess((timezone.now() - item.published).total_seconds(), 60)
|
||||
art = EpisodeArticle.objects.get(episode=self.ep)
|
||||
self.assertEqual((art.item, art.model, art.kind), (item, 'claude', 'Interview'))
|
||||
self.assertEqual(item.episode_article, art)
|
||||
|
||||
def test_image_falls_back_to_feed_artwork(self):
|
||||
self.ep.artwork_url = ''
|
||||
self.ep.save()
|
||||
item = Item.objects.get(pk=self.post().json()['item_id'])
|
||||
self.assertEqual(item.image_url, 'https://p.example/feed.jpg')
|
||||
|
||||
def test_sanitizer_keeps_time_links_only(self):
|
||||
item = Item.objects.get(pk=self.post().json()['item_id'])
|
||||
self.assertNotIn('<script', item.content)
|
||||
self.assertNotIn('alert(1)', item.content)
|
||||
self.assertNotIn('onclick', item.content)
|
||||
self.assertIn('href="#t=754"', item.content)
|
||||
self.assertIn('href="https://example.org/x"', item.content)
|
||||
evil = self.post({'title': 'x', 'body_html': '<a href="#t=1" onfocus="x">a</a><a href="#top">b</a>'})
|
||||
content = Item.objects.get(pk=evil.json()['item_id']).content
|
||||
self.assertNotIn('onfocus', content)
|
||||
self.assertNotIn('#top', content)
|
||||
|
||||
def test_source_made_once(self):
|
||||
other = PodcastEpisode.objects.create(feed=self.feed, guid='e2', title='Folge 2', audio_url='https://p.example/e2.mp3')
|
||||
self.post()
|
||||
self.post(ep=other)
|
||||
src = Source.objects.get(user=self.user, url='diora://podcast-artikel')
|
||||
self.assertEqual((src.title, src.kind, src.fav, src.synthetic), ('Podcast-Artikel', Source.BLOG, True, True))
|
||||
self.assertEqual(Source.objects.count(), 1)
|
||||
self.assertEqual(src.items.count(), 2)
|
||||
|
||||
def test_replace_keeps_one_item(self):
|
||||
first = self.post().json()['item_id']
|
||||
Item.objects.filter(pk=first).update(read_at=timezone.now(), progress=1)
|
||||
second = self.post({'title': 'Neu', 'body_html': '<p>Ganz anderer Text</p>', 'model': 'm2'}).json()['item_id']
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(Item.objects.count(), 1)
|
||||
item = Item.objects.get()
|
||||
self.assertEqual((item.title, item.read_at, item.progress), ('Neu', None, 0))
|
||||
self.assertIn('Ganz anderer Text', item.content)
|
||||
self.assertEqual(EpisodeArticle.objects.get().model, 'm2')
|
||||
|
||||
def test_delete(self):
|
||||
self.post()
|
||||
self.assertEqual(self.client.delete(self.url()).status_code, 200)
|
||||
self.assertFalse(Item.objects.exists())
|
||||
self.assertFalse(EpisodeArticle.objects.exists())
|
||||
self.assertEqual(Source.objects.count(), 1)
|
||||
self.assertEqual(self.client.delete(self.url()).status_code, 200) # again: nothing to do
|
||||
|
||||
def test_validation(self):
|
||||
for data in ({}, {'title': '', 'body_html': BODY}, {'title': 'x' * 301, 'body_html': BODY},
|
||||
{'title': 'x'}, {'title': 'x', 'body_html': '<script>1</script>'},
|
||||
{'title': 'x', 'body_html': 'a' * 1_000_001},
|
||||
{'title': 'x', 'body_html': BODY, 'model': 'm' * 101}):
|
||||
self.assertEqual(self.post(data).status_code, 400, data and list(data))
|
||||
self.assertEqual(self.client.post(self.url(), 'nope', content_type='application/json').status_code, 400)
|
||||
self.assertFalse(Item.objects.exists())
|
||||
self.assertEqual(self.client.get(self.url()).status_code, 405)
|
||||
|
||||
def test_saving_clears_the_wish(self):
|
||||
TranscriptionJob.objects.create(episode=self.ep, state='done', want_article=True, claimed_by='phone',
|
||||
claimed_until=timezone.now() + timezone.timedelta(minutes=10))
|
||||
self.post()
|
||||
job = TranscriptionJob.objects.get()
|
||||
self.assertEqual((job.want_article, job.claimed_by, job.claimed_until), (False, '', None))
|
||||
|
||||
def test_in_favourite_blogs_with_episode_id(self):
|
||||
item_id = self.post().json()['item_id']
|
||||
items = self.client.get('/feeds/blogs/?fav=1').json()['items']
|
||||
self.assertEqual([i['id'] for i in items], [item_id])
|
||||
self.assertEqual(items[0]['episode_id'], self.ep.id)
|
||||
detail = self.client.get(f'/feeds/items/{item_id}/').json()['item']
|
||||
self.assertEqual(detail['episode_id'], self.ep.id)
|
||||
self.assertIn('href="#t=754"', detail['content'])
|
||||
|
||||
def test_ordinary_items_have_no_episode_id(self):
|
||||
src = Source.objects.create(user=self.user, url='https://b.example/feed', title='B', kind=Source.BLOG, fav=True)
|
||||
Item.objects.create(source=src, user=self.user, kind=Source.BLOG, guid_hash='g', title='T',
|
||||
published=timezone.now(), content='<p>x</p>', content_full=True)
|
||||
self.post()
|
||||
items = self.client.get('/feeds/blogs/?fav=1').json()['items']
|
||||
self.assertEqual(len(items), 2)
|
||||
self.assertEqual(sorted('episode_id' in i for i in items), [False, True])
|
||||
|
||||
def test_blog_list_needs_no_query_per_article(self):
|
||||
for n in range(5):
|
||||
self.post(ep=PodcastEpisode.objects.create(feed=self.feed, guid=f'x{n}', title=str(n),
|
||||
audio_url='https://p.example/x.mp3'))
|
||||
with self.assertNumQueries(3):
|
||||
self.client.get('/feeds/blogs/?fav=1')
|
||||
153
podcasts/test_transcription.py
Normal file
153
podcasts/test_transcription.py
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
import io
|
||||
import json
|
||||
import wave
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from django.contrib.auth.models import User
|
||||
from django.test import TestCase, override_settings
|
||||
|
||||
from feeds import logins
|
||||
|
||||
from . import chapters, transcription
|
||||
from .models import TranscriptionSettings
|
||||
|
||||
SETTINGS = '/podcasts/transcription/settings/'
|
||||
TEST = '/podcasts/transcription/test/'
|
||||
|
||||
|
||||
def fake_response(status=200):
|
||||
r = MagicMock()
|
||||
r.status_code = status
|
||||
r.headers = {}
|
||||
r.raw = None # nothing to introspect for the peer check
|
||||
return r
|
||||
|
||||
|
||||
class SettingsTests(TestCase):
|
||||
def setUp(self):
|
||||
self.user = User.objects.create_user(username='alice', password='pw12345678')
|
||||
self.client.force_login(self.user)
|
||||
|
||||
def post(self, data, url=SETTINGS):
|
||||
return self.client.post(url, json.dumps(data), content_type='application/json')
|
||||
|
||||
def test_requires_login(self):
|
||||
self.client.logout()
|
||||
self.assertEqual(self.client.get(SETTINGS).status_code, 401)
|
||||
self.assertEqual(self.post({}, TEST).status_code, 401)
|
||||
|
||||
def test_defaults(self):
|
||||
self.assertEqual(self.client.get(SETTINGS).json(), {
|
||||
'engine': 'builtin', 'url': '', 'model': '', 'language': '', 'parallel': 3,
|
||||
'has_token': False, 'builtin_available': False})
|
||||
self.assertFalse(TranscriptionSettings.objects.exists())
|
||||
|
||||
@override_settings(WHISPER_URL='http://whisper:8000/v1')
|
||||
def test_builtin_available_follows_the_environment_setting(self):
|
||||
self.assertTrue(self.client.get(SETTINGS).json()['builtin_available'])
|
||||
|
||||
def test_roundtrip(self):
|
||||
r = self.post({'engine': 'custom', 'url': 'https://whisper.example.org/v1/', 'model': 'large-v3',
|
||||
'language': 'de', 'parallel': 5, 'token': 'sk-secret'})
|
||||
self.assertEqual(r.status_code, 200)
|
||||
want = {'engine': 'custom', 'url': 'https://whisper.example.org/v1', 'model': 'large-v3',
|
||||
'language': 'de', 'parallel': 5, 'has_token': True, 'builtin_available': False}
|
||||
self.assertEqual(r.json(), want)
|
||||
self.assertEqual(self.client.get(SETTINGS).json(), want)
|
||||
|
||||
def test_token_is_write_only_and_stored_encrypted(self):
|
||||
self.post({'engine': 'custom', 'url': 'https://w.example.org/v1', 'token': 'sk-secret'})
|
||||
conf = TranscriptionSettings.objects.get(user=self.user)
|
||||
self.assertTrue(conf.token_encrypted)
|
||||
self.assertNotIn('sk-secret', conf.token_encrypted)
|
||||
self.assertEqual(transcription.token_of(conf), 'sk-secret')
|
||||
for body in (self.client.get(SETTINGS).content, self.post({'model': 'm'}).content):
|
||||
self.assertNotIn(b'sk-secret', body)
|
||||
self.assertNotIn(conf.token_encrypted.encode(), body)
|
||||
# Not sending a token keeps it; an empty one removes it.
|
||||
conf.refresh_from_db()
|
||||
self.assertTrue(conf.token_encrypted)
|
||||
self.assertFalse(self.post({'token': ''}).json()['has_token'])
|
||||
conf.refresh_from_db()
|
||||
self.assertEqual(conf.token_encrypted, '')
|
||||
|
||||
def test_validation(self):
|
||||
bad = [
|
||||
{'engine': 'cloud'}, {'parallel': 0}, {'parallel': 9}, {'parallel': '3'}, {'parallel': True},
|
||||
{'url': 'ftp://x.org/v1'}, {'url': 'javascript:alert(1)'}, {'url': 'whisper.local/v1'},
|
||||
{'language': 'Deutsch'}, {'model': 5}, {'token': 5},
|
||||
{'engine': 'custom'}, # a custom instance needs a url
|
||||
]
|
||||
for body in bad:
|
||||
self.assertEqual(self.post(body).status_code, 400, body)
|
||||
self.assertEqual(self.client.post(SETTINGS, 'nope', content_type='application/json').status_code, 400)
|
||||
self.assertFalse(TranscriptionSettings.objects.exists())
|
||||
for body in ({'parallel': 1}, {'parallel': 8}, {'language': ''}, {'language': 'pt-BR'}):
|
||||
self.assertEqual(self.post(body).status_code, 200, body)
|
||||
|
||||
def test_settings_are_per_user(self):
|
||||
self.post({'parallel': 7})
|
||||
other = User.objects.create_user(username='bob', password='pw12345678')
|
||||
self.client.force_login(other)
|
||||
self.assertEqual(self.client.get(SETTINGS).json()['parallel'], 3)
|
||||
|
||||
|
||||
class TestEndpointTests(TestCase):
|
||||
def setUp(self):
|
||||
self.user = User.objects.create_user(username='alice', password='pw12345678')
|
||||
self.client.force_login(self.user)
|
||||
|
||||
def configure(self, **kw):
|
||||
TranscriptionSettings.objects.update_or_create(user=self.user, defaults=kw)
|
||||
|
||||
def test_custom_sends_a_second_of_silence(self):
|
||||
self.configure(engine='custom', url='https://whisper.example.org/v1', model='large-v3', language='de',
|
||||
token_encrypted=logins.encrypt('sk-secret'))
|
||||
calls = []
|
||||
|
||||
def post(url, **kw):
|
||||
calls.append((url, kw))
|
||||
return fake_response()
|
||||
|
||||
with patch('podcasts.chapters.assert_safe_url'), patch('podcasts.chapters.requests.post', post):
|
||||
r = self.client.post(TEST)
|
||||
self.assertEqual(r.json(), {'ok': True, 'model': 'large-v3', 'error': ''})
|
||||
url, kw = calls[0]
|
||||
self.assertEqual(url, 'https://whisper.example.org/v1/audio/transcriptions')
|
||||
self.assertEqual(kw['headers']['Authorization'], 'Bearer sk-secret')
|
||||
self.assertEqual(kw['data']['model'], 'large-v3')
|
||||
self.assertEqual(kw['data']['language'], 'de')
|
||||
name, wav, mime = kw['files']['file']
|
||||
self.assertEqual(mime, 'audio/wav')
|
||||
with wave.open(io.BytesIO(wav)) as w:
|
||||
self.assertEqual((w.getframerate(), w.getnchannels(), w.getnframes()), (16000, 1, 16000))
|
||||
|
||||
def test_custom_private_address_is_refused(self):
|
||||
self.configure(engine='custom', url='http://127.0.0.1:9000/v1')
|
||||
with patch('podcasts.chapters.requests.post') as post:
|
||||
r = self.client.post(TEST)
|
||||
self.assertFalse(r.json()['ok'])
|
||||
self.assertTrue(r.json()['error'])
|
||||
post.assert_not_called()
|
||||
|
||||
def test_custom_http_error_is_reported(self):
|
||||
self.configure(engine='custom', url='https://whisper.example.org/v1')
|
||||
with patch('podcasts.chapters.assert_safe_url'), \
|
||||
patch('podcasts.chapters.requests.post', return_value=fake_response(401)):
|
||||
r = self.client.post(TEST)
|
||||
self.assertEqual((r.json()['ok'], r.json()['error']), (False, 'HTTP 401'))
|
||||
|
||||
@override_settings(WHISPER_URL='http://whisper:8000/v1')
|
||||
def test_builtin_is_exempt_from_the_address_check(self):
|
||||
self.configure(engine='builtin')
|
||||
with patch('podcasts.transcription.requests.post', return_value=fake_response()) as post, \
|
||||
patch('podcasts.chapters.assert_safe_url', side_effect=chapters.WebDAVInputError('private')):
|
||||
r = self.client.post(TEST)
|
||||
self.assertTrue(r.json()['ok'])
|
||||
self.assertEqual(post.call_args.args[0], 'http://whisper:8000/v1/audio/transcriptions')
|
||||
self.assertNotIn('Authorization', post.call_args.kwargs['headers'])
|
||||
|
||||
def test_builtin_without_instance(self):
|
||||
r = self.client.post(TEST)
|
||||
self.assertFalse(r.json()['ok'])
|
||||
self.assertTrue(r.json()['error'])
|
||||
152
podcasts/transcription.py
Normal file
152
podcasts/transcription.py
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
"""Transcription settings and the test of the configured Whisper instance
|
||||
(docs/transcription.md). The job itself is podcasts/transcribe.py."""
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import wave
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import requests
|
||||
from django.conf import settings
|
||||
from django.http import JsonResponse
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
from django.views.decorators.http import require_http_methods
|
||||
|
||||
from feeds import logins
|
||||
|
||||
from . import chapters
|
||||
from .models import TranscriptionSettings
|
||||
|
||||
DEFAULT_MODEL = 'whisper-1'
|
||||
TEST_TIMEOUT = 60 # a cold CPU instance may load its model first
|
||||
LANGUAGE_RE = re.compile(r'^[a-z]{2,3}(-[A-Za-z]{2,4})?$')
|
||||
|
||||
|
||||
def _auth(request):
|
||||
if not request.user.is_authenticated:
|
||||
return JsonResponse({'error': 'authentication required'}, status=401)
|
||||
return None
|
||||
|
||||
|
||||
def _body(request):
|
||||
try:
|
||||
body = json.loads(request.body or b'{}')
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return None
|
||||
return body if isinstance(body, dict) else None
|
||||
|
||||
|
||||
def for_user(user):
|
||||
"""The user's settings (unsaved defaults when there are none)."""
|
||||
return TranscriptionSettings.objects.filter(user=user).first() or TranscriptionSettings(user=user)
|
||||
|
||||
|
||||
def token_of(conf):
|
||||
"""The plain API token ('' when there is none or it can't be read)."""
|
||||
return (logins.decrypt(conf.token_encrypted) or '') if conf.token_encrypted else ''
|
||||
|
||||
|
||||
def _data(conf):
|
||||
return {
|
||||
'engine': conf.engine, 'url': conf.url, 'model': conf.model, 'language': conf.language,
|
||||
'parallel': conf.parallel, 'has_token': bool(conf.token_encrypted),
|
||||
'builtin_available': bool(settings.WHISPER_URL),
|
||||
}
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
@require_http_methods(['GET', 'POST'])
|
||||
def transcription_settings(request):
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
conf = for_user(request.user)
|
||||
if request.method == 'GET':
|
||||
return JsonResponse(_data(conf))
|
||||
body = _body(request)
|
||||
if body is None:
|
||||
return JsonResponse({'error': 'invalid JSON'}, status=400)
|
||||
if 'engine' in body:
|
||||
if body['engine'] not in dict(TranscriptionSettings.ENGINES):
|
||||
return JsonResponse({'error': 'engine must be builtin or custom'}, status=400)
|
||||
conf.engine = body['engine']
|
||||
if 'url' in body:
|
||||
url = body['url']
|
||||
if not isinstance(url, str):
|
||||
return JsonResponse({'error': 'url must be text'}, status=400)
|
||||
url = url.strip().rstrip('/')
|
||||
if url and (urlsplit(url).scheme not in ('http', 'https') or not urlsplit(url).hostname):
|
||||
return JsonResponse({'error': 'url must start with http:// or https://'}, status=400)
|
||||
if len(url) > 500:
|
||||
return JsonResponse({'error': 'url too long'}, status=400)
|
||||
conf.url = url
|
||||
if 'model' in body:
|
||||
if not isinstance(body['model'], str) or len(body['model'].strip()) > 100:
|
||||
return JsonResponse({'error': 'invalid model'}, status=400)
|
||||
conf.model = body['model'].strip()
|
||||
if 'language' in body:
|
||||
lang = body['language']
|
||||
if not isinstance(lang, str) or (lang.strip() and not LANGUAGE_RE.match(lang.strip())):
|
||||
return JsonResponse({'error': 'language must be empty or an ISO code'}, status=400)
|
||||
conf.language = lang.strip()
|
||||
if 'parallel' in body:
|
||||
n = body['parallel']
|
||||
if isinstance(n, bool) or not isinstance(n, int) or not 1 <= n <= 8:
|
||||
return JsonResponse({'error': 'parallel must be 1 to 8'}, status=400)
|
||||
conf.parallel = n
|
||||
if 'token' in body:
|
||||
if not isinstance(body['token'], str):
|
||||
return JsonResponse({'error': 'token must be text'}, status=400)
|
||||
conf.token_encrypted = logins.encrypt(body['token']) if body['token'] else ''
|
||||
if conf.engine == TranscriptionSettings.CUSTOM and not conf.url:
|
||||
return JsonResponse({'error': 'a custom instance needs a url'}, status=400)
|
||||
conf.save()
|
||||
return JsonResponse(_data(conf))
|
||||
|
||||
|
||||
def silent_wav(seconds=1, rate=16000):
|
||||
"""`seconds` of 16-bit mono silence as a WAV file, made in memory."""
|
||||
buf = io.BytesIO()
|
||||
with wave.open(buf, 'wb') as w:
|
||||
w.setnchannels(1)
|
||||
w.setsampwidth(2)
|
||||
w.setframerate(rate)
|
||||
w.writeframes(b'\0\0' * rate * seconds)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
@csrf_exempt
|
||||
@require_http_methods(['POST'])
|
||||
def transcription_test(request):
|
||||
"""One second of silence to the configured instance: {ok, model, error}."""
|
||||
if (err := _auth(request)):
|
||||
return err
|
||||
conf = for_user(request.user)
|
||||
builtin = conf.engine == TranscriptionSettings.BUILTIN
|
||||
base = settings.WHISPER_URL if builtin else conf.url
|
||||
model = conf.model or DEFAULT_MODEL
|
||||
if not base:
|
||||
error = 'Keine eingebaute Instanz eingerichtet' if builtin else 'Keine Adresse eingetragen'
|
||||
return JsonResponse({'ok': False, 'model': model, 'error': error})
|
||||
url = base.rstrip('/') + '/audio/transcriptions'
|
||||
headers = {'User-Agent': 'diora transcription test'}
|
||||
if not builtin and conf.token_encrypted:
|
||||
headers['Authorization'] = 'Bearer ' + token_of(conf)
|
||||
files = {'file': ('silence.wav', silent_wav(), 'audio/wav')}
|
||||
data = {'model': model, 'response_format': 'json'}
|
||||
if conf.language:
|
||||
data['language'] = conf.language
|
||||
try:
|
||||
if builtin:
|
||||
# The operator's own instance (DIORA_WHISPER_URL) may sit on a private network.
|
||||
r = requests.post(url, headers=headers, files=files, data=data, timeout=TEST_TIMEOUT)
|
||||
if r.status_code >= 400:
|
||||
raise chapters.ChaptersInvalid(f'HTTP {r.status_code}')
|
||||
else:
|
||||
r = chapters._open(url, headers, method='POST', timeout=TEST_TIMEOUT, files=files, data=data)
|
||||
r.close()
|
||||
except (chapters.ChaptersInvalid, chapters.ChaptersUnavailable) as e:
|
||||
return JsonResponse({'ok': False, 'model': model, 'error': str(e)})
|
||||
except requests.RequestException as e:
|
||||
return JsonResponse({'ok': False, 'model': model, 'error': e.__class__.__name__})
|
||||
return JsonResponse({'ok': True, 'model': model, 'error': ''})
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
from django.urls import path
|
||||
from . import audiobooks, discover, folders, radio, views
|
||||
from . import article, audiobooks, discover, folders, radio, transcription, views
|
||||
|
||||
urlpatterns = [
|
||||
path('search/', views.podcast_search, name='podcast_search'),
|
||||
|
|
@ -28,9 +28,12 @@ urlpatterns = [
|
|||
path('episodes/discard/', views.discard_episodes, name='podcast_discard_episodes'),
|
||||
path('episodes/<int:pk>/', views.episode_detail, name='podcast_episode_detail'),
|
||||
path('episodes/<int:pk>/transcript/', views.episode_transcript, name='podcast_episode_transcript'),
|
||||
path('episodes/<int:pk>/article/', article.episode_article, name='podcast_episode_article'),
|
||||
path('episodes/<int:pk>/silences/', views.episode_silences, name='podcast_episode_silences'),
|
||||
path('episodes/<int:pk>/sponsor/', views.episode_sponsor, name='podcast_episode_sponsor'),
|
||||
path('episodes/<int:pk>/favorite/', views.set_favorite, name='podcast_set_favorite'),
|
||||
path('transcription/settings/', transcription.transcription_settings, name='podcast_transcription_settings'),
|
||||
path('transcription/test/', transcription.transcription_test, name='podcast_transcription_test'),
|
||||
path('stats/', views.podcast_stats, name='podcast_stats'),
|
||||
path('queue/', views.queue_get, name='podcast_queue_get'),
|
||||
path('queue/add/', views.queue_add, name='podcast_queue_add'),
|
||||
|
|
|
|||
|
|
@ -26,7 +26,8 @@
|
|||
const href = a.getAttribute('href');
|
||||
a.removeAttribute('href');
|
||||
if (href.length < 2) return;
|
||||
if (!_isNoteBacklink(a) && _looksLikeFootnoteLink(a, href)) {
|
||||
// "#t=754": a second in a podcast's article, never a note.
|
||||
if (!/^#t=\d+$/.test(href) && !_isNoteBacklink(a) && _looksLikeFootnoteLink(a, href)) {
|
||||
a.dataset.footnoteRef = href.slice(1);
|
||||
a.dataset.footnoteHref = href;
|
||||
a.style.cursor = 'pointer';
|
||||
|
|
|
|||
|
|
@ -720,6 +720,10 @@
|
|||
if (story) h += '<div class="fd-ov">' + (story.overview ? overviewHtml(story) : '') + '</div>';
|
||||
const content = i.content || '';
|
||||
if (i.image_url && content.indexOf(i.image_url) < 0) h += '<img class="fd-art-img" alt="" src="' + esc(i.image_url) + '">';
|
||||
if (i.episode_id) {
|
||||
// The article of a podcast episode (docs/transcription.md): to the episode itself.
|
||||
h += '<p class="fd-art-listen"><button type="button" class="btn" data-r="episode" data-id="' + i.episode_id + '">▶ Folge anhören</button></p>';
|
||||
}
|
||||
h += '<div class="fd-art-body">' + (content || '<p>' + esc(i.summary) + '</p>') + '</div>';
|
||||
if (i.paywalled) {
|
||||
h += '<p class="fd-note">Nur der freie Teil des Artikels. Mit deiner Abo-Anmeldung (⚙ → ' + esc(i.source_title) +
|
||||
|
|
@ -800,12 +804,72 @@
|
|||
});
|
||||
}
|
||||
|
||||
// ---- the episode of a podcast article (docs/transcription.md)
|
||||
|
||||
function podcastHost() {
|
||||
return window.DioraPodcastHost || window.DioraNative || null;
|
||||
}
|
||||
|
||||
function parseJson(text, fallback) {
|
||||
try { return JSON.parse(text); } catch (e) { return fallback; }
|
||||
}
|
||||
|
||||
// Play the episode from `seconds` (null: open its page, where ▶ starts it). The browser and the
|
||||
// diora desktop have the host of podcasts-web.js; the apps' bridge has the same calls
|
||||
// (podcastPlayEpisodeAt is the one that starts at a place; without it the episode starts at its own place).
|
||||
async function playEpisode(episodeId, seconds) {
|
||||
const host = podcastHost();
|
||||
if (!host || !host.playerState) { alert('Der Podcast-Bereich ist hier nicht verfügbar.'); return; }
|
||||
const now = parseJson(host.playerState(), {});
|
||||
const ms = seconds == null ? null : Math.max(0, Math.round(seconds * 1000));
|
||||
let opened = false;
|
||||
if (now.kind === 'podcast' && now.episodeId === episodeId) {
|
||||
if (ms != null && host.podcastSeek) host.podcastSeek(ms);
|
||||
if (ms != null && !now.playing && host.podcastToggle) host.podcastToggle();
|
||||
opened = ms != null;
|
||||
} else {
|
||||
let d;
|
||||
try {
|
||||
d = await api('/podcasts/episodes/' + episodeId + '/');
|
||||
} catch (e) {
|
||||
alert(e.message === 'Failed to fetch' ? 'Keine Verbindung.' : 'Die Folge ist nicht mehr da.');
|
||||
return;
|
||||
}
|
||||
const row = {id: d.id, title: d.title, audio_url: d.audio_url, duration_seconds: d.duration_seconds,
|
||||
artwork_url: d.artwork_url, pub_date: d.pub_date, position_seconds: d.position_seconds,
|
||||
feed__id: d.feed.id, feed__title: d.feed.title, feed__artwork_url: d.feed.artwork_url};
|
||||
closeReader();
|
||||
if (ms != null && host.podcastPlayEpisodeAt) {
|
||||
host.podcastPlayEpisodeAt(JSON.stringify(row), now.episodeId || 0, ms);
|
||||
opened = true;
|
||||
} else if (ms == null && window.__dioraPodcastOpenEpisode && window.__dioraPodcastOpenEpisode(row)) {
|
||||
return;
|
||||
} else if (host.podcastPlayEpisode) {
|
||||
host.podcastPlayEpisode(JSON.stringify(row), now.episodeId || 0);
|
||||
opened = true;
|
||||
}
|
||||
}
|
||||
closeReader();
|
||||
if (opened && window.__dioraOpenPlayer) window.__dioraOpenPlayer('podcast');
|
||||
}
|
||||
|
||||
// Clicks in the article (65-article.js hands them over): its other reports, the next article.
|
||||
function onReaderClick(e) {
|
||||
// A time mark in a podcast article ("▶ ab 12:34", href="#t=754"; the reader keeps it as data-internal-href).
|
||||
const t = e.target.closest('a[data-internal-href^="#t="], a[href^="#t="]');
|
||||
if (t && readerItem && readerItem.episode_id) {
|
||||
const sec = parseInt((t.dataset.internalHref || t.getAttribute('href')).slice(3), 10);
|
||||
if (sec >= 0) {
|
||||
e.preventDefault();
|
||||
playEpisode(readerItem.episode_id, sec);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const b = e.target.closest('[data-r]');
|
||||
if (!b) return;
|
||||
const r = b.dataset.r;
|
||||
if (r === 'item') openItem(parseInt(b.dataset.id, 10), readerStory);
|
||||
if (r === 'episode') playEpisode(parseInt(b.dataset.id, 10), null);
|
||||
else if (r === 'item') openItem(parseInt(b.dataset.id, 10), readerStory);
|
||||
else if (r === 'story') {
|
||||
const other = readerStory && readerStory.overview && (readerStory.overview.related || []).find(x => x.id === parseInt(b.dataset.id, 10));
|
||||
if (other) openItem((other.lead || other.items[0]).id, other);
|
||||
|
|
@ -1464,6 +1528,7 @@ und Ordner. Schlage nur Änderungen vor, die wirklich helfen; eine Quelle ohne s
|
|||
'.fd-artview .fd-art-title { font-size: 1.45em; line-height: 1.25; margin: 10px 0 6px; font-weight: 700; }',
|
||||
'.fd-artview .fd-art-meta { font-family: var(--font); font-size: 12px; color: var(--nui-muted, #9a9a9a); margin: 0 0 14px; }',
|
||||
'.fd-artview .fd-art-img, .fd-artview .fd-art-body img { display: block; max-width: 100%; height: auto; margin: 12px auto; border-radius: 4px; }',
|
||||
'.fd-artview .fd-art-listen { margin: 0 0 12px; }',
|
||||
'.fd-artview .fd-art-body a { color: var(--accent, #e63946); }',
|
||||
'.fd-artview .fd-art-body figure { margin: 12px 0; }',
|
||||
'.fd-artview .fd-art-body figcaption { font-size: 0.8em; color: var(--nui-muted, #9a9a9a); }',
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue