diff --git a/docs/projects.md b/docs/projects.md index 96078f0..6616a2b 100644 --- a/docs/projects.md +++ b/docs/projects.md @@ -62,9 +62,35 @@ state. A note's text then goes beside it as a copy "(Konflikt)"; other data relo - `GET /projects/doi/?q=` → `{doi, csl}` — fetched from doi.org (Crossref, DataCite …) as CSL-JSON, cached a week, nothing else kept. +## Radar + +A project can have a Radar (`projects/radar.py`, `projects/papers.py`): every week the new papers +from PubMed, arXiv, bioRxiv and medRxiv, sorted by Claude into the config's groups (the first +one — "lesenswert" — with what it's about, how solid it is, the political hook), the week's +news (web search), a basic term explained. One Claude batch (Opus, half price): about 0.40 $ +a week. Ported from the "Neuropolitik Grundlagen" session's `F: +europolitik ools` +(fetch_papers.py, digest_batch.py); the digest is their bundle format (version 1). + +- Config per radar (`Radar.config`, presets in `projects/radar_presets.py`): PubMed queries per + topic, arXiv query, keyword pattern for the preprint servers, groups and their labels, the + prompts, weekday and hour (Europe/Berlin), days. **Plain on the server**: it searches with them. + The papers and digests are public material, plain too (`RadarDigest.bundle`). +- The decisions (gelesen / später / verworfen / in Quellen) are the user's: an encrypted item + of kind `radar` (`{decisions: {paper id: state}}`). "→ Quellen" makes a `paper` source + (citation data from doi.org when there's a DOI, else from the record). +- Runs from the host's crontab every half hour: `docker exec diora-prod python manage.py radar` + (submits what is due, collects finished batches). Needs `ANTHROPIC_API_KEY` in the container. + `radar --setup neuropolitik` gives a project its Radar; `radar --import + ` takes a digest made elsewhere; `radar --seen ` ids reported before. +- API: `GET /projects//radar/` → `{radar: {enabled, title, groups, group_labels, weekday, + hour, days, sources, running, submitted, last_run_at, last_error, next_run, key}, digests: + [{id, generated, bundle}]}` (the latest 12); `POST …/radar/settings/` `{enabled}`; + `POST …/radar/run/` (fetch at the next cron run); `POST …/radar/import/` (a bundle, e.g. + `digest_batch.py --diora-url`). + ## Next -Radar (the weekly digests and new papers with lesen / später / verwerfen / → Quellen; who -fetches the papers is open), papers as books (a paper's PDF into the library, its highlights +Papers as books (a paper's PDF into the library, its highlights into the project's notes), Schreiben (drafts with the bibliography in a citation style), the projects in the search, more presets. diff --git a/e2e/navigation.spec.js b/e2e/navigation.spec.js index 158b3fc..b7e3afc 100644 --- a/e2e/navigation.spec.js +++ b/e2e/navigation.spec.js @@ -137,6 +137,48 @@ test('Projekte: a Forschung project, a paper by DOI, a linked book, a note citin expect(bib).toContain('doi = {10.1038/nn1979}'); }); +test('Radar: the week\'s digest, decisions kept encrypted, a paper into the sources', async ({ page }) => { + await page.setViewportSize({ width: 1200, height: 900 }); + const digest = {id: 1, generated: '2026-10-19', bundle: {version: 1, generated: '2026-10-19', + auf_einen_blick: 'Ein Decoder kommt der Absicht zuvor.', grundlagenbegriff: {begriff: 'Decoder', erklaerung: 'Ein Modell …'}, + news_markdown: '- **Paradromics:** Kooperation. [Quelle](https://example.org/n)', + stats: {records: 3, per_source: {PubMed: 2, arXiv: 1}, errors: [], cost_usd: 0.38}, + items: [ + {id: 'pmid:1', doi: '', source: 'PubMed', title: 'Intention decoded', journal: 'Nature', first_author: 'Verbaarschot', + n_authors: 5, date: '2026', url: 'https://example.org/1', gruppe: 'lesenswert', worum_es_geht: 'Ein Spiel.', + einordnung: 'Preprint.', politischer_haken: 'Libet.', kurz: 'Kurz.'}, + {id: 'arxiv:2', source: 'arXiv', title: 'EEG artefacts', first_author: 'Adeika', n_authors: 2, url: 'https://example.org/2', + gruppe: 'technik', worum_es_geht: '', einordnung: '', politischer_haken: '', kurz: 'Muskeln statt Gedanken.'}, + {id: 'pmid:3', source: 'PubMed', title: 'Off topic', gruppe: 'rauschen', kurz: ''}]}}; + await page.route(/\/projects\/\d+\/radar\/$/, route => route.fulfill({json: {radar: {enabled: true, title: 'Neuropolitik-Digest', + groups: ['lesenswert', 'propriozeption', 'gesellschaft', 'technik', 'rauschen'], group_labels: {lesenswert: 'Lesenswert', technik: 'Technik im Überblick', rauschen: 'Rauschen'}, + weekday: 6, hour: 18, days: 7, sources: ['PubMed', 'arXiv'], running: false, next_run: '2026-10-26T18:00:00+01:00', + last_error: '', key: true}, digests: [digest]}})); + await page.goto('/#projekte'); + const tab = page.locator('#tab-projekte'); + await tab.locator('[data-act="new"]').click(); + await tab.locator('input[name="name"]').fill('Radarprojekt'); + await tab.locator('button[type="submit"]').click(); + await expect(tab.locator('.prj-tab.on')).toContainText('Radar'); + await expect(tab.locator('.prj-radar-h')).toContainText('KW 43'); + await expect(tab.locator('.prj-radar-group[open] > summary')).toHaveText([/Lesenswert/, /Technik/, /Branche/]); + await expect(tab.locator('.prj-paper').first()).toContainText('Politischer Haken: Libet.'); + await tab.locator('[data-radar-decide="arxiv:2"][data-state="later"]').click(); + await tab.locator('[data-radar-decide="pmid:1"][data-state="read"]').click(); + await expect(tab.locator('[data-radar-later]')).toHaveText('Später 1'); + // Kept, encrypted: after a reload still there. + await page.waitForTimeout(800); + await page.reload(); + await expect(tab.locator('[data-radar-later]')).toHaveText('Später 1'); + await expect(tab.locator('.prj-paper').first()).toContainText('gelesen'); + const raw = await page.evaluate(() => fetch(location.pathname.replace(/\/$/, '') + '/projects/').then(r => r.json())); + expect(JSON.stringify(raw)).not.toContain('arxiv:2'); + await tab.locator('[data-radar-source="arxiv:2"]').click(); + await expect(tab.locator('.prj-paper').nth(1)).toContainText('✓ in Quellen'); + await tab.locator('[data-view="sources"]').click(); + await expect(tab.locator('.prj-row .prj-title')).toHaveText(['EEG artefacts']); +}); + test('Abos & Quellen: one page for every subscription, reached from the feeds too', async ({ page }) => { await page.setViewportSize({ width: 1200, height: 800 }); await page.goto('/#news'); diff --git a/projects/management/__init__.py b/projects/management/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/projects/management/commands/__init__.py b/projects/management/commands/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/projects/management/commands/radar.py b/projects/management/commands/radar.py new file mode 100644 index 0000000..0b9ff6f --- /dev/null +++ b/projects/management/commands/radar.py @@ -0,0 +1,47 @@ +"""The Radars (projects/radar.py): collect finished batches, submit what is due. +Run from the host's crontab every half hour: + docker exec diora-prod python manage.py radar + ... radar --setup neuropolitik a project gets a Radar from a preset + ... radar --import a digest made elsewhere (also marks its papers seen) +""" +import json + +from django.core.management.base import BaseCommand, CommandError + +from projects import radar +from projects.models import Project, Radar +from projects.radar_presets import PRESETS + + +class Command(BaseCommand): + help = 'Radar: collect finished batches, submit what is due' + + def add_arguments(self, parser): + parser.add_argument('--setup', nargs=2, metavar=('PROJECT', 'PRESET')) + parser.add_argument('--import', dest='import_', nargs=2, metavar=('PROJECT', 'FILE')) + parser.add_argument('--seen', nargs=2, metavar=('PROJECT', 'FILE'), help='ids reported before (JSON list)') + + def handle(self, *args, setup=None, import_=None, seen=None, **opts): + if setup: + p = Project.objects.filter(pk=setup[0]).first() + if not p or setup[1] not in PRESETS: + raise CommandError('project or preset unknown') + r, made = Radar.objects.update_or_create(project=p, defaults={'config': PRESETS[setup[1]]}) + self.stdout.write(f"radar {p.id}: {'set up' if made else 'config updated'}") + return + if import_: + r = Radar.objects.get(project_id=import_[0]) + with open(import_[1], encoding='utf-8') as f: + d = radar.store(r, json.load(f)) + self.stdout.write(f'radar {r.project_id}: digest {d.id} ({d.generated})') + return + if seen: + r = Radar.objects.get(project_id=seen[0]) + with open(seen[1], encoding='utf-8') as f: + ids = json.load(f) + r.seen = list(dict.fromkeys(list(r.seen) + list(ids)))[-radar.SEEN_MAX:] + r.save(update_fields=['seen']) + self.stdout.write(f'radar {r.project_id}: {len(r.seen)} seen') + return + for line in radar.tick(): + self.stdout.write(line) diff --git a/projects/migrations/0002_radar.py b/projects/migrations/0002_radar.py new file mode 100644 index 0000000..733afb0 --- /dev/null +++ b/projects/migrations/0002_radar.py @@ -0,0 +1,46 @@ +# Generated by Django 6.1 on 2026-10-19 18:32 + +import django.db.models.deletion +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('projects', '0001_initial'), + ] + + operations = [ + migrations.AlterField( + model_name='projectitem', + name='kind', + field=models.CharField(choices=[('book', 'Buch'), ('episode', 'Folge'), ('article', 'Artikel'), ('paper', 'Paper'), ('note', 'Notiz'), ('radar', 'Radar')], max_length=10), + ), + migrations.CreateModel( + name='Radar', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('enabled', models.BooleanField(default=True)), + ('config', models.JSONField(default=dict)), + ('seen', models.JSONField(default=list)), + ('pending', models.JSONField(blank=True, null=True)), + ('last_run_at', models.DateTimeField(blank=True, null=True)), + ('last_error', models.TextField(blank=True)), + ('project', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='radar', to='projects.project')), + ], + ), + migrations.CreateModel( + name='RadarDigest', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('generated', models.DateField()), + ('batch_id', models.CharField(blank=True, max_length=100)), + ('bundle', models.JSONField()), + ('created_at', models.DateTimeField(auto_now_add=True)), + ('radar', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='digests', to='projects.radar')), + ], + options={ + 'ordering': ['-generated', '-created_at'], + }, + ), + ] diff --git a/projects/models.py b/projects/models.py index 93b7349..6c537e5 100644 --- a/projects/models.py +++ b/projects/models.py @@ -34,7 +34,10 @@ class ProjectItem(models.Model): ARTICLE = 'article' PAPER = 'paper' NOTE = 'note' - KINDS = [(BOOK, 'Buch'), (EPISODE, 'Folge'), (ARTICLE, 'Artikel'), (PAPER, 'Paper'), (NOTE, 'Notiz')] + # The Radar's decisions ({paper id: read | later | drop}), one per project. + RADAR = 'radar' + KINDS = [(BOOK, 'Buch'), (EPISODE, 'Folge'), (ARTICLE, 'Artikel'), (PAPER, 'Paper'), (NOTE, 'Notiz'), + (RADAR, 'Radar')] LINKED = (BOOK, EPISODE, ARTICLE) project = models.ForeignKey(Project, on_delete=models.CASCADE, related_name='items') @@ -50,3 +53,29 @@ class ProjectItem(models.Model): constraints = [models.UniqueConstraint(fields=['project', 'kind', 'ref'], name='project_item_once', condition=models.Q(ref__isnull=False))] indexes = [models.Index(fields=['kind', 'ref'])] + + +class Radar(models.Model): + """A project's Radar (docs/projects.md): every week new papers from PubMed, arXiv, bioRxiv, + medRxiv (projects/papers.py), sorted and summed up by Claude in a batch (projects/radar.py). + Plain on the server, unlike the rest of a project: the server searches with the config and + the papers are public.""" + project = models.OneToOneField(Project, on_delete=models.CASCADE, related_name='radar') + enabled = models.BooleanField(default=True) + config = models.JSONField(default=dict) # projects/radar_presets.py + seen = models.JSONField(default=list) # paper ids reported before + pending = models.JSONField(null=True, blank=True) # {batch_id, generated, days, records, errors, submitted} + last_run_at = models.DateTimeField(null=True, blank=True) + last_error = models.TextField(blank=True) + + +class RadarDigest(models.Model): + """One week's digest: the bundle as digest_batch.py makes it (version 1).""" + radar = models.ForeignKey(Radar, on_delete=models.CASCADE, related_name='digests') + generated = models.DateField() + batch_id = models.CharField(max_length=100, blank=True) + bundle = models.JSONField() + created_at = models.DateTimeField(auto_now_add=True) + + class Meta: + ordering = ['-generated', '-created_at'] diff --git a/projects/papers.py b/projects/papers.py new file mode 100644 index 0000000..e583ba3 --- /dev/null +++ b/projects/papers.py @@ -0,0 +1,143 @@ +"""New papers for a project's Radar (docs/projects.md): PubMed, arXiv, bioRxiv, medRxiv. + +What is searched comes from the radar's config (projects/radar_presets.py): PubMed queries per +topic, one arXiv query, the preprint servers filtered by a keyword pattern. collect() returns +(records, errors) and keeps nothing — the caller remembers what was reported, so a failed run +loses nothing. Ported from F:\\neuropolitik\\tools\\fetch_papers.py (Neuropolitik session).""" +import json +import re +import time +import xml.etree.ElementTree as ET +from datetime import date, timedelta + +import requests + +UA = 'diora-radar/1.0 (personal research digest; https://diora.creamfresh.xyz)' +ABSTRACT_CHARS = 1200 +RXIV_SERVERS = ('biorxiv', 'medrxiv') + + +def get(url, params=None, retries=3, timeout=60): + for attempt in range(retries): + try: + r = requests.get(url, params=params, headers={'User-Agent': UA}, timeout=timeout) + r.raise_for_status() + return r.text + except requests.RequestException: + if attempt == retries - 1: + raise + time.sleep(3 * (attempt + 1)) + + +def clip(text): + text = re.sub(r'\s+', ' ', text or '').strip() + return text if len(text) <= ABSTRACT_CHARS else text[:ABSTRACT_CHARS].rsplit(' ', 1)[0] + ' …' + + +def fetch_pubmed(queries, days): + base = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/' + records = [] + for topic, query in queries.items(): + ids = json.loads(get(base + 'esearch.fcgi', {'db': 'pubmed', 'term': query, 'reldate': days, + 'datetype': 'edat', 'retmax': 300, 'retmode': 'json'})) + ids = ids['esearchresult']['idlist'] + for i in range(0, len(ids), 100): + root = ET.fromstring(get(base + 'efetch.fcgi', {'db': 'pubmed', 'id': ','.join(ids[i:i + 100]), + 'retmode': 'xml'})) + for art in root.iter('PubmedArticle'): + pmid = art.findtext('.//PMID') + t = art.find('.//ArticleTitle') + abstract = ' '.join(''.join(a.itertext()) for a in art.findall('.//Abstract/AbstractText')) + doi = next((e.text for e in art.findall('.//ArticleId') if e.get('IdType') == 'doi'), None) + authors = art.findall('.//Author') + records.append({ + 'id': f'pmid:{pmid}', 'doi': doi, 'source': 'PubMed', 'topic': topic, + 'title': clip(''.join(t.itertext()) if t is not None else ''), + 'journal': art.findtext('.//Journal/Title'), + 'first_author': authors[0].findtext('LastName') if authors else None, + 'n_authors': len(authors), 'pub_types': [p.text for p in art.findall('.//PublicationType')], + 'date': art.findtext('.//PubDate/Year'), 'url': f'https://pubmed.ncbi.nlm.nih.gov/{pmid}/', + 'abstract': clip(abstract), + }) + time.sleep(0.4) # NCBI: 3 requests a second without a key + return records + + +def fetch_arxiv(query, days): + cutoff = (date.today() - timedelta(days=days)).isoformat() + xml = get('https://export.arxiv.org/api/query', {'search_query': query, 'sortBy': 'submittedDate', + 'sortOrder': 'descending', 'max_results': 150}) + ns = {'a': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'} + records = [] + for e in ET.fromstring(xml).findall('a:entry', ns): + published = e.findtext('a:published', '', ns)[:10] + if published < cutoff: + continue + arxiv_id = e.findtext('a:id', '', ns).rsplit('/abs/', 1)[-1] + cat = e.find('arxiv:primary_category', ns) + authors = e.findall('a:author', ns) + records.append({ + 'id': 'arxiv:' + re.sub(r'v\d+$', '', arxiv_id), 'doi': None, 'source': 'arXiv', 'topic': None, + 'title': clip(e.findtext('a:title', '', ns)), + 'journal': f"arXiv {cat.get('term') if cat is not None else ''}".strip(), + 'first_author': authors[0].findtext('a:name', '', ns) if authors else None, + 'n_authors': len(authors), 'pub_types': ['Preprint'], 'date': published, + 'url': f'https://arxiv.org/abs/{arxiv_id}', 'abstract': clip(e.findtext('a:summary', '', ns)), + }) + return records + + +def fetch_rxiv(server, days, keywords): + start = (date.today() - timedelta(days=days)).isoformat() + end = date.today().isoformat() + records, cursor, total = [], 0, None + while total is None or cursor < total: + data = json.loads(get(f'https://api.biorxiv.org/details/{server}/{start}/{end}/{cursor}')) + msg = data['messages'][0] + if msg.get('status') != 'ok': + break + total = int(msg.get('total', 0)) + batch = data.get('collection', []) + if not batch: + break + for p in batch: + if not keywords.search(f"{p.get('title', '')} {p.get('abstract', '')}"): + continue + authors = (p.get('authors') or '').split(';') + records.append({ + 'id': 'doi:' + p['doi'], 'doi': p['doi'], 'source': server, 'topic': None, + 'title': clip(p.get('title')), 'journal': f"{server} ({p.get('category')})", + 'first_author': authors[0], 'n_authors': len(authors), 'pub_types': ['Preprint'], + 'date': p.get('date'), 'url': f"https://www.{server}.org/content/{p['doi']}v{p.get('version', 1)}", + 'abstract': clip(p.get('abstract')), 'institution': p.get('author_corresponding_institution'), + }) + cursor += len(batch) + time.sleep(0.3) + return records + + +def collect(config, days, seen=frozenset()): + """Every source of the config; doubles (same id, DOI or title) and what was reported + before (seen) left out. (records, errors).""" + keywords = re.compile(config.get('keywords') or r'(?!)', re.I) + jobs = [] + if config.get('pubmed'): + jobs.append(('PubMed', lambda: fetch_pubmed(config['pubmed'], days))) + if config.get('arxiv'): + jobs.append(('arXiv', lambda: fetch_arxiv(config['arxiv'], days))) + for server in config.get('rxiv', RXIV_SERVERS) if config.get('keywords') else (): + jobs.append((server, lambda s=server: fetch_rxiv(s, days, keywords))) + records, errors = [], [] + for name, fn in jobs: + try: + records += fn() + except Exception as e: # noqa: BLE001 - a source down mustn't stop the others + errors.append(f'{name}: {e}') + unique, keys = [], set() + for r in records: + k = {r['id'], (r['doi'] or '').lower(), re.sub(r'\W', '', r['title'].lower())} - {''} + if keys & k or r['id'] in seen or (r['doi'] and f"doi:{r['doi']}" in seen): + continue + keys |= k + unique.append(r) + return unique, errors diff --git a/projects/radar.py b/projects/radar.py new file mode 100644 index 0000000..5b1ebbc --- /dev/null +++ b/projects/radar.py @@ -0,0 +1,379 @@ +"""The Radar of a project (docs/projects.md): once a week the new papers (projects/papers.py) +go to Claude in a batch (half price, done within the hour as a rule): one request sorts every +paper into the config's groups and writes about the worth-reading ones (JSON schema), one looks +for the week's news (web search). The answer becomes a RadarDigest — the bundle format of +F:\\neuropolitik\\tools\\digest_batch.py (version 1), so a digest made elsewhere can come in too. + +Runs from the host's crontab: `manage.py radar` every half hour submits what is due and +collects what is done. Needs ANTHROPIC_API_KEY in the container's environment. +""" +import json +import os +from datetime import date, timedelta +from zoneinfo import ZoneInfo + +from django.db import transaction +from django.http import JsonResponse +from django.utils import timezone +from django.views.decorators.csrf import csrf_exempt +from django.views.decorators.http import require_http_methods + +from . import papers +from .models import Project, Radar, RadarDigest + +DEFAULT_MODEL = 'claude-opus-5-5' +# Batch prices: half of Opus 5.5's list prices ($4 / $20 per million tokens); web search $10 / 1000. +PRICE_IN, PRICE_OUT, PRICE_SEARCH = 2.00 / 1e6, 10.00 / 1e6, 10.00 / 1000 +SEEN_MAX = 20000 +# The weekday and hour of a config are the user's (the server runs on UTC). +ZONE = ZoneInfo('Europe/Berlin') +DIGESTS_SHOWN = 12 + + +def papers_schema(groups): + return { + 'type': 'object', + 'properties': { + 'auf_einen_blick': {'type': 'string'}, + 'items': {'type': 'array', 'items': { + 'type': 'object', + 'properties': { + 'id': {'type': 'string'}, + 'gruppe': {'type': 'string', 'enum': list(groups)}, + 'worum_es_geht': {'type': 'string'}, + 'einordnung': {'type': 'string'}, + 'politischer_haken': {'type': 'string'}, + 'kurz': {'type': 'string'}, + }, + 'required': ['id', 'gruppe', 'worum_es_geht', 'einordnung', 'politischer_haken', 'kurz'], + 'additionalProperties': False, + }}, + 'grundlagenbegriff': { + 'type': 'object', + 'properties': {'begriff': {'type': 'string'}, 'erklaerung': {'type': 'string'}}, + 'required': ['begriff', 'erklaerung'], + 'additionalProperties': False, + }, + }, + 'required': ['auf_einen_blick', 'items', 'grundlagenbegriff'], + 'additionalProperties': False, + } + + +def _model_record(r): + """What the model needs of a paper to sort it.""" + return {k: r.get(k) for k in ('id', 'source', 'topic', 'journal', 'first_author', 'n_authors', 'pub_types', + 'date', 'title', 'abstract', 'institution') if r.get(k) not in (None, '', [])} + + +def _terms(radar): + """The basic terms explained before: not again.""" + out = [] + for b in radar.digests.values_list('bundle', flat=True)[:52]: + t = ((b or {}).get('grundlagenbegriff') or {}).get('begriff') + if t: + out.append(t) + return out + + +def build_requests(config, records, terms, today): + model = config.get('model') or DEFAULT_MODEL + groups = config.get('groups') or ['lesenswert', 'rauschen'] + prompt = config['papers_task'].format( + n=len(records), glossar=', '.join(terms) if terms else '(noch keine)', + records=json.dumps([_model_record(r) for r in records], ensure_ascii=False)) + reqs = [{'custom_id': 'papers', 'params': { + 'model': model, 'max_tokens': 48000, 'thinking': {'type': 'adaptive'}, + 'output_config': {'effort': 'high', 'format': {'type': 'json_schema', 'schema': papers_schema(groups)}}, + 'system': config['system'], 'messages': [{'role': 'user', 'content': prompt}], + }}] + if config.get('news_task'): + reqs.append({'custom_id': 'news', 'params': { + 'model': model, 'max_tokens': 16000, 'thinking': {'type': 'adaptive'}, + 'output_config': {'effort': 'medium'}, 'system': config['system'], + 'tools': [{'type': 'web_search_20260209', 'name': 'web_search', 'max_uses': 4}], + 'messages': [{'role': 'user', 'content': config['news_task'].format(today=today)}], + }}) + return reqs + + +def build_bundle(config, pending, papers_json, news_md, usage, batch_id, project_name=''): + groups = config.get('groups') or ['lesenswert', 'rauschen'] + records = {r['id']: r for r in pending['records']} + items, seen = [], set() + for it in papers_json.get('items') or []: + r = records.get(it.get('id')) + if not r or it['id'] in seen or it.get('gruppe') not in groups: + continue # an id or group the model made up, or twice: left out + seen.add(it['id']) + items.append({**r, **it}) + # What the model forgot shows, unsorted, among the rest. + fallback = 'technik' if 'technik' in groups else groups[-1] + for rid, r in records.items(): + if rid not in seen: + items.append({**r, 'gruppe': fallback, 'worum_es_geht': '', 'einordnung': '', 'politischer_haken': '', + 'kurz': '(nicht eingeordnet)'}) + per_source = {} + for r in pending['records']: + per_source[r['source']] = per_source.get(r['source'], 0) + 1 + cost = (usage['input_tokens'] * PRICE_IN + usage['output_tokens'] * PRICE_OUT + + usage['web_search_requests'] * PRICE_SEARCH) + return { + 'version': 1, 'project': project_name, 'generated': pending['generated'], 'days': pending['days'], + 'model': config.get('model') or DEFAULT_MODEL, 'batch_id': batch_id, + 'auf_einen_blick': papers_json.get('auf_einen_blick', ''), + 'grundlagenbegriff': papers_json.get('grundlagenbegriff') or {'begriff': '', 'erklaerung': ''}, + 'items': items, 'news_markdown': news_md, + 'stats': {'records': len(records), 'per_source': per_source, 'errors': pending.get('errors', []), + 'usage': usage, 'cost_usd': round(cost, 4)}, + } + + +def _client(): + import anthropic # only where the Radar runs + return anthropic.Anthropic() + + +# ---- running --------------------------------------------------------------------------------- + +def due(radar, now=None): + """Its weekday and hour have come (local time), and this week has no digest yet.""" + if not radar.enabled or radar.pending or not radar.config.get('papers_task'): + return False + now = (now or timezone.now()).astimezone(ZONE) + c = radar.config + weekday, hour = int(c.get('weekday', 6)), int(c.get('hour', 18)) + start = (now - timedelta(days=(now.weekday() - weekday) % 7)).replace(hour=hour, minute=0, second=0, + microsecond=0) + if start > now: + start -= timedelta(days=7) + last = radar.last_run_at.astimezone(ZONE) if radar.last_run_at else None + return last is None or last < start + + +def submit(radar, force=False): + """Fetch the new papers and send the batch. Returns what happened, in words.""" + if radar.pending and not force: + return 'Es läuft schon ein Abruf.' + c = radar.config + days = int(c.get('days', 7)) + records, errors = papers.collect(c, days, set(radar.seen)) + radar.last_run_at = timezone.now() + today = date.today().isoformat() + if not records: + radar.last_error = '; '.join(errors) + radar.pending = None + radar.save(update_fields=['last_run_at', 'last_error', 'pending']) + return 'Keine neuen Papers.' + (' Fehler: ' + radar.last_error if errors else '') + batch = _client().messages.batches.create(requests=build_requests(c, records, _terms(radar), today)) + radar.pending = {'batch_id': batch.id, 'generated': today, 'days': days, 'records': records, + 'errors': errors, 'submitted': timezone.now().isoformat(timespec='seconds')} + radar.last_error = '' + radar.save(update_fields=['last_run_at', 'last_error', 'pending']) + return f'{len(records)} Papers an Claude geschickt ({batch.id}).' + + +def _text(message): + return ''.join(b.text for b in message.content if b.type == 'text') + + +def _add_usage(total, message): + u = message.usage + total['input_tokens'] += (u.input_tokens or 0) + (getattr(u, 'cache_read_input_tokens', 0) or 0) \ + + (getattr(u, 'cache_creation_input_tokens', 0) or 0) + total['output_tokens'] += u.output_tokens or 0 + stu = getattr(u, 'server_tool_use', None) + if stu: + total['web_search_requests'] += getattr(stu, 'web_search_requests', 0) or 0 + + +def collect(radar): + """The batch's answer, if it's there: a digest. Returns 'done', 'waiting' or an error text.""" + p = radar.pending + if not p: + return 'done' + client = _client() + batch = client.messages.batches.retrieve(p['batch_id']) + if batch.processing_status != 'ended': + return 'waiting' + usage = {'input_tokens': 0, 'output_tokens': 0, 'web_search_requests': 0} + papers_json, news_md, problems = None, '', [] + for res in client.messages.batches.results(batch.id): + if res.result.type != 'succeeded': + detail = getattr(getattr(res.result, 'error', None), 'type', '') + problems.append(f'{res.custom_id}: {res.result.type} {detail}'.strip()) + continue + msg = res.result.message + _add_usage(usage, msg) + if msg.stop_reason == 'refusal': + problems.append(f'{res.custom_id}: abgelehnt') + elif res.custom_id == 'papers': + if msg.stop_reason == 'max_tokens': + problems.append('papers: max_tokens erreicht, Ausgabe unvollständig') + else: + try: + papers_json = json.loads(_text(msg)) + except ValueError: + problems.append('papers: keine gültige JSON-Antwort') + elif res.custom_id == 'news': + news_md = _text(msg) + if papers_json is None: + # Nothing remembered as seen: the next run gets the papers again. + radar.pending = None + radar.last_error = 'Einordnung fehlgeschlagen: ' + '; '.join(problems) + radar.save(update_fields=['pending', 'last_error']) + return radar.last_error + p['errors'] = list(p.get('errors') or []) + problems + bundle = build_bundle(radar.config, p, papers_json, news_md, usage, batch.id) + store(radar, bundle, mark_seen=[r['id'] for r in p['records']]) + return 'done' + + +def store(radar, bundle, mark_seen=None): + """A digest in; what it reports counts as seen. Replaces one with the same batch.""" + with transaction.atomic(): + if bundle.get('batch_id'): + radar.digests.filter(batch_id=bundle['batch_id']).delete() + d = RadarDigest.objects.create(radar=radar, generated=date.fromisoformat(str(bundle['generated'])[:10]), + batch_id=bundle.get('batch_id') or '', bundle=bundle) + seen = list(dict.fromkeys(list(radar.seen) + list(mark_seen or [i.get('id') for i in bundle.get('items', [])]))) + radar.seen = [s for s in seen if s][-SEEN_MAX:] + radar.pending = None + radar.last_error = '' + radar.save(update_fields=['seen', 'pending', 'last_error']) + return d + + +def tick(now=None): + """What the cron job does: collect what's done, submit what's due. Returns lines to log.""" + out = [] + for radar in Radar.objects.filter(enabled=True).select_related('project'): + try: + if radar.pending: + r = collect(radar) + if r != 'waiting': + out.append(f'radar {radar.project_id}: {r}') + elif due(radar, now): + out.append(f'radar {radar.project_id}: {submit(radar)}') + except Exception as e: # noqa: BLE001 - one radar's trouble mustn't stop the others + radar.last_error = str(e)[:2000] + radar.save(update_fields=['last_error']) + out.append(f'radar {radar.project_id}: Fehler {e}') + return out + + +# ---- API --------------------------------------------------------------------------------------- + +def _auth(request): + if not request.user.is_authenticated: + return JsonResponse({'error': 'authentication required'}, status=401) + return None + + +def next_run(radar, now=None): + """When the cron job will fetch next: now (it's due), else the coming weekday and hour.""" + if not radar.enabled: + return None + now = (now or timezone.now()).astimezone(ZONE) + if radar.pending: + return None + if due(radar, now): + return now.isoformat() + c = radar.config + weekday, hour = int(c.get('weekday', 6)), int(c.get('hour', 18)) + t = (now + timedelta(days=(weekday - now.weekday()) % 7)).replace(hour=hour, minute=0, second=0, microsecond=0) + if t <= now: + t += timedelta(days=7) + return t.isoformat() + + +def _radar_json(radar): + c = radar.config + return {'enabled': radar.enabled, 'title': c.get('title', 'Radar'), 'groups': c.get('groups', []), + 'group_labels': c.get('group_labels', {}), 'weekday': c.get('weekday', 6), 'hour': c.get('hour', 18), + 'days': c.get('days', 7), 'sources': [s for s, on in (('PubMed', c.get('pubmed')), ('arXiv', c.get('arxiv')), + ('bioRxiv/medRxiv', c.get('keywords'))) if on], + 'running': bool(radar.pending), 'submitted': (radar.pending or {}).get('submitted'), + 'last_run_at': radar.last_run_at.isoformat() if radar.last_run_at else None, + 'last_error': radar.last_error, 'next_run': next_run(radar), + 'key': bool(os.environ.get('ANTHROPIC_API_KEY'))} + + +def _digest_json(d): + return {'id': d.id, 'generated': d.generated.isoformat(), 'bundle': d.bundle} + + +@require_http_methods(['GET']) +def radar_view(request, pk): + """The project's Radar and its latest digests (or radar: null).""" + err = _auth(request) + if err: + return err + p = Project.objects.filter(pk=pk, user=request.user).first() + if not p: + return JsonResponse({'error': 'not found'}, status=404) + radar = Radar.objects.filter(project=p).first() + if not radar: + return JsonResponse({'radar': None, 'digests': []}) + return JsonResponse({'radar': _radar_json(radar), + 'digests': [_digest_json(d) for d in radar.digests.all()[:DIGESTS_SHOWN]]}) + + +@csrf_exempt +@require_http_methods(['POST']) +def radar_settings(request, pk): + """{enabled}.""" + err = _auth(request) + if err: + return err + radar = Radar.objects.filter(project__pk=pk, project__user=request.user).first() + if not radar: + return JsonResponse({'error': 'not found'}, status=404) + try: + body = json.loads(request.body or b'{}') + except ValueError: + return JsonResponse({'error': 'invalid JSON'}, status=400) + if 'enabled' in body: + radar.enabled = bool(body['enabled']) + radar.save(update_fields=['enabled']) + return JsonResponse({'radar': _radar_json(radar)}) + + +@csrf_exempt +@require_http_methods(['POST']) +def radar_run(request, pk): + """Fetch now rather than on the weekday: the cron job takes it within half an hour.""" + err = _auth(request) + if err: + return err + radar = Radar.objects.filter(project__pk=pk, project__user=request.user).first() + if not radar: + return JsonResponse({'error': 'not found'}, status=404) + if not radar.pending: + radar.last_run_at = None + radar.enabled = True + radar.save(update_fields=['last_run_at', 'enabled']) + return JsonResponse({'radar': _radar_json(radar)}) + + +@csrf_exempt +@require_http_methods(['POST']) +def radar_import(request, pk): + """A digest made elsewhere (digest_batch.py --diora-url, Bearer token): the bundle.""" + err = _auth(request) + if err: + return err + radar = Radar.objects.filter(project__pk=pk, project__user=request.user).first() + if not radar: + return JsonResponse({'error': 'not found'}, status=404) + try: + bundle = json.loads(request.body or b'{}') + date.fromisoformat(str(bundle['generated'])[:10]) + if bundle.get('version') != 1 or not isinstance(bundle.get('items'), list): + raise ValueError + except (ValueError, KeyError, TypeError): + return JsonResponse({'error': 'kein Digest (version 1)'}, status=400) + if len(request.body) > 5_000_000: + return JsonResponse({'error': 'zu groß'}, status=400) + d = store(radar, bundle) + return JsonResponse({'digest': d.id}) diff --git a/projects/radar_presets.py b/projects/radar_presets.py new file mode 100644 index 0000000..263d348 --- /dev/null +++ b/projects/radar_presets.py @@ -0,0 +1,120 @@ +"""Radar configurations (docs/projects.md): what a project's Radar searches and how Claude +sorts it. NEUROPOLITIK is taken as it was from F:\neuropolitik\tools (fetch_papers.py, +digest_batch.py, the "Neuropolitik Grundlagen" session); generated, edit with care: papers_task +keeps {n}, {glossar} and {records}, news_task {today}.""" + +# flake8: noqa: E501 +NEUROPOLITIK = {'title': 'Neuropolitik-Digest', + 'model': 'claude-opus-5-5', + 'days': 7, + 'weekday': 6, + 'hour': 18, + 'pubmed': {'technik': '"brain-computer interface*"[tiab] OR "brain computer interface*"[tiab] OR ' + '"brain-machine interface*"[tiab] OR "neural interface*"[tiab] OR neuroprosthe*[tiab] ' + 'OR "speech decoding"[tiab] OR "Brain-Computer Interfaces"[Mesh]', + 'gesellschaft': 'neuroethic*[tiab] OR neuroright*[tiab] OR "neural data"[tiab] OR "mental ' + 'privacy"[tiab] OR "cognitive liberty"[tiab] OR "neurotechnolog*"[tiab] OR ' + '"neural privacy"[tiab]', + 'propriozeption': '(propriocept*[tiab] OR "sensory feedback"[tiab] OR "somatosensory ' + 'feedback"[tiab] OR "artificial touch"[tiab] OR "intracortical ' + 'microstimulation"[tiab] OR embodiment[tiab] OR "body schema"[tiab]) AND ' + '(prosthe*[tiab] OR neuroprosthe*[tiab] OR bionic[tiab] OR "neural ' + 'interface*"[tiab] OR "brain-computer"[tiab] OR "brain-machine"[tiab] OR ' + '"nerve stimulation"[tiab] OR "spinal cord stimulation"[tiab] OR ' + 'exoskeleton*[tiab])'}, + 'arxiv': 'abs:"brain-computer interface" OR abs:"brain-machine interface" OR abs:"brain computer interface" ' + 'OR abs:"neural interface" OR abs:neurotechnology OR abs:"neural decoding" OR abs:neuroprosthesis ' + 'OR (abs:proprioception AND (abs:prosthesis OR abs:prosthetic OR abs:"neural interface")) OR ' + 'abs:"intracortical microstimulation"', + 'keywords': 'brain[- ]?(computer|machine) interface|\\bBCIs?\\b|neuroprosthe|neural interface|intracortical ' + '(decod|record|microstim|array|implant)|neural decod|speech decod|brain ' + 'decod|neurotechnolog|neuroethic|neuroright|neural data|mental privacy|cognitive ' + 'liberty|closed[- ]loop (neuro)?(stimulation|modulation)|adaptive deep brain ' + 'stimulation|brain[- ]to[- ](text|speech|image)|intracortical microstimulation|artificial ' + '(touch|sensation|proprioception)|propriocept\\w* ' + '(feedback|restoration|neuroprosthe)|bidirectional (BCI|brain|neural)|bionic ' + '(limb|leg|arm|hand)|myoneural|somatosensory (feedback|neuroprosthe)|(sensory|somatosensory) ' + 'feedback[^.]{0,60}prosthe', + 'rxiv': ['biorxiv', 'medrxiv'], + 'groups': ['lesenswert', 'propriozeption', 'gesellschaft', 'technik', 'rauschen'], + 'group_labels': {'lesenswert': 'Lesenswert', + 'propriozeption': 'Propriozeption und Körper', + 'gesellschaft': 'Gesellschaft, Ethik, Recht', + 'technik': 'Technik im Überblick', + 'rauschen': 'Rauschen'}, + 'system': 'Du schreibst den wöchentlichen Digest für „Neuropolitik“, ein geplantes Publikationsprojekt ' + '(Blog/Podcast) über Brain-Computer-Interfaces (BCIs), Neurotechnologie, Propriozeption und ' + 'Neuroprothetik und ihre Auswirkungen auf die Gesellschaft – aus einer linken Perspektive ' + '(Marxismus, Ljubljana-Schule: Žižek, Zupančič, Dolar; dazu Malabou, Johnston, Pasquinelli, ' + 'Rose/Abi-Rached).\n' + '\n' + 'Die Person hinter dem Projekt ist theoretisch geschult, hat aber keine neurowissenschaftliche ' + 'Ausbildung und arbeitet sich gerade in die Grundlagen ein. Theorie musst du nicht erklären, ' + 'Technik schon – knapp und präzise.\n' + '\n' + 'Ein eigener Schwerpunkt ist Propriozeption: künstliche Rückmeldung von Körperlage und Berührung ' + '(intrakortikale Mikrostimulation, bidirektionale BCIs, Nervenschnittstellen in Prothesen, ' + 'myoneurale Schnittstellen), Embodiment und Körperschema bei Prothesen und Exoskeletten, Verlust ' + 'und Wiederherstellung der Eigenwahrnehmung.\n' + '\n' + 'Regeln:\n' + '- Schreib auf Deutsch, sachlich und präzise. Die linke Perspektive gehört nur in den ' + '„politischen Haken“.\n' + '- Behaupte nichts, was nicht in der Quelle steht. Du kennst von den Papers nur Titel und ' + '(gekürztes) Abstract – schreib „laut Abstract“, wo das wichtig ist. Erfinde keine Zahlen, ' + 'Autor:innen oder Ergebnisse.\n' + '- Gib Abstracts nicht wörtlich wieder, sondern fasse in eigenen Worten zusammen.', + 'papers_task': 'Ordne jedes der folgenden {n} Papers genau einer Gruppe zu und schreib die passenden Texte. ' + 'Jede ID muss genau einmal in `items` vorkommen.\n' + '\n' + 'Gruppen:\n' + '- lesenswert (höchstens 6, möglichst 3–6): Studien an Menschen und klinische BCI-Studien; ' + 'alles zu Propriozeption und sensorischer Rückmeldung; Decoding von Sprache, innerer ' + 'Sprache, Bildern oder Bedeutung (besonders nicht-invasiv: EEG, fMRT, MEG, fNIRS); ' + '„schreibende“ Neurotechnik (Stimulation, Closed-Loop, Tiefe Hirnstimulation mit BCI-Bezug); ' + 'Durchbrüche bei Implantaten; Papers kommerzieller Akteure (Neuralink, Synchron, ' + 'Paradromics, Precision Neuroscience, Blackrock Neurotech, Merge Labs, Meta, Apple, ' + 'chinesische BCI-Firmen); Einsätze in Militär, Arbeitswelt oder Konsumprodukten; neuronale ' + 'Daten; große Reviews. Was politisch oder gesellschaftlich Gewicht hat, geht vor rein ' + 'inkrementeller Technik.\n' + '- propriozeption: Papers mit Propriozeptions-Bezug, die nicht lesenswert sind.\n' + '- gesellschaft: Ethik, Recht, Regulierung, Neurorechte, Datenschutz, Disability Studies, ' + 'Wissenschaftsforschung (STS), Philosophie.\n' + '- technik: übrige relevante BCI- und Neuroprothetik-Technik.\n' + '- rauschen: kein echter BCI- oder Neurotech-Bezug (Schlagwort-Fehltreffer) oder sehr ' + 'inkrementelle ML-Arbeiten zur EEG-Klassifikation.\n' + '\n' + 'Felder je Paper:\n' + '- lesenswert: `worum_es_geht` (2–3 Sätze), `einordnung` (Studientyp: Peer Review oder ' + 'Preprint, Mensch/Tier/Simulation, Zahl der Teilnehmenden falls genannt; invasiv oder ' + 'nicht-invasiv; lesend oder schreibend; Hype-Check: was zeigt das Paper wirklich, was ' + 'nicht), `politischer_haken` (1–2 Sätze als Anregung, nicht als fertige These: Eigentum und ' + 'Plattformabhängigkeit, Arbeit und Überwachung, neuronale Daten als Ware, Militär, ' + 'Behinderung und Legitimationsnarrative, Ideologie des „Gehirnsubjekts“, Gedankenlesen und ' + 'Subjektivität, Körperschema und Eigenleib …) und `kurz` (ein Satz).\n' + '- propriozeption, gesellschaft: `kurz` (1–2 Sätze), die übrigen Felder leer.\n' + '- technik: `kurz` (eine Zeile: was daran neu ist), die übrigen Felder leer.\n' + '- rauschen: alle Textfelder leer.\n' + '\n' + 'Außerdem:\n' + '- `auf_einen_blick`: 3–5 Sätze – was war diese Woche wichtig, gibt es einen Trend?\n' + '- `grundlagenbegriff`: Erkläre in 3–5 Sätzen einen technischen Begriff, der in den Papers ' + 'dieser Woche vorkommt und für Leser:innen mit Theorie-, aber ohne Neuro-Hintergrund wichtig ' + 'ist. Nicht wiederholen: {glossar}.\n' + '\n' + 'Bei einer ruhigen Woche fällt alles kürzer aus. Nichts künstlich aufblähen.\n' + '\n' + 'Papers (JSON):\n' + '{records}', + 'news_task': 'Recherchiere mit höchstens 4 Websuchen Nachrichten der letzten 7 Tage (Stichtag {today}) zu ' + 'Neurotechnologie und BCIs: Neuralink, Synchron und andere BCI-Firmen samt Finanzierung, ' + 'Gesetze zu neuronalen Daten und Neurorechten (USA, EU, Chile, UNESCO), Chinas ' + 'BCI-Industriepolitik, DARPA und Militär.\n' + '\n' + 'Antworte nur mit einer Markdown-Liste, ohne Einleitung, ein Eintrag pro Meldung:\n' + '- **Kurztitel:** 1–2 Sätze. [Quelle](URL)\n' + '\n' + 'Nimm nur Meldungen auf, deren Quelle du in den Suchergebnissen gesehen hast, und nur aus den ' + 'letzten 7 Tagen. Gibt es nichts Relevantes, antworte genau: Keine relevanten Meldungen.'} + +PRESETS = {'neuropolitik': NEUROPOLITIK} diff --git a/projects/test_radar.py b/projects/test_radar.py new file mode 100644 index 0000000..4c40376 --- /dev/null +++ b/projects/test_radar.py @@ -0,0 +1,144 @@ +"""The Radar (projects/radar.py, projects/papers.py) with a pretend Claude.""" +import json +from datetime import datetime +from types import SimpleNamespace as NS +from unittest.mock import patch +from zoneinfo import ZoneInfo + +from django.contrib.auth.models import User +from django.test import TestCase + +from . import papers, radar +from .models import Project, Radar +from .radar_presets import NEUROPOLITIK + +BERLIN = ZoneInfo('Europe/Berlin') +REC = [ + {'id': 'pmid:1', 'doi': '10.1/a', 'source': 'PubMed', 'topic': 'technik', 'title': 'Speech BCI', + 'journal': 'Nature', 'first_author': 'Willett', 'n_authors': 9, 'pub_types': [], 'date': '2026', + 'url': 'https://pubmed.ncbi.nlm.nih.gov/1/', 'abstract': 'A speech neuroprosthesis.'}, + {'id': 'arxiv:2610.1', 'doi': None, 'source': 'arXiv', 'topic': None, 'title': 'EEG noise', + 'journal': 'arXiv eess.SP', 'first_author': 'A', 'n_authors': 2, 'pub_types': ['Preprint'], + 'date': '2026-10-18', 'url': 'https://arxiv.org/abs/2610.1', 'abstract': 'EEG.'}, +] + + +def message(text, stop='end_turn', searches=0): + return NS(content=[NS(type='text', text=text)], stop_reason=stop, + usage=NS(input_tokens=1000, output_tokens=200, cache_read_input_tokens=0, cache_creation_input_tokens=0, + server_tool_use=NS(web_search_requests=searches) if searches else None)) + + +class FakeClient: + def __init__(self, status='ended', papers_text=None): + self.created = [] + self.status = status + self.papers_text = papers_text or json.dumps({ + 'auf_einen_blick': 'Eine ruhige Woche.', + 'grundlagenbegriff': {'begriff': 'Decoder', 'erklaerung': 'Ein Modell, das …'}, + 'items': [{'id': 'pmid:1', 'gruppe': 'lesenswert', 'worum_es_geht': 'Sprache.', 'einordnung': 'Peer Review.', + 'politischer_haken': 'Stimme.', 'kurz': 'Sprache aus dem Kortex.'}, + {'id': 'erfunden', 'gruppe': 'technik', 'worum_es_geht': '', 'einordnung': '', + 'politischer_haken': '', 'kurz': 'x'}]}) + outer = self + + class Batches: + def create(self, requests): + outer.created.append(requests) + return NS(id='msgbatch_1', processing_status='in_progress') + + def retrieve(self, bid): + return NS(id=bid, processing_status=outer.status) + + def results(self, bid): + return [NS(custom_id='papers', result=NS(type='succeeded', message=message(outer.papers_text))), + NS(custom_id='news', result=NS(type='succeeded', message=message('- **X:** y. [Q](https://q)', searches=2)))] + self.messages = NS(batches=Batches()) + + +class RadarTests(TestCase): + def setUp(self): + self.user = User.objects.create_user(username='radar', password='pw12345678') + self.client.force_login(self.user) + self.project = Project.objects.create(user=self.user, data_ct='x', data_iv='a1' * 12) + self.radar = Radar.objects.create(project=self.project, config=NEUROPOLITIK) + + def test_due_on_its_weekday_once_a_week(self): + sat = datetime(2026, 10, 24, 20, 0, tzinfo=BERLIN) + sun_early = datetime(2026, 10, 25, 17, 0, tzinfo=BERLIN) + sun = datetime(2026, 10, 25, 18, 5, tzinfo=BERLIN) + self.radar.last_run_at = datetime(2026, 10, 18, 18, 1, tzinfo=BERLIN) + self.assertFalse(radar.due(self.radar, sat)) + self.assertFalse(radar.due(self.radar, sun_early)) + self.assertTrue(radar.due(self.radar, sun)) + self.radar.last_run_at = sun + self.assertFalse(radar.due(self.radar, datetime(2026, 10, 25, 23, 0, tzinfo=BERLIN))) + self.radar.last_run_at = None + self.assertTrue(radar.due(self.radar, sat), 'never run: at once') + + def test_submit_and_collect(self): + fake = FakeClient(status='in_progress') + with patch.object(radar, '_client', return_value=fake), \ + patch.object(papers, 'collect', return_value=(list(REC), ['medrxiv: down'])): + self.assertIn('2 Papers', radar.submit(self.radar)) + reqs = fake.created[0] + self.assertEqual([r['custom_id'] for r in reqs], ['papers', 'news']) + self.assertIn('Speech BCI', reqs[0]['params']['messages'][0]['content']) + self.assertEqual(radar.collect(self.radar), 'waiting') + fake.status = 'ended' + self.assertEqual(radar.collect(self.radar), 'done') + self.radar.refresh_from_db() + self.assertIsNone(self.radar.pending) + self.assertEqual(set(self.radar.seen), {'pmid:1', 'arxiv:2610.1'}) + b = self.radar.digests.get().bundle + self.assertEqual({i['id']: i['gruppe'] for i in b['items']}, {'pmid:1': 'lesenswert', 'arxiv:2610.1': 'technik'}) + self.assertEqual(b['items'][0]['title'], 'Speech BCI', 'titles from the sources, not the model') + self.assertEqual(b['stats']['usage'], {'input_tokens': 2000, 'output_tokens': 400, 'web_search_requests': 2}) + self.assertEqual(b['stats']['errors'], ['medrxiv: down']) + # The next week the basic term isn't asked for again. + self.assertIn('Decoder', radar.build_requests(NEUROPOLITIK, REC, radar._terms(self.radar), '2026-10-26')[0] + ['params']['messages'][0]['content']) + + def test_a_failed_sorting_forgets_nothing(self): + fake = FakeClient(papers_text='kein json') + self.radar.pending = {'batch_id': 'b', 'generated': '2026-10-19', 'days': 7, 'records': REC, 'errors': []} + self.radar.save() + with patch.object(radar, '_client', return_value=fake): + self.assertIn('fehlgeschlagen', radar.collect(self.radar)) + self.radar.refresh_from_db() + self.assertEqual((self.radar.seen, self.radar.pending, self.radar.digests.count()), ([], None, 0)) + + def test_nothing_new_no_batch(self): + fake = FakeClient() + with patch.object(radar, '_client', return_value=fake), patch.object(papers, 'collect', return_value=([], [])): + self.assertEqual(radar.submit(self.radar), 'Keine neuen Papers.') + self.assertEqual(fake.created, []) + + def test_api_view_run_import(self): + r = self.client.get(f'/projects/{self.project.id}/radar/').json() + self.assertEqual((r['radar']['title'], r['digests']), ('Neuropolitik-Digest', [])) + bundle = {'version': 1, 'generated': '2026-10-19', 'items': [{'id': 'pmid:9', 'gruppe': 'technik'}], + 'batch_id': 'msgbatch_x'} + self.client.post(f'/projects/{self.project.id}/radar/import/', json.dumps(bundle), content_type='application/json') + self.client.post(f'/projects/{self.project.id}/radar/import/', json.dumps(bundle), content_type='application/json') + r = self.client.get(f'/projects/{self.project.id}/radar/').json() + self.assertEqual(len(r['digests']), 1, 'the same batch once') + self.assertIn('pmid:9', Radar.objects.get().seen) + self.assertEqual(self.client.post(f'/projects/{self.project.id}/radar/import/', '{}', + content_type='application/json').status_code, 400) + r = self.client.post(f'/projects/{self.project.id}/radar/run/').json() + self.assertTrue(radar.due(Radar.objects.get())) + other = User.objects.create_user(username='x', password='pw12345678') + self.client.force_login(other) + self.assertEqual(self.client.get(f'/projects/{self.project.id}/radar/').status_code, 404) + + +class PapersTests(TestCase): + def test_collect_leaves_out_doubles_and_seen(self): + dup = dict(REC[0], id='doi:10.1/a', source='biorxiv') + with patch.object(papers, 'fetch_pubmed', return_value=[REC[0]]), \ + patch.object(papers, 'fetch_arxiv', return_value=[REC[1]]), \ + patch.object(papers, 'fetch_rxiv', side_effect=[[dup], OSError('down')]): + got, errors = papers.collect(NEUROPOLITIK, 7, seen={'arxiv:2610.1'}) + self.assertEqual([r['id'] for r in got], ['pmid:1']) + self.assertEqual(errors, ['medrxiv: down']) diff --git a/projects/urls.py b/projects/urls.py index 1accac1..92d4dd4 100644 --- a/projects/urls.py +++ b/projects/urls.py @@ -1,6 +1,6 @@ from django.urls import path -from . import views +from . import radar, views urlpatterns = [ path('', views.projects, name='projects'), @@ -8,6 +8,10 @@ urlpatterns = [ path('doi/', views.doi, name='projects_doi'), path('/', views.project, name='projects_project'), path('/delete/', views.delete, name='projects_delete'), + path('/radar/', radar.radar_view, name='projects_radar'), + path('/radar/settings/', radar.radar_settings, name='projects_radar_settings'), + path('/radar/run/', radar.radar_run, name='projects_radar_run'), + path('/radar/import/', radar.radar_import, name='projects_radar_import'), path('/items/add/', views.item_add, name='projects_item_add'), path('items//', views.item_update, name='projects_item_update'), path('items//delete/', views.item_delete, name='projects_item_delete'), diff --git a/requirements.txt b/requirements.txt index caa4915..7296680 100644 --- a/requirements.txt +++ b/requirements.txt @@ -23,3 +23,5 @@ imageio-ffmpeg==0.6.0 trafilatura==2.3.0 # Subscriber logins of the feed reader are kept encrypted (feeds/logins.py). cryptography==50.0.1 +# The Radar of a project sorts the week's papers with Claude (Batch API, projects/radar.py). +anthropic==1.15.0 diff --git a/static/js/nav.js b/static/js/nav.js index a972b0f..2b5b982 100644 --- a/static/js/nav.js +++ b/static/js/nav.js @@ -201,7 +201,9 @@ if (routing) return; let hash = current; if (current === 'radio') hash += '/' + (load('diora_active_radio_tab') || 'saved'); - if (location.hash.slice(1) !== hash) history.pushState(null, '', '#' + hash); + // A place inside the section (#projekte/12) stays: the section's script reads it. + const now = location.hash.slice(1); + if (now !== hash && now.indexOf(current + '/') !== 0) history.pushState(null, '', '#' + hash); } function route() { diff --git a/static/js/projects.js b/static/js/projects.js index c1ea61b..db37c81 100644 --- a/static/js/projects.js +++ b/static/js/projects.js @@ -19,7 +19,8 @@ research: { label: 'Forschung', hint: 'Papers, Bücher, Folgen und Artikel als Quellen, Zitationen, Notizen.', - tabs: [['sources', 'Quellen'], ['notes', 'Notizen']], + // Radar only where the project has one (a server-side config, docs/projects.md). + tabs: [['radar', 'Radar'], ['sources', 'Quellen'], ['notes', 'Notizen']], }, }; const KIND = { @@ -340,10 +341,14 @@ const d = await api('/projects/' + id + '/'); const items = []; for (const i of d.items) items.push(Object.assign({}, i, {iv: i.data_iv, data: await open_(i.data_ct, i.data_iv)})); + let radar = {radar: null, digests: []}; + try { radar = await api('/projects/' + id + '/radar/'); } catch (e) {} st.open = {id: d.project.id, preset: d.project.preset, archived: d.project.archived, iv: d.project.data_iv, - data: await open_(d.project.data_ct, d.project.data_iv), items: items}; + data: await open_(d.project.data_ct, d.project.data_iv), items: items, radar: radar.radar, digests: radar.digests || []}; if (!keepView) { - st.view = 'sources'; + st.view = st.open.radar ? 'radar' : 'sources'; + st.digest = 0; + st.radarShow = 'all'; st.filter = 'all'; st.expanded = null; st.note = null; @@ -628,7 +633,7 @@ } function sources() { - return st.open.items.filter(i => i.kind !== 'note'); + return st.open.items.filter(i => SOURCE_KINDS.indexOf(i.kind) >= 0); } function notes() { @@ -651,10 +656,11 @@ '' + ''; } - h += '
' + preset.tabs.map(t => '').join('') + '
'; - h += st.view === 'notes' ? notesHtml() : sourcesHtml(); + const count = id => id === 'sources' ? sources().length : id === 'notes' ? notes().length : radarNew(); + h += '
' + preset.tabs.filter(t => t[0] !== 'radar' || p.radar).map(t => + '').join('') + '
'; + h += st.view === 'notes' ? notesHtml() : st.view === 'radar' && p.radar ? radarHtml() : sourcesHtml(); return h; } @@ -727,6 +733,174 @@ ''; } + // ---- the Radar: the week's papers, sorted by Claude on the server; the decisions are ours ---------- + + const DECISIONS = {read: 'gelesen', later: 'später', drop: 'verworfen', source: 'in Quellen'}; + + // The project's item holding the decisions ({id: state}), encrypted like everything else. + function radarItem() { + return st.open.items.filter(i => i.kind === 'radar' && i.data)[0] || null; + } + + function decisions() { + if (st.open.radarDec) return st.open.radarDec; // changed here, maybe not saved yet + const i = radarItem(); + return (i && i.data.decisions) || {}; + } + + // Undecided papers of the latest digest's first group (the worth-reading): the tab's number. + function radarNew() { + const d = st.open.digests[0]; + if (!d || !st.open.radar) return 0; + const dec = decisions(); + return (d.bundle.items || []).filter(it => it.gruppe === (st.open.radar.groups || [])[0] && !dec[it.id]).length; + } + + let deciding = Promise.resolve(); + function decide(id, state) { + // Shown at once; the saves follow one after the other, each with everything decided here. + const open = st.open; + const dec = Object.assign({}, decisions()); + if (state) dec[id] = state; else delete dec[id]; + open.radarDec = dec; + const changed = {}; + changed[id] = state || null; + deciding = deciding.then(async () => { + const it = open.items.filter(i => i.kind === 'radar' && i.data)[0]; + if (!it) { await addItem('radar', null, {decisions: Object.assign({}, open.radarDec)}); return; } + if (await saveItem(it, {decisions: Object.assign({}, open.radarDec)})) return; + // Decided elsewhere meanwhile: theirs with this one on top. + const merged = Object.assign({}, (it.data && it.data.decisions) || {}); + if (changed[id]) merged[id] = changed[id]; else delete merged[id]; + open.radarDec = merged; + await saveItem(it, {decisions: Object.assign({}, merged)}); + if (st.open === open) render(); + }).catch(e => toast('Speichern ging nicht: ' + e.message)); + return deciding; + } + + function paperById(id) { + for (const d of st.open.digests) { + const it = (d.bundle.items || []).filter(x => x.id === id)[0]; + if (it) return it; + } + return null; + } + + // A paper of the Radar as a source: its citation data from doi.org, else from what the Radar has. + async function paperToSource(it) { + let csl = null; + if (it.doi) { + try { csl = (await api('/projects/doi/?q=' + encodeURIComponent(it.doi))).csl; } catch (e) {} + } + if (!csl) { + const pre = /preprint/i.test((it.pub_types || []).join(' ')) || /arxiv|rxiv/i.test(it.source || ''); + csl = {type: pre ? 'article' : 'article-journal', title: it.title, 'container-title': it.journal || '', + author: it.first_author ? names(it.first_author).concat(it.n_authors > 1 ? [{literal: 'u. a.'}] : []) : [], + issued: it.date ? {'date-parts': [String(it.date).split('-').map(Number)]} : undefined, URL: it.url, + DOI: it.doi || undefined}; + } + const data = {csl: csl, note: it.kurz || '', doi: it.doi || '', radar: it.id}; + await keyed(data, st.open.id); + await addItem('paper', null, data); + } + + function weekOf(iso) { + const d = new Date(iso + 'T12:00:00'); + const t = new Date(Date.UTC(d.getFullYear(), d.getMonth(), d.getDate())); + t.setUTCDate(t.getUTCDate() + 4 - (t.getUTCDay() || 7)); + const y = new Date(Date.UTC(t.getUTCFullYear(), 0, 1)); + return Math.ceil(((t - y) / 864e5 + 1) / 7); + } + + function dayText(iso, withTime) { + const d = new Date(iso.length <= 10 ? iso + 'T12:00:00' : iso); + if (isNaN(d)) return ''; + const o = {weekday: 'short', day: 'numeric', month: 'long'}; + if (withTime) { o.hour = '2-digit'; o.minute = '2-digit'; } + return d.toLocaleString('de-DE', o); + } + + function radarStatusHtml(r) { + let s = ''; + if (r.running) s = 'Claude sortiert gerade die Papers' + (r.submitted ? ' (seit ' + dayText(r.submitted, true) + ')' : '') + '.'; + else if (!r.enabled) s = 'Pausiert.'; + else if (r.next_run && new Date(r.next_run) - Date.now() < 60 * 60 * 1000) s = 'Der nächste Abruf kommt in der nächsten halben Stunde.'; + else if (r.next_run) s = 'Nächster Abruf: ' + dayText(r.next_run, true) + '.'; + let h = '
' + esc(s) + + (r.sources.length ? ' ' + esc(r.sources.join(', ')) + ', die letzten ' + r.days + ' Tage.' : '') + ''; + if (!r.running) h += ''; + h += '
'; + if (!r.key) h += '

Auf dem Server fehlt noch der API-Schlüssel (ANTHROPIC_API_KEY): bis dahin kommt kein neuer Digest.

'; + if (r.last_error) h += '

' + esc(r.last_error) + '

'; + return h; + } + + function paperHtml(it, full, dec) { + const state = dec[it.id]; + const who = (it.first_author || '?') + ((it.n_authors || 0) > 1 ? ' u. a.' : ''); + const meta = [who, it.journal || it.source, it.date].filter(Boolean).join(' · '); + let h = '
' + + '
' + (it.url ? '' + esc(it.title) + '' : esc(it.title)) + '
' + + '
' + esc(meta) + (state ? ' · ' + esc(DECISIONS[state]) + '' : '') + '
'; + if (full) { + if (it.worum_es_geht) h += '

Worum es geht: ' + inline(it.worum_es_geht) + '

'; + if (it.einordnung) h += '

Einordnung: ' + inline(it.einordnung) + '

'; + if (it.politischer_haken) h += '

Politischer Haken: ' + inline(it.politischer_haken) + '

'; + } else if (it.kurz) h += '
' + inline(it.kurz) + '
'; + const b = (s, label) => ''; + h += '
' + b('later', 'Später') + b('read', 'Gelesen') + b('drop', 'Verwerfen') + + (state === 'source' ? '✓ in Quellen' + : '') + '
'; + return h; + } + + function radarHtml() { + const r = st.open.radar; + const digests = st.open.digests; + const dec = decisions(); + let h = radarStatusHtml(r); + if (!digests.length) return h + '

Noch kein Digest.

'; + const later = []; + digests.forEach(d => (d.bundle.items || []).forEach(it => { if (dec[it.id] === 'later') later.push(it); })); + h += '
' + digests.map((d, n) => '').join('') + + (later.length ? '' : '') + '
'; + if (st.radarShow === 'later') { + return h + '

Für später

' + later.map(it => paperHtml(it, !!it.worum_es_geht, dec)).join(''); + } + const d = digests[Math.min(st.digest || 0, digests.length - 1)]; + const b = d.bundle; + const groups = r.groups && r.groups.length ? r.groups : ['lesenswert']; + const label = g => (r.group_labels && r.group_labels[g]) || g; + h += '

' + esc(r.title || 'Digest') + ' · KW ' + weekOf(d.generated) + ', ' + esc(dayText(d.generated)) + '

'; + if (b.auf_einen_blick) h += '

' + inline(b.auf_einen_blick) + '

'; + groups.forEach((g, n) => { + const items = (b.items || []).filter(it => it.gruppe === g); + if (!items.length) return; + const noise = n === groups.length - 1 && groups.length > 2; // the last group (Rauschen): closed + h += '
' + esc(label(g)) + + '' + items.length + '' + + items.map(it => paperHtml(it, n === 0 && !!it.worum_es_geht, dec)).join('') + '
'; + }); + if (b.news_markdown && !/^Keine relevanten Meldungen/.test(b.news_markdown.trim())) { + h += '
Aus Branche und Politik
' + + markdown(b.news_markdown) + '
'; + } + const g = b.grundlagenbegriff; + if (g && g.begriff) { + h += '
Grundlagenbegriff der Woche
' + esc(g.begriff) + '. ' + + inline(g.erklaerung || '') + '
'; + } + const s = b.stats || {}; + h += '

' + (s.records || 0) + ' Papers gesichtet' + + (s.per_source ? ' (' + Object.keys(s.per_source).map(k => k + ' ' + s.per_source[k]).join(', ') + ')' : '') + + (typeof s.cost_usd === 'number' ? ' · ' + s.cost_usd.toFixed(2).replace('.', ',') + ' $' : '') + + ((s.errors || []).length ? ' · Fehler: ' + esc(s.errors.join('; ')) : '') + '

'; + return h; + } + function notesHtml() { if (st.note != null) return noteEditorHtml(); const list = notes().filter(n => n.data).sort((a, b) => String(b.updated_at).localeCompare(String(a.updated_at))); @@ -928,6 +1102,35 @@ render(); return; } + if (ds.radarDigest != null) { st.digest = Number(ds.radarDigest); st.radarShow = 'all'; render(); return; } + if (ds.radarLater) { st.radarShow = 'later'; render(); return; } + if (ds.radarDecide) { + const now = decisions()[ds.radarDecide]; + decide(ds.radarDecide, now === ds.state ? null : ds.state); + render(); + return; + } + if (ds.radarSource) { + const it = paperById(ds.radarSource); + if (!it) return; + t.disabled = true; + await paperToSource(it); + await decide(it.id, 'source'); + toast('In den Quellen'); + render(); + return; + } + if (ds.radarRun) { + if (!confirm('Jetzt die Papers der letzten Tage abrufen und von Claude sortieren lassen? Das kostet etwa 0,40 $.')) return; + st.open.radar = (await api('/projects/' + st.open.id + '/radar/run/', {})).radar; + render(); + return; + } + if (ds.radarPause) { + st.open.radar = (await api('/projects/' + st.open.id + '/radar/settings/', {enabled: !st.open.radar.enabled})).radar; + render(); + return; + } if (ds.openNote) { st.note = Number(ds.openNote); st.preview = true; saveState = ''; render(); return; } if (ds.cite) { const i = citeIndex()[ds.cite]; @@ -1167,6 +1370,25 @@ '#tab-projekte .prj-md h3 { font-size: 19px; } #tab-projekte .prj-md h4 { font-size: 16px; } #tab-projekte .prj-md h5, #tab-projekte .prj-md h6 { font-size: 15px; }', '#tab-projekte .prj-md blockquote { margin: 8px 0; padding-left: 12px; border-left: 3px solid var(--border-strong, #444); color: var(--muted, #9a9a9a); }', '#tab-projekte .prj-md a { color: var(--accent, #e63946); }', + '#tab-projekte .prj-radar-status { display: flex; flex-wrap: wrap; gap: 6px; align-items: center; margin: 4px 0 6px; }', + '#tab-projekte .prj-radar-status .prj-muted { flex: 1 1 260px; }', + '#tab-projekte .prj-warn { color: var(--accent, #e63946); font-size: 13px; margin: 6px 0; }', + '#tab-projekte .prj-radar-h { font-size: 17px; margin: 16px 0 6px; }', + '#tab-projekte .prj-radar-glance { line-height: 1.6; font-size: 15px; }', + '#tab-projekte .prj-radar-group summary { cursor: pointer; list-style: none; }', + '#tab-projekte .prj-radar-group summary::-webkit-details-marker { display: none; }', + '#tab-projekte .prj-radar-group summary::before { content: "▸ "; } #tab-projekte .prj-radar-group[open] > summary::before { content: "▾ "; }', + '#tab-projekte .prj-paper { padding: 10px 0 12px; border-bottom: 1px solid var(--border, #2a2a2a); }', + '#tab-projekte .prj-paper p { margin: 6px 0; line-height: 1.55; font-size: 14px; }', + '#tab-projekte .prj-paper-title { font-size: 15px; font-weight: 700; margin-bottom: 2px; }', + '#tab-projekte .prj-paper-title a { color: inherit; text-decoration: none; } #tab-projekte .prj-paper-title a:hover { text-decoration: underline; }', + '#tab-projekte .prj-paper-short { font-size: 14px; margin-top: 4px; line-height: 1.5; }', + '#tab-projekte .prj-paper-acts { display: flex; flex-wrap: wrap; gap: 6px; align-items: center; margin-top: 8px; }', + '#tab-projekte .prj-paper-acts .prj-chip { min-height: 30px; font-size: 12px; }', + '#tab-projekte .prj-dropped { opacity: .45; }', + '#tab-projekte .prj-radar-term { margin: 16px 0; padding: 12px 14px; border-radius: 8px; background: var(--card, #0b0b0b); border: 1px solid var(--border, #2a2a2a); line-height: 1.55; font-size: 14px; }', + '#tab-projekte .prj-radar-term .prj-h3 { margin-top: 0; }', + '#tab-projekte .prj-radar-stats { margin-top: 14px; }', '#tab-projekte .prj-table { overflow-x: auto; margin: 10px 0; }', '#tab-projekte .prj-table table { border-collapse: collapse; font-size: 14px; }', '#tab-projekte .prj-table th, #tab-projekte .prj-table td { border: 1px solid var(--border, #2a2a2a); padding: 6px 8px; text-align: left; vertical-align: top; }', diff --git a/static/js/sw.js b/static/js/sw.js index b808c8a..069a0c2 100644 --- a/static/js/sw.js +++ b/static/js/sw.js @@ -2,7 +2,7 @@ * diora service worker — caches the app shell for offline use. */ -const CACHE = 'diora-v144'; +const CACHE = 'diora-v145'; const SHELL = [ '/static/css/app.css', '/static/js/app/01-core.js',