main
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 KeelrLeek
|
d397b78887 |
news: about the artist, and each story only once
Kyle reported duplicate articles and a few unrelated ones. Measuring the
live feed found it far worse than the handful he saw: 84 items across
seven follows, hand-labelled, 26 junk — and 24 of the 26 came from two
artists. toe returned 0 of 12 genuine, Pond 0 of 12. Round Pond, Goose
Pond Colony, Inlet Pond County Park, "Amanda M. Pond", peep-toe mules, a
funeral-home obituary. Not one item about the Australian band.
The largest single cause was not the query. parse_news_rss kept the first
twelve items and sorted by date afterwards, so it discarded ranks 13-100
of Google's relevance order — and every genuine toe item sits at rank
15-88. Nothing downstream could recover what was thrown away there. All
100 are parsed now; the cap is 40 and applies after filtering and dedup,
because the frontend's band asks for bandCols * 3 and twelve starved a
wide window.
news_relevance is the precision mechanism. Hard gates first: the name
must appear as a whole token run, every occurrence must not abut a hyphen
(peep-toe, head-to-toe, Chorn-Pond), and a canonically capitalised name
in a sentence-case headline must appear cased. Then document-level
signals, then occurrence-level signals computed for every occurrence with
the maximum kept — and the decision is a total AND a separate floor on
the occurrence subscore. That second clause is the whole filter: at a
single combined threshold every false positive scored at threshold from
document signals alone. "Every song on the 'Tip Toe' soundtrack" is a
music-press host plus a quoted title plus a music noun and carries zero
evidence at the name. Publisher reputation must never admit an item on
its own.
Tiers come from a short explicit dictionary, not from name length and not
from a frequency table. Length demoted Weezer to the strict path and cost
it four genuine items; corpus rank sent Wand from 16 kept to 45 because
the feed floods with "Wistoria: Wand and Sword", and took toe's precision
from 1.00 to 0.70.
dedup_news collapses several outlets' wording of one report — containment
rather than Jaccard, because headline lengths are wildly asymmetric and
Jaccard's union denominator punishes exactly that; artist tokens retained,
because stripping them starves a six-token headline to two; leader
clustering, because union-find chained 68 of 93 King Gizzard items into
one "story"; a 2-day window with unparseable dates non-mergeable, since
rfc2822_secs returns 0 on failure and two zeroes always satisfy a window.
The survivor carries an others count for the outlets it stands for.
That still left the other kind of repetition: many different reports about
one record, similar to each other but not to the leader that absorbed the
rest. Six of the visible twelve were the same Alien Metal announcement.
The fix is not a lower dedup threshold — lexical similarity merges
unrelated stories long before it catches rewrites — but a ceiling:
news_story_key names the album or single a headline is about, and
news_story_quota keeps at most two per work, applied after the date sort
so the survivors are a story's newest. Headlines naming no work are never
dropped, so a feed can shrink but can never be emptied.
Two things that were quietly broken. News URLs never qualified in
disk_cache_path, so the entire disk branch in artist_news was unreachable
and every restart refetched every follow against a rate-limited endpoint.
And rfind on a set of dashes plus one lands mid-character on a 3-byte
en-dash: the slice panics, and a panic inside an async Tauri command never
resolves its promise, so the News band hung forever — but only for artists
whose name sits mid-headline.
The host now comes from the source element's url attribute; its text is a
display name ("GamesRadar+") and is not a host. Both new fields are
serde(default), because cached items are read back through an .ok() that
swallows a schema mismatch: a required field would silently invalidate
every cached entry and fire a cold refetch for every follow at once.
Measured live: toe 12 items (0 real) to 5 (5 genuine), Pond 12 (0 real) to
7 (7 genuine), 32 duplicate clusters collapsed covering 71 redundant
reports, and no story takes more than two of the twelve visible slots.
Derivation, the falsified alternatives and a what-not-to-do section are in
research/news/PLAN.md.
|