news: about the artist, and each story only once

Kyle reported duplicate articles and a few unrelated ones. Measuring the
live feed found it far worse than the handful he saw: 84 items across
seven follows, hand-labelled, 26 junk — and 24 of the 26 came from two
artists. toe returned 0 of 12 genuine, Pond 0 of 12. Round Pond, Goose
Pond Colony, Inlet Pond County Park, "Amanda M. Pond", peep-toe mules, a
funeral-home obituary. Not one item about the Australian band.

The largest single cause was not the query. parse_news_rss kept the first
twelve items and sorted by date afterwards, so it discarded ranks 13-100
of Google's relevance order — and every genuine toe item sits at rank
15-88. Nothing downstream could recover what was thrown away there. All
100 are parsed now; the cap is 40 and applies after filtering and dedup,
because the frontend's band asks for bandCols * 3 and twelve starved a
wide window.

news_relevance is the precision mechanism. Hard gates first: the name
must appear as a whole token run, every occurrence must not abut a hyphen
(peep-toe, head-to-toe, Chorn-Pond), and a canonically capitalised name
in a sentence-case headline must appear cased. Then document-level
signals, then occurrence-level signals computed for every occurrence with
the maximum kept — and the decision is a total AND a separate floor on
the occurrence subscore. That second clause is the whole filter: at a
single combined threshold every false positive scored at threshold from
document signals alone. "Every song on the 'Tip Toe' soundtrack" is a
music-press host plus a quoted title plus a music noun and carries zero
evidence at the name. Publisher reputation must never admit an item on
its own.

Tiers come from a short explicit dictionary, not from name length and not
from a frequency table. Length demoted Weezer to the strict path and cost
it four genuine items; corpus rank sent Wand from 16 kept to 45 because
the feed floods with "Wistoria: Wand and Sword", and took toe's precision
from 1.00 to 0.70.

dedup_news collapses several outlets' wording of one report — containment
rather than Jaccard, because headline lengths are wildly asymmetric and
Jaccard's union denominator punishes exactly that; artist tokens retained,
because stripping them starves a six-token headline to two; leader
clustering, because union-find chained 68 of 93 King Gizzard items into
one "story"; a 2-day window with unparseable dates non-mergeable, since
rfc2822_secs returns 0 on failure and two zeroes always satisfy a window.
The survivor carries an others count for the outlets it stands for.

That still left the other kind of repetition: many different reports about
one record, similar to each other but not to the leader that absorbed the
rest. Six of the visible twelve were the same Alien Metal announcement.
The fix is not a lower dedup threshold — lexical similarity merges
unrelated stories long before it catches rewrites — but a ceiling:
news_story_key names the album or single a headline is about, and
news_story_quota keeps at most two per work, applied after the date sort
so the survivors are a story's newest. Headlines naming no work are never
dropped, so a feed can shrink but can never be emptied.

Two things that were quietly broken. News URLs never qualified in
disk_cache_path, so the entire disk branch in artist_news was unreachable
and every restart refetched every follow against a rate-limited endpoint.
And rfind on a set of dashes plus one lands mid-character on a 3-byte
en-dash: the slice panics, and a panic inside an async Tauri command never
resolves its promise, so the News band hung forever — but only for artists
whose name sits mid-headline.

The host now comes from the source element's url attribute; its text is a
display name ("GamesRadar+") and is not a host. Both new fields are
serde(default), because cached items are read back through an .ok() that
swallows a schema mismatch: a required field would silently invalidate
every cached entry and fire a cold refetch for every follow at once.

Measured live: toe 12 items (0 real) to 5 (5 genuine), Pond 12 (0 real) to
7 (7 genuine), 32 duplicate clusters collapsed covering 71 redundant
reports, and no story takes more than two of the twelve visible slots.
Derivation, the falsified alternatives and a what-not-to-do section are in
research/news/PLAN.md.

crates/metadata/tests/news_relevance.rs

@@ -0,0 +1,136 @@
+//! The news filter, measured against real headlines.
+//!
+//! Every case here was taken from a live Google News feed for one of Kyle's
+//! followed artists on 2026-08-05, and hand-labelled. The four `reject_*`
+//! cases are the ones he reported. Treat these numbers as tuning-set
+//! performance: the weights were chosen with this corpus in view, so a new
+//! bug report should arrive as a new row here rather than as a tweak to a
+//! weight that is already fitted.
+use metadata::{news_relevance, news_tier, ArtistNewsItem, NewsTier};
+
+fn item(title: &str, host: &str) -> ArtistNewsItem {
+    ArtistNewsItem {
+        title: title.to_string(),
+        url: String::new(),
+        source: String::new(),
+        date: 1_785_000_000,
+        host: host.to_string(),
+        others: 0,
+    }
+}
+fn keeps(title: &str, host: &str, artist: &str) -> bool {
+    news_relevance(&item(title, host), artist, news_tier(artist)).is_some()
+}
+
+#[test]
+fn tiers_are_assigned_by_dictionary_not_by_length() {
+    // "Weezer" is six characters and perfectly safe; deciding riskiness by
+    // length demoted it to the strict path and cost four genuine items.
+    assert_eq!(news_tier("Weezer"), NewsTier::Single);
+    assert_eq!(news_tier("Wand"), NewsTier::Single);
+    assert_eq!(news_tier("toe"), NewsTier::VeryCommon);
+    assert_eq!(news_tier("Pond"), NewsTier::VeryCommon);
+    assert_eq!(news_tier("King Gizzard & The Lizard Wizard"), NewsTier::Distinctive);
+    assert_eq!(news_tier("Camping In Alaska"), NewsTier::Distinctive);
+}
+
+// ---------------------------------------------------------------- rejects
+
+#[test]
+fn rejects_hyphenated_compounds() {
+    // Hard gate 2 — every occurrence abuts a hyphen.
+    assert!(!keeps(
+        "Charli XCX Launches Her 'Music, Fashion, Film' Era in Patent Peep-toe Mules",
+        "wwd.com",
+        "toe",
+    ));
+    assert!(!keeps(
+        "Hyperpop standout Underscores makes music that can go toe-to-toe with the biggest pop acts",
+        "chicagoreader.com",
+        "toe",
+    ));
+    assert!(!keeps(
+        "The Toe-Tapping Music Genres You're Most Likely To Hear In Trader Joe's",
+        "chowhound.com",
+        "toe",
+    ));
+}
+
+#[test]
+fn rejects_the_stellar_blade_case() {
+    // The occurrence floor earning its keep: this has a music noun and a
+    // quoted title, so document-level signals alone would have passed it.
+    assert!(!keeps(
+        "Stellar Blade: Blood Rain studio CEO defends AI music video because Evie was originally \"created from head to toe\" by the devs",
+        "gamesradar.com",
+        "toe",
+    ));
+}
+
+#[test]
+fn rejects_the_obituary() {
+    assert!(!keeps(
+        "Obituary information for Amanda M. Pond",
+        "cody-whitefuneralhome.com",
+        "Pond",
+    ));
+}
+
+#[test]
+fn rejects_places_and_surnames() {
+    for (title, host) in [
+        ("Rock the Dock 2026 bringing live music to Goose Pond Colony", "rocketcitynow.com"),
+        ("Jud Caswell in Round Pond", "lcnme.com"),
+        ("Bird Song Project sculpture trail opens at Inlet Pond County Park", "northforksun.com"),
+        ("On the record with Aaron Pond and the People's Music Supply", "broadstreetreview.com"),
+        ("Miami Music Week at Toe Jam Backlot", "timeout.com"),
+    ] {
+        let artist = if title.contains("Toe") { "toe" } else { "Pond" };
+        assert!(!keeps(title, host, artist), "should have rejected: {title}");
+    }
+}
+
+// ------------------------------------------------------------------ keeps
+
+#[test]
+fn keeps_genuine_items_for_a_very_common_name() {
+    assert!(keeps("toe Announces October Shows in Osaka and Tokyo", "niew.jp", "toe"));
+    assert!(keeps("Pond Announce New Album 'Terrestrials'", "diymag.com", "Pond"));
+    assert!(keeps("Ride Announce Here And Now Tour For 2027", "nme.com", "Ride"));
+}
+
+#[test]
+fn keeps_a_bare_tour_announcement_with_no_music_vocabulary() {
+    // The false-negative the brief demands: strict enough to kill Stellar
+    // Blade must not kill this.
+    assert!(keeps("Weezer Announces 'The Gathering' Fall Tour", "rockcellarmagazine.com", "Weezer"));
+}
+
+#[test]
+fn keeps_the_short_form_and_the_possessive() {
+    // Without the 4+-token short-form variant and the possessive-tolerant
+    // gate these silently vanish.
+    assert!(keeps(
+        "King Gizzard's Stu Mackenzie on leaving Spotify",
+        "pitchfork.com",
+        "King Gizzard & The Lizard Wizard",
+    ));
+    assert!(keeps(
+        "King Gizzard and The Lizard Wizard debut \"Alien Metal\"",
+        "northerntransmissions.com",
+        "King Gizzard & The Lizard Wizard",
+    ));
+}
+
+#[test]
+fn keeps_masthead_and_review_conventions() {
+    assert!(keeps("Review: toe, 'NOW I SEE THE LIGHT'", "thequietus.com", "toe"));
+    assert!(keeps("Pond 'Terrestrials' Review", "clashmusic.com", "Pond"));
+}
+
+#[test]
+fn a_music_press_host_alone_never_admits_an_item() {
+    // "Every song on the 'Tip Toe' soundtrack" is a music-press host plus a
+    // quoted work title plus a music noun, and zero evidence at the name.
+    assert!(!keeps("Every song on the 'Tip Toe' soundtrack", "nme.com", "toe"));
+}

crates/metadata/tests/news_story_quota.rs

@@ -0,0 +1,194 @@
+//! The per-story display quota, measured against real headlines.
+//!
+//! Dedup collapses rewordings of one report; this bounds the other kind of
+//! repetition — many different reports about the same record. Every headline
+//! here is verbatim from a live Google News feed on 2026-08-05, where seven of
+//! the twelve visible slots were King Gizzard's "Alien Metal" announcement
+//! arriving from seven outlets that shared almost no vocabulary.
+use metadata::{news_story_key, news_story_quota, ArtistNewsItem};
+
+const GIZZ: &str = "King Gizzard & The Lizard Wizard";
+
+fn key(title: &str, artist: &str) -> Option<String> {
+    news_story_key(title, artist)
+}
+fn item(title: &str, date: i64) -> ArtistNewsItem {
+    ArtistNewsItem {
+        title: title.to_string(),
+        url: title.to_string(),
+        source: String::new(),
+        date,
+        host: String::new(),
+        others: 0,
+    }
+}
+
+// ------------------------------------------------------------ extraction
+
+#[test]
+fn every_alien_metal_headline_names_the_same_work() {
+    // Five quote it, two only capitalise it after "Album". All seven are one
+    // story and between them they held seven of twelve panel slots.
+    for title in [
+        "King Gizzard & the Lizard Wizard rave on with some \u{2018}Alien Metal\u{2019}",
+        "King Gizzard & the Lizard Wizard announce 28th album \u{2018}Alien Metal\u{2019}",
+        "King Gizzard & the Lizard Wizard Share Title Track From Forthcoming LP \u{2018}Alien Metal\u{2019}",
+        "King Gizzard and the Lizard Wizard announce EDM album, \u{2018}Alien Metal\u{2019}",
+        "King Gizzard & The Lizard Wizard Announce 28th Album 'Alien Metal,' Brooklyn Rave",
+        "King Gizzard & The Lizard Wizard Release Title Track from Techno Album Alien Metal: Stream",
+        "King Gizzard & The Lizard Wizard Detail New Album Alien Metal",
+    ] {
+        assert_eq!(key(title, GIZZ).as_deref(), Some("alien metal"), "{title}");
+    }
+}
+
+#[test]
+fn a_cue_adjacent_quote_outranks_an_earlier_one() {
+    // The album is the story; "Two Hands" is the track it leads with.
+    assert_eq!(
+        key("Pond Announce New Album 'Terrestrials': Hear \"Two Hands\"", "Pond").as_deref(),
+        Some("terrestrials"),
+    );
+    assert_eq!(
+        key("Pond Share New Single \"Terrestrials\": Listen", "Pond").as_deref(),
+        Some("terrestrials"),
+    );
+}
+
+#[test]
+fn a_curly_quote_may_close_on_its_straight_twin() {
+    // Verbatim from the feed — one headline, two different quote characters.
+    assert_eq!(
+        key("Weezer Enlists Kenny Beats to Produce \u{2018}The Gold Album'", "Weezer").as_deref(),
+        Some("gold album"),
+    );
+}
+
+#[test]
+fn the_artist_name_and_a_leading_article_come_off_the_key() {
+    // Five headlines about one tour. Without this they split into "the
+    // gathering" and "weezer the gathering" and pass the quota twice.
+    for title in [
+        "Weezer Announce \u{2018}The Gathering\u{2019} Arena Tour",
+        "Weezer Announce \u{201c}Weezer: The Gathering\u{201d}",
+        "'Weezer: The Gathering' tour heading to Nationwide Arena in October",
+        "Weezer Play Pop-Up Rooftop Gig to Mark \u{2018}The Gathering\u{2019} Tour",
+    ] {
+        assert_eq!(key(title, "Weezer").as_deref(), Some("gathering"), "{title}");
+    }
+}
+
+#[test]
+fn a_work_cue_inside_a_quote_does_not_introduce_one() {
+    // Three headlines about the single "C.E.O.", live on 2026-08-05. The third
+    // quotes an older song, and the cue inside THAT quote was keying the
+    // headline as a work called "homage" — a third slot for one story.
+    for title in [
+        "Weezer\u{2019}s \u{201c}C.E.O.\u{201d} Video Has Curry Barker, Celeb Cameos, & A \u{201c}Sweater Song\u{201d} Homage",
+        "Weezer Unveil New Single \"C.E.O.\", Expand \"Weezer: The Gathering\" with New Tour Dates",
+        "Weezer Go Back in Time for \u{2018}C.E.O.\u{2019} Music Video",
+    ] {
+        assert_eq!(key(title, "Weezer").as_deref(), Some("c e o"), "{title}");
+    }
+}
+
+#[test]
+fn a_possessive_apostrophe_does_not_open_a_span() {
+    // If it did, the span would run from "'s Rivers…" to the next real quote
+    // and every span after it would be misaligned.
+    assert_eq!(
+        key("Weezer's Rivers Cuomo drops new single 'Shine Again'", "Weezer").as_deref(),
+        Some("shine again"),
+    );
+}
+
+// ------------------------------------------------------- what is NOT a work
+
+#[test]
+fn a_generic_run_after_a_cue_is_not_a_work() {
+    // Each of these keyed as a "work" before the generic list existed —
+    // `review`, `list` and `next week` — grouping unrelated headlines.
+    for (title, artist) in [
+        ("Panda Bear / Sonic Boom: A ? of WHEN Album Review", "Panda Bear"),
+        ("Weezer Concert Setlist: Discover the Average Song List", "Weezer"),
+        (
+            "Weezer Announce North American Tour, Tease New Album, and Are Releasing a New Song Next Week",
+            "Weezer",
+        ),
+        ("Track Review: Paramore \u{2013} Running Out of Time", "Panda Bear"),
+    ] {
+        assert_eq!(key(title, artist), None, "{title}");
+    }
+}
+
+#[test]
+fn an_unclosed_run_is_not_a_work() {
+    // "Album Speculation [Watch]" — a run that neither ends the headline nor
+    // stops at punctuation is ordinary prose.
+    assert_eq!(
+        key("Billy Strings & King Gizzard Share Video From The Studio, Fueling Album Speculation [Watch]", GIZZ),
+        None,
+    );
+    // A capitalised function word ends the run: this is a credit, not a work.
+    assert_eq!(
+        key("Panda Bear Announces Tour and New Album, Shares Video for New Song With Cindy Lee: Watch", "Panda Bear"),
+        None,
+    );
+}
+
+#[test]
+fn a_long_quote_is_speech_not_a_title() {
+    assert_eq!(
+        key("\u{201c}It\u{2019}s 5 days since they released new music \u{2013} was starting to think we\u{2019}d never hear from them again\u{201d}: King Gizzard And The Lizard Wizard\u{2019}s 24 albums in 11 years?", GIZZ),
+        None,
+    );
+}
+
+#[test]
+fn headlines_that_name_no_work_are_never_keyed() {
+    // These are the majority, and they are the reason the quota can never
+    // empty a feed.
+    assert_eq!(key("King Gizzard & The Lizard Wizard Share New Single", GIZZ), None);
+    assert_eq!(key("King Gizzard & The Lizard Wizard Australia 2026 Tour", GIZZ), None);
+    assert_eq!(key("toe Announces October Shows in Osaka and Tokyo", "toe"), None);
+}
+
+// ----------------------------------------------------------------- quota
+
+#[test]
+fn the_quota_keeps_the_first_two_of_a_story_and_all_the_rest() {
+    // The list arrives newest-first, so "first two" is "the two most recent".
+    let titles = [
+        "King Gizzard & the Lizard Wizard rave on with some \u{2018}Alien Metal\u{2019}",
+        "King Gizzard & The Lizard Wizard Share New Single",
+        "King Gizzard & The Lizard Wizard Release Title Track from Techno Album Alien Metal: Stream",
+        "King Gizzard & the Lizard Wizard announce 28th album \u{2018}Alien Metal\u{2019}",
+        "King Gizzard & The Lizard Wizard Detail New Album Alien Metal",
+        "King Gizzard & The Lizard Wizard Australia 2026 Tour",
+    ];
+    let items: Vec<_> = titles
+        .iter()
+        .enumerate()
+        .map(|(i, t)| item(t, 1_785_000_000 - i as i64 * 86_400))
+        .collect();
+    let out = news_story_quota(items, GIZZ);
+    let kept: Vec<&str> = out.iter().map(|i| i.title.as_str()).collect();
+    assert_eq!(kept, vec![titles[0], titles[1], titles[2], titles[5]]);
+}
+
+#[test]
+fn the_quota_never_empties_a_feed() {
+    // Pond's whole in-feed life was five reports of one record. Two survive,
+    // which is the point: an empty row would be worse than a repeated one.
+    let items: Vec<_> = [
+        "Pond Return With Psych\u{2019}d Out Track \u{2018}Terrestrials\u{2019}",
+        "Pond Share New Single \"Terrestrials\": Listen",
+        "Pond Announce North American Tour Behind New Album 'Terrestrials'",
+        "Pond Announce New Album 'Terrestrials': Hear \"Two Hands\"",
+        "Pond \u{2018}Terrestrials\u{2019} Review: Less Sometimes Is Really More",
+    ]
+    .iter()
+    .map(|t| item(t, 1_785_000_000))
+    .collect();
+    assert_eq!(news_story_quota(items, "Pond").len(), 2);
+}

crates/metadata/tests/news_utf8.rs

@@ -0,0 +1,70 @@
+//! Byte-index safety for the news scorer.
+//!
+//! Google's headlines are full of curly quotes and en/em dashes. Every one of
+//! these panicked at some point during development because a byte offset was
+//! advanced by 1 past a multi-byte character. A panic inside an async Tauri
+//! command never resolves its promise, so the symptom was a hung News band —
+//! and only for artists whose name appears mid-headline, which is why it was
+//! invisible on the first artist tried.
+use metadata::{news_relevance, news_tier, ArtistNewsItem};
+
+fn score(title: &str, artist: &str) -> Option<i32> {
+    let item = ArtistNewsItem {
+        title: title.to_string(),
+        url: String::new(),
+        source: String::new(),
+        date: 1_785_000_000,
+        host: "example.com".to_string(),
+        others: 0,
+    };
+    news_relevance(&item, artist, news_tier(artist))
+}
+
+#[test]
+fn multibyte_punctuation_before_the_name_does_not_panic() {
+    for (title, artist) in [
+        ("Listen \u{2013} toe announce a new album", "toe"),
+        ("Review \u{2014} Pond announce 'Terrestrials'", "Pond"),
+        ("Watch: toe share \u{2018}Nothing\u{2019} video", "toe"),
+        ("\u{201c}Terrestrials\u{201d} \u{2013} Pond return", "Pond"),
+        ("Interview \u{2013} King Gizzard & The Lizard Wizard on Spotify", "King Gizzard & The Lizard Wizard"),
+        ("\u{2013}\u{2014}\u{2018}\u{2019}\u{201c}\u{201d} toe", "toe"),
+    ] {
+        let _ = score(title, artist); // must not panic
+    }
+}
+
+#[test]
+fn non_ascii_headlines_score_without_panicking() {
+    for (title, artist) in [
+        ("Elephant Gym\u{2019}s Math Rock journey from Kaohsiung", "Elephant Gym"),
+        ("\u{6771}\u{4EAC}: toe announce October shows", "toe"),
+        ("R\u{00F6}yksopp \u{2013} toe remix premiere", "toe"),
+    ] {
+        let _ = score(title, artist);
+    }
+}
+
+#[test]
+fn a_multibyte_dash_marker_behaves_exactly_like_an_ascii_colon() {
+    // The fix must not merely avoid the panic — the masthead signal has to
+    // still fire. Comparing the two spellings of the same headline tests that
+    // directly, without depending on where a tier's threshold happens to sit.
+    // The lead word must NOT be one of the "Review:/Listen:" prefix markers —
+    // a colon would then earn a second, unrelated bonus and the two spellings
+    // would legitimately differ.
+    for (colon, dash) in [
+        ("Stereogum: toe announce a new album", "Stereogum \u{2013} toe announce a new album"),
+        ("Mojo: Pond return with 'Terrestrials'", "Mojo \u{2014} Pond return with 'Terrestrials'"),
+    ] {
+        let artist = if colon.contains("toe") { "toe" } else { "Pond" };
+        assert_eq!(
+            score(colon, artist),
+            score(dash, artist),
+            "marker spelling changed the score for: {dash}",
+        );
+    }
+    // Whether that shape clears a tier's bar is the relevance suite's job —
+    // it depends on the publisher host, which this file deliberately holds
+    // neutral so only the byte handling is under test.
+}