Keelrleek/SoulStream
2
1 Fork 1
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

news: about the artist, and each story only once

Browse Source 
Kyle reported duplicate articles and a few unrelated ones. Measuring the
live feed found it far worse than the handful he saw: 84 items across
seven follows, hand-labelled, 26 junk — and 24 of the 26 came from two
artists. toe returned 0 of 12 genuine, Pond 0 of 12. Round Pond, Goose
Pond Colony, Inlet Pond County Park, "Amanda M. Pond", peep-toe mules, a
funeral-home obituary. Not one item about the Australian band.

The largest single cause was not the query. parse_news_rss kept the first
twelve items and sorted by date afterwards, so it discarded ranks 13-100
of Google's relevance order — and every genuine toe item sits at rank
15-88. Nothing downstream could recover what was thrown away there. All
100 are parsed now; the cap is 40 and applies after filtering and dedup,
because the frontend's band asks for bandCols * 3 and twelve starved a
wide window.

news_relevance is the precision mechanism. Hard gates first: the name
must appear as a whole token run, every occurrence must not abut a hyphen
(peep-toe, head-to-toe, Chorn-Pond), and a canonically capitalised name
in a sentence-case headline must appear cased. Then document-level
signals, then occurrence-level signals computed for every occurrence with
the maximum kept — and the decision is a total AND a separate floor on
the occurrence subscore. That second clause is the whole filter: at a
single combined threshold every false positive scored at threshold from
document signals alone. "Every song on the 'Tip Toe' soundtrack" is a
music-press host plus a quoted title plus a music noun and carries zero
evidence at the name. Publisher reputation must never admit an item on
its own.

Tiers come from a short explicit dictionary, not from name length and not
from a frequency table. Length demoted Weezer to the strict path and cost
it four genuine items; corpus rank sent Wand from 16 kept to 45 because
the feed floods with "Wistoria: Wand and Sword", and took toe's precision
from 1.00 to 0.70.

dedup_news collapses several outlets' wording of one report — containment
rather than Jaccard, because headline lengths are wildly asymmetric and
Jaccard's union denominator punishes exactly that; artist tokens retained,
because stripping them starves a six-token headline to two; leader
clustering, because union-find chained 68 of 93 King Gizzard items into
one "story"; a 2-day window with unparseable dates non-mergeable, since
rfc2822_secs returns 0 on failure and two zeroes always satisfy a window.
The survivor carries an others count for the outlets it stands for.

That still left the other kind of repetition: many different reports about
one record, similar to each other but not to the leader that absorbed the
rest. Six of the visible twelve were the same Alien Metal announcement.
The fix is not a lower dedup threshold — lexical similarity merges
unrelated stories long before it catches rewrites — but a ceiling:
news_story_key names the album or single a headline is about, and
news_story_quota keeps at most two per work, applied after the date sort
so the survivors are a story's newest. Headlines naming no work are never
dropped, so a feed can shrink but can never be emptied.

Two things that were quietly broken. News URLs never qualified in
disk_cache_path, so the entire disk branch in artist_news was unreachable
and every restart refetched every follow against a rate-limited endpoint.
And rfind on a set of dashes plus one lands mid-character on a 3-byte
en-dash: the slice panics, and a panic inside an async Tauri command never
resolves its promise, so the News band hung forever — but only for artists
whose name sits mid-headline.

The host now comes from the source element's url attribute; its text is a
display name ("GamesRadar+") and is not a host. Both new fields are
serde(default), because cached items are read back through an .ok() that
swallows a schema mismatch: a required field would silently invalidate
every cached entry and fire a cold refetch for every follow at once.

Measured live: toe 12 items (0 real) to 5 (5 genuine), Pond 12 (0 real) to
7 (7 genuine), 32 duplicate clusters collapsed covering 71 redundant
reports, and no story takes more than two of the twelve visible slots.
Derivation, the falsified alternatives and a what-not-to-do section are in
research/news/PLAN.md.
This commit is contained in:
KeelrLeek committed 2026-08-05 09:46:54 -05:00
1 parent 034eac9569
commit d397b78887
4 files changed
+1253 -7
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

crates/metadata/src/lib.rs
+853 -7
View File
@@ -257,12 +257,31 @@ fn parse_news_rss(body: &str) -> Vec<ArtistNewsItem> {
days * 86400 + h * 3600 + mi * 60 + sec
}
/// The `url` attribute of an opening tag: `<source url="https://wwd.com">`.
/// `tag()` above returns element TEXT and has no attribute support.
fn attr(block: &str, name: &str, key: &str) -> Option<String> {
let open = format!("<{name}");
let start = block.find(&open)?;
let head_end = block[start..].find('>')? + start;
let head = &block[start..head_end];
let at = head.find(&format!("{key}=\""))? + key.len() + 2;
let end = head[at..].find('"')? + at;
Some(decode(head[at..end].trim()))
}
/// "https://www.wwd.com/x" → "wwd.com". Bare host, no scheme, no `www.`.
fn host_of(url: &str) -> String {
let rest = url.split_once("://").map_or(url, |(_, r)| r);
let host = rest.split(['/', '?', '#']).next().unwrap_or("");
host.trim_start_matches("www.").to_ascii_lowercase()
}
let mut out = Vec::new();
for block in body.split("<item>").skip(1) {
let block = block.split("</item>").next().unwrap_or("");
let Some(raw_title) = tag(block, "title") else { continue };
let Some(link) = tag(block, "link") else { continue };
let source = tag(block, "source").map(|s| decode(&s)).unwrap_or_default();
let host = attr(block, "source", "url").map(|u| host_of(&u)).unwrap_or_default();
let mut title = decode(&raw_title);
// "Headline - Publication" → "Headline" when the tail names the source.
if !source.is_empty() {
@@ -271,15 +290,822 @@ fn parse_news_rss(body: &str) -> Vec<ArtistNewsItem> {
}
}
let date = tag(block, "pubDate").map(|d| rfc2822_secs(&d)).unwrap_or(0);
out.push(ArtistNewsItem { title, url: decode(&link), source, date });
if out.len() >= 12 {
break;
out.push(ArtistNewsItem { title, url: decode(&link), source, date, host, others: 0 });
}
// NO CAP AND NO DATE SORT HERE.
//
// This used to `break` at 12 and then sort by date. The feed returns 100
// items in Google's RELEVANCE order, so that kept the twelve Google liked
// best and discarded ranks 13-100 — and for an ambiguously named artist
// that is exactly backwards: measured on the live feed, every genuine
// "toe" item sits at rank 15-88 and every genuine "Pond" item starts at
// rank 15. Zero of the top twelve were about either band. No amount of
// filtering downstream could recover what was thrown away up here.
//
// Relevance order is also the better tiebreaker to preserve, so the caller
// filters and dedups against the full feed and caps at the end.
out
}
/* =========================== news relevance ===============================
*
* Deciding whether a headline is about THIS band. Researched and adversarially
* verified 2026-08-05 against seven live feeds; the plan and the measurements
* live in research/news/PLAN.md.
*
* The one idea the whole thing rests on: the signal is the name's SYNTACTIC
* ROLE, not the headline's topic. The query already contains the word "music",
* so every junk item is saturated with music vocabulary — "Every song on the
* 'Tip Toe' soundtrack" has a music-press host, a quoted work title and a
* music noun, and is still not about the band toe. What separates a real item
* is evidence AT THE NAME: a verb it governs, a role noun beside it, its
* position. Hence the two-part decision at the bottom — a total, and a
* separate floor on the occurrence-level subscore that publisher reputation
* can never satisfy on its own.
*/
/// How much evidence a name needs before it is believed.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NewsTier {
/// Multi-token distinctive name ("King Gizzard & The Lizard Wizard").
Distinctive,
/// Any other single-token name ("Weezer", "Wand", "Interpol").
Single,
/// A name that is also an ordinary English word ("toe", "Pond", "Girls").
VeryCommon,
}
impl NewsTier {
const fn threshold(self) -> i32 {
match self {
Self::Distinctive => 0,
Self::Single => 3,
Self::VeryCommon => 6,
}
}
out.sort_by_key(|item| -item.date);
/// The floor on occurrence-level evidence. `Distinctive` opts out.
const fn min_occurrence(self) -> i32 {
match self {
Self::Distinctive => i32::MIN,
Self::Single => 1,
Self::VeryCommon => 3,
}
}
}
/// Names that are ordinary English words. Deliberately a short explicit list,
/// NOT a frequency table: bucketing by corpus rank was measured to send Wand
/// from 16 kept to 45 (the feed floods with "Wistoria: Wand and Sword"), and
/// toe precision from 1.00 to 0.70.
const VERY_COMMON_NAMES: &[&str] = &[
"air", "bush", "cake", "cream", "curve", "eels", "elbow", "everything", "fear", "field",
"fire", "fuel", "garbage", "girls", "gorilla", "grass", "heart", "hole", "horse", "james",
"journey", "kansas", "keane", "low", "lush", "moon", "mountain", "oasis", "pond", "portal",
"prince", "pulp", "queen", "rain", "ride", "river", "rush", "sky", "smoke", "snow", "spirit",
"star", "steel", "stone", "sun", "texas", "toe", "train", "war", "water", "wire", "yes",
];
fn is_very_common(token: &str) -> bool {
VERY_COMMON_NAMES.contains(&token.to_ascii_lowercase().as_str())
}
/// The tier for an artist name.
pub fn news_tier(artist: &str) -> NewsTier {
let tokens: Vec<&str> = artist.split_whitespace().filter(|t| !t.is_empty()).collect();
match tokens.len() {
0 | 1 => {
let t = tokens.first().copied().unwrap_or("");
if is_very_common(t) { NewsTier::VeryCommon } else { NewsTier::Single }
}
_ => {
if tokens.iter().all(|t| is_very_common(t)) {
NewsTier::VeryCommon
} else {
NewsTier::Distinctive
}
}
}
}
/// Publisher hosts whose presence is evidence AGAINST a music story. Negative
/// classes are far more enumerable than a music-press allowlist would be — of
/// fourteen genuine `toe` items only one sits on any obvious list — so hosts
/// are only ever a nudge, never a gate.
const OFF_TOPIC_HOSTS: &[&str] = &[
"legacy.com", "dignitymemorial.com", "echovita.com", "tributearchive.com",
"wwd.com", "footwearnews.com", "hypebeast.com", "pagesix.com", "vogue.com",
"gamesradar.com", "ign.com", "polygon.com", "kotaku.com", "pcgamer.com", "eurogamer.net",
"chowhound.com", "allrecipes.com", "tastingtable.com", "thedailymeal.com",
"espn.com", "realtor.com", "zillow.com",
];
const MUSIC_HOSTS: &[&str] = &[
"pitchfork.com", "stereogum.com", "brooklynvegan.com", "consequence.net", "nme.com",
"rollingstone.com", "spin.com", "billboard.com", "thelineofbestfit.com", "clashmusic.com",
"diymag.com", "loudersound.com", "alternativepress.com", "paste.com", "exclaim.ca",
"thequietus.com", "residentadvisor.net", "ra.co", "faroutmagazine.co.uk", "mojo4music.com",
];
/// Verbs a band GOVERNS. Kept deliberately narrow: adding "brings"/"opens"
/// admitted "bike ride brings history, music and family activities" and
/// "Frog Pond opens 16th season".
const CORE_VERBS: &[&str] = &[
"announce", "announces", "announced", "share", "shares", "shared", "release", "releases",
"released", "drop", "drops", "dropped", "unveil", "unveils", "unveiled", "reveal", "reveals",
"tease", "teases", "teased", "detail", "details", "confirm", "confirms", "confirmed",
"debut", "debuts", "premiere", "premieres", "preview", "previews", "stream", "streams",
"perform", "performs", "headline", "headlines", "tour", "tours", "remix", "remixes",
"reissue", "reissues", "sign", "signs", "cancel", "cancels", "postpone", "postpones",
"reunite", "reunites", "reunited", "add", "adds", "plot", "plots", "map",
];
const SOFT_VERBS: &[&str] = &[
"return", "returns", "launch", "launches", "join", "joins", "take", "takes", "bring",
"brings", "make", "makes", "open", "opens", "set", "sets", "play", "plays", "go", "goes",
];
const ROLE_NOUNS: &[&str] = &[
"band", "bands", "group", "duo", "trio", "quartet", "singer", "rapper", "outfit", "rockers",
"frontman", "frontwoman", "drummer", "guitarist", "bassist", "producer", "act", "collective",
];
const WEAK_ROLES: &[&str] = &["star", "stars", "legend", "legends", "icon", "icons"];
const MUSIC_NOUNS: &[&str] = &[
"album", "albums", "ep", "eps", "lp", "single", "singles", "song", "songs", "track",
"tracks", "tour", "shows", "show", "gig", "gigs", "review", "interview", "video", "festival",
"vinyl", "reissue", "record", "records", "setlist", "concert", "residency",
];
/// Heads that turn a name into a PLACE or an EVENT rather than a band.
const ENTITY_HEADS: &[&str] = &[
"park", "lake", "colony", "county", "hall", "studio", "project", "fest", "festival", "day",
"days", "river", "jam", "lily", "market", "club", "school", "church", "arts", "trail",
"backlot", "road", "street", "lane", "farm", "inn", "resort", "camp", "beach", "bay",
"creek", "valley", "ridge", "grove", "village", "township", "estates", "gardens",
];
const OBIT_MARKERS: &[&str] = &[
"obitu", "passed away", "funeral", "found dead", "in loving memory", "celebration of life",
];
fn words(s: &str) -> Vec<String> {
s.split(|c: char| !c.is_alphanumeric() && c != '\'')
.filter(|w| !w.is_empty())
.map(|w| w.to_ascii_lowercase())
.collect()
}
/// The name forms a headline may legitimately use.
fn name_variants(artist: &str) -> Vec<String> {
let mut out = vec![artist.to_string()];
if artist.contains('&') {
out.push(artist.replace('&', "and"));
}
if artist.to_ascii_lowercase().contains(" and ") {
out.push(artist.replacen(" and ", " & ", 1));
}
let toks: Vec<&str> = artist.split_whitespace().collect();
if toks.len() >= 4 {
// "King Gizzard & The Lizard Wizard" → "King Gizzard". Without this the
// hard gate silently drops genuine stories that use the short form.
out.push(toks[..2].join(" "));
}
out.sort();
out.dedup();
out
}
/// Byte offsets of every whole-token occurrence of `needle` in `hay`, matched
/// case-insensitively on token boundaries. A trailing `'s` still counts —
/// without that, "King Gizzard's Stu Mackenzie" fails the hard gate.
fn token_occurrences(hay: &str, needle: &str) -> Vec<(usize, usize)> {
let h = hay.to_ascii_lowercase();
let n = needle.to_ascii_lowercase();
if n.is_empty() {
return Vec::new();
}
let mut out = Vec::new();
let mut from = 0;
while let Some(rel) = h[from..].find(&n) {
let start = from + rel;
let end = start + n.len();
let before_ok = start == 0
|| !h[..start].chars().next_back().is_some_and(|c| c.is_alphanumeric());
let after = h[end..].chars().next();
let after_ok = match after {
None => true,
Some(c) if !c.is_alphanumeric() => true,
// allow the possessive "'s" which the char scan above already let
// through as an apostrophe; a bare alphanumeric is a different word
_ => false,
};
if before_ok && after_ok {
out.push((start, end));
}
from = end.max(start + 1);
}
out
}
/// Is the character immediately abutting this span an ASCII hyphen? That is
/// what makes "peep-toe", "toe-tapping", "head-to-toe" and "Chorn-Pond" not
/// the band. Spaced en-dashes ("toe – 'NOW I SEE THE LIGHT'") are a different
/// character and are correctly ignored.
fn hyphen_abuts(hay: &str, start: usize, end: usize) -> bool {
let before = hay[..start].chars().next_back() == Some('-');
let after = hay[end..].chars().next() == Some('-');
before || after
}
/// Title Case headlines capitalise everything, so an all-lowercase occurrence
/// carries no information there.
fn is_title_case(title: &str) -> bool {
let long: Vec<&str> = title
.split_whitespace()
.filter(|w| w.chars().filter(|c| c.is_alphabetic()).count() > 3)
.collect();
if long.len() < 3 {
return true; // too short to tell; do not use the case rule
}
let capped = long
.iter()
.filter(|w| w.chars().next().is_some_and(char::is_uppercase))
.count();
capped * 10 > long.len() * 6
}
/// Score a headline for being about `artist`. `None` means rejected outright.
///
/// See research/news/PLAN.md §2 for the derivation of every weight and for the
/// measurements behind the two-part decision at the end.
pub fn news_relevance(item: &ArtistNewsItem, artist: &str, tier: NewsTier) -> Option<i32> {
let title = item.title.as_str();
let lower = title.to_ascii_lowercase();
let variants = name_variants(artist);
// ---- hard gate 1: the name must appear as a whole token run ----------
let mut hits: Vec<(usize, usize)> = Vec::new();
for v in &variants {
hits.extend(token_occurrences(title, v));
}
if hits.is_empty() {
return None;
}
// ---- hard gate 2: every occurrence is inside a hyphenated compound ----
if hits.iter().all(|&(s, e)| hyphen_abuts(title, s, e)) {
return None;
}
hits.retain(|&(s, e)| !hyphen_abuts(title, s, e));
// ---- hard gate 3: canonical capitalisation vs an all-lowercase hit ----
let name_is_capitalised = artist.chars().next().is_some_and(char::is_uppercase);
if name_is_capitalised && !is_title_case(title) {
let any_cased = hits
.iter()
.any(|&(s, e)| title[s..e].chars().next().is_some_and(char::is_uppercase));
if !any_cased {
return None;
}
}
let has = |list: &[&str], w: &str| list.contains(&w);
// ---- document-level signals (counted once) ---------------------------
let name_tokens = artist.split_whitespace().count();
let mut doc = match name_tokens {
0 | 1 => 0,
2 => 2,
_ => 4,
};
if !item.host.is_empty() {
if OFF_TOPIC_HOSTS.iter().any(|h| item.host == *h || item.host.ends_with(&format!(".{h}")))
{
doc -= 6;
} else if MUSIC_HOSTS.iter().any(|h| item.host == *h) {
doc += 2;
}
}
if OBIT_MARKERS.iter().any(|m| lower.contains(m)) {
doc -= 6;
}
if title.contains('"') || title.contains('\u{201c}') || title.contains('\u{2018}') {
doc += 1;
}
// ---- occurrence-level signals — keep the MAX over occurrences --------
// A headline may name the artist several ways; only one of them has to
// carry the evidence.
let mut best_occ = i32::MIN;
for &(s, e) in &hits {
let mut occ = 0i32;
let before_toks = words(&title[..s]);
let after_toks = words(&title[e..]);
let prev = before_toks.last().map(String::as_str);
let next = after_toks.first().map(String::as_str);
let next2 = after_toks.get(1).map(String::as_str);
// "within 2 tokens" is a DISTANCE, so it reaches index 2. Checking only
// the first two positions lost "toe Announces October Shows in Osaka",
// where the music noun is the third word after the name.
let near: Vec<&str> = after_toks.iter().take(3).map(String::as_str).collect();
if next.is_some_and(|w| has(CORE_VERBS, w)) {
occ += 3;
} else if next.is_some_and(|w| has(SOFT_VERBS, w)) {
occ += 1;
}
// possessive followed by a music noun within two tokens
if title[e..].starts_with('\u{2019}') || title[e..].starts_with('\'') {
if next.is_some_and(|w| has(MUSIC_NOUNS, w)) || next2.is_some_and(|w| has(MUSIC_NOUNS, w))
{
occ += 3;
}
}
if prev.is_some_and(|w| has(ROLE_NOUNS, w)) || next.is_some_and(|w| has(ROLE_NOUNS, w)) {
occ += 3;
} else if prev.is_some_and(|w| has(WEAK_ROLES, w)) || next.is_some_and(|w| has(WEAK_ROLES, w))
{
occ += 1;
}
if near.iter().any(|w| has(MUSIC_NOUNS, w)) {
occ += 2;
}
// Head of the headline — or head of what follows a masthead marker,
// which is the same claim: "Review: toe, 'NOW I SEE THE LIGHT'".
// char_indices, not `rfind(..) + 1`: an en/em dash is three bytes, so
// adding one lands mid-character and slicing panics. In an async Tauri
// command that panic surfaces as a promise that never resolves, which
// is why only artists whose name sits mid-headline hung.
let after_marker = title[..s]
.char_indices()
.rev()
.find(|(_, ch)| matches!(ch, ':' | '\u{2013}' | '\u{2014}'))
.is_some_and(|(c, ch)| title[c + ch.len_utf8()..s].trim().is_empty());
if before_toks.is_empty() || after_marker {
occ += 2;
}
// "<Name> — <Work>" / "<Name>: <Work>" masthead convention
let tail = title[e..].trim_start();
if tail.starts_with(':') || tail.starts_with('\u{2013}') || tail.starts_with('\u{2014}') {
occ += 2;
}
// "Review:" / "Listen:" prefix markers
if let Some(colon) = title.find(':') {
if colon < 40 {
let head = title[..colon].to_ascii_lowercase();
if ["review", "interview", "premiere", "listen", "watch", "feature", "video"]
.iter()
.any(|m| head.contains(m))
{
occ += 2;
}
}
}
// ---- negative: the name is really part of a longer proper name ----
// Bare-whitespace adjacency ONLY. Treating any adjacent capitalised
// token as an extension wrongly kills "Review: toe, …" and
// "Pond 'Terrestrials' Review".
if s > 0 {
let gap = &title[..s];
if gap.ends_with(' ') {
let prev_raw = gap.trim_end().rsplit(' ').next().unwrap_or("");
let prev_cap = prev_raw.chars().next().is_some_and(char::is_uppercase);
let sentence_initial = gap.trim_end().is_empty();
let known = prev.is_some_and(|w| {
has(CORE_VERBS, w) || has(SOFT_VERBS, w) || has(ROLE_NOUNS, w)
|| has(WEAK_ROLES, w) || has(MUSIC_NOUNS, w)
});
if prev_cap && !known && !prev_raw.contains('-') {
occ -= if sentence_initial { 2 } else { 4 };
}
// "Amanda M. Pond"
if prev_raw.len() == 2 && prev_raw.ends_with('.') {
occ -= 5;
}
}
}
if next.is_some_and(|w| has(ENTITY_HEADS, w)) {
occ -= 4;
} else if next2.is_some_and(|w| has(ENTITY_HEADS, w)) {
occ -= 3;
}
// "across the Pond", "over the pond"
if before_toks.len() >= 2 {
let a = before_toks[before_toks.len() - 2].as_str();
let b = before_toks[before_toks.len() - 1].as_str();
if b == "the" && ["across", "over", "from", "in", "into", "beyond"].contains(&a) {
occ -= 6;
}
}
if prev.is_some_and(|w| ["a", "an", "the", "to"].contains(&w)) && !artist.starts_with("The ")
{
occ -= 3;
}
best_occ = best_occ.max(occ);
}
if best_occ == i32::MIN {
return None;
}
let total = doc + best_occ;
// The occurrence floor is the whole filter. At a single combined threshold
// every false positive scored at threshold from document-level signals
// alone — "Every song on the 'Tip Toe' soundtrack" is a music-press host
// plus a quoted title plus a music noun and zero evidence at the name.
// Publisher reputation must never admit an item on its own.
if total >= tier.threshold() && best_occ >= tier.min_occurrence() {
Some(total)
} else {
None
}
}
/* ============================ news dedup ==================================
*
* Collapsing several outlets' reports of ONE story. Runs after the filter and
* before the cap. See research/news/PLAN.md §3.
*
* Two counter-intuitive choices, both measured:
* - CONTAINMENT, not Jaccard. Headline lengths are wildly asymmetric —
* Stereogum's six-token 'King Gizzard & The Lizard Wizard – "Alien Metal"'
* against a thirty-token version of the same story — and Jaccard's union
* denominator punishes exactly that (F1 0.61 vs 0.44).
* - KEEP the artist-name tokens. Stripping them starves a six-token headline
* to two and was measured to lose real merges and invent false ones. The
* name is a uniform offset across every pair in a per-artist feed, so it
* moves the operating point rather than the ranking — which the raised
* 0.6 threshold and the non-name-overlap clause compensate for.
*/
const NEWS_STOP: &[&str] = &[
"a", "an", "the", "of", "to", "in", "on", "at", "for", "with", "and", "or", "from", "is",
"are", "was", "were", "be", "been", "by", "as", "it", "its", "this", "that", "new", "now",
"has", "have", "had", "will", "their", "his", "her", "they", "he", "she", "you", "your",
"our", "says", "say", "said", "more", "than", "then", "but", "not", "out", "up", "off",
"over", "into", "about", "after", "before", "some", "all", "how", "why", "what", "when",
"who", "his", "hers", "via", "amp",
];
/// Cap applied AFTER filtering and dedup. The frontend's band asks for up to
/// `bandCols * 3` items, so the old 12 starved a wide window.
const NEWS_KEEP: usize = 40;
/// Score, reject, collapse, cap — the whole pipeline over one parsed feed.
///
/// If the filter empties the list, that is the answer: an empty row is
/// strictly better than the unfiltered feed, which for an ambiguously named
/// artist is 100% junk. There is deliberately no fall back to unfiltered.
fn filter_news(raw: Vec<ArtistNewsItem>, artist: &str, started: Instant) -> Vec<ArtistNewsItem> {
let fetched = raw.len();
let tier = news_tier(artist);
let mut scored: Vec<(ArtistNewsItem, i32)> = raw
.into_iter()
.filter_map(|item| news_relevance(&item, artist, tier).map(|s| (item, s)))
.collect();
let kept = scored.len();
// Google's relevance order is the tiebreaker worth preserving, so only
// sort by date once the set is final.
let mut out = dedup_news(std::mem::take(&mut scored), artist);
let deduped = out.len();
// Sort BEFORE the quota so the two survivors of each story are its newest,
// and quota BEFORE the cap so the forty slots are forty stories.
out.sort_by_key(|item| -item.date);
let mut out = news_story_quota(out, artist);
let storied = out.len();
out.truncate(NEWS_KEEP);
// "fetched 100, kept 0" and "fetched 0" must never read the same — the
// trap already recorded for MusicBrainz probing.
stage_trace(
"artist-news",
started,
&format!(
"{artist} tier={tier:?} fetched={fetched} kept={kept} deduped={deduped} storied={storied}"
),
);
out
}
/// Comparable token set for one headline. The " - Publisher" suffix is already
/// stripped by `parse_news_rss`; do not re-implement it here.
fn news_signature(title: &str) -> std::collections::BTreeSet<String> {
let folded = title
.replace('\u{2019}', "'")
.replace(['\u{201c}', '\u{201d}'], "\"")
.replace(['\u{2013}', '\u{2014}'], "-")
.replace('&', " and ");
folded
.split(|c: char| !c.is_alphanumeric())
.filter(|w| w.len() > 2)
.map(|w| w.to_ascii_lowercase())
.filter(|w| !NEWS_STOP.contains(&w.as_str()))
.collect()
}
/// Collapse near-duplicate reports, keeping the best-scoring member of each
/// cluster and recording how many others it stands for.
///
/// Leader clustering, NOT union-find: union-find at a comparable threshold
/// chained 68 of 93 King Gizzard items — a Nonagon Infinity review and a
/// Phantom Island review included — into a single "story".
fn dedup_news(items: Vec<(ArtistNewsItem, i32)>, artist: &str) -> Vec<ArtistNewsItem> {
let name_toks: std::collections::BTreeSet<String> = news_signature(artist);
let sigs: Vec<_> = items.iter().map(|(i, _)| news_signature(&i.title)).collect();
// leader index -> members
let mut clusters: Vec<(usize, Vec<usize>)> = Vec::new();
'item: for i in 0..items.len() {
for (leader, members) in clusters.iter_mut() {
let (a, b) = (&sigs[*leader], &sigs[i]);
let inter: Vec<&String> = a.intersection(b).collect();
let (da, db) = (items[*leader].0.date, items[i].0.date);
// Google truncates syndicated copies with an ellipsis, which is
// exactly the GamesRadar+/inkl.com pair.
let ta = items[*leader].0.title.trim_end_matches(['\u{2026}', '.', ' ']);
let tb = items[i].0.title.trim_end_matches(['\u{2026}', '.', ' ']);
let prefix = ta.len() >= 24
&& tb.len() >= 24
&& (ta.starts_with(tb) || tb.starts_with(ta));
// `rfc2822_secs` yields 0 on a parse failure, so two unparseable
// items would otherwise show |delta| = 0 and always pass the window.
let dated = da != 0 && db != 0 && (da - db).abs() <= 2 * 86_400;
let denom = a.len().min(b.len()).max(5) as f32;
let containment = inter.len() as f32 / denom;
let non_name = inter.iter().filter(|t| !name_toks.contains(**t)).count();
let same = prefix
|| (dated && containment >= 0.6 && inter.len() >= 3 && non_name >= 2);
if same {
members.push(i);
continue 'item;
}
}
clusters.push((i, vec![i]));
}
let mut out = Vec::new();
for (_, members) in clusters {
// Survivor = best score; tie-break to the earliest report.
let best = *members
.iter()
.min_by(|&&x, &&y| {
items[y].1
.cmp(&items[x].1)
.then_with(|| items[x].0.date.cmp(&items[y].0.date))
})
.expect("cluster is never empty");
let mut item = items[best].0.clone();
item.others = (members.len() - 1) as u32;
out.push(item);
}
out
}
/* ======================= per-story display quota ==========================
*
* What dedup cannot reach. Leader clustering collapses REWORDINGS of one
* report, but it leaves stragglers: twelve outlets covering the "Alien Metal"
* announcement produced one 11-member cluster plus six loners that are similar
* to each other and not to the leader that already absorbed the rest. Measured
* on the live feed, seven of the visible twelve slots were that one record.
*
* The remedy is NOT a lower dedup threshold. Lexical similarity merges
* unrelated stories long before it catches rewrites — Cleveland's "Beloved band
* pulls its music from major streaming service" and InsideHook's "Scammers
* Share Fake King Gizzard Songs on Spotify" are the same story and share
* literally zero content words. Closing that gap needs sentence embeddings.
* See research/news/PLAN.md §3.
*
* So bound the damage instead: at most two items may name the same WORK. The
* key is the album or song title the headline is about, which syndicated
* coverage repeats verbatim even when it shares no other vocabulary. Headlines
* that name no work are never dropped — an artist's feed can shrink but can
* never be emptied by this.
*/
/// Nouns after which a title is a work title. "toe Announces October Shows"
/// carries no work; "Detail New Album Alien Metal" does.
const WORK_CUES: &[&str] = &[
"album", "albums", "lp", "ep", "single", "singles", "track", "song", "mixtape", "record",
];
/// Words that are never a work title on their own. Without these, "…Album
/// Review" keys as `review` and "…New Song Next Week" as `next week`, and two
/// unrelated headlines share a story.
const GENERIC_TITLE_WORDS: &[&str] = &[
"review", "reviews", "interview", "interviews", "stream", "listen", "watch", "premiere",
"premieres", "video", "videos", "news", "tour", "tours", "dates", "list", "setlist", "week",
"weekend", "day", "today", "tomorrow", "release", "releases", "single", "singles", "album",
"albums", "ep", "lp", "song", "songs", "track", "tracks", "next", "first", "new", "out",
"now", "more",
];
/// A capitalised function word does not continue a title: "New Song With Cindy
/// Lee" is a credit, not a work called "With Cindy Lee".
const RUN_STOP_WORDS: &[&str] =
&["with", "from", "by", "featuring", "feat", "and", "for", "on", "in", "at", "to", "of", "via", "ft"];
/// Every quoted span in a headline, as (contents, byte range covering the
/// quotes themselves).
///
/// Two rules earn their keep. A straight quote only OPENS at a word boundary —
/// otherwise a possessive ("Ware's") opens a span that closes on the next real
/// quote and misaligns every span after it. And a curly opener may close on its
/// straight twin, because real feeds mix them inside one headline
/// (`Produce ‘The Gold Album'`).
fn quoted_spans(title: &str) -> Vec<(String, usize, usize)> {
let chars: Vec<(usize, char)> = title.char_indices().collect();
let mut out = Vec::new();
let mut i = 0;
while i < chars.len() {
let (idx, c) = chars[i];
let closers: &[char] = match c {
'\u{2018}' => &['\u{2019}', '\''],
'\u{201c}' => &['\u{201d}', '"'],
'"' => &['"'],
'\'' => &['\''],
_ => {
i += 1;
continue;
}
};
let curly = matches!(c, '\u{2018}' | '\u{201c}');
let opens = curly || i == 0 || !chars[i - 1].1.is_alphanumeric();
let mut advance = 1;
if opens {
let closed = (i + 1..chars.len()).find(|&j| {
closers.contains(&chars[j].1)
&& (curly || j + 1 >= chars.len() || !chars[j + 1].1.is_alphanumeric())
});
if let Some(j) = closed {
let (close_idx, close_ch) = chars[j];
let inner = title[idx + c.len_utf8()..close_idx].to_string();
out.push((inner, idx, close_idx + close_ch.len_utf8()));
advance = j - i + 1;
}
}
i += advance;
}
out
}
/// Normalise a candidate work title into a comparison key, or reject it.
///
/// Artist-name tokens and a leading article come off, which is what unifies
/// "Weezer: The Gathering" with "The Gathering" — five headlines about one
/// tour that would otherwise split into two keys and pass the quota twice.
fn work_key(raw: &str, name_toks: &HashSet<String>) -> Option<String> {
let mut toks: Vec<String> = raw
.split(|c: char| !c.is_alphanumeric())
.filter(|w| !w.is_empty())
.map(|w| w.to_ascii_lowercase())
.collect();
// A span this long is speech, not a title: "It's 5 days since they released
// new music – was starting to think we'd never hear from them again".
if toks.len() > 8 {
return None;
}
toks.retain(|w| !name_toks.contains(w));
while matches!(toks.first().map(String::as_str), Some("the" | "a" | "an")) {
toks.remove(0);
}
if toks.is_empty() || toks.len() > 6 {
return None;
}
if toks
.iter()
.all(|w| GENERIC_TITLE_WORDS.contains(&w.as_str()) || NEWS_STOP.contains(&w.as_str()))
{
return None;
}
let key = toks.join(" ");
(key.len() >= 3).then_some(key)
}
fn bare_word(w: &str) -> String {
w.chars().filter(|c| c.is_alphanumeric()).collect::<String>().to_ascii_lowercase()
}
/// Whitespace-separated words with their byte offsets. `split_whitespace`
/// alone loses the offsets, and the offsets are what tell a work cue inside a
/// quoted span from one outside it.
fn offset_words(s: &str) -> Vec<(usize, &str)> {
let mut out = Vec::new();
let mut start: Option<usize> = None;
for (i, c) in s.char_indices() {
if c.is_whitespace() {
if let Some(at) = start.take() {
out.push((at, &s[at..i]));
}
} else if start.is_none() {
start = Some(i);
}
}
if let Some(at) = start {
out.push((at, &s[at..]));
}
out
}
/// The last whole word before a byte offset.
fn word_before(title: &str, at: usize) -> String {
title[..at]
.split(|c: char| !c.is_alphanumeric())
.rfind(|w| !w.is_empty())
.unwrap_or("")
.to_ascii_lowercase()
}
/// The work a headline is about — an album or song title, normalised — or
/// `None` when it names none. `None` is common and is not a failure: a tour
/// announcement or an interview is its own story and is never quota'd.
pub fn news_story_key(title: &str, artist: &str) -> Option<String> {
let name_toks: HashSet<String> = artist
.split(|c: char| !c.is_alphanumeric())
.filter(|w| !w.is_empty())
.map(|w| w.to_ascii_lowercase())
.collect();
let raw_spans = quoted_spans(title);
let keyed: Vec<(String, usize)> = raw_spans
.iter()
.filter_map(|(raw, at, _)| work_key(raw, &name_toks).map(|k| (k, *at)))
.collect();
// 1. a quoted span right after a work cue IS the work — this outranks a
// bare quote, which is as often speech ("'We are truly doomed'").
if let Some((key, _)) = keyed
.iter()
.find(|(_, at)| WORK_CUES.contains(&word_before(title, *at).as_str()))
{
return Some(key.clone());
}
// 2. an unquoted Title Case run after a work cue, closed by end of headline
// or by `:` `,` or a dash. The closing requirement is what keeps
// "Fueling Album Speculation [Watch]" from keying as a record.
let words = offset_words(title);
for i in 0..words.len() {
let (at, word) = words[i];
if !WORK_CUES.contains(&bare_word(word).as_str()) {
continue;
}
// A cue INSIDE a quote is part of a title, not an introduction to one:
// `A “Sweater Song” Homage` was keying as the work "homage".
if raw_spans.iter().any(|&(_, s, e)| at >= s && at < e) {
continue;
}
let mut run: Vec<&str> = Vec::new();
let mut j = i + 1;
let mut closed = false;
while j < words.len() {
let w = words[j].1;
if !w.chars().next().is_some_and(|c| c.is_uppercase() || c.is_numeric()) {
break;
}
if RUN_STOP_WORDS.contains(&bare_word(w).as_str()) {
break;
}
run.push(w);
j += 1;
if w.ends_with([':', ',', '\u{2013}', '\u{2014}']) {
closed = true;
break;
}
}
if run.is_empty() {
continue;
}
if !closed {
closed =
j >= words.len() || words[j].1.starts_with([':', ',', '\u{2013}', '\u{2014}']);
}
if !closed {
continue;
}
if let Some(key) = work_key(&run.join(" "), &name_toks) {
return Some(key);
}
}
// 3. any other quoted span
keyed.into_iter().next().map(|(key, _)| key)
}
/// At most this many headlines about one work. Two rather than one: a record
/// that is genuinely in the news deserves more than a single line, and the two
/// kept are usually the announcement and the release.
const NEWS_STORY_MAX: usize = 2;
/// Keep at most `NEWS_STORY_MAX` items per work, in the order given — so
/// calling this on a date-sorted list keeps the newest of each story.
pub fn news_story_quota(items: Vec<ArtistNewsItem>, artist: &str) -> Vec<ArtistNewsItem> {
let mut seen: HashMap<String, usize> = HashMap::new();
items
.into_iter()
.filter(|item| match news_story_key(&item.title, artist) {
None => true,
Some(key) => {
let n = seen.entry(key).or_insert(0);
*n += 1;
*n <= NEWS_STORY_MAX
}
})
.collect()
}
/// Wikimedia hosts share one UA policy and one rate limiter, so they share
/// this client's compliant UA, pacer, 429 handling and disk cache. Wikidata is
/// included: it is the identity anchor that turns a MusicBrainz artist into
@@ -839,6 +1665,21 @@ pub struct ArtistNewsItem {
pub url: String,
pub source: String,
pub date: i64,
/// Publisher host from `<source url="…">` — "wwd.com", "pitchfork.com".
/// The element's TEXT is a display name ("GamesRadar+") and is not a host;
/// the attribute is, and it is present on every item.
///
/// `serde(default)` is load-bearing: cached items are read back with
/// `from_value::<Vec<ArtistNewsItem>>(..).ok()`, and `.ok()` swallows a
/// schema mismatch — a required new field would silently invalidate every
/// cached entry and fire a cold refetch for every follow at once, against
/// a rate-limited endpoint.
#[serde(default)]
pub host: String,
/// How many near-duplicate reports of this same story were collapsed into
/// it, so the UI can say "+N more outlets" instead of dropping them mutely.
#[serde(default)]
pub others: u32,
}
/// Lightweight artist identity + artwork response used to paint the profile
@@ -1503,7 +2344,13 @@ impl MetadataClient {
// whole point of using them -- an unpaid ordering must not cost a
// network round on every revisit, which is exactly what TheAudioDB
// suffers for NOT being here.
|| url.starts_with("https://ws.audioscrobbler.com/");
|| url.starts_with("https://ws.audioscrobbler.com/")
// `artist_news` has always written and read a disk entry here, but
// news URLs never qualified, so `disk_cache_path` returned None and
// the whole branch was dead: every restart refetched every follow
// against a rate-limited endpoint. The 24h TTL this reader enforces
// is the outer bound; news carries its own 6h stamp inside the body.
|| url.starts_with("https://news.google.com/rss/");
if !qualifies {
return None;
}
@@ -7041,8 +7888,7 @@ impl MetadataClient {
return Vec::new();
}
};
let items = parse_news_rss(&text);
stage_trace("artist-news", started, artist);
let items = filter_news(parse_news_rss(&text), artist, started);
if let Some(path) = self.disk_cache_path(&url) {
let body = serde_json::json!({ "at": now_secs(), "items": items });
write_disk_cache(&path, &url, &body);