phenom/SoulStream
1
0 Fork 0
forked from Keelrleek/SoulStream
Code Pull Requests Activity
Files
main
SoulStream/crates/metadata/tests/news_story_quota.rs
T
KeelrLeek d397b78887 news: about the artist, and each story only once 
Kyle reported duplicate articles and a few unrelated ones. Measuring the
live feed found it far worse than the handful he saw: 84 items across
seven follows, hand-labelled, 26 junk — and 24 of the 26 came from two
artists. toe returned 0 of 12 genuine, Pond 0 of 12. Round Pond, Goose
Pond Colony, Inlet Pond County Park, "Amanda M. Pond", peep-toe mules, a
funeral-home obituary. Not one item about the Australian band.

The largest single cause was not the query. parse_news_rss kept the first
twelve items and sorted by date afterwards, so it discarded ranks 13-100
of Google's relevance order — and every genuine toe item sits at rank
15-88. Nothing downstream could recover what was thrown away there. All
100 are parsed now; the cap is 40 and applies after filtering and dedup,
because the frontend's band asks for bandCols * 3 and twelve starved a
wide window.

news_relevance is the precision mechanism. Hard gates first: the name
must appear as a whole token run, every occurrence must not abut a hyphen
(peep-toe, head-to-toe, Chorn-Pond), and a canonically capitalised name
in a sentence-case headline must appear cased. Then document-level
signals, then occurrence-level signals computed for every occurrence with
the maximum kept — and the decision is a total AND a separate floor on
the occurrence subscore. That second clause is the whole filter: at a
single combined threshold every false positive scored at threshold from
document signals alone. "Every song on the 'Tip Toe' soundtrack" is a
music-press host plus a quoted title plus a music noun and carries zero
evidence at the name. Publisher reputation must never admit an item on
its own.

Tiers come from a short explicit dictionary, not from name length and not
from a frequency table. Length demoted Weezer to the strict path and cost
it four genuine items; corpus rank sent Wand from 16 kept to 45 because
the feed floods with "Wistoria: Wand and Sword", and took toe's precision
from 1.00 to 0.70.

dedup_news collapses several outlets' wording of one report — containment
rather than Jaccard, because headline lengths are wildly asymmetric and
Jaccard's union denominator punishes exactly that; artist tokens retained,
because stripping them starves a six-token headline to two; leader
clustering, because union-find chained 68 of 93 King Gizzard items into
one "story"; a 2-day window with unparseable dates non-mergeable, since
rfc2822_secs returns 0 on failure and two zeroes always satisfy a window.
The survivor carries an others count for the outlets it stands for.

That still left the other kind of repetition: many different reports about
one record, similar to each other but not to the leader that absorbed the
rest. Six of the visible twelve were the same Alien Metal announcement.
The fix is not a lower dedup threshold — lexical similarity merges
unrelated stories long before it catches rewrites — but a ceiling:
news_story_key names the album or single a headline is about, and
news_story_quota keeps at most two per work, applied after the date sort
so the survivors are a story's newest. Headlines naming no work are never
dropped, so a feed can shrink but can never be emptied.

Two things that were quietly broken. News URLs never qualified in
disk_cache_path, so the entire disk branch in artist_news was unreachable
and every restart refetched every follow against a rate-limited endpoint.
And rfind on a set of dashes plus one lands mid-character on a 3-byte
en-dash: the slice panics, and a panic inside an async Tauri command never
resolves its promise, so the News band hung forever — but only for artists
whose name sits mid-headline.

The host now comes from the source element's url attribute; its text is a
display name ("GamesRadar+") and is not a host. Both new fields are
serde(default), because cached items are read back through an .ok() that
swallows a schema mismatch: a required field would silently invalidate
every cached entry and fire a cold refetch for every follow at once.

Measured live: toe 12 items (0 real) to 5 (5 genuine), Pond 12 (0 real) to
7 (7 genuine), 32 duplicate clusters collapsed covering 71 redundant
reports, and no story takes more than two of the twelve visible slots.
Derivation, the falsified alternatives and a what-not-to-do section are in
research/news/PLAN.md.
2026-08-05 09:46:54 -05:00

195 lines
8.0 KiB
Rust
Raw Permalink Blame History

//! The per-story display quota, measured against real headlines.
//!
//! Dedup collapses rewordings of one report; this bounds the other kind of
//! repetition — many different reports about the same record. Every headline
//! here is verbatim from a live Google News feed on 2026-08-05, where seven of
//! the twelve visible slots were King Gizzard's "Alien Metal" announcement
//! arriving from seven outlets that shared almost no vocabulary.
use metadata::{news_story_key, news_story_quota, ArtistNewsItem};
const GIZZ: &str = "King Gizzard & The Lizard Wizard";
fn key(title: &str, artist: &str) -> Option<String> {
news_story_key(title, artist)
}
fn item(title: &str, date: i64) -> ArtistNewsItem {
ArtistNewsItem {
title: title.to_string(),
url: title.to_string(),
source: String::new(),
date,
host: String::new(),
others: 0,
}
}
// ------------------------------------------------------------ extraction
#[test]
fn every_alien_metal_headline_names_the_same_work() {
// Five quote it, two only capitalise it after "Album". All seven are one
// story and between them they held seven of twelve panel slots.
for title in [
"King Gizzard & the Lizard Wizard rave on with some \u{2018}Alien Metal\u{2019}",
"King Gizzard & the Lizard Wizard announce 28th album \u{2018}Alien Metal\u{2019}",
"King Gizzard & the Lizard Wizard Share Title Track From Forthcoming LP \u{2018}Alien Metal\u{2019}",
"King Gizzard and the Lizard Wizard announce EDM album, \u{2018}Alien Metal\u{2019}",
"King Gizzard & The Lizard Wizard Announce 28th Album 'Alien Metal,' Brooklyn Rave",
"King Gizzard & The Lizard Wizard Release Title Track from Techno Album Alien Metal: Stream",
"King Gizzard & The Lizard Wizard Detail New Album Alien Metal",
] {
assert_eq!(key(title, GIZZ).as_deref(), Some("alien metal"), "{title}");
}
}
#[test]
fn a_cue_adjacent_quote_outranks_an_earlier_one() {
// The album is the story; "Two Hands" is the track it leads with.
assert_eq!(
key("Pond Announce New Album 'Terrestrials': Hear \"Two Hands\"", "Pond").as_deref(),
Some("terrestrials"),
);
assert_eq!(
key("Pond Share New Single \"Terrestrials\": Listen", "Pond").as_deref(),
Some("terrestrials"),
);
}
#[test]
fn a_curly_quote_may_close_on_its_straight_twin() {
// Verbatim from the feed — one headline, two different quote characters.
assert_eq!(
key("Weezer Enlists Kenny Beats to Produce \u{2018}The Gold Album'", "Weezer").as_deref(),
Some("gold album"),
);
}
#[test]
fn the_artist_name_and_a_leading_article_come_off_the_key() {
// Five headlines about one tour. Without this they split into "the
// gathering" and "weezer the gathering" and pass the quota twice.
for title in [
"Weezer Announce \u{2018}The Gathering\u{2019} Arena Tour",
"Weezer Announce \u{201c}Weezer: The Gathering\u{201d}",
"'Weezer: The Gathering' tour heading to Nationwide Arena in October",
"Weezer Play Pop-Up Rooftop Gig to Mark \u{2018}The Gathering\u{2019} Tour",
] {
assert_eq!(key(title, "Weezer").as_deref(), Some("gathering"), "{title}");
}
}
#[test]
fn a_work_cue_inside_a_quote_does_not_introduce_one() {
// Three headlines about the single "C.E.O.", live on 2026-08-05. The third
// quotes an older song, and the cue inside THAT quote was keying the
// headline as a work called "homage" — a third slot for one story.
for title in [
"Weezer\u{2019}s \u{201c}C.E.O.\u{201d} Video Has Curry Barker, Celeb Cameos, & A \u{201c}Sweater Song\u{201d} Homage",
"Weezer Unveil New Single \"C.E.O.\", Expand \"Weezer: The Gathering\" with New Tour Dates",
"Weezer Go Back in Time for \u{2018}C.E.O.\u{2019} Music Video",
] {
assert_eq!(key(title, "Weezer").as_deref(), Some("c e o"), "{title}");
}
}
#[test]
fn a_possessive_apostrophe_does_not_open_a_span() {
// If it did, the span would run from "'s Rivers…" to the next real quote
// and every span after it would be misaligned.
assert_eq!(
key("Weezer's Rivers Cuomo drops new single 'Shine Again'", "Weezer").as_deref(),
Some("shine again"),
);
}
// ------------------------------------------------------- what is NOT a work
#[test]
fn a_generic_run_after_a_cue_is_not_a_work() {
// Each of these keyed as a "work" before the generic list existed —
// `review`, `list` and `next week` — grouping unrelated headlines.
for (title, artist) in [
("Panda Bear / Sonic Boom: A ? of WHEN Album Review", "Panda Bear"),
("Weezer Concert Setlist: Discover the Average Song List", "Weezer"),
(
"Weezer Announce North American Tour, Tease New Album, and Are Releasing a New Song Next Week",
"Weezer",
),
("Track Review: Paramore \u{2013} Running Out of Time", "Panda Bear"),
] {
assert_eq!(key(title, artist), None, "{title}");
}
}
#[test]
fn an_unclosed_run_is_not_a_work() {
// "Album Speculation [Watch]" — a run that neither ends the headline nor
// stops at punctuation is ordinary prose.
assert_eq!(
key("Billy Strings & King Gizzard Share Video From The Studio, Fueling Album Speculation [Watch]", GIZZ),
None,
);
// A capitalised function word ends the run: this is a credit, not a work.
assert_eq!(
key("Panda Bear Announces Tour and New Album, Shares Video for New Song With Cindy Lee: Watch", "Panda Bear"),
None,
);
}
#[test]
fn a_long_quote_is_speech_not_a_title() {
assert_eq!(
key("\u{201c}It\u{2019}s 5 days since they released new music \u{2013} was starting to think we\u{2019}d never hear from them again\u{201d}: King Gizzard And The Lizard Wizard\u{2019}s 24 albums in 11 years?", GIZZ),
None,
);
}
#[test]
fn headlines_that_name_no_work_are_never_keyed() {
// These are the majority, and they are the reason the quota can never
// empty a feed.
assert_eq!(key("King Gizzard & The Lizard Wizard Share New Single", GIZZ), None);
assert_eq!(key("King Gizzard & The Lizard Wizard Australia 2026 Tour", GIZZ), None);
assert_eq!(key("toe Announces October Shows in Osaka and Tokyo", "toe"), None);
}
// ----------------------------------------------------------------- quota
#[test]
fn the_quota_keeps_the_first_two_of_a_story_and_all_the_rest() {
// The list arrives newest-first, so "first two" is "the two most recent".
let titles = [
"King Gizzard & the Lizard Wizard rave on with some \u{2018}Alien Metal\u{2019}",
"King Gizzard & The Lizard Wizard Share New Single",
"King Gizzard & The Lizard Wizard Release Title Track from Techno Album Alien Metal: Stream",
"King Gizzard & the Lizard Wizard announce 28th album \u{2018}Alien Metal\u{2019}",
"King Gizzard & The Lizard Wizard Detail New Album Alien Metal",
"King Gizzard & The Lizard Wizard Australia 2026 Tour",
];
let items: Vec<_> = titles
.iter()
.enumerate()
.map(|(i, t)| item(t, 1_785_000_000 - i as i64 * 86_400))
.collect();
let out = news_story_quota(items, GIZZ);
let kept: Vec<&str> = out.iter().map(|i| i.title.as_str()).collect();
assert_eq!(kept, vec![titles[0], titles[1], titles[2], titles[5]]);
}
#[test]
fn the_quota_never_empties_a_feed() {
// Pond's whole in-feed life was five reports of one record. Two survive,
// which is the point: an empty row would be worse than a repeated one.
let items: Vec<_> = [
"Pond Return With Psych\u{2019}d Out Track \u{2018}Terrestrials\u{2019}",
"Pond Share New Single \"Terrestrials\": Listen",
"Pond Announce North American Tour Behind New Album 'Terrestrials'",
"Pond Announce New Album 'Terrestrials': Hear \"Two Hands\"",
"Pond \u{2018}Terrestrials\u{2019} Review: Less Sometimes Is Really More",
]
.iter()
.map(|t| item(t, 1_785_000_000))
.collect();
assert_eq!(news_story_quota(items, "Pond").len(), 2);
}
View Git Blame Copy Permalink