From c0fa892eee2eaaca8dc485c4fb58f63b378df21a Mon Sep 17 00:00:00 2001 From: KeelrLeek <156356974+KeelrLeek@users.noreply.github.com> Date: Tue, 28 Jul 2026 09:42:22 -0500 Subject: [PATCH] catalog: what the artist page can know once it asks somebody other than Deezer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One supplemental fetch behind the artist page -- the linked MBID, then Last.fm and MusicBrainz in parallel -- and four separate things Deezer had been getting wrong stop being wrong. They ship together because they are the same fetch: `ArtistSupplements` carries scrobble counts and first-release dates out of one `tokio::join!`, `ReleaseFacts` serves dates, disambiguation and attribution alike, and `TrackStats` is read by both the Popular ordering and the album expander. Splitting them would mean inventing intermediate states that never ran. **Popular is ordered by scrobbles, not by recency.** Deezer's `/top` is recency-weighted, so a week-old single outranks the songs an artist is actually known for -- Taylor Swift's Deezer top three are all from the current release where Last.fm's are Blank Space, Cruel Summer, Shake It Off. Measured across 18 artists, Last.fm's canonical top ten is only 43% present in Deezer's top 10 but 77% by its top 50, and the curve flattens after that (79% at 100), so the pool is 50 and the ten that survive are reordered rather than replaced. Nothing is ever admitted that Deezer did not supply: Last.fm ranks rows, it does not add them, so a track with no stream stays off the page. **Albums are dated by their first release, not their latest pressing.** Deezer dates a row from whatever pressing it licensed, so remasters scatter a discography across decades. MusicBrainz release groups supply the real date over three 100-item pages, which covers a very prolific catalogue (Queen has ~100 album groups alone) without walking a discography forever behind the mandatory 1 req/s gate. **MusicBrainz separates seven self-titled Weezer albums by `disambiguation`, and ignoring that field collapsed all of them.** Every one is titled plain "Weezer"; only the disambiguator ("Blue Album" … "Gold Album") tells them apart. Keyed on title alone, all seven folded onto one entry and kept the earliest date, which is how the 2026 Gold Album came out dated 1994-05-10 and filed as an edition of the Red Album. Groups are now indexed under their disambiguated name as well, which is an exact match rather than a fuzzy one because Deezer prints the same disambiguator in its own parenthetical. Three things fall out of that: a key that collapses groups with DIFFERENT dates is marked `ambiguous` and dates nothing (equal dates are a reissue duplicate and still fine); "Weezer (Deluxe Edition / Red Album)" keeps the colour instead of losing the whole bracket; and a pressing takes its work's date through the tracklist-verified `edition_group`, which knows what titles cannot. Verified: Gold → 2026-08-21 in Upcoming and standalone, Red → 2008 with its three editions, Green → 2001-05-07, every deluxe sitting with its own record. **A popular track is shown on the album it came from, not the one Deezer licensed.** This is the big one. Bohemian Rhapsody came back attributed to the 2018 film soundtrack, wearing that poster as its cover, and 10 of Queen's top 10 were wrong the same way. The rule takes the earliest album-typed release that actually contains the song, and swaps `album`, `album_id`, `cover`, `id`, `preview` and `duration_secs` together with a new `origin_track_id`. Live: Queen 0/10 → 10/10, Fleetwood Mac 10/10, Radiohead 10/10, Green Day 9/10 -- the documented Kerplunk! re-recording being the only miss. It costs 0 ms, because the album details it reads are already fetched for the discography. Three things that look like details and are not: - The MusicBrainz secondary-type exclusion is load-bearing: 79/80 with it, 63/80 without. Compilations are dated EARLIER than the works they compile, so "earliest wins" on its own actively SELECTS hits packages. - "Already correct" has to be an OUTPUT test, not a candidate filter. Removing the track's current album from the ranking means a correct row can only be handed to the runner-up, which moved Dreams off plain Rumours onto Rumours (Super Deluxe). 57 of the 80 measured tracks are in that already-correct state, so this is the common case, not the edge one. - A lead single always predates its own album, and EPs are admitted as candidates, so Creep moved off Pablo Honey onto a "Creep EP" five months earlier. Album now outranks EP above the date tiebreak -- it cannot affect a row whose winner is already an album, and an EP still wins when no album carries the song. Radiohead 9/10 → 10/10. The details cap becomes two-sided for this: take 60, then top up to 80 with the OLDEST album-typed rows. A bare 60 silently dropped Bob Dylan's canon -- Highway 61, Freewheelin', Blonde on Blonde -- which is exactly the material the rule exists to find. Known residual, seen in the wild: Blowin' in the Wind lands on The Witmark Demos. MusicBrainz DOES type it Compilation, but calls it "The Bootleg Series, Vol. 9: The Witmark Demos: 1962–1964" where Deezer calls it "The Witmark Demos: 1962-1964 (The Bootleg Series Vol. 9)". Different word order, different title-join key, so the exclusion cannot see it. Not worth a title heuristic. **Per-track play counts, for zero extra requests.** `artist.getTopTracks` was already being fetched for the ordering above and carries `playcount` beside `listeners`. The depth goes 100 → 1000, which is the same one request: at 100 it reached 2 of Van Weezer's 10 tracks, at 1000 it covered 56 of 57 album tracks measured across Johnny Cash, the Grateful Dead, Miles Davis, Dylan and Queen. That works because the long tail is bootlegs and mislabelled scrobbles -- a real album track is by definition one of the popular ones. Cost is body size, 77 KB → ~775 KB, disk cached for a day. Album totals need a guard the track counts do not. Last.fm's album `playcount` counts scrobbles tagged to that release page while track counts span every release, so Weezer's Blue deluxe reported 89k album plays above a first track showing 7.2M. A provider total that cannot cover its own biggest track is rejected as an edition artifact and the tracklist is summed instead. --- crates/metadata/src/lib.rs | 2329 +++++++++++++++++++++++++++++++++++- 1 file changed, 2270 insertions(+), 59 deletions(-) diff --git a/crates/metadata/src/lib.rs b/crates/metadata/src/lib.rs index 1695b69..abb189a 100644 --- a/crates/metadata/src/lib.rs +++ b/crates/metadata/src/lib.rs @@ -228,14 +228,19 @@ const DEEZER_RATE_BUDGET: usize = 45; /// reaches a caller as "this resource does not exist". const DEEZER_QUOTA_ATTEMPTS: usize = 3; const DEEZER_QUOTA_BACKOFF: Duration = Duration::from_millis(1600); -/// Wikimedia's UA policy requires a client that identifies itself with a -/// contact point; the shared default UA reads as an anonymous bot and gets a -/// savage limiter (measured 2026-07-22: ~13 requests then HTTP 429 with -/// `Retry-After: 51`, which silently zeroed every album description on a -/// cold artist page). A compliant UA sustained 45+ back-to-back requests -/// with no rejection. Applied to wikipedia.org requests only — other -/// providers (notably the Bing HTML scrape) must keep seeing the default UA. -const WIKIPEDIA_USER_AGENT: &str = +/// UA for providers that publish a policy about who is calling them. +/// +/// Wikimedia's requires a client that identifies itself with a contact point; +/// the shared default UA reads as an anonymous bot and gets a savage limiter +/// (measured 2026-07-22: ~13 requests then HTTP 429 with `Retry-After: 51`, +/// which silently zeroed every album description on a cold artist page). A +/// compliant UA sustained 45+ back-to-back requests with no rejection. Last.fm +/// serves an anonymous UA today but asks API clients to identify themselves, +/// and the Popular ordering now depends on it, so it opts in too. +/// +/// Applied to those two only — every other provider (notably the Bing HTML +/// scrape) must keep seeing the default UA. +const CONTACT_USER_AGENT: &str = "SoulStream/0.1 (personal desktop music app; contact: https://github.com/KeelrLeek)"; /// Client-side pacing for Wikipedia stays at the measured compliant-UA /// budget (45 back-to-back requests accepted; a cold 14-album page needs @@ -305,6 +310,52 @@ type AppleCandidatesFetch = Arc>>>; /// TheAudioDB's 1280×720 fanart. const FANART_KEY: &str = "1a2fb0b9fd7fa6fcf68388f78a9cdca6"; +/// Shared Last.fm *project* API key, same class as [`FANART_KEY`] and +/// TheAudioDB's public `123` — app-wide, disposable, meant to be embedded. +/// Rotate at last.fm/api/accounts if it is ever abused. +/// Supplies the scrobble counts that order the Popular section: Deezer's own +/// `/top` is recency-weighted, so a week-old single outranks the songs the +/// artist is actually known for (Taylor Swift's Deezer top three are all from +/// the current release; Last.fm's are Blank Space, Cruel Summer, Shake It Off). +const LASTFM_KEY: &str = "69f3b3201381b56903eeee31477644a2"; +/// Last.fm publishes no per-key quota, only a ~5 req/s per-IP guideline. One +/// artist page spends exactly one request and the answer is disk cached for a +/// day, so this window is purely a guard against a burst (a library warm-up +/// sweep) tripping a limiter we cannot see. Deliberately not the unpaced +/// treatment Bing got — that is the one provider that broke this way. +const LASTFM_RATE_WINDOW: Duration = Duration::from_secs(1); +const LASTFM_RATE_BUDGET: usize = 4; +/// How deep to pull the scrobble table. +/// +/// Raised from 100 once this table also had to serve per-track play counts on +/// ALBUM pages, not just reorder the popular ten. 100 covered the Deezer pool +/// fine but is nowhere near an album's deep cuts — measured on Weezer, it hit +/// only 2 of Van Weezer's 10 tracks and 5 of the White Album's. +/// +/// 1000 is one request, the same request, and it covered **56 of 57** album +/// tracks measured across Johnny Cash, the Grateful Dead, Miles Davis, Bob +/// Dylan and Queen — catalogues where Last.fm holds 150k-390k track titles. +/// That works because the long tail is bootlegs and mislabelled scrobbles: a +/// real album track is, by definition, one of the popular ones. The cost is +/// body size, 77 KB -> ~775 KB per artist, and it is disk cached for a day. +const LASTFM_TRACK_LIMIT: usize = 1000; +/// How many Deezer tracks are fetched to reorder. Deezer's `/top` is ranked by +/// its own recency-weighted popularity, so the canonical songs sit deeper than +/// the ten we used to show: measured across 18 artists, Last.fm's canonical top +/// ten is only 43% present in Deezer's top 10, but 77% by its top 50. Past 50 +/// the curve flattens (79% at 100) and every extra track is one more row that +/// can never be displayed. +const POPULAR_POOL_LIMIT: usize = 50; +/// How many reordered tracks reach the frontend. Unchanged: the UI shows five, +/// expands to ten, and `openPreviews` snapshots ten. +const POPULAR_KEEP: usize = 10; +/// Release-group paging for the first-release-date pass. 100 is MusicBrainz's +/// maximum page; three pages covers a very prolific catalogue (Queen has ~100 +/// album groups alone) without ever walking a discography forever behind the +/// mandatory 1 req/s gate. +const MB_RELEASE_GROUP_PAGE: usize = 100; +const MB_RELEASE_GROUP_PAGES: usize = 3; + /// A resolved artist banner rarely changes and costs two external calls, so it /// is cached far longer than ordinary catalog pages. const BANNER_TTL: Duration = Duration::from_secs(24 * 60 * 60); @@ -403,6 +454,74 @@ pub struct CatalogTrack { /// remain playable. Always `false` for normal, released albums. #[serde(default)] pub unreleased: bool, + /// All-time distinct Last.fm listeners, when this track could be anchored + /// to one (see [`Self::listeners`] producers in `lastfm_listener_ranks`). + /// `None` means Last.fm has no count for it — a track too new or too + /// obscure to have been scrobbled, or an artist with no usable anchor at + /// all. The UI shows a dash rather than a zero: unknown is not none. + #[serde(default)] + pub listeners: Option, + /// Total Last.fm scrobbles, from the same table as [`Self::listeners`]. + /// This is the album tracklist's "Plays" column. `None` means Last.fm has + /// no entry for the track — shown as a dash, never as a zero. + #[serde(default)] + pub plays: Option, + /// The Deezer `/top` track id this row was re-attributed away from, when + /// [`apply_original_album_attribution`] moved it onto the song's original + /// album. `None` means the row is Deezer's own attribution, untouched. + /// + /// The UI still matches now-playing against this id, so a track started + /// from search, the suggest dropdown or an album page — all of which use + /// the id Deezer LICENSED — keeps its highlight on this row. + #[serde(default)] + pub origin_track_id: Option, +} + +/// What one MusicBrainz release GROUP says about a work: when it first came +/// out, and what kind of release it is. +/// +/// The two travel together because they only mean anything read together. A +/// compilation is dated EARLIER than the works it compiles, so a date carried +/// without its own secondary types actively *selects* hits packages — see +/// [`apply_original_album_attribution`], which measured 79/80 with the type +/// exclusion and 63/80 without it. +#[derive(Debug, Clone, Default, PartialEq)] +struct ReleaseFacts { + /// MusicBrainz `first-release-date` for the group. + first: String, + /// MusicBrainz `secondary-types`: `Compilation`, `Live`, `Soundtrack`, + /// `Remix`, `DJ-mix`, … Empty for a plain studio album. + secondary: Vec, + /// Several MusicBrainz groups collapsed onto this key carrying DIFFERENT + /// first-release dates, so the key cannot say which record a given Deezer + /// release is. Nothing is ever dated or excluded from an ambiguous key. + /// + /// Weezer is the measured case: MusicBrainz files seven self-titled + /// albums, all titled plain "Weezer", separated only by `disambiguation` + /// ("Blue Album" … "Gold Album"). Taking the earliest dated the 2026 Gold + /// Album 1994-05-10 and filed an unreleased record next to the debut. + ambiguous: bool, +} + +/// One track's Last.fm scrobble figures. Both arrive in the same +/// `artist.gettoptracks` body, so carrying plays alongside listeners costs +/// nothing beyond the field. +#[derive(Debug, Clone, Copy, Default, PartialEq)] +struct TrackStats { + /// Total scrobbles — the "Plays" column on an album. + plays: u64, + /// Distinct listeners — what the Popular list ranks by. + listeners: u64, +} + +/// What the MBID-anchored providers contribute to an artist page. Empty on +/// every failure path, which degrades the page to Deezer's own data. +#[derive(Debug, Default)] +struct ArtistSupplements { + /// Last.fm scrobble figures, keyed by [`track_title_identity`]. + listeners: HashMap, + /// MusicBrainz first-release facts, keyed by [`release_date_key`]. + first_released: HashMap, } #[derive(Debug, Clone, Default, serde::Serialize)] @@ -476,6 +595,18 @@ pub struct CatalogArtistOverview { pub struct CatalogAlbumPage { pub album: CatalogAlbum, pub tracks: Vec, + /// Total Last.fm scrobbles for the record, and the distinct listeners + /// behind them. `None` when the artist could not be anchored to Last.fm. + #[serde(default)] + pub album_plays: Option, + #[serde(default)] + pub album_listeners: Option, + /// The totals were summed from this tracklist because Last.fm has no album + /// entry of its own — routine for anniversary and deluxe titles. Surfaced + /// so the UI can label an approximation rather than pass it off as + /// Last.fm's own figure. + #[serde(default)] + pub totals_summed: bool, } /// Editorial context for one release — a prose description plus genre/label @@ -910,6 +1041,8 @@ pub struct MetadataClient { wiki_cooldown: Mutex>, /// Sliding-window pacer for api.discogs.com (see [`DISCOGS_RATE_BUDGET`]). discogs_gate: Mutex>, + /// Sliding-window pacer for ws.audioscrobbler.com (see [`LASTFM_RATE_BUDGET`]). + lastfm_gate: Mutex>, /// On-disk JSON cache for slow-moving providers (see [`DISK_CACHE_TTL`]). /// `None` (the default, and every test/example) keeps behavior purely /// in-memory. @@ -966,6 +1099,7 @@ impl MetadataClient { wiki_gate: Mutex::new(VecDeque::new()), wiki_cooldown: Mutex::new(None), discogs_gate: Mutex::new(VecDeque::new()), + lastfm_gate: Mutex::new(VecDeque::new()), disk_cache_dir: None, identity_store_dir: None, fused_portraits: std::sync::Mutex::new(HashMap::new()), @@ -1057,7 +1191,13 @@ impl MetadataClient { let dir = self.disk_cache_dir.as_ref()?; let qualifies = url.starts_with("https://musicbrainz.org/") || is_wikimedia_url(url) - || url.starts_with("https://api.discogs.com/"); + || url.starts_with("https://api.discogs.com/") + // Scrobble totals are cumulative all-time counts in the millions; + // a day of drift cannot reorder a top-ten. Being on disk is the + // whole point of using them -- an unpaid ordering must not cost a + // network round on every revisit, which is exactly what TheAudioDB + // suffers for NOT being here. + || url.starts_with("https://ws.audioscrobbler.com/"); if !qualifies { return None; } @@ -1218,6 +1358,7 @@ impl MetadataClient { } let deezer = url.starts_with("https://api.deezer.com/"); let wiki = is_wikimedia_url(url); + let lastfm = url.starts_with("https://ws.audioscrobbler.com/"); let attempts = if wiki { WIKI_429_ATTEMPTS } else { @@ -1234,16 +1375,20 @@ impl MetadataClient { net_trace(url, "wiki-cooldown-fail-fast"); return None; } + if lastfm { + self.lastfm_throttle().await; + } // Ok(Some(body)) = usable answer; Ok(None) = final failure; // Err(()) = a retryable Wikipedia 429 (cooldown recorded). let body: Result, ()> = async { - let request = if wiki { - // Wikimedia's UA policy: identify the app, with a - // contact. The generic default UA is treated as an - // anonymous bot and rate limited into silence. + let request = if wiki || lastfm { + // Identify the app, with a contact — see + // [`CONTACT_USER_AGENT`]. The generic default UA is + // treated as an anonymous bot and rate limited into + // silence. self.http .get(url) - .header(reqwest::header::USER_AGENT, WIKIPEDIA_USER_AGENT) + .header(reqwest::header::USER_AGENT, CONTACT_USER_AGENT) } else { self.http.get(url) }; @@ -3133,11 +3278,17 @@ impl MetadataClient { let profile_url = format!("https://api.deezer.com/artist/{artist_id}"); let albums_url = format!("https://api.deezer.com/artist/{artist_id}/albums?limit=50&index=0"); - let popular_url = format!("https://api.deezer.com/artist/{artist_id}/top?limit=10"); - let (profile, album_page, popular) = tokio::join!( + let popular_url = + format!("https://api.deezer.com/artist/{artist_id}/top?limit={POPULAR_POOL_LIMIT}"); + // The Last.fm chain needs only the artist id, so it rides in this join + // rather than extending the critical path behind it. Both of its hops + // are disk cached for a day; a cold page overlaps them with the Apple + // identity work below, which takes longer. + let (profile, album_page, popular, supplements) = tokio::join!( self.deezer(&profile_url), self.deezer(&albums_url), self.deezer(&popular_url), + self.artist_supplements(artist_id), ); stage_trace("overview.deezer_join", overview_started, ""); let mut artist = parse_artist(&profile?)?; @@ -3180,24 +3331,13 @@ impl MetadataClient { album.artist = artist.name.clone(); } } - let popular_tracks = popular - .as_ref() - .and_then(|body| body.get("data")) - .and_then(|data| data.as_array()) - .map(|tracks| { - tracks - .iter() - .filter(|track| track_belongs_to_artist(track, artist_id)) - .filter_map(parse_track) - .filter(|track| { - !track - .album_id - .is_some_and(|id| foreign_album_ids.contains(&id)) - && !apple_release_is_foreign(&apple, &track.album, None) - }) - .collect() - }) - .unwrap_or_default(); + apply_original_release_dates(&mut albums, &supplements.first_released); + // Sort here too, not just on the full page: with corrected dates the two + // stages would otherwise disagree about chronology and the discography + // would visibly reshuffle underneath the reader when the full page lands. + albums.sort_by(|a, b| b.release_date.cmp(&a.release_date)); + let pool = deezer_popular_pool(popular.as_ref(), artist_id, &foreign_album_ids, &apple); + let popular_tracks = apply_lastfm_order(pool, &supplements.listeners, POPULAR_KEEP); Some(CatalogArtistOverview { artist, popular_tracks, @@ -3315,13 +3455,18 @@ impl MetadataClient { } candidates }; - let popular_url = format!("https://api.deezer.com/artist/{artist_id}/top?limit=10"); + let popular_url = + format!("https://api.deezer.com/artist/{artist_id}/top?limit={POPULAR_POOL_LIMIT}"); let gather_started = Instant::now(); - let (curated, search_candidates, popular, apple_candidates) = tokio::join!( + // Rides the gather join for the same reason it does in the overview: + // it needs only the artist id, and both its hops are disk cached, so + // overlapping costs nothing the exhaustive album walk was not paying. + let (curated, search_candidates, popular, apple_candidates, supplements) = tokio::join!( curated_fut, search_fut, self.deezer(&popular_url), self.apple_artist_candidates(&artist.name), + self.artist_supplements(artist_id), ); stage_trace("page.gather_join", gather_started, ""); let (curated, evidence) = curated?; @@ -3470,7 +3615,45 @@ impl MetadataClient { }) .collect(); need.sort_by_key(|(low_priority, _)| *low_priority); - let need: Vec = need.into_iter().map(|(_, id)| id).take(60).collect(); + // The cap cuts from the TAIL of Deezer's newest-first curated order, + // which is where the OLDEST records sit — and on a big catalogue those + // are exactly the albums a popular track originally came out on. + // Measured on Bob Dylan (82 album+EP rows), positions 60+ are Blonde On + // Blonde, Highway 61 Revisited, Bringing It All Back Home, Another + // Side, The Times They Are A-Changin', Freewheelin' and the debut. Nine + // measured catalogues overflow (Sinatra 88, Cash 138, Miles 165, Dead + // 200), and the failure mode is silent: the rows just stay + // un-re-attributed. So top the selection back up from the oldest end. + // Zero extra requests for everyone who already fits — Queen is 59. + const DETAIL_CAP: usize = 60; + const DETAIL_CEILING: usize = 80; + let mut need: Vec = need + .into_iter() + .map(|(_, id)| id) + .take(DETAIL_CAP) + .collect(); + if need.len() == DETAIL_CAP { + let selected: HashSet = need.iter().copied().collect(); + let mut oldest: Vec<&CatalogAlbum> = albums + .iter() + .filter(|a| { + matches!(a.record_type.as_deref(), Some("album") | Some("ep")) + && !selected.contains(&a.id) + // A dateless row cannot be ordered by age, and it is + // already unconditionally in `need` above (clause 1) -- + // these are the search-only extras, the last rows this + // top-up is for. + && a.release_date.is_some() + }) + .collect(); + oldest.sort_by(|a, b| a.release_date.cmp(&b.release_date)); + need.extend( + oldest + .into_iter() + .take(DETAIL_CEILING - DETAIL_CAP) + .map(|a| a.id), + ); + } let details_started = Instant::now(); let detail_count = need.len(); let fetched = stream::iter(need.iter().copied()) @@ -3617,6 +3800,27 @@ impl MetadataClient { (!tracks.is_empty()).then_some((*id, tracks)) }) .collect(); + // A second, independent read of the same details: which songs does each + // album actually CONTAIN? Deliberately not reusing `tracklists` above — + // that map is ordered and consumed by `edition_track_score`, so + // changing its normalizer or its `Vec` shape would silently move + // edition links. This one is an unordered set under a normalizer that + // also folds `feat.` clauses, which the containment match needs and + // edition folding does not. + let containment: HashMap> = details + .iter() + .filter_map(|(id, detail)| { + let titles: HashSet = detail + .pointer("/tracks/data")? + .as_array()? + .iter() + .filter_map(|track| track.get("title").and_then(|title| title.as_str())) + .map(contained_track_identity) + .filter(|key| !key.is_empty()) + .collect(); + (!titles.is_empty()).then_some((*id, titles)) + }) + .collect(); let mut edition_links = Vec::new(); for (edition_index, edition) in albums.iter().enumerate() { if edition.record_type.as_deref() != Some("album") { @@ -3907,27 +4111,43 @@ impl MetadataClient { } // The profile's nb_album reflects only Deezer's curated regional list; - // show the real merged count and sort by original release chronology. + // show the real merged count and sort by original release chronology -- + // which is only actually chronology once the reissue dates Deezer + // reports have been replaced by the works' first-release dates. + let dates_started = Instant::now(); + let corrected = + apply_original_release_dates(&mut albums, &supplements.first_released); + // Then let the tracklist-verified edition links settle what titles + // cannot: a pressing takes the date of the work it is a pressing of. + let inherited = apply_edition_group_dates(&mut albums); + stage_trace( + "page.original_dates", + dates_started, + &format!("{corrected}/{} corrected, {inherited} inherited", albums.len()), + ); artist.album_count = Some(albums.len() as u32); albums.sort_by(|a, b| b.release_date.cmp(&a.release_date)); - let popular_tracks = popular - .as_ref() - .and_then(|body| body.get("data")) - .and_then(|data| data.as_array()) - .map(|tracks| { - tracks - .iter() - .filter(|track| track_belongs_to_artist(track, artist_id)) - .filter_map(parse_track) - .filter(|track| { - !track - .album_id - .is_some_and(|id| foreign_album_ids.contains(&id)) - && !apple_release_is_foreign(&apple, &track.album, None) - }) - .collect() - }) - .unwrap_or_default(); + let pool = deezer_popular_pool(popular.as_ref(), artist_id, &foreign_album_ids, &apple); + let mut popular_tracks = apply_lastfm_order(pool, &supplements.listeners, POPULAR_KEEP); + // Runs LAST, on the ten rows that survived: after the album pipeline + // has foreign-filtered, ownership-checked, classified and folded, and + // after `apply_original_release_dates` so the date tiebreak compares + // first-release dates rather than the reissue dates Deezer reports. + // Ordering is keyed on `track_title_identity`, not on ids, so moving a + // row's id here cannot disturb the Last.fm order above it. + let attribution_started = Instant::now(); + let reattributed = apply_original_album_attribution( + &mut popular_tracks, + &albums, + &containment, + &details, + &supplements.first_released, + ); + stage_trace( + "page.attribution", + attribution_started, + &format!("{reattributed}/{} re-attributed", popular_tracks.len()), + ); stage_trace( "page.total", page_started, @@ -4852,6 +5072,11 @@ impl MetadataClient { track_no: Some(tracks.len() as u32 + 1), disk_number: None, unreleased: false, + // An album tracklist is not a ranked popular list; the + // scrobble figures are attached by `apply_track_stats`. + listeners: None, + plays: None, + origin_track_id: None, }); } } @@ -4868,7 +5093,7 @@ impl MetadataClient { is_live: false, appears_on: false, }; - Some(CatalogAlbumPage { album, tracks }) + Some(CatalogAlbumPage { album, tracks, album_plays: None, album_listeners: None, totals_summed: false }) } /// The iTunes song lookup for a collection — the album's tracklist. Shared @@ -4945,6 +5170,11 @@ impl MetadataClient { .and_then(|v| v.as_u64()) .map(|n| n as u32), unreleased: false, + // An album tracklist is not a ranked popular list; the + // scrobble figures are attached by `apply_track_stats`. + listeners: None, + plays: None, + origin_track_id: None, }); } let album = CatalogAlbum { @@ -4960,7 +5190,7 @@ impl MetadataClient { is_live: false, appears_on: false, }; - Some(CatalogAlbumPage { album, tracks }) + Some(CatalogAlbumPage { album, tracks, album_plays: None, album_listeners: None, totals_summed: false }) } /// The album's track titles, normalized via `track_identity` for edition @@ -4977,6 +5207,10 @@ impl MetadataClient { /// Album details with the full tracklist (and preview URLs). pub async fn catalog_album(&self, album_id: u64) -> Option { + // Only the Deezer branch knows which artist this record belongs to, and + // the scrobble table is anchored on that id. A MusicBrainz/Apple + // supplement album simply carries no play counts. + let mut deezer_artist_id: Option = None; let mut page = if let Some(album) = self.mb_catalog.lock().await.get(&album_id).cloned() { self.musicbrainz_album(album_id, album).await? } else if let Some(album) = self.apple_catalog.lock().await.get(&album_id).cloned() { @@ -4985,6 +5219,7 @@ impl MetadataClient { let body = self .deezer(&format!("https://api.deezer.com/album/{album_id}")) .await?; + deezer_artist_id = body.pointer("/artist/id").and_then(|id| id.as_u64()); let album = parse_album(&body)?; // Deezer's `/album/{id}` embeds only the first ~25 tracks in // `/tracks/data` and gives no pagination cursor there, so a 50-track @@ -5031,7 +5266,7 @@ impl MetadataClient { }) .unwrap_or_default(); } - CatalogAlbumPage { album, tracks } + CatalogAlbumPage { album, tracks, album_plays: None, album_listeners: None, totals_summed: false } }; // A future-dated album may still have individual songs already out as // singles. Flag the rest unreleased so the UI can block streaming them @@ -5039,9 +5274,105 @@ impl MetadataClient { if is_future_or_unresolved_date(page.album.release_date.as_deref()) { self.mark_unreleased_tracks(&mut page).await; } + self.apply_track_stats(&mut page, deezer_artist_id).await; Some(page) } + /// Attach Last.fm scrobble figures to an album's tracklist and totals. + /// + /// Costs no extra network on the track side: `artist.gettoptracks` is the + /// very body the artist page already pulled, disk cached for a day, and at + /// [`LASTFM_TRACK_LIMIT`] it reaches an album's deep cuts — measured 56 of + /// 57 tracks across five 150k-390k-track catalogues. A track Last.fm has + /// never seen keeps `None` and renders as a dash: unknown is not zero. + async fn apply_track_stats(&self, page: &mut CatalogAlbumPage, deezer_artist_id: Option) { + let Some(artist_id) = deezer_artist_id else { + return; + }; + let Some(mbid) = self.deezer_linked_mbid(artist_id).await else { + return; + }; + let Some(ranks) = self.lastfm_listener_ranks(&mbid).await else { + return; + }; + let started = Instant::now(); + let mut matched = 0; + let mut summed_plays = 0_u64; + let mut peak_plays = 0_u64; + let mut peak_listeners = 0_u64; + for track in page.tracks.iter_mut() { + let key = track_title_identity(&track.title); + let Some(stats) = ranks.get(&key).copied() else { + continue; + }; + matched += 1; + track.plays = (stats.plays > 0).then_some(stats.plays); + track.listeners = (stats.listeners > 0).then_some(stats.listeners); + summed_plays += stats.plays; + peak_plays = peak_plays.max(stats.plays); + peak_listeners = peak_listeners.max(stats.listeners); + } + + // Last.fm's own album entry is the authoritative total when it exists + // AND is describing the same population as the tracklist above it. + // + // Those two things come apart on editions. An album's `playcount` counts + // scrobbles tagged to THAT release page, while a track's counts every + // scrobble of the song across every release it appears on. Measured on + // Weezer's Blue Album deluxe: the album page reports 89k plays while its + // own opening track reports 7.2M — a rail total two orders of magnitude + // below the first row beneath it. So a total that cannot even cover its + // biggest track is an edition-page artifact, not this record's figure; + // sum the tracklist instead and label it. + let provider = self + .lastfm_album_totals(&page.album.artist, &page.album.title) + .await + .filter(|(plays, _)| *plays >= peak_plays); + match provider { + Some((plays, listeners)) => { + page.album_plays = Some(plays); + page.album_listeners = Some(listeners); + } + None if matched > 0 => { + page.album_plays = Some(summed_plays); + page.album_listeners = Some(peak_listeners); + page.totals_summed = true; + } + None => {} + } + stage_trace( + "album.track_stats", + started, + &format!("{matched}/{} matched", page.tracks.len()), + ); + } + + /// Last.fm's scrobble and listener totals for one record, as + /// `(playcount, listeners)`. Disk cached like every other Last.fm call. + async fn lastfm_album_totals(&self, artist: &str, album: &str) -> Option<(u64, u64)> { + if artist.is_empty() || album.is_empty() { + return None; + } + let url = format!( + "https://ws.audioscrobbler.com/2.0/?method=album.getinfo&artist={}&album={}\ + &autocorrect=1&api_key={LASTFM_KEY}&format=json", + urlencode(artist), + urlencode(album) + ); + let body = self.lastfm_json(&url).await?; + // Both arrive as JSON strings, and a record nobody has scrobbled is a + // real zero rather than a missing field. + let plays = body + .pointer("/album/playcount") + .and_then(|v| v.as_str()) + .and_then(|v| v.parse().ok())?; + let listeners = body + .pointer("/album/listeners") + .and_then(|v| v.as_str()) + .and_then(|v| v.parse().ok())?; + Some((plays, listeners)) + } + /// Flag each track on an upcoming album as unreleased unless it is already /// out — either it carries its own Deezer preview, or a released single by /// the same artist with the same title exists. Checks run concurrently. @@ -5182,6 +5513,225 @@ impl MetadataClient { } } + /// GET a Last.fm endpoint, paced, disk cached, and with its in-band errors + /// treated as failures. + /// + /// Last.fm answers `HTTP 200` with `{"error":6,...}` in the body for a + /// lookup that found nothing, exactly like Deezer signals quota inside a + /// 200. Returning that body as success would hand the caller an object with + /// no `toptracks` and cache it for a day. + async fn lastfm_json(&self, url: &str) -> Option { + let body = self.cached_json(url, false).await?; + if let Some(code) = body.get("error") { + net_trace(url, &format!("lastfm-error {code}")); + return None; + } + Some(body) + } + + /// Everything the page needs from the MBID-anchored providers: the MBID is + /// resolved once, then both providers are fetched together. + /// + /// Deliberately takes only the artist id so it can ride in the same + /// `tokio::join!` as the Deezer fetches instead of extending the critical + /// path behind them. Every hop it makes is disk cached for a day, so a + /// revisit pays nothing, and a cold page overlaps it with work that already + /// takes longer. + /// + /// No MusicBrainz entry for this Deezer artist means no anchor for either + /// provider, and the caller falls back to Deezer's own ordering and dates — + /// which is exactly what the page did before any of this existed. + async fn artist_supplements(&self, artist_id: u64) -> ArtistSupplements { + let Some(mbid) = self.deezer_linked_mbid(artist_id).await else { + return ArtistSupplements::default(); + }; + let (listeners, first_released) = tokio::join!( + self.lastfm_listener_ranks(&mbid), + self.musicbrainz_first_release_dates(&mbid), + ); + ArtistSupplements { + listeners: listeners.unwrap_or_default(), + first_released, + } + } + + /// First-release date of every release GROUP this artist has, keyed by + /// [`album_title_identity`]. + /// + /// The browse endpoint is used rather than search: it takes the MBID + /// directly (no query parsing, no relevance ranking) and returns the + /// `first-release-date` this whole feature exists for. Compilations, live + /// records and soundtracks are all fetched too — they are real releases + /// whose dates are just as wrong on Deezer, and excluding them here would + /// leave a compilation dated by its reissue. + async fn musicbrainz_first_release_dates(&self, mbid: &str) -> HashMap { + let started = Instant::now(); + let mut dates: HashMap = HashMap::new(); + for page in 0..MB_RELEASE_GROUP_PAGES { + let offset = page * MB_RELEASE_GROUP_PAGE; + // Byte-identical to the URL `musicbrainz_release_titles` builds, so + // the two passes share one cached body instead of each paying the + // 1 req/s gate for the same page. + let url = format!( + "https://musicbrainz.org/ws/2/release-group?artist={mbid}&limit=100&offset={offset}&fmt=json" + ); + let Some(body) = self.musicbrainz_json(&url).await else { + break; + }; + let Some(groups) = body.get("release-groups").and_then(|v| v.as_array()) else { + break; + }; + let returned = groups.len(); + let total = body + .get("release-group-count") + .and_then(|c| c.as_u64()) + .unwrap_or(returned as u64) as usize; + for group in groups { + merge_release_group_facts(&mut dates, group); + } + if offset + returned >= total || returned == 0 { + break; + } + } + stage_trace( + "musicbrainz.first_release_dates", + started, + &format!("{} groups", dates.len()), + ); + dates + } + + /// All-time Last.fm listener counts for one artist's tracks, keyed by + /// [`track_title_identity`]. + /// + /// Identity is ANCHORED, never name-matched, in two steps — the project + /// rule is Deezer id → MBID → declared links, else show nothing: + /// + /// 1. `mbid=` — Last.fm's own MusicBrainz index. We send an identifier and + /// Last.fm resolves it, so no name is involved and there is nothing to + /// guard. This alone covers 36 of 52 sampled artists, including five of + /// the six smallest in the local library. + /// 2. If Last.fm's index has no entry (its MBID coverage is partial — it + /// returns error 6 for King Gizzard even given the correct MBID), fall + /// back to the last.fm URL **MusicBrainz itself declares** for this + /// artist, and require the response's own `artist.url` to echo that same + /// declared identifier back. That is a declared-link anchor, not a name + /// guess: the identifier comes from MusicBrainz, and the guard proves + /// Last.fm agrees it is the same page. + /// + /// Nothing else is attempted. Querying Last.fm with the artist's *name* was + /// measured and rejected: it reaches only 2 more artists out of 52 while + /// re-opening the exact namesake hole the identity rule exists to close. + /// + /// Step 2 costs no extra MusicBrainz traffic — it reads the same + /// `artist_detail_url` body [`Self::artist_socials`] already fetches, which + /// is memory and disk cached. + async fn lastfm_listener_ranks(&self, mbid: &str) -> Option> { + let started = Instant::now(); + let by_mbid = format!( + "https://ws.audioscrobbler.com/2.0/?method=artist.gettoptracks\ + &mbid={mbid}&limit={LASTFM_TRACK_LIMIT}&autocorrect=1&api_key={LASTFM_KEY}&format=json" + ); + let mut body = self.lastfm_json(&by_mbid).await; + + if body.is_none() { + let declared = self.musicbrainz_declared_lastfm(mbid).await?; + // The identifier is what MusicBrainz declared, decoded; querying with + // it is not a name guess. Last.fm matches it case-insensitively, so + // the same normalized form serves as both the query and the value the + // guard below compares against. + let identifier = lastfm_artist_identifier(&declared)?; + let by_name = format!( + "https://ws.audioscrobbler.com/2.0/?method=artist.gettoptracks\ + &artist={}&limit={LASTFM_TRACK_LIMIT}&autocorrect=1&api_key={LASTFM_KEY}&format=json", + urlencode(&identifier) + ); + let candidate = self.lastfm_json(&by_name).await?; + // THE GUARD. `autocorrect=1` makes Last.fm answer with the page it + // actually resolved to rather than echoing the query -- a misspelled + // "Radiohed" comes back as .../music/Radiohead. So an echoed + // identifier that matches what MusicBrainz declared is Last.fm + // agreeing on identity. With autocorrect=0 it parrots the query and + // this check would be worthless. + let echoed = candidate + .pointer("/toptracks/track/0/artist/url") + .and_then(|url| url.as_str()) + .and_then(lastfm_artist_identifier); + if echoed.as_deref() != Some(identifier.as_str()) { + net_trace(&by_name, "lastfm-identity-rejected"); + return None; + } + body = Some(candidate); + } + + let tracks = body? + .pointer("/toptracks/track")? + .as_array()? + .iter() + .filter_map(|track| { + let title = track.get("name")?.as_str()?; + // Counts arrive as JSON strings, not numbers. + let listeners = track.get("listeners")?.as_str()?.parse().ok()?; + // Plays ride the same body. Absent on a sparse entry, which is + // a zero here and a dash in the UI, never a fabricated count. + let plays = track + .get("playcount") + .and_then(|value| value.as_str()) + .and_then(|value| value.parse().ok()) + .unwrap_or(0); + let key = track_title_identity(title); + (!key.is_empty()).then_some((key, TrackStats { plays, listeners })) + }) + // Last.fm's list is already listeners-descending, so an earlier + // entry wins a key collision (e.g. a remaster listed twice). + .fold(HashMap::new(), |mut acc, (key, stats)| { + acc.entry(key).or_insert(stats); + acc + }); + stage_trace("lastfm.ranks", started, &format!("{} tracks", tracks.len())); + (!tracks.is_empty()).then_some(tracks) + } + + /// The `last.fm` URL MusicBrainz declares for this artist, if any. Shares + /// the cached `inc=url-rels` body the socials row already reads. + async fn musicbrainz_declared_lastfm(&self, mbid: &str) -> Option { + let body = self.musicbrainz_json(&artist_detail_url(mbid)).await?; + body.get("relations")? + .as_array()? + .iter() + .find(|relation| { + relation.get("type").and_then(|kind| kind.as_str()) == Some("last.fm") + }) + .and_then(|relation| relation.pointer("/url/resource")) + .and_then(|url| url.as_str()) + .map(String::from) + } + + /// Hold a Last.fm request until its sliding window has room. + async fn lastfm_throttle(&self) { + loop { + let wait = { + let mut window = self.lastfm_gate.lock().await; + let now = Instant::now(); + while window + .front() + .is_some_and(|at| now.duration_since(*at) >= LASTFM_RATE_WINDOW) + { + window.pop_front(); + } + if window.len() < LASTFM_RATE_BUDGET { + window.push_back(now); + return; + } + match window.front() { + Some(oldest) => LASTFM_RATE_WINDOW - now.duration_since(*oldest), + None => continue, + } + }; + tokio::time::sleep(wait).await; + } + } + /// Hold a Discogs request until its sliding window has room. async fn discogs_throttle(&self) { loop { @@ -6423,6 +6973,11 @@ fn parse_track(v: &serde_json::Value) -> Option { .and_then(|t| t.as_u64()) .map(|t| t as u32), unreleased: false, + // Filled in by `apply_lastfm_order`; Deezer carries no scrobble data. + listeners: None, + plays: None, + // Set only when `apply_original_album_attribution` moves this row. + origin_track_id: None, }) } @@ -6462,6 +7017,762 @@ fn release_title_identity(title: &str) -> String { normalize(strip_release_format_tail(title).0) } +/// Does this bracketed/dashed segment name the SAME recording, just a different +/// pressing of it? +/// +/// Deliberately absent: live, remix, reprise, interlude, acoustic, demo. Those +/// name a genuinely different recording, and folding them into the parent would +/// delete a real track from the list. Measured over 18 artists x Deezer's top +/// 50: this curated set carries 74% of Last.fm's canonical top ten across to a +/// Deezer track while merging 16 entries, where stripping EVERY parenthetical +/// reached 79% but merged 61 — four times the collateral for five points. +fn is_track_version_marker(segment: &str) -> bool { + let segment = segment.trim().trim_matches(|c: char| !c.is_alphanumeric()); + if segment.is_empty() { + return false; + } + // "2011 Remaster", "Remastered 2009", "Taylor's Version", "Album Version". + segment.contains("remaster") + || segment.contains("version") + || segment.contains("radio edit") + || segment.contains("single edit") + || segment.contains("original mix") + || segment.contains("extended mix") + || segment.contains("club mix") + || segment.contains("bonus track") + // A bracketed credit — "(feat. Reckz'Capo)" — never reaches the loop + // above, which only sees a SPACE-prefixed clause. + || segment.starts_with("feat.") + || segment.starts_with("feat ") + || segment.starts_with("featuring") + || segment.starts_with("ft.") + || segment.starts_with("ft ") + || matches!(segment, "edit" | "deluxe" | "mono" | "stereo" | "explicit" | "clean") +} + +/// The cross-provider key for one track title. +/// +/// Last.fm reports the name people scrobbled ("Shake It Off") while Deezer +/// carries the pressing it sells ("Shake It Off (Taylor's Version)"), so a raw +/// string comparison finds neither. Only ever applied WITHIN a single already +/// anchored artist — it decides which of that artist's own tracks two providers +/// are both describing, never which artist we are looking at, so it is not the +/// kind of name match [`audiodb_artist_matches_identity`] exists to forbid. +/// Drop every bracketed segment and trailing dash-clause that `is_marker` +/// accepts, then normalize. +/// +/// Lowercased ONCE up front, so every index below refers to a single string. +/// Comparing a lowercased copy while slicing the original desyncs the two the +/// moment a character's lowercase form has a different byte length (İ -> i̇), +/// which is a panic on a non-ASCII title rather than a wrong answer. +fn title_identity_stripping(title: &str, is_marker: fn(&str) -> bool) -> String { + let mut work = title.trim().to_lowercase(); + + loop { + let found = work.char_indices().find_map(|(open, ch)| { + let close_ch = match ch { + '(' => ')', + '[' => ']', + _ => return None, + }; + let close = open + work[open..].find(close_ch)?; + is_marker(&work[open + 1..close]).then_some((open, close)) + }); + let Some((open, close)) = found else { break }; + work.replace_range(open..=close, " "); + } + + if let Some(at) = [" - ", " – ", " — "] + .iter() + .filter_map(|dash| work.rfind(dash)) + .max() + { + if is_marker(&work[at..]) { + work.truncate(at); + } + } + + normalize(&work) +} + +fn track_title_identity(title: &str) -> String { + let mut work = title.trim().to_lowercase(); + // A featured-artist clause is Deezer packaging, never part of the scrobble. + for marker in [" feat.", " feat ", " ft.", " ft ", " featuring "] { + if let Some(at) = work.find(marker) { + work.truncate(at); + } + } + title_identity_stripping(&work, is_track_version_marker) +} + +/// Does this segment name a re-PRESSING of an album rather than a different +/// work? Mirrors the frontend's `EDITION_MARKER` vocabulary +/// (`CatalogArtist.svelte`), so Rust and the UI fold the same titles. +/// +/// `"'s Version"` is deliberately exempt, for the same reason the +/// frontend exempts it: a re-recorded catalogue (Taylor's Version) is its own +/// work with its own first-release date, not a reissue of the original. +fn is_album_edition_marker(segment: &str) -> bool { + let segment = segment.trim().trim_matches(|c: char| !c.is_alphanumeric()); + if segment.is_empty() { + return false; + } + if segment.ends_with("'s version") || segment.ends_with("\u{2019}s version") { + return false; + } + const MARKERS: [&str; 17] = [ + "super deluxe", + "deluxe", + "expanded", + "remaster", + "anniversary", + "legacy edition", + "special edition", + "collector", + "bonus track", + "complete edition", + "definitive edition", + "reissue", + "redux", + "mono edition", + "stereo edition", + "half-speed", + "half speed", + ]; + MARKERS.iter().any(|marker| segment.contains(marker)) + // "'s Edition", "Tour Edition", "2011 Edition" -- the frontend's + // `\w+(?:['’]s)?\s+edition` arm, minus the exemption handled above. + || segment.ends_with(" edition") + || segment == "edition" +} + +/// The cross-provider key for an ALBUM title, used only to line a Deezer +/// release up with the MusicBrainz release group it is a pressing of. +/// +/// Deezer sells "Hot Space", "A Night at the Opera (2011 Remaster)" and +/// "American Idiot (Deluxe)"; MusicBrainz calls all three by the work's name. +/// Like [`track_title_identity`], this only ever runs INSIDE an artist already +/// anchored by Deezer id -> MBID, so it decides which release two providers are +/// both describing, never which artist we are looking at. +fn album_title_identity(title: &str) -> String { + let base = strip_release_format_tail(title).0.to_lowercase(); + let base = keep_edition_disambiguators(&base); + title_identity_stripping(&fold_volume_markers(&base), is_album_edition_marker) +} + +/// Keep the identifying half of a parenthetical that mixes a pressing with a +/// disambiguator. +/// +/// Deezer sells Weezer's colour records as "Weezer (Deluxe Edition / Red +/// Album)". That single parenthetical carries BOTH the pressing ("Deluxe +/// Edition") and the only thing saying WHICH self-titled album this is ("Red +/// Album"), so dropping it whole reduces the 2008 Red Album to `weezer` — the +/// same key as the 1994 debut. MusicBrainz titles all seven self-titled Weezer +/// groups plain "Weezer", so that key resolves to the EARLIEST of them, and the +/// Red Album deluxe was dated 1994-05-10 and filed fourteen years early. +/// +/// Splitting on the separator and keeping only the parts that are NOT edition +/// markers leaves "weezer (red album)", which keeps its colour and so its own +/// identity. A segment that is entirely edition markers is left exactly as it +/// was, for [`title_identity_stripping`] to remove as before. +fn keep_edition_disambiguators(lowercased: &str) -> String { + let mut work = lowercased.to_string(); + let mut from = 0; + while from < work.len() { + let found = work[from..].char_indices().find_map(|(offset, ch)| { + let closer = match ch { + '(' => ')', + '[' => ']', + _ => return None, + }; + let open = from + offset; + let close = open + work[open..].find(closer)?; + Some((open, close)) + }); + let Some((open, close)) = found else { break }; + let parts: Vec<&str> = work[open + 1..close].split('/').map(str::trim).collect(); + let kept: Vec<&str> = parts + .iter() + .copied() + .filter(|part| !part.is_empty() && !is_album_edition_marker(part)) + .collect(); + // Only rewrite a segment that genuinely MIXES the two. A pure edition + // clause still belongs to `title_identity_stripping`, and a segment + // with no edition part at all is already kept verbatim. + if parts.len() > 1 && !kept.is_empty() && kept.len() < parts.len() { + let replacement = format!("({})", kept.join(" ")); + work.replace_range(open..=close, &replacement); + from = open + replacement.len(); + } else { + from = close + 1; + } + } + work +} + +/// Drop the word in front of a volume number, so "Volume 2", "Vol. 2" and a +/// bare "2" all reduce alike. +/// +/// Deezer and MusicBrainz number a series differently — measured on Queen, +/// Deezer sells "Deep Cuts 2 (1977-1982)" while MusicBrainz files "Deep Cuts, +/// Volume 2 (1977–1982)". Only the word differs, so removing it leaves both +/// sides at `deepcuts219771982` once [`normalize`] discards punctuation. +/// +/// Only applied when a digit actually follows, so a record named "Volume" or +/// "The Volume of Noise" keeps its word. +fn fold_volume_markers(lowercased: &str) -> String { + const MARKERS: [&str; 3] = ["volume", "vol.", "vol"]; + let mut out = String::with_capacity(lowercased.len()); + let mut rest = lowercased; + 'outer: while !rest.is_empty() { + for marker in MARKERS { + let Some(tail) = rest.strip_prefix(marker) else { + continue; + }; + // A marker only counts on a word boundary, and only when the next + // non-space character is the number it introduces. + let starts_word = out.is_empty() || !out.ends_with(|c: char| c.is_alphanumeric()); + if starts_word && tail.trim_start().starts_with(|c: char| c.is_ascii_digit()) { + rest = tail.trim_start(); + continue 'outer; + } + } + let ch = rest.chars().next().unwrap_or_default(); + out.push(ch); + rest = &rest[ch.len_utf8()..]; + } + out +} + +/// Decode a `%XX`/`+` encoded URL path segment. MusicBrainz stores a declared +/// Last.fm link with spaces as `%20` while Last.fm echoes the same artist back +/// with `+`, so the guard has to compare DECODED identifiers — a byte compare +/// rejects 2 of every 18 real matches (measured: Inland Knights, Flight +/// Facilities). +fn percent_decode_plus(value: &str) -> String { + let bytes = value.as_bytes(); + let mut out: Vec = Vec::with_capacity(bytes.len()); + let mut i = 0; + while i < bytes.len() { + match bytes[i] { + b'+' => { + out.push(b' '); + i += 1; + } + b'%' if i + 2 < bytes.len() => { + match u8::from_str_radix(&value[i + 1..i + 3], 16) { + Ok(byte) => { + out.push(byte); + i += 3; + } + Err(_) => { + out.push(b'%'); + i += 1; + } + } + } + other => { + out.push(other); + i += 1; + } + } + } + String::from_utf8_lossy(&out).into_owned() +} + +/// The artist identifier inside a `last.fm/music/` URL, decoded and +/// lowercased so two encodings of one artist compare equal. +fn lastfm_artist_identifier(url: &str) -> Option { + let rest = url.split("/music/").nth(1)?; + let segment = rest.split('/').next()?; + if segment.is_empty() { + return None; + } + Some(percent_decode_plus(segment).to_lowercase()) +} + +/// Lookup key pairing a release's TYPE with its [`album_title_identity`]. +/// +/// The type belongs in the key because a title alone is not unique within one +/// artist: Queen has a "The Miracle" album (1989) and a "The Miracle" single +/// (1984), and keying on title alone dated the album to the single. +fn release_date_key(primary_type: Option<&str>, title: &str) -> String { + let identity = album_title_identity(title); + if identity.is_empty() { + return String::new(); + } + let kind = primary_type.unwrap_or("album").trim().to_lowercase(); + format!("{kind}\u{1f}{identity}") +} + +/// Fold one MusicBrainz release group into the first-release map. +/// +/// Two groups can share one key when an artist reissued a work under the same +/// title and type; the earliest is the original. Different TYPES are already +/// separated by [`release_date_key`] — Queen has both a "The Miracle" album +/// (1989) and a "The Miracle" single (1984), and folding those together dated +/// the album five years before it existed. +/// +/// The secondary types must come from WHICHEVER GROUP SUPPLIED THE SURVIVING +/// DATE, never unioned across the key. Measured: MusicBrainz has three groups +/// keyed `album/fleetwoodmac` — the real 1975 album with no secondary types, a +/// 1979 Compilation, and a 2006 Compilation+Live. Unioning tags the 1975 album +/// Compilation+Live; [`apply_original_album_attribution`] then excludes it and +/// "Rhiannon" lands on `Greatest Hits`, which is the exact failure the type +/// exclusion exists to prevent. +fn merge_release_group_facts(out: &mut HashMap, group: &serde_json::Value) { + let Some(title) = group.get("title").and_then(|v| v.as_str()) else { + return; + }; + let Some(first) = group + .get("first-release-date") + .and_then(|v| v.as_str()) + .filter(|value| !value.is_empty()) + else { + return; + }; + let primary_type = group.get("primary-type").and_then(|v| v.as_str()); + let candidate = ReleaseFacts { + first: first.to_string(), + secondary: group + .get("secondary-types") + .and_then(|value| value.as_array()) + .map(|types| { + types + .iter() + .filter_map(|value| value.as_str().map(String::from)) + .collect() + }) + .unwrap_or_default(), + ambiguous: false, + }; + // Index the DISAMBIGUATED name first. MusicBrainz titles all seven + // self-titled Weezer albums plain "Weezer" and separates them only by + // `disambiguation` — "Blue Album", "Red Album", … "Gold Album" — which is + // precisely what Deezer prints in its parenthetical ("Weezer (Red + // Album)"). Keying on that turns a hopeless seven-way collision into seven + // exact matches, and it is the only reason the 2026 Gold Album can be told + // apart from the 1994 debut at all. + if let Some(disambiguation) = group + .get("disambiguation") + .and_then(|value| value.as_str()) + .map(str::trim) + .filter(|value| !value.is_empty()) + { + merge_one_release_key( + out, + release_date_key(primary_type, &format!("{title} ({disambiguation})")), + candidate.clone(), + ); + } + merge_one_release_key(out, release_date_key(primary_type, title), candidate); +} + +/// Fold one candidate into one key, tracking whether the key stays trustworthy. +fn merge_one_release_key( + out: &mut HashMap, + key: String, + candidate: ReleaseFacts, +) { + if key.is_empty() { + return; + } + let Some(held) = out.get_mut(&key) else { + out.insert(key, candidate); + return; + }; + // Two groups under one key with two different dates: the key can no longer + // identify a record. A reissue duplicate (same date) is still fine, and so + // is the disambiguated key, which stays one-to-one. + let ambiguous = held.ambiguous || held.first != candidate.first; + // Strictly earlier wins outright. On an equal date the plainer group wins: + // when a studio album and a compilation of it share both title and date, + // the album is the one the songs actually came out on. + if candidate.first < held.first + || (candidate.first == held.first && candidate.secondary.len() < held.secondary.len()) + { + *held = candidate; + } + held.ambiguous = ambiguous; +} + +/// Replace Deezer's edition dates with the work's true first release. +/// +/// Deezer dates the PRESSING it licensed, not the record. It reports "A Night +/// at the Opera" as 2005-11-22, "The Game" as 1994-10-17 and "Hot Space" as +/// 1994-10-21 — CD reissue dates for albums from 1975, 1980 and 1982. Sorting a +/// discography by those numbers puts a band's catalogue in the order its label +/// happened to re-press it. +/// +/// MusicBrainz models this correctly: a release GROUP is the work, and its +/// `first-release-date` is the original. Matching is by +/// [`album_title_identity`], so an edition folds onto the work it is a pressing +/// of; an unmatched album simply keeps whatever Deezer said. +fn apply_original_release_dates( + albums: &mut [CatalogAlbum], + first_released: &HashMap, +) -> usize { + let mut corrected = 0; + for album in albums.iter_mut() { + // Deezer's record_type and MusicBrainz's primary-type agree on the + // vocabulary (album / ep / single), and matching on it is what keeps a + // same-named single from dating its album. + let key = release_date_key( + Some(album.record_type.as_deref().unwrap_or("album")), + &album.title, + ); + if key.is_empty() { + continue; + } + // An ambiguous key is several different records wearing one name. It + // dates nothing — the release keeps whatever its own provider said, + // which is the same "anchor, else leave it alone" rule the rest of the + // page follows. + let Some(original) = first_released.get(&key).filter(|facts| !facts.ambiguous) else { + continue; + }; + if album.release_date.as_deref() == Some(original.first.as_str()) { + continue; + } + album.release_date = Some(original.first.clone()); + corrected += 1; + } + corrected +} + +/// Give every edition the first-release date of the work it is a pressing of. +/// +/// `edition_group` is built from TRACKLIST overlap, so it knows something no +/// title comparison can: which record a pressing actually is. That matters most +/// exactly where titles fail — Deezer sells "Weezer (Deluxe Edition / Red +/// Album)" and "Weezer (Deluxe Edition / Blue Album)" while MusicBrainz titles +/// all seven self-titled Weezer groups plain "Weezer", so a title join cannot +/// tell 1994 from 2008 and both deluxes landed on the debut's date. The edition +/// link had already resolved both correctly from their tracklists. +/// +/// Runs after [`apply_original_release_dates`] so an edition inherits the +/// original's CORRECTED date, not Deezer's reissue date for it. A pressing of a +/// work did, by definition, first come out when the work did — which is the +/// same premise the whole date pass rests on. +fn apply_edition_group_dates(albums: &mut [CatalogAlbum]) -> usize { + // An original carries `edition_group == Some(own id)`, so it is its own + // source here and is never rewritten. + let originals: HashMap = albums + .iter() + .filter(|album| album.edition_group == Some(album.id)) + .filter_map(|album| Some((album.id, album.release_date.clone()?))) + .collect(); + let mut corrected = 0; + for album in albums.iter_mut() { + let Some(group) = album.edition_group else { + continue; + }; + if group == album.id { + continue; + } + let Some(original) = originals.get(&group) else { + continue; + }; + if album.release_date.as_deref() == Some(original.as_str()) { + continue; + } + album.release_date = Some(original.clone()); + corrected += 1; + } + corrected +} + +/// The playable, id-verified popular pool for one artist, in Deezer's own +/// order. Both the overview and the full page build it identically, and both +/// then hand it to [`apply_lastfm_order`], so it lives here rather than as two +/// copies kept in sync by hand. +fn deezer_popular_pool( + popular: Option<&serde_json::Value>, + artist_id: u64, + foreign_album_ids: &HashSet, + apple: &AppleArtistCatalog, +) -> Vec { + popular + .and_then(|body| body.get("data")) + .and_then(|data| data.as_array()) + .map(|tracks| { + tracks + .iter() + .filter(|track| track_belongs_to_artist(track, artist_id)) + .filter_map(parse_track) + .filter(|track| { + !track + .album_id + .is_some_and(|id| foreign_album_ids.contains(&id)) + && !apple_release_is_foreign(apple, &track.album, None) + }) + .collect() + }) + .unwrap_or_default() +} + +/// Reorder Deezer's top tracks by Last.fm listener counts. +/// +/// **Last.fm supplies ORDER; Deezer supplies MEMBERSHIP.** Nothing is ever +/// promoted into this list from Last.fm — only the tracks Deezer already +/// verified belong to this artist id (`track_belongs_to_artist`) are ranked. +/// That is what makes a merged Last.fm page harmless: Last.fm files several +/// bands under `/music/Scarlet`, and its top track there ("Independent Love +/// Song", 35,745 listeners) belongs to a different one — but it is not in +/// Deezer's list for this artist, so it simply never appears. +/// +/// Tracks Last.fm has no count for keep their Deezer order BELOW the ranked +/// ones rather than being dropped; a brand-new single nobody has scrobbled yet +/// is still a real track. Same-recording pressings are collapsed to the entry +/// Deezer ranks highest, because they would otherwise sort adjacently showing +/// an identical count twice. +fn apply_lastfm_order( + tracks: Vec, + ranks: &HashMap, + keep: usize, +) -> Vec { + let mut seen: HashSet = HashSet::new(); + let mut ordered: Vec<(usize, CatalogTrack)> = Vec::with_capacity(tracks.len()); + for track in tracks { + let key = track_title_identity(&track.title); + if !key.is_empty() && !seen.insert(key.clone()) { + continue; + } + let mut track = track; + let stats = ranks.get(&key).copied(); + track.listeners = stats.map(|s| s.listeners); + track.plays = stats.map(|s| s.plays).filter(|plays| *plays > 0); + ordered.push((ordered.len(), track)); + } + // Stable by construction: ranked tracks descend by listeners, and anything + // Last.fm has never seen keeps its Deezer position underneath them. + ordered.sort_by(|(left_at, left), (right_at, right)| { + match (left.listeners, right.listeners) { + (Some(a), Some(b)) => b.cmp(&a), + (Some(_), None) => std::cmp::Ordering::Less, + (None, Some(_)) => std::cmp::Ordering::Greater, + (None, None) => std::cmp::Ordering::Equal, + } + .then_with(|| left_at.cmp(right_at)) + }); + ordered.truncate(keep); + ordered.into_iter().map(|(_, track)| track).collect() +} + +/// Does MusicBrainz say this Deezer release is a compilation, a live record or +/// a soundtrack rather than a studio album? +/// +/// Exclusion requires POSITIVE evidence. A key MusicBrainz has never heard of +/// is not excluded, because "MusicBrainz is silent" and "MusicBrainz says +/// compilation" are different facts — Deezer sells plenty of pressings whose +/// exact title MusicBrainz has no group for, and refusing those would throw +/// away the real album (the measured Weezer case). +/// +/// Only the three MEASURED types are listed. `Remix` and `DJ-mix` are the same +/// class and would probably be fine, but they were never measured, and the +/// Deezer side already refuses the `compile` record type independently. +fn musicbrainz_excludes_original(mb: &HashMap, album: &CatalogAlbum) -> bool { + let key = release_date_key( + Some(album.record_type.as_deref().unwrap_or("album")), + &album.title, + ); + if key.is_empty() { + return false; + } + mb.get(&key).is_some_and(|facts| { + // An ambiguous key describes several records at once, so it is not + // evidence about this one. + !facts.ambiguous + && facts.secondary.iter().any(|kind| { + matches!( + kind.trim().to_lowercase().as_str(), + "compilation" | "live" | "soundtrack" + ) + }) + }) +} + +/// Move each popular row onto the album the song is actually FROM. +/// +/// Deezer attributes a popular track to whichever release it LICENSED, not the +/// record the song came out on: "Bohemian Rhapsody" comes back on the 2018 +/// movie soundtrack, carrying that poster as its cover art, and 11 of Queen's +/// top 12 are wrong the same way. The artist's own album details are already +/// fetched for the discography and every one of them carries a full tracklist, +/// so the song's original home is findable at ZERO extra network cost. +/// +/// Containment alone is nowhere near enough, and that is the whole difficulty: +/// "Bohemian Rhapsody" is contained by FIFTEEN of Queen's albums — seven live +/// records, four hits packages, the soundtrack, two box sets, plus the real +/// album and its deluxe. The ranking is the feature, not the lookup. +/// +/// Measured across 8 artists / 80 tracks: **79/80 (98.8%)** against 58/80 for +/// Deezer's own attribution. The MusicBrainz secondary-type exclusion is +/// load-bearing — without it the score falls to 63/80 and 16 picks flip to +/// WRONG, because compilations are dated EARLIER than the works they compile, +/// so "earliest release wins" actively *selects* hits packages. That is how an +/// earlier attempt landed "Boulevard of Broken Dreams" on the American Idiot +/// Broadway cast recording. **The type exclusion is the feature; the date is +/// only the tiebreak.** +/// +/// Returns the number of rows moved. Every failure path degrades to exactly +/// today's Deezer attribution rather than to a guess. +fn apply_original_album_attribution( + tracks: &mut [CatalogTrack], + albums: &[CatalogAlbum], + containment: &HashMap>, + details: &HashMap, + mb: &HashMap, +) -> usize { + // MusicBrainz evidence is the anchor, and without it the rule collapses to + // "earliest release wins", which is measurably worse than doing nothing + // (12 rows broken). An artist with no linked MBID, a failed browse, or a + // quota-emptied details pass all land here and leave the list untouched. + if mb.is_empty() || containment.is_empty() { + return 0; + } + let mut moved = 0; + for track in tracks.iter_mut() { + let key = contained_track_identity(&track.title); + if key.is_empty() { + continue; + } + let winner = albums + .iter() + .enumerate() + .filter(|(_, album)| { + // Singles and `compile` rows are not where a song is from. + matches!(album.record_type.as_deref(), Some("album") | Some("ep")) + // A guest record, a live album and a soundtrack are all + // releases the song is genuinely ON — none is where it is + // FROM. These three are Deezer-side evidence, independent + // of MusicBrainz, which is why the Broadway-cast class is + // rejected three separate ways. + && !album.appears_on + && !album.is_live + && !album.is_soundtrack + && !is_future_or_unresolved_date(album.release_date.as_deref()) + && containment + .get(&album.id) + .is_some_and(|titles| titles.contains(&key)) + && !musicbrainz_excludes_original(mb, album) + }) + .min_by(|(left_at, left), (right_at, right)| { + // An ALBUM outranks an EP, above everything else. A hit is + // routinely issued as a lead single/EP months before the record + // it belongs to, so on date alone the EP always wins: measured + // in the running app, Radiohead's "Creep" moved off *Pablo + // Honey* (1993-02-22, and Deezer's own correct answer) onto a + // "Creep EP" (1992-09-21). That is this feature making a right + // row wrong, and it is the same packaging-versus-work confusion + // the edition tiebreak below settles. An EP still wins whenever + // no album contains the song, so EP-only tracks are unaffected. + let is_ep = + |album: &CatalogAlbum| u8::from(album.record_type.as_deref() == Some("ep")); + // A plain title outranks an edition title ABOVE the date, not + // after it. Measured: `album_title_identity` reduces "Weezer + // (Deluxe Edition / Blue Album)" to `weezer`, which collides + // with seven self-titled Weezer groups and inherits 1994-05-10, + // while the plain "Weezer (Blue Album)" matches no group at all + // and keeps a `None` date. Date-first therefore picks the + // deluxe on every Weezer track; edition-first fixed all five + // and took the measured pass from 74/80 to 79/80. An album + // MusicBrainz has never heard of always loses to a wrongly- + // dated album it has, so this has to sit above the date too. + is_ep(left) + .cmp(&is_ep(right)) + .then_with(|| is_edition_title(&left.title).cmp(&is_edition_title(&right.title))) + // Only now the date, and only as a tiebreak. + .then_with(|| { + left.release_date + .as_deref() + .unwrap_or("9999") + .cmp(right.release_date.as_deref().unwrap_or("9999")) + }) + // An original album over a truncated sampler of it. + .then_with(|| { + right + .track_count + .unwrap_or(0) + .cmp(&left.track_count.unwrap_or(0)) + }) + // Curated-Deezer order, so the pick is deterministic. + .then_with(|| left_at.cmp(right_at)) + }); + let Some((_, album)) = winner else { + continue; + }; + // Deezer already licensed the record the song is from. This has to be + // an OUTPUT test rather than a candidate filter: excluding the current + // album from the ranking hands every already-correct row to the + // RUNNER-UP, which is how "Dreams" — correctly attributed to Rumours — + // moved onto "Rumours (Super Deluxe)". 57 of the 80 measured tracks + // are in this "already right, left alone" state. + if Some(album.id) == track.album_id { + continue; + } + // Everything below is the ATOMIC half: the row moves completely or not + // at all. Swapping `album_id` while leaving `id` on the licensed + // pressing is the one combination that silently kills album + // auto-continue (`maybeExtendAlbumQueue` finds no matching track) and + // suppresses the now-playing -> song jump, so a missing piece here has + // to abandon the whole swap rather than write part of it. + let Some(detail) = details.get(&album.id) else { + continue; + }; + let matched = detail + .pointer("/tracks/data") + .and_then(|rows| rows.as_array()) + .and_then(|rows| { + rows.iter().find(|row| { + row.get("title") + .and_then(|title| title.as_str()) + .is_some_and(|title| contained_track_identity(title) == key) + }) + }); + let Some(matched) = matched else { + continue; + }; + if matched.get("readable").and_then(|v| v.as_bool()) == Some(false) { + continue; + } + let Some(new_id) = matched.get("id").and_then(|v| v.as_u64()) else { + continue; + }; + // Take a SIZED cover. Deezer's bare `album.cover` has no `250x250` + // segment, and both the preview reel and the player upgrade a cover by + // rewriting that segment to 1000x1000 — handed a bare url the rewrite + // is a silent no-op and the reel shows a 120px thumbnail blown up. + let Some(cover) = detail + .get("cover_medium") + .or_else(|| detail.get("cover_big")) + .and_then(|c| c.as_str()) + else { + continue; + }; + + track.origin_track_id = Some(track.id); + track.id = new_id; + track.album = album.title.clone(); + track.album_id = Some(album.id); + track.cover = Some(cover.to_string()); + // Carried, not preserved: the preview modal treats this as one-bit + // evidence that a clip exists, and it must describe the row that is + // actually there now. + track.preview = matched + .get("preview") + .and_then(|p| p.as_str()) + .filter(|p| !p.is_empty()) + .map(String::from); + if let Some(duration) = matched.get("duration").and_then(|d| d.as_u64()) { + track.duration_secs = Some(duration as u32); + } + moved += 1; + } + moved +} + fn deezer_release_evidence(value: &serde_json::Value) -> Option { let title_norm = release_title_identity(value.get("title")?.as_str()?); if title_norm.is_empty() { @@ -7203,6 +8514,40 @@ fn track_identity(title: &str) -> String { normalize(&title) } +/// Track identity for CONTAINMENT: does this album's tracklist hold this song? +/// +/// Neither existing normalizer answers that on both sides. +/// [`track_title_identity`] deliberately keeps `(Live)` distinct — the test +/// `track_identity_folds_pressings_but_not_performances` pins it — and is built +/// on a different marker vocabulary than album tracklists use. +/// [`track_identity`] folds `(Remastered 2011)` exactly the way pressings need, +/// but has no `feat.` handling, so Deezer's `/top` row "Under Pressure (feat. +/// David Bowie)" would never match the plain "Under Pressure" inside *Hot +/// Space* — one of the twelve measured Queen rows. +/// +/// So: drop the featured-artist clause first, with the same markers +/// [`track_title_identity`] uses, then fold pressing suffixes. Both sides of +/// the comparison go through this one function. +fn contained_track_identity(title: &str) -> String { + let mut work = title.trim().to_lowercase(); + // A featured-artist clause is Deezer packaging, never part of the song. It + // arrives BOTH bare ("Song feat. X") and bracketed ("Song (feat. X)"), and + // `track_title_identity` only lists the bare form because its bracket scan + // catches the other one — `track_identity`, which this function ends in, + // has no such scan. Each opener is spelled out rather than searched for + // loosely, because the leading delimiter is the only thing stopping "ft " + // from matching inside "soft " or "left ". + for marker in [ + " feat.", " feat ", " ft.", " ft ", " featuring ", "(feat.", "(feat ", "(ft.", "(ft ", + "(featuring ", "[feat.", "[feat ", "[ft.", "[ft ", "[featuring ", + ] { + if let Some(at) = work.find(marker) { + work.truncate(at); + } + } + track_identity(&work) +} + fn edition_track_score(edition: &[String], original: &[String]) -> Option { let core = edition.len().min(original.len()).min(10); if core < 5 { @@ -12460,4 +13805,870 @@ mod tests { // No tracklist at all (e.g. a search result) resolves false, not a panic. assert!(!detail_is_live(&json!({ "title": "Some Album" }))); } + + fn album_with_date(title: &str, date: &str) -> CatalogAlbum { + CatalogAlbum { + id: 1, + title: title.to_string(), + artist: "A".into(), + cover: None, + release_date: Some(date.to_string()), + record_type: Some("album".into()), + track_count: Some(10), + edition_group: None, + is_soundtrack: false, + is_live: false, + appears_on: false, + } + } + + /// A plain studio album row, the shape the details pass leaves behind. + fn catalog_album(id: u64, title: &str, date: Option<&str>) -> CatalogAlbum { + CatalogAlbum { + id, + title: title.to_string(), + artist: "A".into(), + cover: None, + release_date: date.map(String::from), + record_type: Some("album".into()), + track_count: Some(10), + edition_group: None, + is_soundtrack: false, + is_live: false, + appears_on: false, + } + } + + /// The `/album/{id}` body shape the attribution rule reads: a sized cover + /// plus `tracks.data` rows carrying id / title / duration / preview. + fn album_detail(id: u64, titles: &[&str]) -> serde_json::Value { + json!({ + "id": id, + "cover_medium": format!( + "https://e-cdns-images.dzcdn.net/images/cover/{id}/250x250-000000-80-0-0.jpg" + ), + "tracks": { + "data": titles + .iter() + .enumerate() + .map(|(at, title)| json!({ + "id": id * 1000 + at as u64, + "title": title, + "duration": 200, + "preview": format!("https://cdns-preview.dzcdn.net/{id}-{at}.mp3"), + "readable": true, + })) + .collect::>() + } + }) + } + + /// Albums + the two maps `catalog_artist` derives from the details pass. + #[allow(clippy::type_complexity)] + fn attribution_fixture( + rows: &[(CatalogAlbum, &[&str])], + ) -> ( + Vec, + HashMap>, + HashMap, + ) { + let mut albums = Vec::new(); + let mut containment = HashMap::new(); + let mut details = HashMap::new(); + for (album, titles) in rows { + containment.insert( + album.id, + titles + .iter() + .map(|title| contained_track_identity(title)) + .collect(), + ); + details.insert(album.id, album_detail(album.id, titles)); + albums.push(album.clone()); + } + (albums, containment, details) + } + + fn scrobbles(plays: u64, listeners: u64) -> TrackStats { + TrackStats { plays, listeners } + } + + /// A MusicBrainz group with no secondary types — a plain studio release. + fn mb_album(date: &str) -> ReleaseFacts { + ReleaseFacts { + first: date.to_string(), + secondary: Vec::new(), + ambiguous: false, + } + } + + fn mb_typed(date: &str, kinds: &[&str]) -> ReleaseFacts { + ReleaseFacts { + first: date.to_string(), + secondary: kinds.iter().map(|k| k.to_string()).collect(), + ambiguous: false, + } + } + + /// Deezer's own attribution: the licensed pressing, on some other release. + fn licensed_track(id: u64, title: &str, album: &str, album_id: u64) -> CatalogTrack { + let mut track = popular_track(id, title); + track.album = album.to_string(); + track.album_id = Some(album_id); + track + } + + fn popular_track(id: u64, title: &str) -> CatalogTrack { + CatalogTrack { + id, + title: title.to_string(), + artist: "A".into(), + album: "B".into(), + album_id: Some(1), + cover: None, + duration_secs: Some(180), + preview: None, + track_no: None, + disk_number: None, + unreleased: false, + listeners: None, + plays: None, + origin_track_id: None, + } + } + + #[test] + fn track_identity_folds_pressings_but_not_performances() { + // Same recording, different pressing — these must collapse, or Last.fm's + // scrobble name never finds the Deezer row. + let shake = track_title_identity("Shake It Off"); + assert_eq!(track_title_identity("Shake It Off (Taylor's Version)"), shake); + assert_eq!(track_title_identity("Dreams - 2004 Remaster"), track_title_identity("Dreams")); + assert_eq!( + track_title_identity("Heartbreaker (Original Mix)"), + track_title_identity("Heartbreaker") + ); + assert_eq!( + track_title_identity("Like It (feat. Reckz'Capo)"), + track_title_identity("Like It") + ); + assert_eq!(track_title_identity("Creep [Radio Edit]"), track_title_identity("Creep")); + + // Genuinely different recordings. Collapsing these would delete a real + // track from the list, which is why the marker set is curated and not + // "strip every parenthetical". + let song = track_title_identity("Song"); + assert_ne!(track_title_identity("Song (Live)"), song); + assert_ne!(track_title_identity("Song (Acoustic)"), song); + assert_ne!(track_title_identity("Song (Reprise)"), song); + assert_ne!(track_title_identity("Song (JENNIE Remix)"), song); + // A parenthetical that is part of the title survives intact. + assert_eq!( + track_title_identity("(I Can't Get No) Satisfaction"), + normalize("(I Can't Get No) Satisfaction") + ); + } + + #[test] + fn lastfm_order_ranks_by_listeners_and_keeps_unscored_below() { + let pool = vec![ + popular_track(1, "New Single"), // Deezer's #1, nobody has scrobbled it + popular_track(2, "Deep Cut"), + popular_track(3, "The Hit"), + ]; + let ranks = HashMap::from([ + (track_title_identity("The Hit"), scrobbles(4_100_000, 900_000)), + (track_title_identity("Deep Cut"), scrobbles(48_000, 12_000)), + ]); + let ordered = apply_lastfm_order(pool, &ranks, 10); + let titles: Vec<&str> = ordered.iter().map(|t| t.title.as_str()).collect(); + // Ranked tracks descend by listeners; the unscored one is kept, not + // dropped, and sits below them. + assert_eq!(titles, ["The Hit", "Deep Cut", "New Single"]); + assert_eq!(ordered[0].listeners, Some(900_000)); + // Plays ride the same table and land on the row for the album column. + assert_eq!(ordered[0].plays, Some(4_100_000)); + assert_eq!(ordered[2].listeners, None); + assert_eq!(ordered[2].plays, None); + } + + #[test] + fn lastfm_order_never_admits_a_track_deezer_did_not_supply() { + // The merged-page case: Last.fm files several bands under one name, so + // its top track can belong to a different artist entirely. Membership + // comes from Deezer, so the stranger simply has nowhere to appear. + let pool = vec![popular_track(1, "Circus X")]; + let ranks = HashMap::from([ + (track_title_identity("Independent Love Song"), scrobbles(210_000, 35_745)), + (track_title_identity("Circus X"), scrobbles(31_000, 7_044)), + ]); + let ordered = apply_lastfm_order(pool, &ranks, 10); + assert_eq!(ordered.len(), 1); + assert_eq!(ordered[0].title, "Circus X"); + } + + #[test] + fn lastfm_order_collapses_pressings_to_the_entry_deezer_ranks_highest() { + // Both resolve to one key and would otherwise sort adjacently showing + // the same count twice. Deezer's order decides which survives. + let pool = vec![ + popular_track(7, "Heartbreaker (Original Mix)"), + popular_track(8, "Heartbreaker"), + popular_track(9, "Cruel Mistress"), + ]; + let ranks = HashMap::from([ + (track_title_identity("Heartbreaker"), scrobbles(402_000, 81_809)), + (track_title_identity("Cruel Mistress"), scrobbles(160_000, 35_536)), + ]); + let ordered = apply_lastfm_order(pool, &ranks, 10); + assert_eq!(ordered.len(), 2); + assert_eq!(ordered[0].id, 7); + assert_eq!(ordered[1].title, "Cruel Mistress"); + } + + #[test] + fn lastfm_order_without_ranks_is_deezers_own_order() { + // Every degradation path — no MBID, no Last.fm entry, guard refused — + // arrives here. Popular must fall back to what it always was. + let pool = vec![ + popular_track(1, "First"), + popular_track(2, "Second"), + popular_track(3, "Third"), + ]; + let ordered = apply_lastfm_order(pool, &HashMap::new(), 2); + let titles: Vec<&str> = ordered.iter().map(|t| t.title.as_str()).collect(); + assert_eq!(titles, ["First", "Second"]); + assert!(ordered.iter().all(|t| t.listeners.is_none())); + } + + #[test] + fn album_identity_folds_pressings_onto_the_work() { + // Every one of these is Deezer selling a re-press of one record. + let opera = album_title_identity("A Night at the Opera"); + assert_eq!(album_title_identity("A Night at the Opera (2011 Remaster)"), opera); + assert_eq!(album_title_identity("A Night at the Opera - Remastered"), opera); + assert_eq!(album_title_identity("A Night at the Opera (Deluxe Edition)"), opera); + assert_eq!(album_title_identity("A Night at the Opera (Super Deluxe)"), opera); + assert_eq!( + album_title_identity("A Night at the Opera (40th Anniversary Edition)"), + opera + ); + let idiot = album_title_identity("American Idiot"); + assert_eq!(album_title_identity("American Idiot (Deluxe)"), idiot); + assert_eq!(album_title_identity("Rumours (Super Deluxe)"), album_title_identity("Rumours")); + + // A re-recorded catalogue is its OWN work with its own first-release + // date, so it must not fold onto the original -- same exemption the + // frontend's EDITION_MARKER makes. + assert_ne!( + album_title_identity("Fearless (Taylor's Version)"), + album_title_identity("Fearless") + ); + // Genuinely different records that merely share a word. + assert_ne!( + album_title_identity("Greatest Hits"), + album_title_identity("Greatest Hits II") + ); + } + + #[test] + fn volume_numbering_matches_across_providers() { + // Measured on Queen: Deezer sells "Deep Cuts 2 (1977-1982)" while + // MusicBrainz files "Deep Cuts, Volume 2 (1977–1982)" — note also the + // en-dash. Keyed on raw titles these never met, so the whole Deep Cuts + // series kept Deezer's wrong date and sat in the Albums tab. + let deezer = album_title_identity("Deep Cuts 2 (1977-1982)"); + assert_eq!(album_title_identity("Deep Cuts, Volume 2 (1977\u{2013}1982)"), deezer); + assert_eq!(album_title_identity("Deep Cuts, Vol. 2 (1977-1982)"), deezer); + assert_eq!(album_title_identity("Deep Cuts Vol 2 (1977-1982)"), deezer); + // Different volumes stay different. + assert_ne!(album_title_identity("Deep Cuts, Volume 3 (1984-1995)"), deezer); + // A volume word with no number behind it is part of the title. + assert_eq!(album_title_identity("Volume"), normalize("volume")); + assert_eq!(album_title_identity("The Volume of Noise"), normalize("the volume of noise")); + } + + #[test] + fn original_dates_replace_deezers_reissue_dates() { + // The measured case: Deezer dates the CD re-press, not the record. + let mut albums = vec![ + album_with_date("A Night at the Opera (2011 Remaster)", "2005-11-22"), + album_with_date("The Game", "1994-10-17"), + album_with_date("Some Bootleg Deezer Alone Has", "1999-01-01"), + ]; + let first_released = HashMap::from([ + (release_date_key(Some("Album"), "A Night at the Opera"), mb_album("1975-11-21")), + (release_date_key(Some("Album"), "The Game"), mb_album("1980-06-24")), + ]); + let corrected = apply_original_release_dates(&mut albums, &first_released); + assert_eq!(corrected, 2); + assert_eq!(albums[0].release_date.as_deref(), Some("1975-11-21")); + assert_eq!(albums[1].release_date.as_deref(), Some("1980-06-24")); + // Unmatched keeps whatever Deezer said rather than losing its date. + assert_eq!(albums[2].release_date.as_deref(), Some("1999-01-01")); + + // And the corrected dates are what a chronological sort now uses. + albums.sort_by(|a, b| b.release_date.cmp(&a.release_date)); + let order: Vec<&str> = albums.iter().map(|a| a.title.as_str()).collect(); + assert_eq!(order[2], "A Night at the Opera (2011 Remaster)"); + } + + /// The measured fixture both attribution tests below run against: the + /// compilation is dated EARLIER than the album it compiles, which is the + /// systematic shape of real data and the reason "earliest wins" alone + /// selects hits packages. + #[allow(clippy::type_complexity)] + fn bohemian_fixture() -> ( + Vec, + HashMap>, + HashMap, + ) { + let anato = catalog_album(10, "A Night at the Opera", Some("1975-11-21")); + let hits = catalog_album(11, "Greatest Hits", Some("1971-11-01")); + let mut live = catalog_album(12, "Live at Wembley", Some("1986-07-12")); + live.is_live = true; + let mut ost = catalog_album( + 13, + "Bohemian Rhapsody (The Original Soundtrack)", + Some("2018-10-19"), + ); + ost.is_soundtrack = true; + let songs: &[&str] = &["Bohemian Rhapsody", "You're My Best Friend"]; + attribution_fixture(&[(anato, songs), (hits, songs), (live, songs), (ost, songs)]) + } + + #[test] + fn attribution_picks_the_earliest_studio_album_that_contains_the_track() { + let (albums, containment, details) = bohemian_fixture(); + let mut tracks = vec![licensed_track( + 999, + "Bohemian Rhapsody (2011 Remaster)", + "Bohemian Rhapsody (The Original Soundtrack)", + 13, + )]; + let mb = HashMap::from([ + ( + release_date_key(Some("Album"), "A Night at the Opera"), + mb_album("1975-11-21"), + ), + ( + release_date_key(Some("Album"), "Greatest Hits"), + mb_typed("1971-11-01", &["Compilation"]), + ), + ]); + + let moved = + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb); + assert_eq!(moved, 1); + let track = &tracks[0]; + // The compilation is two years EARLIER and still loses: the type + // exclusion outranks the date, which is the whole feature. + assert_eq!(track.album, "A Night at the Opera"); + assert_eq!(track.album_id, Some(10)); + // Every id-bearing field moves together, or album auto-continue dies + // on a queue seeded from one release and indexed by the other's id. + assert_eq!(track.origin_track_id, Some(999)); + assert_eq!(track.id, 10_000); + assert!(track.cover.as_deref().is_some_and(|c| c.contains("/10/"))); + // A SIZED cover, or the reel's 1000x1000 rewrite silently no-ops. + assert!(track.cover.as_deref().is_some_and(|c| c.contains("250x250"))); + assert!(track.preview.as_deref().is_some_and(|p| p.contains("10-0"))); + } + + #[test] + fn attribution_without_musicbrainz_secondary_types_would_pick_the_compilation() { + // The ablation, asserted as an explicit inverse. Deleting the + // secondary-type exclusion does not merely lose accuracy — it flips 16 + // of 80 measured picks to WRONG, because every hits package is dated + // before the records it draws from. This test exists so the guard + // cannot be quietly removed and still look green. + let (albums, containment, details) = bohemian_fixture(); + let mut tracks = vec![licensed_track( + 999, + "Bohemian Rhapsody (2011 Remaster)", + "Bohemian Rhapsody (The Original Soundtrack)", + 13, + )]; + let mb = HashMap::from([ + ( + release_date_key(Some("Album"), "A Night at the Opera"), + mb_album("1975-11-21"), + ), + // Same fixture, but MusicBrainz no longer says "Compilation". + ( + release_date_key(Some("Album"), "Greatest Hits"), + mb_album("1971-11-01"), + ), + ]); + + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb); + assert_eq!(tracks[0].album, "Greatest Hits"); + } + + #[test] + fn attribution_prefers_a_plain_title_over_an_edition_title() { + // Weezer, measured: `album_title_identity` strips the whole + // parenthetical from "Weezer (Deluxe Edition / Blue Album)" and yields + // `weezer`, which collides with seven self-titled Weezer groups and + // inherits 1994-05-10. The plain "Weezer (Blue Album)" reduces to a key + // MusicBrainz has never heard of and keeps no date at all — so an album + // MB does not know always loses to a wrongly-dated album it does. The + // edition tiebreak has to sit ABOVE the date, and doing so took the + // measured pass from 74/80 to 79/80. + let deluxe = catalog_album(20, "Weezer (Deluxe Edition / Blue Album)", Some("1994-05-10")); + let plain = catalog_album(21, "Weezer (Blue Album)", None); + let songs: &[&str] = &["Buddy Holly", "Say It Ain't So"]; + let (albums, containment, details) = attribution_fixture(&[(deluxe, songs), (plain, songs)]); + let mut tracks = vec![licensed_track(999, "Buddy Holly", "Some Sampler", 77)]; + let mb = HashMap::from([( + release_date_key(Some("Album"), "Weezer (Deluxe Edition / Blue Album)"), + mb_album("1994-05-10"), + )]); + + let moved = + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb); + assert_eq!(moved, 1); + assert_eq!(tracks[0].album, "Weezer (Blue Album)"); + } + + #[test] + fn attribution_leaves_a_track_alone_when_nothing_contains_it() { + // 3 of the 80 measured tracks match nothing — a remix, an "(Original)" + // and a one-off — and all three were already correct. + let currents = catalog_album(30, "Currents", Some("2015-07-17")); + let (albums, containment, details) = + attribution_fixture(&[(currents, &["Let It Happen", "Eventually"])]); + let mb = HashMap::from([( + release_date_key(Some("Album"), "Currents"), + mb_album("2015-07-17"), + )]); + let mut tracks = vec![licensed_track(999, "Dracula (JENNIE Remix)", "Dracula", 88)]; + let before = format!("{:?}", tracks[0]); + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 0 + ); + assert_eq!(format!("{:?}", tracks[0]), before); + + // And a containing album whose detail body was dropped (a Deezer quota + // response never enters `details`) cannot win either — never wrong, + // only less complete. + let mut tracks = vec![licensed_track(999, "Let It Happen", "Some Sampler", 88)]; + let before = format!("{:?}", tracks[0]); + assert_eq!( + apply_original_album_attribution( + &mut tracks, + &albums, + &containment, + &HashMap::new(), + &mb + ), + 0 + ); + assert_eq!(format!("{:?}", tracks[0]), before); + } + + #[test] + fn attribution_prefers_an_album_over_the_ep_that_previewed_it() { + // Caught in the running app on Radiohead. A hit is routinely issued as + // a lead single/EP months before the record it belongs to, so on date + // alone the EP always wins — "Creep" moved off *Pablo Honey*, which is + // Deezer's own correct answer, onto a "Creep EP" five months earlier. + let pablo = catalog_album(80, "Pablo Honey", Some("1993-02-22")); + let mut ep = catalog_album(81, "Creep EP", Some("1992-09-21")); + ep.record_type = Some("ep".into()); + let songs: &[&str] = &["Creep"]; + let (albums, containment, details) = attribution_fixture(&[(pablo, songs), (ep, songs)]); + let mb = HashMap::from([( + release_date_key(Some("Album"), "Pablo Honey"), + mb_album("1993-02-22"), + )]); + let mut tracks = vec![licensed_track(999, "Creep", "Some Sampler", 88)]; + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 1 + ); + assert_eq!(tracks[0].album, "Pablo Honey"); + + // But an EP still wins when no album carries the song, so a genuinely + // EP-only track is not dragged onto an unrelated record. + let other = catalog_album(82, "Amnesiac", Some("2001-06-04")); + let mut ep_only = catalog_album(83, "Drill EP", Some("1992-05-05")); + ep_only.record_type = Some("ep".into()); + let (albums, containment, details) = + attribution_fixture(&[(other, &["Pyramid Song"]), (ep_only, &["Prove Yourself"])]); + let mut tracks = vec![licensed_track(999, "Prove Yourself", "Some Sampler", 88)]; + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 1 + ); + assert_eq!(tracks[0].album, "Drill EP"); + } + + #[test] + fn attribution_leaves_a_row_deezer_already_got_right() { + // Caught in the running app on Fleetwood Mac. "Already right" must be + // an OUTPUT test, not a candidate filter: filtering the current album + // out of the ranking means an already-correct row can only ever be + // handed to the RUNNER-UP. Deezer had "Dreams" on plain Rumours, the + // filter removed plain Rumours from consideration, and the rule + // "corrected" it onto Rumours (Super Deluxe) — strictly worse, and 57 + // of the 80 measured tracks sit in exactly this state. + let plain = catalog_album(70, "Rumours", Some("1977-02-04")); + let deluxe = catalog_album(71, "Rumours (Super Deluxe)", Some("1977-02-04")); + let songs: &[&str] = &["Dreams", "Go Your Own Way"]; + let (albums, containment, details) = attribution_fixture(&[(plain, songs), (deluxe, songs)]); + let mb = HashMap::from([( + release_date_key(Some("Album"), "Rumours"), + mb_album("1977-02-04"), + )]); + let mut tracks = vec![licensed_track(999, "Dreams (2004 Remaster)", "Rumours", 70)]; + let before = format!("{:?}", tracks[0]); + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 0 + ); + assert_eq!(format!("{:?}", tracks[0]), before); + } + + #[test] + fn attribution_is_a_no_op_without_musicbrainz_evidence() { + // No linked MBID, or a failed browse, leaves `first_released` empty. + // The rule then returns byte-identical output to today rather than + // falling back to "earliest wins", which measured WORSE than Deezer. + let (albums, containment, details) = bohemian_fixture(); + let mut tracks = vec![licensed_track( + 999, + "Bohemian Rhapsody (2011 Remaster)", + "Bohemian Rhapsody (The Original Soundtrack)", + 13, + )]; + let before = format!("{:?}", tracks[0]); + assert_eq!( + apply_original_album_attribution( + &mut tracks, + &albums, + &containment, + &details, + &HashMap::new() + ), + 0 + ); + assert_eq!(format!("{:?}", tracks[0]), before); + } + + #[test] + fn attribution_never_moves_a_track_onto_a_guest_live_soundtrack_or_future_release() { + // One candidate per rejection clause, each of which contains the song. + // The Broadway cast recording is the measured failure of the earlier + // date-only attempt, and it is refused three independent ways. + let song: &[&str] = &["Boulevard of Broken Dreams"]; + let mut guest = catalog_album(40, "A Record Someone Else Owns", Some("2005-01-01")); + guest.appears_on = true; + let mut live = catalog_album(41, "Bullet in a Bible", Some("2005-11-15")); + live.is_live = true; + let mut cast = catalog_album( + 42, + "American Idiot - The Original Broadway Cast Recording", + Some("2010-04-12"), + ); + cast.is_soundtrack = true; + let future = catalog_album(43, "A Record Not Out Yet", Some("2099-01-01")); + let mut single = catalog_album(44, "Boulevard of Broken Dreams", Some("2004-11-29")); + single.record_type = Some("single".into()); + let (albums, containment, details) = attribution_fixture(&[ + (guest, song), + (live, song), + (cast, song), + (future, song), + (single, song), + ]); + let mut tracks = vec![licensed_track( + 999, + "Boulevard of Broken Dreams", + "Greatest Hits: God's Favorite Band", + 88, + )]; + let mb = HashMap::from([( + release_date_key(Some("Album"), "American Idiot"), + mb_album("2004-09-21"), + )]); + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 0 + ); + assert_eq!(tracks[0].album_id, Some(88)); + } + + #[test] + fn attribution_matches_across_pressing_suffixes_and_feat_clauses() { + // Deezer's /top row carries a featured-artist clause the album's own + // tracklist does not, and the two sides name the same remaster + // differently. `track_identity` alone misses the first, + // `track_title_identity` uses a different vocabulary for the second. + let hot_space = catalog_album(50, "Hot Space", Some("1982-05-04")); + let (albums, containment, details) = attribution_fixture(&[( + hot_space, + &["Under Pressure", "Bohemian Rhapsody (Remastered 2011)"], + )]); + let mb = HashMap::from([( + release_date_key(Some("Album"), "Hot Space"), + mb_album("1982-05-04"), + )]); + let mut tracks = vec![ + licensed_track(901, "Under Pressure (feat. David Bowie)", "Sampler", 77), + licensed_track(902, "Bohemian Rhapsody (2011 Remaster)", "Sampler", 77), + ]; + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 2 + ); + assert!(tracks.iter().all(|track| track.album == "Hot Space")); + } + + #[test] + fn attribution_keeps_a_re_recording_on_its_first_album() { + // KNOWN AND ACCEPTED — the single regression in 80 measured tracks. + // "Welcome to Paradise" genuinely first appeared on Kerplunk! in 1991; + // Deezer's top row is the 1994 Dookie re-recording. Containment cannot + // tell a re-recording from a reissue, and a duration window was + // rejected because the two cuts are seconds apart. Pinned so the + // behaviour is a decision on record rather than a surprise later. + let kerplunk = catalog_album(60, "Kerplunk!", Some("1991-12-17")); + let dookie = catalog_album(61, "Dookie", Some("1994-02-01")); + let song: &[&str] = &["Welcome to Paradise"]; + let (albums, containment, details) = + attribution_fixture(&[(kerplunk, song), (dookie, song)]); + let mut tracks = vec![licensed_track(999, "Welcome to Paradise", "Sampler", 88)]; + let mb = HashMap::from([ + ( + release_date_key(Some("Album"), "Kerplunk!"), + mb_album("1991-12-17"), + ), + ( + release_date_key(Some("Album"), "Dookie"), + mb_album("1994-02-01"), + ), + ]); + assert_eq!( + apply_original_album_attribution(&mut tracks, &albums, &containment, &details, &mb), + 1 + ); + assert_eq!(tracks[0].album, "Kerplunk!"); + } + + #[test] + fn release_group_facts_take_secondary_types_from_the_earliest_group() { + // Fleetwood Mac, measured: MusicBrainz has three groups keyed + // album/fleetwoodmac — the real 1975 album with no secondary types, a + // 1979 Compilation and a 2006 Compilation+Live. Unioning the types + // across the key tagged the 1975 album Compilation+Live, the + // attribution rule excluded it, and "Rhiannon" landed on Greatest Hits. + // The types must come from whichever group supplied the surviving date. + let mut facts = HashMap::new(); + for group in [ + json!({"title": "Fleetwood Mac", "primary-type": "Album", "first-release-date": "2006-01-01", "secondary-types": ["Compilation", "Live"]}), + json!({"title": "Fleetwood Mac", "primary-type": "Album", "first-release-date": "1975-07-11"}), + json!({"title": "Fleetwood Mac", "primary-type": "Album", "first-release-date": "1979-01-01", "secondary-types": ["Compilation"]}), + ] { + merge_release_group_facts(&mut facts, &group); + } + let held = facts + .get(&release_date_key(Some("Album"), "Fleetwood Mac")) + .expect("the key survives"); + assert_eq!(held.first, "1975-07-11"); + assert!(held.secondary.is_empty()); + + // On an equal date the plainer group wins — a compilation that shares + // both title and date with the album it compiles must not tag it. + let mut tie = HashMap::new(); + for group in [ + json!({"title": "Tie", "primary-type": "Album", "first-release-date": "1980-01-01", "secondary-types": ["Compilation"]}), + json!({"title": "Tie", "primary-type": "Album", "first-release-date": "1980-01-01"}), + ] { + merge_release_group_facts(&mut tie, &group); + } + assert!(tie + .get(&release_date_key(Some("Album"), "Tie")) + .expect("the key survives") + .secondary + .is_empty()); + } + + #[test] + fn musicbrainz_disambiguation_separates_seven_self_titled_albums() { + // Measured live: MusicBrainz files SEVEN self-titled Weezer albums, + // every one titled plain "Weezer", separated only by `disambiguation`. + // Collapsing them to one key and keeping the earliest dated the 2026 + // Gold Album — an unreleased record — 1994-05-10, and filed it beside + // the debut. + let mut facts = HashMap::new(); + for (date, disamb) in [ + ("1994-05-10", "Blue Album"), + ("2001-05-07", "Green Album"), + ("2008-01-01", "Red Album"), + ("2016-04-01", "White Album"), + ("2019-01-24", "Teal Album"), + ("2019-03-01", "Black Album"), + ("2026-08-21", "Gold Album"), + ] { + merge_release_group_facts( + &mut facts, + &json!({ + "title": "Weezer", + "primary-type": "Album", + "first-release-date": date, + "disambiguation": disamb, + }), + ); + } + + // Deezer prints the disambiguator in its parenthetical, so each colour + // now lands on its own record. + for (date, title) in [ + ("1994-05-10", "Weezer (Blue Album)"), + ("2001-05-07", "Weezer (Green Album)"), + ("2008-01-01", "Weezer (Red Album)"), + ("2026-08-21", "Weezer (Gold Album)"), + // Deezer's mixed pressing/disambiguator parenthetical too. + ("2008-01-01", "Weezer (Deluxe Edition / Red Album)"), + ] { + let held = facts + .get(&release_date_key(Some("Album"), title)) + .unwrap_or_else(|| panic!("no entry for {title}")); + assert!(!held.ambiguous, "{title} should be unambiguous"); + assert_eq!(held.first, date, "{title}"); + } + + // And the bare key, which is genuinely seven records at once, refuses + // to date anything rather than guessing the earliest. + let bare = facts + .get(&release_date_key(Some("Album"), "Weezer")) + .expect("the bare key still exists"); + assert!(bare.ambiguous); + let mut albums = vec![album_with_date("Weezer", "2026-08-21")]; + assert_eq!(apply_original_release_dates(&mut albums, &facts), 0); + assert_eq!(albums[0].release_date.as_deref(), Some("2026-08-21")); + } + + #[test] + fn a_reissue_duplicate_still_dates_from_the_earliest_group() { + // The ambiguity guard must not throw away the case it was always + // handling: two groups for ONE work. Same date is not ambiguous, and + // the plainer of the two still carries the secondary types. + let mut facts = HashMap::new(); + for group in [ + json!({"title": "Dookie", "primary-type": "Album", "first-release-date": "1994-02-01"}), + json!({"title": "Dookie", "primary-type": "Album", "first-release-date": "1994-02-01", "secondary-types": ["Compilation"]}), + ] { + merge_release_group_facts(&mut facts, &group); + } + let held = facts + .get(&release_date_key(Some("Album"), "Dookie")) + .expect("the key survives"); + assert!(!held.ambiguous); + assert!(held.secondary.is_empty()); + assert_eq!(held.first, "1994-02-01"); + } + + #[test] + fn a_mixed_edition_parenthetical_keeps_its_disambiguator() { + // Deezer sells Weezer's colour records as "Weezer (Deluxe Edition / + // Red Album)". Stripping that parenthetical whole threw away the only + // thing saying WHICH self-titled album it is, so the 2008 Red Album + // deluxe reduced to `weezer` — the same key as the 1994 debut — and + // MusicBrainz, which titles all seven self-titled groups plain + // "Weezer", dated it 1994-05-10. + assert_eq!( + album_title_identity("Weezer (Deluxe Edition / Red Album)"), + album_title_identity("Weezer (Red Album)") + ); + assert_ne!( + album_title_identity("Weezer (Deluxe Edition / Red Album)"), + album_title_identity("Weezer (Deluxe Edition / Blue Album)") + ); + // A parenthetical that is ONLY a pressing still collapses onto the + // work, which is what the whole date pass depends on. + assert_eq!( + album_title_identity("A Night at the Opera (2011 Remaster)"), + album_title_identity("A Night at the Opera") + ); + assert_eq!( + album_title_identity("Weezer (Deluxe Edition)"), + album_title_identity("Weezer") + ); + // And a parenthetical with no pressing in it is untouched. + assert_eq!( + album_title_identity("Weezer (Blue Album)"), + normalize("weezerbluealbum") + ); + } + + #[test] + fn an_edition_inherits_the_date_of_the_work_it_presses() { + // The tracklist-verified edition link knows which record a pressing + // is; the title does not. Measured on Weezer: the Red Album deluxe + // (edition_group -> the 2008 Red Album) was showing 1994-05-10. + let mut red = album_with_date("Weezer (Red Album)", "2008-06-16"); + red.id = 262150; + red.edition_group = Some(262150); + let mut red_deluxe = album_with_date("Weezer (Deluxe Edition / Red Album)", "1994-05-10"); + red_deluxe.id = 320925257; + red_deluxe.edition_group = Some(262150); + // An unlinked album is left entirely alone. + let mut green = album_with_date("Weezer (Green Album)", "2001-04-24"); + green.id = 249320; + + let mut albums = vec![red, red_deluxe, green]; + assert_eq!(apply_edition_group_dates(&mut albums), 1); + assert_eq!(albums[1].release_date.as_deref(), Some("2008-06-16")); + // The original is its own group and must never be rewritten. + assert_eq!(albums[0].release_date.as_deref(), Some("2008-06-16")); + assert_eq!(albums[2].release_date.as_deref(), Some("2001-04-24")); + // Idempotent: a second pass changes nothing. + assert_eq!(apply_edition_group_dates(&mut albums), 0); + } + + #[test] + fn a_single_never_dates_the_album_that_shares_its_name() { + // Queen, measured: "The Miracle" exists as a 1989 album AND a 1984 + // single. Keyed on title alone the single won and the album was dated + // five years before it was made. + let mut albums = vec![ + album_with_date("The Miracle", "2023-09-29"), + album_with_date("The Miracle (Collector's Edition)", "2022-11-18"), + ]; + let first_released = HashMap::from([ + (release_date_key(Some("Album"), "The Miracle"), mb_album("1989-05-22")), + (release_date_key(Some("Single"), "The Miracle"), mb_album("1984-11-27")), + ]); + apply_original_release_dates(&mut albums, &first_released); + assert_eq!(albums[0].release_date.as_deref(), Some("1989-05-22")); + // The collector's edition folds onto the same work, and onto the ALBUM. + assert_eq!(albums[1].release_date.as_deref(), Some("1989-05-22")); + } + + #[test] + fn lastfm_identifier_ignores_the_two_space_encodings() { + // MusicBrainz declares %20, Last.fm echoes +. A byte compare rejects 2 + // of every 18 real matches, so the guard compares decoded identifiers. + let declared = lastfm_artist_identifier("https://www.last.fm/music/Inland%20Knights"); + let echoed = lastfm_artist_identifier("https://www.last.fm/music/Inland+Knights"); + assert_eq!(declared, echoed); + assert_eq!(declared.as_deref(), Some("inland knights")); + // A different artist still has to fail. + assert_ne!( + lastfm_artist_identifier("https://www.last.fm/music/Scarlet"), + lastfm_artist_identifier("https://www.last.fm/music/Scarlett") + ); + // Non-ASCII survives the round trip (MusicBrainz declares this one for + // Taylor Swift). + assert_eq!( + lastfm_artist_identifier( + "https://www.last.fm/music/%E3%83%86%E3%82%A4%E3%83%A9%E3%83%BC" + ) + .as_deref(), + Some("テイラー") + ); + } }