Fix stuck dedup candidate, enrich timeout/starvation, and qobuz status error
1. enrich-buy-url.py re-queried every no-match track on every daily run (external, rate-limited lookups over the whole backlog), which blew past its 30-min timeout. Cache "tried, no match" with a BUY_URL_TRIED marker tag and skip those for 30 days (--force still re-checks). This is what made "Look up buy links" fail with TIMEOUT after 1800s. 2. Move maintenance:enrich_buy_url to run last (09:45) in the 9am block, after the fingerprint index (09:25) and status report (09:30). enrich starting at 09:10 and holding the shared lock is what left "Rebuild fingerprint index" skipped_lock. 3. pipeline-status.sh: guard the qobuz/app_id read with a file-exists check. `< missing 2>/dev/null` still leaks the shell's redirection error (opened before 2>/dev/null applies), so the daily report logged "qobuz/app_id: No such file or directory" every run. 4. dedup confirm_and_apply: a candidate whose delete target no longer exists (already removed by an earlier dedup/upgrade/hand) was filtered out and the apply silently no-op'd, leaving it stuck in the queue with no way to delete or clear it (the 100 gecs case). Now mark such candidates applied so a Delete click clears them. Covered by tests/test_dedup_review.py. Tests: 46 green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -40,6 +40,18 @@ from pathlib import Path
|
||||
|
||||
UA = "Mozilla/5.0 (X11; Linux x86_64) enrich-buy-url/1.0"
|
||||
BUY_URL_TAG = "COMMERCIAL_INFORMATION"
|
||||
# Marker (unix timestamp) stamped on a FLAC when a buy-link lookup found
|
||||
# nothing, so daily runs don't re-query the whole no-match backlog every time.
|
||||
# Retried after the window below; --force ignores it.
|
||||
BUY_URL_TRIED_TAG = "BUY_URL_TRIED"
|
||||
_TRIED_RETRY_DAYS = 30
|
||||
|
||||
|
||||
def _tried_recently(ts_str):
|
||||
try:
|
||||
return (time.time() - float(ts_str)) < _TRIED_RETRY_DAYS * 86400
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
ITUNES_SEARCH = "https://itunes.apple.com/search"
|
||||
BC_AUTOCOMPLETE = "https://bandcamp.com/api/bcsearch_public_api/1/autocomplete_elastic"
|
||||
QOBUZ_SEARCH = "https://www.qobuz.com/api.json/0.2/track/search"
|
||||
@@ -451,6 +463,14 @@ def main():
|
||||
else:
|
||||
continue # already tagged, not up for upgrade
|
||||
else:
|
||||
# Skip tracks we recently tried and found nothing for, so a daily
|
||||
# run doesn't re-query the whole no-match backlog (external, rate-
|
||||
# limited lookups) every time -- that churn is what pushed this job
|
||||
# past its timeout. --force re-checks everything.
|
||||
if not args.force:
|
||||
tried = flac_tag(sp, BUY_URL_TRIED_TAG)
|
||||
if tried and _tried_recently(tried):
|
||||
continue
|
||||
run_sources = cascade # untagged, or --force
|
||||
artist = flac_tag(sp, "ARTIST").split(";")[0].strip() or flac_tag(sp, "ALBUMARTIST")
|
||||
title = flac_tag(sp, "TITLE")
|
||||
@@ -470,6 +490,10 @@ def main():
|
||||
|
||||
rel = sp[len(args.library):].lstrip("/")
|
||||
if not url:
|
||||
# Record that we checked and found nothing, so the next daily run
|
||||
# skips it (until the retry window). Apply mode only.
|
||||
if args.apply:
|
||||
set_flac_tag(sp, BUY_URL_TRIED_TAG, str(int(time.time())))
|
||||
continue
|
||||
found += 1
|
||||
by_source[src] += 1
|
||||
|
||||
Reference in New Issue
Block a user