andrew/alembic
1
0
Fork 0
Code Pull Requests Actions Releases 18 Wiki Activity

Fix stuck dedup candidate, enrich timeout/starvation, and qobuz status error

Browse Source 
1. enrich-buy-url.py re-queried every no-match track on every daily run (external,
   rate-limited lookups over the whole backlog), which blew past its 30-min
   timeout. Cache "tried, no match" with a BUY_URL_TRIED marker tag and skip
   those for 30 days (--force still re-checks). This is what made "Look up buy
   links" fail with TIMEOUT after 1800s.

2. Move maintenance:enrich_buy_url to run last (09:45) in the 9am block, after
   the fingerprint index (09:25) and status report (09:30). enrich starting at
   09:10 and holding the shared lock is what left "Rebuild fingerprint index"
   skipped_lock.

3. pipeline-status.sh: guard the qobuz/app_id read with a file-exists check.
   `< missing 2>/dev/null` still leaks the shell's redirection error (opened
   before 2>/dev/null applies), so the daily report logged
   "qobuz/app_id: No such file or directory" every run.

4. dedup confirm_and_apply: a candidate whose delete target no longer exists
   (already removed by an earlier dedup/upgrade/hand) was filtered out and the
   apply silently no-op'd, leaving it stuck in the queue with no way to delete
   or clear it (the 100 gecs case). Now mark such candidates applied so a Delete
   click clears them. Covered by tests/test_dedup_review.py.

Tests: 46 green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
andrew
2026-07-11 14:54:52 -06:00
parent fdcba992fd
commit 04c36af6b3
5 changed files with 121 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/services/dedup_review_service.py
+24 -5
View File
@@ -208,15 +208,33 @@ async def confirm_and_apply(candidate_ids: list[int], confirmed_by: str) -> Dedu
try:
candidates = [db.get(DedupCandidate, cid) for cid in candidate_ids]
candidates = [c for c in candidates if c is not None and not c.applied]
candidates = [c for c in candidates if Path(c.delete_path).exists() and Path(c.keep_path).exists()]
if not candidates:
return None
now = time.time()
# A candidate whose delete target no longer exists (already removed by an
# earlier dedup, the mp3->flac upgrade, or by hand) is effectively already
# resolved. Mark it applied so it clears from the pending queue, instead
# of the old behavior of filtering it out and silently no-oping -- which
# left such a stale candidate stuck in the list forever with no way to
# delete or clear it.
already_gone = [c for c in candidates if not Path(c.delete_path).exists()]
for c in already_gone:
c.confirmed = True
c.confirmed_by = confirmed_by
c.confirmed_at = now
c.applied = True
if already_gone:
db.commit()
# The rest need both files present for a safe delete.
actionable = [c for c in candidates if Path(c.delete_path).exists() and Path(c.keep_path).exists()]
by_script: dict[str, list[DedupCandidate]] = {}
for c in candidates:
for c in actionable:
by_script.setdefault(_script_for_pass(c.pass_name), []).append(c)
now = time.time()
finished_at = now
log_paths = []
ran_candidates: list[DedupCandidate] = []
@@ -246,7 +264,8 @@ async def confirm_and_apply(candidate_ids: list[int], confirmed_by: str) -> Dedu
ran_candidates.extend(group)
db.commit()
if not ran_candidates:
# Nothing to report: no stale ones cleared and nothing ran.
if not already_gone and not ran_candidates:
return None
still_there = {c.delete_path for c in ran_candidates if Path(c.delete_path).exists()}
@@ -261,7 +280,7 @@ async def confirm_and_apply(candidate_ids: list[int], confirmed_by: str) -> Dedu
started_at=now,
finished_at=finished_at,
mode="apply",
deleted=actually_deleted,
deleted=actually_deleted + len(already_gone),
kept=len(ran_candidates) - actually_deleted,
log_path=";".join(log_paths) or None,
)