Add dedup_review_service and genre_review_service

dedup-library.sh: additive --json (one NDJSON line per candidate deletion
to stdout, alongside the unchanged human log) and --only-paths FILE (in
--apply mode, only actually delete entries whose path is in FILE; without
it, --apply deletes everything as before -- existing direct callers are
unaffected). Without --only-paths the safety re-verification is free:
--apply --only-paths re-runs all 4 passes from scratch on every invocation,
so if a group's ranking changed since a scan (e.g. the old keep_path is
gone), the fresh pass assigns the previously-"delete" path the KEEP role
instead and the only-paths allowlist naming it is simply never consulted --
no duplicate ranking logic needed in the review service.

spotify-genre.py: additive --json emitting one JSON line per genre change
(dry-run or --apply) for genre_review_service to persist.

pipeline_runner.run_job_capture(): like run_job() but captures stdout as
text (still under the same shared lock, still writes a job_runs row) for
callers that need to parse structured output rather than just log it.

dedup_review_service.scan() persists dry-run candidates into
dedup_runs/dedup_candidates. confirm_and_apply() re-checks confirmed
candidates still exist before invoking --apply --only-paths, so nothing is
ever deleted without an explicit confirm -- matches the false-negative-
biased dedup preference. scheduler_service's maintenance:dedup job now
goes through this (still dry-run only, every day).

genre_review_service.run() wraps spotify-genre.py for both dry-run preview
and the real scheduled --apply run, persisting every run's diff into
genre_runs/genre_candidates either way -- genre writes keep their current
auto-apply behavior (low-risk, reversible, GENRE_LOCK-protected) but are
now reviewable after the fact. lock_artist_genre() gives a one-click revert
path when a run gets something wrong.

Added minimal routers+templates for /dedup (scan, review, confirm-and-
delete) and /genres (preview, review, lock-old-genre).

Verified end-to-end against REAL duplicate files (not mocked): built an
actual FLAC+MP3 duplicate pair in a real beets library, ran dedup-library.sh
--json and confirmed correct JSON output, verified --apply --only-paths
with an empty confirm list deletes nothing and with the real confirmed path
deletes exactly that file (DB + disk) while preserving the FLAC, and ran
the full dedup_review_service scan->confirm->apply flow through the same
fixture. genre_review_service and spotify-genre.py --json verified against
mocked/direct output (spotify-genre.py's own artist-genre lookup needs a
live Spotify API call, out of reach in this sandbox). Confirmed the full
app boots with all five routers registered.
This commit is contained in:
andrew
2026-07-08 14:17:20 -06:00
parent 270c9c0ed1
commit 2075d6cf66
11 changed files with 575 additions and 7 deletions
+18 -3
View File
@@ -6,7 +6,7 @@ from sqlalchemy import select
from app.db import SessionLocal
from app.models import Playlist, ScheduledJob
from app.services import pipeline_runner
from app.services import dedup_review_service, genre_review_service, pipeline_runner
from app.settings import settings
TIMEZONE = "America/Edmonton"
@@ -44,6 +44,21 @@ def _beet(job_key: str, args: list[str]):
return _run
async def _dedup_scan(triggered_by: str = "schedule"):
"""Dry-run only, ever, on the schedule -- see module docstring. Populates
dedup_candidates for review; deletion only ever happens through a
confirmed dedup_review_service.confirm_and_apply() call from the UI."""
return await dedup_review_service.scan(triggered_by=triggered_by)
async def _genre_run(triggered_by: str = "schedule"):
"""Keeps its current auto-apply behavior (unlike dedup) -- genre writes
are low-risk/reversible and GENRE_LOCK-protected -- but now goes through
genre_review_service so every run's diff is persisted for after-the-fact
review instead of only living in a log file."""
return await genre_review_service.run(apply=True, force=True, triggered_by=triggered_by)
async def _log_rotation(triggered_by: str = "schedule"):
"""Replaces `find /var/log/sldl -name '*.log' -mtime +30 -delete`."""
cutoff = time.time() - 30 * 86400
@@ -68,7 +83,7 @@ MAINTENANCE_JOBS: dict[str, tuple[dict, callable]] = {
),
"maintenance:dedup": (
dict(minute=55, hour=8),
_lib("maintenance:dedup", "dedup-library.sh"), # no --apply: dry-run only
_dedup_scan, # no --apply, ever, on schedule -- see _dedup_scan docstring
),
"maintenance:gen_djmix_playlist": (
dict(minute=57, hour=8),
@@ -125,7 +140,7 @@ MAINTENANCE_JOBS: dict[str, tuple[dict, callable]] = {
),
"maintenance:spotify_genre": (
dict(minute=50, hour=8, day_of_week="sun"),
_lib("maintenance:spotify_genre", "spotify-genre.py", ["--apply", "--force"]),
_genre_run, # goes through genre_review_service -- see its docstring
),
"maintenance:beets_update_sync": (
dict(minute=53, hour=8, day_of_week="sun"),