3 Commits

Author SHA1 Message Date
andrew e07e931c45 0.6.12: Redesign the dedup review table for faster, more accurate scanning
The keep/delete columns were raw paths in a fixed-width cell: ellipsis-
truncated, full text only on hover. Slow to scan (mousing over every
row to read the song name) and, once "fixed" with a tag-derived title/
artist header in a first pass of this change, actively worse for the
thing dedup review actually needs -- confirming two files are really
the same recording. Tags can be wrong; the filename on disk can't.

New layout: each candidate gets a title row (song title, large,
unclipped -- parsed from beets' known singleton path template,
Artist/Album/Title.ext) followed by a compact detail row. The detail
row shows a color-coded format pill (FLAC/MP3/etc, matching the same
quality judgment dedup-library.sh's rank_file() already makes) and the
full relative file path, always rendered as visible text -- never
hover-only -- so a real difference in filename, album, or folder is
still plainly visible even when tags line up. Shared artist collapses
onto the title row so it isn't repeated per side; a colored left rail
marks which side survives without having to read the words.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 10:15:27 -06:00
andrew 6114e6dc7a 0.6.11: Auto-apply exact-tag format upgrades; self-heal stale pending rows
Two gaps left obvious cases stuck in the manual dedup queue:

1. Format-upgrade pairs found by Pass 2/3 (case-insensitive/normalized
   tag match) still required manual confirm even though the pass
   itself already proved same song via identical tags, and rank_file()
   already guarantees FLAC beats any other format regardless of
   size/dirty-name. A directory-casing difference (e.g. "All About
   This Ep" vs "...EP") meant these never matched the narrower
   same-directory numbered-twin rule from 0.6.10. New is_format_upgrade()
   auto-deletes any Pass 2/3 pair whose extensions differ, gated on
   pass_name so Pass 4 (cross-album fuzzy -- different masters, DJ-mix
   edits, genuinely ambiguous) is untouched and still requires a human.

2. A pending row can go stale without ever being confirmed -- some
   other action (a later auto-apply, upgrade-mp3-to-flac, a manual fix)
   already resolved one side of the pair -- and nothing pruned it, so
   an already-fixed duplicate kept surfacing in the queue indefinitely.
   scan() now sweeps and auto-resolves any pending candidate whose
   keep_path or delete_path no longer exists before persisting new
   ones, so the queue reflects current reality on every run instead of
   accumulating dead entries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 09:50:55 -06:00
andrew f05f076464 0.6.10: Auto-delete identical numbered-sibling duplicates, no review needed
Every dedup pass previously funneled through the same manual-confirm
queue, including the obviously-safe case: two files in the same
directory with the same name and extension, differing only by the
".N" collision suffix sldl/beets inserts when a filename collides.
That's not a fuzzy match or a different version -- it's the same
download landing twice -- so it doesn't need a human in the loop the
way case-insensitive tag matches or cross-album fuzzy matches do
(different masters, DJ-mix versions, etc., which still require
confirmation).

dedup-library.sh gains --auto-apply-numbered: an is_numbered_twin()
filename check (same dir, same name modulo the numeric infix, same
extension) that deletes matching pairs unconditionally, regardless of
which pass found them -- Pass 1 only fires when the canonical name is
a beets ghost; the common case where both copies are already tracked
in beets defers to Pass 2's tag-based grouping instead, and still gets
the same free pass here. When the auto-deleted pair's survivor is
itself the numbered-named file, it's renamed back to canonical so the
library doesn't accumulate ".1."/".2." names for tracks that no longer
have a duplicate.

dedup_review_service.scan() (both the manual "Scan now" button and the
nightly scheduled job) now passes this flag and records auto-applied
deletions as pre-confirmed DedupCandidate rows for audit visibility --
they never appear as pending review items. Tag-based and cross-album
fuzzy passes are unaffected and still require manual confirmation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 09:34:16 -06:00
6 changed files with 374 additions and 74 deletions
+3 -3
View File
@@ -73,10 +73,10 @@ Optional, add these later if you want them:
Pull the prebuilt image onto your Docker host:
```bash
docker pull git.kretzer.club/andrew/alembic:0.6.9
docker pull git.kretzer.club/andrew/alembic:0.6.12
```
That is the whole install. You do not need to download the source or build anything. The `0.6.9` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
That is the whole install. You do not need to download the source or build anything. The `0.6.12` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
(If you would rather build it yourself from source, you can, but you do not need to.)
@@ -136,7 +136,7 @@ Create a file called `docker-compose.yml` on your server (put it wherever you ke
```yaml
services:
alembic:
image: git.kretzer.club/andrew/alembic:0.6.9
image: git.kretzer.club/andrew/alembic:0.6.12
container_name: alembic
ports:
- "8420:8420"
+68 -5
View File
@@ -1,3 +1,6 @@
import os
import re
from fastapi import APIRouter, BackgroundTasks, Depends, Request
from fastapi.responses import RedirectResponse
from fastapi.templating import Jinja2Templates
@@ -30,16 +33,76 @@ def _display_path(path: str) -> str:
return path[len(_LIBRARY_PREFIX):] if path.startswith(_LIBRARY_PREFIX) else path
# Compilation-style import paths carry a "$track " prefix on the filename
# (see pipeline/configs/beets config.yaml paths: comp/albumtype_soundtrack);
# singleton paths (the vast majority of this library) don't. Strip it either
# way so the title reads clean.
_TRACK_PREFIX_RE = re.compile(r"^\d{1,3}[\s.\-]+")
# rank_file() in dedup-library.sh always ranks FLAC above every other format
# and WAV below MP3 (a download glitch, never desirable) -- mirror that
# judgment in the format pill's color so it reads as "this one's the keeper"
# at a glance, not just a bare extension string.
_FORMAT_BADGE = {"flac": "badge-success", "wav": "badge-warning"}
def _split_display(path: str) -> dict:
"""Break a library display path into (artist, album, title, ext).
Beets' singleton path template is always
%the{$albumartist}/$album/$title (pipeline/configs/beets config.yaml,
paths:) -- every track in this library lands at exactly that shape, so
the last two segments are reliably album/filename. A path that doesn't
fit (an unexpected root-level file) degrades to showing the raw display
path as the title instead of guessing at structure that isn't there.
"""
display = _display_path(path)
parts = display.split("/")
if len(parts) >= 2:
artist, album, filename = parts[0], parts[-2], parts[-1]
title, ext = os.path.splitext(filename)
title = _TRACK_PREFIX_RE.sub("", title)
else:
artist, album = "", ""
title, ext = os.path.splitext(display)
return {
"path": path,
"display": display,
"artist": artist,
"album": album,
"title": title,
"ext": ext.lstrip(".").lower(),
}
def _file_size(path: str) -> int | None:
try:
return os.path.getsize(path)
except OSError:
return None
def _to_row(c) -> dict:
keep = _split_display(c.keep_path)
delete = _split_display(c.delete_path)
keep["size_bytes"] = _file_size(c.keep_path)
keep["badge"] = _FORMAT_BADGE.get(keep["ext"], "badge-muted")
delete["size_bytes"] = c.delete_size_bytes if c.delete_size_bytes is not None else _file_size(c.delete_path)
delete["badge"] = _FORMAT_BADGE.get(delete["ext"], "badge-muted")
return {
"id": c.id,
"pass_name": c.pass_name,
"pass_label": _pass_label(c.pass_name),
"keep_path": c.keep_path,
"delete_path": c.delete_path,
"keep_display": _display_path(c.keep_path),
"delete_display": _display_path(c.delete_path),
"delete_size_bytes": c.delete_size_bytes,
"keep": keep,
"delete": delete,
# Same-tag passes (numbered_sibling/case_insensitive/normalized) share
# artist/title almost always; cross_album_fuzzy and fuzzy_audio can
# legitimately differ (different credit, different album entirely) --
# the template only collapses shared context onto one line when it's
# actually shared, otherwise shows both sides in full.
"same_title": keep["title"].lower() == delete["title"].lower(),
"same_artist": keep["artist"].lower() == delete["artist"].lower(),
"same_album": keep["album"].lower() == delete["album"].lower(),
}
+83 -10
View File
@@ -27,6 +27,38 @@ def _script_for_pass(pass_name: str) -> str:
return _FUZZY_SCRIPT if pass_name == _FUZZY_PASS else _SCRIPT
def _prune_stale_pending(db) -> int:
"""Mark pending candidates whose delete_path or keep_path no longer
exists as resolved, instead of leaving them to surface forever.
A pending row can go stale without ever being confirmed through the UI --
e.g. a later scan's numbered-twin/format-upgrade auto-apply already
removed one side, or upgrade-mp3-to-flac replaced it, or someone deleted
it by hand. confirm_and_apply() already does this "already_gone" check
for candidates a user explicitly acts on; this generalizes it to run at
the start of every scan, so the pending list reflects current reality
instead of stale groups that something else already resolved."""
now = time.time()
rows = db.execute(
select(DedupCandidate).where(
DedupCandidate.applied == False, # noqa: E712
DedupCandidate.confirmed == False, # noqa: E712
DedupCandidate.ignored == False, # noqa: E712
)
).scalars().all()
pruned = 0
for c in rows:
if not Path(c.delete_path).exists() or not Path(c.keep_path).exists():
c.confirmed = True
c.confirmed_by = "auto:stale"
c.confirmed_at = now
c.applied = True
pruned += 1
if pruned:
db.commit()
return pruned
def _is_pair_ignored(db, path_a: str, path_b: str) -> bool:
"""True if this path pair was ever marked 'keep both', regardless of
which path was on the keep/delete side that time -- a later scan can
@@ -58,7 +90,9 @@ def _is_pair_already_pending(db, keep_path: str, delete_path: str) -> bool:
return db.execute(query).first() is not None
async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupRun | None:
async def _run_scan(
job_key: str, script_name: str, triggered_by: str, extra_args: tuple[str, ...] = ()
) -> DedupRun | None:
"""Returns None (persisting nothing) if the job never actually ran --
e.g. skipped_lock because something else was using the pipeline lock
at that moment. Same lesson as genre_review_service.run(): recording a
@@ -67,7 +101,7 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
pending-candidates list isn't scoped to "latest run only"."""
script = str(settings.pipeline_dir / "lib" / script_name)
job_run, output = await pipeline_runner.run_job_capture(
job_key, [script, "--json"], triggered_by=triggered_by, timeout=_JOB_TIMEOUT_SECONDS
job_key, [script, "--json", *extra_args], triggered_by=triggered_by, timeout=_JOB_TIMEOUT_SECONDS
)
if job_run.status != "success":
return None
@@ -76,6 +110,8 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
db = SessionLocal()
try:
_prune_stale_pending(db)
now = time.time()
dedup_run = DedupRun(
started_at=job_run.started_at,
finished_at=job_run.finished_at,
@@ -91,7 +127,30 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
skipped_ignored = 0
skipped_duplicate = 0
auto_applied = 0
for c in candidates:
if c.get("auto_applied"):
# dedup-library.sh already deleted this pair unconditionally
# (identical numbered-sibling twin: same dir/name/ext, no
# tag/fuzzy ambiguity involved) -- record it pre-applied for
# audit history; it never shows up as a pending review item.
db.add(
DedupCandidate(
dedup_run_id=dedup_run.id,
pass_name=c.get("pass", "unknown"),
keep_path=c["keep_path"],
delete_path=c["delete_path"],
delete_id=c.get("delete_id"),
delete_size_bytes=c.get("delete_size_bytes"),
confirmed=True,
confirmed_by="auto:numbered_twin",
confirmed_at=now,
applied=True,
)
)
auto_applied += 1
db.flush()
continue
if _is_pair_ignored(db, c["keep_path"], c["delete_path"]):
skipped_ignored += 1
continue
@@ -116,8 +175,11 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
db.commit()
db.refresh(dedup_run)
skipped_total = skipped_ignored + skipped_duplicate
if skipped_total:
dedup_run.kept = (dedup_run.kept or 0) + skipped_total
if skipped_total or auto_applied:
if skipped_total:
dedup_run.kept = (dedup_run.kept or 0) + skipped_total
if auto_applied:
dedup_run.deleted = auto_applied
db.commit()
db.refresh(dedup_run)
return dedup_run
@@ -126,12 +188,23 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
async def scan(triggered_by: str = "manual") -> DedupRun | None:
"""Dry-run dedup-library.sh --json, persist every candidate deletion
into a fresh dedup_runs/dedup_candidates pair. Never deletes anything
-- the scheduled maintenance:dedup job also only ever calls this (no
--apply), matching the false-negative-biased dedup preference; actual
deletion only ever happens through confirm_and_apply() below."""
return await _run_scan("dedup:scan", _SCRIPT, triggered_by)
"""Dry-run dedup-library.sh --json (plus --auto-apply-numbered), persist
every candidate deletion into a fresh dedup_runs/dedup_candidates pair.
Two kinds of match need no human judgment and get deleted unconditionally
by dedup-library.sh itself (see is_numbered_twin/is_format_upgrade there):
identical numbered-sibling twins (same dir, same name, same extension,
differing only by the ".N" collision suffix), and format upgrades within
an EXACT tag match from Pass 2/3 (case-insensitive/normalized already
proved same song via identical tags, so a differing extension there is
just "better format vs. worse"). Everything else -- including Pass 4's
cross-album fuzzy matches, where a wrong auto-delete could take out a
genuinely different version (a different album pressing, a DJ-mix edit,
etc.) -- stays a dry-run candidate awaiting manual confirm_and_apply()
below; that's the false-negative-biased preference for anything with real
ambiguity. Auto-applied deletions are reported here already applied,
purely for audit visibility."""
return await _run_scan("dedup:scan", _SCRIPT, triggered_by, extra_args=("--auto-apply-numbered",))
async def scan_fuzzy(triggered_by: str = "manual") -> DedupRun | None:
+54 -5
View File
@@ -475,11 +475,60 @@ tbody tr:hover { background: rgba(167, 139, 250, 0.055); }
tbody td.actions-cell { display: flex; gap: 0.5rem; flex-wrap: wrap; }
.dedup-table { table-layout: fixed; }
.dedup-table .path-cell {
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
max-width: 0; /* forces the cell to respect the colgroup width instead of the content's natural width */
/* Each candidate is its own <tbody> (title row + detail row) so the pair
highlights together on hover, and so :hover can bubble from either row
up to the shared group without JS. That means every detail row is now
a tbody's last-child, which would otherwise strip the separator between
one candidate and the next (the generic `tbody tr:last-child` rule above
assumed one shared tbody) -- restore it here, and only drop it for the
actual last group in the table. */
.dedup-table .dedup-row-group:hover td { background: rgba(167, 139, 250, 0.06); }
.dedup-table .dedup-detail-row td { border-bottom: 1px solid var(--border); }
.dedup-table .dedup-row-group:last-of-type .dedup-detail-row td { border-bottom: none; }
.dedup-title-row td { padding: 0.75rem 1.1rem 0.2rem; border-bottom: none; }
.dedup-detail-row td { padding-top: 0.15rem; }
.dedup-title {
font-family: var(--font-display);
font-size: 1.02rem;
font-weight: 700;
color: var(--fg);
line-height: 1.35;
overflow-wrap: anywhere;
}
.dedup-title-alt { font-weight: 400; color: var(--muted); }
.dedup-context { font-size: 0.8rem; margin-top: 0.15rem; }
/* A thin colored rail on each side echoes the keep/delete verdict itself --
green for the copy that survives, muted-pink for the one on the chopping
block -- so which side is which reads before you've even read the words. */
.dedup-side {
display: flex;
align-items: center;
gap: 0.55rem;
padding-left: 0.65rem;
border-left: 2px solid transparent;
min-height: 1.6rem;
}
.dedup-side-keep { border-left-color: var(--status-success); }
.dedup-side-delete { border-left-color: var(--status-danger); }
.dedup-side-size { font-size: 0.78rem; white-space: nowrap; flex-shrink: 0; }
/* The actual filename/path -- always visible, never truncated. Tags can be
wrong; this is the ground truth for judging whether two files are really
the same recording, so it can't be hover-only the way it was in the first
pass of this redesign. */
.dedup-side-path {
padding-left: 0.65rem;
margin-top: 0.2rem;
font-size: 0.78rem;
color: var(--muted-2);
overflow-wrap: anywhere;
line-height: 1.4;
}
.empty-row td { padding: 2.5rem 1rem; text-align: center; }
+75 -46
View File
@@ -1,5 +1,65 @@
{% extends "base.html" %}
{% block title %}Dedup — alembic{% endblock %}
{% macro candidate_rows(c, mode) %}
<tbody class="dedup-row-group">
<tr class="dedup-title-row">
<td colspan="{{ 5 if mode == 'pending' else 4 }}">
{% if c.same_title %}
<div class="dedup-title">{{ c.keep.title }}</div>
{% else %}
<div class="dedup-title">{{ c.keep.title }} <span class="dedup-title-alt">/ {{ c.delete.title }}</span></div>
{% endif %}
{% if c.same_artist and c.same_album and c.keep.album %}
<div class="dedup-context muted">{{ c.keep.artist }} · {{ c.keep.album }}</div>
{% elif c.same_artist %}
<div class="dedup-context muted">{{ c.keep.artist }}</div>
{% else %}
<div class="dedup-context muted">{{ c.keep.artist }} / {{ c.delete.artist }}</div>
{% endif %}
</td>
</tr>
<tr class="dedup-detail-row">
{% if mode == 'pending' %}
<td><input type="checkbox" name="candidate_id" value="{{ c.id }}" form="bulk-delete-form"></td>
{% endif %}
<td><span class="badge {{ 'badge-warning' if c.pass_label == 'Acoustically Similar' else 'badge-info' }}" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
<td>
<div class="dedup-side dedup-side-keep">
<span class="badge {{ c.keep.badge }}">{{ c.keep.ext or '?' }}</span>
{% if c.keep.size_bytes %}<span class="muted dedup-side-size">{{ (c.keep.size_bytes / 1024 / 1024) | round(1) }} MB</span>{% endif %}
</div>
<div class="dedup-side-path mono muted">{{ c.keep.display }}</div>
</td>
<td>
<div class="dedup-side dedup-side-delete">
<span class="badge {{ c.delete.badge }}">{{ c.delete.ext or '?' }}</span>
{% if c.delete.size_bytes %}<span class="muted dedup-side-size">{{ (c.delete.size_bytes / 1024 / 1024) | round(1) }} MB</span>{% endif %}
</div>
<div class="dedup-side-path mono muted">{{ c.delete.display }}</div>
</td>
<td class="actions-cell">
{% if mode == 'pending' %}
<form method="post" action="/dedup/confirm" class="inline"
onsubmit="return confirm('Permanently delete this file from disk?\n{{ c.delete.display }}')">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-danger">Delete</button>
</form>
<form method="post" action="/dedup/ignore" class="inline">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-ghost" title="Keep both copies and never flag this pair again">Keep both</button>
</form>
{% else %}
<form method="post" action="/dedup/unignore" class="inline">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-ghost">Un-ignore</button>
</form>
{% endif %}
</td>
</tr>
</tbody>
{% endmacro %}
{% block content %}
<div class="page-header">
<div>
@@ -18,48 +78,30 @@
{% endif %}
<h2>Pending candidates</h2>
<p class="muted" style="margin-top:-0.5rem;">Paths are relative to the library root. Hover a name for the full path.</p>
<p class="muted" style="margin-top:-0.5rem;">Song title up top so you can scan straight down the list; the actual file path is always shown underneath each side so you can check they're really the same file before confirming.</p>
<form id="bulk-delete-form" method="post" action="/dedup/confirm"
onsubmit="return confirm('Permanently delete every selected file from disk? This cannot be undone.')"></form>
<div class="table-wrap scroll">
<table class="dedup-table">
<colgroup>
<col style="width:2.2rem"><col style="width:9.5rem"><col style="width:28%">
<col style="width:28%"><col style="width:5rem"><col style="width:9rem">
<col style="width:2.2rem"><col style="width:9.5rem"><col style="width:38%">
<col style="width:38%"><col style="width:10rem">
</colgroup>
<thead>
<tr><th></th><th>Caught by</th><th>Keep</th><th>Delete</th><th>Size</th><th>Action</th></tr>
<tr><th></th><th>Caught by</th><th>Keep</th><th>Delete</th><th>Action</th></tr>
</thead>
{% for c in candidates %}
{{ candidate_rows(c, 'pending') }}
{% else %}
<tbody>
{% set pass_badge = {"File Naming": "badge-info", "Acoustically Similar": "badge-warning"} %}
{% for c in candidates %}
<tr>
<td><input type="checkbox" name="candidate_id" value="{{ c.id }}" form="bulk-delete-form"></td>
<td><span class="badge {{ pass_badge.get(c.pass_label, 'badge-info') }}" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
<td class="muted mono path-cell" title="{{ c.keep_path }}">{{ c.keep_display }}</td>
<td class="mono path-cell" title="{{ c.delete_path }}">{{ c.delete_display }}</td>
<td class="muted">{{ (c.delete_size_bytes / 1024 / 1024) | round(1) if c.delete_size_bytes else '?' }} MB</td>
<td class="actions-cell">
<form method="post" action="/dedup/confirm" class="inline"
onsubmit="return confirm('Permanently delete this file from disk?\n{{ c.delete_display }}')">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-danger">Delete</button>
</form>
<form method="post" action="/dedup/ignore" class="inline">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-ghost" title="Keep both copies and never flag this pair again">Keep both</button>
</form>
</td>
</tr>
{% else %}
<tr class="empty-row"><td colspan="6">
<tr class="empty-row"><td colspan="5">
<div class="empty-state">
<span class="sparkle" style="position:relative; display:inline-block; margin-bottom:0.5rem;"></span>
<p class="muted" style="margin:0;">No pending candidates. Run a scan.</p>
</div>
</td></tr>
{% endfor %}
</tbody>
{% endfor %}
</table>
</div>
{% if candidates %}
@@ -72,28 +114,15 @@
<div class="table-wrap scroll">
<table class="dedup-table">
<colgroup>
<col style="width:9.5rem"><col style="width:33%">
<col style="width:33%"><col style="width:5rem"><col style="width:7rem">
<col style="width:9.5rem"><col style="width:40%">
<col style="width:40%"><col style="width:7rem">
</colgroup>
<thead>
<tr><th>Caught by</th><th>Keep</th><th>Delete</th><th>Size</th><th></th></tr>
<tr><th>Caught by</th><th>Keep</th><th>Delete</th><th></th></tr>
</thead>
<tbody>
{% for c in ignored %}
<tr>
<td><span class="badge badge-muted" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
<td class="muted mono path-cell" title="{{ c.keep_path }}">{{ c.keep_display }}</td>
<td class="muted mono path-cell" title="{{ c.delete_path }}">{{ c.delete_display }}</td>
<td class="muted">{{ (c.delete_size_bytes / 1024 / 1024) | round(1) if c.delete_size_bytes else '?' }} MB</td>
<td>
<form method="post" action="/dedup/unignore" class="inline">
<input type="hidden" name="candidate_id" value="{{ c.id }}">
<button type="submit" class="btn btn-sm btn-ghost">Un-ignore</button>
</form>
</td>
</tr>
{% endfor %}
</tbody>
{% for c in ignored %}
{{ candidate_rows(c, 'ignored') }}
{% endfor %}
</table>
</div>
{% endif %}
+91 -5
View File
@@ -34,11 +34,26 @@ set -euo pipefail
APPLY=0
JSON_MODE=0
ONLY_PATHS_FILE=""
AUTO_APPLY_NUMBERED=0
while [[ $# -gt 0 ]]; do
case "$1" in
--apply) APPLY=1; shift ;;
--json) JSON_MODE=1; shift ;;
--only-paths) ONLY_PATHS_FILE="$2"; shift 2 ;;
# Delete two kinds of match unconditionally, independent of
# --apply/--only-paths, while everything else stays dry-run/log-only:
# - numbered-sibling twins (see is_numbered_twin): same dir, same name,
# same extension, just the ".N" collision suffix -- the same download
# landing twice.
# - format upgrades within an exact tag match (see is_format_upgrade):
# Pass 2/3 already proved same song via identical tags; a different
# extension there just means "better format vs. worse", never a
# different version.
# Excludes Pass 4 (cross-album fuzzy) entirely -- matching across
# different albums/directories is exactly where a different master or
# DJ-mix edit can share tags without being interchangeable, so it always
# needs a human. See dedup_review_service.scan().
--auto-apply-numbered) AUTO_APPLY_NUMBERED=1; shift ;;
*) shift ;;
esac
done
@@ -97,6 +112,44 @@ host_path() {
SEP=$'\x1c' # ASCII file separator — safe with any music metadata
# True if two paths are the SAME directory, SAME extension, and identical
# basenames except for a numeric ".N" infix sldl/beets inserts on a filename
# collision (e.g. "Title.flac" vs "Title.1.flac"). This is deliberately
# narrower than any tag-based pass: no fuzzy matching, no cross-album
# ambiguity, just the literal same download landing twice. Used to
# auto-delete regardless of which pass found the pair -- Pass 1 only fires
# when the canonical name is a beets ghost; the very common case where BOTH
# copies are already tracked in beets (so Pass 1 defers) surfaces instead via
# Pass 2/3's tag-based grouping, and still deserves the same free pass.
is_numbered_twin() {
local a="$1" b="$2"
[[ "${a%/*}" == "${b%/*}" ]] || return 1
local ba="${a##*/}" bb="${b##*/}"
local ext="${ba##*.}"
[[ "${ext,,}" == "${bb##*.}" ]] || return 1
[[ "$(echo "$ba" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" \
== "$(echo "$bb" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" ]]
}
# True if this pair is a same-song format upgrade found by an EXACT tag match
# (Pass 2 case-insensitive or Pass 3 normalized -- both require identical
# albumartist+album+title after case/punctuation folding, so "same song" is
# already proven by the pass itself) where the two files simply have
# different extensions. rank_file() always ranks FLAC ahead of any other
# format regardless of size/dirty-name/etc, so within a tag-exact group a
# cross-extension pair unconditionally means "the format-preferred copy vs.
# a lesser one" -- no judgment call. Deliberately excludes Pass 4
# (cross_album_fuzzy): that pass matches by artist+title-base ACROSS
# different albums/directories, which is exactly where a different master,
# DJ-mix edit, or reissue can share tags without being an interchangeable
# copy -- those still need a human to look at album/context before deleting
# either side.
is_format_upgrade() {
local pass="$1" keep="$2" delete="$3"
[[ "$pass" == "case_insensitive" || "$pass" == "normalized" ]] || return 1
[[ "${keep##*.}" != "${delete##*.}" ]]
}
# Rank a file: lower = better. FLAC > MP3 > other (WAV/etc.); clean filename
# > sync-conflict artifact (-DESKTOP-, (1), .1.); extended/club mix > radio
# edit at same format; then larger wins.
@@ -177,7 +230,7 @@ process_group() {
esac
local first=1
local keep_path=""
local keep_path="" keep_id="" any_auto=0
while IFS="$SEP" read -r _sk id hp; do
[[ -z "$hp" ]] && continue
local size_bytes; size_bytes=$(stat -c '%s' "$hp" 2>/dev/null || echo 0)
@@ -185,14 +238,31 @@ process_group() {
if [[ $first -eq 1 ]]; then
log " KEEP ($size_h) $hp"
keep_path="$hp"
keep_id="$id"
first=0
else
log " DELETE ($size_h) $hp"
json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes}}"
# Auto-apply-numbered deletes numbered_sibling matches unconditionally,
# bypassing --only-paths (that gate exists only for the manual confirm
# flow, which never runs alongside this flag). All other passes stay
# dry-run/log-only here regardless.
local do_delete=0 auto_applied=0
if [[ $APPLY -eq 1 ]]; then
do_delete=1
elif [[ $AUTO_APPLY_NUMBERED -eq 1 ]] \
&& { is_numbered_twin "$keep_path" "$hp" || is_format_upgrade "$pass_name" "$keep_path" "$hp"; }; then
do_delete=1
auto_applied=1
any_auto=1
fi
local auto_json="false"; [[ $auto_applied -eq 1 ]] && auto_json="true"
json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes},\"auto_applied\":${auto_json}}"
if [[ $do_delete -eq 1 ]]; then
# With --only-paths given, only delete entries the caller explicitly
# confirmed; without it, delete everything (original behavior).
if [[ -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
if [[ $auto_applied -eq 0 && -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
log " (skipped — not in --only-paths confirm list)"
elif [[ -n "$id" ]]; then
# In beets — beet remove -d removes from DB + disk. Best-effort: a
@@ -206,6 +276,18 @@ process_group() {
fi
fi
done < <(echo "$ranked" | grep -v '^$' | sort -n)
# An auto-applied numbered-twin deletion can leave the numbered-named file
# as the sole survivor (it won on size despite the dirty-filename penalty).
# Rename it back to the canonical name so the library doesn't accumulate
# ".1."/".2." filenames for tracks that no longer have a duplicate. Only
# auto_applied triggers this (never a manual --only-paths confirm, which
# may leave the sibling undeleted if the user didn't confirm it -- renaming
# then would collide); only when it's actually in beets (a ghost survivor
# has nothing to move); only when the name still has the dirty infix.
if [[ $any_auto -eq 1 && -n "$keep_id" && "$keep_path" =~ \.[0-9]+\.[A-Za-z0-9]+$ ]]; then
beet move "id:${keep_id}" >> "$LOG" 2>&1 || log " WARN: beet move id:${keep_id} failed (rename survivor)"
fi
return 0 # explicit: the loop's last command status must not leak out under set -e
}
@@ -248,16 +330,20 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
# Canonical wins: delete numbered (in beets), keep canonical (ghost)
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
"$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
if [[ $APPLY -eq 1 ]]; then
if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
# canonical is now a ghost file; import it into beets
beet import -q -s "${canonical_cp}" >> "$LOG" 2>&1 || log " WARN: beet import failed for ${canonical_cp}"
fi
else
# Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
# the numbered file to the canonical path (id query — see process_group).
# Safe to reach with just AUTO_APPLY_NUMBERED: this loop only ever builds
# genuine numbered-sibling pairs (see is_numbered_twin), and process_group
# above already deleted the ghost canonical before this runs, so the move
# target is guaranteed clear -- no rename collision.
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
"${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
if [[ $APPLY -eq 1 ]]; then
if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log " WARN: beet move id:${numbered_id} failed"
fi
fi