andrew/alembic
1
0
Fork 0
Code Pull Requests Actions Releases 18 Wiki Activity

0.6.10: Auto-delete identical numbered-sibling duplicates, no review needed

Browse Source 
Every dedup pass previously funneled through the same manual-confirm
queue, including the obviously-safe case: two files in the same
directory with the same name and extension, differing only by the
".N" collision suffix sldl/beets inserts when a filename collides.
That's not a fuzzy match or a different version -- it's the same
download landing twice -- so it doesn't need a human in the loop the
way case-insensitive tag matches or cross-album fuzzy matches do
(different masters, DJ-mix versions, etc., which still require
confirmation).

dedup-library.sh gains --auto-apply-numbered: an is_numbered_twin()
filename check (same dir, same name modulo the numeric infix, same
extension) that deletes matching pairs unconditionally, regardless of
which pass found them -- Pass 1 only fires when the canonical name is
a beets ghost; the common case where both copies are already tracked
in beets defers to Pass 2's tag-based grouping instead, and still gets
the same free pass here. When the auto-deleted pair's survivor is
itself the numbered-named file, it's renamed back to canonical so the
library doesn't accumulate ".1."/".2." names for tracks that no longer
have a duplicate.

dedup_review_service.scan() (both the manual "Scan now" button and the
nightly scheduled job) now passes this flag and records auto-applied
deletions as pre-confirmed DedupCandidate rows for audit visibility --
they never appear as pending review items. Tag-based and cross-album
fuzzy passes are unaffected and still require manual confirmation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
andrew
2026-07-22 09:34:16 -06:00
parent 62251d764e
commit f05f076464
3 changed files with 113 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipeline/lib/dedup-library.sh
+64 -5
View File
@@ -34,11 +34,19 @@ set -euo pipefail
APPLY=0
JSON_MODE=0
ONLY_PATHS_FILE=""
AUTO_APPLY_NUMBERED=0
while [[ $# -gt 0 ]]; do
case "$1" in
--apply) APPLY=1; shift ;;
--json) JSON_MODE=1; shift ;;
--only-paths) ONLY_PATHS_FILE="$2"; shift 2 ;;
# Delete Pass 1 (numbered-sibling) matches unconditionally, independent of
# --apply/--only-paths, while Passes 2-4 stay dry-run/log-only. Safe to
# run unattended: a numbered-sibling pair is the SAME title, in the SAME
# directory, with the SAME extension -- unlike the tag-based/fuzzy passes,
# there's no fuzzy matching or cross-album ambiguity to get wrong, so it
# doesn't need a human in the loop. See dedup_review_service.scan().
--auto-apply-numbered) AUTO_APPLY_NUMBERED=1; shift ;;
*) shift ;;
esac
done
@@ -97,6 +105,25 @@ host_path() {
SEP=$'\x1c' # ASCII file separator — safe with any music metadata
# True if two paths are the SAME directory, SAME extension, and identical
# basenames except for a numeric ".N" infix sldl/beets inserts on a filename
# collision (e.g. "Title.flac" vs "Title.1.flac"). This is deliberately
# narrower than any tag-based pass: no fuzzy matching, no cross-album
# ambiguity, just the literal same download landing twice. Used to
# auto-delete regardless of which pass found the pair -- Pass 1 only fires
# when the canonical name is a beets ghost; the very common case where BOTH
# copies are already tracked in beets (so Pass 1 defers) surfaces instead via
# Pass 2/3's tag-based grouping, and still deserves the same free pass.
is_numbered_twin() {
local a="$1" b="$2"
[[ "${a%/*}" == "${b%/*}" ]] || return 1
local ba="${a##*/}" bb="${b##*/}"
local ext="${ba##*.}"
[[ "${ext,,}" == "${bb##*.}" ]] || return 1
[[ "$(echo "$ba" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" \
== "$(echo "$bb" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" ]]
}
# Rank a file: lower = better. FLAC > MP3 > other (WAV/etc.); clean filename
# > sync-conflict artifact (-DESKTOP-, (1), .1.); extended/club mix > radio
# edit at same format; then larger wins.
@@ -177,7 +204,7 @@ process_group() {
esac
local first=1
local keep_path=""
local keep_path="" keep_id="" any_auto=0
while IFS="$SEP" read -r _sk id hp; do
[[ -z "$hp" ]] && continue
local size_bytes; size_bytes=$(stat -c '%s' "$hp" 2>/dev/null || echo 0)
@@ -185,14 +212,30 @@ process_group() {
if [[ $first -eq 1 ]]; then
log " KEEP ($size_h) $hp"
keep_path="$hp"
keep_id="$id"
first=0
else
log " DELETE ($size_h) $hp"
json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes}}"
# Auto-apply-numbered deletes numbered_sibling matches unconditionally,
# bypassing --only-paths (that gate exists only for the manual confirm
# flow, which never runs alongside this flag). All other passes stay
# dry-run/log-only here regardless.
local do_delete=0 auto_applied=0
if [[ $APPLY -eq 1 ]]; then
do_delete=1
elif [[ $AUTO_APPLY_NUMBERED -eq 1 ]] && is_numbered_twin "$keep_path" "$hp"; then
do_delete=1
auto_applied=1
any_auto=1
fi
local auto_json="false"; [[ $auto_applied -eq 1 ]] && auto_json="true"
json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes},\"auto_applied\":${auto_json}}"
if [[ $do_delete -eq 1 ]]; then
# With --only-paths given, only delete entries the caller explicitly
# confirmed; without it, delete everything (original behavior).
if [[ -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
if [[ $auto_applied -eq 0 && -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
log " (skipped — not in --only-paths confirm list)"
elif [[ -n "$id" ]]; then
# In beets — beet remove -d removes from DB + disk. Best-effort: a
@@ -206,6 +249,18 @@ process_group() {
fi
fi
done < <(echo "$ranked" | grep -v '^$' | sort -n)
# An auto-applied numbered-twin deletion can leave the numbered-named file
# as the sole survivor (it won on size despite the dirty-filename penalty).
# Rename it back to the canonical name so the library doesn't accumulate
# ".1."/".2." filenames for tracks that no longer have a duplicate. Only
# auto_applied triggers this (never a manual --only-paths confirm, which
# may leave the sibling undeleted if the user didn't confirm it -- renaming
# then would collide); only when it's actually in beets (a ghost survivor
# has nothing to move); only when the name still has the dirty infix.
if [[ $any_auto -eq 1 && -n "$keep_id" && "$keep_path" =~ \.[0-9]+\.[A-Za-z0-9]+$ ]]; then
beet move "id:${keep_id}" >> "$LOG" 2>&1 || log " WARN: beet move id:${keep_id} failed (rename survivor)"
fi
return 0 # explicit: the loop's last command status must not leak out under set -e
}
@@ -248,16 +303,20 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
# Canonical wins: delete numbered (in beets), keep canonical (ghost)
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
"$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
if [[ $APPLY -eq 1 ]]; then
if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
# canonical is now a ghost file; import it into beets
beet import -q -s "${canonical_cp}" >> "$LOG" 2>&1 || log " WARN: beet import failed for ${canonical_cp}"
fi
else
# Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
# the numbered file to the canonical path (id query — see process_group).
# Safe to reach with just AUTO_APPLY_NUMBERED: this loop only ever builds
# genuine numbered-sibling pairs (see is_numbered_twin), and process_group
# above already deleted the ghost canonical before this runs, so the move
# target is guaranteed clear -- no rename collision.
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
"${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
if [[ $APPLY -eq 1 ]]; then
if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log " WARN: beet move id:${numbered_id} failed"
fi
fi