0.6.8: Pin sldl skip index + fix dedup scan blinded by the path rewrite
Two bugs conspired to fill the library with .1.flac duplicates on the night of 2026-07-15/16: 1. sldl derives its skip-index folder from the input: the Spotify playlist display name before 0.6.2, the CSV filename after. The switch stranded every playlist's download history in the old emoji-named subfolder, so sldl started from an empty index and re-downloaded every playlist in full. Fix: pin index-path in _template.conf, seed the pinned file from all stranded indexes (merge-sldl-indexes.py, called from run-playlist.sh whenever the pinned index is missing or older than a stranded one). Recovered rows keep the CSV-era identity fields but are marked downloaded, so tracks that failed last night yet exist in the library since months ago are not fetched again. 2. The safety net that should have flagged the flood, dedup-library.sh, has silently reported 0 groups since the 2026-07-08 path rewrite: host_path() still assumed /music/... display paths and prepended the library dir to already-correct /data/music/Library/... paths, so every candidate failed the -f check. Fix: pass real paths through unchanged, keep the /music prefix swap only for legacy entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
andrew
@@ -78,8 +78,22 @@ fi
|
||||
# Counters: tracked via log grep at the end (avoids bash subshell pitfalls).
|
||||
# Functions write KEEP/DELETE lines with consistent prefixes; summary greps them.
|
||||
|
||||
# Convert container path (/music/...) to host path (${MUSIC_DATA_DIR:-/data/music}/Library/...)
|
||||
host_path() { echo "${MUSIC_DATA_DIR:-/data/music}/Library${1#/music}"; }
|
||||
# Resolve a beets-displayed path to a real path in this container. Since the
|
||||
# 2026-07-08 path rewrite, beets displays ${MUSIC_DATA_DIR:-/data/music}/Library/...
|
||||
# paths that are directly usable — pass those through UNCHANGED. Only legacy
|
||||
# /music/... display paths (the pre-rewrite transitional mount) still need the
|
||||
# prefix swap. Blindly prepending the library dir to an already-correct path
|
||||
# (the old behavior) produced /data/music/Library/data/music/Library/... —
|
||||
# nonexistent, so every candidate failed the -f check and every pass reported
|
||||
# 0 groups from 2026-07-08 until this fix.
|
||||
host_path() {
|
||||
local p="$1"
|
||||
if [[ "$p" == /music/* ]]; then
|
||||
echo "${MUSIC_DATA_DIR:-/data/music}/Library${p#/music}"
|
||||
else
|
||||
echo "$p"
|
||||
fi
|
||||
}
|
||||
|
||||
SEP=$'\x1c' # ASCII file separator — safe with any music metadata
|
||||
|
||||
@@ -202,9 +216,10 @@ process_group() {
|
||||
log ""
|
||||
log "[$(date -Iseconds)] === Pass 1: numbered siblings ==="
|
||||
|
||||
# Dump id + container path from beets once. beets normalizes $path for
|
||||
# display (always /music/-prefixed) even though the DB stores a mix of
|
||||
# absolute and relative — so the display paths are safe to compare/convert,
|
||||
# Dump id + display path from beets once. beets normalizes $path for display
|
||||
# (real /data/music/Library/... paths since the 2026-07-08 rewrite; /music/...
|
||||
# on any legacy entry) even though the DB stores a mix of absolute and
|
||||
# relative — so the display paths are safe to compare/convert via host_path(),
|
||||
# and the ids are what we hand to beet remove/move.
|
||||
beet ls -f "\$id${SEP}\$path" 2>/dev/null > /tmp/beets-id-paths.txt
|
||||
cut -d"$SEP" -f2- /tmp/beets-id-paths.txt > /tmp/beets-all-paths.txt
|
||||
@@ -231,7 +246,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
|
||||
|
||||
if [[ $score_canonical -le $score_numbered ]]; then
|
||||
# Canonical wins: delete numbered (in beets), keep canonical (ghost)
|
||||
process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
|
||||
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
|
||||
"$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
|
||||
if [[ $APPLY -eq 1 ]]; then
|
||||
# canonical is now a ghost file; import it into beets
|
||||
@@ -240,7 +255,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
|
||||
else
|
||||
# Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
|
||||
# the numbered file to the canonical path (id query — see process_group).
|
||||
process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
|
||||
process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
|
||||
"${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
|
||||
if [[ $APPLY -eq 1 ]]; then
|
||||
beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log " WARN: beet move id:${numbered_id} failed"
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fold every sldl skip index under a playlist's dropbox into one pinned file.
|
||||
|
||||
sldl names its per-playlist index folder after the *input*: the Spotify
|
||||
playlist's display name for spotify input, the CSV filename stem for csv
|
||||
input. So when 0.6.2 switched input from Spotify URL to CSV, sldl started a
|
||||
fresh empty index in a new subfolder, saw no download history, and
|
||||
re-downloaded every playlist in full (2026-07-16). The rendered confs now pin
|
||||
index-path to <dropbox>/<playlist>/_index.csv; this script seeds that pinned
|
||||
file from all the indexes sldl left behind (run-playlist.sh calls it whenever
|
||||
the pinned file is missing or older than a stranded one).
|
||||
|
||||
Usage: merge-sldl-indexes.py <playlist_dropbox_dir> <output_index_csv>
|
||||
|
||||
Merge rules, per (artist, title) lowercased:
|
||||
- the row from the newest index file wins;
|
||||
- EXCEPT when that row is a failure (state 2) and any older index has the
|
||||
track as downloaded (state 1 or 3): then the newest row's identity
|
||||
(artist/album/title/length -- matching what the current input will
|
||||
present; Spotify length rounding drifted between the old extractor and
|
||||
our CSV) is kept but marked state 3 (already downloaded), so sldl does
|
||||
not re-fetch a track the library already holds;
|
||||
- rows only present in older indexes are kept as-is (tracks since removed
|
||||
from the playlist; harmless, and they keep their history if re-added).
|
||||
"""
|
||||
|
||||
import csv
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HEADER = ["filepath", "artist", "album", "title", "length", "tracktype", "state", "failurereason"]
|
||||
DOWNLOADED_STATES = {"1", "3"} # 1 = downloaded this run, 3 = found in index previously
|
||||
FAILED_STATE = "2"
|
||||
|
||||
|
||||
def read_rows(path: Path) -> list[dict]:
|
||||
rows = []
|
||||
with open(path, newline="", encoding="utf-8") as fh:
|
||||
reader = csv.DictReader(fh)
|
||||
for row in reader:
|
||||
if row.get("artist") is None or row.get("title") is None:
|
||||
continue
|
||||
rows.append({k: (row.get(k) or "") for k in HEADER})
|
||||
return rows
|
||||
|
||||
|
||||
def norm_key(row: dict) -> tuple:
|
||||
# Primary artist only: sldl's old Spotify extractor recorded just the
|
||||
# first artist, while spotify-playlist-csv.py joins all of them with
|
||||
# ", " -- keying on the full string would miss every multi-artist track
|
||||
# when recovering history across the two index generations. First
|
||||
# comma-segment matches both forms (and both sides of a comma-in-name
|
||||
# artist like "Tyler, The Creator" truncate identically).
|
||||
return (row["artist"].split(",")[0].strip().lower(), row["title"].strip().lower())
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) != 3:
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
dropbox = Path(sys.argv[1])
|
||||
out_path = Path(sys.argv[2])
|
||||
if not dropbox.is_dir():
|
||||
print(f"[merge-sldl-indexes] not a directory: {dropbox}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
# Every index at the dropbox root or one level down (sldl's input-named
|
||||
# subfolders), including the pinned output itself if it already exists --
|
||||
# newest first, so the most recent record of each track wins.
|
||||
candidates = sorted(
|
||||
set(dropbox.glob("_index.csv")) | set(dropbox.glob("*/_index.csv")),
|
||||
key=lambda p: p.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
if not candidates:
|
||||
print(f"[merge-sldl-indexes] no _index.csv found under {dropbox}; nothing to seed")
|
||||
return 0
|
||||
|
||||
merged: dict[tuple, dict] = {}
|
||||
recovered = 0
|
||||
for path in candidates:
|
||||
try:
|
||||
rows = read_rows(path)
|
||||
except (OSError, csv.Error) as exc:
|
||||
print(f"[merge-sldl-indexes] skipping unreadable {path}: {exc}", file=sys.stderr)
|
||||
continue
|
||||
print(f"[merge-sldl-indexes] {path}: {len(rows)} rows")
|
||||
for row in rows:
|
||||
key = norm_key(row)
|
||||
kept = merged.get(key)
|
||||
if kept is None:
|
||||
merged[key] = row
|
||||
elif kept["state"] == FAILED_STATE and row["state"] in DOWNLOADED_STATES:
|
||||
# Newest attempt failed but an older index proves we already
|
||||
# have this track: keep the newest identity fields, take the
|
||||
# old filepath (informational only; skip-mode index never
|
||||
# checks the file on disk), and mark it downloaded.
|
||||
kept["filepath"] = row["filepath"]
|
||||
kept["state"] = "3"
|
||||
kept["failurereason"] = "0"
|
||||
recovered += 1
|
||||
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = out_path.with_suffix(".csv.tmp")
|
||||
with open(tmp, "w", newline="", encoding="utf-8") as fh:
|
||||
writer = csv.DictWriter(fh, fieldnames=HEADER)
|
||||
writer.writeheader()
|
||||
writer.writerows(merged.values())
|
||||
tmp.replace(out_path)
|
||||
|
||||
downloaded = sum(1 for r in merged.values() if r["state"] in DOWNLOADED_STATES)
|
||||
print(
|
||||
f"[merge-sldl-indexes] wrote {out_path}: {len(merged)} tracks "
|
||||
f"({downloaded} downloaded, {recovered} recovered from older indexes)"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user