andrew/alembic
1
0
Fork 0
Code Pull Requests Actions Releases 18 Wiki Activity

0.6.8: Pin sldl skip index + fix dedup scan blinded by the path rewrite

Browse Source 
Two bugs conspired to fill the library with .1.flac duplicates on the
night of 2026-07-15/16:

1. sldl derives its skip-index folder from the input: the Spotify
   playlist display name before 0.6.2, the CSV filename after. The
   switch stranded every playlist's download history in the old
   emoji-named subfolder, so sldl started from an empty index and
   re-downloaded every playlist in full. Fix: pin index-path in
   _template.conf, seed the pinned file from all stranded indexes
   (merge-sldl-indexes.py, called from run-playlist.sh whenever the
   pinned index is missing or older than a stranded one). Recovered
   rows keep the CSV-era identity fields but are marked downloaded, so
   tracks that failed last night yet exist in the library since months
   ago are not fetched again.

2. The safety net that should have flagged the flood, dedup-library.sh,
   has silently reported 0 groups since the 2026-07-08 path rewrite:
   host_path() still assumed /music/... display paths and prepended the
   library dir to already-correct /data/music/Library/... paths, so
   every candidate failed the -f check. Fix: pass real paths through
   unchanged, keep the /music prefix swap only for legacy entries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
andrew
2026-07-16 09:23:53 -06:00
parent 24738d815f
commit d756aab7fc
5 changed files with 173 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pipeline/bin/run-playlist.sh
+15
View File
@@ -96,6 +96,21 @@ else
log "Fetched $TRACK_COUNT track(s) from Spotify"
fi
# ==== Seed the pinned sldl skip index if it's missing or stranded ====
# The conf pins index-path to $DROPBOX/_index.csv (see _template.conf: sldl's
# default index location is derived from the input, so an input change strands
# the old index and the whole playlist re-downloads -- that's what produced
# the 2026-07-16 duplicate flood). If the pinned file is missing, or an index
# in one of sldl's old input-named subfolders is newer (i.e. sldl last wrote
# somewhere else), fold them all into the pinned location first.
INDEX_FILE="$DROPBOX/_index.csv"
if [[ ! -s "$INDEX_FILE" ]] || \
[[ -n "$(find "$DROPBOX" -mindepth 2 -maxdepth 2 -name _index.csv -newer "$INDEX_FILE" -print -quit 2>/dev/null)" ]]; then
log "Seeding pinned sldl index at $INDEX_FILE from prior indexes"
python3 "${PIPELINE_DIR:-/app/pipeline}/lib/merge-sldl-indexes.py" "$DROPBOX" "$INDEX_FILE" >> "$LOG" 2>&1 \
|| log "WARNING: index merge failed -- sldl may re-download tracks the library already has"
fi
# ==== Run sldl (vendored binary, subprocess of this container) ====
log "Running sldl for: $PLAYLIST_NAME"