0.6.9: Finish the Stage 4 path revisit in the three remaining scripts

The 2026-07-08 path rewrite changed beets' displayed paths from the
transitional /music/... mount to real /data/music/Library/... paths.
dedup-library.sh was fixed in 0.6.8; three more scripts still assumed
the old prefix, each failing silently:

- replace-with-better.sh resolved every library match to a doubled
  nonexistent path, logged [stale-db], skipped the in-place replace,
  and then imported the staged FLAC as a NEW track via the
  leftover-import step. Net effect: the mp3-to-flac upgrade flow
  produced flac+mp3 twin pairs in the library (surfacing in the dedup
  queue) instead of replacing the MP3 in place.

- fix-track-metadata.py queried beets only by the legacy /music/...
  form, matched nothing, and silently skipped beet update/move after
  retagging.

- export-laptop-playlists.py filtered out every library track (no
  displayed path starts with /music/ anymore), producing empty
  exports.

All three now use the real displayed path and keep the /music/... form
only as a legacy fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

@@ -73,10 +73,10 @@ Optional, add these later if you want them:
 Pull the prebuilt image onto your Docker host:
 
 ```bash
-docker pull git.kretzer.club/andrew/alembic:0.6.6
+docker pull git.kretzer.club/andrew/alembic:0.6.9
 ```
 
-That is the whole install. You do not need to download the source or build anything. The `0.6.6` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
+That is the whole install. You do not need to download the source or build anything. The `0.6.9` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
 
 (If you would rather build it yourself from source, you can, but you do not need to.)
 
@@ -136,7 +136,7 @@ Create a file called `docker-compose.yml` on your server (put it wherever you ke
 ```yaml
 services:
   alembic:
-    image: git.kretzer.club/andrew/alembic:0.6.6
+    image: git.kretzer.club/andrew/alembic:0.6.9
     container_name: alembic
     ports:
       - "8420:8420"

pipeline/bin/run-playlist.sh

@@ -96,6 +96,21 @@ else
   log "Fetched $TRACK_COUNT track(s) from Spotify"
 fi
 
+# ==== Seed the pinned sldl skip index if it's missing or stranded ====
+# The conf pins index-path to $DROPBOX/_index.csv (see _template.conf: sldl's
+# default index location is derived from the input, so an input change strands
+# the old index and the whole playlist re-downloads -- that's what produced
+# the 2026-07-16 duplicate flood). If the pinned file is missing, or an index
+# in one of sldl's old input-named subfolders is newer (i.e. sldl last wrote
+# somewhere else), fold them all into the pinned location first.
+INDEX_FILE="$DROPBOX/_index.csv"
+if [[ ! -s "$INDEX_FILE" ]] || \
+   [[ -n "$(find "$DROPBOX" -mindepth 2 -maxdepth 2 -name _index.csv -newer "$INDEX_FILE" -print -quit 2>/dev/null)" ]]; then
+  log "Seeding pinned sldl index at $INDEX_FILE from prior indexes"
+  python3 "${PIPELINE_DIR:-/app/pipeline}/lib/merge-sldl-indexes.py" "$DROPBOX" "$INDEX_FILE" >> "$LOG" 2>&1 \
+    || log "WARNING: index merge failed -- sldl may re-download tracks the library already has"
+fi
+
 # ==== Run sldl (vendored binary, subprocess of this container) ====
 log "Running sldl for: $PLAYLIST_NAME"
 

pipeline/configs/_template.conf

@@ -31,6 +31,18 @@ path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME
 playlist-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_sldl.m3u8
 write-playlist = true
 
+# ==== Skip index (pinned!) ====
+# sldl's already-downloaded index defaults to {path}/{playlist-name}/_index.csv
+# where {playlist-name} is derived from the INPUT: the Spotify playlist's
+# display name for spotify input, the CSV filename for csv input. beets moves
+# every download out of the dropbox, so this index is the ONLY thing standing
+# between a nightly run and re-downloading the whole playlist. When 0.6.2
+# switched input from Spotify URL to CSV, the derived name changed, sldl
+# started a fresh empty index in a new subfolder, and every playlist
+# re-downloaded in full on 2026-07-16. Pinning the path here decouples the
+# index from input naming so that can never happen again.
+index-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_index.csv
+
 # ==== Naming ====
 name-format = {artist} - {title}
 

pipeline/lib/dedup-library.sh

@@ -78,8 +78,22 @@ fi
 # Counters: tracked via log grep at the end (avoids bash subshell pitfalls).
 # Functions write KEEP/DELETE lines with consistent prefixes; summary greps them.
 
-# Convert container path (/music/...) to host path (${MUSIC_DATA_DIR:-/data/music}/Library/...)
-host_path() { echo "${MUSIC_DATA_DIR:-/data/music}/Library${1#/music}"; }
+# Resolve a beets-displayed path to a real path in this container. Since the
+# 2026-07-08 path rewrite, beets displays ${MUSIC_DATA_DIR:-/data/music}/Library/...
+# paths that are directly usable — pass those through UNCHANGED. Only legacy
+# /music/... display paths (the pre-rewrite transitional mount) still need the
+# prefix swap. Blindly prepending the library dir to an already-correct path
+# (the old behavior) produced /data/music/Library/data/music/Library/... —
+# nonexistent, so every candidate failed the -f check and every pass reported
+# 0 groups from 2026-07-08 until this fix.
+host_path() {
+  local p="$1"
+  if [[ "$p" == /music/* ]]; then
+    echo "${MUSIC_DATA_DIR:-/data/music}/Library${p#/music}"
+  else
+    echo "$p"
+  fi
+}
 
 SEP=$'\x1c'  # ASCII file separator — safe with any music metadata
 
@@ -202,9 +216,10 @@ process_group() {
 log ""
 log "[$(date -Iseconds)] === Pass 1: numbered siblings ==="
 
-# Dump id + container path from beets once. beets normalizes $path for
-# display (always /music/-prefixed) even though the DB stores a mix of
-# absolute and relative — so the display paths are safe to compare/convert,
+# Dump id + display path from beets once. beets normalizes $path for display
+# (real /data/music/Library/... paths since the 2026-07-08 rewrite; /music/...
+# on any legacy entry) even though the DB stores a mix of absolute and
+# relative — so the display paths are safe to compare/convert via host_path(),
 # and the ids are what we hand to beet remove/move.
 beet ls -f "\$id${SEP}\$path" 2>/dev/null > /tmp/beets-id-paths.txt
 cut -d"$SEP" -f2- /tmp/beets-id-paths.txt > /tmp/beets-all-paths.txt
@@ -231,7 +246,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
 
   if [[ $score_canonical -le $score_numbered ]]; then
     # Canonical wins: delete numbered (in beets), keep canonical (ghost)
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
       "$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
     if [[ $APPLY -eq 1 ]]; then
       # canonical is now a ghost file; import it into beets
@@ -240,7 +255,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
   else
     # Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
     # the numbered file to the canonical path (id query — see process_group).
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
       "${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
     if [[ $APPLY -eq 1 ]]; then
       beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log "  WARN: beet move id:${numbered_id} failed"

pipeline/lib/export-laptop-playlists.py

@@ -57,11 +57,12 @@ OUT_DIR = f"{_MUSIC_DATA_DIR}/playlists-laptop"
 # Absolute Windows path to the laptop's library root.
 LIBRARY_LAPTOP_ROOT = r"C:\Music\Library"
 
-# Beets stores paths with this prefix (the container-side mount of the
-# library); strip it to get the path relative to the library root. This is
-# still "/music/" through the migration's Stage 0-3 transitional mount;
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
-BEETS_LIBRARY_PREFIX = "/music/"
+# Prefixes beets may display library paths under, tried in order: the real
+# library dir (everything since the 2026-07-08 path rewrite) and the legacy
+# transitional mount (any stale pre-rewrite entry). Strip whichever matches
+# to get the path relative to the library root; when neither matches the
+# track is outside the library and is skipped.
+BEETS_LIBRARY_PREFIXES = (f"{_MUSIC_DATA_DIR}/Library/", "/music/")
 
 M3U_EXT = ".m3u"
 # Older formats from earlier iterations of this script — swept by reconcile.
@@ -157,9 +158,12 @@ def build_beets_index() -> dict[tuple[str, str], list[tuple[str, str, str, str]]
         if len(parts) != 4:
             continue
         aa, alb, ti, path = parts
-        if not path.startswith(BEETS_LIBRARY_PREFIX):
+        rel = next(
+            (path[len(p):] for p in BEETS_LIBRARY_PREFIXES if path.startswith(p)),
+            None,
+        )
+        if rel is None:
             continue
-        rel = path[len(BEETS_LIBRARY_PREFIX):]
         idx.setdefault((_normkey(alb), _normkey(ti)), []).append((aa, alb, ti, rel))
     return idx
 

pipeline/lib/fix-track-metadata.py

@@ -49,8 +49,10 @@ _ALEMBIC_CONFIG_DIR = os.environ.get("ALEMBIC_CONFIG_DIR", "/config")
 _MUSIC_DATA_DIR = os.environ.get("MUSIC_DATA_DIR", "/data/music")
 
 LIBRARY = f"{_MUSIC_DATA_DIR}/Library"
-# Still "/music" through the migration's Stage 0-3 transitional beets mount;
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
+# Legacy prefix from the migration's transitional beets mount. Since the
+# 2026-07-08 path rewrite beets displays real LIBRARY paths, so this only
+# matters for accepting pasted /music/... paths as input and as a fallback
+# beets query form for any stale pre-rewrite DB entry.
 CONTAINER_PREFIX = "/music"
 SPOTIFY_ENV = f"{_ALEMBIC_CONFIG_DIR}/pipeline/_spotify.env"
 SPOTIFY_GENRE = f"{os.environ.get('PIPELINE_DIR', '/app/pipeline')}/lib/spotify-genre.py"
@@ -316,15 +318,19 @@ def resolve_input_path(arg):
 
 
 def beets_id_for(host_path):
-    cp = host_to_container(host_path)
-    r = subprocess.run(
-        ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
-        capture_output=True, text=True, check=True)
-    for line in r.stdout.splitlines():
-        if "|||" in line:
-            id_str, p = line.split("|||", 1)
-            if p == cp:
-                return int(id_str)
+    # Real path first (how beets displays everything since the 2026-07-08
+    # path rewrite), legacy /music/... form second for any stale entry.
+    # Querying only the legacy form (the old behavior) matched nothing after
+    # the rewrite, so retag-from-url silently skipped beet update/move.
+    for cp in (host_path, host_to_container(host_path)):
+        r = subprocess.run(
+            ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
+            capture_output=True, text=True, check=True)
+        for line in r.stdout.splitlines():
+            if "|||" in line:
+                id_str, p = line.split("|||", 1)
+                if p == cp:
+                    return int(id_str)
     return None
 
 

pipeline/lib/merge-sldl-indexes.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Fold every sldl skip index under a playlist's dropbox into one pinned file.
+
+sldl names its per-playlist index folder after the *input*: the Spotify
+playlist's display name for spotify input, the CSV filename stem for csv
+input. So when 0.6.2 switched input from Spotify URL to CSV, sldl started a
+fresh empty index in a new subfolder, saw no download history, and
+re-downloaded every playlist in full (2026-07-16). The rendered confs now pin
+index-path to <dropbox>/<playlist>/_index.csv; this script seeds that pinned
+file from all the indexes sldl left behind (run-playlist.sh calls it whenever
+the pinned file is missing or older than a stranded one).
+
+Usage: merge-sldl-indexes.py <playlist_dropbox_dir> <output_index_csv>
+
+Merge rules, per (artist, title) lowercased:
+  - the row from the newest index file wins;
+  - EXCEPT when that row is a failure (state 2) and any older index has the
+    track as downloaded (state 1 or 3): then the newest row's identity
+    (artist/album/title/length -- matching what the current input will
+    present; Spotify length rounding drifted between the old extractor and
+    our CSV) is kept but marked state 3 (already downloaded), so sldl does
+    not re-fetch a track the library already holds;
+  - rows only present in older indexes are kept as-is (tracks since removed
+    from the playlist; harmless, and they keep their history if re-added).
+"""
+
+import csv
+import sys
+from pathlib import Path
+
+HEADER = ["filepath", "artist", "album", "title", "length", "tracktype", "state", "failurereason"]
+DOWNLOADED_STATES = {"1", "3"}  # 1 = downloaded this run, 3 = found in index previously
+FAILED_STATE = "2"
+
+
+def read_rows(path: Path) -> list[dict]:
+    rows = []
+    with open(path, newline="", encoding="utf-8") as fh:
+        reader = csv.DictReader(fh)
+        for row in reader:
+            if row.get("artist") is None or row.get("title") is None:
+                continue
+            rows.append({k: (row.get(k) or "") for k in HEADER})
+    return rows
+
+
+def norm_key(row: dict) -> tuple:
+    # Primary artist only: sldl's old Spotify extractor recorded just the
+    # first artist, while spotify-playlist-csv.py joins all of them with
+    # ", " -- keying on the full string would miss every multi-artist track
+    # when recovering history across the two index generations. First
+    # comma-segment matches both forms (and both sides of a comma-in-name
+    # artist like "Tyler, The Creator" truncate identically).
+    return (row["artist"].split(",")[0].strip().lower(), row["title"].strip().lower())
+
+
+def main() -> int:
+    if len(sys.argv) != 3:
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    dropbox = Path(sys.argv[1])
+    out_path = Path(sys.argv[2])
+    if not dropbox.is_dir():
+        print(f"[merge-sldl-indexes] not a directory: {dropbox}", file=sys.stderr)
+        return 2
+
+    # Every index at the dropbox root or one level down (sldl's input-named
+    # subfolders), including the pinned output itself if it already exists --
+    # newest first, so the most recent record of each track wins.
+    candidates = sorted(
+        set(dropbox.glob("_index.csv")) | set(dropbox.glob("*/_index.csv")),
+        key=lambda p: p.stat().st_mtime,
+        reverse=True,
+    )
+    if not candidates:
+        print(f"[merge-sldl-indexes] no _index.csv found under {dropbox}; nothing to seed")
+        return 0
+
+    merged: dict[tuple, dict] = {}
+    recovered = 0
+    for path in candidates:
+        try:
+            rows = read_rows(path)
+        except (OSError, csv.Error) as exc:
+            print(f"[merge-sldl-indexes] skipping unreadable {path}: {exc}", file=sys.stderr)
+            continue
+        print(f"[merge-sldl-indexes] {path}: {len(rows)} rows")
+        for row in rows:
+            key = norm_key(row)
+            kept = merged.get(key)
+            if kept is None:
+                merged[key] = row
+            elif kept["state"] == FAILED_STATE and row["state"] in DOWNLOADED_STATES:
+                # Newest attempt failed but an older index proves we already
+                # have this track: keep the newest identity fields, take the
+                # old filepath (informational only; skip-mode index never
+                # checks the file on disk), and mark it downloaded.
+                kept["filepath"] = row["filepath"]
+                kept["state"] = "3"
+                kept["failurereason"] = "0"
+                recovered += 1
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = out_path.with_suffix(".csv.tmp")
+    with open(tmp, "w", newline="", encoding="utf-8") as fh:
+        writer = csv.DictWriter(fh, fieldnames=HEADER)
+        writer.writeheader()
+        writer.writerows(merged.values())
+    tmp.replace(out_path)
+
+    downloaded = sum(1 for r in merged.values() if r["state"] in DOWNLOADED_STATES)
+    print(
+        f"[merge-sldl-indexes] wrote {out_path}: {len(merged)} tracks "
+        f"({downloaded} downloaded, {recovered} recovered from older indexes)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pipeline/lib/pipeline-status.sh

@@ -18,8 +18,10 @@
 # swallow the whole message.
 #
 # Reports on:
-#   - Sibling service reachability (slskd, navidrome) via HTTP, not docker ps —
-#     alembic has no Docker socket access
+#   - Sibling service reachability (navidrome) via HTTP, not docker ps —
+#     alembic has no Docker socket access. slskd is deliberately NOT checked:
+#     it is not part of the pipeline (sldl is its own Soulseek client) — it
+#     runs on the host purely as the user's own file-sharing presence.
 #   - Today's runs: playlist syncs, Bandcamp sync, manual imports, dedup
 #   - Weekly maintenance freshness: strip-mb-tags, strip-watermark-art,
 #     scrub-watermark-text, clean-sldl-index, spotify-genre
@@ -188,7 +190,6 @@ dedup_today_status() {
       mark_warn "$name" "unreachable at $url"
     fi
   }
-  check_http slskd     "http://gluetun:5030/"
   check_http navidrome "http://navidrome:4533/rest/ping.view"
 
   # 2. Today's runs

pipeline/lib/replace-with-better.sh

@@ -195,7 +195,18 @@ while IFS= read -r -d '' new_file; do
   existing_id="${match%%$'\x1f'*}"
   existing_container="${match#*$'\x1f'}"
 
-  existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  # beets displays real ${MUSIC_DATA_DIR}/Library/... paths since the
+  # 2026-07-08 path rewrite -- use them as-is; only a legacy /music/...
+  # display path still needs the prefix swap. Blindly prepending (the old
+  # behavior) doubled the prefix, so every match logged [stale-db], the MP3
+  # never got replaced, and the leftover-import step at the bottom imported
+  # the staged FLAC as a NEW track -- producing exactly the flac+mp3 twins
+  # this script exists to prevent.
+  if [[ "$existing_container" == /music/* ]]; then
+    existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  else
+    existing="$existing_container"
+  fi
   if [[ ! -f "$existing" ]]; then
     echo "[stale-db]       beets has $existing_container but file missing" | tee -a "$LOG"
     continue