0.6.14: Fix Bandcamp sync silently failing to import new purchases

sync-bandcamp.sh set a hardcoded PATH at startup that left out /opt/venv/bin,
which is where beet lives in this image. Every time the Bandcamp sync had a
new purchase to import, its call into import-track.sh would run beet import
with that broken PATH, fail with "command not found", and import-track.sh
would just log the exit code and carry on, so sync-bandcamp.sh still reported
success. The purchase sat on disk, never entered the beets library, and never
showed up in Navidrome. This went unnoticed for weeks because most daily
syncs have nothing new to import, so the broken code path rarely ran.

Fixed the same copy-pasted PATH line in upgrade-mp3-to-flac.sh (which also
calls beet directly) and notify-telegram.sh (harmless there, but fixed for
consistency).

Also moved the post-import duplicate cleanup (replace-with-better.sh and
dedup-library.sh) out of sync-bandcamp.sh and into import-track.sh itself, so
every import gets the same cleanup, not just Bandcamp purchases. A manual
import or SMB drop that happens to match something already in the library no
longer leaves a duplicate copy sitting there until someone runs the dedup
review by hand.

README.md

@@ -73,10 +73,10 @@ Optional, add these later if you want them:
 Pull the prebuilt image onto your Docker host:
 
 ```bash
-docker pull git.kretzer.club/andrew/alembic:0.6
+docker pull git.kretzer.club/andrew/alembic:0.6.14
 ```
 
-That is the whole install. You do not need to download the source or build anything. The `0.6` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
+That is the whole install. You do not need to download the source or build anything. The `0.6.14` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
 
 (If you would rather build it yourself from source, you can, but you do not need to.)
 
@@ -136,7 +136,7 @@ Create a file called `docker-compose.yml` on your server (put it wherever you ke
 ```yaml
 services:
   alembic:
-    image: git.kretzer.club/andrew/alembic:0.6
+    image: git.kretzer.club/andrew/alembic:0.6.14
     container_name: alembic
     ports:
       - "8420:8420"
@@ -211,9 +211,11 @@ Steps:
 
 That's it, done once. alembic stores the resulting OAuth refresh token (encrypted, same as your other credentials) and uses it to silently mint a fresh access token whenever it needs one — you never have to repeat this unless you disconnect or revoke access on Spotify's side.
 
+One more limit that comes with a new Spotify app: Spotify only returns a playlist's contents to the account that owns it (or collaborates on it), even after you connect. So sync playlists that belong to the account you connect, and if someone shares a playlist with you, save your own copy of it first (or have them add you as a collaborator).
+
 If you happen to be running a Spotify app created *before* the 2025/2026 change, plain Client ID/Secret still works and this step is optional — but connecting is harmless either way, so there's no reason not to.
 
-> **If you connected Spotify before version 0.6.1:** click **Connect Spotify** again. Versions before 0.6.1 requested a narrower OAuth scope than sldl actually needs, which could get you a token that refreshes fine but then gets 403 Forbidden loading a playlist (see Troubleshooting). Reconnecting grants the correct scope; your old connection doesn't upgrade itself.
+> **If you connected Spotify before version 0.6.1:** click **Connect Spotify** again. Versions before 0.6.1 requested a narrower OAuth scope than playlist reads actually need, which could get you a token that refreshes fine but then gets 403 Forbidden loading a playlist (see Troubleshooting). Reconnecting grants the correct scope; your old connection doesn't upgrade itself.
 
 ## Adding your first playlist
 
@@ -290,11 +292,13 @@ Your Spotify app can't read the playlist with app-only (client-credentials) acce
 
 The fix: go to **Settings → Credentials** and click **Connect Spotify** to complete the OAuth login (see "Connecting your Spotify account (OAuth)" above). This mints a user token that works regardless of when your app was created. If you haven't already, add the redirect URI `https://<your-host>/connect/spotify/callback` in your Spotify app's dashboard first, or the connect step itself will fail with a redirect_uri mismatch.
 
-**A playlist's job log shows `sldl crashed (exit 134)` mentioning a 403 Forbidden, even though you've connected Spotify.**
+**A playlist's job log shows `Spotify returned 403 loading this playlist`, `Spotify returned this playlist without its contents`, or `0 visible tracks` even though you've connected Spotify.** (On versions before 0.6.2 these showed up as `sldl crashed (exit 134)` mentioning a 403 Forbidden; on 0.6.2 a playlist you don't own could finish "successfully" while downloading nothing.)
-Two possible causes, in order of likelihood:
+Possible causes, in order of likelihood:
 
-1. **You connected before version 0.6.1.** Earlier versions requested a narrower OAuth scope than sldl actually needs, so the token refreshes fine but then gets 403 loading a playlist regardless of ownership. Fix: go to **Settings → Credentials** and click **Connect Spotify** again to grant the correct scope.
+1. **The playlist isn't owned by the connected account.** For Spotify apps created after the 2025/2026 API change, Spotify only returns a playlist's contents to its owner or a collaborator on it. Being public is NOT enough. Fix: recreate the playlist under the account you connected via **Connect Spotify**, or have the owner make it collaborative and add you.
-2. **The connected account genuinely can't see this playlist.** Once connected, playlist reads are scoped to what that account can actually see — its own playlists, ones it collaborates on, or ones marked Public — not any arbitrary playlist by ID the way app-only access could. Fix: make sure the playlist is owned by the connected account, or set it to Public on Spotify (Playlist → ⋯ → Make public).
+2. **You connected before version 0.6.1.** Earlier versions requested a narrower OAuth scope than playlist reads actually need, so the token refreshes fine but then gets 403 loading a playlist regardless of ownership. Fix: go to **Settings → Credentials** and click **Connect Spotify** again to grant the correct scope.
+
+(If your Spotify app is old enough to be grandfathered, both public playlists and your own keep working like before.)
 
 Either way, re-run the playlist after fixing it.
 

app/main.py

@@ -8,7 +8,7 @@ from fastapi.templating import Jinja2Templates
 from starlette.exceptions import HTTPException as StarletteHTTPException
 from starlette.middleware.sessions import SessionMiddleware
 
-from app.db import enable_beets_db_wal, init_db, mark_interrupted_runs
+from app.db import SessionLocal, enable_beets_db_wal, init_db, mark_interrupted_runs
 from app.routers import artist_casing, auth, connect, credentials, dashboard, dedup, genres, health, import_, jobs, library, playlists
 from app.security.session import resolve_session_secret
 from app.services import scheduler_service
@@ -48,12 +48,31 @@ def _warn_if_key_colocated_with_config() -> None:
         )
 
 
+def _render_confs_on_startup() -> None:
+    """Re-render every playlist .conf from the current template once per boot,
+    so confs rendered by an older version converge on upgrade without waiting
+    for a playlist or credential change. Concretely: 0.6.2 dropped the Spotify
+    credential lines from confs (sldl no longer talks to Spotify), and this is
+    what scrubs those secrets from already-deployed confs. Best-effort -- a
+    render failure shouldn't stop the app from starting."""
+    from app.services import credential_service
+
+    db = SessionLocal()
+    try:
+        credential_service.render_playlist_confs(db)
+    except Exception:
+        log.exception("Startup playlist .conf render failed; continuing")
+    finally:
+        db.close()
+
+
 @asynccontextmanager
 async def lifespan(_app: FastAPI):
     _warn_if_key_colocated_with_config()
     init_db()
     mark_interrupted_runs()
     enable_beets_db_wal()
+    _render_confs_on_startup()
 
     scheduler = scheduler_service.create_scheduler()
     scheduler_service.register_all_jobs(scheduler)

app/routers/dedup.py

@@ -1,3 +1,6 @@
+import os
+import re
+
 from fastapi import APIRouter, BackgroundTasks, Depends, Request
 from fastapi.responses import RedirectResponse
 from fastapi.templating import Jinja2Templates
@@ -30,16 +33,76 @@ def _display_path(path: str) -> str:
     return path[len(_LIBRARY_PREFIX):] if path.startswith(_LIBRARY_PREFIX) else path
 
 
+# Compilation-style import paths carry a "$track " prefix on the filename
+# (see pipeline/configs/beets config.yaml paths: comp/albumtype_soundtrack);
+# singleton paths (the vast majority of this library) don't. Strip it either
+# way so the title reads clean.
+_TRACK_PREFIX_RE = re.compile(r"^\d{1,3}[\s.\-]+")
+
+# rank_file() in dedup-library.sh always ranks FLAC above every other format
+# and WAV below MP3 (a download glitch, never desirable) -- mirror that
+# judgment in the format pill's color so it reads as "this one's the keeper"
+# at a glance, not just a bare extension string.
+_FORMAT_BADGE = {"flac": "badge-success", "wav": "badge-warning"}
+
+
+def _split_display(path: str) -> dict:
+    """Break a library display path into (artist, album, title, ext).
+
+    Beets' singleton path template is always
+    %the{$albumartist}/$album/$title (pipeline/configs/beets config.yaml,
+    paths:) -- every track in this library lands at exactly that shape, so
+    the last two segments are reliably album/filename. A path that doesn't
+    fit (an unexpected root-level file) degrades to showing the raw display
+    path as the title instead of guessing at structure that isn't there.
+    """
+    display = _display_path(path)
+    parts = display.split("/")
+    if len(parts) >= 2:
+        artist, album, filename = parts[0], parts[-2], parts[-1]
+        title, ext = os.path.splitext(filename)
+        title = _TRACK_PREFIX_RE.sub("", title)
+    else:
+        artist, album = "", ""
+        title, ext = os.path.splitext(display)
+    return {
+        "path": path,
+        "display": display,
+        "artist": artist,
+        "album": album,
+        "title": title,
+        "ext": ext.lstrip(".").lower(),
+    }
+
+
+def _file_size(path: str) -> int | None:
+    try:
+        return os.path.getsize(path)
+    except OSError:
+        return None
+
+
 def _to_row(c) -> dict:
+    keep = _split_display(c.keep_path)
+    delete = _split_display(c.delete_path)
+    keep["size_bytes"] = _file_size(c.keep_path)
+    keep["badge"] = _FORMAT_BADGE.get(keep["ext"], "badge-muted")
+    delete["size_bytes"] = c.delete_size_bytes if c.delete_size_bytes is not None else _file_size(c.delete_path)
+    delete["badge"] = _FORMAT_BADGE.get(delete["ext"], "badge-muted")
     return {
         "id": c.id,
         "pass_name": c.pass_name,
         "pass_label": _pass_label(c.pass_name),
-        "keep_path": c.keep_path,
+        "keep": keep,
-        "delete_path": c.delete_path,
+        "delete": delete,
-        "keep_display": _display_path(c.keep_path),
+        # Same-tag passes (numbered_sibling/case_insensitive/normalized) share
-        "delete_display": _display_path(c.delete_path),
+        # artist/title almost always; cross_album_fuzzy and fuzzy_audio can
-        "delete_size_bytes": c.delete_size_bytes,
+        # legitimately differ (different credit, different album entirely) --
+        # the template only collapses shared context onto one line when it's
+        # actually shared, otherwise shows both sides in full.
+        "same_title": keep["title"].lower() == delete["title"].lower(),
+        "same_artist": keep["artist"].lower() == delete["artist"].lower(),
+        "same_album": keep["album"].lower() == delete["album"].lower(),
     }
 
 

app/routers/playlists.py

@@ -14,12 +14,16 @@ templates = Jinja2Templates(directory="app/templates")
 def _friendly_status_error(exc: Exception) -> str:
     """Turn a raw Spotify API error into something a non-technical user can act
     on. 403 here almost always means the Spotify app can't read the playlist:
-    either the credentials are wrong or the playlist isn't public."""
+    wrong credentials, or the connected account can't see it. (The other
+    can't-read case, contents hidden for playlists the account doesn't own,
+    arrives as a RuntimeError from spotify_client with its own message.)"""
     if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code == 403:
         return (
             "Spotify denied access (403). Check your Spotify credentials under "
-            "Settings then Credentials, and make sure the playlist is set to public "
+            "Settings then Credentials, and make sure the account connected via "
-            "on Spotify -- alembic can't read private playlists."
+            "Connect Spotify can see this playlist. For newly created Spotify "
+            "apps the connected account must own the playlist or be a "
+            "collaborator on it."
         )
     return str(exc)
 

app/services/credential_service.py

@@ -116,8 +116,10 @@ def get_scope(db: Session, scope: str) -> dict[str, str]:
 _SAFE_CONF_NAME = re.compile(r"^[A-Za-z0-9 _-]{1,64}$")
 
 # The credential lines the renderer fills in from the encrypted store. Everything
-# else in a rendered .conf comes verbatim from _template.conf.
+# else in a rendered .conf comes verbatim from _template.conf. Spotify creds are
-_CONF_CRED_KEYS = ("user", "pass", "spotify-id", "spotify-secret", "spotify-refresh")
+# NOT here: sldl never talks to Spotify itself (see _template.conf and
+# run-playlist.sh) -- the scripts that do read them from _spotify.env instead.
+_CONF_CRED_KEYS = ("user", "pass")
 
 
 def _set_conf_field(text: str, key: str, value: str) -> str:
@@ -134,27 +136,24 @@ def _set_conf_field(text: str, key: str, value: str) -> str:
 
 def render_playlist_confs(db: Session) -> None:
     """Render every playlist's sldl .conf directly from _template.conf, with
-    the path placeholders and the Soulseek/Spotify credentials substituted in
+    the path placeholders and the Soulseek credentials substituted in one
-    one pass, written 0600.
+    pass, written 0600. No Spotify credentials here -- sldl never talks to
+    Spotify itself (see _template.conf); those live only in _spotify.env.
 
     This is the single source of .conf rendering. It replaces the older
     DB -> playlists.json -> regen.sh -> regex-patch-back chain: the app owns
     every input (playlists in its DB, credentials in its encrypted store), so
     there is no reason to round-trip through an intermediate file and a
     subprocess. Called on any playlist change (playlist_service) and on any
-    Spotify/Soulseek credential change (render_scope)."""
+    Soulseek credential change (render_scope)."""
     from app.services import playlist_service
 
     template = (settings.pipeline_dir / "configs" / "_template.conf").read_text()
     dropbox_root = str(settings.music_data_dir / "sldl-dropbox")
-    spotify = get_scope(db, "spotify")
     soulseek = get_scope(db, "soulseek")
     creds = {
         "user": soulseek.get("username", ""),
         "pass": soulseek.get("password", ""),
-        "spotify-id": spotify.get("client_id", ""),
-        "spotify-secret": spotify.get("client_secret", ""),
-        "spotify-refresh": spotify.get("refresh_token", ""),
     }
     # Path placeholders are substituted as LITERAL text in a single left-to-right
     # pass (never re-scanned), so a value can't be reinterpreted as another
@@ -193,13 +192,14 @@ def render_scope(db: Session, scope: str) -> None:
     if scope == "spotify":
         cid = values.get("client_id", "")
         csec = values.get("client_secret", "")
-        # _spotify.env is read by the pipeline python scripts (spotify-genre,
+        # _spotify.env is the ONLY place Spotify creds are rendered -- sldl
-        # spotify-retag, fix-track-metadata). SPOTIFY_REFRESH_TOKEN is only set
+        # itself never talks to Spotify (see _template.conf), so playlist
-        # once an account is connected via /connect/spotify -- its presence is
+        # .confs don't need them. Read by run-playlist.sh (to generate the
-        # what switches every reader from client-credentials to a user token
+        # search CSV) and the pipeline python scripts (spotify-retag,
-        # (see _spotify_auth.get_token). The rendered playlist .conf gets the
+        # spotify-genre, fix-track-metadata). SPOTIFY_REFRESH_TOKEN is only
-        # same refresh token too (spotify-refresh, in render_playlist_confs),
+        # set once an account is connected via /connect/spotify -- its
-        # which is what lets sldl itself use a user token.
+        # presence is what switches every reader from client-credentials to
+        # a user token (see _spotify_auth.get_token).
         _write_env_file(
             settings.pipeline_config_dir / "_spotify.env",
             {
@@ -208,8 +208,6 @@ def render_scope(db: Session, scope: str) -> None:
                 "SPOTIFY_REFRESH_TOKEN": values.get("refresh_token", ""),
             },
         )
-        # Re-render every playlist .conf so the new Spotify creds land in them.
-        render_playlist_confs(db)
 
     elif scope == "soulseek":
         render_playlist_confs(db)

app/services/dedup_review_service.py

@@ -27,6 +27,38 @@ def _script_for_pass(pass_name: str) -> str:
     return _FUZZY_SCRIPT if pass_name == _FUZZY_PASS else _SCRIPT
 
 
+def _prune_stale_pending(db) -> int:
+    """Mark pending candidates whose delete_path or keep_path no longer
+    exists as resolved, instead of leaving them to surface forever.
+
+    A pending row can go stale without ever being confirmed through the UI --
+    e.g. a later scan's numbered-twin/format-upgrade auto-apply already
+    removed one side, or upgrade-mp3-to-flac replaced it, or someone deleted
+    it by hand. confirm_and_apply() already does this "already_gone" check
+    for candidates a user explicitly acts on; this generalizes it to run at
+    the start of every scan, so the pending list reflects current reality
+    instead of stale groups that something else already resolved."""
+    now = time.time()
+    rows = db.execute(
+        select(DedupCandidate).where(
+            DedupCandidate.applied == False,  # noqa: E712
+            DedupCandidate.confirmed == False,  # noqa: E712
+            DedupCandidate.ignored == False,  # noqa: E712
+        )
+    ).scalars().all()
+    pruned = 0
+    for c in rows:
+        if not Path(c.delete_path).exists() or not Path(c.keep_path).exists():
+            c.confirmed = True
+            c.confirmed_by = "auto:stale"
+            c.confirmed_at = now
+            c.applied = True
+            pruned += 1
+    if pruned:
+        db.commit()
+    return pruned
+
+
 def _is_pair_ignored(db, path_a: str, path_b: str) -> bool:
     """True if this path pair was ever marked 'keep both', regardless of
     which path was on the keep/delete side that time -- a later scan can
@@ -58,7 +90,9 @@ def _is_pair_already_pending(db, keep_path: str, delete_path: str) -> bool:
     return db.execute(query).first() is not None
 
 
-async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupRun | None:
+async def _run_scan(
+    job_key: str, script_name: str, triggered_by: str, extra_args: tuple[str, ...] = ()
+) -> DedupRun | None:
     """Returns None (persisting nothing) if the job never actually ran --
     e.g. skipped_lock because something else was using the pipeline lock
     at that moment. Same lesson as genre_review_service.run(): recording a
@@ -67,7 +101,7 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
     pending-candidates list isn't scoped to "latest run only"."""
     script = str(settings.pipeline_dir / "lib" / script_name)
     job_run, output = await pipeline_runner.run_job_capture(
-        job_key, [script, "--json"], triggered_by=triggered_by, timeout=_JOB_TIMEOUT_SECONDS
+        job_key, [script, "--json", *extra_args], triggered_by=triggered_by, timeout=_JOB_TIMEOUT_SECONDS
     )
     if job_run.status != "success":
         return None
@@ -76,6 +110,8 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
 
     db = SessionLocal()
     try:
+        _prune_stale_pending(db)
+        now = time.time()
         dedup_run = DedupRun(
             started_at=job_run.started_at,
             finished_at=job_run.finished_at,
@@ -91,7 +127,30 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
 
         skipped_ignored = 0
         skipped_duplicate = 0
+        auto_applied = 0
         for c in candidates:
+            if c.get("auto_applied"):
+                # dedup-library.sh already deleted this pair unconditionally
+                # (identical numbered-sibling twin: same dir/name/ext, no
+                # tag/fuzzy ambiguity involved) -- record it pre-applied for
+                # audit history; it never shows up as a pending review item.
+                db.add(
+                    DedupCandidate(
+                        dedup_run_id=dedup_run.id,
+                        pass_name=c.get("pass", "unknown"),
+                        keep_path=c["keep_path"],
+                        delete_path=c["delete_path"],
+                        delete_id=c.get("delete_id"),
+                        delete_size_bytes=c.get("delete_size_bytes"),
+                        confirmed=True,
+                        confirmed_by="auto:numbered_twin",
+                        confirmed_at=now,
+                        applied=True,
+                    )
+                )
+                auto_applied += 1
+                db.flush()
+                continue
             if _is_pair_ignored(db, c["keep_path"], c["delete_path"]):
                 skipped_ignored += 1
                 continue
@@ -116,8 +175,11 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
         db.commit()
         db.refresh(dedup_run)
         skipped_total = skipped_ignored + skipped_duplicate
-        if skipped_total:
+        if skipped_total or auto_applied:
-            dedup_run.kept = (dedup_run.kept or 0) + skipped_total
+            if skipped_total:
+                dedup_run.kept = (dedup_run.kept or 0) + skipped_total
+            if auto_applied:
+                dedup_run.deleted = auto_applied
             db.commit()
             db.refresh(dedup_run)
         return dedup_run
@@ -126,12 +188,23 @@ async def _run_scan(job_key: str, script_name: str, triggered_by: str) -> DedupR
 
 
 async def scan(triggered_by: str = "manual") -> DedupRun | None:
-    """Dry-run dedup-library.sh --json, persist every candidate deletion
+    """Dry-run dedup-library.sh --json (plus --auto-apply-numbered), persist
-    into a fresh dedup_runs/dedup_candidates pair. Never deletes anything
+    every candidate deletion into a fresh dedup_runs/dedup_candidates pair.
-    -- the scheduled maintenance:dedup job also only ever calls this (no
+
-    --apply), matching the false-negative-biased dedup preference; actual
+    Two kinds of match need no human judgment and get deleted unconditionally
-    deletion only ever happens through confirm_and_apply() below."""
+    by dedup-library.sh itself (see is_numbered_twin/is_format_upgrade there):
-    return await _run_scan("dedup:scan", _SCRIPT, triggered_by)
+    identical numbered-sibling twins (same dir, same name, same extension,
+    differing only by the ".N" collision suffix), and format upgrades within
+    an EXACT tag match from Pass 2/3 (case-insensitive/normalized already
+    proved same song via identical tags, so a differing extension there is
+    just "better format vs. worse"). Everything else -- including Pass 4's
+    cross-album fuzzy matches, where a wrong auto-delete could take out a
+    genuinely different version (a different album pressing, a DJ-mix edit,
+    etc.) -- stays a dry-run candidate awaiting manual confirm_and_apply()
+    below; that's the false-negative-biased preference for anything with real
+    ambiguity. Auto-applied deletions are reported here already applied,
+    purely for audit visibility."""
+    return await _run_scan("dedup:scan", _SCRIPT, triggered_by, extra_args=("--auto-apply-numbered",))
 
 
 async def scan_fuzzy(triggered_by: str = "manual") -> DedupRun | None:

app/services/pipeline_runner.py

@@ -12,6 +12,15 @@ from app.settings import settings
 # all-jobs-share-one-lock behavior exactly rather than over-engineering it.
 _lock = asyncio.Lock()
 
+# How long a SCHEDULED run waits for the lock before giving up (recorded as
+# skipped_lock). The old flock -n instant-skip starves fixed-time schedules:
+# the Sunday 08:30 strip-mb-tags run took 10m19s on 2026-07-12, so the 08:35
+# and 08:40 jobs both hit the held lock and silently lost their only slot of
+# the week. Scheduled jobs have nobody watching, so queueing (bounded) beats
+# skipping; manual runs keep the instant skip because a person clicking "Run
+# now" should get an immediate answer, not a silent 30-minute wait.
+SCHEDULED_LOCK_WAIT_SECONDS = 1800.0
+
 _ENV_PASSTHROUGH_KEYS = ("PATH", "HOME", "LANG", "LC_ALL", "TZ")
 
 
@@ -94,18 +103,30 @@ async def _execute(
     timeout: float | None,
     use_lock: bool,
     capture: bool,
+    lock_wait: float | None,
 ) -> tuple[JobRun, str]:
     """Shared core for run_job/run_job_capture: acquire the lock (unless
     use_lock=False), record the run, exec the subprocess, and finalize. When
     capture=True, stdout+stderr is captured as text and returned; otherwise it
     streams straight to the log file. Returns (JobRun, output_text) -- the text
-    is "" in the non-capture and skipped-lock cases."""
+    is "" in the non-capture and skipped-lock cases.
+
+    lock_wait: how long to wait for a held lock before recording
+    skipped_lock. None picks the policy default: SCHEDULED_LOCK_WAIT_SECONDS
+    for triggered_by="schedule", instant skip for everything else."""
     started_at = time.time()
+    if lock_wait is None:
+        lock_wait = SCHEDULED_LOCK_WAIT_SECONDS if triggered_by == "schedule" else 0.0
 
     acquired = False
     if use_lock:
         if not await _try_acquire_nowait():
-            return _record_run(job_key, started_at, triggered_by, finished_at=started_at, status="skipped_lock"), ""
+            if lock_wait <= 0:
+                return _record_run(job_key, started_at, triggered_by, finished_at=started_at, status="skipped_lock"), ""
+            try:
+                await asyncio.wait_for(_lock.acquire(), timeout=lock_wait)
+            except asyncio.TimeoutError:
+                return _record_run(job_key, started_at, triggered_by, finished_at=time.time(), status="skipped_lock"), ""
         acquired = True
 
     try:
@@ -170,18 +191,21 @@ async def run_job(
     triggered_by: str = "schedule",
     timeout: float | None = None,
     use_lock: bool = True,
+    lock_wait: float | None = None,
 ) -> JobRun:
     """Run a pipeline command under the shared mutual-exclusion lock,
-    recording one job_runs row start-to-finish. If the lock is already
+    recording one job_runs row start-to-finish. If the lock is already held,
-    held, records status='skipped_lock' immediately and returns without
+    scheduled runs (triggered_by="schedule") wait up to
-    running anything -- the old flock -n behavior, now visible in the UI
+    SCHEDULED_LOCK_WAIT_SECONDS for it before recording skipped_lock -- see
-    instead of silently skipping.
+    that constant for why -- while manual/other runs record skipped_lock
+    immediately (the old flock -n behavior, visible in the UI instead of
+    silently skipping). Pass lock_wait to override either way.
 
     use_lock=False runs the command WITHOUT taking the pipeline lock, for
     read-only jobs that never touch the library (e.g. the status report).
     Those must never be starved by a long-running write job, and running
     them concurrently is safe."""
-    run, _ = await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=False)
+    run, _ = await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=False, lock_wait=lock_wait)
     return run
 
 
@@ -191,6 +215,7 @@ async def run_job_capture(
     triggered_by: str = "manual",
     timeout: float | None = None,
     use_lock: bool = True,
+    lock_wait: float | None = None,
 ) -> tuple[JobRun, str]:
     """Like run_job(), but captures stdout+stderr as text and returns it
     alongside the JobRun, instead of only writing it to the log file --
@@ -198,7 +223,7 @@ async def run_job_capture(
     dedup_review_service's dry-run scan. The full output is still written
     to a log file afterward so job_runs.log_path works the same as any
     other job."""
-    return await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=True)
+    return await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=True, lock_wait=lock_wait)
 
 
 async def run_playlist(playlist_name: str, no_m3u: bool = False, triggered_by: str = "schedule") -> JobRun:

app/services/playlist_service.py

@@ -115,7 +115,8 @@ def update(db: Session, playlist_id: int, **fields) -> Playlist:
 
 
 def delete(db: Session, playlist_id: int) -> None:
-    """Remove the playlist from the DB and delete its rendered .conf file.
+    """Remove the playlist from the DB and delete its rendered .conf and
+    generated search .csv files.
     Does NOT touch anything already downloaded/imported for it — that's a
     library decision, not a playlist-definition one."""
     playlist = db.get(Playlist, playlist_id)
@@ -126,6 +127,9 @@ def delete(db: Session, playlist_id: int) -> None:
     db.commit()
     conf_path = settings.pipeline_config_dir / f"{name}.conf"
     conf_path.unlink(missing_ok=True)
+    # The search CSV run-playlist.sh generates next to the conf each run.
+    csv_path = settings.pipeline_config_dir / f"{name}.csv"
+    csv_path.unlink(missing_ok=True)
 
     from app.services import scheduler_service
 

app/services/scheduler_service.py

@@ -155,8 +155,19 @@ MAINTENANCE_JOBS: dict[str, tuple[dict, callable]] = {
         _log_rotation,
     ),
     # ==== Weekly (Sunday) ====
+    # strip_mb_tags runs first here at 01:00 -- not 08:30 like the rest of
+    # this chain -- because its runtime isn't stable: 10m19s on 2026-07-12,
+    # 39.6min on 2026-07-19 (MusicBrainz lookup latency scales with library
+    # size and isn't under our control). At 08:30 that variance repeatedly
+    # starved every job behind it: strip_watermark_art waited the full
+    # SCHEDULED_LOCK_WAIT_SECONDS and gave up every week from 07-12 onward,
+    # and on 07-19 the starvation cascaded all the way through genre:run,
+    # normalize_casing, beets_update_sync, dedup:scan, and both gen_*_playlist
+    # jobs. 01:00 sits in the dead zone before the first playlist sync (05:00)
+    # and well after log_rotation (00:00), so even a run several times slower
+    # than 07-19's is guaranteed to release the lock long before 08:30.
     "maintenance:strip_mb_tags": (
-        dict(minute=30, hour=8, day_of_week="sun"),
+        dict(minute=0, hour=1, day_of_week="sun"),
         _lib("maintenance:strip_mb_tags", "strip-mb-tags.sh"),
     ),
     "maintenance:strip_watermark_art": (

app/services/spotify_client.py

@@ -77,18 +77,34 @@ def get_playlist_tracks(db: Session, playlist_url: str) -> list[dict]:
 
     tracks = []
     # /items, not /tracks: Spotify removed GET /playlists/{id}/tracks in its
-    # February 2026 API changes in favor of /items (same response shape). New
+    # February 2026 API changes. The response shape ALSO changed, but only for
-    # apps are 403'd on the old endpoint; grandfathered apps still tolerate it
+    # apps on the new behavior: each entry's payload moved from "track" to
-    # for now, but /items is the correct, future-proof one.
+    # "item" (tracks.tracks.track -> items.items.item). Extended Quota Mode
+    # (grandfathered) apps keep the old "track" key, so parse both. No
+    # `fields` filter: it selects by key name, so on the renamed shape a
+    # track(...) filter silently returns empty pages -- exactly the failure
+    # we're avoiding.
     url = f"{API_BASE}/playlists/{playlist_id}/items"
-    params = {"limit": 100, "fields": "items(track(name,artists(name),external_ids)),next"}
+    params = {"limit": 100}
 
     while url:
         resp = httpx.get(url, params=params, headers=headers, timeout=15)
         resp.raise_for_status()
         data = resp.json()
-        for item in data.get("items", []):
+        if "items" not in data:
-            track = item.get("track")
+            # New-behavior apps get metadata only (no items field at all) for
+            # playlists the connected account doesn't own or collaborate on --
+            # public is no longer sufficient. Same message style the playlist
+            # page shows for a 403.
+            raise RuntimeError(
+                "Spotify returned this playlist without its contents. For newly "
+                "created Spotify apps, the account connected via Connect Spotify "
+                "must own the playlist (or be a collaborator on it) -- ask the "
+                "owner to share it as collaborative, or recreate it under the "
+                "connected account."
+            )
+        for item in data["items"]:
+            track = item.get("track") or item.get("item")
             if not track:
                 continue  # local files / removed tracks show up as null
             artists = track.get("artists") or []

app/static/style.css

@@ -475,11 +475,60 @@ tbody tr:hover { background: rgba(167, 139, 250, 0.055); }
 tbody td.actions-cell { display: flex; gap: 0.5rem; flex-wrap: wrap; }
 
 .dedup-table { table-layout: fixed; }
-.dedup-table .path-cell {
+
-  overflow: hidden;
+/* Each candidate is its own <tbody> (title row + detail row) so the pair
-  text-overflow: ellipsis;
+   highlights together on hover, and so :hover can bubble from either row
-  white-space: nowrap;
+   up to the shared group without JS. That means every detail row is now
-  max-width: 0; /* forces the cell to respect the colgroup width instead of the content's natural width */
+   a tbody's last-child, which would otherwise strip the separator between
+   one candidate and the next (the generic `tbody tr:last-child` rule above
+   assumed one shared tbody) -- restore it here, and only drop it for the
+   actual last group in the table. */
+.dedup-table .dedup-row-group:hover td { background: rgba(167, 139, 250, 0.06); }
+.dedup-table .dedup-detail-row td { border-bottom: 1px solid var(--border); }
+.dedup-table .dedup-row-group:last-of-type .dedup-detail-row td { border-bottom: none; }
+
+.dedup-title-row td { padding: 0.75rem 1.1rem 0.2rem; border-bottom: none; }
+.dedup-detail-row td { padding-top: 0.15rem; }
+
+.dedup-title {
+  font-family: var(--font-display);
+  font-size: 1.02rem;
+  font-weight: 700;
+  color: var(--fg);
+  line-height: 1.35;
+  overflow-wrap: anywhere;
+}
+.dedup-title-alt { font-weight: 400; color: var(--muted); }
+
+.dedup-context { font-size: 0.8rem; margin-top: 0.15rem; }
+
+/* A thin colored rail on each side echoes the keep/delete verdict itself --
+   green for the copy that survives, muted-pink for the one on the chopping
+   block -- so which side is which reads before you've even read the words. */
+.dedup-side {
+  display: flex;
+  align-items: center;
+  gap: 0.55rem;
+  padding-left: 0.65rem;
+  border-left: 2px solid transparent;
+  min-height: 1.6rem;
+}
+.dedup-side-keep { border-left-color: var(--status-success); }
+.dedup-side-delete { border-left-color: var(--status-danger); }
+
+.dedup-side-size { font-size: 0.78rem; white-space: nowrap; flex-shrink: 0; }
+
+/* The actual filename/path -- always visible, never truncated. Tags can be
+   wrong; this is the ground truth for judging whether two files are really
+   the same recording, so it can't be hover-only the way it was in the first
+   pass of this redesign. */
+.dedup-side-path {
+  padding-left: 0.65rem;
+  margin-top: 0.2rem;
+  font-size: 0.78rem;
+  color: var(--muted-2);
+  overflow-wrap: anywhere;
+  line-height: 1.4;
 }
 
 .empty-row td { padding: 2.5rem 1rem; text-align: center; }
@@ -509,6 +558,13 @@ tbody td.actions-cell { display: flex; gap: 0.5rem; flex-wrap: wrap; }
 .badge-pulse::before { animation: pulse-dot 1.4s ease-in-out infinite; }
 @keyframes pulse-dot { 0%, 100% { opacity: 1; } 50% { opacity: 0.35; } }
 
+/* The dedup "Caught by" column is only 10.75rem wide -- "Acoustically
+   Similar" at the default badge size overflowed past it and rendered on
+   top of the keep-side format pill. Smaller size/padding/tracking keeps it
+   inside the column instead of shrinking the column itself, which would
+   just crowd the Keep/Delete path columns next to it. */
+.badge-caughtby { font-size: 0.6rem; padding: 0.24rem 0.55rem; letter-spacing: 0.02em; }
+
 /* ---- stacked status bar (playlist track reconciliation) ---- */
 
 .status-bar {

app/templates/dedup/index.html

@@ -1,5 +1,65 @@
 {% extends "base.html" %}
 {% block title %}Dedup — alembic{% endblock %}
+
+{% macro candidate_rows(c, mode) %}
+<tbody class="dedup-row-group">
+  <tr class="dedup-title-row">
+    <td colspan="{{ 5 if mode == 'pending' else 4 }}">
+      {% if c.same_title %}
+      <div class="dedup-title">{{ c.keep.title }}</div>
+      {% else %}
+      <div class="dedup-title">{{ c.keep.title }} <span class="dedup-title-alt">/ {{ c.delete.title }}</span></div>
+      {% endif %}
+      {% if c.same_artist and c.same_album and c.keep.album %}
+      <div class="dedup-context muted">{{ c.keep.artist }} · {{ c.keep.album }}</div>
+      {% elif c.same_artist %}
+      <div class="dedup-context muted">{{ c.keep.artist }}</div>
+      {% else %}
+      <div class="dedup-context muted">{{ c.keep.artist }} / {{ c.delete.artist }}</div>
+      {% endif %}
+    </td>
+  </tr>
+  <tr class="dedup-detail-row">
+    {% if mode == 'pending' %}
+    <td><input type="checkbox" name="candidate_id" value="{{ c.id }}" form="bulk-delete-form"></td>
+    {% endif %}
+    <td><span class="badge badge-caughtby {{ 'badge-warning' if c.pass_label == 'Acoustically Similar' else 'badge-info' }}" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
+    <td>
+      <div class="dedup-side dedup-side-keep">
+        <span class="badge {{ c.keep.badge }}">{{ c.keep.ext or '?' }}</span>
+        {% if c.keep.size_bytes %}<span class="muted dedup-side-size">{{ (c.keep.size_bytes / 1024 / 1024) | round(1) }} MB</span>{% endif %}
+      </div>
+      <div class="dedup-side-path mono muted">{{ c.keep.display }}</div>
+    </td>
+    <td>
+      <div class="dedup-side dedup-side-delete">
+        <span class="badge {{ c.delete.badge }}">{{ c.delete.ext or '?' }}</span>
+        {% if c.delete.size_bytes %}<span class="muted dedup-side-size">{{ (c.delete.size_bytes / 1024 / 1024) | round(1) }} MB</span>{% endif %}
+      </div>
+      <div class="dedup-side-path mono muted">{{ c.delete.display }}</div>
+    </td>
+    <td class="actions-cell">
+      {% if mode == 'pending' %}
+      <form method="post" action="/dedup/confirm" class="inline"
+            onsubmit="return confirm('Permanently delete this file from disk?\n{{ c.delete.display }}')">
+        <input type="hidden" name="candidate_id" value="{{ c.id }}">
+        <button type="submit" class="btn btn-sm btn-danger">Delete</button>
+      </form>
+      <form method="post" action="/dedup/ignore" class="inline">
+        <input type="hidden" name="candidate_id" value="{{ c.id }}">
+        <button type="submit" class="btn btn-sm btn-ghost" title="Keep both copies and never flag this pair again">Keep both</button>
+      </form>
+      {% else %}
+      <form method="post" action="/dedup/unignore" class="inline">
+        <input type="hidden" name="candidate_id" value="{{ c.id }}">
+        <button type="submit" class="btn btn-sm btn-ghost">Un-ignore</button>
+      </form>
+      {% endif %}
+    </td>
+  </tr>
+</tbody>
+{% endmacro %}
+
 {% block content %}
 <div class="page-header">
   <div>
@@ -18,48 +78,30 @@
 {% endif %}
 
 <h2>Pending candidates</h2>
-<p class="muted" style="margin-top:-0.5rem;">Paths are relative to the library root. Hover a name for the full path.</p>
+<p class="muted" style="margin-top:-0.5rem;">Song title up top so you can scan straight down the list; the actual file path is always shown underneath each side so you can check they're really the same file before confirming.</p>
 <form id="bulk-delete-form" method="post" action="/dedup/confirm"
       onsubmit="return confirm('Permanently delete every selected file from disk? This cannot be undone.')"></form>
 <div class="table-wrap scroll">
   <table class="dedup-table">
     <colgroup>
-      <col style="width:2.2rem"><col style="width:9.5rem"><col style="width:28%">
+      <col style="width:2.2rem"><col style="width:10.75rem"><col style="width:37.5%">
-      <col style="width:28%"><col style="width:5rem"><col style="width:9rem">
+      <col style="width:37.5%"><col style="width:10rem">
     </colgroup>
     <thead>
-      <tr><th></th><th>Caught by</th><th>Keep</th><th>Delete</th><th>Size</th><th>Action</th></tr>
+      <tr><th></th><th>Caught by</th><th>Keep</th><th>Delete</th><th>Action</th></tr>
     </thead>
+    {% for c in candidates %}
+    {{ candidate_rows(c, 'pending') }}
+    {% else %}
     <tbody>
-      {% set pass_badge = {"File Naming": "badge-info", "Acoustically Similar": "badge-warning"} %}
+      <tr class="empty-row"><td colspan="5">
-      {% for c in candidates %}
-      <tr>
-        <td><input type="checkbox" name="candidate_id" value="{{ c.id }}" form="bulk-delete-form"></td>
-        <td><span class="badge {{ pass_badge.get(c.pass_label, 'badge-info') }}" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
-        <td class="muted mono path-cell" title="{{ c.keep_path }}">{{ c.keep_display }}</td>
-        <td class="mono path-cell" title="{{ c.delete_path }}">{{ c.delete_display }}</td>
-        <td class="muted">{{ (c.delete_size_bytes / 1024 / 1024) | round(1) if c.delete_size_bytes else '?' }} MB</td>
-        <td class="actions-cell">
-          <form method="post" action="/dedup/confirm" class="inline"
-                onsubmit="return confirm('Permanently delete this file from disk?\n{{ c.delete_display }}')">
-            <input type="hidden" name="candidate_id" value="{{ c.id }}">
-            <button type="submit" class="btn btn-sm btn-danger">Delete</button>
-          </form>
-          <form method="post" action="/dedup/ignore" class="inline">
-            <input type="hidden" name="candidate_id" value="{{ c.id }}">
-            <button type="submit" class="btn btn-sm btn-ghost" title="Keep both copies and never flag this pair again">Keep both</button>
-          </form>
-        </td>
-      </tr>
-      {% else %}
-      <tr class="empty-row"><td colspan="6">
         <div class="empty-state">
           <span class="sparkle" style="position:relative; display:inline-block; margin-bottom:0.5rem;"></span>
           <p class="muted" style="margin:0;">No pending candidates. Run a scan.</p>
         </div>
       </td></tr>
-      {% endfor %}
     </tbody>
+    {% endfor %}
   </table>
 </div>
 {% if candidates %}
@@ -72,28 +114,15 @@
 <div class="table-wrap scroll">
   <table class="dedup-table">
     <colgroup>
-      <col style="width:9.5rem"><col style="width:33%">
+      <col style="width:10.75rem"><col style="width:39.25%">
-      <col style="width:33%"><col style="width:5rem"><col style="width:7rem">
+      <col style="width:39.25%"><col style="width:7rem">
     </colgroup>
     <thead>
-      <tr><th>Caught by</th><th>Keep</th><th>Delete</th><th>Size</th><th></th></tr>
+      <tr><th>Caught by</th><th>Keep</th><th>Delete</th><th></th></tr>
     </thead>
-    <tbody>
+    {% for c in ignored %}
-      {% for c in ignored %}
+    {{ candidate_rows(c, 'ignored') }}
-      <tr>
+    {% endfor %}
-        <td><span class="badge badge-muted" title="{{ c.pass_name }}">{{ c.pass_label }}</span></td>
-        <td class="muted mono path-cell" title="{{ c.keep_path }}">{{ c.keep_display }}</td>
-        <td class="muted mono path-cell" title="{{ c.delete_path }}">{{ c.delete_display }}</td>
-        <td class="muted">{{ (c.delete_size_bytes / 1024 / 1024) | round(1) if c.delete_size_bytes else '?' }} MB</td>
-        <td>
-          <form method="post" action="/dedup/unignore" class="inline">
-            <input type="hidden" name="candidate_id" value="{{ c.id }}">
-            <button type="submit" class="btn btn-sm btn-ghost">Un-ignore</button>
-          </form>
-        </td>
-      </tr>
-      {% endfor %}
-    </tbody>
   </table>
 </div>
 {% endif %}

pipeline/bin/import-track.sh

@@ -242,6 +242,30 @@ BEETS_EXIT=0
 beet import -q -s "$DEST_DIR" >> "$LOG" 2>&1 || BEETS_EXIT=$?
 log "beets import finished with exit code $BEETS_EXIT"
 
+# ==== Safety net: clean up stragglers + exact duplicates ====
+# With duplicate_action=keep, beets always imports rather than rejecting a
+# real conflict (see beets/config.yaml) -- any file whose (albumartist,
+# album, title) already exists in the library gets imported anyway as a
+# `.1.ext` sibling. That's true whether this run came from the web UI, an
+# SMB drop, or sync-bandcamp.sh, so the cleanup has to run here rather than
+# per-caller (a 2026-07-23 incident: a Bandcamp sync's own call into this
+# script failed and silently stranded ~150 already-owned files in import-me/
+# for two weeks; a later unrelated manual import swept them back in and
+# duplicated them, and nothing had deduped since).
+log "Running replace-with-better safety pass on /downloads stragglers"
+if ${PIPELINE_DIR:-/app/pipeline}/lib/replace-with-better.sh --apply >> "$LOG" 2>&1; then
+  log "replace-with-better OK"
+else
+  log "WARN: replace-with-better.sh exited non-zero (exit $?)"
+fi
+
+log "Running dedup pass (exact-match duplicates only; FLAC > MP3, then largest file)"
+if ${PIPELINE_DIR:-/app/pipeline}/lib/dedup-library.sh --apply >> "$LOG" 2>&1; then
+  log "dedup OK"
+else
+  log "WARN: dedup-library.sh exited non-zero (exit $?)"
+fi
+
 # ==== Regenerate M3U if playlist was specified ====
 if [[ -n "$PLAYLIST_NAME" ]]; then
   log "Regenerating M3U for $PLAYLIST_NAME"

pipeline/bin/run-playlist.sh

@@ -25,6 +25,8 @@ PLAYLIST_NAME="${1:?Usage: $0 [--no-m3u] <playlist_name>}"
 
 # ==== Paths ====
 CONFIG_FILE=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/${PLAYLIST_NAME}.conf
+SPOTIFY_ENV=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/_spotify.env
+CSV_FILE=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/${PLAYLIST_NAME}.csv
 DROPBOX=${MUSIC_DATA_DIR:-/data/music}/sldl-dropbox/${PLAYLIST_NAME}
 PLAYLISTS_DIR=${MUSIC_DATA_DIR:-/data/music}/playlists
 QUARANTINE_DIR=${MUSIC_DATA_DIR:-/data/music}/Songs/untagged
@@ -54,6 +56,61 @@ if [[ ! -f "$CONFIG_FILE" ]]; then
   exit 1
 fi
 
+# ==== Read the playlist from Spotify into a CSV sldl can search from ====
+# sldl's own vendored Spotify client still calls GET /playlists/{id}/tracks,
+# which Spotify removed in its February 2026 API changes. Grandfathered apps
+# still get a pass there; apps created after that change get a hard 403
+# regardless of auth method (client-credentials, or even a correctly-scoped,
+# correctly-owned OAuth user token -- confirmed live, exit 134 unhandled
+# APIException: Forbidden from Extractors.Spotify.GetPlaylist). sldl has no
+# fallback to the still-working /items endpoint, so it can never read a
+# playlist for a new app no matter what alembic hands it. Reading the
+# playlist ourselves (via /items) and handing sldl a plain CSV instead
+# sidesteps sldl's Spotify client entirely -- works the same for
+# grandfathered and new apps alike.
+SPOTIFY_URL=$(sed -n 's/^input *= *//p' "$CONFIG_FILE" | tr -d ' ')
+[[ -f "$SPOTIFY_ENV" ]] && source "$SPOTIFY_ENV"
+
+if [[ -z "$SPOTIFY_URL" || -z "${SPOTIFY_CLIENT_ID:-}" || -z "${SPOTIFY_CLIENT_SECRET:-}" ]]; then
+  log "ERROR: missing playlist URL in $CONFIG_FILE or Spotify credentials in $SPOTIFY_ENV"
+  exit 1
+fi
+
+log "Fetching playlist from Spotify: $SPOTIFY_URL"
+if ! SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
+    SPOTIFY_REFRESH_TOKEN="${SPOTIFY_REFRESH_TOKEN:-}" \
+    python3 "${PIPELINE_DIR:-/app/pipeline}/lib/spotify-playlist-csv.py" "$SPOTIFY_URL" "$CSV_FILE" >> "$LOG" 2>&1; then
+  log "ERROR: failed to fetch playlist from Spotify — see $LOG"
+  exit 1
+fi
+
+# Surface the fetched count in the timestamped summary. Zero tracks from a
+# successful fetch is almost never a truly empty playlist -- it usually means
+# Spotify hid the contents from the connected account (see the pointed errors
+# in spotify-playlist-csv.py), so call it out instead of letting sldl no-op
+# with a clean exit 0.
+TRACK_COUNT=$(($(wc -l < "$CSV_FILE") - 1))
+if [[ "$TRACK_COUNT" -le 0 ]]; then
+  log "WARNING: Spotify returned 0 visible tracks for this playlist -- nothing to download. If the playlist isn't actually empty, the connected Spotify account can't see its contents (it must own or collaborate on the playlist for newly created Spotify apps)."
+else
+  log "Fetched $TRACK_COUNT track(s) from Spotify"
+fi
+
+# ==== Seed the pinned sldl skip index if it's missing or stranded ====
+# The conf pins index-path to $DROPBOX/_index.csv (see _template.conf: sldl's
+# default index location is derived from the input, so an input change strands
+# the old index and the whole playlist re-downloads -- that's what produced
+# the 2026-07-16 duplicate flood). If the pinned file is missing, or an index
+# in one of sldl's old input-named subfolders is newer (i.e. sldl last wrote
+# somewhere else), fold them all into the pinned location first.
+INDEX_FILE="$DROPBOX/_index.csv"
+if [[ ! -s "$INDEX_FILE" ]] || \
+   [[ -n "$(find "$DROPBOX" -mindepth 2 -maxdepth 2 -name _index.csv -newer "$INDEX_FILE" -print -quit 2>/dev/null)" ]]; then
+  log "Seeding pinned sldl index at $INDEX_FILE from prior indexes"
+  python3 "${PIPELINE_DIR:-/app/pipeline}/lib/merge-sldl-indexes.py" "$DROPBOX" "$INDEX_FILE" >> "$LOG" 2>&1 \
+    || log "WARNING: index merge failed -- sldl may re-download tracks the library already has"
+fi
+
 # ==== Run sldl (vendored binary, subprocess of this container) ====
 log "Running sldl for: $PLAYLIST_NAME"
 
@@ -62,37 +119,24 @@ log "Running sldl for: $PLAYLIST_NAME"
 # download. A non-zero exit here is logged but not fatal.
 #
 # Hard timeout: without it a stuck sldl hangs FOREVER holding the shared lock,
-# which silently skips every later-scheduled playlist for the night. (Seen
+# which silently skips every later-scheduled playlist for the night. timeout
-# 2026-06-18: a transient Spotify client-creds failure made sldl fall back to
+# TERMs the run and KILLs after a grace period; there's no leftover container
-# interactive OAuth — "manually open: https://accounts.spotify.com/..." — and
+# to clean up now that sldl is a plain subprocess instead of a docker-compose
-# it waited 7h on a browser callback that never comes. Fixed by always
+# service.
-# rendering `spotify-refresh` into the conf once a Spotify account is
+#
-# connected via /connect/spotify — sldl's own docs confirm supplying a
+# -c "$CONFIG_FILE" still supplies Soulseek login, download path, quality
-# refresh token skips the login-flow requirement entirely. This timeout stays
+# settings, port, etc. -- the CSV positional arg + --input-type csv override
-# as a backstop for any other cause of a stuck run.) timeout TERMs the run
+# just the input source (confirmed: sldl ignores the conf's own `input =`/
-# and KILLs after a grace period; there's no leftover container to clean up
+# `input-type =` lines when both are given).
-# now that sldl is a plain subprocess instead of a docker-compose service.
 SLDL_TIMEOUT="${SLDL_TIMEOUT:-2700}"        # 45 min; override via env
 SLDL_EXIT=0
 timeout --kill-after=30s "$SLDL_TIMEOUT" \
-  "$SLDL_BIN" -c "$CONFIG_FILE" >> "$LOG" 2>&1 \
+  "$SLDL_BIN" "$CSV_FILE" --input-type csv -c "$CONFIG_FILE" >> "$LOG" 2>&1 \
   || SLDL_EXIT=$?
 if [[ "$SLDL_EXIT" -eq 124 || "$SLDL_EXIT" -eq 137 ]]; then
   log "ERROR: sldl exceeded ${SLDL_TIMEOUT}s and was killed (likely a hang)"
 elif [[ "$SLDL_EXIT" -eq 134 ]]; then
-  # sldl aborts (SIGABRT) on ANY unhandled .NET exception rather than failing
+  log "ERROR: sldl crashed (exit 134, unhandled exception) -- see the log above for the exception detail"
-  # cleanly. The one case we've actually seen in the wild: a valid, freshly-
-  # refreshed user token still gets 403 Forbidden loading the playlist,
-  # because Spotify scopes OAuth playlist reads to what the CONNECTED account
-  # can see (its own playlists, ones it collaborates on, or ones marked
-  # Public) -- unlike client-credentials, which could read any public
-  # playlist regardless of whose app it was. Give a pointed hint when that's
-  # the visible cause; otherwise just flag that sldl crashed outright.
-  if grep -q "APIException: Forbidden" "$LOG"; then
-    log "ERROR: sldl crashed (exit 134) -- Spotify returned 403 loading this playlist. The connected Spotify account can't see it: make sure it's owned by, or shared/followed by, whichever account is connected via /connect/spotify, or set it to Public on Spotify."
-  else
-    log "ERROR: sldl crashed (exit 134, unhandled exception) -- see the log above for the exception detail"
-  fi
 fi
 log "sldl finished with exit code $SLDL_EXIT"
 
@@ -125,27 +169,19 @@ log "Cleaning up .incomplete files in dropbox"
 find "$DROPBOX" -name "*.incomplete" -type f -delete 2>>"$LOG" || true
 
 # ==== Rewrite tags from Spotify (source of truth for matched tracks) ====
-# Reads Spotify creds + playlist URL from the same conf sldl used. For each file
+# Uses the playlist URL and _spotify.env credentials already loaded for the
-# in the dropbox that matches a Spotify playlist track, overwrites ARTIST
+# CSV fetch above (confs no longer carry Spotify creds). For each file in the
+# dropbox that matches a Spotify playlist track, overwrites ARTIST
 # (semicolon-joined), ALBUMARTIST (primary), ALBUM, TITLE, TRACKNUMBER,
 # DISCNUMBER, DATE. GROUPING is preserved. Files that don't match are left for
 # the fallback step below.
-SPOTIFY_URL=$(sed -n 's/^input *= *//p' "$CONFIG_FILE" | tr -d ' ')
+log "Rewriting tags from Spotify for files in $DROPBOX"
-SPOTIFY_CLIENT_ID=$(sed -n 's/^spotify-id *= *//p' "$CONFIG_FILE" | tr -d ' ')
+if SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
-SPOTIFY_CLIENT_SECRET=$(sed -n 's/^spotify-secret *= *//p' "$CONFIG_FILE" | tr -d ' ')
+    SPOTIFY_REFRESH_TOKEN="${SPOTIFY_REFRESH_TOKEN:-}" \
-SPOTIFY_REFRESH_TOKEN=$(sed -n 's/^spotify-refresh *= *//p' "$CONFIG_FILE" | tr -d ' ')
+    python3 ${PIPELINE_DIR:-/app/pipeline}/lib/spotify-retag.py "$SPOTIFY_URL" "$DROPBOX" >> "$LOG" 2>&1; then
-
+  log "Spotify retag complete"
-if [[ -n "$SPOTIFY_URL" && -n "$SPOTIFY_CLIENT_ID" && -n "$SPOTIFY_CLIENT_SECRET" ]]; then
-  log "Rewriting tags from Spotify for files in $DROPBOX"
-  if SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
-      SPOTIFY_REFRESH_TOKEN="$SPOTIFY_REFRESH_TOKEN" \
-      python3 ${PIPELINE_DIR:-/app/pipeline}/lib/spotify-retag.py "$SPOTIFY_URL" "$DROPBOX" >> "$LOG" 2>&1; then
-    log "Spotify retag complete"
-  else
-    log "WARNING: Spotify retag failed (exit $?) — continuing with sldl-supplied tags"
-  fi
 else
-  log "Skipping Spotify retag — missing input/spotify-id/spotify-secret in $CONFIG_FILE"
+  log "WARNING: Spotify retag failed (exit $?) — continuing with sldl-supplied tags"
 fi
 
 # ==== Quarantine any files still missing essential tags ====

pipeline/configs/_template.conf

@@ -14,19 +14,15 @@
 user = SOULSEEK_USER
 pass = SOULSEEK_PASS
 
-# ==== Spotify API credentials ====
-spotify-id = SPOTIFY_CLIENT_ID
-spotify-secret = SPOTIFY_CLIENT_SECRET
-# Set once an account is connected via /connect/spotify (blank otherwise).
-# sldl's own docs confirm supplying a refresh token skips its interactive
-# login-flow requirement entirely -- this is what lets newly created Spotify
-# apps (which Spotify blocks from reading playlists with client-credentials
-# alone) work without sldl ever trying to open a browser.
-spotify-refresh = SPOTIFY_REFRESH_TOKEN
-
 # ==== Input (Spotify playlist URL) ====
+# sldl itself never reads Spotify at all -- its vendored Spotify client still
+# calls GET /playlists/{id}/tracks, which Spotify removed in its February 2026
+# API changes (new apps get a hard 403 there regardless of auth method; see
+# spotify-playlist-csv.py). run-playlist.sh reads the playlist itself (via the
+# still-working /items endpoint) and hands sldl a CSV via --input-type csv on
+# the command line, which overrides this line entirely -- it's kept only so
+# run-playlist.sh has somewhere to read the playlist URL from per-playlist.
 input = SPOTIFY_URL
-input-type = spotify
 
 # ==== Output paths ====
 # SLDL_DROPBOX_ROOT is substituted at render time from $MUSIC_DATA_DIR
@@ -35,6 +31,18 @@ path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME
 playlist-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_sldl.m3u8
 write-playlist = true
 
+# ==== Skip index (pinned!) ====
+# sldl's already-downloaded index defaults to {path}/{playlist-name}/_index.csv
+# where {playlist-name} is derived from the INPUT: the Spotify playlist's
+# display name for spotify input, the CSV filename for csv input. beets moves
+# every download out of the dropbox, so this index is the ONLY thing standing
+# between a nightly run and re-downloading the whole playlist. When 0.6.2
+# switched input from Spotify URL to CSV, the derived name changed, sldl
+# started a fresh empty index in a new subfolder, and every playlist
+# re-downloaded in full on 2026-07-16. Pinning the path here decouples the
+# index from input naming so that can never happen again.
+index-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_index.csv
+
 # ==== Naming ====
 name-format = {artist} - {title}
 

pipeline/lib/dedup-library.sh

@@ -34,11 +34,26 @@ set -euo pipefail
 APPLY=0
 JSON_MODE=0
 ONLY_PATHS_FILE=""
+AUTO_APPLY_NUMBERED=0
 while [[ $# -gt 0 ]]; do
   case "$1" in
     --apply) APPLY=1; shift ;;
     --json) JSON_MODE=1; shift ;;
     --only-paths) ONLY_PATHS_FILE="$2"; shift 2 ;;
+    # Delete two kinds of match unconditionally, independent of
+    # --apply/--only-paths, while everything else stays dry-run/log-only:
+    #   - numbered-sibling twins (see is_numbered_twin): same dir, same name,
+    #     same extension, just the ".N" collision suffix -- the same download
+    #     landing twice.
+    #   - format upgrades within an exact tag match (see is_format_upgrade):
+    #     Pass 2/3 already proved same song via identical tags; a different
+    #     extension there just means "better format vs. worse", never a
+    #     different version.
+    # Excludes Pass 4 (cross-album fuzzy) entirely -- matching across
+    # different albums/directories is exactly where a different master or
+    # DJ-mix edit can share tags without being interchangeable, so it always
+    # needs a human. See dedup_review_service.scan().
+    --auto-apply-numbered) AUTO_APPLY_NUMBERED=1; shift ;;
     *) shift ;;
   esac
 done
@@ -78,11 +93,63 @@ fi
 # Counters: tracked via log grep at the end (avoids bash subshell pitfalls).
 # Functions write KEEP/DELETE lines with consistent prefixes; summary greps them.
 
-# Convert container path (/music/...) to host path (${MUSIC_DATA_DIR:-/data/music}/Library/...)
+# Resolve a beets-displayed path to a real path in this container. Since the
-host_path() { echo "${MUSIC_DATA_DIR:-/data/music}/Library${1#/music}"; }
+# 2026-07-08 path rewrite, beets displays ${MUSIC_DATA_DIR:-/data/music}/Library/...
+# paths that are directly usable — pass those through UNCHANGED. Only legacy
+# /music/... display paths (the pre-rewrite transitional mount) still need the
+# prefix swap. Blindly prepending the library dir to an already-correct path
+# (the old behavior) produced /data/music/Library/data/music/Library/... —
+# nonexistent, so every candidate failed the -f check and every pass reported
+# 0 groups from 2026-07-08 until this fix.
+host_path() {
+  local p="$1"
+  if [[ "$p" == /music/* ]]; then
+    echo "${MUSIC_DATA_DIR:-/data/music}/Library${p#/music}"
+  else
+    echo "$p"
+  fi
+}
 
 SEP=$'\x1c'  # ASCII file separator — safe with any music metadata
 
+# True if two paths are the SAME directory, SAME extension, and identical
+# basenames except for a numeric ".N" infix sldl/beets inserts on a filename
+# collision (e.g. "Title.flac" vs "Title.1.flac"). This is deliberately
+# narrower than any tag-based pass: no fuzzy matching, no cross-album
+# ambiguity, just the literal same download landing twice. Used to
+# auto-delete regardless of which pass found the pair -- Pass 1 only fires
+# when the canonical name is a beets ghost; the very common case where BOTH
+# copies are already tracked in beets (so Pass 1 defers) surfaces instead via
+# Pass 2/3's tag-based grouping, and still deserves the same free pass.
+is_numbered_twin() {
+  local a="$1" b="$2"
+  [[ "${a%/*}" == "${b%/*}" ]] || return 1
+  local ba="${a##*/}" bb="${b##*/}"
+  local ext="${ba##*.}"
+  [[ "${ext,,}" == "${bb##*.}" ]] || return 1
+  [[ "$(echo "$ba" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" \
+     == "$(echo "$bb" | sed -E "s/\.[0-9]+\.${ext}\$/.${ext}/")" ]]
+}
+
+# True if this pair is a same-song format upgrade found by an EXACT tag match
+# (Pass 2 case-insensitive or Pass 3 normalized -- both require identical
+# albumartist+album+title after case/punctuation folding, so "same song" is
+# already proven by the pass itself) where the two files simply have
+# different extensions. rank_file() always ranks FLAC ahead of any other
+# format regardless of size/dirty-name/etc, so within a tag-exact group a
+# cross-extension pair unconditionally means "the format-preferred copy vs.
+# a lesser one" -- no judgment call. Deliberately excludes Pass 4
+# (cross_album_fuzzy): that pass matches by artist+title-base ACROSS
+# different albums/directories, which is exactly where a different master,
+# DJ-mix edit, or reissue can share tags without being an interchangeable
+# copy -- those still need a human to look at album/context before deleting
+# either side.
+is_format_upgrade() {
+  local pass="$1" keep="$2" delete="$3"
+  [[ "$pass" == "case_insensitive" || "$pass" == "normalized" ]] || return 1
+  [[ "${keep##*.}" != "${delete##*.}" ]]
+}
+
 # Rank a file: lower = better. FLAC > MP3 > other (WAV/etc.); clean filename
 # > sync-conflict artifact (-DESKTOP-, (1), .1.); extended/club mix > radio
 # edit at same format; then larger wins.
@@ -163,7 +230,7 @@ process_group() {
   esac
 
   local first=1
-  local keep_path=""
+  local keep_path="" keep_id="" any_auto=0
   while IFS="$SEP" read -r _sk id hp; do
     [[ -z "$hp" ]] && continue
     local size_bytes; size_bytes=$(stat -c '%s' "$hp" 2>/dev/null || echo 0)
@@ -171,14 +238,31 @@ process_group() {
     if [[ $first -eq 1 ]]; then
       log "  KEEP   ($size_h) $hp"
       keep_path="$hp"
+      keep_id="$id"
       first=0
     else
       log "  DELETE ($size_h) $hp"
-      json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes}}"
+
+      # Auto-apply-numbered deletes numbered_sibling matches unconditionally,
+      # bypassing --only-paths (that gate exists only for the manual confirm
+      # flow, which never runs alongside this flag). All other passes stay
+      # dry-run/log-only here regardless.
+      local do_delete=0 auto_applied=0
       if [[ $APPLY -eq 1 ]]; then
+        do_delete=1
+      elif [[ $AUTO_APPLY_NUMBERED -eq 1 ]] \
+        && { is_numbered_twin "$keep_path" "$hp" || is_format_upgrade "$pass_name" "$keep_path" "$hp"; }; then
+        do_delete=1
+        auto_applied=1
+        any_auto=1
+      fi
+      local auto_json="false"; [[ $auto_applied -eq 1 ]] && auto_json="true"
+      json_emit "{\"pass\":\"${pass_name}\",\"keep_path\":\"$(json_escape "$keep_path")\",\"delete_path\":\"$(json_escape "$hp")\",\"delete_size_bytes\":${size_bytes},\"auto_applied\":${auto_json}}"
+
+      if [[ $do_delete -eq 1 ]]; then
         # With --only-paths given, only delete entries the caller explicitly
         # confirmed; without it, delete everything (original behavior).
-        if [[ -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
+        if [[ $auto_applied -eq 0 && -n "$ONLY_PATHS_FILE" && -z "${ONLY_PATHS[$hp]:-}" ]]; then
           log "    (skipped — not in --only-paths confirm list)"
         elif [[ -n "$id" ]]; then
           # In beets — beet remove -d removes from DB + disk. Best-effort: a
@@ -192,6 +276,18 @@ process_group() {
       fi
     fi
   done < <(echo "$ranked" | grep -v '^$' | sort -n)
+
+  # An auto-applied numbered-twin deletion can leave the numbered-named file
+  # as the sole survivor (it won on size despite the dirty-filename penalty).
+  # Rename it back to the canonical name so the library doesn't accumulate
+  # ".1."/".2." filenames for tracks that no longer have a duplicate. Only
+  # auto_applied triggers this (never a manual --only-paths confirm, which
+  # may leave the sibling undeleted if the user didn't confirm it -- renaming
+  # then would collide); only when it's actually in beets (a ghost survivor
+  # has nothing to move); only when the name still has the dirty infix.
+  if [[ $any_auto -eq 1 && -n "$keep_id" && "$keep_path" =~ \.[0-9]+\.[A-Za-z0-9]+$ ]]; then
+    beet move "id:${keep_id}" >> "$LOG" 2>&1 || log "    WARN: beet move id:${keep_id} failed (rename survivor)"
+  fi
   return 0  # explicit: the loop's last command status must not leak out under set -e
 }
 
@@ -202,9 +298,10 @@ process_group() {
 log ""
 log "[$(date -Iseconds)] === Pass 1: numbered siblings ==="
 
-# Dump id + container path from beets once. beets normalizes $path for
+# Dump id + display path from beets once. beets normalizes $path for display
-# display (always /music/-prefixed) even though the DB stores a mix of
+# (real /data/music/Library/... paths since the 2026-07-08 rewrite; /music/...
-# absolute and relative — so the display paths are safe to compare/convert,
+# on any legacy entry) even though the DB stores a mix of absolute and
+# relative — so the display paths are safe to compare/convert via host_path(),
 # and the ids are what we hand to beet remove/move.
 beet ls -f "\$id${SEP}\$path" 2>/dev/null > /tmp/beets-id-paths.txt
 cut -d"$SEP" -f2- /tmp/beets-id-paths.txt > /tmp/beets-all-paths.txt
@@ -231,18 +328,22 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
 
   if [[ $score_canonical -le $score_numbered ]]; then
     # Canonical wins: delete numbered (in beets), keep canonical (ghost)
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
       "$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
-    if [[ $APPLY -eq 1 ]]; then
+    if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
       # canonical is now a ghost file; import it into beets
       beet import -q -s "${canonical_cp}" >> "$LOG" 2>&1 || log "  WARN: beet import failed for ${canonical_cp}"
     fi
   else
     # Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
     # the numbered file to the canonical path (id query — see process_group).
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    # Safe to reach with just AUTO_APPLY_NUMBERED: this loop only ever builds
+    # genuine numbered-sibling pairs (see is_numbered_twin), and process_group
+    # above already deleted the ghost canonical before this runs, so the move
+    # target is guaranteed clear -- no rename collision.
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
       "${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
-    if [[ $APPLY -eq 1 ]]; then
+    if [[ $APPLY -eq 1 || $AUTO_APPLY_NUMBERED -eq 1 ]]; then
       beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log "  WARN: beet move id:${numbered_id} failed"
     fi
   fi

pipeline/lib/enrich-buy-url.py

@@ -458,7 +458,7 @@ def main():
           + (f"  upgrade-from={sorted(upgrade_from)}" if upgrade_from else ""))
     print(f"[enrich] {len(flacs)} FLAC files in library\n")
 
-    looked = found = upgraded = 0
+    looked = found = upgraded = write_failed = 0
     by_source = {s: 0 for s in cascade}
     touched = []
     for p in flacs:
@@ -519,17 +519,25 @@ def main():
             if set_flac_tag(sp, BUY_URL_TAG, url):
                 touched.append(sp)
             else:
-                print("      ! metaflac write failed")
+                write_failed += 1
+                print("      ! metaflac write failed (permissions? disk full?) -- NOT tagged")
         else:
             touched.append(sp)
 
         if found % 50 == 0:
             print(f"  ...{found} links from {looked} lookups so far", flush=True)
 
-    print(f"\n[enrich] {looked} lookups, {found} {'tagged' if args.apply else 'would-tag'}"
+    # touched is only appended to on an actual successful write (apply mode)
+    # or a would-tag match (dry run) -- len(touched) is what really landed on
+    # disk. `found` counts matches regardless of write outcome, so reporting
+    # `found` here as "tagged" lied about full success on 2026-07-22, when
+    # every write in the run failed (root-owned files under explo/) but the
+    # summary line still read "51 tagged".
+    print(f"\n[enrich] {looked} lookups, {len(touched)} {'tagged' if args.apply else 'would-tag'}"
           f"  ({', '.join(f'{s}={by_source[s]}' for s in cascade)})"
           f"  no-match={looked - found}"
-          + (f"  upgraded={upgraded}" if upgrade_from else ""))
+          + (f"  upgraded={upgraded}" if upgrade_from else "")
+          + (f"  WRITE-FAILED={write_failed}" if write_failed else ""))
 
     if args.apply and touched and az_key and args.azuracast_base:
         print("[enrich] telling AzuraCast to reprocess touched files...")

pipeline/lib/export-laptop-playlists.py

@@ -57,11 +57,12 @@ OUT_DIR = f"{_MUSIC_DATA_DIR}/playlists-laptop"
 # Absolute Windows path to the laptop's library root.
 LIBRARY_LAPTOP_ROOT = r"C:\Music\Library"
 
-# Beets stores paths with this prefix (the container-side mount of the
+# Prefixes beets may display library paths under, tried in order: the real
-# library); strip it to get the path relative to the library root. This is
+# library dir (everything since the 2026-07-08 path rewrite) and the legacy
-# still "/music/" through the migration's Stage 0-3 transitional mount;
+# transitional mount (any stale pre-rewrite entry). Strip whichever matches
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
+# to get the path relative to the library root; when neither matches the
-BEETS_LIBRARY_PREFIX = "/music/"
+# track is outside the library and is skipped.
+BEETS_LIBRARY_PREFIXES = (f"{_MUSIC_DATA_DIR}/Library/", "/music/")
 
 M3U_EXT = ".m3u"
 # Older formats from earlier iterations of this script — swept by reconcile.
@@ -157,9 +158,12 @@ def build_beets_index() -> dict[tuple[str, str], list[tuple[str, str, str, str]]
         if len(parts) != 4:
             continue
         aa, alb, ti, path = parts
-        if not path.startswith(BEETS_LIBRARY_PREFIX):
+        rel = next(
+            (path[len(p):] for p in BEETS_LIBRARY_PREFIXES if path.startswith(p)),
+            None,
+        )
+        if rel is None:
             continue
-        rel = path[len(BEETS_LIBRARY_PREFIX):]
         idx.setdefault((_normkey(alb), _normkey(ti)), []).append((aa, alb, ti, rel))
     return idx
 

pipeline/lib/fix-track-metadata.py

@@ -49,8 +49,10 @@ _ALEMBIC_CONFIG_DIR = os.environ.get("ALEMBIC_CONFIG_DIR", "/config")
 _MUSIC_DATA_DIR = os.environ.get("MUSIC_DATA_DIR", "/data/music")
 
 LIBRARY = f"{_MUSIC_DATA_DIR}/Library"
-# Still "/music" through the migration's Stage 0-3 transitional beets mount;
+# Legacy prefix from the migration's transitional beets mount. Since the
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
+# 2026-07-08 path rewrite beets displays real LIBRARY paths, so this only
+# matters for accepting pasted /music/... paths as input and as a fallback
+# beets query form for any stale pre-rewrite DB entry.
 CONTAINER_PREFIX = "/music"
 SPOTIFY_ENV = f"{_ALEMBIC_CONFIG_DIR}/pipeline/_spotify.env"
 SPOTIFY_GENRE = f"{os.environ.get('PIPELINE_DIR', '/app/pipeline')}/lib/spotify-genre.py"
@@ -316,15 +318,19 @@ def resolve_input_path(arg):
 
 
 def beets_id_for(host_path):
-    cp = host_to_container(host_path)
+    # Real path first (how beets displays everything since the 2026-07-08
-    r = subprocess.run(
+    # path rewrite), legacy /music/... form second for any stale entry.
-        ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
+    # Querying only the legacy form (the old behavior) matched nothing after
-        capture_output=True, text=True, check=True)
+    # the rewrite, so retag-from-url silently skipped beet update/move.
-    for line in r.stdout.splitlines():
+    for cp in (host_path, host_to_container(host_path)):
-        if "|||" in line:
+        r = subprocess.run(
-            id_str, p = line.split("|||", 1)
+            ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
-            if p == cp:
+            capture_output=True, text=True, check=True)
-                return int(id_str)
+        for line in r.stdout.splitlines():
+            if "|||" in line:
+                id_str, p = line.split("|||", 1)
+                if p == cp:
+                    return int(id_str)
     return None
 
 

pipeline/lib/merge-sldl-indexes.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Fold every sldl skip index under a playlist's dropbox into one pinned file.
+
+sldl names its per-playlist index folder after the *input*: the Spotify
+playlist's display name for spotify input, the CSV filename stem for csv
+input. So when 0.6.2 switched input from Spotify URL to CSV, sldl started a
+fresh empty index in a new subfolder, saw no download history, and
+re-downloaded every playlist in full (2026-07-16). The rendered confs now pin
+index-path to <dropbox>/<playlist>/_index.csv; this script seeds that pinned
+file from all the indexes sldl left behind (run-playlist.sh calls it whenever
+the pinned file is missing or older than a stranded one).
+
+Usage: merge-sldl-indexes.py <playlist_dropbox_dir> <output_index_csv>
+
+Merge rules, per (artist, title) lowercased:
+  - the row from the newest index file wins;
+  - EXCEPT when that row is a failure (state 2) and any older index has the
+    track as downloaded (state 1 or 3): then the newest row's identity
+    (artist/album/title/length -- matching what the current input will
+    present; Spotify length rounding drifted between the old extractor and
+    our CSV) is kept but marked state 3 (already downloaded), so sldl does
+    not re-fetch a track the library already holds;
+  - rows only present in older indexes are kept as-is (tracks since removed
+    from the playlist; harmless, and they keep their history if re-added).
+"""
+
+import csv
+import sys
+from pathlib import Path
+
+HEADER = ["filepath", "artist", "album", "title", "length", "tracktype", "state", "failurereason"]
+DOWNLOADED_STATES = {"1", "3"}  # 1 = downloaded this run, 3 = found in index previously
+FAILED_STATE = "2"
+
+
+def read_rows(path: Path) -> list[dict]:
+    rows = []
+    with open(path, newline="", encoding="utf-8") as fh:
+        reader = csv.DictReader(fh)
+        for row in reader:
+            if row.get("artist") is None or row.get("title") is None:
+                continue
+            rows.append({k: (row.get(k) or "") for k in HEADER})
+    return rows
+
+
+def norm_key(row: dict) -> tuple:
+    # Primary artist only: sldl's old Spotify extractor recorded just the
+    # first artist, while spotify-playlist-csv.py joins all of them with
+    # ", " -- keying on the full string would miss every multi-artist track
+    # when recovering history across the two index generations. First
+    # comma-segment matches both forms (and both sides of a comma-in-name
+    # artist like "Tyler, The Creator" truncate identically).
+    return (row["artist"].split(",")[0].strip().lower(), row["title"].strip().lower())
+
+
+def main() -> int:
+    if len(sys.argv) != 3:
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    dropbox = Path(sys.argv[1])
+    out_path = Path(sys.argv[2])
+    if not dropbox.is_dir():
+        print(f"[merge-sldl-indexes] not a directory: {dropbox}", file=sys.stderr)
+        return 2
+
+    # Every index at the dropbox root or one level down (sldl's input-named
+    # subfolders), including the pinned output itself if it already exists --
+    # newest first, so the most recent record of each track wins.
+    candidates = sorted(
+        set(dropbox.glob("_index.csv")) | set(dropbox.glob("*/_index.csv")),
+        key=lambda p: p.stat().st_mtime,
+        reverse=True,
+    )
+    if not candidates:
+        print(f"[merge-sldl-indexes] no _index.csv found under {dropbox}; nothing to seed")
+        return 0
+
+    merged: dict[tuple, dict] = {}
+    recovered = 0
+    for path in candidates:
+        try:
+            rows = read_rows(path)
+        except (OSError, csv.Error) as exc:
+            print(f"[merge-sldl-indexes] skipping unreadable {path}: {exc}", file=sys.stderr)
+            continue
+        print(f"[merge-sldl-indexes] {path}: {len(rows)} rows")
+        for row in rows:
+            key = norm_key(row)
+            kept = merged.get(key)
+            if kept is None:
+                merged[key] = row
+            elif kept["state"] == FAILED_STATE and row["state"] in DOWNLOADED_STATES:
+                # Newest attempt failed but an older index proves we already
+                # have this track: keep the newest identity fields, take the
+                # old filepath (informational only; skip-mode index never
+                # checks the file on disk), and mark it downloaded.
+                kept["filepath"] = row["filepath"]
+                kept["state"] = "3"
+                kept["failurereason"] = "0"
+                recovered += 1
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = out_path.with_suffix(".csv.tmp")
+    with open(tmp, "w", newline="", encoding="utf-8") as fh:
+        writer = csv.DictWriter(fh, fieldnames=HEADER)
+        writer.writeheader()
+        writer.writerows(merged.values())
+    tmp.replace(out_path)
+
+    downloaded = sum(1 for r in merged.values() if r["state"] in DOWNLOADED_STATES)
+    print(
+        f"[merge-sldl-indexes] wrote {out_path}: {len(merged)} tracks "
+        f"({downloaded} downloaded, {recovered} recovered from older indexes)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pipeline/lib/notify-telegram.sh

@@ -5,13 +5,30 @@
 # Usage:
 #   echo "single line message" | notify-telegram.sh
 #   notify-telegram.sh "single line message"
-#   notify-telegram.sh < ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
+#   notify-telegram.sh --html < ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
 #
-# Telegram messages are capped at 4096 chars. Anything longer is truncated
+# --html sends with parse_mode=HTML for messages authored as Telegram-HTML
-# with a "...<truncated>" tail. Returns exit 0 on send-OK, non-zero otherwise.
+# (pipeline-status.sh's digest: <b>/<i>/<code>/<pre>/<blockquote>). The
+# CALLER is responsible for escaping &, <, > in any free text; this script
+# only guarantees the truncation below can't cut a tag in half. Without the
+# flag, messages go as plain text exactly as before.
+#
+# Telegram messages are capped at 4096 chars. Anything longer is truncated;
+# in HTML mode the cut lands on a line boundary and re-closes an open <pre>
+# so the truncated message still parses (a mid-tag cut, or a bare "<" in the
+# tail marker, makes the Bot API reject the ENTIRE message with a 400).
+# Returns exit 0 on send-OK, non-zero otherwise.
 
 set -euo pipefail
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# /opt/venv/bin leads PATH for consistency with the other pipeline scripts,
+# even though this one only shells out to curl.
+PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
+PARSE_MODE=""
+if [[ "${1:-}" == "--html" ]]; then
+  PARSE_MODE="HTML"
+  shift
+fi
 
 CONFIG="${ALEMBIC_CONFIG_DIR:-/config}/pipeline/telegram/notify.env"
 if [[ ! -f "$CONFIG" ]]; then
@@ -30,24 +47,37 @@ else
   MSG=$(cat)
 fi
 
-# Telegram caps at 4096 chars (UTF-8 codepoints). Trim conservatively at 3900.
+# Telegram caps at 4096 chars. Trim conservatively at 3900. Use python for
-# Use python for proper UTF-8 length handling.
+# proper UTF-8 length handling. Plain mode appends a literal marker; HTML
+# mode cuts at the last newline inside the budget (our tags never span
+# lines except <pre> blocks) and re-closes an unbalanced <pre>.
 MSG=$(python3 -c "
 import sys
 s = sys.argv[1]
+html = sys.argv[2] == 'HTML'
 if len(s) > 3900:
-    s = s[:3900] + '\n...<truncated>'
+    s = s[:3900]
+    if html:
+        cut = s.rfind('\n')
+        if cut > 0:
+            s = s[:cut]
+        if s.count('<pre>') > s.count('</pre>'):
+            s += '</pre>'
+        s += '\n<i>… truncated</i>'
+    else:
+        s += '\n...(truncated)'
 print(s, end='')
-" "$MSG")
+" "$MSG" "${PARSE_MODE:-plain}")
 
-# Send via Bot API. Disable web-page preview and use plain text (no parse_mode)
+# Send via Bot API. Preview disabled; parse_mode only when requested so log
-# so log content with special chars doesn't get interpreted as Markdown.
+# content with special chars can't be misread as markup in plain sends.
 # `|| true`: a curl timeout/network error must fall through to the explicit
 # ok-check below (which reports "send failed" and exits 2), not abort here.
 resp=$(curl -s --max-time 15 -X POST \
   "https://api.telegram.org/bot${TG_BOT_TOKEN}/sendMessage" \
   --data-urlencode "chat_id=${TG_CHAT_ID}" \
   --data-urlencode "text=${MSG}" \
+  ${PARSE_MODE:+--data-urlencode "parse_mode=${PARSE_MODE}"} \
   --data-urlencode "disable_web_page_preview=true" || true)
 
 ok=$(echo "$resp" | python3 -c "import sys,json;print(json.load(sys.stdin).get('ok',False))" 2>/dev/null || echo False)

pipeline/lib/pipeline-status.sh

@@ -4,9 +4,24 @@
 # Writes a one-screen summary to ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log (overwritten daily)
 # and ships it to Telegram via notify-telegram.sh.
 #
+# Formatting note: the digest uses Telegram HTML entities (<b>, <i>, <code>,
+# <blockquote>) — the brand's chrome/dot-badge language translated into what
+# Telegram can actually render (no color, no custom font, so status "dots"
+# become colored circle emoji). Deliberately NO <pre>/monospace column layout:
+# a code block forces fixed-width columns that wrap into unreadable garbage on
+# a phone ("54 tracks in / M3U"). Instead every row is plain reflowing text —
+# a bold label + " — value" — so it wraps cleanly at any screen width. This
+# REQUIRES notify-telegram.sh to be called with --html (parse_mode=HTML) —
+# sent as plain text the tags would show up literally. All free-text going
+# through mark_ok/mark_warn/mark_skip/section is escaped (esc()) for &, <, >
+# so a stray angle bracket in a log line can't break the HTML parse and
+# swallow the whole message.
+#
 # Reports on:
-#   - Sibling service reachability (slskd, navidrome) via HTTP, not docker ps —
+#   - Sibling service reachability (navidrome) via HTTP, not docker ps —
-#     alembic has no Docker socket access
+#     alembic has no Docker socket access. slskd is deliberately NOT checked:
+#     it is not part of the pipeline (sldl is its own Soulseek client) — it
+#     runs on the host purely as the user's own file-sharing presence.
 #   - Today's runs: playlist syncs, Bandcamp sync, manual imports, dedup
 #   - Weekly maintenance freshness: strip-mb-tags, strip-watermark-art,
 #     scrub-watermark-text, clean-sldl-index, spotify-genre
@@ -27,7 +42,10 @@
 # the opposite of what a health report should do. It writes no library state,
 # so there is nothing to leave half-applied.
 set -u
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# /opt/venv/bin first: `beet` lives in the app venv. Without it every beet
+# probe here ("added today", "Library by format", the mp3-now count) silently
+# came up empty behind its 2>/dev/null.
+PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 
 OUT=${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
 TODAY=$(date +%Y%m%d)
@@ -44,12 +62,39 @@ OK=0
 WARN=0
 SKIP=0
 
-mark_ok()   { OK=$((OK+1));     printf "  ✓ %s\n" "$*"; }
+# Escapes &, <, > so free text (log snippets, exception messages, filenames)
-mark_warn() { WARN=$((WARN+1)); printf "  ⚠ %s\n" "$*"; }
+# can't be mistaken for an HTML entity by Telegram's parser.
-mark_skip() { SKIP=$((SKIP+1)); printf "  ⊘ %s\n" "$*"; }
+esc() {
-mark_info() {                   printf "    %s\n" "$*"; }
+  local s=$1
+  s=${s//&/&amp;}
+  s=${s//</&lt;}
+  s=${s//>/&gt;}
+  printf '%s' "$s"
+}
 
-section() { printf "\n▎ %s\n" "$*"; }
+# One status row: a colored dot, a bold label, and (optionally) " — value".
+# Plain proportional text — NO padding, NO monospace — so it reflows on mobile
+# instead of wrapping mid-column. The dot is the brand's badge-state color,
+# emoji being Telegram's only color channel.
+_row() {
+  local dot=$1 label=$2 detail=${3:-}
+  if [[ -n "$detail" ]]; then
+    printf "%s <b>%s</b> — %s\n" "$dot" "$(esc "$label")" "$(esc "$detail")"
+  else
+    printf "%s <b>%s</b>\n" "$dot" "$(esc "$label")"
+  fi
+}
+mark_ok()   { OK=$((OK+1));     _row "🟢" "$@"; }
+mark_warn() { WARN=$((WARN+1)); _row "🟠" "$@"; }
+mark_skip() { SKIP=$((SKIP+1)); _row "⚪" "$@"; }
+# Free-text info line (no dot, no counter) — for sub-breakdowns.
+mark_info() { printf "   %s\n" "$(esc "$*")"; }
+
+# Bold section header (reads like the web app's <h2>). No <pre> — the rows
+# under it are plain reflowing text.
+section() {
+  printf "\n<b>▎ %s</b>\n" "$(esc "$*")"
+}
 
 # syslog / `logger` is not present in the container image. Only call it if it
 # exists, so these lines don't spew "logger: command not found" into the job
@@ -70,17 +115,6 @@ human_age() {
   fi
 }
 
-# Newest matching log → "<age_seconds>|<path>", empty if no match.
-newest_log() {
-  local glob="$1"
-  local newest
-  newest=$(ls -1t $glob 2>/dev/null | head -1)
-  [[ -z "$newest" ]] && return
-  local mtime
-  mtime=$(stat -c %Y "$newest" 2>/dev/null) || return
-  echo "$((NOW_TS - mtime))|$newest"
-}
-
 # Per-playlist log status.
 playlist_status() {
   local log="$1"
@@ -91,7 +125,7 @@ playlist_status() {
     m3u=$(awk 'match($0, /updated with [0-9]+ tracks/) {
                  n=substr($0, RSTART, RLENGTH); gsub(/[^0-9]/, "", n); last=n
                } END { print last }' "$log")
-    echo "OK|${m3u:-0} tracks in M3U"
+    echo "OK|${m3u:-0} tracks"
   elif grep -q "=== Starting playlist run" "$log" 2>/dev/null; then
     echo "WARN|started but never finished"
   else
@@ -151,12 +185,11 @@ dedup_today_status() {
   check_http() {
     local name="$1" url="$2"
     if curl -s -o /dev/null --connect-timeout 3 --max-time 6 "$url"; then
-      mark_ok "$(printf '%-11s reachable' "$name")"
+      mark_ok "$name" "reachable"
     else
-      mark_warn "$(printf '%-11s unreachable at %s' "$name" "$url")"
+      mark_warn "$name" "unreachable at $url"
     fi
   }
-  check_http slskd   "http://gluetun:5030/"
   check_http navidrome "http://navidrome:4533/rest/ping.view"
 
   # 2. Today's runs
@@ -197,14 +230,12 @@ dedup_today_status() {
         fi
         ;;
     esac
-    line=$(printf '%-13s %s' "$short" "$detail")
+    if [[ "$tag" == "OK" ]]; then mark_ok "$short" "$detail"; else mark_warn "$short" "$detail"; fi
-    if [[ "$tag" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
   done
   if [[ -n "$newest_dedup_today" ]]; then
     any_today=1
     IFS='|' read -r tag detail < <(dedup_today_status "$newest_dedup_today")
-    line=$(printf '%-13s %s' "dedup" "$detail")
+    if [[ "$tag" == "OK" ]]; then mark_ok "dedup" "$detail"; else mark_warn "dedup" "$detail"; fi
-    if [[ "$tag" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
   fi
   [[ "$any_today" -eq 0 ]] && mark_info "(nothing has run yet today)"
 
@@ -216,30 +247,28 @@ dedup_today_status() {
   ADDED_TODAY=$(beet ls -f '$grouping' \
     "added:$(date +%Y-%m-%d).." 2>/dev/null || true)
   added_total=$(echo -n "$ADDED_TODAY" | grep -c '^' || true)
-  mark_ok "$(printf '%-13s %d new tracks in beets' 'added today' "$added_total")"
+  mark_ok "added today" "$added_total new tracks in beets"
   if [[ "$added_total" -gt 0 ]]; then
     # Per-playlist breakdown. Empty $grouping (bandcamp/manual) shown as "(none)".
+    # Piped through the same &/</> escaping as esc(), since this prints
+    # straight into the message without going through _row.
     echo "$ADDED_TODAY" \
       | awk '{ if ($0 == "") print "(none)"; else print }' \
       | sort | uniq -c | sort -rn \
-      | awk '{ g=$2; for(i=3;i<=NF;i++) g=g" "$i; printf "                  %3d  %s\n", $1, g }'
+      | awk '{ c=$1; g=$2; for(i=3;i<=NF;i++) g=g" "$i; printf "   • %s (%d)\n", g, c }' \
+      | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g'
   fi
 
   # 3. Maintenance freshness
-  # Each weekly/monthly task: find its newest log and check age.
+  # Read from the app's job_runs table (the scheduler's own record) instead of
+  # globbing for log files. Jobs run through pipeline_runner write their logs
+  # to logs/<job_key>/<timestamp>.log, so the old cron-era filename globs
+  # matched nothing for jobs that don't also write their own log, and healthy
+  # jobs were reported as "never run yet". The DB also distinguishes lock
+  # skips and failures, which a missing log file can't.
   # Limits: weekly tasks should be <9 days old; daily <2; monthly <35.
   section "Maintenance (last run)"
 
-  declare -A MAINT_GLOB=(
-    [strip-mb-tags]="${ALEMBIC_CONFIG_DIR:-/config}/logs/mb-strip-*.log"
-    [strip-watermark-art]="${ALEMBIC_CONFIG_DIR:-/config}/logs/strip-watermark-*.log"
-    [scrub-watermark-text]="${ALEMBIC_CONFIG_DIR:-/config}/logs/scrub-text-*.log"
-    [clean-sldl-index]="${ALEMBIC_CONFIG_DIR:-/config}/logs/clean-index-*.log"
-    [clear-bad-genres]="${ALEMBIC_CONFIG_DIR:-/config}/logs/clear-bad-genres-*.log"
-    [spotify-genre]="${ALEMBIC_CONFIG_DIR:-/config}/logs/spotify-genre-*.log"
-    [dedup-library]="${ALEMBIC_CONFIG_DIR:-/config}/logs/dedup-*.log"
-    [upgrade-mp3-to-flac]="${ALEMBIC_CONFIG_DIR:-/config}/logs/upgrade-mp3-*.log"
-  )
   declare -A MAINT_LIMIT=(
     [strip-mb-tags]=$((9*86400))
     [strip-watermark-art]=$((9*86400))
@@ -250,45 +279,110 @@ dedup_today_status() {
     [dedup-library]=$((2*86400))
     [upgrade-mp3-to-flac]=$((35*86400))
   )
+  # Emits one line per task: task|latest_status|latest_age_s|success_age_s|success_log
+  # (ages are -1 when there is no such run). Read-only DB open; prints nothing
+  # if the DB is missing so every task falls through to "never run yet".
+  maint_rows=$(python3 - "${ALEMBIC_CONFIG_DIR:-/config}/alembic.db" <<'PYEOF'
+import sqlite3
+import sys
+import time
+
+# Digest label -> job_runs.job_key. The scheduled genre refresh records under
+# genre:run (via genre_review_service) and dedup under dedup:scan (via
+# dedup_review_service); everything else is its MAINTENANCE_JOBS key.
+KEYS = [
+    ("strip-mb-tags", "maintenance:strip_mb_tags"),
+    ("strip-watermark-art", "maintenance:strip_watermark_art"),
+    ("scrub-watermark-text", "maintenance:scrub_watermark_text"),
+    ("clean-sldl-index", "maintenance:clean_sldl_index"),
+    ("clear-bad-genres", "maintenance:clear_bad_genres"),
+    ("spotify-genre", "genre:run"),
+    ("dedup-library", "dedup:scan"),
+    ("upgrade-mp3-to-flac", "maintenance:upgrade_mp3_to_flac"),
+]
+now = time.time()
+try:
+    conn = sqlite3.connect(f"file:{sys.argv[1]}?mode=ro", uri=True)
+    conn.execute("SELECT 1 FROM job_runs LIMIT 1")
+except sqlite3.Error:
+    sys.exit(0)
+for task, key in KEYS:
+    latest = conn.execute(
+        "SELECT status, started_at FROM job_runs WHERE job_key = ? ORDER BY started_at DESC LIMIT 1",
+        (key,),
+    ).fetchone()
+    if latest is None:
+        print(f"{task}|none|-1|-1|")
+        continue
+    success = conn.execute(
+        "SELECT started_at, log_path FROM job_runs WHERE job_key = ? AND status = 'success' "
+        "ORDER BY started_at DESC LIMIT 1",
+        (key,),
+    ).fetchone()
+    s_age = int(now - success[0]) if success else -1
+    s_log = (success[1] or "") if success else ""
+    print(f"{task}|{latest[0]}|{int(now - latest[1])}|{s_age}|{s_log}")
+PYEOF
+)
+
   for task in strip-mb-tags strip-watermark-art scrub-watermark-text clean-sldl-index clear-bad-genres spotify-genre dedup-library upgrade-mp3-to-flac; do
-    res=$(newest_log "${MAINT_GLOB[$task]}")
+    row=$(grep "^${task}|" <<< "$maint_rows" || true)
-    if [[ -z "$res" ]]; then
+    IFS='|' read -r _ latest_status latest_age age_s log <<< "$row"
-      mark_skip "$(printf '%-22s never run yet' "$task")"
+    if [[ -z "$row" || "$latest_status" == "none" ]]; then
+      mark_skip "$task" "never run yet"
+      continue
+    fi
+    if [[ "$age_s" -lt 0 ]]; then
+      # attempted, but never once succeeded -- say what the last attempt did
+      case "$latest_status" in
+        skipped_lock) note="skipped, lock busy" ;;
+        running)      note="running now" ;;
+        *)            note="$latest_status" ;;
+      esac
+      mark_warn "$task" "never succeeded (last attempt $(human_age "$latest_age"): $note)"
       continue
     fi
-    age_s=${res%%|*}
-    log=${res##*|}
     age_h=$(human_age "$age_s")
-    case "$task" in
+    snip=""
-      spotify-genre)
+    if [[ -n "$log" && -f "$log" ]]; then
-        snip=$(grep -oE "written: [0-9]+, unchanged: [0-9]+" "$log" | tail -1) ;;
+      case "$task" in
-      clear-bad-genres)
+        spotify-genre)
-        snip=$(grep -oE "(would-clear|cleared): [0-9]+ tracks" "$log" | tail -1) ;;
+          snip=$(grep -oE "written: [0-9]+, unchanged: [0-9]+" "$log" | tail -1) ;;
-      clean-sldl-index)
+        clear-bad-genres)
-        snip=$(grep -oE "[0-9]+ m3us, total [0-9]+ kept, [0-9]+ dropped" "$log" | tail -1) ;;
+          snip=$(grep -oE "rewrote: [0-9]+ tracks, blanked: [0-9]+ tracks|(would-clear|cleared): [0-9]+ tracks" "$log" | tail -1) ;;
-      strip-watermark-art)
+        clean-sldl-index)
-        snip=$(grep -oE "stripped from [0-9]+ tracks|no images shared" "$log" | tail -1) ;;
+          snip=$(grep -oE "[0-9]+ m3us, total [0-9]+ kept, [0-9]+ dropped" "$log" | tail -1) ;;
-      scrub-watermark-text)
+        strip-watermark-art)
-        snip=$(grep -oE "stripped [0-9]+ frame\(s\) across [0-9]+ file" "$log" | tail -1) ;;
+          snip=$(grep -oE "stripped from [0-9]+ tracks|no images shared" "$log" | tail -1) ;;
-      strip-mb-tags)
+        scrub-watermark-text)
-        snip=$(grep -oE "Done\. Log:" "$log" | head -1 | sed 's/Done\. Log:/done/') ;;
+          snip=$(grep -oE "stripped [0-9]+ frame\(s\) across [0-9]+ file" "$log" | tail -1) ;;
-      dedup-library)
+        strip-mb-tags)
-        snip=$(grep "=== Summary:" "$log" | tail -1 \
+          snip=$(grep -oE "Done\. Log:" "$log" | head -1 | sed 's/Done\. Log:/done/') ;;
-               | sed -E 's/.*=== Summary: //; s/ ===.*//; s/ \([^)]*\)//') ;;
+        dedup-library)
-      upgrade-mp3-to-flac)
+          snip=$(grep "=== Summary:" "$log" | tail -1 \
-        # The log's "to attempt" count is frozen at the last monthly run, so on
+                 | sed -E 's/.*=== Summary: //; s/ ===.*//; s/ \([^)]*\)//') ;;
-        # its own it looks stale against the live "Library by format" section.
+        upgrade-mp3-to-flac)
-        # Pair it with the current format:mp3 count so the drop (upgrades that
+          # The log's "to attempt" count is frozen at the last monthly run, so on
-        # have landed since) is visible instead of looking like a mismatch.
+          # its own it looks stale against the live "Library by format" section.
-        attempted=$(grep -oE "[0-9]+ MP3 tracks to attempt" "$log" | grep -oE "^[0-9]+" | tail -1)
+          # Pair it with the current format:mp3 count so the drop (upgrades that
-        mp3_now=$(beet ls 'format:mp3' 2>/dev/null | grep -c '^')
+          # have landed since) is visible instead of looking like a mismatch.
-        snip="${attempted:-?} attempted · ${mp3_now} MP3 now" ;;
+          attempted=$(grep -oE "[0-9]+ MP3 tracks to attempt" "$log" | grep -oE "^[0-9]+" | tail -1)
+          mp3_now=$(beet ls 'format:mp3' 2>/dev/null | grep -c '^')
+          snip="${attempted:-?} attempted · ${mp3_now} MP3 now" ;;
+      esac
+    fi
+    # Age/snippet describe the last SUCCESS; if a newer attempt failed or was
+    # lock-skipped, say so rather than hiding it behind the healthy line.
+    case "$latest_status" in
+      failed)       snip="${snip:+$snip, }latest attempt failed" ;;
+      skipped_lock) snip="${snip:+$snip, }latest attempt skipped, lock busy" ;;
     esac
-    line=$(printf '%-22s %-9s %s' "$task" "$age_h" "${snip:-}")
+    detail="$age_h"
-    if (( age_s > MAINT_LIMIT[$task] )); then
+    [[ -n "$snip" ]] && detail="$age_h · $snip"
-      mark_warn "$line"
+    if (( age_s > MAINT_LIMIT[$task] )) || [[ "$latest_status" == "failed" ]]; then
+      mark_warn "$task" "$detail"
     else
-      mark_ok "$line"
+      mark_ok "$task" "$detail"
     fi
   done
 
@@ -300,13 +394,16 @@ dedup_today_status() {
     exp=$(awk -F'\t' '$6=="identity" {print $5; exit}' ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)
     if [[ -n "${exp:-}" && "$exp" =~ ^[0-9]+$ && "$exp" -gt 0 ]]; then
       days=$(( (exp - NOW_TS) / 86400 ))
-      line=$(printf '%-22s %d days left' "Bandcamp cookie" "$days")
+      if (( days < 14 )); then
-      if (( days < 14 )); then mark_warn "$line — re-export soon"; else mark_ok "$line"; fi
+        mark_warn "Bandcamp cookie" "$days days left — re-export soon"
+      else
+        mark_ok "Bandcamp cookie" "$days days left"
+      fi
     else
-      mark_warn "Bandcamp cookie         could not parse expiry"
+      mark_warn "Bandcamp cookie" "could not parse expiry"
     fi
   else
-    mark_warn "Bandcamp cookie         missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)"
+    mark_warn "Bandcamp cookie" "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)"
   fi
 
   # Qobuz Web Player token (buy-link enrich cascade #2). The token is opaque
@@ -326,12 +423,20 @@ dedup_today_status() {
       -H "X-App-Id: $q_appid" -H "X-User-Auth-Token: $q_token" -w '%{http_code}' \
       "https://www.qobuz.com/api.json/0.2/track/search?query=test&limit=1&app_id=$q_appid" 2>/dev/null)
     case "$q_code" in
-      200) mark_ok   "$(printf '%-22s %s' 'Qobuz token' 'valid (buy-link lookup live)')" ;;
+      200) mark_ok   "Qobuz token" "valid (buy-link lookup live)" ;;
-      401) mark_warn "$(printf '%-22s %s' 'Qobuz token' "EXPIRED — re-export X-User-Auth-Token to ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token")" ;;
+      401) mark_warn "Qobuz token" "EXPIRED — re-export X-User-Auth-Token to ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token" ;;
-      *)   mark_warn "$(printf '%-22s %s' 'Qobuz token' "check failed (HTTP ${q_code:-none})")" ;;
+      # A genuinely expired/bad token gets a JSON 401 from Qobuz's own API.
+      # 403 instead means the request never reached that code at all -- Qobuz's
+      # Akamai edge is rejecting the request outright (seen 2026-07-22: the
+      # exact same request got this "Access Denied"/edgesuite.net block from
+      # alembic's VPN egress IP, but a clean 401 from a non-VPN IP). Re-
+      # exporting the token does nothing for that -- it's the egress IP's
+      # reputation, not the credential. Say so, so it isn't mistaken for 401.
+      403) mark_warn "Qobuz token" "blocked (HTTP 403, likely Akamai/CDN, not the token) — VPN egress IP may be flagged; re-exporting the token won't fix this" ;;
+      *)   mark_warn "Qobuz token" "check failed (HTTP ${q_code:-none})" ;;
     esac
   else
-    mark_warn "$(printf '%-22s %s' 'Qobuz token' "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token)")"
+    mark_warn "Qobuz token" "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token)"
   fi
 
   # VPN egress
@@ -341,17 +446,20 @@ dedup_today_status() {
   # asking Docker to introspect gluetun's health, and needs no Docker socket.
   egress_ip=$(curl -s --connect-timeout 5 --max-time 10 https://ifconfig.me 2>/dev/null)
   if [[ -n "$egress_ip" ]]; then
-    mark_ok "$(printf '%-22s %s' 'VPN egress' "$egress_ip")"
+    mark_ok "VPN egress" "$egress_ip"
   else
-    mark_warn "$(printf '%-22s %s' 'VPN egress' 'could not determine egress IP — VPN may be down')"
+    mark_warn "VPN egress" "could not determine egress IP — VPN may be down"
   fi
 
   # 5. Storage
   section "Storage"
   while read -r mp pct used total; do
-    line=$(printf '%-22s %s used (%s of %s)' "$mp" "$pct" "$used" "$total")
     pct_n=${pct%\%}
-    if (( pct_n > 90 )); then mark_warn "$line"; else mark_ok "$line"; fi
+    if (( pct_n > 90 )); then
+      mark_warn "$mp" "$pct used ($used of $total)"
+    else
+      mark_ok "$mp" "$pct used ($used of $total)"
+    fi
   done < <(df -h ${MUSIC_DATA_DIR:-/data/music} / 2>/dev/null | awk 'NR>1 && $1!~/tmpfs/ {print $6, $5, $3, $2}')
 
   # 6. Navidrome
@@ -376,24 +484,40 @@ except Exception as e:
     print(f\"WARN|unreachable: {e}\")
 " 2>/dev/null)
   state=${ND_LINE%%|*}; detail=${ND_LINE##*|}
-  line=$(printf '%-22s %s' "Navidrome" "$detail")
+  if [[ "$state" == "OK" ]]; then mark_ok "Navidrome" "$detail"; else mark_warn "Navidrome" "$detail"; fi
-  if [[ "$state" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
 
-  # 7. Library by format (info-only, no status pill)
+  # 7. Library by format (info-only, no status dot)
   section "Library by format"
   beet ls -f '$format' 2>/dev/null | sort | uniq -c | sort -rn \
-    | awk '{printf "    %-6s %d tracks\n", $2, $1}'
+    | awk '{printf "   • %s — %d tracks\n", $2, $1}' \
+    | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g'
 
 } > "$OUT"
 
 # ---- Build header banner and prepend ----
+# Bold brand mark (⚗️ is literally the "ALEMBIC" unicode glyph — no generic
+# music-note stand-in needed), a hostname chip in <code>, and an italic tally
+# that reuses the same 🟢/🟠/⚪ dots as the body. When something needs
+# attention, a <blockquote> callout mirrors the web app's .notice.warning
+# panel; an all-clear run gets the calm .notice.success equivalent instead.
 HOSTNAME_SHORT=$(hostname -s)
 DATE_HUMAN=$(TZ="${TZ:-UTC}" date '+%a %Y-%m-%d %H:%M %Z')
-BANNER_TITLE="🎵 Music pipeline · $HOSTNAME_SHORT · $DATE_HUMAN"
+HEADER="⚗️ <b>Alembic</b> · music pipeline · <code>${HOSTNAME_SHORT}</code> · ${DATE_HUMAN}
-BANNER_TALLY="   $OK OK · $WARN warn · $SKIP skip"
+<i>🟢 ${OK} OK   🟠 ${WARN} warn   ⚪ ${SKIP} skip</i>"
-sed -i "1c\\
+if (( WARN > 0 )); then
-${BANNER_TITLE}\\
+  HEADER="${HEADER}
-${BANNER_TALLY}" "$OUT"
+<blockquote>🟠 <b>${WARN} issue(s)</b> need attention — see below</blockquote>"
+else
+  HEADER="${HEADER}
+<blockquote>🟢 All clear — nothing needs attention.</blockquote>"
+fi
+
+# Swap the __HEADER_PLACEHOLDER__ line for the (2-3 line) banner above. A
+# straight prepend, not sed 1c — the banner's line count varies with the
+# blockquote, which sed's `c` command can't take as a variable easily.
+tail -n +2 "$OUT" > "${OUT}.body"
+{ printf '%s\n' "$HEADER"; cat "${OUT}.body"; } > "$OUT"
+rm -f "${OUT}.body"
 
 # Always mirror the one-line summary to syslog so journalctl shows last-run
 # state even if Telegram is broken. Warnings get a separate WARN tag.
@@ -402,10 +526,12 @@ if (( WARN > 0 )); then
   slog "WARN: pipeline-status flagged $WARN issue(s) — see $OUT"
 fi
 
-# Send to Telegram. If it fails, log the failure to syslog so the absence of
+# Send to Telegram with --html: the digest is Telegram-HTML (see header
-# a message in the chat has a corresponding journal entry to grep for.
+# comment), so notify-telegram.sh must send it with parse_mode=HTML. If the
+# send fails, log to syslog so the absence of a message in the chat has a
+# corresponding journal entry to grep for.
 if [[ -x ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh ]]; then
-  if ! ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh < "$OUT" 2>/tmp/tg-err; then
+  if ! ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh --html < "$OUT" 2>/tmp/tg-err; then
     slog "WARN: Telegram send failed: $(cat /tmp/tg-err 2>/dev/null | head -c 200)"
     rm -f /tmp/tg-err
   fi

pipeline/lib/replace-with-better.sh

@@ -195,7 +195,18 @@ while IFS= read -r -d '' new_file; do
   existing_id="${match%%$'\x1f'*}"
   existing_container="${match#*$'\x1f'}"
 
-  existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  # beets displays real ${MUSIC_DATA_DIR}/Library/... paths since the
+  # 2026-07-08 path rewrite -- use them as-is; only a legacy /music/...
+  # display path still needs the prefix swap. Blindly prepending (the old
+  # behavior) doubled the prefix, so every match logged [stale-db], the MP3
+  # never got replaced, and the leftover-import step at the bottom imported
+  # the staged FLAC as a NEW track -- producing exactly the flac+mp3 twins
+  # this script exists to prevent.
+  if [[ "$existing_container" == /music/* ]]; then
+    existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  else
+    existing="$existing_container"
+  fi
   if [[ ! -f "$existing" ]]; then
     echo "[stale-db]       beets has $existing_container but file missing" | tee -a "$LOG"
     continue

pipeline/lib/scrub-watermark-text.py

@@ -29,7 +29,7 @@ Usage:
   scrub-watermark-text.py                # dry run
   scrub-watermark-text.py --apply        # actually strip
 """
-import sys, re, argparse
+import os, sys, re, argparse
 from pathlib import Path
 from mutagen import File as MFile
 from mutagen.id3 import ID3, ID3NoHeaderError

pipeline/lib/spotify-playlist-csv.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""
+spotify-playlist-csv.py — dump a Spotify playlist to a CSV sldl can search from.
+
+sldl's own vendored Spotify client still calls Spotify's GET /playlists/{id}/tracks
+endpoint, which Spotify removed in its February 2026 API changes (see
+https://developer.spotify.com/documentation/web-api/references/changes/february-2026)
+in favor of /items. Grandfathered apps still get a pass on /tracks for now, but
+apps created after that change get a hard 403 there regardless of auth method --
+client-credentials, or a correctly-scoped, correctly-owned OAuth user token, none
+of it matters, because the endpoint itself is gone for them. sldl has no separate
+code path to fall back to /items, so it can never read a playlist for a new app
+no matter what alembic hands it.
+
+Rather than depend on sldl's Spotify client at all, this script reads the
+playlist itself (via /items, which alembic's own code already migrated to) and
+writes the same Artist,Title,Album,Length CSV format upgrade-mp3-to-flac.sh
+already feeds to sldl via --input-type csv. run-playlist.sh runs this first and
+then points sldl at the CSV instead of the playlist URL, sidestepping sldl's
+Spotify client entirely -- for grandfathered and new apps alike.
+
+Usage:
+  spotify-playlist-csv.py <playlist_url> <out_csv>
+
+Credentials are read from env vars SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET.
+SPOTIFY_REFRESH_TOKEN, if set, mints a user token instead of client-credentials
+(required for newly created Spotify apps -- see _spotify_auth.get_token).
+"""
+import csv
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.request
+
+from _spotify_auth import get_token
+
+API = "https://api.spotify.com/v1"
+
+
+def fetch_playlist(url: str, token: str) -> list[dict]:
+    m = re.search(r"playlist[/:]([A-Za-z0-9]+)", url)
+    if not m:
+        sys.exit(f"Could not parse playlist ID from {url!r}")
+    pid = m.group(1)
+    tracks = []
+    # No `fields` filter: Spotify's February 2026 changes renamed each entry's
+    # payload from "track" to "item" for apps on the new behavior (extended
+    # quota / grandfathered apps keep "track"), and a fields filter selects by
+    # key name -- filtering on track(...) silently returns EMPTY pages on the
+    # renamed shape. Fetch unfiltered and parse both key names below.
+    next_url = f"{API}/playlists/{pid}/items?limit=100"
+    while next_url:
+        req = urllib.request.Request(next_url, headers={"Authorization": f"Bearer {token}"})
+        try:
+            with urllib.request.urlopen(req, timeout=15) as r:
+                data = json.loads(r.read())
+        except urllib.error.HTTPError as e:
+            if e.code in (403, 404):
+                sys.exit(
+                    f"Spotify returned {e.code} loading this playlist. The connected "
+                    "Spotify account can't see it: it must own the playlist, be a "
+                    "collaborator on it, or (for grandfathered apps) the playlist "
+                    "must be Public. Check which account is connected via "
+                    "/connect/spotify."
+                )
+            raise
+        if "items" not in data:
+            # New-behavior apps get metadata only (no items field) for
+            # playlists the connected account doesn't own or collaborate on.
+            sys.exit(
+                "Spotify returned this playlist without its contents. For newly "
+                "created Spotify apps, the connected account must OWN the playlist "
+                "or be a collaborator on it (public is no longer enough). Ask the "
+                "owner to make it collaborative and add you, or recreate the "
+                "playlist under the connected account."
+            )
+        for item in data["items"]:
+            t = item.get("track") or item.get("item")
+            if not t or t.get("is_local"):
+                continue
+            # All artists, comma-joined -- matches what sldl's own Spotify
+            # extractor fed its search, so multi-artist tracks match no worse
+            # than they did before the CSV detour.
+            artists = [a["name"] for a in (t.get("artists") or []) if a.get("name")]
+            tracks.append(
+                {
+                    "artist": ", ".join(artists),
+                    "title": t.get("name", ""),
+                    "album": (t.get("album") or {}).get("name", ""),
+                    "length": round((t.get("duration_ms") or 0) / 1000),
+                }
+            )
+        next_url = data.get("next")
+    return tracks
+
+
+def main() -> int:
+    if len(sys.argv) != 3:
+        sys.exit(f"Usage: {sys.argv[0]} <playlist_url> <out_csv>")
+    url, out_path = sys.argv[1], sys.argv[2]
+
+    cid = os.environ.get("SPOTIFY_CLIENT_ID")
+    csec = os.environ.get("SPOTIFY_CLIENT_SECRET")
+    refresh = os.environ.get("SPOTIFY_REFRESH_TOKEN") or None
+    if not cid or not csec:
+        sys.exit("SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET must be set")
+
+    token = get_token(cid, csec, refresh)
+    tracks = fetch_playlist(url, token)
+
+    with open(out_path, "w", newline="") as f:
+        writer = csv.writer(f)
+        writer.writerow(["Artist", "Title", "Album", "Length"])
+        for t in tracks:
+            writer.writerow([t["artist"], t["title"], t["album"], t["length"]])
+
+    print(f"[spotify-playlist-csv] wrote {len(tracks)} tracks to {out_path}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pipeline/lib/spotify-retag.py

@@ -56,8 +56,11 @@ def fetch_playlist(url: str, token: str) -> list[dict]:
         )
         with urllib.request.urlopen(req, timeout=15) as r:
             data = json.loads(r.read())
-        for item in data["items"]:
+        # Entries carry "track" (grandfathered apps) or "item" (apps on the
-            t = item.get("track")
+        # February 2026 renamed shape); items is absent entirely when the
+        # connected account can't read the playlist's contents.
+        for item in data.get("items", []):
+            t = item.get("track") or item.get("item")
             if not t or t.get("is_local"):
                 continue
             tracks.append(

pipeline/lib/strip-watermark-art.py

@@ -51,10 +51,17 @@ def extract_flac_picture(flac_path):
     with tempfile.NamedTemporaryFile(delete=False, suffix=".pic") as tf:
         tmp = tf.name
     try:
-        r = subprocess.run(
+        try:
-            ["metaflac", f"--export-picture-to={tmp}", flac_path],
+            r = subprocess.run(
-            capture_output=True, timeout=15
+                ["metaflac", f"--export-picture-to={tmp}", flac_path],
-        )
+                capture_output=True, timeout=60
+            )
+        except subprocess.TimeoutExpired:
+            # A transient IO stall on one file must not abort the whole weekly
+            # run (seen 2026-07-15: a healthy 190KB cover took >15s under disk
+            # contention and the raised TimeoutExpired failed the entire job).
+            print(f"[strip-art] WARN: metaflac timed out on {flac_path}, skipping file")
+            return None
         if r.returncode != 0 or not os.path.exists(tmp):
             return None
         sz = os.path.getsize(tmp)

pipeline/lib/sync-bandcamp.sh

@@ -7,7 +7,15 @@
 # Run as root (cron). Logs to ${ALEMBIC_CONFIG_DIR:-/config}/logs/bandcamp-YYYYMMDD.log.
 
 set -euo pipefail
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# /opt/venv/bin must lead PATH -- that's where `beet` lives (the app's own
+# subprocess env puts it there too, see pipeline_runner._subprocess_env()).
+# This script used to run as a bare root cron job (pre-2026-07-08 cutover to
+# the app scheduler) where a minimal hardened PATH made sense; omitting the
+# venv here silently broke every downstream `beet import` call inside
+# import-track.sh once this started running through the app instead --
+# import-track.sh swallows that failure and still reports OK, so a purchase
+# would sit unimported until someone noticed it missing from Navidrome.
+PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 export PATH
 
 CONFIG="${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/config.env"
@@ -87,7 +95,10 @@ shopt -u dotglob nullglob
 
 # Hand off to the standard manual-import pipeline. No playlist tag — Bandcamp
 # purchases aren't part of any Spotify playlist. The import-track.sh guard,
-# albumartist fallback, beets import, and Navidrome scan all kick in normally.
+# albumartist fallback, beets import, straggler/dedup safety net, and
+# Navidrome scan all kick in normally (import-track.sh runs the
+# replace-with-better + dedup-library safety net itself now, for every
+# caller, not just this one).
 log "Calling import-track.sh"
 if ${PIPELINE_DIR:-/app/pipeline}/bin/import-track.sh >> "$LOG" 2>&1; then
   log "import-track.sh OK"
@@ -95,28 +106,5 @@ else
   log "WARN: import-track.sh exited non-zero (exit $?)"
 fi
 
-# Safety net: if anything is stranded in ${MUSIC_DATA_DIR:-/data/music}/downloads (rare with
-# duplicate_action=keep set in beets config, but possible for files beets
-# couldn't process), find each one's library counterpart and replace if the
-# new file is higher quality, OR import as new if there's no counterpart.
-log "Running replace-with-better safety pass on /downloads stragglers"
-if ${PIPELINE_DIR:-/app/pipeline}/lib/replace-with-better.sh --apply >> "$LOG" 2>&1; then
-  log "replace-with-better OK"
-else
-  log "WARN: replace-with-better.sh exited non-zero (exit $?)"
-fi
-
-# Now dedupe across the library. With duplicate_action=keep beets imported
-# every Bandcamp file even when it conflicts with an existing track at the
-# same path (it creates `.1.ext` siblings). dedup-library.sh's policy is
-# "FLAC > MP3, then largest file" — Bandcamp version wins, soulseek version
-# is removed from both the beets DB and disk.
-log "Running dedup pass (Bandcamp FLAC wins over older Soulseek copies)"
-if ${PIPELINE_DIR:-/app/pipeline}/lib/dedup-library.sh --apply >> "$LOG" 2>&1; then
-  log "dedup OK"
-else
-  log "WARN: dedup-library.sh exited non-zero (exit $?)"
-fi
-
 log "=== Bandcamp sync done ==="
 exit 0

pipeline/lib/upgrade-mp3-to-flac.sh

@@ -20,7 +20,8 @@
 #   upgrade-mp3-to-flac.sh --csv-only  # just write the CSV; don't run sldl
 
 set -euo pipefail
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# /opt/venv/bin must lead PATH -- `beet` (used heavily below) lives there.
+PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 export PATH
 
 # These helpers are the same ones import-track.sh sources and runs under its

tests/test_credential_service.py

@@ -43,22 +43,41 @@ def test_render_playlist_confs_injects_creds_safely(db, config_dir):
     text = conf.read_text()
     assert "user = seek" in text
     assert "pass = p'a$(id)ss" in text
-    assert "spotify-id = CID" in text
+    # run-playlist.sh scrapes the playlist URL from this line to build the CSV
-    assert "spotify-secret = CSECRET" in text
+    assert "input = https://open.spotify.com/playlist/Z" in text
-    # no account connected yet -- rendered blank, not left as the placeholder
+    # Spotify creds must NOT be rendered into confs: sldl never talks to
-    assert "spotify-refresh = \n" in text
+    # Spotify (run-playlist.sh feeds it a CSV); the creds live in _spotify.env
+    assert "spotify-id" not in text
+    assert "spotify-secret" not in text
+    assert "spotify-refresh" not in text
+    assert "CID" not in text
+    assert "CSECRET" not in text
     # rendered config must be owner-only (holds the Soulseek password)
     assert stat.S_IMODE(os.stat(conf).st_mode) == 0o600
 
 
-def test_render_playlist_confs_includes_spotify_refresh_when_connected(db, config_dir):
+def test_spotify_env_renders_creds_and_refresh_token(db, config_dir):
-    cs.set_credentials(db, "spotify", {"client_id": "CID", "client_secret": "CSECRET", "refresh_token": "RTOK"})
+    # _spotify.env is the ONLY rendered home of Spotify creds; run-playlist.sh
-    cs.set_credentials(db, "soulseek", {"username": "seek", "password": "pw"})
+    # sources it to fetch the playlist CSV, so values must be shell-quoted
-    pl.create(db, name="rtest2", spotify_url="https://open.spotify.com/playlist/Z")
+    cs.set_credentials(
+        db, "spotify", {"client_id": "CID", "client_secret": "CS'EC$(id)RET", "refresh_token": "RTOK"}
+    )
 
-    conf = config_dir / "pipeline" / "rtest2.conf"
+    env = config_dir / "pipeline" / "_spotify.env"
-    text = conf.read_text()
+    text = env.read_text()
-    assert "spotify-refresh = RTOK" in text
+    assert "SPOTIFY_CLIENT_ID=CID" in text
+    # shlex.quote keeps a metacharacter-laden secret a single literal value
+    assert "SPOTIFY_CLIENT_SECRET='CS'\"'\"'EC$(id)RET'" in text
+    assert "SPOTIFY_REFRESH_TOKEN=RTOK" in text
+    assert stat.S_IMODE(os.stat(env).st_mode) == 0o600
+
+
+def test_spotify_env_refresh_token_blank_until_connected(db, config_dir):
+    cs.set_credentials(db, "spotify", {"client_id": "CID", "client_secret": "CSECRET"})
+
+    text = (config_dir / "pipeline" / "_spotify.env").read_text()
+    # rendered blank (''), which sources cleanly and stays falsy in bash
+    assert "SPOTIFY_REFRESH_TOKEN=''" in text
 
 
 def test_azuracast_renders_and_clears_base_url(db, config_dir):

tests/test_pipeline_runner.py

@@ -25,11 +25,41 @@ def test_run_job_failure():
     assert run.exit_code == 1
 
 
-def test_lock_skip_when_held():
+def test_manual_run_skips_immediately_when_lock_held():
     async def scenario():
         await pr._lock.acquire()
         try:
-            return await pr.run_job("test:locked", ["true"], use_lock=True)
+            return await pr.run_job("test:locked", ["true"], use_lock=True, triggered_by="manual")
+        finally:
+            pr._lock.release()
+
+    run = _run(scenario())
+    assert run.status == "skipped_lock"
+
+
+def test_scheduled_run_waits_for_lock_by_default(monkeypatch):
+    # scheduled runs must QUEUE behind a held lock (bounded), not skip --
+    # instant-skip starved the Sunday maintenance block (see
+    # SCHEDULED_LOCK_WAIT_SECONDS)
+    monkeypatch.setattr(pr, "SCHEDULED_LOCK_WAIT_SECONDS", 5.0)
+
+    async def scenario():
+        await pr._lock.acquire()
+        task = asyncio.ensure_future(pr.run_job("test:queued", ["true"], triggered_by="schedule"))
+        await asyncio.sleep(0.2)  # let the job start waiting on the lock
+        pr._lock.release()
+        return await task
+
+    run = _run(scenario())
+    assert run.status == "success"
+    assert not pr._lock.locked()
+
+
+def test_scheduled_run_skips_after_lock_wait_timeout():
+    async def scenario():
+        await pr._lock.acquire()
+        try:
+            return await pr.run_job("test:waited_out", ["true"], triggered_by="schedule", lock_wait=0.2)
         finally:
             pr._lock.release()
 

tests/test_spotify_client.py

@@ -0,0 +1,71 @@
+"""get_playlist_tracks must parse BOTH /items response shapes: entries keyed
+"track" (grandfathered / extended-quota apps) and "item" (apps on Spotify's
+February 2026 renamed shape), and must fail loudly, not return [], when
+Spotify withholds a playlist's contents (no items field at all)."""
+import pytest
+
+from app.services import spotify_client
+
+URL = "https://open.spotify.com/playlist/XYZ123"
+
+
+class _Resp:
+    def __init__(self, data):
+        self._data = data
+
+    def raise_for_status(self):
+        pass
+
+    def json(self):
+        return self._data
+
+
+def _fake_api(monkeypatch, pages):
+    it = iter(pages)
+    monkeypatch.setattr(spotify_client, "_get_token", lambda db: "tok")
+    monkeypatch.setattr(
+        spotify_client.httpx,
+        "get",
+        lambda url, params=None, headers=None, timeout=None: _Resp(next(it)),
+    )
+
+
+def _track(name, artist, isrc=None):
+    return {
+        "name": name,
+        "artists": [{"name": artist}],
+        "external_ids": {"isrc": isrc} if isrc else {},
+    }
+
+
+def test_parses_legacy_track_shape(monkeypatch):
+    _fake_api(monkeypatch, [{"items": [{"track": _track("Sandstorm", "Darude", "FIDAR9900011")}], "next": None}])
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert tracks == [{"artist": "Darude", "title": "Sandstorm", "isrc": "FIDAR9900011"}]
+
+
+def test_parses_renamed_item_shape(monkeypatch):
+    # February 2026 shape: payload under "item", not "track"
+    _fake_api(monkeypatch, [{"items": [{"item": _track("Sandstorm", "Darude")}], "next": None}])
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert tracks == [{"artist": "Darude", "title": "Sandstorm", "isrc": None}]
+
+
+def test_missing_items_field_raises_not_empty(monkeypatch):
+    # metadata-only response (playlist not owned by the connected account on a
+    # new app) must be an error the UI can show, never a silent empty list
+    _fake_api(monkeypatch, [{"name": "DJ-USB", "public": True}])
+    with pytest.raises(RuntimeError, match="without its contents"):
+        spotify_client.get_playlist_tracks(None, URL)
+
+
+def test_skips_null_entries_and_paginates(monkeypatch):
+    _fake_api(
+        monkeypatch,
+        [
+            {"items": [{"track": None}, {"track": _track("A", "X")}], "next": "page2"},
+            {"items": [{"item": _track("B", "Y")}], "next": None},
+        ],
+    )
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert [t["title"] for t in tracks] == ["A", "B"]