0.6.9: Finish the Stage 4 path revisit in the three remaining scripts

The 2026-07-08 path rewrite changed beets' displayed paths from the transitional /music/... mount to real /data/music/Library/... paths. dedup-library.sh was fixed in 0.6.8; three more scripts still assumed the old prefix, each failing silently: - replace-with-better.sh resolved every library match to a doubled nonexistent path, logged [stale-db], skipped the in-place replace, and then imported the staged FLAC as a NEW track via the leftover-import step. Net effect: the mp3-to-flac upgrade flow produced flac+mp3 twin pairs in the library (surfacing in the dedup queue) instead of replacing the MP3 in place. - fix-track-metadata.py queried beets only by the legacy /music/... form, matched nothing, and silently skipped beet update/move after retagging. - export-laptop-playlists.py filtered out every library track (no displayed path starts with /music/ anymore), producing empty exports. All three now use the real displayed path and keep the /music/... form only as a legacy fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0.6.8: Pin sldl skip index + fix dedup scan blinded by the path rewrite
2026-07-16 11:51:55 -06:00 · 2026-07-16 09:23:53 -06:00 · 2026-07-15 13:50:20 -06:00 · 2026-07-15 13:44:26 -06:00 · 2026-07-15 13:30:58 -06:00 · 2026-07-15 10:13:02 -06:00
32 changed files with 1226 additions and 242 deletions
@@ -1,6 +1,7 @@
 __pycache__/
 *.pyc
 .pytest_cache/
+.venv/
 *.db
 *.db-journal
 *.db-wal
@@ -56,7 +56,7 @@ Before you begin, make sure you have:
 1. **A server that runs Docker**, with `docker compose` available. Any Linux box, NAS, or mini PC works.
 2. **A music server that reads a plain folder of tagged files**, such as [Navidrome](https://www.navidrome.org/). alembic builds your library on disk; Navidrome (or similar) is what you actually listen with.
 3. **A Soulseek account.** This is how tracks get downloaded. Sign up free at [soulseek.com](https://www.soulseek.com/).
-4. **A free Spotify Developer app.** This lets alembic read playlist contents (it does not download from Spotify itself, only Soulseek). Takes two minutes, see the credentials section below.
+4. **A Spotify Developer app, plus an OAuth login (not just keys).** alembic reads your playlists' track lists from Spotify (it does not download from Spotify itself, only Soulseek). Create an app free at [developer.spotify.com/dashboard](https://developer.spotify.com/dashboard) to get a Client ID and Secret — but for any app created since Spotify's 2025/2026 API change, those two values alone (plain "client-credentials" auth) are no longer enough to read playlists. You additionally need to complete a one-time **OAuth login** — alembic's **Connect Spotify** button, which redirects you to Spotify to log in and grants alembic a user token — the first time you set things up. Plan on doing both steps; see the credentials section below for exactly how.
 5. **A login provider that speaks OpenID Connect (OIDC).** alembic doesn't have its own username/password login, it delegates to something you already trust. [Pocket ID](https://github.com/pocket-id/pocket-id) is a small self-hosted option built exactly for this, but Authentik, Keycloak, Authelia, or any other OIDC provider works too.

 Optional, add these later if you want them:
@@ -73,10 +73,10 @@ Optional, add these later if you want them:
 Pull the prebuilt image onto your Docker host:

 ```bash
-docker pull git.kretzer.club/andrew/alembic:0.5
+docker pull git.kretzer.club/andrew/alembic:0.6.9
 ```

-That is the whole install. You do not need to download the source or build anything. The `0.5` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.
+That is the whole install. You do not need to download the source or build anything. The `0.6.9` is the version; you can pin to it so nothing changes under you, or use `latest` to always get the newest.

 (If you would rather build it yourself from source, you can, but you do not need to.)

@@ -136,7 +136,7 @@ Create a file called `docker-compose.yml` on your server (put it wherever you ke
 ```yaml
 services:
  alembic:
-    image: git.kretzer.club/andrew/alembic:0.5
+    image: git.kretzer.club/andrew/alembic:0.6.9
    container_name: alembic
    ports:
      - "8420:8420"
@@ -191,14 +191,32 @@ Spotify, Soulseek, and Navidrome are marked **Required** — every playlist sync

 | Service | What to enter | Where to get it |
 |---|---|---|
-| **Spotify** *(required)* | Client ID, Client Secret | Create a free app at [developer.spotify.com/dashboard](https://developer.spotify.com/dashboard). You don't need any special access, the basic free tier is enough to read playlist contents. |
+| **Spotify** *(required)* | Client ID, Client Secret, then **Connect Spotify** (OAuth login) | Reads your playlists' track lists. Create an app at [developer.spotify.com/dashboard](https://developer.spotify.com/dashboard) for the Client ID/Secret, save them, then click **Connect Spotify** to complete an OAuth login with your Spotify account — required for any app created after Spotify's 2025/2026 API change, since keys alone (client-credentials) are no longer sufficient (see Troubleshooting below). |
 | **Soulseek** *(required)* | Username, Password | Your normal Soulseek login. |
 | **Navidrome** *(required)* | Base URL, admin username, admin password | The address of your Navidrome server and an admin account on it. Used so alembic can trigger a library rescan after downloads. |
 | **Bandcamp** *(optional)* | Username, format preference, cookies | Export your Bandcamp cookies while logged into bandcamp.com in your browser, using an extension like "Get cookies.txt LOCALLY", and paste the contents in. Format preference is a space-separated list like `flac mp3-320`. |
 | **Qobuz** *(optional)* | Token, App ID, region | A logged-in Qobuz session token — used (alongside Bandcamp) as a source for "buy this" links stamped onto tracks you didn't purchase there. |
-| **AzuraCast** *(optional)* | API key | Only relevant if you also run an AzuraCast radio station off the same library. It's not a buy-link source itself (that's Bandcamp/Qobuz, written straight to the file); this just lets alembic tell AzuraCast to immediately pick up a tag change instead of waiting for its own periodic scan. |
+| **AzuraCast** *(optional)* | Base URL, API key | Only relevant if you also run an AzuraCast radio station off the same library. It's not a buy-link source itself (that's Bandcamp/Qobuz, written straight to the file); this just lets alembic tell AzuraCast to immediately pick up a tag change instead of waiting for its own periodic scan. Base URL is your AzuraCast instance's address, e.g. `https://radio.example.com`. |
 | **Telegram** *(optional)* | Bot token, chat ID | Create a bot via [@BotFather](https://t.me/BotFather) if you want the daily status digest sent to a chat. |

+### Connecting your Spotify account (OAuth) — required for any new Spotify app
+
+Spotify Client ID/Secret alone ("client-credentials" auth) used to be enough for alembic to read your playlists. Since Spotify's 2025/2026 API change, that stopped working for any app created after the change — Spotify now requires a real user to authorize the app via OAuth before it will hand over playlist data at all. If your Spotify app is that new (almost certainly, if you just created one in the step above), you must complete this OAuth login or every playlist sync will fail with a 403/401 (see Troubleshooting).
+
+Steps:
+
+1. In your Spotify app's dashboard (Developer Dashboard → your app → Settings → Redirect URIs), add this redirect URI: `https://<your-host>/connect/spotify/callback`.
+2. In alembic, go to **Settings → Credentials**, enter your Spotify Client ID and Secret, and save.
+3. Click **Connect Spotify**. You'll be redirected to Spotify's own login page to authorize alembic, then bounced back to the Credentials page, which now shows **Connected**.
+
+That's it, done once. alembic stores the resulting OAuth refresh token (encrypted, same as your other credentials) and uses it to silently mint a fresh access token whenever it needs one — you never have to repeat this unless you disconnect or revoke access on Spotify's side.
+
+One more limit that comes with a new Spotify app: Spotify only returns a playlist's contents to the account that owns it (or collaborates on it), even after you connect. So sync playlists that belong to the account you connect, and if someone shares a playlist with you, save your own copy of it first (or have them add you as a collaborator).
+
+If you happen to be running a Spotify app created *before* the 2025/2026 change, plain Client ID/Secret still works and this step is optional — but connecting is harmless either way, so there's no reason not to.
+
+> **If you connected Spotify before version 0.6.1:** click **Connect Spotify** again. Versions before 0.6.1 requested a narrower OAuth scope than playlist reads actually need, which could get you a token that refreshes fine but then gets 403 Forbidden loading a playlist (see Troubleshooting). Reconnecting grants the correct scope; your old connection doesn't upgrade itself.
+
 ## Adding your first playlist

 1. Go to **Playlists** → **Add playlist**.
@@ -220,7 +238,7 @@ That's it. On its next scheduled run (or immediately, using the **Run now** butt

 ## Updating alembic

-When a new version is released, change the version in your `docker-compose.yml` (for example `:0.5` to the new number), then pull and restart:
+When a new version is released, change the version in your `docker-compose.yml` (for example `:0.6` to the new number), then pull and restart:

 ```bash
 docker compose pull
@@ -269,6 +287,21 @@ Double check the redirect URI registered with your OIDC provider exactly matches
 **A playlist says it's synced but nothing downloaded.**
 Check that playlist's job history under **Settings → Jobs**, the log will usually show whether Soulseek couldn't find a track, or a credential is missing. The playlist's own page will also show everything as "Waiting to download" if nothing came through.

+**The playlist page shows "Spotify denied access (403)" or a 401, or nothing downloads and the logs mention one of those.**
+Your Spotify app can't read the playlist with app-only (client-credentials) access alone — you haven't completed the OAuth login yet. Spotify closed off key-only access for apps created after its 2025/2026 API change, on both the old `/tracks` endpoint (403) and the current `/items` endpoint (401 "valid user authentication required") — this affects even public playlists. The "extended quota mode" that would lift it is granted only to organizations, not individuals, since May 2025.
+
+The fix: go to **Settings → Credentials** and click **Connect Spotify** to complete the OAuth login (see "Connecting your Spotify account (OAuth)" above). This mints a user token that works regardless of when your app was created. If you haven't already, add the redirect URI `https://<your-host>/connect/spotify/callback` in your Spotify app's dashboard first, or the connect step itself will fail with a redirect_uri mismatch.
+
+**A playlist's job log shows `Spotify returned 403 loading this playlist`, `Spotify returned this playlist without its contents`, or `0 visible tracks` even though you've connected Spotify.** (On versions before 0.6.2 these showed up as `sldl crashed (exit 134)` mentioning a 403 Forbidden; on 0.6.2 a playlist you don't own could finish "successfully" while downloading nothing.)
+Possible causes, in order of likelihood:
+
+1. **The playlist isn't owned by the connected account.** For Spotify apps created after the 2025/2026 API change, Spotify only returns a playlist's contents to its owner or a collaborator on it. Being public is NOT enough. Fix: recreate the playlist under the account you connected via **Connect Spotify**, or have the owner make it collaborative and add you.
+2. **You connected before version 0.6.1.** Earlier versions requested a narrower OAuth scope than playlist reads actually need, so the token refreshes fine but then gets 403 loading a playlist regardless of ownership. Fix: go to **Settings → Credentials** and click **Connect Spotify** again to grant the correct scope.
+
+(If your Spotify app is old enough to be grandfathered, both public playlists and your own keep working like before.)
+
+Either way, re-run the playlist after fixing it.
+
 **Dedup found something that isn't actually a duplicate.**
 Use the "Keep both" button on that pair. alembic will remember your decision and won't flag that exact pair again.

@@ -8,8 +8,8 @@ from fastapi.templating import Jinja2Templates
 from starlette.exceptions import HTTPException as StarletteHTTPException
 from starlette.middleware.sessions import SessionMiddleware

-from app.db import enable_beets_db_wal, init_db, mark_interrupted_runs
-from app.routers import artist_casing, auth, credentials, dashboard, dedup, genres, health, import_, jobs, library, playlists
+from app.db import SessionLocal, enable_beets_db_wal, init_db, mark_interrupted_runs
+from app.routers import artist_casing, auth, connect, credentials, dashboard, dedup, genres, health, import_, jobs, library, playlists
 from app.security.session import resolve_session_secret
 from app.services import scheduler_service
 from app.settings import settings
@@ -48,12 +48,31 @@ def _warn_if_key_colocated_with_config() -> None:
        )


+def _render_confs_on_startup() -> None:
+    """Re-render every playlist .conf from the current template once per boot,
+    so confs rendered by an older version converge on upgrade without waiting
+    for a playlist or credential change. Concretely: 0.6.2 dropped the Spotify
+    credential lines from confs (sldl no longer talks to Spotify), and this is
+    what scrubs those secrets from already-deployed confs. Best-effort -- a
+    render failure shouldn't stop the app from starting."""
+    from app.services import credential_service
+
+    db = SessionLocal()
+    try:
+        credential_service.render_playlist_confs(db)
+    except Exception:
+        log.exception("Startup playlist .conf render failed; continuing")
+    finally:
+        db.close()
+
+
@asynccontextmanager
 async def lifespan(_app: FastAPI):
    _warn_if_key_colocated_with_config()
    init_db()
    mark_interrupted_runs()
    enable_beets_db_wal()
+    _render_confs_on_startup()

    scheduler = scheduler_service.create_scheduler()
    scheduler_service.register_all_jobs(scheduler)
@@ -82,6 +101,7 @@ def create_app() -> FastAPI:
    app.include_router(genres.router)
    app.include_router(playlists.router)
    app.include_router(credentials.router)
+    app.include_router(connect.router)
    app.include_router(jobs.router)
    app.include_router(artist_casing.router)

@@ -0,0 +1,106 @@
+"""Spotify OAuth "Connect Spotify" flow.
+
+Not on the /auth prefix or the /settings/credentials prefix: Spotify's app
+dashboard needs one fixed redirect URI to register, and this is it in full:
+https://<host>/connect/spotify/callback.
+
+Unlike Pocket ID (app/security/oidc.py), Spotify's client id/secret live in
+the encrypted credential store, not app.settings -- so this builds the
+authorize/token requests directly with httpx instead of going through an
+authlib client registered once at import time with fixed credentials.
+"""
+import base64
+import secrets
+import urllib.parse
+
+import httpx
+from fastapi import APIRouter, Depends, HTTPException, Request
+from fastapi.responses import RedirectResponse
+
+from app.db import get_db
+from app.security.deps import require_auth
+from app.services import credential_service
+
+router = APIRouter(tags=["spotify-connect"])
+
+AUTHORIZE_URL = "https://accounts.spotify.com/authorize"
+TOKEN_URL = "https://accounts.spotify.com/api/token"
+# Matches exactly what sldl's own built-in OAuth login flow requests (seen in
+# its printed authorize URL: user-library-read + these two) -- granting a
+# narrower scope than sldl expects gets a token sldl can refresh fine but
+# then gets 403 Forbidden from Spotify partway through loading a playlist,
+# which sldl turns into an unhandled-exception crash (exit 134) rather than a
+# clean scope error.
+SCOPES = "playlist-read-private playlist-read-collaborative user-library-read"
+
+
+def _callback_redirect_uri(request: Request) -> str:
+    """request.url_for() reflects the scheme uvicorn saw on the actual TCP
+    connection, not what the reverse proxy served over -- alembic sits behind
+    a TLS-terminating proxy (per every deployment example in the README) and
+    uvicorn isn't told to trust its X-Forwarded-Proto, so this comes back
+    `http://`. Spotify requires an exact scheme match against the redirect
+    URI registered in its app dashboard (always https for a real hostname),
+    so force it here rather than trust the raw connection's scheme."""
+    return str(request.url_for("spotify_callback").replace(scheme="https"))
+
+
+@router.get("/connect/spotify")
+async def connect_spotify(request: Request, user: dict = Depends(require_auth), db=Depends(get_db)):
+    client_id = credential_service.get_scope(db, "spotify").get("client_id")
+    if not client_id:
+        # Nothing to connect to yet -- send them to save client_id/secret first.
+        return RedirectResponse(url="/settings/credentials", status_code=303)
+
+    # CSRF guard on the callback: the state we get back must match what we
+    # handed out for this session, not just be present.
+    state = secrets.token_urlsafe(24)
+    request.session["spotify_oauth_state"] = state
+
+    params = {
+        "client_id": client_id,
+        "response_type": "code",
+        "redirect_uri": _callback_redirect_uri(request),
+        "scope": SCOPES,
+        "state": state,
+    }
+    return RedirectResponse(url=f"{AUTHORIZE_URL}?{urllib.parse.urlencode(params)}")
+
+
+@router.get("/connect/spotify/callback", name="spotify_callback")
+async def spotify_callback(request: Request, user: dict = Depends(require_auth), db=Depends(get_db)):
+    if error := request.query_params.get("error"):
+        raise HTTPException(400, f"Spotify authorization failed: {error}")
+
+    state = request.query_params.get("state")
+    if not state or state != request.session.pop("spotify_oauth_state", None):
+        raise HTTPException(400, "Spotify authorization state mismatch")
+
+    code = request.query_params.get("code")
+    if not code:
+        raise HTTPException(400, "Spotify did not return an authorization code")
+
+    creds = credential_service.get_scope(db, "spotify")
+    client_id = creds.get("client_id")
+    client_secret = creds.get("client_secret")
+    if not client_id or not client_secret:
+        raise HTTPException(400, "Spotify client id/secret not configured")
+
+    basic = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
+    resp = httpx.post(
+        TOKEN_URL,
+        data={
+            "grant_type": "authorization_code",
+            "code": code,
+            "redirect_uri": _callback_redirect_uri(request),
+        },
+        headers={"Authorization": f"Basic {basic}"},
+        timeout=15,
+    )
+    resp.raise_for_status()
+    refresh_token = resp.json().get("refresh_token")
+    if not refresh_token:
+        raise HTTPException(400, "Spotify did not return a refresh token")
+
+    credential_service.set_credentials(db, "spotify", {"refresh_token": refresh_token})
+    return RedirectResponse(url="/settings/credentials", status_code=303)
@@ -6,7 +6,7 @@ from sqlalchemy import select
 from app.db import get_db
 from app.models import JobRun, ScheduledJob
 from app.security.deps import require_auth
-from app.services import scheduler_service
+from app.services import pipeline_runner, playlist_service, scheduler_service

 router = APIRouter(prefix="/settings/jobs", tags=["jobs"])
 templates = Jinja2Templates(directory="app/templates")
@@ -67,7 +67,9 @@ async def runs_table_partial(request: Request, user: dict = Depends(require_auth


@router.post("/{job_key:path}/run")
-async def run_now(job_key: str, background: BackgroundTasks, user: dict = Depends(require_auth)):
+async def run_now(
+    job_key: str, background: BackgroundTasks, user: dict = Depends(require_auth), db=Depends(get_db)
+):
    scheduler = scheduler_service.get_scheduler()
    if scheduler is None:
        raise HTTPException(503, "scheduler not running")
@@ -75,10 +77,23 @@ async def run_now(job_key: str, background: BackgroundTasks, user: dict = Depend
    # background and redirect immediately. A playlist sync can take the better
    # part of an hour; awaiting it here would hang the browser/reverse proxy.
    # Progress shows up in the runs table below, which polls every few seconds.
-    if scheduler.get_job(job_key) is None:
-        raise HTTPException(404, f"no such registered job: {job_key}")
-    background.add_task(scheduler_service.trigger_now, scheduler, job_key)
-    return RedirectResponse(url="/settings/jobs?started=1", status_code=303)
+    if scheduler.get_job(job_key) is not None:
+        background.add_task(scheduler_service.trigger_now, scheduler, job_key)
+        return RedirectResponse(url="/settings/jobs?started=1", status_code=303)
+
+    # Playlists with no cron_expr (or paused) are never registered with the
+    # scheduler by sync_playlist_jobs -- that's "unscheduled/manual-only" by
+    # design, not missing. Run it directly through pipeline_runner instead of
+    # requiring a scheduler job to exist, so "Run now" works for those too.
+    if job_key.startswith("playlist:"):
+        playlist = playlist_service.get_by_name(db, job_key.removeprefix("playlist:"))
+        if playlist is not None:
+            background.add_task(
+                pipeline_runner.run_playlist, playlist.name, playlist.no_m3u, "manual"
+            )
+            return RedirectResponse(url="/settings/jobs?started=1", status_code=303)
+
+    raise HTTPException(404, f"no such registered job: {job_key}")


@router.post("/{job_key:path}/toggle")
@@ -1,3 +1,4 @@
+import httpx
 from fastapi import APIRouter, Depends, Form, HTTPException, Request
 from fastapi.responses import RedirectResponse
 from fastapi.templating import Jinja2Templates
@@ -10,6 +11,23 @@ router = APIRouter(prefix="/playlists", tags=["playlists"])
 templates = Jinja2Templates(directory="app/templates")


+def _friendly_status_error(exc: Exception) -> str:
+    """Turn a raw Spotify API error into something a non-technical user can act
+    on. 403 here almost always means the Spotify app can't read the playlist:
+    wrong credentials, or the connected account can't see it. (The other
+    can't-read case, contents hidden for playlists the account doesn't own,
+    arrives as a RuntimeError from spotify_client with its own message.)"""
+    if isinstance(exc, httpx.HTTPStatusError) and exc.response.status_code == 403:
+        return (
+            "Spotify denied access (403). Check your Spotify credentials under "
+            "Settings then Credentials, and make sure the account connected via "
+            "Connect Spotify can see this playlist. For newly created Spotify "
+            "apps the connected account must own the playlist or be a "
+            "collaborator on it."
+        )
+    return str(exc)
+
+
@router.get("")
 async def playlists_index(request: Request, user: dict = Depends(require_auth), db=Depends(get_db)):
    playlists = playlist_service.list_all(db)
@@ -54,7 +72,7 @@ async def playlist_detail(
    try:
        status = status_service.playlist_status(db, playlist.name, playlist.spotify_url)
    except Exception as exc:
-        error = str(exc)
+        error = _friendly_status_error(exc)

    return templates.TemplateResponse(
        request,
@@ -13,11 +13,11 @@ from app.settings import settings
 # Which fields exist per scope, and whether each is safe to display back to
 # the UI once saved (short opaque strings need never be shown again).
 SCOPE_FIELDS = {
-    "spotify": ["client_id", "client_secret"],
+    "spotify": ["client_id", "client_secret", "refresh_token"],
    "soulseek": ["username", "password"],
    "navidrome": ["base_url", "admin_user", "admin_pass"],
    "bandcamp": ["username", "format_pref", "cookies_txt"],
-    "azuracast": ["api_key"],
+    "azuracast": ["base_url", "api_key"],
    "qobuz": ["token", "app_id", "region"],
    "telegram": ["bot_token", "chat_id"],
 }
@@ -56,6 +56,7 @@ SECRET_KEYS = {
    "api_key",
    "token",
    "bot_token",
+    "refresh_token",
 }


@@ -115,8 +116,10 @@ def get_scope(db: Session, scope: str) -> dict[str, str]:
 _SAFE_CONF_NAME = re.compile(r"^[A-Za-z0-9 _-]{1,64}$")

 # The credential lines the renderer fills in from the encrypted store. Everything
-# else in a rendered .conf comes verbatim from _template.conf.
-_CONF_CRED_KEYS = ("user", "pass", "spotify-id", "spotify-secret")
+# else in a rendered .conf comes verbatim from _template.conf. Spotify creds are
+# NOT here: sldl never talks to Spotify itself (see _template.conf and
+# run-playlist.sh) -- the scripts that do read them from _spotify.env instead.
+_CONF_CRED_KEYS = ("user", "pass")


 def _set_conf_field(text: str, key: str, value: str) -> str:
@@ -133,26 +136,24 @@ def _set_conf_field(text: str, key: str, value: str) -> str:

 def render_playlist_confs(db: Session) -> None:
    """Render every playlist's sldl .conf directly from _template.conf, with
-    the path placeholders and the Soulseek/Spotify credentials substituted in
-    one pass, written 0600.
+    the path placeholders and the Soulseek credentials substituted in one
+    pass, written 0600. No Spotify credentials here -- sldl never talks to
+    Spotify itself (see _template.conf); those live only in _spotify.env.

    This is the single source of .conf rendering. It replaces the older
    DB -> playlists.json -> regen.sh -> regex-patch-back chain: the app owns
    every input (playlists in its DB, credentials in its encrypted store), so
    there is no reason to round-trip through an intermediate file and a
    subprocess. Called on any playlist change (playlist_service) and on any
-    Spotify/Soulseek credential change (render_scope)."""
+    Soulseek credential change (render_scope)."""
    from app.services import playlist_service

    template = (settings.pipeline_dir / "configs" / "_template.conf").read_text()
    dropbox_root = str(settings.music_data_dir / "sldl-dropbox")
-    spotify = get_scope(db, "spotify")
    soulseek = get_scope(db, "soulseek")
    creds = {
        "user": soulseek.get("username", ""),
        "pass": soulseek.get("password", ""),
-        "spotify-id": spotify.get("client_id", ""),
-        "spotify-secret": spotify.get("client_secret", ""),
    }
    # Path placeholders are substituted as LITERAL text in a single left-to-right
    # pass (never re-scanned), so a value can't be reinterpreted as another
@@ -191,13 +192,22 @@ def render_scope(db: Session, scope: str) -> None:
    if scope == "spotify":
        cid = values.get("client_id", "")
        csec = values.get("client_secret", "")
-        # _spotify.env is read by the pipeline python scripts (spotify-genre etc.).
+        # _spotify.env is the ONLY place Spotify creds are rendered -- sldl
+        # itself never talks to Spotify (see _template.conf), so playlist
+        # .confs don't need them. Read by run-playlist.sh (to generate the
+        # search CSV) and the pipeline python scripts (spotify-retag,
+        # spotify-genre, fix-track-metadata). SPOTIFY_REFRESH_TOKEN is only
+        # set once an account is connected via /connect/spotify -- its
+        # presence is what switches every reader from client-credentials to
+        # a user token (see _spotify_auth.get_token).
        _write_env_file(
            settings.pipeline_config_dir / "_spotify.env",
-            {"SPOTIFY_CLIENT_ID": cid, "SPOTIFY_CLIENT_SECRET": csec},
+            {
+                "SPOTIFY_CLIENT_ID": cid,
+                "SPOTIFY_CLIENT_SECRET": csec,
+                "SPOTIFY_REFRESH_TOKEN": values.get("refresh_token", ""),
+            },
        )
-        # Re-render every playlist .conf so the new Spotify creds land in them.
-        render_playlist_confs(db)

    elif scope == "soulseek":
        render_playlist_confs(db)
@@ -239,6 +249,10 @@ def render_scope(db: Session, scope: str) -> None:
        key_path = az_dir / "api_key"
        key_path.write_text(values.get("api_key", ""))
        key_path.chmod(0o600)
+        # base_url isn't secret, but still keyfile-based (not env) so it's
+        # readable via the same fallback cascade as api_key -- see
+        # enrich-buy-url.py's --azuracast-base resolution.
+        (az_dir / "base_url").write_text(values.get("base_url", ""))

    elif scope == "qobuz":
        qobuz_dir = settings.pipeline_config_dir / "qobuz"
@@ -289,7 +303,9 @@ def _clear_rendered(scope: str) -> None:
    notify-telegram.sh call). Re-enabling calls render_scope() again to
    recreate the file from the still-stored values."""
    if scope == "azuracast":
-        (settings.pipeline_config_dir / "azuracast" / "api_key").unlink(missing_ok=True)
+        az_dir = settings.pipeline_config_dir / "azuracast"
+        (az_dir / "api_key").unlink(missing_ok=True)
+        (az_dir / "base_url").unlink(missing_ok=True)
    elif scope == "qobuz":
        qobuz_dir = settings.pipeline_config_dir / "qobuz"
        for name in ("token", "app_id", "region"):
@@ -360,6 +376,8 @@ def auth_states(db: Session) -> list[dict]:
        values = get_scope(db, scope)
        configured = bool(values)
        entry = {"scope": scope, "configured": configured}
+        if scope == "spotify":
+            entry["spotify_connected"] = bool(values.get("refresh_token"))
        if scope == "bandcamp" and values.get("cookies_txt"):
            expiry = _bandcamp_cookie_expiry(values["cookies_txt"])
            if expiry is not None:
@@ -12,6 +12,15 @@ from app.settings import settings
 # all-jobs-share-one-lock behavior exactly rather than over-engineering it.
 _lock = asyncio.Lock()

+# How long a SCHEDULED run waits for the lock before giving up (recorded as
+# skipped_lock). The old flock -n instant-skip starves fixed-time schedules:
+# the Sunday 08:30 strip-mb-tags run took 10m19s on 2026-07-12, so the 08:35
+# and 08:40 jobs both hit the held lock and silently lost their only slot of
+# the week. Scheduled jobs have nobody watching, so queueing (bounded) beats
+# skipping; manual runs keep the instant skip because a person clicking "Run
+# now" should get an immediate answer, not a silent 30-minute wait.
+SCHEDULED_LOCK_WAIT_SECONDS = 1800.0
+
 _ENV_PASSTHROUGH_KEYS = ("PATH", "HOME", "LANG", "LC_ALL", "TZ")


@@ -94,18 +103,30 @@ async def _execute(
    timeout: float | None,
    use_lock: bool,
    capture: bool,
+    lock_wait: float | None,
 ) -> tuple[JobRun, str]:
    """Shared core for run_job/run_job_capture: acquire the lock (unless
    use_lock=False), record the run, exec the subprocess, and finalize. When
    capture=True, stdout+stderr is captured as text and returned; otherwise it
    streams straight to the log file. Returns (JobRun, output_text) -- the text
-    is "" in the non-capture and skipped-lock cases."""
+    is "" in the non-capture and skipped-lock cases.
+
+    lock_wait: how long to wait for a held lock before recording
+    skipped_lock. None picks the policy default: SCHEDULED_LOCK_WAIT_SECONDS
+    for triggered_by="schedule", instant skip for everything else."""
    started_at = time.time()
+    if lock_wait is None:
+        lock_wait = SCHEDULED_LOCK_WAIT_SECONDS if triggered_by == "schedule" else 0.0

    acquired = False
    if use_lock:
        if not await _try_acquire_nowait():
-            return _record_run(job_key, started_at, triggered_by, finished_at=started_at, status="skipped_lock"), ""
+            if lock_wait <= 0:
+                return _record_run(job_key, started_at, triggered_by, finished_at=started_at, status="skipped_lock"), ""
+            try:
+                await asyncio.wait_for(_lock.acquire(), timeout=lock_wait)
+            except asyncio.TimeoutError:
+                return _record_run(job_key, started_at, triggered_by, finished_at=time.time(), status="skipped_lock"), ""
        acquired = True

    try:
@@ -170,18 +191,21 @@ async def run_job(
    triggered_by: str = "schedule",
    timeout: float | None = None,
    use_lock: bool = True,
+    lock_wait: float | None = None,
 ) -> JobRun:
    """Run a pipeline command under the shared mutual-exclusion lock,
-    recording one job_runs row start-to-finish. If the lock is already
-    held, records status='skipped_lock' immediately and returns without
-    running anything -- the old flock -n behavior, now visible in the UI
-    instead of silently skipping.
+    recording one job_runs row start-to-finish. If the lock is already held,
+    scheduled runs (triggered_by="schedule") wait up to
+    SCHEDULED_LOCK_WAIT_SECONDS for it before recording skipped_lock -- see
+    that constant for why -- while manual/other runs record skipped_lock
+    immediately (the old flock -n behavior, visible in the UI instead of
+    silently skipping). Pass lock_wait to override either way.

    use_lock=False runs the command WITHOUT taking the pipeline lock, for
    read-only jobs that never touch the library (e.g. the status report).
    Those must never be starved by a long-running write job, and running
    them concurrently is safe."""
-    run, _ = await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=False)
+    run, _ = await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=False, lock_wait=lock_wait)
    return run


@@ -191,6 +215,7 @@ async def run_job_capture(
    triggered_by: str = "manual",
    timeout: float | None = None,
    use_lock: bool = True,
+    lock_wait: float | None = None,
 ) -> tuple[JobRun, str]:
    """Like run_job(), but captures stdout+stderr as text and returns it
    alongside the JobRun, instead of only writing it to the log file --
@@ -198,7 +223,7 @@ async def run_job_capture(
    dedup_review_service's dry-run scan. The full output is still written
    to a log file afterward so job_runs.log_path works the same as any
    other job."""
-    return await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=True)
+    return await _execute(job_key, argv, triggered_by, timeout, use_lock, capture=True, lock_wait=lock_wait)


 async def run_playlist(playlist_name: str, no_m3u: bool = False, triggered_by: str = "schedule") -> JobRun:
@@ -115,7 +115,8 @@ def update(db: Session, playlist_id: int, **fields) -> Playlist:


 def delete(db: Session, playlist_id: int) -> None:
-    """Remove the playlist from the DB and delete its rendered .conf file.
+    """Remove the playlist from the DB and delete its rendered .conf and
+    generated search .csv files.
    Does NOT touch anything already downloaded/imported for it — that's a
    library decision, not a playlist-definition one."""
    playlist = db.get(Playlist, playlist_id)
@@ -126,6 +127,9 @@ def delete(db: Session, playlist_id: int) -> None:
    db.commit()
    conf_path = settings.pipeline_config_dir / f"{name}.conf"
    conf_path.unlink(missing_ok=True)
+    # The search CSV run-playlist.sh generates next to the conf each run.
+    csv_path = settings.pipeline_config_dir / f"{name}.csv"
+    csv_path.unlink(missing_ok=True)

    from app.services import scheduler_service

@@ -11,8 +11,11 @@ TOKEN_URL = "https://accounts.spotify.com/api/token"
 API_BASE = "https://api.spotify.com/v1"

 # Module-level cache: one alembic process, one Spotify app registration --
-# a single shared client-credentials token is fine (no per-user tokens here).
-_token_cache: dict = {"token": None, "expires_at": 0.0}
+# a single shared token is fine (single-user app, no per-request identity).
+# Keyed by mode too, so connecting/disconnecting an account (switching
+# between a user token and a client-credentials token) can't serve a stale
+# token minted under the other grant type.
+_token_cache: dict = {"token": None, "expires_at": 0.0, "mode": None}

 _PLAYLIST_ID_RE = re.compile(r"playlist/([A-Za-z0-9]+)")

@@ -26,19 +29,34 @@ def _extract_playlist_id(playlist_url: str) -> str:

 def _get_token(db: Session) -> str:
    now = time.time()
-    if _token_cache["token"] and _token_cache["expires_at"] > now + 30:
-        return _token_cache["token"]
-
    creds = credential_service.get_scope(db, "spotify")
    client_id = creds.get("client_id")
    client_secret = creds.get("client_secret")
    if not client_id or not client_secret:
        raise RuntimeError("Spotify credentials not configured (settings/credentials)")
+    refresh_token = creds.get("refresh_token")
+    # A user token (from /connect/spotify) can read playlists on newly created
+    # Spotify apps; client-credentials can't (Spotify blocks GET /items for
+    # new apps without user auth). Prefer the user token whenever one is
+    # connected, falling back to client-credentials for grandfathered apps.
+    mode = "user" if refresh_token else "client_credentials"
+
+    if (
+        _token_cache["token"]
+        and _token_cache["mode"] == mode
+        and _token_cache["expires_at"] > now + 30
+    ):
+        return _token_cache["token"]

    basic = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
+    grant_data = (
+        {"grant_type": "refresh_token", "refresh_token": refresh_token}
+        if refresh_token
+        else {"grant_type": "client_credentials"}
+    )
    resp = httpx.post(
        TOKEN_URL,
-        data={"grant_type": "client_credentials"},
+        data=grant_data,
        headers={"Authorization": f"Basic {basic}"},
        timeout=15,
    )
@@ -46,6 +64,7 @@ def _get_token(db: Session) -> str:
    data = resp.json()
    _token_cache["token"] = data["access_token"]
    _token_cache["expires_at"] = now + data["expires_in"]
+    _token_cache["mode"] = mode
    return _token_cache["token"]


@@ -57,15 +76,35 @@ def get_playlist_tracks(db: Session, playlist_url: str) -> list[dict]:
    headers = {"Authorization": f"Bearer {token}"}

    tracks = []
-    url = f"{API_BASE}/playlists/{playlist_id}/tracks"
-    params = {"limit": 100, "fields": "items(track(name,artists(name),external_ids)),next"}
+    # /items, not /tracks: Spotify removed GET /playlists/{id}/tracks in its
+    # February 2026 API changes. The response shape ALSO changed, but only for
+    # apps on the new behavior: each entry's payload moved from "track" to
+    # "item" (tracks.tracks.track -> items.items.item). Extended Quota Mode
+    # (grandfathered) apps keep the old "track" key, so parse both. No
+    # `fields` filter: it selects by key name, so on the renamed shape a
+    # track(...) filter silently returns empty pages -- exactly the failure
+    # we're avoiding.
+    url = f"{API_BASE}/playlists/{playlist_id}/items"
+    params = {"limit": 100}

    while url:
        resp = httpx.get(url, params=params, headers=headers, timeout=15)
        resp.raise_for_status()
        data = resp.json()
-        for item in data.get("items", []):
-            track = item.get("track")
+        if "items" not in data:
+            # New-behavior apps get metadata only (no items field at all) for
+            # playlists the connected account doesn't own or collaborate on --
+            # public is no longer sufficient. Same message style the playlist
+            # page shows for a 403.
+            raise RuntimeError(
+                "Spotify returned this playlist without its contents. For newly "
+                "created Spotify apps, the account connected via Connect Spotify "
+                "must own the playlist (or be a collaborator on it) -- ask the "
+                "owner to share it as collaborative, or recreate it under the "
+                "connected account."
+            )
+        for item in data["items"]:
+            track = item.get("track") or item.get("item")
            if not track:
                continue  # local files / removed tracks show up as null
            artists = track.get("artists") or []
@@ -45,8 +45,22 @@
      <p class="cred-disabled-note">Disabled — its scheduled automation is paused (or, for buy-link sources, its rendered credential file is removed) until you turn this back on. Saved values below are kept.</p>
      {% endif %}

+      {% if scope == "spotify" %}
+      <div class="field wide" style="margin-bottom:1rem;">
+        <label>Account connection</label>
+        <p class="muted" style="margin:0 0 0.5rem;">Newly created Spotify apps can no longer read playlists without this — connecting an account mints a user token that works where the app-only credentials above are blocked. Grandfathered apps keep working without it.</p>
+        {% if "refresh_token" in configured.get(scope, []) %}
+        <span class="badge badge-success">Connected</span>
+        <a href="/connect/spotify" class="btn" style="margin-left:0.5rem;">Reconnect Spotify</a>
+        {% else %}
+        <a href="/connect/spotify" class="btn">Connect Spotify</a>
+        {% endif %}
+      </div>
+      {% endif %}
+
      <form method="post" action="/settings/credentials/{{ scope }}">
        {% for field in fields %}
+        {% if not (scope == "spotify" and field == "refresh_token") %}
        <div class="field wide">
          <label for="{{ scope }}_{{ field }}">
            {{ field }}
@@ -60,6 +74,7 @@
          <input type="text" id="{{ scope }}_{{ field }}" name="{{ field }}" placeholder="leave blank to keep current" autocomplete="off" spellcheck="false">
          {% endif %}
        </div>
+        {% endif %}
        {% endfor %}
        <button type="submit" class="btn">Save {{ scope }}</button>
      </form>
@@ -63,7 +63,7 @@
 <h2>Credential status</h2>
 <ul class="chip-list auth-chip-list">
  {% for a in auth_states %}
-  <li title="{{ a.scope }}: {{ 'configured' if a.configured else 'not configured' }}{% if a.days_left is defined %}, cookie expires in {{ a.days_left }}d{% endif %}"><span class="status-dot {{ 'status-dot-success' if a.configured else 'status-dot-muted' }}{% if a.days_left is defined and a.days_left < 14 %} status-dot-warning{% endif %}"></span>{{ a.scope }}{% if a.days_left is defined %}<span class="muted"> {{ a.days_left }}d</span>{% endif %}</li>
+  <li title="{{ a.scope }}: {{ 'configured' if a.configured else 'not configured' }}{% if a.days_left is defined %}, cookie expires in {{ a.days_left }}d{% endif %}{% if a.spotify_connected is defined %}, {{ 'OAuth connected' if a.spotify_connected else 'app-only (client-credentials)' }}{% endif %}"><span class="status-dot {{ 'status-dot-success' if a.configured else 'status-dot-muted' }}{% if a.days_left is defined and a.days_left < 14 %} status-dot-warning{% endif %}"></span>{{ a.scope }}{% if a.days_left is defined %}<span class="muted"> {{ a.days_left }}d</span>{% endif %}{% if a.spotify_connected %}<span class="muted"> (OAuth)</span>{% endif %}</li>
  {% endfor %}
 </ul>
 <p><a href="/settings/credentials">Manage credentials &rarr;</a></p>
@@ -25,6 +25,8 @@ PLAYLIST_NAME="${1:?Usage: $0 [--no-m3u] <playlist_name>}"

 # ==== Paths ====
 CONFIG_FILE=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/${PLAYLIST_NAME}.conf
+SPOTIFY_ENV=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/_spotify.env
+CSV_FILE=${ALEMBIC_CONFIG_DIR:-/config}/pipeline/${PLAYLIST_NAME}.csv
 DROPBOX=${MUSIC_DATA_DIR:-/data/music}/sldl-dropbox/${PLAYLIST_NAME}
 PLAYLISTS_DIR=${MUSIC_DATA_DIR:-/data/music}/playlists
 QUARANTINE_DIR=${MUSIC_DATA_DIR:-/data/music}/Songs/untagged
@@ -54,6 +56,61 @@ if [[ ! -f "$CONFIG_FILE" ]]; then
  exit 1
 fi

+# ==== Read the playlist from Spotify into a CSV sldl can search from ====
+# sldl's own vendored Spotify client still calls GET /playlists/{id}/tracks,
+# which Spotify removed in its February 2026 API changes. Grandfathered apps
+# still get a pass there; apps created after that change get a hard 403
+# regardless of auth method (client-credentials, or even a correctly-scoped,
+# correctly-owned OAuth user token -- confirmed live, exit 134 unhandled
+# APIException: Forbidden from Extractors.Spotify.GetPlaylist). sldl has no
+# fallback to the still-working /items endpoint, so it can never read a
+# playlist for a new app no matter what alembic hands it. Reading the
+# playlist ourselves (via /items) and handing sldl a plain CSV instead
+# sidesteps sldl's Spotify client entirely -- works the same for
+# grandfathered and new apps alike.
+SPOTIFY_URL=$(sed -n 's/^input *= *//p' "$CONFIG_FILE" | tr -d ' ')
+[[ -f "$SPOTIFY_ENV" ]] && source "$SPOTIFY_ENV"
+
+if [[ -z "$SPOTIFY_URL" || -z "${SPOTIFY_CLIENT_ID:-}" || -z "${SPOTIFY_CLIENT_SECRET:-}" ]]; then
+  log "ERROR: missing playlist URL in $CONFIG_FILE or Spotify credentials in $SPOTIFY_ENV"
+  exit 1
+fi
+
+log "Fetching playlist from Spotify: $SPOTIFY_URL"
+if ! SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
+    SPOTIFY_REFRESH_TOKEN="${SPOTIFY_REFRESH_TOKEN:-}" \
+    python3 "${PIPELINE_DIR:-/app/pipeline}/lib/spotify-playlist-csv.py" "$SPOTIFY_URL" "$CSV_FILE" >> "$LOG" 2>&1; then
+  log "ERROR: failed to fetch playlist from Spotify — see $LOG"
+  exit 1
+fi
+
+# Surface the fetched count in the timestamped summary. Zero tracks from a
+# successful fetch is almost never a truly empty playlist -- it usually means
+# Spotify hid the contents from the connected account (see the pointed errors
+# in spotify-playlist-csv.py), so call it out instead of letting sldl no-op
+# with a clean exit 0.
+TRACK_COUNT=$(($(wc -l < "$CSV_FILE") - 1))
+if [[ "$TRACK_COUNT" -le 0 ]]; then
+  log "WARNING: Spotify returned 0 visible tracks for this playlist -- nothing to download. If the playlist isn't actually empty, the connected Spotify account can't see its contents (it must own or collaborate on the playlist for newly created Spotify apps)."
+else
+  log "Fetched $TRACK_COUNT track(s) from Spotify"
+fi
+
+# ==== Seed the pinned sldl skip index if it's missing or stranded ====
+# The conf pins index-path to $DROPBOX/_index.csv (see _template.conf: sldl's
+# default index location is derived from the input, so an input change strands
+# the old index and the whole playlist re-downloads -- that's what produced
+# the 2026-07-16 duplicate flood). If the pinned file is missing, or an index
+# in one of sldl's old input-named subfolders is newer (i.e. sldl last wrote
+# somewhere else), fold them all into the pinned location first.
+INDEX_FILE="$DROPBOX/_index.csv"
+if [[ ! -s "$INDEX_FILE" ]] || \
+   [[ -n "$(find "$DROPBOX" -mindepth 2 -maxdepth 2 -name _index.csv -newer "$INDEX_FILE" -print -quit 2>/dev/null)" ]]; then
+  log "Seeding pinned sldl index at $INDEX_FILE from prior indexes"
+  python3 "${PIPELINE_DIR:-/app/pipeline}/lib/merge-sldl-indexes.py" "$DROPBOX" "$INDEX_FILE" >> "$LOG" 2>&1 \
+    || log "WARNING: index merge failed -- sldl may re-download tracks the library already has"
+fi
+
 # ==== Run sldl (vendored binary, subprocess of this container) ====
 log "Running sldl for: $PLAYLIST_NAME"

@@ -62,19 +119,24 @@ log "Running sldl for: $PLAYLIST_NAME"
 # download. A non-zero exit here is logged but not fatal.
 #
 # Hard timeout: without it a stuck sldl hangs FOREVER holding the shared lock,
-# which silently skips every later-scheduled playlist for the night. (Seen
-# 2026-06-18: a transient Spotify client-creds failure made sldl fall back to
-# interactive OAuth — "manually open: https://accounts.spotify.com/..." — and
-# it waited 7h on a browser callback that never comes.) timeout TERMs the run
-# and KILLs after a grace period; there's no leftover container to clean up
-# now that sldl is a plain subprocess instead of a docker-compose service.
+# which silently skips every later-scheduled playlist for the night. timeout
+# TERMs the run and KILLs after a grace period; there's no leftover container
+# to clean up now that sldl is a plain subprocess instead of a docker-compose
+# service.
+#
+# -c "$CONFIG_FILE" still supplies Soulseek login, download path, quality
+# settings, port, etc. -- the CSV positional arg + --input-type csv override
+# just the input source (confirmed: sldl ignores the conf's own `input =`/
+# `input-type =` lines when both are given).
 SLDL_TIMEOUT="${SLDL_TIMEOUT:-2700}"        # 45 min; override via env
 SLDL_EXIT=0
 timeout --kill-after=30s "$SLDL_TIMEOUT" \
-  "$SLDL_BIN" -c "$CONFIG_FILE" >> "$LOG" 2>&1 \
+  "$SLDL_BIN" "$CSV_FILE" --input-type csv -c "$CONFIG_FILE" >> "$LOG" 2>&1 \
  || SLDL_EXIT=$?
 if [[ "$SLDL_EXIT" -eq 124 || "$SLDL_EXIT" -eq 137 ]]; then
  log "ERROR: sldl exceeded ${SLDL_TIMEOUT}s and was killed (likely a hang)"
+elif [[ "$SLDL_EXIT" -eq 134 ]]; then
+  log "ERROR: sldl crashed (exit 134, unhandled exception) -- see the log above for the exception detail"
 fi
 log "sldl finished with exit code $SLDL_EXIT"

@@ -107,25 +169,19 @@ log "Cleaning up .incomplete files in dropbox"
 find "$DROPBOX" -name "*.incomplete" -type f -delete 2>>"$LOG" || true

 # ==== Rewrite tags from Spotify (source of truth for matched tracks) ====
-# Reads Spotify creds + playlist URL from the same conf sldl used. For each file
-# in the dropbox that matches a Spotify playlist track, overwrites ARTIST
+# Uses the playlist URL and _spotify.env credentials already loaded for the
+# CSV fetch above (confs no longer carry Spotify creds). For each file in the
+# dropbox that matches a Spotify playlist track, overwrites ARTIST
 # (semicolon-joined), ALBUMARTIST (primary), ALBUM, TITLE, TRACKNUMBER,
 # DISCNUMBER, DATE. GROUPING is preserved. Files that don't match are left for
 # the fallback step below.
-SPOTIFY_URL=$(sed -n 's/^input *= *//p' "$CONFIG_FILE" | tr -d ' ')
-SPOTIFY_CLIENT_ID=$(sed -n 's/^spotify-id *= *//p' "$CONFIG_FILE" | tr -d ' ')
-SPOTIFY_CLIENT_SECRET=$(sed -n 's/^spotify-secret *= *//p' "$CONFIG_FILE" | tr -d ' ')
-
-if [[ -n "$SPOTIFY_URL" && -n "$SPOTIFY_CLIENT_ID" && -n "$SPOTIFY_CLIENT_SECRET" ]]; then
-  log "Rewriting tags from Spotify for files in $DROPBOX"
-  if SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
-      python3 ${PIPELINE_DIR:-/app/pipeline}/lib/spotify-retag.py "$SPOTIFY_URL" "$DROPBOX" >> "$LOG" 2>&1; then
-    log "Spotify retag complete"
-  else
-    log "WARNING: Spotify retag failed (exit $?) — continuing with sldl-supplied tags"
-  fi
+log "Rewriting tags from Spotify for files in $DROPBOX"
+if SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
+    SPOTIFY_REFRESH_TOKEN="${SPOTIFY_REFRESH_TOKEN:-}" \
+    python3 ${PIPELINE_DIR:-/app/pipeline}/lib/spotify-retag.py "$SPOTIFY_URL" "$DROPBOX" >> "$LOG" 2>&1; then
+  log "Spotify retag complete"
 else
-  log "Skipping Spotify retag — missing input/spotify-id/spotify-secret in $CONFIG_FILE"
+  log "WARNING: Spotify retag failed (exit $?) — continuing with sldl-supplied tags"
 fi

 # ==== Quarantine any files still missing essential tags ====
@@ -14,13 +14,15 @@
 user = SOULSEEK_USER
 pass = SOULSEEK_PASS

-# ==== Spotify API credentials ====
-spotify-id = SPOTIFY_CLIENT_ID
-spotify-secret = SPOTIFY_CLIENT_SECRET
-
 # ==== Input (Spotify playlist URL) ====
+# sldl itself never reads Spotify at all -- its vendored Spotify client still
+# calls GET /playlists/{id}/tracks, which Spotify removed in its February 2026
+# API changes (new apps get a hard 403 there regardless of auth method; see
+# spotify-playlist-csv.py). run-playlist.sh reads the playlist itself (via the
+# still-working /items endpoint) and hands sldl a CSV via --input-type csv on
+# the command line, which overrides this line entirely -- it's kept only so
+# run-playlist.sh has somewhere to read the playlist URL from per-playlist.
 input = SPOTIFY_URL
-input-type = spotify

 # ==== Output paths ====
 # SLDL_DROPBOX_ROOT is substituted at render time from $MUSIC_DATA_DIR
@@ -29,6 +31,18 @@ path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME
 playlist-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_sldl.m3u8
 write-playlist = true

+# ==== Skip index (pinned!) ====
+# sldl's already-downloaded index defaults to {path}/{playlist-name}/_index.csv
+# where {playlist-name} is derived from the INPUT: the Spotify playlist's
+# display name for spotify input, the CSV filename for csv input. beets moves
+# every download out of the dropbox, so this index is the ONLY thing standing
+# between a nightly run and re-downloading the whole playlist. When 0.6.2
+# switched input from Spotify URL to CSV, the derived name changed, sldl
+# started a fresh empty index in a new subfolder, and every playlist
+# re-downloaded in full on 2026-07-16. Pinning the path here decouples the
+# index from input naming so that can never happen again.
+index-path = SLDL_DROPBOX_ROOT/PLAYLIST_NAME/_index.csv
+
 # ==== Naming ====
 name-format = {artist} - {title}

@@ -12,19 +12,27 @@ import urllib.parse
 import urllib.request

 _TOKEN_URL = "https://accounts.spotify.com/api/token"
-_cache: dict[tuple[str, str], str] = {}
+_cache: dict[tuple[str, str, str | None], str] = {}


-def get_token(client_id: str, client_secret: str) -> str:
-    """Return a client-credentials access token, cached per (id, secret) for
-    the life of the process."""
-    key = (client_id, client_secret)
+def get_token(client_id: str, client_secret: str, refresh_token: str | None = None) -> str:
+    """Return an access token, cached per (id, secret, refresh_token) for the
+    life of the process. If refresh_token is set (an account has been
+    connected via /connect/spotify), mints a user token -- required for
+    playlist reads on newly created Spotify apps, which reject
+    client-credentials tokens. Otherwise falls back to client-credentials."""
+    key = (client_id, client_secret, refresh_token)
    if key in _cache:
        return _cache[key]
    creds = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
+    grant = (
+        {"grant_type": "refresh_token", "refresh_token": refresh_token}
+        if refresh_token
+        else {"grant_type": "client_credentials"}
+    )
    req = urllib.request.Request(
        _TOKEN_URL,
-        data=urllib.parse.urlencode({"grant_type": "client_credentials"}).encode(),
+        data=urllib.parse.urlencode(grant).encode(),
        headers={
            "Authorization": f"Basic {creds}",
            "Content-Type": "application/x-www-form-urlencoded",
@@ -189,7 +189,7 @@ def main() -> int:
    elapsed = time.time() - start
    print(f"[index] done in {elapsed:.0f}s. wrote {done - failed} rows, {failed} failed. "
          f"index now has {total} entries.", flush=True)
-    return 0 if failed == 0 else 1
+    return 0 if (done - failed) > 0 else 1


 if __name__ == "__main__":
@@ -78,8 +78,22 @@ fi
 # Counters: tracked via log grep at the end (avoids bash subshell pitfalls).
 # Functions write KEEP/DELETE lines with consistent prefixes; summary greps them.

-# Convert container path (/music/...) to host path (${MUSIC_DATA_DIR:-/data/music}/Library/...)
-host_path() { echo "${MUSIC_DATA_DIR:-/data/music}/Library${1#/music}"; }
+# Resolve a beets-displayed path to a real path in this container. Since the
+# 2026-07-08 path rewrite, beets displays ${MUSIC_DATA_DIR:-/data/music}/Library/...
+# paths that are directly usable — pass those through UNCHANGED. Only legacy
+# /music/... display paths (the pre-rewrite transitional mount) still need the
+# prefix swap. Blindly prepending the library dir to an already-correct path
+# (the old behavior) produced /data/music/Library/data/music/Library/... —
+# nonexistent, so every candidate failed the -f check and every pass reported
+# 0 groups from 2026-07-08 until this fix.
+host_path() {
+  local p="$1"
+  if [[ "$p" == /music/* ]]; then
+    echo "${MUSIC_DATA_DIR:-/data/music}/Library${p#/music}"
+  else
+    echo "$p"
+  fi
+}

 SEP=$'\x1c'  # ASCII file separator — safe with any music metadata

@@ -202,9 +216,10 @@ process_group() {
 log ""
 log "[$(date -Iseconds)] === Pass 1: numbered siblings ==="

-# Dump id + container path from beets once. beets normalizes $path for
-# display (always /music/-prefixed) even though the DB stores a mix of
-# absolute and relative — so the display paths are safe to compare/convert,
+# Dump id + display path from beets once. beets normalizes $path for display
+# (real /data/music/Library/... paths since the 2026-07-08 rewrite; /music/...
+# on any legacy entry) even though the DB stores a mix of absolute and
+# relative — so the display paths are safe to compare/convert via host_path(),
 # and the ids are what we hand to beet remove/move.
 beet ls -f "\$id${SEP}\$path" 2>/dev/null > /tmp/beets-id-paths.txt
 cut -d"$SEP" -f2- /tmp/beets-id-paths.txt > /tmp/beets-all-paths.txt
@@ -231,7 +246,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do

  if [[ $score_canonical -le $score_numbered ]]; then
    # Canonical wins: delete numbered (in beets), keep canonical (ghost)
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
      "$host_canonical" "${numbered_id}${SEP}${numbered_cp}"
    if [[ $APPLY -eq 1 ]]; then
      # canonical is now a ghost file; import it into beets
@@ -240,7 +255,7 @@ while IFS="$SEP" read -r numbered_id numbered_cp; do
  else
    # Numbered wins (canonical is ghost): rm the ghost, then beet move to rename
    # the numbered file to the canonical path (id query — see process_group).
-    process_group "P1 numbered-sibling: ${numbered_cp##/music/}" \
+    process_group "P1 numbered-sibling: ${numbered_cp##*/Library/}" \
      "${numbered_id}${SEP}${numbered_cp}" "$host_canonical"
    if [[ $APPLY -eq 1 ]]; then
      beet move "id:${numbered_id}" >> "$LOG" 2>&1 || log "  WARN: beet move id:${numbered_id} failed"
@@ -294,6 +294,9 @@ def set_flac_tag(path, name, value):

 # ---- AzuraCast reprocess (same as backfill-buy-url.py) ----------------------
 def az_reprocess(base, key, station, host_paths, library, log):
+    if not base:
+        log("  no AzuraCast base URL configured — skipping reprocess")
+        return
    root = library.rstrip("/") + "/"
    want = {p[len(root):] for p in host_paths if p.startswith(root)}
    headers = {"X-API-Key": key, "Accept": "application/json"}
@@ -365,7 +368,7 @@ def refresh_qobuz_links(flacs, args, az_key):
          f"{converted} {'converted' if args.apply else 'would-convert'}, "
          f"{unconvertible} left intact (not purchasable / unresolved)")

-    if args.apply and touched and az_key:
+    if args.apply and touched and az_key and args.azuracast_base:
        print("[refresh-qobuz] telling AzuraCast to reprocess touched files...")
        az_reprocess(args.azuracast_base, az_key, args.station,
                     touched, args.library, lambda m: print(m))
@@ -415,6 +418,16 @@ def main():
        if os.path.exists(keyfile):
            az_key = Path(keyfile).read_text().strip()

+    # Same cascade for the base URL: --azuracast-base, then env, then the file
+    # credential_service renders from Settings -> Credentials -> AzuraCast.
+    # The scheduled job (scheduler_service) invokes this script with neither
+    # the flag nor the env var set, so the keyfile is the only way it ever
+    # gets one.
+    if not args.azuracast_base:
+        basefile = f"{_ALEMBIC_CONFIG_DIR}/pipeline/azuracast/base_url"
+        if os.path.exists(basefile):
+            args.azuracast_base = Path(basefile).read_text().strip()
+
    cascade = [s.strip() for s in args.sources.split(",") if s.strip() in SOURCES]
    if not cascade:
        sys.exit(f"[enrich] no valid sources in {args.sources!r}")
@@ -518,12 +531,12 @@ def main():
          f"  no-match={looked - found}"
          + (f"  upgraded={upgraded}" if upgrade_from else ""))

-    if args.apply and touched and az_key:
+    if args.apply and touched and az_key and args.azuracast_base:
        print("[enrich] telling AzuraCast to reprocess touched files...")
        az_reprocess(args.azuracast_base, az_key, args.station,
                     touched, args.library, lambda m: print(m))
-    elif args.apply and touched and not az_key:
-        print("[enrich] no AzuraCast key (flag/env/keyfile) — tags written; "
+    elif args.apply and touched and (not az_key or not args.azuracast_base):
+        print("[enrich] no AzuraCast key/base URL (flag/env/keyfile) — tags written; "
              "AzuraCast's periodic media scan will pick them up.")
    elif not args.apply:
        print("[enrich] re-run with --apply to write (key via --azuracast-key, "
@@ -57,11 +57,12 @@ OUT_DIR = f"{_MUSIC_DATA_DIR}/playlists-laptop"
 # Absolute Windows path to the laptop's library root.
 LIBRARY_LAPTOP_ROOT = r"C:\Music\Library"

-# Beets stores paths with this prefix (the container-side mount of the
-# library); strip it to get the path relative to the library root. This is
-# still "/music/" through the migration's Stage 0-3 transitional mount;
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
-BEETS_LIBRARY_PREFIX = "/music/"
+# Prefixes beets may display library paths under, tried in order: the real
+# library dir (everything since the 2026-07-08 path rewrite) and the legacy
+# transitional mount (any stale pre-rewrite entry). Strip whichever matches
+# to get the path relative to the library root; when neither matches the
+# track is outside the library and is skipped.
+BEETS_LIBRARY_PREFIXES = (f"{_MUSIC_DATA_DIR}/Library/", "/music/")

 M3U_EXT = ".m3u"
 # Older formats from earlier iterations of this script — swept by reconcile.
@@ -157,9 +158,12 @@ def build_beets_index() -> dict[tuple[str, str], list[tuple[str, str, str, str]]
        if len(parts) != 4:
            continue
        aa, alb, ti, path = parts
-        if not path.startswith(BEETS_LIBRARY_PREFIX):
+        rel = next(
+            (path[len(p):] for p in BEETS_LIBRARY_PREFIXES if path.startswith(p)),
+            None,
+        )
+        if rel is None:
            continue
-        rel = path[len(BEETS_LIBRARY_PREFIX):]
        idx.setdefault((_normkey(alb), _normkey(ti)), []).append((aa, alb, ti, rel))
    return idx

@@ -49,8 +49,10 @@ _ALEMBIC_CONFIG_DIR = os.environ.get("ALEMBIC_CONFIG_DIR", "/config")
 _MUSIC_DATA_DIR = os.environ.get("MUSIC_DATA_DIR", "/data/music")

 LIBRARY = f"{_MUSIC_DATA_DIR}/Library"
-# Still "/music" through the migration's Stage 0-3 transitional beets mount;
-# revisit at Stage 4 once beets' directory: becomes MUSIC_DATA_DIR/Library.
+# Legacy prefix from the migration's transitional beets mount. Since the
+# 2026-07-08 path rewrite beets displays real LIBRARY paths, so this only
+# matters for accepting pasted /music/... paths as input and as a fallback
+# beets query form for any stale pre-rewrite DB entry.
 CONTAINER_PREFIX = "/music"
 SPOTIFY_ENV = f"{_ALEMBIC_CONFIG_DIR}/pipeline/_spotify.env"
 SPOTIFY_GENRE = f"{os.environ.get('PIPELINE_DIR', '/app/pipeline')}/lib/spotify-genre.py"
@@ -89,7 +91,8 @@ def _spotify_token():
                continue
            k, v = line.split("=", 1)
            env[k] = v.strip().strip("'").strip('"')
-    return get_token(env["SPOTIFY_CLIENT_ID"], env["SPOTIFY_CLIENT_SECRET"])
+    return get_token(env["SPOTIFY_CLIENT_ID"], env["SPOTIFY_CLIENT_SECRET"],
+                      env.get("SPOTIFY_REFRESH_TOKEN") or None)


 def _spotify_get(path, token):
@@ -315,15 +318,19 @@ def resolve_input_path(arg):


 def beets_id_for(host_path):
-    cp = host_to_container(host_path)
-    r = subprocess.run(
-        ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
-        capture_output=True, text=True, check=True)
-    for line in r.stdout.splitlines():
-        if "|||" in line:
-            id_str, p = line.split("|||", 1)
-            if p == cp:
-                return int(id_str)
+    # Real path first (how beets displays everything since the 2026-07-08
+    # path rewrite), legacy /music/... form second for any stale entry.
+    # Querying only the legacy form (the old behavior) matched nothing after
+    # the rewrite, so retag-from-url silently skipped beet update/move.
+    for cp in (host_path, host_to_container(host_path)):
+        r = subprocess.run(
+            ["beet", "ls", "-f", "$id|||$path", f"path:{cp}"],
+            capture_output=True, text=True, check=True)
+        for line in r.stdout.splitlines():
+            if "|||" in line:
+                id_str, p = line.split("|||", 1)
+                if p == cp:
+                    return int(id_str)
    return None


@@ -10,10 +10,21 @@ set -u
 CONFIGS="${ALEMBIC_CONFIG_DIR:-/config}/pipeline"
 LOG="${ALEMBIC_CONFIG_DIR:-/config}/logs/backfill-$(date +%Y%m%d-%H%M%S).log"
 RETAG="${PIPELINE_DIR:-/app/pipeline}/lib/spotify-retag.py"
+SPOTIFY_ENV="${ALEMBIC_CONFIG_DIR:-/config}/pipeline/_spotify.env"

 mkdir -p "$(dirname "$LOG")"
 echo "[$(date -Iseconds)] === Backfill start ===" | tee -a "$LOG"

+# Spotify credentials (client id/secret, and a refresh token if an account is
+# connected via /connect/spotify) are rendered once to _spotify.env as well as
+# into each playlist .conf; read from the single shared file here rather than
+# re-scraping every conf individually.
+[ -f "$SPOTIFY_ENV" ] && source "$SPOTIFY_ENV"
+if [[ -z "${SPOTIFY_CLIENT_ID:-}" || -z "${SPOTIFY_CLIENT_SECRET:-}" ]]; then
+  echo "[$(date -Iseconds)] ERROR: Spotify credentials not found at $SPOTIFY_ENV" | tee -a "$LOG"
+  exit 1
+fi
+
 TOTAL_TRACKS=0
 TOTAL_PLAYLISTS=0

@@ -22,11 +33,9 @@ for conf in "$CONFIGS"/*.conf; do
  playlist=$(basename "$conf" .conf)

  url=$(sed -n 's/^input *= *//p' "$conf" | tr -d ' ')
-  cid=$(sed -n 's/^spotify-id *= *//p' "$conf" | tr -d ' ')
-  csec=$(sed -n 's/^spotify-secret *= *//p' "$conf" | tr -d ' ')

-  if [[ -z "$url" || -z "$cid" || -z "$csec" ]]; then
-    echo "[$playlist] SKIP: missing url/creds in conf" | tee -a "$LOG"
+  if [[ -z "$url" ]]; then
+    echo "[$playlist] SKIP: missing url in conf" | tee -a "$LOG"
    continue
  fi

@@ -41,7 +50,8 @@ for conf in "$CONFIGS"/*.conf; do
  fi

  echo "[$playlist] retagging $count tracks via Spotify" | tee -a "$LOG"
-  if echo "$paths" | SPOTIFY_CLIENT_ID="$cid" SPOTIFY_CLIENT_SECRET="$csec" \
+  if echo "$paths" | SPOTIFY_CLIENT_ID="$SPOTIFY_CLIENT_ID" SPOTIFY_CLIENT_SECRET="$SPOTIFY_CLIENT_SECRET" \
+        SPOTIFY_REFRESH_TOKEN="${SPOTIFY_REFRESH_TOKEN:-}" \
        python3 "$RETAG" "$url" - >> "$LOG" 2>&1; then
    echo "[$playlist] retag OK" | tee -a "$LOG"
    TOTAL_PLAYLISTS=$((TOTAL_PLAYLISTS + 1))
@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Fold every sldl skip index under a playlist's dropbox into one pinned file.
+
+sldl names its per-playlist index folder after the *input*: the Spotify
+playlist's display name for spotify input, the CSV filename stem for csv
+input. So when 0.6.2 switched input from Spotify URL to CSV, sldl started a
+fresh empty index in a new subfolder, saw no download history, and
+re-downloaded every playlist in full (2026-07-16). The rendered confs now pin
+index-path to <dropbox>/<playlist>/_index.csv; this script seeds that pinned
+file from all the indexes sldl left behind (run-playlist.sh calls it whenever
+the pinned file is missing or older than a stranded one).
+
+Usage: merge-sldl-indexes.py <playlist_dropbox_dir> <output_index_csv>
+
+Merge rules, per (artist, title) lowercased:
+  - the row from the newest index file wins;
+  - EXCEPT when that row is a failure (state 2) and any older index has the
+    track as downloaded (state 1 or 3): then the newest row's identity
+    (artist/album/title/length -- matching what the current input will
+    present; Spotify length rounding drifted between the old extractor and
+    our CSV) is kept but marked state 3 (already downloaded), so sldl does
+    not re-fetch a track the library already holds;
+  - rows only present in older indexes are kept as-is (tracks since removed
+    from the playlist; harmless, and they keep their history if re-added).
+"""
+
+import csv
+import sys
+from pathlib import Path
+
+HEADER = ["filepath", "artist", "album", "title", "length", "tracktype", "state", "failurereason"]
+DOWNLOADED_STATES = {"1", "3"}  # 1 = downloaded this run, 3 = found in index previously
+FAILED_STATE = "2"
+
+
+def read_rows(path: Path) -> list[dict]:
+    rows = []
+    with open(path, newline="", encoding="utf-8") as fh:
+        reader = csv.DictReader(fh)
+        for row in reader:
+            if row.get("artist") is None or row.get("title") is None:
+                continue
+            rows.append({k: (row.get(k) or "") for k in HEADER})
+    return rows
+
+
+def norm_key(row: dict) -> tuple:
+    # Primary artist only: sldl's old Spotify extractor recorded just the
+    # first artist, while spotify-playlist-csv.py joins all of them with
+    # ", " -- keying on the full string would miss every multi-artist track
+    # when recovering history across the two index generations. First
+    # comma-segment matches both forms (and both sides of a comma-in-name
+    # artist like "Tyler, The Creator" truncate identically).
+    return (row["artist"].split(",")[0].strip().lower(), row["title"].strip().lower())
+
+
+def main() -> int:
+    if len(sys.argv) != 3:
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    dropbox = Path(sys.argv[1])
+    out_path = Path(sys.argv[2])
+    if not dropbox.is_dir():
+        print(f"[merge-sldl-indexes] not a directory: {dropbox}", file=sys.stderr)
+        return 2
+
+    # Every index at the dropbox root or one level down (sldl's input-named
+    # subfolders), including the pinned output itself if it already exists --
+    # newest first, so the most recent record of each track wins.
+    candidates = sorted(
+        set(dropbox.glob("_index.csv")) | set(dropbox.glob("*/_index.csv")),
+        key=lambda p: p.stat().st_mtime,
+        reverse=True,
+    )
+    if not candidates:
+        print(f"[merge-sldl-indexes] no _index.csv found under {dropbox}; nothing to seed")
+        return 0
+
+    merged: dict[tuple, dict] = {}
+    recovered = 0
+    for path in candidates:
+        try:
+            rows = read_rows(path)
+        except (OSError, csv.Error) as exc:
+            print(f"[merge-sldl-indexes] skipping unreadable {path}: {exc}", file=sys.stderr)
+            continue
+        print(f"[merge-sldl-indexes] {path}: {len(rows)} rows")
+        for row in rows:
+            key = norm_key(row)
+            kept = merged.get(key)
+            if kept is None:
+                merged[key] = row
+            elif kept["state"] == FAILED_STATE and row["state"] in DOWNLOADED_STATES:
+                # Newest attempt failed but an older index proves we already
+                # have this track: keep the newest identity fields, take the
+                # old filepath (informational only; skip-mode index never
+                # checks the file on disk), and mark it downloaded.
+                kept["filepath"] = row["filepath"]
+                kept["state"] = "3"
+                kept["failurereason"] = "0"
+                recovered += 1
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = out_path.with_suffix(".csv.tmp")
+    with open(tmp, "w", newline="", encoding="utf-8") as fh:
+        writer = csv.DictWriter(fh, fieldnames=HEADER)
+        writer.writeheader()
+        writer.writerows(merged.values())
+    tmp.replace(out_path)
+
+    downloaded = sum(1 for r in merged.values() if r["state"] in DOWNLOADED_STATES)
+    print(
+        f"[merge-sldl-indexes] wrote {out_path}: {len(merged)} tracks "
+        f"({downloaded} downloaded, {recovered} recovered from older indexes)"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -5,14 +5,29 @@
 # Usage:
 #   echo "single line message" | notify-telegram.sh
 #   notify-telegram.sh "single line message"
-#   notify-telegram.sh < ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
+#   notify-telegram.sh --html < ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
 #
-# Telegram messages are capped at 4096 chars. Anything longer is truncated
-# with a "...<truncated>" tail. Returns exit 0 on send-OK, non-zero otherwise.
+# --html sends with parse_mode=HTML for messages authored as Telegram-HTML
+# (pipeline-status.sh's digest: <b>/<i>/<code>/<pre>/<blockquote>). The
+# CALLER is responsible for escaping &, <, > in any free text; this script
+# only guarantees the truncation below can't cut a tag in half. Without the
+# flag, messages go as plain text exactly as before.
+#
+# Telegram messages are capped at 4096 chars. Anything longer is truncated;
+# in HTML mode the cut lands on a line boundary and re-closes an open <pre>
+# so the truncated message still parses (a mid-tag cut, or a bare "<" in the
+# tail marker, makes the Bot API reject the ENTIRE message with a 400).
+# Returns exit 0 on send-OK, non-zero otherwise.

 set -euo pipefail
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

+PARSE_MODE=""
+if [[ "${1:-}" == "--html" ]]; then
+  PARSE_MODE="HTML"
+  shift
+fi
+
 CONFIG="${ALEMBIC_CONFIG_DIR:-/config}/pipeline/telegram/notify.env"
 if [[ ! -f "$CONFIG" ]]; then
  echo "[notify-telegram] no config at $CONFIG" >&2
@@ -30,24 +45,37 @@ else
  MSG=$(cat)
 fi

-# Telegram caps at 4096 chars (UTF-8 codepoints). Trim conservatively at 3900.
-# Use python for proper UTF-8 length handling.
+# Telegram caps at 4096 chars. Trim conservatively at 3900. Use python for
+# proper UTF-8 length handling. Plain mode appends a literal marker; HTML
+# mode cuts at the last newline inside the budget (our tags never span
+# lines except <pre> blocks) and re-closes an unbalanced <pre>.
 MSG=$(python3 -c "
 import sys
 s = sys.argv[1]
+html = sys.argv[2] == 'HTML'
 if len(s) > 3900:
-    s = s[:3900] + '\n...<truncated>'
+    s = s[:3900]
+    if html:
+        cut = s.rfind('\n')
+        if cut > 0:
+            s = s[:cut]
+        if s.count('<pre>') > s.count('</pre>'):
+            s += '</pre>'
+        s += '\n<i>… truncated</i>'
+    else:
+        s += '\n...(truncated)'
 print(s, end='')
-" "$MSG")
+" "$MSG" "${PARSE_MODE:-plain}")

-# Send via Bot API. Disable web-page preview and use plain text (no parse_mode)
-# so log content with special chars doesn't get interpreted as Markdown.
+# Send via Bot API. Preview disabled; parse_mode only when requested so log
+# content with special chars can't be misread as markup in plain sends.
 # `|| true`: a curl timeout/network error must fall through to the explicit
 # ok-check below (which reports "send failed" and exits 2), not abort here.
 resp=$(curl -s --max-time 15 -X POST \
  "https://api.telegram.org/bot${TG_BOT_TOKEN}/sendMessage" \
  --data-urlencode "chat_id=${TG_CHAT_ID}" \
  --data-urlencode "text=${MSG}" \
+  ${PARSE_MODE:+--data-urlencode "parse_mode=${PARSE_MODE}"} \
  --data-urlencode "disable_web_page_preview=true" || true)

 ok=$(echo "$resp" | python3 -c "import sys,json;print(json.load(sys.stdin).get('ok',False))" 2>/dev/null || echo False)
@@ -4,9 +4,24 @@
 # Writes a one-screen summary to ${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log (overwritten daily)
 # and ships it to Telegram via notify-telegram.sh.
 #
+# Formatting note: the digest uses Telegram HTML entities (<b>, <i>, <code>,
+# <blockquote>) — the brand's chrome/dot-badge language translated into what
+# Telegram can actually render (no color, no custom font, so status "dots"
+# become colored circle emoji). Deliberately NO <pre>/monospace column layout:
+# a code block forces fixed-width columns that wrap into unreadable garbage on
+# a phone ("54 tracks in / M3U"). Instead every row is plain reflowing text —
+# a bold label + " — value" — so it wraps cleanly at any screen width. This
+# REQUIRES notify-telegram.sh to be called with --html (parse_mode=HTML) —
+# sent as plain text the tags would show up literally. All free-text going
+# through mark_ok/mark_warn/mark_skip/section is escaped (esc()) for &, <, >
+# so a stray angle bracket in a log line can't break the HTML parse and
+# swallow the whole message.
+#
 # Reports on:
-#   - Sibling service reachability (slskd, navidrome) via HTTP, not docker ps —
-#     alembic has no Docker socket access
+#   - Sibling service reachability (navidrome) via HTTP, not docker ps —
+#     alembic has no Docker socket access. slskd is deliberately NOT checked:
+#     it is not part of the pipeline (sldl is its own Soulseek client) — it
+#     runs on the host purely as the user's own file-sharing presence.
 #   - Today's runs: playlist syncs, Bandcamp sync, manual imports, dedup
 #   - Weekly maintenance freshness: strip-mb-tags, strip-watermark-art,
 #     scrub-watermark-text, clean-sldl-index, spotify-genre
@@ -27,7 +42,10 @@
 # the opposite of what a health report should do. It writes no library state,
 # so there is nothing to leave half-applied.
 set -u
-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+# /opt/venv/bin first: `beet` lives in the app venv. Without it every beet
+# probe here ("added today", "Library by format", the mp3-now count) silently
+# came up empty behind its 2>/dev/null.
+PATH=/opt/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

 OUT=${ALEMBIC_CONFIG_DIR:-/config}/logs/STATUS.log
 TODAY=$(date +%Y%m%d)
@@ -44,12 +62,39 @@ OK=0
 WARN=0
 SKIP=0

-mark_ok()   { OK=$((OK+1));     printf "  ✓ %s\n" "$*"; }
-mark_warn() { WARN=$((WARN+1)); printf "  ⚠ %s\n" "$*"; }
-mark_skip() { SKIP=$((SKIP+1)); printf "  ⊘ %s\n" "$*"; }
-mark_info() {                   printf "    %s\n" "$*"; }
+# Escapes &, <, > so free text (log snippets, exception messages, filenames)
+# can't be mistaken for an HTML entity by Telegram's parser.
+esc() {
+  local s=$1
+  s=${s//&/&amp;}
+  s=${s//</&lt;}
+  s=${s//>/&gt;}
+  printf '%s' "$s"
+}

-section() { printf "\n▎ %s\n" "$*"; }
+# One status row: a colored dot, a bold label, and (optionally) " — value".
+# Plain proportional text — NO padding, NO monospace — so it reflows on mobile
+# instead of wrapping mid-column. The dot is the brand's badge-state color,
+# emoji being Telegram's only color channel.
+_row() {
+  local dot=$1 label=$2 detail=${3:-}
+  if [[ -n "$detail" ]]; then
+    printf "%s <b>%s</b> — %s\n" "$dot" "$(esc "$label")" "$(esc "$detail")"
+  else
+    printf "%s <b>%s</b>\n" "$dot" "$(esc "$label")"
+  fi
+}
+mark_ok()   { OK=$((OK+1));     _row "🟢" "$@"; }
+mark_warn() { WARN=$((WARN+1)); _row "🟠" "$@"; }
+mark_skip() { SKIP=$((SKIP+1)); _row "⚪" "$@"; }
+# Free-text info line (no dot, no counter) — for sub-breakdowns.
+mark_info() { printf "   %s\n" "$(esc "$*")"; }
+
+# Bold section header (reads like the web app's <h2>). No <pre> — the rows
+# under it are plain reflowing text.
+section() {
+  printf "\n<b>▎ %s</b>\n" "$(esc "$*")"
+}

 # syslog / `logger` is not present in the container image. Only call it if it
 # exists, so these lines don't spew "logger: command not found" into the job
@@ -70,17 +115,6 @@ human_age() {
  fi
 }

-# Newest matching log → "<age_seconds>|<path>", empty if no match.
-newest_log() {
-  local glob="$1"
-  local newest
-  newest=$(ls -1t $glob 2>/dev/null | head -1)
-  [[ -z "$newest" ]] && return
-  local mtime
-  mtime=$(stat -c %Y "$newest" 2>/dev/null) || return
-  echo "$((NOW_TS - mtime))|$newest"
-}
-
 # Per-playlist log status.
 playlist_status() {
  local log="$1"
@@ -91,7 +125,7 @@ playlist_status() {
    m3u=$(awk 'match($0, /updated with [0-9]+ tracks/) {
                 n=substr($0, RSTART, RLENGTH); gsub(/[^0-9]/, "", n); last=n
               } END { print last }' "$log")
-    echo "OK|${m3u:-0} tracks in M3U"
+    echo "OK|${m3u:-0} tracks"
  elif grep -q "=== Starting playlist run" "$log" 2>/dev/null; then
    echo "WARN|started but never finished"
  else
@@ -151,12 +185,11 @@ dedup_today_status() {
  check_http() {
    local name="$1" url="$2"
    if curl -s -o /dev/null --connect-timeout 3 --max-time 6 "$url"; then
-      mark_ok "$(printf '%-11s reachable' "$name")"
+      mark_ok "$name" "reachable"
    else
-      mark_warn "$(printf '%-11s unreachable at %s' "$name" "$url")"
+      mark_warn "$name" "unreachable at $url"
    fi
  }
-  check_http slskd   "http://gluetun:5030/"
  check_http navidrome "http://navidrome:4533/rest/ping.view"

  # 2. Today's runs
@@ -197,14 +230,12 @@ dedup_today_status() {
        fi
        ;;
    esac
-    line=$(printf '%-13s %s' "$short" "$detail")
-    if [[ "$tag" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
+    if [[ "$tag" == "OK" ]]; then mark_ok "$short" "$detail"; else mark_warn "$short" "$detail"; fi
  done
  if [[ -n "$newest_dedup_today" ]]; then
    any_today=1
    IFS='|' read -r tag detail < <(dedup_today_status "$newest_dedup_today")
-    line=$(printf '%-13s %s' "dedup" "$detail")
-    if [[ "$tag" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
+    if [[ "$tag" == "OK" ]]; then mark_ok "dedup" "$detail"; else mark_warn "dedup" "$detail"; fi
  fi
  [[ "$any_today" -eq 0 ]] && mark_info "(nothing has run yet today)"

@@ -216,30 +247,28 @@ dedup_today_status() {
  ADDED_TODAY=$(beet ls -f '$grouping' \
    "added:$(date +%Y-%m-%d).." 2>/dev/null || true)
  added_total=$(echo -n "$ADDED_TODAY" | grep -c '^' || true)
-  mark_ok "$(printf '%-13s %d new tracks in beets' 'added today' "$added_total")"
+  mark_ok "added today" "$added_total new tracks in beets"
  if [[ "$added_total" -gt 0 ]]; then
    # Per-playlist breakdown. Empty $grouping (bandcamp/manual) shown as "(none)".
+    # Piped through the same &/</> escaping as esc(), since this prints
+    # straight into the message without going through _row.
    echo "$ADDED_TODAY" \
      | awk '{ if ($0 == "") print "(none)"; else print }' \
      | sort | uniq -c | sort -rn \
-      | awk '{ g=$2; for(i=3;i<=NF;i++) g=g" "$i; printf "                  %3d  %s\n", $1, g }'
+      | awk '{ c=$1; g=$2; for(i=3;i<=NF;i++) g=g" "$i; printf "   • %s (%d)\n", g, c }' \
+      | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g'
  fi

  # 3. Maintenance freshness
-  # Each weekly/monthly task: find its newest log and check age.
+  # Read from the app's job_runs table (the scheduler's own record) instead of
+  # globbing for log files. Jobs run through pipeline_runner write their logs
+  # to logs/<job_key>/<timestamp>.log, so the old cron-era filename globs
+  # matched nothing for jobs that don't also write their own log, and healthy
+  # jobs were reported as "never run yet". The DB also distinguishes lock
+  # skips and failures, which a missing log file can't.
  # Limits: weekly tasks should be <9 days old; daily <2; monthly <35.
  section "Maintenance (last run)"

-  declare -A MAINT_GLOB=(
-    [strip-mb-tags]="${ALEMBIC_CONFIG_DIR:-/config}/logs/mb-strip-*.log"
-    [strip-watermark-art]="${ALEMBIC_CONFIG_DIR:-/config}/logs/strip-watermark-*.log"
-    [scrub-watermark-text]="${ALEMBIC_CONFIG_DIR:-/config}/logs/scrub-text-*.log"
-    [clean-sldl-index]="${ALEMBIC_CONFIG_DIR:-/config}/logs/clean-index-*.log"
-    [clear-bad-genres]="${ALEMBIC_CONFIG_DIR:-/config}/logs/clear-bad-genres-*.log"
-    [spotify-genre]="${ALEMBIC_CONFIG_DIR:-/config}/logs/spotify-genre-*.log"
-    [dedup-library]="${ALEMBIC_CONFIG_DIR:-/config}/logs/dedup-*.log"
-    [upgrade-mp3-to-flac]="${ALEMBIC_CONFIG_DIR:-/config}/logs/upgrade-mp3-*.log"
-  )
  declare -A MAINT_LIMIT=(
    [strip-mb-tags]=$((9*86400))
    [strip-watermark-art]=$((9*86400))
@@ -250,45 +279,110 @@ dedup_today_status() {
    [dedup-library]=$((2*86400))
    [upgrade-mp3-to-flac]=$((35*86400))
  )
+  # Emits one line per task: task|latest_status|latest_age_s|success_age_s|success_log
+  # (ages are -1 when there is no such run). Read-only DB open; prints nothing
+  # if the DB is missing so every task falls through to "never run yet".
+  maint_rows=$(python3 - "${ALEMBIC_CONFIG_DIR:-/config}/alembic.db" <<'PYEOF'
+import sqlite3
+import sys
+import time
+
+# Digest label -> job_runs.job_key. The scheduled genre refresh records under
+# genre:run (via genre_review_service) and dedup under dedup:scan (via
+# dedup_review_service); everything else is its MAINTENANCE_JOBS key.
+KEYS = [
+    ("strip-mb-tags", "maintenance:strip_mb_tags"),
+    ("strip-watermark-art", "maintenance:strip_watermark_art"),
+    ("scrub-watermark-text", "maintenance:scrub_watermark_text"),
+    ("clean-sldl-index", "maintenance:clean_sldl_index"),
+    ("clear-bad-genres", "maintenance:clear_bad_genres"),
+    ("spotify-genre", "genre:run"),
+    ("dedup-library", "dedup:scan"),
+    ("upgrade-mp3-to-flac", "maintenance:upgrade_mp3_to_flac"),
+]
+now = time.time()
+try:
+    conn = sqlite3.connect(f"file:{sys.argv[1]}?mode=ro", uri=True)
+    conn.execute("SELECT 1 FROM job_runs LIMIT 1")
+except sqlite3.Error:
+    sys.exit(0)
+for task, key in KEYS:
+    latest = conn.execute(
+        "SELECT status, started_at FROM job_runs WHERE job_key = ? ORDER BY started_at DESC LIMIT 1",
+        (key,),
+    ).fetchone()
+    if latest is None:
+        print(f"{task}|none|-1|-1|")
+        continue
+    success = conn.execute(
+        "SELECT started_at, log_path FROM job_runs WHERE job_key = ? AND status = 'success' "
+        "ORDER BY started_at DESC LIMIT 1",
+        (key,),
+    ).fetchone()
+    s_age = int(now - success[0]) if success else -1
+    s_log = (success[1] or "") if success else ""
+    print(f"{task}|{latest[0]}|{int(now - latest[1])}|{s_age}|{s_log}")
+PYEOF
+)
+
  for task in strip-mb-tags strip-watermark-art scrub-watermark-text clean-sldl-index clear-bad-genres spotify-genre dedup-library upgrade-mp3-to-flac; do
-    res=$(newest_log "${MAINT_GLOB[$task]}")
-    if [[ -z "$res" ]]; then
-      mark_skip "$(printf '%-22s never run yet' "$task")"
+    row=$(grep "^${task}|" <<< "$maint_rows" || true)
+    IFS='|' read -r _ latest_status latest_age age_s log <<< "$row"
+    if [[ -z "$row" || "$latest_status" == "none" ]]; then
+      mark_skip "$task" "never run yet"
+      continue
+    fi
+    if [[ "$age_s" -lt 0 ]]; then
+      # attempted, but never once succeeded -- say what the last attempt did
+      case "$latest_status" in
+        skipped_lock) note="skipped, lock busy" ;;
+        running)      note="running now" ;;
+        *)            note="$latest_status" ;;
+      esac
+      mark_warn "$task" "never succeeded (last attempt $(human_age "$latest_age"): $note)"
      continue
    fi
-    age_s=${res%%|*}
-    log=${res##*|}
    age_h=$(human_age "$age_s")
-    case "$task" in
-      spotify-genre)
-        snip=$(grep -oE "written: [0-9]+, unchanged: [0-9]+" "$log" | tail -1) ;;
-      clear-bad-genres)
-        snip=$(grep -oE "(would-clear|cleared): [0-9]+ tracks" "$log" | tail -1) ;;
-      clean-sldl-index)
-        snip=$(grep -oE "[0-9]+ m3us, total [0-9]+ kept, [0-9]+ dropped" "$log" | tail -1) ;;
-      strip-watermark-art)
-        snip=$(grep -oE "stripped from [0-9]+ tracks|no images shared" "$log" | tail -1) ;;
-      scrub-watermark-text)
-        snip=$(grep -oE "stripped [0-9]+ frame\(s\) across [0-9]+ file" "$log" | tail -1) ;;
-      strip-mb-tags)
-        snip=$(grep -oE "Done\. Log:" "$log" | head -1 | sed 's/Done\. Log:/done/') ;;
-      dedup-library)
-        snip=$(grep "=== Summary:" "$log" | tail -1 \
-               | sed -E 's/.*=== Summary: //; s/ ===.*//; s/ \([^)]*\)//') ;;
-      upgrade-mp3-to-flac)
-        # The log's "to attempt" count is frozen at the last monthly run, so on
-        # its own it looks stale against the live "Library by format" section.
-        # Pair it with the current format:mp3 count so the drop (upgrades that
-        # have landed since) is visible instead of looking like a mismatch.
-        attempted=$(grep -oE "[0-9]+ MP3 tracks to attempt" "$log" | grep -oE "^[0-9]+" | tail -1)
-        mp3_now=$(beet ls 'format:mp3' 2>/dev/null | grep -c '^')
-        snip="${attempted:-?} attempted · ${mp3_now} MP3 now" ;;
+    snip=""
+    if [[ -n "$log" && -f "$log" ]]; then
+      case "$task" in
+        spotify-genre)
+          snip=$(grep -oE "written: [0-9]+, unchanged: [0-9]+" "$log" | tail -1) ;;
+        clear-bad-genres)
+          snip=$(grep -oE "rewrote: [0-9]+ tracks, blanked: [0-9]+ tracks|(would-clear|cleared): [0-9]+ tracks" "$log" | tail -1) ;;
+        clean-sldl-index)
+          snip=$(grep -oE "[0-9]+ m3us, total [0-9]+ kept, [0-9]+ dropped" "$log" | tail -1) ;;
+        strip-watermark-art)
+          snip=$(grep -oE "stripped from [0-9]+ tracks|no images shared" "$log" | tail -1) ;;
+        scrub-watermark-text)
+          snip=$(grep -oE "stripped [0-9]+ frame\(s\) across [0-9]+ file" "$log" | tail -1) ;;
+        strip-mb-tags)
+          snip=$(grep -oE "Done\. Log:" "$log" | head -1 | sed 's/Done\. Log:/done/') ;;
+        dedup-library)
+          snip=$(grep "=== Summary:" "$log" | tail -1 \
+                 | sed -E 's/.*=== Summary: //; s/ ===.*//; s/ \([^)]*\)//') ;;
+        upgrade-mp3-to-flac)
+          # The log's "to attempt" count is frozen at the last monthly run, so on
+          # its own it looks stale against the live "Library by format" section.
+          # Pair it with the current format:mp3 count so the drop (upgrades that
+          # have landed since) is visible instead of looking like a mismatch.
+          attempted=$(grep -oE "[0-9]+ MP3 tracks to attempt" "$log" | grep -oE "^[0-9]+" | tail -1)
+          mp3_now=$(beet ls 'format:mp3' 2>/dev/null | grep -c '^')
+          snip="${attempted:-?} attempted · ${mp3_now} MP3 now" ;;
+      esac
+    fi
+    # Age/snippet describe the last SUCCESS; if a newer attempt failed or was
+    # lock-skipped, say so rather than hiding it behind the healthy line.
+    case "$latest_status" in
+      failed)       snip="${snip:+$snip, }latest attempt failed" ;;
+      skipped_lock) snip="${snip:+$snip, }latest attempt skipped, lock busy" ;;
    esac
-    line=$(printf '%-22s %-9s %s' "$task" "$age_h" "${snip:-}")
-    if (( age_s > MAINT_LIMIT[$task] )); then
-      mark_warn "$line"
+    detail="$age_h"
+    [[ -n "$snip" ]] && detail="$age_h · $snip"
+    if (( age_s > MAINT_LIMIT[$task] )) || [[ "$latest_status" == "failed" ]]; then
+      mark_warn "$task" "$detail"
    else
-      mark_ok "$line"
+      mark_ok "$task" "$detail"
    fi
  done

@@ -300,13 +394,16 @@ dedup_today_status() {
    exp=$(awk -F'\t' '$6=="identity" {print $5; exit}' ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)
    if [[ -n "${exp:-}" && "$exp" =~ ^[0-9]+$ && "$exp" -gt 0 ]]; then
      days=$(( (exp - NOW_TS) / 86400 ))
-      line=$(printf '%-22s %d days left' "Bandcamp cookie" "$days")
-      if (( days < 14 )); then mark_warn "$line — re-export soon"; else mark_ok "$line"; fi
+      if (( days < 14 )); then
+        mark_warn "Bandcamp cookie" "$days days left — re-export soon"
+      else
+        mark_ok "Bandcamp cookie" "$days days left"
+      fi
    else
-      mark_warn "Bandcamp cookie         could not parse expiry"
+      mark_warn "Bandcamp cookie" "could not parse expiry"
    fi
  else
-    mark_warn "Bandcamp cookie         missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)"
+    mark_warn "Bandcamp cookie" "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/bandcamp/cookies.txt)"
  fi

  # Qobuz Web Player token (buy-link enrich cascade #2). The token is opaque
@@ -326,12 +423,12 @@ dedup_today_status() {
      -H "X-App-Id: $q_appid" -H "X-User-Auth-Token: $q_token" -w '%{http_code}' \
      "https://www.qobuz.com/api.json/0.2/track/search?query=test&limit=1&app_id=$q_appid" 2>/dev/null)
    case "$q_code" in
-      200) mark_ok   "$(printf '%-22s %s' 'Qobuz token' 'valid (buy-link lookup live)')" ;;
-      401) mark_warn "$(printf '%-22s %s' 'Qobuz token' "EXPIRED — re-export X-User-Auth-Token to ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token")" ;;
-      *)   mark_warn "$(printf '%-22s %s' 'Qobuz token' "check failed (HTTP ${q_code:-none})")" ;;
+      200) mark_ok   "Qobuz token" "valid (buy-link lookup live)" ;;
+      401) mark_warn "Qobuz token" "EXPIRED — re-export X-User-Auth-Token to ${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token" ;;
+      *)   mark_warn "Qobuz token" "check failed (HTTP ${q_code:-none})" ;;
    esac
  else
-    mark_warn "$(printf '%-22s %s' 'Qobuz token' "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token)")"
+    mark_warn "Qobuz token" "missing (${ALEMBIC_CONFIG_DIR:-/config}/pipeline/qobuz/token)"
  fi

  # VPN egress
@@ -341,17 +438,20 @@ dedup_today_status() {
  # asking Docker to introspect gluetun's health, and needs no Docker socket.
  egress_ip=$(curl -s --connect-timeout 5 --max-time 10 https://ifconfig.me 2>/dev/null)
  if [[ -n "$egress_ip" ]]; then
-    mark_ok "$(printf '%-22s %s' 'VPN egress' "$egress_ip")"
+    mark_ok "VPN egress" "$egress_ip"
  else
-    mark_warn "$(printf '%-22s %s' 'VPN egress' 'could not determine egress IP — VPN may be down')"
+    mark_warn "VPN egress" "could not determine egress IP — VPN may be down"
  fi

  # 5. Storage
  section "Storage"
  while read -r mp pct used total; do
-    line=$(printf '%-22s %s used (%s of %s)' "$mp" "$pct" "$used" "$total")
    pct_n=${pct%\%}
-    if (( pct_n > 90 )); then mark_warn "$line"; else mark_ok "$line"; fi
+    if (( pct_n > 90 )); then
+      mark_warn "$mp" "$pct used ($used of $total)"
+    else
+      mark_ok "$mp" "$pct used ($used of $total)"
+    fi
  done < <(df -h ${MUSIC_DATA_DIR:-/data/music} / 2>/dev/null | awk 'NR>1 && $1!~/tmpfs/ {print $6, $5, $3, $2}')

  # 6. Navidrome
@@ -376,24 +476,40 @@ except Exception as e:
    print(f\"WARN|unreachable: {e}\")
 " 2>/dev/null)
  state=${ND_LINE%%|*}; detail=${ND_LINE##*|}
-  line=$(printf '%-22s %s' "Navidrome" "$detail")
-  if [[ "$state" == "OK" ]]; then mark_ok "$line"; else mark_warn "$line"; fi
+  if [[ "$state" == "OK" ]]; then mark_ok "Navidrome" "$detail"; else mark_warn "Navidrome" "$detail"; fi

-  # 7. Library by format (info-only, no status pill)
+  # 7. Library by format (info-only, no status dot)
  section "Library by format"
  beet ls -f '$format' 2>/dev/null | sort | uniq -c | sort -rn \
-    | awk '{printf "    %-6s %d tracks\n", $2, $1}'
+    | awk '{printf "   • %s — %d tracks\n", $2, $1}' \
+    | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g'

 } > "$OUT"

 # ---- Build header banner and prepend ----
+# Bold brand mark (⚗️ is literally the "ALEMBIC" unicode glyph — no generic
+# music-note stand-in needed), a hostname chip in <code>, and an italic tally
+# that reuses the same 🟢/🟠/⚪ dots as the body. When something needs
+# attention, a <blockquote> callout mirrors the web app's .notice.warning
+# panel; an all-clear run gets the calm .notice.success equivalent instead.
 HOSTNAME_SHORT=$(hostname -s)
 DATE_HUMAN=$(TZ="${TZ:-UTC}" date '+%a %Y-%m-%d %H:%M %Z')
-BANNER_TITLE="🎵 Music pipeline · $HOSTNAME_SHORT · $DATE_HUMAN"
-BANNER_TALLY="   $OK OK · $WARN warn · $SKIP skip"
-sed -i "1c\\
-${BANNER_TITLE}\\
-${BANNER_TALLY}" "$OUT"
+HEADER="⚗️ <b>Alembic</b> · music pipeline · <code>${HOSTNAME_SHORT}</code> · ${DATE_HUMAN}
+<i>🟢 ${OK} OK   🟠 ${WARN} warn   ⚪ ${SKIP} skip</i>"
+if (( WARN > 0 )); then
+  HEADER="${HEADER}
+<blockquote>🟠 <b>${WARN} issue(s)</b> need attention — see below</blockquote>"
+else
+  HEADER="${HEADER}
+<blockquote>🟢 All clear — nothing needs attention.</blockquote>"
+fi
+
+# Swap the __HEADER_PLACEHOLDER__ line for the (2-3 line) banner above. A
+# straight prepend, not sed 1c — the banner's line count varies with the
+# blockquote, which sed's `c` command can't take as a variable easily.
+tail -n +2 "$OUT" > "${OUT}.body"
+{ printf '%s\n' "$HEADER"; cat "${OUT}.body"; } > "$OUT"
+rm -f "${OUT}.body"

 # Always mirror the one-line summary to syslog so journalctl shows last-run
 # state even if Telegram is broken. Warnings get a separate WARN tag.
@@ -402,10 +518,12 @@ if (( WARN > 0 )); then
  slog "WARN: pipeline-status flagged $WARN issue(s) — see $OUT"
 fi

-# Send to Telegram. If it fails, log the failure to syslog so the absence of
-# a message in the chat has a corresponding journal entry to grep for.
+# Send to Telegram with --html: the digest is Telegram-HTML (see header
+# comment), so notify-telegram.sh must send it with parse_mode=HTML. If the
+# send fails, log to syslog so the absence of a message in the chat has a
+# corresponding journal entry to grep for.
 if [[ -x ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh ]]; then
-  if ! ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh < "$OUT" 2>/tmp/tg-err; then
+  if ! ${PIPELINE_DIR:-/app/pipeline}/lib/notify-telegram.sh --html < "$OUT" 2>/tmp/tg-err; then
    slog "WARN: Telegram send failed: $(cat /tmp/tg-err 2>/dev/null | head -c 200)"
    rm -f /tmp/tg-err
  fi
@@ -195,7 +195,18 @@ while IFS= read -r -d '' new_file; do
  existing_id="${match%%$'\x1f'*}"
  existing_container="${match#*$'\x1f'}"

-  existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  # beets displays real ${MUSIC_DATA_DIR}/Library/... paths since the
+  # 2026-07-08 path rewrite -- use them as-is; only a legacy /music/...
+  # display path still needs the prefix swap. Blindly prepending (the old
+  # behavior) doubled the prefix, so every match logged [stale-db], the MP3
+  # never got replaced, and the leftover-import step at the bottom imported
+  # the staged FLAC as a NEW track -- producing exactly the flac+mp3 twins
+  # this script exists to prevent.
+  if [[ "$existing_container" == /music/* ]]; then
+    existing="${MUSIC_DATA_DIR:-/data/music}/Library${existing_container#/music}"
+  else
+    existing="$existing_container"
+  fi
  if [[ ! -f "$existing" ]]; then
    echo "[stale-db]       beets has $existing_container but file missing" | tee -a "$LOG"
    continue
@@ -54,9 +54,9 @@ def _load_spotify_creds():
                continue
            k, v = line.split("=", 1)
            env[k] = v.strip().strip("'").strip('"')
-    return env["SPOTIFY_CLIENT_ID"], env["SPOTIFY_CLIENT_SECRET"]
+    return env["SPOTIFY_CLIENT_ID"], env["SPOTIFY_CLIENT_SECRET"], env.get("SPOTIFY_REFRESH_TOKEN") or None

-SPOTIFY_CID, SPOTIFY_CSEC = _load_spotify_creds()
+SPOTIFY_CID, SPOTIFY_CSEC, SPOTIFY_REFRESH = _load_spotify_creds()
 WHITELIST_FILE = f"{ALEMBIC_CONFIG_DIR}/pipeline/genres-whitelist.txt"
 MAX_GENRES = 3
 SEPARATOR = "; "
@@ -76,7 +76,7 @@ def load_whitelist():


 def _spotify_token():
-    return get_token(SPOTIFY_CID, SPOTIFY_CSEC)
+    return get_token(SPOTIFY_CID, SPOTIFY_CSEC, SPOTIFY_REFRESH)


 # Cache: artist_name → list[str] of Spotify genres (or None if not found)
@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""
+spotify-playlist-csv.py — dump a Spotify playlist to a CSV sldl can search from.
+
+sldl's own vendored Spotify client still calls Spotify's GET /playlists/{id}/tracks
+endpoint, which Spotify removed in its February 2026 API changes (see
+https://developer.spotify.com/documentation/web-api/references/changes/february-2026)
+in favor of /items. Grandfathered apps still get a pass on /tracks for now, but
+apps created after that change get a hard 403 there regardless of auth method --
+client-credentials, or a correctly-scoped, correctly-owned OAuth user token, none
+of it matters, because the endpoint itself is gone for them. sldl has no separate
+code path to fall back to /items, so it can never read a playlist for a new app
+no matter what alembic hands it.
+
+Rather than depend on sldl's Spotify client at all, this script reads the
+playlist itself (via /items, which alembic's own code already migrated to) and
+writes the same Artist,Title,Album,Length CSV format upgrade-mp3-to-flac.sh
+already feeds to sldl via --input-type csv. run-playlist.sh runs this first and
+then points sldl at the CSV instead of the playlist URL, sidestepping sldl's
+Spotify client entirely -- for grandfathered and new apps alike.
+
+Usage:
+  spotify-playlist-csv.py <playlist_url> <out_csv>
+
+Credentials are read from env vars SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET.
+SPOTIFY_REFRESH_TOKEN, if set, mints a user token instead of client-credentials
+(required for newly created Spotify apps -- see _spotify_auth.get_token).
+"""
+import csv
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.request
+
+from _spotify_auth import get_token
+
+API = "https://api.spotify.com/v1"
+
+
+def fetch_playlist(url: str, token: str) -> list[dict]:
+    m = re.search(r"playlist[/:]([A-Za-z0-9]+)", url)
+    if not m:
+        sys.exit(f"Could not parse playlist ID from {url!r}")
+    pid = m.group(1)
+    tracks = []
+    # No `fields` filter: Spotify's February 2026 changes renamed each entry's
+    # payload from "track" to "item" for apps on the new behavior (extended
+    # quota / grandfathered apps keep "track"), and a fields filter selects by
+    # key name -- filtering on track(...) silently returns EMPTY pages on the
+    # renamed shape. Fetch unfiltered and parse both key names below.
+    next_url = f"{API}/playlists/{pid}/items?limit=100"
+    while next_url:
+        req = urllib.request.Request(next_url, headers={"Authorization": f"Bearer {token}"})
+        try:
+            with urllib.request.urlopen(req, timeout=15) as r:
+                data = json.loads(r.read())
+        except urllib.error.HTTPError as e:
+            if e.code in (403, 404):
+                sys.exit(
+                    f"Spotify returned {e.code} loading this playlist. The connected "
+                    "Spotify account can't see it: it must own the playlist, be a "
+                    "collaborator on it, or (for grandfathered apps) the playlist "
+                    "must be Public. Check which account is connected via "
+                    "/connect/spotify."
+                )
+            raise
+        if "items" not in data:
+            # New-behavior apps get metadata only (no items field) for
+            # playlists the connected account doesn't own or collaborate on.
+            sys.exit(
+                "Spotify returned this playlist without its contents. For newly "
+                "created Spotify apps, the connected account must OWN the playlist "
+                "or be a collaborator on it (public is no longer enough). Ask the "
+                "owner to make it collaborative and add you, or recreate the "
+                "playlist under the connected account."
+            )
+        for item in data["items"]:
+            t = item.get("track") or item.get("item")
+            if not t or t.get("is_local"):
+                continue
+            # All artists, comma-joined -- matches what sldl's own Spotify
+            # extractor fed its search, so multi-artist tracks match no worse
+            # than they did before the CSV detour.
+            artists = [a["name"] for a in (t.get("artists") or []) if a.get("name")]
+            tracks.append(
+                {
+                    "artist": ", ".join(artists),
+                    "title": t.get("name", ""),
+                    "album": (t.get("album") or {}).get("name", ""),
+                    "length": round((t.get("duration_ms") or 0) / 1000),
+                }
+            )
+        next_url = data.get("next")
+    return tracks
+
+
+def main() -> int:
+    if len(sys.argv) != 3:
+        sys.exit(f"Usage: {sys.argv[0]} <playlist_url> <out_csv>")
+    url, out_path = sys.argv[1], sys.argv[2]
+
+    cid = os.environ.get("SPOTIFY_CLIENT_ID")
+    csec = os.environ.get("SPOTIFY_CLIENT_SECRET")
+    refresh = os.environ.get("SPOTIFY_REFRESH_TOKEN") or None
+    if not cid or not csec:
+        sys.exit("SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET must be set")
+
+    token = get_token(cid, csec, refresh)
+    tracks = fetch_playlist(url, token)
+
+    with open(out_path, "w", newline="") as f:
+        writer = csv.writer(f)
+        writer.writerow(["Artist", "Title", "Album", "Length"])
+        for t in tracks:
+            writer.writerow([t["artist"], t["title"], t["album"], t["length"]])
+
+    print(f"[spotify-playlist-csv] wrote {len(tracks)} tracks to {out_path}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -22,6 +22,8 @@ Usage:
  spotify-retag.py <playlist_url> -              # read newline-separated paths from stdin

 Credentials are read from env vars SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET.
+SPOTIFY_REFRESH_TOKEN, if set, mints a user token instead of client-credentials
+(required for newly created Spotify apps -- see _spotify_auth.get_token).

 Matching strategy (per file):
  1. Parse "Artist - Title.ext" from filename. If multiple " - " separators,
@@ -46,15 +48,19 @@ def fetch_playlist(url: str, token: str) -> list[dict]:
        sys.exit(f"Could not parse playlist ID from {url!r}")
    pid = m.group(1)
    tracks = []
-    next_url = f"{API}/playlists/{pid}/tracks?limit=100"
+    # /items (not the removed /tracks) -- see February 2026 Spotify API changes.
+    next_url = f"{API}/playlists/{pid}/items?limit=100"
    while next_url:
        req = urllib.request.Request(
            next_url, headers={"Authorization": f"Bearer {token}"}
        )
        with urllib.request.urlopen(req, timeout=15) as r:
            data = json.loads(r.read())
-        for item in data["items"]:
-            t = item.get("track")
+        # Entries carry "track" (grandfathered apps) or "item" (apps on the
+        # February 2026 renamed shape); items is absent entirely when the
+        # connected account can't read the playlist's contents.
+        for item in data.get("items", []):
+            t = item.get("track") or item.get("item")
            if not t or t.get("is_local"):
                continue
            tracks.append(
@@ -222,6 +228,7 @@ def main() -> int:
    url, target = sys.argv[1], sys.argv[2]
    cid = os.environ.get("SPOTIFY_CLIENT_ID")
    csec = os.environ.get("SPOTIFY_CLIENT_SECRET")
+    refresh = os.environ.get("SPOTIFY_REFRESH_TOKEN") or None
    if not cid or not csec:
        sys.exit("SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET must be set")

@@ -231,7 +238,7 @@ def main() -> int:
        files_iter = Path(target).rglob("*")

    print(f"[spotify-retag] Fetching playlist {url}")
-    token = get_token(cid, csec)
+    token = get_token(cid, csec, refresh)
    tracks = fetch_playlist(url, token)
    print(f"[spotify-retag] Got {len(tracks)} tracks from Spotify")

@@ -51,10 +51,17 @@ def extract_flac_picture(flac_path):
    with tempfile.NamedTemporaryFile(delete=False, suffix=".pic") as tf:
        tmp = tf.name
    try:
-        r = subprocess.run(
-            ["metaflac", f"--export-picture-to={tmp}", flac_path],
-            capture_output=True, timeout=15
-        )
+        try:
+            r = subprocess.run(
+                ["metaflac", f"--export-picture-to={tmp}", flac_path],
+                capture_output=True, timeout=60
+            )
+        except subprocess.TimeoutExpired:
+            # A transient IO stall on one file must not abort the whole weekly
+            # run (seen 2026-07-15: a healthy 190KB cover took >15s under disk
+            # contention and the raised TimeoutExpired failed the entire job).
+            print(f"[strip-art] WARN: metaflac timed out on {flac_path}, skipping file")
+            return None
        if r.returncode != 0 or not os.path.exists(tmp):
            return None
        sz = os.path.getsize(tmp)
@@ -43,12 +43,58 @@ def test_render_playlist_confs_injects_creds_safely(db, config_dir):
    text = conf.read_text()
    assert "user = seek" in text
    assert "pass = p'a$(id)ss" in text
-    assert "spotify-id = CID" in text
-    assert "spotify-secret = CSECRET" in text
+    # run-playlist.sh scrapes the playlist URL from this line to build the CSV
+    assert "input = https://open.spotify.com/playlist/Z" in text
+    # Spotify creds must NOT be rendered into confs: sldl never talks to
+    # Spotify (run-playlist.sh feeds it a CSV); the creds live in _spotify.env
+    assert "spotify-id" not in text
+    assert "spotify-secret" not in text
+    assert "spotify-refresh" not in text
+    assert "CID" not in text
+    assert "CSECRET" not in text
    # rendered config must be owner-only (holds the Soulseek password)
    assert stat.S_IMODE(os.stat(conf).st_mode) == 0o600


+def test_spotify_env_renders_creds_and_refresh_token(db, config_dir):
+    # _spotify.env is the ONLY rendered home of Spotify creds; run-playlist.sh
+    # sources it to fetch the playlist CSV, so values must be shell-quoted
+    cs.set_credentials(
+        db, "spotify", {"client_id": "CID", "client_secret": "CS'EC$(id)RET", "refresh_token": "RTOK"}
+    )
+
+    env = config_dir / "pipeline" / "_spotify.env"
+    text = env.read_text()
+    assert "SPOTIFY_CLIENT_ID=CID" in text
+    # shlex.quote keeps a metacharacter-laden secret a single literal value
+    assert "SPOTIFY_CLIENT_SECRET='CS'\"'\"'EC$(id)RET'" in text
+    assert "SPOTIFY_REFRESH_TOKEN=RTOK" in text
+    assert stat.S_IMODE(os.stat(env).st_mode) == 0o600
+
+
+def test_spotify_env_refresh_token_blank_until_connected(db, config_dir):
+    cs.set_credentials(db, "spotify", {"client_id": "CID", "client_secret": "CSECRET"})
+
+    text = (config_dir / "pipeline" / "_spotify.env").read_text()
+    # rendered blank (''), which sources cleanly and stays falsy in bash
+    assert "SPOTIFY_REFRESH_TOKEN=''" in text
+
+
+def test_azuracast_renders_and_clears_base_url(db, config_dir):
+    cs.set_credentials(db, "azuracast", {"base_url": "https://radio.example.com", "api_key": "KEY"})
+
+    az_dir = config_dir / "pipeline" / "azuracast"
+    assert (az_dir / "base_url").read_text() == "https://radio.example.com"
+    assert (az_dir / "api_key").read_text() == "KEY"
+
+    cs.set_scope_enabled(db, "azuracast", False)
+    assert not (az_dir / "base_url").exists()
+    assert not (az_dir / "api_key").exists()
+
+    cs.set_scope_enabled(db, "azuracast", True)
+    assert (az_dir / "base_url").read_text() == "https://radio.example.com"
+
+
 def test_bandcamp_cookie_expiry_parsing():
    jar = "\n".join([
        "# Netscape HTTP Cookie File",
@@ -25,11 +25,41 @@ def test_run_job_failure():
    assert run.exit_code == 1


-def test_lock_skip_when_held():
+def test_manual_run_skips_immediately_when_lock_held():
    async def scenario():
        await pr._lock.acquire()
        try:
-            return await pr.run_job("test:locked", ["true"], use_lock=True)
+            return await pr.run_job("test:locked", ["true"], use_lock=True, triggered_by="manual")
+        finally:
+            pr._lock.release()
+
+    run = _run(scenario())
+    assert run.status == "skipped_lock"
+
+
+def test_scheduled_run_waits_for_lock_by_default(monkeypatch):
+    # scheduled runs must QUEUE behind a held lock (bounded), not skip --
+    # instant-skip starved the Sunday maintenance block (see
+    # SCHEDULED_LOCK_WAIT_SECONDS)
+    monkeypatch.setattr(pr, "SCHEDULED_LOCK_WAIT_SECONDS", 5.0)
+
+    async def scenario():
+        await pr._lock.acquire()
+        task = asyncio.ensure_future(pr.run_job("test:queued", ["true"], triggered_by="schedule"))
+        await asyncio.sleep(0.2)  # let the job start waiting on the lock
+        pr._lock.release()
+        return await task
+
+    run = _run(scenario())
+    assert run.status == "success"
+    assert not pr._lock.locked()
+
+
+def test_scheduled_run_skips_after_lock_wait_timeout():
+    async def scenario():
+        await pr._lock.acquire()
+        try:
+            return await pr.run_job("test:waited_out", ["true"], triggered_by="schedule", lock_wait=0.2)
        finally:
            pr._lock.release()

@@ -0,0 +1,71 @@
+"""get_playlist_tracks must parse BOTH /items response shapes: entries keyed
+"track" (grandfathered / extended-quota apps) and "item" (apps on Spotify's
+February 2026 renamed shape), and must fail loudly, not return [], when
+Spotify withholds a playlist's contents (no items field at all)."""
+import pytest
+
+from app.services import spotify_client
+
+URL = "https://open.spotify.com/playlist/XYZ123"
+
+
+class _Resp:
+    def __init__(self, data):
+        self._data = data
+
+    def raise_for_status(self):
+        pass
+
+    def json(self):
+        return self._data
+
+
+def _fake_api(monkeypatch, pages):
+    it = iter(pages)
+    monkeypatch.setattr(spotify_client, "_get_token", lambda db: "tok")
+    monkeypatch.setattr(
+        spotify_client.httpx,
+        "get",
+        lambda url, params=None, headers=None, timeout=None: _Resp(next(it)),
+    )
+
+
+def _track(name, artist, isrc=None):
+    return {
+        "name": name,
+        "artists": [{"name": artist}],
+        "external_ids": {"isrc": isrc} if isrc else {},
+    }
+
+
+def test_parses_legacy_track_shape(monkeypatch):
+    _fake_api(monkeypatch, [{"items": [{"track": _track("Sandstorm", "Darude", "FIDAR9900011")}], "next": None}])
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert tracks == [{"artist": "Darude", "title": "Sandstorm", "isrc": "FIDAR9900011"}]
+
+
+def test_parses_renamed_item_shape(monkeypatch):
+    # February 2026 shape: payload under "item", not "track"
+    _fake_api(monkeypatch, [{"items": [{"item": _track("Sandstorm", "Darude")}], "next": None}])
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert tracks == [{"artist": "Darude", "title": "Sandstorm", "isrc": None}]
+
+
+def test_missing_items_field_raises_not_empty(monkeypatch):
+    # metadata-only response (playlist not owned by the connected account on a
+    # new app) must be an error the UI can show, never a silent empty list
+    _fake_api(monkeypatch, [{"name": "DJ-USB", "public": True}])
+    with pytest.raises(RuntimeError, match="without its contents"):
+        spotify_client.get_playlist_tracks(None, URL)
+
+
+def test_skips_null_entries_and_paginates(monkeypatch):
+    _fake_api(
+        monkeypatch,
+        [
+            {"items": [{"track": None}, {"track": _track("A", "X")}], "next": "page2"},
+            {"items": [{"item": _track("B", "Y")}], "next": None},
+        ],
+    )
+    tracks = spotify_client.get_playlist_tracks(None, URL)
+    assert [t["title"] for t in tracks] == ["A", "B"]
Author	SHA1	Message	Date
andrew	62251d764e	0.6.9: Finish the Stage 4 path revisit in the three remaining scripts The 2026-07-08 path rewrite changed beets' displayed paths from the transitional /music/... mount to real /data/music/Library/... paths. dedup-library.sh was fixed in 0.6.8; three more scripts still assumed the old prefix, each failing silently: - replace-with-better.sh resolved every library match to a doubled nonexistent path, logged [stale-db], skipped the in-place replace, and then imported the staged FLAC as a NEW track via the leftover-import step. Net effect: the mp3-to-flac upgrade flow produced flac+mp3 twin pairs in the library (surfacing in the dedup queue) instead of replacing the MP3 in place. - fix-track-metadata.py queried beets only by the legacy /music/... form, matched nothing, and silently skipped beet update/move after retagging. - export-laptop-playlists.py filtered out every library track (no displayed path starts with /music/ anymore), producing empty exports. All three now use the real displayed path and keep the /music/... form only as a legacy fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-16 11:51:55 -06:00
andrew	d756aab7fc	0.6.8: Pin sldl skip index + fix dedup scan blinded by the path rewrite Two bugs conspired to fill the library with .1.flac duplicates on the night of 2026-07-15/16: 1. sldl derives its skip-index folder from the input: the Spotify playlist display name before 0.6.2, the CSV filename after. The switch stranded every playlist's download history in the old emoji-named subfolder, so sldl started from an empty index and re-downloaded every playlist in full. Fix: pin index-path in _template.conf, seed the pinned file from all stranded indexes (merge-sldl-indexes.py, called from run-playlist.sh whenever the pinned index is missing or older than a stranded one). Recovered rows keep the CSV-era identity fields but are marked downloaded, so tracks that failed last night yet exist in the library since months ago are not fetched again. 2. The safety net that should have flagged the flood, dedup-library.sh, has silently reported 0 groups since the 2026-07-08 path rewrite: host_path() still assumed /music/... display paths and prepended the library dir to already-correct /data/music/Library/... paths, so every candidate failed the -f check. Fix: pass real paths through unchanged, keep the /music prefix swap only for legacy entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-16 09:23:53 -06:00
andrew	24738d815f	0.6.7: Drop slskd from the digest health check slskd is not part of the pipeline -- sldl is its own Soulseek client and nothing in alembic calls slskd's API; the digest's reachability ping was its only mention. It keeps running on the host purely as the user's own file-sharing presence, so checking it here just risked a permanent bogus warning line for something alembic doesn't depend on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 13:50:20 -06:00
andrew	900bbff8aa	0.6.6: Digest format final: plain reflowing rows instead of <pre> columns Iterating on the 0.6.5 HTML digest on a real phone: <pre> column layout forces fixed-width lines that wrap into unreadable fragments ("54 tracks in / M3U"), and one <pre> per section rendered as a stack of copy-button widgets. Every row is now plain proportional text -- colored dot, bold label, "-- value" -- which reflows cleanly at any screen width; breakdowns (added-today per playlist, library by format) become bullet lists. Header banner (brand line, dot tally, all-clear/N-issues blockquote) unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 13:44:26 -06:00
andrew	be33664be2	0.6.5: HTML Telegram digest + fix silently-empty beet sections + watermark-art resilience The daily digest is now formatted with Telegram HTML: bold section headers with <pre>-aligned columns, colored circle emoji as status dots, a branded header line, and a top blockquote callout (all clear vs N issues). All free text is &/</> escaped so a stray angle bracket in a log snippet can't break the parse and swallow the whole message. notify-telegram.sh gains an --html flag (parse_mode=HTML) used by the digest send; plain-text callers are unchanged. Truncation is now tag-safe in HTML mode: cut on a line boundary, re-close an open <pre>, and use an escaped marker -- the old literal "...<truncated>" tail would itself have 400'd the send. Two real bugs surfaced while testing: - pipeline-status.sh pins its own PATH, which lacked /opt/venv/bin where beet lives, so every beet probe ("added today", "Library by format", the mp3-now count) has been silently empty behind 2>/dev/null since the cron migration. PATH now includes the venv; both sections show real numbers. - strip-watermark-art.py aborted its entire weekly run when metaflac stalled on ONE file (seen today: a healthy 190KB cover took >15s under disk contention, TimeoutExpired killed the job). Per-file timeout is now 60s and a timeout skips that file with a warning instead of failing the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 13:30:58 -06:00
andrew	7a49e2d507	0.6.4: Truthful maintenance digest + scheduled jobs queue for the lock instead of skipping The Telegram digest reported healthy weekly jobs as "never run yet": its maintenance section still globbed for cron-era per-script log filenames (clean-index-*.log etc.) that jobs run through pipeline_runner no longer write. It now reads job_runs, the scheduler's own record, so it reports the last success (age + summary snippet from that run's log), flags a newer failed or lock-skipped attempt on the same line, and distinguishes "never succeeded (last attempt: skipped, lock busy)" from genuinely never scheduled. Also matched the clear-bad-genres snippet grep to the script's current summary wording, and dropped the now-unused newest_log helper. The DB also showed WHY two jobs had never run: on Sunday the 08:30 strip-mb-tags run took 10m19s while holding the shared pipeline lock, so strip-watermark-art (08:35) and scrub-watermark-text (08:40) hit flock-style instant skip and lost their only slot of the week -- the 08:40 job missed by 19 seconds. Scheduled runs now wait up to 30 minutes for the lock (SCHEDULED_LOCK_WAIT_SECONDS) and only then record skipped_lock, so fixed-time blocks queue instead of starving; manual "Run now" keeps the instant skip since a person expects an immediate answer. Tests: scheduled-run queueing, lock-wait timeout, and manual instant-skip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 10:13:02 -06:00
andrew	8ecff44811	0.6.3: Parse Spotify's renamed /items response shape (new apps get item, not track) Spotify's February 2026 changes did not just move /playlists/{id}/tracks to /items: for apps on the new behavior the per-entry payload key was renamed from "track" to "item" (tracks.tracks.track -> items.items.item). Extended Quota Mode (grandfathered) apps keep the old key, which is why this never reproduced locally. Every parser in alembic read only "track", so on a new app each entry looked like a null/local track and was silently skipped: the CSV came out header-only, sldl no-opped with exit 0, and the playlist page showed an empty tracklist. Confirmed live on a new app against a playlist the connected account owns. All /items consumers (spotify_client.py, spotify-playlist-csv.py, spotify-retag.py) now parse both key names, and the fields query filter is gone: it selects by key name, so filtering on track(...) is itself what returned empty pages on the renamed shape. Also per the migration guide, new apps only receive playlist contents for playlists the connected account owns or collaborates on; other playlists return metadata with no items field at all (public is no longer enough). That case now raises a pointed error (UI and CSV fetch) instead of reading as an empty playlist, run-playlist.sh logs the fetched track count and warns loudly when it is zero, and the README guidance is updated to match. New tests pin get_playlist_tracks against both response shapes, the metadata-only error, and pagination. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 09:50:57 -06:00
andrew	2e3210cc01	README: point install and compose examples at 0.6.2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 08:54:17 -06:00
andrew	3fbf580b88	0.6.2: Bypass sldl's Spotify client -- fetch the playlist ourselves, feed sldl a CSV sldl's vendored Spotify client still calls GET /playlists/{id}/tracks, which Spotify removed in its February 2026 API changes. Grandfathered apps still get a pass; apps created after the change get a hard 403 there no matter how they authenticate (client-credentials or a correctly-scoped OAuth user token), so sldl can never load a playlist for a new app regardless of what we hand it. Instead of depending on sldl's Spotify client at all, run-playlist.sh now reads the playlist itself via the still-working /items endpoint (new spotify-playlist-csv.py, creds sourced from _spotify.env) and invokes sldl with the CSV + --input-type csv, which override the conf's input lines while -c still supplies Soulseek login, paths, and quality settings (verified live). The same CSV format upgrade-mp3-to-flac.sh already feeds sldl. All artists are comma-joined in the CSV so multi-artist tracks search no worse than before, and a 403/404 on the fetch prints the account-visibility hint instead of a bare traceback. Since sldl no longer talks to Spotify, playlist .confs no longer carry spotify-id/spotify-secret/spotify-refresh: _template.conf drops them, render_playlist_confs stops injecting them, and a Spotify credential save no longer re-renders confs (only _spotify.env). The retag step in run-playlist.sh reuses the sourced _spotify.env creds instead of scraping the conf lines that no longer exist. Confs are also re-rendered once at app startup so already-deployed confs converge on upgrade (and shed the stale secret lines) without waiting for a playlist or credential change. Playlist delete now also removes the generated .csv. Tests updated: conf rendering must NOT contain Spotify creds but must keep the input URL line; new coverage for _spotify.env rendering (quoting, refresh-token presence/blank, 0600). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-15 08:48:30 -06:00
andrew	9976c430aa	0.6.1: Fix Spotify OAuth scope missing user-library-read connect.py's OAuth scope request (playlist-read-private playlist-read-collaborative) didn't match what sldl's own built-in login flow requests (same two, plus user-library-read). The narrower scope alembic granted let sldl refresh a valid access token, but Spotify then returned 403 Forbidden loading the playlist regardless of ownership, which sldl turns into an unhandled-exception crash (exit 134) instead of a clean scope error. Found by reproducing a friend's crash: his playlist was his own, connected via his own OAuth, correct redirect URI -- ruling out the ownership/ visibility explanation and pointing at the scope mismatch instead. Also: - run-playlist.sh: detect exit 134 specifically and log a pointed hint (distinguishing the Forbidden-loading-playlist case from any other unhandled sldl exception) instead of just "sldl finished with exit code 134" with no context. - README: document that anyone who connected Spotify before this version needs to click Connect Spotify again -- existing refresh tokens carry the old, narrower scope and don't upgrade themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-15 08:07:30 -06:00
andrew	29d6249e88	README: make the Spotify OAuth requirement explicit, not a footnote Previously described the Connect Spotify step in soft "one click, optional either way" language and never used the word OAuth at all, even though it's now a required step for any Spotify app created after Spotify's 2025/2026 API change. Rewrote "What you need before you start", the credentials table, and the connect section to say plainly that keys alone no longer work and walk through the OAuth login as a required numbered step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-14 11:25:05 -06:00
andrew	e5d9c7f043	0.6: Spotify OAuth connect flow, AzuraCast base URL, fingerprint/buy-url fixes - Add "Connect Spotify" OAuth flow (/connect/spotify) so newly created Spotify apps can read playlists: Spotify now requires a user token for playlist reads, which client-credentials alone can no longer provide. The stored refresh token is rendered as spotify-refresh into each playlist's sldl .conf -- sldl's own docs confirm supplying it skips its interactive login flow, which is what was causing multi-hour hangs on a stuck sync. Grandfathered apps keep working unchanged if no account is connected. - Fix the connect flow's redirect_uri: request.url_for() reflected the raw connection scheme (http) rather than what the reverse proxy actually served over, which Spotify's exact-match redirect_uri check rejected. Force https rather than trust the unproxied scheme. - Add an AzuraCast base URL credential field alongside the API key, with a keyfile fallback so the scheduled enrich-buy-url.py job can actually reach AzuraCast (previously it had no way to receive a base URL at all when run from the scheduler, so the reprocess call was silently always skipped). - Fix build-fingerprint-index.py exiting non-zero on any single fingerprint failure instead of only when nothing was written. - Guard enrich-buy-url.py's AzuraCast reprocess against an empty --azuracast-base (raised ValueError since PD2 removed the hardcoded base). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-14 11:16:55 -06:00
andrew	91f3982eff	README: state plainly that a new Spotify app can't read playlists (both endpoints blocked) Testing a brand-new app confirmed /tracks returns 403 and /items returns 401 'valid user authentication required' for app-only (client-credentials) access. Update the prerequisite, the credentials table, and Troubleshooting so new self-hosters know they need a grandfathered app's Client ID/Secret rather than creating their own. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-14 10:19:05 -06:00
andrew	97672ffdcd	0.5.2: Run now for unscheduled playlists, Spotify /items migration, friendlier 403 - Run now (jobs.run_now): playlists without a daily sync time are never registered as scheduler jobs, so the old registered-job check 404'd. Fall back to running the playlist directly via pipeline_runner for any playlist:<name> key that isn't registered. - Spotify removed GET /playlists/{id}/tracks in its February 2026 API changes in favor of /items (identical response shape). New apps are already 403'd on the old endpoint. Migrate both callers (app/services/spotify_client.py and pipeline/lib/spotify-retag.py) to /items. (The vendored sldl downloader uses its own bundled Spotify library and would need an upstream update when /tracks is fully removed.) - Friendlier on-screen error for a Spotify 403 (check credentials + playlist must be public) and a README troubleshooting entry covering the 403, the public-playlist requirement, and the org-only extended-quota reality. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-14 10:12:42 -06:00
andrew	9a879b5322	README: bump pull/compose examples to 0.5.1 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-11 14:58:57 -06:00