tag-guard.sh checked MP3 tags with the id3v2 CLI (id3lib-3.8.3, last released
in 2013), which cannot parse ID3v2.4 tags at all. It reported "No ID3 tag" on
files that were completely and correctly tagged, just written in the modern
tag version most taggers default to now (Mp3tag, yt-dlp, foobar2000). Every
manual import of ID3v2.4 MP3s got its files quarantined as untagged and the
whole import failed once nothing survived the quarantine pass.
Replaced the id3v2-based reads in tag-guard.sh with mutagen, which handles
both tag versions correctly. Also replaced the ALBUMARTIST-fallback write
(previously done via `id3v2 --TPE2`) with a mutagen-based write, because
testing turned up something worse: writing a single frame with the id3v2 CLI
to a file that already has an ID3v2.4 tag silently destroys every other
frame on the file (title, album, genre, date, artwork, all of it), since
id3lib doesn't understand v2.4 and rewrites the whole tag as v2.3 containing
only the frame it was told to set. The mutagen write sets just the one frame
and leaves everything else untouched.
Other id3v2 CLI writes still exist elsewhere in the pipeline (playlist
grouping tags, genre fixes, buy-url tagging) and carry the same corruption
risk against ID3v2.4 files; not touched in this release.
sync-bandcamp.sh set a hardcoded PATH at startup that left out /opt/venv/bin,
which is where beet lives in this image. Every time the Bandcamp sync had a
new purchase to import, its call into import-track.sh would run beet import
with that broken PATH, fail with "command not found", and import-track.sh
would just log the exit code and carry on, so sync-bandcamp.sh still reported
success. The purchase sat on disk, never entered the beets library, and never
showed up in Navidrome. This went unnoticed for weeks because most daily
syncs have nothing new to import, so the broken code path rarely ran.
Fixed the same copy-pasted PATH line in upgrade-mp3-to-flac.sh (which also
calls beet directly) and notify-telegram.sh (harmless there, but fixed for
consistency).
Also moved the post-import duplicate cleanup (replace-with-better.sh and
dedup-library.sh) out of sync-bandcamp.sh and into import-track.sh itself, so
every import gets the same cleanup, not just Bandcamp purchases. A manual
import or SMB drop that happens to match something already in the library no
longer leaves a duplicate copy sitting there until someone runs the dedup
review by hand.
scrub-watermark-text.py crashed on every run (missing import os). strip-watermark-art.py
was starved every Sunday by lock contention from strip-mb-tags' growing runtime -- moved
it to 01:00 so the rest of the weekly chain has room. enrich-buy-url.py's summary line
counted matches found, not successful writes, so a run where every metaflac write failed
(root-owned files) still reported "N tagged". pipeline-status.sh now tells a 403 from
Qobuz (Akamai/CDN block) apart from a 401 (actually expired token) -- re-exporting the
token does nothing for the former. Also fixes the dedup review table's "Caught by" badge
overflowing into the Keep column's format pill.
The keep/delete columns were raw paths in a fixed-width cell: ellipsis-
truncated, full text only on hover. Slow to scan (mousing over every
row to read the song name) and, once "fixed" with a tag-derived title/
artist header in a first pass of this change, actively worse for the
thing dedup review actually needs -- confirming two files are really
the same recording. Tags can be wrong; the filename on disk can't.
New layout: each candidate gets a title row (song title, large,
unclipped -- parsed from beets' known singleton path template,
Artist/Album/Title.ext) followed by a compact detail row. The detail
row shows a color-coded format pill (FLAC/MP3/etc, matching the same
quality judgment dedup-library.sh's rank_file() already makes) and the
full relative file path, always rendered as visible text -- never
hover-only -- so a real difference in filename, album, or folder is
still plainly visible even when tags line up. Shared artist collapses
onto the title row so it isn't repeated per side; a colored left rail
marks which side survives without having to read the words.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps left obvious cases stuck in the manual dedup queue:
1. Format-upgrade pairs found by Pass 2/3 (case-insensitive/normalized
tag match) still required manual confirm even though the pass
itself already proved same song via identical tags, and rank_file()
already guarantees FLAC beats any other format regardless of
size/dirty-name. A directory-casing difference (e.g. "All About
This Ep" vs "...EP") meant these never matched the narrower
same-directory numbered-twin rule from 0.6.10. New is_format_upgrade()
auto-deletes any Pass 2/3 pair whose extensions differ, gated on
pass_name so Pass 4 (cross-album fuzzy -- different masters, DJ-mix
edits, genuinely ambiguous) is untouched and still requires a human.
2. A pending row can go stale without ever being confirmed -- some
other action (a later auto-apply, upgrade-mp3-to-flac, a manual fix)
already resolved one side of the pair -- and nothing pruned it, so
an already-fixed duplicate kept surfacing in the queue indefinitely.
scan() now sweeps and auto-resolves any pending candidate whose
keep_path or delete_path no longer exists before persisting new
ones, so the queue reflects current reality on every run instead of
accumulating dead entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every dedup pass previously funneled through the same manual-confirm
queue, including the obviously-safe case: two files in the same
directory with the same name and extension, differing only by the
".N" collision suffix sldl/beets inserts when a filename collides.
That's not a fuzzy match or a different version -- it's the same
download landing twice -- so it doesn't need a human in the loop the
way case-insensitive tag matches or cross-album fuzzy matches do
(different masters, DJ-mix versions, etc., which still require
confirmation).
dedup-library.sh gains --auto-apply-numbered: an is_numbered_twin()
filename check (same dir, same name modulo the numeric infix, same
extension) that deletes matching pairs unconditionally, regardless of
which pass found them -- Pass 1 only fires when the canonical name is
a beets ghost; the common case where both copies are already tracked
in beets defers to Pass 2's tag-based grouping instead, and still gets
the same free pass here. When the auto-deleted pair's survivor is
itself the numbered-named file, it's renamed back to canonical so the
library doesn't accumulate ".1."/".2." names for tracks that no longer
have a duplicate.
dedup_review_service.scan() (both the manual "Scan now" button and the
nightly scheduled job) now passes this flag and records auto-applied
deletions as pre-confirmed DedupCandidate rows for audit visibility --
they never appear as pending review items. Tag-based and cross-album
fuzzy passes are unaffected and still require manual confirmation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-08 path rewrite changed beets' displayed paths from the
transitional /music/... mount to real /data/music/Library/... paths.
dedup-library.sh was fixed in 0.6.8; three more scripts still assumed
the old prefix, each failing silently:
- replace-with-better.sh resolved every library match to a doubled
nonexistent path, logged [stale-db], skipped the in-place replace,
and then imported the staged FLAC as a NEW track via the
leftover-import step. Net effect: the mp3-to-flac upgrade flow
produced flac+mp3 twin pairs in the library (surfacing in the dedup
queue) instead of replacing the MP3 in place.
- fix-track-metadata.py queried beets only by the legacy /music/...
form, matched nothing, and silently skipped beet update/move after
retagging.
- export-laptop-playlists.py filtered out every library track (no
displayed path starts with /music/ anymore), producing empty
exports.
All three now use the real displayed path and keep the /music/... form
only as a legacy fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs conspired to fill the library with .1.flac duplicates on the
night of 2026-07-15/16:
1. sldl derives its skip-index folder from the input: the Spotify
playlist display name before 0.6.2, the CSV filename after. The
switch stranded every playlist's download history in the old
emoji-named subfolder, so sldl started from an empty index and
re-downloaded every playlist in full. Fix: pin index-path in
_template.conf, seed the pinned file from all stranded indexes
(merge-sldl-indexes.py, called from run-playlist.sh whenever the
pinned index is missing or older than a stranded one). Recovered
rows keep the CSV-era identity fields but are marked downloaded, so
tracks that failed last night yet exist in the library since months
ago are not fetched again.
2. The safety net that should have flagged the flood, dedup-library.sh,
has silently reported 0 groups since the 2026-07-08 path rewrite:
host_path() still assumed /music/... display paths and prepended the
library dir to already-correct /data/music/Library/... paths, so
every candidate failed the -f check. Fix: pass real paths through
unchanged, keep the /music prefix swap only for legacy entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
slskd is not part of the pipeline -- sldl is its own Soulseek client and
nothing in alembic calls slskd's API; the digest's reachability ping was its
only mention. It keeps running on the host purely as the user's own
file-sharing presence, so checking it here just risked a permanent bogus
warning line for something alembic doesn't depend on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Iterating on the 0.6.5 HTML digest on a real phone: <pre> column layout
forces fixed-width lines that wrap into unreadable fragments ("54 tracks
in / M3U"), and one <pre> per section rendered as a stack of copy-button
widgets. Every row is now plain proportional text -- colored dot, bold
label, "-- value" -- which reflows cleanly at any screen width; breakdowns
(added-today per playlist, library by format) become bullet lists. Header
banner (brand line, dot tally, all-clear/N-issues blockquote) unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The daily digest is now formatted with Telegram HTML: bold section headers
with <pre>-aligned columns, colored circle emoji as status dots, a branded
header line, and a top blockquote callout (all clear vs N issues). All free
text is &/</> escaped so a stray angle bracket in a log snippet can't break
the parse and swallow the whole message. notify-telegram.sh gains an --html
flag (parse_mode=HTML) used by the digest send; plain-text callers are
unchanged. Truncation is now tag-safe in HTML mode: cut on a line boundary,
re-close an open <pre>, and use an escaped marker -- the old literal
"...<truncated>" tail would itself have 400'd the send.
Two real bugs surfaced while testing:
- pipeline-status.sh pins its own PATH, which lacked /opt/venv/bin where
beet lives, so every beet probe ("added today", "Library by format", the
mp3-now count) has been silently empty behind 2>/dev/null since the cron
migration. PATH now includes the venv; both sections show real numbers.
- strip-watermark-art.py aborted its entire weekly run when metaflac stalled
on ONE file (seen today: a healthy 190KB cover took >15s under disk
contention, TimeoutExpired killed the job). Per-file timeout is now 60s
and a timeout skips that file with a warning instead of failing the run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Telegram digest reported healthy weekly jobs as "never run yet": its
maintenance section still globbed for cron-era per-script log filenames
(clean-index-*.log etc.) that jobs run through pipeline_runner no longer
write. It now reads job_runs, the scheduler's own record, so it reports the
last success (age + summary snippet from that run's log), flags a newer
failed or lock-skipped attempt on the same line, and distinguishes "never
succeeded (last attempt: skipped, lock busy)" from genuinely never
scheduled. Also matched the clear-bad-genres snippet grep to the script's
current summary wording, and dropped the now-unused newest_log helper.
The DB also showed WHY two jobs had never run: on Sunday the 08:30
strip-mb-tags run took 10m19s while holding the shared pipeline lock, so
strip-watermark-art (08:35) and scrub-watermark-text (08:40) hit flock-style
instant skip and lost their only slot of the week -- the 08:40 job missed by
19 seconds. Scheduled runs now wait up to 30 minutes for the lock
(SCHEDULED_LOCK_WAIT_SECONDS) and only then record skipped_lock, so
fixed-time blocks queue instead of starving; manual "Run now" keeps the
instant skip since a person expects an immediate answer.
Tests: scheduled-run queueing, lock-wait timeout, and manual instant-skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spotify's February 2026 changes did not just move /playlists/{id}/tracks to
/items: for apps on the new behavior the per-entry payload key was renamed
from "track" to "item" (tracks.tracks.track -> items.items.item). Extended
Quota Mode (grandfathered) apps keep the old key, which is why this never
reproduced locally. Every parser in alembic read only "track", so on a new
app each entry looked like a null/local track and was silently skipped: the
CSV came out header-only, sldl no-opped with exit 0, and the playlist page
showed an empty tracklist. Confirmed live on a new app against a playlist
the connected account owns.
All /items consumers (spotify_client.py, spotify-playlist-csv.py,
spotify-retag.py) now parse both key names, and the fields query filter is
gone: it selects by key name, so filtering on track(...) is itself what
returned empty pages on the renamed shape.
Also per the migration guide, new apps only receive playlist contents for
playlists the connected account owns or collaborates on; other playlists
return metadata with no items field at all (public is no longer enough).
That case now raises a pointed error (UI and CSV fetch) instead of reading
as an empty playlist, run-playlist.sh logs the fetched track count and warns
loudly when it is zero, and the README guidance is updated to match.
New tests pin get_playlist_tracks against both response shapes, the
metadata-only error, and pagination.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sldl's vendored Spotify client still calls GET /playlists/{id}/tracks, which
Spotify removed in its February 2026 API changes. Grandfathered apps still get
a pass; apps created after the change get a hard 403 there no matter how they
authenticate (client-credentials or a correctly-scoped OAuth user token), so
sldl can never load a playlist for a new app regardless of what we hand it.
Instead of depending on sldl's Spotify client at all, run-playlist.sh now reads
the playlist itself via the still-working /items endpoint (new
spotify-playlist-csv.py, creds sourced from _spotify.env) and invokes sldl with
the CSV + --input-type csv, which override the conf's input lines while -c still
supplies Soulseek login, paths, and quality settings (verified live). The same
CSV format upgrade-mp3-to-flac.sh already feeds sldl. All artists are
comma-joined in the CSV so multi-artist tracks search no worse than before, and
a 403/404 on the fetch prints the account-visibility hint instead of a bare
traceback.
Since sldl no longer talks to Spotify, playlist .confs no longer carry
spotify-id/spotify-secret/spotify-refresh: _template.conf drops them,
render_playlist_confs stops injecting them, and a Spotify credential save no
longer re-renders confs (only _spotify.env). The retag step in run-playlist.sh
reuses the sourced _spotify.env creds instead of scraping the conf lines that
no longer exist. Confs are also re-rendered once at app startup so
already-deployed confs converge on upgrade (and shed the stale secret lines)
without waiting for a playlist or credential change. Playlist delete now also
removes the generated .csv.
Tests updated: conf rendering must NOT contain Spotify creds but must keep the
input URL line; new coverage for _spotify.env rendering (quoting, refresh-token
presence/blank, 0600).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
connect.py's OAuth scope request (playlist-read-private
playlist-read-collaborative) didn't match what sldl's own built-in login
flow requests (same two, plus user-library-read). The narrower scope alembic
granted let sldl refresh a valid access token, but Spotify then returned 403
Forbidden loading the playlist regardless of ownership, which sldl turns
into an unhandled-exception crash (exit 134) instead of a clean scope error.
Found by reproducing a friend's crash: his playlist was his own, connected
via his own OAuth, correct redirect URI -- ruling out the ownership/
visibility explanation and pointing at the scope mismatch instead.
Also:
- run-playlist.sh: detect exit 134 specifically and log a pointed hint
(distinguishing the Forbidden-loading-playlist case from any other
unhandled sldl exception) instead of just "sldl finished with exit code
134" with no context.
- README: document that anyone who connected Spotify before this version
needs to click Connect Spotify again -- existing refresh tokens carry the
old, narrower scope and don't upgrade themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously described the Connect Spotify step in soft "one click, optional
either way" language and never used the word OAuth at all, even though it's
now a required step for any Spotify app created after Spotify's 2025/2026
API change. Rewrote "What you need before you start", the credentials
table, and the connect section to say plainly that keys alone no longer
work and walk through the OAuth login as a required numbered step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Add "Connect Spotify" OAuth flow (/connect/spotify) so newly created Spotify
apps can read playlists: Spotify now requires a user token for playlist
reads, which client-credentials alone can no longer provide. The stored
refresh token is rendered as spotify-refresh into each playlist's sldl
.conf -- sldl's own docs confirm supplying it skips its interactive login
flow, which is what was causing multi-hour hangs on a stuck sync.
Grandfathered apps keep working unchanged if no account is connected.
- Fix the connect flow's redirect_uri: request.url_for() reflected the raw
connection scheme (http) rather than what the reverse proxy actually
served over, which Spotify's exact-match redirect_uri check rejected.
Force https rather than trust the unproxied scheme.
- Add an AzuraCast base URL credential field alongside the API key, with a
keyfile fallback so the scheduled enrich-buy-url.py job can actually reach
AzuraCast (previously it had no way to receive a base URL at all when run
from the scheduler, so the reprocess call was silently always skipped).
- Fix build-fingerprint-index.py exiting non-zero on any single fingerprint
failure instead of only when nothing was written.
- Guard enrich-buy-url.py's AzuraCast reprocess against an empty
--azuracast-base (raised ValueError since PD2 removed the hardcoded base).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Testing a brand-new app confirmed /tracks returns 403 and /items returns 401
'valid user authentication required' for app-only (client-credentials) access.
Update the prerequisite, the credentials table, and Troubleshooting so new
self-hosters know they need a grandfathered app's Client ID/Secret rather than
creating their own.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Run now (jobs.run_now): playlists without a daily sync time are never
registered as scheduler jobs, so the old registered-job check 404'd. Fall
back to running the playlist directly via pipeline_runner for any
playlist:<name> key that isn't registered.
- Spotify removed GET /playlists/{id}/tracks in its February 2026 API changes
in favor of /items (identical response shape). New apps are already 403'd on
the old endpoint. Migrate both callers (app/services/spotify_client.py and
pipeline/lib/spotify-retag.py) to /items. (The vendored sldl downloader uses
its own bundled Spotify library and would need an upstream update when
/tracks is fully removed.)
- Friendlier on-screen error for a Spotify 403 (check credentials + playlist
must be public) and a README troubleshooting entry covering the 403, the
public-playlist requirement, and the org-only extended-quota reality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. enrich-buy-url.py re-queried every no-match track on every daily run (external,
rate-limited lookups over the whole backlog), which blew past its 30-min
timeout. Cache "tried, no match" with a BUY_URL_TRIED marker tag and skip
those for 30 days (--force still re-checks). This is what made "Look up buy
links" fail with TIMEOUT after 1800s.
2. Move maintenance:enrich_buy_url to run last (09:45) in the 9am block, after
the fingerprint index (09:25) and status report (09:30). enrich starting at
09:10 and holding the shared lock is what left "Rebuild fingerprint index"
skipped_lock.
3. pipeline-status.sh: guard the qobuz/app_id read with a file-exists check.
`< missing 2>/dev/null` still leaks the shell's redirection error (opened
before 2>/dev/null applies), so the daily report logged
"qobuz/app_id: No such file or directory" every run.
4. dedup confirm_and_apply: a candidate whose delete target no longer exists
(already removed by an earlier dedup/upgrade/hand) was filtered out and the
apply silently no-op'd, leaving it stuck in the queue with no way to delete
or clear it (the 100 gecs case). Now mark such candidates applied so a Delete
click clears them. Covered by tests/test_dedup_review.py.
Tests: 46 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Point the install at the published image (git.kretzer.club/andrew/alembic:0.5)
now that v0.5 is on the registry: docker pull instead of clone+build, the
compose example uses the registry image, and Updating is docker compose
pull/up instead of git pull + build. Setup, credentials, and day-to-day
sections are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run-playlist.sh and import-track.sh had the same beets->#EXTM3U block, but
import-track.sh wrote the file non-atomically. Move it to pipeline/lib/m3u.sh
as regen_m3u <name> <out>, sourced by both. The helper always writes atomically
(temp + rename), so a reader like Navidrome never sees a partial file and a
stray wrong-owner leftover .m3u8 is replaced without a chown. Returns 0 always
(empty result is a warning), so it's safe as a standalone call under set -e.
Verified: tests/test_m3u.py (atomic write with tracks, no file when empty; 44
tests green) plus a live regen against the real library (techno -> 229 tracks,
no temp leftover).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two were ~90% identical (lock/skip, job_runs create, subprocess exec,
timeout/kill, finalize) and had already drifted once during the use_lock
change. Extract _execute(capture=bool) plus _record_run/_finalize_run helpers;
run_job and run_job_capture become thin wrappers. run_job_capture also gains
the use_lock parameter for free. Behavior is unchanged.
Covered by tests/test_pipeline_runner.py, now including run_job_capture output
capture + persistence and its skipped-lock case (42 tests green).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- The client-credentials token fetch was copy-pasted across spotify-retag.py,
spotify-genre.py, and fix-track-metadata.py (and the app's spotify_client).
Add pipeline/lib/_spotify_auth.get_token (cached per id/secret); the three
scripts now source their own credentials but delegate the request to it. The
scripts run with pipeline/lib on sys.path, so the plain `from _spotify_auth
import get_token` resolves.
- The identical _parse_json_lines helper in dedup_review_service and
genre_review_service is now a single app/services/_ndjson.parse_json_lines.
Verified: unit test of the token helper (cache + request), the NDJSON parser
(tests/test_ndjson.py), full suite green (40), and a live spotify-genre dry-run
that fetched a token and queried Spotify through the shared helper.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First automated coverage. 38 tests, run against the built image (which has the
app deps); tests/ and requirements-dev.txt are not baked into the runtime image.
- tests/test_dedup_gating.py: the safety-critical one. Drives dedup-library.sh's
process_group with --apply + --only-paths and asserts ONLY the confirmed path
is deleted and the higher-ranked FLAC is kept.
- tests/test_pipeline_runner.py: lock skip, use_lock=False bypass, lock release,
success/failure/timeout recording.
- tests/test_db.py: schema version + idempotent migrations, FK-safe job-run
pruning, stale-run reconciliation.
- tests/test_credential_service.py: conf-field substitution incl. newline-
injection safety, secret/non-secret classification, cookie-expiry parsing.
- tests/test_playlist_service.py: cron round-trips, name validation, render.
- tests/test_diagnostics.py: /health + /setup checks.
- conftest.py points every path setting at throwaway temp dirs and a temp DB, so
tests never touch a real library, /config, or the network. See tests/README.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Navidrome password is now passed to curl via stdin (--data-urlencode "p@-")
in navidrome-scan.sh and pipeline-status.sh, so it no longer appears in
ps/proc. Verified the query sent is identical and a live scan still triggers.
- MIN_ARTIST_DIRS (the share-health gate) is now a setting, threaded through to
the pipeline env, so a user with a small library can lower it instead of the
scan/sync being permanently blocked by the hardcoded 500.
- /auth/logout is now POST-only (with a nav form + aria-label), so a drive-by
GET can't log the user out; enforced allowed_email already landed separately.
- view_log now confirms the run's log_path resolves under the logs dir before
serving it (defense in depth).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- R11: pin requirements.txt to the versions running in the image, so a rebuild
can't pull a breaking upstream release. Documented how to upgrade.
- R12: log rotation now also prunes job_runs rows older than 90 days (nulling
the manual_imports FK reference first), so the table doesn't grow without
bound. dedup/genre candidates are review state and left alone.
- U5: credential form shows non-secret fields (URLs, usernames, prefs) as plain
text so they can be verified while typing; only real secrets stay masked.
Dashboard IP card reworded from gluetun-specific to generic "Outbound IP".
- S9: enforce ALLOWED_EMAIL at the OIDC callback, before any user row or session
is created, instead of only on later requests.
Verified: pinned build resolves; prune deletes only old rows and keeps the
manual_imports record; credentials page renders text+password inputs; a
disallowed email gets 403 with no user row, an allowed one succeeds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The app owns every input to a playlist's sldl .conf (playlists in its DB,
credentials in its encrypted store), so the old
DB -> playlists.json -> regen.sh subprocess -> regex credential patch-back
chain was three serialization hops for no reason. Collapse it:
- credential_service.render_playlist_confs(db) renders each <playlist>.conf
from _template.conf in one pass: path placeholders substituted as literal
text, then the four credential lines set, written 0600. This is now the
single source of .conf rendering.
- playlist_service loses _write_playlists_json and regenerate_confs; _sync_to_disk
just calls render_playlist_confs and syncs the scheduler. No subprocess, no
intermediate JSON file.
- render_scope for spotify/soulseek re-renders confs via the same function
(spotify still also writes _spotify.env for the python scripts). The dead
_patch_conf_field / _every_playlist_conf helpers are removed.
- Delete pipeline/configs/regen.sh and drop it from the Dockerfile chmod;
update _template.conf's comments.
Nothing outside playlist_service consumed regen.sh or playlists.json (verified).
Also closes the last remnants of the S2/S3 injection surface: names are
re-validated and path/credential values are substituted as literals, never
through sed or a shell.
Verified: end-to-end render in a throwaway DB (paths, creds incl. a password
with shell metacharacters, 0600, bad-name rejection) and a live re-render of
all 17 playlist configs with credentials preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move six scripts that are not wired into the app, scheduler, or web UI out of
the active pipeline/lib into pipeline/lib/manual, so the scheduled scripts are
easy to see and these stay available for hands-on use: convert-m4a.sh,
fix-empty-album.sh, backfill-spotify-tags.sh, backfill-buy-url.py,
recover-azuracast-playlists.py, import-dj-collection.py. Adds a README
explaining each, and extends the Dockerfile chmod to cover the new folder.
Nothing referenced these from active code (verified), so no wiring changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- fix-genre.sh: set -euo pipefail; guard the tag-remove metaflac calls and the
'beet ls | head -5' verify (SIGPIPE) that would otherwise abort.
- sync-bandcamp.sh: set -euo pipefail; the destructive chain already used
'if cmd; then..else WARN' (set -e exempt); guard the staging mv.
- upgrade-mp3-to-flac.sh: set -euo pipefail; capture sldl/replace-with-better
exit codes without aborting (was 'cmd; RC=$?', which set -e breaks); read
Soulseek creds from the first available .conf instead of a hardcoded y2k.conf,
guarded; guard the xargs purge and a grep -c.
- pipeline-status.sh: deliberately kept at set -u (documented). It is read-only
and assembles ~30 independent probes; set -e would abort the whole digest on
one missing optional file, which is exactly how the 9am notification broke.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds strict mode to four maintenance scripts, with guards so expected non-zero
exits (empty beet queries, a missing config, a curl timeout) log/skip instead
of aborting: gen-djmix bails cleanly if djmix-albums.txt is absent and tolerates
a bad single query; gen-vgm tolerates an empty query; strip-mb-tags quotes its
path arg and best-efforts the trailing beet update; notify-telegram guards curl
so a network error still reaches its explicit "send failed" branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of a missed 9am notification: the read-only pipeline_status_report
runs at 09:30, but enrich_buy_url (09:10, --apply) held the single global
pipeline lock long past 09:30, so the status report recorded skipped_lock and
never sent its Telegram digest. build_fingerprint_index (09:25) was starved the
same way.
- run_job/run_lib_script gain use_lock; the status report runs with
use_lock=False since it only reads logs, pings Navidrome, and sends Telegram.
It can no longer be blocked by a long write job, and running concurrently is
safe.
- _lib was silently dropping its timeout argument, so maintenance jobs had no
timeout and a hung one held the lock until the next container restart. Thread
timeout through _lib -> run_lib_script -> run_job, cap enrich_buy_url at 30m
and build_fingerprint_index at 60m, and give the status report a 5m cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds strict mode to the file-replacement script (the destructive path used by
the Bandcamp sync and the mp3->flac upgrade).
Uses `set -eu` deliberately WITHOUT pipefail: the script reads tags through
many `cmd | head -1` pipelines, and under pipefail `head` closing the pipe
early makes the producer die with SIGPIPE, turning good reads into false
failures. pipefail adds no safety here since every piped read feeds a value
that is immediately emptiness-checked.
Guards added so a single expected non-zero doesn't abort a destructive run:
- the four progressive `beet ls` match queries fall through on empty/error
- beet remove / beet import / rm / leftover-import / find -delete log a warning
instead of aborting mid-replace
- copy_tags_from_existing is called with `|| true` (also suspends set -e for its
best-effort per-tag metaflac calls)
- explicit `exit 0` so a successful run reports success
Verified: function-level tests (ranking, match fall-through under failing beet)
and a full dry-run against the real library, all exit 0 under set -e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn on strict mode for the file that deletes music, and neutralize the
set -e footguns it exposed:
- json_emit returned non-zero in the default (non-JSON) path; as a standalone
call that aborted the run on the first candidate. Now returns 0 explicitly.
- process_group returned 1 when a group had no files on disk, aborting the
whole run since callers don't check it; now returns 0, with an explicit
return 0 at the end so the loop's last status can't leak out.
- The cross-album keep-list pipeline returns non-zero on an all-comments file
(the default) under pipefail; guarded with || true (empty = protect nobody).
- Apply-mode beet remove/import/move and rm are now best-effort (|| log WARN)
so one failed deletion doesn't abort the rest of a confirmed apply.
- Added an explicit `exit 0`: the script previously exited non-zero in --apply
mode because its last line was a failing `[[ $APPLY -eq 0 ]]` test, making a
successful apply look like a failed run to the caller.
Verified: full dry-run over the real library exits 0 through all four passes;
process_group ranks FLAC over MP3, emits JSON, and handles empty groups without
aborting; and dedup:scan records success through pipeline_runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- init_db applies each schema file with executescript (semicolons in triggers
and string literals no longer break it) and records schema_version itself
after each file, so a forgotten in-file bump can't re-run an ALTER and crash
startup. Re-running is a no-op.
- On startup, sweep any job_runs left in 'running' (killed by a restart) to
'interrupted' so the UI/history don't show a run stuck running forever.
- prep-audio.sh skips loudgain and cue_file cleanly when they aren't installed
(they aren't in the base image) instead of failing on a missing binary and
logging errors; pipeline-status.sh routes syslog through a guard so a missing
`logger` doesn't spew into the job log.
- Add a branded 404 page (rendered to static HTML, no external runtime) via a
custom exception handler that leaves all other responses, including the login
redirect, untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Run now (jobs and playlists), dedup scan, genre preview/apply, and manual
import now dispatch via BackgroundTasks and redirect immediately, instead
of awaiting a job that can run for the better part of an hour and hang the
browser or reverse proxy. Progress shows in the Jobs runs table (which
already polls); if the pipeline is busy the run records skipped_lock there.
- Fix the import banner, which claimed work was ongoing after the request
had actually blocked to completion; it now reflects the backgrounded start.
- Add confirmation prompts to the dedup per-row and bulk delete and to
"Fix genres now", matching the existing confirms on library and playlist
deletes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security (P0):
- Remove committed session-secret default; auto-generate and persist a
random secret to the config volume when SESSION_SECRET is unset
(prevents forgeable session cookies / auth bypass).
- Validate playlist names to a safe charset and render sldl configs via
literal Python substitution instead of sed (closes a command-injection
and path-traversal path through playlist names).
- shlex-quote credential values written to shell-sourced env files, and
strip newlines from values patched into .conf files.
- Render playlist .conf files 0600; warn at startup if the master key is
co-located with the config volume; document keeping it separate.
Portability:
- Configurable timezone via TZ (default UTC) instead of hardcoded Edmonton.
- Remove personal defaults (navidrome user "andrew", ephemeral.club URLs).
- Ship generic example seeds; move the cross-album dedup keep-list and the
legacy playlist import to editable config files; drop the personal
_upgrade.csv.
- Generic VPN reference in docker-compose.snippet.yml.
First-run experience:
- Redirect to /setup instead of 500 when OIDC is unconfigured; surface a
missing master key inline; entrypoint exits with an actionable message
when the config folder is not writable.
- Add unauthenticated /health (JSON) and /setup (checklist) diagnostics.
Docs:
- Write docs/ARCHITECTURE.md and docs/MIGRATION.md (previously referenced
but missing); expand README with ownership, backups, advanced settings,
and migration guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
genre_review_service.run() and dedup_review_service._run_scan() both
unconditionally created a new run row even when the underlying job never
actually executed (skipped_lock, or any other non-success status). For
genres this was directly user-visible: the review page always shows the
most recent run, so a skipped click created an empty run that displaced
the real previous preview, making it look like every pending change had
vanished. Both now return None (persisting nothing) when the job didn't
succeed, and both routers surface a "didn't run, something else was
using the pipeline" notice instead of silently redirecting. Cleaned up
the one phantom empty genre_runs row already sitting in production,
restoring the real 20-change preview.
Also added a "Run now" button to each row on the Playlists list page
(previously only on a playlist's own detail page), for the common case
of adding new tracks and wanting to sync immediately.
Nav references were stale (Manage credentials/playlists -> Settings >
Credentials, Playlists), the credentials table was missing the
required/optional split and enable-disable toggles, and AzuraCast's
description was wrong (it's not a buy-link source, that's Bandcamp/Qobuz --
AzuraCast's key just triggers an immediate reprocess). Added the playlist
compare view, dashboard public-IP check, and Artist Casing page to the
feature list and day-to-day usage section.
A handful of confirmed candidates reference files that are no longer in
the beets DB (removed by some other pass, or never re-imported after a
move) but are still sitting on disk -- beet remove can't match them by
id or path either way, so they failed forever. If the file still exists
after a "No matching items found" error, remove it directly instead of
counting it as a failure. Applies to both --apply and --apply-pairs.
Playlist detail page's status section (and the new two-column compare
view) was never rendering: the router awaited status_service.playlist_status(),
but that function is plain sync, so awaiting its dict return raised
"object dict can't be used in 'await' expression" on every load, silently
caught and shown as a generic fetch error. Removed the erroneous await.
find-fuzzy-dupes.py's --apply and --apply-pairs both deleted by a `path::`
regex query even though the beets id was already available in scope --
same mixed-path-storage issue (pre/post beets-2.11-upgrade items store
absolute vs. library-relative paths) dedup-library.sh already worked
around by deleting via id instead. This silently failed to match for most
confirmed fuzzy-audio deletions (verified: 19 of 20 in one run). Added a
delete_id column to dedup_candidates, threaded through the scan/apply
pipeline, and switched both delete call sites to `id:`.
Dedup scans also never checked whether a pair was already sitting in the
pending list, so every re-scan (including the daily schedule) added a new
row for the same unreviewed duplicate -- cleaned up 38 redundant rows
already in production and added a check so future scans skip a pair
that's already pending.
Dashboard gets a Public IP stat card (cached 10 min, fetched via ipify)
as a quick confidence check that outbound traffic is actually routed
through gluetun's VPN and not the home connection.
Credentials page now shows a fact-checked one-line description per
service and marks Spotify/Soulseek/Navidrome as Required (no toggle --
they're load-bearing for every playlist sync). The four optional
integrations (Bandcamp, AzuraCast, Qobuz, Telegram) get a real on/off
switch: Bandcamp pauses its scheduled job (same mechanism as the Jobs
page), the other three just remove their rendered credential file, which
the scripts that read them already treat as "not configured, skip" --
no script changes needed. Disabled optional scopes also drop off the
dashboard's credential status row.
New Settings > Artist Casing page to view/add/remove entries in
artist-canonical.list without shelling in -- adding a name that already
matches case-insensitively replaces its casing in place instead of
duplicating.
Playlist detail page's status section is now two side-by-side columns:
the original Spotify tracklist on the left, and a tabbed Downloaded /
Waiting-to-download view on the right (client-side tabs via Alpine),
both with an internal scroll area so long playlists don't pull the
columns out of sync. Uses the existing status_service reconciliation
(sldl index + beets + quarantine) -- no backend changes needed there.
.field's margin-bottom (meant for vertically-stacked fields) pushed each
field's box lower than the bare Filter button under align-items: flex-end,
making the button look higher than the search/playlist/format boxes next
to it. New .filter-bar class zeroes that margin instead of the previous
ad-hoc inline flex styling.
Settings is now one top-nav entry with Credentials and Jobs as sidebar
sub-pages (jobs moved from /jobs to /settings/jobs). Manual import page
replaces the bare file input and free-text filename field with a drag-drop
dropzone and a proper file-picker table. Genres page drops the "force"
checkbox for two explicit buttons (Preview / Fix genres now). Dedup's
"Scan now" runs both the file-naming and acoustic passes together, and the
table labels which pass caught each candidate instead of showing the raw
pass name. .btn-ghost gets a visible border so it reads as a real button
next to Delete/Danger actions instead of looking unaligned. Job names
throughout the UI are now human-readable instead of raw job_key strings.
Also includes the fpcalc exit-code fix from earlier (fingerprint index
was discarding valid fingerprints on files with a benign decode warning).
README covers what alembic does, prerequisites, a full deploy quickstart,
credential setup, and a page-by-page usage tour, written for someone
running Docker but not necessarily comfortable coding or sysadmin work.
Favicon/icons were prepared in a separate theme scratch dir but never
wired in; moved them into app/static and pointed base.html at the full
icon set (SVG, PNG fallbacks, apple-touch-icon, manifest) instead of the
single inline data-URI SVG it had before.
confirm_and_apply no longer marks candidates confirmed before their apply
job actually succeeds (a stuck/skipped_lock run was silently vanishing
confirmed deletes without deleting anything), switches the fuzzy pass to
--apply-pairs to avoid a full rescan on every confirm, and adds a job
timeout so a hung subprocess can't hold the global pipeline lock forever.
dedup-library.sh's Pass 2-4 use gawk-only features (gensub, POSIX interval
regex) but were running under mawk (Debian's default awk) in the deployed
image, throwing silent syntax errors on every run -- switched those
invocations to gawk explicitly and added it to the Dockerfile.
Also: per-track delete button on the library list/detail pages, and two
CSS fixes (the primary button's hover gradient was getting clobbered by
the base .btn:hover rule's plain background, and the select dropdown
arrow had no background-size so it rendered oversized).
Lets a candidate be marked ignored instead of confirmed/applied; future
scans skip re-flagging the same path pair once it's been ignored, since
a rescan can flip which side ranks as keep/delete without changing the
user's earlier decision that the pair is fine as two copies. Also adds
--apply-pairs to find-fuzzy-dupes.py to apply pre-confirmed pairs
without a full rescan.
Playlist M3U regen now writes to a temp file and renames it into place,
so a stray wrong-owner leftover file (root:root, from pre-migration
host-cron runs) can't block nightly writes the way it did last night.
Dashboard now lists tracks added to the beets library in the last 24h
with playlist and format, sourced from a new beets_service.recently_added().
andrew