Move six scripts that are not wired into the app, scheduler, or web UI out of
the active pipeline/lib into pipeline/lib/manual, so the scheduled scripts are
easy to see and these stay available for hands-on use: convert-m4a.sh,
fix-empty-album.sh, backfill-spotify-tags.sh, backfill-buy-url.py,
recover-azuracast-playlists.py, import-dj-collection.py. Adds a README
explaining each, and extends the Dockerfile chmod to cover the new folder.
Nothing referenced these from active code (verified), so no wiring changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- fix-genre.sh: set -euo pipefail; guard the tag-remove metaflac calls and the
'beet ls | head -5' verify (SIGPIPE) that would otherwise abort.
- sync-bandcamp.sh: set -euo pipefail; the destructive chain already used
'if cmd; then..else WARN' (set -e exempt); guard the staging mv.
- upgrade-mp3-to-flac.sh: set -euo pipefail; capture sldl/replace-with-better
exit codes without aborting (was 'cmd; RC=$?', which set -e breaks); read
Soulseek creds from the first available .conf instead of a hardcoded y2k.conf,
guarded; guard the xargs purge and a grep -c.
- pipeline-status.sh: deliberately kept at set -u (documented). It is read-only
and assembles ~30 independent probes; set -e would abort the whole digest on
one missing optional file, which is exactly how the 9am notification broke.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds strict mode to four maintenance scripts, with guards so expected non-zero
exits (empty beet queries, a missing config, a curl timeout) log/skip instead
of aborting: gen-djmix bails cleanly if djmix-albums.txt is absent and tolerates
a bad single query; gen-vgm tolerates an empty query; strip-mb-tags quotes its
path arg and best-efforts the trailing beet update; notify-telegram guards curl
so a network error still reaches its explicit "send failed" branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds strict mode to the file-replacement script (the destructive path used by
the Bandcamp sync and the mp3->flac upgrade).
Uses `set -eu` deliberately WITHOUT pipefail: the script reads tags through
many `cmd | head -1` pipelines, and under pipefail `head` closing the pipe
early makes the producer die with SIGPIPE, turning good reads into false
failures. pipefail adds no safety here since every piped read feeds a value
that is immediately emptiness-checked.
Guards added so a single expected non-zero doesn't abort a destructive run:
- the four progressive `beet ls` match queries fall through on empty/error
- beet remove / beet import / rm / leftover-import / find -delete log a warning
instead of aborting mid-replace
- copy_tags_from_existing is called with `|| true` (also suspends set -e for its
best-effort per-tag metaflac calls)
- explicit `exit 0` so a successful run reports success
Verified: function-level tests (ranking, match fall-through under failing beet)
and a full dry-run against the real library, all exit 0 under set -e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn on strict mode for the file that deletes music, and neutralize the
set -e footguns it exposed:
- json_emit returned non-zero in the default (non-JSON) path; as a standalone
call that aborted the run on the first candidate. Now returns 0 explicitly.
- process_group returned 1 when a group had no files on disk, aborting the
whole run since callers don't check it; now returns 0, with an explicit
return 0 at the end so the loop's last status can't leak out.
- The cross-album keep-list pipeline returns non-zero on an all-comments file
(the default) under pipefail; guarded with || true (empty = protect nobody).
- Apply-mode beet remove/import/move and rm are now best-effort (|| log WARN)
so one failed deletion doesn't abort the rest of a confirmed apply.
- Added an explicit `exit 0`: the script previously exited non-zero in --apply
mode because its last line was a failing `[[ $APPLY -eq 0 ]]` test, making a
successful apply look like a failed run to the caller.
Verified: full dry-run over the real library exits 0 through all four passes;
process_group ranks FLAC over MP3, emits JSON, and handles empty groups without
aborting; and dedup:scan records success through pipeline_runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- init_db applies each schema file with executescript (semicolons in triggers
and string literals no longer break it) and records schema_version itself
after each file, so a forgotten in-file bump can't re-run an ALTER and crash
startup. Re-running is a no-op.
- On startup, sweep any job_runs left in 'running' (killed by a restart) to
'interrupted' so the UI/history don't show a run stuck running forever.
- prep-audio.sh skips loudgain and cue_file cleanly when they aren't installed
(they aren't in the base image) instead of failing on a missing binary and
logging errors; pipeline-status.sh routes syslog through a guard so a missing
`logger` doesn't spew into the job log.
- Add a branded 404 page (rendered to static HTML, no external runtime) via a
custom exception handler that leaves all other responses, including the login
redirect, untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security (P0):
- Remove committed session-secret default; auto-generate and persist a
random secret to the config volume when SESSION_SECRET is unset
(prevents forgeable session cookies / auth bypass).
- Validate playlist names to a safe charset and render sldl configs via
literal Python substitution instead of sed (closes a command-injection
and path-traversal path through playlist names).
- shlex-quote credential values written to shell-sourced env files, and
strip newlines from values patched into .conf files.
- Render playlist .conf files 0600; warn at startup if the master key is
co-located with the config volume; document keeping it separate.
Portability:
- Configurable timezone via TZ (default UTC) instead of hardcoded Edmonton.
- Remove personal defaults (navidrome user "andrew", ephemeral.club URLs).
- Ship generic example seeds; move the cross-album dedup keep-list and the
legacy playlist import to editable config files; drop the personal
_upgrade.csv.
- Generic VPN reference in docker-compose.snippet.yml.
First-run experience:
- Redirect to /setup instead of 500 when OIDC is unconfigured; surface a
missing master key inline; entrypoint exits with an actionable message
when the config folder is not writable.
- Add unauthenticated /health (JSON) and /setup (checklist) diagnostics.
Docs:
- Write docs/ARCHITECTURE.md and docs/MIGRATION.md (previously referenced
but missing); expand README with ownership, backups, advanced settings,
and migration guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A handful of confirmed candidates reference files that are no longer in
the beets DB (removed by some other pass, or never re-imported after a
move) but are still sitting on disk -- beet remove can't match them by
id or path either way, so they failed forever. If the file still exists
after a "No matching items found" error, remove it directly instead of
counting it as a failure. Applies to both --apply and --apply-pairs.
Playlist detail page's status section (and the new two-column compare
view) was never rendering: the router awaited status_service.playlist_status(),
but that function is plain sync, so awaiting its dict return raised
"object dict can't be used in 'await' expression" on every load, silently
caught and shown as a generic fetch error. Removed the erroneous await.
find-fuzzy-dupes.py's --apply and --apply-pairs both deleted by a `path::`
regex query even though the beets id was already available in scope --
same mixed-path-storage issue (pre/post beets-2.11-upgrade items store
absolute vs. library-relative paths) dedup-library.sh already worked
around by deleting via id instead. This silently failed to match for most
confirmed fuzzy-audio deletions (verified: 19 of 20 in one run). Added a
delete_id column to dedup_candidates, threaded through the scan/apply
pipeline, and switched both delete call sites to `id:`.
Dedup scans also never checked whether a pair was already sitting in the
pending list, so every re-scan (including the daily schedule) added a new
row for the same unreviewed duplicate -- cleaned up 38 redundant rows
already in production and added a check so future scans skip a pair
that's already pending.
Dashboard gets a Public IP stat card (cached 10 min, fetched via ipify)
as a quick confidence check that outbound traffic is actually routed
through gluetun's VPN and not the home connection.
Settings is now one top-nav entry with Credentials and Jobs as sidebar
sub-pages (jobs moved from /jobs to /settings/jobs). Manual import page
replaces the bare file input and free-text filename field with a drag-drop
dropzone and a proper file-picker table. Genres page drops the "force"
checkbox for two explicit buttons (Preview / Fix genres now). Dedup's
"Scan now" runs both the file-naming and acoustic passes together, and the
table labels which pass caught each candidate instead of showing the raw
pass name. .btn-ghost gets a visible border so it reads as a real button
next to Delete/Danger actions instead of looking unaligned. Job names
throughout the UI are now human-readable instead of raw job_key strings.
Also includes the fpcalc exit-code fix from earlier (fingerprint index
was discarding valid fingerprints on files with a benign decode warning).
confirm_and_apply no longer marks candidates confirmed before their apply
job actually succeeds (a stuck/skipped_lock run was silently vanishing
confirmed deletes without deleting anything), switches the fuzzy pass to
--apply-pairs to avoid a full rescan on every confirm, and adds a job
timeout so a hung subprocess can't hold the global pipeline lock forever.
dedup-library.sh's Pass 2-4 use gawk-only features (gensub, POSIX interval
regex) but were running under mawk (Debian's default awk) in the deployed
image, throwing silent syntax errors on every run -- switched those
invocations to gawk explicitly and added it to the Dockerfile.
Also: per-track delete button on the library list/detail pages, and two
CSS fixes (the primary button's hover gradient was getting clobbered by
the base .btn:hover rule's plain background, and the select dropdown
arrow had no background-size so it rendered oversized).
Lets a candidate be marked ignored instead of confirmed/applied; future
scans skip re-flagging the same path pair once it's been ignored, since
a rescan can flip which side ranks as keep/delete without changing the
user's earlier decision that the pair is fine as two copies. Also adds
--apply-pairs to find-fuzzy-dupes.py to apply pre-confirmed pairs
without a full rescan.
- beets_service: filter params now use truthy checks instead of `is not
None`, since a real <select> left on "(any)" submits an empty string,
not an absent param -- the filter form silently matched zero rows for
any real submission. Also match grouping tokens within "; "-joined
multi-playlist values instead of requiring an exact string match.
- dashboard: credential status renders as compact dot indicators instead
of badge+text chips, so long scope names (telegram) stay on one line.
- credentials page: two-column grid layout instead of one long column.
- find-fuzzy-dupes.py: add --json/--only-paths flags matching
dedup-library.sh's convention, so it can plug into the same review
queue.
- dedup_review_service: add scan_fuzzy() and route confirm_and_apply()
to the correct underlying script (dedup-library.sh vs
find-fuzzy-dupes.py) per candidate's pass_name, since tag-based passes
miss duplicates whose tags differ even when the audio is identical
(e.g. a remix credited to different artists between two copies).
- dedup page: add a "Scan for audio duplicates" trigger alongside the
existing tag-based scan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dedup-library.sh: additive --json (one NDJSON line per candidate deletion
to stdout, alongside the unchanged human log) and --only-paths FILE (in
--apply mode, only actually delete entries whose path is in FILE; without
it, --apply deletes everything as before -- existing direct callers are
unaffected). Without --only-paths the safety re-verification is free:
--apply --only-paths re-runs all 4 passes from scratch on every invocation,
so if a group's ranking changed since a scan (e.g. the old keep_path is
gone), the fresh pass assigns the previously-"delete" path the KEEP role
instead and the only-paths allowlist naming it is simply never consulted --
no duplicate ranking logic needed in the review service.
spotify-genre.py: additive --json emitting one JSON line per genre change
(dry-run or --apply) for genre_review_service to persist.
pipeline_runner.run_job_capture(): like run_job() but captures stdout as
text (still under the same shared lock, still writes a job_runs row) for
callers that need to parse structured output rather than just log it.
dedup_review_service.scan() persists dry-run candidates into
dedup_runs/dedup_candidates. confirm_and_apply() re-checks confirmed
candidates still exist before invoking --apply --only-paths, so nothing is
ever deleted without an explicit confirm -- matches the false-negative-
biased dedup preference. scheduler_service's maintenance:dedup job now
goes through this (still dry-run only, every day).
genre_review_service.run() wraps spotify-genre.py for both dry-run preview
and the real scheduled --apply run, persisting every run's diff into
genre_runs/genre_candidates either way -- genre writes keep their current
auto-apply behavior (low-risk, reversible, GENRE_LOCK-protected) but are
now reviewable after the fact. lock_artist_genre() gives a one-click revert
path when a run gets something wrong.
Added minimal routers+templates for /dedup (scan, review, confirm-and-
delete) and /genres (preview, review, lock-old-genre).
Verified end-to-end against REAL duplicate files (not mocked): built an
actual FLAC+MP3 duplicate pair in a real beets library, ran dedup-library.sh
--json and confirmed correct JSON output, verified --apply --only-paths
with an empty confirm list deletes nothing and with the real confirmed path
deletes exactly that file (DB + disk) while preserving the FLAC, and ran
the full dedup_review_service scan->confirm->apply flow through the same
fixture. genre_review_service and spotify-genre.py --json verified against
mocked/direct output (spotify-genre.py's own artist-genre lookup needs a
live Spotify API call, out of reach in this sandbox). Confirmed the full
app boots with all five routers registered.
playlist_service: CRUD for the playlists table, playlists.json export,
regen.sh invocation, and a one-time seed_legacy() importing the 17-entry
legacy array. Creating/updating a playlist automatically re-renders confs
and re-patches credentials into any newly-generated .conf file.
credential_service: encrypted credential storage (via security/crypto.py)
and per-scope rendering to the exact files the pipeline scripts read --
every <playlist>.conf (spotify/soulseek), navidrome/admin.env,
bandcamp/config.env+cookies.txt, azuracast/api_key, qobuz/{token,app_id,region},
telegram/notify.env. set_credentials() batches multi-field saves so a render
never sees a half-populated scope.
Also fixes a real bug found via integration testing: regen.sh's embedded
python3 -c snippet used backslash-escaped quotes inside a single-quoted
bash string, which is invalid Python syntax (backslash passed through
literally) -- switched to double-quoted bash wrapper + single-quoted Python
strings.
Verified end-to-end: seeded 17 playlists, regenerated all 17 .conf files,
saved spotify/soulseek/navidrome credentials and confirmed correct
patching into rendered files (including bash-sourcing admin.env with
special characters in the password), and confirmed delete/create both
correctly add/remove .conf files with credentials pre-patched on create.
Moves all ~30 pipeline scripts, configs, and the vendored sldl binary into
this repo (source /opt/sldl left untouched). Removes all docker exec/docker
compose dependencies now that beets and sldl run in-process/as a subprocess
of this container instead of via soulbeet/on-demand sldl containers.
Replaces hardcoded host paths, Navidrome credentials, and Spotify credential
sourcing with env-var-driven paths and shared credential loaders. Adds
Dockerfile, entrypoint.sh, requirements.txt, docker-compose.snippet.yml, and
the initial app DB schema.
andrew