slskd is not part of the pipeline -- sldl is its own Soulseek client and
nothing in alembic calls slskd's API; the digest's reachability ping was its
only mention. It keeps running on the host purely as the user's own
file-sharing presence, so checking it here just risked a permanent bogus
warning line for something alembic doesn't depend on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Iterating on the 0.6.5 HTML digest on a real phone: <pre> column layout
forces fixed-width lines that wrap into unreadable fragments ("54 tracks
in / M3U"), and one <pre> per section rendered as a stack of copy-button
widgets. Every row is now plain proportional text -- colored dot, bold
label, "-- value" -- which reflows cleanly at any screen width; breakdowns
(added-today per playlist, library by format) become bullet lists. Header
banner (brand line, dot tally, all-clear/N-issues blockquote) unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The daily digest is now formatted with Telegram HTML: bold section headers
with <pre>-aligned columns, colored circle emoji as status dots, a branded
header line, and a top blockquote callout (all clear vs N issues). All free
text is &/</> escaped so a stray angle bracket in a log snippet can't break
the parse and swallow the whole message. notify-telegram.sh gains an --html
flag (parse_mode=HTML) used by the digest send; plain-text callers are
unchanged. Truncation is now tag-safe in HTML mode: cut on a line boundary,
re-close an open <pre>, and use an escaped marker -- the old literal
"...<truncated>" tail would itself have 400'd the send.
Two real bugs surfaced while testing:
- pipeline-status.sh pins its own PATH, which lacked /opt/venv/bin where
beet lives, so every beet probe ("added today", "Library by format", the
mp3-now count) has been silently empty behind 2>/dev/null since the cron
migration. PATH now includes the venv; both sections show real numbers.
- strip-watermark-art.py aborted its entire weekly run when metaflac stalled
on ONE file (seen today: a healthy 190KB cover took >15s under disk
contention, TimeoutExpired killed the job). Per-file timeout is now 60s
and a timeout skips that file with a warning instead of failing the run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Telegram digest reported healthy weekly jobs as "never run yet": its
maintenance section still globbed for cron-era per-script log filenames
(clean-index-*.log etc.) that jobs run through pipeline_runner no longer
write. It now reads job_runs, the scheduler's own record, so it reports the
last success (age + summary snippet from that run's log), flags a newer
failed or lock-skipped attempt on the same line, and distinguishes "never
succeeded (last attempt: skipped, lock busy)" from genuinely never
scheduled. Also matched the clear-bad-genres snippet grep to the script's
current summary wording, and dropped the now-unused newest_log helper.
The DB also showed WHY two jobs had never run: on Sunday the 08:30
strip-mb-tags run took 10m19s while holding the shared pipeline lock, so
strip-watermark-art (08:35) and scrub-watermark-text (08:40) hit flock-style
instant skip and lost their only slot of the week -- the 08:40 job missed by
19 seconds. Scheduled runs now wait up to 30 minutes for the lock
(SCHEDULED_LOCK_WAIT_SECONDS) and only then record skipped_lock, so
fixed-time blocks queue instead of starving; manual "Run now" keeps the
instant skip since a person expects an immediate answer.
Tests: scheduled-run queueing, lock-wait timeout, and manual instant-skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1. enrich-buy-url.py re-queried every no-match track on every daily run (external,
rate-limited lookups over the whole backlog), which blew past its 30-min
timeout. Cache "tried, no match" with a BUY_URL_TRIED marker tag and skip
those for 30 days (--force still re-checks). This is what made "Look up buy
links" fail with TIMEOUT after 1800s.
2. Move maintenance:enrich_buy_url to run last (09:45) in the 9am block, after
the fingerprint index (09:25) and status report (09:30). enrich starting at
09:10 and holding the shared lock is what left "Rebuild fingerprint index"
skipped_lock.
3. pipeline-status.sh: guard the qobuz/app_id read with a file-exists check.
`< missing 2>/dev/null` still leaks the shell's redirection error (opened
before 2>/dev/null applies), so the daily report logged
"qobuz/app_id: No such file or directory" every run.
4. dedup confirm_and_apply: a candidate whose delete target no longer exists
(already removed by an earlier dedup/upgrade/hand) was filtered out and the
apply silently no-op'd, leaving it stuck in the queue with no way to delete
or clear it (the 100 gecs case). Now mark such candidates applied so a Delete
click clears them. Covered by tests/test_dedup_review.py.
Tests: 46 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Navidrome password is now passed to curl via stdin (--data-urlencode "p@-")
in navidrome-scan.sh and pipeline-status.sh, so it no longer appears in
ps/proc. Verified the query sent is identical and a live scan still triggers.
- MIN_ARTIST_DIRS (the share-health gate) is now a setting, threaded through to
the pipeline env, so a user with a small library can lower it instead of the
scan/sync being permanently blocked by the hardcoded 500.
- /auth/logout is now POST-only (with a nav form + aria-label), so a drive-by
GET can't log the user out; enforced allowed_email already landed separately.
- view_log now confirms the run's log_path resolves under the logs dir before
serving it (defense in depth).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- fix-genre.sh: set -euo pipefail; guard the tag-remove metaflac calls and the
'beet ls | head -5' verify (SIGPIPE) that would otherwise abort.
- sync-bandcamp.sh: set -euo pipefail; the destructive chain already used
'if cmd; then..else WARN' (set -e exempt); guard the staging mv.
- upgrade-mp3-to-flac.sh: set -euo pipefail; capture sldl/replace-with-better
exit codes without aborting (was 'cmd; RC=$?', which set -e breaks); read
Soulseek creds from the first available .conf instead of a hardcoded y2k.conf,
guarded; guard the xargs purge and a grep -c.
- pipeline-status.sh: deliberately kept at set -u (documented). It is read-only
and assembles ~30 independent probes; set -e would abort the whole digest on
one missing optional file, which is exactly how the 9am notification broke.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- init_db applies each schema file with executescript (semicolons in triggers
and string literals no longer break it) and records schema_version itself
after each file, so a forgotten in-file bump can't re-run an ALTER and crash
startup. Re-running is a no-op.
- On startup, sweep any job_runs left in 'running' (killed by a restart) to
'interrupted' so the UI/history don't show a run stuck running forever.
- prep-audio.sh skips loudgain and cue_file cleanly when they aren't installed
(they aren't in the base image) instead of failing on a missing binary and
logging errors; pipeline-status.sh routes syslog through a guard so a missing
`logger` doesn't spew into the job log.
- Add a branded 404 page (rendered to static HTML, no external runtime) via a
custom exception handler that leaves all other responses, including the login
redirect, untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Security (P0):
- Remove committed session-secret default; auto-generate and persist a
random secret to the config volume when SESSION_SECRET is unset
(prevents forgeable session cookies / auth bypass).
- Validate playlist names to a safe charset and render sldl configs via
literal Python substitution instead of sed (closes a command-injection
and path-traversal path through playlist names).
- shlex-quote credential values written to shell-sourced env files, and
strip newlines from values patched into .conf files.
- Render playlist .conf files 0600; warn at startup if the master key is
co-located with the config volume; document keeping it separate.
Portability:
- Configurable timezone via TZ (default UTC) instead of hardcoded Edmonton.
- Remove personal defaults (navidrome user "andrew", ephemeral.club URLs).
- Ship generic example seeds; move the cross-album dedup keep-list and the
legacy playlist import to editable config files; drop the personal
_upgrade.csv.
- Generic VPN reference in docker-compose.snippet.yml.
First-run experience:
- Redirect to /setup instead of 500 when OIDC is unconfigured; surface a
missing master key inline; entrypoint exits with an actionable message
when the config folder is not writable.
- Add unauthenticated /health (JSON) and /setup (checklist) diagnostics.
Docs:
- Write docs/ARCHITECTURE.md and docs/MIGRATION.md (previously referenced
but missing); expand README with ownership, backups, advanced settings,
and migration guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves all ~30 pipeline scripts, configs, and the vendored sldl binary into
this repo (source /opt/sldl left untouched). Removes all docker exec/docker
compose dependencies now that beets and sldl run in-process/as a subprocess
of this container instead of via soulbeet/on-demand sldl containers.
Replaces hardcoded host paths, Navidrome credentials, and Spotify credential
sourcing with env-var-driven paths and shared credential loaders. Adds
Dockerfile, entrypoint.sh, requirements.txt, docker-compose.snippet.yml, and
the initial app DB schema.
andrew