e5d9c7f0434f1391991e6f3cacd6f442e4a0bdd7
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 andrew
|
a9e1263b01 |
Render playlist configs directly in-app; retire regen.sh and playlists.json (R3)
The app owns every input to a playlist's sldl .conf (playlists in its DB, credentials in its encrypted store), so the old DB -> playlists.json -> regen.sh subprocess -> regex credential patch-back chain was three serialization hops for no reason. Collapse it: - credential_service.render_playlist_confs(db) renders each <playlist>.conf from _template.conf in one pass: path placeholders substituted as literal text, then the four credential lines set, written 0600. This is now the single source of .conf rendering. - playlist_service loses _write_playlists_json and regenerate_confs; _sync_to_disk just calls render_playlist_confs and syncs the scheduler. No subprocess, no intermediate JSON file. - render_scope for spotify/soulseek re-renders confs via the same function (spotify still also writes _spotify.env for the python scripts). The dead _patch_conf_field / _every_playlist_conf helpers are removed. - Delete pipeline/configs/regen.sh and drop it from the Dockerfile chmod; update _template.conf's comments. Nothing outside playlist_service consumed regen.sh or playlists.json (verified). Also closes the last remnants of the S2/S3 injection surface: names are re-validated and path/credential values are substituted as literals, never through sed or a shell. Verified: end-to-end render in a throwaway DB (paths, creds incl. a password with shell metacharacters, 0600, bad-name rejection) and a live re-render of all 17 playlist configs with credentials preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
andrew
|
b135f11557 |
Security hardening and first-run/portability improvements
Security (P0): - Remove committed session-secret default; auto-generate and persist a random secret to the config volume when SESSION_SECRET is unset (prevents forgeable session cookies / auth bypass). - Validate playlist names to a safe charset and render sldl configs via literal Python substitution instead of sed (closes a command-injection and path-traversal path through playlist names). - shlex-quote credential values written to shell-sourced env files, and strip newlines from values patched into .conf files. - Render playlist .conf files 0600; warn at startup if the master key is co-located with the config volume; document keeping it separate. Portability: - Configurable timezone via TZ (default UTC) instead of hardcoded Edmonton. - Remove personal defaults (navidrome user "andrew", ephemeral.club URLs). - Ship generic example seeds; move the cross-album dedup keep-list and the legacy playlist import to editable config files; drop the personal _upgrade.csv. - Generic VPN reference in docker-compose.snippet.yml. First-run experience: - Redirect to /setup instead of 500 when OIDC is unconfigured; surface a missing master key inline; entrypoint exits with an actionable message when the config folder is not writable. - Add unauthenticated /health (JSON) and /setup (checklist) diagnostics. Docs: - Write docs/ARCHITECTURE.md and docs/MIGRATION.md (previously referenced but missing); expand README with ownership, backups, advanced settings, and migration guidance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
andrew
|
c87ddc2649 |
Dashboard detail, library filtering/pagination, simplified playlist times,
job descriptions Dashboard: format breakdown (FLAC/MP3/etc. counts), approximate library size (bitrate*length/8, same approximation `beet stats` itself uses -- verified to match its "100.5 GiB" output exactly against the real library), and a credential auth-state summary per scope (configured y/n plus, for Bandcamp, cookie expiry parsed locally from the stored cookie jar -- deliberately not a live network probe like pipeline-status.sh's Qobuz check, since this renders on every dashboard load). Library: was one unfiltered page dumping all 4072 tracks. Added search (artist/title/album), playlist and format filter dropdowns, and real SQL-level pagination (LIMIT/OFFSET, not fetch-everything-then-slice). beets_service gained count_items()/distinct_formats() to support this. Playlists: cron_expr is still the stored/scheduled representation, but the UI now shows and edits a plain daily time picker instead of raw cron syntax -- every playlist schedule today is a simple daily HH:MM anyway. playlist_service.cron_to_time()/time_to_cron() convert at the router boundary; verified round-trip against all real playlist cron values. Jobs: each maintenance job now shows a short one-line description of what it actually does (MAINTENANCE_JOB_DESCRIPTIONS), and "next run" is formatted consistently with playlists' time style (HH:MM, with a day qualifier for non-today runs -- maintenance jobs can be weekly/monthly, unlike playlists' plain daily schedule). Verified end-to-end against the real production data (4072 tracks, 100.5 GiB, real credentials): stats/format-breakdown/search/pagination all correct, all 7 credential scopes report configured with Bandcamp's real cookie expiry (325 days), and a full authenticated page-render sweep (dashboard, library with every filter combination, playlist detail, jobs) all returned 200 with no template errors. |
||
|
andrew
|
a1f5bbb121 |
Fix regen.sh PATH bug; add legacy credential import script
playlist_service.regenerate_confs() was passing a bare PATH=/usr/bin:/bin to the regen.sh subprocess, but python3 (which regen.sh shells out to for parsing playlists.json) lives at /opt/venv/bin/python3 in this image. Every call silently generated zero .conf files -- no error surfaced, since the failure was inside a `python3 -c ... | while read` pipeline whose input was just empty. Found this deploying to the real host: seed_legacy() reported "17 playlists" successfully, but zero .conf files existed. Fixed to match pipeline_runner's already-correct PATH. scripts/import_legacy_credentials.py: the Stage 1/2 one-time migration script -- reads spotify/soulseek creds from a sample playlist .conf, Navidrome's password (extracted from navidrome-scan.sh, never lived in a file), and bandcamp/azuracast/qobuz/telegram from their legacy scattered locations, then populates credential_service. Read-only against the legacy files, idempotent. Run as a one-off `docker run --entrypoint python` with the legacy paths bind-mounted read-only (documented in the script's own docstring) -- entrypoint.sh always execs uvicorn regardless of CMD, so --entrypoint must override it for one-off invocations like this. Verified against the real production host: seeded all 17 legacy playlists with their historical cron schedules, confirmed all 17 .conf files render correctly after the PATH fix, ran the credential import for all 7 scopes, and byte-diffed the rendered spotify/soulseek values in a sample .conf against the original (identical, verified via diff -q without displaying the actual secret values in any tool output). Also had to chown 4 root:root mode-600 legacy credential files (bandcampconfig/cookies.txt, azuracastconfig/api_key, qobuzconfig/token, telegramconfig/notify.env) to 1000:1000 so the container could read them for the import -- root retains read access regardless, so this doesn't affect the still-running host-cron pipeline. |
||
|
andrew
|
4e0cfb8463 |
Add scheduler_service and pipeline_runner
pipeline_runner: subprocess execution wrapper replacing flock -n with a single asyncio.Lock (deliberately one global lock, not per-resource -- matches the old flock's all-jobs-share-one-lock behavior rather than over-engineering it). Records one job_runs row per invocation (started/finished/status/exit_code/summary/log_path/triggered_by). If the lock is already held, records status='skipped_lock' immediately instead of silently dropping the run. Found and fixed a real concurrency bug via testing: the obvious asyncio.wait_for(lock.acquire(), timeout=0) idiom for a non-blocking try-acquire is broken in asyncio -- the wrapping Task's first iteration and the timeout-0 callback race with no guaranteed ordering, so it timed out on literally every call, including the very first uncontended one. Fixed using lock.locked() + acquire(), relying on acquire()'s fast path never suspending when uncontended. scheduler_service: AsyncIOScheduler with the default in-memory job store (NOT SQLAlchemyJobStore -- it pickles job functions to persist them, and the _lib()/_beet() factory closures here aren't picklable; MemoryJobStore avoids this since register_all_jobs() rebuilds the whole schedule from code + DB on every startup anyway). All 19 maintenance jobs ported 1:1 from /etc/cron.d/sldl-maintenance with their exact cron expressions; dedup deliberately has no --apply (dry-run only on schedule, per the false-negative-biased dedup preference). Playlist jobs are synced live from the playlists table (sync_playlist_jobs), so add/remove-in-the-UI takes effect with no redeploy -- wired into playlist_service's create/update/ delete. Maintenance job enable/disable persists to the scheduled_jobs table so a paused job stays paused across a restart despite the in-memory job store. trigger_now() supports "Run now" by invoking a job's function immediately with triggered_by='manual', bypassing its schedule. Wired into main.py's lifespan (start on boot, shutdown on exit). Verified via integration tests: 36 jobs register correctly (17 seeded playlists + 19 maintenance) with correct cron triggers; live playlist create/delete correctly adds/removes its scheduler job; maintenance enable/disable persists to the DB and takes effect live; trigger_now correctly bypasses the schedule; pipeline_runner correctly handles success/failure/timeout/concurrent-lock-contention with real subprocesses; and the full app boots with the scheduler running and shuts down cleanly. |
||
|
andrew
|
5eaae8e813 |
Add playlist_service and credential_service
playlist_service: CRUD for the playlists table, playlists.json export,
regen.sh invocation, and a one-time seed_legacy() importing the 17-entry
legacy array. Creating/updating a playlist automatically re-renders confs
and re-patches credentials into any newly-generated .conf file.
credential_service: encrypted credential storage (via security/crypto.py)
and per-scope rendering to the exact files the pipeline scripts read --
every <playlist>.conf (spotify/soulseek), navidrome/admin.env,
bandcamp/config.env+cookies.txt, azuracast/api_key, qobuz/{token,app_id,region},
telegram/notify.env. set_credentials() batches multi-field saves so a render
never sees a half-populated scope.
Also fixes a real bug found via integration testing: regen.sh's embedded
python3 -c snippet used backslash-escaped quotes inside a single-quoted
bash string, which is invalid Python syntax (backslash passed through
literally) -- switched to double-quoted bash wrapper + single-quoted Python
strings.
Verified end-to-end: seeded 17 playlists, regenerated all 17 .conf files,
saved spotify/soulseek/navidrome credentials and confirmed correct
patching into rendered files (including bash-sourcing admin.env with
special characters in the password), and confirmed delete/create both
correctly add/remove .conf files with credentials pre-patched on create.
|