6114e6dc7aa79fa5372bdd3096f90ef787f2449e
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 andrew
|
3fbf580b88 |
0.6.2: Bypass sldl's Spotify client -- fetch the playlist ourselves, feed sldl a CSV
sldl's vendored Spotify client still calls GET /playlists/{id}/tracks, which
Spotify removed in its February 2026 API changes. Grandfathered apps still get
a pass; apps created after the change get a hard 403 there no matter how they
authenticate (client-credentials or a correctly-scoped OAuth user token), so
sldl can never load a playlist for a new app regardless of what we hand it.
Instead of depending on sldl's Spotify client at all, run-playlist.sh now reads
the playlist itself via the still-working /items endpoint (new
spotify-playlist-csv.py, creds sourced from _spotify.env) and invokes sldl with
the CSV + --input-type csv, which override the conf's input lines while -c still
supplies Soulseek login, paths, and quality settings (verified live). The same
CSV format upgrade-mp3-to-flac.sh already feeds sldl. All artists are
comma-joined in the CSV so multi-artist tracks search no worse than before, and
a 403/404 on the fetch prints the account-visibility hint instead of a bare
traceback.
Since sldl no longer talks to Spotify, playlist .confs no longer carry
spotify-id/spotify-secret/spotify-refresh: _template.conf drops them,
render_playlist_confs stops injecting them, and a Spotify credential save no
longer re-renders confs (only _spotify.env). The retag step in run-playlist.sh
reuses the sourced _spotify.env creds instead of scraping the conf lines that
no longer exist. Confs are also re-rendered once at app startup so
already-deployed confs converge on upgrade (and shed the stale secret lines)
without waiting for a playlist or credential change. Playlist delete now also
removes the generated .csv.
Tests updated: conf rendering must NOT contain Spotify creds but must keep the
input URL line; new coverage for _spotify.env rendering (quoting, refresh-token
presence/blank, 0600).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
andrew
|
e5d9c7f043 |
0.6: Spotify OAuth connect flow, AzuraCast base URL, fingerprint/buy-url fixes
- Add "Connect Spotify" OAuth flow (/connect/spotify) so newly created Spotify apps can read playlists: Spotify now requires a user token for playlist reads, which client-credentials alone can no longer provide. The stored refresh token is rendered as spotify-refresh into each playlist's sldl .conf -- sldl's own docs confirm supplying it skips its interactive login flow, which is what was causing multi-hour hangs on a stuck sync. Grandfathered apps keep working unchanged if no account is connected. - Fix the connect flow's redirect_uri: request.url_for() reflected the raw connection scheme (http) rather than what the reverse proxy actually served over, which Spotify's exact-match redirect_uri check rejected. Force https rather than trust the unproxied scheme. - Add an AzuraCast base URL credential field alongside the API key, with a keyfile fallback so the scheduled enrich-buy-url.py job can actually reach AzuraCast (previously it had no way to receive a base URL at all when run from the scheduler, so the reprocess call was silently always skipped). - Fix build-fingerprint-index.py exiting non-zero on any single fingerprint failure instead of only when nothing was written. - Guard enrich-buy-url.py's AzuraCast reprocess against an empty --azuracast-base (raised ValueError since PD2 removed the hardcoded base). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
andrew
|
fa86e992ad |
Robustness: crash-safe migrations, stale-run sweep, optional prep tools; branded 404
- init_db applies each schema file with executescript (semicolons in triggers and string literals no longer break it) and records schema_version itself after each file, so a forgotten in-file bump can't re-run an ALTER and crash startup. Re-running is a no-op. - On startup, sweep any job_runs left in 'running' (killed by a restart) to 'interrupted' so the UI/history don't show a run stuck running forever. - prep-audio.sh skips loudgain and cue_file cleanly when they aren't installed (they aren't in the base image) instead of failing on a missing binary and logging errors; pipeline-status.sh routes syslog through a guard so a missing `logger` doesn't spew into the job log. - Add a branded 404 page (rendered to static HTML, no external runtime) via a custom exception handler that leaves all other responses, including the login redirect, untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
andrew
|
b135f11557 |
Security hardening and first-run/portability improvements
Security (P0): - Remove committed session-secret default; auto-generate and persist a random secret to the config volume when SESSION_SECRET is unset (prevents forgeable session cookies / auth bypass). - Validate playlist names to a safe charset and render sldl configs via literal Python substitution instead of sed (closes a command-injection and path-traversal path through playlist names). - shlex-quote credential values written to shell-sourced env files, and strip newlines from values patched into .conf files. - Render playlist .conf files 0600; warn at startup if the master key is co-located with the config volume; document keeping it separate. Portability: - Configurable timezone via TZ (default UTC) instead of hardcoded Edmonton. - Remove personal defaults (navidrome user "andrew", ephemeral.club URLs). - Ship generic example seeds; move the cross-album dedup keep-list and the legacy playlist import to editable config files; drop the personal _upgrade.csv. - Generic VPN reference in docker-compose.snippet.yml. First-run experience: - Redirect to /setup instead of 500 when OIDC is unconfigured; surface a missing master key inline; entrypoint exits with an actionable message when the config folder is not writable. - Add unauthenticated /health (JSON) and /setup (checklist) diagnostics. Docs: - Write docs/ARCHITECTURE.md and docs/MIGRATION.md (previously referenced but missing); expand README with ownership, backups, advanced settings, and migration guidance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
andrew
|
9a116f3da4 |
Credential enable/disable + descriptions, artist casing settings page, playlist compare view
Credentials page now shows a fact-checked one-line description per service and marks Spotify/Soulseek/Navidrome as Required (no toggle -- they're load-bearing for every playlist sync). The four optional integrations (Bandcamp, AzuraCast, Qobuz, Telegram) get a real on/off switch: Bandcamp pauses its scheduled job (same mechanism as the Jobs page), the other three just remove their rendered credential file, which the scripts that read them already treat as "not configured, skip" -- no script changes needed. Disabled optional scopes also drop off the dashboard's credential status row. New Settings > Artist Casing page to view/add/remove entries in artist-canonical.list without shelling in -- adding a name that already matches case-insensitively replaces its casing in place instead of duplicating. Playlist detail page's status section is now two side-by-side columns: the original Spotify tracklist on the left, and a tabbed Downloaded / Waiting-to-download view on the right (client-side tabs via Alpine), both with an internal scroll area so long playlists don't pull the columns out of sync. Uses the existing status_service reconciliation (sldl index + beets + quarantine) -- no backend changes needed there. |
||
|
andrew
|
5601de3d44 |
Add playlists, credentials, and jobs pages; wire up dashboard
Vendored htmx.min.js (1.9.12) and alpine.min.js (3.14.1) into static/ -- self-hosted per the no-CDN/CSP requirement, included in base.html. routers/playlists.py: list/add/remove playlists, a detail page showing live per-playlist status via status_service (Spotify vs. beets vs. sldl index vs. quarantine), edit form for active/cron_expr/no_m3u/notes, and a "Run now" button wired to the jobs run-now endpoint. routers/credentials.py: write-only credential forms per scope (a saved secret is never sent back to the browser -- fields just show "(set)" and leaving a field blank keeps its current value, matching credential_service.set_credentials' partial-update semantics). routers/jobs.py: maintenance job list with run-now/enable-disable, and a recent-runs table that auto-refreshes via HTMX polling (hx-trigger="every 5s") against a partial-only endpoint -- the one place in the UI where avoiding a full-page reload actually matters, since job status changes while the page is sitting open. Per-run log viewing. Dashboard now shows real data (beets stats, playlist counts, recent job runs) instead of the placeholder from Task 2. Verified all 10 authenticated pages render successfully (200, no template errors) via TestClient with require_auth overridden, including the playlist detail page's graceful-degradation path when Spotify credentials aren't configured yet (renders a "couldn't fetch live status" message instead of a 500). |
||
|
andrew
|
2075d6cf66 |
Add dedup_review_service and genre_review_service
dedup-library.sh: additive --json (one NDJSON line per candidate deletion to stdout, alongside the unchanged human log) and --only-paths FILE (in --apply mode, only actually delete entries whose path is in FILE; without it, --apply deletes everything as before -- existing direct callers are unaffected). Without --only-paths the safety re-verification is free: --apply --only-paths re-runs all 4 passes from scratch on every invocation, so if a group's ranking changed since a scan (e.g. the old keep_path is gone), the fresh pass assigns the previously-"delete" path the KEEP role instead and the only-paths allowlist naming it is simply never consulted -- no duplicate ranking logic needed in the review service. spotify-genre.py: additive --json emitting one JSON line per genre change (dry-run or --apply) for genre_review_service to persist. pipeline_runner.run_job_capture(): like run_job() but captures stdout as text (still under the same shared lock, still writes a job_runs row) for callers that need to parse structured output rather than just log it. dedup_review_service.scan() persists dry-run candidates into dedup_runs/dedup_candidates. confirm_and_apply() re-checks confirmed candidates still exist before invoking --apply --only-paths, so nothing is ever deleted without an explicit confirm -- matches the false-negative- biased dedup preference. scheduler_service's maintenance:dedup job now goes through this (still dry-run only, every day). genre_review_service.run() wraps spotify-genre.py for both dry-run preview and the real scheduled --apply run, persisting every run's diff into genre_runs/genre_candidates either way -- genre writes keep their current auto-apply behavior (low-risk, reversible, GENRE_LOCK-protected) but are now reviewable after the fact. lock_artist_genre() gives a one-click revert path when a run gets something wrong. Added minimal routers+templates for /dedup (scan, review, confirm-and- delete) and /genres (preview, review, lock-old-genre). Verified end-to-end against REAL duplicate files (not mocked): built an actual FLAC+MP3 duplicate pair in a real beets library, ran dedup-library.sh --json and confirmed correct JSON output, verified --apply --only-paths with an empty confirm list deletes nothing and with the real confirmed path deletes exactly that file (DB + disk) while preserving the FLAC, and ran the full dedup_review_service scan->confirm->apply flow through the same fixture. genre_review_service and spotify-genre.py --json verified against mocked/direct output (spotify-genre.py's own artist-genre lookup needs a live Spotify API call, out of reach in this sandbox). Confirmed the full app boots with all five routers registered. |
||
|
andrew
|
270c9c0ed1 |
Add manual fix / manual import services and routers
library_edit.update_track_fields: single-track tag edits via `beet modify -y -m` (subprocess with an argument list, not beets.library.Library in-process -- an arg-list subprocess has no shell-injection surface to avoid in the first place, and correctly reuses beets' own configured path-format/move logic rather than reimplementing it against a hand-built Library object with guessed config). Every changed field gets a manual_fix_audit row with old/new values; triggers a Navidrome rescan afterward if anything changed. Uses "genres" (plural) as the field name, matching this deployment's actual beets column and every existing script's convention, not the generic beets docs' singular "genre". genre_fix.set_artist_genre and retag.retag_from_url wrap the existing fix-genre.sh and fix-track-metadata.py as callable services rather than reimplementing their per-format mutagen tag-writing logic. manual_import: ties pipeline_runner.run_import_track to the manual_imports table, mirrors import-track.sh's own argument-count-based disambiguation (0/1/2 args), plus import-me/ listing and upload-with-path-traversal-guard. routers/library.py + routers/import_.py: browsable library list, a track detail/edit page (tag edit, artist-wide genre override, retag-from-URL forms), and the import page (upload + trigger). Minimal templates for now, Task 9 covers full UI polish. Verified update_track_fields end-to-end against a REAL beets library (not mocked): generated a tagged FLAC with ffmpeg/metaflac, imported it via a real `beet import`, edited artist+title through the service, and confirmed the file was physically moved to the new path-template location, tags were rewritten on disk, the DB updated, and both changes landed correctly in manual_fix_audit. Also verified the full app boots with the new routers registered and all four new routes correctly redirect to login when unauthenticated. |
||
|
andrew
|
4e0cfb8463 |
Add scheduler_service and pipeline_runner
pipeline_runner: subprocess execution wrapper replacing flock -n with a single asyncio.Lock (deliberately one global lock, not per-resource -- matches the old flock's all-jobs-share-one-lock behavior rather than over-engineering it). Records one job_runs row per invocation (started/finished/status/exit_code/summary/log_path/triggered_by). If the lock is already held, records status='skipped_lock' immediately instead of silently dropping the run. Found and fixed a real concurrency bug via testing: the obvious asyncio.wait_for(lock.acquire(), timeout=0) idiom for a non-blocking try-acquire is broken in asyncio -- the wrapping Task's first iteration and the timeout-0 callback race with no guaranteed ordering, so it timed out on literally every call, including the very first uncontended one. Fixed using lock.locked() + acquire(), relying on acquire()'s fast path never suspending when uncontended. scheduler_service: AsyncIOScheduler with the default in-memory job store (NOT SQLAlchemyJobStore -- it pickles job functions to persist them, and the _lib()/_beet() factory closures here aren't picklable; MemoryJobStore avoids this since register_all_jobs() rebuilds the whole schedule from code + DB on every startup anyway). All 19 maintenance jobs ported 1:1 from /etc/cron.d/sldl-maintenance with their exact cron expressions; dedup deliberately has no --apply (dry-run only on schedule, per the false-negative-biased dedup preference). Playlist jobs are synced live from the playlists table (sync_playlist_jobs), so add/remove-in-the-UI takes effect with no redeploy -- wired into playlist_service's create/update/ delete. Maintenance job enable/disable persists to the scheduled_jobs table so a paused job stays paused across a restart despite the in-memory job store. trigger_now() supports "Run now" by invoking a job's function immediately with triggered_by='manual', bypassing its schedule. Wired into main.py's lifespan (start on boot, shutdown on exit). Verified via integration tests: 36 jobs register correctly (17 seeded playlists + 19 maintenance) with correct cron triggers; live playlist create/delete correctly adds/removes its scheduler job; maintenance enable/disable persists to the DB and takes effect live; trigger_now correctly bypasses the schedule; pipeline_runner correctly handles success/failure/timeout/concurrent-lock-contention with real subprocesses; and the full app boots with the scheduler running and shuts down cleanly. |
||
|
andrew
|
bd56c8153f |
Add FastAPI app skeleton: settings, db, models, security, auth
- settings.py: pydantic-settings config for the two mount points (MUSIC_DATA_DIR, ALEMBIC_CONFIG_DIR), OIDC client config, encryption key path - db.py: SQLite engine with WAL mode, plain-SQL schema migration runner (schema/*.sql, tracked by schema_version -- not the alembic migration tool) - models.py: SQLAlchemy ORM models matching schema/0001_init.sql - security/crypto.py: Fernet encrypt/decrypt for the secrets table - security/oidc.py + routers/auth.py: Pocket ID OIDC login/callback/logout - security/deps.py: require_auth dependency (redirect-to-login on no session) - main.py: app factory, lifespan (init_db + beets WAL enable), session middleware, minimal dashboard route Verified boots end-to-end via TestClient: unauthenticated GET / redirects to /auth/login, static files serve, and all 13 schema tables get created on first run. |