c87ddc26492652628acb6558e612dfa02f94c509
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 andrew
|
c87ddc2649 |
Dashboard detail, library filtering/pagination, simplified playlist times,
job descriptions Dashboard: format breakdown (FLAC/MP3/etc. counts), approximate library size (bitrate*length/8, same approximation `beet stats` itself uses -- verified to match its "100.5 GiB" output exactly against the real library), and a credential auth-state summary per scope (configured y/n plus, for Bandcamp, cookie expiry parsed locally from the stored cookie jar -- deliberately not a live network probe like pipeline-status.sh's Qobuz check, since this renders on every dashboard load). Library: was one unfiltered page dumping all 4072 tracks. Added search (artist/title/album), playlist and format filter dropdowns, and real SQL-level pagination (LIMIT/OFFSET, not fetch-everything-then-slice). beets_service gained count_items()/distinct_formats() to support this. Playlists: cron_expr is still the stored/scheduled representation, but the UI now shows and edits a plain daily time picker instead of raw cron syntax -- every playlist schedule today is a simple daily HH:MM anyway. playlist_service.cron_to_time()/time_to_cron() convert at the router boundary; verified round-trip against all real playlist cron values. Jobs: each maintenance job now shows a short one-line description of what it actually does (MAINTENANCE_JOB_DESCRIPTIONS), and "next run" is formatted consistently with playlists' time style (HH:MM, with a day qualifier for non-today runs -- maintenance jobs can be weekly/monthly, unlike playlists' plain daily schedule). Verified end-to-end against the real production data (4072 tracks, 100.5 GiB, real credentials): stats/format-breakdown/search/pagination all correct, all 7 credential scopes report configured with Bandcamp's real cookie expiry (325 days), and a full authenticated page-render sweep (dashboard, library with every filter combination, playlist detail, jobs) all returned 200 with no template errors. |
||
|
andrew
|
a1f5bbb121 |
Fix regen.sh PATH bug; add legacy credential import script
playlist_service.regenerate_confs() was passing a bare PATH=/usr/bin:/bin to the regen.sh subprocess, but python3 (which regen.sh shells out to for parsing playlists.json) lives at /opt/venv/bin/python3 in this image. Every call silently generated zero .conf files -- no error surfaced, since the failure was inside a `python3 -c ... | while read` pipeline whose input was just empty. Found this deploying to the real host: seed_legacy() reported "17 playlists" successfully, but zero .conf files existed. Fixed to match pipeline_runner's already-correct PATH. scripts/import_legacy_credentials.py: the Stage 1/2 one-time migration script -- reads spotify/soulseek creds from a sample playlist .conf, Navidrome's password (extracted from navidrome-scan.sh, never lived in a file), and bandcamp/azuracast/qobuz/telegram from their legacy scattered locations, then populates credential_service. Read-only against the legacy files, idempotent. Run as a one-off `docker run --entrypoint python` with the legacy paths bind-mounted read-only (documented in the script's own docstring) -- entrypoint.sh always execs uvicorn regardless of CMD, so --entrypoint must override it for one-off invocations like this. Verified against the real production host: seeded all 17 legacy playlists with their historical cron schedules, confirmed all 17 .conf files render correctly after the PATH fix, ran the credential import for all 7 scopes, and byte-diffed the rendered spotify/soulseek values in a sample .conf against the original (identical, verified via diff -q without displaying the actual secret values in any tool output). Also had to chown 4 root:root mode-600 legacy credential files (bandcampconfig/cookies.txt, azuracastconfig/api_key, qobuzconfig/token, telegramconfig/notify.env) to 1000:1000 so the container could read them for the import -- root retains read access regardless, so this doesn't affect the still-running host-cron pipeline. |
||
|
andrew
|
2075d6cf66 |
Add dedup_review_service and genre_review_service
dedup-library.sh: additive --json (one NDJSON line per candidate deletion to stdout, alongside the unchanged human log) and --only-paths FILE (in --apply mode, only actually delete entries whose path is in FILE; without it, --apply deletes everything as before -- existing direct callers are unaffected). Without --only-paths the safety re-verification is free: --apply --only-paths re-runs all 4 passes from scratch on every invocation, so if a group's ranking changed since a scan (e.g. the old keep_path is gone), the fresh pass assigns the previously-"delete" path the KEEP role instead and the only-paths allowlist naming it is simply never consulted -- no duplicate ranking logic needed in the review service. spotify-genre.py: additive --json emitting one JSON line per genre change (dry-run or --apply) for genre_review_service to persist. pipeline_runner.run_job_capture(): like run_job() but captures stdout as text (still under the same shared lock, still writes a job_runs row) for callers that need to parse structured output rather than just log it. dedup_review_service.scan() persists dry-run candidates into dedup_runs/dedup_candidates. confirm_and_apply() re-checks confirmed candidates still exist before invoking --apply --only-paths, so nothing is ever deleted without an explicit confirm -- matches the false-negative- biased dedup preference. scheduler_service's maintenance:dedup job now goes through this (still dry-run only, every day). genre_review_service.run() wraps spotify-genre.py for both dry-run preview and the real scheduled --apply run, persisting every run's diff into genre_runs/genre_candidates either way -- genre writes keep their current auto-apply behavior (low-risk, reversible, GENRE_LOCK-protected) but are now reviewable after the fact. lock_artist_genre() gives a one-click revert path when a run gets something wrong. Added minimal routers+templates for /dedup (scan, review, confirm-and- delete) and /genres (preview, review, lock-old-genre). Verified end-to-end against REAL duplicate files (not mocked): built an actual FLAC+MP3 duplicate pair in a real beets library, ran dedup-library.sh --json and confirmed correct JSON output, verified --apply --only-paths with an empty confirm list deletes nothing and with the real confirmed path deletes exactly that file (DB + disk) while preserving the FLAC, and ran the full dedup_review_service scan->confirm->apply flow through the same fixture. genre_review_service and spotify-genre.py --json verified against mocked/direct output (spotify-genre.py's own artist-genre lookup needs a live Spotify API call, out of reach in this sandbox). Confirmed the full app boots with all five routers registered. |
||
|
andrew
|
270c9c0ed1 |
Add manual fix / manual import services and routers
library_edit.update_track_fields: single-track tag edits via `beet modify -y -m` (subprocess with an argument list, not beets.library.Library in-process -- an arg-list subprocess has no shell-injection surface to avoid in the first place, and correctly reuses beets' own configured path-format/move logic rather than reimplementing it against a hand-built Library object with guessed config). Every changed field gets a manual_fix_audit row with old/new values; triggers a Navidrome rescan afterward if anything changed. Uses "genres" (plural) as the field name, matching this deployment's actual beets column and every existing script's convention, not the generic beets docs' singular "genre". genre_fix.set_artist_genre and retag.retag_from_url wrap the existing fix-genre.sh and fix-track-metadata.py as callable services rather than reimplementing their per-format mutagen tag-writing logic. manual_import: ties pipeline_runner.run_import_track to the manual_imports table, mirrors import-track.sh's own argument-count-based disambiguation (0/1/2 args), plus import-me/ listing and upload-with-path-traversal-guard. routers/library.py + routers/import_.py: browsable library list, a track detail/edit page (tag edit, artist-wide genre override, retag-from-URL forms), and the import page (upload + trigger). Minimal templates for now, Task 9 covers full UI polish. Verified update_track_fields end-to-end against a REAL beets library (not mocked): generated a tagged FLAC with ffmpeg/metaflac, imported it via a real `beet import`, edited artist+title through the service, and confirmed the file was physically moved to the new path-template location, tags were rewritten on disk, the DB updated, and both changes landed correctly in manual_fix_audit. Also verified the full app boots with the new routers registered and all four new routes correctly redirect to login when unauthenticated. |
||
|
andrew
|
4e0cfb8463 |
Add scheduler_service and pipeline_runner
pipeline_runner: subprocess execution wrapper replacing flock -n with a single asyncio.Lock (deliberately one global lock, not per-resource -- matches the old flock's all-jobs-share-one-lock behavior rather than over-engineering it). Records one job_runs row per invocation (started/finished/status/exit_code/summary/log_path/triggered_by). If the lock is already held, records status='skipped_lock' immediately instead of silently dropping the run. Found and fixed a real concurrency bug via testing: the obvious asyncio.wait_for(lock.acquire(), timeout=0) idiom for a non-blocking try-acquire is broken in asyncio -- the wrapping Task's first iteration and the timeout-0 callback race with no guaranteed ordering, so it timed out on literally every call, including the very first uncontended one. Fixed using lock.locked() + acquire(), relying on acquire()'s fast path never suspending when uncontended. scheduler_service: AsyncIOScheduler with the default in-memory job store (NOT SQLAlchemyJobStore -- it pickles job functions to persist them, and the _lib()/_beet() factory closures here aren't picklable; MemoryJobStore avoids this since register_all_jobs() rebuilds the whole schedule from code + DB on every startup anyway). All 19 maintenance jobs ported 1:1 from /etc/cron.d/sldl-maintenance with their exact cron expressions; dedup deliberately has no --apply (dry-run only on schedule, per the false-negative-biased dedup preference). Playlist jobs are synced live from the playlists table (sync_playlist_jobs), so add/remove-in-the-UI takes effect with no redeploy -- wired into playlist_service's create/update/ delete. Maintenance job enable/disable persists to the scheduled_jobs table so a paused job stays paused across a restart despite the in-memory job store. trigger_now() supports "Run now" by invoking a job's function immediately with triggered_by='manual', bypassing its schedule. Wired into main.py's lifespan (start on boot, shutdown on exit). Verified via integration tests: 36 jobs register correctly (17 seeded playlists + 19 maintenance) with correct cron triggers; live playlist create/delete correctly adds/removes its scheduler job; maintenance enable/disable persists to the DB and takes effect live; trigger_now correctly bypasses the schedule; pipeline_runner correctly handles success/failure/timeout/concurrent-lock-contention with real subprocesses; and the full app boots with the scheduler running and shuts down cleanly. |
||
|
andrew
|
fe98afaaf4 |
Add beets_service, spotify_client, status_service
beets_service: read-only SQLite queries against the beets DB (mode=ro, matches the WAL setup in db.py's enable_beets_db_wal). Decodes the `path` column, which beets stores as a BLOB not TEXT. stats() gives dashboard/ migration-verification parity with `beet stats`. spotify_client: client-credentials OAuth (token cached in-process), paginated playlist-track fetch. Verified with mocked httpx responses: pagination across pages, null-track filtering (removed/local tracks), ISRC extraction, and token reuse across calls. status_service: reconciles a playlist's live Spotify tracklist against beets (IN_LIBRARY, by ISRC or normalized artist+title), sldl's per-playlist _sldl.m3u8 index (DOWNLOADED_PENDING_IMPORT), and the tag-guard quarantine dir (QUARANTINED, matched by filename since quarantined files have no tags by definition) -- everything else is NOT_YET_ATTEMPTED. ATTEMPTED_NO_MATCH is deliberately not implemented (the only signal is grepping sldl's per-run text logs, which the migration plan already flags as unreliable rather than authoritative). Verified end-to-end against a fake beets DB + sldl index + quarantine dir: all four statuses reconcile correctly, including both the ISRC-match and normalized-artist+title-fallback paths. |
||
|
andrew
|
5eaae8e813 |
Add playlist_service and credential_service
playlist_service: CRUD for the playlists table, playlists.json export,
regen.sh invocation, and a one-time seed_legacy() importing the 17-entry
legacy array. Creating/updating a playlist automatically re-renders confs
and re-patches credentials into any newly-generated .conf file.
credential_service: encrypted credential storage (via security/crypto.py)
and per-scope rendering to the exact files the pipeline scripts read --
every <playlist>.conf (spotify/soulseek), navidrome/admin.env,
bandcamp/config.env+cookies.txt, azuracast/api_key, qobuz/{token,app_id,region},
telegram/notify.env. set_credentials() batches multi-field saves so a render
never sees a half-populated scope.
Also fixes a real bug found via integration testing: regen.sh's embedded
python3 -c snippet used backslash-escaped quotes inside a single-quoted
bash string, which is invalid Python syntax (backslash passed through
literally) -- switched to double-quoted bash wrapper + single-quoted Python
strings.
Verified end-to-end: seeded 17 playlists, regenerated all 17 .conf files,
saved spotify/soulseek/navidrome credentials and confirmed correct
patching into rendered files (including bash-sourcing admin.env with
special characters in the password), and confirmed delete/create both
correctly add/remove .conf files with credentials pre-patched on create.
|
||
|
andrew
|
bd56c8153f |
Add FastAPI app skeleton: settings, db, models, security, auth
- settings.py: pydantic-settings config for the two mount points (MUSIC_DATA_DIR, ALEMBIC_CONFIG_DIR), OIDC client config, encryption key path - db.py: SQLite engine with WAL mode, plain-SQL schema migration runner (schema/*.sql, tracked by schema_version -- not the alembic migration tool) - models.py: SQLAlchemy ORM models matching schema/0001_init.sql - security/crypto.py: Fernet encrypt/decrypt for the secrets table - security/oidc.py + routers/auth.py: Pocket ID OIDC login/callback/logout - security/deps.py: require_auth dependency (redirect-to-login on no session) - main.py: app factory, lifespan (init_db + beets WAL enable), session middleware, minimal dashboard route Verified boots end-to-end via TestClient: unauthenticated GET / redirects to /auth/login, static files serve, and all 13 schema tables get created on first run. |