pipeline_runner: subprocess execution wrapper replacing flock -n with a
single asyncio.Lock (deliberately one global lock, not per-resource --
matches the old flock's all-jobs-share-one-lock behavior rather than
over-engineering it). Records one job_runs row per invocation
(started/finished/status/exit_code/summary/log_path/triggered_by).
If the lock is already held, records status='skipped_lock' immediately
instead of silently dropping the run.
Found and fixed a real concurrency bug via testing: the obvious
asyncio.wait_for(lock.acquire(), timeout=0) idiom for a non-blocking
try-acquire is broken in asyncio -- the wrapping Task's first iteration and
the timeout-0 callback race with no guaranteed ordering, so it timed out on
literally every call, including the very first uncontended one. Fixed using
lock.locked() + acquire(), relying on acquire()'s fast path never
suspending when uncontended.
scheduler_service: AsyncIOScheduler with the default in-memory job store
(NOT SQLAlchemyJobStore -- it pickles job functions to persist them, and
the _lib()/_beet() factory closures here aren't picklable; MemoryJobStore
avoids this since register_all_jobs() rebuilds the whole schedule from code
+ DB on every startup anyway). All 19 maintenance jobs ported 1:1 from
/etc/cron.d/sldl-maintenance with their exact cron expressions; dedup
deliberately has no --apply (dry-run only on schedule, per the
false-negative-biased dedup preference). Playlist jobs are synced live from
the playlists table (sync_playlist_jobs), so add/remove-in-the-UI takes
effect with no redeploy -- wired into playlist_service's create/update/
delete. Maintenance job enable/disable persists to the scheduled_jobs table
so a paused job stays paused across a restart despite the in-memory job
store. trigger_now() supports "Run now" by invoking a job's function
immediately with triggered_by='manual', bypassing its schedule.
Wired into main.py's lifespan (start on boot, shutdown on exit).
Verified via integration tests: 36 jobs register correctly (17 seeded
playlists + 19 maintenance) with correct cron triggers; live playlist
create/delete correctly adds/removes its scheduler job; maintenance
enable/disable persists to the DB and takes effect live; trigger_now
correctly bypasses the schedule; pipeline_runner correctly handles
success/failure/timeout/concurrent-lock-contention with real subprocesses;
and the full app boots with the scheduler running and shuts down cleanly.
andrew
4e0cfb8463