Scheduling LMS Data Refreshes with Cron and Python

A metrics dashboard is only as dependable as the job that refreshes it, and the job is only dependable if it is safe to run unattended. This guide builds a cron-safe refresh entrypoint: a single Python script that a scheduler invokes on a fixed cadence, that refuses to overlap with a previous run still in progress, that only pulls data since it last succeeded, and that leaves the metrics store in a consistent state whether it runs once or a hundred times. The serving-layer design it refreshes is covered in the parent Dashboard Automation for LMS Metrics reference; here we focus on making the refresh robust under a scheduler with no human watching.

Two failure modes dominate scheduled jobs, and both are addressed by design rather than by hope. Overlap — a slow run still executing when cron fires the next one — is prevented by a lock file. Non-idempotency — a retry double-counting or a missed window leaving a gap — is prevented by a last-run guard that makes the refresh window explicit and the write a delete-and-replace.

Prerequisites

Confirm each of these before scheduling — the script assumes all of them are in place.

Step-by-step implementation

1. Acquire an exclusive lock before doing any work. Open a lock file and take an OS-level exclusive lock; if it is already held, another run is still going, so exit cleanly rather than piling on. This makes overlap impossible even if a run runs long.

2. Read the last successful run marker. The refresh window starts where the previous success ended, so the job pulls only new data and never re-processes a closed period. A missing marker means a first run — fall back to a sensible bootstrap window.

3. Compute an explicit (since, until) window. Making the window an explicit argument — not “now minus a guess” — is what lets the same code run on cron, in a backfill, or in a manual re-run with identical, predictable behaviour.

4. Run the idempotent refresh. Materialize the store with delete-and-replace by partition so a retry after a crash converges to the same result instead of double-counting.

5. Advance the marker only on success. Write the new last-run marker as the final step, so a failure mid-refresh leaves the marker untouched and the next run safely re-processes the same window.

6. Release the lock and exit with a meaningful code. Exit 0 on success, non-zero on failure, so cron’s MAILTO or a wrapper can alert on a bad run. Always release the lock, even on error, via a context manager.

7. Schedule it with an absolute-path crontab line. Cron runs with a minimal environment, so invoke the interpreter and script by absolute path and redirect output to a log.

Complete runnable code block

python
#!/usr/bin/env python3
"""Idempotent LMS metrics refresh — safe to run unattended from cron."""
import fcntl
import logging
import sys
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from pathlib import Path

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    handlers=[logging.FileHandler("/var/lib/lms-refresh/refresh.log"),
              logging.StreamHandler()],
)
logger = logging.getLogger("lms_refresh")

STATE_DIR = Path("/var/lib/lms-refresh")
LOCK_FILE = STATE_DIR / "refresh.lock"
LAST_RUN_FILE = STATE_DIR / "last_success.txt"
BOOTSTRAP_DAYS = 7   # first-run window when no marker exists


@contextmanager
def single_instance(lock_path: Path):
    """Exclusive, non-blocking lock so two runs can never overlap."""
    lock_path.parent.mkdir(parents=True, exist_ok=True)
    handle = open(lock_path, "w")
    try:
        fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        logger.warning("Another refresh holds the lock — exiting without work.")
        handle.close()
        sys.exit(0)          # not an error: overlap avoidance is expected
    try:
        yield
    finally:
        fcntl.flock(handle, fcntl.LOCK_UN)
        handle.close()


def read_last_success() -> datetime:
    """Last-run guard: the window starts where the previous success ended."""
    if LAST_RUN_FILE.exists():
        return datetime.fromisoformat(LAST_RUN_FILE.read_text().strip())
    logger.info("No last-run marker; bootstrapping %d days.", BOOTSTRAP_DAYS)
    return datetime.now(timezone.utc) - timedelta(days=BOOTSTRAP_DAYS)


def write_last_success(until: datetime) -> None:
    """Advance the marker ONLY after a fully successful refresh."""
    LAST_RUN_FILE.write_text(until.isoformat())


def refresh_metrics(since: datetime, until: datetime) -> int:
    """Idempotent materialization for the explicit window. Returns rows written.

    Replace the body with the delete-and-replace materialization from the
    Dashboard Automation reference; it must be safe to run twice for one window.
    """
    logger.info("Refreshing metrics window %s → %s", since.date(), until.date())
    # ... pull incremental data, aggregate, suppress small cells, write partition
    rows_written = 0
    return rows_written


def main() -> int:
    with single_instance(LOCK_FILE):
        until = datetime.now(timezone.utc)
        since = read_last_success()
        if since >= until:
            logger.info("Window already up to date; nothing to do.")
            return 0
        try:
            rows = refresh_metrics(since, until)
        except Exception:
            logger.exception("Refresh failed; marker NOT advanced, window re-runs.")
            return 1
        write_last_success(until)     # only on success
        logger.info("Refresh OK — %d rows; marker advanced to %s", rows, until.isoformat())
        return 0


if __name__ == "__main__":
    sys.exit(main())

Schedule it with an absolute-path crontab entry — cron’s environment is minimal, so nothing may rely on your interactive PATH. This runs the refresh every weekday at 04:15 and appends both streams to a log:

cron
# m  h  dom mon dow   command
  15 4  *   *   1-5   /opt/lms/.venv/bin/python /opt/lms/refresh.py >> /var/lib/lms-refresh/cron.log 2>&1

Verification and output validation

Confirm the entrypoint is genuinely cron-safe before trusting it to a scheduler:

  • Overlap is impossible. Start the script, then start a second copy while the first runs; the second must log “Another refresh holds the lock” and exit 0 without touching the store.
  • Idempotent on retry. Run the job twice for the same window (temporarily pin since/until); the metrics store row counts must be identical after both runs — delete-and-replace, never append.
  • Marker advances only on success. Force refresh_metrics to raise; confirm last_success.txt is unchanged and the next run re-processes the same window.
  • Bootstrap works. Delete last_success.txt and run; the log must show the bootstrap window and the run must complete without a FileNotFoundError.
  • Exit codes are meaningful. echo $? after a successful run is 0; after a forced failure it is 1, so cron MAILTO or a wrapper can alert.
  • Runs under cron’s environment. Test with env -i /opt/lms/.venv/bin/python /opt/lms/refresh.py to mimic cron’s bare environment and catch any hidden PATH or virtualenv assumption.

Troubleshooting

  • The job never runs, but the crontab looks right. Cron could not find the interpreter, because the line used python instead of an absolute path. Always invoke the venv interpreter by full path, as in the crontab example.
  • Two runs clobber each other’s output. The lock file path is on a filesystem where flock is unreliable (some network mounts), or each run opens a different lock path. Keep the lock on local disk and use one fixed absolute path.
  • The window keeps re-processing the same period. The marker write is failing (permissions) or is placed before the refresh. write_last_success must run after a successful refresh and the state dir must be writable by the cron user.
  • A gap appears in the metrics after a failed night. The marker advanced despite a partial refresh. Confirm the except branch returns non-zero without calling write_last_success, so the window is retried intact.
  • Logs are empty even though the job ran. Cron discarded stdout because the crontab line omitted the >> ... 2>&1 redirect, and no FileHandler was configured. Add both.
  • Duplicate rows accumulate over weeks. The refresh appends instead of replacing its partition. Swap the write for the delete-and-replace materialization from Dashboard Automation for LMS Metrics; the schedule cannot fix a non-idempotent write.

Part of: Dashboard Automation for LMS Metrics