CSV vs API Ingestion for Moodle Attendance

Every institution that reports on attendance eventually faces the same fork: pull the Moodle Attendance activity’s data through the built-in CSV export, or through the mod_attendance web-service functions. Both describe the identical academic fact — this student was Present, Absent, Late, or Excused in this session — but they differ sharply in latency, completeness, operational cost, and how much personally identifiable information crosses the wire. Choosing wrongly means either a brittle nightly file drop that silently misses a session, or an over-engineered polling job hammering a token-limited endpoint for data that changes twice a week. This guide compares the two ingestion paths on the axes that actually decide the question, gives a concrete decision rule, and closes with one runnable script that reads both paths into the same canonical attendance shape — so the choice is an ingestion detail, not a schema commitment. It builds on the LMS CSV Export Format Standards contract and the state model in attendance state normalization rules.

Prerequisites

Confirm each item before running the procedure — the script demonstrates both paths and assumes all are in place.

• Python 3.10+ (the code uses str | None unions and modern type hints).
• pandas==2.2.* and requests==2.32.* in the virtualenv.
• For the CSV path: an Attendance activity export saved locally. Export it from Attendance → Export, choosing a CSV format with UTF-8 encoding; the file carries one row per student-per-session with a Status acronym column (P/A/L/E or an institution-defined set).
• For the API path: a Moodle web-service token with the mod_attendance_get_sessions function enabled on its external service, exported as MOODLE_WS_TOKEN, plus the numeric attendance instance id. The mod_attendance plugin must be installed and its external functions turned on.
• A TOKEN_SALT environment variable holding a per-institution secret — the salt for the student pseudonym must live inside the compliance boundary, never in code.
• The canonical target: one row per (student_token, session_id) carrying a normalized status_code, the session_date, and a source column recording which path produced the row.

The comparison, axis by axis

Two consequences follow. First, the CSV path’s zero rate limit and single-file completeness make it the low-risk default for scheduled, whole-term loads — the same reasoning behind standardizing LMS CSV headers for data lakes, where a stable header map neutralizes the schema fragility. Second, the API path’s session-level freshness and field-scoped PII minimization make it the right choice whenever attendance must feed a same-day intervention — and its pacing is governed by the Python requests for LMS APIs patterns for tokens and retries.

Decision rule

Pick the CSV export when all of these hold: the consumer tolerates data that is at most one scheduled cycle old, the volume fits a single file comfortably, and you can pin the export locale (delimiter, encoding, date format) so the header map stays stable. Pick the web service when any of these hold: an intervention or dashboard needs same-day session data, you want incremental timemodified pulls rather than full re-exports, or FERPA data-minimization requires pulling attendance fields without the identity columns a CSV export bundles alongside them. When both would work, default to CSV for its lower operational surface — a file read cannot be throttled, expire, or leak a field you did not ask for.

Step-by-step implementation

1. Define the canonical shape first, then make each path fill it. The whole point is that ingestion is swappable. Fix (student_token, session_id, session_date, status_code, source) as the contract before writing either reader.

2. Normalize status the same way regardless of source. Both paths deliver a raw acronym or code; map it through one STATUS_MAP to the canonical P/A/L/E set so a CSV Late and an API L collapse to the same value, exactly as attendance state normalization rules prescribe.

3. Tokenize the student key at the boundary in both readers. The CSV carries an ID number column and the API carries a studentid; hash whichever you get with the same salted SHA-256 so rows from the two paths join on an identical student_token.

4. Stamp a source column. Recording csv vs api on every row lets an audit trace lineage and lets you reconcile the two paths against each other during a cutover.

5. Read defensively. The CSV reader forces dtype=str and an explicit delimiter/encoding; the API reader gates on the exception envelope and reads dates as UTC epochs. Neither trusts positional access.

Complete runnable code block

import os
import csv
import hashlib
import logging
from datetime import datetime, timezone
import pandas as pd
import requests

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("moodle_attendance_ingest")

WS = "https://moodle.example.edu/webservice/rest/server.php"
TOKEN = os.environ.get("MOODLE_WS_TOKEN", "")
SALT = os.environ["TOKEN_SALT"].encode()  # kept inside the compliance boundary

# One canonical status set for both paths.
STATUS_MAP = {
    "P": "P", "PRESENT": "P",
    "A": "A", "ABSENT": "A",
    "L": "L", "LATE": "L",
    "E": "E", "EXCUSED": "E",
}
CANONICAL_COLS = ["student_token", "session_id", "session_date", "status_code", "source"]


def tokenize(raw_id: str) -> str:
    return "stu_" + hashlib.sha256(SALT + raw_id.strip().encode()).hexdigest()[:24]


def _norm_status(raw: str | None) -> str | None:
    if not raw:
        return None
    return STATUS_MAP.get(raw.strip().upper())


def read_csv_path(path: str) -> pd.DataFrame:
    """Path A: parse a Moodle Attendance activity CSV export."""
    rows: list[dict] = []
    with open(path, newline="", encoding="utf-8-sig") as fh:
        reader = csv.DictReader(fh, delimiter=",")   # pin the locale delimiter
        for r in reader:
            raw_id = (r.get("ID number") or r.get("Student ID") or "").strip()
            if not raw_id:                            # quarantine keyless rows
                continue
            # Moodle exports dates like "2026-03-04" (site-locale dependent).
            date_raw = (r.get("Date") or "").strip()
            rows.append({
                "student_token": tokenize(raw_id),
                "session_id": (r.get("Session ID") or r.get("Description") or "").strip(),
                "session_date": pd.to_datetime(date_raw, errors="coerce").date()
                                if date_raw else None,
                "status_code": _norm_status(r.get("Status acronym") or r.get("Status")),
                "source": "csv",
            })
    return _finalize(rows)


def _ws_call(wsfunction: str, **params) -> dict | list:
    payload = {"wstoken": TOKEN, "wsfunction": wsfunction,
               "moodlewsrestformat": "json", **params}
    resp = requests.post(WS, data=payload, timeout=30)
    resp.raise_for_status()
    data = resp.json()
    if isinstance(data, dict) and data.get("exception"):
        raise RuntimeError(f"{wsfunction}: {data.get('errorcode')} {data.get('message')}")
    return data


def read_api_path(attendance_id: int) -> pd.DataFrame:
    """Path B: pull the same data via mod_attendance web service."""
    sessions = _ws_call("mod_attendance_get_sessions", attendanceid=attendance_id)
    rows: list[dict] = []
    for s in sessions:
        epoch = s.get("sessdate")                     # UTC Unix epoch
        session_date = (datetime.fromtimestamp(epoch, tz=timezone.utc).date()
                        if epoch else None)
        for log in s.get("attendance_log", s.get("statuses", [])):
            raw_id = str(log.get("studentid") or log.get("userid") or "").strip()
            if not raw_id:
                continue
            rows.append({
                "student_token": tokenize(raw_id),
                "session_id": str(s.get("id")),
                "session_date": session_date,
                "status_code": _norm_status(log.get("acronym") or log.get("status")),
                "source": "api",
            })
    return _finalize(rows)


def _finalize(rows: list[dict]) -> pd.DataFrame:
    if not rows:
        return pd.DataFrame(columns=CANONICAL_COLS)
    df = pd.DataFrame(rows)[CANONICAL_COLS]
    return df.astype({"student_token": "string", "session_id": "string",
                      "status_code": "string", "source": "string"})


if __name__ == "__main__":
    csv_df = read_csv_path(os.environ["ATTENDANCE_CSV"])
    logger.info("CSV path rows: %d", len(csv_df))
    if TOKEN:
        api_df = read_api_path(int(os.environ["ATTENDANCE_ID"]))
        logger.info("API path rows: %d", len(api_df))
        combined = pd.concat([csv_df, api_df], ignore_index=True)
        # Prefer the API row when both paths cover the same grain.
        combined = combined.sort_values("source").drop_duplicates(
            subset=["student_token", "session_id"], keep="first")
        print(combined.head())
    else:
        print(csv_df.head())

Verification and output validation

Confirm both paths yield the same canonical shape before wiring either into production:

• Identical schema. assert list(read_csv_path(p).columns) == CANONICAL_COLS and the same for read_api_path — the two readers must be drop-in interchangeable.
• Grain is unique per source. Within one source, assert len(df) == df[["student_token", "session_id"]].drop_duplicates().shape[0].
• Status normalized, never raw. assert df["status_code"].dropna().isin(["P", "A", "L", "E"]).all(); a stray Present means the STATUS_MAP missed a locale label.
• No raw identifiers leaked. assert df["student_token"].str.startswith("stu_").all() and no ID number/studentid column survives.
• Cross-path agreement. For overlapping grain, assert (csv_df.merge(api_df, on=["student_token", "session_id"], suffixes=("_c", "_a")).eval("status_code_c == status_code_a")).all() — a mismatch flags a normalization or freshness gap, not a schema bug.

Troubleshooting

• CSV status_code is all <NA>. The export used institution-specific acronyms (Pr, Ab) the STATUS_MAP does not know. Extend the map with your site’s acronyms; do not silently coerce unknown labels to A.
• session_date is NaT from the CSV. The site locale exported dates as 04/03/2026 (day-first), which pd.to_datetime mis-parses. Pin format= or dayfirst=True, and pin the export locale so it stays stable, per standardizing LMS CSV headers for data lakes.
• invalidtoken / accessexception on the API path. The token lacks mod_attendance_get_sessions on its external service, or the plugin’s external functions are disabled. Enable the function and confirm the token’s role can view the activity.
• API session_date is a day off. sessdate is a UTC epoch but the CSV shows the site-timezone date, so late-evening sessions cross midnight. Normalize both to UTC (or both to the institutional timezone) before comparing — never mutate the stored epoch.
• Duplicate rows after pd.concat. Both paths covered the same session. The dedup on (student_token, session_id) resolves it; keep the source you trust for freshness (the script keeps api).
• API path returns fewer sessions than the CSV. The token’s role cannot see hidden or future sessions the export included, or the session list mutated mid-walk. Snapshot the session ids first and reconcile against the CSV source during a cutover.

Related

• LMS CSV Export Format Standards — the canonical export schema and delimiter/encoding rules the CSV path must respect.
• Standardizing LMS CSV Headers for Data Lakes — the header-mapping layer that neutralizes the CSV path’s schema fragility.
• Attendance State Normalization Rules — the canonical status model both readers map into.
• Python Requests for LMS APIs — the token, retry, and pacing patterns the web-service path depends on.

Part of: LMS CSV Export Format Standards