Early-Warning At-Risk Scoring for LMS Engagement Pipelines

An early-warning system turns the exhaust of a learning platform — page views, submission timestamps, gradebook rows, attendance marks — into a single actionable signal: which learners are drifting toward failure while there is still time to intervene? For EdTech engineers and institutional data analysts, the engineering problem is not the model. Logistic regression on a handful of features has been the workhorse of student-success analytics for over a decade, and it still outperforms most black-box alternatives on the metrics that matter here. The hard part is everything around the model: assembling honest features from three misaligned source systems, calibrating a probability into a defensible threshold, and doing all of it inside a FERPA boundary where a risk label is one of the most sensitive fields an institution can compute.

This page defines the feature and entity model that a risk score is built from, the assembly logic that joins the normalized engagement metric to the gradebook and attendance layers, the scoring model itself with its calibration and threshold mechanics, the compliance constraints that govern who may ever see a risk label, a runnable reference implementation, and the failure modes — label leakage, survivorship bias, feedback loops — that quietly invalidate an early-warning system in production. Its child page walks through building an at-risk student flag in Python end to end.

The risk feature entity model

A risk score is a function of a learner-term feature vector: one row per learner per course-term, holding a small set of engineered features that each proxy a different dimension of academic trajectory. The grain matters — scoring at learner-term (not learner-course-week, not learner globally) is what lets a threshold mean the same thing across a cohort and lets an intervention attach to a specific enrollment.

Risk feature assembly from three normalized source facts Three source facts on the left — the engagement index, grade trajectory, and attendance flags — each contribute two or three engineered features. Their columns converge into a central learner-term feature vector keyed by a tokenized learner id. That vector feeds a calibrated logistic model on the right, which outputs a risk probability and then a boolean at-risk flag once a threshold is applied. Engagement index engagement_index (0–1) login_recency_days activity_trend Grade trajectory running_grade (0–1) grade_slope missing_submissions Attendance flags absence_rate consecutive_absences late_submission_rate Feature vector grain · learner × term learner_token PK † 8 standardized features Logistic model σ(w·x + b) → risk prob p → flag if p ≥ τ † learner_token is the salted-hash join key — no raw student id ever enters the feature store. Assemble features from normalized facts, not raw payloads.

Five feature families carry almost all of the predictive weight, and each maps to a source layer already normalized elsewhere on this site:

  • Engagement index. A single engagement_index in [0, 1] from the engagement metric normalization layer, plus login_recency_days and a short-window activity_trend. This is deliberately a normalized metric, not a raw page-view count — raw counts are incomparable across a lecture course and a lab, and folding them straight into a model bakes in course-size bias.
  • Grade trajectory. The running_grade to date (a normalized weighted mean), a grade_slope over the last few graded items, and a count of missing_submissions. Reconstructing the running grade correctly is non-trivial and is handled by the weighted grade calculation engines — the risk layer consumes their output, it does not re-derive it.
  • Attendance. An absence_rate, the longest run of consecutive_absences, and a late_submission_rate. These are drawn from normalized attendance states; the sharper anomaly signals live in attendance anomaly detection, and a risk model does well to consume its flags rather than recompute z-scores itself.
  • Submission timeliness. How consistently a learner submits before the due datetime, expressed as a rate rather than a raw lateness in hours, so it is comparable across courses with different deadline conventions.
  • Trajectory, not level. Wherever possible, a feature is a change — a slope, a recency, a trend — because early warning is about deviation from a learner’s own baseline, and level features alone over-flag learners who were always low-engagement but stable.

Every feature is keyed by a tokenized learner_token, never a raw student identifier. The feature store sits inside the analytical zone, downstream of the tokenization boundary, so the join key that stitches engagement to grades to attendance is the same salted hash used everywhere else in the pipeline. This is the discipline documented in FERPA-compliant PII handling, and it is not optional for a risk feature store specifically, because a risk table is a magnet for re-identification attempts.

Assembling features from normalized facts

Feature assembly is a left join from an enrollment spine outward, and the order is load-bearing. Start from the roster — every active enrollment for the term — so that a learner with zero engagement rows still appears with an explicit “no activity” feature rather than silently vanishing. This single decision prevents the most damaging quiet bug in early-warning systems: the learner who stopped logging in entirely, who is by definition the highest-risk case, disappearing from the scored set because an inner join dropped them.

From the spine, each source contributes a pre-aggregated fact at the learner-term grain. The engagement layer already emits one engagement_index per learner-term; the gradebook layer emits a running grade and missing-count; the attendance layer emits rates and run-lengths. Assembly is therefore a sequence of left joins followed by imputation of the structural nulls that a left join creates — and structural nulls carry meaning. A missing engagement row means “never engaged,” which you encode as engagement_index = 0.0 and login_recency_days = term_length, not as a median-imputed value. Median imputation here would smuggle an average learner’s behaviour onto exactly the learners you most need to flag.

Two normalization rules keep features honest across courses. First, standardize continuous features within cohort — subtract the cohort mean and divide by the cohort standard deviation — so a coefficient learned on one term transfers to another. Second, cap and floor before standardizing, because a single learner with 4,000 page views in a week (a scripted download, usually) otherwise dominates the scale. The output of assembly is a dense, standardized, tokenized feature matrix with no nulls remaining — every absence has been converted to an explicit, meaningful value.

The scoring model

The model is a weighted logistic regression. Given a feature vector x\mathbf{x} for a learner-term, a learned weight vector w\mathbf{w}, and bias bb, the raw score is the linear combination z=w⋅x+bz = \mathbf{w}\cdot\mathbf{x} + b, and the risk probability is the logistic (sigmoid) transform of that score:

p(at\_risk∣x)=σ(z)=11+e−(w⋅x+b)p(\text{at\_risk} \mid \mathbf{x}) = \sigma(z) = \frac{1}{1 + e^{-(\mathbf{w}\cdot\mathbf{x} + b)}}

The logistic function is the right default for three reasons that are specific to this problem, not generic model-selection folklore. It outputs a calibrated-able probability in [0,1][0, 1] rather than an unbounded score, which is what a threshold and an alert budget need. Its coefficients are directly interpretable — a positive wiw_i on consecutive_absences means exactly “more consecutive absences raises risk, holding everything else fixed,” which is the kind of statement an academic-affairs committee can review and sign off on. And it is monotonic and additive, so a counselor asking “why was this learner flagged?” gets an honest, decomposable answer: the per-feature contributions wixiw_i x_i sum to the score, and the largest positive terms are the explanation.

You can fit w\mathbf{w} by maximum likelihood on a labeled history (learners who did / did not fail the course), but even a hand-set weight vector reviewed by faculty is a legitimate and often more defensible starting point, because it sidesteps the label-leakage traps discussed below. Whether learned or hand-set, the raw probabilities almost always need calibration before a threshold is applied.

Calibration and threshold selection

A model can rank learners perfectly and still be badly calibrated — if it says “0.9” for a group that actually fails 60% of the time, every downstream alert budget computed from that probability is wrong. Calibration fixes the mapping from score to true frequency, typically with Platt scaling (a second logistic fit on the scores) or isotonic regression on a held-out term. After calibration, a predicted 0.30 means roughly 30% of such learners historically struggled, and only then does a probability threshold mean anything.

Threshold selection is where the model meets institutional reality. The threshold τ\tau converts a probability into a flag — flag = p ≥ τ — and it is a policy lever, not a statistical one. Lowering τ\tau catches more genuinely at-risk learners (higher recall) at the cost of flagging more who were fine (lower precision); raising it does the reverse. The right operating point depends on the cost of the two errors, which in this domain are wildly asymmetric and institution-specific: a missed at-risk learner (false negative) may mean a withdrawal that a timely nudge would have prevented, while a false positive costs a counselor’s fifteen minutes and, more insidiously, erodes trust in the system.

Precision-recall tradeoff and the alert-budget operating band A plot with recall on the x-axis from zero to one and rate on the y-axis. A precision curve slopes downward as recall rises, and a recall-versus-threshold relationship is annotated. A shaded vertical operating band in the middle marks the alert budget where precision stays high enough to prevent alert fatigue while recall is still useful. Lowering the threshold moves right into higher recall but lower precision; raising it moves left. Recall (share of at-risk learners caught) → Rate → alert-budget band precision recall ← raise τ lower τ → precision high enough to avoid alert fatigue 0 1.0

The practical way to set τ\tau is to invert the question: instead of choosing a probability, choose an alert budget — the number of flags a support team can act on meaningfully in a week — and pick the τ\tau that produces that many flags at the highest achievable precision. This grounds the threshold in the institution’s actual capacity to intervene, and it directly addresses alert fatigue: a system that flags a third of the cohort every week trains counselors to ignore it, which makes a perfectly accurate model operationally worthless. The operating band in the diagram is where precision is high enough that acting on every flag is worth a counselor’s time, and recall is high enough that the system earns its keep.

Compliance constraints on risk labels

A risk label is not an ordinary analytical column. Under FERPA, a computed at-risk score derived from a learner’s educational records is itself part of the educational record, and it is unusually sensitive because it is interpretive — it asserts something predictive about a person. Three constraints follow, and they shape the schema, not just the access policy.

Who may see a label is narrower than who may see the inputs. An instructor may legitimately see engagement and grades for their own section; it does not follow that they should see a system-generated “at-risk” flag, which can bias grading and interaction. Risk labels are gated behind role-based access control for LMS data, typically visible only to designated advising or success staff with a documented legitimate-educational-interest, and never exposed in a broad dashboard. The FERPA-compliant PII handling guide covers the audit-log schema that records every read of a risk label.

No proxies for protected classes. A model must not use — and must be tested to ensure it has not learned — features that proxy for race, disability, national origin, or other protected characteristics. Zip code, name-derived features, and country-of-origin are obvious exclusions; subtler ones (a specific course-taking pattern that correlates with a disability accommodation) require auditing the fitted coefficients and checking flag rates across groups. The safest posture is a small, reviewed feature set of behavioural signals — engagement, grades, attendance, timeliness — that have a defensible causal story toward academic outcome and no path through a protected attribute.

Labels are decision support, not decisions. The output is a prompt for a human to reach out, never an automated adverse action. This is both an ethical line and a practical one: it keeps the institution out of automated-decision territory and keeps a human in the loop who can catch the false positives the model will inevitably produce.

Reference implementation

The implementation below builds a risk score from an already-assembled feature DataFrame. It standardizes features within cohort, applies a calibrated logistic model, thresholds against an alert budget, and emits flags keyed by tokenized ids only. It assumes the upstream layers have delivered a clean, learner-term feature frame — the assembly and I/O belong in the child walkthrough.

python
import hashlib
import hmac
from dataclasses import dataclass, field

import numpy as np
import pandas as pd

_SALT = b"rotate-me-from-secrets-manager"

# Features whose HIGHER value RAISES risk carry positive weights; grade and
# engagement are protective, so their weights are negative. Reviewed by faculty.
WEIGHTS: dict[str, float] = {
    "engagement_index": -2.1,
    "login_recency_days": 0.8,
    "running_grade": -3.4,
    "grade_slope": -1.2,
    "missing_submissions": 0.9,
    "absence_rate": 1.6,
    "consecutive_absences": 0.7,
    "late_submission_rate": 1.1,
}
BIAS = -0.5


def tokenize(raw_id: str) -> str:
    """FERPA-safe, non-reversible learner token for the scored output."""
    digest = hmac.new(_SALT, raw_id.encode("utf-8"), hashlib.sha256).hexdigest()
    return f"stu_{digest[:16]}"


@dataclass
class RiskModel:
    weights: dict[str, float] = field(default_factory=lambda: dict(WEIGHTS))
    bias: float = BIAS
    platt_a: float = 1.0   # calibration slope, fit on a held-out term
    platt_b: float = 0.0   # calibration intercept

    def _standardize(self, df: pd.DataFrame) -> pd.DataFrame:
        cols = list(self.weights)
        x = df[cols].astype("float64")
        # Cap outliers, then standardize WITHIN this cohort so weights transfer.
        x = x.clip(lower=x.quantile(0.01), upper=x.quantile(0.99), axis=1)
        return (x - x.mean()) / x.std(ddof=0).replace(0.0, 1.0)

    def score(self, df: pd.DataFrame) -> pd.Series:
        x = self._standardize(df)
        z = x.mul(pd.Series(self.weights)).sum(axis=1) + self.bias
        z_cal = self.platt_a * z + self.platt_b          # calibration
        return 1.0 / (1.0 + np.exp(-z_cal))              # logistic → probability

    def flag(self, df: pd.DataFrame, alert_budget: int) -> pd.DataFrame:
        p = self.score(df)
        # Pick the threshold that yields exactly the alert budget, highest-p first.
        tau = p.sort_values(ascending=False).iloc[min(alert_budget, len(p)) - 1]
        out = pd.DataFrame({
            "learner_token": df["raw_learner_id"].map(tokenize),
            "risk_probability": p.round(4),
            "at_risk": p >= tau,
        })
        return out.sort_values("risk_probability", ascending=False).reset_index(drop=True)

The two design choices worth underscoring: standardization happens inside score, so the model is self-contained and a caller cannot forget it; and flag derives the threshold from an alert budget rather than a hard-coded probability, so the operating point tracks the support team’s real capacity. The raw learner id is tokenized on the way out and never persists in the flag frame.

Failure modes and edge cases

Early-warning systems fail in ways that are invisible on a validation metric and only surface as eroded trust months later. The recurring ones are specific enough to design against.

Label leakage. A feature that encodes the outcome — a final-grade-derived column, a “withdrew” flag, or an engagement window that extends past the intervention point — produces a model that scores beautifully in backtest and is useless in production, because at scoring time that feature does not yet exist. Detection: freeze every feature at an explicit as-of date and forbid any column computed from post-cutoff data; audit the top coefficients for anything suspiciously predictive.

Survivorship bias. Training only on learners who completed the term omits exactly the population the system exists to help — those who withdrew early. A model fit on survivors learns the wrong boundary. Detection: include withdrawn and dropped enrollments in the training frame with their true outcomes, and start the enrollment spine from the roster, not from learners with activity.

Feedback loops. Once flags drive interventions, a flagged-and-helped learner who then succeeds looks, in next term’s training data, like a false positive — the intervention worked, but the label says the flag was wrong. Naively retraining on this data teaches the model to stop flagging the cases it was right about. Detection: record which flags led to interventions and either hold them out of retraining or model the intervention explicitly.

Cohort drift. A model calibrated on a fall cohort mis-predicts a summer or fully-online cohort whose baseline engagement differs. Detection: standardize within cohort (as the reference code does) and re-check calibration each term against a held-out slice before trusting the probabilities.

Threshold decay. A fixed probability threshold slowly flags too many or too few as the cohort shifts. Mitigation: re-derive τ\tau from the alert budget each run rather than freezing a number.

The unifying discipline is the same one that governs the rest of the pipeline: features come from normalized, versioned, tokenized facts; the model is interpretable and reviewed; and the output is decision support that a human acts on, logged and access-controlled the whole way. A risk score built that way is defensible in front of a review board and useful to the counselor who acts on it.

Part of: Engagement Analytics & Reporting Automation