Engagement Metric Normalization: One Comparable Index from Three Vendors’ Activity Exhaust

Raw activity counts lie. A learner who logged 400 page views in a 500-student lecture is not more engaged than one who logged 60 in a 12-person seminar, and a Moodle log row is not interchangeable with a Canvas participation. Engagement metric normalization is the stage that makes those numbers comparable — converting heterogeneous, vendor-specific activity events into a single 0–1 engagement index that means the same thing across courses, cohorts, and platforms. It is the first stage under Engagement Analytics & Reporting Automation, and everything downstream — the at-risk scoring model and the automated dashboards — depends on the comparability this stage guarantees. If normalization is wrong, every consumer is confidently wrong in the same direction.

This page defines the entity model for activity events, the endpoints each vendor exposes them through, the baselining and standardization logic that produces the index, the FERPA constraints that bound the whole thing, a runnable reference implementation, and the failure modes that make an engagement index quietly meaningless.

Entity model for activity events

An activity event is the atomic unit of engagement. The normalized model is deliberately tiny — five fields — because a small, frozen schema is what lets three incompatible vendor payloads land in one table. Every vendor event is mapped into this shape during ingestion; vendor quirks are resolved here and never leak downstream.

Entity model for activity-event normalization A central fact_activity_event table sits between two lookups. It is keyed by a tokenized learner_token and a course_key and holds event_type, occurred_at, iso_week and a numeric weight. On the left, a dim_event_taxonomy table maps each canonical event_type to its weight and vendor source. On the right, a dim_course dimension carries term and cohort_size used to baseline the index. The learner_token column is tokenized at the FERPA boundary, marked with a dagger. fact_activity_event grain · one row per activity event learner_token FK † course_key FK event_type (canonical) occurred_at · iso_week weight (from taxonomy) dim_event_taxonomy event_type PK weight vendor_source taxonomy_version dim_course course_key PK term, title cohort_size start_of_term_roster engagement_index grain · learner_token × course × iso_week 0–1, baselined within cohort weight × baseline × squash † learner_token tokenized at the FERPA boundary · PK primary key · FK foreign key

The event_type column is canonical, not vendor-native: a Canvas participation, a Moodle mod_forum post, and a Blackboard discussion interaction all resolve to forum_post. The weight is not stored per event by hand — it is looked up from dim_event_taxonomy, so re-weighting forum_post from 3.0 to 4.0 is a single versioned change that reprices every historical event on the next recompute. occurred_at is stored as a normalized UTC timestamp and iso_week is derived from it, because the engagement index is a weekly-cadence metric and ISO week numbering is the only week definition that behaves consistently across year boundaries. The learner_token is already tokenized when it arrives at this table; the raw identifier never enters, following the boundary defined in FERPA-Compliant PII Handling.

The choice of a weekly grain is deliberate and load-bearing. Daily engagement is too noisy to act on — a learner who does all their coursework on Sunday looks disengaged Monday through Saturday — while a term-level rollup arrives far too late for any early-warning intervention to matter. A week is the natural cadence of academic work: it aligns with how assignments are set, how advisors schedule outreach, and how a learner’s own rhythm settles. Storing the fact table at the raw-event grain but serving the index at the learner-week grain gives you both replayability and an actionable reporting unit, and it means a change to the week definition or the taxonomy weights can be recomputed from the durable event rows without re-pulling anything from a vendor whose logs may already have aged out.

One subtlety in the model deserves emphasis: event_type is a closed vocabulary, not an open string. Every vendor event either maps to one of the six canonical types or is explicitly dropped with a zero weight — there is no other bucket that quietly accumulates unmapped activity. This is what makes the taxonomy auditable: at any point you can ask “what fraction of Moodle log rows this term found no mapping,” and a rising answer is an early signal that a plugin changed or a new eventname appeared, rather than a silent distortion of every affected course’s index.

API endpoints and request patterns

Each vendor exposes activity through a different surface, and knowing which endpoint yields which event type is half the normalization battle.

  • Canvas — the Analytics API returns per-course and per-user rollups (GET /api/v1/courses/:course_id/analytics/student_summaries), while GET /api/v1/users/:id/page_views returns near-raw page-view events with a created_at and an action. Participations and page views overlap, so the taxonomy must decide which one maps to content_view to avoid double-counting. These reads are cost-expensive; pace them with Handling Canvas API Rate Limits and page them with the pagination strategies for bulk exports.
  • Moodle — the canonical source is logstore_standard_log, exposed through the Logs report or a web-service query. Each row carries eventname (e.g. \mod_forum\event\post_created), component, timecreated (Unix epoch integer), and userid. The normalization job maps hundreds of distinct eventnames onto the small canonical taxonomy; unmapped events default to a zero weight so a new plugin cannot silently inflate engagement.
  • Blackboard — course activity and accesses come through the Learn REST API (GET /learn/api/public/v1/courses/:courseId/...) and activity reports. Events are coarser — a single “course access” hides many in-page interactions — so Blackboard events carry a resolution caveat: they anchor the login and content_view buckets but undercount fine-grained interaction.

The critical request-pattern discipline is pull tokenized, or tokenize immediately on landing. Activity endpoints return raw user_id and login_id values; those are tokenized at the ingestion boundary before the events reach the normalization table, so nothing in this stage ever handles a direct identifier.

Normalization logic: from raw counts to a 0–1 index

Normalization is three deterministic transforms applied in order: weight, baseline, squash.

Weight. Sum each learner’s events per week, multiplying each by its taxonomy weight, to get a single weighted_activity scalar per (learner_token, course_key, iso_week). This collapses “six logins and one forum post” into one comparable effort total instead of two incommensurable counts.

Baseline. A raw weighted total is still course-specific, so baseline it against the course cohort’s distribution for that week. Two standard transforms apply. Min-max rescales the learner’s total xx against the cohort’s observed range:

eminmax=x−xminxmax−xmine_\text{minmax} = \frac{x - x_\text{min}}{x_\text{max} - x_\text{min}}

which is intuitive but fragile — one hyperactive learner stretches xmaxx_\text{max} and compresses everyone else toward zero. Z-score standardization instead expresses the total in standard deviations from the cohort mean μc\mu_c with spread σc\sigma_c:

z=x−μcσcz = \frac{x - \mu_c}{\sigma_c}

which is robust to a single outlier but unbounded, so a zz of +4+4 is meaningless on a dashboard gauge. The z-score is the better default for skewed engagement distributions, provided it is bounded in the next step.

Squash. Map the standardized value into a comparable, bounded 0–1 range with a logistic function, so that the index is directly interpretable as “how engaged relative to the cohort”:

e=11+e−ze = \frac{1}{1 + e^{-z}}

A learner at the cohort mean lands at exactly 0.5, one standard deviation above at roughly 0.73, and one below at roughly 0.27 — a scale an advisor can read without a statistics degree. This is the index that the single-vendor case in Computing Canvas Page-View Engagement Scores and the cross-vendor case in Normalizing LMS Login Frequency Across Platforms both produce.

The reason to baseline within the course cohort rather than against a global institutional distribution is comparability of the right kind. What an advisor and an early-warning model actually need is not “is this learner more active than the average student at the university,” but “is this learner falling behind the peers sitting in the same course under the same expectations.” A heavily-online course and a lab-based course generate wildly different absolute activity, and a global baseline would rank every lab student as disengaged and every online student as thriving purely as an artifact of course design. Cohort baselining cancels that design effect, so the index measures relative participation against a like-for-like reference group. The tradeoff is that the index is explicitly not an absolute measure — a 0.5 in a highly-engaged honors seminar and a 0.5 in a struggling gateway course both mean “middle of this room,” which is exactly the semantics early-warning wants and exactly the semantics a naive cross-course leaderboard would get wrong.

A second decision hides inside the baseline: whether to compute μc\mu_c and σc\sigma_c over the whole term or over a rolling recent window. A term-wide baseline is stable but slow to react — a learner who was highly active for ten weeks and then goes dark still scores near their historical mean for a while. A rolling three-to-four-week window reacts faster to a genuine drop, which is what early-warning is for, at the cost of more noise. The normalization stage exposes both and lets the at-risk scoring model choose, because the right window is a policy decision about false-positive tolerance, not a property of the data.

Compliance constraints

Engagement events are education records under FERPA the moment they are attributable to a named learner, so normalization operates entirely on tokenized identifiers. Three constraints bind this stage specifically:

  • Tokenize before the event table, never after. The learner_token is produced at the ingestion boundary with a salted SHA-256 pattern, and the reversible mapping stays in the restricted vault covered by implementing salted SHA-256 tokenization for student ids. The normalization job cannot re-identify a learner even if compromised.
  • Baseline needs cohort size, not cohort identities. Computing μc\mu_c and σc\sigma_c requires the distribution of weighted activity, which is aggregate. No individual learner’s identity is needed to normalize another learner, so the whole stage runs de-identified.
  • Data minimization on event detail. Store the canonical event_type and weight, not the raw vendor payload with its URLs and IP addresses. A content_view event does not need to record which page — that detail is both PII-adjacent and irrelevant to a weekly index.

Reference Python implementation

The function below takes a DataFrame of already-tokenized activity events and produces the normalized 0–1 engagement index per learner-week, z-scored within each course cohort and logistically squashed. It uses Python 3.10+ syntax and pandas nullable dtypes so that a cold-start week (zero cohort variance) degrades to a neutral index rather than a divide-by-zero.

python
import numpy as np
import pandas as pd

# Canonical event taxonomy: weight reflects genuine learning effort, versioned.
TAXONOMY: dict[str, float] = {
    "login": 1.0,
    "content_view": 1.5,
    "video_play": 2.0,
    "quiz_attempt": 3.0,
    "submission": 4.0,
    "forum_post": 4.0,
}
NEUTRAL_INDEX = 0.5  # cold-start / zero-variance fallback


def normalize_engagement(events: pd.DataFrame) -> pd.DataFrame:
    """Turn tokenized activity events into a 0-1 engagement index per learner-week.

    Expected columns: learner_token (str, 'stu_...'), course_key (str),
    iso_week (str, e.g. '2026-W07'), event_type (str, canonical).
    """
    if events.empty:
        return pd.DataFrame(
            columns=["learner_token", "course_key", "iso_week",
                     "weighted_activity", "z", "engagement_index"]
        )

    df = events.copy()
    # FERPA guard: refuse to process anything that is not already tokenized.
    if not df["learner_token"].str.startswith("stu_").all():
        raise ValueError("untokenized learner id reached normalization boundary")

    # 1. WEIGHT — map each canonical event_type to its taxonomy weight, sum per week.
    df["weight"] = df["event_type"].map(TAXONOMY).fillna(0.0)  # unknown events = 0
    per_week = (
        df.groupby(["course_key", "iso_week", "learner_token"], as_index=False)["weight"]
        .sum()
        .rename(columns={"weight": "weighted_activity"})
    )

    # 2. BASELINE — z-score within each (course, week) cohort.
    grp = per_week.groupby(["course_key", "iso_week"])["weighted_activity"]
    mu = grp.transform("mean")
    sigma = grp.transform("std", ddof=0)
    # Zero variance (cold-start or single-learner cohort) -> undefined z.
    per_week["z"] = np.where(sigma > 0, (per_week["weighted_activity"] - mu) / sigma, np.nan)

    # 3. SQUASH — logistic map to a bounded, comparable 0-1 index.
    per_week["engagement_index"] = (
        (1.0 / (1.0 + np.exp(-per_week["z"]))).fillna(NEUTRAL_INDEX).round(4)
    )
    return per_week.sort_values(["course_key", "iso_week", "engagement_index"])


if __name__ == "__main__":
    sample = pd.DataFrame(
        {
            "learner_token": ["stu_a1", "stu_a1", "stu_b2", "stu_c3", "stu_c3"],
            "course_key": ["BIO101"] * 5,
            "iso_week": ["2026-W07"] * 5,
            "event_type": ["login", "forum_post", "login", "submission", "video_play"],
        }
    )
    print(normalize_engagement(sample).to_string(index=False))

The function is pure and idempotent: given the same events it returns the same index, so it can be replayed against immutable staging without side effects. The startswith("stu_") guard makes the FERPA boundary a runtime invariant rather than a convention, and the fillna(NEUTRAL_INDEX) is what keeps a cold-start week from emitting NaN into a dashboard.

Failure modes and edge cases

  • Zero-inflation. Most learners generate few events in any given week, so the weighted-activity distribution is a spike at zero with a heavy right-hand skew. A z-score over a zero-inflated distribution pins the median learner well below the mean, making “average” look disengaged. Fix: baseline over learners with at least one event that week, and treat total silence as a separate signal the at-risk model consumes directly rather than as a low index.
  • Course-size skew. A 12-person cohort has an unstable σc\sigma_c — one active learner swings it — so small-section indices are noisy. Fix: apply a minimum-cohort threshold below which the index falls back to a neutral value, and never compare a raw index across cohorts of wildly different size without noting it.
  • Cold-start weeks. Week one has no history and near-zero variance, so z-scores are undefined. Fix: the NEUTRAL_INDEX fallback above, plus suppressing downstream flags during the warm-up window, mirroring the cold-start handling flagged in the Engagement Analytics & Reporting Automation overview.
  • Taxonomy drift. A vendor renames an eventname or ships a new plugin, and events silently map to weight zero, deflating engagement for a whole course. Fix: alert on the share of events falling through to the zero-weight default, exactly as schema drift is caught in the LMS Data Architecture & Schema Mapping reference.
  • Double-counting Canvas participations and page views. Both endpoints describe overlapping activity; summing both inflates Canvas engagement relative to Moodle and Blackboard. Fix: the taxonomy maps exactly one Canvas primitive to content_view and drops the other, a decision made once and versioned.

Part of: Engagement Analytics & Reporting Automation