Webhook Event Ingestion for LMS Data Pipelines

A polling pipeline asks the same question on a timer — what changed since my last cursor? — and pays for the answer whether anything changed or not. Event-driven ingestion inverts the relationship: the learning management system tells you the moment a submission is graded, an enrollment is created, or a discussion post is published, and your infrastructure only does work when there is work to do. For latency-sensitive signals — an early-alert system that wants an attendance marker within seconds, or a live dashboard that should reflect a grade change before the instructor closes the tab — a webhook feed is not a nice-to-have; it is the only way to hit the freshness target without polling every course section every few seconds. This page scopes the push side of API Ingestion & Sync Workflows and treats an inbound event as a first-class entity that must be authenticated, deduplicated, ordered, and durably staged before a single value reaches the warehouse.

The engineering difficulty of webhooks is the mirror image of polling’s. A poller controls its own cadence and reads from a source of truth; a receiver is a public HTTP endpoint that anyone on the internet can POST to, that fires at least once rather than exactly once, that delivers out of order under retry, and that carries raw student identifiers in every payload. The discipline that makes an ingestion loop trustworthy — verify authenticity, make every write idempotent, tokenize at the compliance boundary — applies with even more force here, because unlike a poll you cannot simply re-run a failed request: a dropped webhook is gone unless you designed for replay. Event ingestion is best understood as the complement to async polling for grade syncs, not a replacement: the webhook tells you that something changed, and a bounded reconciliation poll remains the backstop that guarantees you never permanently lose an event the network ate.

The Inbound-Event Envelope

Before a single handler is written, model the thing you receive. Every webhook, regardless of vendor, resolves to one canonical entity — an inbound-event envelope — and the durability of the whole pipeline rests on persisting the right fields for each arrival. Vendors wrap the interesting payload in a great deal of transport chrome; your job is to collapse that variety into one row.

Two of these fields carry the weight. event_id is the axis of idempotency: because delivery is at least once, the same event will arrive twice under retry, and the only defence is a durable uniqueness constraint on this column so a repeat collapses into a no-op. occurred_at is the axis of ordering: webhooks arrive out of order under retry and parallel delivery, so a grade_change that happened at 10:00 can land after one that happened at 10:05, and a handler that trusts arrival order will overwrite the newer value with the older. Persisting the source timestamp lets the store apply last-writer-wins on occurred_at rather than on arrival, which is the difference between a correct final grade and a silently stale one.

The relational grain that matters is partition_key → ordered event stream. Events belonging to the same subject form an ordered sequence that must be applied in occurred_at order; events across subjects are independent and can be processed in parallel. This is the same per-key ordering guarantee a Kafka partition provides, and modelling it explicitly is what lets the pipeline fan out for throughput without corrupting any single learner’s record.

Webhook Sources: Canvas Live Events, Blackboard, and Moodle

The three dominant platforms all expose an event feed, but they differ in transport, authentication, and delivery semantics enough that a portable receiver must normalize all three into the envelope above. The per-vendor schema differences behind these payloads belong to the LMS data architecture and schema mapping reference; the delivery mechanics are summarized here.

Canvas Live Events is the reference case. Instructure emits a stream of learning events — grade_change, submission_created, enrollment_updated, asset_accessed — as Caliper-shaped or Canvas-native JSON, delivered to an HTTPS endpoint you register through Canvas Data Services. Each POST carries an HMAC-SHA256 signature computed over the raw request body with a shared secret, and the body wraps the interesting payload in a metadata header carrying the event_name and an event id. Delivery is at-least-once with retry on any non-2xx response, and ordering is explicitly not guaranteed — the two properties that dominate every design decision below. The concrete receiver for this feed is the subject of handling Canvas Live Events webhooks in Python.

Blackboard Learn exposes webhook subscriptions through its REST platform, signing each delivery with a secret issued alongside the OAuth application registration. The retry behaviour is similar to Canvas — non-2xx triggers a backing-off redelivery — so the same idempotency and ordering machinery applies unchanged.

Moodle is the ragged case. Core Moodle raises internal events through its observer system, but externalizing them as HTTP webhooks depends on a plugin, and delivery guarantees vary: some configurations fire once with no retry, which shifts the burden of loss-detection onto your reconciliation poll. Where a Moodle deployment cannot emit reliable events at all, the pipeline falls back to the incremental pull model in incremental sync state management, tracking a last-seen timestamp instead of consuming a stream.

Verifying Authenticity: HMAC Signatures

A webhook endpoint is a publicly reachable URL, so the first line of every handler is authentication, not parsing. The standard mechanism is a hash-based message authentication code: the producer computes HMAC-SHA256(secret, raw_body) and sends the result in a header; the receiver recomputes the same value over the bytes it received and accepts the request only if the two match. Because the secret never travels on the wire, a forged request cannot produce a valid signature without it.

The verification has three non-negotiable properties. First, it runs over the raw request body, byte for byte, before any JSON parsing — because re-serializing a parsed object changes whitespace and key order and therefore the hash. Second, the comparison uses a constant-time equality check (hmac.compare_digest), never ==, so that an attacker cannot recover the expected signature one byte at a time by measuring how long a comparison takes. Third, the signature covers a timestamp the receiver bounds, so that a valid request captured off the wire cannot be replayed hours later. The formal statement of the check is simply

$\text{accept} \iff \operatorname{HMAC}_{K}(m) \stackrel{\text{ct}}{=} \sigma \;\wedge\; \lvert t_{\text{now}} - t_{\text{sent}}\rvert \le \Delta$

where $K$ is the shared secret, $m$ the raw body, $\sigma$ the supplied signature, $\stackrel{\text{ct}}{=}$ a constant-time comparison, and $\Delta$ the replay window (a few minutes). A request that fails either conjunct is rejected with a 401 and never enters the inbox — an unverified event is not data, it is an attack surface.

Idempotent Handling and the Durable Inbox

At-least-once delivery means the pipeline will see duplicates, and the only robust response is to make processing idempotent rather than to hope for exactly-once transport. The pattern has two parts: a durable inbox that decouples acknowledgement from processing, and a deduplication key that collapses repeats.

The durable inbox is an append-only queue (a database table, a Kafka topic, or an SQS queue) that sits between the HTTP receiver and the business logic. The receiver’s only job is to verify the signature, write the raw envelope to the inbox, and return 200 — fast. It does not recompute grades, hit the warehouse, or call another service inline, because every second the receiver spends processing is a second the producer waits, and a slow endpoint earns a delivery timeout and a redelivery, which manufactures the very duplicates you are trying to avoid. Acknowledge on durable receipt; process asynchronously behind the queue. This is the same submit-then-defer discipline that governs async polling for grade syncs, applied in the opposite direction.

Deduplication happens where the handler drains the inbox. Every event carries a producer-assigned event_id, and the handler upserts on it — either a unique constraint that turns a duplicate insert into a no-op, or a seen-set check before processing. The canonical store’s primary key is therefore event_id (or (source_platform, event_id)), so that no matter how many times the network redelivers, the effect on the warehouse is applied exactly once. Combined with last-writer-wins on occurred_at, this yields the two guarantees downstream analytics assume: every event is applied, and it is applied in the order it happened, regardless of the order it arrived.

Ordering is the subtlety idempotency alone does not solve. Deduplication stops a repeat from being counted twice, but it does not stop an old event from clobbering a new one when they arrive reversed. The fix is to make writes commutative on the source timestamp: an update applies only if its occurred_at is newer than the row already stored, so a late-arriving stale event is dropped rather than regressing the value. Ordering only needs to hold within a partition_key, which is what lets the handler pool process different courses in parallel while keeping each learner’s stream consistent.

Replay is what turns the inbox from a buffer into a safety net. Because the raw envelope is stored durably before processing, a transformation bug — a wrong normalization rule, a mis-parsed field — is recoverable by re-draining the inbox against fixed code, with idempotency guaranteeing the replay does not double-apply anything already correct. This is also the recovery path for the events a lossy Moodle plugin drops: a periodic reconciliation pull compares the event stream against a source-of-truth poll and re-injects anything missing, so the push feed’s speed is backstopped by the pull feed’s completeness.

Compliance: PII in Every Payload

A webhook payload is unusually dense with regulated data because it describes a specific student doing a specific thing — the exact person-plus-record linkage the Family Educational Rights and Privacy Act (FERPA) governs. A grade_change event names the learner, the assignment, and the new score in one small object. Three constraints therefore bind the ingestion path, and all three must be enforced before the payload reaches durable storage, not after.

• Verify, then tokenize, then persist — in that order. The signature is checked against the raw body first; the direct identifiers (user_id, sis_user_id, login_id, email) are replaced with a salted SHA-256 surrogate as the envelope crosses into the inbox; only the tokenized form is written anywhere durable. The same deterministic tokenization that cross-LMS student ID mapping relies on keeps the surrogate joinable across platforms while making it useless without the salt.
• Log the envelope, never the body. A verification failure or a poison payload is logged by event_id, event_type, and partition_key — never by dumping the raw JSON, which would spray student data into a log aggregator. The signature itself is never logged; it is a credential.
• Audit every canonical row. Each processed event stamps received_at, source_platform, and the originating event_id, so any warehouse value is traceable to the exact delivery that produced it — the lineage an institutional audit and a deletion request both require. These rules flow down from the pipeline-wide FERPA tokenization boundary.

The practical test is the same one every stage of the pipeline must pass: can a raw student identifier reach durable storage or a log line? If the answer is anything but “no, tokenization at the inbox boundary makes it impossible,” the receiver is mis-designed.

Reference Python Implementation

The receiver below is the production shape of the pattern: it reads the raw body, verifies the HMAC-SHA256 signature in constant time, bounds a replay window, tokenizes identifiers, and enqueues the envelope idempotently — returning fast and doing no heavy work inline. It models placeholder tokenization; a real deployment injects a per-institution salt from a secrets manager and swaps the in-memory inbox for a durable queue.

import hashlib
import hmac
import json
import os
import time

# Shared secret registered with the LMS webhook subscription; rotated on schedule.
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"].encode()
ID_SALT = os.environ["EVENT_ID_SALT"].encode()
REPLAY_WINDOW_S = 300           # reject anything older than five minutes
PII_FIELDS = ("user_id", "sis_user_id", "login_id", "email")

_seen: set[str] = set()          # stand-in for a durable uniqueness constraint


def tokenize(value: str | int) -> str:
    """Irreversible FERPA-safe surrogate for a direct identifier."""
    return "stu_" + hashlib.sha256(ID_SALT + str(value).encode()).hexdigest()[:24]


def verify(raw_body: bytes, signature: str, sent_at: float) -> bool:
    """Constant-time HMAC check plus a bounded replay window."""
    expected = hmac.new(WEBHOOK_SECRET, raw_body, hashlib.sha256).hexdigest()
    fresh = abs(time.time() - sent_at) <= REPLAY_WINDOW_S
    return fresh and hmac.compare_digest(expected, signature)


def to_envelope(body: dict) -> dict:
    """Collapse a vendor payload into the canonical inbound-event envelope."""
    clean = {k: v for k, v in body.get("payload", {}).items() if k not in PII_FIELDS}
    for f in PII_FIELDS:
        if body.get("payload", {}).get(f) is not None:
            clean[f"{f}_token"] = tokenize(body["payload"][f])
    return {
        "event_id": body["event_id"],
        "event_type": body["event_type"],
        "occurred_at": body["occurred_at"],
        "received_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "partition_key": clean.get("course_id", "unknown"),
        "payload": clean,
    }


def enqueue(envelope: dict) -> str:
    """Idempotent write to the durable inbox; a repeat is a no-op."""
    if envelope["event_id"] in _seen:
        return "duplicate"           # at-least-once delivery absorbed here
    _seen.add(envelope["event_id"])
    # inbox.put(envelope)  -> durable queue in production
    return "enqueued"


def receive(raw_body: bytes, signature: str, sent_at: float) -> tuple[int, str]:
    """HTTP entry point: verify, tokenize, enqueue, ack fast."""
    if not verify(raw_body, signature, sent_at):
        return 401, "invalid signature"          # never enters the inbox
    body = json.loads(raw_body)                   # parse only AFTER verifying
    result = enqueue(to_envelope(body))
    return 200, result                            # ack immediately; process async

Four properties make this receiver production-grade: it verifies over the raw bytes before parsing, it compares signatures in constant time, it tokenizes every identifier before the envelope is durable, and it deduplicates on event_id so at-least-once delivery is harmless. The heavy work — recomputing weighted grades, upserting the warehouse — happens behind the inbox in a separate worker, which is where the ordering and last-writer-wins logic lives.

Failure Modes and Edge Cases

• Spoofed or unsigned events. A public endpoint attracts forged and probing requests. Verifying the HMAC over the raw body before parsing, and rejecting a missing signature outright, is the only defence — never process an unverified payload, and never let a parse of an unsigned body happen first.
• Duplicate delivery. At-least-once transport guarantees repeats, especially after a slow response triggers a redelivery. The event_id uniqueness constraint collapses them; without it, a redelivered grade_change double-applies. Keep the receiver fast so timeouts do not manufacture extra duplicates in the first place.
• Out-of-order arrival. Retry and parallel delivery reorder events, so an older occurred_at can land after a newer one. Last-writer-wins on the source timestamp — not arrival time — prevents a stale event from regressing a value. Deduplication alone does not solve this.
• Poison payloads. A malformed or unparseable body that fails processing repeatedly will otherwise block the queue forever. Cap the retry count and divert exhausted events to a dead-letter queue for inspection, keyed by event_id, so one bad message does not stall the stream. The retry-budget mechanics are shared with error retry logic for sync jobs.
• Silent event loss. A dropped webhook — a receiver outage, a lossy Moodle plugin, a delivery the producer gave up retrying — leaves a permanent gap that no re-request can fill. A periodic reconciliation pull against the incremental cursor is the backstop; pair this feed with incremental sync state management so nothing is lost forever.
• Secret rotation mid-flight. Rotating the signing secret invalidates in-flight signatures and starts rejecting valid events. Accept both the old and new secret during an overlap window, then retire the old one, so rotation never drops a legitimate delivery.

Related

• Async Polling for Grade Syncs — the pull counterpart; webhooks say that something changed, polling reconciles and backstops loss.
• Incremental Sync State Management — the last-seen cursor that reconciles dropped events and closes webhook gaps.
• Error Retry Logic for Sync Jobs — retry budgets, backoff, and the dead-letter path poison payloads divert to.
• Handling Canvas Live Events Webhooks in Python — a concrete, runnable receiver for the Canvas Live Events feed.

Part of: API Ingestion & Sync Workflows