The whole system on one page: ingest, score, transform, serve and erase. Open the image for the full-size animated version.

EuroStream started from a conflict we kept running into in European data work. Message logs append forever and lakehouse files are immutable by design, while GDPR Article 17 says a person can ask for their data to be erased. Bolting a delete endpoint onto that stack leaves copies behind in partitions, aggregates and in-memory state.

So we built the pipeline around erasure instead. A deletion request runs as an atomic six-layer transaction, and a verification endpoint proves what remains: zero rows in Silver and Gold, zero clear-text PII in Bronze, and one audit entry carrying a truncated hash.

The problem in one paragraph

Kafka partitions, Parquet files and materialized aggregates are append-only by nature. Erasing one person means rewriting history across every layer, and the usual fix is a manual script that misses a copy somewhere. A missed copy can mean a fine under Article 83(5) and a public incident, so deletion here is a pipeline operation with tests, not a support ticket.

How it works

1. A producer writes synthetic EU orders, clicks and payments onto the bus. Locally that is SQLite in WAL mode; in production it is Aiven Kafka over SASL_SSL with SCRAM-SHA-256.
2. The streaming engine consumes payments and scores them against three rules: a velocity spike (more than 5 payments in 300 seconds), an amount z-score above 3.0 computed with sample variance, and a billing-to-merchant country mismatch. Suppressed customers are dropped before any rule runs.
3. Every four hours the batch DAG runs watermarked incremental merges. Bronze keeps raw events, Silver deduplicates on natural keys and replaces PII with salted SHA-256 pseudonyms, and Gold builds the consent-gated marts.
4. Six quality assertions run after the merge, including uniqueness on both marts, a check that no clear-text email or IBAN pattern survives in Silver, and consent gating against the upstream flag.
5. Changed Silver and Gold tables export to de-identified Parquet and sync to the Hugging Face dataset. Only files whose content hash changed are pushed.
6. The FastAPI app reads the local DuckDB file during development or the Turso cloud replica in production, and serves the dashboard, the REST API and the Prometheus exposition.

The six-layer erasure cascade

Every layer that could hold a copy of the customer gets touched in one ordered transaction:

1. Suppression registry. The customer id enters an in-memory set and the governance.suppression_registry table, so replayed or delayed events are dropped before they reach a consumer.
2. Bronze anonymization. Raw rows are masked in place (email, iban, ip become <anonymized>) so ledger ordering and row counts survive.
3. Silver hard delete. Pseudonymized customer, order and payment rows are removed.
4. Gold hard delete. The Customer 360 aggregate, order facts and fraud summary rows are removed, so no ghost record keeps lifetime spend.
5. Streaming memory purge. The fraud scorer evicts the customer's window state and alert history, then their Bronze alerts are deleted.
6. Lake re-snapshot. Silver and Gold Parquet partitions are rewritten and synced, and the audit log stores sha256(request_id : customer_id)[0:16].

The verification endpoint returns the counts a regulator would ask for:

{
  "customer_id": "cust_424242",
  "verified": true,
  "is_suppressed": true,
  "gold_rows_remaining": 0,
  "silver_rows_remaining": 0,
  "bronze_clear_text_rows": 0,
  "bronze_anonymized_rows": 60,
  "audit_log_entries": 1
}

The benchmark suite runs 50 erasures end to end: 66.95 ms mean, 61.84 ms median, 109.20 ms p95 and 110.07 ms worst case. The statutory window is 60 seconds, so the slowest run finished with plenty of room, and erasure_sla_breaches_total stays at 0.

Three fraud rules and one gate

• Velocity. A customer with more than 5 payments inside a 300 second window gets one alert per tumbling window, which keeps a burst from becoming an alert flood.
• Amount z-score. Each payment is compared against the customer's own history with sample standard deviation, and the score has to pass 3.0. The history is a bounded deque of 200 amounts with a sweep every 50 events, so memory cannot grow without limit.
• Geo mismatch. The issuing bank country and the merchant country are compared directly, which catches card testing that stays under the amount threshold.
• Suppression gate. Every rule checks the suppression registry first. An erased customer cannot produce an alert, and their state is gone from memory as well as from storage.

Governance that runs in CI

The PII classifier does not trust patterns alone. European IBANs are validated with the ISO 13616 / ISO 7064 mod-97 checksum, which stops the false positives that regex matching produces on UUIDs and long order ids. An early version of this check flagged valid UUIDs, and the repository keeps the postmortem for it.

Schema drift is handled the same way. eurostream contracts --baseline governance/contracts.json compares the live models against a committed baseline, and CI blocks any pull request that adds an unclassified column or breaks an existing contract. The rest of the gate is ruff, mypy in strict mode across 23 source files, and the pytest suite.

Local now, cloud later

Every component sits behind one interface, so the same code runs on a laptop and in production:

• Event bus. SqliteBus locally, KafkaBus on Aiven when EUROSTREAM_EVENT_BUS_BACKEND=kafka.
• Warehouse. Embedded DuckDB for analytics, with every write mirrored to a Turso libSQL replica over HTTP.
• Lake. Parquet under data/lake/, synced to the swadhinbiswas/eustream dataset on Hugging Face.
• API. Uvicorn on port 7860 locally, a Docker container on Render in production.
• Orchestration. The CLI or cron locally, a scheduled GitHub Actions DAG every four hours in production.

The dual-engine setup is what keeps governance state alive. Containers restart, but suppression sets and watermarks live in DuckDB and Turso, so a fresh process reconstitutes them on boot instead of losing them with the container.

Observability

The app exposes Prometheus counters and a latency summary at /metrics/prometheus, which is enough for Grafana to scrape erasure throughput, SLA breaches and fraud alerts split by rule. The dashboard also has five live views: overview, fraud intelligence, Medallion and Customer 360, the Article 17 console, and a Prometheus explorer.

Stack

Part	Technology
Language	Python 3.11+, mypy strict
Event bus	SQLite WAL, Aiven Kafka
Warehouse	DuckDB, Turso libSQL
Lake	Parquet on Hugging Face
API	FastAPI, Uvicorn
Orchestration	GitHub Actions, 4-hour cron
Deploy	Docker, Render
Quality	pytest, ruff, contract gate

Links

• Live demo: eurostream-api.onrender.com (free instance, the first request wakes it in about 20 seconds)
• Dataset lake: swadhinbiswas/eustream
• Source: github.com/swadhinbiswas/eurostream
• Docs: architecture RFC, ADRs and the erasure flow
• Paper: JOSS manuscript draft