← back
Using RL Agent to Detect and Remediate ETL Pipeline Failures - Anna Marie Benzon
Takeaway
Use learning for bounded remediation choices while keeping operational authority and escalation rules outside the learned policy.
Summary
- An AWS Glue failure triggers EventBridge and Lambda; the agent reads CloudWatch logs and Glue Data Catalog metadata before proposing a bounded remediation action.
- Deterministic rules establish schema and data-quality facts, Q-learning selects contextual actions, and an independent safety layer can override unsafe or passive choices with escalation.
- The demonstration uses synthetic incidents and a modeled manual-recovery baseline; the speaker proposes shadow-mode validation before production authority and emphasizes post-action checks, policy versioning and rollback.
reinforcement-learningetlaws-glue
Original description
Cloud ETL failures often require engineers to manually inspect logs, diagnose schema or data-quality issues, select a repair, rerun the job, and validate recovery. This talk presents an RL-guided pipeline health agent that automates this workflow through deterministic anomaly detection, interpretable Q-learning, bounded remediation actions, and an external safety layer. The system detects schema drift, null-rate spikes, type changes, and runtime failures, then selects actions such as retry, schema coercion, rollback, quarantine, or escalation. Evaluation across 30 controlled synthetic runs demonstrates minutes-scale recovery for successfully resolved cases while highlighting the importance of deterministic rules and safety guardrails. Speakers: Anna Marie Benzon: Anna Marie Benzon is a World Economic Forum–recognized technology leader, startup founder, and PhD researcher in AI with 9+ years of experience building AI-powered products and scaling multidisciplinary teams. LinkedIn: / anna-marie-benzon GitHub: https://github.com/ambenzon27