Agentic Data Pipeline

Overview

This project was built to solve the fragility of traditional ETL pipelines when upstream schemas change frequently. By introducing an LLM agent into the Airflow DAG, the pipeline can autonomously heal and adapt mapping logic without human intervention.

Technical Implementation

The core controller is written in Python using LangChain. We use a locally hosted Llama 3 instance to keep PII data secure.

Results

- 95% reduction in pipeline breakage alerts. - Saving 20 eng hours per week.