|
|
--- |
|
|
title: Lineage Graph Accelerator |
|
|
emoji: 🔥 |
|
|
colorFrom: purple |
|
|
colorTo: blue |
|
|
sdk: gradio |
|
|
sdk_version: 6.0.0 |
|
|
app_file: app.py |
|
|
pinned: true |
|
|
license: mit |
|
|
short_description: AI data lineage extraction & export to data catalogs |
|
|
tags: |
|
|
- data-lineage |
|
|
- mcp |
|
|
- gradio |
|
|
- data-governance |
|
|
- dbt |
|
|
- airflow |
|
|
- etl |
|
|
- mcp-in-action-track-productivity |
|
|
- hackathon |
|
|
--- |
|
|
|
|
|
# Lineage Graph Accelerator 🔥 |
|
|
|
|
|
**AI-powered data lineage extraction and visualization for modern data platforms** |
|
|
|
|
|
[](https://huggingface.co/spaces/aamanlamba/Lineage-graph-accelerator) |
|
|
[](https://opensource.org/licenses/MIT) |
|
|
[](https://gradio.app) |
|
|
|
|
|
> 🎉 **Built for the Gradio Agents & MCP Hackathon - Winter 2025** 🎉 |
|
|
> |
|
|
> Celebrating MCP's 1st Birthday! This project demonstrates the power of MCP integration for enterprise data governance. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🌟 What is Lineage Graph Accelerator? |
|
|
|
|
|
Lineage Graph Accelerator is an AI-powered tool that helps data teams: |
|
|
|
|
|
- **Extract** data lineage from dbt, Airflow, BigQuery, Snowflake, and more |
|
|
- **Visualize** complex data dependencies with interactive Mermaid diagrams |
|
|
- **Export** lineage to enterprise data catalogs (Collibra, Microsoft Purview, Alation) |
|
|
- **Integrate** with MCP servers for enhanced AI-powered processing |
|
|
|
|
|
### Why Data Lineage Matters |
|
|
|
|
|
Understanding where your data comes from and where it goes is critical for: |
|
|
- **Data Quality**: Track data transformations and identify issues |
|
|
- **Compliance**: Document data flows for GDPR, CCPA, and other regulations |
|
|
- **Impact Analysis**: Understand downstream effects of schema changes |
|
|
- **Data Discovery**: Help analysts find and trust data assets |
|
|
|
|
|
--- |
|
|
|
|
|
## 🎯 Key Features |
|
|
|
|
|
### Multi-Source Support |
|
|
| Source | Status | Description | |
|
|
|--------|--------|-------------| |
|
|
| dbt Manifest | ✅ | Parse dbt's manifest.json for model dependencies | |
|
|
| Airflow DAG | ✅ | Extract task dependencies from DAG definitions | |
|
|
| SQL DDL | ✅ | Parse CREATE statements for table lineage | |
|
|
| BigQuery | ✅ | Query INFORMATION_SCHEMA for metadata | |
|
|
| Custom JSON | ✅ | Flexible node/edge format for any source | |
|
|
| Snowflake | 🔄 | Coming via MCP integration | |
|
|
|
|
|
### Export to Data Catalogs |
|
|
| Catalog | Status | Format | |
|
|
|---------|--------|--------| |
|
|
| OpenLineage | ✅ | Universal open standard | |
|
|
| Collibra | ✅ | Data Intelligence Platform | |
|
|
| Microsoft Purview | ✅ | Azure Data Governance | |
|
|
| Alation | ✅ | Data Catalog | |
|
|
| Apache Atlas | 🔄 | Coming soon | |
|
|
|
|
|
### Visualization Options |
|
|
- **Mermaid Diagrams**: Interactive, client-side rendering |
|
|
- **Subgraph Grouping**: Organize by data layer (raw, staging, marts) |
|
|
- **Color-Coded Nodes**: Distinguish sources, tables, models, reports |
|
|
- **Edge Labels**: Show transformation types |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 Quick Start |
|
|
|
|
|
### Try Online (HuggingFace Space) |
|
|
|
|
|
1. Visit [Lineage Graph Accelerator on HuggingFace](https://huggingface.co/spaces/YOUR_SPACE) |
|
|
2. Click "Load Sample" to load example data |
|
|
3. Click "Extract Lineage" to see the visualization |
|
|
4. Explore the Demo Gallery for more examples |
|
|
|
|
|
### Run Locally |
|
|
|
|
|
```bash |
|
|
# Clone the repository |
|
|
git clone https://github.com/YOUR_REPO/lineage-graph-accelerator.git |
|
|
cd lineage-graph-accelerator |
|
|
|
|
|
# Create virtual environment |
|
|
python3 -m venv .venv |
|
|
source .venv/bin/activate |
|
|
|
|
|
# Install dependencies |
|
|
pip install -r requirements.txt |
|
|
|
|
|
# Run the app |
|
|
python app.py |
|
|
``` |
|
|
|
|
|
Open http://127.0.0.1:7860 in your browser. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📖 Usage Guide |
|
|
|
|
|
### 1. Text/File Metadata Tab |
|
|
|
|
|
Paste your metadata directly: |
|
|
|
|
|
```json |
|
|
{ |
|
|
"nodes": [ |
|
|
{"id": "source_db", "type": "source", "name": "Source Database"}, |
|
|
{"id": "staging", "type": "table", "name": "Staging Table"}, |
|
|
{"id": "analytics", "type": "table", "name": "Analytics Table"} |
|
|
], |
|
|
"edges": [ |
|
|
{"from": "source_db", "to": "staging"}, |
|
|
{"from": "staging", "to": "analytics"} |
|
|
] |
|
|
} |
|
|
``` |
|
|
|
|
|
### 2. Sample Data |
|
|
|
|
|
Load pre-built samples to explore different scenarios: |
|
|
- **Simple JSON**: Basic node/edge lineage |
|
|
- **dbt Manifest**: Full dbt project with 15+ models |
|
|
- **Airflow DAG**: ETL pipeline with 15 tasks |
|
|
- **Data Warehouse**: Snowflake-style multi-layer architecture |
|
|
- **ETL Pipeline**: Complex multi-source pipeline |
|
|
- **Complex Demo**: 50+ node e-commerce platform |
|
|
|
|
|
### 3. Export to Data Catalogs |
|
|
|
|
|
1. Extract lineage from your metadata |
|
|
2. Expand "Export to Data Catalog" |
|
|
3. Select format (OpenLineage, Collibra, Purview, Alation) |
|
|
4. Click "Generate Export" |
|
|
5. Copy the JSON for import into your catalog |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔌 MCP Integration |
|
|
|
|
|
Connect to MCP (Model Context Protocol) servers for enhanced processing: |
|
|
|
|
|
``` |
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ |
|
|
│ Lineage Graph │────▶│ MCP Server │────▶│ AI Model │ |
|
|
│ Accelerator │ │ (HuggingFace) │ │ (Claude) │ |
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘ |
|
|
``` |
|
|
|
|
|
### Configuration |
|
|
|
|
|
1. Expand "MCP Server Configuration" in the UI |
|
|
2. Enter your MCP server URL |
|
|
3. Add API key (if required) |
|
|
4. Click "Test Connection" |
|
|
|
|
|
### Run Local MCP Server |
|
|
|
|
|
```bash |
|
|
uvicorn mcp_example.server:app --reload --port 9000 |
|
|
``` |
|
|
|
|
|
Then use `http://localhost:9000/mcp` as your server URL. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🏗️ Architecture |
|
|
|
|
|
```mermaid |
|
|
flowchart TD |
|
|
A[User Interface - Gradio] --> B[Input Parser] |
|
|
B --> C{Source Type} |
|
|
C -->|dbt| D[dbt Parser] |
|
|
C -->|Airflow| E[Airflow Parser] |
|
|
C -->|SQL| F[SQL Parser] |
|
|
C -->|JSON| G[JSON Parser] |
|
|
D & E & F & G --> H[LineageGraph] |
|
|
H --> I[Mermaid Generator] |
|
|
H --> J[Export Engine] |
|
|
I --> K[Visualization] |
|
|
J --> L[OpenLineage] |
|
|
J --> M[Collibra] |
|
|
J --> N[Purview] |
|
|
J --> O[Alation] |
|
|
|
|
|
subgraph Optional |
|
|
P[MCP Server] --> H |
|
|
end |
|
|
``` |
|
|
|
|
|
### Project Structure |
|
|
|
|
|
``` |
|
|
lineage-graph-accelerator/ |
|
|
├── app.py # Main Gradio application |
|
|
├── exporters/ # Data catalog exporters |
|
|
│ ├── __init__.py |
|
|
│ ├── base.py # Base classes |
|
|
│ ├── openlineage.py # OpenLineage format |
|
|
│ ├── collibra.py # Collibra format |
|
|
│ ├── purview.py # Microsoft Purview format |
|
|
│ └── alation.py # Alation format |
|
|
├── samples/ # Sample data files |
|
|
│ ├── sample_metadata.json |
|
|
│ ├── dbt_manifest_sample.json |
|
|
│ ├── airflow_dag_sample.json |
|
|
│ ├── sql_ddl_sample.sql |
|
|
│ ├── warehouse_lineage_sample.json |
|
|
│ ├── etl_pipeline_sample.json |
|
|
│ └── complex_lineage_demo.json |
|
|
├── mcp_example/ # Example MCP server |
|
|
│ └── server.py |
|
|
├── tests/ # Unit tests |
|
|
│ └── test_app.py |
|
|
├── memories/ # Agent configuration |
|
|
├── USER_GUIDE.md # Comprehensive user guide |
|
|
├── BUILD_PLAN.md # Development roadmap |
|
|
└── requirements.txt |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧪 Testing |
|
|
|
|
|
```bash |
|
|
# Activate virtual environment |
|
|
source .venv/bin/activate |
|
|
|
|
|
# Run unit tests |
|
|
python -m unittest tests.test_app -v |
|
|
|
|
|
# Run setup validation |
|
|
python test_setup.py |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📋 Requirements |
|
|
|
|
|
- Python 3.9+ |
|
|
- Gradio 5.49.1+ |
|
|
- See `requirements.txt` for full dependencies |
|
|
|
|
|
--- |
|
|
|
|
|
## 🎖️ Competition Submission |
|
|
|
|
|
**Track**: Track 2 - MCP in Action (Productivity) |
|
|
|
|
|
**Team Members**: |
|
|
|
|
|
- [Aaman Lamba](https://aamanlamba.com) | [HuggingFace](https://huggingface.co/aamanlamba) | [GitHub](https://github.com/aamanlamba) |
|
|
|
|
|
### Judging Criteria Alignment |
|
|
|
|
|
| Criteria | Implementation | |
|
|
|----------|----------------| |
|
|
| **UI/UX Design** | Clean, professional interface with tabs, accordions, and color-coded visualizations | |
|
|
| **Functionality** | Full MCP integration, multiple input formats, 5 export formats | |
|
|
| **Creativity** | Novel approach to data lineage visualization with AI-powered parsing | |
|
|
| **Documentation** | Comprehensive README, USER_GUIDE.md, inline comments | |
|
|
| **Real-world Impact** | Solves critical enterprise need for data governance and compliance | |
|
|
|
|
|
### Demo Video |
|
|
|
|
|
📺 **YouTube**: [Watch the Demo](https://youtu.be/U4Dfc7txa_0) |
|
|
🎥 **Loom**: [Alternative Link](https://www.loom.com/share/3de27e88e01f4e97bfd13e4f0031f416) |
|
|
|
|
|
**Highlights**: |
|
|
|
|
|
- AI Assistant with Google Gemini generating lineage from natural language |
|
|
- MCP Integration with Local Demo server |
|
|
- Demo Gallery with 50+ node complex pipelines |
|
|
- Export to Collibra, Purview, and Apache Atlas |
|
|
- Interactive Mermaid visualizations with zoom and download |
|
|
|
|
|
### Social Media Post |
|
|
|
|
|
📱 **LinkedIn**: [View the announcement post](https://www.linkedin.com/posts/aamanlamba_lineage-graph-accelerator-a-hugging-face-activity-7400658296166297600-n9a6) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔜 Roadmap |
|
|
|
|
|
- [x] Gradio 6 upgrade for enhanced UI components |
|
|
- [x] Agentic chatbot for natural language queries (Google Gemini) |
|
|
- [x] Apache Atlas export support |
|
|
- [ ] File upload functionality |
|
|
- [x] Graph export as PNG/SVG |
|
|
- [ ] Batch processing API |
|
|
- [ ] Column-level lineage |
|
|
|
|
|
--- |
|
|
|
|
|
## 🤝 Contributing |
|
|
|
|
|
Contributions welcome! Please: |
|
|
|
|
|
1. Fork the repository |
|
|
2. Create a feature branch |
|
|
3. Make your changes |
|
|
4. Submit a pull request |
|
|
|
|
|
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📄 License |
|
|
|
|
|
MIT License - see [LICENSE](LICENSE) for details. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🙏 Acknowledgments |
|
|
|
|
|
- **Anthropic** - MCP Protocol and Claude |
|
|
- **Gradio Team** - Amazing UI framework |
|
|
- **HuggingFace** - Hosting and community |
|
|
- **dbt Labs** - Inspiration for metadata standards |
|
|
- **OpenLineage** - Open lineage specification |
|
|
|
|
|
--- |
|
|
|
|
|
## 📞 Support |
|
|
|
|
|
- **Documentation**: [USER_GUIDE.md](USER_GUIDE.md) |
|
|
- **Author Website**: [aamanlamba.com](https://aamanlamba.com) |
|
|
- **Issues**: [GitHub Issues](https://github.com/aamanlamba/lineage-graph-accelerator/issues) |
|
|
- **Discussion**: [HuggingFace Community](https://huggingface.co/spaces/aamanlamba/Lineage-graph-accelerator/discussions) |
|
|
|
|
|
--- |
|
|
|
|
|
<p align="center"> |
|
|
Built with ❤️ by <a href="https://aamanlamba.com"><strong>Aaman Lamba</strong></a> for the <strong>Gradio Agents & MCP Hackathon - Winter 2025</strong> |
|
|
<br> |
|
|
Celebrating MCP's 1st Birthday! 🎂 |
|
|
</p> |
|
|
|
