Job Description
Key Responsibilities:
- Automate data quality and reconciliation checks across varied storage layers, including Snowflake, SQL, and RDF/SPARQL databases
- Test and verify data lineage, governance, and visualization components using Snowflake, data catalogs (ie. DataHub), Thoughtspot, and other visualization tools
- Integrate test suites into the core infrastructure orchestrated by Apache Airflow and utilizing Iceberg table formats, while monitoring data pipeline health, alerting, and observability metrics using Prometheus and Grafana Cloud
- Establish AI Evaluation Loops (Evals) and Guardrails: Build rigorous verification protocols— including structural tests, checks, and watchdog agents—to validate AI-generated artifacts, catch false positives, and ensure all automated outputs are secure, reliable, and free from hallucinations.
- Integrate automated testing workflows into CI/CD pipelines using GitHub Actions, ensuring continuous stabili...