Job Description
About the Role
We are seeking an experienced Senior Databricks Data Quality Engineer to lead the design, implementation, and automation of enterprise-scale data quality frameworks within a Databricks environment. The successful candidate will play a key role in establishing data quality controls, profiling frameworks, remediation processes, and AI-assisted quality monitoring across a large-scale data platform consisting of 170+ datasets and over 1,300 Critical Data Elements (CDEs).
This role requires strong expertise in Databricks, PySpark, Delta Lake, MLflow, and modern data quality management practices.
Key Responsibilities
Data Platform & Databricks Configuration
- Configure and manage Databricks workspaces, compute clusters, PySpark notebooks, Delta Lake architecture, and Unity Catalog integrations.
- Design scalable data quality processing frameworks across 170+ datasets and 1,346 prioritized Critical Data Elements (CDE...