Job Description
Responsibilities:
- Build and optimise scalable data pipelines using PySpark and SQL
- Design and implement ETL/ELT processes for batch and streaming data
- Develop data solutions using Databricks Lakehouse and Delta Lake
- Ingest and integrate data from internal and external sources (e.g. Kafka, CDC)
- Optimise Spark jobs and data workflows for performance, scalability, and cost efficiency
- Manage infrastructure and environments using Terraform (IaC)
- Ensure data quality, monitoring, and reliability
- Implement governance and access controls (e.g. Unity Catalog)
- Deliver clean, structured, and accessible data for analytics and business use
- Collaborate with cross-functional teams to support analytics, reporting, and AI/ML initiatives
Qualifications:
- Demonstrated experience in data engineering, with a proven ability to build scalable data solutions