Job Description
Key Responsibilities: Design, build, and maintain robust ETL/ELT data pipelines using Apache Spark on Databricks. Implement data Lakehouse architecture using Delta Lake for cost-effective data storage and analytics. Use Databricks Workflows for orchestrating batch and streaming pipelines. Develop and maintain CI/CD pipelines for data applications using tools such as Azure DevOps, GitHub Actions or Databricks Repos. Monitor pipeline performance and troubleshoot data issues in real-time and batch environments. Document solutions, workflows, and technical standards. Required Skills: Experience in data engineering with a strong focus on Databricks and Apache Spark. Proficiency in PySpark, SQL, and Python. Experience with Delta Lake, Databricks SQL, and Unity Catalog. Hands-on experience with cloud platforms Familiarity with data lakehouse architecture, data warehousing and streaming data Strong understanding of ETL best practices, data partitioning, and performance tuning. Experience with ...