Job Description
Primary Responsibilities:
- Design, develop, and implement scalable data pipelines using Azure Databricks
- Develop PySpark-based data transformations and integrate structured and unstructured data from various sources
- Optimize Databricks clusters for performance, scalability, and cost-efficiency within the Azure ecosystem
- Monitor, troubleshoot, and resolve performance bottlenecks in Databricks workloads
- Manage orchestration and scheduling of end to end data pipeline using tool like Apache airflow, ADF scheduling, logic apps
- Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise...