Job Description
Responsibilities
- Design, develop, and maintain scalable data pipelines using Python, PySpark, and AWS Glue, EMR, Step function.
- Build and optimize distributed data processing workflows using EMR and cluster programming techniques.
- Develop Airflow DAGs to orchestrate complex ETL pipelines.
- Work extensively with AWS services to build end-to-end data engineering solutions.
- Ensure reliability, performance, and scalability of all data workflows.
- Troubleshoot production issues quickly and implement corrective actions.
- Collaborate with cross-functional teams to deliver high-quality data products.
- Independently drive deliverables with minimal supervision.