Job Description
Scope of Work: Data Pipeline Development & Management
- Design, build, and maintain robust data pipelines using AWS Glue
- Implement ETL/ELT processes for data ingestion from multiple sources
- Optimize data workflows for performance and scalability
- Monitor and troubleshoot data pipeline failures and performance issues
- Manage and optimize AWS Redshift data warehouse operations
- Configure and maintain data storage solutions (AWS S3, data lakes)
- Implement data partitioning, indexing, and compression strategies
- Support Infrastructure as Code (IaC) for data infrastructure deployment
- Develop and maintain CI/CD pipelines for data workflows using GitLab
- Implement automated testing for data pipelines and data quality ...