Job Description
Job Description:
- Design, develop, and maintain ETL pipelines using Informatica, Azure Data Factory (ADF), and Databricks
- Perform migration of legacy ETL workflows (Informatica) to Databricks using Python/PySpark
- Analyze existing ETL workflows and re-engineer into optimized Spark-based transformations
- Develop data processing and transformation solutions using Python and PySpark
- Apply AI/ML techniques for data enrichment, anomaly detection, and predictive insights (where applicable)
- Build and optimize SQL queries, data models, and transformations
- Schedule and monitor jobs using AutoSys
- Integrate data from multiple sources: Relational databases (SQL Server, DB2), Files (CSV, XML, JSON) and mainframe systems
- Streaming platforms like Kafka
- Perform data validation, reconciliation, and ensure data quality
- Troubleshoot ETL/pipeline failures and optimize performance
- Collaborate...