Job Description
We are seeking an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. The role requires strong hands-on experience in big data processing, pipeline optimization, and deployment using modern data engineering tools and frameworks.
Key Responsibilities
Design, develop, and maintain robust, scalable data pipelines using Python and PySpark
Perform data ingestion, transformation, cleansing, and validation across structured and unstructured datasets
Conduct Exploratory Data Analysis (EDA) to identify data patterns, anomalies, and quality issues
Apply data imputation techniques, data linking, and cleansing to ensure high data quality
Implement feature engineering pipelines to support analytics and downstream use cases
Optimize Spark jobs for performance, scalability, and cost ...