Job Description
Job Description
Must Have Technical/Functional Skills
AWS Data Engineering Services (EMR/Glue,Redshift,Aurora, S3, Lambda), Spark, Python, Collibra, Snowflake/Databricks, Tableau.
Roles & Responsibilities
Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python.
Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks.
Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access.
Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, ...