Job Description
The Cloud Platform Engineer focuses on building and operating the environment and ensures a reliable, well-governed, high-performance platform. This role also owns infrastructure, access, tooling, observability, and standards across the full stack.
Main Accountabilities
- Provisions, configures, and tunes Databricks workspaces, Unity Catalog, and Apache Spark cluster settings (memory, executors, autoscaling) for automated, cost-efficient compute.
- Defines and enforces Spark usage standards, query patterns, resource quotas, and Delta Table partitioning guidelines to optimize storage and prevent cluster contention.
- Manages S3 bucket structures, lifecycle policies, access controls, and overall storage cost governance.
- Administers and tunes Airflow environments (upgrades, plugins, monitoring), enabling automated DAG authoring and workflow management.
- Owns IAM roles, policies, and least-privilege access management acro...