Senior Data Engineer - Databricks

MediaRadar, an industry leader in marketing intelligence, is seeking a Senior Data Engineer to join their data team. The role involves building and maintaining scalable, high-performance data solutions using Azure Databricks, Apache Spark, and modern data lakehouse architectures.

Requirements

• Design, development, and maintenance of scalable ETL/ELT pipelines on Azure Databricks using Apache Spark (PySpark/Spark SQL).
• Design and implement both batch and real-time data ingestion and transformation processes.
• Build and manage Delta Lake tables, schemas, and data models to support efficient querying and analytics.
• Consolidate and process large-scale datasets from various structured and semi-structured sources (e.g., JSON, Parquet, Avro).
• Write optimized SQL queries for large datasets using Spark SQL and PostgreSQL.
• Develop, schedule, and monitor workflows using Databricks Workflows, Airflow or similar orchestration tools.
• Design, build, and deploy cloud-native, containerized applications on Azure Kubernetes Service (AKS) and integrate with Azure services.
• Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
• Collaborate with data analysts, data architects, and business stakeholders to translate requirements into technical solutions.
• Contribute to and enforce best practices in data engineering, including version control (Git), CI/CD pipelines, and coding standards.
• Continuously enhance data systems for improved performance, reliability, and scalability.
• Mentor junior engineers and help evolve team practices and documentation.
• Stay up to date on emerging trends, technologies, and best practices in the data engineering space.
• Work effectively within an agile, cross-functional project team.

Benefits

• Full-time exempt role with base salary plus benefits.
• Equal Opportunity Employer.
• Diversity of backgrounds, perspectives, and experiences is valued and encouraged.