Lead AI Platform Engineer

Ecolab is looking for a Data Engineer to be part of a dynamic team that's at the forefront of technological innovation. We're leveraging cutting-edge AI to create novel solutions that optimize operations for our clients, particularly within the restaurant industry.

Requirements

• Design, develop, and maintain scalable and robust data pipelines on Databricks (Spark SQL, PySpark, Delta Lake).
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Optimize and troubleshoot existing data pipelines for performance and reliability.
• Ensure data quality and integrity across various data sources.
• Implement data security and compliance best practices.
• Monitor data pipeline performance, implement data quality checks, and conduct necessary maintenance and updates.
• Document data pipeline processes and technical specifications.
• Implement robust pipeline orchestration using tools like Databricks Workflows, dbt, or similar.
• Generate and maintain data quality reports and dashboards.
• Implement Infrastructure as Code (IaC) principles for managing Databricks infrastructure.

Benefits

• Full-time job