Ecolab is looking for a Data Engineer to be part of a dynamic team that's at the forefront of technological innovation. We're leveraging cutting-edge AI to create novel solutions that optimize operations for our clients, particularly within the restaurant industry.
Requirements
- Design, develop, and maintain scalable and robust data pipelines on Databricks (Spark SQL, PySpark, Delta Lake).
- Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
- Optimize and troubleshoot existing data pipelines for performance and reliability.
- Ensure data quality and integrity across various data sources.
- Implement data security and compliance best practices.
- Monitor data pipeline performance, implement data quality checks, and conduct necessary maintenance and updates.
- Document data pipeline processes and technical specifications.
- Implement robust pipeline orchestration using tools like Databricks Workflows, dbt, or similar.
- Generate and maintain data quality reports and dashboards.
- Implement Infrastructure as Code (IaC) principles for managing Databricks infrastructure.
Benefits