Data & Databricks Test Automation Engineer

Citco, a global leader in financial services, is seeking a Data & Databricks Test Automation Engineer to ensure the quality and reliability of their data solutions. The role involves designing and implementing automated testing frameworks for Databricks-based solutions, data pipelines, and data quality validation, working closely with data engineering teams. This involves testing Databricks notebooks, data pipelines, and model tracking while implementing integration tests with AWS and Databricks-AWS integrations.

Requirements

• Design and implement automated testing for Databricks notebooks and workflows
• Create test frameworks for Delta Lake tables and ACID transactions
• Develop automated validation for structured streaming pipelines
• Test MLflow integrations and model tracking
• Validate Delta Live Tables implementations
• Automate testing for ETL/ELT processes in Databricks
• Implement Spark job testing and optimization validation
• Create test cases for data ingestion and processing workflows
• Develop automated checks for data transformations
• Test Unity Catalog features and access controls
• Implement automated testing for Databricks-AWS integrations
• Create test cases for S3, Glue catalog, and Lambda functions
• Validate data lake storage and access patterns
• Test secure networking configurations
• Design and execute data quality test strategies
• Implement automated data reconciliation processes
• Create performance testing for large-scale Spark jobs
• Create cluster configuration testing
• Implement pipeline monitoring test frameworks
• Create automated test dashboards
• Generate quality metrics and testing reports
• Maintain comprehensive test documentation