We are looking for a Senior Data Engineer with strong technology and data understanding in big data engineering space, having proven delivery capability. The ideal candidate will have 3 to 7 years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases, NoSQL, and data warehouse solutions.
Requirements
- Develop & deploy Azure Databricks in a cloud environment using Azure Cloud services
- Configure and replicate the Azure Databricks platform, including workspaces, clusters, and storage based on existing Hadoop workloads.
- Migration of Hadoop jobs (YARN, SPARK), ETL workflows (Sqoop scripts, Hive), Kafka streams, Impala queries and database objects to Azure Databricks, ensuring efficient data cleaning, transformation, and loading.
- Set up and manage workflow orchestration using Databricks Jobs Scheduler, handling dependencies and scheduling of data processing tasks.
- Implement monitoring solutions to track the health and performance of the Databricks environment, configuring alerting mechanisms for proactive issue resolution.
- Ensure all code is maintained in the existing Git repository, undergoing peer reviews to maintain quality and readability.
- Conduct System Integration Testing (SIT) and QA testing parallel to development, ensuring all critical workflows and reports function correctly.
- Document and address any issues identified during testing, compiling detailed reports for stakeholders.
- Implement the cutover strategy and production migration according to the deployment plan, communicating with stakeholders and ensuring non-restricted hours migration.
- Implement data validation checks to confirm the integrity and completeness of the migrated data.
- Document the deployment process, including encountered issues and their resolutions, and provide stakeholders with information on how to report any issues.
- Provide hyper care support post-deployment, addressing any issues promptly and transitioning to standard operational support as the environment stabilizes.
Benefits
- Support, coaching and feedback from some of the most engaging colleagues around
- Opportunities to develop new skills and progress your career
- The freedom and flexibility to handle your role in a way that’s right for you
