Senior Data Engineer (Data Bricks)
AllataPune, Maharashtra, India • New York, New York, United States • São Paulo, Brazil
Posted December 17, 2024
Full Time
About the Company
Allata is a strategy, architecture, and enterprise-level application development firm focused on enhancing or scaling business opportunities through custom technologies. We specialize in strategic advisory services, transformational program management, and emerging technologies like AI, IoT, and cloud computing to help businesses become more agile and disruptive. Our company culture emphasizes family, client satisfaction, and community involvement.
Job Description
Allata is a global consulting and technology services firm seeking a skilled Data Engineer to guide the evolution of client data ecosystems. The role focuses on architecting, building, and implementing robust data solutions, mentoring team members, and collaborating with clients to achieve analytics goals. Allata utilizes a variety of advanced technologies, including cloud platforms and AI, to create engaging digital experiences.
Requirements
- Proven expertise in Data Bricks, Delta Live Tables, SQL, and Pyspark for processing and managing large data volumes.
- Strong experience in designing and implementing dimensional models and medallion architecture.
- Strong experience in designing and migrating existing databricks workspaces and models to Unity Catalog enabled workspaces.
- Strong Experience creatinging and managing group Access Control Lists (ACL) and compute and governance policies in Databricks Unity Catalog.
- Hands-on experience with modern data pipeline tools (e.g. AWS Glue, Azure Data Factory) and cloud platforms (e.g. Databricks).
- Knowledge of cloud data lakes (e.g., Data Bricks Delta Lake, Azure Storage and/or AWS S3).
- Demonstrated experience applying DevOps principles using Version Control and CICD for IaC and code base deployments(e.g. AzureDevops, Git, CI/CD).