LLM Training Resilience Engineer
Together AISan Francisco, California, United States
Posted August 12, 2025
160000 - 230000 USD year
Full Time
About the Company
Together AI is a research-driven AI cloud infrastructure provider offering a purpose-built GPU platform for training and running advanced AI models. Serving leading SaaS companies and pioneering startups, Together AI champions open source AI and decentralized computing, advocating for transparency to drive innovation and societal benefits.
Job Description
Together.ai is seeking a Large-scale Training Resilience Engineer to ensure the reliability, fault tolerance, and scalability of their large-scale training infrastructure. The role focuses on designing and implementing resilience strategies, optimizing error detection, and maintaining observability systems. This is a hands-on position for individuals passionate about solving complex distributed systems problems.
Requirements
- 5+ years of experience in distributed systems, cloud infrastructure, or large-scale machine learning training.
- Proficiency in distributed computing frameworks (e.g., PyTorch DDP, TensorFlow, Horovod).
- Strong knowledge of resilience strategies in distributed systems (e.g., leader election, consensus, retry mechanisms).
- Hands-on experience with observability tools (e.g., Prometheus, Grafana, ELK stack).
- Proficient in Python, Go, or a similar programming language.
- Experience working with cloud platforms (e.g., AWS, GCP, Azure) and Kubernetes for workload orchestration.
- Strong analytical, problem-solving, and debugging skills.
Benefits
- Competitive compensation
- Startup equity
- Health insurance