engineering
1 weeks ago*
Software Engineer, Ray Data
at Anyscale
📍 Bengaluru, India·🏢 On-site
Requirements
- With Anyscale, we’re building the best place to run Ray, so that any developer or data scientist can scale an ML application from their laptop to the cluster without needing to be a distributed systems expert.
- About the role: Ray aims to provide a universal API for building distributed applications (e.g. a machine learning pipeline of feature engineering, model training, and evaluation).
- Ray Datasets is a Python library built on top of Apache Arrow and Ray Core (Ray’s C++ backend), and the Ray Data team interacts closely with Ray Core components including the scheduler and the memory & I/O subsystems.
- The Ray Data team also works closely with Ray’s ML libraries including Train, RLlib, and Serve.
- A snapshot of projects you will work on: - Performance of Ray Datasets https://docs.ray.io/en/latest/data/dataset.html at large scale (leveraging Arrow primitives, optimizing Ray object manager, etc.) - Integration with ML training and data sources - Stability and stress testing infrastructure - Lead future work integrating streaming workloads into Ray such as Beam on Ray - Differentiate Data operations in Anyscale hosted Ray service As part of this role,
- Experience in building scalable and fault-tolerant distributed systems -
- Experience with data processing, database internals including Spark or Dask (streaming is a plus) Anyscale Inc. is an Equal Opportunity Employer.
Contact
- About Anyscale: At Anyscale https://www.anyscale.com/, we're on a mission to democratize distributed computing and make it accessible to software developers of all skill levels.
- We’re commercializing Ray https://docs.ray.io/en/latest/, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning.
- Companies like OpenAI https://thenewstack.io/how-ray-a-distributed-ai-framework-helps-power-chatgpt/, Uber https://www.uber.com/blog/horovod-ray/, Spotify https://engineering.atspotify.com/2023/02/unleashing-ml-innovation-at-spotify-with-ray/, Instacart https://www.youtube.com/watch?v=3t26ucTy0Rs&list=PLzTswPQNepXmLUiL4F_1VHrPcCz1OeILw&index=23&pp=iAQB, Cruise https://www.youtube.com/watch?v=gj0BqvfX_wI&list=PLzTswPQNepXmLUiL4F_1VHrPcCz1OeILw&index=46&pp=iAQB, and many more, have Ray in their tech stacks
- Proud to be backed by Andreessen Horowitz, NEA, and Addition https://www.wsj.com/articles/ai-startup-anyscale-adds-99-million-to-andressen-horowitz-led-funding-round-11661254200 with $250+ million raised to date.