Petals lets users run large LLMs at home via BitTorrent-style distributed inference
Petals, a project from the BigScience research workshop, enables running large language models like Llama 3.1 (up to 405B), Mixtral, Falcon, and BLOOM on consumer GPUs or Google Colab by distributing model parts across a peer-to-peer network. It supports fine-tuning, custom sampling, and access to hidden states, with single-batch inference speeds up to 6 tokens/sec for Llama 2 70B and 4 tokens/sec for Falcon 180B. The project is featured on Hacker News as of July 23, 2026.
Coverage timeline
Hacker Newssnorbleck
Run large language models at home, BitTorrent‑style * Generate text with **Llama 3.1** (up to 405B), **Mixtral** (8x22B), **Falcon** (40B+) or **BLOOM** (176B) and fine‑tune them for your tasks — using a consumer-grade GPU or Google Colab. * You load a part of the model, then join a network of people serving its other parts. Single‑batch inference runs at up to **6 tokens/sec** for **Llama 2** (70B) and up to **4 tokens/sec** for **Falcon** (180B) — enough for chatbots and interactive apps. * Beyond classic LLM APIs — you can employ any fine-tuning and sampling methods, execute custom paths through the model, or see its hidden states. You get the comforts of an API with the flexibility of **PyTorch** and 🤗**Transformers**. **Top contributors** right now: Loading... Follow development in Discord or via email: We send updates once a few months. No spam. We sent you an email to confirm your address. Click it and you're in! Featured on: This project is a part of the BigScience research wo