在家中使用日常设备运行自己的 AI 集群。由 exo labs 维护。

Alex Cheema dd882fe6bc experimental notice hace 1 año
exo 5bbde22a23 move everything under exo module hace 1 año
.gitignore 850b72d3ea make StatefulShardedModel callable, add some tests for mlx sharded inference hace 1 año
README.md dd882fe6bc experimental notice hace 1 año
example_user.py 5bbde22a23 move everything under exo module hace 1 año
example_user_2.py 5bbde22a23 move everything under exo module hace 1 año
main.py 32f2e36fd3 main rename hace 1 año
main_static.py 32f2e36fd3 main rename hace 1 año
requirements.txt c851644a43 update requirements, specify exact versions hace 1 año

README.md

exo logo exo: Run your own AI cluster at home with everyday devices. Maintained by [exo labs](https://x.com/exolabs_).

[Homepage](https://github.com/exo-explore/exo) | [Documentation](https://github.com/exo-explore/exo/docs) | [Discord](TODO)


Forget NVIDIA, unify your existing devices into one powerful GPU: iPhone, iPad, Android, Mac, Linux, pretty much any device!

Get Involved

exo is experimental software. We welcome contributions from the community.

We have a list of bounties in this sheet.

Features

Wide Model Support

exo supports LLaMA and other popular models.

Dynamic Model Partitioning

exo optimally splits up models based on the current network topology and device resources available. This enables you to run larger models than you would be able to on any single device.

Automatic Device Discovery

exo will automatically discover other devices using the best method available. Zero manual configuration.

ChatGPT-compatible API

exo provides a ChatGPT-compatible API for running models. It's a one-line change in your application to run models on your own hardware using exo.

Installation

The current recommended way to install exo is from source.

From source

git clone https://github.com/exo-explore/exo.git
cd exo
pip install -r requirements.txt

Documentation

Example Usage on Multiple Devices

Device 1:

./run.sh

Device 2:

./run.sh

That's it! No configuration required - exo will automatically discover the other device(s).

A ChatGPT-like web interface will be available on each device on port 8000 http://localhost:8000.

An API endpoint will be available on port 8001. Example usage:

curl -X POST http://localhost:8001/api/v1/chat -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "What is the meaning of life?"}]}'

Inference Engines

exo supports the following inference engines:

Networking Modules