Home Explore Help
Sign In
lqb
/
exo
mirror of https://github.com/exo-explore/exo
1
0
Fork 0
Files Issues 0 Wiki
Tree: 2644fd02c8
Branches Tags
exo
/
exo
/
inference
/
mlx
/
perf_improvements.md

perf_improvements.md 268 B
History Raw

Perf improvements

Target: 460 tok/sec

  • removing sample goes from 369 -> 402
  • performance degrades as we generate more tokens
  • make mlx inference engien synchronous, removing thread pool executor: 402 -> 413
  • remove self.on_opaque_status.trigger_all: 413 -> 418