Tutorials
So cool to see that open-source, with open experimentation (and with the help of someone posting blog posts about their personal research), …
So cool to see that open-source, with open experimentation (and with the help of someone posting blog posts about their personal research), can yield a very robust method for MoE balancing. This metho
So cool to see that open-source, with open experimentation (and with the help of someone posting blog posts about their personal research), can yield a very robust method for MoE balancing. This method seems more elegant than all other methods I have seen. Open source is Awesome! Marin is using quantile balancing from @Jianlin_S (who developed RoPE, which was also a good idea) to train our current 1e23 FLOPs MoE. The idea is elegant: assigning tokens to experts by solving a linear program. No hyperparameters to tune. Yields stable training.
Related
- We @TeraflopAI have worked together with @johngfriedman and @daftengine to open-sourced all major filings from SEC EDGAR completely for free…
- It really took me a while to 'get it' when it comes to nbdev. But I gotta hand it to @jeremyphoward this way of working makes too much sense…
- Introducing DDTree: accelerates speculative decoding by drafting a tree with one block diffusion pass, then verifying multiple likely contin…
- Google just casually disrupted the open-source AI narrative…
Source: Jeremy Howard (X) | 2026-04-17