There's a broadly held misconception in AI that methods that scale well are simple methods -- even, that simple methods usually scale. This …

There's a broadly held misconception in AI that methods that scale well are simple methods -- even, that simple methods usually scale. This is completely wrong. Pretty much none of the truly simple methods in ML scale well. SVM, kNN, random forests are some of the simplest methods out there, and they don't scale at all. Meanwhile "train a transformer via backprop and gradient descent" is a very high-entropy method, easily 10x more complex than random forest fitting. But it scales very well. Further, given a simple method that doesn't scale, it is usually the case that you alter it to make it scale by adding a lot of complication. For instance, take a simple a simple combinatorial search-based method (not scalable at all) -- you can make it scale by adding deep learning guidance (which blows up complexity). Scalability usually belongs to high-entropy, complex systems.

Related

• Simplicity is a very strong signal of model quality. Galileo's heliocentric model was directionally correct, but its predictive power was qu…
• The reason symmetry is so important in physics is because symmetry is a highly effective compression operator. If a system is invariant unde…
• Science needs a way to process models that are only 'mostly correct' in terms of their predictions, but are very compressive (high ratio bet…
• Good design is the art of packing 1,000 'hows' into a single 'what'. Good design is compression: making the numerator trend towards infinity…
• We should view the history of physics as a long-running program synthesis task. Kepler and Newton were searching the space of possible symbo…

Source: Francois Chollet (X) | 2026-04-16