Local Ai
Overfitting and creativity
I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets. The problem with overfitting a model to the training data is that it becomes
I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets. The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data its predictions are wrong. This is what we see with current models, they are extremely good at predicting, especially code, but can't generalise to real creativity. In theory, a larger dataset should allow to get better predictions on unseen data. This is even more true for neural nets. So my question is: to have a creative model, should a model be trained on a huge dataset, but fitting it less than what is done on current models? Somebody has an idea about training epochs, learning rate, and other parameters of last generation models? Do you think that a model that use a last gen training recipe, but slightly undercooked (or very undercooked) would be better at creative writing and worse at coding? submitted by /u/Expensive-Paint-9490 [link] [comments]
Related
- Were designing a tiny autonomous research agent
- Could we all crowdsource a dataset/model/finetune?
- Making a synthetic dataset for fine-tuning
- How many of you are using Kiwix/offline Wikipedia for training?
Source: r/LocalLLaMA | 2026-10-15