Research
Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models
arXiv:2307.12896v5 Announce Type: replace Abstract: The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. T
arXiv:2307.12896v5 Announce Type: replace Abstract: The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions. The first one is the standard urn model which predicts that marginal frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text. The second assumption posits that the hapax rate is a simple function of the text length. Four such functions are discussed: the constant model, the cancelation model, the linear model, and the logistic model. As a simple illustration, it is shown that the logistic model yields the best fit for a sample of 14 texts in English. The need and the availability of more complex mixture models that reflect two-regime vocabularies for larger corpora is also discussed.
Related
Source: arXiv cs.CL | 2026-08-20