Talkie: a 13B vintage language model from 1930

The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point:

> Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks into the training corpus from after the intended knowledge cutoff

But as they acknowledge later, there are multiple major data leakage issues in their training pipeline, and their model does in fact have quite a bit of anachronistic knowledge. So it fails at what they call the most important objective. It's fair to say that they are working toward something that meets their definition of "vintage", but they're not there yet.

> A language model trained from scratch exclusively on data from certain places and time periods to reduce modern bias and emulate the voice, vocabulary, and worldview of the era.

Discussed here: https://news.ycombinator.com/item?id=46590280

https://lifearchitect.ai/models-table/

This one is easiest to talk to in a HF space:

https://huggingface.co/spaces/tventurella/mr_chatterbox

It's going to be more like corresponding with someone from the past. We don't have much in the way of recorded speech from that area, so this will be built from written records. Much more than now, the written records are going to be formal and edited, reflecting a different pattern than casual speech or writing.

Having said that, this is cool. I recently had to OCR a two-hundred year old book with the usual garish fonts from that era. It was remarkably easy to do, and accurate.

Fun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1].

Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible.

[1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [video]:

https://youtu.be/yolkEfuUaGs

Post World War 2, some people had the odds per year at 10%. Some of that is probably a mix of recency bias + not understanding how to use new weapons etc etc but as Silver points out, the odds were much lower.

I mention this only b/c the "could something trained on LLMs of the time predict the future" always makes me think of it.

Tell me about Winston Churchill:

> Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constituency ever since. Mr. Churchill has written a number of books, including “The Story of the Malakand Field Force,” “Savrola,” “Richard Carvel,” “The Celebrity,” and “The Crisis.” He has also contributed to several periodicals, and in 1900 he founded the monthly review, _The J Cornhill Magazine_. Mr. Churchill is an ardent sportsman, and has shot big game in Africa. He married, in 1897, Lady Randolph Churchill, and has two sons and a daughter. Politically, he is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs. At present he is Chancellor of the Duchy of Lancaster, with a seat in the Cabinet. Mr. Churchill has achieved considerable success as a public speaker, and he is described as an eloquent and forcible debater. His residence is at 42, Grosvener Place, London, S.W.

The colonialism is... wow.. Tell me about the likelihood of independence of India:

> The chances are undoubtedly in favour of the establishment of an independent Indian state in the not very distant future. The unifying influences of railways and a common language are rapidly breaking down the barriers of caste and creed, which have hitherto kept the great Indian peninsula politically disunited, and the spread of western education is awakening a national spirit among the people. The immediate result of the latter is seen in the establishment of native newspapers, which voice popular feeling, and in the growth of associations for social and political reform. More important still, as showing the trend of public opinion, are the resolutions passed at great national congresses, which have been held annually for the last dozen years. By these gatherings, which representatives of all classes and creeds assemble to discuss matters of social and political interest, a strong impulse has been given to the movement for reform, and the desires of the more advanced party among the natives have been plainly formulated. The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy, and which shall legislate for and administer the internal affairs of the country, subject to the control of the imperial legislature at Westminster. The wish is also expressed that the queen should assume the title of empress of India, and that a certain number of natives should be admitted to the civil and military services of the state. Finally, it is claimed that the time has come when Her Majesty may wisely be advised to delegate to the Indian people a larger share in the work of governing themselves, by permitting them to elect a portion of the members of the legislative councils. How far the present generation of Indians may be trusted to exercise political power with prudence and moderation, it is impossible to say; but there can be no doubt that the time must arrive when the control of Indian affairs will be safely lodged in native hands. The process may be hastened or retarded, but come it must. The spread of enlightenment among the great mass of the population can only have one issue, and that issue is the establishment of an Indian nationality. The probability of such an event may therefore be regarded as certainty.