Hi, I’m currently starting to learn how LLM works in depth, so I started using nanoGPT to understand how to train a model and I’d like to play around with the code a little more. So I set myself a goal to train a model that can write basic French, it doesn’t to be coherent or deep in its writing, just French with correct grammar. I only have a laptop that doesn’t have a proper GPU, so I can’t really train a model with billions of parameters. Do you think it’s possible without too much dataset or intensive training? Is it a better idea if I use something different from nanoGPT?
TLDR: I’d like to train my own LLM on my laptop which doesn’t have a GPU. It’s only for learning purpose, so my goal is that it can write basic French. Is it doable? If it is, do you have any tips to make this easier?
I believe the answer is, unfortunately, no.
Long answer: In the past, an ML researcher trying to do this would have used either manual labels (for example a dictionary of parts of speech for each word) or multiple sub-models trained to solve each sub-problem before combining into a full prediction model, and even then performance is not great.
However, once the models grew to billions of parameters it turned out that none of this external linguistic knowledge is necessary and the model can learn it all on its own. But it takes billions to trillions of examples to learn all these weights, which means a double hit to the training time: each step is slower due to more parameters, and more steps are needed to train on the full dataset.
None of these models are trainable without a cluster of GPUs, which massively parallelizes the training process.
That doesn’t mean you can’t try, but my results training a small toy model from scratch for 20-30 hours on a consumer GPU have been underwhelming. You get some nearly-grammatical sentences but also a lot of garbage, repetition, and incoherence.
That seems pretty disappointing. It seemed to me like it could have been somewhat possible. I’ve trained a 0.8M parameters model and it was spitting out something that looked like French, not French though. So I need to test it but I feel like if I do it with some millions of parameters it could work. It still wouldn’t have a coherence but at least it could form real sentences. Again I don’t know much about this, so I’m surely wrong. I also think the dataset may be the issue, I didn’t use a general purposed dataset, only French books in a txt file.
How did you determine the dataset size? I mean if it’s just a few megabytes of French books, I’m not surprised you don’t get any results out of that. And it also depends how you feed it in and what parameters you choose for training and model architecture. There are several scientific papers researching for example the needed dataset size to corresponding parameter count of the model.
Once you choose the correct dataset size, have a look at your loss graphs. Do they converge? Did you run training long enough? I suppose it should take weeks (to months?) on an (old) laptop CPU before you see any results, even at that model size.