I want to fine tune an LLM to “steer” it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.
Basically, for my dataset doing 100 “steers” on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these “steer” operations has value and could be used for training.
So maybe I’m looking for some kind of localized DPO. Does anyone know if something like this exists?
I dont know what you mean with steering?
- Do you want a given output structure, like json or toml?
- Do you want to align the model, with your dataset of question and answer pairs?
First of all, have you tried giving the model multiple examples of input output pairs in the context, this already helps the model a lot to output the correct format.
Second you can force a specific output structure by using a regex or grammar: https://python.langchain.com/docs/integrations/chat/outlines/#constrained-generation https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md
And third, in case you want to train a model to respond differently and the previous steps were not good enough, you can fine-tune. I can recommend this project to you, as it teaches how to fine-tune a model: https://github.com/huggingface/smol-course
Depending on the size of the model, that you want to fine-tune and the amount of compute that you have available you can either train by updating all parameters like ORPO or you can train via PEFT (LoRA)
Thanks for your answer. I think to be clear, what I’m looking for is a kind of masked fine-tuning. You see, I want to “steer” a particular output instead of providing complete examples, which are costly to create.
The steering would be something like this:
- I have an LLM generate a sequence.
- I find exactly where the LLM goes “off track” and correct it there (for only maybe 10-20 tokens instead of correcting the rest of the generation manually).
- The LLM continues “on track” until it goes off track again.
What I would like to do is train the model based on these corrections I give it, where many corrections might be part of the same overall generation. Conceptually I think each correction must have some training value. I don’t know much about masking, but what I mean here is that I don’t want it to train on a few tens or hundreds of (incomplete) samples but rather thousands of (masked) “steers” that correct the course of the rest of the sample’s generated text.
Here’s a guide: https://medium.com/@yuxiaojian/fine-tuning-llama3-1-and-deploy-to-ollama-f500a6579090
Not sure what the impact of a few 100 examples will be, though, even if you freeze most parameters.
Can SFT be used on partial generations? What I mean by a “steer” is a correction to only a portion, and not even the end, of model output.
For example, a “bad” partial output might be:
<assistant> Here are four examples: 1. High-quality example 1 2. Low-quality example 2and the “steer” might be:
<assistant> Here are four examples: 1. High-quality example 1 2. High-quality example 2but the full response will eventually be:
<assistant> Here are four examples: 1. High-quality example 1 2. High-quality example 2 3. High-quality example 3 4. High-quality example 4The corrections don’t include the full output.
I do not know what SFT means. So I can’t comment on that, I’m afraid.
Models only predict the distribution of the next token. So “partial response” vs “full response” is a consequence repeated inference untill the stop token is reached. It’s mostly unrelated to the model parameters.
For training, it makes no difference.
The article you linked to uses SFT (supervised fine tuning, a specific training technique) as its alignment strategy. There are other ways to fine-tune a model.
I guess I’m wondering if you can train on these partial responses without needing the full rest of the output, without the stop token, or if you need full examples as the article hints to.
I was unaware of that acronym, thank you. It does make me wonder: is there unsupervised training of LLMs?
Yes, you can train without the stop token. The stop token is just that: another token. I do not expect the model to unlearn the usage of the stop token from training on a few 100 new examples.
Unsupervised training happens during the pre-training phase when you dump all kinds of quality documents and it learns the relationship between tokens
Could you perhaps share a reference for this? I’m eager to learn as I don’t quite understand.
I’ve always trained LLM supervised: predict token N+1 based on tokens 1 to N.
This pre-training was done by Meta. It’s what Llama-3.1-405B is (in contrast to Llama-3.1-405B-Instruct). https://huggingface.co/meta-llama/Llama-3.1-405B
Training Data
Overview: Llama 3.1 was pretrained on ~15 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples.