I want to fine tune an LLM to “steer” it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 “steers” on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these “steer” operations has value and could be used for training.

So maybe I’m looking for some kind of localized DPO. Does anyone know if something like this exists?

  • iii@mander.xyzEnglish
    1·
    7 days ago

    Would you call token (N+1), given tokens (1 to N) as a ground truth?

    • hok@lemmy.dbzer0.comOPEnglish
      1·
      7 days ago

      No, in that case there’s no labelling required. That would be unsupervised learning.

      https://en.wikipedia.org/wiki/Unsupervised_learning

      Conceptually, unsupervised learning divides into the aspects of data, training, algorithm, and downstream applications. Typically, the dataset is harvested cheaply “in the wild”, such as massive text corpus obtained by web crawling, with only minor filtering (such as Common Crawl). This compares favorably to supervised learning, where the dataset (such as the ImageNet1000) is typically constructed manually, which is much more expensive.

      • iii@mander.xyzEnglish
        2·
        7 days ago

        So supervised vs unsupervised, according to you, is a property of the dataset?

        • hok@lemmy.dbzer0.comOPEnglish
          11·
          7 days ago

          Sorry, I really don’t care to continue talking about the difference between supervised and unsupervised learning. It’s a pattern used to describe how you are doing ML. It’s not a property of a dataset (you wouldn’t call Dataset A “unsupervised”). Read the Wikipedia articles for more details.