In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost O(1) inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at this https URL.

  • kakes@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    They mention the possibility of parallelization in training. Is this something that could allow (or lead to allowing) distributed training? Something like Folding@Home for LLMs?

    If so, I’m beyond excited. I honestly think that’ll be a major step forward in the democratization of AI, if we can crowdsource training.

    • noneabove1182@sh.itjust.worksOPM
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      That’s definitely a nifty idea, we’ve got people getting distributed inferencing, I can’t see why we couldn’t do something similar for training, especially if we learn better ways to combine training samples

  • Yahma@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Its about time we start looking into alternatives to the transformer model. We are starting to come up on the limits of what is capable with local hardware and there are obvious missing components to the transformer model.

    • Kerfuffle@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Its about time we start looking into alternatives to the transformer model.

      People have been looking into alternatives. If you read the paper, you can see that they compare their approach to a bunch of different alternatives/modifications. Naturally they claim it comes out looking very favorable, but we’ll have to wait and see if the models/code they release actually perform as well as they’re saying and non-obvious downsides.

      It’s not an easy thing to get right.

  • rufus
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    Nice. Seems like it outperforms everything prior in every way possible. Now we have to see if that still holds true at scale. I’d love to see that trained with a trillion tokens and more than 7b parameters, compared to an fully optimized and quantized LLaMA implementation.