And if so, how do they label headphones, contact lenses etc?

  • nandeEbisu@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    9 months ago

    They don’t process words as unified tokens for something like an LLM, but they do process them as multi-letter encoding, like byte-pair encoding or more advanced techniques.