This is an interesting demo, but it has some drawbacks I can already see:

  • It’s Windows only (maybe Win11 only, the documentation isn’t clear)
  • It only works with RTX 30 series and up
  • It’s closed source, so you have no idea if they’re uploading your data somewhere

The concept is great, having an LLM to sort through your local files and help you find stuff, but it seems really limited.

I think you could get the same functionality(and more) by writing an API for text-gen-webui.

more info here: https://videocardz.com/newz/nvidia-unveils-chat-with-rtx-ai-chatbot-powered-locally-by-geforce-rtx-30-40-gpus

  • TheFinn
    link
    fedilink
    English
    arrow-up
    6
    ·
    4 months ago

    It works with windows 10 as well; it didn’t like a lot of the older pdfs that were in my documents; and it didn’t work offline. The last bit is probably the python dependencies trying to update themselves, but it doesn’t fail gracefully into a working state when the network isn’t available.

    • OpticalMooseOP
      link
      fedilink
      English
      arrow-up
      6
      ·
      4 months ago

      Thanks for the info. I’m not on Windows, so I can’t try it. Doesn’t work offline? That’s what I was afraid of.

      It’s a great proof of concept. Hopefully we’ll see more text-gen-webui extensions soon.

  • Donut@leminal.space
    link
    fedilink
    English
    arrow-up
    4
    ·
    4 months ago

    It will also be available as an open-source reference project, so developers can easily implement the same capabilities in their own applications.

    Not truly open source but at least you can build your own and control what happens with all the data, if I understand this correctly?

    • General_Effort@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      4 months ago

      It means that they want people to consult the code as a reference for how to best use the hardware acceleration.

      If all software uses their cards to best effect, that makes their cards more useful and thus more valuable; making them money. If only their own frontend can do that, they lose out on most of that, while also having to spend money to make sure that the rest of the software, like the UI, is competitive.