The biggest issue will be your VRAM. If you don’t have enough of it (which is very likely, even the 8B models I use need ~10gb), you’ll have to use a GGUF model which will need to use your system RAM and CPU for the parts that don’t fit in the VRAM, which will heavily slow it down.
The biggest issue will be your VRAM. If you don’t have enough of it (which is very likely, even the 8B models I use need ~10gb), you’ll have to use a GGUF model which will need to use your system RAM and CPU for the parts that don’t fit in the VRAM, which will heavily slow it down.