this post was submitted on 14 Jul 2023
17 points (87.0% liked)

Selfhosted

50923 readers
667 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
17
Can I run local LLMs on Intel ARC/AMD with 8GB of RAM? (lemmy.world)
submitted 2 years ago by MigratingtoLemmy@lemmy.world to c/selfhosted@lemmy.world
16 comments fedilink hide all child comments
 

publication croisée depuis : https://lemmy.world/post/1474932

Hi there.

I wanted to run LLMs locally on my server (for better privacy), and was wondering if:

  1. I could use Intel ARC/AMD GPUs - these are often less expensive and AMD has open source drivers, which is something I like.
  2. If a PCIe x4 Gen 3 slot would be enough (it's an x16 slot with x4 speeds) - this is an important consideration.
  3. Would 8GB of RAM (in the GPU, I believe it's called VRAM?) be enough?

I'm looking at language models to train on my Reddit and Lemmy content, in an aim to make it write like me (and maybe even better than me? Who knows). I don't quite know which models I will train, or how I will do so (I certainly won't be writing anything from scratch), but I was wondering; with the explosion of FOSS AI models, maybe something like this would be possible with the hardware constraints I mentioned above?

Does the speed of the connection between the GPU and the CPU really matter in such applications?

Thanks!

you are viewing a single comment's thread
view the rest of the comments
[–] theterrasque 6 points 2 years ago (7 children)

You can probably run a 7b LLM comfortably in system RAM, maybe one of the smaller 13b ones.

Software to use

Models

In general, you want small GGML models. https://huggingface.co/TheBloke has a lot of them. There are some superHOT version of models, but I'd avoid them for now. They're trained to handle bigger context sizes, but it seems that made them dumber too. There's a lot of new things coming out on bigger context lengths, so you should probably revisit that when you need it.

Each have different strengths, orca is supposed to be better at reasoning, airoboros is good at longer and more storylike answers, vicuna is a very good allrounder, wizardlm is also a notably good allrounder.

For training, there are some tricks like qlora, but results aren't impressive from what I've read. Also, training LLM's can be pretty difficult to get the results you want. You should probably start with just running them and get comfortable with that, maybe try few-shot prompts (prompts with a few examples of writing styles), and then go from there.

[–] MigratingtoLemmy@lemmy.world 0 points 2 years ago (6 children)

Thank you. I did have llama.cpp in mind but didn't know where or how to start!

Do these models have a limit on how much information they can injest and how much they can improve relative to the information fed to them?

[–] theterrasque 1 points 2 years ago (1 children)

Another thing, llama.cpp support offloading layers to gpu, you could try opencl backend for that for non-nvidia gpu's. But llama.cpp can also run on cpu-only, with usable speed. On my system, it does about 150ms per token on a 13b model.

koboldcpp is probably the most straight forward to get running, since you don't have to compile, it has a simple UI to set launch parameters, and it also have a web ui to chat with the bot in. And since it use llama.cpp it support everything that does, including opencl (clblast in launcher)

[–] MigratingtoLemmy@lemmy.world 1 points 2 years ago

Thanks, I'll take a look

load more comments (4 replies)
load more comments (4 replies)