this post was submitted on 24 Jul 2024
197 points (92.6% liked)
Technology
74854 readers
3549 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
What’s the resources requirements for the 405B model? I did some digging but couldn’t find any documentation during my cursory search.
405b ain't running local unless you got a proepr set up is enterpise grade lol
I think 70b is possible but I haven't find anyone confirming it yet
Also would like to know specs on whoever did it
I've run quantized 70B models on CPU with 32 gigs but it is very slow
I gonna add some RAM with hope I can split original 70b between GPU and RAM. 8b is great what it is as is
Looks like it should be possible, not sure how much performance hit offloading to RAM will do. Fafo