this post was submitted on 23 Jul 2024
48 points (83.3% liked)
Technology
75324 readers
3611 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
IMO the more interesting models are 70B and 8B, aka the first models you can host yourself and (for basically the first time) the first open models distilled from such a large "parent" model.
But the release is a total dud among testers because they're bugged with llama.cpp, lol.
I've got llama 3.1 8b running locally in open webui. What do you mean it's bugged with llama.cpp?
llama.cpp, the underlying engine, doesn't support extended RoPE yet. Basically this means long context doesnt work and short context could be messed up too.
I am also hearing rumblings of a messed up chat template?
Basically with any LLM in any UI that uses a GGUF, you have to be very careful of bugs you wouldn't get in the huggingface-based backends. A lot of models run without errors, but not quite right.
I wouldn't call it a "dud" on that basis. Lots of models come out with lagging support on the various inference engines, it's a fast-movibg field.
Yeah, but it leaves a bad initial impression when all the frontends ship it and the users aren't aware its bugged.