Unfortunately, the AI community prefers rushed buggy development over proper, tested releases, so the quants and maybe the PR weren’t fully working.
As of 3 hours ago, unsloth was still updating their quants and guide. I don’t have time to test now but I wouldn’t judge the base model performance in the first few days when the bugs are still being worked out.
They also recommend some unconventional parameters in the Unsloth guide.
It could also be that the model is truly shit of course.
Edit I just took a look at the llama.cpp repo and there are still issues with the implementation as well.
Unfortunately, the AI community prefers rushed buggy development over proper, tested releases, so the quants and maybe the PR weren’t fully working.
As of 3 hours ago, unsloth was still updating their quants and guide. I don’t have time to test now but I wouldn’t judge the base model performance in the first few days when the bugs are still being worked out.
They also recommend some unconventional parameters in the Unsloth guide.
It could also be that the model is truly shit of course.
Edit I just took a look at the llama.cpp repo and there are still issues with the implementation as well.