I’ve got a homelab setup that could benefit from low-power AI acceleration, which could let me run Whisper and distilled models locally and integrate with my serviced ie. home assistant. Plus, the less data I send over my network the happier I’ll be.
I don’t really want to stuff a GPU into my system right now, I dont have much power budget and GPUs can get pricey for the cost of one that’s useful. I’ve seen a few examples of “Edge accelerators” which boast a super tiny (2-5w) power envelope and 40 TOPs, but that doesn’t tell me much about how well models will actually work in practice.
Is there any kind of mapping between TOPs and, say, tokens per second for X model? Maybe recommended TOPs for X model?
Idk about now, but the low powered ai accelerators of olde weren’t meant for that.
The google (nee coral!) ones for example really shined at object recognition but weren’t good for text to text or tts (I didn’t try hard).
If you’re not willing to get a gpu, you’re better off ram maxing and doing stuff like that in cpu.
If you are willing to get a gpu, you can still do what you need using the old ass Maxwell and pascal ones. They’ll be awful at image generation but fine for text.
I also want to carry the good word of not worrying about power consumption to you! It doesn’t matter! Pcs aren’t expensive to run! They have low idle draw! Power is cheap!
If you have to know for sure about the power impact, get a kill-a-watt and plug your shit into it and be confident in your new knowledge.
Yeah I could do that tbh. DDR4 isn’t so bad price-wise…
I’ll see what the lower budget cards of the last few years look like. I’m a lazy sod.
Idle power consumption isn’t a massive issue for me, but I’m more finnicky about it with my NAS as I’d prefer have the cooling and power reserved for my drives (and expansion thereof).