this post was submitted on 24 Jul 2024
62 points (94.3% liked)

Tech

447 readers
1 users here now

A community for high quality news and discussion around technological advancements and changes

Things that fit:

Things that don't fit

Community Wiki

founded 8 months ago
MODERATORS
Vacant@programming.dev
62
The first GPT-4-class AI model anyone can download has arrived: Llama 405B (arstechnica.com)
submitted 3 months ago by sabreW4K3@lazysoci.al to c/tech@programming.dev
10 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] Naz@sh.itjust.works 38 points 3 months ago* (last edited 3 months ago) (3 children)

Anyone can download, but practically no one can run it.

With the absolute highest memory compression settings, the largest model that you can fit inside of 24GB of VRAM is 109 billion parameters.

Which means even with crazy compression, you need at the very least, ~100GB of VRAM to run it. That's only in the realm of the larger workstation cards which cost around $24,000 - $40,000 each, so y'know.

[–] xionzui@sh.itjust.works 9 points 3 months ago* (last edited 3 months ago) (1 children)

You can get usable performance on a CPU with good memory bandwidth. Apple studios are the best way to get that right now, but a good Epyc with 256GB of RAM works too.

Of course, you could also just run 5 GPUs

[–] IsThisAnAI@lemmy.world 4 points 3 months ago

"usable" sure if you want to wait 10 minutes a word.

[–] Varyk@sh.itjust.works 6 points 3 months ago

Thanks, I was wondering