LocalLLaMA

2151 readers

1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago

MODERATORS

SkySyrup@sh.itjust.works

pax@sh.itjust.works

noneabove1182@sh.itjust.works

How usable are AMD GPUs? (lemmy.dbzer0.com)

submitted 1 year ago* (last edited 1 year ago) by RandomLegend@lemmy.dbzer0.com to c/localllama@sh.itjust.works

3 comments fedilink hide all child comments

Heyho, I'm currently on a RTX3070 but want to upgrade to a RX 7900 XT

I see that AMD installers are there, but is it all smooth sailing? How well do AMD cards compare to NVidia in terms of performance?

I'd mainly use oobabooga but would also love to try some other backends.

Anyone here with one of the newer AMD cards that could talk about their experience?

EDIT: To clear things up a little bit. I am on Linux, and i'd say i am quite experienced with it. I know how to handle a card swap and i know where to get my drivers from. I know of the gaming performance difference between NVidia and AMD. Those are the main reasons i want to switch to AMD. Now i just want to hear from someone who ALSO has Linux + AMD what their experience with Oobabooga and Automatic1111 are when using ROCm for example.

top 3 comments

sorted by: hot top controversial new old

[–] EddyBot@feddit.de 0 points 1 year ago (1 children)

You probably want to use the amd driver out of the box on your linux distro + ROCm instead of whatever AMD gives you as a driver download on their landing page

Gaming wise the AMD card would win in rasterization performance but PyTorch is made for CUDA (Nvidia only) first instead of OpenCL/HID (which AMD uses)
I couldn't get my AMD card to run reliable in half-precision (16fp) which actually hurts performance A LOT in comparison to no-half or 32fp

interestingly enough setting up AMD cards on Linux with ROCm is actually easier compared to Windows

anyway my experience is mostly stable difussion and some early gpt4all stuff but oobabooga uses PyTorch too so its probably similar

[–] RandomLegend@lemmy.dbzer0.com 0 points 1 year ago (1 children)

I've had AMD cards my whole life and only switched to NVidia 3 years ago where that whole local LLM and ImageAI thing wasn't even on the table...now i am just pissed that NVidia gives us so little VRAM to play with unless you pay the same price as used car -.-

AMD drivers are available from within the kernel so yeah, i won't do any downloading for AMD drivers on Linux^^

Oobabooga and Automatic1111 are my main questions - i could actually live with a downgrade in terms of performance if i then atleast can run the bigger models due to having way more VRAM. Can't even run 17b models on my current 8GB VRAM card...can't even make 1024x1024 images on Auto1111 without getting Issues aswell. If i can do those things but a bit slower, thats fine for me^^

[–] EddyBot@feddit.de 0 points 1 year ago

I only ever used 7b large language models on my RX 6950 XT but PyTorch had or still has some nasty AMD VRAM bugs which didn't fully utilized all of my VRAM (more like only a quarter of it)

it seems the sad truth is high performance/training of models are just not good on AMD cards as of now