this post was submitted on 16 Jul 2024
32 points (97.1% liked)
Hardware
625 readers
222 users here now
All things related to technology hardware, with a focus on computing hardware.
Rules (Click to Expand):
-
Follow the Lemmy.world Rules - https://mastodon.world/about
-
Be kind. No bullying, harassment, racism, sexism etc. against other users.
-
No Spam, illegal content, or NSFW content.
-
Please stay on topic, adjacent topics (e.g. software) are fine if they are strongly relevant to technology hardware. Another example would be business news for hardware-focused companies.
-
Please try and post original sources when possible (as opposed to summaries).
-
If posting an archived version of the article, please include a URL link to the original article in the body of the post.
Some other hardware communities across Lemmy:
- AMD
- Augmented Reality
- Gaming Laptops
- Laptops
- Linux Hardware
- Mechanical Keyboards
- Microcontrollers
- Monitors
- Retro Computing
- Single Board Computers
- Virtual Reality
Icon by "icon lauk" under CC BY 3.0
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
My question would be why do I need to run "30B parameter models at 100 tokens per second" on my PC?
I understand the benefits of running things locally, but why not just use Google's or OpenAI's LLM? You shouldn't be sharing sensitive information with such tools in the first place, so that leaves level low-impact business queries and random "lifestyle" queries. Why wouldn't I use cloud infrastructure for such queries?
I understand the benefits of cutting down sugar, but why not just binge on cake and ice cream?
Sounds like you don’t understand the benefits of running things, and specifically LLMs and other kinds of AI models locally.
So what are the benefits with respect to local LLMs in the context I described?
If you’re doing it locally, more sensitive queries become ok, because that data is never leaving your computer…..
Even when you’re not sending data that you consider sensitive, it’s helping train their models (and you’re paying for it!).
Also what’s not sensitive to one person might be extremely sensitive to another.
Also something you run locally, by definition, can be used with no Internet connection (like writing code on a plane or in a train tunnel).
For me as a consultant, it means I can generally use an assistant without worrying about privacy policies on the LLM provider or client policies related to AI and third parties in general.
For me as an individual, it means I can query the model away without worrying that every word I send it will be used to build a profile of who I am that can later be exploited by ad companies and other adversaries.