90% of mine is just programming syntax. The rest is shit that Google can't answer anymore. Then 1% is me trying to trick it into telling me about illegal stuff
this post was submitted on 07 Aug 2024
96 points (97.1% liked)
Technology
59612 readers
3232 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
What illegal stuff?
.... Asking for a friend.
Nothing special, I got it to give me a recipe for meth and a list of sites to pirate games. Stuff like that
There are models you can download and run at home that doesn't have the politically correct censorship inside. It's very nice to not have artificial politeness for example, and the models actually answers your actual questions.
You need a powerful computer for some of them though.
Such as? Where would technologically proficient AI-beginner start?
For a user without much technical experience using a ready-made gui like Jan.ai with automatic model download and ability to run models with the ggml library on consumer grade hardware like mac M-series chips or cheap GPUs by either Nvidia or AMD is probably a good start.
For a little bit more technically proficient users Ollama is probably a great choice to start to host your own OpenAI-like API for local models. I mostly run gemma2 or small llama 3.1 like models with that.
I was also kind of blown away by the Firefox nightly version, where they have a new sidebar. In that sidebar, you have buttons for having chat gpt open if you want. But that's not the impressive part. It also lets you choose from other models like huggingface, so anyone can try them and understand how the open models are without any installation.
Very cool.
OpenWebUI is also a great and simple solution, that's using Ollama under the hood. Was pretty easy to setup with Docker.
Ollama software and you can pick a model that dose what you want. Mistral and Llama are currently best IMO but it changes often.
Oobabooga is a pretty beginner-friendly solution for running LLMs locally. Models are freely available on Huggingface, but look for GGUF quantizations that will fit in your VRAM. The good thing about GGUFs is that they're typically offered in a wide range of sizes so you can pick one that will fit on your GPU. If you use all your VRAM and start offloading to system memory then the generation will be far slower.
I've had the best results with Noromaid20B and Rose20B quants running on a 16GB 4080. Don't expect it to be as smart as GPT 4.0, but those models do a pretty good job of following instruction and writing decent prose.
Once you mess around with Oobabooga a bit, I'd highly recommend picking up the SillyTavern front-end. Oobabooga runs the actual model while SillyTavern manages characters, world lore, and offers a wide range of other features including a "visual novel" mode where you can set up character sprites that emote based on the content of the messages. It takes a while to get the hang of but it's pretty cool.
I asked it yesterday, why touchscreen input is separate on Linux (and needs software's support) and how to map it to mouse input. Can't really google that.
I had a roommate who reviewed and scored responses for Google Bard. A ton of it was people generating posts for whatever business/crypto/alpha-male grift they were running. The main thing though was really really specific fetish stuff.