Point us to where the coin slot is. E.g. Patreon. We insert coin πͺ, you upgrade.
this post was submitted on 04 Jun 2023
0 points (NaN% liked)
Lemmy
12438 readers
1 users here now
Everything about Lemmy; bugs, gripes, praises, and advocacy.
For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.
founded 4 years ago
MODERATORS
I guess every bigger instance needs to constantly promote a monthly support fee to keep the servers runnin. Something that most people already use and don't have to register first. Like PayPal or Patreon.
Edit: And I support Lemmy now on Patreon.
You might wanna consider temporarily closing sign-up requests on lemmy.ml similarly to how mastodon.social did it during its large influx. Making a sign-up request and just receiving an infinite loading icon is a very frustrating experience.
Similarly, you want to make it as easy as possible to financially contribute to lemmy, even if it means using proprietary platforms like Patreon.
Overall, the current Reddit API change is probably one of the largest opportunities for lemmy right now, so smoothing over the user experience as fast as possible in the coming days will be of atmost importance if we want lemmy to become a viable Reddit alternative...
@ devs:
Can you please focus your work on optimising performance for the UI? It will greatly reduce the amount of electricity and money spent, so you're actually multiplying every tenth of a second you can shave off of CPU time...
Thank you a lot for writing this software. It's been a great little project so far and it seems to go down the Mastodon route of increased popularity. Be proud!
We're about 80% done with a big rework of the UI to switch from websockets to http. This should solve most of the stability and performance issues of both the front and back end.
Damn I am veeery excited!! Do you have an ETA? No pressure though :)
Great to hear, thank you for the update!
Hi, as one of the new people, is there a way to transfer to another instance or would I have to create a new account there?
You have to create a new account. But that's easy ;)
That's kind of wrong though, isn't it? What about stuff like GDPR data exports? Users should be able to export their data, then import it into another instance, effectively migrating instances.
You are free to learn to program and write a user import routine for lemmy: https://github.com/LemmyNet/lemmy
I know how to program, I also know how to wonder how many instances are running off the docker-compose with publicly exposed postgres... that would make import/export really easy, wouldn't it? π
Anyway, would you say this isn't the right place to discuss this stuff?
Why would this be not the right place?
I think lemmy will be bitten in the ass by not having considered clustering/horizontal scaling from the start. Federation alone as a scaling mechanism is only feasible for "nerds". But if the network wants to grow, we will need a few scale-able large hosted instances. And if their only choice is to scale vertically, there will be a hard limit (unless we put a good old Mainframe somewhere ^^).
Another downside of this design is: you can't run it with high availability. If there's only one process per instance, updating it will mean the whole instance is down. Sure, if all goes well this downtime is under a second. But if it doesn't go well or if a migration is needed, this might quickly become hours.
I think you probably underestimate how far one can get with "vertical" scaling. Here's the dockerfile: https://raw.githubusercontent.com/LemmyNet/lemmy/release/v0.17/docker/prod/docker-compose.yml
- It includes 4 different containers... so there's a way to scale out to 4 machines right away. Maybe not every container is doing an equal amount of work... but there's some amount of immediately available machine-splitting.
- I'm no expert, but I believe that at least the lemmy and lemmy-ui containers are stateless. If so, they're horizontally scalable already.
- Postgres then would likely be the main bottleneck. But postgres offers read-replicas, so again the write-load and the read-load can be hosted on separate machines. And if there's enough read-load, you can have many replicas.
Other comments from the admins have shown that lemmy.ml today is running on a single eight-core box and it's currently hosting 30k registered users and over 1k active. So how much more compute capacity can we throw at "vertical" scaling on the current software architecture?
- Just by going to a bigger single box, we can get 128 cores with no problem, a 16x bump in capacity. Does that get us to at least to 300k registered + 10k active?
- Splitting the containers onto 4 separate machines. Does that get us 2x more?
- Adding PG read-replicas and additional lemmy/lemm-ui containers would allow us to expand our instance footprint to maybe 6 physical machines should get us another 2x or more in performance.
Conservatively, that's 100x the computing capacity of the current hardware and could potentially support 1m registered users and 50k active. Now, I don't REALLY expect this to be possible today, there will be many software bottlenecks found along the way to scaling a single instance this large. But my point is that there's already a medium amount of horizontal scalability built into lemmy, and if the software doesn't fall over for algorithmic reasons (which is will at first), the current infrastructure architecture allows quite a lot of growth. There's plenty of time between now and a federation of million user instances to adopt a truly distributed storage backend if needed.
Doesn't solve the availability issues, though. I know of no seriously hosted system that doesn't have at least two replicas in different availability zones. I don't expect any hobby instance to offer any kind of availability guarantee. But if we want to have one or two central instances that the typical reddit user can flock to, this would IMO be essential to have.
Also, in my experience it is FAR cheaper to have a few low to mid range systems for vertical scaling, than to throw a high end machine at it for vertical scaling. If you look the the pricing, the monthly costs for vertical scaling goes up exponentially once you want much more RAM and CPU cores (and storage, and so on).
Being able to scale horizontally solves both issues: hardware is cheaper and reliability is higher.
That lemmy is so damn efficient would then simply mean, that we can achieve excessively good results with low resources, where Reddit would already struggly and needs to put much more machines in place. That would be a nice "business" advantage.
Doesnβt solve the availability issues, though. I know of no seriously hosted system that doesnβt have at least two replicas in different availability zones.
I'm not sure why you think the setup I've described can't have coverage in multiple availability zones. If the lemmy and lemmy-ui containers are stateless as I suspect, you can autoscale them. Pictrs is new to me, not sure there... but it appears to support object-storage which would likely make it stateless and the object-storage can replicate to multiple-az's. Postgres read-replicas can be placed in multiple az's as well. The only component that presents an issue is the Postgres write-leader, and failovers there can be done in minutes. Many many popular sites run with an infrastructure like this and achieve excellent uptimes.
I do get the power of horizontal scalability, I specialize in distributed databases. But they come at a cost in flexibility relative to something like Postgres... and we're very far from "needing" horizontally scaling database writes here. Everything else looks like it can be scaled horizontally if someone wants to take on the headache of doing so.
Well, one could try to swap postgres for cockroachdb. But a ticket in github that asked for clustering support was closed with being out of scope. So might be lemmy is not stateless. Haven't checked the code yet, though.
If cockroach is truly PG compatible, lemmy admins can swap it in without developer support. I suspect Cockroach constrains some SQL features and has poor performance on others, but that or AWS Aurora are things you can experiment with without dev support if you're passionate about the proving out the value of scale-out.
The statement that spawned my response though was this:
I think lemmy will be bitten in the ass by not having considered clustering/horizontal scaling from the start. Federation alone as a scaling mechanism is only feasible for βnerdsβ. But if the network wants to grow, we will need a few scale-able large hosted instances.
I still don't think it's true that we need horizontal scaling to support sufficiently large instances. The amount of vertical and horizontal scaling ability built into Lemmy today is both useful, and likely to outstrip the current ability of its code to scale a single instance. Any algorithms that scale super-linearly with respect to comment-count, post-count, user-count, or community-count, will fail just as hard with distributed backends as they do with an RDBMS. And as you note, PG-compatible distributed systems provide a potential lower-engineering-cost on-ramp to distributed systems once the codebase is efficient-enough to warrant such a transition to scale further. I suspect I've contributed everything of use I have to this thread though, and don't expect to respond further.
Thank you for your thorough explanations and input. It definitely gave me a few things to think about. And if I have some spare time I might even try to spin up lemmy in some local k8s to see how it reacts to being scaled up and down.
Indeed. If a big instance like lemmy.ml was to be shut down all the communities would be lost. This is simply not sustainable. Why would users put effort building a community if it could be gone at any time?
That however would be a different problem. A horizontally scaled instance would be able to cope with more users, but if it shuts down for monetary, personal, or whatever reason, it's still down.
Protecting a community from this is what the decentralized part is for. That is already in place.
(Although there is a middle ground where you could design the system in a way that one instance is mirrored and load-balanced across different hosters. That would actually also be quite interesting to have. But that's another layer of complexity on top.)
Protecting a community from this is what the decentralized part is for. That is already in place.
What? How is it solved exactly? If say lemmy.ml is down, what's the point of other servers existing, if most of the content and users are here? Like, I created a few new communities on lemmy.ml, which don't exist on say Beehaw because for some strange reason, the Beehaw admins don't allow users to create communities. So how is going to Beehaw help me, if lemmy.ml is unavailable? Okay, so you tell me I should go to a different server then. Maybe even make a new server. Done and done. But there's very few to zero users on that server, so those new communities and content created there might as well not exist. Also, even though Lemmy is federated, the homepage defaults to "local", so all the new users coming in may miss out on all the other federated communities, and, if I'm reading this correctly, the federation isn't even a fully automatic process, and some admins may even choose to put there server in a whitelist mode. All of it makes the whole "advantage" of federation, or at least Lemmy's version of it, seem kind of pointless.
It's like saying, "Hey, Gmail is down so you should just use Hotmail instead." Okay, so I can still send and receive emails, but I can't access any of my old emails for context, and none of my contacts can reach me using my Gmail address, and none of my filters, address book and other content is available so I may not even be able to reach out to my contacts and let them know what my new email is.
IMO the way the way the federation should've been designed is to use something like blockchain technology, so every instance basically has all the content and there's only one source of truth for user accounts and data (distributed ledger), or maybe even just implement the whole thing as a plain old high-availability cluster with load balancing.
Unless I'm missing something fundamental, I don't see how this decentralization is of any use if the content isn't there.
What? How is it solved exactly? If say lemmy.ml is down, whatβs the point of other servers existing, [...]
Because you want to rely on someone else's instance. The idiomatic solution would be for a community to host their own lemmy/activitypub instance and join the federation. Then the community has control over their own data. In every sense. If they want to delete something (for breaching law, protocol, or whatever), they are free to do so and don't have to ask anyone else.
IMO the way the way the federation shouldβve been designed is to use something like blockchain technology [...]
Please no. I mean there is IPFS out there that somewhat works like that, but I don't really like that. First, the ever-growing amount of data means that every instance has to keep up with it. If they wouldn't replicate it, the deletion of a single instance would still eliminate the data, even if there were references in a block-chain.
Also: the ability to "forget" is important. Not everything needs to live on forever. That it currently does, can already be a big problem. Look how peoples lives got almost ruined because someone dug up tweets from 10 years ago that were stupid. Solving the issue of data ownership is IMO one of the bigger things we need to keep in mind when designing a better web. Federation with the ability to "just" bring your own instance along where you are the owner is one of these options.
Is there a feature to send a read-only/static link to Lemmy pages?
Iβm envisioning a pre-cached version of the page that is updated hourly or so, rather than querying the database live for every comment on the thread. In a perfect world, these could also be offloaded to a CDN as static pagesβ¦
That's basically what Reddit does when you're not logged in (and a bunch of similar sites too I guess).
Another thing:
We do need more site admins to help us handle the applications and moderation.
For obvious reasons, we prefer ppl who have been here for a long time, and post / comment consistently. If you'd like to help us out, so that nutomic and I can focus on coding, that would be splendid.
If needed, I'd also be more than happy to help.
I'm going to set up a general purpose instance tomorrow with the intention of handling a relatively large number of users. The main problem is choosing a domain!
I was also contemplating setting up a new instance for this. I have 100s of gigs of unused ram, CPUs on idle and a 10gbit connection looking for something to do. The only issue I couldn't figure out was the name. I own itjust.works was thinking of something clever subdomain to use with it. I'm glad I'm not the only one with this issue