Hey all, We are planning to do a Docker upgrade tomorrow 6/30 Sunday, around 1800 UTC. The whole site will be down during the upgrade, but should only be down for a short while. Just wanted to give everyone a heads up. Thanks!

• @jelloeater85@lemmy.world

Hello world!

We would like to start by saying thank you ❤, no really 🙏 THANK YOU to ALL the moderators out there!

Without you folks, we would have no one to help keep our community safe and help build the communities both here on Lemmy.World and on other fine instances. To this end, we want to make sure your voices are heard 📣 loud and clear📣.

So, in the spirit of transparency, we would like everyone to know that we are looking to help out the folks working on Sublinks. Over the last several months we have grown to be more than just Lemmy.World. We've added platforms such as Pixelfed and Sharkey to help offer our users more diverse options for expressing themselves online. We still are very committed to Mastodon as well.

We DO NOT plan on moving away from Lemmy as a software platform at this time. Any changes in our core services would need to be discussed extensively internally AND externally with our community members. We firmly believe in the growth of the Fediverse and without the users, there would only be software, and that's no fun!

Sooo...

The Sublinks team has written up a little survey, which we feel is both thorough and inclusive. It covers a wide range of topics, such as user privacy, and community engagement, along with trying to gauge things that are difficult when moderating.

Also please be aware the information collected by this survey is completely anonymous. As many of us in the social sciences background know, if you want the REAL feelings of individuals, they need to feel safe to express themselves.

👉Moderation Survey HERE👈

Please feel free to comment in this thread, we will do our best to respond to any genuine questions.

We look forward to hearing from each and every one of you!

=Sincerely,
Fedihosting Foundation

PS ... also if this sounds like a corporate press release to you folks, we still punk 🤘😜🤘

Removal of piracy communities

Hello world!

Some of you will already have noticed that we have removed some piracy related communities from Lemmy.World during the last day.

Lack of communication

First off, we want to address the lack of communication.

Not everyone in our current admin team has been with us long enough to be aware of the previous issues and discussions related to these communities and the impact this has on our community.

We should absolutely have published this announcement when or before we removed the communities, not hours later. After realizing this mistake, we would have liked to write this a lot earlier already, but we were all busy with irl things, that we just didn’t have time for it.

Lemmy.World is run by volunteers on their personal time, nobody here gets paid for what we do.

Removed communities

Next, we want to explain how we got to the decision to remove these communities.

[!crackwatch@lemmy.dbzer0.com](/c/crackwatch@lemmy.dbzer0.com)

A lot of the recent content posted to this community included images instructing users to visit a specific website to obtain a copy of the release that the post is about. These instructions were in the form of Type in Google: visit-this.domain. The domain referenced in these posts is entirely focused on video game piracy and providing people with access to copyright infringing material.

While there may be legal differences between whether one is linking to specific content on a domain or just linking to the domain itself, such as linking to https://en.wikipedia.org/wiki/Online_piracy compared to linking to https://en.wikipedia.org/, we do not consider this to be clear enough in laws and previous lawsuits that linking to just the domain is acceptable, if that domain is primarily about distributing copyright infringing material. We therefore do not allow linking to such domains. Additionally, we do not see a significant difference between posting a link directly to a website and embedding said link in an image, so we treat them equally.

[!piracy@lemmy.dbzer0.com](/c/piracy@lemmy.dbzer0.com)

This community is, for the most part, just about discussing various topics related to piracy. We do not at all mind discussion about this topic, and if it had been limited to that, this community would be fine.

This community, however, contains a pinned Megathread post by a community moderator, which, through a few levels of a pastebin-like site, provides an aggregated overview of various sources of content. Some of these sources are entirely legal content, but it intentionally includes various other references, such as the website referred to from the CrackWatch community, which are primarily intended for copyright infringement.

lemmy.dbzer0.com is willing to accept this content on their instance, as well as the potential legal risk coming from this, which they’re free to do.

We do not plan to defederate from lemmy.dbzer0.com, but we will continue to remove communities that are directly facilitating copyright infringement. @db0@lemmy.dbzer0.com, the admin of lemmy.dbzer0.com, is a great person, and we have no problems with him as a person. This is just a matter of different risk tolerance.

[!piracy@lemmy.ml](/c/piracy@lemmy.ml)

Same as [!piracy@lemmy.dbzer0.com](/c/piracy@lemmy.dbzer0.com).

Why have the piracy communities been restored previously? What changed?

Currently, based on the memories of team members involved in the decision back then, it appears that there was a misunderstanding between the community moderators and Lemmy.World admins in how the community will be moderated going forward, as well as which types of content are allowed.

Lemmy.World expected/assumed that links to websites primarily focused on facilitating distribution of pirated content would be disallowed in these communities.

The community moderators however do tolerate references to such websites, as long as people are not linking to individual content directly.

We suspect that this may have been missed during our original review when restoring the communities, which lead us to previously restoring these communities.

Why now?

We have recently received a takedown request for content not directly related to these communities, but it prompted us to review other piracy related content and communities.

Terms of Service clarification

Last, as we’ve reviewed our Terms of Service, we have updated our wording here to make it more clear what is and what isn’t allowed when it comes to piracy. This was already covered by "Do not post illegal content of any type. Do not engage in any activity that may […] facilitate or provide access to illegal transactions" in section 4, but we have now added section 4.1 to better explain this.

We apologize for the delays in communication.

[Update - March 3rd 2024 , 21:14 CET]

The update seems to have worked well. The database migration took some time, but after that it looks everything works as far as we tested. Let us know what you think! And if you like our work, check the donations links in the sidebar ;-)

(As you might notice, the new comments are shown YELLOW, that's not a bug, it's apparently a new feature in Lemmy...)

Hey everyone,

We wanted to announce that we will be attempting to upgrade Lemmy.World to version 0.19.3 on Sunday. The update will start at 2024-03-03 1900 UTC and should hopefully take no more than 2 hours. In the event there are issues, we will roll back to 0.18.5. In the event of a roll back, any data created by users will be lost, so we'll try our best to avoid this!

Hopefully downtime should be minimal <3

During the upgrade, please feel free to make yourself at home on our Matrix and Discord chat rooms.

Lemmy.World Pub

Matrix

Please follow Lemmy.World (@LemmyWorld@mastodon.world) - Mastodon for updates.

As usual, status can be tracked by visiting https://dash.lemmy.world and https://status.lemmy.world

Thanks for your patience in waiting for us. Stability is king for us!

-LW Tech Group

Hello World,

You may have seen the LW holiday themed logos we have used for Halloween and Thanksgiving. LW's users represent many countries around the world and we want to celebrate holidays and other special days that are local to you, but our team is fairly small and we aren't aware of a lot of the local customs out there. So we're asking you what you would like to see represented in a LW themed logo. What are some holidays/special days in your area and how do you celebrate them? And not just major holidays, we would like to celebrate festivals, days of remembrance, and other special days.

Please, comment below your suggestions and ideas on how we could represent them in the LW logo.

EDIT: Mostly looking for events throughout the year. What's left of 2023 is already in work. Thanks!

Hello World,

Today, after careful consideration and evaluation of recent events, we have decided to defederate from Lemmygrad.

Regrettably, we have observed a significant increase in hate speech and calls to violence originating from the Lemmygrad instance. Due to the severity of the posts and comments, we are not waiting for the next Lemmy update that will allow users to block instances.

At Lemmy.world, we have always strived to foster an inclusive and welcoming user environment. However, recent posts and comments from Lemmygrad have clearly violated our server rules and, more importantly, our core values. We firmly believe that hate speech and incitement of violence have no place in our community, regardless of personal beliefs or affiliations.

As always, we encourage all users to report any content they deem inappropriate or harmful. No matter one's stance in any conflict, Lemmy.world will always take immediate action to remove and ban any posts or comments that incite violence or propagate hatred.

We encourage everyone to continue engaging in discussions within the boundaries of respect and understanding. As we move forward with this decision, we remain committed to providing all community members with a safe and welcoming space. We appreciate your continued support and cooperation in upholding our shared principles.

Thank you,

The Lemmy.World Team

We know most of you were thrilled about our Discord server, so we decided to make you even happier with our very own space on matrix!

• There is an announcement room for all the latest and greatest about Lemmy World: https://matrix.to/#/#announcements:lemmy.world

• A General channel where everyone is welcome to join and talk: https://matrix.to/#/#general:lemmy.world

And lastly, this one is more for admins of other instances but we have opened a "LW Defense HQ" room which we want to use in our ongoing fight against spammers: https://matrix.to/#/#defense:lemmy.world

Come join us!

EDIT: Madlad @ruud@lemmy.world was easily convinced to set up a lemmy.world matrix server. So I edited the links to the new space and rooms. The old ones still work too!

That means you can now also sign up for a Lemmy World matrix account. Instructions in this comment: https://lemmy.world/comment/3443207

Earlier, after review, we blocked and removed several communities that were providing assistance to access copyrighted/pirated material, which is currently not allowed per Rule #1 of our Code of Conduct. The communities that were removed due to this decision were:

• !piracy@lemmy.dbzer0.com
• !piracy@lemmy.ml
• !steamdeckpirates@lemmy.dbzer0.com

We took this action to protect lemmy.world, lemmy.world's users, and lemmy.world staff as the material posted in those communities could be problematic for us, because of potential legal issues around copyrighted material and services that provide access to or assistance in obtaining it.

This decision is about liability and does not mean we are otherwise hostile to any of these communities or their users. As the Lemmyverse grows and instances get big, precautions may happen. We will keep monitoring the situation closely, and if in the future we deem it safe, we would gladly reallow these communities.

The discussions that have happened in various threads on Lemmy make it very clear that removing the communites before we announced our intent to remove them is not the level of transparency the community expects, and that as stewards of this community we need to be extremely transparent before we do this again in the future as well as make sure that we get feedback around what the planned changes are, because lemmy.world is yours as much as it is ours.

Hello there!

It has been a while since our last update, but it's about time to address the elephant in the room: downtimes. Lemmy.World has been having multiple downtimes a day for quite a while now. And we want to take the time to address some of the concerns and misconceptions that have been spread in chatrooms, memes and various comments in Lemmy communities.

So let's go over some of these misconceptions together.

"Lemmy.World is too big and that is bad for the fediverse".

While one thing is true, we are the biggest Lemmy instance, we are far from the biggest in the Fediverse. If you want actual numbers you can have a look here: https://fedidb.org/network

The entire Lemmy fediverse is still in its infancy and even though we don't like to compare ourselves to Reddit it gives you something comparable. The entire amount of Lemmy users on all instances combined is currently 444,876 which is still nothing compared to a medium sized subreddit. There are some points that can be made that it is better to spread the load of users and communities across other instances, but let us make it clear that this is not a technical problem.

And even in a decentralised system, there will always be bigger and smaller blocks within; such would be the nature of any platform looking to be shaped by its members. 

"Lemmy.World should close down registrations"

Lemmy.World is being linked in a number of Reddit subreddits and in Lemmy apps. Imagine if new users land here and they have no way to sign up. We have to assume that most new users have no information on how the Fediverse works and making them read a full page of what's what would scare a lot of those people off. They probably wouldn't even take the time to read why registrations would be closed, move on and not join the Fediverse at all. What we want to do, however, is inform the users before they sign up, without closing registrations. The option is already built into Lemmy but only available on Lemmy.ml - so a ticket was created with the development team to make these available to other instance Admins. Here is the post on Lemmy Github.

Which brings us to the third point:

"Lemmy.World can not handle the load, that's why the server is down all the time"

This is simply not true. There are no financial issues to upgrade the hardware, should that be required; but that is not the solution to this problem.

The problem is that for a couple of hours every day we are under a DDOS attack. It's a never-ending game of whack-a-mole where we close one attack vector and they'll start using another one. Without going too much into detail and expose too much, there are some very 'expensive' sql queries in Lemmy - actions or features that take up seconds instead of milliseconds to execute. And by by executing them by the thousand a minute you can overload the database server.

So who is attacking us? One thing that is clear is that those responsible of these attacks know the ins and outs of Lemmy. They know which database requests are the most taxing and they are always quick to find another as soon as we close one off. That's one of the only things we know for sure about our attackers. Being the biggest instance and having defederated with a couple of instances has made us a target.  

"Why do they need another sysop who works for free"

Everyone involved with LW works as a volunteer. The money that is donated goes to operational costs only - so hardware and infrastructure. And while we understand that working as a volunteer is not for everyone, nobody is forcing anyone to do anything. As a volunteer you decide how much of your free time you are willing to spend on this project, a service that is also being provided for free.

We will leave this thread pinned locally for a while and we will try to reply to genuine questions or concerns as soon as we can.

We've launced a public discord server at https://discord.gg/lemmyworld

The reason for why we choose discord is because it was easier to moderate and manage than other options. Besides we also had a discord-bot guru by the name of Rooki who created a neat bot that allowed us to connect/verify discord users to their Lemmy World account.

So if you are a moderator and you are looking for some extra hands to moderate your community, or if you need to contact anyone from the LW team, this is the most efficient way to do it!

Come have a look!

Lemmy.World is looking for 4 new Systems operators to help with our growing community.

Volunteers will assist our existing systems team with monitoring and maintenance.

We’re ideally looking for chill folks that want to give back to their community and work on our back-end infrastructure. Must have 4+ years of professional experience working in systems administration. We are not looking for junior admins at this time. Please keep in mind that, while this is a volunteer gig, we would ask you to be able to help at least 5-10 hours a week. We also understand this is a hobby and that family and work comes first.

Applicants must be okay with providing their CV and/or LinkedIn profile AND sitting for a video interview. This is due to the sensitivity of the infrastructure you will have access to.

We are an international team that works from both North America EST time (-4) and Europe CEST (+2) so we would ask that candidates be flexible with their availability.

If you are in AEST (+10) or JST (+9) please let us know, as we are looking for at least one Sysadmin to help out during our overnight.

You may be asked to participate in an on-call pool. Please keep in mind that this is a round-robin style pool, so it's alright if you're busy as it will just move along the chain.

If you're interested and want to apply, click here.

Update:
The comments from this post will not be removed as to preserve the discussion around the announcement. Any continued discussions outside of this thread that violate server rules will be removed. We feel that everyone that has an opinion, and wanted to vent, has been heard.

————-

Original post:
Yesterday, we received information about the planned federation by Hexbear. The announcement thread can be found here: https://www.hexbear.net/post/280770. After reviewing the thread and the comments, it became evident that allowing Hexbear to federate would violate our rules.

Our code of conduct and server rules can be found here.

The announcement included several concerning statements, as highlighted below:

• “Please try to keep the dirtbag lib-dunking to hexbear itself. Do not follow the Chapo Rules of Posting, instead try to engage utilizing informed rhetoric with sources to dismantle western propaganda. Posting the western atrocity propaganda and pig poop balls is hilarious but will pretty quickly get you banned and if enough of us do it defederated.”
• “The West's role in the world, through organizations such as NATO, the IMF, and the World Bank - among many others - are deeply harmful to the billions of people living both inside and outside of their imperial core.”
• “These organizations constitute the modern imperial order, with the United States at its heart - we are not fooled by the term "rules-based international order." It is in the Left's interest for these organizations to be demolished. When and how this will occur, and what precisely comes after, is the cause of great debate and discussion on this site, but it is necessary for a better world.”

The rhetoric and goal of Hexbar are clear based on their announcement: to "dismantle western propaganda" and "demolish organizations such as NATO” shows that Hexbar has no intention of "respecting the rules of the community instance in which they are posting/commenting.” It’s to push their beliefs and ideology.

In addition, several comments from a Hexbear admin, demonstrate that instance rules will not be respected.

Here are some examples:

“I can assure you there will be no lemmygrad brigades, that energy would be better funneled into the current war against liberalism on the wider fediverse.”

“All loyal, honest, active and upright Communists must unite to oppose the liberal tendencies shown by certain people among us, and set them on the right path. This is one of the tasks on our ideological front.”

• https://lemmy.world/comment/121850
• https://lemmy.world/comment/1487168
• https://lemmy.world/comment/1476084
• https://lemmy.world/comment/171595
• https://www.hexbear.net/comment/3648500

Overall community comments:

• https://www.hexbear.net/comment/3526128
• https://www.hexbear.net/comment/3526086
• https://www.hexbear.net/comment/3652828

To clarify, for those who have inquired about why Hexbear versus Lemmygrad, it should be noted that we are currently exploring the possibility of defederating from Lemmygrad as well based on similar comments Hexbear has made.

• https://lemmygrad.ml/post/158656
• https://lemmygrad.ml/comment/882559
• https://lemmygrad.ml/comment/540170
• https://lemmygrad.ml/comment/446529

Defederation should only be considered as a last resort. However, based on their comments and behavior, no positive outcomes can be expected.

We made the decision to preemptively defederate from Hexbear for these reasons. While we understand that not everyone may agree with our decision, we believe it is important to prioritize the best interests of our community.

We've installed Voyager and it's reachable at https://m.lemmy.world, you can browse Lemmy, and login there (also if your account isn't on lemmy.world)

PS Thanks go out to @stux@stux@geddit.social , he came up with the idea (see https://m.geddit.social).

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

• There is an vulnerability which was exploited
• Several people had their JWT cookies leaked, including at least one admin
• Attackers started changing site settings and posting fake announcements etc

Our mitigations:

• We removed the vulnerability
• Deleted all comments and private messages that contained the exploit
• Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

There has been significant discussion in recent weeks regarding Meta/Threads. We would like to express our disappointment with the negative and threatening tone of some of these discussions. We kindly ask everyone to engage in civil discourse and remember that not everyone will share the same opinions, which is perfectly acceptable.

When considering whether or not to defederate from Threads, we're looking for a decision based on facts that prioritize your safety. We strive to remain neutral to make an informed choice.

First, there seem to be some misconceptions about how the Fediverse operates based on several posts. We’ve compiled some resource links to help explain the details and address any misunderstandings.

Fed Tips , Fediverse , ActivityPub

Initial Thoughts:

It seems unlikely that Meta will federate with Lemmy. When/if Meta adopts ActivityPub, it will likely affect Mastodon only rather than Lemmy, given Meta's focus on being a Twitter alternative at the moment.

Please note that we have a few months before Threads will even federate with Mastodon, so we have some time to make the right decision.

Factors to Consider:
Factors to consider if Meta federates with Lemmy:

Privacy - While it’s true that Meta's privacy settings for the app are excessive, it’s important to note that these settings only apply to users of the official Threads app and do not impact Lemmy users. It’s worth mentioning that Lemmy does not collect any personal data, and Meta has no means of accessing such data from this platform. In addition, when it comes to scraping data from your post/comments, Meta doesn’t need ActivityPub to do that. Anyone can read your profile and public posts as it is today.

Moderation - If a server hosts a substantial amount of harmful content without performing efficient and comprehensive moderation, it will create an excessive workload for our moderators. Currently, Meta is utilizing its existing Instagram moderation tools. Considering there were 95 million posts on the first day, this becomes worrisome, as it could potentially overwhelm us and serve as a sufficient reason for defederation.

Ads - It’s possible if Meta presents them as posts.

Promoting Posts - It’s possible with millions of users upvoting a post for it to trend.

Embrace, extend, and extinguish (EEE) - We don't think they can. If anyone can explain how they technically would, please let us know. Even if Meta forks Lemmy and gets rid of the original software, Lemmy will survive.

Instance Blocking - Unlike Mastodon, Lemmy does not provide a feature for individual users to block an instance (yet). This creates a dilemma where we must either defederate, disappointing those who desire interaction with Threads, or choose not to defederate, which will let down those who prefer no interaction with Threads.

Blocking Outgoing Federation - There is currently no tool available to block outgoing federation from lemmy.world to other instances. We can only block incoming federation. This means that if we choose to defederate with our current capabilities, Threads will still receive copies of lemmy.world posts. However, only users on Threads will be able to interact with them, while we would not be able to see their interactions. This situation is similar to the one with Beehaw at the moment. Consequently, it leads to significant fragmentation of content, which has real and serious implications.

Conclusion:
From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

If you have any added points or remarks on the above, please send them to @mwadmin@mastodon.world.

Another day, another update.

More troubleshooting was done today. What did we do:

• Yesterday evening @phiresky@phiresky@lemmy.world did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
• @cetra3@lemmy.ml created a docker image containing 3PR's: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
• We started using this image, and saw a big drop in CPU usage and disk load.
• We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
• We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
• We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set ~~proxy_next_upstream timeout;~~ max_fails=5 in nginx.

Currently we're running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the ~~proxy_next_upstream timeout;~~ max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @phiresky@lemmy.world , @cetra3@lemmy.ml , @stanford@discuss.as200950.com, @db0@lemmy.dbzer0.com , @jelloeater85@lemmy.world , @TragicNotCute@lemmy.world for their help!

And not to forget, thanks to @nutomic@lemmy.ml and @dessalines@lemmy.ml for their continuing hard work on Lemmy!

And thank you all for your patience, we'll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that's now started, and I noticed the proxy_next_upstream timeout setting didn't work (or I didn't set it properly) so I used max_fails=5 for each upstream, that does actually work.

Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won't work as desired.. we see several issues:

Performance

• Loading is mostly OK, but sometimes things take forever
• We (and you) see many 502 errors, resulting in empty pages etc.
• System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

• Replying to a DM doesn't seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
• 2FA seems to be a problem for many people. It doesn't always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @nutomic@lemmy.ml is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we'll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn't too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we're very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

So I've been troubleshooting the federation issues with some other admins:

(Thanks for the help)

So what we see is that when there are many federation workers running at the same time, they get too slow, causing them to timeout and fail.

I had federation workers set to 200000. I've now lowered that to 8192, and set the activitypub logging to debugging to get queue stats. RUST_LOG="warn,lemmy_server=warn,lemmy_api=warn,lemmy_api_common=warn,lemmy_api_crud=warn,lemmy_apub=warn,lemmy_db_schema=warn,lemmy_db_views=warn,lemmy_db_views_actor=warn,lemmy_db_views_moderator=warn,lemmy_routes=warn,lemmy_utils=warn,lemmy_websocket=warn,activitypub_federation=debug"

Also, I saw that there were many workers retrying to servers that are unreachable. So, I've blocked some of these servers:

commallama.social,mayheminc.win,lemmy.name,lm.runnerd.net,frostbyrne.io,be-lemmy.org,lemmonade.marbledfennec.net,lemmy.sarcasticdeveloper.com,lemmy.kosapps.com,pawb.social,kbin.wageoffsite.com,lemmy.iswhereits.at,lemmy.easfrq.live,lemmy.friheter.com,lmy.rndmm.us,kbin.korgen.xyz

This gave good results, way less active workers, so less timeouts. (I see that above 3000 active workers, timeouts start).

(If you own one of these servers, let me know once it's back up, so I can un-block it)

Now it's after midnight so I'm going to bed. Surely more troubleshooting will follow tomorrow and in the weekend.

Please let me know if you see improvements, or have many issues still.

EDIT: it seems that this is a known bug and will be fixed in the next release! Thank you guys for letting me know

https://github.com/LemmyNet/lemmy/pull/3090#issuecomment-1591009865

Lemmy.World is running amazing for the amount of users that is has but when I am trying to scroll, every 2 seconds the post im looking at gets shoved to the bottom, with Lemmy skyrocketing in popularity, having the new posts appear on the front page constantly is making it very hard to browse.

Does anyone know if there is a way I can stop that? Or do we just have to wait for an update?

I want to add that I love lemmy and I will continue to use it anyways! The community here is so much better than reddit and with the protest going on reddit is a dumpster fire of toxicity right now.

I noticed my feed on Lemmy was pretty dry today, even for Lemmy. Took me a while to realize lemmy.ml has been going up and down all morning, and isn't federating new posts.

But, since this is all still federated, I can still create and read posts on other instances while I wait. Even this one! Any other service would just be unavailable completely right now.

I do miss the larger communities on lemmy.ml - asklemmy, memes, and I really wanted to watch the reddit fallout on /c/reddit. Maybe I'll look around for some good replacements for those. Open to suggestions!