ruud

joined 1 year ago
MODERATOR OF
 

So I've been troubleshooting the federation issues with some other admins:

(Thanks for the help)

So what we see is that when there are many federation workers running at the same time, they get too slow, causing them to timeout and fail.

I had federation workers set to 200000. I've now lowered that to 8192, and set the activitypub logging to debugging to get queue stats. RUST_LOG="warn,lemmy_server=warn,lemmy_api=warn,lemmy_api_common=warn,lemmy_api_crud=warn,lemmy_apub=warn,lemmy_db_schema=warn,lemmy_db_views=warn,lemmy_db_views_actor=warn,lemmy_db_views_moderator=warn,lemmy_routes=warn,lemmy_utils=warn,lemmy_websocket=warn,activitypub_federation=debug"

Also, I saw that there were many workers retrying to servers that are unreachable. So, I've blocked some of these servers:

commallama.social,mayheminc.win,lemmy.name,lm.runnerd.net,frostbyrne.io,be-lemmy.org,lemmonade.marbledfennec.net,lemmy.sarcasticdeveloper.com,lemmy.kosapps.com,pawb.social,kbin.wageoffsite.com,lemmy.iswhereits.at,lemmy.easfrq.live,lemmy.friheter.com,lmy.rndmm.us,kbin.korgen.xyz

This gave good results, way less active workers, so less timeouts. (I see that above 3000 active workers, timeouts start).

(If you own one of these servers, let me know once it's back up, so I can un-block it)

Now it's after midnight so I'm going to bed. Surely more troubleshooting will follow tomorrow and in the weekend.

Please let me know if you see improvements, or have many issues still.

0
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/moderators@lemmy.world
 

Guidelines

  • First of all, every community on Lemmy.world should follow the lemmy.world rules
  • Please create a sidebar with some contents, at least a description of the community and some extra rules when applicable
  • Adding a banner and icon for the community makes it prettier. Please do.
  • Every community needs enough moderators.

About moderators

Moderation is very important. This site needs to be a safe place for everyone. The more subscribers and posts, the more moderators you'll need. Make sure you have moderators in all timezones, so if bad stuff is reported in your community, it doesn't need to sit there until you're off work... As the community grows, add even more moderators.

Reports

A moderator will receive reports for reported content in their community. (The admins will receive a copy of all reports in all communities). Please resolve the reports according to the site rules and community rules, as soon as possible. If reports are open for too long (more than 24 hours), the admins will contact the moderator(s). If this keeps happening, the moderators might be replaced.

These guidelines are under construction. Please check back after a few days, hopefully it's more complete ;-)

 

So some spam signups just happened (all username12345678@gmail.com format e-mail) This caused bounced mail to increase, causing Mailgun to block our domain to prevent it getting blacklisted.

So:

  • Mail temporarily doesn't work
  • I closed signups for now
  • I will ban the spam accounts
  • I will check how to prevent (maybe approval required again?)

Stay tuned.

Edit: so apparently there is a captcha option which I now enabled. Let's see if this prevents spam. Registrations open again.

Edit2 : Hmm Mailgun isn't that fast in unblocking the domain. Closing signups again because validation mails aren't sent

Edit 3: I convinced Mailgun to lift the block. Signups open again.

 

So after we've extended the virtual cloud server twice, we're at the max for the current configuration. And with this crazy growth (almost 12k users!!) even now the server is more and more reaching capacity.

Therefore I decided to order a dedicated server. Same one as used for mastodon.world.

So the bad news... we will need some downtime. Hopefully, not too much. I will prepare the new server, copy (rsync) stuff over, stop Lemmy, do last rsync and change the DNS. If all goes well it would take maybe 10 minutes downtime, 30 at most. (With mastodon.world it took 20 minutes, mainly because of a typo :-) )

For those who would like to donate, to cover server costs, you can do so at our OpenCollective or Patreon

Thanks!

Update The server was migrated. It took around 4 minutes downtime. For those who asked, it now uses a dedicated server with a AMD EPYC 7502P 32 Cores "Rome" CPU and 128GB RAM. Should be enough for now.

I will be tuning the database a bit, so that should give some extra seconds of downtime, but just refresh and it's back. After that I'll investigate further to the cause of the slow posting. Thanks @veroxii@lemmy.world for assisting with that.

1
submitted 1 year ago* (last edited 1 year ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world
 

(I'm creating a starting guide post here. Have patience, it will take some time...)

Disclaimer: I am new to Lemmy like most of you. Still finding my way. If you see something that isn't right, let me know. Also additions, please comment!

Welcome!

Welcome to Lemmy (on whichever server you're reading this)

About Lemmy

Lemmy is a federated platform for news aggregagtion / discussion. It's being developed by the Lemmy devs: https://github.com/LemmyNet

About Federation

What does this federation mean?

It means Lemmy is using a protocol (Activitypub) which makes it possible for all Lemmy servers to interact.

  • You can search and view communities on remote servers from here
  • You can create posts in remote communities
  • You can respond to remote posts
  • You will be notified (if you wish) of comments on your remote posts
  • You can follow Lemmy users/communities on other platforms that also use Activitypub (like Mastodon, Calckey etc) (There's currently a known issue with that, see here

Please note that a server only starts indexing a server/community once it has been interacted with by a user of this server.

A great image describing this, made by @ulu_mulu@lemmy.world : https://imgur.com/a/uyoYySY

About Lemmy.world

Lemmy.world is one of the many servers hosting the Lemmy software. It was started on June 1st, 2023 by @ruud@lemmy.world , who is also running https://mastodon.world, https://calckey.world and others.

A list of Lemmy servers and their statistics can be found at FediDB

Quick start guide

Account

You can use your account you created to log in to the server on which you created it. Not on other servers. Content is federated to other servers, users/accounts are not.

Searching

In the top menu, you'll see the search icon. There, you can search for posts, communities etc.

You can just enter a search-word and it will find the Post-titles, post-content, communities etc containing that word that the server knows of. So any content any user of this server ever interacted with.

You can also search for a community by it's link, e.g. [!Netherlands@lemmy.nl](/c/Netherlands@lemmy.nl). Even if the server hasn't ever seen that community, it will look it up remotely. Sometimes it takes some time for it to fetch the info (and displays 'No results' meanwhile..) so just be patient and search a second time after a few seconds.

Creating communities

First, make sure the community doesn't already exist. Use search (see above). Also try https://browse.feddit.de/ to see if there are remote communities on other Lemmy instances that aren't known to Lemmy.world yet.

If you're sure it doesn't exist yet, go to the homepage and click 'Create a Community'.

It will open up the following page:

Here you can fill out:

  • Name: should be all lowercase letters. This will be the /c/
  • Display name: As to be expected, this will be the displayed name.
  • You can upload an icon and banner image. Looks pretty.
  • The sidebar should contain things like description, rules, links etc. You can use Markdown (yey!)
  • If the community will contain mainly NSFW content, check the NSFW mark. NSFW is allowed as long as it doesn't break the rules
  • If you only want moderators to be able to post, check that checkbox.
  • Select any language you want people to be able to post in. Apparently you shouldn't de-select 'Undetermined'. I was told some apps use 'Undetermined' as default language so don't work if you don't have it selected

Reading

I think the reading is obvious. Just click the post and you can read it. SOmetimes when there are many comments, they will partly be collapsed.

Posting

When viewing a community, you can create a new post in it. First of all make sure to check the community's rules, probably stated in the sidebar.

In the Create Post page these are the fields:

  • URL: Here you can paste a link which will be shown at the top of the post. Also the thumbnail of the post will link there. Alternatively you can upload an image using the image icon to the right of the field. That image will also be displayed as thumbnail for the post.
  • Title: The title of the post.
  • Body: Here you can type your post. You can use Markdown if you want.
  • Community: select the community where you want this post created, defaults to the community you were in when you clicked 'create post'
  • NSFW: Select this if you post any NSFW material, this blurs the thumbnail and displays 'NSFW' behind the post title.
  • Language: Specify in which language your post is.

Also see the Lemmy documentation on formatting etc.

Commenting

Moderating / Reporting

Client apps

There are some apps available or in testing. See this post for a list!

Issues

When you find any issue, please report so here: https://lemmy.world/post/15786 if you think it's server related (or not sure).

Report any issues or improvement requests for the Lemmy software itself here: https://github.com/LemmyNet

Known issues

Known issues can be found in the beforementioned post, one of the most annoying ones is the fact that post/reply in a somewhat larger community can take up to 10 seconds. It seems like that's related to the number of subscribers of the community.

I'll be looking into that one, and hope the devs are too.

 

lemmy.world just reached 1000 users.

Please remember that the server was created June 1st! So still might notice some startup issues... but so far so good! Welcome @all!

view more: ‹ prev next ›