Lemmy Federate

95 readers
1 users here now

Updates and questions about lemmy-federate.com

founded 5 months ago
MODERATORS
1
19
submitted 2 months ago* (last edited 2 months ago) by iso@lemy.lol to c/lemmyfederate@lemy.lol
 
 

I’ve seen many arguments claiming that Lemmy Federate creates a significant load on instances in terms of data storage and networking. I used to believe this wasn’t a very valid concern, as I assumed that communities without followers wouldn’t generate many posts, comments, likes, etc.

Today, I had the chance to test this on my own instance, and here are the results:

  • Communities
    • Total count: 33,920
    • Federated by LF: 5,863
    • Percentage: 17.28%
  • Posts
    • Total count: 3,217,783
    • Federated by LF: 114,067
    • Percentage: 3.54%
  • Comments
    • Total count: 14,222,401
    • Federated by LF: 192,925
    • Percentage: 1.36%

LF = Lemmy Federate. Posts and comments refer to the number of posts and comments in communities federated with Lemmy Federate.

As you can see from the statistics, almost 20% of the communities are federated by Lemmy Federate. Although this seems like a high number, only 3.5% of the posts are created in communities federated by Lemmy Federate. The number of comments is even less than the number of posts.

So to answer, Lemmy Federate creates an average load of around 2-3% in an instance with about 1000 users. You can compare it with its benefits and decide whether to use it or not.

If you want to get stats for your own instance, you can run the SQL script below. Let us know the results please :)

the script

-- make sure to replace the <lemmy_federate_bot_person_id>
WITH
-- Total community count
total_communities AS (
    SELECT COUNT(id) AS total FROM community where local != true
),

-- Communities federated by LF (assuming person_id <lemmy_federate_bot_person_id> represents LF bot)
-- A community is considered federated by LF if LF follows it and it has local subscribers
lf_communities AS (
    SELECT ca.community_id
    FROM community_aggregates ca
    JOIN community c on c.id = ca.community_id
    JOIN community_follower cf ON ca.community_id = cf.community_id
    WHERE ca.subscribers_local > 0 and c.local != true AND cf.person_id = <lemmy_federate_bot_person_id>
),
lf_communities_count AS (
    SELECT COUNT(*) AS count FROM lf_communities
),

-- Total post count
total_posts AS (
    SELECT COUNT(id) AS total FROM post
),

-- Posts in LF communities
lf_posts AS (
    SELECT COUNT(p.id) AS count
    FROM post p
    WHERE p.community_id IN (SELECT community_id FROM lf_communities)
),

-- Total comment count
total_comments AS (
    SELECT COUNT(id) AS total FROM comment
),

-- Comments on posts in LF communities
lf_comments AS (
    SELECT COUNT(c.id) AS count
    FROM comment c
    JOIN post p ON c.post_id = p.id
    WHERE p.community_id IN (SELECT community_id FROM lf_communities)
)

-- Final output
SELECT
    tc.total AS community_count,
    lfc.count AS community_count_lf,
    ROUND((lfc.count::decimal / tc.total) * 100, 2) AS community_lf_percent,
    tp.total AS post_count,
    lp.count AS post_count_lf,
    ROUND((lp.count::decimal / tp.total) * 100, 2) AS post_lf_percent,
    tcom.total AS comment_count,
    lcom.count AS comment_count_lf,
	ROUND((lcom.count::decimal / tcom.total) * 100, 2) AS comment_lf_percent
FROM total_communities tc
JOIN lf_communities_count lfc ON TRUE
JOIN total_posts tp ON TRUE
JOIN lf_posts lp ON TRUE
JOIN total_comments tcom ON TRUE
JOIN lf_comments lcom ON TRUE;

2
1
submitted 1 week ago* (last edited 1 week ago) by activistPnk@slrpnk.net to c/lemmyfederate@lemy.lol
 
 

After grabbing your dataset and comparing to another, FWiW these lemmy nodes are apparently missing from lemmy-federate.com:

0xdd.org.ru			lemmy.libertarianfellowship.org
1337lemmy.com			lemmy.lidstah.info
50501.chat			lemmy.linuxuserspace.show
acqrs.co.uk			lemmy.lqx.net
alien.top			lemmy.lukeog.com
altgag.net			lemmy.lupiverse.com
altmedia.house			lemmy.mair.io
api-dev.rabotaem.app		lemmy.manganiello.tech
ascy.mooo.com			lemmy.maquis196.io
awful.systems			lemmy.mariusdavid.fr
badatbeing.social		lemmy.market
baraza.africa			lemmy.masto.community
beehaw.org			lemmy.mats.ooo
belfry.rip			lemmy.max-p.me
bfed.uk				lemmy.mbl.social
blendit.bsd.cafe		lemmy.meissners.me
blockchained.world		lemmy.mengsk.org
blog.kaki87.net			lemmy.menos.gotdns.org
board.minimally.online		lemmy.michaelsasser.org
bolha.forum			lemmy.minecloud.ro
bookwormstory.social		lemmy.minigubben.se
buddyverse.one			lemmy.mkwarman.com
canzuk.zone			lemmy.mlaga97.space
chachara.club			lemmy.moocloud.party
champserver.net			lemmy.mrm.one
chinese.lol			lemmy.multivers.cc
civilloquy.com			lemmy.nekusoul.de
communick.news			lemmy.nerdcore.social
corndog.social			lemmy.nexus
corrigan.space			lemmy.nocturnal.garden
cyberia.hydrar.de		lemmy.noellesporn.de
diggita.com			lemmy.nope.ly
digipres.cafe			lemmy.norbipeti.eu
discover.deltanauten.de		lemmy.nothavingthis.co.uk
discuss.hadan.social		lemmy.nowsci.com
discuss.icewind.me		lemmy.obrell.se
discuss.jacen.moe		lemmy.obscuro.be
dit.reformed.social		lemmy.okr765.com
doomscroll.n8e.dev		lemmy.pe1uca.dev
downonthestreet.eu		lemmy.peoplever.se
dullsters.net			lemmy.phschaad.com
enterprise.lemmy.ml		lemmy.pierre-couy.fr
expats.zone			lemmy.pifferi.io
falconry.party			lemmy.pit.ninja
fanaticus.social		lemmy.plaureano.nohost.me
fed.dyne.org			lemmy.pos005.com
feddit.bg			lemmy.projetretro.io
feddit.cl			lemmy.pt
feddit.dk			lemmy.quirijngb.com
feddit.it			lemmy.radio
feddit.rocks			lemmy.rhymelikedi.me
feddit.world			lemmy.rimkus.it
feditown.com			lemmy.rochegmr.com
fenmou.cyou			lemmy.runesmite.com
flamewar.social			lemmy.sador.me
foros.fediverso.gal		lemmy.schlunker.com
forum.ayom.media		lemmy.secnd.me
forum.novopoder.org		lemmy.securitycafe.ca
fstab.sh			lemmy.seekpie.nohost.me
gearhead.town			lemmy.self-hosted.site
gioia.news			lemmy.services.coupou.fr
hardware.watch			lemmy.shiny-task.com
healthy.community		lemmy.sieprawski.pl
helvetiverse.ch			lemmy.sietch.online
hi-fi.community			lemmy.simpl.website
hobbit.world			lemmy.skoops.social
hoihoi.superboi.eu.org		lemmy.skyjake.fi
hub.astromagna.com		lemmy.snoot.tube
ibbit.at			lemmy.snoozetown.org
info.prou.be			lemmy.sotu.casa
journal.sdelaem.agency		lemmy.spaceships.me
krabb.org			lemmy.spv.sh
kuu.kohana.fi			lemmy.squidcopter.homeip.net
kyu.de				lemmy.squids.ca
l.7rg1nt.moe			lemmy.ssba.com
l.henlo.fi			lemmy.stad.social
l.hostux.net			lemmy.staphup.nl
l.iri.cx			lemmy.starlightkel.xyz
l.mathers.fr			lemmy.stefanoprenna.com
l.mchome.net			lemmy.sumuun.net
l.posterdati.it			lemmy.super.ynh.fr
l.shoddy.site			lemmy.syrasu.com
l.swee.codes			lemmy.t-rg.ws
leaf.dance			lemmy.techhaven.io
lebowski.social			lemmy.technowizardry.net
lef.li				lemmy.tespia.org
leftopia.org			lemmy.tgxn.net
legbeard.xyz			lemmy.thc.sh
lem.afiz.org			lemmy.thefloatinglab.world
lem.free.as			lemmy.thewooskeys.com
lem.ph3j.com			lemmy.tobyvin.dev
lem.serkozh.me			lemmy.toot.pt
lem.trashbrain.org		lemmy.trippy.pizza
lem.ugh.im			lemmy.uhhoh.com
lemminielettrici.it		lemmy.umany222.net
lemmit.nyc.what.if.ua		lemmy.uninsane.org
lemmus.org			lemmy.vyizis.tech
lemmy-api.ten4ward.social	lemmy.w9r.de
lemmy.0upti.me			lemmy.waynetec.us
lemmy.100010101.xyz		lemmy.xaetacore.net
lemmy.30p87.de			lemmy.zhukov.al
lemmy.411426.xyz		lemmy.zimage.com
lemmy.4d2.org			lemmy.zwanenburg.info
lemmy.86thumbs.net		lemmy2.greatpyramid.social
lemmy.8bitar.io			lemmybefree.net
lemmy.8th.world			lemmygrad.ml
lemmy.activitypub.academy	lemmyis.fun
lemmy.ahall.se			lemmyjan.xyz
lemmy.amethyst.name		lemmyland.com
lemmy.amxl.com			lemmyusa.com
lemmy.ananace.dev		level-up.zone
lemmy.anonion.social		libretechni.ca
lemmy.anymore.nl		linkage.ds8.zone
lemmy.az.social			links.gayfr.online
lemmy.azamserver.com		links.nadia.moe
lemmy.b0tt0m.xyz		links.rocks
lemmy.baguenau.de		linux.community
lemmy.baie.me			linz.city
lemmy.balamb.fr			lm.inu.is
lemmy.beagle.quest		lm.madiator.cloud
lemmy.beru.co			lm.preferlinux.de
lemmy.bestiver.se		lmm.dru5k1.io
lemmy.blackeco.com		lmy.sagf.io
lemmy.bosio.info		lonestarlemmy.mooo.com
lemmy.bowyerhub.uk		lsmu.schmurian.xyz
lemmy.bp99.eu			lt.harding.dev
lemmy.brandyapple.com		lu.skbo.net
lemmy.brdsnest.net		martinlm.mypi.co
lemmy.brief.guru		matchpoint.zone
lemmy.caliban.io		metawire.eu
lemmy.casasnow.noho.st		mlem.eldritch.gift
lemmy.chigityk.com		monyet.cc
lemmy.cleberg.net		mouse.chitanda.moe
lemmy.cogindo.net		mtgzone.com
lemmy.conorab.com		mujico.org
lemmy.corbin.sh			natur.23.nu
lemmy.coupou.fr			nba.space
lemmy.cringecollective.io	netheads.online
lemmy.criticalbasics.xyz	news.idlestate.org
lemmy.cronyakatsuki.xyz		no.lastname.nz
lemmy.cultimean.group		nodesphere.site
lemmy.curiana.net		notdigg.com
lemmy.darkc0de.one		nsfwaiclub.com
lemmy.darvit.nl			orbiting.observer
lemmy.deadca.de			orcas.enjoying.yachts
lemmy.decronym.xyz		overctrl.dbzer0.com
lemmy.deedium.nl		parenti.sh
lemmy.dev.sebathefox.dk		poeng.link
lemmy.dexlit.xyz		ponder.cat
lemmy.digitalcharon.in		popplesburger.hilciferous.nl
lemmy.digitalfall.net		potato-guy.space
lemmy.dokmelody.org		poweruser.forum
lemmy.dryja.dev			preserve.games
lemmy.duckpond.social		pridehaven.social
lemmy.dudeami.win		providence.root.sx
lemmy.elbmatsch.de		ragu.meadow.cafe
lemmy.elxpd.com			rdrama.co
lemmy.enchanted.social		realbitcoin.cash
lemmy.enticedwanderer.com	reddeet.com
lemmy.esquiretheduke.nohost.me	rekabu.ru
lemmy.eus			rentadrunk.org
lemmy.evangineer.net		roanoke.social
lemmy.fait.ch			rollenspiel.forum
lemmy.federate.cc		rqd2.net
lemmy.fedi.zutto.fi		rss.ponder.cat
lemmy.fedifriends.social	sammich.es
lemmy.fediverse.jp		sappho.social
lemmy.ferg.al			selfhosted.forum
lemmy.fish			sfw.community
lemmy.flicke.red		sha1.nl
lemmy.freakaria.com		shibanu.app
lemmy.freewilltiger.page	slangenettet.pyjam.as
lemmy.fromshado.ws		slrpnk.net
lemmy.frozeninferno.xyz		soccer.forum
lemmy.fwgx.uk			social.belowland.com
lemmy.gf4.pw			social.dn42.us
lemmy.giftedmc.com		social.ggbox.fr
lemmy.glasgow.social		social.jears.at
lemmy.glue.io			social.nerdhouse.io
lemmy.graphics			social.p80.se
lemmy.grys.it			social.packetloss.gg
lemmy.halfbro.xyz		social.sour.is
lemmy.halis.io			spgrn.com
lemmy.hangdaan.com		stammtisch.hallertau.social
lemmy.helios42.de		sub.wetshaving.social
lemmy.helvetet.eu		supernova.place
lemmy.hogru.ch			swg-empire.de
lemmy.horwood.cloud		switter.su
lemmy.hybridsarcasm.xyz		szmer.info
lemmy.igl.ooo			tech.pr0n.pl
lemmy.imagisphe.re		test2.lealternative.net
lemmy.inbutts.lol		timeperiods.fr
lemmy.izanami.moe		timesink.p3nguin.org
lemmy.jamesj999.co.uk		tkohhh.social
lemmy.javant.xyz		toast.ooo
lemmy.jaypg.pw			twun.io
lemmy.jelliefrontier.net	ukfli.uk
lemmy.jhjacobs.nl		unlemmy.com
lemmy.johnnei.org		upvote.au
lemmy.jonaharagon.net		usenet.lol
lemmy.kaytse.fun		va11halla.bar
lemmy.kde.social		viewfinder.pro
lemmy.keniir.wtf		waterloolemmy.ca
lemmy.killtime.online		welppp.com
lemmy.kjkalle.net		wired.bluemarch.art
lemmy.klein.ruhr		wolf3d.space
lemmy.kmoneyserver.com		xn--mh-fkaaaaaa.schuetze.link
lemmy.kokomo.cloud		ymous.au
lemmy.lama-corp.space		zenith.digitalunderworlds.com
lemmy.lantian.pub		zonenranslite.de
3
 
 

I fed this output:

https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D

Into json_pp | grep name, and got:

"name" : "science_memes",
               "name" : "al_gore",
               "name" : "applied_paranoia",
               "name" : "windowmanagers",
               "name" : "hihihi",
               "name" : "media_reviews",
               "name" : "petits_animaux",
               "name" : "twnw",
               "name" : "niagaraonthelake",
               "name" : "niagarafalls",

That’s it. There are no more names. Inspecting the dataset seems to show a lot of communities, but only their number. Is there a separate table that maps community numbers to names?

(previous discussion for reference)

4
 
 

cross-posted from: https://slrpnk.net/post/24729225

Are there any open datasets that track who is federated / defederated with who? I have the Lemmyverse datasets but it has nothing on node relationships.

lemmy-federate.com appears to have the info I am after, but it only appears to be reachable in a GUI webpage. Any way to get the dataset?

5
 
 

Could have sworn it used to work and I can't think of anything I've changed? Yet when I look every single thing is being denied from quokk.au. Any ideas why it's no longer syncing?

Is PieFed not compatible with the auto-adding?

6
 
 

I made this change to increase software support for Lemmy Federate. This way, besides Lemmy and Mbin, other software that supports FEP-1b12 such as NodeBB, Piefed, Guppe, Friendica or other custom implementations (like blogs) will be able to register to Lemmy Federate.

However, I must admit that this change makes the login mechanism more difficult. But I believe it was worth it.

An example authentication flow:

  • Enter instance host and click login

  • Add given record to DNS, and click login again

  • At this point, API key becomes reusable between logins.

If you experience any problems please let me know.

7
 
 

I was looking for a particular instance by it's not listed on the tool. I couldn't find info on what to do in this case. Is this something the instance admin must do?

8
 
 

Right now Lemmy federate says piefed.social isn't registered. I went to their matrix and asked the admin about adding lemmy-federate. They said they had attempted to do so but the Lemmy federate bot didn't recognize their account as an admin account.

With the end of lemm.ee and the piefed.social migration tools we are going to see a lot of communities pop up on piefed.

Would it be possible to either add piefed support or manually flag the piefed social admin account as validated?

9
 
 

What do you plan on doing with lemmy federate once this feature is implemented?

https://github.com/LemmyNet/lemmy/issues/2951

10
 
 

Report of the same issue in lemmy support - https://hackertalks.com/post/9701951

If a user is the only subscriber on a instance, and gets banned, and unbanned, then unban wont federate.

If lemmy-federate would subscribe even if there are already local subscribers, then this lemmy issue would be moot for lemmy-federated communities (there would always be at least one subscriber on the instance with the lemmy-federate bot, so the unban gets federated).

11
7
submitted 3 months ago* (last edited 3 months ago) by iso@lemy.lol to c/lemmyfederate@lemy.lol
 
 

Sometimes instances can crash or slow down while following other instances. Or the remote instance may be blocked or the bot may be stuck at the rate limit.

To troubleshoot such issues I added a logs section to the instance management page.

Note: these logs are responses of requests sent to your instance. You can only see the logs for your own instance.

Some techie friends may ask why I didn't use something like Grafana. My answer is, I was just lazy :)

12
 
 

Hey how can we troubleshoot when communities are not being federated?

For example, this community shows that there was an error federating to Lemmy.nz (but not what the error was). Previously it was showing as not allowed. screen shot showing metabolic_health@lemm.ee is not allowed to federate

Here are my settings: settings showing things like that it's enabled, disables NSFW, requires fediseer endorsement

There are other communities listed in this post, though really I'd like the tools to troubleshoot myself: https://lemmy.nz/post/21531352

Any help appreciated 🙂

13
 
 

Generic Threadiverse support

Thanks to @rikudou@lemmings.world's contribution (#28), Lemmy Federate now supports all software types that implements group federation such as PieFed, NodeBB, Guppe 🎉

But unfortunately, not everything is perfect. Since there is no Fediverse standard for verifying whether a user is an admin, I have to register admins manually. I am also considering manually approving instances that are not guaranteed in Fediseer against spam attacks. Please contact me for this.

Note: Lemmy and Mbin works as before.

Top instances of Lemmy

With the addition of Lemmy.ml, the top 25 largest instances on Lemmy now use Lemmy Federate (except slrpnk.net). I think we can now consider that we have fixed the accessibility issue that was the reason I created this tool. Even if we didn't fix it, at least we band-aided it :)

Instance blocking feature

In addition to the allow list, a block list has been added.

  • If you allow at least one instance, you will not follow any other instances.
  • If you block an instance, you will continue to follow instances other than those you blocked.

Dedicated community

I didn't want to open it before, but now that we are trying to be compatible with more software, I believe a dedicated community could be useful. That's why I created a community here !lemmyfederate@lemy.lol. If I make an update from now on, I'll probably post it there.

https://lemmy-federate.com/ https://github.com/ismailkarsli/lemmy-federate