Lemmy Federate

95 readers

1 users here now

Updates and questions about lemmy-federate.com

founded 5 months ago

MODERATORS

iso@lemy.lol

Accessing the dataset for communities -- and their names (slrpnk.net)

submitted 2 weeks ago* (last edited 2 weeks ago) by activistPnk@slrpnk.net to c/lemmyfederate@lemy.lol

5 comments fedilink hide all child comments

I fed this output:

https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D

Into json_pp | grep name, and got:

"name" : "science_memes",
               "name" : "al_gore",
               "name" : "applied_paranoia",
               "name" : "windowmanagers",
               "name" : "hihihi",
               "name" : "media_reviews",
               "name" : "petits_animaux",
               "name" : "twnw",
               "name" : "niagaraonthelake",
               "name" : "niagarafalls",

That’s it. There are no more names. Inspecting the dataset seems to show a lot of communities, but only their number. Is there a separate table that maps community numbers to names?

(previous discussion for reference)

you are viewing a single comment's thread
view the rest of the comments

[–] activistPnk@slrpnk.net 2 points 1 week ago* (last edited 1 week ago) (1 children)

The communities w/relationships is a much bigger dataset. I tried grabbing 100 records per fetch in a loop with no sleep or throttle. Page 523 had 100 records and page 524 was an empty file. I restarted with the skip at 523 and got to page 531. It died again, this time leaving a file that ended in the middle of a JSON field.

Any suggestions? I wonder if I should put a 1 or 2 second delay between pages so the server is not overloaded.

(update) wow, this is bulkier than I expected. 966mb. Hope that didn’t cause any problems. I guess I won’t do that full fetch again. I don’t suppose there an API parameter to select records with updatedAt newer than a specified date?

(update 2) is skip the number of pages, or records? I treated it as pages but it’s starting to look like that’s number of records -- which would mean I grabbed a lot of dupes. Sorry! (if that’s the case)

(update 3) Shit.. looks like skip is the number of records, which makes sense. Sorry for the waste! I’ll fix my script.

[–] iso@lemy.lol 2 points 1 week ago

Good to hear the problem was that!