How I Subscribed to Every Melonlander's RSS Feed!

[Dan Q]

The Melonlander Superfeed!

tl;dr: I made an OPML file which you can use to tell your feed reader to instantly subscribe to every Melonlander's RSS feed! Want it? here it is!


The what now?

As I was saying in the other thread, I love RSS feeds and my RSS reader (FreshRSS) is the first and last thing I look at most days. (Like @mittens, I use FreshRSS; that doesn't matter much to this post, but there's one cool feature about it that I'll mention later!)

Wouldn't it be cool, I thought, if I could subscribe to every Melonlander's RSS feed (I can always prune the ones that don't interest me later). That way, I'd be able to hear the latest updates from so many of the most-wonderful people on the Internet, all at once.

This post describes (a) how I did it and (b) how you can do it too (it's not hard, now I've done the leg-work!). Here we go...


How I made an OPML file
(skip this bit if you just want to know how to add Melonlanders to your own feed reader!)

Step 1: Getting the data

Every Melonland profile page has space for (a) your website, and (b) your RSS feed (a way of subscribing to updates from websites). If you've got one (or both), you'll hopefully have filled these bits of your profile in.

There's no nice way to export all this data, so I used a sneaky (read: hacky, dirty) trick. First, I used the member search to find everybody who had "http" written somewhere in the "website" field on their profile (my assumption is that nobody will have listed their RSS feed without also listing their website). Next, I wrote a userscript (a piece of Javascript that runs inside a plugin in your browser - I use Violentmonkey - to improve websites you visit in a way that you choose): my userscript gradually clicks through every page of those search results and on every member who matched the search. Obviously I didn't want to hammer the forum, so it does it very gradually (with a three second gap between looking at one page and the next): it took about an hour and a half to complete!

(Obviously I'd love a faster way to do this: if @Melooon wants to e.g. add an RSS feed column to the output of search then that'd speed things up to under a minute! Or JSON output would be fine too! Or if he shares his DB schema I'm happy to write the code for him.)

As it visited each of about 1,600 profile pages, it collected the website address and, if present, RSS feed address from each. Once it was done, it output an OPML file (basically a way to list lots of web links, particularly those with feeds: this will become relevant later!).

Step 2: Tidying the data

I anticipated that some of the data might be invalid: links to dead websites, invalid RSS feeds, etc. So I wrote another program, this time in Ruby, which would check them all. It opened the OPML file and then tried to connect to every single URL in there. If a web address or feed seemed invalid (404'd, 500'd, wasn't the right type e.g. got HTML back when asking for RSS, etc.), it was removed from the data.

Another way I did some tidying up was that if a website was listed but not a feed, I attempted to detect where a feed might be. Web developers can put a bit of metadata in their pages to say "the RSS feed is over there", so I tried to detect that and "fix" the missing RSS URLs where the website seemed to have one (and it was valid).

This gave me two further outputs: (1) the same OPML file but with only believed-valid data in it, and (2) a filtered version of the OPML file that contained only the sites with working, valid RSS feeds (that's the one I'm interested in!).


Using the OPML file

Most RSS readers can import an OPML file, which automatically makes them subscribe to everything listed within. If you want to do that, go to https://gist.githubusercontent.com/Dan-Q/8bff16a01c4896972efa7801651643eb/raw/9763922a43821825ee79e965c0c0ae2d3cdf841c/melonland.opml.feeds.xml, save the file, and upload it to your RSS reader. Done!

Some RSS readers, including my beloved FreshRSS, can subscribe to an OPML file. This means that if the OPML file gets modified later on (e.g. somebody new comes along and adds their feed to their Melonland profile, and I run my scripts above again!) then FreshRSS will automatically pick it up!

In the hope that one day I can make it do that, I used this FreshRSS feature:

• I clicked Subscription Management > Add a feed or category > Add dynamic OPML
• I entered the name "Melonland"
• I added the URL https://gist.githubusercontent.com/Dan-Q/8bff16a01c4896972efa7801651643eb/raw/9763922a43821825ee79e965c0c0ae2d3cdf841c/melonland.opml.feeds.xml.
• I clicked to save
• I waited for a surprisingly long time while the RSS reader did its initial fetches of the new feeds
• I asked it to delete everything over a month old (some of you keep a lot of archived data in your RSS feeds!)

Seeing Melonlanders' Updates!

The attached screenshot shows how my feed reader displays the latest updated by around 76 Melonlanders (except for those I've already muted!). My feed reader will retain their latest updates until I read them (or until they're marked as expired: my feed reader can be configured to purge old entries if they're not read in a timely fashion, for example): you'll see I'm reading a blog post about "Blogcember" by @Loebas and I can see plenty of posts by other people too (maybe you can spot your own!).

If this is useful to anybody else, let me know: I'm happy to expand it into something better/smarter/automated (especially if I can get easier access to the right data so I can generate it in closer to 90 seconds, rather than an hour!). Otherwise: it's useful to me, but if it's only me it helps I'll just run the code as-and-when I need it and not think about automation!

(A tertiary benefit of my script is that it identified lots of people whose links weren't valid; if that's you, I'll try to reach out by commenting on your profile pages, so you know!)

[Melooon]

This is cool, thanks for putting this together! I did set up an experiment at https://forum.melonland.net/feeds.php last year (I cant guarantee this page will stay online, but we can definitely come up with a better alternative); your system seems much more comprehensive and scalable, the issue I ran into was that processing feeds server side was really not practical as more members joined and promoting it would put a bunch of extra load on the server; plus I never thought to account for unlisted feeds on peoples linked websites

I attached the feeds draft below if you're interested; at any rate I'll definitely try this one out!

[Dan Q]

That's cool! I like the thinking!

If you're able to throw together a page that outputs all members, their website names and URLS, and their feed URLs (paginated if you need), in any format you like (JSON's nice, but HTML can be read by humans as well!) then I can streamline my code enough that it might be usable your-side (my PHP's solid; and I'd happily host an open-source aggregator that you can then steal from if you want it!). I've got some experience running aggregators (PlanetPlanet, back in the day, but a few custom ones) and would consider running a public one.

Without that, my scraper is... well, it's slow. An hour and a half to update in a "don't hammer the server" way is still more-aggressive than just getting the data in a single query, of course!

'cos making things is cool. But making data more-accessible means everybody can make things.

[candycanearter07]

(referencing the attached image)

aw you removed mine? :( for shame

(cool idea :D)

[Dan Q]

aw you removed mine? :( for shame

Nope, I'd just already read/marked-as-read them before I took that screenshot!

Your feed has a couple of minor bugs in it, including the fact that it needs an uppercase 'D' in the <pubDate> elements. As a result, my RSS reader decided to date them all "today", and they were getting in the way of a good screenshot! So I looked through them first and then got them out of the way.

I promise you're in my reader, just with everything published so-far marked as read! See new screenshot below!

[Melooon]

@Dan Q Here u go, all link-able services that offer RSS feeds 

Code

https://forum.melonland.net/api.php?memberWebsites - Linked profile websites
https://forum.melonland.net/api.php?rssFeeds - Linked profile RSS feeds
https://forum.melonland.net/api.php?statusCafe - Linked profile StatusCafe names
https://forum.melonland.net/api.php?imood - Linked profile iMood names
https://forum.melonland.net/api.php?spaceHey - Linked profile SpaceHey names, though you need to convert these to IDs to get SpaceHey RSS feeds so I'm not sure how useful it is
https://forum.melonland.net/api.php?memberNames - Display names associated with IDs

Each will return a JSON list of Member Id's and their associated data. You can add an "id" parameter if you want to filter it down to a specific member id, otherwise it will return every nonempty result.

The return values are not validated in any way, so the URLs might not be valid and should be checked after the fact, HOWEVER websites do return a surfclub value, 0false being their failed to pass the surfclub validation (or linked a host such as carrd that we don't accept) and 1 true meaning they where actively online for the last surfclub check.

Hope that helps

(also for any one wondering if its safe to just hand out data like this, Dan already demonstrated that with a quick bit of code you can already scrape this data from the forum using a crawler script, so there's nothing being hidden by not making it accessible  )

EDIT:
Corrected booleans to true and false!