Reading RSS with newsboat and sfeed

RSS is a wonderful technology. It allows one to follow a bunch of media creators without getting ensnared in an endless algorithmic dopamine chase. For more information on RSS, check out: aboutfeeds.com and my related reading on feeds. In this note, I’m going to write up how up how my RSS feed reader works.

I use two tools for reading RSS feeds: newsboat and a home-brewed static blogroll based off sfeed. The latter is a very stripped down RSS reader suitable for hacking around with. You can see my setup here:

https://pgadey.ca/feeds/sfeed.html : All the feeds that I follow and all their posts.
https://pgadey.ca/feeds/sfeed-mini.html : A stripped down summary of all the feeds which just shows the posts made within the last twenty four hours.

Attention Conservation Warning: I’m about to describe a really convoluted way to read RSS. If you’re a reasonable person, you should sign up for a simple RSS reader service like mire. It was fun for me to hack this thing together, but reading RSS doesn’t need to be so complicated. I just like to live in a steaming heap of bash script.

Overview

The high-level view of how things work is this: I add/remove/manage feeds in newsboat. Then, I use a bash alias newsboat-send-feeds to update a list of feeds on my personal server. Lists of feeds are stored in OPML¹ files. The personal server periodically runs a bash script rss-update.sh which uses sfeed to create the full and mini version of my feeds. The bash script also sends a copy of my feeds to ctrl-c.club.

rss-update.sh

And so, let’s get in to the details. The shell script that I use to generated the full feed list and the mini list is here. It comes with no warranty, and needs a lot of tweaking to work with other setups. It takes an input feeds.opml and creates the two static HTML pages listed above.

The code of rss-update.sh

 #!/bin/bash
 
 # output the date for logging purposes
 date --iso=second 
 
 # pgadey's feeds
 # import all feeds from opml to sfeed
 cat "/home/pgadey/public_html/pgadey/feeds/feeds.opml" | sfeed_opml_import > "/home/pgadey/.sfeed/sfeedrc"
 sfeed_update 

 # generate the full feeds list
 sfeed_html /home/pgadey/.sfeed/feeds/* > "/home/pgadey/.sfeed/feeds.html"

 # create a mini-fied version as static html
 echo "Minifying!"
 cat >/home/pgadey/.sfeed/feeds-mini.html <<EOF
 <!DOCTYPE html>
 
 <html>
 <head>
 <title>~pgadey's mini-sfeed</title>
 <meta charset="UTF-8" />
 <link rel="stylesheet" type="text/css" href="style.css">
 </head>
 <body>
 <pre>
 EOF
 
 echo -e "Generated at: $(date --iso=second)">> /home/pgadey/.sfeed/feeds-mini.html

 # find the feeds which have been recently updated (shown in bold <b>)
 # you can hackily include / exclude stuff by modifying the grep here.
 # for example ~gome suggested limiting the number of lines, 
 # an added "head -n10 " would do the trick for ten lines

 cat /home/pgadey/.sfeed/feeds.html | grep '<b>' | grep 'https' >> /home/pgadey/.sfeed/feeds-mini.html
 
 cat >>/home/pgadey/.sfeed/feeds-mini.html <<EOF
 <a href="sfeed.html">Full feed list</a>
 </pre>
 </body>
 </html>
 EOF
 
 echo -e "Done minifying: $(wc -c /home/pgadey/.sfeed/feeds-mini.html)."
 
 # copy over the pages to ctrl-c.club and pgadey.ca
 cp -v /home/pgadey/.sfeed/feeds.html /home/pgadey/public_html/pgadey/feeds/sfeed.html
 cp -v /home/pgadey/.sfeed/feeds-mini.html /home/pgadey/public_html/pgadey/feeds/sfeed-mini.html
 scp -v /home/pgadey/.sfeed/feeds.html ctrl-c.club:/home/pgadey/public_html/sfeed.html
 scp -v /home/pgadey/.sfeed/feeds-mini.html ctrl-c.club:/home/pgadey/public_html/sfeed-mini.html

newsboat

I use newsboat to read RSS locally on my laptop, and to generate the OPML¹ file feeds.opml. It’s great except for the non-obvious default behaviour that it doesn’t automatically refresh all the feeds when you open it. This feels weird to me, and so I added the following to ~/.newsboat/config.

 # load all feeds on startup
 refresh-on-startup yes

The following bash alias sends the feeds to my server and runs rss-update.sh.

 alias newsboat-send-feeds='cd ~/.newsboat; newsboat --export-to-opml > feeds.opml; scp feeds.opml cloudbox:/home/pgadey/public_html/pgadey/feeds/; ssh cloudbox "/home/pgadey/bin/rss-update.sh"'

Once the feeds.opml file is in place, sfeed (and rss-update.sh) will do all the work. rss-update.sh is run via a cron job on my personal server² daily.

How Does This Actually Work in Practice?

If I finding a creator that I want to follow, then I open up newsboat, add their feed, and run newsboat-send-feeds. It takes about thirty seconds. Their feed is then added to the OPML file sitting on my server. The feeds pages will get updated once per day, which is just enough to keep things interesting.

Acknowledgments

My ctrl-c.club friend ~gome wanted to know how my RSS reader worked. Thanks, ~gome for asking about it.

Before writing this, I had no idea what OPML stood for. Turns out that it’s “outline processor markup language”. This is the standard format for exchanging lists of RSS feeds. ↩︎
You might wonder: Why not do all this on ctrl-c.club? That seems like a better place for hacking around with this stuff. I decided to do it on my home server to save our administrator the task of installing sfeed. ↩︎

Meta

Published: Nov 20, 2023

Last Modified: May 13, 2024