Twigfeeds RSS Feed Labelling/Categorisation

Hi. Yes Ill try again today. I need to clarify some issues for reference, and in case any of this is relevant. Sorry for long reply.

Im not clear what you mean when you say

But because of the format BlueSky uses, I’m afraid the parser

because the sentence isnt complete.

  1. Its not Bluesky feeds that were breaking Twigfeeds beta 5.1 - I wasn’t running any BS feeds in the main tests (because they didnt work at all, in any Twigfeeds versions). Mastodon feeds, which also do not have the title element, work flawlessly in previous versions of Twigfeeds (4 & 5) and probably this version too, which is interesting. So, its unlikely the lack of title element stops BS feeds from working.

I believe it might be to do with the lack of xml header in the BS feed. If you compare the raw RSS 2.0 from BS and Mastodon - both of which lack the title element), you can see the different formatting and the additional header in the Mastodon code.

Header in the Mastodon RSS 2.0, plus additional RSS 2 attributes:

The Bluesky RSS2.0 without any header or any RSS 2 attributes.

  1. Both Bluesky and Mastodon run without problems when using basic php to fetch. I ran tests earlier this year with basic php SimpleXML, to test various feed types. This test page (not styled much) has the BS and Masto examples.

Example code showing Bluesky and Mastodon feed:

Bluesky

 <div class="grid-item">  
          <h4>brian merchant</h4>
            <ul class="fa-ul" style="--fa-li-margin: 5em;"> 
                <?php
                $rss_feed = simplexml_load_file("https://bsky.app/profile/bcmerchant.bsky.social/rss");
                if (! empty($rss_feed)) {
                    $i = 0;
                    foreach ($rss_feed->channel->item as $feed_item) {
                        if ($i >= 5)
                            break;
                        ?>               
                 <!--  <li> <a class="feed-title-url" href="<?php //echo $feed_item->link; ?>" target="_blank"><?php //echo $feed_item->title; ?></a></li>  -->
               <li class="description"><span class="fa-li"><i class="fa-solid fa-plus"></i></span>
                <a class="feed-desc-url"  target="_blank" href="<?php  echo $feed_item->link; ?>"><?php  echo implode(' ', array_slice(explode(' ', $feed_item->description), 0, 14)) . "..."; ?></a></li>
                <?php
                    $i ++;
                   }
                 }
                ?>
            </ul>   
        </div>

Mastodon

        <div class="grid-item">  
            <h4>paris marx</h4>
               <ul class="fa-ul" style="--fa-li-margin: 5em;"> 
                <?php
                $rss_feed = simplexml_load_file("https://mastodon.online/@parismarx.rss");
                if (! empty($rss_feed)) {
                    $i = 0;
                    foreach ($rss_feed->channel->item as $feed_item) {
                        if ($i >= 5)
                            break;
                        ?>              
                <!--  <li> <a class="feed-title-url" href="<?php //echo $feed_item->link; ?>"><?php //echo $feed_item->title; ?></a></li> -->
                <li class="description"><span class="fa-li"><i class="fa-solid fa-plus"></i></span>
                    <a class="feed-desc-url" target="_blank" href="<?php echo $feed_item->link; ?>"><?php echo implode(' ', array_slice(explode(' ', $feed_item->description), 0, 10)) . "..."; ?></a></li>
                  <?php
                    $i ++;
                   }
                 }
                ?>
            </ul>   
        </div>

I also tested this basic code with usual feeds from blogs or websites and it works fine, but with the additional title element also being used.

  1. When I tested TF5.1 I used standard feeds, but in my working Grav app - so it did have the local user/config/plugins twigfeeds yaml file in place. Next test I’ll do it with all previous Twigfeeds files removed. I will test 5.2 over next few days. Along with BBC and NY Times I will test with a Grav feed from my own site (for reference, with full post text), another blog (substack maybe) and a mastodon and bluesky feed.

NB FetchRSS isnt great in my opinion. To check using another widely used web app Id recommend RSS.app, its much better, but only has a 7 day trial. FetchRSS cannot even capture the individual post links from Bluesky, so it’s very poor. The basic php SimpleXML does a great job at capturing all the detail, for both Mastodon and Bluesky.

Sorry, I hit enter before finishing it. I meant to write that the parser does not return BlueSky’s feed because the parser’s internals seemingly fails to comprehend the XML. I might need a more elaborate test-suite, as you point out that Mastodon does not have the same issue. Most of my tests have been with well-formed, exhaustively defined XML.

It might fix it to just forcibly add that metadata, when recognizing the <rss> as opposed to the <feed> tag. The underlying parser-library, FeedIO, hasn’t been updated much in the past few years, but recently got some more attention. Whilst there are others, and a basic approach also works, the value of it is some conformity in the parsing. I considered unifying multiple libraries, but their dependencies made that untenable.

quick update on todays tests, sadly Im not having any luck with this. (beta 1 and beta 2)
localhost and remote both fail. Im pretty sure this is related to the user/blueprints/plugins/twigfeeds/options.yaml in some way because every time I comment it out (underscore the folder name) the site and admin backend returns to functional.

Errors

I can get it to show up in the admin, and have the cat/tag info from the user/config/plugins/twigfeeds.yaml showing. But, it will then suddenly fail, causing timeout errors. eg these in localhost. This breaks front and back end.

**Fatal error**: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in **F:\WEB\laragon\ www\joyndfyi\cache\compiled\files\efc504e1ba6b299578a8cd10117c6ce3.yaml.php** on line **3**  
  
**Fatal error**: Allowed memory size of 536870912 bytes exhausted (tried to allocate 32768 bytes) in **[site folder path]\vendor\monolog\monolog\src\Monolog\Formatter\LineFormatter.php** on line **1**

**Fatal error**: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in **[site folder path]\vendor\filp\whoops\src\Whoops\Util\SystemFacade.php** on line **117**  
  
**Fatal error**: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in **Unknown** on line **0**

**Fatal error**: Allowed memory size of 536870912 bytes exhausted (tried to allocate 262144 bytes) in **[site folder path]\vendor\rockettheme\toolbox\File\src\AbstractFile.php** on line **284**``` 

When its not timing out, the front end twig template does not parse with feed.config.category or feed.config.tag, and breaks the page, giving an error of undefined data in twigfeeds.php line 324.

    $end = $feed['end'];
                } else {
                    $end = 50;
                }
                $feed['amount'] = abs($start - $end);
                $resource = $parser->parseFeed($feed);
                if ($config['silence_security'] && $resource == null) {
                    continue;
                }
                if (isset($feed['name'])) {
                    $name = $feed['name'];
                } elseif (isset($resource['data']['title'])) {
                    $name = $resource['data']['title'];
                } else {
                    $name = $feed['source'];
                }
                $feed['amount'] = count($resource['data']['items']);
                if (isset($resource['data']) && !empty($resource['data'])) {
                    $feed_items[$name] = array_merge($feed, $resource['data']);
                }
                $debug ? $this->debug($resource['data']) : null;
            }
        }
        $this->grav['twig']->twig_vars['twig_feeds'] = $feed_items;
        if ($debug && (array) $this->config->get('system.debugger.enabled')) {
            $this->grav['debugger']->stopTimer('twigfeeds');
        }
    }
}
 
Arguments
"Undefined array key "data""

The {{ name }} call in the twig template also pulls the source feed url, not the name, for some reason.

Possible issues/causes

  • Plugin conflict? Im only running two main other plugins - SImpleMDE and Add Page by Form. I have tried to test with those disabled but it seems to make no difference.
  • Is there a sitewide config for Blueprints that I dont have? You may have something in user/config/systems.yaml or site.yaml or elsewhere that is making this work.
  • I ended up only testing with BBC, NY Times and my own Grav blog RSS, to remove the possibility that other less well formed RSS was the cause.

Ill keep trying other possible solutions, but I cannot work out why v5 works so easily, and what is going wrong with the path/info in the user blueprints options yaml.

For the blueprints I’m fairly confident the Rockettheme\Toolbox is causing some recursion, because I started getting the same issues after a few cache-hits. At first it loads, saves, renders fine, but then starts exhausting memory even without high memory-use. I’ll need to look into it further, so for now just avoid using user/blueprints/plugins/twigfeeds/options.yaml.

As for BlueSky, and after a lot of digging, the cause is actually both simple and complex: They do not accept HEAD-requests. A GET-request – curl https://bsky.app/profile/did:plc:eclio37ymobqex2ncko63h4r/rss – works fine, adding --head gives 404-response. This should of course be a 405-response. The underlying FeedIO-library uses HEAD for performance-reasons, and so any requests fail. I can imagine a few workarounds that maintains compatibility with the plugin as is, and will look into that further too.

Ok, yes I could see it was about cache and cleared all cache constantly, including manually.

I’ll probably return to the customised v5 fttb as Id like to scope out templates for how to view RSS content from more feeds - eg up to 50 feeds/5 posts each in the plugin, but only viewed by category or tag(s) so not swamping page load. This would make the plugin hugely useful in my opinion. In smol web (aka small tech) circles, ‘leveraging RSS’ is a big thing. I believe the work is worth it.

Having Bluesky work would be a big bonus. Anything you can do is really appreciated.

NB small issue, but my errors for undefined data in twigfeeds.php are probably caused by me testing whether to have debug mode enabled or not. When it’s enabled, it breaks. Disabling it returns the page to parse properly.

Really appreciate your attention on this.

I’ve traced the BlueSky-issue back to an invalid interface-implementation. For now, TwigFeeds v5.1.0-beta.4 patches that, as well as some other code quality-issues.

1 Like

Ill test this asap. Thank for the ref link too.

v5.1.0-beta.6 adds native support for categories and tags. Extending the blueprints has some deep-rooted issues, and does not reliably work as expected.

Ill download and install beta.6. This is great news, that cats and tags will be included as standard functionality.

I tested beta.4 yesterday and on the remote site it works fine, including Bluesky, no problems. Very nice work. Will keep updated.

Update:

Testing mainly on remote site (Ionos LAMP shared).

  • Categiory and tag information will not save. The user/config/plugins/twigfeeds.yaml does not save any category information unless a category or tag already pre-exists in the Grav taxonomy.
  • Writing the twigfeeds.yaml category and tags manually does not show up in the twigfeeds plugin admin panel form
  • I tried testing the twig feed.config.category and feed.config.tags in templates but they do not call any info - even if the category or tag is an existing taxonomy
  • Ive tested using your code patterns as well as mine
  • I note youve added ‘Mode’ to the config fields. Bluesky feeds work with this set as default, it makes no difference as far as I can tell
  • Im not using user/blueprints/plugins/twigfeeds/options.yaml. I assume this is no longer required but perhaps this is not correct.

For information, if I call a Bluesky (or Mastodon) feed, I have a template that uses the first part of the description to act as the title, and have a simple if/else conditional. This means the topic of the post is more clearly visible.

Eg (incl my classes but they are not relevant):

{% for item in feed.items %}

  {% if item.title != true %}

        <li class="description"><span class="fa-li"><i class="fa-solid fa-plus"></i></span>
            <h5><a class="feed-desc-url" href="{{ item.link }}">{{ item.content|safe_truncate_html(8)|striptags|raw}}</a></h5>
           <div class="feed-content">{{ item.content|safe_truncate_html(50)|raw}}</div>
            <div class="feed-pubdate">PubDate: <a class="feed-desc-url" href="{{ item.link }}"><time>{{ item.lastModified }}</time></a></div>
        </li>
              
 {% else %}

        <li class="description"><span class="fa-li"><i class="fa-solid fa-plus"></i></span>
          <h5><a class="feed-desc-url" href="{{ item.link }}">{{ item.title }}</a></h5>
          <div class="feed-content">{{ item.content|safe_truncate_html(50)|raw}}</div>
          <div class="feed-pubdate">PubDate: <a class="feed-desc-url" href="{{ item.link }}"><time>{{ item.lastModified }}</time></a></div>
        </li>

 {% endif %}

I tested both with and without pre-existing taxonomy data from Pages, and it saves either way. Existing taxonomy only shows up in the typeahead-field if it exists for categories or tags. It also renders as expected in Admin the other way around; writing it directly to the config-file.

Do you get any errors logged? It sounds like a cache-issue or config-conflict. After these rounds of testing and changes, it’s sane to set it up from scratch again, to ensure there are no hidden server-errors.

I forgot to change the property in README to categories, but this renders both within the feeds-loop:

{% for category in feed.config.categories %}
    <strong>C: {{ category }}, </strong>
{% endfor %}
{% for tag in feed.config.tags %}
    <strong>T: {{ tag }}, </strong>
{% endfor %}

The mode-setting is a bit of future-proofing, as well as potential debugger: It gets the presumed XML at the target URL without any parsing except directly from XML to JSON. In cases where the parser-library cannot handle the target’s response, such as when BlueSky improperly implements header-responses on their server, it would be a shorter way of getting the data rather than setting up timeouts and middleware.

The customizations to the blueprints are not necessary, but the extends@-method remains because it allows easier composition of blueprints going forward. I need to work out a minimally reproducible example for when and how the upstream-library runs haywire, as the approach to overrides is sound.

Thanks for your patience.

Im clearing cache regularly with cli, as its the most reliable method imo.

Ive thought my issues may arise from plugin conflict? Im running Add Page By Form and SimpleMDE (most likely candidates to potentially cause issues, though they work flawlessly) so will test plugin compatibility generally. Update - no, I dont think this is it bc it makes no difference)

I will also set up a clean install to test (local and remote).

Thanks for these code snippets. I did try ‘categories’ as that is what is in the blueprint, but this may work better.

(For some reason Im baffled by, I cannot get the beta versions of Twigfeeds to work in localhost but Ive set that problem aside for now.)

I only shared the code snippet for bluesky/mastodon in case its relevant to add something like this to the README.

ERROR logs - Im not getting errors afaik in the remote. In localhost the front end breaks altogether and gives "Call to a member function getFeed() on array", from …\user\plugins\twigfeeds\classes\Parser.php124. Im checking SSL cert in Laragon Apache and its path (this may be the issue) and have turned off verify peer/host in grav config. Ill send a link in pm for remote and local error log files you can download.

But, Im focusing on the remote clean install for now.

Update.

(I deleted the pm with link to old logs. Can provide new logs if you wish.)

Great news - Twigfeeds appears to work fine with tags and categories with no issue on the clean remote install. This install includes the plugins Im running on the other site so there mustve been some file corrupted in the old build that I cannot trace.

Ive tested the two news feeds plus a mastodon and a bluesky. All cats and tags are being saved. Note that the {{ name }} field will fetch the feed channel title, not the config name unless you use {{ feed.config.name }} in the twig.
UPDATE2: the cats and tags are saving in user/config/plugins/twigfeeds.yaml but do not show up in the admin form when you return to the twigfeeds plugin later.

Im now looking at how the urls work for feed category and tag links. :slight_smile: As they are not added to the main taxonomy list they need to be called from the twigfeeds yaml. Ill think about that.

Its really great work, thanks.

Localhost is still throwing an error that breaks the interface when twigfeeds is enabled, with errors in Guzzlehttp on multiple lines. The error is cURL error 60: SSL certificate problem: unable to get local issuer certificate . I sorted out previous certificate links for Apache in Laragon (this was \laragon\etc\ssl\cacert.pem, the drive letter had changed!), but this is different and hard to sort out, the laragon cert is installed on the windows system and present in the portable laragon, with a link in the php.ini., but still breaking the site.

Ah yes, the old, obscured cURL-certificate-issue. I had the same issue when switching from PHP 8.1 to 8.2 yesterday, because I forget to update that and error_logs. With my setup the most efficient way is to scoop install cacert and change php.ini to curl.cainfo = "C:\Scoop\apps\cacert\current\cacert.pem". I run most testing off of the internal php bin/grav server.

Do you still have any overrides in users/blueprints/? The categories and tags selectize-fields should accept any valid form of List-definition in YAML.

TwigFeeds does not hook into the regular Taxonomy-functionality of Grav, it only reads these two keys and suggests them in Admin, to keep them separate and simple. The feed name is indeed meant to handle the defined name, whereas title is the fetched one, falling back where necessary – because feeds are so differently used.

scoop install cacert and change php.ini to curl.cainfo = "C:\Scoop\apps\cacert\current\cacert.pem" . I run most testing off of the internal php bin/grav server

Ill check into this. I use Laragon on an external drive so this may be different.

Do you still have any overrides in users/blueprints/ ? The categories and tags selectize-fields should accept any valid form of List-definition in YAML.

can you explain this a bit more? No Im not using anything in user blueprints now.

TwigFeeds does not hook into the regular Taxonomy-functionality of Grav

Yes I know Twigfeeds cats/tags are not part of the main taxonomy. This is a good thing as otherwise youd have problems with duplication and calling data you didnt want in templates - eg posts instead of feed updates.

Im thinking about how Id call tagged content, eg tagged as ‘tech’ feed items/sources, which Im looking at today. If you set up a basic a href the link works (domain.com/feeds/tag:tech) but the trick is in getting Grav to recognise other feed sources tagged with that tag and only returning those in a list. Even though the tags are not stored as a related list, they might be recognised using some conditional if/else with {{ uri.param('tag')| ?? }}.

Ive used this in tagcloud behaviour to assign a title for a returned tags page (eg <h2>{{ page.header.title }} in <em>'{{ uri.param('tag')|humanize }}'</em> </h2>). But this uses taxonomylist plugin to get the tags links. But, if I can make the equivalent of a tagcloud for the feed tag/cat sources, and generate the links, then recognise them with uri:param, then it might work.

.

That would be a possible reason why list-values do not show up in Admin even if they are in a valid YAML config-file.

You’d probably want to loop over the items in the feed-list, |reduce or |map them, and read the resulting values back into a list keyed-map with the matching items added to a list with the same key. Then you can |count each list and spit out their keys to have a size-weighted tagcloud.

why list-values do not show up in Admin

I removed everything in the clean install. The user/blueprints/plugins/twigfeeds/... was causing a lot of issues previously. I thought we no longer need that? The data in the Admin form isnt persistant even though it is saving touser/config/plugins/twigfeed.yaml. Maybe its a cookie thing? Its not great for users if this data isn’t visible in the form. NB Im doing a cli clear cache regularly.

how Id call tagged content

thanks for these tips -Ill investigate to see if I can understand what youre describing.
Ive made a tag list template partial for twigfeeds tags, just modifying fairly basic code from my other site (inherited from taxonomylist plugin). You dont need that plugin for this to work. This is just a list, not weighted, but I think is adequate considering we are only tagging sources, not posts.

  {% for name, feed in twig_feeds %}
          <ul class="tagcloud">
                  {% for tag in feed.config.tags %}
                  <li> <a href="{{ config.site.networks.route }}/tag{{ config.system.param_sep }}{{ tag|e('url') }}">{{ tag }}</a></li>  
                  {% endfor %}
          </ul>
  {% endfor %}

(The config.site.networks.route is just to make a consistent user pathway to a single page that would fetch the sources tagged with the tag or category being clicked.)

The idea is to then only need one page & template for category sources and one for tag sources, and that page acts dynamically to call the relevant data that is tagged/categorised with a particular tag etc. Im thinking of possible ways to do this but we’ll see. I do this with tags in my other site but thats with the Grav taxonomy.

user/blueprints/plugins/twigfeeds/ is no longer necessary, and would likely be the cause of a memory-leak leading to a crash. What is the current contents of your user/config/plugins/twigfeeds.yaml? And after saving, are there any errors logged to error_log or in the browser’s console?

For flexibility I would write a macro that can be imported and used in various contexts, doing the iteration, filtering, and rendering given a specific taxonomy type and key, as the twig_feeds data is available to all templates.

Hi.

Hope youre having a good Sunday.
Ive been doing a bit more and made some headway. But, its better in this Obsidian shared note than all added to this thread: Twigfeeds 5.1.0-beta. 6 notes 2

It covers

  • my current twigfeeds yaml contents
  • caching issues THIS IS THE MAIN CONCERN
  • using url.param to call the tagged content (I got this to work)
  • duplicate issues in the tag list (solved 19-08-25)

I think macros are a bit beyond my tech ability currently but I have looked up how to make one.

Quick update. As Ive figured out the main template issues I’ll carry on now with doing categories as Ive done for tags. It works very well if caching is on and nothing is saved through the Admin panel:

  1. The Twigfeeds caching issue is a bit worrying, that tags and categories stop working (parsing in templates) if caching is not enabled. Static cache seems to make no difference to this. As if the connection to user/config/plugins/twigfeeds.yaml only persists via the cache.

  2. The issue of the tag/category data not showing in the Admin panel means it is lost as soon as the user saves another source (for example), as the user/config/plugins/twigfeeds.yaml is replaced, ad all tag/category info is lost altogether.