Twigfeeds RSS Feed Labelling/Categorisation

Took me some time to get back to this; beta.7 is a quick fix that addresses the Admin-issues. It is because of changes or regression-issues with the selectize-field. I’ve been that road before, and the same problems seem to have returned. Note that contrary to previously, both categories and tags are pluralized and defined as Lists, thus cannot be a string-value and won’t get picked up by category or tag.

user/config/plugins/twigfeeds.yaml bears no relation to the cache, and only defines the settings for the parser and is interpreted by Admin for the GUI. I can see the cache-issue however, it is because the config is not carried over into the result. I’ll look for a fix shortly. EDIT: Should be fixed with beta.8.

Thanks for this, and your continued great effort to evolve the plugin. Please dont feel harrassed by me. Soon I will have less time also :wink:

Update on tests/experience with beta.6:

  • Im testing about 25 feeds. I’ll limit it at 30 I think.
  • Im running categories, tags, a merged full feed and a ‘snapshot’ random 3 entries of the full feed, plus a source list.
  • All tags & categories behave exactly as expected. Using a dynamically rendered page for tag and one for category to call items works brilliantly.
  • Full feed works great, overall as expected. Pagination works great.
  • Anomaly with caching/refreshing Blusesky feeds (see below)
  • Anomaly while using randomize when calling 3 random items from the merged feed as a small preview of the full feed. (see below)

Bluesky cache_time and lastModified

lastModified is calling the most recent refresh of the BS rss by twigfeeds, NOT the original pubDate of the BS item, so it adds the newest refresh date as if its the pubDate - to all the items Im pulling from BS (3 items each for 2 feeds). This only happens with BS feeds, all other feeds retain the original item pubDate and populate the merged feed etc as expected. But when BS feeds are refreshed, all 6 items have the same or very similar new lastModified date and go to the top of the merged feed list and stay there until another feed is refreshed/updated later. They very quickly return to the top if they have short or null cache_time.
I tested this by adding cache_time: 3600 to every feed except the two Bluseky feeds, which have cache_time: 21600. This prevents the BS feeds continuously occupying the top 6 items of the merged feed list. Checking the BS original pubDate in the raw RSS confirms what Im thinking, that lastModified in the case of BS is the time twigfeeds refreshes the BS feed update, and the original pubDate of an item is lost in the twigfeeds cached data. I hope this is understandable.

Odd randomize behaviour

Im doing this to create a small preview of random items from the merged feed. When calling the retrieveTitle field, an odd behaviour occurs - the feed name often just does not show up in the template, even though the item always shows. I cannot work out why this is happening. It affects any feed source. Ive tried numerous configurations of how to call the random 3 entries.

  • For retrievedTitle as
    {% set item = item|merge({ 'retrievedTitle': feed.config.name }) %} or {% set item = item|merge({ 'retrievedTitle': name }) %} both work, but only randomly.

  • For the item call:
    {% for index, item in feed_items|sort_by_key('sortDate')|randomize|slice(0, 3) %}
    Or
    {% for item in feed_items|ksort('retrievedTitle')|randomize|slice(0, 3) %}
    Or
    {% for item in feed_items|sort('sortDate')|randomize|slice(0, 3) %}
    etc

  • The item is called as usual:

   {#  if it has no title, use the description #}
        {% if item.title != true %}
             <li class="">
                <em style="font-weight: 600">{{ item.retrievedTitle }}</em> 
                <a class="feed-desc-url" href="{{ item.link }}">{{ item.content|safe_truncate_html(8)|striptags|raw}}</a>               
                <small>{{ item.lastModified }}</small>
             </li>
        {% else %}         
            {#  otherwise, show the title field #}
             <li class="">
                <em style="font-weight: 600">{{ item.retrievedTitle }}</em> 
                <a class="feed-desc-url" href="{{ item.link }}">{{ item.title }}</a>           
                <small>{{ item.lastModified }}</small>
             </li>
        {% endif %}    

I cannot explain why retrievedTitle only shows up randomly. Any ideas appreciated on this. Maybe I need to do a set random thing but I dont know how.

Ill keep updating the shared Obsidian note with other info as I go along.

BlueSky’s lack of conformance and server-setup will cause this, and as you’ve discovered the solution with aligning TwigFeed’s cache_time to make it behave is insufficient. The offending line is in Parser.php, used as a fallback for this lack of conformance. The parser-library’s getLastModified should handle pubDate, but for the time being this looks like an edge-case with BlueSky that would need further debugging.

Thanks for the sort_by_key reminder, and updating the Obsidian-note, I’d forgotten if Twig can do datetime-parsing and comparison by itself. Seemingly not through the native |sort((a, b) => a.lastModified - b.lastModified).

retrievedTitle is not a standardized property, but seemingly based on the pagination-example. My best guess would be misalignment between TwigFeed’s and Twig’s caching, but the use of the underlying properties feed.config.name and name shouldn’t be affected. A trick for debugging is to add in

echo '<script>window.twig_feeds = ' . json_encode($feed_items) . ';</script>';

before line 372 in twigfeeds.php, to view the actual data passed to Twig on runtime. That is taken directly from the raw data, cached or not. Beyond that, all cached data is in cache://twigfeeds or user://data/twigfeeds, and you’d have to compare if there are any changes to the files that manifest.json refers to in order to discover unexpected variations between processing.

For posterity, the following is an example of flipping the feed-data to group and sort it by taxonomy. Grouping depends on a bit of pre-arranging of metadata:

{# Iterate and find unique tag-values #}
{% set twig_feeds_tags = [] %}
{% for name, feed in twig_feeds %}
  {% for value in feed.config['tags'] %}
    {% if value not in twig_feeds_tags %}
      {% set twig_feeds_tags = twig_feeds_tags|merge([value]) %}
    {% endif %}
  {% endfor %}
{% endfor %}
{# Iterate and find unique category-values #}
{% set twig_feeds_categories = [] %}
{% for name, feed in twig_feeds %}
  {% for value in feed.config['categories'] %}
    {% if value not in twig_feeds_categories %}
      {% set twig_feeds_categories = twig_feeds_categories|merge([value]) %}
    {% endif %}
  {% endfor %}
{% endfor %}

So that you could efficiently regroup, sort, and render them:

{% set twig_feeds_tags_items = [] %}
{% for value in twig_feeds_tags %}
  {% set twig_feeds_filtered = twig_feeds|filter(v => value in v.config.tags) %}
  {% set twig_feeds_filtered_items = [] %}
  {% for name, feed in twig_feeds_filtered %}
    {% set twig_feeds_filtered_items = twig_feeds_filtered_items|merge(feed.items) %}
  {% endfor %}
  <h4>{{ value }} ({{ print_r(twig_feeds_filtered_items|count) }})</h4>
  {% for item in twig_feeds_filtered_items|sort_by_key('lastModified') %}
    <time>{{ item.lastModified }}</time>
    <small>
      <a href="{{ item.link }}">{{ item.title|default(item.link) }}</a>
    </small>
    <br />
  {% endfor %}
{% endfor %}

(forgot to mention, I’ll test beta.7 asap over next few days.)

The offending line is in Parser.php … parser-library’s getLastModified should handle pubDate , but for the time being this looks like an edge-case with BlueSky

Yes definitely the fault of Bluesky, I can see how its behaving. No matter if there are updated items or not, all Bluesky sources appear again at the top whenever Twigfeeds checks the server at cache_time (atm every 6hrs). Every other feed behaves as it should re pubDate and refreshing so I keep that at 30minutes.

updating the Obsidian-note

I’ll tidy this up so its a more logical and usable, for you to see how users encounter and deal with things, and documenting the issues I come across more clearly.

retrievedTitle is not a standardized property

Yes I know, but I was using that for convenience to call the name or feed.config.name, as per the merged feed code pattern/snippet. I will test the code line in twigfeeds.php and see what I can find out. Its very odd. This does not affect item randomization at all.

Thanks also for the more expert code on the tag and category sorting and parsing. I do have a simpler version of doing this but will test your advanced way.

btw I just sorted the source list alphabetically, which is a useful thing to also have in this more advanced and dynamic way of using Twigfeeds with a lot of feeds, it’s very simple - {% set twig_feeds = twig_feeds|sort_by_key('name') %} , then calling as normal.
It looks like this currently

A note on overriding a plugin’s blueprints: I have replicated the issue in Extending a blueprint causes fatal error · Issue #2458 · getgrav/grav-plugin-admin · GitHub.