@movq@www.uninformativ.de Is there a good way to get jenny to do a one-off fetch of a feed, for when you want to fill in missing parts of a thread? I just added @slashdot@feeds.twtxt.net to my private follow file just because @prologic@twtxt.net keeps responding to the feed :-P and I want to know what he’s commenting on even though I don’t want to see every new slashdot twt.

@falsifian@www.falsifian.org You mean fetching the feed temporarily and then discarding all its twts again? 🤔 I don’t think there’s an easy way to do that, other than filtering in your mail client, yeah. 🤔

@falsifian@www.falsifian.org Ah, I see. 🤔 Maybe I’ll add that. To be honest, I have the same “problem” regarding the slashdot feed. 😅 It’s mostly stuff that I’m not interested in – but from time to time someone replies and then I want to see what it’s about.

@prologic@twtxt.net @falsifian@www.falsifian.org This just popped up in my head: How about adding a “fetch context” feature? Point jenny to some mail file that contains a twt (or pipe it to stdin) and it will try to auto-discover and fetch all related things. Like, if it sees something like @<falsifian https://www.falsifian.org/twtxt.txt>, then it will look in https://www.falsifian.org/twtxt.txt for a twt with hash tkjafka. Maybe even do this recursively until there are no new references anymore. This process could include explicitly querying some user-configurable Yarn pods as well. 🤔

It won’t always work. There’s no guarantee that tkjafka will be present in the given URL.

Hmm. 🤔

(The jenny code is getting a bit long and convoluted. I feel the need to refactor this quite a bit. That’s why I’m not implementing any of this right away.)

@falsifian@www.falsifian.org I’ve pushed a draft into the git repo.

You can now do a “oneshot fetch” for a URL:

jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest

This fetches the entire feed, which might be too much. So there’s also this, which only fetches a single twt:

jenny oneshot-fetch --url https://feeds.twtxt.net/hacker-news-newest/twtxt.txt --nick hacker-news-newest --only-twt-hash r6rbinq

Let me know what you think. 🤔

@falsifian@www.falsifian.org @bender@twtxt.net I pushed an alternative implementation to the fetch-context branch. This integrates the whole thing into mutt/jenny.

You will want to configure a new mutt hotkey, similar to the “reply” hotkey:

macro index,pager <esc>C "\
<enter-command> set my_pipe_decode=\$pipe_decode nopipe_decode<Enter>\
<pipe-message> jenny -c<Enter>\
<enter-command> set pipe_decode=\$my_pipe_decode; unset my_pipe_decode<Enter>" \
"Try to fetch context of current twt, like a missing root twt"

This pipes the mail to jenny -c. jenny will try to find the thread hash and the URL and then fetch it. (If there’s no URL or if the specific twt cannot be found in that particular feed, it could query a Yarn pod. That is not yet implemented, though.)

The whole thing looks like this:

https://movq.de/v/0d0e76a180/jenny.mp4

In other words, when there’s a missing root twt, you press a hotkey to fetch it, done.

I think I like this version better. 🤔

(This needs a lot of testing. 😆)

@prologic@twtxt.net How does yarn.social’s API fix the problem of centralization? I still need to know whose API to use.

Say I see a twt beginning (#hash) and I want to look up the start of the thread. Is the idea that if that twt is hosted by a a yarn.social pod, it is likely to know the thread start, so I should query that particular pod for the hash? But what if no yarn.social pods are involved?

The community seems small enough that a registry server should be able to keep up, and I can have a couple of others as backups. Or I could crawl the list of feeds followed by whoever emitted the twt that prompted my query.

I have successfully used registry servers a little bit, e.g. to find a feed that mentioned a tag I was interested in. Was even thinking of making my own, if I get bored of my too many other projects :-)

@falsifian@www.falsifian.org So yes, you would ask a pod about the missing Twt by hash, or whatever. Pods do this already, even though there aren’t that many now, so it maybe a bit less effective today. However it’s more of a small/tiny “distributed” protocol, you ask any pod.

On registries however, I think a registry is the wrong approach. I see far greater value in feed crawlers and search engines like the (half baked one) I built over at https://search.twtxt.net/

@prologic@twtxt.net What’s the difference between search.twtxt.net and the /api/plain/tweets endpoint of a registry? In my mind, a registry is a twtxt search engine. Or are registries not supposed to do their own crawling to discover new feeds?

@falsifian@www.falsifian.org to my knowledge registries were never designed to crawl the Twtxt space. If they did, they would be considered a search engine 🤣

@prologic@twtxt.net I guess I thought they were search engines. Anyway, the registry API looks like a decent one for searching for tweets. Could/should yarn.social pods implement the same API?

@falsifian@www.falsifian.org I think I’m missing something in my description. When I say “search engine” I also mean “with a crawler” that is able to self-discover feeds. A registry (as designed today, or as the spec described) required users to add their feeds to one or more registries, putting the burden on the user(s). I for example do not bother adding my feed to a registry (which one would I add it to anyway?)

@prologic@twtxt.net I believe you when you say registries as designed today do not crawl. But when I first read the spec, it conjured in my mind a search engine. Now I don’t know how things work out in practice, but just based on reading, I don’t see why it can’t be an API for a crawling search engine. (In fact I don’t see anything in the spec indicating registry servers shouldn’t crawl.)

(I also noticed that https://twtxt.readthedocs.io/en/latest/user/registry.html recommends “The registries should sync each others user list by using the users endpoint”. If I understood that right, registering with one should be enough to appear on others, even if they don’t crawl.)

Does yarnd provide an API for finding twts? Is it similar?

@falsifian@www.falsifian.org You are totally right. The specs are at least “open enough” for us to consider that as an implementation detail. We, and by we I mean @movq@www.uninformativ.de @lyse@lyse.isobeef.org @bender@twtxt.net @xuu@txt.sour.is and others should discuss this in more detail I believe and try to see if we can agree on what we’re trying to solve.

Does yarnd provide an API for finding twts? Is it similar?

No, it doesn’t. But yarns (the search engine/crawler wrote) seems more fitting here. It’s been discussed before, the possibility of building a “Twtxt Register v1” compatible API for yarns. I think a search engine + crawler + registry (especially ones that can form a bit of a “distributed network) are far more useful I think in order to support the actual decentralised Twtxt / Yarn ecosystem (which is how I prefer to describe it).