@prologic@twtxt.net Thanks for writing that up!

I hope it can remain a living document (or sequence of draft revisions) for a good long time while we figure out how this stuff works in practice.

I am not sure how I feel about all this being done at once, vs. letting conventions arise.

For example, even today I could reply to twt abc1234 with “(#abc1234) Edit: …” and I think all you humans would understand it as an edit to (#abc1234). Maybe eventually it would become a common enough convention that clients would start to support it explicitly.

Similarly we could just start using 11-digit hashes. We should iron out whether it’s sha256 or whatever but there’s no need get all the other stuff right at the same time.

I have similar thoughts about how some users could try out location-based replies in a backward-compatible way (append the replyto: stuff after the legacy (#hash) style).

However I recognize that I’m not the one implementing this stuff, and it’s less work to just have everything determined up front.

Misc comments (I haven’t read the whole thing):

• Did you mean to make hashes hexadecimal? You lose 11 bits that way compared to base32. I’d suggest gaining 11 bits with base64 instead.

• “Clients MUST preserve the original hash” — do you mean they MUST preserve the original twt?

• Thanks for phrasing the bit about deletions so neutrally.

• I don’t like the MUST in “Clients MUST follow the chain of reply-to references…”. If someone writes a client as a 40-line shell script that requires the user to piece together the threading themselves, IMO we shouldn’t declare the client non-conforming just because they didn’t get to all the bells and whistles.

• Similarly I don’t like the MUST for user agents. For one thing, you might want to fetch a feed without revealing your identty. Also, it raises the bar for a minimal implementation (I’m again thinking again of the 40-line shell script).

• For “who follows” lists: why must the long, random tokens be only valid for a limited time? Do you have a scenario in mind where they could leak?

• Why can’t feeds be served over HTTP/1.0? Again, thinking about simple software. I recently tried implementing HTTP/1.1 and it wasn’t too bad, but 1.0 would have been slightly simpler.

• Why get into the nitty-gritty about caching headers? This seems like generic advice for HTTP servers and clients.

• I’m a little sad about other protocols being not recommended.

• I don’t know how I feel about including markdown. I don’t mind too much that yarn users emit twts full of markdown, but I’m more of a plain text kind of person. Also it adds to the length. I wonder if putting a separate document would make more sense; that would also help with the length.

@prologic@twtxt.net the basic idea was to stem the hash.. so you have a hash abcdef0123456789... any sub string of that hash after the first 6 will match. so abcdef, abcdef012, abcdef0123456 all match the same. on the case of a collision i think we decided on matching the newest since we archive off older threads anyway. the third rule was about growing the minimum hash size after some threshold of collisions were detected.

@sorenpeter@darch.dk hmm, how does your client handles “a little editing”? I am sure threads would break just as well. 😉

@movq@www.uninformativ.de going a little sideways on this, “*If twtxt/Yarn was to grow bigger, then this would become a concern again. But even Mastodon allows editing, so how much of a problem can it really be? 😅*”, wouldn’t it preparing for a potential (even if very, very, veeeeery remote) growth be a good thing? Mastodon signs all messages, keeps a history of edits, and it doesn’t break threads. It isn’t a problem there.😉 It is here.

I think keeping hashes is a must. If anything for that “feels good” feeling.

@quark@ferengi.one Oh, sure, it would be nice if edits didn’t break threads. I was just pondering the circumstances under which I get annoyed about data being irrecoverably deleted or otherwise lost.

@quark@ferengi.one I don’t really mind if the twt gets edited before I even fetch it. I think it’s the idea of my computer discarding old versions it’s fetched, especially if it’s shown them to me, that bugs me.

But I do like @movq@www.uninformativ.de’s suggestion on this thread that feeds could contain both the original and the edited twt. I guess it would be up to the author.

@prologic@twtxt.net sorry but nope. Neither jenny, nor yarnd supports it at all. This was treated as a thread because I picked one of @falsifian@www.falsifian.org’s twtxts (with the “old subject”), and replied to it (hence starting the thread).

@prologic@twtxt.net based on @falsifian@www.falsifian.org’s findings, I don’t believe this is quite accurate.

“yarnd(_at least_) doesn't support creating such a custom TwtSubject, but it will reply and respect and thread one if one was constructed."

Hmm, but yarnd also isn’t showing these twts as being part of a thread. @prologic@twtxt.net you said yarnd respects customs subjects. Shouldn’t these twts count as having a custom subject, and get threaded together?

@sorenpeter@darch.dk I like this idea. Just for fun, I’m using a variant in this twt. (Also because I’m curious how it non-hash subjects appear in jenny and yarn.)

URLs can contain commas so I suggest a different character to separate the url from the date. Is this twt I’ve used space (also after “replyto”, for symmetry).

I think this solves:

• Changing feed identities: although @mckinley@twtxt.net points out URLs can change, I think this syntax should be okay as long as the feed at that URL can be fetched, and as long as the current canonical URL for the feed lists this one as an alternate.
  
• editing, if you don’t care about message integrity
  
• finding the root of a thread, if you’re not following the author
  

An optional hash could be added if message integrity is desired. (E.g. if you don’t trust the feed author not to make a misleading edit.) Other recent suggestions about how to deal with edits and hashes might be applicable then.

People publishing multiple twts per second should include sub-second precision in their timestamps. As you suggested, the timestamp could just be copied verbatim.

@prologic@twtxt.net

They’re in Section 6:

• Receiver should adopt UDP GRO. (Something about saving CPU processing UDP packets; I’m a but fuzzy about it.) And they have suggestions for making GRO more useful for QUIC.

• Some other receiver-side suggestions: “sending delayed QUICK ACKs”; “using recvmsg to read multiple UDF packets in a single system call”.

• Use multiple threads when receiving large files.

So this is a great thread. I have been thinking about this too.. and what if we are coming at it from the wrong direction? Identity being tied to a given URL has always been a pain point. If i get a new URL its almost as if i have a new identity because not only am I serving at a new location but all my previous communications are broken because the hashes are all wrong.

What if instead we used this idea of signatures to thread the URLs together into one identity? We keep the URL to Hash in place. Changing that now is basically a no go. But we can create a signature chain that can link identities together. So if i move to a new URL i update the chain hosted by my primary identity to include the new URL. If i have an archived feed that the old URL is now dead, we can point to where it is now hosted and use the current convention of hashing based on the first url:

The signature chain can also be used to rotate to new keys over time. Just sign in a new key or revoke an old one. The prior signatures remain valid within the scope of time the signatures were made and the keys were active.

The signature file can be hosted anywhere as long as it can be fetched by a reasonable protocol. So say we could use a webfinger that directs to the signature file? you have an identity like frank@beans.co that will discover a feed at some URL and a signature chain at another URL. Maybe even include the most recent signing key?

From there the client can auto discover old feeds to link them together into one complete timeline. And the signatures can validate that its all correct.

I like the idea of maybe putting the chain in the feed preamble and keeping the single self contained file.. but wonder if that would cause lots of clutter? The signature chain would be something like a log with what is changing (new key, revoke, add url) and a signature of the change + the previous signature.

# chain: ADDKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w 
# sig: BEGIN SALTPACK SIGNED MESSAGE. ... 
# chain: ADDURL https://txt.sour.is/user/xuu
# sig: BEGIN SALTPACK SIGNED MESSAGE. ...
# chain: REVKEY kex14zwrx68cfkg28kjdstvcw4pslazwtgyeueqlg6z7y3f85h29crjsgfmu0w
# sig: ...

@lyse@lyse.isobeef.org This looks like a nice way to do it.

Another thought: if clients can’t agree on the url (for example, if we switch to this new way, but some old clients still do it the old way), that could be mitigated by computing many hashes for each twt: one for every url in the feed. So, if a feed has three URLs, every twt is associated with three hashes when it comes time to put threads together.

A client stills need to choose one url to use for the hash when composing a reply, but this might add some breathing room if there’s a period when clients are doing different things.

(From what I understand of jenny, this would be difficult to implement there since each pseudo-email can only have one msgid to match to the in-reply-to headers. I don’t know about other clients.)

@movq@www.uninformativ.de @prologic@twtxt.net Another option would be: when you edit a twt, prefix the new one with (#[old hash]) and some indication that it’s an edited version of the original tweet with that hash. E.g. if the hash used to be abcd123, the new version should start “(#abcd123) (redit)”.

What I like about this is that clients that don’t know this convention will still stick it in the same thread. And I feel it’s in the spirit of the old pre-hash (subject) convention, though that’s before my time.

I guess it may not work when the edited twt itself is a reply, and there are replies to it. Maybe that could be solved by letting twts have more than one (subject) prefix.

But the great thing about the current system is that nobody can spoof message IDs.

I don’t think twtxt hashes are long enough to prevent spoofing.

The actual end-user problem is that I can’t see the thread properly when using neomutt+jenny.

@prologic@twtxt.net How does yarn.social’s API fix the problem of centralization? I still need to know whose API to use.

Say I see a twt beginning (#hash) and I want to look up the start of the thread. Is the idea that if that twt is hosted by a a yarn.social pod, it is likely to know the thread start, so I should query that particular pod for the hash? But what if no yarn.social pods are involved?

The community seems small enough that a registry server should be able to keep up, and I can have a couple of others as backups. Or I could crawl the list of feeds followed by whoever emitted the twt that prompted my query.

I have successfully used registry servers a little bit, e.g. to find a feed that mentioned a tag I was interested in. Was even thinking of making my own, if I get bored of my too many other projects :-)

@quark@ferengi.one Check out this thread if you haven’t already: https://mastodon.social/@sundogplanets/112464533481477428

I think we already know It’s likely to be a disaster.

@movq@www.uninformativ.de I think I have got it, but need to test upon receiving further posts. I added:

set uncollapse_new     = yes  # open threads when new mail
set uncollapse_jump    = yes  # jump to unread message when uncollapse
set collapse_unread    = no   # don't collapse threads with unread mails

Let’s see how it goes.

Collapsed threads, that is. If I un-collapse a thread, new/unread messages show on the intended new colour, but while the thread is in collapsed state, there is no highlight.

@bender@twtxt.net, cool, so I can join the threads, but your edit to the original will never show at my end. Will have @bender@twtxt.net show the screenshot.

For the mutt/neomutt users out here, what’s the trick to highlight threads with new messages? No user interaction, just upon opening, or while opened, have threads with new, unread messages in it highlighted. Thanks!

@movq@www.uninformativ.de, using the branch on topic right now, it works perfect. The only thing I found was that I had to quit neomutt, and re-open, to see the perfect thread. Other than that, I love it!

@movq@www.uninformativ.de Is there a good way to get jenny to do a one-off fetch of a feed, for when you want to fill in missing parts of a thread? I just added @slashdot@feeds.twtxt.net to my private follow file just because @prologic@twtxt.net keeps responding to the feed :-P and I want to know what he’s commenting on even though I don’t want to see every new slashdot twt.

@bender@twtxt.net He is running on the latest macbook pro with 128G memory. though the chrome app seems to be sitting at 125MB. i am a bit suspicious about that stat since we dont see all the worker threads and he is currently sitting on 40GB of non cache ram.

Started writing something from scratch yesterday using thread(3) and wow do I miss writing in Limbo instead. :-/ #plan9

Twtxt spec enhancement proposal thread 🧵

Adding attributes to individual twts similar to adding feed attributes in the heading comments.

https://git.mills.io/yarnsocial/go-lextwt/pulls/17

The basic use case would be for multilingual feeds where there is a default language and some twts will be written a different language.

As seen in the wild: https://eapl.mx/twtxt.txt

The attributes are formatted as [key=value]

They can show up in the twt anywhere it is not enclosed by another element such as codeblock or part of a markdown link.

@sorenpeter@darch.dk this makes sense as a quote twt that references a direct URL. If we go back to how it developed on twitter originally it was RT @nick: original text because it contained the original text the twitter algorithm would boost that text into trending.

i like the format (#hash) @<nick url> > "Quoted text"\nThen a comment
as it preserves the human read able. and has the hash for linking to the yarn. The comment part could be optional for just boosting the twt.

The only issue i think i would have would be that that yarn could then become a mess of repeated quotes. Unless the client knows to interpret them as multiple users have reposted/boosted the thread.

The format is also how iphone does reactions to SMS messages with +number liked: original SMS

@movq@www.uninformativ.de It took a little over a minute on my machine.. i should try to make it multi threaded.. 🤔

Executed in   68.96 secs    fish           external
   usr time   60.84 secs  242.00 micros   60.84 secs
   sys time   12.52 secs  252.00 micros   12.52 secs

So far it all seems prey snappy. No long pauses when pulling up threads at all.

@prologic@twtxt.net was this in reply to a different thread? Or maybe a hash collision?

I’ve only been using snac/the fediverse for a few days and already I’ve had to mute somebody. I know I come on strongly with my opinions sometimes and some people don’t like that, but this person had already started going ad hominem (in my reading of it), and was using what felt to me like sketchy tactics to distract from the point I was trying to make and to shut down conversation. They were doing similar things to other people in the thread so rather than wait for it to get bad for me I just muted them. People get so weirdly defensive so fast when you disagree with something they said online. Not sure I fully understand that.

There’s all this talk and speculation about Twitter, Threads, Mastodon, Minds and so on. Meanwhile twtxt sits there, forever free and open.

Interesting thoughts about multi thread vs single thread performance.

I found this to be a good thread on the subject of how the media is covering the dam explosion. The author, Timothy Snyder, is a history professor at Yale and has consistently good commentary on the war in Ukraine.

What is a good device for home virtualization these days? I have been looking at the Intel NUC 13 pro’s. Basically I want something “quiet” (ie not a screaming banshee 1U), smallish, but with lots of threads and rams. Disk will come from an external NAS.

I maintain keys for my email addresses.. but like most in this thread i almost never receive encrypted emails.. other than the BTC exchange i use that sends automated mail encrypted.

Wob3.11 is a scam. Pass it on. https://threadreaderapp.com/thread/1455625844504743938.html

@movq@www.uninformativ.de No argument that threading is an improvement. But I think (#hash) does that, and I think figuring out how to search should mostly be up to the client.

@prologic@twtxt.net we would want:

• a way to reply to the current thread. We have this.
  
• a way to reply to a specific twt. Need this. Maybe make all the replies start new conversations?
  
• check if twt is start of a conversation.. we kinda have this in the main feed with the conversation button. need to extend it for forked convs
  
• a way to inline first replies. maybe show one or two in the sub thread with a link to view.
  
• for convenience have a link to parent conv?

Posted to Entropy Arbitrage: Experimenting with Worker Threads https://john.colagioia.net/blog/2020/04/15/worker.html #techtips #programming #javascript #threads