Of course we still have to fix the hashing algorithm and length.

I finally resolved my issues with hashing twts… with REGEX!

Dates in JavaScript are truly strange creatures.

SHA2 Fatal Flaw? (Hash Length Extension Attack) - Computerphile ⌘ Read more

@lyse@lyse.isobeef.org I don’t think there’s any point in continuing the discussion of Location vs. Content based addressing.

I want us to preserve Content based addressing.

Let’s improve the user experience and fix the hash commission problems.

@prologic@twtxt.net I know we won’t ever convince each other of the other’s favorite addressing scheme. :-D But I wanna address (haha) your concerns:

1. I don’t see any difference between the two schemes regarding link rot and migration. If the URL changes, both approaches are equally terrible as the feed URL is part of the hashed value and reference of some sort in the location-based scheme. It doesn’t matter.

2. The same is true for duplication and forks. Even today, the “cannonical URL” has to be chosen to build the hash. That’s exactly the same with location-based addressing. Why would a mirror only duplicate stuff with location- but not content-based addressing? I really fail to see that. Also, who is using mirrors or relays anyway? I don’t know of any such software to be honest.

3. If there is a spam feed, I just unfollow it. Done. Not a concern for me at all. Not the slightest bit. And the byte verification is THE source of all broken threads when the conversation start is edited. Yes, this can be viewed as a feature, but how many times was it actually a feature and not more behaving as an anti-feature in terms of user experience?

4. I don’t get your argument. If the feed in question is offline, one can simply look in local caches and see if there is a message at that particular time, just like looking up a hash. Where’s the difference? Except that the lookup key is longer or compound or whatever depending on the cache format.

5. Even a new hashing algorithm requires work on clients etc. It’s not that you get some backwards-compatibility for free. It just cannot be backwards-compatible in my opinion, no matter which approach we take. That’s why I believe some magic time for the switch causes the least amount of trouble. You leave the old world untouched and working.

If these are general concerns, I’m completely with you. But I don’t think that they only apply to location-based addressing. That’s how I interpreted your message. I could be wrong. Happy to read your explanations. :-)

Here is just a small list of things™ that I’m aware will break, some quite badly, others in minor ways:

1. Link rot & migrations: domain changes, path reshuffles, CDN/mirror use, or moving from txt → jsonfeed will orphan replies unless every reader implements perfect 301/410 history, which they won’t.
   
2. Duplication & forks: mirrors/relays produce multiple valid locations for the same post; readers see several “parents” and split the thread.
   
3. Verification & spam-resistance: content addressing lets you dedupe and verify you’re pointing at exactly the post you meant (hash matches bytes). Location anchors can be replayed or spoofed more easily unless you add signing and canonicalization.
   
4. Offline/cached reading: without the original URL being reachable, readers can’t resolve anchors; with hashes they can match against local caches/archives.
   
5. Ecosystem churn: all existing clients, archives, and tools that assume content-derived IDs need migrations, mapping layers, and fallback logic. Expect long-lived threads to fracture across implementations.

We’ve been discussing the idea of changing the threading model from Content-based Addressing to Location-based addressing for years now. The problem is quite complex, but I feel I have to keep reminding y’all of the potential perils of changing this and the pros/cons of each model:

With content-addressed threading, a reply points at something that’s intrinsically identified (hash of author/feed URI + timestamp + content). That ID never changes as long as the content doesn’t. Switching to location-based anchors makes the reply target extrinsic—it now depends on where the post currently lives. In a pull-based, decentralised network, locations drift. The moment they do, thread identity fragments.

@zvava@twtxt.net There would be only one hash for a message. Some to be defined magic date selects which hash to use. If the message creation timestamp is before this epoch, hash it with v1, otherwise hammer it through v2. Eventually, support for v1 could be dropped as nobody interacts with the old stuff anymore. But I’d keep it around in my client, because why not.

If users choose a client which supports the extensions, they don’t have to mess around with v1 and v2 hashing, just like today.

As for the school of thought, personally, I’d prefer something else, too. I’m in camp location-based addressing, or whatever it is called. There more I think about it, a complete redesign of twtxt and its extensions would be necessary in my opinion. Retrofitting has its limits. Of course, this is much more work, though.

@zvava@twtxt.net I was about to suggest that you post some examples. By now, we’re pretty good at debugging hashing issues, because that happens so often. 😂 But it looks like you figured it out on your own. ✌️

@zvava@twtxt.net we have to amend the spec and increase the hash length. We just haven’t done so yet 😆

@zvava@twtxt.net may I recommend to change the mention format upon hitting reply to something similar to what it’s used in Yarn, and perhaps hiding the hash on the post too? Looking good!

@movq@www.uninformativ.de Yeah, we’ve seen how this plays out in practice 🤣 @dce@hashnix.club My advice, do what @movq@www.uninformativ.de has hinted at and don’t change the 1st # url = field in your feed. I’m not sure if you had already, but the first url field is kind of important in your feed as it is used as the “Hashing URI” for threading.

@dce@hashnix.club Ah, oh, well then. 🥴

My client supports that, if you set multiple url = fields in your feed’s metadata (the top-most one must be the “main” URL, that one is used for hashing).

But yeah, multi-protocol feeds can be problematic and some have considered it a mistake to support them. 🤔

Functional hash tables explained
Comments ⌘ Read more

rapidhash: Very fast, high quality, platform-independent hashing algorithm
Comments ⌘ Read more

huh.. so not even trying to be compatible with existing hashes?

I have zero mental energy for programming at the moment. 🫤

I’ll try to implement the new hashing stuff in jenny before the “deadline”. But I don’t think you’ll see any texudus development from me in the near future. ☹️

Hash Collisions & The Birthday Paradox - Computerphile ⌘ Read more

@ About the URL, since it no longer used for hashing there might be no need to change it. I agree that we keep all the parts that already are out there for the most parts. Instead of a contact field you could also just use links like: link = Email mailto:user@example.dk or link = Signal https://signal.me/sthF4raI5Lg_ybpJwB1sOptDla4oU7p[...]

@lyse@lyse.isobeef.org Yeah to avoid cutting off bits at the end making hashes end in either q or a 🤣

Adaptive Hashing
Comments ⌘ Read more

The reason I think this can work so well and I’m in full support of it is that it’s the least disruptive way to resolve the issue of:

where did this hash come from?

@prologic@twtxt.net Not sure I’d attach any if clauses to this. My point is: Every time I see a hash, I’d like to have a hint as to where to find the corresponding twt.

@movq@www.uninformativ.de If we’re focusing on solving the “missing roots” problems. I would start to think about “client recommendations”. The first recommendation would be:

1. Replying to a Twt that has no initial Subject must itself have a Subject of the form (hash; url).
   

This way it’s a hint to fetching clients that follow B, but not A (in the case of no mentions) that the Subject/Root might (very likely) is in the feed url.

If we must stick to hashes for threading, can we maybe make it mandatory to always include a reference to the original twt URL when writing replies?

Instead of

(<a href="https://we.loveprivacy.club/search?q=%23123467">#123467</a>) hello foo bar

you would have

(<a href="https://we.loveprivacy.club/search?q=%23123467">#123467</a> http://foo.com/tw.txt) hello foo bar

or maybe even:

(<a href="https://we.loveprivacy.club/search?q=%23123467">#123467</a> 2025-04-30T12:30:31Z http://foo.com/tw.txt) hello foo bar

This would greatly help in reconstructing broken threads, since hashes are obviously unfortunately one-way tickets. The URL/timestamp would not be used for threading, just for discovery of feeds that you don’t already follow.

I don’t insist on including the timestamp, but having some idea which feed we’re talking about would help a lot.

I will be adding the code in for yarnd very soon™ for this change, with a if the date is >= 2025-07-01 then compute_new_hashes else compute_old_hashes

Finally I propose that we increase the Twt Hash length from 7 to 12 and use the first 12 characters of the base32 encoded blake2b hash. This will solve two problems, the fact that all hashes today either end in q or a (oops) 😅 And increasing the Twt Hash size will ensure that we never run into the chance of collision for ions to come. Chances of a 50% collision with 64 bits / 12 characters is roughly ~12.44B Twts. That ought to be enough! – I also propose that we modify all our clients and make this change from the 1st July 2025, which will be Yarn.social’s 5th birthday and 5 years since I started this whole project and endeavour! 😱 #Twtxt #Update

I had Chick-fil-A breakfast today (sausage, egg, and cheese biscuit, hash browns, coffee, and orange juice). Then at lunch my work place offered hot dogs. I had two (kosher, if that matters), plus a coke, a macadamia nuts cookie, and a small chocolate brownie.

So, here I am, at home, feeling hungry but guilty and refusing to eat anything else for the rest of the day. To top it off, I have only clocked 4,000 steps today (and I don’t feel like walking). I am going to hell, am I?

yes it used be http:// only and to keep hashes from breaking i added # url = http://... and now we are stock with it due to the curret specs.

Hmmm there’s a bug somewhere in the way I’m ingesting archived feeds 🤔

sqlite> select * from twts where content like 'The web is such garbage these days%';
      hash = 37sjhla
  feed_url = https://twtxt.net/user/prologic/twtxt.txt/1
   content = The web is such garbage these days 😔 Or is it the garbage search engines? 🤔
   created = 2024-11-14T01:53:46Z
created_dt = 2024-11-14 01:53:46
   subject = #37sjhla
  mentions = []
      tags = []
     links = []
sqlite>

**Hmmm there’s a bug somewhere in the way I’m ingesting archived feeds 🤔

sqlite> select * from twts where content like 'The web is such ga ...**
Hmmm there’s a bug somewhere in the way I’m ingesting archived feeds 🤔

sqlite> select * from twts where content like ‘The web is such garbage these days%’;

  hash = 37sjhla

feed_url = https://twtxt.net/user/prologic/twtxt.txt/1
content = The web is such garbage these days 😔 Or is it the garbage search engines? 🤔
created = 2024-11-14T01:53:46Z
created_dt = 2024-11-14 01:53:46
… ⌘ Read more

Some A hole has been trying to pull every single Twtxt feed that existed/still exists since forever. How do I know? Welp’ They’ve been querying my Timeline™ instance for all of it, every single twtxt file and twt Hash they can find. 😆🤦 It must have been going on for days and I have just noticed… + it’s all coming from the same ASN AS136907 HWCLOUDS-AS-AP HUAWEI CLOUDS

Thank you Huawei for the DDos you sons of Glitches!!!

@quark@ferengi.one No editing old Twts that are the root of a thread with replies in the ecosystem. Just results in a fork. Unless the client has an implementation that does not store Twts keyed by Hash.

(#mkhkhuq) @quark@quark No editing old Twts that are the root of a thread with replies in the ecosystem. Just results in a fork. Unless th …
@quark @ferengi.one No editing old Twts that are the root of a thread with replies in the ecosystem. Just results in a fork. Unless the client has an implementation that does not store Twts keyed by Hash. ⌘ Read more

@david@collantes.us @andros@twtxt.andros.dev The correct hash would be si4er3q. See https://twtxt.dev/exts/twt-hash.html, a timezone offset of +00:00 or -00:00 must be replaced by Z.

(That said, there’s a bug in jenny as well. It only replaces +00:00, not -00:00. 🤡)

@prologic@twtxt.net interesting. What would happen on a hash collision? 🤔

@bender@twtxt.net It’s a bug in the UI for sure. The hash is the primary key.

(#yyriova) @bender@bender It’s a bug in the UI for sure. The hash is the primary key.
@bender @twtxt.net It’s a bug in the UI for sure. The hash is the primary key. ⌘ Read more

@david@collantes.us Yeah, we’ve been debugging that a bit yesterday. Looks like the wrong input (sometimes) gets fed to the hash function → broken threads.

The actual hash is fs7673q.

@prologic@twtxt.net that’s not what I see. The hash znf6csa cannot be found.

@prologic@twtxt.net There was no edit according to my Git history. 🤔 On my end, the hash is fs7673q and that’s also what kat used to reply.

Doesn’t look like it Hmmm

sqlite> select * from twts where content LIKE '%Linux installation%';
    hash = znf6csa
feed_url = https://www.uninformativ.de/twtxt.txt
 content = I wonder if my current Linux installation will actually make it to 20 years:

    $ head -n 1 /var/log/pacman.log
    [2011-07-07 11:19] installed filesystem (2011.04-1)

It’s not toooo far into the future.

It would be crazy … 20 years without reinstalling once … phew. 🥴
 created = 2025-04-07T19:59:51Z
 subject = (#znf6csa)
mentions = []
    tags = []
   links = []

**(#2znenta) Doesn’t look like it Hmmm

sqlite> select * from twts where content LIKE '%Linux installation%';
    hash = znf6csa
feed_url = ht ...**
Doesn’t look like it Hmmm

sqlite> select * from twts where content LIKE ‘%Linux installation%’;

hash = znf6csa

feed_url = https://www.uninformativ.de/twtxt.txt
content = I wonder if my current Linux installation will actually make it to 20 years:

$ head -n 1 /var/log/pacman.log
[2011-07-07 11:19] installed filesystem (2011.04-1)

It’s not toooo far into the future.

It wou … ⌘ Read more

@movq@www.uninformativ.de Apparently you wrote it :D The hash doesn’t lie? 🤣 https://twtxt.net/twt/znf6csa

(#2znenta) @movq@movq Apparently you wrote it :D The hash doesn’t lie? 🤣
@movq @www.uninformativ.de Apparently you wrote it :D The hash doesn’t lie? 🤣 https://twtxt.net/twt/znf6csa ⌘ Read more

@prologic@twtxt.net What happened here – did I edit my twt or is this hash wrong? 🥴

@andros@twtxt.andros.dev sha256 hash of twt in json. Look at converter script

Hello, i want to present my new revolution twtxt v3 format - twjson
That’s why you should use it:

1. It’s easy to to parse
   
2. It’s easy to read (in formatted mode :D)
   
3. It used actually \n for newlines, you don’t need unprintable symbols
   
4. Forget about hash collisions because using full hash
   Here is my twjson feed: https://doesnm.p.psf.lt/twjson.json
   And twtxt2json converter: https://doesnm.p.psf.lt/twjson.js

@bender@twtxt.net Yeah, as you mentioned in the other thread, @andros@twtxt.andros.dev’s hashes appear to be not quite right. 🤔