Almost a week to this and not a lot of drama on Indian Twitter, so I thought I could write about it.

TLDR: Malicious code with a backdoor was added to one of the core packages of Linux affecting almost all Debian and Redhat distros; was later discovered by a developer at Microsoft because of a 500ms delay in ssh response.

How did we get here?

The two latest releases (as of March 9th 2024) on the xz package apparently contained this malicious code. This code wasn't directly visible on the repo, but was disguised as a part of the tests which in turn modify the function in the liblzma library that uses the xz package. You would get this code upon downloading the specific release tarballs.

This was committed by user JiaT75 on GitHub (also known as Jia Tan; real identities, obviously unknown), who has been a regular maintainer on this project for about 2 years. I'm pretty sure there's way more into this story of how well-crafted this was and what the potential motive behind it could be; but that's not for this article.

How was this found?

Andres Freund, a developer at Microsoft, who is also a maintainer at Postgres thought something was suspicious when he observed a ~500ms delay in his ssh login process (refer to the image below). He was able to analyse further and remember facing similar issues with valgrind in the past with Postgres testing (absolute legend).

I've read a lot about this and I still do not know how liblzma and sshd are related. From what I've understood, sshd internally uses liblzma in some Debian distros (long story short).

I do not know how someone could get to such conclusions with very little detail, gotta respect what Andres has done here. If you want to read more about his analysis, it's available here on OpenWall. Worth the read even though I didn't understand more than 5% of it 🤣

What's the current update?

All major distros that had this xz version in their beta releases have been safely reverted, as confirmed by Debian, Fedora and Arch as of March 29th. From what I read, no stable versions of any distros were affected.

It's fascinating how little we know about the open source software that we use everyday. Things could've gone south very quickly if not for this discovery from Andres. The maintainers were also proactive on getting changes reverted to safety. Stories like these make us realise how little we know; and as tech enthusiasts, I believe it only makes us curious to keep learning.