What was the exploit
In recent weeks he had noticed that ssh—a system to log on securely to another device over the internet—was running about 500 milliseconds more slowly than expected. Closer inspection revealed malicious code embedded deep inside xz Utils, a piece of software designed for compressing data used inside the Linux operating system, which runs on virtually all publicly accessible internet servers
Effectively, someone used the package XZ Utils to piggy back into systemd loading liblzma, which in turn loaded XZ, allowing sshd to be hooked to trojan it on Linux distributions that use systemd.
The trojan allows somebody a private key to hijack sshd to execute commands, amongst other functions
These changes were committed to Github back in February, and made their way into test releases of Debian, Fedora and Kali Linux. Nobody noticed the problem. Additionally, a request was opened to make the threat actor a Linux kernel module maintainer for XZ Embedded.
Who is Andres
From a recent podcast, Andres describes what he does
https://oxide.computer/podcasts/oxide-and-friends/1843393 I've been working on Postgres for, I don't know, 15 years or so about. And as part over time, more and more contributions got more and more technically involved or deeply technically involved And I've a lot of work around performance, replication, scalability over the years, and a lot of, I don't know, just finding. And at at some point, I became a committer in the project and I've been at for about 10 years, I would guess. And yeah, so I've done a lot of reasonably sized pieces of work in Postgres and probably one of the people in Postgres that works most on performance and particularly around micro benchmarking or benchmarking and profiling and seeing what's causing scalability issues and stuff like that. So I guess that I had a good, starting point for finding any issues like this.
How was the exploit discovered
Andres:“I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
One key fact that stood out was that a lot of coincidences had to happen in his discovery of the exploit. Andres mentions a few of them below
One more aspect that I think emphasizes the number of coincidences that had to come together to find this:I run a number “buildfarm” instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.Without having seen the odd complaints in valgrind, I don’t think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().”
Eventually, Andres published a report to OpenWall to expose the exploit, mitigating a global threat.
Details by time
The details highlight the amount of effort and careful planning for the exploit revolving around a malicious actor JiaT75(Jia Tan)
Date | Event |
2021 | Jia Tan creates a GitHub account |
2021 | Jia Tan merges an MR in libarchive
This PR was merged without discussion, but after the XZ utils exploit, people found out that this PR also introduces a vulnerability |
Apr 2022 | Jia Tan submits a patch (MR) for XZ via email
At the same time, a random actor Jigar Kumar starts to pressure Lasse Collin to add another maintainer for XZ
https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html
Around this time, Lasse Collin was also taking a break due to his mental health
Jia Tan starts to make commits for XZ
Shortly after, another random actor Dennis Ens also starts to participate in the pressure tactics
https://www.mail-archive.com/search?l=xz-devel@tukaani.org&q=from:"Dennis+Ens"
Both accounts are never active again after this period |
Jan 2023 | Jia Tan merges his first commit, which is when he starts to gain trust |
Mar 2023 | The primary contact email in Google’s oss-fuzz is updated to be Jia’s, instead of Lasse Collin |
Jun 2023 | Another account Hans Jansen created a MR the infrastructure for the exploit (test files)
Jia Tan reviewed and merged the MR
Hans Jansen’s account was seemingly made specifically to create this pull request. There is very little activity before and after |
July 2023 | A PR was opened in oss-fuzz to disable ifunc for fuzzing builds, due to issues introduced by the changes above.
This appears to be deliberate to mask the malicious changes that will be introduced soon
|
Feb 2024 | A commit containing the final steps required to execute this backdoor is added to the repository:
Tests: Add a few test files
Tests: Update two test files |
March 2024 | The exploit was discovered and an email was sent to oss-security
https://www.openwall.com/lists/oss-security/2024/03/29/4
At the same time Hans Jansen and multiple other account start to push for inclusion of the vulnerable version of XZ into Debian
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
Jia Tan also attempted to get it into Ubuntu before its release freeze
https://boehs.org/node/everything-i-know-about-the-xz-backdoor?source=post_page-----bbfe628fafdd-------------------------------- |
ㅤ | ㅤ |
Conclusion
The whole world got lucky because one developer chanced upon the exploit in his spare time
Some other notes
- The efforts to obfuscate the backdoor led to increased latency which led to its discovery
- Observe the benefits of version control and logs
- Social engineering attacks is perhaps more dangerous than cyber attacks