On March 24, 2026, a PyPI software supply chain attack hit LiteLLM — one of the most widely used AI libraries on the planet — and the fallout is still unfolding. This wasn’t a fake package. It wasn’t a typo-squatted name designed to fool developers who can’t spell. Attackers used real, stolen credentials to publish malicious code as the genuine article, right there in the official PyPI repository. LiteLLM gets pulled into developer environments 97 million times a month. The math on potential impact is not comforting.
This blog post summarizes the main points of my latest podcast episode. If you’d like, you can listen to it or watch it at https://www.backupwrapup.com/pypi-software-supply-chain-attack.
How the PyPI Software Supply Chain Attack Actually Worked
In our book Learning Ransomware Response and Recovery, Dr. Mike Saylor uses a phrase I keep coming back to: getting poisoned by your own medicine cabinet. That’s exactly what happened here. You go to a trusted source — the same place you’ve gone a hundred times — and this time it kills you.
The attack didn’t start with LiteLLM. It started five days earlier with Trivy, an open source security scanner that LiteLLM used in its own CI/CD pipeline. Attackers compromised Trivy first, then used that foothold to steal the PyPI publish credentials from LiteLLM’s build environment. With those credentials in hand, they published two malicious versions — 1.82.7 and 1.82.8 — directly to PyPI.
Version 1.82.7 ran its payload when you imported the proxy module. Version 1.82.8 was nastier. It used a Python mechanism called a .pth file that fires every single time the Python interpreter starts — including when you run pip install. You didn’t have to run LiteLLM. You didn’t have to touch the proxy. Just having it installed was enough.
Here’s the part that makes standard security tools useless against this class of attack: the package passed every integrity check. No hash mismatch. No suspicious filename. The malicious content was published using legitimate credentials, so from PyPI’s perspective, everything looked fine.
What the Malware Stole — and What It Left Behind
The payload went after everything it could reach: SSH keys, AWS and Azure credentials, Kubernetes service account tokens, database passwords, .env files, shell history, CI/CD secrets, Docker configs, and cryptocurrency wallet files. It encrypted the haul and sent it to an attacker-controlled server. Then it installed a persistent backdoor — a systemd service called sysmon.service, disguised as “System Telemetry Service” — and started polling for follow-on instructions every five minutes.
In Kubernetes environments, it tried to escalate to the whole cluster. If it found a service account token, it read secrets across every namespace and attempted to deploy a privileged pod to every node in kube-system.
Here’s what matters most from a recovery standpoint: just deleting the package is not remediation. The backdoor is already there. Open sessions are not killed by rotating credentials — you have to revoke them first, then rotate. Treating this as a simple package removal is like trying to stomp out a grass fire with moccasins. The fire will keep coming back.
PyPI Software Supply Chain Attack Action Items for Right Now
If you installed LiteLLM on March 24, 2026 — or if you pull in litellm without a pinned version — treat the affected system as compromised and work through this list.
Check your version first. Run pip show litellm and look for 1.82.7 or 1.82.8. If you see either of those, stop what you’re doing.
Look for persistence artifacts. Check for ~/.config/sysmon/sysmon.py and ~/.config/systemd/user/sysmon.service. Check your Python site-packages directory for a file called litellm_init.pth.
Revoke, then rotate. Every credential that was accessible on that system — cloud keys, SSH keys, API keys, Kubernetes tokens, database passwords — needs to be revoked and reissued. Not just changed. Revoked. Active sessions don’t care that you changed a password.
Check Kubernetes. Look for pods named node-setup-* in kube-system. Audit cluster secrets for unauthorized access.
Restore from a trusted backup. This is where immutable backups matter. If your backups can be deleted — by you, by an admin, by anyone — they are not truly immutable. Configure your backup storage so that not even you can delete backups before they expire. If that’s not where you are today, get there. The credential-harvesting that defines this PyPI software supply chain attack is exactly the kind of access that leads to ransomware deployment down the road.
What This PyPI Software Supply Chain Attack Teaches Us About the Future
Two things would have broken this attack chain before it started. First: version pinning. Aider — another popular AI tool that uses LiteLLM — was confirmed unaffected because they pin to a specific version in their requirements.txt. That one habit kept them clean.
Second: your CI/CD tools are part of your attack surface. LiteLLM got hit because Trivy wasn’t pinned. The security scanner became the weapon. Every unpinned tool in your build pipeline carries the same risk.
The group behind this — TeamPCP — is on Phase 9 of an ongoing campaign. They’ve already hit Trivy, dozens of npm packages, Checkmarx, and VS Code extensions. They are not done.
Keep a software inventory. Know what libraries you’re running, what versions, and what they depend on. Subscribe to CISA alerts at cisa.gov. Turn on logging in your cloud environments and keep those logs long enough to be useful when you need them. And get serious about immutable backups — because when credential harvesting leads to ransomware, a backup that nobody could touch is the only clean path back.
Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at S2|DATA, which helps companies manage their legacy data
