In November 2023, S-RM published an urgent Cyber Threat Advisory on Citrix Bleed, a critical vulnerability first disclosed in October 2023 that allows remote attackers to successfully authenticate to susceptible devices and gain access to victim networks.
In this article, Virginia Romero discusses insights from S-RM's Incident Response team, whom in recent weeks have responded to numerous ransomware attacks that had Citrix vulnerabilities at the root. With new Citrix zero days being recently disclosed, we predict more of the same to come in the next few months. We dive into how organisations using Citrix can protect themselves, tackling alternatives to the often impossible advice of ‘patch more and faster’.
Missed last week’s article? Read What’s next in incident response? 4 key trends to watch out for in 2024 now.
Citrix in the crosshairs
In December 2023, S-RM was brought in to respond to a major ransomware incident affecting a European travel group. During the initial call, our team asked our client what for us is a routine, information gathering, question: ‘how do users remotely access your network?’. As responders, we typically want to get an understanding of all methods of remote access as soon as we are involved in an incident, including VPNs, remote support agents, and externally accessible remote desktop infrastructure, as these continue to be a favourite for threat actors to get initial access to corporate networks. Understanding how, in normal circumstances, remote users access the network helps us determine how a cybercriminal sitting in Russia may have accomplished the same. This context allows our team to offer immediate containment advice and recommendations, designed to kick out the threat actor from the network by locking down each avenue of remote access, with these only being securely restored once resets have been carried out and additional security measures have been implemented.
This particular client had a corporate VPN in place for some users, running the latest version of the software and secured by Multi-Factor Authentication ('MFA'). Pending our validation checks, this meant the VPN was likely not at the root cause of the incident. Crucially, however, they also had two separate Citrix environments that provided access to group applications. Since mid-October, when S-RM first responded to a case where Citrix Bleed had been exploited by the NoEscape Ransomware-as-a-Service ('RaaS') operation to gain unauthorised access to one of our client’s networks, the number of incidents involving this infrastructure had skyrocketed. Therefore, any mention of Citrix was an immediate red flag: for us, back in December, if Citrix was in use, chances were it had been exploited. This was also not limited to a couple of RaaS groups: Akira, PLAY, LockBit, BlackBasta… all major ransomware players were targeting Citrix in incidents our team were responding to.
What is Citrix?
Citrix provides software solutions that allow corporate networks and resources to be accessed remotely by leveraging virtualisation technology. Some of the key components in Citrix environments are:
- Virtual Desktop Infrastructure ('VDI'): this technology provides a virtual version of a computer or desktop environment stored in a remote server, accessible over the internet. Instead of having a physical computer, users can access the operating system, files, applications and data on that system from anywhere once successfully authenticated.
- Citrix Application Delivery Controller ('ADC', formerly known as ‘Netscaler’): this component serves as a ‘traffic director’ of sorts for a network, responsible for optimising data flows and ensuring availability of applications and resources for users. Citrix ADCs can also integrate a Gateway, a feature that is responsible for authenticating users to the environment and which can be thought of as a security checkpoint.
A playbook of ransom via Citrix
As was the case in the incident affecting our client in the travel industry, the Citrix-to-ransomware incidents we responded to in Q4 of 2023 all followed a very similar pattern, and one that we often see in cases with software vulnerabilities at the root. In over 90% of cases involving Citrix infrastructure, the threat actor had exploited the well-known Citrix Bleed vulnerability, with only a small proportion having been traced back to the exploitation of a separate Citrix ShareFile vulnerability, CVE 2023-24489.
Our incident data suggests that threat actors were actively exploiting these vulnerabilities as soon as they were disclosed. For example, S-RM's team first responded to a Citrix Bleed incident approximately two weeks after disclosure. In these cases, there was also a pattern of relatively long dwell times, where attackers would actively exploit the vulnerability when first announced (and before victims had patched their vulnerable Citrix appliances) by dropping remote access malware on a system for persistence, and only returning to carry out the post-exploitation phases of the attack weeks later, potentially as a means to evade detection or the increased scrutiny of the victim’s infosec team.
Regardless of which vulnerability had been exploited, whether Citrix Bleed or ShareFile, a typical attack chain involved the following steps:
- External reconnaissance: threat actors scan the internet looking for vulnerable Citrix infrastructure. Most threat actors are set up to do this on an automated, full-time basis, making the process of identifying targets relatively easy. Often, public vulnerabilities will have exploits (the code that allows abusing a flaw in the software) written by an individual or individuals that are widely shared among the criminal underground. Therefore, for RaaS groups, the process of finding a target and subsequently gaining an initial foothold on a victim’s network can be as simple as entering a few commands. This stage of the attack is typically automated while casting a wide, indiscriminate, net - often with little effort or even expertise on the part of cybercriminals.
- Initial access: threat actors exploit an existing vulnerability to circumvent security measures on a Citrix appliance, bypassing routine login controls. In most cases, they either authenticate as a valid user by stealing a token from an active session or gain the ability to execute remote code and upload malicious files on the system. Exploitation of a vulnerability typically leads to a threat actor getting access to virtualised remote computers within the Citrix VDI environment.
- Persistence: at this stage, we observed threat actors deploying persistence to a target VDI system, either in the form of the popular post-exploitation tool, CobaltStrike, or legitimate remote access solutions such as AnyDesk, Splashtop, or ScreenConnect. Where the VDI machine they landed was non-persistent (meaning data would be wiped upon shut down), threat actors would quickly laterally move to persistent servers within a victim network to retain their access. In many cases, after establishing persistence, what would then follow would be a period of inactivity, with the threat actor only returning to the network some time after the initial access.
- Privilege escalation and lateral movement: upon establishing a reliable means of repeatedly accessing the network, threat actors would then traverse the environment, looking to access high value servers such as domain controllers and file servers hosting sensitive corporate information. This was typically done over network shares, but also leveraging the in-built Windows administrative tool, Remote Desktop Protocol ('RDP'). To do so, threat actors would also have previously gained access to highly privileged accounts, often with domain administrator privileges. The privilege escalation phase of the attack would vary depending on the threat actor – some relied on dumping copies of hashed passwords for users across the domain, others on old favourites such as Mimikatz, or credential-stealing functionality built into bespoke malware.
From here, threat actors followed their usual ransomware playbook: further penetrating the network, stealing sensitive data, deleting backups, and eventually encrypting files hosted across a victim’s IT infrastructure.
Protection beyond patching
As with any critical vulnerabilities that are actively being exploited by threat actors in the wild, ensuring externally accessible Citrix appliances are up to date with the latest security patches remains a crucial part of preventing potential attacks. A robust patch management policy should guarantee that critical updates are installed as soon as possible and that, after applying the patches, these are validated by checking the version number. The short time between disclosure and active exploitation suggests that getting the timing right is critical.
However, there a lot of pitfalls and challenges to relying on patching alone. Our IR team helped multiple clients in the last few months where, despite a Citrix patch being applied, an unknown error prevented the application from successfully installing the update, leading to exploitation of the underlying vulnerability and a paralysing ransomware incident. Many of our clients also struggled to apply patches quickly enough as scheduling downtime of their Citrix platform was delayed for pragmatic business uptime reasons. Often a window of two weeks between patches being released and them being applied was enough for a ransomware group to get into the network. This scenario is made even more impossible when you consider the fact that many of these vulnerabilities in fact start as zero days, unknown vulnerabilities actively abused by threat actors for which no patch is immediately available.
What to do then? Beyond patching, our IR team found several key measures made the difference for our clients between those who experienced severe disruption, and those who were able to limit the impact of falling victim to the latest Citrix vulnerability. We have collected our top recommendations here:
Expect patching to fail and compensate
- The right network segmentation will not stop the initial intrusion but can greatly limit the potential spread from there. We recommend restricting connectivity from Citrix VDIs, gateways and appliances to servers containing your crown jewels. This can be complicated if – for example – you have high privileged users carrying out work via Citrix VDIs; however, ensuring that their access to critical systems is contingent on them fulfilling a secondary form of authentication not reliant on Citrix itself can slow attackers down.
- Protecting assets on your corporate network with a well-configured and monitored Endpoint Detection and Response ('EDR') solution will not stop threat actors successfully exploiting Citrix vulnerabilities. But again, it will stop a lot of attacks in their tracks just after the initial intrusion. Many Citrix appliances themselves will support EDR tools and we would recommend ensuring full coverage where possible. While EDR should not be considered a silver bullet, if it is correctly configured (do make sure to take into account considerations around non-persistent assets for Citrix VDIs), deployed everywhere possible, and appropriately monitored, your chances of spotting and containing an active intrusion via Citrix in the first few hours of an attack will increase dramatically.
- While this will not be easy for everyone, deploying an Intrusion Detection System/Intrusion Prevention System ('IDS'/'IPS') in addition to an EDR tool will give you very high chances of detecting an intrusion, even if a threat actor manages to bypass the Citrix Gateway (or external firewall).
Be ready for the initial intrusion and reduce your time to respond
- If your window to patch was longer than you hoped and you have heard the vulnerability in question is being actively exploited, consider conducting rapid forensic triage of your Citrix appliances and related systems as a routine process. Often, software vendors – Citrix included – will disclose critical vulnerabilities with patching notes, but limited to no guidance around what to do to be sure a threat actor did not exploit the vulnerability before it was patched. If you are unsure what to do, S-RM can help, but at a basic level we would recommend searching for common persistence, privilege escalation and lateral movement signs on both your Citrix appliances and network-adjacent systems as a good starting point.
- Carry out preliminary containment actions designed to remove a threat actor’s access, even when said access has not been confirmed. This should ideally include revoking all active Citrix sessions, resetting passwords for all user accounts, and temporarily isolating Citrix appliances from the rest of the network while a preliminary investigation is ongoing. While this will necessarily lead to some disruption, the risk you remove by doing so will be worth it, and any disruption will pale in comparison to the consequences of a successful ransomware attack.
What’s next for Citrix?
Citrix is, of course, not alone – other popular technologies are routinely targeted by threat actors, and much of our analysis and recommendations here apply elsewhere. Citrix appears to be in the crosshairs of a lot of the major ransomware groups, likely due to it becoming a victim of its own popularity in a world where remote working is more important and prevalent than ever. Nevertheless, considering that Citrix has already disclosed two critical zero-day vulnerabilities in 2024 (and we’re still in January), S-RM expects more campaigns by ransomware groups and other threat actors who use Citrix as their preferred means of accessing victim networks.
Now more than ever, we would encourage organisations seeking to protect themselves from this threat to focus on refining their patching as usual, but to also turn their attention to other compensating controls that can help limit the impact of Citrix-related intrusions when they happen.
