It’s been nearly a month since Oakland was hit by a ransomware attack that’s left non-emergency systems like 311 partially inoperable. The city, with the help of undisclosed third-party cybersecurity firms, has recovered some of its systems including Wifi, phone lines, and the Oakland Public Library’s online services. Still, local businesses are unable to pay their taxes because the payment system is down.
The city has disclosed little to no information about the nature of the cyberattack that began the evening of Feb. 8 and impacted systems the following morning. To learn more about what a ransomware attack is and how cities respond to them, The Oaklandside decided to check in with Ryan Chapman, principal consultant for Palo Alto Networks, a leading cybersecurity firm.
Chapman’s interest in cybersecurity began when he was a teenager growing up in Salinas, California.
“I went through some phases in high school where I thought it was cool to pirate programs, steal software and sell it to people, you know, music CDs,” Chapman said. “I got hit with a [computer] virus myself and it destroyed about a year’s worth of my high school work and it kind of just blew my mind. It got me to realize the importance of protecting our computing lives.”
Since then he’s gone on to work for IT security companies that help cities recover from ransomware attacks.
This interview has been edited for length and clarity.
Ransomware attacks are pretty common. Have you ever been contracted to work with cities for ransomware-related issues?
Yes, I have. I can’t say which cities for obvious reasons, but many actually. Some of the cases that bother me the most are healthcare facilities, universities, and colleges—especially community colleges— that are just blasted by these groups. When those people are impacted, you really start to realize the severity of these kinds of attacks.
They’re actually very simplistic in nature even though they have devastating consequences from the technical side of things. Oftentimes cities and community colleges that get hit by an attack aren’t adhering to security 101 best practices.
We’ll often find that there’s a phishing vector involved, or there’s what’s known as a remote desktop protocol.
What is a phishing vector? What is a remote desktop protocol?
When cities or any type of entity gets hit with a ransomware attack, it’s usually through one of three intrusion vectors or entry vectors as we call them.
One vector is phishing, which is basically when a threat actor [hacker] sends out a mass email. They usually call that a “lure.” So it’s just like when you’re fishing with a fishing rod, but it’s spelled with a “ph.” The term originated back when lots of people were using the America Online system. It started with instant messages being sent to people basically purporting to be employees of AOL saying, “Hey, I need your password.”
Nowadays, a lure can be a fake invoice, a fake shipping document, or something saying that your account has been suspended somewhere, just something to make the user want to click and enter their password to fix the fake problem. Once the user gives their credentials, that provides the [hackers] remote access to their computer and systems.
Another vector is what’s known as a remote desktop protocol, or RDP.
RDP is designed to allow remote connectivity into a desktop environment. That’s why it’s called a remote desktop protocol. By design, it is meant to allow and facilitate remote access. That’s why it exists.
The problem comes into play when organizations don’t secure it properly. They’ll have a password that’s easily guessable. They’ll have a password that’s commonly used, something of that nature.
So threat actors [hackers] will scan the internet and if they’re targeting cities, colleges, or universities they just throw a bunch of passwords at it until they get one.
Once they get in, they’re literally moving a mouse and controlling the computer remotely. From that point, they do their dirty deeds. Those are the top two ways [hackers] gain access to city systems.
What is the third way?
Number three—which is becoming increasingly popular—is the exploitation of software vulnerabilities. That basically translates to not patching [or updating] your software. You’ll have some type of internet-facing service that will be running a version of whatever service or software that’s not been patched.
The media portrays a lot of these attacks as though it’s some kind of crazy, new, amazing threat attack that’s never been seen before. Those are what’s referred to as a zero-day. A zero-day is an exploit for which there’s no patch. There’s no way to fix it because the software or hardware appliance vendor has no idea the attack is coming. It just shows up one day and you get hit with it.
Zero-day attacks are not as common as people would think.
The reality is that unpatched software is when there is for sure a patch available, and yet it just isn’t applied in time.
I take these kinds of cases all the time. I’ll work a case where a threat actor gets into a particular service and the client ends up asking, “How’d they get in?” Of course, they always want to know that. And we say, “You had this particular software that hadn’t been patched for, you know, 11 months.” So software patching is very important because that’s one of the top three entry vectors.
What makes cities like Oakland or other organizations vulnerable to these sorts of attacks?
This is my personal opinion, and not my employer’s, but I personally believe that cities, universities, and especially community colleges are susceptible because they often don’t have much of a budget for IT security. When they don’t have enough money for security, specifically security engineers to engineer the network to be defensible against these types of attacks, they’re basically just sitting ducks.
We find this a lot of times with cities because they often will have IT systems that are outdated. The city itself will offer these various services and the IT computing systems that you’re logging into will be running Windows 2008. It’s 2023! We just find that they’re not updated as frequently.
When you look at large corporations, they have their eyes on the prize in terms of being able to allocate money specifically to IT security. We see that a lot of cities just aren’t able to do that.
I imagine it’s hard to budget for strengthening its IT department until this kind of thing happens, right?
Yes. You run into the type of situation where you try to tell the voter base, “Hey, we’re going to spend an extra two point something million dollars on this security program for the next three years,” and the voters’ responses are like, “What? No, you’re not.”
Now, keep in mind, I don’t know exactly what funding has been allocated for IT security for the city of Oakland, but what I can assume is that it’s been an uphill battle because it is with many cities.
How effective are ransomware attacks? In your opinion, is it better for a city like Oakland to pay the ransom or hire a third-party firm to try and resolve it?
This is the great debate: to pay or not to pay?
Personally—and this is my opinion—I can’t stand the idea of paying any ransom because if we were to simply stop paying, ransomware attacks would just go away. It’s as simple as that. But obviously, it’s not truly as simple as that because we don’t live in a society where we can always make the decision we’re not going to pay.
A lot of groups think if they pay, they’ll be up and running right away. But in reality, a city can pay and still be down for multiple weeks or months. Oftentimes people don’t realize the amount of rebuilding, redesigning, and redeploying that you have to do in a network environment that has been hit by ransomware.
My rule of thumb is if a ransomware threat actor has compromised a machine, I don’t ever want that machine put back on the internet. I want that machine rebuilt. So what I mean by rebuilt is either restored prior to the attack date or have Windows reinstalled on the system.
What almost any consulting firm should say is that we don’t want to ever see an infected PC that wasn’t fully restored put back into the environment.
How often do cities end up caving in and paying?
According to Coveware, a company that helps people recover from these kinds of attacks, fewer ransomware victims are paying. Many of us in the industry believe the reason that payments are down is specifically linked to the conflict in Ukraine.
A very high percentage of ransomware actors are Russian speaking and reside in areas where they align with Russia. One such group called Conti was responsible for a lot of attacks. If you got hit by Conti, you were having a bad time. They were large and they had a massive group of people who worked for them.
Their downfall was that they came out and they said, “We are Russians. We are in support of Russia. Anyone who opposes Russia, we will come after you.” Well, apparently they kind of forgot that they had a number of affiliates who were Ukrainian, and one of those Ukrainian gentlemen said, “uh, no,” and got really aggravated. That person ended up leaking out a ton of data about the group and we’re talking private chat communications for, you know, a year or two or something like that. That happened in early March 2022 and they disbanded shortly after.
So a lot of ransomware incidents are related to organized crime?
Yes. Many people believe that ransomware is a virus, but it’s actually an attack campaign that’s carried out usually by humans with their hands on the keyboard. This is why we often refer to ransomware attacks as human-operated ransomware. The acronym is HUMOR. They’ll go from machine to machine, they’ll compromise accounts, they’ll access data, they’ll exfiltrate the data, which is the term that we use to say that they steal the data, and use that for extortion. When they’re done, that’s when they deploy the encryption piece of the attack.
Ransomware attacks can take days, hours, or weeks. But you’ll have a threat actor in your environment just moving around, stealing stuff, clicking on things, and doing all the bad things they want to do. All of this is organized through the developers of the various ransomware variants and what they do is hire affiliates. The affiliate is basically someone who comes in and says, “I want to make money from ransomware.” The developers say, “OK, you are an affiliate of ours now, so we are going to hire you basically, or sometimes you’ll pay us to use our stuff.”
And if they’re successful, the affiliate will split the money with the ransomware developer. So you don’t have to know how to write ransomware software, and you don’t have to know how to write computer code. You just have to basically follow an instruction playbook.
