PHP2Shell or: How We Learned to Stop Worrying and Loved PHP

What is it all about?

PHP is a programming language and interpreter that has been used for many years to serve dynamic websites, providing a “core” for the greater part of modern-day internet services. Recently detected vulnerabilities in that core software allow a remote attacker to use the running PHP server to bypass the actual web application and instead issue any commands right into the service user’s terminal—essentially taking control of the entire web app and potentially even more.

Imagine going to the city office—usually you go there, fill some forms, and then you are granted your permit, or passport, or whatever you came to get there. So let’s say you fill a couple of forms in a very specific way and suddenly you become the manager of that office. It then allows you to get or change any data you want, change the office’s regulations, or even impersonate the office to a third party. That’s the scenario we wouldn’t like to happen in the real world, and it can be as bad in the virtual one, as well. Now imagine you can hack into almost every city office the same way—and basically that’s what happened with PHP.

When did you first become aware of the vulnerabilities?

We started gathering the first news about the issue from various sources in the very morning of October 21, with the first internal discussion starting as early as 4:58z. Not much was known about it at the time, so we started assessing different possibilities and temporary remediation strategies. We were able to find some temporary solutions (or work-arounds) for the problem and implemented them around 8:00z into our production servers.

Imagine going to the city office (…) you fill a couple of forms in a very specific way and suddenly you become the manager of that office. It then allows you to get or change any data you want, change the office’s regulations, or even impersonate the office to a third party.

When was the actual fix available and when was it applied?

The fix became available at around 13:30z and we immediately started to prepare the so-called “base images” for all of our services. As we expected, the entire world started to download these updates right after we did, so the process stalled for a bit. Luckily, we were able to switch to an alternative repository with more capacity and in the end it all went smoothly. Then we started to replace the images on our virtual infrastructure, starting from Leon itself, and then proceeding to update all our internal tools as well. All our services were patched by 15:00z the same day.

What made such a quick mitigation possible?

Our infrastructure, definitely. It wouldn’t be possible without the containerization of all services using Docker. Combined with the infrastructure-as-a-service approach, these are the two directions of our infrastructure development we have been aligned with since 2016 and once again, it has paid off greatly. Thanks to this approach, we were able to replace all Leon app servers without any downtime. Our internal tools also allowed us to monitor the service before, during and after update, ensuring that nothing fishy is going on.

What would happen if we didn’t apply the patch?

As it turned out, not much. Not because the vulnerability isn’t severe (it is!) but due to other protection tools we are using, especially WAF (both AWS WAF and OWASP CRF) which is a dedicated tool capable of detecting and immediately blocking dangerous payloads that can be used for very sophisticated attack patterns, like this one. On top of that, Upwind monitors our infrastructure and detects any suspicious anomalies. The important thing about tools of this kind is that they’re pro-active—they can stop a wide range of attacks by just noticing that something is “off”. This makes it especially useful against the zero-day attacks. So even if we didn’t apply the patch in time, the real risk of affecting our software or our clients’ data would be very limited.

When did clients start asking about the issue?

Actually, very early—first enquiries came in the morning the same day, although most customers didn’t become aware of the issue until the next day. Funnily enough, the fact that we were able to protect the servers without any downtime or maintenance made it much more difficult for us to convince everyone that the situation had been already fixed! But that’s probably the standard of service you would expect from a ISO 9001 and ISO 27001 certified company.

(…) due to other protection tools we are using (…) even if we didn’t apply the patch in time, the real risk of affecting our software or our clients’ data would be very limited.