When working with a healthtech startup a few years ago, a strategic player that would open up a huge market required a security assessment from an external Infosec company in order to partner up with us. The application dealt with sensitive user information, and they couldn’t take any security risks.
The CEO found an Infosec company, and they scheduled the assessment. They would be running a battery of automated and manual tests, testing every component of the application.
They didn’t say when they were going to do this test, so we had no way to prepare – which shouldn’t be an issue. We should be ready.
The test
One day, I noticed the platform became very slow. Almost like someone was conducting a DDOS attack (sending a lot of traffic in order to overload our system). At some points the application was completely unresponsive, taking up to 40 seconds to load a page.
Under normal circumstances I would think this is weird, but since we knew someone was likely testing the platform, we let it be. The traffic we got under normal operation didn’t justify getting a bigger server or having any complex scaling in place – yet.
A week passes, and we get the test results back.
It’s an understatement to say that they looked terrible. Dozens of vulnerabilities, lots of red fonts, and the word “critical” thrown around very often.
The CEO was very afraid. If the test results were this bad, is the potential make-or-break-it partnership going to be lost? They will not approve an application in this sorry state.
Panic mode.
The CEO understandably goes into panic mode. Made dozens of calls to engineer friends who were senior engineers working for the likes of Deloitte and Capgemini. To my surprise, they all jumped into the code, and started looking at where the problems might be.
However, I was not so certain that the test had been conducted properly. It didn’t take longer than a glance at the audit to become very skeptical of the results. I communicated that I didn’t think the audit was valid.
Here’s why I didn’t think it was valid:
When looking at the audit, I noticed a few lines mentioning vulnerabilities that were not possible on our server.
One of the lines claimed we had a “Ruby code injection”. Ruby is a programming language – one of the many they tested the application against.
But we didn’t have ruby installed on that server. Such an injection with that setup was impossible. Which forced me to doubt the results of the audit.
Imagine a security assessment company does a security audit of your house and says your windows were unlocked – when you live in a basement with no windows – they were either looking at the wrong house or made some other mistake!
That same mistake appeared a few times for different programming languages that were not possible to exist.
The bad results
The audit documents were thankfully very comprehensive, and included the exact code and methods used to test for the vulnerabilities they say they found.
To keep this short, the audit ran a bunch of tests very rapidly. Remember the application being slow that one time? It was because the application was being tested thousands of times per second, making the small (at the time) server overwhelmed, and slowing the whole application down.
Now, one of the ways they checked for ruby code execution was by injecting code that asked the server to “sleep” for 10 seconds, and to then reply with some arbitrary response. The code would then measure if the server “slept” for at least those 10 seconds.
They repeated this exact test in multiple languages, hundreds of times.
But the problem was, the server was overwhelmed by all those requests, and the responses were up to 40 seconds long, and essentially was “sleeping” the entire time! The test was not valid.
Every single “code injection” they tested was done with this method, and came up as a vulnerability, when in fact it was not!
Jumping into code immediately was a mistake I didn’t expect senior engineers of that caliber to make, given that the audit hadn’t been properly audited. A common issue with engineers is that often they have tunnel vision and end up missing critical big-picture information.
All in all, a few people wasted a week chasing something that never existed – and unnecessary stress built up for no reason whatsoever.
After days of pushing for a line of communication with the Infosec company, we ended up getting new, squeaky-clean audit results. We got a couple of non critical “warnings” that we were able to fix within a couple of hours.
The strategic partnership ended up happening, and a lot of extra business came (and still comes) from it!