Slashdot Mirror
Ask Slashdot: Share Your Security Review Tales
New submitter TreZ writes: If you write software, you are most likely subject to a "security review" at some point. A large portion of this is common sense like don't put plain text credentials into github, don't write your own encryption algorithms, etc. Once you get past that there is a "subjective" nature to these reviews.
What is the worst "you can't do" or "you must do" that you've been subjected to in a security review? A fictitious example would be: you must authenticate all clients with a client certificate, plus basic auth, plus MFA token. Tell your story here, omitting incriminating details.
What is the worst "you can't do" or "you must do" that you've been subjected to in a security review? A fictitious example would be: you must authenticate all clients with a client certificate, plus basic auth, plus MFA token. Tell your story here, omitting incriminating details.
If you write software, you are most likely subject to a "security review" at some point
Wrong! My code has never been subjected to any such stupid security review.
Disclaimer: Opinions expressed here are mine, not my employer Equifax.
Disclaimer to disclaimer: Nah! I'm not really working for Equifax
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Working for a semi-well known mesh networking company in Seattle, I was hired for DevOps because, despite 20+ years experience with C/C++, the gasseous CTO didn't believe I was qualified to do development. About a month into the job, I got called into a code review for one of the senior developers, and I quickly caught several buffer overruns on the "rookie mistake" level with strncpy overflowing the allocated space.
Gotta wonder how many of those Mr. Senior Developer committed to the code base.
I was a high-ranking official in the state department. The FBI sent me a subpoena for my private email server because I used it to discuss classified government business, so I had my IT guy wipe my private email server before I handed it over to the FBI. Later he was discovered on Reddit and confessed to the FBI, but I made sure they couldn't trace the decision back to me.
This happened to a customer of us: They were told by an auditor that they absolute must have anti-virus on all machines, as per policy. Hence they built a tunnel into a completely isolated environment with absolutely no malware-vectors in order to be able to get updated AV signatures to the AV they installed on these machines. The really bad thing was that they did not seem to understand when we explained to them that they now did not have an isolated environment anymore and that the AV vendor as well as anybody successfully attacking the AV vendor could now attack them and export data at their leisure. What they should have done is to get an exception.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I'm not sure there is such a thing as a completely isolated environment anymore. There are too many air-gap bridging attacks. (See also Stuxnet).
Now, those attacks require far more work than the anti-virus vector. And it's not likely to be used. But it should be expected that something valuable enough (to a nation state) will be breached.
Your ad here. Ask me how!
Some software I was involved with developing had to undergo a security review. Little did I realize how unprepared for this event I was!
It started off relatively benignly. I was sitting in one of the conference rooms, waiting for the external security consultant to come in. He arrived a minute or two after I had arrived.
"Hello.", he started off. "I'm Steve and today we'll be performing a security review of the software you and your team have developed." He opened his laptop and started loading up the source code we'd provided to him earlier. He didn't even bother with any sort of friendly small talk.
"So I see you chose C++." he said after a minute or so of looking at the code.
"Yes, C++14.", I confirmed.
"Unacceptable.", he stated without hesitation.
I was slightly taken aback. "Pardon?", I asked.
"C++ is unacceptable.", he stated.
"We're using modern C++ techniques, including smart pointers and RAII. We also run our code through several static and dynamic analysis tools.", I explained.
"C++ is unacceptable.", he repeated.
We sat in silence for a couple of minutes as he continued to scroll through the code.
"Why didn't you use Rust?", he finally asked.
"Rust?", I replied. "We started this project before Rust 1.0 had been released. Plus our team is more familiar with C++."
"C++ is unacceptable.", he repeated once again.
He was starting to get agitated. "Why the fuck didn't you use Rust?!" he asked once more.
"I just explained why.", I responded.
"Don't you give a fuck about guaranteed memory safety? Don't you give a fuck about threads without data races?", he asked loudly.
"Well, yes, I do care about such things. But we can achieve those by using modern C++ sensibly."
As expected, he replied "C++ is unacceptable. C++ is fucking unacceptable."
I wasn't really sure what to do at this point. Clearly he didn't think C++ was an acceptable language to use.
My pondering was cut short. He abruptly started screaming, "WHY THE FUCK DIDN'T YOU USE RUST?! DON'T YOU GIVE A FUCK ABOUT ZERO-COST ABSTRACTIONS?!"
"C++ usually has zero-cost abstractions.", I pointed out.
This sent him over the edge. His face started getting a very deep red color, and I could see he was getting extraordinarily angry. "C++ IS UNACCEPTABLE! C++ IS UNACCEPTABLE! YOU HAVE TO USE RUST! RUST IS THE ONLY PROGRAMMING LANGUAGE THAT RUNS BLAZINGLY FAST, PREVENTS SEGFAULTS, AND GUARANTEES THREAD SAFETY!"
At this point I was starting to fear for my safety. I had read comments from Rust fanatics online, at places like Hacker News and Stack Overflow. But I had never expected these Rust advocates to be as egregiously agitated as this security consultant was.
Noticing that the door to the conference room was slightly open, and thankful that I was sitting closer to the door than the consultant was, I made a dash for freedom. I slipped through the door, and immediately started running toward my manager's office.
All the way I could hear the consultant screaming, "C++ IS UNACCEPTABLE! YOU NEED TO USE RUST BECAUSE IT HAS TRAIT-BASED GENERICS AND PATTERN MATCHING!"
I quickly explained the situation to my manager, who was wondering what all of the yelling was about. He quickly dialed the office building's security team, but they must have been alerted beforehand by somebody else, because the consultant's yelling abruptly stopped mid-way through a rant about the importance of move semantics.
To be perfectly honest, I have no idea what happened in the end. I assume the security consultant was promptly removed from the building. As for the security review of our software, I haven't heard about having to do any additional ones. Perhaps management realized that there were better uses for our time than listening to some lunatic berate us for using C++ instead of Rust.
Some nosyparker busybody customer did a "strings" on our product and found the string and ratted out to our CTO. Nothing serious happened, just a slap on the wrist. But another colleague told me the same customer found the full "man from nantucket" in his test strings for the stringutil library he wrote. And another said that customer also found the "Fuck! Got null pointer again!" in his code.
We think he was looking for some kind of debug switches and env settings that will disable license check.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I was summoned by a contract firm to a 500 person company that had been a victim of an inside job. They wanted a security review and fixes for "whatever that guy did". Turns out the guy was a half-assed developer. The client had spotty and in some cases non-existent backups. They wanted to pass a SOX audit (hahahaha!) while 20-30 machines were completely pwned and backdoored. He'd used everything from sub7 to more modern remote access & control tools. Some of the tools looked like ones he'd cobbled together himself from other tools. He'd also got in and falsified and buried a bunch of code hacks in their version control repo. Luckily, I was able to get that off tape and they only lost about a MONTH of code/work. The FBI got involved because the guy was out of state. I spent about 3 weeks gathering evidence and rebuilding servers, routers, print-servers, and other devices he'd hacked or otherwise tainted. My fees amounted to around $30k. A federal DA charged him with about 10 different hacking related and felony vandalism charges. After a pretty short trial (no jury) he was found guilty and he's still in the same federal prison in Louisiana. He actually has a cell near Bernie Ebbers. I had to talk to him once while he was in prison to get some passwords. The whole thing was surreal. Now get this, on the SOX audit? They passed! They got dinged for the hack but they still passed even before I was done cleaning up. That's when I realized that CISSP/SOX/GLBA/PCI security and *actual* IT security aren't always aligned. Audit all you like, but ... "stay frosty and alert. You can't afford to let one of those bastards in here."
Worst:
I use to work for a company, about a year ago, where no one had even the most basic concept of data security. During my time there I implemented MFA on all Servers, programmed in Data Encryption, Data Validation, Client Verification, DB Security and other such improvements. Well, I was showing the three other existing employees how the software worked and how the new infrastructure worked, they didn't like that it was now "hard" to log into the servers and that they would now have to use password and keys to access Data. They told me to revert to how it was before, as they knew better then I did, so I quit. They reverted all my changes and claim it's now more secure and better!
The software product they're developing, without a developer (they still don't have one), is an iSCSI based Desktop Protection System, but it's so riddled with holes and such a massive lack of security that they're committing fraud by selling what they have as a security solution.
Best:
The best security I've ever seen and been involved with developing had multilayer client authentication, certificate binding, transaction queue verification. It had a routine that went through the software and tweaked it's ports and accesses. Every piece of data was run through an AES-192-GCM based function that signed all the transactions and messages. The infrastructure this software was running on was just as impressive, ever server had at least 3FA+ turned on for logging in, active port based monitoring, which used MongoDB Clusters to validate logins, clients and pretty much everything you could imagine.
I have been a developer since about 1990 and I have been occasionally re-purposed to perform security reviews
The first time was in 2000, we were a data center that was part of a fairly new fiber company.
We were in a partnership with a large document management company and some open source organization.
The website that we were hosting kept crashing and the PM responsible for it had lost the admin password.
It ran on Oracle and I was able to use some default passwords and os level functionality in Oracle to grep the file system and identify the system passwords.
Unsurprisingly, I found that they were using commonly known passwords that were present throughout their training materials
When I asked if they could change the default passwords to new values, and rotate them regularly, the response was "No, we think that will break everything"
This resulted in two things, I because the Unix security guru, and I got them kicked out of our datacenter.
Similarly, one day our SAN admin noticed that DVD images were being stored on our SAN. We traced it back to another start-up that had been using TELNET to log into their box as root. Apparently this traffic had been sniffed out and our systems had been compromised. This resulted in a clean-room rebuild of all active systems and me being tasked with writing security policies to publish to our customers, follow them and be welcome, don't follow them and buh-bye...
You may notice that this is all being done in a completely reactive manner since at that time, apparently, nobody in the fucking planet had a clue about how to build and admin a secure system.
Since that time I have fought the "make a system where you can change the default passwords", "Keep your systems patched up the currently available level", "why the fuck are you passing strings to your database without scrubbing them", and "sure, that is what you think your firewall is doing, but it is not really doing that" battles over and over and over.
They say that security starts at the code level, and you can really fuck yourself over by taking a poor approach, but most security problems are just plain piss-poor admin skills. IMHO
Common Sense? A set of assumptions, usually false acquired before age 18.
Like rust?
Gamingmuseum.com: Give your 3D accelerator a rest.
False Positives during automated audit tools is my own personal hell. PCI Compliance demands these audits be ran every quarter. And every quarter, our Windows 2012r2 server which is only used for a couple of people to work remotely fails the audit. Which test does it fail? The audit claims it is vulnerable to a Windows NT4 terminal services exploit. The exploits have long been patched by Microsoft, plus the effected cyphers have also long been disabled. Yet every single goddamn quarter, we fail the audit, and it is usually a month long battle with one-way messaging to the audit company to let them know their still a bunch of morons. And guess what? The quarter just started this week!
So basically I should start using 64-bit keys.
In 1999 I was hired by a Midwestern telco -- in the interests of not getting sued I won't say which: I'll just say their market cap used to be in the billions and now you could buy them with the lint in your pocket -- to do security remediation on their billing system. I spent weeks poring over architectural diagrams, going through source code, examining protocols. After a while I realized I had some really scary information, so I asked my manager for a safe.
"Just put it all in a binder," she said. "We trust you to keep an eye on it."
The Binder of Doom was a nondescript black binder about three inches thick. It had no cover page and no markings: I didn't want anyone to realize the secrets that were in it. I carried it around with me everywhere. I slept with it in bed with me. That's how terrified I was these secrets would come out.
Then the Binder of Doom got worse. Having completed my survey, I now devised attacks on the system. I found ways enterprising individuals could fleece the company out of truly mind-boggling sums, and how difficult it would be to detect these attacks with the then-current security infrastructure. By the end of six months the Binder of Doom was stuffed to bursting and I was giving serious thought to filing for a concealed-carry permit. I wondered if the sheriff's department would understand if I told them I was routinely carrying around a binder with a *conservative* worth to a criminal syndicate of $100 million.
I went back to my manager. I told her I was done. It was time to remediate the risks. "Oh, excellent," she told me, "because we just ran out of money for the remediation."
Uh. What?
"Management has decided the main risk is in unsecured communications links, so just ensure we're using PGP on everything and we'll call it good."
I asked if she wanted the Binder of Doom.
"No, you hold onto it for a while."
So I became increasingly disgruntled, bitter, and sarcastic. I told everyone I worked with that I'd been retasked to "secure" our network using PGP -- and even old-school PGP 2.6, not GnuPG (which had just reached 1.0), either -- and oh God this is awful and if this company lasts another year it'll be a miracle and...
I was shortly thereafter cashiered for having a toxic attitude towards work. I walked into the parking lot, got into my car, and tossed the Binder of Doom into the passenger seat. As I drove away I realized something was horribly wrong, but didn't realize what until I was pulling out of the lot:
I HAD THE BINDER OF DOOM IN MY PASSENGER SEAT.
I returned to the office and tried to walk inside, but was met by an HR rep at the door who told me if I didn't leave they'd call the police and file a trespass charge. I held up the Binder of Doom to the HR rep. "Do you want this back?" I asked.
"No," she told me clearly. "Keep it. We just want you to leave."
I turned around, gobsmacked, and left the company holding detailed plans for how to embezzle $100 million or more... which the company had just thoughtfully delivered into the hands of a disgruntled former employee.
(And if you're wondering what I did with the Binder of Doom, it sat on my bookshelf for a few days tempting me before I threw it into an incinerator and threw the ashes into a strong wind.)
Most folks, including many so-called "experts", lack both the knowledge and ability to do anything close to a "real" security check. So the best route is to rely on "canned" testing that has been created and is maintained by a reputable group.
First, scan the platform (with the application installed) for known vulnerabilities, including updates, configuration (CVRs, STIGs), rootkits and antivirus.
Second, scan the source code with all available static analysis tools. Start with lint, then do as many more as you can afford.
Third, do as much run-time testing as the schedule permits, being sure to do not only functional testing, but also fuzzing at all levels.
Do the above using Free/Open tools, but also be sure to check out commercial tools. Yes, some tools are very much worth paying for.
That will get you 99% of the way there, with minimal time and effort invested.
I worked at a place where we had a lot of disk (~2TB) with data that were accessible to the public. We also had a web site in place where users could upload new data, which would then be vetted by staff and then published to the public. This was all okay.
The bad part starts when we hired a new guy who, among other duties, wound up redoing the upload interface. So he redesigns and implements the system. I wasn't part of that process, and I wasn't paying any attention to how he was doing it. Later on, he quit, and his codebase was passed to me to maintain. That's when I started looking at the code and discovered that he had implemented a server-side API for uploading data that required no credentials whatsoever--he had set up a password authentication on the web front-end, but the API itself was open to the world. Oh, and the new API also stored uploaded data directly in the publicly accessible disk space. Any rando on the Internet who discovered this API would be able to upload hundreds of GB of whatever porn and warez they wanted, and just pass the URLs out freely. This code had been running in production for months.
Luckily, apparently nobody noticed. I audited the file system and its contents were exactly the files we expected to be there, and with the correct hashes. But it all made me wish we had a better review process, if this was the kind of coder they were going to hire.
Through a client referral, I was introduced to a company that was in sudden need of a new web host. Their current Australian host was shutting down, and they had two weeks (by the time I was referred) to move their small Canadian site elsewhere.
When I say "small Canadian site", I mean the site was a small, promotional, site, with little more than five pages and a signup form.
Little did I know...
This was ultimately the consumer-brand of a large telecom provider -- a very large, national, telecom provider. This "small" site, was a mass-market allowing consumers to sign-up, and to also pay their monthly long-distance bill. This was circa 2010.
We shook hands, I said: "sure, I can move your site in the two weeks, just give me the credentials to it, and I'll figure it out."
Wow was that a mistake. Anyone heard of CakePHP?. I had to figure it out pretty fast.
It was late one evening, when I discovered the page that allowed customers to pay their bill online -- something no one had told me was a part of this tiny site. There was no https/ssl to even hint at it. And then I saw the MySQL insert statement, and the variable "card_number". And I was scared.
I said, to myself, "no, it can't be!" There must be some part of the platform wrapping the database call that must mask-out the card number. Or this must not be the actual card number. Or maybe it's not used anymore. Or something.
Then I logged into the phpMyAdmin, with the credentials given to me.
So, this is when you need to understand something. I'm a small independent web developer. At the time, I was teeny tiny. I had no written contract. The e-mailed and in-person job discussions said nothing of sensitive information of any kind. No money would be transferred until the job was done. So at this point, there is effectively zero legal agreement between us.
I looked at the table, I saw over forty-thousand records, each with real, live, credit card numbers. . .and expiry dates, and card holder names, and purchase amounts, and confirmation/approval codes.
I was stunned.
Obviously, being the non-criminal that I was at the time, I told them. I told them that I was appalled. I told them that it can't stay this way. I told them that I was going to charge them a few hundred dollars to encrypt the field, and the very least -- I was too young to know that I should have been charging way more.
They said they didn't care, I should just leave it as-is.
That was over a decade ago. Ever since then, I've learned that there are very few clients who will pay five cents towards security, backup, or encryption of any kind. In my entire 25-year career (so far), I've met only two clients who'll invest in that kind of safety.
So I no longer bother even suggesting that security or backup is a good idea. My legal contracts ensure that I'm not legally liable for the consequences of doing anything that they've explicitly told me to do, and that's good enough for me, I guess.
So to all those youngin's not yet jaded for failed efforts to be good, enjoy having the hero-skills to save people; but if your career is anything like mine, you'll quickly learn that those skills carry a perfectly zero-dollar value.
In the days of Equifax, riddle me this: where's the law that says you can't store millions of archived data all in one place, forever, online? Some of these 40'000 records hadn't been charged in over a year -- clearly old/former customers. And aside from those from the current day, all of them were old records that were no longer needed at all. Equifax had e-mails from ten years ago. How about a very simple law saying that things get taken offline eventually? Your ten-year-old e-mail can be accessible from that machine in the corner of the office, or through a request for the tape backup, and that's good enough 99% of the time.
But hey, where's the law that says one model of gun is illegal.
Thanks for the freedom.
You jest, but that's what we did.
Very similar setup, a completely isolated network with no way to bring data in an automated way into it. Data was entered manually only and extracted on CD-Roms. Similar problem, getting any kind of data line in would have required unacceptable security breaches. And similar requirements, i.e. all machines need antivirus software.
So we wrote one. What that software basically did was to routinely check all hashes of all files on the machine (with the exception of the data files that contained text-only and some system files that didn't want to play nice) and locked the machine if they found a single file that did not match the whitelist.
A "signature update" was basically us recalculating the hashes whenever a planned change occurred on a machine. Since that happened with a frequency of about once a month (i.e. at MS patchday) and we had mostly identical machines, that was not really a big deal. We simply created a "template" machine, patched it, created the hashes and rolled them out to the other machines along with the patches.
In other words, we reduced the problem of "you have to have an AV" to what the AV is supposed to do: Ensure that nothing bad exists on the computer. Since our approach was more restrictive than the required one (i.e. nothing unknown allowed at all instead of nothing bad allowed), it was considered acceptable.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
If you allow changing of the hardware, then the term "isolated system" loses all meaning. Feel free to do so, but know that your statements become nonsense if you do this.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
"Common sense" is just a term that means "agrees with what I already think."
The Quirkz Handbook of Self-Improvement for People Who Are Already Pretty Okay