Security: Why should software engineers care?
While technology has continued to improve, it feels like security in technology has taken a back seat. In today’s article, I want to discuss why it is important to a software engineer and what some of the things you should do are. Let’s peel this onion and improve our security practices.
Story Time
Before I talk a lot about security, I want to share a story. I’m not going to go into specifics, but I have had to handle a data breach. This data breach resulted in parts of several million customers' personal identification information (PII) being stolen.
PII refers to any combination of information that helps identify a specific person. Several different pieces of data are considered PII. The more commonly known are the social security number, driver's license, and birth date. Essentially, it is any combination of details that can be used to steal someone’s identity.
I was actually on vacation when the data breach occurred. Little did I know that my whole vacation was about to be ruined when the on-call alerts started going off as a result of the data breach. I didn’t get involved until later in the day, but I vividly remember the moment I realized what was happening and why. It is not a day I will forget. At close to 1:00 AM, I sighed deeply and called my boss to let him know what happened. I can still hear his sigh as he prepared to call his boss.
If I’m honest, there are times I can’t help but wonder if I should have just closed my laptop, went back to sleep, and pretended I didn’t see anything. I didn’t do this, nor do I regret not doing it, but for the next three months, it was as close to hell on earth as I’ve ever experienced. Everyone on the team was putting in 10+ hours a day as a result of the breach, with myself putting in 15+ hours as I worked with the head of security and lawyers.
What are the ramifications of a data breach?
More often than not, security takes a back seat. Oftentimes, it’s treated as not a problem…until it is. When it does become a problem, there is no going back. All you can do is push your way through the fallout. While a lot of this is on the business level, it all eventually affects the frontline, that is to say, the engineers.
Customer Trust is Lost
The second your customer finds out their data is stolen, all bets are off. You’ve lost their trust. It’ll be difficult to get it back. Undoubtedly, you are going to lose customers to your competitors. Your dedicated clients aren’t likely to leave, but you can be assured that they will be taking a different look at your security.
Cyber Insurance Goes Through the Roof
If you’ve ever had minimum coverage on your vehicle and got into a wreck, you’ve probably seen your insurance rates go up as a result. This is why some insurance companies advertise how your rates won’t go up if you get into a wreck that isn’t your fault.
Cyber insurance is the same way. A data breach can triple, if not quadruple, or worse, your annual cyber insurance cost. This alone is enough to shut down a smaller company.
The Company May Get Sued
We tend to live in a lawsuit-happy society. Chances are high that some of your customers will get together and file a class-action lawsuit against the company. One customer suing your company is bad enough. Class actions are far worse from a total cost perspective.
Development Grinds to a Halt
The second there is a data breach, all future development has to stop. You can’t work on any new features, even if they are a day away from completion. It’s an all-stop, all hands on-board scenario where everyone has no choice but to identify, locate, and fix any and every possible data hole that remains. This is a process that will ultimately depend on the size of the application and how diligent you’ve been in your code reviews and whatnot. It may take you a few days, a few weeks, or even months.
Package Management Hell
As a result of the holes you find, you have to update everything to the latest versions of softwares to ensure all third-party packages have no known vulnerabilities. If you haven’t kept your packages updated before this, good luck. The longer it’s been since you’ve updated, the harder it’s going to be to update. What’s more, if you have no automated testing in your application, you just doubled if not tripled the time that this process will take, because you have no way to easily know everything is correct.
Your Job May Be On the Line
Not all companies are going to be forgiving. If you’re the one who added the code that resulted in the data breach, you’re on the chopping block. You might make it through it, you might not. Who knows. It depends on the company. Additionally, even if you stay on, that doesn’t mean you won’t be the first pick if they do a mass layoff or similar.
You Will Never Be The Same
Once you experience a data breach first-hand, you change. You won’t be the same. It’s such a long, drawn-out, soul-draining experience that you will never want to go through it again. From that point on, you’ll be thinking of security in a completely new light.
The Company May Close its Doors
While this is the worst-case scenario, the company may close. Between the massive increase in cyber insurance, paying out lawsuits, hiring lawyers to handle the legal side of things, and no new development to bring in new customers, the company is going to be forking out a lot of cash. I’m not talking in the thousands, I’m talking in the millions. If the company can’t afford to pay it, good luck. Heaven forbid you have no Cyber Insurance, it’ll be even worse then.
What Can You Do to Prevent a Data Breach?
Now that we’ve talked about the ramifications of a data breach, let's talk about the things you can do to prevent them from happening in the first place. Remember that you are only as strong as your weakest link.
These ideas are not in any specific order.
Adopt a Zero-Trust Policy
Zero-Trust is about only giving people the access they need, and only when they need it. This means adding features like double authentication to the administration menus. For example, you can make a user enter their administrative password when updating a user account in the admin.
Use a Password Manager
Use something like KeePassXC to store your passwords. They are encrypted and cannot be accessed without a password. You can use other flavors as well. It’s really up to you. In the end, this isn’t so much about the manager itself, but rather ensuring that you have unique passwords for every application and/or service you use.
To be clear, the most common form of “hacking” is probably “Phishing”, that is to say, a fake e-mail taking you to a fake form that asks for your password. Most people use the same password over and over again, regardless of the service. All they have to do is identify the password for the weakest account, at somewhere like a forum you signed up for 10 years ago that you haven’t been to in 8 years, and bam they have your password for your e-mail. At the very least, if you are not going to use a password manager, always make sure your e-mail/active directory password is different from everything else.
I once received a Phishing email and didn’t pay attention to it. Had I paid more attention, I would have realized that the email, demanding payment for an expired account, went to the wrong address for the account. I was on mobile at the time and should have waited till I got home. Instead, I immediately clicked the link and ‘logged in’ to submit the payment. I inputted my credit card and was preparing to submit it when I slowed down and realized something wasn’t right. I immediately reset the password on my account. I also put a lock on my card. Not 30 minutes later, I received an email for an unauthorized attempt on the account. I think I may have received it for my email account too, but I don’t recall for sure. I also received a notice on the locked account that a charge had been attempted and denied since the account was locked. Needless to say, I had to get a new card.
Implement Multi-Factor Authentication
Multi-factor authentication has all but become a must-have for anyone on the internet. You should focus on implementing MFA into your applications and enforcing their use. Yes, this may be inconvenient for your customers, but it’s to protect them as much as it is to protect you and your company.
Run Regular In-House Penetration Tests
Tools like Arachni or Acunetix are excellent penetration testing tools that you should incorporate into your CI/CD pipeline, if possible. If you can’t incorporate them into your CI/CD, then you should at least schedule regular tests.
Don’t skimp on pen testing. That is to say, don’t get caught up in the thought that you don’t have to test it under an admin because it’s ‘unlikely’ to cause an issue. I once worked on a product where savvy tech people figured out how to log in as the CEO of a customer (the admin) without a username or password.
Run Annual Out-of-House Penetration Tests
Similar to In-House penetration tests, consider hiring an out-of-house security company to run the pen testing directly. They are likely to identify things you didn’t think of.
Don’t Assume Anything
This is likely the most important lesson you can learn here. Never, Never Never! Assume anything about your code, your environment, or your overall security. It might be tempting to say ‘I’m not going to worry about ‘XYZ’ because that will never happen’. By the time it does happen, it’s too late. For example, you can’t assume that just because a page is ‘admin only,’ it will never be the target of an attack.
Today’s market has several automated tools for various purposes like one made explicitly for trying to download a database by SQL Injections.
Leverage Multiple Code Reviews
Most every engineer knows what a code review is. Despite this, it’s still common to see companies not implementing them. As engineers, we make mistakes. After spending hours if not days in the code it can be easy to get lost in the minutia of it all and miss something as simple as forgetting to sanitize your inputs. Not only should you implement code reviews, you should implement them such that multiple people on the team review every ticket.
Familiarize Yourself With the OWASP Top 10
The OWASP Top 10 are the most common forms of security issues found in applications today. You should familiarize yourself with them regularly.
Admittedly, I can’t name them all myself. I am at least aware of their existence and do reference them.
Adopt a Shift-Left SDLC
At this point in your career, you’ve likely heard of DevOps. One of the newest iterations of DevOps is the concept of DevSecOps. The idea between DevSecOps and ‘shifting left’ is that you are moving security further left in the SDLC. It’s common for security to be a secondary thought. In most cases, ‘securing’ the software isn’t done until after all the work is otherwise completed. Shifting Left means talking about security as early as the planning phase when it’s just a scribble on a napkin.
Implement Automated Testing
Automated Testing isn’t technically a security thing, but rather a piece of mind. When you start worrying about maintaining your code and dependency hell, having automated testing in place will make these much easier to manage and will allow you to respond to security issues much faster.
Avoid Storing Unnecessary Data
While it might be cool to send your user an e-mail on their birthday, the full birthday is a component of PII. You have to take extra, sometimes complicated, care to store and protect this data. If you instead, just avoid storing the data, to begin with, you eliminate another vector of data that can be stolen or used to identify the user. Either just don’t store the data unless you need to, or store it in a manner that doesn’t matter. For example, instead of storing the whole birthdate, instead only store the month and day. You don’t need the year to be able to give them a happy birthday.
Adopt Deployment as Code
Also, not technically a security thing, but adopting Deployment as Code helps enforce a Zero Trust policy. You configure your deployments through software like Octopus and remove developer access to be able to get to product machines.
Many years ago, one of my coworker’s work computer was infected by malware. This malware scanned their Filezilla FTP configurations and uploaded the credentials to a remote site. This remote site then started connecting to every website and introducing a base 64 encoded javascript snippet into the website that automatically downloaded the same malware and installed it. Rinse and repeat. Fortunately, I was using Avast Anti-Virus at the time. I went to one of our client’s websites and suddenly got a virus notice. It took us several hours to track down what happened and it took many more days to clean up all the websites infected by the malware.
Harden, Harden, Harden
As an engineer, we must take every chance to harden the software in any way we can. This can mean anything from sanitizing your inputs to using both client and serverside validation. Security should be something you are constantly thinking about. Anything you can do to harden your code, no matter how small, will go a long way. You can consider things like casting variables, sanitization, encrypting values through JWTs, or other similar concepts.
Hardening of course doesn’t apply to just your code. If your team is responsible for the infrastructure, then proper hardening of the infrastructure matters too. Implementing a Web Application Firewall, using fail2ban, and proper port blocking on the firewalls are all examples of things you can do to harden the infrastructure.
Think Twice Before Accepting a Risk
This should seem like a no-brainer, but be sure you fully consider the ramifications of accepting any “risk”. There will be times when something may be a false positive. For example, you may determine that an identified vulnerability doesn’t matter to you because your product doesn’t implement the specific code that creates that vulnerability. Something you must consider here is what happens if later someone does implement those changes without retriggering the error.
I cannot stress enough to consider the ramifications of marking anything as not a risk. “You’re only as strong as your weakest link” applies here more than ever. If you ever think something “couldn’t happen”, assume it will.
Implement Security Training
Implement security training for your engineers. Places like Secure Code Warriors will help developers train on security-specific scenarios and raise general awareness across the team.
Keep All Third Party Packages Up-To-Date
No amount of hardening you do to your codebase will do any good if the supporting 3rd party packages aren’t updated regularly as well. Determine if the packages are using Semantic Versioning or not. If they are, then you know you can safely update the patch version all the time, and usually the minor version as well. This allows you to implement regular updates without much concern. You may even be able to automate these updates. If the package isn’t using a semantic version, this will be more difficult. The longer you go between package updates, the harder it is to update the packages without far greater strides.
Scan for Issues Now
You don’t want to put off scanning. Even if you are unable to start fixing things immediately, you need to know how many Critical, High, and Medium issues you have. Focusing on the OWASP Top 10 and getting them fixed as soon as possible is crucial to ensuring your protection.
Don’t Rely on Client-Side Validation
It always amazes me how many modern applications only implement client-side validation in their web applications. JavaScript is something that can be easily avoided, or even disabled. Furthermore, a hacker need only monitor the network to see what requests are being made and then use software like Postman to force requests on the server. You should always implement server-side validation, even if you have client-side validation. Client-side validation is to prevent clients using the software from doing something ‘stupid’. Server-side validation is to prevent hackers from doing something nefarious.
Use Parameterized SQL
Parameterized SQL has been around for a long time, but it still amazes me how many applications don’t use them, instead using just raw queries with no validation. Parameterized SQL naturally protects you from SQL Injection even if you don’t put in server-side validation.
Avoid Reinventing the Wheel
It can be tempting to implement your own API or logic for a common functionality. Instead, consider using a frequently maintained 3rd party package over rolling your own. Admittedly, I’ve seen few engineers in recent years doing this.
I’m mostly referencing a recent surge in the use of ‘stateless’ mentalities. To be clear, I have nothing against stateless applications. Rather, what I have seen is engineers implementing ‘stateless’ applications using something like JWT or OAuth tokens, but building this on top of the built-in or established session management, like Spring Framework, or even implementing their own session management.
Established, frequently updated, frequently used packages will usually always have better security management for things you haven’t thought of.
Adding SAST Scanning
Leverage tools like SonarQube to implement SAST Scanning. SAST, or Static Application Security Testing, is a method of scanning the code without actually running the code. There are multiple SAST libraries out there for all major coding languages. These tools statically scan the code base, looking for known issues or commonly seen issues in the codebase.
Implement Mutation Testing
I talked earlier about automated testing. However, how do you know if your automated testing is good enough? This is where Mutation Testing comes into play. Mutation Testing isn’t as prevalent as Automated Testing, so not every language will have available options.
Mutation testing is an add-on to the automation testing. It works by first running all the automated tests, with code coverage enabled. The mutation framework will then use the code coverage to determine what lines in the code are covered by a given unit test. It will then scan each line covered for known ‘mutations’ that can be made. For example, if the line returns an object, a known mutation might be returning null. The framework will then generate patches for every mutation. It will apply those patches, one at a time, and re-run all unit tests that covered that line of code. If the unit test fails, you have ‘squashed’ the mutant. If the unit test passes, the ‘mutant’ escapes. If the mutant escaped, it means that your unit test failed to catch the mutant and your unit test is incomplete. Using the returning null example, it means that your unit test isn’t confirming the return value of the method.
Industry Awareness
This is a bit harder to come by, but every industry may have additional concepts that you need to ensure everyone on the team is aware of. A good example of this is Health Insurance and HIPAA. HIPAA, or the Health Insurance Portability and Accountability Act, is a set of laws in the Insurance industry that sets certain regulations about what type of data is kept, how that data is handled, and who has access to that data. If you have ever heard the phrase Doctor/Patient Confidentiality, you can thank HIPAA for that in many ways.
Another example is PCI Compliance, or Payment Card Industry Compliance, which are the rules and regulations around how you work with Credit Card Data. Were you aware that if a Credit Card Number passes through your infrastructure at all, you are required to be PCI Compliant, even if you don’t store the Credit Card?
SOC 2
In recent years, SOC 2 has become more prominent. It is very common nowadays for a customer to require you to show proof of your SOC 2, a SOC 2 Audit as it is usually called. The customer wants to know that you are handling their data in a very secure method. Being SOC 2 compliant is a big process, and is one of those things you should consider doing now instead of when the client asks. Adding SOC 2 sooner rather than later is part of the Shift-Left mentality.
It’s worth noting that implementing SOC 2 is no easy task. It involves all layers of the company and requires setting up very rigorous procedures that have to be vetted and approved. The one good thing about SOC 2 is that unlike HIPAA or PCI Compliance, it’s not as rigid. With HIPAA or PCI Compliance, you have explicit rules you must follow. For SOC 2, the type of data you are working with ultimately dictates the degree to which you configure SOC 2.
Raise Awareness
Make sure that everyone in the company is aware of the ramifications of a security breach. Ignorance is not bliss when it comes to security. Ensure that the company has proper controls in place to raise awareness. This can be simple videos or in-depth training. Encourage the company to introduce a security department, if there isn’t one already. Determine the type of data you store and if it warrants getting Cyber Insurance. You must make them understand.
Implement a Captcha
Everyone has probably seen that checkbox that says I'm not a robot. This is called a Captcha and it uses a variety of methods to ensure a form cannot be submitted by a nefarious bot, or any bot for that matter.
Conclusion
While I hope I haven’t scared you, I hope I have at least made you understand what the concerns with security are. I also hope that you will start looking at security in a new light. Take my word for it, you don’t want to deal with the fallout of any type of data breach. I know we all hope we never have to deal with hackers, but so long as technology exists, so will the hackers. There is no changing this. We have to do everything we can and ensure security doesn’t take a back seat at any time.
