In this world of lightning-fast development cycles, MVPs, and DevOps, it may intuitively feel like security gets left behind. You might be thinking, “Aren’t the security guys the ones who want to stop everything and look at our code to tell us how broken it is right before we try to deliver it?” Many feel that DevOps security is a pipe dream.
Is it possible to be fast and secure? Lately, I’ve been drooling over a sports car—namely, the Alfa Romeo Giulia Quadrifoglio. Long name, fast car. It holds some impressive racing records and sports 505 horsepower but also is a Motor Trend Car of the Year and an IIHS Top Safety Pick. These awards are due to automatic braking technology, forward-collision warning, lane-keeping assistance, blind-spot monitoring, and rear cross-traffic alert. It is possible to be fast and safe.
The key to DevOps security is to move forward with development. Security teams need to understand why DevOps practices are so effective and learn to adopt them.
The Three Ways of Security
DevOps is based on a set of practices known as “the three ways.” If you need the details, check out The Phoenix Project and The DevOps Handbook. I’ll be giving you a primer on what the three ways mean for your team.
Development and infrastructure teams have had time to understand how to apply the three ways. However, the application of these ways into security teams is not as well understood. In fact, many companies are scrambling to find out how to do DevOps security effectively. While exploring how security can enable DevOps, we’ll also see how security teams should apply DevOps to themselves.
Let’s get started on the path to enlightenment.
The Principles of Flow
The first of the three ways is the principle of flow. Flow involves three things:
- Making work visible.
- Reducing batch sizes and intervals of work.
- Building quality in.
Flow is all about reducing the lead time required to fulfill internal and external customer requests. Development teams at mature DevOps shops implement new features and deliver them every day.
Security teams can apply the principles of flow in two ways: in supporting the development teams and within their own processes.
DevOps Security: Go With The Flow
Development and security teams have the same goal at the end of the day: deliver high-quality software that will either make or save your business money.
Flow is what makes DevOps go. Development teams deliver value more quickly than ever. However, you don’t want just a faster way to deliver security bugs that cause a data breach. If this happens, your business will come grinding to a halt. Security teams need to build security into the SDLC without affecting development flow.
A great way to do this is through automated security test cases. Create unit and integration tests within the codebase that exercise security functionality, and ensure that functionality is complete. Also, many vulnerability-scanning tools can be automated so that they run with the automated build process. Include these scanners and fail the build if any critical vulnerabilities are found.
The best way to secure your code is to design security in from the start. If you’re not already threat modeling, start now. Identify threats before you build so that what you build will defend against these threats,
Don’t forget to secure your actual pipeline as well. Closely monitor who has access to build servers. Don’t allow any changes to production that don’t go through the pipeline. Artifacts should be cryptographically signed so that they can’t change after they’re created.
Make sure that the security functions you add don’t affect developer flow. Protecting that flow is the number one priority of DevOps teams. Security should impact it as little as possible.
Applying Flow to Security: The AppSec Pipeline
Before moving on to the next “way” of DevOps, I’m going to briefly touch on how security teams can apply the principle of flow within their own teams. Many security teams don’t “flow” very well.
Application security teams at large companies, especially highly regulated companies, tend to be the bottleneck in workflows. Developers don’t want to engage them because it takes so much time and paperwork to get anything done. This should not be the way. The era of long Word documents of procedures that no one reads and manual processes needs to come to an end.
Therefore, application security teams should apply the principle of flow within their own teams. Get rid of processes that no longer add value. Ask your developers what their pain points are and fix them. Use automation within the application security team’s own processes to eliminate manual steps and human intervention.
If you need help wrapping your head around this concept, then check out OWASP’s AppSec Pipeline project. I love the idea of using DevOps and lean practices within the application security team itself. It helps the security team and the development teams to keep their flow as high as possible.
DevOps Security: Gathering Feedback
The second way of DevOps is amplifying feedback loops. Effective feedback involves understanding what your application is actually doing in production. Unfortunately, this is usually not on developers’ minds when they’re building software. The principle of fast feedback aims to change that.
Monitoring your software properly is just as important for security as it is for operations. Despite your best efforts, your software will have vulnerabilities. Any complex system does. However, instead of allowing this fact to paralyze you with fear, meet the problem head-on.
Prevention is preferred, but detection is better. You need to quickly see when problems occur, and you need to react just as quickly.
At a recent training I attended, the trainer relayed a story that illustrates the importance of fast feedback. Etsy, the online craft marketplace, hired a penetration tester. During the course of the test, the tester found a vulnerability and exploited it. However, after a short time, he tried the same exploit and it didn’t work anymore. What happened?
His first attack set off alarms that the operations team caught. They saw the attack, found the vulnerability, and immediately fixed it and moved it to production. Due to Etsy’s ability to provide fast feedback, including security indicators, the operations team stopped an attack that was in progress. Of course, this instance was a test, but you can imagine that this capability is brought to bear on real attacks just as effectively.
Your applications can behave in the same manner. Add logging around key functions, such as the login page, and monitor those logs for anomalies. Those anomalies might be attacks in progress. This allows you to act on these attacks before any real damage is done.
DevOps Security: Never Stop Learning
The final way is the principle of continual learning and experimentation. It’s important to encourage a culture that embraces these ideals. This may be more of a culture shift than you think.
When something goes wrong, what is the attitude of those that investigate the problem? Do they look for someone to blame and punish, or do they look for answers so that all can improve? This tells you all you need to know about your current culture around learning.
Security is no different. Bad guys are always coming up with new ways of breaking in. When your advanced monitoring and logging capabilities find that something is wrong, don’t look to place blame. Instead, use it as a learning opportunity for both the development and security teams.
Encourage the security team to learn and experiment as well. Give them the freedom to try new things, learn new things, and break things while trying to make them better.
Many companies speak about security awareness and how important it is. Why not experiment with new ways of educating end users about security? Run a security carnival that shows how bad guys get into a system or trick you. Make it personal for the end user, and they’ll be more engaged. If you show them a cross-site scripting vulnerability, they’ll be bored immediately. But do some live social engineering to trick them into doing something, and that will stick. Make it fun and interactive.
The bottom line is that security can be just as fun as writing code, if not more so. So encourage your security team to have fun and discover great things along the way.
Security Is a Turbocharger, Not a Parking Brake
The Alfa Romeo Giulia Quadrifoglio rocks a supercharged engine that pushes the car to amazing speed and ability. However, the car is still safe. Your software can move just as quickly and just as safely as the Giulia. Don’t view security as the “parking brake” that stops you from moving forward. Instead, security can turbocharge your applications by providing the safety features that make your application truly robust and profitable.
Flow is the turbocharged engine that drives DevOps. Security should be the blind-spot monitor that looks for what developers don’t always see. Fast feedback provides the monitoring needed to react quickly. Security in monitoring acts as the forward-collision warning and automatic braking system, making sure you safely navigate the dangers of the digital world. Finally, your learning and experimentation are key to moving your application forward. The Alfa Romeo Giulia doesn’t exist without experimenting with what is possible. Your next big security improvement or solution could come from a small experiment.
Now go get behind that wheel and drive your company to success.
This post was written by Justin Boyer. Justin has been a software developer for eight years, doing development across all layers and using multiple languages and frameworks. He’s recently made the switch to security, becoming CSSLP and Security+ certified. He blogs about application security practices and principles at Green Machine Security.
