Building Holistic Security Into your DevOps and Throughout your Organization

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.”    ~ Bill Gates

 

The motivation to introduce DevOps in most organizations is to increase the speed by which feature enhancements and new developments are brought to users in a way that aligns with business goals. Speed, in and of itself, is not a bad motivation; especially when aligned with business goals that include security.

The heavy reliance on automation within the DevOps organizational model will benefit organizations that have included security in addition to speed as the driving business goals for their DevOps initiatives. The result will be efficiency in those areas (speed and security) being magnified, and an overall reduction of cybersecurity risk in the process.

Conversely, those organizations that only include speed as the driving business goal for their DevOps initiatives will find that inefficiencies related to security will be magnified; and therefore, increase exposure to cybersecurity risk in the process.

If luck favors the well-prepared mind, DevOps favors the well-prepared organization. Especially when it comes to security in DevOps.

Most organizations like to believe that they operate with the best of intentions. However, competitive market forces, poor past decisions that lead to the accumulations of technical debt, and organizational culture that slowly evolves over time to include both positive and negative attributes, all contribute to produce challenging operating conditions. These challenging operating conditions are the areas that will be magnified once the automation implicit in a DevOps organizational model are implemented. This is especially true for security weaknesses present prior to the implementation of DevOps.

Five examples of challenging operating conditions that will impact security:

1. Lack of clearly understood business objectives
2. Poor work completion and time tracking metrics for development and IT teams
3. Opaque work commitment procedures
4. Production constraints like unidentified and/or misunderstood bottlenecks
5. Poor Quality Assurance (QA) procedures that result in unplanned for work

Each of these challenging operating conditions make it more difficult for organizations to establish an accurate base-line understanding of their IT production capabilities and the throughput of the overall IT system. These are the areas that automation will magnify the strengths and weaknesses of once implemented as part of DevOps. If security hasn’t been addressed from the outset and throughout the Software Development Lifecycle (SDLC), DevOps will magnify the security gaps produced by this oversight.

Implicit in the DevOps organizational model is the very important goal to increase the throughput of the overall IT system. In order to increase that throughput the organization must:

1. Realistically establish clear business objectives for the work that is placed into the system.
2. Determine an accurate understanding of the system’s capacity.

Though these two items don’t address security specifically by name, if security is not factored into the establishment, assessment, and understanding of both, they will increase the risk of introducing security vulnerabilities and increase the overall risk to the organization.

That said, how can organizations align the operational challenges with the DevOps requirements for increasing the throughput of the overall IT system while also integrating security throughout the process in the most holistic way?

1. Define realistic business objectives that prioritize security. This is perhaps the most important step. By placing a top priority on security as a business objective, it ensures that these considerations are a part of the work that is placed into the system from the start, which is the goal. This will have positive downstream impacts, and help to ensure that security is central to the DevOps organizational model. Something that is increasingly being called DevSecOps. For this to work security must be constructed into the DevOps structure at a foundational level.
2. Implement tight work completion and time tracking metrics for development and IT teams with the goal of accurately measuring production capacity that includes security as a foundational attribute. Focus projects in areas with identified technical debt — with the top priority being placed on security-related technical debt — so that development resources are appropriately applied to these areas. Recognize that unplanned for work related to systems and applications that are already in production is usually due to unresolved technical debt that would have better been addressed upstream. As already mentioned, DevOps automation will magnify inefficiencies related to technical debt, especially security-related technical debt. Shops with a long history of collecting this kind of technical debt will confront much greater operational obstacles when pursuing DevOps organizational models. If approached correctly though, DevOps can become one of the best ways to overcome security-related technical debt – but only if it is approached and planned for as a primary business objective when measuring production capacity.
3. Implement strong IT project proposal and acceptance procedures that incorporate agreed upon project scoping criteria and includes QA throughout the project lifecycle. QA procedures should include security as a priority, including risk assessments of scoped projects prior to any development work is launched, and the establishment of security testing procedures, that includes security practitioners working alongside engineers throughout the process, not just at the beginning and end. This will strengthen the bonds between security and engineering staffs, and assist them in better understanding the pros and cons of the considerations and decisions that impact both. Any QA testing – security related, or otherwise – that can be automated, should be.
4. Emphasize identifying where production bottlenecks exist, be prepared to take action that will decrease them, and thereby increase the throughput of the IT system. This is especially important when factoring in security personnel and their availability at key points, so to minimize the potential of security unnecessarily slowing down the process. In a DevOps model, security should play a bigger role throughout the process, so it is important to have appropriate security staff in place to ensure capacity will be met. Identify any areas of needless work that may be contributing to the bottlenecks, and remove them. This will require the accurate definition of business objectives and priorities (number 1), so that any items that are not matched with these objectives and priorities can be identified and removed. Focus improvements on the bottlenecks, and not on areas before or after them – as these areas will not increase throughput, and instead will only increase inefficiency. This is a great opportunity to ensure that security is not a bottleneck and that security considerations and checkpoints are appropriately placed throughout the process.
5. Introduce QA procedures that are tightly aligned with work completion and time tracking metrics (number 2), and IT project proposal and acceptance procedures (number 3), in a way that incorporate security testing throughout the SDLC. This is an area where automation will be needed in order to limit operational variables that negatively impact security. This requires that things like automated version control be implemented, hardened base-line server configurations for development, testing, and production environments established (and are maintained and synced), and that automation systems (Puppet, Chef, MSCCM, etc.) be implemented for consistent deployment across environments. Through automation, environmental variables will be reduced, so that the testing and verification of security-related developments can be addressed in the most streamlined fashion possible.

By focusing attention in these areas, organizations stand to benefit from DevOps in a way that will drive security throughout the enterprise. Efforts already made (or in progress) in these areas will be of benefit when adopting a DevOps organizational model. Those organizations that are looking for quick wins by implementing DevOps with the main goal of increasing speed at the expense of other considerations should be warned, the automation brought on by DevOps will only magnify security weaknesses, and thereby increase overall exposure to cybersecurity risk. Always remember: just as luck favors the well-prepared mind, DevOps favors the well-prepared organization. Especially when it comes to security.