#94 - Easier, Better, Faster, & Cheaper Software

Hello, and welcome to another episode of CISO Tradecraft, the podcast that provides you with the information, knowledge, and wisdom to be a more effective cybersecurity leader.  My name is G. Mark Hardy, and today we're going to try to balance the impossible equation of better, faster, and cheaper.  As always, please follow us on LinkedIn, and subscribe if you have not already done so.

Shigeo Shingo, who lived from 1909-1990, helped to improve efficiency at Toyota by teaching thousands of engineers the Toyota Production System, and even influenced the creation of Kaizen.  He wrote, "There are four purposes for improvement: easier, better, faster, cheaper. These four goals appear in order of priority."

Satya Nadella, the CEO of Microsoft, stated that, “Every company is a software company.  You have to start thinking and operating like a digital company.  It’s no longer just about procuring one solution and deploying one solution… It’s really you yourself thinking of your own future as a digital company, building out what we refer to as systems of intelligence.”

The first time I heard this I didn’t really fully understand it.  But after reflection it makes a ton of sense.  For example, let’s say your company couldn’t send email.  How much would that hurt the business?  What if your company couldn’t use Salesforce to look up customer information?  How might that impact future sales?  What if your core financial systems had database integrity issues?  Any of these examples would greatly impact most businesses.  So, getting high-quality software applications that enable the business is a huge win.

If every company is a software or digital company, then the CISO has a rare opportunity.  That is, we can create one of the largest competitive advantages for our businesses.

What if we could create an organization that builds software cheaper, faster, and better than all of our competitors?

Sounds good right?  That is the focus of today’s show, and we are going to teach you how to excel in creating a world class organization through a focused program in Secure Software Development.  Now if you like the sound of better, faster, cheaper, as most executives do, you might be thinking, where can I buy that?  Let's start at the back and work our way forward.

• We can make our software development costs cheaper by increasing productivity from developers.
• We can make our software development practices faster by increasing convenience and reducing waste.
• We can make our software better by increasing security.

Let’s first look at increasing productivity.  To increase productivity, we need to under    stand the Resistance Pyramid.  If you know how to change people and the culture within an organization, then you can significantly increase your productivity.  However, people and culture are difficult to change, and different people require different management approaches.

• At the bottom of the pyramid are people who are unknowing.  These individuals Don’t know what to do.  You can think of the interns in your company.  They just got to your company, but don't understand what practices and processes to follow.  If you want to change the interns, then you need to communicate what is best practice and what is expected from their performance.  Utilize an inquiry approach to decrease fear of not knowing, for example, "do you know to whom I should speak about such-and-such?" or "do you know how we do such-and-such here?"  An answer of "no" allows you to inform them of the missing knowledge in a conversational rather than a directional manner.
• The middle part of the pyramid is people who believe they are unable to adapt to change.  These are individuals that don’t know how to do the task at hand.  Here, communications are important, but also skills training.  Compare your team members here to an unskilled labor force -- they're willing to work but need an education to move forward.  If you give them that, then the unskilled can become skilled. However, if you never invest in them, then you will not increase your company’s productivity and lowers your costs.
• At the Top of the resistance pyramid are the people who are unwilling.  These individuals Don’t Want to Change.  We might call these folks the curmudgeons that say we tried it before, and it doesn’t work.  Or I’m too old to learn that.  If you want to change these individuals and the culture of an organization, then you need to create motivation.

As leaders, our focus to stimulate change will be to focus on communicating, educating, and motivating.  The first thing that we need to communicate is the Why.  Why is Secure Software Development important?  The answer is money.  There are a variety of studies that have found that when software vulnerabilities get detected in the early development processes, they are cheaper than later in the production phases.  Research from the Ponemon Institute in 2017 found that the average cost to address a defect in the development phase was $80, in the build phase was $240, in the QA/Test Phase was $960, and in the Production phase was $7,600.  Think of that difference.  $80 is about 1% of $7,600.  So if a developer finds bugs in the development code then they don’t just save their time, they save the time of second developer who doesn’t have to do a failed code review, they save the time of an infrastructure engineer who has to put the failed code on a server, they save the time of another tester who has to create regression tests which fail, they save the time of a wasted change approval board on a failed release, and they save the customer representatives time who will respond to customers when the software is detected as having issues.  As you see there’s a lot of time to be saved by increasing productivity, as well as a 99% cost savings for what has to be done anyway.  Saving their own time is something that will directly appeal to every development team member.

To do this we need to do something called Shift Left Testing.  The term shift left refers to finding vulnerabilities earlier in development.  To properly shift left we need to create two secure software development programs.

• The first program needs to focus on is the processes that an organization needs to follow to build software the right way.  This is something you have to build in house.  For example, think about how you want software to create a network diagram that architects can look at in your organization.  Think about the proper way to register an application into a Configuration Management Database so that there is a POC who can answer questions when an application is down.  Think about how a developer needs to get a DNS entry created for new websites.  Think about how someone needs to get a website into the various security scanning tools that your organization requires (SAST, DAST, Vuln Management, Container Scanning, etc.)  Think about how developers should retire servers at the end of life.  These practices are unique to your company.  They may require a help desk ticket to make something happen or if you don't have a ticketing system, an email.  We need to document all of these into one place where they can be communicated to the staff members who will be following the processes.  Then our employee has a checklist of activities they can follow.  Remember if it’s not in the checklist, then it won’t get done.  If it doesn’t get done, then bad security outcomes are more likely happen.  So, work with your architects and security gurus to document all of the required practices for Secure Software Development in your company.  You can place this knowledge into a Wikipedia article, a SharePoint site, a Confluence Page, or some kind of website.  Make sure to communicate this frequently.  For example, have the CIO or CISO share it at the IT All Hands meeting.  Send it out in monthly newsletters.  Refer to it in security discussions and architecture review boards.  The more it’s communicated the more unknowing employees will hear about it and change their behavior.
• The second program that you should consider building is a secure code training platform.  You can think of things such as Secure Code Warrior, HackEDU (now known as Security Journey), or Checkmarx Code Bashing.  These secure code training solutions are usually bought by organizations instead of being created in-house.  They teach developers how to write more secure code.  For example, "How do I write JavaScript code that validates user input, sanitizes database queries, and avoids risky program calls that could create vulnerabilities in an application?"  If developers gain an education in secure programming, then they are less likely to introduce vulnerabilities into their code.  Make these types of training programs available to every developer in your company.
• Lastly, we need to find a way to motivate the curmudgeons.  One way to do that is the following:
  • Let’s say you pick one secure coding platform and create an initial launch.  The first two hundred people in the organization that pass the secure developer training get a one-time bonus of $200.  This perk might get a lot of people interested in the platform.  You might even get 10-20% of your organization taking the training in the first quarter of the program.  The second quarter your organization announces that during performance reviews anyone who passed the secure software training will be viewed more favorable than their peers.  Guess what?  You will see more and more people taking the training class.  Perhaps you see that 50% of your developer population becomes certified.  Then the following year you say since so many developers are now certified, to achieve the rank of Senior Developer within the organization, it is now expected to pass this training.  It becomes something HR folks look for during promotion panels.  This gradual approach to move the ball in training can work and has been proven to increase the secure developer knowledgebase.
  • Here's a pro tip:  Be sure to create some kind of badges or digital certificates that employees can share.  You might even hand out stickers upon completion that developers can proudly place on their laptops.  Simple things like this can increase visibility.  They can also motivate people you didn't think would change.

Now that we have increased productivity from the two development programs (building software the right way and a secure code training platform), it’s time to increase convenience and reduce waste.  Do you know what developers hate?  Well, other than last-minute change requests.  They hate inefficiencies.  Imagine if you get a vulnerability that says you have a bug on line 242 in your code.  So you go to the code, and find there really isn’t a bug, it's just a false positive in the tool.  This false bug detection really, well, bugs developers.  So, when your organization picks a new SAST, DAST, or IAST tool, be sure to test the true and false positive rates of the tool.  One way to do this is to run the tools you are considering against the OWASP Benchmark.  (We have a link to the OWASP Benchmark in our show notes.)  The OWASP Benchmark allows companies to test tools against a deliberately vulnerable website with vulnerable code.  In reality, testing tools find both good code and bad code.  These results should be compared against the ground truth data to determine how many true/false positives were found.  For example, if the tool you choose has a 90% True Positive Rate and a 90% False Positive Rate then that means the tool pretty much reports everything is vulnerable.  This means valuable developer time is wasted and they will hate the tool despite its value.  If the tool has a 50% True Positive Rate and a 50% False positive rate, then the tool is essentially reporting randomly.  Once again, this results in lost developer confidence in the tool.  You really want tools that have high True Positive Rates and low False Positive Rates.  Optimize accordingly.

Another developer inefficiency is the amount of tools developers need to leverage.  If a developer has to log into multiple tools such as Checkmarx for SAST findings, Qualys for Vulnerability Management findings, Web Inspect for DAST findings, Prisma for Container Findings, Truffle Hog for Secrets scanning, it becomes a burden.  If ten systems require two minutes of logging in and setup each that's twenty minutes of unproductive time.  Multiply that time the number of developers in your organization and you can see just how much time is lost by your team just to get setup to perform security checks.  Let's provide convenience and make development faster.  We can do that by centralizing the security scanning results into one tool.  We recommend putting all the security findings into a Source Code Repository such as GitHub  or GitLab.  This allows a developer to log into GitHub every day and see code scanning vulnerabilities, dependency vulnerabilities, and secret findings in one place.  This means that they are more likely to make those fixes since they actually see them.  You can provide this type of view to developers by buying tools such as GitHub Advanced Security.  Now this won’t provide all of your security tools in one place by itself.  You still might need to show container or cloud findings which are not in GitHub Advanced Security.  But this is where you can leverage your Source Code Repository’s native CI/CD tooling.  GitHub has Actions and GitLab has Runners.  With this CI/CD function developers don’t need to go to Jenkins and other security tools.  They can use a GitHub Actions to integrate Container and Cloud findings from a tool like Prisma.  This means that developers have even fewer tools from CI/CD perspectives as well less logging into security tools.  Therefore, convenience improves.  Now look at it from a longer perspective.  If we get all of our developers integrating with these tools in one place, then we can look in our GitHub repositories to determine what vulnerabilities a new software release will introduce.  This could be reviewed at Change Approval Board.  You could also fast track developer who are coding securely.  If a developer has zero findings observed in GitHub, then that code can be auto approved for the Change Approval.  However, if you have high/critical findings then you need manager approvals first.  These approvals can be codified using GitHub code scanning, which has subsumed the tool Looks Good To Me (LGTM), which stopped accepting new user sign-ups last week (31 August 2022).  This process can be streamlined into DevSecOps pipelines that improve speed and convenience when folks can skip change approval meetings.

Another key way we can make software faster is by performing value stream mapping exercises.  Here’s an example of how that reduces waste.  Let’s say from the time Nessus finds a vulnerability there’s actually fifteen steps that need to occur within an organization to fix the vulnerability.  For example, the vulnerability needs to be assigned to the right team, the team needs to look at the vulnerability to confirm it’s a legitimate finding, a patch needs to be available, a patch needs to be tested, a change window needs to be available, etc.  Each of these fifteen steps take time and often require different handoffs between teams.  These activities often mean that things sit in queues.  This can result in waste and inefficiencies.  Have your team meet with the various stakeholders and identify two time durations.  One is the best-case time for how long something should go through in an optimal process.  The second is the average time it takes things to go through in the current process.  At the end of it you might see that the optimal case is that it takes twenty days to complete the fifteen activities whereas the average case takes ninety days.  This insight can show you where you are inefficient.  You can identify ways to speed up from ninety to twenty days.  If you can do this faster, then developer time is gained.  Now, developers don’t have to wait for things to happen.  Making it convenient and less wasteful through value stream mapping exercises allows your teams to deploy faster, patch faster, and perform faster.

OK last but not least is making software better by increasing security.   At the end of the day, there are many software activities that we do which provide zero value to the business.  For example, patching operating systems on servers does not increase sales.  What makes the sales team sell more products?  The answer is more features on a website such as product recommendations, more analysis of the data to better target consumers, and more recommendations from the reporting to identify better widgets to sell.  Now, I know you are thinking, did CISO Tradecraft just say to not patch your operating systems?  No, we did not.  We are saying patching operating systems is not a value-add exercise.  Here’s what we do recommend.  Ask every development team to identify what ike patching.  Systems that have a plethora of maintenance activities are wasteful and should be shortlisted for replacement.  You know the ones: solutions still running via on-premises VMWare software, software needing monthly java patching, and software if the wind blows the wrong way you have an unknown error.  These systems are ripe for replacement.  It can also be a compelling sell to executives.  For example, imagine going to the CIO and CEO of Acme corporation.  You highlight the Acme app is run by a staff of ten developers which fully loaded cost us about $250K each.  Therefore, developing, debugging, and maintaining that app costs our organization roughly $2,500,000 in developer time alone plus hosting fees.  You have analyzed this application and found that roughly 80% of the time, or $2,000,000, is spent on maintenance activities such as patching. You believe if the team were to rewrite the application in a modern programming language using a serverless technology approach the team could lower maintenance activities from 80% to 30%.  This means that the maintenance costs would decrease from $2 million to $750K each year.  Therefore, you can build a financial case that leadership fund a $1.25 million initiative to rewrite the application in a more supportable language and environment, which will pay for itself at the end of the second year.  No, I didn't get my math wrong -- don't forget that you're still paying the old costs while developing the new system.)

Now if you just did a lift and shift to AWS and ran the servers on EC-2 or ECS, then you still have to patch the instance operating systems, middle ware, and software -- all of which is a non-value add.  This means that you won’t reduce the maintenance activities from 80% to 30%.  Don't waste developer time on these expensive transition activities; you're not going to come out ahead.  Now let’s instead look at how to make that maintenance go away by switching to a serverless approach.  Imagine if the organization rewrote the VMware application to run on either:

• A third party hosted SaaS platform such as Salesforce or Office 365

or

• A serverless AWS application consisting of Amazon S3 buckets to handle front-end code, an Amazon API Gateway to make REST API calls to endpoints, AWS Lambda to run code to retrieve information from a Database, and Dynamo DB to store data by the application

This new software shift to a serverless architecture means you no longer have to worry about patching operating systems or middleware.  It also means developers don’t spend time fixing misconfigurations and vulnerabilities at the operating system or middleware level.  This means you made the software more secure and gave the developers more time to write new software features which can impact the business profitability.  This serverless approach truly is better and more secure.  There’s a great story from Capital One you can look up in our show notes that discusses how they moved from EC-2 Servers to Lambda for their Credit Offers Application Interface.  The executive summary states that the switch to serverless resulted in 70% performance gains, 90% cost savings, and increased team velocity by 30% since time was not spent patching, fixing, and taking care of servers.  Capital One uses this newfound developer time to innovate, create, and expand on business requirements.  So, if you want to make cheaper, faster, and better software, then focus on reducing maintenance activities that don’t add value to the business.

Let's recap.  World class CISOs create a world class software development organization.  They do this by focusing on cheaper, faster, and better software. To perform this function CISOs increase productivity from developers by creating documentation that teaches developers how to build software the right way as well as creating a training program that promotes secure coding practices.  World Class CISOs increase the convenience to developers by bringing high-confidence vulnerability lists to developers which means time savings in not weeding out false positives.  Developers live in Source Code Repositories such as GitHub or GitLab, not the ten different software security tools that security organizations police.  World Class CISOs remove waste by performing value stream exercises to lean out processes and make it easier for developers to be more efficient.  Finally, World Class CISOs make software better by changing the legacy architecture with expensive maintenance activities to something that is a winnable game.  These CISOs partner with the business to focus on finding systems that when re-architected to become serverless increase performance gains, promote cost savings, and increase developer velocity.

We appreciate your time listening to today’s episode.  If this sparks a new idea in your head. please write it down, share it on LinkedIn and tag CISO Tradecraft in the comment.  We would love to see how you are taking these cyber lessons into your organization to make better software for all of us.

Thanks again for listening to CISO Tradecraft.  This is G. Mark Hardy, and until next time, stay safe out there.

References

• https://www.sixsigmadaily.com/who-was-shigeo-shingo-and-why-is-he-important-to-process-improvement/

• https://news.microsoft.com/speeches/satya-nadella-and-chris-capossela-envision-2016/ 

• Galpin, T.J. (1996).  The Human Side of Change: A Practical Guide to Organization Redesign.  Jossey-Bass 

• https://www.businesscoaching.co.uk/news/blog/how-to-break-down-barriers-to-change 

• Ponemon Institute and IBM. (2017) The State of Vulnerability Management in the Cloud and On-Premises 

• https://www.bmc.com/blogs/what-is-shift-left-shift-left-testing-explained/ 

• https://www.securecodewarrior.com/ 

• https://www.securityjourney.com/ 

• https://checkmarx.com/product/codebashing-secure-code-training/ 

• https://owasp.org/www-project-benchmark/ 

• https://docs.github.com/en/get-started/learning-about-github/about-github-advanced-security 

• https://medium.com/capital-one-tech/a-serverless-and-go-journey-credit-offers-api-74ef1f9fde7f