This episode of Breakfast Bytes is Part 2 of a series where Felicia King and Dan Moyer of QPC Security continue their conversation on Vulnerability Management. Listen to Part 1 at https://qpcsecurity.podbean.com/e/vulnerability-management-part-1/.
In today’s episode, Felicia and Dan discuss vulnerability management workflows, supply chain risk management, starting with security on the front end rather than retrofitting, and proper patch management.
Workflow management
01:10
This episode of Breakfast Bytes is Part 2 of a series where Felicia King and Dan Moyer of QPC Security continue their conversation on Vulnerability Management. Listen to Part 1 at https://qpcsecurity.podbean.com/e/vulnerability-management-part-1/.
In today’s episode, Felicia and Dan discuss vulnerability management workflows, supply chain risk management, starting with security on the front end rather than retrofitting, and proper patch management.
Workflow management
01:10 CISO-related (Chief Information Security Officer) workflows are at the core of what is today’s necessity, and we will only see it become more mandatory within the next couple of years. Organizations that do not have vulnerability management workflows in place in a comprehensive way are going to find they have too much technical debt, deferred maintenance, or deferred security to be able to dig themselves out. This won’t be from a lack of money either, but a lack of manpower and time in the day to rectify the issue.
Supply chain risk management
02:43 SaaS vendors have vulnerabilities and very few of them have in their contracts your rights and their obligations. What kind of questions should you be asking your SaaS vendors that in many cases you are responsible for as an organization? Here are just a few:
- Do they have continuous vulnerability management scanning going on with regards to their SaaS platform?
- How are they classifying vulnerabilities?
- How quickly are they going to resolve vulnerabilities?
- How are they communicating these issues to you?
- Do they use API security scanning?
- How do they adhere to OWASP API standards and best practices?
- What are they doing for you in terms of supply chain risk management or software bill of materials?
Your organization’s CISO or vCISO should be in your court getting answers to these questions if they are not being addressed by your SaaS vendor or addressed in your contract. Having a proactive, highly functional, highly communicative, and open, honest working relationship with your CISO will ensure you have the protections your organization needs.
Proper patch management
04:51 Let's walk through an example of patch management in an environment with Hyper-V hosts, Dell PowerEdge server, domain controllers, business critical SQL servers with essential business applications, virtual machines, remote sites, on-site and offsite backups, hardware at different speeds, and then all these third-party software on these workloads – how do you patch all these things?
06:11 It is exceptionally important to note that some patches will step on or over each other, be required to be put in place and rebooted first, and then other patches applied on top of it. The time it takes to patch a server can be exacerbated by trying to accomplish, say, five patches in one changewindow rather than one patch/reboot followed by another one patch/reboot, and so on.
07:48 Watching the servers reboot is an important piece to verify the workload comes back up reiterating the point made in Part 1 of this series that adequate patch management of an entire server for $50/month cannot be done.
Domain controllers
09:19 There tends to be multiple domain controllers or, in the case of just one, it has been designed so that it can reboot whenever it needs to allow for patching. The domain controller is the brain of everything, and since it can reboot whenever needed to apply patches, it can facilitate that while staying available when everything else comes back up.
Typically we will start with domain controllers as the first thing patched and verified. Now if there are multiple, and depending on how critical the environment is, a rolling out patch might be done so that these secondary domain controllers or ones that are not on the best hardware are patched and then they sit for a period.
Backup plans and backstops
11:29 Part of that patching methodology is your backup plans and backstops – having the tools and everything else in place to uninstall a patch if needed. When we set up our servers, we always have Command Prompt and PowerShell already queued up on those devices when we log in. Then we have the availability of pre-planned scripts that we can adjust as we go but most importantly, all the tools are there and available.
Importance of roles on servers
12:25 Part of your ability to have resiliency in the environment is the ability to reboot whenever you need, because you have redundancy and resiliency. Because it is a single role server, it gives you that agility to be able to resolve and prevent issues.
Therefore, workload design is the name of the game. Whatever you think that cost is of that additional virtual machine, that is nothing compared to the problems that you cannot solve because you tried to shove a bunch of stuff together in workloads that did not meet because they were mismatched workloads.
Many patch managers are not comprehensive and there is a lack of consistency in of what is getting patched on a well-designed domain controller versus a third-party party application server.
Physical servers
16:09 Watching a virtual machine reboot while maintaining efficiency and not biting off more than one can chew is crucial, but we are also finding is increasingly important to watch the physical servers and that can only be possible with the right hardware.
How are you auditing and confirming that patches are being applied and which ones have not? At QPC Security, we bring all the virtual machines down and reboot the host as a prerequisite for patching because it gives you a clean slate to start your patches. Then we will use the patching methodology to push specific patches down to it. We use our patching piece to push specific ones because not everything is needed for hosts and other pieces that we have identified will cause an issue, is a multi-patch, or a multi-patch/multi-reboot process.
Taking one step at a time, pull it down, apply patches, make sure everything is happy coming up. Go through that entire process again. While we are connected to iDRAC, we watch the server, reboot, apply patches, come back up, make sure all the VM's are checking in properly, we are making sure everything is available, then they go through that process two to three times. It depends on how many patches are available and what things got pushed out.
Everything has patches
20:39 If you have a hypervisor that is not giving you patches; you should not be using them. Likewise, if there is no product improvement then there is no security management from that vendor. There is no easy button or a set it and forget it.
21:42 When IT is not confident in how a process is going to work, they do not want to touch it and that is exactly where a vulnerability arises. Say a consultant installs Cisco, but without a brand expert or budget in place keep the consultant to maintain it, it remains unpatched and therefore vulnerable. That is precisely why organizations need to have a business continuity and disaster recovery (BCDR) plan in place and a procurement policy that drives effective vulnerability management.
Incremental patching
25:26 When people are too afraid to patch the hardware, it does not get patch which accumulates over time in terms of technical debt and the technical issues it accumulates. Attempting to patch too many patches at once or jump too many versions results in the reboot cycle of death or a very time-consuming reboot because you are not running a vetted, tested, and supported configuration. The more time and versions you allow to pass between patches, the more divergent from manufacturer’s tested config those updates become.
Buying the right hardware to begin with saves you money down the line
33:20 A crucial piece to vulnerability management in your workstations is BIOS, drivers, firmware. If you buy the right hardware to begin with that has the automation engine built into it and when you deploy it you are configuring it accordingly, it becomes far less expensive than paying a human being to manually babysit your vulnerability management.
Not all workloads are created equal
34:59 A word of caution when an IT service provider quotes patch management for your organization. When it comes to patching business line apps that need high uptimes because it costs a business thousands of dollars per hour to be down, what patches does the ITSP apply and with what preparation for back out plan?
In many cases, an ITSP is giving a client the perception of patch management, certainly not vulnerability management, but in reality they are simply doing a Windows update and only some third-party patching, which might only be five third party applications. At QPC Security, our catalog of patches of over 9500 software titles that we are patching and there is no automation. Visit https://www.ivanti.com/partners/ivanti-software-catalog to learn more about the normalization of software titles.
Cybersecurity insurance applications require continuous vulnerability assessment and vulnerability management. However, most IT service providers do not offer comprehensive patch management. Their vulnerability management claims are grossly misrepresented to the point of malfeasance.
Vendor documentation & software bill of materials
37:43 You cannot keep your head in the sand – all these things must be considered when receiving a quote from an IT service provider.
In cases when the software vendor is not offering competent documentation, your organization must rely on the legwork of your IT service provider to offer timely patches at opportune times. Do not forget that many ITSPs will charge you to run patches on the weekend or evenings when there will be minimal impact to your business.
"Titrics"
43:02 Your ITSP should have vetted and tested procedures and protocols for implementing patches, yet all too many do not. So many times, we see the priority of IT companies are how quickly they can close a ticket and rely on the software companies to do it for them. This focus on first-call closures and ticket metrics (termed here as “titrics”) is grossly underserving their clients and their clients’ organizations. Proper documentation allows for better time management and to offer effective support to best serve the needs of the clients without requiring the assistance of the third-party software vendor.
47:05 Gaps in change management, change control, and documentation for server workloads arise when an ITSP is focused on ticket-based productivity rather than quality of service. The original scope of the project by the ITSP requires evaluation from someone who can accurately evaluate the needs of the client’s organization. When the bid is too low, the needs of the client are not going to be met, the work will not be completed, and the organization is left vulnerable.
50:03 Unfortunately, an incompetent ITSP will leave out what services they had to cut out on the race to the bottom of the pricing model and that leaves it up to you, as the business owner, to be aware of your organization’s cybersecurity insurance policy requirements and how they are being fulfilled.
Questions? Reach out to us
QPC Security proudly serves businesses with virtual CISO services for our clients. If you are interested in learning more about how QPC Security can serve the needs of your organization please visit https://www.qpcsecurity.com/ or call one of our experts directly on (262) 553-6510.
Stay up to date on the most recent episode of Breakfast Bytes by following the podcast on Podbean at https://qpcsecurity.podbean.com/.
Learn more: https://www.complianceforge.com/faq/word-crimes/policy-vs-standard-vs-control-vs-procedure
