Vulnerability management (VM) is the process of discovering and remediating system vulnerabilities with the purpose of keeping information security risks at acceptable level. All IT systems have vulnerabilities. A vulnerability is a weakness which allows an attacker to reduce the confidentiality, integrity and/or availability of information. We still remember the major vulnerability discovered this year in OpenSSL, Heartbleed, or another big one from a few years ago – DNS cache poisoning affecting many DNS implementations.
Vulnerabilities exist in operating systems, applications, databases and network devices. Each month a number of new vulnerabilities are being published by vendors and independent researchers, so it is critical to discover and remediate them according to the risks they introduce. In this blog post I am sharing the details of the ThousandEyes VM process.
At ThousandEyes we automate a VM process using a service called QualysGuard by Qualys. The process includes the following sub-processes (or phases of VM):
- Network mapping
- Asset classification
- Network scanning
- Vulnerability prioritization
- Vulnerability remediation
- Closure of vulnerability
Network mapping is an essential step in discovering vulnerabilities and consists of enumeration of all IP addresses in registered networks in an attempt to find live hosts. Network mapping is implemented using QualysGuard Vulnerability Management Scans – Maps. Mapping is implemented as a weekly task, one day prior to scanning.
All discovered hosts are entered into the QualysGuard asset inventory. Refer to the next section to understand how we manage assets.
All technology assets have a criticality level assigned to them. At ThousandEyes we use a very simple and effective approach – asset classification by location within the network. QualysGuard uses Asset Groups to manage assets. We have one Asset Group created for each subnet. Refer to the table below for an example of asset management (IP addresses are provided here as an example).
|1||208.185.7.x||5 – most critical||Permiter network|
|6||192.168.20.x||1 – least critical||Internal lab|
You can apply different strategies to classify the Asset Groups by their criticality; here I am using an example of an exposure-driven approach. You can also select data-driven approach or a combination of these two. Based on the table above, a system located in 10.20.10.x subnet will be assigned criticality level 2, a system located in 10.10.10.x subnet will be assigned criticality level 4, etc.
Based on the results of mapping, Information Security operations personnel add newly-discovered systems into appropriate Asset Groups for upcoming scans.
We have implemented two types of vulnerability scans on a weekly basis – external (using the Qualys cloud scanner) and internal (using an internal QualysGuard appliance). The external scan shows us our exposure to the bad guys on the internet and are being used for general reporting only. To maximize results and improve their quality, internal scans run as an anonymous process and as an authenticated process, against all defined Asset Groups. Results of the scans feed into the Qualys remediation module for prioritization and ticket creation.
Not all Qualys users are aware of this, but Qualys comes with a powerful policy-based ticketing module called Remediation (exists in Express Suite and above) at no additional cost.
To take full advantage of the Qualys Remediation module, you have to document your risk management policies related to vulnerabilities and set your remediation targets (operating level agreement, or OLA) with those who will fix the vulnerability. Feel free to reuse what we have put in place at ThousandEyes.
Each discovered vulnerability has a risk level associated with it. These risk levels are calculated using a combination of Criticality of an asset (1 through 5, see table 1) and a severity assigned by QualysGuard (1 through 5, according to Qualys Severity Levels). See the table below for detailed information about the risks associated with vulnerabilities:
Our Information Security standards set the following remediation targets:
|No.||Risk Level||Risk Rating||Remediation Target|
|1||HighRisk vulnerabilities||25 – 100||30 days|
|2||MediumRisk vulnerabilities||10 – 25||60 days|
|3||LowRisk vulnerabilities||5 – 10||90 days|
|4||SlightRisk vulnerabilities||1 – 5||120 days|
Tables 2 and 3 are documented in the Policies section of the Qualys Remediation module. It is important to keep in mind that if you add new Asset Group, you need to update the policies.
I have a small trick for ease of reporting: each vulnerability gets a risk level assigned as the user name. As you can see from Figure 5, I have created 4 users and called them 4 High Risk, 3 Medium Risk, 2 Low Risk, and 1 Slight Risk. Let’s look at what’s inside those policies.
This is the policy “High Risk – Asset 5 Severity 3 4 5”. It has the Asset Group with criticality level 5 from Table 1 (208,185.7.x). According to Table 2, vulnerabilities with severity levels 3, 4, and 5 in the “Criticality 5” asset group will result in HIGH RISK vulnerability. The remediation target for those is 30 days. Qualys also assigns this vulnerability to the user 4 High Risk (see Figure 5), so we can clearly identify this vulnerability as high risk.
Essentially, to document your policies in Qualys you have to go through your risk calculation table line by line and enter all data in the remediation policies. For example, to fully document table 2 you’ll need to create 17 policies in Qualys.
What we have done so far was only the preparation, and now the real work starts. After the first scan you’ll have the list of prioritized vulnerabilities with due dates defined and risks associated. Go under “Tickets” in the Remediation module of VM to see the list. All resource administrators work off this prioritized list to investigate and remediate vulnerabilities.
I will not be sharing any screenshots of our vulnerabilities; however, I will share the metrics I use for reporting. As part of the Security operations metrics, each month our CEO is presented the following data:
- Number of vulnerabilities remediated last month
- New vulnerabilities discovered last month
- Currently open vulnerabilities
- Vulnerabilities in violation of OLA (expected to be none)
Figure 7 shows our reporting screenshot related to vulnerability management. Feel free to reuse it. The same metrics are provided for the past 12 months.
- * Weighted is calculated based on risk ratings as:
(100 x High) + (25 x Medium) + (10 x Low) + (5 x Slight)
- ** Absolute is calculated as:
High + Medium + Low + Slight
Vulnerability tickets will be closed by QualysGuard automatically when it no longer detects them as open. If a vulnerability ticket was opened by an authenticated scan, it can only be confirmed as closed by an authenticated scan, and the same applies to anonymous scans. We do not close tickets manually (unless there is an exception); everything is done by QualysGuard automatically.
Suggestions to Qualys Team
Below are the items which would improve my experience with the Qualys service and save even more of my time:
- Elevate Remediation module in the solution architecture, so it can cover both VM and Web application scanning vulnerabilities.
- In addition to severity of vulnerability, add a risk to the ticket (though could be back calculated as a combination of severity and timeframe between due date and created date, ticket OLA). IN our standard they are High, Medium, Low, Slight.
- Add reporting in time as described in Remediation section.
An effective vulnerability management process is a must in today’s fast-changing environment. The same vulnerability in a production database server and in the IT lab most probably poses different levels of risk and must be treated with different priority. There are tools available to track service exposure and criticality of assets. If you already use QualysGuard for scanning, you can implement an efficient process by maximizing its use.
All other trademarks are the property of their respective owners.