Patch Management: taming the beast

David Ianetta

Steven King said, “Nothing is so frightening as what’s behind the closed door.”

As we all know, he makes a very comfortable living out of writing scary and suspenseful tales. Other than being a very talented writer, King understands something about fear.

HAndsWe fear what we do not know. The monster is always the most terrifying when it stays in the imagination, unseen to the eye, behind the closed door. This sums up all the anxiety and stress around Patch Management.

It is late at night and the only light in the room radiates from the computer screen. A user is working, trying to meet a deadline.

Suddenly there is a pop up, Java Update Available! Like Jack Nicolson in the Shinning, hacking a hole in the door…the update has shown its ugly face, “here’s Java!”  Another unexpected update demanding attention and a choice. For the end user, it’s a frightening choice that leads into the unknown.

The end user gasps…

Is this a real update? Is this a virus or piece of malware? If it is real, will it slow down my machine as it installs? Maybe I should install it…but then, maybe it will break something…I’ll just put it off.

And put it off… and put it off.

Eventually, when left to the user, a machine can be years behind in updates. And that makes it vulnerable to cybercrime.

The cure can be worse than the disease
Administrators are not exempt from this stress. We know one wrong update on a production server and we are on hold with Microsoft for the next decade. It feels that way anyway… “press or say, TWO… I’m sorry, I did not understand you…press or say…”

Obviously, putting off updates is not the answer, nor is blindly installing them all. We know all too well the issues that can occur even when a patch is installed under perfect conditions.

There are too many instances of the cure being worse than the disease. When a patch is released there is a strong possibility that it can cause very negative consequences. For example, check out KB2982791 (https://technet.microsoft.com/library/security/ms14-045).

  • • V1.0 (August 12, 2014): Bulletin published.
  • • V2.0 (August 15, 2014): Bulletin revised to remove Download Center links for Microsoft security update 2982791. Microsoft recommends that customers uninstall this update.

Yes, you read that right. Install the update, then three days later Microsoft is telling you to uninstall it. This happens, all the time.

I’m glad they don’t make motorcycles… anyway…

This is just one example. What about the times where perfectly functioning updates are not compatible with third-party software?

So what is the answer? We cannot install every update automatically, and we cannot ignore them?

Planning ahead… your knight in shining armour
Data, knowledge, a plan. Open the door, look at the monster, study it and create a plan.

monsterI’m going to share some tips of the trade. If you are reading this blog, chances are you already know how important these patches are. Not installing them is not an option. Let’s move forward together and face the monster head on. Unless of course it really is Jack with the axe… if that's the case you’re on your own…just sayin’.

Anyway, back to the plan.

The accepted best practices for installing updates are the following:

  1. Scan for missing updates (Microsoft and Third Party)
  2. Research the updates, read the KB articles and determine if they apply to your environment.
  3. Have a test environment. This should be machines with the exact setup as your production network. Or choose a few machines with user who can afford to be down if something goes wrong.
  4. Install the updates on this test environment, Reboot the machines, test the business critical software to make sure everything is working.
  5. Push the patches out to the rest of the production environment. If possible, in stages. Make sure everything is working.

That really is it. The monster is not as large at it seems. Following these practices will reduce the stress related to patching.

A few extra tips…
Some other tips would be:

1. Get completely up to date on all applicable patches. When you first begin working on Patch Management, scan your machines and install the oldest patches first. Do this in stages until you are completely up to date. After that you will be amazed at how few patches there really are to install when you actually stay on top of the process.

2. Know the patch cycle. Microsoft releases patches on Patch Tuesday, the second Tuesday of every month. Mark this on your calendar, know it is coming up. Oracle (Java) updates are less frequent etc (four times a year). Know what third-party updates apply to your network and locate the message boards for these updates.

3. Read other administrator blogs and patch update summaries. There is no need to re invent the wheel here. Many are out there testing the new patches and blogging about it. A quick google search can help you come up with some great resources to follow.

For Microsoft, start here for Security Bulletin Summaries.

4. Do not allow users to have say over if or when patches will be installed on their machines. If you do, they will never be installed. Setup a weekly time for remote users to know they need to leave their machines on, and that you will be installing patches and rebooting their machines. If some stragglers don’t leave the machines on, have a back up plan.

5. Reboot weekly. Setup a regular schedule of rebooting servers and workstations regardless of whether or not patches have been installed. This will help catch up the stragglers.

I cannot stress enough how critical Patch Management is to keeping your network safe and sound.

Time to face your fears and move forward.