Patch Management philosophy and procedures
Whether you practice break/fix or managed services, it is good form to advocate maintaining client computers professionally. And while I prefer automated, perpetual maintenance, this approach is not practical for everyone. Let's look at three aspects of patch management:
- Manual updates
- Automated updates
Patch Management Philosophy
I know some people think "philosophy" seems out of place in the world of technology. But we all have philosophies that guide us. For example, when do you sell a new operating system? It used to be common wisdom that you don't install a new version of Windows until Service Pack 1 is released.
Such an approach hasn’t been true for a long time: For more than 10 years now, Microsoft has had massive public beta programs that put their OS in the hands of millions of users for many months before the release date.
So the release version is very stable.
Service packs are another story. Here, too, Microsoft has improved dramatically. But there's still good reason to wait: Firstly, service packs are 98% just a collection of previously released fixes. So a well-patched machine will have the critical elements already. As a result, modern service packs provide a certain level of consistency and stability, but their deployment is not critical.
For non-Microsoft service packs and patches, we're more cautious. Because we don't see them as often or as consistently, we don't know how reliable or safe they are. As a result, we are very careful with installs and very watchful when we perform them.
If you support a specific application, you may know its quirks and patterns. If so, you may have a philosophy for that application. If not, then fall back to the more generic (careful) philosophy for lesser-known application updates.
Beware of the Zero Day Attack…
One final example has emerged in recent years: The Zero Day Attack. These are viruses or attacks that become widely known on the same day. There is no time (zero days) to prepare for the attack.
Sound like there's nothing you can do? Not true. As long as you talk about WHAT you will do when the day comes, you can have some level of preparedness.
In this case, the Standard Operating Procedures (SOP) involves having pre-defined lines of command and communication. When a zero day attack happens, who will decide how your company responds? How will you communicate with the team? How will you communicate with the clients?
As I said, we all have these philosophies. And every technician you hire has a set of philosophies as well. But each company needs to operate on one philosophy. If you're the owner, or service manager, then YOUR philosophy is the one that matters. You need to write it out and educate the rest of the team.
When major service packs or patches come out, you need to orchestrate a plan for installing them. This probably means creating one service request for each client server or one service request for each client.
You might schedule these for the same time as the monthly maintenance, or maybe assign the whole lot to one or two technicians.
The key to success is that you:
- Have a philosophy and;
- Propagate it to your staff.
This is best accomplished by writing out your philosophies and putting them in your SOP document/folder. After that, you need to have a training session, talk this out with your staff, and agree on how you'll proceed.
Manual Patch Management
Don’t have a Remote Monitoring and Management (RMM) system? If this is the case you’ll be installing these updates "by hand." Alternatively, one of your philosophies may be to enable automatic updates from Microsoft. But you still need to apply updates to Office applications. Many non-critical updates won't be applied by automatic updates. And, of course, you need to apply updates to all those non-Microsoft products as well.
If you don't have an RMM system, you will have to create a schedule for updates.
One possible policy is to apply updates whenever you find yourself logged into a client computer. That's not bad, but if you do this, make sure you keep a list of all updates for that client on the c:\!tech directory of their server, or in your own PSA system. That way, you can make sure that you do a thorough job each time.
The problem with this approach is that you might not touch every machine.
Therefore, you should have a policy to check updates on all machines on a regular basis. For servers, this is part of the regular monthly maintenance. Whether performed remotely or onsite, all updates will be applied at least once a month. For desktops, you will need to create a service request to update all machines. I recommend one service request per client (not one per machine). You will need to have a document for keeping track of which machines have been completed.
You should go through this process at least once a year. Ideally, you will do it once per quarter. You can see why an RMM tool can save you a LOT of time. These "sweeps" can be extremely time consuming if you manage a lot of machines. If you have an RMM tool, you can apply updates weekly or monthly with almost zero additional effort.
Automated Patch Management
Automated patch management is a life-saver for managed service providers. If you have a thousand desktops and 100 servers, patching them adequately would be a full time job. The more you can automate the better.
The key with automated patch management is to decide how much you want to be involved. You can simply pass things through (e.g. if Microsoft released it, install it). Or you can test each patch for yourself. Here's the process we follow:
- Patch is released by Microsoft (Tuesday)
- One business day later we look to see whether the patch has been recalled
- We deploy the patch to our internal servers/workstations
- Three to five business days later (assuming no problems), we deploy to client servers/workstations
If your RMM company has a patch approval feature, you could simply set up groups so that patches are deployed once they have been white listed. In this way, you simply rely on your RMM vendor to do the vetting for you.
For a variety of reasons, you might choose to blacklist some patches so they do not get installed. Sometimes this is applied throughout all your clients (for example, if there's a Microsoft patch that just causes problems). More commonly, you might have a specific customer's Line of Business (LOB) application that is incompatible with a specific patch or service pack.
Patches have three possible statuses: Pending Approval; Approved; and Denied.
When patches are denied, you need to be very certain that they do not get applied. That means you need to have strict policies in-house to make sure that everyone respects the "denied" status.
Are there are any final tips?
Desktops can always be fixed, and don't normally affect the entire company. Servers are much more critical: They should always be backed up before patches are applied. Whether manual or automatic, stop the process until you have a good backup. I know things almost never go wrong with "simple" patches, but they sometimes DO go wrong.
You need a way back without wasting a great deal of downtime.
As a general rule, the RMM companies’ patch approval process is probably good enough to rely on. But you know your clients and their computers better than anyone. You need to decide on a patch management level that makes sense and is comfortable for you.
You need to articulate that philosophy and make sure your staff understands it.
Three Take-Aways from this chapter:
- Don’t just let patch installations “happen.” Have a philosophy – your approach to how patches are installed. Write it down.
- Define your process. Whatever it is, implement it consistently.
- Create a blacklist policy. Even if you only have a few clients who need to blacklist patches, there might be a lot of work involved in ignoring this.
(Used with permission of Karl W. Palachuk, SmallBizThoughts.com)