Logging and Monitoring Best Practices

Your audit or event log, the document that records significant events in your IT system, can be an invaluable resource in understanding your network—as long as you follow best practices for logging and monitoring. When you’re experiencing a capacity issue, dealing with a cybersecurity threat, or just want to seek out better, more efficient ways of allocating resources and managing your systems, a properly maintained audit log can provide all the answers you need.

The challenge for IT professionals is dealing with a massive trove of different logs that can be about as disparate and distributed as the systems themselves. Maintaining and archiving all this data for years at a time can ironically contribute to the very capacity issues that IT pros would want to use audit logs to solve. Without a clear idea of exactly what information you’re supposed to be storing, you may find that your log data doesn’t give you the insight you need once the time comes to analyze it.

These problems can be easily overcome with a primer on how to create, manage, and analyze audit logs. In this piece, we’ll cover the basics of event log management, explain some logging best practices, and answer some frequently asked questions about effective log retention, audit log security, and log storage management.

What information should my audit logs feature?

Any event in an IT system can be included in an audit log. The needs and risks associated with every application and server instance will differ greatly, and it’s these factors that should determine what information you are continually logging and analyzing. However, it’s safe to say that in almost all scenarios, your audit logs should feature these elements:

• User ID
• Terminal identity
• Log on and log off time and date
• Systems, data, applications, files, and networks accessed
• Failed attempts to access systems, data, applications, files, and networks
• Changes to system configurations and use of system utilities
• Alarms and other security events
• Activity from cybersecurity tools like the firewall or antivirus software

How long should audit logs be kept?

The amount of time you should archive event logs depends on the type of log you’re keeping. Your client or organization may have particular requirements and recommendations regarding audit logging, and most forms of logging are subject to regulation. However, if you remain unsure as to how long you should be keeping a given audit log, logging best practices suggest keeping everything for at least one year.

When setting a length of time with event log management, it may help to remember that distributed IT networks have significantly changed the practice of audit logging and monitoring. A large enterprise network can feature thousands of server instances or containers, and each of those instances and containers is constantly generating audit logs. As a result, more log data is being created today than ever before. These volumes of data are so massive that companies may reasonably struggle with the question of how to manage and store it all, let alone how to regularly comb through it for important insights about their network’s security and performance. In order for an enterprise to successfully store all this data for months or a full year, it’s often logistically necessary to use a cloud-based, managed services solution.

What is application logging? 

Just like with any other component of your network, activity within your applications needs to be regularly saved and analyzed. One major difference between an application log and other event logs in your IT system, however, is that the log’s format and content are determined by the application rather than your operating system. This is to say, unless you’re developing the application yourself you have little control over what information is featured in log files.

An application normally contains code to write various types of events to an application log file. The log file can reveal message flow issues and application problems. It can also contain information about user and system actions that have occurred. Logged events typically include the following:

• Application exceptions
• Major events like startups, stops, and restarts, as well as security events.
• Error events that prevent the application from starting
• Some debug information
• SQL logs

What is audit logs security?

In cybersecurity, we have a number of active protection measures we can take, including antivirus software, some form of user authentication, and firewalls. These tools are at the disposal of network security specialists to prevent unauthorized users or users with malicious intent from stealing or destroying assets within that network, while protecting those who are authorized to use those networks. But what happens when, despite all these measures, an attack occurs? Security professionals can turn to their event logs to search for answers.

When security breaks down and your application or network is compromised, event logging and monitoring can notify you that a problem exists as well as where the breach has occurred, enabling you to stop or limit the damage. It can also help you understand the vulnerabilities that have been exploited by an outside threat so you can attempt to recover or protect that data—or at least do what’s necessary to avoid similar breaches in the future.

But the mere presence of audit logs isn’t sufficient to protect you from cyberattacks, just like security cameras can’t offer you any intelligence if they aren’t trained on the area you’re trying to protect. Here are some logging and monitoring best practices for ensuring that you are not only logging significant IT events, but that you’re doing so in a way that will be easier to assess in the event of a security breach.

1. Automate reviewsA log management software solution is a necessary tool in any IT manager’s arsenal, but it isn’t enough on its own. Logs must not only be collected but carefully reviewed—and in the case of particularly high-risk applications, these reviews should be automatically conducted on an hourly basis. Ideally, the solution you use to do this would not only detect security threats in logs but deploy automated responses, such as blocking IP addresses, changing privileges, and disabling accounts.
2. Maintain manual administrator logsBecause administrators have so many more permissions than other users, their accounts must be monitored and protected with more vigilance. These users could exercise caution by manually logging their activities, including the times they logged on and off. These manual logs should be handled and analyzed with special attention if possible.
3. Frequently review fault logsErrors reported by servers, applications, or by the people who use them are incredibly vital to the work of troubleshooting. Understanding whether a recurring problem is the result of faulty equipment or user error, for instance, can be incredibly difficult without a well-maintained fault log. Logs for incredibly important and/or high-risk applications like eCommerce platforms should be reviewed and analyzed every day. Other applications and servers can have their fault logs checked every week or so.
4. Create log redundancyCybercriminals will often try to break into your log files in order to delete any evidence of the breach they committed. That’s why it’s important in logging best practices to record logs both locally and to a remote server that will be harder for criminals to access—discrepancy between the two files will trigger an alarm and prevent a breach from going unnoticed.
5. Make sure system clocks are synchronizedIn the world of forensics, understanding the exact order of events is essential to piecing together an accurate account of what crimes were committed and by whom. Doing that becomes very difficult if the clock on a particular device is inaccurate, even if it’s only by a minute or two. Regularly check the clocks of all devices in a system to ensure they’re all in sync.

What is unique about the Security Log in Windows? 

Microsoft offers an activity log specifically for the purposes of detecting attempts at unauthorized access. While the operating system uses its own criteria for determining what events are significant enough to record in the Security Log, administrators have the ability to configure the tool to include any operating system activity they choose.

The problem is that these specific event log management policies have also become a popular target among hackers. Because administrators have this ability to configure the Security Log, for example, it’s very common for attackers to attempt to compromise administrator accounts and tamper with these records, prompting many companies to create redundancy in their logging as recommended above. Similarly, because the Security Log can only hold a certain number of events, hackers will sometimes try to overload the system by generating so many events that incriminating evidence is overwritten.

Going forward

These best practices represent a good start, but as we’ve covered here, there’s simply too much information to analyze without the help of an effective tool. IT pros looking to leverage all their audit log data should turn to a solution that’s capable of both helping to demonstrate  compliance as well as responding to security threats as they’re detected in activity log data. SolarWinds^® Threat Monitor takes the guesswork out of logging and monitoring. The platform provides a single, holistic view so you can better examine the disparate event logs across network infrastructure. This solution keeps logs for a full year and can analyze thousands of logs quickly, comparing incidents with known threat intelligence. With a solution created with best practices in mind, your clients can rest assured that you are helping to keep their business networks secure.