« SecViz », a log analysis tool (Part I)

Introduction

"SecViz" is the short name for "Security data visualization". It covers the activity that aims at graphically displaying data log. This term now is more illustrating a 'fashionable' trend: sites such as secviz.org or vizsec.org and articles on the subject are increasing. Therefore, it seemed appropriate to us to address this topic.

In general, there are several steps in the exploitation of logs within a company:

• The first step is to realize that some logs exist and that we could do something with them.
• The second step is the archiving of these logs often for legal reasons.
• The third step is to concentrate all the logs of the company in a common location to pool and share the resources used for archiving and secondly to establish a global process of exploitation.

From that point, the process of exploitation can follow two different paths, complementary but not necessarily exclusive of each other.

• The process of real time supervision: This process is based on the use of a SOC (Security Operation Center) and the deployment of software that allow the aggregation and correlation of these logs, generating alarm monitoring in real time.
• The process of viewing logs: This process can be a real-time analysis or offline (following an incident report for example).

The implementation of the visualization process is more or less iterative, depending on the degree of knowledge that one gain on the processed logs.

Logs correlation

In general, the correlation of logs consists first in aggregating similar logs (same source, same message, same time moment) and then, thanks to made-up scenarios defined by the operator, in generating new meta-events with more information.

Simple scenario example: If the log is destined to port 21 and the following log is destined to port 22 and next log is destined to port 23, and that the 3 logs are held in less than 1 minute, then a port scanning is probably ongoing towards that machine.

The correlation of logs is a powerful activity, with undeniable interests. However, be aware that it has also limitations.

• The problem with the correlation of logs is that they are linked to various scenarios. These different scenarios must indeed be written, and a scenario that is not imagined will not be written, and therefore the attempt (or failure) of intrusion will not be detected.
• In addition, it is sometimes impossible to correlate an event log with a previous one, because of a lack of context. For example, it is sometimes impossible in case of simultaneous SSH sessions on a machine, to link a SUDO event to a session rather than another.
• Another difficulty is that correlation is a process that consumes a lot of memory and CPU power during the analysis.

Logs visualization

The visualization of logs is a complementary approach to the correlation. It focuses on watching the logs in their overall and without the help of scenarios.

In this approach, it is the operator's eye that must perform the intuitive correlation work, based on the more or less complex visual signatures.

Logs visualization aims at highlighting:

• The recurring phenomena or the phenomena that produce a regular pattern among a fog (cloud) of logs that seem unlikely (random).
• The random phenomena among a scheme of logs planned and expected which ought to be regular or repetitive.

For the logs to be displayable, they must have certain characteristics:

• They must be numerous. It is useless to view only a few lines of logs, a simple analysis of the file is enough.
• They must contain digital information to position the values on a visualization axis. This information might be native (numbers, port numbers) or easily reconstructed (IP addresses, dates).
• They must contain several digital dimensions (at least 2).

The constraints of viewing logs are:

• The operator must know what he wants to represent, how he wants to represent it and what the image will be a priori.
• If the output image is not what it should have been, the task of interpreting the result will be more difficult.

Conclusion

Through this first part of the article, we presented the basic concepts related to the analysis approach based on log visualization. Next month we will cover the experience that the CNES acquired in such a field.

In the meantime, if you want to know more about log visualization, you can:

• visit the site dedicated to log visualization www.secviz.org and its gallery,
• explore the DAVIX "live-CD" which offers a set of tools to view logs.