Mask sensitive information in logs

Requires OneAgent 1.147 or later

Your log data contains information that may be considered sensitive. Specific log messages may include user names, email addresses, URL parameters, and other information that you may not want to disclose. Log Monitoring features the ability to mask any information by modifying the configuration file on each of the OneAgents that handles information you consider to be sensitive.

Mask personal data

Each OneAgent provides a ruxitagentloganalytics.conf file where you can set configuration options. To mask personal data in Log Monitoring, a masking rule and a masking rule scope need to be added to the configuration file for each OneAgent. The masking rule defines what data is masked, while the masking rule scope defines to what log files the rule is applied.

The masking is done in the OneAgent. No personal data will be sent to and stored in the DESK server.

For each OneAgent:

Edit the ruxitagentloganalytics.conf file, where you can set the configuration options:

On Linux, the file is located in:
/var/lib/desk/oneagent/agent/config/
On Windows, the file is located in:
C:\ProgramData\desk\oneagent\agent\config\

Create MaskingRule.
Append all your masking rules to the ruxitagentloganalytics.conf file.

This MaskingRule is a prerequisite for defining a scope. Whether you plan to use MaskingRulePGIDScope, MaskingRuleFileScope or both, first you must define the MaskingRule. This rule defines what to search for and how to replace it.

See MaskingRule

Define the scope where the MaskingRule should be applied.
Append all your defined scopes to the ruxitagentloganalytics.conf file.

You can create:

MaskingRulePGIDScope
This scope specifies that the MaskingRule should be applied to the specified process group ID.

See MaskingRulePGIDScope.
MaskingRuleFileScope
This scope specifies that the MaskingRule should be applied to the specified file or file groups.

See MaskingRuleFileScope.
Both MaskingRulePGIDScope and MaskingRuleFileScope In this case the, MaskingRule rule is applied to each defined scope associated with it in the order in which they were defined.

Save the file.
The masking configuration will be applied (ready for use) in less than a minute.
Changes in the configuration don't require OneAgent restart.

Your log files will now have sensitive information masked based on the rules you've defined.

Priority for multiple rules

Each consecutive masking rule is applied to the result of the previous masking rule.

For example, assume that you have created three masking rules:

The first masking rule is applied to the original log files.
The second masking rule is applied to the result of the first masking rule.
The third masking rule is applied to the result of the first and second masking rules.

The result is a log file sequentially processed by masking rules based on the rule order in the configuration file.

Entering log paths

MS Windows system logs

Masking sensitive information works on all Windows-based application logs, but not on MS Windows operating system logs.

The log file paths should be presented in a specific way:

The log file path is case sensitive.
Certain Windows log file paths are presented with the drive letter capitalized. Make sure that your Windows log file path reflects the capitalization of your Windows version.
No spaces are accepted between definition components.
Valid: MaskingRuleFileScope=Username_replace,"C:\logs\errors.log"
Invalid: MaskingRuleFileScope=Username_replace, "C:\logs\errors.log"
Do not use regular expressions or an asterisk wild card (*) to indicate multiple log files. Instead, use the DESK mapped names. For example, access.#.log.

Example of valid masking rule and scope rules

MaskingRule=Username_replace,"Username: {*},",sha1
MaskingRulePGIDScope=Username_replace,0X6714946EC3018AB3
MaskingRuleFileScope=Username_replace,"/var/log/apache2/access.#.log"
MaskingRuleFileScope=Username_replace,"/var/log/apache2/error.log"
MaskingRuleFileScope=Username_replace,"C:\logs\errors.log"

In this example, the masking rule named Username_replace will search for an instance of Username: and, for each occurrence, replace all characters between Username: and , strings with an SHA-1 hash for matching characters.

Next, it will apply the Username_replace rule to all log files in the 0X6714946EC3018AB3 process group.
Then, it will apply that same Username_replace rule to all log files under mapped name access.#.log and specifically to the error.log file.

Finally, it will apply that Username_replace rule to the Windows errors.log file (if OneAgent is operating on a Windows machine).

Example of a log file transformation

For this example log file /var/log/apache2/error.log containing the following log lines:

2018-01-12 15:44:12 INFO Successful transaction for Username: Maria Brava in e-trading, Product-Name: KS221
2018-01-12 15:44:12 INFO New worker family started with PID 2332af67
2018-01-12 15:44:12 WARN Authentication failure (Username: Arthur Brand, Times: 3)

The server will transmit the following log file:

2018-01-12 15:44:12 INFO Successful transaction for Username: ************************, Product-Name: KS221
2018-01-12 15:44:12 INFO New worker family started with PID 2332af67
2018-01-12 15:44:12 WARN Authentication failure (Username: ************, Times: 3)

MaskingRule

Syntax: MaskingRule=<name>,"<search_pattern>","<replace_string_or_hash>"

Masking rule name

A masking rule must have a unique name that consists of letters [a-zA-Z], numbers [0-9], and underscore (_) character.

Examples of valid Masking Rule names

ApacheAccessMethodRule
credit_card_mask
UserNameRule3

Masking rule search pattern

A search pattern consists of any literal string including a {*} wildcard that specifies the part of the string that should be masked. The search pattern must be placed in double quotes (" ") and cannot begin with the {*} wildcard. The {*} wildcard may be used at the end or in the middle of a search pattern.

Input line	Expression	Result
text123_text234_text123_text234	`{*}`text	not allowed (invalid)
text123text234text123text234	text1`{*}`text2	text1`###`text234text1`###`text234
text123text234text123text234	text`{*}`	text123text234text123text`###`
text123text234text123text234	text`{*}`text1	text`###`text123text234

Examples of valid masking rule search patterns

search pattern	result
`"Username={*}"`	All characters following `Username=` string will be masked.
`"Username={*} logged in"`	All characters between `Username=` and `logged in` strings will be masked.
`"CreditCardNo: {*}"`	All characters following `CreditCardNo:` string will be masked.
`"CreditCardNo: {*}, Customer:"`	All characters between `CreditCardNo:` and `, Customer:` strings will be masked.

Masking rule replace string or hash

The replace string can be a fixed phrase or an SHA-1 hash.

The provided string will literally replace the {*} wildcard in the search pattern result. Be sure to put the replace string in double quotes. The length of the replace string is fixed and does not correspond to the length of the pattern matched.

Examples of valid Masking Rule replace string

"#########"
"... MASKED ..."
"xxx"

Examples of valid Masking Rule replace string or hash

MaskingRule=ApplicationErrorRule,"Error {*}(",">>> MASKED <<< "

The masking rule ApplicationErrorRule searches for an instance with the Error string and, for each occurrence, replaces all characters between Error and ( with a fixed >>> MASKED <<< string.

MaskingRule=Username_replace,"Username: {*},",sha1

The masking rule Username_replace searches for an instance with Username: and, for each occurrence, replaces all characters between Username: and , with an SHA-1 hash for found characters.

MaskingRule=loginRuleServer1,"Successfully logged in as: '{*}' user","#####"

The masking rule loginRuleServer1 searches for an instance with Successfully logged in as: ' and, for each occurrence, replaces all characters between Successfully logged in as: ' and ' user with a fixed ##### string.

MaskingRule=ApacheAccessRule,"1.0\" {*} 462","Error code"

The masking rule ApacheAccessRule searches for an instance with 1.0" (notice that double quotes must be escaped with a backslash) and, for each occurrence, replaces all characters between 1.0" and 462 with a fixed Error code string.

MaskingRulePGIDScope

Syntax: MaskingRulePGIDScope=<name>,<pgid_as_hexadecimal_value>

MaskingRulePGIDScope defines where the linked MaskingRule should be applied based on the specified Process Group ID.

The MaskingRulePGIDScope is linked with the MaskingRule through a common name.

For example, MaskingRulePGIDScope=ApplicationErrorRule,0x6714946EC3018AB3 indicates that the MaskingRule named ApplicationErrorRule should be applied to all log files in the 0x6714946EC3018AB3 process group (the process group is represented as a hexadecimal value).
You can have multiple definitions of MaskingRulePGIDScope for the same MaskingRule.

Example of MaskingRulePGIDScope using the same masking rule

The following scope definitions apply the same UsernameMask MaskingRule but to a different process group:

MaskingRulePGIDScope=UsernameMask,0x6714946EC3018AB3
MaskingRulePGIDScope=UsernameMask,0XB34081EFF9E5F516

The process group is represented as a hexadecimal value that must begin with either 0x or 0X to indicate that it is a hex value.

MaskingRuleFileScope

Syntax: MaskingRuleFileScope=<name>,"<path_to_physical_or_mapped_filename>"

MaskingRuleFileScope defines where the linked MaskingRule should be applied based on the specified file or file groups.

The MaskingRuleFileScope is linked with the MaskingRule through a common name.

For example, MaskingRuleFileScope=ApplicationErrorRule,"/var/log/apache2/access.#.log" indicates that the MaskingRule named ApplicationErrorRule should be applied to all access.#.log files.
You can have multiple definitions of MaskingRuleFileScope using the same MaskingRule.

Example of MaskingRuleFileScope using the same masking rule

The following scope definitions apply the same UsernameMask MaskingRule but to different log files:

MaskingRuleFileScope=UsernameMask,"/var/log/apache2/error.log"
MaskingRuleFileScope=UsernameMask,"/var/log/apache2/mycustom.log"
MaskingRuleFileScope=UsernameMask,"/var/log/apache2/access.#.log"
MaskingRuleFileScope=UsernameMask,"C:\logs\errors.log"

The file scope definition can point to a specific log file on the host (for example, /var/log/apache2/error.log) or a mapped name that is used for rotated files (for example, /var/log/apache2/access.#.log). In either case, be sure to place the file path in double quotes.
The mapped names that represent rotated files use the same wildcard syntax as the Log viewer.

For example, the access.#.log mapped name represents files access.1.log, access.2.log, and any consecutive log files matching that pattern.