【实践成果】Splunk 9.0 Configuration Change Tracking

news2024/11/29 5:35:30

Splunk 9.0 引入了新的功能，一个很重要的一个，就是跟踪conguration 文件的变化：

这个很重要的特性，在splunk 9.0 以后才引入，就看server.conf 配置中，9.0 以后的版本才有：

server.conf - Splunk Documentation

Configuration Change Tracker

[config_change_tracker]
disabled = <boolean>
* Whether or not splunkd writes configuration changes to the 
  configuration change log at $SPLUNK_HOME/var/log/splunk/configuration_change.log.
* If set to "false", configuration changes are captured in
  $SPLUNK_HOME/var/log/splunk/configuration_change.log.
* If set to "true", configuration changes are not captured
  in $SPLUNK_HOME/var/log/splunk/configuration_change.log.
* Default: false

mode = [auto|diff|track-only]
* Determines the method used by 'config_change_tracker' to track and record
  changes to .conf files.
* A value of "auto" or "diff" means splunkd logs all configuration changes made to
  .conf files, including changes to setting values. In this mode, config change
  tracking only includes changes that could have an effect on your environment.
  For example, if a file with a stanza and setting-value pair is created, updated,
  or deleted, splunkd logs the change. But if an empty file or a stanza without any
  setting-value pairs is added or deleted, splunkd does not log the change since it
  will not have an impact. Similarly, splunkd does not track any comments that are
  added to or removed from files.
* A value of "track-only" means splunkd logs .conf file changes, but excludes
  configuration setting values. In this mode, config change tracking includes
  changes whether or not they can have an effect on your environment. For example,
  splunkd logs a change for any updates to file content, or that come from a change
  by the operating system. Splunkd also sees a comment that has been added to a .conf
  file as a change, because that change results in a different file checksum.
* Splunkd tracks all .conf files under the following directories:
  * $SPLUNK_HOME/etc/system
  * $SPLUNK_HOME/etc/apps
  * $SPLUNK_HOME/etc/users
  * $SPLUNK_HOME/etc/peer-apps
  It also tracks changes to the following:
  * $SPLUNK_HOME/etc/instance.cfg
* The values "auto" and "diff" have the same behavior at this time. Setting the 
  value to "auto" ensures that the instance will always use the latest feature set.
* Default: auto

denylist = <regular expression>
* If set, splunkd does not monitor files for configuration change tracker if 
  their path matches the specified regex.
* No default.

log_throttling_disabled = <boolean>
* Describes whether or not splunkd logs config changes to a .conf file
  that occur within the 'log_throttling_threshold_ms' time span as a single event.
* A value of "false" means that splunkd logs all changes to a conf file within
  the time span 'log_throttling_threshold_ms' as a single event.
* A value of "true" means that splunkd logs all changes individually as
  soon as it detects them.
* This setting requires a Linux system with the "inotify" API for
  file system event monitoring.
* Do not change this setting without first consulting with Splunk Support.
* Default: true

log_throttling_threshold_ms = <positive integer>
* The span of time, in milliseconds, during which splunkd logs multiple
  changes to a .conf file as a single configuration change event.
* If multiple changes are made to a conf file within the time span
  'log_throttling_threshold_ms' milliseconds, splunkd logs those changes
  as a single event.
* Default: 10000

exclude_fields = <comma-separated list>
* If set, splunkd excludes the stanza key that you specify when it writes to the 
  configuration_change.log file.
* The format for each entry is '<conf-file>:<stanza>:<key>'. Separate multiple 
  entries with commas.
* To exclude all keys under a stanza, use the '<conf-file>:<stanza>:*' format.
* This setting has no effect when mode is set to "track-only".
* Example setting: 
  'server.conf:general:pass4SymmKey, authentication.conf:authentication:*'
* No default.

* NOTE: The [config_change_audit] stanza, which was previously mentioned in 
  the Splunk version 8.2.0 documentation and configuration specification files,
  is now DEPRECATED.


-

上面明确提到splunk8.2 是没有这个功能的。

下面实践一下这个功能：

先创建一个alert:

这样，就在如下的path 下面产生文件：

/opt/splunk/etc/users/sheng/search/下面产生savedsearches.conf 文件。

Everyweek / expire: 24 / is equal to=0

后来的修改：

Everyday / expire: 48 / is equal to=2

这个时候：看到如下的跟踪log 变化：

/opt/splunk/var/log/splunk/configuration_change.log

{"datetime":"09-30-2023 13:32:23.725 +0800","log_level":"INFO ","component":"ConfigChange","data":{"path":"/opt/splunk/etc/users/admin/search/local/savedsearches.conf","action":"update","modtime":"Sat Sep 30 13:32:22 2023","epoch_time":"1696051943","new_checksum":"0xd41d8cd98f00b204","old_checksum":"0x7e9b3519314c7f2b","changes":[{"stanza":"coubase_alert","properties":[{"name":"action.email","new_value":"","old_value":"1"},{"name":"action.email.to","new_value":"","old_value":"shp@163.com"},{"name":"action.email.useNSSubject","new_value":"","old_value":"1"},{"name":"action.webhook.enable_allowlist","new_value":"","old_value":"0"},{"name":"alert.expires","new_value":"","old_value":"48h"},{"name":"alert.suppress","new_value":"","old_value":"0"},{"name":"alert.track","new_value":"","old_value":"0"},{"name":"counttype","new_value":"","old_value":"number of events"},{"name":"cron_schedule","new_value":"","old_value":"0 0 * * *"},{"name":"dispatch.earliest_time","new_value":"","old_value":"-1d"},{"name":"dispatch.latest_time","new_value":"","old_value":"now"},{"name":"enableSched","new_value":"","old_value":"1"},{"name":"quantity","new_value":"","old_value":"2"},{"name":"relation","new_value":"","old_value":"equal to"},{"name":"request.ui_dispatch_app","new_value":"","old_value":"search"},{"name":"request.ui_dispatch_view","new_value":"","old_value":"search"},{"name":"search","new_value":"","old_value":"index=abc_base"}]}]}}

很好：下面开始change alert 来对比一下：

index=_configtracker "data.path"="*savedsearches.conf"
| spath output=modtime data.modtime,
| spath output=path data.path,
| spath output=stanza data.changes{}.stanza,
| spath output=name data.changes{}.properties{}.name,
| spath output=new_value data.changes{}.properties{}.new_value,
| spath output=old_value data.changes{}.properties{}.old_value,
| table modtime path name prop_name new_value old_value