关于logdata-anomaly-miner
logdata-anomaly-miner是一款安全日志解析与异常检测工具，该工具旨在以有限的资源和尽可能低的权限运行分析，以使其适合生产服务器使用。
为了确保 logdata-anomaly-miner的正常运行，推荐安装了python >= 3.6的Linux 系统。更具体地说，支持的系统包括 Debian Buster、Debian Bullseye、Debian Bookworm、Ubuntu 20.04、Ubuntu 22.04、Fedora（docker image fedora:latest）和 RedHat（docker image redhat/ubi9）。
工具架构
工具要求
scipy==1.10.0
pylibacl==0.5.4
kafka_python==2.0.2
pytz==2020.4
urllib3==1.26.19
numpy==1.22.0
Cerberus==1.3.2
psutil==5.7.3
kafka==1.3.5
pyzmq==20.0.0
python_dateutil==2.8.1
PyYAML==5.4
statsmodels==0.12.2
工具安装
由于该工具基于Python 3开发，因此我们首先需要在本地设备上安装并配置好最新版本的Python 3环境。
源码安装
接下来，广大研究人员可以直接使用下列命令将该项目源码克隆至本地：
git clone https://github.com/ait-aecid/logdata-anomaly-miner.git
然后切换到项目目录中，使用pip命令和项目提供的requirements.txt安装该工具所需的其他依赖组件：
cd logdata-anomaly-miner pip install -r requirements.txt
Debian安装
在官方 Debian/Ubuntu 存储库中，有适用于 logdata-anomaly-miner 的 Debian 软件包：
apt-get update && apt-get install logdata-anomaly-miner
Wget安装
以下命令将安装最新的稳定版本：
cd $HOME wget https://raw.githubusercontent.com/ait-aecid/logdata-anomaly-miner/main/scripts/aminer_install.sh chmod +x aminer_install.sh ./aminer_install.sh
工具配置
现在让我们将 Apache 解析器模型添加到 aminer-config：
alice@ubuntu2004:~$ sudo ln -s /etc/aminer/conf-available/generic/ApacheAccessModel.py /etc/aminer/conf-enabled/ alice@ubuntu2004:~$
在以前版本的 aminer 中，我们必须用 python 编写配置文件。在当前版本中，我们可以使用以 yaml 编写的配置。现在创建并编辑文件 /etc/aminer/config.yml：
LearnMode: True LogResourceList: - 'file:///var/log/apache2/access.log' Parser: - id: 'START' start: True type: ApacheAccessModel name: 'apache' Input: timestamp_paths: "/accesslog/time" Analysis: - type: "NewMatchPathValueDetector" paths: ["/accesslog/status"] output_logline: True EventHandlers: - id: "stpe" type: "StreamPrinterEventHandler"
如果我们现在启动 aminer，它将读取 access.log 并了解所有解析器路径。我们将在启动 aminer 之前使用“-C”参数清除持久性。（请注意，您可以使用 CTRL+c 终止 aminer）
alice@ubuntu2004:~$ sudo cat /var/log/apache2/access.log
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
alice@ubuntu2004:~$ sudo aminer -C --config /etc/aminer/config.yml
2021-05-17 12:12:36 New path(es) detected
NewMatchPathDetector: "DefaultNewMatchPathDetector" (1 lines)
/accesslog: 127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/host: 127.0.0.1
/accesslog/sp0:
/accesslog/ident: -
/accesslog/sp1:
/accesslog/user: -
/accesslog/sp2:
/accesslog/time: 1621250714
/accesslog/sp3: ] "
/accesslog/fm/request: GET / HTTP/1.1
/accesslog/fm/request/method: 0
/accesslog/fm/request/sp5:
/accesslog/fm/request/request: /
/accesslog/fm/request/sp6:
/accesslog/fm/request/version: HTTP/1.1
/accesslog/sp6: "
/accesslog/status: 200
/accesslog/sp7:
/accesslog/size: 11229
/accesslog/combined: "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/combined/combined: "-" "Wget/1.20.3 (linux-gnu)"
/accesslog/combined/combined/sp9: "
/accesslog/combined/combined/referer: -
/accesslog/combined/combined/sp10: " "
/accesslog/combined/combined/user_agent: Wget/1.20.3 (linux-gnu)
/accesslog/combined/combined/sp11: "
['/accesslog', '/accesslog/host', '/accesslog/sp0', '/accesslog/ident', '/accesslog/sp1', '/accesslog/user', '/accesslog/sp2', '/accesslog/time', '/accesslog/sp3', '/accesslog/fm/request', '/accesslog/sp6', '/accesslog/status', '/accesslog/sp7', '/accesslog/size', '/accesslog/combined', '/accesslog/combined/combined', '/accesslog/combined/combined/sp9', '/accesslog/combined/combined/referer', '/accesslog/combined/combined/sp10', '/accesslog/combined/combined/user_agent', '/accesslog/combined/combined/sp11', '/accesslog/fm/request/method', '/accesslog/fm/request/sp5', '/accesslog/fm/request/request', '/accesslog/fm/request/sp6', '/accesslog/fm/request/version']
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
2021-05-17 12:12:36 New value(s) detected
NewMatchPathValueDetector: "NewMatchPathValueDetector2" (1 lines)
{'/accesslog/status': 200}
127.0.0.1 - - [17/May/2021:11:25:14 +0000] "GET / HTTP/1.1" 200 11229 "-" "Wget/1.20.3 (linux-gnu)"
工具运行演示
演示视频：【点我观看】
许可证协议
本项目的开发与发布遵循GPL-3.0开源许可协议。
项目地址
logdata-anomaly-miner：【GitHub传送门】
参考资料
AMiner: A Modular Log Data Analysis Pipeline for Anomaly-based Intrusion Detection | Digital Threats: Research and Practice
SciTePress - Publication Details
AECID-PG: A Tree-Based Log Parser Generator To Enable Log Analysis | IEEE Conference Publication | IEEE Xplore
https://securitylab.no/cyberhunt2019/
