文章目录
- Kdump简介
- Crash简介
- Ubuntu下安装使用方法
- Crash相关命令
- crash基本用法
- crash的基本命令
- help：crash所提供的调试命令
- log：查看日志信息，类似dmesg输出
- bt：查看异常时候的堆栈信息
- dev: 查看设备的情况
- dis：反汇编代码
- files：查看进程打开的文件清单等
- fuser：打开文件的进程信息
- ipcs：系统中使用共享内存的情况
- irq： 查看中断号和中断资源的对应关系
- mod：查看模块加载的情况
- mount： 查看设备挂载情况
- net： 网络配置信息
- ps： 查看系统的进程状态
- runq： CPU上的运行队列情况
- sym： 查看符号列表信息
- sys： 查看所有系统调用的实现
- task： 显示task_struct结构体的变量情况
- timer：定时器的状态信息
- 应用场景
- log + bt + dis
- dev+irq+mod+mount+net+ps+runq+ipcs
Kdump简介
- Kdump 是一种的新的crash dump捕获机制，用来捕获kernel crash时候产生的crash dump。Kdump需要配置两个不同目的的kernel，其中一个称作standard(production) kernel；另外一个称之为Crash(capture)kernel。
- standard（production）kernel，是指正在使用的kernel，当standard kernel在使用的过程中出现crash的时候， kdump会切换到crash kernel, 简单来说，standard kernel在正运行时发生crash，而crash(capture) Kernel 会被用来捕获production kernel crash时候产生的crash dump。
- 捕获crash dump是在新的crash(capture) kernel 的上下文中来捕获的，而不是在standard kernel上下文进行。
- 具体是当standard kernel方式crash的时候，kdump通过kexec自动启动进入到crash kernel当中。如果启动了kdump服务，standard kernel会预留一部分内存， 这部分内存用来启动crash kernel。
kdump机制主要包括两个组件：kdump和kexec： - kexec是一个快速启动机制，允许通过已经运行的内核的上下文启动一个Linux内核，不需要经过BIOS。BIOS可能会消耗很多时间，特别是带有众多数量的外设的大型服务器。这种办法可以为经常启动机器的开发者节省很多时间。Kexec是实现kdump机制的关键，它包括2个组成部分：一是内核空间的系统调用kexec_load，负责在生产内核（production kernel 或 first kernel）启动时将捕获内核（capture kernel或sencond kernel）加载到指定地址。二是用户空间的工具kexec-tools，他将捕获内核的地址传递给生产内核，从而在系统崩溃的时候能够找到捕获内核的地址并运行。没有kexec就没有kdump。先有kexec实现了在一个内核中可以启动另一个内核，才让kdump有了用武之地。
- kdump是一种先进的基于kexec的内核崩溃转储机制。当系统崩溃时，kdump使用kexec 启动到第二个内核。第二个内核通常叫做捕获内核，以很小内存启动以捕获转储镜像。第一个内核保留了内存的一部分给第二内核启动用。由于kdump利用kexec启动捕获内核，绕过了 BIOS，所以第一个内核的内存得以保留。这是内核崩溃转储的本质。kdump需要两个不同目的的内核，生产内核和捕获内核。生产内核是捕获内核服务的对像。捕获内核会在生产内核崩溃时启动起来，与相应的ramdisk一起组建一个微环境，用以对生产内核下的内存进行收集和转存。注意，在启动时，kdump保留了一定数量的重要的内存，为了计算系统需要的真正最小内存，加上kdump使用的内存数量，以决定真正的最小内存的需求。
- Kexec的设计是用新内核去覆盖原内核位置；而KDUMP是预留一块内存来加载第二个内核(和相关数据)，Crash后第二个内核在原位置运行(不然就达不到相关目的了)，收集第一个内核的相关内存信息。
Crash简介
当 linux 系统内核发生崩溃的时候，可以通过 kdump 等方式收集内核崩溃之前的内存，生成一个转储文件 vmcore。而后通过分析该 vmcore 文件就可以诊断出内核崩溃的原因，从而进行操作系统的代码改进。而 crash 就是一个被广泛使用的内核崩溃转储文件分析工具，掌握 crash 的使用技巧，对于定位问题有着十分重要的作用。
使用crash的先决条件：
- kernel 映像文件 vmlinux 在编译的时候必须指定了 -g 参数，即带有调试信息；
- 需要有一个内存崩溃转储文件（例如 vmcore），或者可以通过 /dev/mem 或 /dev/crash 访问的实时系统内存。如果 crash 命令行没有指定转储文件，则 crash 默认使用实时系统内存；
- crash 支持的平台处理器包括：x86, x86_64, ia64, ppc64, arm, s390, s390x ( 也有部分 crash 版本支持 Alpha 和 32-bit PowerPC，但是对于这两种平台的支持不保证长期维护 )；
- crash 支持 2.2.5-15（含）以后的 Linux 内核版本。随着 Linux 内核的更新，crash 也在不断升级以适应新的内核。
Ubuntu下安装使用方法
在Ubuntu上进行kdump与crash的相关配置：
（1）内核配置支持kdump与kexec：
在内核中需要打开如下选项：
CONFIG_KEXEC=y
CONFIG_SYSFS=y
CONFIG_DEBUG_INFO=Y
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y
（2）在Ubuntu上安装crash、kexec-tools等相关工具：
首先通过：apt-get install linux-crashdump 安装；
在安装完成后，需要调整crashkernel内存大小为768M(默认192M内存太小)
修改 /etc/defatul/grub.d/kdump-tools.cfg 中 crashkernel=512M-:768M
然后再重新生成grub.cfg：grub-mkconfig -o /boot/grub/grub.cfg
之后reboot，重启机器生效，在系统重新启动后可通过kdump-config show查看crash的相关配置及是否已然生效：
（3）通过sysrq机制手动测试kdump是否正常工作：
首先在通过sysrq的指令来触发一次panic，来测试capture内核是否能够正常运行，这需要在Linux内核配置时，打开sysrq功能的支持，以及panic on oops选型的支持：
之后使用
echo c > /proc/sysrq-trigger
类似alt + sysrq + c键会触发一个空指针的kernel panic.
等待内核转储完毕，系统会重新启动，重启完成后会在/var/crash/[time]/目录下会生成dump.[time]的转储文件。
等待内核转储完毕，系统会重新启动，重启完成后会在/var/crash/[time]/目录下会生成dump.[time]的转储文件。
然后使用
crash /lib/modules/5.3.18+/build/vmlinuz-5.3.18+ dump.202201120843
进行crash分析.
（4）Crash的使用：
使用 crash 调试转储文件，需要在命令行输入两个参数：debug kernel 和 dump file，其中 dump file 是内核转储文件的名称，debug kernel 是由内核调试信息包安装的
Crash命令：crash <system-map-file> <vmlinux-file> coredump
常用参数选项：
-h：打印帮助信息
-d：设置调试级别
-S：使用 /boot/System.map 作为默认的映射文件
-s：不显示版本、初始调试信息等，直接进入命令行
-i file：启动之后自动运行 file 中的命令，再接受用户输入
Crash相关命令
crash基本用法
root@root-PC:/var/crash/202201120843# crash /lib/modules/5.3.18+/build/vmlinuz-5.3.18+ dump.202201120843
KERNEL: /data/linux-source-5.3.0/vmlinux //debug 内核
DUMPFILE: dump.202204091433 [PARTIAL DUMP] //dump文件
CPUS: 8 //CPU数量
DATE: Sat Apr 9 14:33:07 2022 //发生dump的日期
UPTIME: 00:04:03 //表示内核已正常运行的时间
LOAD AVERAGE: 2.31, 0.82, 0.29 //内核崩溃时的系统负载
TASKS: 343 //内核崩溃时系统运行的任务数
NODENAME: root-PC //主机名
RELEASE: 5.3.18+ //内核版本
VERSION: #2 SMP Sat Apr 9 14:26:36 CST 2022
MACHINE: x86_64 (2494 Mhz) //CPU架构与主频信息
MEMORY: 16 GB //发生内核崩溃的系统的内存大小
PANIC: "Kernel panic - not syncing: sysrq triggered crash" //内核崩溃的类型
PID: 1341 //导致内核崩溃的进程号
COMMAND: "bash" //内核崩溃的进程名称，或命令
TASK: ffff8ede003bdc00 [THREAD_INFO: ffff8ede003bdc00] //内核崩溃的进程访问的内存地址
CPU: 6 //表示导致内核崩溃的进程占用的 CPU
STATE: TASK_RUNNING (PANIC) //表示导致内核崩溃的进程的运行状态
crash的基本命令
help：crash所提供的调试命令
crash> help
* extend log rd task
alias files mach repeat timer
ascii foreach mod runq tree
bpf fuser mount search union
bt gdb net set vm
btop help p sig vtop
dev ipcs ps struct waitq
dis irq pte swap whatis
eval kmem ptob sym wr
exit list ptov sys q
常用命令如下：
alias: 别名
ascii: 打印ASCII编码表，16进制转换为字符。
bpf: 加载eBPF 脚本程序。
eval: 进展转变，把数字按照各种格式打印出来。
btop: 地址转换为对应的页号
mod: 和lsmod的功能一样
log：查看日志信息，类似dmesg输出
crash> log
......
......
[ 45.762046] wlan0: associated
[ 45.784751] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 45.837528] wlan0: Limiting TX power to 30 (30 - 0) dBm as advertised by d4:68:ba:08:f8:7d
[ 74.928113] sysrq: Trigger a crash
[ 74.931668] Kernel panic - not syncing: sysrq triggered crash
[ 74.937558] CPU: 6 PID: 1341 Comm: bash Kdump: loaded Tainted: P IOE 5.3.18+ #2
[ 74.946263] Hardware name: To be filled by O.E.M. To be filled by O.E.M./QM87, BIOS 4.6.5 11/10/2015
[ 74.955630] Call Trace:
[ 74.958167] dump_stack+0x6d/0x95
[ 74.961573] panic+0xfe/0x2d4
[ 74.964615] sysrq_handle_crash+0x15/0x20
[ 74.968753] __handle_sysrq+0x93/0x150
[ 74.972653] write_sysrq_trigger+0x2f/0x40
[ 74.976816] proc_reg_write+0x3e/0x60
[ 74.980504] __vfs_write+0x1b/0x40
[ 74.983943] vfs_write+0xb1/0x1a0
[ 74.987298] ksys_write+0xa7/0xe0
[ 74.990646] __x64_sys_write+0x1a/0x20
[ 74.994451] do_syscall_64+0x5a/0x130
[ 74.998151] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 75.003273] RIP: 0033:0x7f86862cb264
[ 75.006886] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 a1 06 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5
[ 75.026016] RSP: 002b:00007ffebd5172d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 75.033788] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f86862cb264
[ 75.041084] RDX: 0000000000000002 RSI: 0000561b90df2590 RDI: 0000000000000001
[ 75.048390] RBP: 0000561b90df2590 R08: 000000000000000a R09: 0000000000000001
[ 75.055748] R10: 000000000000000a R11: 0000000000000246 R12: 00007f86865a7760
[ 75.063097] R13: 0000000000000002 R14: 00007f86865a32a0 R15: 00007f86865a2760
bt：查看异常时候的堆栈信息
可以查看调用的代码行号
crash> bt -l
PID: 1341 TASK: ffff8ede003bdc00 CPU: 6 COMMAND: "bash"
#0 [ffffb47b81953c58] machine_kexec at ffffffff8766b833
/data/linux-source-5.3.0/arch/x86/kernel/machine_kexec_64.c: 441
#1 [ffffb47b81953cb8] __crash_kexec at ffffffff8774bb82
/data/linux-source-5.3.0/kernel/kexec_core.c: 957
#2 [ffffb47b81953d88] panic at ffffffff8769778a
/data/linux-source-5.3.0/./arch/x86/include/asm/smp.h: 72
#3 [ffffb47b81953e10] sysrq_handle_crash at ffffffff87db3a85
/data/linux-source-5.3.0/drivers/tty/sysrq.c: 140
#4 [ffffb47b81953e20] __handle_sysrq at ffffffff87db3ec3
/data/linux-source-5.3.0/drivers/tty/sysrq.c: 580
#5 [ffffb47b81953e58] write_sysrq_trigger at ffffffff87db439f
/data/linux-source-5.3.0/drivers/tty/sysrq.c: 1108
#6 [ffffb47b81953e70] proc_reg_write at ffffffff879530fe
/data/linux-source-5.3.0/fs/proc/inode.c: 238
#7 [ffffb47b81953e90] __vfs_write at ffffffff878c86cb
/data/linux-source-5.3.0/fs/read_write.c: 500
#8 [ffffb47b81953ea0] vfs_write at ffffffff878c93b1
/data/linux-source-5.3.0/fs/read_write.c: 584
#9 [ffffb47b81953ed8] ksys_write at ffffffff878cb977
/data/linux-source-5.3.0/fs/read_write.c: 638
#10 [ffffb47b81953f20] __x64_sys_write at ffffffff878cb9ca
/data/linux-source-5.3.0/fs/read_write.c: 646
#11 [ffffb47b81953f30] do_syscall_64 at ffffffff876043ba
/data/linux-source-5.3.0/arch/x86/entry/common.c: 296
#12 [ffffb47b81953f50] entry_SYSCALL_64_after_hwframe at ffffffff8840008c
/data/linux-source-5.3.0/arch/x86/entry/entry_64.S: 184
RIP: 00007f86862cb264 RSP: 00007ffebd5172d8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f86862cb264
RDX: 0000000000000002 RSI: 0000561b90df2590 RDI: 0000000000000001
RBP: 0000561b90df2590 R8: 000000000000000a R9: 0000000000000001
R10: 000000000000000a R11: 0000000000000246 R12: 00007f86865a7760
R13: 0000000000000002 R14: 00007f86865a32a0 R15: 00007f86865a2760
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
dev: 查看设备的情况
一切皆文件，fops，pci设备，disk io的状态信息。
crash> dev
CHRDEV NAME CDEV OPERATIONS
1 mem ffff9e944bd8a180 memory_fops
2 pty ffff9e944aca8380 tty_fops
3 ttyp ffff9e944aca8780 tty_fops
4 /dev/vc/0 ffffffff97a01640 console_fops
4 tty ffff9e944bd8a780 tty_fops
4 ttyS ffff9e944aca8c80 tty_fops
5 /dev/tty ffffffff97a00540 tty_fops
5 /dev/console ffffffff97a004c0 console_fops
5 /dev/ptmx ffffffff97a006e0 ptmx_fops
5 ttyprintk ffff9e9448a9cb00 tty_fops
7 vcs ffff9e944bd8ad80 vcs_fops
10 misc ffff9e944b5ea680 misc_fops
13 input (none)
21 sg ffff9e944b4ebb00 sg_fops
29 fb ffff9e944b5ea600 fb_fops
81 video4linux (none)
89 i2c ffff9e944ad382e8 i2cdev_fops
100 ts_config ffff9e9448a9cb80 ts_config_fops
116 alsa ffff9e94485d3b00 snd_fops
128 ptm ffff9e944aca8100 tty_fops
136 pts ffff9e944aca8400 tty_fops
166 ttyACM (none)
180 usb ffff9e944b5aee80 usb_fops
188 ttyUSB (none)
189 usb_device ffffffff97a13060 usbdev_file_operations
195 nvidia-frontend ffff9e94485d3180 nv_frontend_fops
226 drm ffff9e944b27e080 drm_stub_fops
238 nvidia-uvm ffffffffc1bd3d40 uvm_fops
239 nvidia-nvswitch ffffffffc1523ec8 device_fops
240 nvidia-nvlink ffffffffc1523e28 nvlink_fops
241 hidraw ffffffff97a16f60 hidraw_ops
242 ttyGS (none)
243 aux ffff9e944b27ec00 auxdev_fops
244 nvme (none)
245 ttynull ffff9e9448a9c080 tty_fops
246 bsg ffffffff979f6be0 bsg_fops
247 watchdog (none)
248 cec (none)
249 rtc ffff9e94474e9308 rtc_dev_fops
250 dax (none)
251 dimmctl ffff9e944b5ae800 nvdimm_fops
252 ndctl ffff9e944b5aec00 nvdimm_bus_fops
253 tpm (none)
254 gpiochip (none)
BLKDEV NAME GENDISK OPERATIONS
259 blkext (none)
7 loop ffff9e9448488800 lo_fops
8 sd ffff9e944b73c800 sd_fops
11 sr (none)
65 sd (none)
66 sd (none)
67 sd (none)
dis：反汇编代码
通过地址信息查看代码，行号，反汇编函数等
crash> dis -s vfs_write+87
FILE: fs/read_write.c
LINE: 575
570
571 if (!(file->f_mode & FMODE_WRITE))
572 return -EBADF;
573 if (!(file->f_mode & FMODE_CAN_WRITE))
574 return -EINVAL;
* 575 if (unlikely(!access_ok(buf, count)))
576 return -EFAULT;
577
578 ret = rw_verify_area(WRITE, file, pos, count);
579 if (!ret) {
580 if (count > MAX_RW_COUNT)
581 count = MAX_RW_COUNT;
582 file_start_write(file);
583 ret = __vfs_write(file, buf, count, pos);
584 if (ret > 0) {
585 fsnotify_modify(file);
586 add_wchar(current, ret);
587 }
588 inc_syscw(current);
589 file_end_write(file);
590 }
591
592 return ret;
593 }
files：查看进程打开的文件清单等
通过地址信息查看代码，行号，反汇编函数等
crash> files 3406
PID: 3406 TASK: ffff9e9404475c00 CPU: 1 COMMAND: "bash"
ROOT: / CWD: /home/system/root
FD FILE DENTRY INODE TYPE PATH
0 ffff9e943ff3fd00 ffff9e93ed9f3600 ffff9e943543f840 CHR /dev/pts/0
1 ffff9e9441486900 ffff9e943ab88900 ffff9e943ab7a268 REG /proc/sysrq-trigger
2 ffff9e943ff3fd00 ffff9e93ed9f3600 ffff9e943543f840 CHR /dev/pts/0
10 ffff9e943ff3fd00 ffff9e93ed9f3600 ffff9e943543f840 CHR /dev/pts/0
255 ffff9e943ff3fd00 ffff9e93ed9f3600 ffff9e943543f840 CHR /dev/pts/0
fuser：打开文件的进程信息
crash> fuser /dev/nvidia-uvm
PID TASK COMM USAGE
3221 ffff88040af12340 "UltrasoundBack fd
3228 ffff88040b7c9780 "UltrasoundMain fd
3553 ffff88040a9c4680 "tmpdata" fd
3554 ffff8800d944bac0 "tmpdata" fd
3555 ffff8800d944e9c0 "tmpdata" fd
3556 ffff88040a9c0bc0 "tmpdata" fd
3576 ffff88040b7a4680 "tmpdata" fd
3577 ffff88040aa00000 "KVmSignal" fd
3578 ffff88040b7a3ac0 "KVmSignal" fd
3579 ffff88040b7a5240 "LogWriting" fd
3580 ffff88040aa02340 "LogWriting" fd
3581 ffff88040aa01780 "QXcbEventReade fd
3582 ffff88040b7a2f00 "QXcbEventReade fd
3585 ffff8800d9449780 "tmpdata" fd
3586 ffff880407e12340 "tmpdata" fd
3587 ffff8800d944af00 "tmpdata" fd
3588 ffff880407e10000 "tmpdata" fd
3589 ffff88040b0469c0 "tmpdata" fd
3590 ffff88040acd2340 "tmpdata" fd
3607 ffff88040aa069c0 "SonoIotDispatc fd
3608 ffff88040ab20000 "SonoIotClient" fd
3609 ffff88040af10000 "UltrasoundIO" fd
3610 ffff88040af12f00 "UltrasoundFile fd
3746 ffff880407e10bc0 "usb2int" fd
3748 ffff88040aa8c680 "KPeripheralMon fd
3874 ffff88040aa8de00 "screenbrightne fd
3877 ffff88040ab02340 "TouchscreenRec fd
3881 ffff88040ab00bc0 "PrintThread" fd
3889 ffff8803b626d240 "MonitorThread" fd
4088 ffff88040aa88000 "UltrasoundMain fd
4089 ffff88040aa8bac0 "UltrasoundMain fd
4090 ffff88040aa8e9c0 "KAutotestServe fd
4092 ffff88040aa8af00 "IoEventThread" fd
4093 ffff88040aa8a340 "httpthread" fd
4114 ffff8803b613af00 "KMwaThread" fd
4115 ffff8803b6278bc0 "IPostProcess2D fd
4116 ffff8803b6279780 "IPreProcess2D" fd
4117 ffff8803b627e9c0 "KPostProcess1D fd
4118 ffff8803b6278000 "KPreProcess1D" fd
4119 ffff8803b627a340 "KPostProcessWv fd
4120 ffff8803b627af00 "KImgParseThrea fd
4121 ffff8803b627bac0 "KImgSave" fd
4122 ffff8803b627c680 "KImgDisplayThr fd
4123 ffff8803b627d240 "KImgPick" fd
4124 ffff880392130000 "KPreProcess4D" fd
4125 ffff880392130bc0 "KPostProcess4D fd
4286 ffff880392131780 "FrontendHeader fd
4328 ffff8803b629d240 "KDicomServer" fd
4329 ffff8803b6299780 "KDcmCmdSender" fd
4330 ffff8803921c5e00 "UltrasoundMain fd
......
ipcs：系统中使用共享内存的情况
crash> ipcs
SHMID_KERNEL KEY SHMID UID PERMS BYTES NATTCH STATUS
ffff9e9449917400 00000000 2 0 600 56800 2 dest
ffff9e9449917c00 00000000 3 0 600 4019200 2 dest
ffff9e940051aa00 00000000 4 0 600 90000 2 dest
ffff9e940051ae00 00000000 5 0 600 56800 2 dest
ffff9e940051a300 00000000 6 0 600 4019200 2 dest
ffff9e940051bf00 00000000 7 0 600 8294400 2 dest
ffff9e940051b900 00000000 8 0 600 773568 2 dest
SEM_ARRAY KEY SEMID UID PERMS NSEMS
ffff9e94367be000 00010001 0 0 666 1
ffff9e94367be400 00060002 1 0 666 1
MSG_QUEUE KEY MSQID UID PERMS USED-BYTES MESSAGES
ffff9e9448556d00 0000232a 0 0 666 0 0
ffff9e9448556100 00002330 1 0 666 0 0
ffff9e9447d86700 00002329 2 0 666 0 0
ffff9e9441484900 00002331 3 0 666 0 0
irq： 查看中断号和中断资源的对应关系
crash> irq
IRQ IRQ_DESC/_DATA IRQACTION NAME
0 ffff9e944bd4d800 ffffffff9722d840 "timer"
1 ffff9e944bd4e800 ffff9e944ad06d80 "i8042"
2 ffff9e944bd4d400 (unused)
3 ffff9e944bd4cc00 (unused)
4 ffff9e944bd4c000 ffff9e944b237900 "ttyS0"
5 ffff9e944bd4c400 (unused)
6 ffff9e944bd4ec00 (unused)
7 ffff9e944bd4e200 ffff9e9435037800 "ttyS2"
8 ffff9e944bd4de00 ffff9e944b237480 "rtc0"
9 ffff9e944bd4f400 ffff9e944b5eab80 "acpi"
10 ffff9e944bd4f200 (unused)
11 ffff9e944bd4e000 (unused)
12 ffff9e944bd4c600 ffff9e944ad06580 "i8042"
13 ffff9e944bd4c200 (unused)
14 ffff9e944bd4ce00 (unused)
15 ffff9e944bd4dc00 (unused)
16 ffff9e944b14dc00 ffff9e944b27e580 "ehci_hcd:usb1"
17 ffff9e9448857000 ffff9e944b3ff800 "snd_hda_intel:card1"
18 ffff9e944b0e6c00 ffff9e944ad06100 "i801_smbus"
19 ffff9e944842b800 (unused)
20 (unused) (unused)
21 (unused) (unused)
22 ffff9e9448855e00 (unused)
23 ffff9e94485d8400 ffff9e944b27e480 "ehci_hcd:usb2"
24 ffff9e944b14d000 ffff9e944b5ead00 "PCIe PME"
25 ffff9e944b14ee00 ffff9e944b5eaf80 "PCIe PME"
26 ffff9e9448857600 ffff9e944b3fff00 "PCIe PME"
27 ffff9e944842bc00 ffff9e9448b4e080 "ahci[0000:00:1f.2]"
28 ffff9e94485dac00 ffff9e944b237180 "xhci_hcd"
29 ffff9e9449d55400 ffff9e944b237200 "eth0"
30 ffff9e9449d55800 ffff9e944b237f80 "eth1"
31 ffff9e9448855400 ffff9e944b3ff280 "snd_hda_intel:card0"
32 ffff9e9447ce3600 ffff9e943eeb9f00 "nvidia"
mod：查看模块加载的情况
crash> mod
MODULE NAME SIZE OBJECT FILE
ffffffffc01a5380 x_tables 40960 (not loaded) [CONFIG_KALLSYMS]
ffffffffc01b0280 ip_tables 32768 (not loaded) [CONFIG_KALLSYMS]
ffffffffc01e4140 sch_fq_codel 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0200000 soundcore 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0210340 snd 81920 (not loaded) [CONFIG_KALLSYMS]
ffffffffc021e0c0 snd_timer 36864 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0224140 snd_seq_device 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0229140 ie31200_edac 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc022e040 snd_seq_midi_event 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0233000 ledtrig_audio 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc023a1c0 lpc_ich 24576 (not loaded) [CONFIG_KALLSYMS]
ffffffffc02412c0 rtc_cmos 24576 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0247180 snd_seq_midi 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc025a340 snd_seq 69632 (not loaded) [CONFIG_KALLSYMS]
ffffffffc027b640 psmouse 131072 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0288240 snd_rawmidi 36864 (not loaded) [CONFIG_KALLSYMS]
ffffffffc02be000 e1000e 253952 (not loaded) [CONFIG_KALLSYMS]
ffffffffc02dd580 snd_pcm 102400 (not loaded) [CONFIG_KALLSYMS]
ffffffffc02f3040 snd_hda_core 90112 (not loaded) [CONFIG_KALLSYMS]
ffffffffc02feb80 intel_cstate 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0305040 snd_intel_dspcfg 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc0310e40 snd_hda_intel 53248 (not loaded) [CONFIG_KALLSYMS]
ffffffffc03214c0 snd_hda_codec_hdmi 57344 (not loaded) [CONFIG_KALLSYMS]
ffffffffc032a080 snd_hwdep 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc15222c0 nvidia 20459520 (not loaded) [CONFIG_KALLSYMS]
ffffffffc16b3040 irqbypass 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc16b9100 coretemp 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc16bf140 intel_powerclamp 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc16d83c0 snd_hda_codec 131072 (not loaded) [CONFIG_KALLSYMS]
ffffffffc17d6880 nvidia_modeset 1114112 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1803400 snd_hda_codec_generic 81920 (not loaded) [CONFIG_KALLSYMS]
ffffffffc181f500 snd_hda_codec_realtek 122880 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1839000 glue_helper 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc183f100 cryptd 24576 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1845040 crypto_simd 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc184a440 ghash_clmulni_intel 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc184f880 rapl 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc185c600 nvidia_drm 49152 (not loaded) [CONFIG_KALLSYMS]
ffffffffc18631c0 aes_x86_64 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc18b4dc0 aesni_intel 372736 (not loaded) [CONFIG_KALLSYMS]
ffffffffc18c4240 crc32_pclmul 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc18c9240 crct10dif_pclmul 16384 (not loaded) [CONFIG_KALLSYMS]
ffffffffc19450c0 kvm 655360 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1995f00 kvm_intel 204800 (not loaded) [CONFIG_KALLSYMS]
ffffffffc19ea180 x86_pkg_temp_thermal 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc19f6540 intel_rapl_common 24576 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1aad800 mac80211 663552 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1ada780 rt2x00lib 61440 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1afa600 rt2800lib 122880 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1b02180 rt2x00usb 20480 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1b0a280 rt2800usb 28672 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1b27140 usb_sono 24576 (not loaded) [CONFIG_KALLSYMS]
ffffffffc1bd3980 nvidia_uvm 942080 (not loaded) [CONFIG_KALLSYMS]
mount： 查看设备挂载情况
crash> mount
MOUNT SUPERBLK TYPE DEVNAME DIRNAME
ffff9e944bdc6140 ffff9e944bc12800 rootfs rootfs /
ffff9e944bdc6780 ffff9e9448667800 sysfs sysfs /sys
ffff9e944bdc7040 ffff9e944bc15800 proc proc /proc
ffff9e944bdc6000 ffff9e944b56a800 devtmpfs udev /dev
ffff9e944bdc63c0 ffff9e9448660000 devpts devpts /dev/pts
ffff9e944bdc6640 ffff9e9448664800 tmpfs tmpfs /run
ffff9e94483412c0 ffff9e944b7ad800 ext4 /dev/sda1 /
ffff9e944b00d2c0 ffff9e944b4c7000 securityfs securityfs /sys/kernel/security
ffff9e944b00da40 ffff9e9448fcc000 tmpfs tmpfs /dev/shm
ffff9e944b00de00 ffff9e9448fce000 tmpfs tmpfs /run/lock
ffff9e944b00cc80 ffff9e9448fcb800 tmpfs tmpfs /sys/fs/cgroup
ffff9e944b00dcc0 ffff9e9448fcb000 cgroup2 cgroup /sys/fs/cgroup/unified
ffff9e944b00d680 ffff9e9448fcf800 cgroup cgroup /sys/fs/cgroup/systemd
ffff9e944b00d7c0 ffff9e9448fc8000 pstore pstore /sys/fs/pstore
ffff9e944b00c780 ffff9e9448fcc800 cgroup cgroup /sys/fs/cgroup/freezer
ffff9e944b00d040 ffff9e944adb9800 cgroup cgroup /sys/fs/cgroup/net_cls,net_prio
ffff9e944b00c000 ffff9e944adb9000 cgroup cgroup /sys/fs/cgroup/rdma
ffff9e944b00c3c0 ffff9e944adbc000 cgroup cgroup /sys/fs/cgroup/perf_event
ffff9e944b00c640 ffff9e944adbd800 cgroup cgroup /sys/fs/cgroup/devices
ffff9e944b00c8c0 ffff9e944adbe000 cgroup cgroup /sys/fs/cgroup/memory
ffff9e944b00d900 ffff9e944adbb800 cgroup cgroup /sys/fs/cgroup/blkio
ffff9e944b00d540 ffff9e944adbb000 cgroup cgroup /sys/fs/cgroup/pids
ffff9e944b00ca00 ffff9e944adbf800 cgroup cgroup /sys/fs/cgroup/hugetlb
ffff9e944b00cf00 ffff9e944adb8000 cgroup cgroup /sys/fs/cgroup/cpuset
ffff9e944b00c280 ffff9e944adbc800 cgroup cgroup /sys/fs/cgroup/cpu,cpuacct
ffff9e944ac88500 ffff9e944848c800 hugetlbfs hugetlbfs /dev/hugepages
ffff9e94498a97c0 ffff9e944b4c2800 debugfs debugfs /sys/kernel/debug
ffff9e944b0e2f00 ffff9e944b4c3000 rpc_pipefs sunrpc /run/rpc_pipefs
ffff9e94498a8780 ffff9e9448fc8800 mqueue mqueue /dev/mqueue
ffff9e944ac88dc0 ffff9e944848d000 fusectl fusectl /sys/fs/fuse/connections
ffff9e944ac89400 ffff9e944ac30000 configfs configfs /sys/kernel/config
ffff9e9448847180 ffff9e944ac36800 ramfs none /tmp
ffff9e944b0e2280 ffff9e944b454000 ext4 /dev/sda2 /var
ffff9e944bdc68c0 ffff9e9447cba800 ext4 /dev/sda3 /home
ffff9e944bdc7540 ffff9e9447cb9800 ext4 /dev/sda4 /data
net： 网络配置信息
crash> net
NET_DEVICE NAME IP ADDRESS(ES)
ffff9e944b3e2000 lo 127.0.0.1
ffff9e943f75c000 eth0 192.168.33.3
ffff9e943f524000 eth1 192.0.2.2
ffff9e94499f6000 wlan0 192.168.93.102
ps： 查看系统的进程状态
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 0 0 0 ffffffff97213780 RU 0.0 0 0 [swapper/0]
0 0 1 ffff9e944bea9700 RU 0.0 0 0 [swapper/1]
> 0 0 2 ffff9e944beaae00 RU 0.0 0 0 [swapper/2]
> 0 0 3 ffff9e944bff1700 RU 0.0 0 0 [swapper/3]
> 0 0 4 ffff9e944bff2e00 RU 0.0 0 0 [swapper/4]
> 0 0 5 ffff9e944bff0000 RU 0.0 0 0 [swapper/5]
> 0 0 6 ffff9e944bff4500 RU 0.0 0 0 [swapper/6]
> 0 0 7 ffff9e944bff5c00 RU 0.0 0 0 [swapper/7]
1 0 7 ffff9e944be80000 IN 0.1 78732 10020 systemd
2 0 5 ffff9e944be84500 IN 0.0 0 0 [kthreadd]
3 2 0 ffff9e944be85c00 ID 0.0 0 0 [rcu_gp]
4 2 0 ffff9e944be81700 ID 0.0 0 0 [rcu_par_gp]
5 2 0 ffff9e944be82e00 ID 0.0 0 0 [kworker/0:0]
6 2 0 ffff9e944bea4500 ID 0.0 0 0 [kworker/0:0H]
7 2 0 ffff9e944bea5c00 ID 0.0 0 0 [kworker/0:1]
8 2 7 ffff9e944bea1700 ID 0.0 0 0 [kworker/u16:0]
9 2 0 ffff9e944bea2e00 ID 0.0 0 0 [mm_percpu_wq]
runq： CPU上的运行队列情况
crash> runq
CPU 0 RUNQUEUE: ffff9e944da2a6c0
CURRENT: PID: 0 TASK: ffffffff97213780 COMMAND: "swapper/0"
RT PRIO_ARRAY: ffff9e944da2a980
[no tasks queued]
CFS RB_ROOT: ffff9e944da2a7f0
[no tasks queued]
CPU 1 RUNQUEUE: ffff9e944da6a6c0
CURRENT: PID: 3406 TASK: ffff9e9404475c00 COMMAND: "bash"
RT PRIO_ARRAY: ffff9e944da6a980
[no tasks queued]
CFS RB_ROOT: ffff9e944da6a7f0
[no tasks queued]
CPU 2 RUNQUEUE: ffff9e944daaa6c0
CURRENT: PID: 0 TASK: ffff9e944beaae00 COMMAND: "swapper/2"
RT PRIO_ARRAY: ffff9e944daaa980
[no tasks queued]
CFS RB_ROOT: ffff9e944daaa7f0
[no tasks queued]
sym： 查看符号列表信息
crash> sym usb
symbol not found: usb
possible alternatives:
ffffffff95aa7cd0 (T) kill_pid_usb_asyncio
ffffffff960d11c0 (t) quirk_gpu_usb_typec_ucsi
ffffffff960d11e0 (t) quirk_gpu_usb
ffffffff962f7430 (T) usb_ep_type_string
ffffffff962f7460 (T) usb_otg_state_string
ffffffff962f7490 (T) usb_speed_string
ffffffff962f74c0 (T) usb_state_string
ffffffff962f74f0 (T) usb_get_maximum_speed
ffffffff962f7560 (T) usb_get_dr_mode
ffffffff962f75d0 (T) usb_disabled
ffffffff962f76a0 (T) usb_find_common_endpoints
ffffffff962f7760 (T) usb_find_common_endpoints_reverse
ffffffff962f7840 (T) usb_ifnum_to_if
ffffffff962f78a0 (T) usb_altnum_to_altsetting
ffffffff962f7920 (t) usb_dev_prepare
ffffffff962f7930 (T) __usb_get_extra_descriptor
ffffffff962f79b0 (T) usb_find_interface
ffffffff962f7a30 (T) usb_put_dev
ffffffff962f7a50 (T) usb_put_intf
ffffffff962f7a70 (T) usb_for_each_dev
ffffffff962f7ad0 (t) usb_dev_restore
ffffffff962f7af0 (t) usb_dev_thaw
ffffffff962f7b10 (t) usb_dev_resume
ffffffff962f7b30 (t) usb_dev_poweroff
sys： 查看所有系统调用的实现
crash> sys -c
NUM SYSTEM CALL FILE AND LINE NUMBER
0 __x64_sys_read ../fs/read_write.c: 621
1 __x64_sys_write ../fs/read_write.c: 646
2 __x64_sys_open ../fs/open.c: 1104
3 __x64_sys_close ../fs/open.c: 1185
4 __x64_sys_newstat ../fs/stat.c: 337
5 __x64_sys_newfstat ../fs/stat.c: 375
6 __x64_sys_newlstat ../fs/stat.c: 348
7 __x64_sys_poll ../fs/select.c: 1047
8 __x64_sys_lseek ../fs/read_write.c: 322
9 __x64_sys_mmap ../arch/x86/kernel/sys_x86_64.c: 91
10 __x64_sys_mprotect ../mm/mprotect.c: 613
11 __x64_sys_munmap ../mm/mmap.c: 2869
12 __x64_sys_brk ../mm/mmap.c: 187
13 __x64_sys_rt_sigaction ../kernel/signal.c: 4234
14 __x64_sys_rt_sigprocmask ../kernel/signal.c: 3019
15 __ia32_sys_rt_sigreturn ../arch/x86/kernel/signal.c: 641
16 __x64_sys_ioctl ../fs/ioctl.c: 718
17 __x64_sys_pread64 ../fs/read_write.c: 672
18 __x64_sys_pwrite64 ../fs/read_write.c: 698
19 __x64_sys_readv ../fs/read_write.c: 1148
20 __x64_sys_writev ../fs/read_write.c: 1154
21 __x64_sys_access ../fs/open.c: 452
22 __x64_sys_pipe ../fs/pipe.c: 876
23 __x64_sys_select ../fs/select.c: 722
24 __ia32_sys_sched_yield ../kernel/sched/core.c: 5459
25 __x64_sys_mremap ../mm/mremap.c: 595
26 __x64_sys_msync ../mm/msync.c: 32
27 __x64_sys_mincore ../mm/mincore.c: 253
28 __x64_sys_madvise ../mm/madvise.c: 813
29 __x64_sys_shmget ../ipc/shm.c: 745
30 __x64_sys_shmat ../ipc/shm.c: 1591
31 __x64_sys_shmctl ../ipc/shm.c: 1194
32 __x64_sys_dup ../fs/file.c: 979
33 __x64_sys_dup2 ../fs/file.c: 949
task： 显示task_struct结构体的变量情况
crash> task
PID: 3406 TASK: ffff9e9404475c00 CPU: 1 COMMAND: "bash"
struct task_struct {
thread_info = {
flags = 2147483648,
status = 0
},
state = 0,
stack = 0xffffb4a581d08000,
usage = {
refs = {
counter = 2
}
},
flags = 4194560,
ptrace = 0,
wake_entry = {
next = 0x0
},
on_cpu = 1,
cpu = 1,
wakee_flips = 2,
wakee_flip_decay_ts = 4294925859,
last_wakee = 0xffff9e944b1a8000,
recent_used_cpu = 1,
wake_cpu = 1,
on_rq = 1,
prio = 120,
static_prio = 120,
normal_prio = 120,
rt_priority = 0,
sched_class = 0xffffffff96c3b000,
se = {
load = {
weight = 1048576,
inv_weight = 4194304
},
runnable_weight = 1048576,
run_node = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
group_node = {
next = 0xffff9e944da6b190,
prev = 0xffff9e944da6b190
},
timer：定时器的状态信息
crash> timer
JIFFIES
4294925895
TIMER_BASES[0][BASE_STD]: ffff9e944da1b600
EXPIRES TTE TIMER_LIST FUNCTION
4294925938 43 ffffffff97399ac0 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294925955 60 ffff9e943f52d448 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294926012 117 ffff9e9449012448 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294926299 404 ffff9e943f52f448 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294967500 41605 ffff9e944da16c20 ffffffff95a4b710 <mce_timer_fn>
TIMER_BASES[0][BASE_DEF]: ffff9e944da1c880
EXPIRES TTE TIMER_LIST FUNCTION
4294926000 105 ffffffff972e1740 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294926000 105 ffff9e944da279a0 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294967411 41516 ffff9e944da2a088 ffffffff95ab7180 <idle_worker_timeout>
TIMER_BASES[1][BASE_STD]: ffff9e944da5b600
EXPIRES TTE TIMER_LIST FUNCTION
4294925997 102 ffff9e94389ca348 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294926115 220 ffff9e944b0e4850 ffffffff9648f030 <neigh_timer_handler>
4294926224 329 ffff9e944b3e1448 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294967497 41602 ffff9e944da56c20 ffffffff95a4b710 <mce_timer_fn>
4296714848 1788953 ffff9e9437ef2d58 ffffffff96517b70 <tcp_keepalive_timer>
TIMER_BASES[1][BASE_DEF]: ffff9e944da5c880
EXPIRES TTE TIMER_LIST FUNCTION
4294929383 3488 ffffffff97440ea8 ffffffff95ab5ac0 <delayed_work_timer_fn>
4294967435 41540 ffff9e944bc18c48 ffffffff95ab7180 <idle_worker_timeout>
4294968970 43075 ffff9e944da6a088 ffffffff95ab7180 <idle_worker_timeout>
应用场景
log + bt + dis
使用log查看日志，bt查看堆栈情况，dis查看异常时的代码行及反汇编代码。
dev+irq+mod+mount+net+ps+runq+ipcs
查看系统的设备列表，中断分配，模块使用情况，网络配置情况，进程状态，cpu上的运行队列信 息,ipcs共享内存使用情况
