1. 介绍

本文主要介绍crash工具的使用以及常用的命令。crash工具，常用来分析内核的coredump以及应用的coredump，功能非常强大。

crash工具官方介绍

使用crash分析内核crash情况，需要准备以下内容：

1. 内核crash时生成的coredump文件或者raw data
2. 运行内核对应的符号表vmlinux
3. crash 工具

2. crash基本使用

2.1 解释命令

使用crash工具解析内核crash产生的raw data，命令如下：

crash cluster-vmlinux ramdump@0x57400000

解析成功后，会进入crash工具cmdline模式，如下图所示：

使用crash工具解析内核crash产生的coredump文件，命令如下：

crash vmlinux vmcore

解析成功后，也会进入crash工具的cmdline模式

2.2 crash工具基本输出

      KERNEL: cluster-vmlinux  [TAINTED]
   DUMPFILES: /var/tmp/ramdump_elf_Tp0C6d [temporary ELF header]
              ramdump
        CPUS: 6
        DATE: Mon Apr 18 11:33:58 CST 2022                      //panic时间
      UPTIME: 01:28:40                                          //panic时，已运行时间
LOAD AVERAGE: 1.31, 1.32, 1.29
       TASKS: 134                                               //panic时，进程总数
    NODENAME: x9h_ms                                            //硬件信息
     RELEASE: 4.14.61                                           //内核版本信息
     VERSION: #1 SMP PREEMPT Fri Apr 15 14:25:47 UTC 2022
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 544 MB                                            //Linux系统内存数
       PANIC: "sysrq: SysRq : Trigger a crash"                  //panic原因
         PID: 2496                                              //panic进程号
     COMMAND: "sh"                                              //panic进程名词
        TASK: ffff800023c5b800  [THREAD_INFO: ffff800023c5b800] //panic进程task和thread_info信息
         CPU: 5                                                 //系统cpu核数
       STATE: TASK_RUNNING (SYSRQ)                              //panic时任务状态

2.2.1 crash常用命令

1. help
   该命令是查看crash工具的帮助信息，譬如支持的命令、具体某个命令的使用方法等。

crash> help
*              extend         log            rd             task
alias          files          mach           repeat         timer
ascii          foreach        mod            runq           tree
bpf            fuser          mount          search         union
bt             gdb            net            set            vm
btop           help           p              sig            vtop
dev            ipcs           ps             struct         waitq
dis            irq            pte            swap           whatis
eval           kmem           ptob           sym            wr
exit           list           ptov           sys            q

crash version: 8.0.0++   gdb version: 10.2
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".

//查看某个命令使用方法
crash> help ps

NAME
  ps - display process status information

SYNOPSIS
  ps [-k|-u|-G|-y policy] [-s] [-p|-c|-t|-[l|m][-C cpu]|-a|-g|-r|-S|-A]
     [pid | task | command] ...

DESCRIPTION
  This command displays process status for selected, or all, processes
  in the system.  If no arguments are entered, the process data is
  is displayed for all processes.  Specific processes may be selected
  by using the following identifier formats:

       pid  a process PID.
      task  a hexadecimal task_struct pointer.
   command  a command name.  If a command name is made up of letters that
            are all numerical values, precede the name string with a "\".
            If the command string is enclosed within "'" characters, then
            the encompassed string must be a POSIX extended regular expression
            that will be used to match task names.

  The process list may be further restricted by the following options:

        -k  restrict the output to only kernel threads.
        -u  restrict the output to only user tasks.
        -G  display only the thread group leader in a thread group.
 -y policy  restrict the output to tasks having a specified scheduling policy
            expressed by its integer value or by its (case-insensitive) name;
            multiple policies may be entered in a comma-separated list:
              0 or NORMAL
              1 or FIFO
              2 or RR
              3 or BATCH
              4 or ISO
              5 or IDLE
              6 or DEADLINE

  The process identifier types may be mixed.  For each task, the following
  items are displayed:

    1. the process PID.
    2. the parent process PID.
    3. the CPU number that the task ran on last.
    4. the task_struct address or the kernel stack pointer of the process.
       (see -s option below)
    5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW, WA, PA, ID, NE).
    6. the percentage of physical memory being used by this task.
    7. the virtual address size of this task in kilobytes.
    8. the resident set size of this task in kilobytes.
    9. the command name.

  The default output shows the task_struct address of each process under a
  column titled "TASK".  This can be changed to show the kernel stack
  pointer under a column titled "KSTACKP".

       -s  replace the TASK column with the KSTACKP column.

  On SMP machines, the active task on each CPU will be highlighted by an
  angle bracket (">") preceding its information.  If the crash variable
  "offline" is set to "hide", the active task on an offline CPU will
  be highlighted by a "-" preceding its information.

  Alternatively, information regarding parent-child relationships,
  per-task time usage data, argument/environment data, thread groups,
  or resource limits may be displayed:

       -p  display the parental hierarchy of selected, or all, tasks.
       -c  display the children of selected, or all, tasks.
       -t  display the task run time, start time, and cumulative user
           and system times.
       -l  display the task's last-run timestamp value, using either the
           task_struct's last_run value, the task_struct's timestamp value
           or the task_struct's sched_entity last_arrival value, whichever
           applies, of selected, or all, tasks; the list is sorted with the
           most recently-run task (with the largest timestamp) shown first,
           followed by the task's current state.
       -m  similar to -l, but the timestamp value is translated into days,
           hours, minutes, seconds, and milliseconds since the task was
           last run on a cpu.
  -C cpus  only usable with the -l or -m options, dump the timestamp data
           in per-cpu blocks, where the cpu[s] can be specified as "1,3,5",
           "1-3", "1,3,5-7,10", "all", or "a" (shortcut for "all").
       -a  display the command line arguments and environment strings of
           selected, or all, user-mode tasks.
       -g  display tasks by thread group, of selected, or all, tasks.
       -r  display resource limits (rlimits) of selected, or all, tasks.
       -S  display a summary consisting of the number of tasks in a task state.
       -A  display only the active task on each cpu.

EXAMPLES
  Show the process status of all current tasks:
    crash> ps
       PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
    >     0      0   3  c024c000  RU   0.0     0     0  [swapper]
    >     0      0   0  c0dce000  RU   0.0     0     0  [swapper]
          0      0   1  c0fa8000  RU   0.0     0     0  [swapper]
    >     0      0   2  c009a000  RU   0.0     0     0  [swapper]
          1      0   1  c0098000  IN   0.0  1096   476  init
          2      1   1  c0090000  IN   0.0     0     0  [kflushd]
          3      1   1  c000e000  IN   0.0     0     0  [kpiod]
          4      1   3  c000c000  IN   0.0     0     0  [kswapd]
          5      1   1  c0008000  IN   0.0     0     0  [mdrecoveryd]
        253      1   2  fbc4c000  IN   0.0  1088   376  portmap

2. bt命令
   该命令是查看进程栈信息的，可以通过pid/cpu等选项指定相关信息

crash> bt
PID: 2496   TASK: ffff800023c5b800  CPU: 5   COMMAND: "sh"
 #0 [ffff000013573800] __crash_kexec at ffff00000817cee8
 #1 [ffff000013573890] (null) at 108142390
 #2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
 #3 [ffff000013573940] machine_restart at ffff000008085b94
 #4 [ffff000013573960] emergency_restart at ffff0000080fddc4
 #5 [ffff000013573970] panic at ffff0000080d8c04
 #6 [ffff000013573a50] die at ffff00000808ae10
 #7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
 #8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
 #9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
     PC: ffff0000086095b0  [sysrq_handle_crash+32]
     LR: ffff00000860959c  [sysrq_handle_crash+12]
     SP: ffff000013573d30  PSTATE: 60400145
    X29: ffff000013573d30  X28: ffff800023c5b800  X27: ffff000008cc1000
    X26: 0000000000000040  X25: 0000000000000124  X24: 0000000000000000
    X23: 0000000000000004  X22: ffff000009217000  X21: ffff000009217f10
    X20: 0000000000000063  X19: ffff00000919e000  X18: 0000000000000010
    X17: 000000000049b4d8  X16: ffff000008277b80  X15: ffffffffffffffff
    X14: ffff0000892fc7a7  X13: ffff0000092fc7b5  X12: ffff00000919e000
    X11: ffff00000917ae78  X10: ffff000008434f78   X9: 00000000ffffffd0
     X8: 0000000000000015   X7: 6767697254203a20   X6: 00000000000004a3
     X5: 0000000000000000   X4: 0000000000000000   X3: 0000000000000000
     X2: ffff80002731efb8   X1: 0000000000000000   X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
     PC: 0000ffff9e5aaeac   LR: 0000000000408cc4   SP: 0000ffffeea390d0
    X29: 0000ffffeea390d0  X28: 0000000000000000  X27: 0000000000000000
    X26: 00000000071a3cc5  X25: 000000000049b000  X24: 00000000071a6372
    X23: 0000000000000000  X22: 0000000000000001  X21: 00000000071a6370
    X20: 0000000000000002  X19: 0000000000000001  X18: 00000000000002c8
    X17: 000000000049b4d8  X16: 0000ffff9e5aae80  X15: 0000ffff9e4ebde0
    X14: 0000ffff9e4f92c8  X13: 000000000000270f  X12: 0101010101010101
    X11: 0000000000000000  X10: 0101010101010101   X9: fffffffffffffff0
     X8: 0000000000000040   X7: 7f7f7f7f7f7f7f7f   X6: 0080000080808080
     X5: 0000000000000000   X4: 00000000071a0063   X3: 0000ffff9e63f190
     X2: 0000000000000002   X1: 00000000071a6370   X0: 0000000000000001
    ORIG_X0: 0000000000000001  SYSCALLNO: 40  PSTATE: 80000000

3. set
   该命令获取crash的进程信息

crash> set
    PID: 2496
COMMAND: "sh"
   TASK: ffff800023c5b800  [THREAD_INFO: ffff800023c5b800]
    CPU: 5
  STATE: TASK_RUNNING (SYSRQ)

4. sym
   该命令查看符号和符号对应的地址信息，可以根据符号给出对应的地址，也可以根据给出的地址给出对应的符号

crash> sym proc_reg_write
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym ffff0000082ea1d0
ffff0000082ea1d0 (t) proc_reg_write /linux/fs/proc/inode.c: 224
crash> sym __log_buf
ffff0000092ff020 (b) __log_buf
crash> sym ffff0000092ff020
ffff0000092ff020 (b) __log_buf

5. ps
   查看系统的进程信息，类似linux的ps

crash> ps
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffff000009186300  RU   0.0       0      0  [swapper/0]
>     0      0   1  ffff800025ba9c00  RU   0.0       0      0  [swapper/1]
>     0      0   2  ffff800025baaa00  RU   0.0       0      0  [swapper/2]
>     0      0   3  ffff800025bab800  RU   0.0       0      0  [swapper/3]
>     0      0   4  ffff800025bac600  RU   0.0       0      0  [swapper/4]
      0      0   5  ffff800025bad400  RU   0.0       0      0  [swapper/5]
      1      0   2  ffff800025b70000  IN   0.1    1824   1276  init
      2      0   2  ffff800025b70e00  IN   0.0       0      0  [kthreadd]
      4      2   0  ffff800025b72a00  ID   0.0       0      0  [kworker/0:0H]
      6      2   0  ffff800025b74600  ID   0.0       0      0  [mm_percpu_wq]
      7      2   0  ffff800025b75400  IN   0.0       0      0  [ksoftirqd/0]
      8      2   5  ffff800025b76200  ID   0.0       0      0  [rcu_preempt]
      9      2   1  ffff800025b77000  ID   0.0       0      0  [rcu_sched]
     10      2   0  ffff800025ba8000  ID   0.0       0      0  [rcu_bh]
     11      2   0  ffff800025ba8e00  IN   0.0       0      0  [migration/0]
     12      2   0  ffff800025bae200  IN   0.0       0      0  [cpuhp/0]
     13      2   1  ffff800025baf000  IN   0.0       0      0  [cpuhp/1]
     14      2   1  ffff800025be8000  IN   0.0       0      0  [migration/1]
...

6. vtop
   根据一个虚拟地址转换成对应的物理地址

crash> vtop ffff800025065400
VIRTUAL           PHYSICAL
ffff800025065400  65065400

PAGE DIRECTORY: ffff00000938b000
   PGD: ffff00000938b800 => 673f8803
   PUD: ffff8000273f8000 => 673f7803
   PMD: ffff8000273f7940 => f8000065000f11
  PAGE: 65000000  (2MB)

     PTE        PHYSICAL  FLAGS
f8000065000f11  65000000  (VALID|SHARED|AF|NG|PXN|UXN|DIRTY)

      PAGE       PHYSICAL      MAPPING       INDEX CNT FLAGS
ffff7e0000941940 65065000 dead000000000400        0  0 fffc00000000000

7. ptov
   根据一个物理地址，转存成对应的内核态虚拟地址

crash> ptov 65065400
VIRTUAL           PHYSICAL
ffff800025065400  65065400

8. struct
   查看结构体信息，如成员偏移，某个结构体变量内存中的值等

crash> struct thread_info -o
struct thread_info {
   [0] unsigned long flags;
   [8] mm_segment_t addr_limit;
  [16] int preempt_count;
}
SIZE: 24

crash> struct thread_info -o -x
struct thread_info {
   [0x0] unsigned long flags;
   [0x8] mm_segment_t addr_limit;
  [0x10] int preempt_count;
}
SIZE: 0x18

crash> struct thread_info 0xffff000009186300
struct thread_info {
  flags = 32,
  addr_limit = 281474976710655,
  preempt_count = 65537
}

crash> task_struct.thread_info 0xffff000009186300 -p
  thread_info = {
    flags = 32,
    addr_limit = 281474976710655,
    preempt_count = 65537
  },

9. dis
   反汇编，根据给出的PC地址，反编译出汇编源码，一般结合bt提供的栈信息和寄存器信息和反编译的汇编源码信息分析。

crash> bt
PID: 2496   TASK: ffff800023c5b800  CPU: 5   COMMAND: "sh"
 #0 [ffff000013573800] __crash_kexec at ffff00000817cee8
 #1 [ffff000013573890] (null) at 108142390
 #2 [ffff000013573920] psci_sys_reset at ffff000008965ae0
 #3 [ffff000013573940] machine_restart at ffff000008085b94
 #4 [ffff000013573960] emergency_restart at ffff0000080fddc4
 #5 [ffff000013573970] panic at ffff0000080d8c04
 #6 [ffff000013573a50] die at ffff00000808ae10
 #7 [ffff000013573a90] __do_kernel_fault at ffff00000809feb8
 #8 [ffff000013573ac0] do_page_fault at ffff00000809ffdc
 #9 [ffff000013573b30] do_translation_fault at ffff0000080a0390
#10 [ffff000013573b40] do_mem_abort at ffff00000808130c
#11 [ffff000013573d20] el1_ia at ffff000008083050
     PC: ffff0000086095b0  [sysrq_handle_crash+32]
     LR: ffff00000860959c  [sysrq_handle_crash+12]
     SP: ffff000013573d30  PSTATE: 60400145
    X29: ffff000013573d30  X28: ffff800023c5b800  X27: ffff000008cc1000
    X26: 0000000000000040  X25: 0000000000000124  X24: 0000000000000000
    X23: 0000000000000004  X22: ffff000009217000  X21: ffff000009217f10
    X20: 0000000000000063  X19: ffff00000919e000  X18: 0000000000000010
    X17: 000000000049b4d8  X16: ffff000008277b80  X15: ffffffffffffffff
    X14: ffff0000892fc7a7  X13: ffff0000092fc7b5  X12: ffff00000919e000
    X11: ffff00000917ae78  X10: ffff000008434f78   X9: 00000000ffffffd0
     X8: 0000000000000015   X7: 6767697254203a20   X6: 00000000000004a3
     X5: 0000000000000000   X4: 0000000000000000   X3: 0000000000000000
     X2: ffff80002731efb8   X1: 0000000000000000   X0: 0000000000000001
#12 [ffff000013573d30] sysrq_handle_crash at ffff0000086095ac
#13 [ffff000013573d40] __handle_sysrq at ffff000008609b5c
#14 [ffff000013573d80] write_sysrq_trigger at ffff00000860a100
#15 [ffff000013573da0] proc_reg_write at ffff0000082ea22c
#16 [ffff000013573dc0] __vfs_write at ffff00000827760c
#17 [ffff000013573e40] vfs_write at ffff000008277900
#18 [ffff000013573e80] sys_write at ffff000008277bc4
#19 [ffff000013573ff0] el0_svc_naked at ffff000008083abc
     PC: 0000ffff9e5aaeac   LR: 0000000000408cc4   SP: 0000ffffeea390d0
    X29: 0000ffffeea390d0  X28: 0000000000000000  X27: 0000000000000000
    X26: 00000000071a3cc5  X25: 000000000049b000  X24: 00000000071a6372
    X23: 0000000000000000  X22: 0000000000000001  X21: 00000000071a6370
    X20: 0000000000000002  X19: 0000000000000001  X18: 00000000000002c8
    X17: 000000000049b4d8  X16: 0000ffff9e5aae80  X15: 0000ffff9e4ebde0
    X14: 0000ffff9e4f92c8  X13: 000000000000270f  X12: 0101010101010101
    X11: 0000000000000000  X10: 0101010101010101   X9: fffffffffffffff0
     X8: 0000000000000040   X7: 7f7f7f7f7f7f7f7f   X6: 0080000080808080
     X5: 0000000000000000   X4: 00000000071a0063   X3: 0000ffff9e63f190
     X2: 0000000000000002   X1: 00000000071a6370   X0: 0000000000000001
    ORIG_X0: 0000000000000001  SYSCALLNO: 40  PSTATE: 80000000

//查看栈数据
crash> bt -f 2
PID: 2      TASK: ffff800179d30000  CPU: 1   COMMAND: "kthreadd"
 #0 [ffff00000a56bd90] __switch_to at ffff000008086690
    ffff00000a56bd90: ffff00000a56bdb0 ffff000008ea5fe8 
    ffff00000a56bda0: ffff80017feae280 ffff800179d6e580 
 #1 [ffff00000a56bdb0] __schedule at ffff000008ea5fe4
    ffff00000a56bdb0: ffff00000a56be40 ffff000008ea6708 
    ffff00000a56bdc0: ffff800179d30000 ffff000009a2bda0 
    ffff00000a56bdd0: ffff000009b8df60 ffff8000f4276600 
    ffff00000a56bde0: 0000000000000000 ffff000008108b68 
    ffff00000a56bdf0: 0000000000000001 ffff000008108000 
    ffff00000a56be00: 0000000000000000 0000000000000000 
    ffff00000a56be10: ffff000008109f54 ffff00000a56be60 
    ffff00000a56be20: ffff000008109e80 80838e611e66f600 
    ffff00000a56be30: ffff000000000004 80838e611e66f600 
 #2 [ffff00000a56be40] schedule at ffff000008ea6704
    ffff00000a56be40: ffff00000a56be60 ffff000008109f54 
    ffff00000a56be50: ffff8000f4276628 ffff000008109e80 
 #3 [ffff00000a56be60] kthreadd at ffff000008109f50

//查看异常栈
crash> bt -e
PID: 1803   TASK: ffff8001794fab80  CPU: 5   COMMAND: "pvr_defer_free"

KERNEL-MODE EXCEPTION FRAME AT: ffff00000e7dbbc0
     PC: ffff00000876d444  [DeviceMemSet+140]
     LR: ffff0000087609c0  [_ZeroPageArray+120]
     SP: ffff00000e7dbd00  PSTATE: 20c00145
    X29: ffff00000e7dbd00  X28: ffff80015bc9fb00  X27: 00000000000186a0
    X26: 0000000000000000  X25: 00e8000000000f0f  X24: 0000000000000080
    X23: ffff80012d0db000  X22: ffff00001fb7c000  X21: 0000000000000000
    X20: 0000000000080000  X19: ffff00001fbaf000  X18: 0000ffffac000bcc
    X17: 0000ffff97c22ec8  X16: ffff00000818ded0  X15: 0000ffffac000bc8
    X14: 0140000000000000  X13: ffff00001fbfc000  X12: 0000000000000000
    X11: 0000000000000000  X10: 0000000000000040   X9: 0040000000000041
     X8: 0040000000000001   X7: 0000000000000001   X6: 000000017fffd7e8
     X5: ffff8001398feb98   X4: ffff8001398feb98   X3: ffff00001fbfbfff
     X2: 0000000000000000   X1: 0000000000000000   X0: ffff00001fbfc000

10. irq
    查看系统的中断信息

crash> irq
 IRQ   IRQ_DESC/_DATA      IRQACTION      NAME
  0       (unused)          (unused)
  1   ffff800025807400      (unused)
  2   ffff800025807800      (unused)
  3   ffff800025807a00      (unused)
  4   ffff800025807c00  ffff80002588df00  "arch_timer"
  5   ffff800025807e00      (unused)
  6   ffff800024c7e400      (unused)
  7   ffff800024c7e600  ffff800024fcc100  "ttyS0"
  8   ffff800024c7e800      (unused)
  9   ffff800024c7ea00  ffff800024b10680  "30b00000.i2c"
 10   ffff800024c7ec00  ffff800024efc200  "30b10000.i2c"
 11   ffff800024c7ee00  ffff800024c72f00  "30b30000.i2c"
 ...

11. files
    查看进程打开的文件

crash> files
PID: 2496   TASK: ffff800023c5b800  CPU: 5   COMMAND: "sh"
ROOT: /    CWD: /home/root
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR  /dev/ttyS0
  1 ffff800024de9500 ffff8000239b29c0 ffff80002399e508 REG  /proc/sysrq-trigger
  2 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR  /dev/ttyS0
 10 ffff8000236a3b00 ffff8000245ff300 ffff800025095668 CHR  /dev/tty
 11 ffff800023772500 ffff8000246089c0 ffff8000240b2f28 CHR  /dev/ttyS0

 crash> foreach files -R /dev/console       //结合foreach命令搜索打开/dev/console文件的进程
PID: 2331   TASK: ffff800024a1b800  CPU: 2   COMMAND: "messagecenter"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2460   TASK: ffff800024f8aa00  CPU: 5   COMMAND: "adbd"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2462   TASK: ffff80002421f000  CPU: 2   COMMAND: "usb ffs open"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2463   TASK: ffff80002421aa00  CPU: 5   COMMAND: "server socket"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2464   TASK: ffff800024a1aa00  CPU: 5   COMMAND: "->transport"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2465   TASK: ffff800024a1c600  CPU: 5   COMMAND: "<-transport"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  1 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800022ec1100 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

PID: 2478   TASK: ffff800024a1d400  CPU: 1   COMMAND: "start_getty"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  1 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console
  2 ffff800023772c00 ffff8000245ff3c0 ffff800025095928 CHR  /dev/console

12. dev
    查看系统注册的设备信息

crash> dev
CHRDEV    NAME                 CDEV        OPERATIONS
   1      mem            ffff8000253f5b00  memory_fops
   2      pty            ffff800024a6ed80  tty_fops
   3      ttyp           ffff8000249e3280  tty_fops
   4      /dev/vc/0      ffff00000934b7a8  console_fops
   4      tty            ffff80002487e180  tty_fops
   4      ttyS           ffff800024f02e80  tty_fops
   5      /dev/tty       ffff00000934a2d0  tty_fops
   5      /dev/console   ffff00000934a338  console_fops
   5      /dev/ptmx      ffff00000934a4b0  ptmx_fops
...
BLKDEV    NAME                GENDISK      OPERATIONS
 259      blkext              (none)
   7      loop           ffff800024e85000  lo_fops
   8      sd                  (none)
   9      md             ffff800023db8800  md_fops
  31      mtdblock            (none)
...

13. mount
    查看系统挂载的文件系统

crash> mount
     MOUNT           SUPERBLK     TYPE   DEVNAME   DIRNAME
ffff800025b4c000 ffff800025812000 rootfs rootfs    /
ffff800023d0c000 ffff800023db9000 ext4   /dev/root /
ffff800023d0c1c0 ffff8000250b8000 devtmpfs devtmpfs /dev
ffff800023e0c000 ffff800025815000 proc   proc      /proc
ffff800023e0c1c0 ffff800023e60000 sysfs  sysfs     /sys
ffff800023e0c380 ffff800025817000 debugfs debugfs  /sys/kernel/debug
ffff800023e0c540 ffff80002403f800 configfs configfs /sys/kernel/config
ffff800023e0c700 ffff800023e60800 tmpfs  tmpfs     /run
ffff800023e0c8c0 ffff800023e61000 tmpfs  tmpfs     /var/volatile
ffff8000250a8540 ffff8000234e4000 devpts devpts    /dev/pts
ffff8000250a8700 ffff800023e61000 tmpfs  tmpfs     /var/lib
ffff800023e0ca80 ffff80002403f800 configfs none    /config
ffff800023e0cc40 ffff800023e62000 functionfs adb   /dev/usb-ffs/adb
ffff800023d0c380 ffff800024e19800 tracefs tracefs  /sys/kernel/debug/tracing
ffff8000250a88c0 ffff8000234e6800 ext4   /dev/block/by-name/userdata /data

14. kmem
    查看内核态下内存的使用情况，如slab信息、vmalloc信息等

crash> kmem -i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM   120690     471.4 MB         ----
         FREE    76395     298.4 MB   63% of TOTAL MEM
         USED    44295       173 MB   36% of TOTAL MEM
       SHARED     7202      28.1 MB    5% of TOTAL MEM
      BUFFERS      649       2.5 MB    0% of TOTAL MEM
       CACHED    10890      42.5 MB    9% of TOTAL MEM
         SLAB     7869      30.7 MB    6% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

   TOTAL SWAP        0            0         ----
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE        0            0    0% of TOTAL SWAP

 COMMIT LIMIT    60345     235.7 MB         ----
    COMMITTED    43807     171.1 MB   72% of TOTAL LIMIT
crash> kmem -s  //查看kmalloc信息
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff800024ada180      136       3052      3060    102     4k  ext4_groupinfo_1k
ffff800023c96000      144          3        28      1     4k  ext4_groupinfo_4k
ffff800024164480     1408          2        46      2    32k  UDPv6
ffff800024164300      232          0         0      0     4k  tw_sock_TCPv6
ffff800024164180      304          0         0      0     8k  request_sock_TCPv6
ffff800024164000     2112          5        75      5    32k  TCPv6
ffff8000241c7e00      224        233       882     49     4k  nf_conntrack_expect
ffff8000241c7c80      256          0         0      0     8k  nf_conntrack
ffff8000241c7b00      312          0         0      0     8k  ashmem_area_cache
ffff8000241c7980     3312          0         0      0    32k  kcopyd_job
ffff8000241c7800      120          0         0      0     4k  dm_rq_target_io
ffff8000241c7680       72          0         0      0     4k  isp1760_qtd
ffff8000241c7500      112         13       216      6     4k  cfq_io_cq
ffff8000241c7380      240         14       102      6     4k  cfq_queue
ffff8000241c7200      216          0         0      0     4k  bsg_cmd
ffff8000241c7080      896          1        17      1    16k  mqueue_inode_cache
ffff8000241c6f00      656          0         0      0    16k  v9fs_inode_cache
...
crash> kmem -S    //查看所有slab对象信息
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff800024ada180      136       3052      3060    102     4k  ext4_groupinfo_1k
CPU 0 KMEM_CACHE_CPU:
  ffff7dffbfecbfb0
CPU 0 SLAB:
  (empty)
CPU 0 PARTIAL:
  (empty)
CPU 1 KMEM_CACHE_CPU:
  ffff7dffbfee4fb0
CPU 1 SLAB:
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  ffff7e0000721f80  ffff80001c87e000     0     30         22     8
  FREE / [ALLOCATED]
  [ffff80001c87e000]
  [ffff80001c87e088]
...
crash> kmem -p     //查看page信息
      PAGE       PHYSICAL      MAPPING       INDEX CNT FLAGS
ffff7e0000000000 40000000                0        0  0 0
ffff7e0000000040 40001000                0        0  0 0
ffff7e0000000080 40002000                0        0  0 0
ffff7e00000000c0 40003000                0        0  0 0
ffff7e0000000100 40004000                0        0  0 0
ffff7e0000000140 40005000                0        0  0 0
...

15. foreach
    遍历系统上task的信息，如查看系统上处于UNINTERRUPT状态的task的栈信息

crash> foreach UN bt

PID: 561    TASK: ffff8000e3d33a00  CPU: 0   COMMAND: "kworker/u12:1"
 #0 [ffff00002038b5f0] __switch_to at ffff000008086690
 #1 [ffff00002038b610] __schedule at ffff000008ea5fe4
 #2 [ffff00002038b6a0] schedule at ffff000008ea6704
 #3 [ffff00002038b6c0] wait_transaction_locked at ffff0000083b5d8c
 #4 [ffff00002038b730] add_transaction_credits at ffff0000083b602c
 #5 [ffff00002038b7a0] start_this_handle at ffff0000083b624c
 #6 [ffff00002038b840] jbd2__journal_start at ffff0000083b6a50
 #7 [ffff00002038b8a0] __ext4_journal_start_sb at ffff0000083564d8
 #8 [ffff00002038b8f0] ext4_writepages at ffff0000083760cc
 #9 [ffff00002038ba40] do_writepages at ffff000008225400
#10 [ffff00002038bac0] __writeback_single_inode at ffff0000082ee774
#11 [ffff00002038bb10] writeback_sb_inodes at ffff0000082ef180
#12 [ffff00002038bc10] __writeback_inodes_wb at ffff0000082ef4b0
#13 [ffff00002038bc60] wb_writeback at ffff0000082ef828
#14 [ffff00002038bd10] wb_workfn at ffff0000082f0400
#15 [ffff00002038bdd0] process_one_work at ffff000008101950
#16 [ffff00002038be20] worker_thread at ffff000008101c6c
#17 [ffff00002038be70] kthread at ffff000008108c9c

PID: 814    TASK: ffff8001793f8000  CPU: 0   COMMAND: "saftey_heart_be"
 #0 [ffff00000c053c10] __switch_to at ffff000008086690
 #1 [ffff00000c053c30] __schedule at ffff000008ea5fe4
 #2 [ffff00000c053cc0] schedule at ffff000008ea6704
 #3 [ffff00000c053ce0] schedule_timeout at ffff000008eaa428
 #4 [ffff00000c053d80] msleep at ffff000008178d64
 #5 [ffff00000c053da0] saftey_hb_thread at ffff000008ad925c
 #6 [ffff00000c053e70] kthread at ffff000008108c9c

PID: 1767   TASK: ffff800020776580  CPU: 0   COMMAND: "AVDemuxer::Read"
 #0 [ffff000015dc3780] __switch_to at ffff000008086690
 #1 [ffff000015dc37a0] __schedule at ffff000008ea5fe4
 #2 [ffff000015dc3830] schedule at ffff000008ea6704
 #3 [ffff000015dc3850] wait_transaction_locked at ffff0000083b5d8c
 #4 [ffff000015dc38c0] add_transaction_credits at ffff0000083b602c
 #5 [ffff000015dc3930] start_this_handle at ffff0000083b624c
 #6 [ffff000015dc39d0] jbd2__journal_start at ffff0000083b6a50
 #7 [ffff000015dc3a30] __ext4_journal_start_sb at ffff0000083564d8
 #8 [ffff000015dc3a80] ext4_da_write_begin at ffff000008378448
 #9 [ffff000015dc3c10] generic_perform_write at ffff00000821263c
#10 [ffff000015dc3ca0] __generic_file_write_iter at ffff000008215884
#11 [ffff000015dc3cf0] ext4_file_write_iter at ffff000008362760
#12 [ffff000015dc3d80] __vfs_write at ffff0000082b5180
#13 [ffff000015dc3e30] vfs_write at ffff0000082b53e0
#14 [ffff000015dc3e70] sys_write at ffff0000082b56f8
#15 [ffff000015dc3ff0] el0_svc_naked at ffff000008083af4
     PC: 0000ffffa8eaee60   LR: 0000ffffa8eaee48   SP: 0000ffff93ffcdc0
    X29: 0000ffff93ffcdc0  X28: 0000000000008000  X27: 0000000002faf080
    X26: 0000000000000000  X25: 0000000000000000  X24: 0000000000001000
    X23: 0000ffff84004de0  X22: 0000000000001000  X21: 0000ffff84004de0
    X20: 0000000000001000  X19: 000000000000001f  X18: 0000ffffa8f5ba70
    X17: 0000ffffa8e50398  X16: 0000ffffa9b00420  X15: 0000074895555910
    X14: 0000000000000000  X13: 52e6647d9bae90b9  X12: 936c08cc82e6992c
    X11: 2de7fcffe5393767  X10: 41a928c963510c66   X9: 44377654e95c11b2
     X8: 0000000000000040   X7: 042d1f3b698dddd1   X6: 0000000000000000
     X5: 0000ffff93ffede0   X4: 00000000ffffffbb   X3: 0000000000000000
     X2: 0000000000001000   X1: 0000ffff84004de0   X0: 000000000000001f
    ORIG_X0: 000000000000001f  SYSCALLNO: 40  PSTATE: 80000000
...

16. p
    打印变量或者表达式的值

crash> p jiffies
jiffies = $1 = 4298283231

crash> p vm_event_states
PER-CPU DATA TYPE:
  struct vm_event_state vm_event_states;
PER-CPU ADDRESSES:
  [0]: ffff80017fe91ab8
  [1]: ffff80017feaaab8
  [2]: ffff80017fec3ab8
  [3]: ffff80017fedcab8
  [4]: ffff80017fef5ab8
  [5]: ffff80017ff0eab8

//查看cpu1对应的变量的值
crash> p vm_event_states:1
per_cpu(vm_event_states, 1) = $3 = {
  event = {318552, 739584, 0, 0, 558108, 2889268, 0, 0, 0, 0, 0, 0, 0, 4189877, 38713, 21935, 255, 2454551, 259, 0, 27971, 34455, 0, 63883, 0, 0, 0, 0, 62080, 233, 2, 1, 14, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4576, 0, 0, 0, 46844, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 622, 34, 48, 0, 0, 0, 0, 0, 2119, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0}
}

//查看数组元素
crash> p &page_wait_table
$4 = (wait_queue_head_t (*)[256]) 0xffff000009a07d80 <page_wait_table>
crash> p page_wait_table[4]
$5 = {
  lock = {
    {
      rlock = {
        raw_lock = {
          owner = 6378,
          next = 6378
        }
      }
    }
  },
  head = {
    next = 0xffff000009a07de8 <page_wait_table+104>,
    prev = 0xffff000009a07de8 <page_wait_table+104>
  }
}

17. list
    遍历由指针或者链表连接起来的内容

//查看挂在panic_notifier_list的notify

crash> p &panic_notifier_list                                                  
$3 = (struct atomic_notifier_head *) 0xffff000009b8a670 <panic_notifier_list>
crash> struct atomic_notifier_head                                             
struct atomic_notifier_head {
    spinlock_t lock;
    struct notifier_block *head;
}
SIZE: 16
crash> struct notifier_block
struct notifier_block {
    notifier_fn_t notifier_call;
    struct notifier_block *next;
    int priority;
}
SIZE: 24
crash> list atomic_notifier_head.head 0xffff000009b8a670 -s notifier_block.next
ffff000009b8a670
  next = 0xffff000009afaad0 <sdrv_wdt_panic_event_nb>,
ffff000009afaad0
  next = 0xffff000009a3b540 <trace_panic_notifier>,
ffff000009a3b540
  next = 0xffff000009a322a0 <rcu_panic_block>,
ffff000009a322a0
  next = 0xffff000009a3ad50 <panic_block>,
ffff000009a3ad50
  next = 0xffff000009aa5b80 <crashdump_panic_event_nb>,
ffff000009aa5b80
  next = 0xffff000009a1b190 <kernel_offset_notifier>,
ffff000009a1b190
  next = 0xffff000009a1b588 <cpu_hwcaps_notifier>,
ffff000009a1b588
  next = 0xffff000009a1d3b0 <mem_limit_notifier>,
ffff000009a1d3b0
  next = 0xffff000009afd460 <heartbeat_panic_nb>,
ffff000009afd460
  next = 0x0,