crash

发表于 | 更新于 | 阅读次数：

ubuntu 版本: Ubuntu 22.04.1 LTS 5.15.0-58.64-generic 5.15.74

crash白皮书

crash github

Ubuntu安装Kernel-debuginfo

配置安装源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
baoze@baoze:~/workspace$ cat <<EOF | sudo tee /etc/apt/sources.list.d/ddebs.list
deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
#deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse
EOF
baoze@baoze:~/workspace$ sudo apt update   #### 更新时提示没有公钥无法验证签名，需要添加公钥信息
......
W: GPG error: http://ddebs.ubuntu.com jammy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY C8CAB6595FDFF622
E: The repository 'http://ddebs.ubuntu.com jammy Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
......
baoze@baoze:~/workspace$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
baoze@baoze:~/workspace$ sudo apt update
baoze@baoze:~/workspace$ sudo apt install linux-image-unsigned-$(uname -r)-dbgsym

直接下载安装

到网站 http://ddebs.ubuntu.com/pool/main/l/linux/ 下载对应的debug-info包，然后进行安装，vmlinux默认安装在/usr/lib/debug/boot/vmlinux-5.15.0-58-generic

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
baoze@baoze:~/workspace$ wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-unsigned-5.15.0-58-generic-dbgsym_5.15.0-58.64_amd64.ddeb
baoze@baoze:~/workspace$ sudo dpkg -i linux-image-unsigned-5.15.0-58-generic-dbgsym_5.15.0-58.64_amd64.ddeb
baoze@baoze:~/workspace$ sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-58-generic 

crash 8.0.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...


      KERNEL: /usr/lib/debug/boot/vmlinux-5.15.0-58-generic
    DUMPFILE: /proc/kcore
        CPUS: 2
        DATE: Sat Jan 14 03:00:56 UTC 2023
      UPTIME: 00:33:55
LOAD AVERAGE: 0.77, 0.64, 0.56
       TASKS: 441
    NODENAME: baoze
     RELEASE: 5.15.0-58-generic
     VERSION: #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023
     MACHINE: x86_64  (2399 Mhz)
      MEMORY: 8 GB
         PID: 3333
     COMMAND: "crash"
        TASK: ffff9a01d1994b00  [THREAD_INFO: ffff9a01d1994b00]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash>

安装crash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
baoze@baoze:~/workspace$ sudo apt install crash
baoze@baoze:~/workspace$ sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-58-generic 

crash 8.0.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...


      KERNEL: /usr/lib/debug/boot/vmlinux-5.15.0-58-generic
    DUMPFILE: /proc/kcore
        CPUS: 2
        DATE: Sat Jan 14 03:00:56 UTC 2023
      UPTIME: 00:33:55
LOAD AVERAGE: 0.77, 0.64, 0.56
       TASKS: 441
    NODENAME: baoze
     RELEASE: 5.15.0-58-generic
     VERSION: #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023
     MACHINE: x86_64  (2399 Mhz)
      MEMORY: 8 GB
         PID: 3333
     COMMAND: "crash"
        TASK: ffff9a01d1994b00  [THREAD_INFO: ffff9a01d1994b00]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash>

crash常用调试命令

命令 说明
help 查看<某个命令>帮助信息
log 查看系统的日志
bt 查看堆栈信息
set 切换调试进程
struct task_struct ffff8b7df3cdae00 -x 把指定地址的内容以task_struct结构体解析打印，如果不带地址会显示结构体定义和大小
dis -r ffffffff9a6010ae dis命令进行返汇编，查看对应地址的代码逻辑
ps 查看所有进程信息
mod 查看当前加载的module，通过其他参数也可以加载对应的module
files 查看打开的文件信息
p init_task 查看init_task变量的值
vm 查看虚拟内存
list 查看链表信息，可以遍历数据结构中的链表数据

list命令使用

list [[-o] offset][-e end][-[s|S] struct[.member[,member] [-l offset]] -[x|d]] [-r|-B] [-h [-O head_offset]|-H] start

list命令解析链表的内容。内核中链表的方式通常有两种

  • 包含next指针的单链表
  • 使用struct list_head的双链表

-o参数： 结构体中执行next指针的偏移量（默认为0），通常可以采用struct.member来表示。-o可以省略输入
-e参数： list的结束地址，通常情况下crash根据链表的规则自动结束
-s参数： 需要输出的结构体中的成员，采用struct.member1,member2来表示，需要输出多个成员时，用,隔开。如果不输入该参数，则只打印地址信息。
-S参数： 类似于-s，但不是解析gdb输出，而是直接从内存中读取成员值，因此该命令对于1-、2-、4-和8字节成员的工作速度要快得多
-x参数： 将默认输出格式改为十六进制格式
-d参数： 使用十进制格式覆盖默认输出格式
-r参数： 对于一个链接到list_head结构的列表，使用“prev”指针而不是“next”，以相反的顺序遍历列表
start参数： 第一个数据结构的地址。可以用十六进制的形式表示，也可以用表达式求值为一个地址，它取决于-h或-h选项是否在前面
-h start参数： 包含嵌入list_head的数据结构的地址。
-H start参数： list_head结构的地址，通常是外部独立的list_head()的地址。

示例一

全局变量struct file_system_type file_systems通过next指针来保存下一个文件系统的类型的数据

1
2
3
struct file_system_type {
  struct file_system_type * next;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
crash> p file_systems
file_systems = $5 = (struct file_system_type *) 0xffffffffa3065620 <sysfs_fs_type>
crash> list file_system_type.next -s file_system_type.name,fs_flags 0xffffffffa3065620
ffffffffa3065620
  name = 0xffffffffa264a1c8 "sysfs",
  fs_flags = 8,
ffffffffa300b760
  name = 0xffffffffa267e0ef "tmpfs",
  fs_flags = 8200,
ffffffffa30aebc0
  name = 0xffffffffa260b14a "bdev",
  fs_flags = 0,
......

示例二：

全局变量super_blocks是struct list_head类型，链表中保存的是struct super_block类型的数据。

1
2
3
4
5
6
static LIST_HEAD(super_blocks);
struct super_block {
	struct list_head	s_list;
  struct file_system_type	*s_type;
  ......
}
1
2
3
4
5
6
7
8
9
10
11
crash> p super_blocks
super_blocks = $7 = {
  next = 0xffff9a01c004b800,
  prev = 0xffff9a01cbe74800
}
crash> list super_block.s_list -s super_block.s_type -H super_blocks
ffff9a01c004b800
  s_type = 0xffffffffa300b760 <shmem_fs_type>,
ffff9a01c004c000
  s_type = 0xffffffffa2e1aea0 <rootfs_fs_type>,
......

示例三：

struct super_block中有个成员 struct list_head s_mounts，该成员作为链表头，通过struct mount->mnt_instance成员挂了struct mount结构体。

1
2
3
4
5
6
7
8
9
10
11
12
13
crash> list -o mount.mnt_instance -s mount.mnt_mp,mnt_mountpoint -O super_block.s_mounts -h 0xffff90ab00b06800
ffff90ab001ff3c0
  mnt_mp = 0x0,
  mnt_mountpoint = 0xffff90ab00440240,
ffff90ab00b2cc80
  mnt_mp = 0xffff90ab00b30d00,
  mnt_mountpoint = 0xffff90ab00440840,
ffff90ab00b2c640
  mnt_mp = 0xffff90ab01293640,
  mnt_mountpoint = 0xffff90ab1bb3e600,
ffff90ab1ac48280
  mnt_mp = 0xffff90ab01293640,
  mnt_mountpoint = 0xffff90ab1bb3e600,