Linux 内核 crash 时 Oops(Kernel Panic)信息解读

30 分钟阅读次阅读

日志信息

内核崩溃时，使用 dmesg 查看，会打印如下的错误信息

[   65.086700] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000100
[   65.092841] Mem abort info:
[   65.093103]   ESR = 0x96000005
[   65.093383]   EC = 0x25: DABT (current EL), IL = 32 bits
[   65.093889]   SET = 0, FnV = 0
[   65.094174]   EA = 0, S1PTW = 0
[   65.094453] Data abort info:
[   65.094708]   ISV = 0, ISS = 0x00000005
[   65.095087]   CM = 0, WnR = 0
[   65.095366] user pgtable: 4k pages, 39-bit VAs, pgdp=00000001f973d000
[   65.095976] [0000000000000100] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
[   65.096755] Internal error: Oops: 96000005 [#1] SMP
[   65.097195] Modules linked in:
[   65.097481] CPU: 1 PID: 327 Comm: wpa_supplicant Not tainted 5.10.198-gce7d97244615-dirty #80
[   65.098233] Hardware name: Rockchip RK3568 EVB1 DDR4 V10 Board (DT)
[   65.098785] pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
[   65.099320] pc : wl_cfg80211_add_if+0x440/0xd90
[   65.099724] lr : wl_cfg80211_add_if+0x6a8/0xd90
[   65.100125] sp : ffffffc00cceb780
[   65.100424] x29: ffffffc00cceb780 x28: ffffff81f9673700
[   65.100897] x27: 0000000000000000 x26: 0000000000000000
[   65.101369] x25: ffffff81f78db800 x24: ffffffc00a2b2000
[   65.101839] x23: ffffffc0095be588 x22: ffffff81fa21d000
[   65.102310] x21: 0000000000000007 x20: ffffff800c151800
[   65.102781] x19: ffffffc0095bf268 x18: 0000000000000030
[   65.103252] x17: 0000000000000000 x16: 0000000000000000
[   65.103722] x15: ffffffffffffffff x14: ffffffc00a480208
[   65.104193] x13: 0000000000000006 x12: ffffffc00a4801fe
[   65.104663] x11: 695f6464615f3131 x10: ffffffc00a153258
[   65.105134] x9 : 0000000000000000 x8 : 69662065726f6665
[   65.105604] x7 : 62204c4c554e2073 x6 : ffffff800f00620a
[   65.106074] x5 : 0000000000000031 x4 : 0000000000000008
[   65.106545] x3 : 000000000000000a x2 : 0000000000000000
[   65.107016] x1 : 0000000000000000 x0 : 0000000000000023
[   65.107487] Call trace:
[   65.107711]  wl_cfg80211_add_if+0x440/0xd90
[   65.108081]  wl_cfg80211_add_virtual_iface+0x154/0x274
[   65.108543]  nl80211_new_interface+0x140/0x3f0
[   65.108936]  genl_family_rcv_msg_doit+0xc8/0x150
[   65.109350]  genl_rcv_msg+0xe4/0x1e0
[   65.109672]  netlink_rcv_skb+0x5c/0x130
[   65.110017]  genl_rcv+0x38/0x50
[   65.110294]  netlink_unicast+0x258/0x340
[   65.110639]  netlink_sendmsg+0x198/0x3f0
[   65.110987]  ____sys_sendmsg+0x1e0/0x240
[   65.111332]  ___sys_sendmsg+0x7c/0xc0
[   65.111654]  __sys_sendmsg+0x68/0xc4
[   65.111977]  __arm64_sys_sendmsg+0x24/0x30
[   65.112349]  el0_svc_common.constprop.0+0x68/0x1c4
[   65.112774]  do_el0_svc+0x1c/0x2c
[   65.113075]  el0_svc+0x20/0x30
[   65.113353]  el0_sync_handler+0xb0/0xb4
[   65.113699]  el0_sync+0x1a0/0x1c0
[   65.113999]
               PC: 0xffffffc008bdce60:
[   65.114433] cc60  d2800001 52800000 90006ba3 91086063 97fe4a30 52800002 d0006c23 d2800001
[   65.115163] cc80  91160063 52800000 97fe4a2a 14000068 b94e4300 370035e0 d2800019 d5384100
[   65.115889] cca0  f9405fe1 f9426002 eb020021 d2800002 54002b21 aa1903e0 a94153f3 a9425bf5
[   65.116614] ccc0  a94363f7 a9446bf9 a8cc7bfd d50323bf d65f03c0 b94e4300 3607fe00 913382f7
[   65.117339] cce0  90006ba0 aa1703e1 91090000 94204db6 d2800019 d0006c20 91046000 94204db2
[   65.118064] cd00  97fe49b8 aa0003e4 aa1703e5 52800002 d2800001 52800000 90006ba3 91086063
[   65.118790] cd20  97fe4a04 52800002 d0006c23 d2800001 9104e063 52800000 97fe49fe 17ffffd8
[   65.119516] cd40  913382f9 90006ba0 aa1903e1 91078000 f90043e6 94204d9c d0006c20 91100000
[   65.120242] cd60  94204d99 97fe499f aa0003e4 aa1903e5 52800002 d2800001 52800000 90006ba3
[   65.120971] cd80  91086063 97fe49eb 52800000 52800002 d0006c23 d2800001 9110a063 97fe49e5
[   65.121697] cda0  91404680 f94043e6 395ca000 350014e0 b94e4300 121f0001 35003321 37003000
[   65.122423] cdc0  12804160 b9009be0 7940ebe3 2a1503e2 f9416ec0 52800041 97fffd3e b9409be0
[   65.123148] cde0  3108301f 54000120 aa1603e0 52801fe1 97fff558 f9403ff3 aa1303e0 97fdf879
[   65.123874] ce00  72001c1f 54001b40 f94037e0 d2800019 94213bd4 17ffffa2 710026bf 54fffce1
[   65.124599] ce20  350039a1 92800bd9 370035e0 91404694 7940ebe3 52800021 aa1903e0 2a1503e2
[   65.125325] ce40  39691284 0b010084 39291284 97fffd21 b94e4300 37083160 f9401321 b4001061
[   65.126051] ce60* b9410021 350005a1 37000280 b9400b21 34000061 39691281 34001461 37083a80
[   65.126776] ce80  f94037e0 94213bb7 17ffff85 3900001f f94043e0 b5ffe880 1400008f b40011f9
[   65.127502] cea0  910273e6 b9400320 b9009fe0 79400b20 790008c0 17ffff42 913382f3 90006ba0
[   65.128228] cec0  aa1303e1 91090000 94204d3f d0006c20 91196000 94204d3c 97fe4942 aa0003e4
[   65.128953] cee0  aa1303e5 52800002 d2800001 52800000 90006ba3 91086063 97fe498e 52800000
[   65.129679] cf00  52800002 d0006c23 d2800001 911a6063 97fe4988 b94e4300 b9400b21 35fffac1
[   65.130405] cf20  3607fae0 913382f3 90006ba0 aa1303e1 91090000 94204d24 d0006c20 911b6000
[   65.131132] cf40  94204d21 97fe4927 aa0003e4 aa1303e5 52800002 d2800001 52800000 90006ba3
[   65.131858] cf60  91086063 97fe4973 52800000 52800002 d0006c23 d2800001 911c4063 97fe496d
[   65.132583] cf80  b94e4300 17ffffbc 913382f9 90006ba8 91078108 aa1903e1 aa0803e0 f90047e6
[   65.133310] cfa0  94204d09 d0006c20 91066000 94204d06 97fe490c aa0003e4 90006ba7 910860e7
[   65.134036] cfc0  aa1903e5 aa0703e3 52800002 d2800001 52800000 f90043e7 97fe4956 52800002
[   65.134762] cfe0  d2800001 52800000 d0006c23 91070063 97fe4950 b94e4300 90006ba2 710022bf
[   65.135487] d000  91078048 121f0001 a9481be7 54ffde21 35003a01 d503201f 37001020 370816a0
[   65.136212] d020  b9409be0 3108301f 54ffed00 12800240 b9009be0 17ffff65 35ffe841 91404681
[   65.136941] d040  395ca021 34ffebc1 aa0603e3 aa1303e2 910263e4 2a1503e1 aa1403e0 97ff7c81
[   65.137670]
               LR: 0xffffffc008bdd0c8:
[   65.138104] cec8  94204d3f d0006c20 91196000 94204d3c 97fe4942 aa0003e4 aa1303e5 52800002
[   65.138830] cee8  d2800001 52800000 90006ba3 91086063 97fe498e 52800000 52800002 d0006c23
[   65.139556] cf08  d2800001 911a6063 97fe4988 b94e4300 b9400b21 35fffac1 3607fae0 913382f3
[   65.140282] cf28  90006ba0 aa1303e1 91090000 94204d24 d0006c20 911b6000 94204d21 97fe4927
[   65.141008] cf48  aa0003e4 aa1303e5 52800002 d2800001 52800000 90006ba3 91086063 97fe4973
[   65.141733] cf68  52800000 52800002 d0006c23 d2800001 911c4063 97fe496d b94e4300 17ffffbc
[   65.142459] cf88  913382f9 90006ba8 91078108 aa1903e1 aa0803e0 f90047e6 94204d09 d0006c20
[   65.143184] cfa8  91066000 94204d06 97fe490c aa0003e4 90006ba7 910860e7 aa1903e5 aa0703e3
[   65.143909] cfc8  52800002 d2800001 52800000 f90043e7 97fe4956 52800002 d2800001 52800000
[   65.144635] cfe8  d0006c23 91070063 97fe4950 b94e4300 90006ba2 710022bf 91078048 121f0001
[   65.145364] d008  a9481be7 54ffde21 35003a01 d503201f 37001020 370816a0 b9409be0 3108301f
[   65.146089] d028  54ffed00 12800240 b9009be0 17ffff65 35ffe841 91404681 395ca021 34ffebc1
[   65.146814] d048  aa0603e3 aa1303e2 910263e4 2a1503e1 aa1403e0 97ff7c81 aa0003f9 17fffef0
[   65.147539] d068  3607f020 913382f3 f0006b80 aa1303e1 91090000 94204cd2 b0006c20 9117a000
[   65.148264] d088  94204ccf 97fe48d5 aa0003e4 aa1303e5 52800002 d2800001 52800000 f0006b83
[   65.148990] d0a8  91086063 97fe4921 d2800001 52800000 52800002 b0006c23 91188063 97fe491b
[   65.149716] d0c8* b94e4300 f9401321 17ffff64 a94573fb 910273e6 794113e1 aa0603e2 aa1403e0
[   65.150441] d0e8  f90043e6 97ff61dd f94043e6 34ffd640 128002a0 b9009be0 17ffff32 3607ebc0
[   65.151166] d108  913382f3 f0006b80 aa1303e1 91090000 94204cab b0006c20 911d2000 94204ca8
[   65.151892] d128  97fe48ae aa0003e4 aa1303e5 52800002 d2800001 52800000 f0006b83 91086063
[   65.152617] d148  97fe48fa 52800000 52800002 b0006c23 d2800001 911e0063 97fe48f4 b94e4300
[   65.153342] d168  17ffff45 52800021 128ffe80 39153a61 790ace60 f9400280 f9401000 97fe6fec
[   65.154068] d188  17ffff20 b94e4300 3607e180 913382f7 f0006b80 aa1703e1 91090000 94204c88
[   65.154793] d1a8  b955da81 b0006c20 910e4000 94204c84 97fe488a aa0003e4 aa1703e5 52800002
[   65.155520] d1c8  d2800001 52800000 f0006b83 91086063 97fe48d6 b955da84 b0006c23 52800002
[   65.156244] d1e8  910f2063 d2800001 52800000 97fe48cf 17fffef2 aa1303e3 aa0603e2 2a1503e1
[   65.156970] d208  97ff7daf aa0003f9 17fffe85 a90573fb 94212c26 9133e2f3 f0006b80 aa1303e1
[   65.157695] d228  91090000 94204c66 b0006c20 9108a000 94204c63 97fe4869 aa0003e4 aa1303e5
[   65.158420] d248  52800002 d2800001 52800000 f0006b83 91086063 97fe48b5 b0006c23 52800002
[   65.159146] d268  91094063 d2800001 52800000 97fe48af b94e4300 370803a0 b94e4300 3607ece0
[   65.159870] d288  913382f7 f0006b80 aa1703e1 91090000 94204c4b b9409be1 b0006c20 91146000
[   65.160595] d2a8  94204c47 97fe484d aa0003e4 aa1703e5 52800002 d2800001 52800000 f0006b83
[   65.161322]
               SP: 0xffffffc00cceb780:
[   65.161757] b580  f9473180 ffffff81 6f096600 20c10bb3 0cceb5b0 ffffffc0 08034dd0 ffffffc0
[   65.162481] b5a0  96000005 00000000 0cceb630 ffffffc0 0cceb5f0 ffffffc0 09426b64 ffffffc0
[   65.163206] b5c0  0cceb630 ffffffc0 96000005 00000000 00000100 00000000 08bdce60 ffffffc0
[   65.163931] b5e0  0cceb630 ffffffc0 0a2b2000 ffffffc0 0cceb620 ffffffc0 094271dc ffffffc0
[   65.164656] b600  095bf268 ffffffc0 ffffffff 0000007f 0cceb780 ffffffc0 0cceb780 ffffffc0
[   65.165381] b620  0cceb760 ffffffc0 08011c08 ffffffc0 00000023 00000000 00000000 00000000
[   65.166105] b640  00000000 00000000 0000000a 00000000 00000008 00000000 00000031 00000000
[   65.166829] b660  0f00620a ffffff80 554e2073 62204c4c 726f6665 69662065 00000000 00000000
[   65.167554] b680  0a153258 ffffffc0 615f3131 695f6464 0a4801fe ffffffc0 00000006 00000000
[   65.168278] b6a0  0a480208 ffffffc0 ffffffff ffffffff 00000000 00000000 00000000 00000000
[   65.169003] b6c0  00000030 00000000 095bf268 ffffffc0 0c151800 ffffff80 00000007 00000000
[   65.169728] b6e0  fa21d000 ffffff81 095be588 ffffffc0 0a2b2000 ffffffc0 f78db800 ffffff81
[   65.170452] b700  00000000 00000000 00000000 00000000 f9673700 ffffff81 0cceb780 ffffffc0
[   65.171178] b720  08bdd0c8 ffffffc0 0cceb780 ffffffc0 08bdce60 ffffffc0 60400009 00000000
[   65.171903] b740  00000000 00000000 00000000 00000000 ffffffff 0000007f 6f096600 20c10bb3
[   65.172627] b760  0cceb780 ffffffc0 08bdce60 ffffffc0 0f0061d9 ffffff80 00000000 00000000
[   65.173352] b780* 0cceb840 ffffffc0 08bdd904 ffffffc0 00000001 00000000 2a969c20 ffffff80
[   65.174077] b7a0  0000000a 00000000 0a2b2000 ffffffc0 fa21d000 ffffff81 0c140800 ffffff80
[   65.174802] b7c0  0c140e58 ffffff80 00000001 00000000 00000000 00000000 f9673700 ffffff81
[   65.175526] b7e0  0a402000 ffffffc0 0c140ee8 ffffff80 2abf9000 00000000 f79480a8 ffffff81
[   65.176250] b800  0cceb81c ffffffc0 0cceb81c ffffffc0 0cceb840 ffffffc0 00000000 fbf441aa
[   65.176975] b820  0000e7ac 00000000 00000000 00000000 00000000 00000000 6f096600 20c10bb3
[   65.177700] b840  0cceb8a0 ffffffc0 09332640 ffffffc0 2a969c20 ffffff80 0cceb960 ffffffc0
[   65.178424] b860  0c140000 ffffff80 0000000a 00000000 f3fd3700 ffffff81 0c140300 ffffff80
[   65.179148] b880  0a300400 ffffffc0 0cceba78 ffffffc0 0a300400 00000007 6f096600 20c10bb3
[   65.179873] b8a0  0cceb910 ffffffc0 090ec7ec ffffffc0 00000000 00000000 0cceba08 ffffffc0
[   65.180598] b8c0  2abff000 ffffff80 f3fd3d00 ffffff81 09a2d980 ffffffc0 0cceba78 ffffffc0
[   65.181324] b8e0  2a969c00 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
[   65.182048] b900  00000000 00000000 6f096600 20c10bb3 0cceb9b0 ffffffc0 090ecbc4 ffffffc0
[   65.182773] b920  09a2d980 ffffffc0 f3fd3d00 ffffff81 0a309738 ffffffc0 0cceba78 ffffffc0
[   65.183497] b940  0a300400 ffffffc0 2a969c00 ffffff80 00000004 00000000 0cceb9b0 ffffffc0
[   65.184221] b960  68c29fec a0800147 2a969c00 ffffff80 2a969c10 ffffff80 2a969c14 ffffff80
[   65.184948]
               X6: 0xffffff800f00620a:
[   65.185382] 6008  5b0a3765 36202020 35382e32 38353934 6c77205d 6766635f 31323038 64615f31
[   65.186108] 6028  66695f64 6c77203a 6766635f 31323038 64615f31 66695f64 74697720 72616568
[   65.186834] 6048  0a312074 2020205b 382e3236 36393435 77205d35 66635f6c 32303867 615f3131
[   65.187559] 6068  695f6464 77203a66 66635f6c 32303867 615f3131 695f6464 69772066 61656874
[   65.188283] 6088  35207472 20205b0a 2e323620 36353538 3a5d3437 434c5720 5445475f 5241565f
[   65.189008] 60a8  6d63203a 32203a64 202c3236 3a67736d 72756320 6874655f 64617265 76207264
[   65.189734] 60c8  203a6c61 2c307830 6e656c20 3532203a 73202c36 203a7465 74202c30 692d6e78
[   65.190458] 60e8  36203a64 35333535 20205b0a 2e323620 36353538 205d3539 635f6c77 32706766
[   65.191183] 6108  64615f70 32705f64 69645f70 695f6373 50203a66 69205032 7265746e 65636166
[   65.191908] 6128  67657220 65747369 0a646572 2020205b 382e3236 30373535 77205d36 66635f6c
[   65.192633] 6148  32303867 615f3131 695f6464 77203a66 66635f6c 32303867 615f3131 695f6464
[   65.193358] 6168  69772066 61656874 38207472 20205b0a 2e323620 37353538 205d3431 635f6c77
[   65.194083] 6188  30386766 5f313132 5f646461 203a6669 635f6c77 30386766 5f313132 5f646461
[   65.194807] 61a8  77206669 65687469 20747261 205b0a39 32362020 3535382e 5d323237 5f6c7720
[   65.195532] 61c8  38676663 31313230 6464615f 3a66695f 74697720 72616568 77203a74 2d766564
[   65.196257] 61e8  74656e3e 20766564 4e207369 204c4c55 6f666562 66206572 6c616e69 676f6c20
[   65.196982] 6208* 00000a21 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.197707] 6228  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.198431] 6248  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.199155] 6268  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.199879] 6288  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.200603] 62a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.201329] 62c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.202054] 62e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.202779] 6308  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.203504] 6328  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.204229] 6348  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.204952] 6368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.205677] 6388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.206401] 63a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.207125] 63c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.207850] 63e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.208575] 6408  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.209302]
               X10: 0xffffffc00a153258:
[   65.209746] 3058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.210471] 3078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.211196] 3098  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.211920] 30b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.212645] 30d8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.213370] 30f8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.214094] 3118  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.214819] 3138  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.215543] 3158  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.216268] 3178  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.216992] 3198  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.217716] 31b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.218440] 31d8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.219165] 31f8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.219888] 3218  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.220614] 3238  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.221338] 3258* ffffe000 80000000 fffc0000 ffffffff fffc0040 ffffffff ffffe001 80000000
[   65.222063] 3278  fffc0040 ffffffff fffc0188 ffffffff ffffe002 80000000 fffc0188 ffffffff
[   65.222788] 3298  fffc01c8 ffffffff ffffe003 80000000 fffc01c8 ffffffff fffc0210 ffffffff
[   65.223512] 32b8  ffffe004 80000000 fffc0210 ffffffff fffc0240 ffffffff ffffe005 80000000
[   65.224237] 32d8  fffc0240 ffffffff fffc0260 ffffffff ffffe006 80000000 fffc0260 ffffffff
[   65.224961] 32f8  fffc02b8 ffffffff ffffe007 80000000 fffc02b8 ffffffff fffc0310 ffffffff
[   65.225686] 3318  ffffe008 80000000 fffc0310 ffffffff fffc0328 ffffffff ffffe009 80000000
[   65.226411] 3338  fffc0328 ffffffff fffc0368 ffffffff ffffe00a 80000000 fffc0368 ffffffff
[   65.227136] 3358  fffc0380 ffffffff ffffe00b 80000000 fffc0380 ffffffff fffc03c0 ffffffff
[   65.227861] 3378  ffffe00c 80000000 fffc03c0 ffffffff fffc03e8 ffffffff ffffe00d 80000000
[   65.228586] 3398  fffc03e8 ffffffff fffc0408 ffffffff ffffe00e 80000000 fffc0408 ffffffff
[   65.229311] 33b8  fffc0448 ffffffff ffffe00f 80000000 fffc0448 ffffffff fffc0488 ffffffff
[   65.230036] 33d8  ffffe010 80000000 fffc0488 ffffffff fffc04c8 ffffffff ffffe011 80000000
[   65.230761] 33f8  fffc04c8 ffffffff fffc0510 ffffffff ffffe012 80000000 fffc0510 ffffffff
[   65.231486] 3418  fffc0538 ffffffff ffffe013 80000000 fffc0538 ffffffff fffc0568 ffffffff
[   65.232212] 3438  ffffe014 80000000 fffc0568 ffffffff fffc0590 ffffffff ffffe015 80000000
[   65.232938]
               X12: 0xffffffc00a4801fe:
[   65.233383] fffc  72657669 3031203a 30312e30 2033362e 36373028 35393137 20200a29 6d726946
[   65.234108] 001c  65726177 6c77203a 4a203a30 20206c75 30322032 30203931 31323a32 2033323a
[   65.234833] 003c  73726576 206e6f69 36342e37 2e38352e 28203932 37313772 20313435 20295943
[   65.235559] 005c  44495746 2d313020 32373661 39626436 34736520 2e33632e 0a20346e 68432020
[   65.236283] 007c  203a7069 36613961 76655220 50203120 3420676b 00000000 00000000 00000000
[   65.237008] 009c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.237732] 00bc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.238457] 00dc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.239181] 00fc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.239908] 011c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.240632] 013c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.241357] 015c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.242082] 017c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.242806] 019c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.243531] 01bc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.244256] 01dc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 36202020
[   65.244981] 01fc* 35382e32 32323735 00000000 00000000 00000000 f79480a8 ffffff81 00000001
[   65.245706] 021c  00000000 0f100000 ffffff80 000f9df6 00000000 0f00620a ffffff80 0f000000
[   65.246431] 023c  ffffff80 0f000000 ffffff80 00000000 00000000 f79480a8 ffffff81 00000001
[   65.247155] 025c  00000000 0f180000 ffffff80 00080000 00000000 0f100000 ffffff80 0f100000
[   65.247880] 027c  ffffff80 0f100000 ffffff80 00000000 00000000 f79480a8 ffffff81 00000001
[   65.248604] 029c  00000000 f9792000 ffffff81 00002000 00000000 f9790000 ffffff81 f9790000
[   65.249329] 02bc  ffffff81 f9790000 ffffff81 00000000 00000000 00000000 00000000 6e65762f
[   65.250053] 02dc  2f726f64 2f637465 6d726966 65726177 7779632f 33343334 7a615f38 32373377
[   65.250779] 02fc  7478742e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.251503] 031c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.252228] 033c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.252952] 035c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.253676] 037c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.254400] 039c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.255124] 03bc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.255848] 03dc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.256573] 03fc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.257299]
               X14: 0xffffffc00a480208:
[   65.257745] 0008  2033362e 36373028 35393137 20200a29 6d726946 65726177 6c77203a 4a203a30
[   65.258470] 0028  20206c75 30322032 30203931 31323a32 2033323a 73726576 206e6f69 36342e37
[   65.259195] 0048  2e38352e 28203932 37313772 20313435 20295943 44495746 2d313020 32373661
[   65.259920] 0068  39626436 34736520 2e33632e 0a20346e 68432020 203a7069 36613961 76655220
[   65.260645] 0088  50203120 3420676b 00000000 00000000 00000000 00000000 00000000 00000000
[   65.261370] 00a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.262095] 00c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.262819] 00e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.263543] 0108  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.264268] 0128  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.264993] 0148  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.265717] 0168  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.266441] 0188  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.267167] 01a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.267892] 01c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.268618] 01e8  00000000 00000000 00000000 00000000 36202020 35382e32 32323735 00000000
[   65.269344] 0208* 00000000 00000000 f79480a8 ffffff81 00000001 00000000 0f100000 ffffff80
[   65.270068] 0228  000f9df6 00000000 0f00620a ffffff80 0f000000 ffffff80 0f000000 ffffff80
[   65.270793] 0248  00000000 00000000 f79480a8 ffffff81 00000001 00000000 0f180000 ffffff80
[   65.271518] 0268  00080000 00000000 0f100000 ffffff80 0f100000 ffffff80 0f100000 ffffff80
[   65.272243] 0288  00000000 00000000 f79480a8 ffffff81 00000001 00000000 f9792000 ffffff81
[   65.272967] 02a8  00002000 00000000 f9790000 ffffff81 f9790000 ffffff81 f9790000 ffffff81
[   65.273692] 02c8  00000000 00000000 00000000 00000000 6e65762f 2f726f64 2f637465 6d726966
[   65.274416] 02e8  65726177 7779632f 33343334 7a615f38 32373377 7478742e 00000000 00000000
[   65.275141] 0308  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.275865] 0328  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.276590] 0348  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.277314] 0368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.278039] 0388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.278764] 03a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.279490] 03c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.280214] 03e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.280940]
               X19: 0xffffffc0095bf268:
[   65.281386] f068  63746977 6f6e5f68 79666974 00000000 615f6c77 68635f70 656e6e61 6e695f6c
[   65.282111] f088  00000064 00000000 615f6c77 74735f70 5f747261 00646e69 635f6c77 635f6173
[   65.282837] f0a8  6c706d6f 5f657465 00646e69 00000000 635f6c77 30386766 5f313132 61656c63
[   65.283562] f0c8  65735f72 69727563 00007974 00000000 635f6c77 30386766 5f313132 6e776f64
[   65.284288] f0e8  00000000 00000000 6c775f5f 6766635f 31323038 6f645f31 00006e77 00000000
[   65.285012] f108  635f6c77 30386766 5f313132 5f6c6564 5f703270 76656477 00000000 00000000
[   65.285737] f128  635f6c77 30386766 5f313132 5f746573 63657073 00000074 665f6c77 6873756c
[   65.286462] f148  5f77665f 5f676f6c 66667562 00007265 635f6c77 30386766 5f313132 616d6572
[   65.287187] f168  6f5f6e69 68635f6e 656e6e61 0000006c 635f6c77 30386766 5f313132 6e616863
[   65.287912] f188  625f6567 6f636165 0000006e 00000000 635f6c77 30386766 5f313132 74736f68
[   65.288636] f1a8  5f647061 00636573 6e5f6c77 6669746f 696d5f79 74735f63 73757461 00000000
[   65.289361] f1c8  635f6c77 30386766 5f313132 736c6474 6e6f635f 00676966 635f6c77 30386766
[   65.290085] f1e8  5f313132 63616669 74735f65 5f657461 0073706f 00000000 09965dc0 ffffffc0
[   65.290810] f208  09965dd8 ffffffc0 09965df0 ffffffc0 09965e08 ffffffc0 09965e20 ffffffc0
[   65.291535] f228  09965e38 ffffffc0 09965e50 ffffffc0 775f6c77 685f7370 6c646e61 66695f65
[   65.292260] f248  006c6564 00000000 775f6c77 5f63666c 62616e65 0000656c 00000000 00000000
[   65.292985] f268* 635f6c77 30386766 5f313132 5f646461 00006669 00000000 635f6c77 30386766
[   65.293710] f288  5f313132 5f646461 73736269 00000000 635f6c77 30386766 5f313132 5f646461
[   65.294436] f2a8  696e6f6d 5f726f74 00006669 00000000 635f6c77 30386766 5f313132 5f646461
[   65.295160] f2c8  74726976 5f6c6175 63616669 00000065 635f6c77 30386766 5f313132 6e616863
[   65.295885] f2e8  765f6567 75747269 695f6c61 65636166 00000000 00000000 635f6c77 30386766
[   65.296610] f308  5f313132 736c6474 65706f5f 00000072 635f6c77 30386766 5f313132 6e6e6f63
[   65.297335] f328  00746365 00000000 ffffffff 0000ffff 735f6c77 6b5f7465 6d5f7965 00746d67
[   65.298060] f348  000fac01 00000040 000fac05 00001000 000fac06 00008000 000fac02 00000080
[   65.298785] f368  000fac03 00004040 000fac04 00004080 000fac0e 00010000 000fac0f 00020000
[   65.299509] f388  000fac0b 00200000 000fac0c 00400000 000fac12 00000400 000fac04 00000800
[   65.300234] f3a8  000fac08 00040000 000fac0d 02004000 506f9a02 00200000 635f6c77 30386766
[   65.300959] f3c8  5f313132 5f746573 0070666d 00000000 635f6c77 6f745f68 6168635f 6570736e
[   65.301683] f3e8  00000063 00000000 6e5f6c77 6669746f 6f635f79 63656e6e 74735f74 73757461
[   65.302408] f408  7362695f 00000073 6e5f6c77 6669746f 6f635f79 63656e6e 74735f74 73757461
[   65.303133] f428  00000000 00000000 635f6c77 65686361 7373615f 725f636f 5f707365 00736569
[   65.303858] f448  695f6c77 696c5f73 70756b6e 00000000 00000000 00000000 635f6c77 30386766
[   65.304584]
               X20: 0xffffff800c151800:
[   65.305029] 1600  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.305753] 1620  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.306478] 1640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.307203] 1660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.307927] 1680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.308651] 16a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.309376] 16c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.310101] 16e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.310825] 1700  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.311550] 1720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.312274] 1740  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.312999] 1760  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.313723] 1780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.314448] 17a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.315172] 17c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.315896] 17e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.316620] 1800* 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.317344] 1820  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.318068] 1840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.318793] 1860  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.319518] 1880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.320242] 18a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.320966] 18c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.321690] 18e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.322415] 1900  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.323139] 1920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.323863] 1940  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.324587] 1960  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.325312] 1980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.326037] 19a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.326761] 19c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.327486] 19e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.328212]
               X22: 0xffffff81fa21d000:
[   65.328657] ce00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.329382] ce20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.330106] ce40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.330830] ce60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.331554] ce80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.332278] cea0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.333003] cec0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.333728] cee0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.334452] cf00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.335178] cf20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.335902] cf40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.336627] cf60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.337352] cf80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.338076] cfa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.338800] cfc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.339524] cfe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.340248] d000* 6e616c77 00000030 00000000 00000000 f9536000 ffffff81 00000000 00000000
[   65.340973] d020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.341697] d040  00000003 00000000 0a300490 ffffffc0 f23cc048 ffffff81 fa21d058 ffffff81
[   65.342423] d060  fa21d058 ffffff81 fa21d068 ffffff81 fa21d068 ffffff81 fa21d078 ffffff81
[   65.343148] d080  fa21d078 ffffff81 fa21d088 ffffff81 fa21d088 ffffff81 f1cc5d78 ffffff81
[   65.343873] d0a0  0c09b578 ffffff80 fa21d0a8 ffffff81 fa21d0a8 ffffff81 fa21d0b8 ffffff81
[   65.344598] d0c0  fa21d0b8 ffffff81 00006000 00000000 00004800 02000200 00004800 00000000
[   65.345323] d0e0  00000020 00000000 10000001 00000000 00000001 00000000 00000000 00000000
[   65.346048] d100  00000006 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.346772] d120  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.347496] d140  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.348221] d160  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.348946] d180  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.349670] d1a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.350395] d1c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.351119] d1e0  095b91b8 ffffffc0 00000000 00000000 0a2ac550 ffffffc0 0a2ac780 ffffffc0
[   65.351845]
               X23: 0xffffffc0095be588:
[   65.352290] e388  6973765f 65645f65 6574656c 00000000 635f6c77 30386766 5f313132 74736f70
[   65.353016] e3a8  6366695f 74616572 00000065 00000000 725f6c77 5f656c6f 635f6f74 30386766
[   65.353741] e3c8  5f313132 65707974 00000000 00000000 635f6c77 30386766 5f313132 5f703270
[   65.354466] e3e8  615f6669 00006464 635f6c77 30386766 5f313132 61657263 695f6574 65636166
[   65.355190] e408  00000000 00000000 635f6c77 30386766 5f313132 74736f70 6466695f 00006c65
[   65.355915] e428  635f6c77 30386766 5f313132 61656c63 5f70756e 74726976 5f6c6175 63616669
[   65.356640] e448  00007365 00000000 635f6c77 30386766 5f313132 74736669 5f737461 6e756f63
[   65.357366] e468  73726574 00000000 635f6c77 30386766 5f313132 5f6c6564 5f6c6c61 00617473
[   65.358090] e488  635f6c77 30386766 5f313132 5f746573 5f63616d 006c6361 755f6c77 74616470
[   65.358815] e4a8  6d705f65 73696c6b 00000074 00000000 635f6c77 30386766 5f313132 73756c66
[   65.359539] e4c8  6d705f68 0061736b 635f6c77 30386766 5f313132 5f6c6564 736b6d70 00000061
[   65.360264] e4e8  635f6c77 30386766 5f313132 5f746573 736b6d70 00000061 735f6c77 775f7465
[   65.360989] e508  5f636573 6f666e69 676c615f 0000736f 765f6c77 64696c61 5f657461 32617077
[   65.361713] e528  00006569 00000000 635f6c77 30386766 5f313132 5f6e6362 6e697262 5f707567
[   65.362438] e548  00007061 00000000 775f6c77 735f7370 69737365 755f6e6f 74616470 00000065
[   65.363164] e568  775f6c77 685f7370 6c646e61 386d5f65 6e65735f 00000074 775f6c77 685f7370
[   65.363890] e588* 6c646e61 74735f65 696c5f61 6f646b6e 00006e77 00000000 775f6c77 685f7370
[   65.364615] e5a8  6c646e61 65705f65 74737265 696c5f61 6f646b6e 00006e77 775f6c77 685f7370
[   65.365341] e5c8  6c646e61 74735f65 696c5f61 70756b6e 00000000 00000000 775f6c77 685f7370
[   65.366066] e5e8  6c646e61 65705f65 74737265 696c5f61 70756b6e 00000000 775f6c77 685f7370
[   65.366790] e608  6c646e61 69645f65 6e6f6373 7463656e 696c635f 00746e65 775f6c77 685f7370
[   65.367516] e628  6c646e61 69645f65 6e6f6373 7463656e 00000000 00000000 775f6c77 685f7370
[   65.368241] e648  6c646e61 6f635f65 63656e6e 61665f74 00006c69 00000000 775f6c77 685f7370
[   65.368966] e668  6c646e61 75615f65 726f6874 00657a69 635f6c77 30386766 5f313132 6e616863
[   65.369690] e688  735f6567 69746174 00006e6f 00000000 635f6c77 30386766 5f313132 5f6c6564
[   65.370416] e6a8  74617473 006e6f69 635f6c77 30386766 5f313132 5f74656e 61747461 00006863
[   65.371140] e6c8  635f6c77 30386766 5f313132 61747461 00006863 00000000 735f6c77 70757465
[   65.371865] e6e8  7069775f 00007968 095bf820 ffffffc0 00000003 00050006 00000000 00000000
[   65.372592] e708  000fac01 000fac05 000fac02 000fac04 000fac06 000fac0c 000fac0b 000fac0d
[   65.373316] e728  00147201 00904c00 000fac08 000fac09 000fac0b 000fac0c 00000000 2000ffff
[   65.374041] e748  2810ffff 3c15ffff 3c15ffff 00000000 00000000 00000000 2010ffff 3c15ffff
[   65.374766] e768  2010ffff 00000000 00000000 00000000 00000001 00000008 000000ff 00000001
[   65.375492]
               X24: 0xffffffc00a2b2000:
[   65.375937] 1e00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.376661] 1e20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.377386] 1e40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.378110] 1e60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.378835] 1e80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.379560] 1ea0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.380284] 1ec0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.381008] 1ee0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.381733] 1f00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.382456] 1f20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.383180] 1f40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.383905] 1f60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.384629] 1f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.385354] 1fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.386078] 1fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.386801] 1fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.387525] 2000* 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.388249] 2020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.388973] 2040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.389698] 2060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.390424] 2080  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.391148] 20a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.391873] 20c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.392599] 20e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.393323] 2100  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.394048] 2120  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.394773] 2140  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.395497] 2160  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.396221] 2180  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.396945] 21a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.397670] 21c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.398395] 21e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.399119]
               X25: 0xffffff81f78db800:
[   65.399564] b600  f78dac00 ffffff81 00000000 00000000 00000000 00000000 00000000 00000000
[   65.400289] b620  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.401013] b640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.401738] b660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.402462] b680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.403186] b6a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.403910] b6c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.404634] b6e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.405358] b700  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.406083] b720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.406808] b740  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.407533] b760  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.408257] b780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.408981] b7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.409708] b7c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.410433] b7e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.411158] b800* 0c140300 ffffff80 0000000a 00000000 00000000 00000000 00000000 00000000
[   65.411882] b820  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.412606] b840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.413331] b860  00000000 00000000 41aa0000 e7acfbf4 00000000 00000000 00000000 00000000
[   65.414056] b880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.414781] b8a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.415505] b8c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.416231] b8e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.416956] b900  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.417681] b920  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.418406] b940  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.419130] b960  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.419855] b980  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.420579] b9a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.421302] b9c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.422027] b9e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.422753]
               X28: 0xffffff81f9673700:
[   65.423198] 3500  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.423923] 3520  00000000 00000000 00000000 00000000 00000010 01000000 00000000 00000000
[   65.424648] 3540  00000003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.425373] 3560  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.426098] 3580  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.426822] 35a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.427548] 35c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.428272] 35e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.428997] 3600  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.429721] 3620  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.430445] 3640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.431170] 3660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.431894] 3680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.432619] 36a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.433344] 36c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.434070] 36e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.434795] 3700* 00000022 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
[   65.435520] 3720  0cce8000 ffffffc0 00000001 00400100 00000000 00000001 00000000 00000000
[   65.436245] 3740  00000030 00000000 00000001 00000035 fffc607b 00000000 f9433700 ffffff81
[   65.436970] 3760  00000001 00000001 00000001 00000078 00000078 00000078 00000000 00000000
[   65.437694] 3780  09a2c720 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[   65.438419] 37a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.439144] 37c0  00100000 00000000 00400000 00000000 00000001 00000000 00000000 00000000
[   65.439868] 37e0  00000000 00000000 ff71d890 ffffff81 f19c5368 ffffff81 00000001 00000000
[   65.440592] 3800  abb29a0d 0000000e 0bcddbb3 00000000 cf160e10 00000008 02983763 00000000
[   65.441316] 3820  0000000e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.442040] 3840  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.442764] 3860  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.443489] 3880  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.444214] 38a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.444938] 38c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.445662] 38e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   65.446387]
               X29: 0xffffffc00cceb780:
[   65.446833] b580  f9473180 ffffff81 6f096600 20c10bb3 0cceb5b0 ffffffc0 08034dd0 ffffffc0
[   65.447556] b5a0  96000005 00000000 0cceb630 ffffffc0 0cceb5f0 ffffffc0 09426b64 ffffffc0
[   65.448282] b5c0  0cceb630 ffffffc0 96000005 00000000 00000100 00000000 08bdce60 ffffffc0
[   65.449006] b5e0  0cceb630 ffffffc0 0a2b2000 ffffffc0 0cceb620 ffffffc0 094271dc ffffffc0
[   65.449731] b600  095bf268 ffffffc0 ffffffff 0000007f 0cceb780 ffffffc0 0cceb780 ffffffc0
[   65.450456] b620  0cceb760 ffffffc0 08011c08 ffffffc0 00000023 00000000 00000000 00000000
[   65.451183] b640  00000000 00000000 0000000a 00000000 00000008 00000000 00000031 00000000
[   65.451907] b660  0f00620a ffffff80 554e2073 62204c4c 726f6665 69662065 00000000 00000000
[   65.452633] b680  0a153258 ffffffc0 615f3131 695f6464 0a4801fe ffffffc0 00000006 00000000
[   65.453357] b6a0  0a480208 ffffffc0 ffffffff ffffffff 00000000 00000000 00000000 00000000
[   65.454082] b6c0  00000030 00000000 095bf268 ffffffc0 0c151800 ffffff80 00000007 00000000
[   65.454807] b6e0  fa21d000 ffffff81 095be588 ffffffc0 0a2b2000 ffffffc0 f78db800 ffffff81
[   65.455532] b700  00000000 00000000 00000000 00000000 f9673700 ffffff81 0cceb780 ffffffc0
[   65.456257] b720  08bdd0c8 ffffffc0 0cceb780 ffffffc0 08bdce60 ffffffc0 60400009 00000000
[   65.456982] b740  00000000 00000000 00000000 00000000 ffffffff 0000007f 6f096600 20c10bb3
[   65.457706] b760  0cceb780 ffffffc0 08bdce60 ffffffc0 0f0061d9 ffffff80 00000000 00000000
[   65.458431] b780* 0cceb840 ffffffc0 08bdd904 ffffffc0 00000001 00000000 2a969c20 ffffff80
[   65.459155] b7a0  0000000a 00000000 0a2b2000 ffffffc0 fa21d000 ffffff81 0c140800 ffffff80
[   65.459879] b7c0  0c140e58 ffffff80 00000001 00000000 00000000 00000000 f9673700 ffffff81
[   65.460604] b7e0  0a402000 ffffffc0 0c140ee8 ffffff80 2abf9000 00000000 f79480a8 ffffff81
[   65.461329] b800  0cceb81c ffffffc0 0cceb81c ffffffc0 0cceb840 ffffffc0 00000000 fbf441aa
[   65.462054] b820  0000e7ac 00000000 00000000 00000000 00000000 00000000 6f096600 20c10bb3
[   65.462779] b840  0cceb8a0 ffffffc0 09332640 ffffffc0 2a969c20 ffffff80 0cceb960 ffffffc0
[   65.463503] b860  0c140000 ffffff80 0000000a 00000000 f3fd3700 ffffff81 0c140300 ffffff80
[   65.464228] b880  0a300400 ffffffc0 0cceba78 ffffffc0 0a300400 00000007 6f096600 20c10bb3
[   65.464953] b8a0  0cceb910 ffffffc0 090ec7ec ffffffc0 00000000 00000000 0cceba08 ffffffc0
[   65.465678] b8c0  2abff000 ffffff80 f3fd3d00 ffffff81 09a2d980 ffffffc0 0cceba78 ffffffc0
[   65.466402] b8e0  2a969c00 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
[   65.467127] b900  00000000 00000000 6f096600 20c10bb3 0cceb9b0 ffffffc0 090ecbc4 ffffffc0
[   65.467851] b920  09a2d980 ffffffc0 f3fd3d00 ffffff81 0a309738 ffffffc0 0cceba78 ffffffc0
[   65.468576] b940  0a300400 ffffffc0 2a969c00 ffffff80 00000004 00000000 0cceb9b0 ffffffc0
[   65.469300] b960  68c29fec a0800147 2a969c00 ffffff80 2a969c10 ffffff80 2a969c14 ffffff80
[   65.470028] Code: b94e4300 37083160 f9401321 b4001061 (b9410021)
[   65.470567] ---[ end trace 3ddce5e5b8f41422 ]---

这段信息是 Linux 内核错误报告（通常称为 “Oops” 或 “Kernel Panic” 信息），它记录了内核在运行过程中遇到了一个无法恢复的严重错误（在本例中是空指针解引用），导致系统崩溃或特定进程终止。

📌 信息类型与解读方法

类型： Linux Kernel Oops / Kernel Panic 日志。
解读目的： 诊断内核、驱动程序或硬件中的严重错误。
解读方法：
1. 定位错误源头： 找到错误发生的位置（函数、模块、指令地址）。
2. 理解错误类型： 确定是什么错误（空指针、内存访问违规等）。
3. 分析上下文： 查看寄存器状态、调用栈、内存映射等，了解错误发生时的系统状态。
4. 关联模块： 确定是哪个内核模块或驱动程序导致了问题。
5. 查找线索： 结合错误信息、寄存器值、调用栈和内存转储，寻找代码逻辑或数据结构的异常。

📊 各部分含义详解

错误摘要：
- [ 65.086700] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000100
  - 含义： 核心错误！内核尝试访问一个 NULL 指针（地址 0x100），这是非法操作。这是导致 Oops 的直接原因。
内存异常信息 (Mem abort info / Data abort info)：
- ESR = 0x96000005
- EC = 0x25: DABT (current EL), IL = 32 bits
- SET = 0, FnV = 0, EA = 0, S1PTW = 0
- ISV = 0, ISS = 0x00000005, CM = 0, WnR = 0
页表信息：
- user pgtable: 4k pages, 39-bit VAs, pgdp=00000001f973d000
  - 含义： 描述用户空间页表配置：页大小 4KB，虚拟地址 39 位宽，顶级页目录地址。
- [0000000000000100] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
  - 含义： 尝试访问地址 0x100 时各级页表项的值。全为 0 表示该地址根本没有映射到任何物理内存，证实了空指针解引用。
错误类型与严重性：
- Internal error: Oops: 96000005 [#1] SMP
模块信息：
- Modules linked in:
  - 含义： 列出当前加载的内核模块。日志中这一行是空的，但这通常显示加载的驱动模块，有助于定位问题驱动。
CPU 和进程信息：
- CPU: 1 PID: 327 Comm: wpa_supplicant Not tainted 5.10.198-gce7d97244615-dirty #80
  - 含义： 错误发生在 CPU 1 上。触发错误的进程 PID 是 327，进程名是 wpa_supplicant（无线网络配置守护进程）。Not tainted 表示错误不是由外部模块（非 GPL 等）引起的。内核版本是 5.10.198，编译描述 gce7d97244615-dirty（dirty 表示编译后有本地修改未提交），编译编号 #80。
硬件信息：
- Hardware name: Rockchip RK3568 EVB1 DDR4 V10 Board (DT)
  - 含义： 硬件平台是 Rockchip RK3568 的开发板（EVB1），使用设备树（DT）描述硬件。
处理器状态：
- pstate: 60400009 (nZCv daif +PAN -UAO -TCO BTYPE=--)
程序计数器和链接寄存器：
- pc : wl_cfg80211_add_if+0x440/0xd90
- lr : wl_cfg80211_add_if+0x6a8/0xd90
  - 含义： pc 是程序计数器，指向当前执行的指令。lr 是链接寄存器，通常指向函数调用后的返回地址。这里都指向函数 wl_cfg80211_add_if 内部的偏移量（0x440 和 0x6a8），表明错误发生在这个函数内部或其调用的函数中。0xd90 是该函数的大致大小。这是定位错误代码的关键线索！ wl_cfg80211 通常与博通（Broadcom）的 Wi-Fi 驱动相关。
栈指针和寄存器转储：
- sp : ffffffc00cceb780
  - 含义： 栈指针寄存器（SP）的值。
- x29: ... x0: ...
  - 含义： ARM64 架构的通用寄存器 X0 到 X30（X29 通常用作帧指针 FP，X30 是 LR）在错误发生时的值。这些值对于理解函数参数、局部变量和调用上下文至关重要。例如，x0 : 0000000000000023 可能是一个参数或返回值。
调用栈回溯 (Call trace)：
- ```
[   65.107487] Call trace:
[   65.107711]  wl_cfg80211_add_if+0x440/0xd90
[   65.108081]  wl_cfg80211_add_virtual_iface+0x154/0x274
[   65.108543]  nl80211_new_interface+0x140/0x3f0
[   65.108936]  genl_family_rcv_msg_doit+0xc8/0x150
... (省略后续调用栈)
[   65.113999]  el0_sync_handler+0xb0/0xb4
[   65.113999]  el0_sync+0x1a0/0x1c0
```
- 含义： 最重要的调试信息之一！ 它显示了从错误发生点（wl_cfg80211_add_if）开始，逐级回溯的函数调用链。这清晰地展示了导致错误的代码路径：
  - 用户空间进程 wpa_supplicant 通过 netlink 套接字 (nl80211, genl) 与内核通信。
  - 请求创建一个新的无线网络接口 (new_interface, add_virtual_iface)。
  - 最终调用到博通 Wi-Fi 驱动的 wl_cfg80211_add_if 函数。
  - 在该函数内部 (+0x440) 发生了空指针解引用。
内存转储：
- PC: 0xffffffc008bdce60: ..., LR: 0xffffffc008bdd0c8: ..., SP: 0xffffffc00cceb780: ..., X6: 0xffffff800f00620a: ... 等。
- 含义： 内核转储了程序计数器（PC）、链接寄存器（LR）、栈指针（SP）以及部分通用寄存器（如 X6, X10, X12, X14, X19, X20, X22）所指向的内存地址附近的内容。这些是原始的机器指令码（或数据），用于高级调试，通常需要结合内核符号表和源代码才能解读。* 标记了 PC 和 LR 当前指向的具体位置。

🔍 总结本次错误

错误类型： 内核空指针解引用（NULL pointer dereference）。
触发地址： 虚拟地址 0x100（这是一个典型的无效地址）。
直接原因： 博通 Wi-Fi 驱动 (wl_cfg80211) 中的函数 wl_cfg80211_add_if 在偏移 0x440 处执行时，尝试读取地址 0x100。
调用路径： 用户空间的 wpa_supplicant 进程通过 netlink (nl80211) 请求创建一个新的无线网络接口，最终调用了出问题的驱动函数。
环境： Rockchip RK3568 开发板，运行自定义编译的 Linux 内核 5.10.198。

排查方式

1 打log排查

pc : wl_cfg80211_add_if+0x440/0xd90
lr : wl_cfg80211_add_if+0x6a8/0xd90 这部分说明错误出现在wl_cfg80211_add_if这个函数中，可以在这个函数中各个位置加入Log（或者使用二分法加入log），通过内核崩溃的时间点和日志的打印时序，定位崩溃代码所在位置，如下：可以确认崩溃发生在编号9的日志之后

2 使用GDB解析vmlinux

编译时，如果编译出了vmlinux(通常位于kernel根目录下)，可以使用gdb进行定位

gdb vmlinux

输出如下

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vmlinux...
(gdb)

使用info line命令查看PC指向的具体代码，直接定位(记得使用*解引用)

info line *0xffffffc008bdce60

输出如下

Line 3225 of "drivers/net/wireless/rockchip_wlan/infineon/cyw43438/wl_cfg80211.c" starts at address 0x8bdce60 <wl_cfg80211_add_if+1088> and ends at 0x8bdce68 <wl_cfg80211_add_if+1096>.

复制链接复制短链接 QQ空间微博

吴思含(Witheart)

Linux 内核 crash 时 Oops(Kernel Panic)信息解读

日志信息

📌 信息类型与解读方法

📊 各部分含义详解

🔍 总结本次错误

排查方式

1 打log排查

2 使用GDB解析vmlinux

分享

留下评论

猜您还喜欢

PMBUS Linear-11数据编码解析

RK3568 Ubuntu 20.04 显卡mali驱动配置与桌面加速

高速信号AC耦合电容

UEFI 与 ACPI 简介