Abstract
A production server crashed unexpectedly and rebooted. The crash happened at a strange time, and we doubt it was a simple hardware fault. A kernel crash dump was captured. Your mission is to analyze it to find the real cause of the crash and determine if any other suspicious activity was present on the system.
The Sherlock provides a memory dump of a Ubuntu 22.04 machine with kernel 5.15.0-25.
Questions
Question 1
What is the hostname of the crashed server?
Loading the dump and the symbols into crash (apt install crash) prints some basic information about the kernel dump, basically the same as the sys command.
$ crash ubuntu22.04-5.15.0-25-generic-202511032103.kdump vmlinux-dbgsym
crash 9.0.2
--- SNIP ---
KERNEL: vmlinux-dbgsym [TAINTED]
DUMPFILE: ubuntu22.04-5.15.0-25-generic-202511032103.kdump [PARTIAL DUMP]
CPUS: 2
DATE: Tue Nov 4 03:03:29 CET 2025
UPTIME: 00:38:58
LOAD AVERAGE: 0.15, 0.08, 0.12
TASKS: 548
NODENAME: ubuntu-2204
RELEASE: 5.15.0-25-generic
VERSION: #25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022
MACHINE: x86_64 (2687 Mhz)
MEMORY: 2 GB
PANIC: "Oops: 0002 [#1] SMP PTI" (check log for details)
PID: 9236
COMMAND: "cat"
TASK: ffff8e0479346200 [THREAD_INFO: ffff8e0479346200]
CPU: 0
STATE: TASK_RUNNING (PANIC)
ubuntu-2204
Question 2
What is the assigned IP address of the server at time of crash?
Running the net command shows the IP address assigned to the ens33 interface.
192.168.1.135
Question 3
When did the server crash? (UTC)
The output from loading the crash dump in Question 1 has the current date in CET and this is one hour ahead of UTC.
2025-11-04 02:03:29
Question 4
The crash was triggered by a specific process. What was the PID of the active process that caused the panic?
Also output from Question 1, but can be seen by using ps.
9236
Question 5
Which command-line utility was leveraged by the previous process to trigger the crash?
Also output from Question 1, but can be seen by using ps.
cat
Question 6
What was the kernel's fatal panic bug message?
The log command shows the messages from the kernel and contains the BUG text.
crash> log
--- SNIP ---
[ 2270.499970] core_helper: loading out-of-tree module taints kernel.
[ 2270.500164] core_helper: module verification failed: signature and/or required key missing - tainting kernel
[ 2270.500758] core_helper: helper module v1.1.2 initialized.
[ 2338.667536] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 2338.667546] #PF: supervisor write access in kernel mode
[ 2338.667549] #PF: error_code(0x0002) - not-present page
[ 2338.667551] PGD 0 P4D 0
[ 2338.667556] Oops: 0002 [#1] SMP PTI
[ 2338.667561] CPU: 0 PID: 9236 Comm: cat Kdump: loaded Tainted: G OE 5.15.0-25-generic #25-Ubuntu
[ 2338.667615] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 2338.667619] RIP: 0010:core_helper_read+0x8/0x15 [core_helper]
[ 2338.667633] Code: Unable to access opcode bytes at RIP 0xffffffffc0a93fde.
[ 2338.667635] RSP: 0018:ffff9a64c611fe60 EFLAGS: 00010246
[ 2338.667639] RAX: 0000000000000000 RBX: ffff8e040e909780 RCX: ffff9a64c611fed0
[ 2338.667641] RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: ffff8e04107cb500
[ 2338.667643] RBP: ffff9a64c611fe80 R08: 0000000000000001 R09: ffff8e0419f77ab0
[ 2338.667646] R10: 0000000000020000 R11: 0000000000000000 R12: 0000000000000000
[ 2338.667648] R13: ffff8e04107cb500 R14: ffff9a64c611fed0 R15: 00007fae8be3d000
[ 2338.667650] FS: 00007fae8cc45740(0000) GS:ffff8e047be00000(0000) knlGS:0000000000000000
[ 2338.667653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2338.667655] CR2: ffffffffc0a93fde CR3: 000000004cdd0004 CR4: 00000000003706f0
[ 2338.667660] Call Trace:
[ 2338.667663] <TASK>
[ 2338.667667] ? proc_reg_read+0x5a/0x90
[ 2338.667676] vfs_read+0x9f/0x1a0
[ 2338.667683] ksys_read+0x67/0xe0
[ 2338.667685] __x64_sys_read+0x19/0x20
[ 2338.667688] do_syscall_64+0x5c/0xc0
[ 2338.667694] ? exc_page_fault+0x89/0x160
[ 2338.667697] ? asm_exc_page_fault+0x8/0x30
[ 2338.667704] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 2338.667708] RIP: 0033:0x7fae8cd5c852
[ 2338.667711] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 1a b4 0c 00 e8 a5 1d 02 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24
[ 2338.667715] RSP: 002b:00007fff5f513348 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2338.667718] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007fae8cd5c852
[ 2338.667720] RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: 0000000000000003
[ 2338.667722] RBP: 00007fae8be3d000 R08: 00007fae8be3c010 R09: 00007fae8be3c010
[ 2338.667724] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000022000
[ 2338.667726] R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
[ 2338.667730] </TASK>
[ 2338.667732] Modules linked in: core_helper(OE) tls rfcomm cpuid binfmt_misc bnep intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock snd_ens1371 snd_ac97_codec gameport ac97_bus snd_pcm nls_iso8859_1 vmw_balloon snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device joydev snd_timer input_leds serio_raw snd btusb btrtl btbcm btintel bluetooth ecdh_generic ecc soundcore vmw_vmci mac_hid sch_fq_codel vmwgfx ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler msr parport_pc ppdev lp drm parport ip_tables x_tables autofs4 hid_generic crc32_pclmul usbhid psmouse hid e1000 mptspi mptscsih ahci mptbase libahci scsi_transport_spi pata_acpi i2c_piix4
[ 2338.667900] CR2: 0000000000000000
kernel NULL pointer dereference, address: 0000000000000000
Question 7
What is the absolute path of the malicious file that caused the kernel crash?
Information about the files used by the process can be seen with the files command.
crash> files
PID: 9236 TASK: ffff8e0479346200 CPU: 0 COMMAND: "cat"
ROOT: / CWD: /root/module
FD FILE DENTRY INODE TYPE PATH
0 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR /dev/pts/1
1 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR /dev/pts/1
2 ffff8e040c25db00 ffff8e04072300c0 ffff8e0431368780 CHR /dev/pts/1
3 ffff8e04107cb500 ffff8e04072d8000 ffff8e0419f77ab0 REG /proc/jiffies_ext
/proc/jiffies_ext
Question 8
What is the name of the function at the top of the kernel's call stack at time of the crash?
Using the stack backtrace with bt the command shows the function where the exception occured.
crash> bt
PID: 9236 TASK: ffff8e0479346200 CPU: 0 COMMAND: "cat"
#0 [ffff9a64c611fb60] machine_kexec at ffffffff85685e40
#1 [ffff9a64c611fbc0] __crash_kexec at ffffffff8578cc22
#2 [ffff9a64c611fc90] crash_kexec at ffffffff8578e3a8
#3 [ffff9a64c611fca0] oops_end at ffffffff8563fa06
#4 [ffff9a64c611fcc8] page_fault_oops at ffffffff85697a0e
#5 [ffff9a64c611fd28] do_user_addr_fault at ffffffff856981c9
#6 [ffff9a64c611fd80] exc_page_fault at ffffffff86348ec7
#7 [ffff9a64c611fdb0] asm_exc_page_fault at ffffffff86400ace
[exception RIP: core_helper_read+8]
RIP: ffffffffc0a94008 RSP: ffff9a64c611fe60 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8e040e909780 RCX: ffff9a64c611fed0
RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: ffff8e04107cb500
RBP: ffff9a64c611fe80 R8: 0000000000000001 R9: ffff8e0419f77ab0
R10: 0000000000020000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8e04107cb500 R14: ffff9a64c611fed0 R15: 00007fae8be3d000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff9a64c611fe68] proc_reg_read at ffffffff85a23a4a
#9 [ffff9a64c611fe88] vfs_read at ffffffff85970f8f
#10 [ffff9a64c611fec8] ksys_read at ffffffff85973937
#11 [ffff9a64c611ff08] __x64_sys_read at ffffffff859739c9
#12 [ffff9a64c611ff18] do_syscall_64 at ffffffff863451bc
#13 [ffff9a64c611ff50] entry_SYSCALL_64_after_hwframe at ffffffff8640007c
RIP: 00007fae8cd5c852 RSP: 00007fff5f513348 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007fae8cd5c852
RDX: 0000000000020000 RSI: 00007fae8be3d000 RDI: 0000000000000003
RBP: 00007fae8be3d000 R8: 00007fae8be3c010 R9: 00007fae8be3c010
R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000022000
R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000020000
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
core_helper_read
Question 9
This function belongs to a malicious kernel module. What is the base memory address of this module?
The register RIP points towards the next instruction so it’s safe to assume it belongs to the function. Asking about the kernel memory with kmem and providing the RIP shows the kernel module that is associated with this function (core_helper) and the memory region where it’s loaded.
crash> kmem ffffffffc0a94008
ffffffffc0a94008 (t) core_helper_read+8 [core_helper]
VMAP_AREA VM_STRUCT ADDRESS RANGE SIZE
ffff8e040d203c80 ffff8e0442ca0e80 ffffffffc0a94000 - ffffffffc0a99000 20480
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffc5d281ed8480 7b612000 0 0 1 fffffc0000000
ffffffffc0a94000
Question 10
What is the function name in the malicious kernel module that performs cleanup?
Inspecting the kernel module, it does not seem to have a special function assigned for exit, so it calls the cleanup_module by default1 and this function is present.
crash> struct module ffffffffc0a94000
struct module {
state = 4464399,
list = {
next = 0x58000000002504c6,
prev = 0xf63155c35de58948
},
--- SNIP ---
exit = 0x0,
refcnt = {
counter = 0
},
ei_funcs = 0x0,
num_ei_funcs = 0
}
crash> sym -m core_helper
ffffffffc0a94000 MODULE START: core_helper
ffffffffc0a94000 (t) core_helper_read
ffffffffc0a94015 (t) core_helper_exit
ffffffffc0a94015 (T) cleanup_module
ffffffffc0a95024 (?) _note_9
ffffffffc0a9503c (?) _note_8
ffffffffc0a950e0 (?) proc_file_ops
ffffffffc0a96000 (?) __this_module
ffffffffc0a98000 MODULE END: core_helper
cleanup_module
Question 11
Before the kernel panic, a suspicious process was running with sudo privileges, What was the process name?
The task tree via ps -p shows multiple command started with sudo, among them also the cat process that crashed the kernel. Suspicious here is the httpd-worker as this is usually started by apache.
crash> ps -p
--- SNIP ---
PID: 0 TASK: ffffffff8741b440 CPU: 0 COMMAND: "swapper/0"
PID: 1 TASK: ffff8e04012dc980 CPU: 1 COMMAND: "systemd"
PID: 3778 TASK: ffff8e041067b100 CPU: 1 COMMAND: "systemd"
PID: 4486 TASK: ffff8e0410440000 CPU: 1 COMMAND: "gnome-terminal-"
PID: 4504 TASK: ffff8e041044b100 CPU: 0 COMMAND: "bash"
PID: 4552 TASK: ffff8e0410681880 CPU: 0 COMMAND: "sudo"
PID: 4553 TASK: ffff8e040e884980 CPU: 1 COMMAND: "sudo"
PID: 4554 TASK: ffff8e040e880000 CPU: 0 COMMAND: "su"
PID: 4555 TASK: ffff8e0410680000 CPU: 0 COMMAND: "bash"
PID: 8923 TASK: ffff8e04300e1880 CPU: 1 COMMAND: "sudo"
PID: 8924 TASK: ffff8e04300e6200 CPU: 0 COMMAND: "sudo"
PID: 8925 TASK: ffff8e041046b100 CPU: 0 COMMAND: "httpd-worker"
PID: 0 TASK: ffffffff8741b440 CPU: 0 COMMAND: "swapper/0"
PID: 1 TASK: ffff8e04012dc980 CPU: 1 COMMAND: "systemd"
PID: 3778 TASK: ffff8e041067b100 CPU: 1 COMMAND: "systemd"
PID: 8931 TASK: ffff8e04300e4980 CPU: 0 COMMAND: "threaded-ml"
PID: 0 TASK: ffffffff8741b440 CPU: 0 COMMAND: "swapper/0"
PID: 2 TASK: ffff8e04012d9880 CPU: 1 COMMAND: "kthreadd"
PID: 9056 TASK: ffff8e0410674980 CPU: 0 COMMAND: "kworker/u256:0"
PID: 0 TASK: ffffffff8741b440 CPU: 0 COMMAND: "swapper/0"
PID: 2 TASK: ffff8e04012d9880 CPU: 1 COMMAND: "kthreadd"
PID: 9230 TASK: ffff8e040dc4c980 CPU: 0 COMMAND: "kworker/0:1"
PID: 0 TASK: ffffffff8741b440 CPU: 0 COMMAND: "swapper/0"
PID: 1 TASK: ffff8e04012dc980 CPU: 1 COMMAND: "systemd"
PID: 3778 TASK: ffff8e041067b100 CPU: 1 COMMAND: "systemd"
PID: 4486 TASK: ffff8e0410440000 CPU: 1 COMMAND: "gnome-terminal-"
PID: 4504 TASK: ffff8e041044b100 CPU: 0 COMMAND: "bash"
PID: 4552 TASK: ffff8e0410681880 CPU: 0 COMMAND: "sudo"
PID: 4553 TASK: ffff8e040e884980 CPU: 1 COMMAND: "sudo"
PID: 4554 TASK: ffff8e040e880000 CPU: 0 COMMAND: "su"
PID: 4555 TASK: ffff8e0410680000 CPU: 0 COMMAND: "bash"
PID: 9236 TASK: ffff8e0479346200 CPU: 0 COMMAND: "cat"
httpd-worker