KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #0 r330544M: Tue Mar 6 18:19:16 CET 2018 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 6.0.0 (branches/release_60 325330) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 7 boot pages configured startup_alloc from "UMA Kegs", 6 boot pages left startup_alloc from "UMA Zones", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Zones", 2 boot pages left startup_alloc from "UMA Hash", 1 boot pages left Entering uma_startup1 with 0 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.08-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33179262976 (31642 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads arc4random: no preloaded entropy cache ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #11 Launched! Timecounter "TSC-low" frequency 1750038802 Hz quality 1000 random: entropy device external interface module_register_init: MOD_LOAD (vesa, 0xffffffff80fec590, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 netmap: loaded module [ath_hal] loaded nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 6 queue msgs: 9 admincnt: 1 igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: using 6 rx queues 6 tx queues iuhub0: 21 ports with 21 removable, self powered Mounting local file systems:. ugen2.2: at usbus2 uhub3 on uhub1 uhub3: on usbus2 Writing entropy file:ugen1.2: at usbus1 uhub4 on uhub2 uhub4: on usbus1 . Setting hostname: mercat1.netperf.freebsd.org. ugen0.2: at usbus0 uhub5 on uhub0 uhub5: on usbus0 uhub3: 8 ports with 8 removable, self powered uhub4: 6 ports with 6 removable, self powered uhub5: 4 ports with 3 removable, self powered ugen0.3: at usbus0 ukbd0 on uhub5 ukbd0: on usbus0 kbd2 at ukbd0 Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo igb0: flags=8843 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 nd6 options=21 media: Ethernet autoselect status: no carrier igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier ums0 on uhub5 ums0: on usbus0 Starting pflogd:ums0: 3 buttons and [Z] coordinates ID=0 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Setting date via ntp. 7 Mar 09:33:19 ntpdate[517]: no servers can be used, exiting Starting rpcbind. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 NFS access cache time=60 Starting casperd. lock order reversal: 1st 0xfffffe0000ec29f0 bufwait (bufwait) @ kern/vfs_bio.c:3736 2nd 0xfffff80004010800 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0xffffffff80b15413 at witness_debugger+0x73 #1 0xffffffff80b15294 at witness_checkorder+0xe34 #2 0xffffffff80ab7a08 at _sx_xlock+0x68 #3 0xffffffff80db5ded at ufsdirhash_add+0x4d #4 0xffffffff80dba90f at ufs_direnter+0x49f #5 0xffffffff80dc2f0c at ufs_mkdir+0x8dc #6 0xffffffff810e325d at VOP_MKDIR_APV+0xfd #7 0xffffffff80b8f056 at kern_mkdirat+0x1d6 #8 0xffffffff80f6ce3b at amd64_syscall+0x7ab #9 0xffffffff80f47a6d at fast_syscall_common+0x101 Clearing /tmp (X related). Starting statd. Starting lockd. Updating motd:Link state changed to up Wed Mar 7 09:33 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: root Mar 7 09:35:47 mercat1 login: ROOT LOGIN (root) ON ttyu1 Last login: Wed Mar 7 09:11:26 on ttyu1 FreeBSD 12.0-CURRENT (PHO) #0 r330544M: Tue Mar 6 18:19:16 CET 2018 root@mercat1:~ # Mar 7 09:36:19 mercat1 su: pho to root on /dev/pts/0 20180307 09:36:29 all (1/1): link2.sh lock order reversal: 1st 0xfffff8015332e068 ufs (ufs) @ kern/vfs_subr.c:2602 2nd 0xfffffe0000fbfd40 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:282 3rd 0xfffff80067adecc0 ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xffffffff80b15413 at witness_debugger+0x73 #1 0xffffffff80b15294 at witness_checkorder+0xe34 #2 0xffffffff80a840bb at lockmgr_xlock_hard+0x6b #3 0xffffffff80a84a9a at __lockmgr_args+0x51a #4 0xffffffff80db0baf at ffs_lock+0xaf #5 0xffffffff810e3f71 at VOP_LOCK1_APV+0xe1 #6 0xffffffff80b91f46 at _vn_lock+0x66 #7 0xffffffff80b802f8 at vget+0x98 #8 0xffffffff80b71e71 at vfs_hash_get+0xd1 #9 0xffffffff80dac45f at ffs_vgetf+0x3f #10 0xffffffff80da2366 at softdep_sync_buf+0xd16 #11 0xffffffff80db19b4 at ffs_syncvnode+0x294 #12 0xffffffff80daf14d at ffs_sync+0x20d #13 0xffffffff80b935b5 at vfs_write_suspend+0x175 #14 0xffffffff80b937e7 at vfs_write_suspend_umnt+0x47 #15 0xffffffff80daea2d at ffs_unmount+0xad #16 0xffffffff80b781c9 at dounmount+0x519 #17 0xffffffff80b77c20 at sys_unmount+0x310 lock order reversal: 1st 0xfffff801533937d0 ufs (ufs) @ kern/vfs_mount.c:1280 2nd 0xfffff801533852e0 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1371 stack backtrace: #0 0xffffffff80b15413 at witness_debugger+0x73 #1 0xffffffff80b15294 at witness_checkorder+0xe34 #2 0xffffffff80a83bee at lockmgr_lock_fast_path+0x17e #3 0xffffffff810e3f71 at VOP_LOCK1_APV+0xe1 #4 0xffffffff80b91f46 at _vn_lock+0x66 #5 0xffffffff80dac314 at ffs_flushfiles+0xa4 #6 0xffffffff80d8f6a2 at softdep_flushfiles+0x82 #7 0xffffffff80daea59 at ffs_unmount+0xd9 #8 0xffffffff80b781c9 at dounmount+0x519 #9 0xffffffff80b77c20 at sys_unmount+0x310 #10 0xffffffff80f6ce3b at amd64_syscall+0x7ab #11 0xffffffff80f47a6d at fast_syscall_common+0x101 GEOM leak: 204/285 20180307 09:40:00 all: done 20180307 10:40:27 all (1/1): link2.sh GEOM leak: 204/489 20180307 10:43:27 all: done ./all.sh -l 5 link2.sh ./all.sh: Command not found. root@mercat1:~ # ./all.sh -l 5 link2.sh ./all.sh: Command not found. root@mercat1:~ # 20180307 10:44:14 all (1/1): link2.sh GEOM leak: 204/693 20180307 10:47:46 all (1/1): link2.sh GEOM leak: 204/897 20180307 10:51:40 all (1/1): link2.sh GEOM leak: 204/1101 20180307 10:54:54 all (1/1): link2.sh GEOM leak: 204/1305 20180307 10:58:53 all (1/1): link2.sh GEOM leak: 204/1509 20180307 11:03:28 all: done 20180307 11:16:22 all (1/1): link2.sh GEOM leak: 204/1713 20180307 11:20:09 all (1/1): link2.sh GEOM leak: 204/1917 20180307 11:24:05 all (1/1): link2.sh GEOM leak: 204/2121 20180307 11:29:10 all (1/1): link2.sh GEOM leak: 204/2325 20180307 11:33:09 all (1/1): link2.sh GEOM leak: 204/2529 20180307 11:38:14 all (1/1): link2.sh GEOM leak: 204/2733 20180307 11:42:10 all (1/1): link2.sh GEOM leak: 204/2937 20180307 11:46:19 all (1/1): link2.sh GEOM leak: 204/3141 20180307 11:51:11 all (1/1): link2.sh GEOM leak: 204/3345 20180307 11:53:39 all (1/1): link2.sh GEOM leak: 204/3549 20180307 11:57:37 all (1/1): link2.sh GEOM leak: 204/3753 20180307 12:02:42 all (1/1): link2.sh GEOM leak: 204/3957 20180307 12:06:15 all (1/1): link2.sh GEOM leak: 204/4161 20180307 12:09:56 all (1/1): link2.sh GEOM leak: 204/4365 20180307 12:14:49 all (1/1): link2.sh GEOM leak: 204/4569 20180307 12:19:03 all (1/1): link2.sh GEOM leak: 204/4773 20180307 12:23:30 all (1/1): link2.sh GEOM leak: 204/4977 20180307 12:27:30 all (1/1): link2.sh GEOM leak: 204/5181 20180307 12:32:35 all (1/1): link2.sh Expensive timeout(9) function: 0xffffffff808e3190(0xfffffe009a5eecd8) 0.029814304 s GEOM leak: 204/5385 20180307 12:36:22 all (1/1): link2.sh GEOM leak: 204/5589 20180307 12:40:48 all (1/1): link2.sh GEOM leak: 204/5793 20180307 12:43:56 all (1/1): link2.sh GEOM leak: 204/5997 20180307 12:48:48 all (1/1): link2.sh GEOM leak: 204/6201 20180307 12:51:13 all (1/1): link2.sh pts leak: 1/3 GEOM leak: 204/6405 20180307 12:55:19 all (1/1): link2.sh GEOM leak: 204/6609 20180307 13:00:24 all: done 20180307 13:50:12 all (1/561): 256m.sh 20180307 13:50:13 all (2/561): advlock.sh GEOM leak: 4/6613 20180307 13:51:15 all (3/561): alternativeFlushPath.sh Mar 7 13:54:03 mercat1 kernel: pid 26943 (alternativeFlushPat), uid 0 inumber 240770 on /tmp: out of inodes freework leak: 6/17. 20180307 14:11:16 all (4/561): audit.sh 20180307 14:11:17 all (5/561): bench.sh GEOM leak: 20/6633 20180307 14:12:17 all (6/561): bio.sh lock order reversal: 1st 0xfffff80149212a48 syncer (syncer) @ kern/vfs_subr.c:2152 2nd 0xfffff8059ef0d068 ufs (ufs) @ kern/vfs_subr.c:2602 stack backtrace: #0 0xffffffff80b15413 at witness_debugger+0x73 #1 0xffffffff80b15294 at witness_checkorder+0xe34 #2 0xffffffff80a840bb at lockmgr_xlock_hard+0x6b #3 0xffffffff80a84a9a at __lockmgr_args+0x51a #4 0xffffffff80db0baf at ffs_lock+0xaf #5 0xffffffff810e3f71 at VOP_LOCK1_APV+0xe1 #6 0xffffffff80b91f46 at _vn_lock+0x66 #7 0xffffffff80b802f8 at vget+0x98 #8 0xffffffff80b82646 at vfs_msync+0xa6 #9 0xffffffff80b87666 at sync_fsync+0xc6 #10 0xffffffff810e2aed at VOP_FSYNC_APV+0xfd #11 0xffffffff80b854d4 at sched_sync+0x284 #12 0xffffffff80a6f614 at fork_exit+0x84 #13 0xffffffff80f47f3e at fork_trampoline+0xe GEOM leak: 4/6637 20180307 14:17:40 all (7/561): buildworld.sh freework leak: 1/18. GEOM leak: 4/6641 20180307 14:38:18 all (8/561): buildworld2.sh freework leak: 2/20. 20180307 14:38:18 all (9/561): burnin.sh GEOM leak: 4/6645 20180307 14:59:05 all (10/561): callout_reset_on.sh 20180307 15:01:31 all (11/561): callout_reset_on2.sh 20180307 15:11:33 all (12/561): churn.sh 20180307 15:21:35 all (13/561): cluster.sh 20180307 15:21:36 all (14/561): cmp.sh GEOM leak: 276/6921 20180307 15:27:07 all (15/561): context.sh 20180307 15:32:11 all (16/561): context2.sh 20180307 15:37:25 all (17/561): contigmalloc.sh Mar 7 15:37:29 mercat1 kernel: pid 774 (top), uid 2006, was killed: out of swap space Mar 7 15:37:30 mercat1 kernel: pid 586 (rpc.statd), uid 0, was killed: out of swap space Mar 7 15:37:30 mercat1 kernel: pid 589 (rpc.lockd), uid 0, was killed: out of swap space Mar 7 15:37:31 mercat1 kernel: pid 440 (devd), uid 0, was killed: out of swap space Mar 7 15:37:31 mercat1 kernel: pid 761 (csh), uid 0, was killed: out of swap space Mar 7 15:37:31 mercat1 kernel: pid 778 (csh), uid 0, was killed: out of swap space FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: Mar 7 15:37:32 mercat1 kernel: pid 781 (bash), uid 0, was killed: out of swap space Mar 7 15:37:32 mercat1 kernel: pid 773 (bash), uid 2006, was killed: out of swap space Mar 7 15:37:33 mercat1 kernel: pid 768 (bash), uid 2006, was killed: out of swap space Mar 7 15:37:33 mercat1 kernel: pid 23534 (bash), uid 2006, was killed: out of swap space pid 520 (rpcbind), uid 0, was killed: out of swap space pid 758 (dhclient), uid 65, was killed: out of swap space pid 696 (dhclient), uid 0, was killed: out of swap space pid 50974 (awk), uid 0, was killed: out of swap space pid 32764 (awk), uid 0, was killed: out of swap space pid 65926 (sh), uid 0, was killed: out of swap space panic: mutex vm page not owned at ../../../vm/vm_page.c:3121 cpuid = 6 time = 1520433454 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x18d/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- KDB: enter: panic [ thread pid 65994 tid 100747 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> run pho db:0:pho> bt Tracing pid 65994 tid 100747 td 0xfffff80248705560 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- db:0:pho> show allpcpu Current CPU: 6 cpuid = 0 dynamic pcpu = 0xa0f240 curthread = 0xfffff80003938000: pid 11 tid 100003 "idle: cpu0" curpcb = 0xfffffe007895ab80 fpcurthread = none idlethread = 0xfffff80003938000: tid 100003 "idle: cpu0" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69710 commontssp = 0xffffffff81f69710 rsp0 = 0xfffffe007895ab80 gs32p = 0xffffffff81f6ff58 ldt = 0xffffffff81f6ff98 tss = 0xffffffff81f6ff88 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00856c6240 curthread = 0xfffff80003937560: pid 11 tid 100004 "idle: cpu1" curpcb = 0xfffffe007895fb80 fpcurthread = none idlethread = 0xfffff80003937560: tid 100004 "idle: cpu1" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69778 commontssp = 0xffffffff81f69778 rsp0 = 0xfffffe007895fb80 gs32p = 0xffffffff81f6ffc0 ldt = 0xffffffff81f70000 tss = 0xffffffff81f6fff0 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00856cf240 curthread = 0xfffff80003937000: pid 11 tid 100005 "idle: cpu2" curpcb = 0xfffffe0078964b80 fpcurthread = none idlethread = 0xfffff80003937000: tid 100005 "idle: cpu2" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f697e0 commontssp = 0xffffffff81f697e0 rsp0 = 0xfffffe0078964b80 gs32p = 0xffffffff81f70028 ldt = 0xffffffff81f70068 tss = 0xffffffff81f70058 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00856d8240 curthread = 0xfffff80003936560: pid 11 tid 100006 "idle: cpu3" curpcb = 0xfffffe0078969b80 fpcurthread = none idlethread = 0xfffff80003936560: tid 100006 "idle: cpu3" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69848 commontssp = 0xffffffff81f69848 rsp0 = 0xfffffe0078969b80 gs32p = 0xffffffff81f70090 ldt = 0xffffffff81f700d0 tss = 0xffffffff81f700c0 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00856e1240 curthread = 0xfffff80003936000: pid 11 tid 100007 "idle: cpu4" curpcb = 0xfffffe007896eb80 fpcurthread = none idlethread = 0xfffff80003936000: tid 100007 "idle: cpu4" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f698b0 commontssp = 0xffffffff81f698b0 rsp0 = 0xfffffe007896eb80 gs32p = 0xffffffff81f700f8 ldt = 0xffffffff81f70138 tss = 0xffffffff81f70128 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00856ea240 curthread = 0xfffff80003935560: pid 11 tid 100008 "idle: cpu5" curpcb = 0xfffffe0078973b80 fpcurthread = none idlethread = 0xfffff80003935560: tid 100008 "idle: cpu5" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69918 commontssp = 0xffffffff81f69918 rsp0 = 0xfffffe0078973b80 gs32p = 0xffffffff81f70160 ldt = 0xffffffff81f701a0 tss = 0xffffffff81f70190 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00856f3240 curthread = 0xfffff80248705560: pid 65994 tid 100747 "ctest" curpcb = 0xfffffe00a1d58b80 fpcurthread = 0xfffff80248705560: pid 65994 "ctest" idlethread = 0xfffff80003935000: tid 100009 "idle: cpu6" curpmap = 0xfffff800139d5130 tssp = 0xffffffff81f69980 commontssp = 0xffffffff81f69980 rsp0 = 0xfffffe00a1d58b80 gs32p = 0xffffffff81f701c8 ldt = 0xffffffff81f70208 tss = 0xffffffff81f701f8 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00856fc240 curthread = 0xfffff80003949560: pid 11 tid 100010 "idle: cpu7" curpcb = 0xfffffe007897db80 fpcurthread = none idlethread = 0xfffff80003949560: tid 100010 "idle: cpu7" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f699e8 commontssp = 0xffffffff81f699e8 rsp0 = 0xfffffe007897db80 gs32p = 0xffffffff81f70230 ldt = 0xffffffff81f70270 tss = 0xffffffff81f70260 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe0085705240 curthread = 0xfffff80003949000: pid 11 tid 100011 "idle: cpu8" curpcb = 0xfffffe0078982b80 fpcurthread = none idlethread = 0xfffff80003949000: tid 100011 "idle: cpu8" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69a50 commontssp = 0xffffffff81f69a50 rsp0 = 0xfffffe0078982b80 gs32p = 0xffffffff81f70298 ldt = 0xffffffff81f702d8 tss = 0xffffffff81f702c8 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe008570e240 curthread = 0xfffff80003948560: pid 11 tid 100012 "idle: cpu9" curpcb = 0xfffffe0078987b80 fpcurthread = none idlethread = 0xfffff80003948560: tid 100012 "idle: cpu9" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69ab8 commontssp = 0xffffffff81f69ab8 rsp0 = 0xfffffe0078987b80 gs32p = 0xffffffff81f70300 ldt = 0xffffffff81f70340 tss = 0xffffffff81f70330 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe0085717240 curthread = 0xfffff80003948000: pid 11 tid 100013 "idle: cpu10" curpcb = 0xfffffe007898cb80 fpcurthread = none idlethread = 0xfffff80003948000: tid 100013 "idle: cpu10" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69b20 commontssp = 0xffffffff81f69b20 rsp0 = 0xfffffe007898cb80 gs32p = 0xffffffff81f70368 ldt = 0xffffffff81f703a8 tss = 0xffffffff81f70398 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe007f1e8240 curthread = 0xfffff80003947560: pid 11 tid 100014 "idle: cpu11" curpcb = 0xfffffe0078991b80 fpcurthread = none idlethread = 0xfffff80003947560: tid 100014 "idle: cpu11" curpmap = 0xffffffff81f3f698 tssp = 0xffffffff81f69b88 commontssp = 0xffffffff81f69b88 rsp0 = 0xfffffe0078991b80 gs32p = 0xffffffff81f703d0 ldt = 0xffffffff81f70410 tss = 0xffffffff81f70400 curvnet = 0 spin locks held: db:0:pho> show alllocks db:0:pho> show lockedvnods Locked vnodes db:0:pho> show allchains chain 1: thread 100161 (pid 65997, getty) sleeping on 0xfffff800041fd4b0 "ttyin" chain 2: thread 101430 (pid 65996, tail) sleeping on 0xfffff8027fb83000 "piperd" chain 3: thread 101733 (pid 65925, tee) sleeping on 0xfffff807bac538e8 "piperd" chain 4: thread 101693 (pid 65656, sleep) sleeping on 0xffffffff81d40760 "nanslp" chain 5: thread 100158 (pid 65651, sleep) sleeping on 0xffffffff81d4076b "nanslp" chain 6: thread 101122 (pid 50973, sh) sleeping on 0xfffff8060000c000 "wait" chain 7: thread 100374 (pid 50962, sh) sleeping on 0xfffff80013ddba70 "wait" chain 8: thread 100197 (pid 32763, sh) sleeping on 0xfffff800138f3a70 "wait" chain 9: thread 100232 (pid 32636, sh) sleeping on 0xfffff80153298000 "wait" chain 10: thread 100257 (pid 24670, sh) sleeping on 0xfffff80153853000 "wait" chain 11: thread 100165 (pid 723, getty) sleeping on 0xfffff800041fd8b0 "ttyin" chain 12: thread 100191 (pid 722, getty) sleeping on 0xfffff800133048b0 "ttyin" chain 13: thread 100190 (pid 721, getty) sleeping on 0xfffff80013304cb0 "ttyin" chain 14: thread 100189 (pid 720, getty) sleeping on 0xfffff800133050b0 "ttyin" chain 15: thread 100188 (pid 719, getty) sleeping on 0xfffff800133054b0 "ttyin" chain 16: thread 100187 (pid 718, getty) sleeping on 0xfffff800133058b0 "ttyin" chain 17: thread 100186 (pid 717, getty) sleeping on 0xfffff80013305cb0 "ttyin" chain 18: thread 100170 (pid 716, getty) sleeping on 0xfffff800133060b0 "ttyin" chain 19: thread 100140 (pid 715, getty) sleeping on 0xfffff800133064b0 "ttyin" chain 20: thread 100171 (pid 658, cron) sleeping on 0xffffffff81d40765 "nanslp" chain 21: thread 100157 (pid 654, sshd) sleeping on 0xfffff8001320f0c0 "select" chain 22: thread 100178 (pid 561, casperd) sleeping on 0xfffff80003f210c0 "select" chain 23: thread 100177 (pid 560, casperd) sleeping on 0xfffff80003ef78c0 "select" chain 24: thread 100132 (pid 33, syncer) sleeping on 0xffffffff81ee8b80 "syncer" chain 25: thread 100130 (pid 32, vnlru) sleeping on 0xfffff80013165000 "vlruwt" chain 26: thread 100128 (pid 31, bufdaemon) sleeping on 0xffffffff81ee8028 "psleep" chain 27: thread 100131 (pid 31, bufspacedaemon-0) sleeping on 0xffffffff81962c40 "-" chain 28: thread 100133 (pid 31, bufspacedaemon-1) sleeping on 0xffffffff8196ae40 "-" chain 29: thread 100135 (pid 31, bufspacedaemon-2) sleeping on 0xffffffff81973040 "-" chain 30: thread 100136 (pid 31, bufspacedaemon-3) sleeping on 0xffffffff8197b240 "-" chain 31: thread 100137 (pid 31, bufspacedaemon-4) sleeping on 0xffffffff81983440 "-" chain 32: thread 100138 (pid 31, bufspacedaemon-5) sleeping on 0xffffffff8198b640 "-" chain 33: thread 100139 (pid 31, bufspacedaemon-6) sleeping on 0xffffffff81993840 "-" chain 34: thread 100150 (pid 31, / worker) sleeping on 0xfffff80013c344e8 "sdflush" chain 35: thread 100154 (pid 31, /tmp worker) sleeping on 0xfffff800132b30e8 "sdflush" chain 36: thread 100127 (pid 30, vmdaemon) sleeping on 0xffffffff81ef4598 "psleep" chain 37: thread 100126 (pid 29, dom0) sleeping on 0xffffffff81f63520 "psleep" chain 38: thread 100129 (pid 29, laundry: dom0) sleeping on 0xffffffff81f63528 "launds" chain 39: thread 100134 (pid 29, uma) sleeping on 0xffffffff80dcec50 "umarcl" chain 40: thread 100125 (pid 28, enc_daemon1) sleeping on 0xfffff80013065000 "idle" chain 41: thread 100124 (pid 27, enc_daemon0) sleeping on 0xfffff80013065538 "idle" chain 42: thread 100118 (pid 26, rand_harvestq) sleeping on 0xffffffff81c13748 "-" chain 43: thread 100117 (pid 25, sctp_iterator) sleeping on 0xffffffff81f61ff0 "waiting_for_work" chain 44: thread 100116 (pid 24, soaiod4) sleeping on 0xffffffff81ee7a24 "-" chain 45: thread 100115 (pid 23, soaiod3) sleeping on 0xffffffff81ee7a24 "-" chain 46: thread 100114 (pid 22, soaiod2) sleeping on 0xffffffff81ee7a24 "-" chain 47: thread 100113 (pid 21, soaiod1) sleeping on 0xffffffff81ee7a24 "-" chain 48: thread 100094 (pid 20, usbus0) sleeping on 0xfffffe009c7e1460 "-" chain 49: thread 100095 (pid 20, usbus0) sleeping on 0xfffffe009c7e14b8 "-" chain 50: thread 100096 (pid 20, usbus0) sleeping on 0xfffffe009c7e1510 "-" chain 51: thread 100097 (pid 20, usbus0) sleeping on 0xfffffe009c7e1568 "-" chain 52: thread 100098 (pid 20, usbus0) sleeping on 0xfffffe009c7e15c0 "-" chain 53: thread 100100 (pid 20, usbus1) sleeping on 0xfffffe009a5eed10 "-" chain 54: thread 100101 (pid 20, usbus1) sleeping on 0xfffffe009a5eed68 "-" chain 55: thread 100102 (pid 20, usbus1) sleeping on 0xfffffe009a5eedc0 "-" chain 56: thread 100103 (pid 20, usbus1) sleeping on 0xfffffe009a5eee18 "-" chain 57: thread 100104 (pid 20, usbus1) sleeping on 0xfffffe009a5eee70 "-" chain 58: thread 100105 (pid 20, usbus2) sleeping on 0xfffffe009ca62d10 "-" chain 59: thread 100106 (pid 20, usbus2) sleeping on 0xfffffe009ca62d68 "-" chain 60: thread 100107 (pid 20, usbus2) sleeping on 0xfffffe009ca62dc0 "-" chain 61: thread 100108 (pid 20, usbus2) sleeping on 0xfffffe009ca62e18 "-" chain 62: thread 100109 (pid 20, usbus2) sleeping on 0xfffffe009ca62e70 "-" chain 63: thread 100089 (pid 19, doneq0) sleeping on 0xffffffff81aeea40 "-" chain 64: thread 100090 (pid 19, doneq1) sleeping on 0xffffffff81aeeac0 "-" chain 65: thread 100091 (pid 19, doneq2) sleeping on 0xffffffff81aeeb40 "-" chain 66: thread 100123 (pid 19, scanner) sleeping on 0xffffffff81aee908 "-" chain 67: thread 100088 (pid 18, crypto returns 11) sleeping on 0xfffff80003ad2450 "crypto_ret_wait" chain 68: thread 100087 (pid 17, crypto returns 10) sleeping on 0xfffff80003ad23f0 "crypto_ret_wait" chain 69: thread 100086 (pid 16, crypto returns 9) sleeping on 0xfffff80003ad2390 "crypto_ret_wait" chain 70: thread 100085 (pid 15, crypto returns 8) sleeping on 0xfffff80003ad2330 "crypto_ret_wait" chain 71: thread 100084 (pid 14, crypto returns 7) sleeping on 0xfffff80003ad22d0 "crypto_ret_wait" chain 72: thread 100083 (pid 9, crypto returns 6) sleeping on 0xfffff80003ad2270 "crypto_ret_wait" chain 73: thread 100082 (pid 8, crypto returns 5) sleeping on 0xfffff80003ad2210 "crypto_ret_wait" chain 74: thread 100081 (pid 7, crypto returns 4) sleeping on 0xfffff80003ad21b0 "crypto_ret_wait" chain 75: thread 100080 (pid 6, crypto returns 3) sleeping on 0xfffff80003ad2150 "crypto_ret_wait" chain 76: thread 100079 (pid 5, crypto returns 2) sleeping on 0xfffff80003ad20f0 "crypto_ret_wait" chain 77: thread 100078 (pid 4, crypto returns 1) sleeping on 0xfffff80003ad2090 "crypto_ret_wait" chain 78: thread 100077 (pid 3, crypto returns 0) sleeping on 0xfffff80003ad2030 "crypto_ret_wait" chain 79: thread 100076 (pid 2, crypto) sleeping on 0xffffffff81eeedd8 "crypto_wait" chain 80: thread 100060 (pid 13, g_event) sleeping on 0xffffffff81f3ea60 "-" chain 81: thread 100061 (pid 13, g_up) sleeping on 0xffffffff81f3ea68 "-" chain 82: thread 100062 (pid 13, g_down) sleeping on 0xffffffff81f3ea70 "-" chain 83: thread 100002 (pid 1, init) sleeping on 0xfffff80003934538 "wait" chain 84: thread 100001 (pid 10, audit) sleeping on 0xffffffff81f62cd8 "audit_worker_cv" chain 85: thread 100000 (pid 0, swapper) sleeping on 0xffffffff81f3eaa8 "swapin" chain 86: thread 100015 (pid 0, aiod_kick taskq) sleeping on 0xfffff80003970c00 "-" chain 87: thread 100018 (pid 0, thread taskq) sleeping on 0xfffff80003970400 "-" chain 88: thread 100020 (pid 0, kqueue_ctx taskq) sleeping on 0xfffff8000396fe00 "-" chain 89: thread 100021 (pid 0, if_io_tqg_0) sleeping on 0xfffff8000397e000 "-" chain 90: thread 100022 (pid 0, if_io_tqg_1) sleeping on 0xfffff8000397de00 "-" chain 91: thread 100023 (pid 0, if_io_tqg_2) sleeping on 0xfffff8000397dd00 "-" chain 92: thread 100024 (pid 0, if_io_tqg_3) sleeping on 0xfffff8000397dc00 "-" chain 93: thread 100025 (pid 0, if_io_tqg_4) sleeping on 0xfffff8000397db00 "-" chain 94: thread 100026 (pid 0, if_io_tqg_5) sleeping on 0xfffff8000397da00 "-" chain 95: thread 100027 (pid 0, if_io_tqg_6) sleeping on 0xfffff8000397d900 "-" chain 96: thread 100028 (pid 0, if_io_tqg_7) sleeping on 0xfffff8000397d800 "-" chain 97: thread 100029 (pid 0, if_io_tqg_8) sleeping on 0xfffff8000397d700 "-" chain 98: thread 100030 (pid 0, if_io_tqg_9) sleeping on 0xfffff8000397d600 "-" chain 99: thread 100031 (pid 0, if_io_tqg_10) sleeping on 0xfffff8000397d500 "-" chain 100: thread 100032 (pid 0, if_io_tqg_11) sleeping on 0xfffff8000397d400 "-" chain 101: thread 100033 (pid 0, if_config_tqg_0) sleeping on 0xfffff8000397d300 "-" chain 102: thread 100034 (pid 0, softirq_0) sleeping on 0xfffff8000397d200 "-" chain 103: thread 100035 (pid 0, softirq_1) sleeping on 0xfffff8000397d100 "-" chain 104: thread 100036 (pid 0, softirq_2) sleeping on 0xfffff8000397d000 "-" chain 105: thread 100037 (pid 0, softirq_3) sleeping on 0xfffff8000397ce00 "-" chain 106: thread 100038 (pid 0, softirq_4) sleeping on 0xfffff8000397cd00 "-" chain 107: thread 100039 (pid 0, softirq_5) sleeping on 0xfffff8000397cc00 "-" chain 108: thread 100040 (pid 0, softirq_6) sleeping on 0xfffff8000397cb00 "-" chain 109: thread 100041 (pid 0, softirq_7) sleeping on 0xfffff8000397ca00 "-" chain 110: thread 100042 (pid 0, softirq_8) sleeping on 0xfffff8000397c900 "-" chain 111: thread 100043 (pid 0, softirq_9) sleeping on 0xfffff8000397c800 "-" chain 112: thread 100044 (pid 0, softirq_10) sleeping on 0xfffff8000397c700 "-" chain 113: thread 100045 (pid 0, softirq_11) sleeping on 0xfffff8000397c600 "-" chain 114: thread 100063 (pid 0, firmware taskq) sleeping on 0xfffff80003a0c100 "-" chain 115: thread 100064 (pid 0, crypto_0) sleeping on 0xfffff80003a0be00 "-" chain 116: thread 100065 (pid 0, crypto_1) sleeping on 0xfffff80003a0be00 "-" chain 117: thread 100066 (pid 0, crypto_2) sleeping on 0xfffff80003a0be00 "-" chain 118: thread 100067 (pid 0, crypto_3) sleeping on 0xfffff80003a0be00 "-" chain 119: thread 100068 (pid 0, crypto_4) sleeping on 0xfffff80003a0be00 "-" chain 120: thread 100069 (pid 0, crypto_5) sleeping on 0xfffff80003a0be00 "-" chain 121: thread 100070 (pid 0, crypto_6) sleeping on 0xfffff80003a0be00 "-" chain 122: thread 100071 (pid 0, crypto_7) sleeping on 0xfffff80003a0be00 "-" chain 123: thread 100072 (pid 0, crypto_8) sleeping on 0xfffff80003a0be00 "-" chain 124: thread 100073 (pid 0, crypto_9) sleeping on 0xfffff80003a0be00 "-" chain 125: thread 100074 (pid 0, crypto_10) sleeping on 0xfffff80003a0be00 "-" chain 126: thread 100075 (pid 0, crypto_11) sleeping on 0xfffff80003a0be00 "-" chain 127: thread 100112 (pid 0, mca taskq) blocked on lockmgr (null)EXCL KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57e10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57e20 trap() at trap+0x5c/frame 0xfffffe00a1d57f30 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57f30 --- trap 0xc, rip = 0xffffffff80b0cfca, rsp = 0xfffffe00a1d58000, rbp = 0xfffffe00a1d58020 --- print_lockchain() at print_lockchain+0x3a/frame 0xfffffe00a1d58020 db_show_allchains() at db_show_allchains+0x80/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Script command 'show allchains' returned error db:0:pho> show mount 0xfffff80013881000 /dev/gpt/work on / (ufs) 0xfffff80013882000 devfs on /dev (devfs) 0xfffff80013c96000 /dev/gpt/tmp on /tmp (ufs) More info: show mount db:0:pho> show page vm_cnt.v_free_count: 7937405 vm_cnt.v_inactive_count: 10 vm_cnt.v_active_count: 836 vm_cnt.v_laundry_count: 471 vm_cnt.v_wire_count: 191741 vm_cnt.v_free_reserved: 10652 vm_cnt.v_free_min: 51303 vm_cnt.v_free_target: 173256 vm_cnt.v_inactive_target: 259884 db:0:pho> show bufqueues bqempty: 57866 bqdirty: 67 Buf domain 0 freebufs 14526 lofreebufs 219 hifreebufs 329 bufspace 3026944 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 567 wakeup 0 lim 25 CPU 0, 9, 4, 8, 1, 0, 0, 1, 1, 5, 2, 8, Buf domain 1 freebufs 14295 lofreebufs 219 hifreebufs 329 bufspace 9748480 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 736 wakeup 0 lim 25 CPU 2, 12, 6, 9, 3, 4, 0, 0, 2, 3, 1, 13, Buf domain 2 freebufs 11504 lofreebufs 219 hifreebufs 329 bufspace 64770048 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 3504 wakeup 0 lim 25 CPU 14, 18, 7, 2, 11, 2, 19, 0, 15, 1, 15, 10, Buf domain 3 freebufs 14435 lofreebufs 219 hifreebufs 329 bufspace 5361664 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 596 wakeup 0 lim 25 CPU 6, 19, 6, 18, 16, 1, 0, 3, 11, 5, 7, 8, Buf domain 4 freebufs 14476 lofreebufs 219 hifreebufs 329 bufspace 3162112 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 625 wakeup 0 lim 25 CPU 0, 16, 2, 0, 1, 1, 0, 0, 6, 1, 3, 1, Buf domain 5 freebufs 14459 lofreebufs 219 hifreebufs 329 bufspace 3801088 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 617 wakeup 0 lim 25 CPU 5, 15, 2, 2, 3, 1, 0, 2, 6, 1, 0, 17, Buf domain 6 freebufs 14459 lofreebufs 219 hifreebufs 329 bufspace 5267456 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 cleanq count 620 wakeup 0 lim 25 CPU 0, 9, 6, 3, 6, 5, 1, 2, 1, 3, 4, 12, db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 65997 1 65997 0 Ss+ ttyin 0xfffff800041fd4b0 getty 65996 1 24670 0 S+ piperd 0xfffff8027fb83000 tail 65994 1 24670 0 R+ CPU 6 ctest 65925 24670 24670 0 S+ piperd 0xfffff807bac538e8 tee 65656 50973 50962 0 SW+ nanslp 0xffffffff81d40760 sleep 65651 32763 32636 0 SW+ nanslp 0xffffffff81d4076b sleep 50973 50962 50962 0 SW+ wait 0xfffff8060000c000 sh 50962 1 50962 0 S+ wait 0xfffff80013ddba70 sh 32763 32636 32636 0 SW+ wait 0xfffff800138f3a70 sh 32636 1 32636 0 S+ wait 0xfffff80153298000 sh 24670 1 24670 0 S+ wait 0xfffff80153853000 sh 723 1 723 0 Ss+ ttyin 0xfffff800041fd8b0 getty 722 1 722 0 Ss+ ttyin 0xfffff800133048b0 getty 721 1 721 0 Ss+ ttyin 0xfffff80013304cb0 getty 720 1 720 0 Ss+ ttyin 0xfffff800133050b0 getty 719 1 719 0 Ss+ ttyin 0xfffff800133054b0 getty 718 1 718 0 Ss+ ttyin 0xfffff800133058b0 getty 717 1 717 0 Ss+ ttyin 0xfffff80013305cb0 getty 716 1 716 0 Ss+ ttyin 0xfffff800133060b0 getty 715 1 715 0 Ss+ ttyin 0xfffff800133064b0 getty 658 1 658 0 SWs nanslp 0xffffffff81d40765 cron 654 1 654 0 Ss select 0xfffff8001320f0c0 sshd 561 1 561 0 Ss select 0xfffff80003f210c0 casperd 560 1 560 0 Ss select 0xfffff80003ef78c0 casperd 33 0 0 0 DL syncer 0xffffffff81ee8b80 [syncer] 32 0 0 0 DL vlruwt 0xfffff80013165000 [vnlru] 31 0 0 0 DL (threaded) [bufdaemon] 100128 D psleep 0xffffffff81ee8028 [bufdaemon] 100131 D - 0xffffffff81962c40 [bufspacedaemon-0] 100133 D - 0xffffffff8196ae40 [bufspacedaemon-1] 100135 D - 0xffffffff81973040 [bufspacedaemon-2] 100136 D - 0xffffffff8197b240 [bufspacedaemon-3] 100137 D - 0xffffffff81983440 [bufspacedaemon-4] 100138 D - 0xffffffff8198b640 [bufspacedaemon-5] 100139 D - 0xffffffff81993840 [bufspacedaemon-6] 100150 D sdflush 0xfffff80013c344e8 [/ worker] 100154 D sdflush 0xfffff800132b30e8 [/tmp worker] 30 0 0 0 DL psleep 0xffffffff81ef4598 [vmdaemon] 29 0 0 0 DL (threaded) [pagedaemon] 100126 D psleep 0xffffffff81f63520 [dom0] 100129 D launds 0xffffffff81f63528 [laundry: dom0] 100134 D umarcl 0xffffffff80dcec50 [uma] 28 0 0 0 DL idle 0xfffff80013065000 [enc_daemon1] 27 0 0 0 DL idle 0xfffff80013065538 [enc_daemon0] 26 0 0 0 DL - 0xffffffff81c13748 [rand_harvestq] 25 0 0 0 DL waiting 0xffffffff81f61ff0 [sctp_iterator] 24 0 0 0 DL - 0xffffffff81ee7a24 [soaiod4] 23 0 0 0 DL - 0xffffffff81ee7a24 [soaiod3] 22 0 0 0 DL - 0xffffffff81ee7a24 [soaiod2] 21 0 0 0 DL - 0xffffffff81ee7a24 [soaiod1] 20 0 0 0 DL (threaded) [usb] 100094 D - 0xfffffe009c7e1460 [usbus0] 100095 D - 0xfffffe009c7e14b8 [usbus0] 100096 D - 0xfffffe009c7e1510 [usbus0] 100097 D - 0xfffffe009c7e1568 [usbus0] 100098 D - 0xfffffe009c7e15c0 [usbus0] 100100 D - 0xfffffe009a5eed10 [usbus1] 100101 D - 0xfffffe009a5eed68 [usbus1] 100102 D - 0xfffffe009a5eedc0 [usbus1] 100103 D - 0xfffffe009a5eee18 [usbus1] 100104 D - 0xfffffe009a5eee70 [usbus1] 100105 D - 0xfffffe009ca62d10 [usbus2] 100106 D - 0xfffffe009ca62d68 [usbus2] 100107 D - 0xfffffe009ca62dc0 [usbus2] 100108 D - 0xfffffe009ca62e18 [usbus2] 100109 D - 0xfffffe009ca62e70 [usbus2] 19 0 0 0 DL (threaded) [cam] 100089 D - 0xffffffff81aeea40 [doneq0] 100090 D - 0xffffffff81aeeac0 [doneq1] 100091 D - 0xffffffff81aeeb40 [doneq2] 100123 D - 0xffffffff81aee908 [scanner] 18 0 0 0 DL crypto_ 0xfffff80003ad2450 [crypto returns 11] 17 0 0 0 DL crypto_ 0xfffff80003ad23f0 [crypto returns 10] 16 0 0 0 DL crypto_ 0xfffff80003ad2390 [crypto returns 9] 15 0 0 0 DL crypto_ 0xfffff80003ad2330 [crypto returns 8] 14 0 0 0 DL crypto_ 0xfffff80003ad22d0 [crypto returns 7] 9 0 0 0 DL crypto_ 0xfffff80003ad2270 [crypto returns 6] 8 0 0 0 DL crypto_ 0xfffff80003ad2210 [crypto returns 5] 7 0 0 0 DL crypto_ 0xfffff80003ad21b0 [crypto returns 4] 6 0 0 0 DL crypto_ 0xfffff80003ad2150 [crypto returns 3] 5 0 0 0 DL crypto_ 0xfffff80003ad20f0 [crypto returns 2] 4 0 0 0 DL crypto_ 0xfffff80003ad2090 [crypto returns 1] 3 0 0 0 DL crypto_ 0xfffff80003ad2030 [crypto returns 0] 2 0 0 0 DL crypto_ 0xffffffff81eeedd8 [crypto] 13 0 0 0 DL (threaded) [geom] 100060 D - 0xffffffff81f3ea60 [g_event] 100061 D - 0xffffffff81f3ea68 [g_up] 100062 D - 0xffffffff81f3ea70 [g_down] 12 0 0 0 WL (threaded) [intr] 100016 I [swi6: task queue] 100017 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100046 I [swi3: vm] 100047 I [swi4: clock (0)] 100048 I [swi4: clock (1)] 100049 I [swi4: clock (2)] 100050 I [swi4: clock (3)] 100051 I [swi4: clock (4)] 100052 I [swi4: clock (5)] 100053 I [swi4: clock (6)] 100054 I [swi4: clock (7)] 100055 I [swi4: clock (8)] 100056 I [swi4: clock (9)] 100057 I [swi4: clock (10)] 100058 I [swi4: clock (11)] 100059 I [swi1: netisr 0] 100092 I [irq264: ahci0] 100093 I [irq19: xhci0] 100099 I [irq18: ehci0 ehci1] 100110 I [irq279: ahci1] 100111 I [swi0: uart uart] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 CanRun [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffff80003934538 [init] 10 0 0 0 DL audit_w 0xffffffff81f62cd8 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81f3eaa8 [swapper] 100015 D - 0xfffff80003970c00 [aiod_kick taskq] 100018 D - 0xfffff80003970400 [thread taskq] 100020 D - 0xfffff8000396fe00 [kqueue_ctx taskq] 100021 D - 0xfffff8000397e000 [if_io_tqg_0] 100022 D - 0xfffff8000397de00 [if_io_tqg_1] 100023 D - 0xfffff8000397dd00 [if_io_tqg_2] 100024 D - 0xfffff8000397dc00 [if_io_tqg_3] 100025 D - 0xfffff8000397db00 [if_io_tqg_4] 100026 D - 0xfffff8000397da00 [if_io_tqg_5] 100027 D - 0xfffff8000397d900 [if_io_tqg_6] 100028 D - 0xfffff8000397d800 [if_io_tqg_7] 100029 D - 0xfffff8000397d700 [if_io_tqg_8] 100030 D - 0xfffff8000397d600 [if_io_tqg_9] 100031 D - 0xfffff8000397d500 [if_io_tqg_10] 100032 D - 0xfffff8000397d400 [if_io_tqg_11] 100033 D - 0xfffff8000397d300 [if_config_tqg_0] 100034 D - 0xfffff8000397d200 [softirq_0] 100035 D - 0xfffff8000397d100 [softirq_1] 100036 D - 0xfffff8000397d000 [softirq_2] 100037 D - 0xfffff8000397ce00 [softirq_3] 100038 D - 0xfffff8000397cd00 [softirq_4] 100039 D - 0xfffff8000397cc00 [softirq_5] 100040 D - 0xfffff8000397cb00 [softirq_6] 100041 D - 0xfffff8000397ca00 [softirq_7] 100042 D - 0xfffff8000397c900 [softirq_8] 100043 D - 0xfffff8000397c800 [softirq_9] 100044 D - 0xfffff8000397c700 [softirq_10] 100045 D - 0xfffff8000397c600 [softirq_11] 100063 D - 0xfffff80003a0c100 [firmware taskq] 100064 D - 0xfffff80003a0be00 [crypto_0] 100065 D - 0xfffff80003a0be00 [crypto_1] 100066 D - 0xfffff80003a0be00 [crypto_2] 100067 D - 0xfffff80003a0be00 [crypto_3] 100068 D - 0xfffff80003a0be00 [crypto_4] 100069 D - 0xfffff80003a0be00 [crypto_5] 100070 D - 0xfffff80003a0be00 [crypto_6] 100071 D - 0xfffff80003a0be00 [crypto_7] 100072 D - 0xfffff80003a0be00 [crypto_8] 100073 D - 0xfffff80003a0be00 [crypto_9] 100074 D - 0xfffff80003a0be00 [crypto_10] 100075 D - 0xfffff80003a0be00 [crypto_11] 100112 D - 0xfffff8000609bc00 [mca taskq] 100119 D - 0xfffff80013174600 [acpi_task_0] 100120 D - 0xfffff80013174600 [acpi_task_1] 100121 D - 0xfffff80013174600 [acpi_task_2] 100122 D - 0xfffff80003971300 [CAM taskq] db:0:pho> allt Tracing command getty pid 65997 tid 100161 td 0xfffff80013b05000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee72610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee72640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee72680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee726e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee72710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee72770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ee727a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ee72810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ee72840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ee728a0 dofileread() at dofileread+0xba/frame 0xfffffe009ee728e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ee72930 sys_read() at sys_read+0x86/frame 0xfffffe009ee72980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee72ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee72ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command tail pid 65996 tid 101430 td 0xfffff80509259560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a8f44690 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a8f446c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a8f44700 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe00a8f44760 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00a8f44790 _sleep() at _sleep+0x359/frame 0xfffffe00a8f44840 pipe_read() at pipe_read+0x345/frame 0xfffffe00a8f448a0 dofileread() at dofileread+0xba/frame 0xfffffe00a8f448e0 kern_readv() at kern_readv+0x68/frame 0xfffffe00a8f44930 sys_read() at sys_read+0x86/frame 0xfffffe00a8f44980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a8f44ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a8f44ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800413cca, rsp = 0x7fffffffe478, rbp = 0x7fffffffe490 --- Tracing command ctest pid 65994 tid 100747 td 0xfffff80248705560 (CPU 6) kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command tee pid 65925 tid 101733 td 0xfffff802f79c0000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a9535690 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a95356c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a9535700 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe00a9535760 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00a9535790 _sleep() at _sleep+0x359/frame 0xfffffe00a9535840 pipe_read() at pipe_read+0x345/frame 0xfffffe00a95358a0 dofileread() at dofileread+0xba/frame 0xfffffe00a95358e0 kern_readv() at kern_readv+0x68/frame 0xfffffe00a9535930 sys_read() at sys_read+0x86/frame 0xfffffe00a9535980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a9535ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a9535ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800410cca, rsp = 0x7fffffffe708, rbp = 0x7fffffffe750 --- Tracing command sleep pid 65656 tid 101693 td 0xfffff804ee054000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57db0 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57dc0 trap() at trap+0x5c/frame 0xfffffe00a1d57ed0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57ed0 --- trap 0xc, rip = 0xffffffff80f45de6, rsp = 0xfffffe00a1d57fa0, rbp = 0xfffffe00a1d57fb0 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe00a1d57fb0 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command sleep pid 65651 tid 100158 td 0xfffff80003c5c000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57db0 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57dc0 trap() at trap+0x5c/frame 0xfffffe00a1d57ed0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57ed0 --- trap 0xc, rip = 0xffffffff80f45de6, rsp = 0xfffffe00a1d57fa0, rbp = 0xfffffe00a1d57fb0 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe00a1d57fb0 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command sh pid 50973 tid 101122 td 0xfffff80153309000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57db0 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57dc0 trap() at trap+0x5c/frame 0xfffffe00a1d57ed0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57ed0 --- trap 0xc, rip = 0xffffffff80f45de6, rsp = 0xfffffe00a1d57fa0, rbp = 0xfffffe00a1d57fb0 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe00a1d57fb0 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command sh pid 50962 tid 100374 td 0xfffff803213cf000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a1ca4550 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a1ca4580 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a1ca45c0 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe00a1ca4620 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00a1ca4650 _sleep() at _sleep+0x359/frame 0xfffffe00a1ca4700 kern_wait6() at kern_wait6+0x4c4/frame 0xfffffe00a1ca4790 sys_wait4() at sys_wait4+0x78/frame 0xfffffe00a1ca4980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1ca4ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1ca4ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003fe81a, rsp = 0x7fffffffe828, rbp = 0x7fffffffe8e0 --- Tracing command sh pid 32763 tid 100197 td 0xfffff800138fe560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57db0 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57dc0 trap() at trap+0x5c/frame 0xfffffe00a1d57ed0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57ed0 --- trap 0xc, rip = 0xffffffff80f45de6, rsp = 0xfffffe00a1d57fa0, rbp = 0xfffffe00a1d57fb0 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe00a1d57fb0 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command sh pid 32636 tid 100232 td 0xfffff801533e7000 sched_switch() at sched_switch+0x942/frame 0xfffffe009efd5550 mi_switch() at mi_switch+0x18c/frame 0xfffffe009efd5580 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009efd55c0 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009efd5620 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009efd5650 _sleep() at _sleep+0x359/frame 0xfffffe009efd5700 kern_wait6() at kern_wait6+0x4c4/frame 0xfffffe009efd5790 sys_wait4() at sys_wait4+0x78/frame 0xfffffe009efd5980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009efd5ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009efd5ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003fe81a, rsp = 0x7fffffffe828, rbp = 0x7fffffffe8e0 --- Tracing command sh pid 24670 tid 100257 td 0xfffff80153856560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a1c54550 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a1c54580 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a1c545c0 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe00a1c54620 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00a1c54650 _sleep() at _sleep+0x359/frame 0xfffffe00a1c54700 kern_wait6() at kern_wait6+0x4c4/frame 0xfffffe00a1c54790 sys_wait4() at sys_wait4+0x78/frame 0xfffffe00a1c54980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1c54ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1c54ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003fe81a, rsp = 0x7fffffffe5e8, rbp = 0x7fffffffe6a0 --- Tracing command getty pid 723 tid 100165 td 0xfffff80067042560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee86610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee86640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee86680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee866e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee86710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee86770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ee867a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ee86810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ee86840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ee868a0 dofileread() at dofileread+0xba/frame 0xfffffe009ee868e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ee86930 sys_read() at sys_read+0x86/frame 0xfffffe009ee86980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee86ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee86ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 722 tid 100191 td 0xfffff800678f8000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef08610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef08640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef08680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ef086e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ef08710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ef08770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ef087a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ef08810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ef08840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ef088a0 dofileread() at dofileread+0xba/frame 0xfffffe009ef088e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ef08930 sys_read() at sys_read+0x86/frame 0xfffffe009ef08980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef08ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef08ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 721 tid 100190 td 0xfffff800678f8560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef03610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef03640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef03680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ef036e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ef03710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ef03770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ef037a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ef03810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ef03840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ef038a0 dofileread() at dofileread+0xba/frame 0xfffffe009ef038e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ef03930 sys_read() at sys_read+0x86/frame 0xfffffe009ef03980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef03ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef03ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 720 tid 100189 td 0xfffff800678f9000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eefe610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eefe640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eefe680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eefe6e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eefe710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eefe770 tty_wait() at tty_wait+0x42/frame 0xfffffe009eefe7a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009eefe810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009eefe840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009eefe8a0 dofileread() at dofileread+0xba/frame 0xfffffe009eefe8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009eefe930 sys_read() at sys_read+0x86/frame 0xfffffe009eefe980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eefeab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eefeab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 719 tid 100188 td 0xfffff8001385d000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eef9610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eef9640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eef9680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eef96e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eef9710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eef9770 tty_wait() at tty_wait+0x42/frame 0xfffffe009eef97a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009eef9810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009eef9840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009eef98a0 dofileread() at dofileread+0xba/frame 0xfffffe009eef98e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009eef9930 sys_read() at sys_read+0x86/frame 0xfffffe009eef9980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eef9ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eef9ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 718 tid 100187 td 0xfffff8001385d560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eef4610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eef4640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eef4680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eef46e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eef4710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eef4770 tty_wait() at tty_wait+0x42/frame 0xfffffe009eef47a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009eef4810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009eef4840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009eef48a0 dofileread() at dofileread+0xba/frame 0xfffffe009eef48e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009eef4930 sys_read() at sys_read+0x86/frame 0xfffffe009eef4980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eef4ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eef4ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 717 tid 100186 td 0xfffff80013869560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eeef610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eeef640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eeef680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eeef6e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eeef710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eeef770 tty_wait() at tty_wait+0x42/frame 0xfffffe009eeef7a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009eeef810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009eeef840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009eeef8a0 dofileread() at dofileread+0xba/frame 0xfffffe009eeef8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009eeef930 sys_read() at sys_read+0x86/frame 0xfffffe009eeef980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eeefab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eeefab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 716 tid 100170 td 0xfffff8001386a560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee9f610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee9f640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee9f680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee9f6e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee9f710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee9f770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ee9f7a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ee9f810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ee9f840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ee9f8a0 dofileread() at dofileread+0xba/frame 0xfffffe009ee9f8e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ee9f930 sys_read() at sys_read+0x86/frame 0xfffffe009ee9f980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee9fab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee9fab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command getty pid 715 tid 100140 td 0xfffff80013900000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee09610 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee09640 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee09680 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee096e0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee09710 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee09770 tty_wait() at tty_wait+0x42/frame 0xfffffe009ee097a0 ttydisc_read() at ttydisc_read+0x22c/frame 0xfffffe009ee09810 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe009ee09840 devfs_read_f() at devfs_read_f+0xe0/frame 0xfffffe009ee098a0 dofileread() at dofileread+0xba/frame 0xfffffe009ee098e0 kern_readv() at kern_readv+0x68/frame 0xfffffe009ee09930 sys_read() at sys_read+0x86/frame 0xfffffe009ee09980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee09ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee09ab0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80042bcca, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 --- Tracing command cron pid 658 tid 100171 td 0xfffff8006702c560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00a1d57db0 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe00a1d57dc0 trap() at trap+0x5c/frame 0xfffffe00a1d57ed0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d57ed0 --- trap 0xc, rip = 0xffffffff80f45de6, rsp = 0xfffffe00a1d57fa0, rbp = 0xfffffe00a1d57fb0 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe00a1d57fb0 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe00a1d58050 db_command() at db_command+0x279/frame 0xfffffe00a1d58120 db_script_exec() at db_script_exec+0x188/frame 0xfffffe00a1d581c0 db_command() at db_command+0x279/frame 0xfffffe00a1d58290 db_command_loop() at db_command_loop+0x64/frame 0xfffffe00a1d582a0 db_trap() at db_trap+0xef/frame 0xfffffe00a1d58330 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe00a1d583a0 trap() at trap+0x328/frame 0xfffffe00a1d584b0 calltrap() at calltrap+0x8/frame 0xfffffe00a1d584b0 --- trap 0x3, rip = 0xffffffff80af40db, rsp = 0xfffffe00a1d58580, rbp = 0xfffffe00a1d58590 --- kdb_enter() at kdb_enter+0x3b/frame 0xfffffe00a1d58590 vpanic() at vpanic+0x1aa/frame 0xfffffe00a1d585f0 panic() at panic+0x43/frame 0xfffffe00a1d58650 __mtx_assert() at __mtx_assert+0xb4/frame 0xfffffe00a1d58660 vm_page_free_prep() at vm_page_free_prep+0xa6/frame 0xfffffe00a1d58690 vm_page_reclaim_contig_domain() at vm_page_reclaim_contig_domain+0x91d/frame 0xfffffe00a1d58800 kmem_alloc_contig_domain() at kmem_alloc_contig_domain+0x19c/frame 0xfffffe00a1d588a0 kmem_alloc_contig() at kmem_alloc_contig+0x6c/frame 0xfffffe00a1d58910 contigmalloc() at contigmalloc+0x25/frame 0xfffffe00a1d58950 cmalloc() at cmalloc+0x99/frame 0xfffffe00a1d58980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe00a1d58ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00a1d58ab0 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x80080fd8a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6c0 --- Tracing command sshd pid 654 tid 100157 td 0xfffff80013c54000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee5e5b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee5e5e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee5e620 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee5e680 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee5e6b0 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee5e710 seltdwait() at seltdwait+0x8d/frame 0xfffffe009ee5e750 kern_select() at kern_select+0x886/frame 0xfffffe009ee5e940 sys_select() at sys_select+0x56/frame 0xfffffe009ee5e980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee5eab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee5eab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800874caa, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command casperd pid 561 tid 100178 td 0xfffff80067041000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eec75b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eec75e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eec7620 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eec7680 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eec76b0 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eec7710 seltdwait() at seltdwait+0x8d/frame 0xfffffe009eec7750 kern_select() at kern_select+0x886/frame 0xfffffe009eec7940 sys_select() at sys_select+0x56/frame 0xfffffe009eec7980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eec7ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eec7ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800850caa, rsp = 0x7fffffffe7f8, rbp = 0x7fffffffece0 --- Tracing command casperd pid 560 tid 100177 td 0xfffff80067041560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eec25b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eec25e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eec2620 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009eec2680 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009eec26b0 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009eec2710 seltdwait() at seltdwait+0x8d/frame 0xfffffe009eec2750 kern_select() at kern_select+0x886/frame 0xfffffe009eec2940 sys_select() at sys_select+0x56/frame 0xfffffe009eec2980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eec2ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eec2ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800850caa, rsp = 0x7fffffffe658, rbp = 0x7fffffffe720 --- Tracing command syncer pid 33 tid 100132 td 0xfffff80013866000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005e18c0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005e18f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005e1930 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005e1970 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x221/frame 0xfffffe00005e19e0 sched_sync() at sched_sync+0x609/frame 0xfffffe00005e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vnlru pid 32 tid 100130 td 0xfffff80013866560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005d2870 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005d28a0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005d28e0 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005d2920 _sleep() at _sleep+0x307/frame 0xfffffe00005d29d0 vnlru_proc() at vnlru_proc+0xa4/frame 0xfffffe00005d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100128 td 0xfffff80013869000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005cd8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005cd910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005cd950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005cd990 _sleep() at _sleep+0x307/frame 0xfffffe00005cda40 buf_daemon() at buf_daemon+0xe8/frame 0xfffffe00005cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100131 td 0xfffff80003c5f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005d78b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005d78e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005d7920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005d7960 _sleep() at _sleep+0x307/frame 0xfffffe00005d7a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100133 td 0xfffff80003c5f000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005e68b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005e68e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005e6920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005e6960 _sleep() at _sleep+0x307/frame 0xfffffe00005e6a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100135 td 0xfffff80003c5e560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005f08b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005f08e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005f0920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005f0960 _sleep() at _sleep+0x307/frame 0xfffffe00005f0a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100136 td 0xfffff80003c5e000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005f58b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005f58e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005f5920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005f5960 _sleep() at _sleep+0x307/frame 0xfffffe00005f5a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100137 td 0xfffff80003c5d560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005fa8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005fa8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005fa920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005fa960 _sleep() at _sleep+0x307/frame 0xfffffe00005faa10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100138 td 0xfffff80003c5d000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005ff8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005ff8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005ff920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005ff960 _sleep() at _sleep+0x307/frame 0xfffffe00005ffa10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100139 td 0xfffff80003c5c560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee048b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee048e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee04920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee04960 _sleep() at _sleep+0x307/frame 0xfffffe009ee04a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe009ee04a70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee04ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee04ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100150 td 0xfffff8001386b560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee3b8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee3b910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee3b950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee3b990 _sleep() at _sleep+0x307/frame 0xfffffe009ee3ba40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe009ee3ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee3bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee3bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100154 td 0xfffff80013067000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee4f8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee4f910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee4f950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee4f990 _sleep() at _sleep+0x307/frame 0xfffffe009ee4fa40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe009ee4fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee4fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee4fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 30 tid 100127 td 0xfffff8001307a560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005c88a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005c88d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005c8910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005c8940 _sleep() at _sleep+0x367/frame 0xfffffe00005c89f0 vm_daemon() at vm_daemon+0x85/frame 0xfffffe00005c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100126 td 0xfffff80013167000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005c3810 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005c3840 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005c3880 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005c38c0 _sleep() at _sleep+0x307/frame 0xfffffe00005c3970 vm_pageout_worker() at vm_pageout_worker+0x294/frame 0xfffffe00005c3a50 vm_pageout() at vm_pageout+0x133/frame 0xfffffe00005c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100129 td 0xfffff8001386c560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005dc840 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005dc870 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005dc8b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005dc8e0 _sleep() at _sleep+0x367/frame 0xfffffe00005dc990 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xc05/frame 0xfffffe00005dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100134 td 0xfffff8001386c000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005eb8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005eb910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005eb950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005eb990 _sleep() at _sleep+0x307/frame 0xfffffe00005eba40 uma_reclaim_worker() at uma_reclaim_worker+0xda/frame 0xfffffe00005eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 28 tid 100125 td 0xfffff80013077560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005be8a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005be8d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005be910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005be940 _sleep() at _sleep+0x367/frame 0xfffffe00005be9f0 enc_daemon() at enc_daemon+0x1ca/frame 0xfffffe00005bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 27 tid 100124 td 0xfffff80013078000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005b98a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005b98d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005b9910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005b9940 _sleep() at _sleep+0x367/frame 0xfffffe00005b99f0 enc_daemon() at enc_daemon+0x1ca/frame 0xfffffe00005b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 26 tid 100118 td 0xfffff80013171000 sched_switch() at sched_switch+0x942/frame 0xfffffe000059b8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000059b8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000059b920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe000059b960 _sleep() at _sleep+0x307/frame 0xfffffe000059ba10 random_kthread() at random_kthread+0x57/frame 0xfffffe000059ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000059bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000059bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sctp_iterator pid 25 tid 100117 td 0xfffff80013171560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000596900 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000596930 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000596970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005969a0 _sleep() at _sleep+0x367/frame 0xfffffe0000596a50 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe0000596a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000596ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000596ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 24 tid 100116 td 0xfffff80013078560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005908d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000590900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000590940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000590980 _sleep() at _sleep+0x307/frame 0xfffffe0000590a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000590a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000590ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000590ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 23 tid 100115 td 0xfffff80013079000 sched_switch() at sched_switch+0x942/frame 0xfffffe000058b8d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000058b900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000058b940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe000058b980 _sleep() at _sleep+0x307/frame 0xfffffe000058ba30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe000058ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000058bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000058bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 22 tid 100114 td 0xfffff80013079560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005868d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000586900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000586940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000586980 _sleep() at _sleep+0x307/frame 0xfffffe0000586a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000586a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000586ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000586ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 21 tid 100113 td 0xfffff8001307a000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005818d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000581900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000581940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000581980 _sleep() at _sleep+0x307/frame 0xfffffe0000581a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000581a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000581ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000581ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100094 td 0xfffff80004020560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000522930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000522960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005229a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005229d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000522a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000522a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000522ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000522ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100095 td 0xfffff80004020000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000527930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000527960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005279a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005279d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000527a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000527a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000527ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000527ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100096 td 0xfffff8000401f560 sched_switch() at sched_switch+0x942/frame 0xfffffe000052c930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000052c960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000052c9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000052c9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000052ca30 usb_process() at usb_process+0x12b/frame 0xfffffe000052ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000052cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000052cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100097 td 0xfffff8000401f000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000531930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000531960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005319a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005319d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000531a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000531a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000531ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000531ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100098 td 0xfffff80004017560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000536930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000536960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005369a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005369d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000536a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000536a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000536ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000536ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100100 td 0xfffff80003c60560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000540930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000540960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005409a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005409d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000540a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000540a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000540ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000540ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100101 td 0xfffff80004194000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000545930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000545960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005459a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005459d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000545a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000545a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000545ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000545ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100102 td 0xfffff80004193560 sched_switch() at sched_switch+0x942/frame 0xfffffe000054a930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000054a960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000054a9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000054a9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000054aa30 usb_process() at usb_process+0x12b/frame 0xfffffe000054aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000054aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000054aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100103 td 0xfffff80004193000 sched_switch() at sched_switch+0x942/frame 0xfffffe000054f930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000054f960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000054f9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000054f9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000054fa30 usb_process() at usb_process+0x12b/frame 0xfffffe000054fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000054fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000054fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100104 td 0xfffff80004192560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000554930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000554960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005549a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005549d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000554a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000554a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000554ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000554ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100105 td 0xfffff80006072560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000559930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000559960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005599a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005599d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000559a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000559a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000559ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000559ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100106 td 0xfffff80006072000 sched_switch() at sched_switch+0x942/frame 0xfffffe000055e930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000055e960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000055e9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000055e9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000055ea30 usb_process() at usb_process+0x12b/frame 0xfffffe000055ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000055eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000055eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100107 td 0xfffff80006071560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000563930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000563960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005639a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005639d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000563a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000563a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000563ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000563ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100108 td 0xfffff80006071000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000568930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000568960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005689a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005689d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000568a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000568a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000568ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000568ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100109 td 0xfffff80006070560 sched_switch() at sched_switch+0x942/frame 0xfffffe000056d930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000056d960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000056d9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000056d9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000056da30 usb_process() at usb_process+0x12b/frame 0xfffffe000056da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000056dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000056dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100089 td 0xfffff80003ad9000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005098d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000509900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000509940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000509970 _sleep() at _sleep+0x367/frame 0xfffffe0000509a20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0000509a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000509ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000509ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100090 td 0xfffff80003ace560 sched_switch() at sched_switch+0x942/frame 0xfffffe000050e8d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000050e900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000050e940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000050e970 _sleep() at _sleep+0x367/frame 0xfffffe000050ea20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe000050ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000050eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000050eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100091 td 0xfffff80003c60000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005138d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000513900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000513940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000513970 _sleep() at _sleep+0x367/frame 0xfffffe0000513a20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0000513a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000513ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000513ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100123 td 0xfffff80013167560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005b48c0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005b48f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005b4930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005b4960 _sleep() at _sleep+0x367/frame 0xfffffe00005b4a10 xpt_scanner_thread() at xpt_scanner_thread+0x93/frame 0xfffffe00005b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 11 pid 18 tid 100088 td 0xfffff80003ad9560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005048b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005048e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000504920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000504950 _sleep() at _sleep+0x367/frame 0xfffffe0000504a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe0000504a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000504ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000504ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 10 pid 17 tid 100087 td 0xfffff80003ada000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004ff8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004ff8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004ff920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004ff950 _sleep() at _sleep+0x367/frame 0xfffffe00004ffa00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 9 pid 16 tid 100086 td 0xfffff80003ada560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004fa8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004fa8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004fa920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004fa950 _sleep() at _sleep+0x367/frame 0xfffffe00004faa00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 8 pid 15 tid 100085 td 0xfffff80003adb000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004f58b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004f58e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004f5920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004f5950 _sleep() at _sleep+0x367/frame 0xfffffe00004f5a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 7 pid 14 tid 100084 td 0xfffff80003adb560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004f08b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004f08e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004f0920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004f0950 _sleep() at _sleep+0x367/frame 0xfffffe00004f0a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 6 pid 9 tid 100083 td 0xfffff80003adc000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004eb8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004eb8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004eb920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004eb950 _sleep() at _sleep+0x367/frame 0xfffffe00004eba00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 5 pid 8 tid 100082 td 0xfffff80003adc560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004e68b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004e68e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004e6920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004e6950 _sleep() at _sleep+0x367/frame 0xfffffe00004e6a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 4 pid 7 tid 100081 td 0xfffff80003aca000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004e18b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004e18e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004e1920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004e1950 _sleep() at _sleep+0x367/frame 0xfffffe00004e1a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 3 pid 6 tid 100080 td 0xfffff80003aca560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004dc8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004dc8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004dc920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004dc950 _sleep() at _sleep+0x367/frame 0xfffffe00004dca00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 2 pid 5 tid 100079 td 0xfffff80003acb000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004d78b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004d78e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004d7920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004d7950 _sleep() at _sleep+0x367/frame 0xfffffe00004d7a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 1 pid 4 tid 100078 td 0xfffff80003acb560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004d28b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004d28e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004d2920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004d2950 _sleep() at _sleep+0x367/frame 0xfffffe00004d2a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 0 pid 3 tid 100077 td 0xfffff80003acc000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004cd8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004cd8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004cd920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004cd950 _sleep() at _sleep+0x367/frame 0xfffffe00004cda00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100076 td 0xfffff80003acc560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004c88e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004c8910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004c8950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004c8980 _sleep() at _sleep+0x367/frame 0xfffffe00004c8a30 crypto_proc() at crypto_proc+0x2ba/frame 0xfffffe00004c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100060 td 0xfffff80003a2f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004788e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000478910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000478950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000478980 _sleep() at _sleep+0x367/frame 0xfffffe0000478a30 g_run_events() at g_run_events+0x4f/frame 0xfffffe0000478a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000478ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000478ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100061 td 0xfffff80003a2f000 sched_switch() at sched_switch+0x942/frame 0xfffffe000047d8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000047d910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000047d950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000047d980 _sleep() at _sleep+0x367/frame 0xfffffe000047da30 g_io_schedule_up() at g_io_schedule_up+0xa5/frame 0xfffffe000047da60 g_up_procbody() at g_up_procbody+0x6d/frame 0xfffffe000047da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000047dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000047dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100062 td 0xfffff80003a2e560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004828d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000482900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000482940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000482970 _sleep() at _sleep+0x367/frame 0xfffffe0000482a20 g_io_schedule_down() at g_io_schedule_down+0x14c/frame 0xfffffe0000482a60 g_down_procbody() at g_down_procbody+0x6d/frame 0xfffffe0000482a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000482ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000482ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100016 td 0xfffff80003946560 sched_switch() at sched_switch+0x942/frame 0xfffffe007899b9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe007899ba20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe007899ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe007899bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007899bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100017 td 0xfffff80003946000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789a09f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789a0a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe00789a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffff80003976000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100046 td 0xfffff80003a26560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100047 td 0xfffff80003a26000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004379f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000437a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000437a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000437ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000437ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100048 td 0xfffff80003a25560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100049 td 0xfffff80003a25000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100050 td 0xfffff80003a24560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xfffff80003a24000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xfffff80003a18560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100053 td 0xfffff80003a18000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xfffff800039fe560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100055 td 0xfffff80003a32000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100056 td 0xfffff80003a31560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100057 td 0xfffff80003a31000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100058 td 0xfffff80003a30560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xfffff80003a30000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004739f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000473a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000473a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000473ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000473ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100092 td 0xfffff80004021560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005189f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000518a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000518a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000518ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000518ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100093 td 0xfffff80004021000 sched_switch() at sched_switch+0x942/frame 0xfffffe000051d9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000051da20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe000051da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000051dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000051dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100099 td 0xfffff80004017000 sched_switch() at sched_switch+0x942/frame 0xfffffe000053b9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000053ba20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe000053ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000053bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000053bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100110 td 0xfffff80006070000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100111 td 0xfffff80006068560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005779f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000577a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000577a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000577ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000577ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff80003938000 (CPU 0) cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff81ef88d0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xffffffff81ef88f0 trap() at trap+0x49/frame 0xffffffff81ef8a00 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81ef8a00 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe007895a910, rbp = 0xfffffe007895a910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff80003937560 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00007ffdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00007ffe10 trap() at trap+0x49/frame 0xfffffe00007fff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00007fff20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe007895f910, rbp = 0xfffffe007895f910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007895f910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895f960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895f980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895f9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff80003937000 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071b2df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071b2e10 trap() at trap+0x49/frame 0xfffffe00071b2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071b2f20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078964910, rbp = 0xfffffe0078964910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078964910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078964960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078964980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789649a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078964a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078964ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078964ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff80003936560 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071bbdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071bbe10 trap() at trap+0x49/frame 0xfffffe00071bbf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071bbf20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078969910, rbp = 0xfffffe0078969910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078969910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078969960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078969980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789699a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078969a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078969ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078969ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff80003936000 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071c4df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071c4e10 trap() at trap+0x49/frame 0xfffffe00071c4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071c4f20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe007896e910, rbp = 0xfffffe007896e910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007896e910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007896e960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007896e980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007896e9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007896ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe007896eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007896eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff80003935560 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071cddf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071cde10 trap() at trap+0x49/frame 0xfffffe00071cdf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071cdf20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078973910, rbp = 0xfffffe0078973910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078973910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078973960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078973980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789739a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078973a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078973ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078973ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff80003935000 sched_switch() at sched_switch+0x942/frame 0xfffffe0078978850 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078978880 critical_exit() at critical_exit+0x8c/frame 0xfffffe00789788a0 ipi_bitmap_handler() at ipi_bitmap_handler+0x7b/frame 0xfffffe00789788e0 ipi_intr_bitmap_handler_u() at ipi_intr_bitmap_handler_u+0x98/frame 0xfffffe0078978a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078978ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078978ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff80003949560 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071dfdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071dfe10 trap() at trap+0x49/frame 0xfffffe00071dff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071dff20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe007897d910, rbp = 0xfffffe007897d910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007897d910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007897d960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007897d980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007897d9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007897da70 fork_exit() at fork_exit+0x84/frame 0xfffffe007897dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007897dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff80003949000 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071e8df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071e8e10 trap() at trap+0x49/frame 0xfffffe00071e8f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071e8f20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078982910, rbp = 0xfffffe0078982910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078982910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078982960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078982980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789829a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078982a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078982ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078982ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff80003948560 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071f1df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071f1e10 trap() at trap+0x49/frame 0xfffffe00071f1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071f1f20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078987910, rbp = 0xfffffe0078987910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078987910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078987960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078987980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789879a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078987a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078987ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078987ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff80003948000 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071fadf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071fae10 trap() at trap+0x49/frame 0xfffffe00071faf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071faf20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe007898c910, rbp = 0xfffffe007898c910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007898c910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007898c960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007898c980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007898c9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007898ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe007898cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007898cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff80003947560 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071ffdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071ffe10 trap() at trap+0x49/frame 0xfffffe00071fff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071fff20 --- trap 0x13, rip = 0xffffffff810c86e6, rsp = 0xfffffe0078991910, rbp = 0xfffffe0078991910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078991910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078991960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078991980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789919a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078991a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078991ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078991ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff80003938560 sched_switch() at sched_switch+0x942/frame 0xfffffe0078955550 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078955580 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789555c0 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe0078955620 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0078955650 _sleep() at _sleep+0x359/frame 0xfffffe0078955700 kern_wait6() at kern_wait6+0x4c4/frame 0xfffffe0078955790 sys_wait4() at sys_wait4+0x78/frame 0xfffffe0078955980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe0078955ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0078955ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2a716a, rsp = 0x7fffffffe788, rbp = 0x7fffffffe7f0 --- Tracing command audit pid 10 tid 100001 td 0xfffff80003939000 sched_switch() at sched_switch+0x942/frame 0xfffffe0078950910 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078950940 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0078950980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789509b0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0078950a10 audit_worker() at audit_worker+0x93/frame 0xfffffe0078950a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078950ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078950ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81f3efe0 sched_switch() at sched_switch+0x942/frame 0xffffffff824f29c0 mi_switch() at mi_switch+0x18c/frame 0xffffffff824f29f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffffff824f2a30 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xffffffff824f2a70 _sleep() at _sleep+0x307/frame 0xffffffff824f2b20 swapper() at swapper+0x247/frame 0xffffffff824f2b70 btext() at btext+0x2c Tracing command kernel pid 0 tid 100015 td 0xfffff80003947000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789968f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078996920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0078996960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0078996990 _sleep() at _sleep+0x367/frame 0xfffffe0078996a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0078996a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078996ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078996ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffff80003939560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789a58f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789a5920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789a5960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789a5990 _sleep() at _sleep+0x367/frame 0xfffffe00789a5a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00789a5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789a5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789a5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffff80003975560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789af990 _sleep() at _sleep+0x367/frame 0xfffffe00789afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00789afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffff80003975000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789b4920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789b4950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789b4990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789b49c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789b4a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffff80003974560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789b9920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789b9950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789b9990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789b99c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789b9a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffff80003974000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789be920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789be950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789be990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789be9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789bea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffff80003973560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789c3920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789c3950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789c3990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789c39c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789c3a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffff80003973000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789c8920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789c8950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789c8990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789c89c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789c8a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffff80003972560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789cd920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789cd950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789cd990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789cd9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789cda40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffff80003972000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789d2920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789d2950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789d2990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789d29c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789d2a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffff8000398a560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789d7920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789d7950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789d7990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789d79c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789d7a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffff8000398a000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789dc920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789dc950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789dc990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789dc9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789dca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffff80003985560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789e1920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789e1950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789e1990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789e19c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789e1a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffff80003985000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789e6920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789e6950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789e6990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789e69c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789e6a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffff80003984560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789eb920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789eb950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789eb990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789eb9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789eba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffff80003984000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789f0920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789f0950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789f0990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789f09c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789f0a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffff80003983560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789f5920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789f5950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789f5990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789f59c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789f5a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffff80003983000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789fa920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789fa950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789fa990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789fa9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789faa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffff80003976560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789ff920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789ff950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789ff990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789ff9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789ffa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffff800039fe000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000405920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000405950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000405990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004059c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000405a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000405a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000405ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000405ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffff800039fd560 sched_switch() at sched_switch+0x942/frame 0xfffffe000040a920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000040a950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000040a990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000040a9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000040aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000040aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000040aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000040aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffff800039fd000 sched_switch() at sched_switch+0x942/frame 0xfffffe000040f920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000040f950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000040f990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000040f9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000040fa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000040fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000040fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000040fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffff800039fc560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000414920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000414950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000414990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004149c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000414a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000414a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000414ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000414ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffff800039fc000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000419920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000419950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000419990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004199c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000419a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000419a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000419ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000419ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffff800039fb560 sched_switch() at sched_switch+0x942/frame 0xfffffe000041e920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000041e950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000041e990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000041e9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000041ea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000041ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000041eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000041eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffff800039fb000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000423920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000423950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000423990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004239c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000423a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000423a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000423ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000423ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffff800039fa560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000428920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000428950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000428990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004289c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000428a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000428a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000428ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000428ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffff800039fa000 sched_switch() at sched_switch+0x942/frame 0xfffffe000042d920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000042d950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000042d990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000042d9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000042da40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000042da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000042dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000042dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffff80003a2e000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004878f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000487920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000487960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000487990 _sleep() at _sleep+0x367/frame 0xfffffe0000487a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000487a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000487ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000487ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffff80003ac9560 sched_switch() at sched_switch+0x942/frame 0xfffffe000048c8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000048c920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000048c960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000048c990 _sleep() at _sleep+0x367/frame 0xfffffe000048ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe000048ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000048cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000048cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffff80003ac9000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004918f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000491920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000491960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000491990 _sleep() at _sleep+0x367/frame 0xfffffe0000491a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000491a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000491ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000491ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffff80003ac8560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004968f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000496920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000496960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000496990 _sleep() at _sleep+0x367/frame 0xfffffe0000496a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000496a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000496ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000496ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffff80003ac8000 sched_switch() at sched_switch+0x942/frame 0xfffffe000049b8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000049b920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000049b960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000049b990 _sleep() at _sleep+0x367/frame 0xfffffe000049ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe000049ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000049bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000049bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffff80003ac7560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004a08f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004a0920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004a0960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004a0990 _sleep() at _sleep+0x367/frame 0xfffffe00004a0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffff80003ac7000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004a58f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004a5920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004a5960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004a5990 _sleep() at _sleep+0x367/frame 0xfffffe00004a5a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004a5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004a5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004a5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffff80003ac6560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004aa8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004aa920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004aa960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004aa990 _sleep() at _sleep+0x367/frame 0xfffffe00004aaa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004aaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004aaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004aaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffff80003ac6000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004af990 _sleep() at _sleep+0x367/frame 0xfffffe00004afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffff80003a32560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004b48f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004b4920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004b4960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004b4990 _sleep() at _sleep+0x367/frame 0xfffffe00004b4a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffff80003ace000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004b98f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004b9920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004b9960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004b9990 _sleep() at _sleep+0x367/frame 0xfffffe00004b9a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffff80003acd560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004be8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004be920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004be960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004be990 _sleep() at _sleep+0x367/frame 0xfffffe00004bea40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffff80003acd000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004c38f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004c3920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004c3960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004c3990 _sleep() at _sleep+0x367/frame 0xfffffe00004c3a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xfffff80006068000 sched_switch() at sched_switch+0x942/frame 0xfffffe000057c920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000057c950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000057c990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000057c9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000057ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000057ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000057cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000057cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100119 td 0xfffff80013170560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005a0920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005a0950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005a0990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005a09c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00005a0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00005a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100120 td 0xfffff80013170000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005a5920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005a5950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005a5990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005a59c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00005a5a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00005a5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005a5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100121 td 0xfffff8001316f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005aa920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005aa950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005aa990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005aa9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00005aaa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00005aaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005aaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005aaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100122 td 0xfffff8001316f000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005af990 _sleep() at _sleep+0x367/frame 0xfffffe00005afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00005afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:pho> dump Dumping 1592 out of 32625 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:0:pho> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 6 (kgdb) bt #0 doadump (textdump=0x0) at pcpu.h:230 #1 0xffffffff803e196b in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:574 #2 0xffffffff803e1739 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #3 0xffffffff803e64c8 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #4 0xffffffff803e1739 in db_command (cmd_table=) at ../../../ddb/db_command.c:481 #5 0xffffffff803e14b4 in db_command_loop () at ../../../ddb/db_command.c:534 #6 0xffffffff803e46df in db_trap (type=, code=) at ../../../ddb/db_main.c:250 #7 0xffffffff80af4a03 in kdb_trap (type=0x3, code=0xffff0ff0, tf=) at ../../../kern/subr_kdb.c:697 #8 0xffffffff80f6b868 in trap (frame=0xfffffe00a1d584c0) at ../../../amd64/amd64/trap.c:547 #9 0xffffffff80f4721c in calltrap () at ../../../amd64/amd64/exception.S:232 #10 0xffffffff80af40db in kdb_enter (why=0xffffffff814dcdf2 "panic", msg=) at cpufunc.h:65 #11 0xffffffff80aaf44a in vpanic (fmt=, ap=0xfffffe00a1d58630) at ../../../kern/kern_shutdown.c:801 #12 0xffffffff80aaf4d3 in panic (fmt=) at ../../../kern/kern_shutdown.c:739 #13 0xffffffff80a8ea34 in __mtx_assert (c=0xffffffff81d3cdf8, what=0x80, file=0xfffffe00a1d58480 "", line=0x80) at ../../../kern/kern_mutex.c:1065 #14 0xffffffff80dec2d6 in vm_page_free_prep (m=0xfffff80874f55518, pagequeue_locked=0x0) at ../../../vm/vm_page.c:4145 #15 0xffffffff80deb6dd in vm_page_reclaim_contig_domain (domain=, req=, npages=0x1579, low=0x0, high=, alignment=, boundary=0x0) at ../../../vm/vm_page.c:3265 #16 0xffffffff80dd6a8c in kmem_alloc_contig_domain (domain=0x0, size=0x1579000, flags=, low=0x0, high=0xffffffffffffffff, alignment=0x1000, boundary=0x0, memattr=0x6) at ../../../vm/vm_kern.c:281 #17 0xffffffff80dd6c5c in kmem_alloc_contig (vmem=, size=0x1579000, flags=, low=0x0, high=0xffffffffffffffff, alignment=0x1000, boundary=0x0, memattr=0x6) at ../../../vm/vm_kern.c:323 #18 0xffffffff80a89375 in contigmalloc (size=0x1579000, type=0xffffffff81a677a0, flags=, low=, high=, alignment=, boundary=0x0) at ../../../kern/kern_malloc.c:418 #19 0xffffffff826300b9 in ?? () #20 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) f 14 #14 0xffffffff80dec2d6 in vm_page_free_prep (m=0xfffff80874f55518, pagequeue_locked=0x0) at ../../../vm/vm_page.c:4145 4145 mtx_assert_(vm_page_lockptr(m), a, file, line); (kgdb) p *m $1 = {plinks = {q = {tqe_next = 0x0, tqe_prev = 0xffffffff81f634b0}, s = {ss = {sle_next = 0x0}, pv = 0xffffffff81f634b0}, memguard = {p = 0x0, v = 0xffffffff81f634b0}}, listq = {tqe_next = 0xfffff8087194b6a0, tqe_prev = 0xfffff808760e2790}, object = 0x0, pindex = 0x7, phys_addr = 0x69a17f000, md = {pv_list = {tqh_first = 0x0, tqh_last = 0xfffff80874f55550}, pv_gen = 0xd42, pat_mode = 0x6}, wire_count = 0x0, busy_lock = 0x1, hold_count = 0x0, flags = 0x0, aflags = 0x0, oflags = 0x0, queue = 0xff, psind = 0x0, segind = 0x7, order = 0xd, pool = 0x0, act_count = 0x0, valid = 0x0, dirty = 0x0} (kgdb) f 15 #15 0xffffffff80deb6dd in vm_page_reclaim_contig_domain (domain=, req=, npages=0x1579, low=0x0, high=, alignment=, boundary=0x0) at ../../../vm/vm_page.c:3265 3265 if (vm_page_free_prep(m, false)) (kgdb) l 3260 count = 0; 3261 TAILQ_INIT(&pgl); 3262 while ((m = SLIST_FIRST(free)) != NULL) { 3263 count++; 3264 SLIST_REMOVE_HEAD(free, plinks.s.ss); 3265 if (vm_page_free_prep(m, false)) 3266 TAILQ_INSERT_TAIL(&pgl, m, listq); 3267 } 3268 3269 vm_page_free_phys_pglist(&pgl); (kgdb) info loc m_runs = 0xfffffe00a1d586a0 curr_low = req_class = count = 0x9 options = m_run = reclaimed = Cannot access memory at address 0x0 (kgdb) what /usr/src/sys/amd64/compile/PHO/kernel.debug /usr/src/sys/amd64/compile/PHO/kernel.debug: ls -l /usr/src/sys/amd64/compile/PHO/kernel.debug /var/crash/vmcore.53 -rwxr-xr-x 1 root wheel 82397680 Mar 6 18:22 /usr/src/sys/amd64/compile/PHO/kernel.debug -rw------- 1 root wheel 1669545984 Mar 7 16:42 /var/crash/vmcore.53 vmstat -N /usr/src/sys/amd64/compile/PHO/kernel.debug -M /var/crash/vmcore.53 -m Type InUse MemUse HighUse Requests Size(s) feeder 7 1K - 7 32 CAM I/O Scheduler 1 1K - 1 128 CAM queue 29 9K - 89 16,32,512 UART 6 5K - 6 16,1024 USB 53 72K - 58 16,128,256,512,1024,2048,4096 USBdev 39 6K - 39 32,64,128,256,512 CAM dev queue 13 1K - 13 64 vtbuf 24 1968K - 46 4096 vt 11 6K - 11 512 DEVFS3 150 38K - 13926 256 DEVFS1 127 64K - 3633 512 DEVFS 17 1K - 18 16,128 DEVFSP 0 0K - 2 64 NFSD V4client 1 1K - 1 256 NFSD lckfile 1 1K - 1 256 NFSD session 1 1K - 1 1024 pfs_nodes 20 5K - 20 256 GEOM 6921 54812K - 295773 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768 raid_data 0 0K - 62340 32,128,256 isadev 7 1K - 7 128 SCSI ENC 46 199K - 6602 16,64,256,2048,32768 ddb_capture 1 64K - 1 65536 cdev 3 1K - 3 256 filedesc 6 49K - 4188 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 sigio 0 0K - 4 64 filecaps 9 1K - 18879 16,32,64 kdtrace 224 46K - 1932674 64,256 kenv 146 14K - 160 16,32,64,128,8192 kqueue 58 4K - 964526 64 proc-args 119 5K - 575099 16,32,64,128,256 hhook 13 4K - 13 256 ithread 162 30K - 162 32,128,256 KTRACE 100 13K - 100 128 linker 249 133K - 272 16,32,64,128,256,512,1024,2048,4096,8192,65536 lockf 8 1K - 429864 64,128 loginclass 3 1K - 3 64 devbuf 17817 34761K - 27999 16,32,64,128,256,512,1024,2048,4096,8192 temp 88 41334K - 1980765 16,32,64,128,256,512,1024,2048,4096,16384,32768,65536 module 518 65K - 519 128 mtx_pool 3 84K - 3 8192 osd 3 1K - 9 16,32,64,128,256 pmchooks 1 1K - 1 128 pgrp 18 3K - 17388 128 session 16 2K - 186 128 proc 2 128K - 2 65536 subproc 665 536K - 965133 512,4096 cred 31 8K - 11326 256 acpiintr 1 1K - 1 64 acpica 24513 2400K - 359710 16,32,64,128,256,512,1024,2048,4096 acpitask 1 64K - 1 65536 plimit 13 4K - 2000 256 uidinfo 4 17K - 104 128,16384 dumper 1 1K - 1 512 kbdmux 7 22K - 7 16,512,1024,2048,16384 sysctl 0 0K - 3 64 sysctloid 7407 377K - 7550 16,32,64,128 sysctltmp 0 0K - 6998 16,64,256,512,1024 LED 40 3K - 40 16,128 tidhash 1 128K - 1 callout 13 4232K - 13 umtx 2008 251K - 2008 128 p1003.1b 1 1K - 1 16 SWAP 1 4372K - 1 bus 1662 181K - 77366 16,32,64,128,256,1024 bus-sc 124 1911K - 17623 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 devstat 8 17K - 8 32,4096 eventhandler 124 11K - 124 64,128 gtaskqueue 78 32K - 78 16,32,256,8192 kobj 354 1416K - 62953 4096 Per-cpu 1 1K - 1 32 rman 307 35K - 665 16,32,128 sbuf 0 0K - 239353 16,32,64,128,256,512,1024,2048,4096,8192,16384 toponodes 32 4K - 32 128 md_disk 0 0K - 38 4096 taskqueue 30 4K - 30 16,32,128,256 terminal 11 3K - 11 256 Unitno 31 2K - 1072887 32,64 vmem 3 336K - 15 512,1024,2048,4096,8192,16384,32768,65536 Witness 1539 3282K - 1539 2048,16384 ioctlops 0 0K - 147 256,512,1024 select 229 29K - 229 128 iov 0 0K - 8315907 16,32,64,128,256,512 msg 4 30K - 4 2048,4096,8192,16384 sem 4 106K - 4 2048,4096 shm 1 32K - 1 32768 tty 15 15K - 17 1024 pts 1 1K - 3 256 mbuf_tag 0 0K - 92 32 shmfd 1 8K - 1 8192 soname 1 1K - 164675 16,32,128 pcb 16 4247K - 19063 16,32,128,1024,2048,8192 vfscache 4 8289K - 4 256,32768,65536 cl_savebuf 0 0K - 44298 64 vfs_hash 1 4096K - 1 vnodes 1 1K - 1 256 mount 31 2K - 49084 16,32,64,128,256 statfs 0 0K - 44219 4096 vnodemarker 0 0K - 22658 1024 chacha20random 1 4K - 1 4096 BPF 3 1K - 11 128,512,4096 ifnet 4 7K - 4 128,2048 ifaddr 50 14K - 51 16,32,64,128,256,512,2048,4096 ether_multi 45 3K - 52 16,32,64 clone 8 1K - 8 128 ipsec 1 1K - 1 256 lltable 15 5K - 15 256,512 iflib 128 793K - 136 64,128,1024,8192,16384,32768 routetbl 26 4K - 29 32,64,128,512 vnet 1 1K - 1 64 vnet_data 1 64K - 1 65536 vnet_data_free 1 1K - 1 32 igmp 3 1K - 3 128 in_multi 2 1K - 3 256 encap_export_host 2 2K - 2 1024 sctp_a_it 0 0K - 4 16 sctp_vrf 1 1K - 1 64 sctp_ifa 5 1K - 5 128 sctp_ifn 2 1K - 2 128 sctp_iter 0 0K - 4 256 tfo_ccache 1 128K - 1 hostcache 1 32K - 1 32768 LRO 24 480K - 24 8192,32768 tcpfunc 1 1K - 1 64 syncache 1 68K - 1 in6_mfilter 0 0K - 1 1024 in6_multi 26 4K - 27 32,256 ip6_moptions 0 0K - 2 32,256 mld 3 1K - 3 128 ip6ndp 5 1K - 6 64,256 inpcbpolicy 4 1K - 27277 32 secasvar 1 1K - 1 1024 sahead 1 1K - 1 1024 ipsecpolicy 2 2K - 2 256,1024 ipsec-saq 2 2K - 2 1024 NLM 0 0K - 1 32 crypto 2 3K - 2 512,2048 rpc 5 10K - 196 16,32,64,128,256,512,1024,4096 audit_evclass 230 8K - 286 32 pagedep 5 1025K - 792024 256 inodedep 20 8201K - 72092813 512 bmsafemap 5 17K - 35545658 256,8192 newblk 11 16387K - 245703 256 indirdep 0 0K - 17830 128,32768 freefrag 1 1K - 2253 128 freeblks 0 0K - 18782 256 freefile 4 1K - 35525061 64 diradd 12 2K - 74074306 128 mkdir 2 1K - 7186 128 dirrem 2 1K - 74061732 128 newdirblk 1 1K - 84183 64 freework 2 1K - 22648 32,128 sbdep 0 0K - 5661 64 savedino 0 0K - 262797 256 softdep 2 1K - 3236 512 ufs_dirhash 135 48K - 32374 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 ufs_quota 1 4096K - 1 ufs_mount 6 57K - 13520 512,2048,4096,8192,32768 vm_pgdata 1 1K - 1 128 UMAHash 7 132K - 33 512,1024,2048,4096,8192,16384,32768,65536 memdesc 1 4K - 1 4096 pci_link 16 2K - 16 64,128 acpisem 31 4K - 31 128 acpidev 183 12K - 183 64 CAM SIM 13 4K - 13 256 CAM XPT 48 5K - 165 16,32,64,128,256,512,1024,2048,65536 apmdev 1 1K - 1 128 madt_table 0 0K - 2 128,4096 entropy 1 1K - 73 32,4096 CAM DEV 16 32K - 35 2048 CAM CCB 0 0K - 5134613 2048 CAM path 19 1K - 112 32 CAM periph 8 2K - 32 16,32,64,128,256 io_apic 2 4K - 2 2048 local_apic 1 16K - 1 16384 MCA 35 9K - 35 128,512 cpus 2 1K - 2 64 msi 24 3K - 24 128 nexusdev 5 1K - 5 16 acpi_perf 12 6K - 12 512 pmc 1130 11809K - 1130 16,128,256,512,1024,4096,8192,65536 tmpfs mount 0 0K - 389 128 tmpfs name 0 0K - 570627 16 vmstat -N /usr/src/sys/amd64/compile/PHO/kernel.debug -M /var/crash/vmcore.53 -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 280, 0, 208, 0, 986, 0, 0 UMA Zones: 1160, 0, 225, 0, 1003, 0, 0 UMA Slabs: 112, 0, 15009, 1301, 257617, 0, 0 UMA Hash: 256, 0, 3, 12, 10, 0, 0 4 Bucket: 32, 0, 997, 2971, 189063,31233, 0 6 Bucket: 48, 0, 36, 3035, 153769, 0, 0 8 Bucket: 64, 0, 40, 2998, 794263, 17, 0 12 Bucket: 96, 0, 78, 1521, 81446,18404, 0 16 Bucket: 128, 0, 128, 1391, 1255866, 372, 0 32 Bucket: 256, 0, 1491, 1179, 3535795,1471, 0 64 Bucket: 512, 0, 290, 256, 1326678,7380362, 0 128 Bucket: 1024, 0, 196, 152, 2524033,4760295, 0 256 Bucket: 2048, 0, 324, 454, 2085726,29428013, 0 vmem: 1792, 0, 1, 1, 1, 0, 0 vmem btag: 56, 0, 23841, 5056, 73512, 205, 0 VM OBJECT: 256, 0, 16129, 606971,51946834, 0, 0 RADIX NODE: 144, 0, 8292, 67983,65183827, 0, 0 MAP: 232, 0, 3, 65, 3, 0, 0 KMAP ENTRY: 120, 0, 8, 124, 8, 0, 0 MAP ENTRY: 120, 0, 694, 8480,43369894, 0, 0 VMSPACE: 2512, 0, 25, 507, 964456, 0, 0 fakepg: 104, 0, 0, 152, 8, 0, 0 64 pcpu: 8, 0, 2635, 1077, 2905, 0, 0 mt_zone: 16400, 0, 417, 0, 417, 0, 0 16: 16, 0, 19, 3218, 4486, 0, 0 16: 16, 0, 472, 1271, 159447, 0, 0 16: 16, 0, 280, 2708, 13998, 0, 0 16: 16, 0, 2495, 2236, 183481, 0, 0 16: 16, 0, 235, 2255, 575812, 0, 0 16: 16, 0, 42, 1701, 30222, 0, 0 16: 16, 0, 0, 0, 0, 0, 0 16: 16, 0, 40, 1703, 27854, 0, 0 32: 32, 0, 127, 3097, 534285, 0, 0 32: 32, 0, 512, 976, 8731, 0, 0 32: 32, 0, 172, 1316, 72945, 0, 0 32: 32, 0, 2858, 1606, 35372, 0, 0 32: 32, 0, 406, 1826, 17767, 0, 0 32: 32, 0, 25, 3075, 1072494, 0, 0 32: 32, 0, 8, 240, 9, 0, 0 32: 32, 0, 32, 588, 13650, 0, 0 64: 64, 0, 43, 2561, 48903, 0, 0 64: 64, 0, 9317, 3083, 8321385, 0, 0 64: 64, 0, 650, 838, 15073, 0, 0 64: 64, 0, 629, 3091, 1015138, 0, 0 64: 64, 0, 8672, 2612, 1158598, 0, 0 64: 64, 0, 134, 672, 15623, 0, 0 64: 64, 0, 6, 1048,35609245, 0, 0 64: 64, 0, 13, 359, 10330, 0, 0 128: 128, 0, 36, 925, 24954, 0, 0 128: 128, 0, 14339, 510, 173674, 0, 0 128: 128, 0, 606, 417, 43500, 0, 0 128: 128, 0, 2214, 1382, 212029, 0, 0 128: 128, 0, 548, 1498,148831522, 0, 0 128: 128, 0, 2666, 2573, 10024, 0, 0 128: 128, 0, 2, 401, 2254, 0, 0 128: 128, 0, 23, 504, 54291, 0, 0 256: 256, 0, 65, 250, 6237, 0, 0 256: 256, 0, 65, 430, 794989, 0, 0 256: 256, 0, 390, 615,35604494, 0, 0 256: 256, 0, 42, 438, 16715, 0, 0 256: 256, 0, 619, 896, 1247590, 0, 0 256: 256, 0, 9, 111, 3782, 0, 0 256: 256, 0, 11, 544, 242469, 0, 0 256: 256, 0, 52, 113, 59040, 0, 0 512: 512, 0, 12, 72, 12, 0, 0 512: 512, 0, 23, 40, 52, 0, 0 512: 512, 0, 1, 20, 3975, 0, 0 512: 512, 0, 18, 45, 51, 0, 0 512: 512, 0, 341, 107, 16793, 0, 0 512: 512, 0, 653, 845,72093596, 0, 0 512: 512, 0, 11, 10, 12, 0, 0 512: 512, 0, 2, 12, 93778, 0, 0 1024: 1024, 0, 7, 9, 7, 0, 0 1024: 1024, 0, 44, 20, 6582, 0, 0 1024: 1024, 0, 20, 36, 2645, 0, 0 1024: 1024, 0, 4, 120, 1704, 0, 0 1024: 1024, 0, 41, 159, 1525823, 0, 0 1024: 1024, 0, 6, 94, 22670, 0, 0 1024: 1024, 0, 1, 3, 2, 0, 0 1024: 1024, 0, 2, 6, 3619, 0, 0 2048: 2048, 0, 3, 1, 3, 0, 0 2048: 2048, 0, 1559, 69, 5136194, 0, 0 2048: 2048, 0, 15, 3, 475, 0, 0 2048: 2048, 0, 4, 2, 12, 0, 0 2048: 2048, 0, 94, 4, 2367, 0, 0 2048: 2048, 0, 5, 3, 14, 0, 0 2048: 2048, 0, 0, 0, 3, 0, 0 2048: 2048, 0, 2, 2, 10412, 0, 0 4096: 4096, 0, 1025, 0, 1025, 0, 0 4096: 4096, 0, 1, 0, 13, 0, 0 4096: 4096, 0, 5, 5, 1207, 0, 0 4096: 4096, 0, 356, 7, 107182, 0, 0 4096: 4096, 0, 41, 5, 9459, 0, 0 4096: 4096, 0, 79, 48, 967936, 0, 0 4096: 4096, 0, 12, 0, 96, 0, 0 4096: 4096, 0, 20, 0, 6844, 0, 0 8192: 8192, 0, 6, 0, 6, 0, 0 8192: 8192, 0, 38, 0, 38, 0, 0 8192: 8192, 0, 12, 0, 4442, 0, 0 8192: 8192, 0, 2, 0, 10, 0, 0 8192: 8192, 0, 16, 4, 719, 0, 0 8192: 8192, 0, 4, 0, 6759, 0, 0 8192: 8192, 0, 0, 0, 1, 0, 0 8192: 8192, 0, 6827, 0, 42009, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 6, 0, 6, 0, 0 16384: 16384, 0, 1, 0, 1845, 0, 0 16384: 16384, 0, 1, 0, 9, 0, 0 16384: 16384, 0, 5, 10, 298, 0, 0 16384: 16384, 0, 3, 0, 5, 0, 0 16384: 16384, 0, 1, 0, 1, 0, 0 16384: 16384, 0, 0, 0, 12, 0, 0 32768: 32768, 0, 7, 7, 1467, 0, 0 32768: 32768, 0, 13, 0, 13, 0, 0 32768: 32768, 0, 0, 0, 1381, 0, 0 32768: 32768, 0, 1, 0, 1, 0, 0 32768: 32768, 0, 12, 0, 43, 0, 0 32768: 32768, 0, 2, 0, 4, 0, 0 32768: 32768, 0, 0, 0, 2, 0, 0 32768: 32768, 0, 4, 0, 8, 0, 0 65536: 65536, 0, 26, 0, 26, 0, 0 65536: 65536, 0, 1, 0, 17, 0, 0 65536: 65536, 0, 3, 0, 1084, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 2, 0, 98, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 2, 0, 3, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 SLEEPQUEUE: 88, 0, 1005, 390, 1005, 0, 0 Files: 80, 0, 38, 501624,45424893, 0, 0 filedesc0: 1104, 0, 58, 200, 964526, 0, 0 rl_entry: 40, 0, 572, 2200, 572, 0, 0 TURNSTILE: 136, 0, 1005, 235, 1005, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 umtx_shm: 88, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1336, 0, 57, 549, 964525, 0, 0 THREAD: 1368, 0, 714, 290, 4228, 0, 0 cpuset: 104, 0, 142, 428, 3438, 0, 0 domainset: 32, 0, 0, 0, 0, 0, 0 audit_record: 1280, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 13008735, 28, 3893, 45552, 0, 0 mbuf: 256, 13008735, 6242, 3889,447801238, 0, 0 mbuf_cluster: 2048, 2560000, 8957, 23, 131796, 0, 0 mbuf_jumbo_page: 4096, 1016306, 0, 0, 21, 0, 0 mbuf_jumbo_9k: 9216, 903381, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 677536, 0, 0, 0, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 g_bio: 408, 0, 0, 702,33167695, 0, 0 ttyinq: 160, 0, 315, 237, 1245, 0, 0 ttyoutq: 256, 0, 165, 210, 649, 0, 0 nvme_request: 128, 0, 0, 0, 0, 0, 0 cryptop: 128, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 632, 0, 16088, 1750,38146625, 0, 0 VNODEPOLL: 120, 0, 0, 0, 0, 0, 0 BUF TRIE: 144, 0, 388, 105560, 3005668, 0, 0 NAMEI: 1024, 0, 0, 144,182888665, 0, 0 rentr: 24, 0, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 16255, 4395, 1260247, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 10, 98, 4725, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 176, 48, 1251040, 0, 0 NCLNODE: 744, 0, 0, 0, 0, 0, 0 pipe: 760, 0, 4, 336, 499381, 0, 0 procdesc: 136, 0, 0, 29, 12, 0, 0 Mountpoints: 2896, 0, 3, 43, 3770, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0 AIOP: 32, 0, 0, 0, 0, 0, 0 AIOCB: 752, 0, 0, 0, 0, 0, 0 AIOLIO: 280, 0, 0, 0, 0, 0, 0 ksiginfo: 112, 0, 582, 1833, 7689907, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0 KNOTE: 160, 0, 0, 0, 0, 0, 0 socket: 856, 1044004, 8, 368, 31961, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0 unpcb: 232, 1044004, 4, 387, 4606, 0, 0 ipq: 56, 80017, 0, 0, 0, 0, 0 udp_inpcb: 424, 1044009, 2, 43, 275, 0, 0 udpcb: 32, 1044080, 2, 370, 275, 0, 0 tcp_inpcb: 424, 1044009, 2, 43, 26991, 0, 0 tcpcb: 904, 1044004, 2, 26, 26991, 0, 0 tcptw: 88, 27810, 0, 0, 92, 0, 0 syncache: 176, 15378, 0, 0, 8046, 0, 0 hostcache: 96, 15375, 2, 162, 2, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 tfo: 4, 0, 0, 0, 0, 0, 0 tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0 tcpreass: 40, 160083, 0, 0, 0, 0, 0 sctp_ep: 1216, 1044006, 0, 0, 0, 0, 0 sctp_asoc: 2408, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 0, 4, 0, 0 sctp_raddr: 736, 80000, 0, 0, 0, 0, 0 sctp_chunk: 152, 400010, 0, 0, 0, 0, 0 sctp_readq: 152, 400010, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400015, 0, 0, 0, 0, 0 sctp_asconf: 40, 400059, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 udplite_inpcb: 424, 1044009, 0, 0, 0, 0, 0 ripcb: 424, 1044009, 0, 9, 11, 0, 0 rtentry: 208, 0, 12, 102, 13, 0, 0 selfd: 64, 0, 240, 3170, 3690447, 0, 0 swpctrie: 144, 4065255, 101, 12616, 302180, 0, 0 swblk: 136, 4065249, 588, 165843, 3955006, 0, 0 FFS inode: 160, 0, 16057, 6335,37571715, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 16057, 4988,37571504, 0, 0