KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.0-CURRENT #1 r330873M: Tue Mar 13 21:19:44 CET 2018 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 7 boot pages configured startup_alloc from "UMA Kegs", 6 boot pages left startup_alloc from "UMA Zones", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Zones", 2 boot pages left startup_alloc from "UMA Hash", 1 boot pages left Entering uma_startup1 with 0 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33179262976 (31642 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads arc4random: no preloaded entropy cache ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #7 Launched! Timecounter "TSC-low" frequency 1750032874 Hz quality 1000 random: entropy device external interface module_register_init: MOD_LOAD (vesa, 0xffffffff80fec5d0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" kbd1 at kbdmux0 netmap: loaded module [ath_hal] loaded nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: attach_pre capping queues at 8 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 8 igb0: pxm cpus: 6 queue msgs: 9 admincnt: 1 igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: using 6 rx queues 6 tx queues iugen0.2: at usbus0 uhub5 on uhub1 uhub5: on usbus0 uhub4: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered uhub5: 4 ports with 3 removable, self powered Writing entropy file:. Setting hostname: mercat1.netperf.freebsd.org. ugen0.3: at usbus0 ukbd0 on uhub5 ukbd0: on usbus0 kbd2 at ukbd0 Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo igb0: flags=8843 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 nd6 options=21 media: Ethernet autoselect status: no carrier igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=e505bb ether 0c:c4:7a:a8:cd:db nd6 options=29 media: Ethernet autoselect status: no carrier ums0 on uhub5 ums0: on usbus0 Starting pflogd:ums0: 3 buttons and [Z] coordinates ID=0 add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Setting date via ntp. 13 Mar 21:25:20 ntpdate[525]: no servers can be used, exiting Starting rpcbind. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 NFS access cache time=60 rpc.umntall: 192.168.5.1: MOUNTPROG: RPC: Port mapper failure - RPC: Unable to send Starting casperd. lock order reversal: 1st 0xfffffe0000ec7aa8 bufwait (bufwait) @ kern/vfs_bio.c:3892 2nd 0xfffff8000c09c800 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0xffffffff80b15573 at witness_debugger+0x73 #1 0xffffffff80b153f4 at witness_checkorder+0xe34 #2 0xffffffff80ab7b58 at _sx_xlock+0x68 #3 0xffffffff80db648d at ufsdirhash_add+0x4d #4 0xffffffff80dbafaf at ufs_direnter+0x49f #5 0xffffffff80dc35ac at ufs_mkdir+0x8dc #6 0xffffffff810e28fd at VOP_MKDIR_APV+0xfd #7 0xffffffff80b8f746 at kern_mkdirat+0x1d6 #8 0xffffffff80f6ce3b at amd64_syscall+0x7ab #9 0xffffffff80f4801d at fast_syscall_common+0x101 Clearing /tmp (X related). Starting statd. Starting lockd. Updating motd:. Mounting late file systems:. Configuring syscons: blanktime. Link state changed to up Starting sshd. Starting cron. Tue Mar 13 21:25 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: root Mar 13 21:25:25 mercat1 login: ROOT LOGIN (root) ON ttyu1 Last login: Tue Mar 13 20:32:57 on ttyu1 FreeBSD 12.0-CURRENT (PHO) #1 r330873M: Tue Mar 13 21:19:44 CET 2018 root@mercat1:~ # Mar 13 21:25:32 mercat1 su: pho to root on /dev/pts/0 20180313 21:27:14 all (1/3): stealer.sh Mar 13 21:27:18 mercat1 su: pho to root on /dev/pts/1 lock order reversal: 1st 0xfffffe0000e23d38 bufwait (bufwait) @ vm/vm_pager.c:374 2nd 0xfffff8016e5daa48 ufs (ufs) @ dev/md/md.c:952 stack backtrace: #0 0xffffffff80b15573 at witness_debugger+0x73 #1 0xffffffff80b153f4 at witness_checkorder+0xe34 #2 0xffffffff80a844cb at lockmgr_xlock_hard+0x6b #3 0xffffffff80a84eaa at __lockmgr_args+0x51a #4 0xffffffff80db124f at ffs_lock+0xaf #5 0xffffffff810e3611 at VOP_LOCK1_APV+0xe1 #6 0xffffffff80b92636 at _vn_lock+0x66 #7 0xffffffff80640eb0 at mdstart_vnode+0x440 #8 0xffffffff8063f4ce at md_kthread+0x1fe #9 0xffffffff80a6fa24 at fork_exit+0x84 #10 0xffffffff80f484ee at fork_trampoline+0xe swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed : : swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(25): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed freework leak: 10/18. pts leak: 1/2 20180313 21:37:03 all (2/3): sendfile13.sh lock order reversal: 1st 0xfffff8073bad2068 ufs (ufs) @ kern/vfs_mount.c:1280 2nd 0xfffff80496fbea48 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1583 stack backtrace: #0 0xffffffff80b15573 at witness_debugger+0x73 #1 0xffffffff80b153f4 at witness_checkorder+0xe34 #2 0xffffffff80a83ffe at lockmgr_lock_fast_path+0x17e #3 0xffffffff810e3611 at VOP_LOCK1_APV+0xe1 #4 0xffffffff80b92636 at _vn_lock+0x66 #5 0xffffffff80daf8ac at ffs_sync+0x2cc #6 0xffffffff80b78885 at dounmount+0x555 #7 0xffffffff80b782a0 at sys_unmount+0x310 #8 0xffffffff80f6ce3b at amd64_syscall+0x7ab #9 0xffffffff80f4801d at fast_syscall_common+0x101 fsync: giving up on dirty (error = 35) 0xfffff80496fbe9e0: tag devfs, type VCHR usecount 1, writecount 0, refcount 28 rdev 0xfffff804964eda00 flags (VI_ACTIVE) v_object 0xfffff802d882a700 ref 0 pages 194 cleanbuf 23 dirtybuf 3 lock type devfs: EXCL by thread 0xfffff801f38f6000 (pid 1771, umount, tid 100280) #0 0xffffffff80a84063 at lockmgr_lock_fast_path+0x1e3 #1 0xffffffff810e3611 at VOP_LOCK1_APV+0xe1 #2 0xffffffff80b92636 at _vn_lock+0x66 #3 0xffffffff80daf8ac at ffs_sync+0x2cc #4 0xffffffff80b78885 at dounmount+0x555 #5 0xffffffff80b782a0 at sys_unmount+0x310 #6 0xffffffff80f6ce3b at amd64_syscall+0x7ab #7 0xffffffff80f4801d at fast_syscall_common+0x101 dev md11a GEOM leak: 45/126 20180313 21:44:30 all (3/3): fdgrowtable.sh 20180313 21:59:26 all (1/3): fdgrowtable.sh 20180313 22:14:23 all (2/3): sendfile13.sh GEOM leak: 16/142 20180313 22:22:12 all (3/3): stealer.sh swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed : : swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(3): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(18): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(18): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(32): failed swap_pager_getswapspace(20): failed swap_pager_getswapspace(18): failed swap_pager_getswapspace(9): failed swap_pager_getswapspace(3): failed swap_pager_getswapspace(32): failed 20180314 01:51:55 all (2/3): fdgrowtable.sh Mar 14 02:08:08 mercat1 kernel: pid 23285 (swap), uid 0, was killed: out of swap space Mar 14 02:09:16 mercat1 kernel: pid 23324 (swap), uid 0, was killed: out of swap space Mar 14 02:09:53 mercat1 kernel: pid 23321 (swap), uid 0, was killed: out of swap space Mar 14 02:09:56 mercat1 kernel: pid 23320 (swap), uid 0, was killed: out of swap space Mar 14 02:09:58 mercat1 kernel: pid 23318 (swap), uid 0, was killed: out of swap space Mar 14 02:10:00 mercat1 kernel: pid 23317 (swap), uid 0, was killed: out of swap space Mar 14 02:10:02 mercat1 kernel: pid 23314 (swap), uid 0, was killed: out of swap space Mar 14 02:10:04 mercat1 kernel: pid 23313 (swap), uid 0, was killed: out of swap space Mar 14 02:10:06 mercat1 kernel: pid 23312 (swap), uid 0, was killed: out of swap space Mar 14 02:10:09 mercat1 kernel: pid 23309 (swap), uid 0, was killed: out of swap space Mar 14 02:10:11 mercat1 kernel: pid 23306 (swap), uid 0, was killed: out of swap space Mar 14 02:10:13 mercat1 kernel: pid 23305 (swap), uid 0, was killed: out of swap space Mar 14 02:10:15 mercat1 kernel: pid 23304 (swap), uid 0, was killed: out of swap space Mar 14 02:10:18 mercat1 kernel: pid 23303 (swap), uid 0, was killed: out of swap space Mar 14 02:10:20 mercat1 kernel: pid 23302 (swap), uid 0, was killed: out of swap space Mar 14 02:10:22 mercat1 kernel: pid 23301 (swap), uid 0, was killed: out of swap space Mar 14 02:10:24 mercat1 kernel: pid 23300 (swap), uid 0, was killed: out of swap space Mar 14 02:10:27 mercat1 kernel: pid 23299 (swap), uid 0, was killed: out of swap space Mar 14 02:10:29 mercat1 kernel: pid 23297 (swap), uid 0, was killed: out of swap space Mar 14 02:10:31 mercat1 kernel: pid 23296 (swap), uid 0, was killed: out of swap space Mar 14 02:10:33 mercat1 kernel: pid 23295 (swap), uid 0, was killed: out of swap space Mar 14 02:10:35 mercat1 kernel: pid 23294 (swap), uid 0, was killed: out of swap space Mar 14 02:10:38 mercat1 kernel: pid 23291 (swap), uid 0, was killed: out of swap space Mar 14 02:10:40 mercat1 kernel: pid 23290 (swap), uid 0, was killed: out of swap space Mar 14 02:10:42 mercat1 kernel: pid 23288 (swap), uid 0, was killed: out of swap space Mar 14 02:10:44 mercat1 kernel: pid 23287 (swap), uid 0, was killed: out of swap space Mar 14 02:10:46 mercat1 kernel: pid 23286 (swap), uid 0, was killed: out of swap space Mar 14 02:10:49 mercat1 kernel: pid 23308 (swap), uid 0, was killed: out of swap space Mar 14 02:10:51 mercat1 kernel: pid 23316 (swap), uid 0, was killed: out of swap space Mar 14 02:10:53 mercat1 kernel: pid 23323 (swap), uid 0, was killed: out of swap space Mar 14 02:10:55 mercat1 kernel: pid 23319 (swap), uid 0, was killed: out of swap space Mar 14 02:10:58 mercat1 kernel: pid 23315 (swap), uid 0, was killed: out of swap space Mar 14 02:11:00 mercat1 kernel: pid 23289 (swap), uid 0, was killed: out of swap space Mar 14 02:11:02 mercat1 kernel: pid 23307 (swap), uid 0, was killed: out of swap space Mar 14 02:11:04 mercat1 kernel: pid 23310 (swap), uid 0, was killed: out of swap space Mar 14 02:11:07 mercat1 kernel: pid 23322 (swap), uid 0, was killed: out of swap space Mar 14 02:11:09 mercat1 kernel: pid 23298 (swap), uid 0, was killed: out of swap space Mar 14 02:11:11 mercat1 kernel: pid 23311 (swap), uid 0, was killed: out of swap space Mar 14 02:11:13 mercat1 kernel: pid 23293 (swap), uid 0, was killed: out of swap space Mar 14 02:11:15 mercat1 kernel: pid 23292 (swap), uid 0, was killed: out of swap space Mar 14 02:11:17 mercat1 kernel: pid 1103 (top), uid 0, was killed: out of swap space Mar 14 02:11:20 mercat1 kernel: pid 595 (rpc.statd), uid 0, was killed: out of swap space Mar 14 02:11:29 mercat1 kernel: pid 598 (rpc.lockd), uid 0, was killed: out of swap space Mar 14 02:11:32 mercat1 kernel: pid 448 (devd), uid 0, was killed: out of swap space Mar 14 02:11:34 mercat1 kernel: pid 770 (csh), uid 0, was killed: out of swap space Mar 14 02:11:37 mercat1 kernel: pid 1100 (csh), uid 0, was killed: out of swap space Mar 14 02:11:39 mercat1 kernel: pid 779 (csh), uid 0, was killed: out of swap space Mar 14 02:11:41 mercat1 kernel: pid 1102 (bash), uid 0, was killed: out of swap space Mar 14 02:11:43 mercat1 kernel: pid 781 (bash), uid 0, was killed: out of swap space Mar 14 02:11:46 mercat1 kernel: pid 1098 (bash), uid 2006, was killed: out of swap space Mar 14 02:11:48 mercat1 kernel: pid 777 (bash), uid 2006, was killed: out of swap space Mar 14 02:11:52 mercat1 kernel: pid 776 (sshd), uid 2006, was killed: out of swap space Mar 14 02:11:54 mercat1 kernel: pid 773 (sshd), uid 0, was killed: out of swap space Mar 14 02:12:07 mercat1 kernel: pid 1097 (sshd), uid 2006, was killed: out of swap space Mar 14 02:12:11 mercat1 kernel: pid 1099 (su), uid 0, was killed: out of swap space Mar 14 02:12:13 mercat1 kernel: pid 778 (su), uid 0, was killed: out of swap space Mar 14 02:12:15 mercat1 kernel: pid 22308 (su), uid 0, was killed: out of swap space Mar 14 02:12:17 mercat1 kernel: pid 769 (login), uid 0, was killed: out of swap space pid 1089 (sshd), uid 0, was killed: out of swap space pid 528 (rpcbind), uid 0, was killed: out of swap space pid 720 (dhclient), uid 65, was killed: out of swap space pid 22297 (sh), uid 0, was killed: out of swap space pid 673 (dhclient), uid 0, was killed: out of swap space pid 784 (sh), uid 0, was killed: out of swap space pid 22296 (tee), uid 0, was killed: out of swap space pid 767 (getty), uid 0, was killed: out of swap space pid 766 (getty), uid 0, was killed: out of swap space pid 765 (getty), uid 0, was killed: out of swap space pid 764 (getty), uid 0, was killed: out of swap space pid 763 (getty), uid 0, was killed: out of swap space pid 762 (getty), uid 0, was killed: out of swap space pid 761 (getty), uid 0, was killed: out of swap space pid 760 (getty), uid 0, was killed: out of swap space pid 768 (getty), uid 0, was killed: out of swap space pid 22352 (fdgrowtable), uid 2007, was killed: out of swap space pid 22295 (sh), uid 0, was killed: out of swap space pid 570 (casperd), uid 0, was killed: out of swap space pid 569 (casperd), uid 0, was killed: out of swap space pid 676 (dhclient), uid 0, was killed: out of swap space pid 23408 (getty), uid 0, was killed: out of swap space pid 23407 (getty), uid 0, was killed: out of swap space pid 23406 (getty), uid 0, was killed: out of swap space pid 23405 (getty), uid 0, was killed: out of swap space pid 23404 (getty), uid 0, was killed: out of swap space pid 23403 (getty), uid 0, was killed: out of swap space pid 23402 (getty), uid 0, was killed: out of swap space pid 23401 (getty), uid 0, was killed: out of swap space pid 23400 (getty), uid 0, was killed: out of swap space pid 23284 (swap), uid 0, was killed: out of swap space pid 23280 (swap), uid 0, was killed: out of swap space KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_alt_break_internal+0x106: movq $0,kdb_why db> ps pid ppid pgrp uid state wmesg wchan cmd 23408 1 1 0 D ufs 0xfffff8001b2747d0 getty 23407 1 1 0 D ufs 0xfffff8001b2747d0 getty 23406 1 1 0 D ufs 0xfffff8001b2747d0 getty 23405 1 1 0 D ufs 0xfffff8001b2747d0 getty 23404 1 1 0 D ufs 0xfffff8001b2747d0 getty 23403 1 1 0 D ufs 0xfffff8001b2747d0 getty 23402 1 1 0 D ufs 0xfffff8001b2747d0 getty 23401 1 1 0 D ufs 0xfffff8001b2747d0 getty 23400 1 1 0 D ufs 0xfffff8001b2747d0 getty 23324 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23323 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23322 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23321 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23320 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23319 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23318 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23317 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23316 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23315 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap --More-- db> run pho db:0:pho> run pho1 db:1:pho1> bt Tracing pid 11 tid 100003 td 0xfffff80003938000 kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:1:pho1> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0xa101c0 curthread = 0xfffff80003938000: pid 11 tid 100003 "idle: cpu0" curpcb = 0xfffffe007895ab80 fpcurthread = none idlethread = 0xfffff80003938000: tid 100003 "idle: cpu0" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ac10 commontssp = 0xffffffff81f6ac10 rsp0 = 0xfffffe007895ab80 gs32p = 0xffffffff81f71458 ldt = 0xffffffff81f71498 tss = 0xffffffff81f71488 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00856c51c0 curthread = 0xfffff80003937560: pid 11 tid 100004 "idle: cpu1" curpcb = 0xfffffe007895fb80 fpcurthread = none idlethread = 0xfffff80003937560: tid 100004 "idle: cpu1" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ac78 commontssp = 0xffffffff81f6ac78 rsp0 = 0xfffffe007895fb80 gs32p = 0xffffffff81f714c0 ldt = 0xffffffff81f71500 tss = 0xffffffff81f714f0 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00856ce1c0 curthread = 0xfffff80003937000: pid 11 tid 100005 "idle: cpu2" curpcb = 0xfffffe0078964b80 fpcurthread = none idlethread = 0xfffff80003937000: tid 100005 "idle: cpu2" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ace0 commontssp = 0xffffffff81f6ace0 rsp0 = 0xfffffe0078964b80 gs32p = 0xffffffff81f71528 ldt = 0xffffffff81f71568 tss = 0xffffffff81f71558 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00856d71c0 curthread = 0xfffff80003936560: pid 11 tid 100006 "idle: cpu3" curpcb = 0xfffffe0078969b80 fpcurthread = none idlethread = 0xfffff80003936560: tid 100006 "idle: cpu3" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ad48 commontssp = 0xffffffff81f6ad48 rsp0 = 0xfffffe0078969b80 gs32p = 0xffffffff81f71590 ldt = 0xffffffff81f715d0 tss = 0xffffffff81f715c0 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00856e01c0 curthread = 0xfffff80003936000: pid 11 tid 100007 "idle: cpu4" curpcb = 0xfffffe007896eb80 fpcurthread = none idlethread = 0xfffff80003936000: tid 100007 "idle: cpu4" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6adb0 commontssp = 0xffffffff81f6adb0 rsp0 = 0xfffffe007896eb80 gs32p = 0xffffffff81f715f8 ldt = 0xffffffff81f71638 tss = 0xffffffff81f71628 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00856e91c0 curthread = 0xfffff80003935560: pid 11 tid 100008 "idle: cpu5" curpcb = 0xfffffe0078973b80 fpcurthread = none idlethread = 0xfffff80003935560: tid 100008 "idle: cpu5" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ae18 commontssp = 0xffffffff81f6ae18 rsp0 = 0xfffffe0078973b80 gs32p = 0xffffffff81f71660 ldt = 0xffffffff81f716a0 tss = 0xffffffff81f71690 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00856f21c0 curthread = 0xfffff80003935000: pid 11 tid 100009 "idle: cpu6" curpcb = 0xfffffe0078978b80 fpcurthread = none idlethread = 0xfffff80003935000: tid 100009 "idle: cpu6" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6ae80 commontssp = 0xffffffff81f6ae80 rsp0 = 0xfffffe0078978b80 gs32p = 0xffffffff81f716c8 ldt = 0xffffffff81f71708 tss = 0xffffffff81f716f8 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00856fb1c0 curthread = 0xfffff80003949560: pid 11 tid 100010 "idle: cpu7" curpcb = 0xfffffe007897db80 fpcurthread = none idlethread = 0xfffff80003949560: tid 100010 "idle: cpu7" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6aee8 commontssp = 0xffffffff81f6aee8 rsp0 = 0xfffffe007897db80 gs32p = 0xffffffff81f71730 ldt = 0xffffffff81f71770 tss = 0xffffffff81f71760 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00857041c0 curthread = 0xfffff80003949000: pid 11 tid 100011 "idle: cpu8" curpcb = 0xfffffe0078982b80 fpcurthread = none idlethread = 0xfffff80003949000: tid 100011 "idle: cpu8" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6af50 commontssp = 0xffffffff81f6af50 rsp0 = 0xfffffe0078982b80 gs32p = 0xffffffff81f71798 ldt = 0xffffffff81f717d8 tss = 0xffffffff81f717c8 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe008570d1c0 curthread = 0xfffff80003948560: pid 11 tid 100012 "idle: cpu9" curpcb = 0xfffffe0078987b80 fpcurthread = none idlethread = 0xfffff80003948560: tid 100012 "idle: cpu9" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6afb8 commontssp = 0xffffffff81f6afb8 rsp0 = 0xfffffe0078987b80 gs32p = 0xffffffff81f71800 ldt = 0xffffffff81f71840 tss = 0xffffffff81f71830 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00857161c0 curthread = 0xfffff80003948000: pid 11 tid 100013 "idle: cpu10" curpcb = 0xfffffe007898cb80 fpcurthread = none idlethread = 0xfffff80003948000: tid 100013 "idle: cpu10" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6b020 commontssp = 0xffffffff81f6b020 rsp0 = 0xfffffe007898cb80 gs32p = 0xffffffff81f71868 ldt = 0xffffffff81f718a8 tss = 0xffffffff81f71898 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe007f1e71c0 curthread = 0xfffff80003947560: pid 11 tid 100014 "idle: cpu11" curpcb = 0xfffffe0078991b80 fpcurthread = none idlethread = 0xfffff80003947560: tid 100014 "idle: cpu11" curpmap = 0xffffffff81f40758 tssp = 0xffffffff81f6b088 commontssp = 0xffffffff81f6b088 rsp0 = 0xfffffe0078991b80 gs32p = 0xffffffff81f718d0 ldt = 0xffffffff81f71910 tss = 0xffffffff81f71900 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 23408 (getty) thread 0xfffff8001bdb8000 (100200) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23407 (getty) thread 0xfffff8001bad3560 (100140) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23406 (getty) thread 0xfffff8001bdf2000 (100187) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23405 (getty) thread 0xfffff8001bdf2560 (100157) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23404 (getty) thread 0xfffff8001bdf1560 (100195) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23403 (getty) thread 0xfffff8001bdf1000 (100196) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23402 (getty) thread 0xfffff8001bdf0560 (100197) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23401 (getty) thread 0xfffff8001bdf0000 (100198) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23400 (getty) thread 0xfffff8001bdb8560 (100199) shared lockmgr ufs (ufs) r = 0 (0xfffff8001b274cc0) locked @ kern/vfs_subr.c:2602 Process 23308 (swap) thread 0xfffff8001ba6c560 (100169) shared sx vm map (user) (vm map (user)) r = 0 (0xfffff802415f9078) locked @ vm/vm_map.c:4065 Process 23293 (swap) thread 0xfffff8001bcbb560 (100182) shared sx vm map (user) (vm map (user)) r = 0 (0xfffff8016e3eb078) locked @ vm/vm_map.c:4065 Process 23280 (swap) thread 0xfffff80756572560 (100496) shared sx vm map (user) (vm map (user)) r = 0 (0xfffff801f3da2078) locked @ vm/vm_map.c:4065 Process 29 (pagedaemon) thread 0xfffff8001ba6e560 (100129) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe0002978e48) locked @ kern/vfs_bio.c:1674 exclusive lockmgr ufs (ufs) r = 0 (0xfffff8001b2747d0) locked @ kern/vfs_subr.c:2602 db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffff8001b274c58: tag ufs, type VDIR usecount 9, writecount 0, refcount 10 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffff8000c08c100 ref 0 pages 0 cleanbuf 0 dirtybuf 0 lock type ufs: SHARED (count 9) #0 0xffffffff80a8440b at lockmgr_slock_hard+0x32b #1 0xffffffff80a84ed7 at __lockmgr_args+0x547 #2 0xffffffff80db124f at ffs_lock+0xaf #3 0xffffffff810e3611 at VOP_LOCK1_APV+0xe1 #4 0xffffffff80b92636 at _vn_lock+0x66 #5 0xffffffff80b809e8 at vget+0x98 #6 0xffffffff80b65dda at cache_lookup+0x6fa #7 0xffffffff80b6acfc at vfs_cache_lookup+0xac #8 0xffffffff810e0007 at VOP_LOOKUP_APV+0xf7 #9 0xffffffff80b744a2 at lookup+0x692 #10 0xffffffff80b739eb at namei+0x53b #11 0xffffffff80b91e67 at vn_open_cred+0x207 #12 0xffffffff80b8a7cc at kern_openat+0x20c #13 0xffffffff80f6ce3b at amd64_syscall+0x7ab #14 0xffffffff80f4801d at fast_syscall_common+0x101 ino 56339712, on dev gpt/work vnode 0xfffff8001b274768: tag ufs, type VREG usecount 172, writecount 0, refcount 247 flags (VI_ACTIVE) v_object 0xfffff8000c08bd00 ref 171 pages 205 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0xfffff8001ba6e560 (pid 29, pagedaemon, tid 100129) with shared waiters pending #0 0xffffffff80a8480c at lockmgr_xlock_hard+0x3ac #1 0xffffffff80a84eaa at __lockmgr_args+0x51a #2 0xffffffff80db124f at ffs_lock+0xaf #3 0xffffffff810e3611 at VOP_LOCK1_APV+0xe1 #4 0xffffffff80b92636 at _vn_lock+0x66 #5 0xffffffff80b809e8 at vget+0x98 #6 0xffffffff80df02fd at vm_pageout_laundry_worker+0x8cd #7 0xffffffff80a6fa24 at fork_exit+0x84 #8 0xffffffff80f484ee at fork_trampoline+0xe ino 56340986, on dev gpt/work db:1:pho1> show allchains chain 1: thread 100200 (pid 23408, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 2: thread 100140 (pid 23407, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 3: thread 100187 (pid 23406, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 4: thread 100157 (pid 23405, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 5: thread 100195 (pid 23404, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 6: thread 100196 (pid 23403, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 7: thread 100197 (pid 23402, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 8: thread 100198 (pid 23401, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 9: thread 100199 (pid 23400, getty) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 10: thread 100495 (pid 23324, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 11: thread 100491 (pid 23323, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 12: thread 100498 (pid 23322, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 13: thread 100497 (pid 23321, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 14: thread 100307 (pid 23320, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 15: thread 100223 (pid 23319, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 16: thread 100222 (pid 23318, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 17: thread 100166 (pid 23317, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 18: thread 100208 (pid 23316, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 19: thread 100192 (pid 23315, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 20: thread 100190 (pid 23314, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 21: thread 100457 (pid 23313, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 22: thread 100456 (pid 23312, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 23: thread 100455 (pid 23311, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 24: thread 100217 (pid 23310, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 25: thread 100144 (pid 23309, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 26: thread 100169 (pid 23308, swap) sleeping on 0xffffffff819a0e40 "vmwait" chain 27: thread 100189 (pid 23307, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 28: thread 100423 (pid 23306, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 29: thread 100147 (pid 23305, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 30: thread 100167 (pid 23304, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 31: thread 100287 (pid 23303, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 32: thread 100288 (pid 23302, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 33: thread 100164 (pid 23301, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 34: thread 100165 (pid 23300, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 35: thread 100212 (pid 23299, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 36: thread 100424 (pid 23298, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 37: thread 100401 (pid 23297, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 38: thread 100402 (pid 23296, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 39: thread 100206 (pid 23295, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 40: thread 100185 (pid 23294, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 41: thread 100182 (pid 23293, swap) sleeping on 0xffffffff819a0e40 "vmwait" chain 42: thread 100426 (pid 23292, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 43: thread 100181 (pid 23291, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 44: thread 100159 (pid 23290, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 45: thread 100213 (pid 23289, swap) sleeping on 0xffffffff819a0e40 "pfault" chain 46: thread 100158 (pid 23288, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 47: thread 100414 (pid 23287, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 48: thread 100415 (pid 23286, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 49: thread 100142 (pid 23285, swap) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 50: thread 100496 (pid 23280, swap) sleeping on 0xffffffff819a0e40 "vmwait" chain 51: thread 100168 (pid 1097, sshd) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 52: thread 100191 (pid 773, sshd) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 53: thread 100143 (pid 722, sshd) sleeping on 0xfffff80003ef1ac0 "select" chain 54: thread 100172 (pid 720, dhclient) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 55: thread 100175 (pid 569, casperd) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 56: thread 100171 (pid 528, rpcbind) blocked on lockmgr ufsEXCL thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 57: thread 100133 (pid 33, vnlru) sleeping on 0xfffff8001b382a70 "vlruwt" chain 58: thread 100130 (pid 32, syncer) sleeping on 0xffffffff81ee9c00 "syncer" chain 59: thread 100128 (pid 31, bufdaemon) sleeping on 0xffffffff81ee90a8 "qsleep" chain 60: thread 100131 (pid 31, bufspacedaemon-0) sleeping on 0xffffffff81963738 "bufspace" chain 61: thread 100132 (pid 31, bufspacedaemon-1) sleeping on 0xffffffff8196ba40 "-" chain 62: thread 100134 (pid 31, bufspacedaemon-2) sleeping on 0xffffffff81973d00 "-" chain 63: thread 100136 (pid 31, bufspacedaemon-3) sleeping on 0xffffffff8197bfc0 "-" chain 64: thread 100137 (pid 31, bufspacedaemon-4) sleeping on 0xffffffff81984280 "-" chain 65: thread 100138 (pid 31, bufspacedaemon-5) sleeping on 0xffffffff8198c540 "-" chain 66: thread 100139 (pid 31, bufspacedaemon-6) sleeping on 0xffffffff81994800 "-" chain 67: thread 100150 (pid 31, / worker) sleeping on 0xfffff8001be126e8 "sdflush" chain 68: thread 100155 (pid 31, /tmp worker) sleeping on 0xfffff8001be7c8e8 "sdflush" chain 69: thread 100127 (pid 30, vmdaemon) sleeping on 0xffffffff81ef5618 "psleep" chain 70: thread 100126 (pid 29, dom0) sleeping on 0xffffffff81d4057b "pwait" chain 71: thread 100129 (pid 29, laundry: dom0) sleeping on 0xfffff8084ad12508 "grbmaw" chain 72: thread 100135 (pid 29, uma) sleeping on 0xffffffff80dcf2f0 "umarcl" chain 73: thread 100125 (pid 28, enc_daemon1) sleeping on 0xfffff8001b28d000 "idle" chain 74: thread 100124 (pid 27, enc_daemon0) sleeping on 0xfffff8001b28d538 "idle" chain 75: thread 100121 (pid 26, rand_harvestq) sleeping on 0xffffffff81c147c8 "-" chain 76: thread 100117 (pid 25, sctp_iterator) sleeping on 0xffffffff81f630c0 "waiting_for_work" chain 77: thread 100116 (pid 24, soaiod4) sleeping on 0xffffffff81ee8aa4 "-" chain 78: thread 100115 (pid 23, soaiod3) sleeping on 0xffffffff81ee8aa4 "-" chain 79: thread 100114 (pid 22, soaiod2) sleeping on 0xffffffff81ee8aa4 "-" chain 80: thread 100113 (pid 21, soaiod1) sleeping on 0xffffffff81ee8aa4 "-" chain 81: thread 100094 (pid 20, usbus0) sleeping on 0xfffffe009c7e1460 "-" chain 82: thread 100095 (pid 20, usbus0) sleeping on 0xfffffe009c7e14b8 "-" chain 83: thread 100096 (pid 20, usbus0) sleeping on 0xfffffe009c7e1510 "-" chain 84: thread 100097 (pid 20, usbus0) sleeping on 0xfffffe009c7e1568 "-" chain 85: thread 100098 (pid 20, usbus0) sleeping on 0xfffffe009c7e15c0 "-" chain 86: thread 100100 (pid 20, usbus1) sleeping on 0xfffffe009a5eed10 "-" chain 87: thread 100101 (pid 20, usbus1) sleeping on 0xfffffe009a5eed68 "-" chain 88: thread 100102 (pid 20, usbus1) sleeping on 0xfffffe009a5eedc0 "-" chain 89: thread 100103 (pid 20, usbus1) sleeping on 0xfffffe009a5eee18 "-" chain 90: thread 100104 (pid 20, usbus1) sleeping on 0xfffffe009a5eee70 "-" chain 91: thread 100105 (pid 20, usbus2) sleeping on 0xfffffe009ca62d10 "-" chain 92: thread 100106 (pid 20, usbus2) sleeping on 0xfffffe009ca62d68 "-" chain 93: thread 100107 (pid 20, usbus2) sleeping on 0xfffffe009ca62dc0 "-" chain 94: thread 100108 (pid 20, usbus2) sleeping on 0xfffffe009ca62e18 "-" chain 95: thread 100109 (pid 20, usbus2) sleeping on 0xfffffe009ca62e70 "-" chain 96: thread 100089 (pid 19, doneq0) sleeping on 0xffffffff81aefac0 "-" chain 97: thread 100090 (pid 19, doneq1) sleeping on 0xffffffff81aefb40 "-" chain 98: thread 100091 (pid 19, doneq2) sleeping on 0xffffffff81aefbc0 "-" chain 99: thread 100123 (pid 19, scanner) sleeping on 0xffffffff81aef988 "-" chain 100: thread 100088 (pid 18, crypto returns 11) sleeping on 0xfffff80003ad2450 "crypto_ret_wait" chain 101: thread 100087 (pid 17, crypto returns 10) sleeping on 0xfffff80003ad23f0 "crypto_ret_wait" chain 102: thread 100086 (pid 16, crypto returns 9) sleeping on 0xfffff80003ad2390 "crypto_ret_wait" chain 103: thread 100085 (pid 15, crypto returns 8) sleeping on 0xfffff80003ad2330 "crypto_ret_wait" chain 104: thread 100084 (pid 14, crypto returns 7) sleeping on 0xfffff80003ad22d0 "crypto_ret_wait" chain 105: thread 100083 (pid 9, crypto returns 6) sleeping on 0xfffff80003ad2270 "crypto_ret_wait" chain 106: thread 100082 (pid 8, crypto returns 5) sleeping on 0xfffff80003ad2210 "crypto_ret_wait" chain 107: thread 100081 (pid 7, crypto returns 4) sleeping on 0xfffff80003ad21b0 "crypto_ret_wait" chain 108: thread 100080 (pid 6, crypto returns 3) sleeping on 0xfffff80003ad2150 "crypto_ret_wait" chain 109: thread 100079 (pid 5, crypto returns 2) sleeping on 0xfffff80003ad20f0 "crypto_ret_wait" chain 110: thread 100078 (pid 4, crypto returns 1) sleeping on 0xfffff80003ad2090 "crypto_ret_wait" chain 111: thread 100077 (pid 3, crypto returns 0) sleeping on 0xfffff80003ad2030 "crypto_ret_wait" chain 112: thread 100076 (pid 2, crypto) sleeping on 0xffffffff81eefe58 "crypto_wait" chain 113: thread 100060 (pid 13, g_event) sleeping on 0xffffffff81f3fb20 "-" chain 114: thread 100061 (pid 13, g_up) sleeping on 0xffffffff81f3fb28 "-" chain 115: thread 100062 (pid 13, g_down) sleeping on 0xffffffff81f3fb30 "-" chain 116: thread 100002 (pid 1, init) sleeping on 0xfffff80003934538 "wait" chain 117: thread 100001 (pid 10, audit) sleeping on 0xffffffff81f63da8 "audit_worker_cv" chain 118: thread 100000 (pid 0, swapper) sleeping on 0xffffffff819a0e40 "vmwait" chain 119: thread 100017 (pid 0, thread taskq) sleeping on 0xfffff80003970600 "-" chain 120: thread 100019 (pid 0, kqueue_ctx taskq) sleeping on 0xfffff80003970100 "-" chain 121: thread 100020 (pid 0, aiod_kick taskq) sleeping on 0xfffff8000396fe00 "-" chain 122: thread 100021 (pid 0, if_config_tqg_0) sleeping on 0xfffff8000397e000 "-" chain 123: thread 100022 (pid 0, softirq_0) sleeping on 0xfffff8000397de00 "-" chain 124: thread 100023 (pid 0, softirq_1) sleeping on 0xfffff8000397dd00 "-" chain 125: thread 100024 (pid 0, softirq_2) sleeping on 0xfffff8000397dc00 "-" chain 126: thread 100025 (pid 0, softirq_3) sleeping on 0xfffff8000397db00 "-" chain 127: thread 100026 (pid 0, softirq_4) sleeping on 0xfffff8000397da00 "-" chain 128: thread 100027 (pid 0, softirq_5) sleeping on 0xfffff8000397d900 "-" chain 129: thread 100028 (pid 0, softirq_6) sleeping on 0xfffff8000397d800 "-" chain 130: thread 100029 (pid 0, softirq_7) sleeping on 0xfffff8000397d700 "-" chain 131: thread 100030 (pid 0, softirq_8) sleeping on 0xfffff8000397d600 "-" chain 132: thread 100031 (pid 0, softirq_9) sleeping on 0xfffff8000397d500 "-" chain 133: thread 100032 (pid 0, softirq_10) sleeping on 0xfffff8000397d400 "-" chain 134: thread 100033 (pid 0, softirq_11) sleeping on 0xfffff8000397d300 "-" chain 135: thread 100034 (pid 0, if_io_tqg_0) sleeping on 0xfffff8000397d200 "-" chain 136: thread 100035 (pid 0, if_io_tqg_1) sleeping on 0xfffff8000397d100 "-" chain 137: thread 100036 (pid 0, if_io_tqg_2) sleeping on 0xfffff8000397d000 "-" chain 138: thread 100037 (pid 0, if_io_tqg_3) sleeping on 0xfffff8000397ce00 "-" chain 139: thread 100038 (pid 0, if_io_tqg_4) sleeping on 0xfffff8000397cd00 "-" chain 140: thread 100039 (pid 0, if_io_tqg_5) sleeping on 0xfffff8000397cc00 "-" chain 141: thread 100040 (pid 0, if_io_tqg_6) sleeping on 0xfffff8000397cb00 "-" chain 142: thread 100041 (pid 0, if_io_tqg_7) sleeping on 0xfffff8000397ca00 "-" chain 143: thread 100042 (pid 0, if_io_tqg_8) sleeping on 0xfffff8000397c900 "-" chain 144: thread 100043 (pid 0, if_io_tqg_9) sleeping on 0xfffff8000397c800 "-" chain 145: thread 100044 (pid 0, if_io_tqg_10) sleeping on 0xfffff8000397c700 "-" chain 146: thread 100045 (pid 0, if_io_tqg_11) sleeping on 0xfffff8000397c600 "-" chain 147: thread 100063 (pid 0, firmware taskq) sleeping on 0xfffff80003a0a100 "-" chain 148: thread 100064 (pid 0, crypto_0) sleeping on 0xfffff80003a09e00 "-" chain 149: thread 100065 (pid 0, crypto_1) sleeping on 0xfffff80003a09e00 "-" chain 150: thread 100066 (pid 0, crypto_2) sleeping on 0xfffff80003a09e00 "-" chain 151: thread 100067 (pid 0, crypto_3) sleeping on 0xfffff80003a09e00 "-" chain 152: thread 100068 (pid 0, crypto_4) sleeping on 0xfffff80003a09e00 "-" chain 153: thread 100069 (pid 0, crypto_5) sleeping on 0xfffff80003a09e00 "-" chain 154: thread 100070 (pid 0, crypto_6) sleeping on 0xfffff80003a09e00 "-" chain 155: thread 100071 (pid 0, crypto_7) sleeping on 0xfffff80003a09e00 "-" chain 156: thread 100072 (pid 0, crypto_8) sleeping on 0xfffff80003a09e00 "-" chain 157: thread 100073 (pid 0, crypto_9) sleeping on 0xfffff80003a09e00 "-" chain 158: thread 100074 (pid 0, crypto_10) sleeping on 0xfffff80003a09e00 "-" chain 159: thread 100075 (pid 0, crypto_11) sleeping on 0xfffff80003a09e00 "-" chain 160: thread 100112 (pid 0, mca taskq) sleeping on 0xfffff8000c0c3c00 "-" chain 161: thread 100118 (pid 0, acpi_task_0) sleeping on 0xfffff8001b39b700 "-" chain 162: thread 100119 (pid 0, acpi_task_1) sleeping on 0xfffff8001b39b700 "-" chain 163: thread 100120 (pid 0, acpi_task_2) sleeping on 0xfffff8001b39b700 "-" chain 164: thread 100122 (pid 0, CAM taskq) sleeping on 0xfffff80003971300 "-" db:1:pho1> show mount 0xfffff8001ba83000 /dev/gpt/work on / (ufs) 0xfffff8001ba84000 devfs on /dev (devfs) 0xfffff8001be50000 /dev/gpt/tmp on /tmp (ufs) More info: show mount db:1:pho1> show page vm_cnt.v_free_count: 31720 vm_cnt.v_inactive_count: 89 vm_cnt.v_active_count: 493941 vm_cnt.v_laundry_count: 7492994 vm_cnt.v_wire_count: 111586 vm_cnt.v_free_reserved: 10652 vm_cnt.v_free_min: 51303 vm_cnt.v_free_target: 173256 vm_cnt.v_inactive_target: 259884 db:1:pho1> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000091 | 000021 06 (000256K) | 000031 | 000007 05 (000128K) | 000073 | 000002 04 (000064K) | 000085 | 000009 03 (000032K) | 000110 | 000023 02 (000016K) | 000106 | 000068 01 (000008K) | 000280 | 000141 00 (000004K) | 005181 | 000361 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000000 | 000000 05 (000128K) | 000000 | 000000 04 (000064K) | 000000 | 000000 03 (000032K) | 000000 | 000000 02 (000016K) | 000000 | 000000 01 (000008K) | 000000 | 000000 00 (000004K) | 000000 | 000000 db:1:pho1> show pageq pq_free 31720 dom 0 page_cnt 8130451 free 31720 pq_act 493941 pq_inact 89 pq_laund 7492994 pq_unsw 0 db:1:pho1> show bufqueues bqempty: 58147 Buf domain 0 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 14439 space 0 Buf domain 1 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 15217 space 0 Buf domain 2 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 14246 space 0 Buf domain 3 freebufs 15132 lofreebufs 219 hifreebufs 329 bufspace 32768 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 1 space 0 Total buffers: 12077 space 0 Buf domain 4 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 14371 space 0 Buf domain 5 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 33539 space 0 Buf domain 6 freebufs 15133 lofreebufs 219 hifreebufs 329 bufspace 0 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 1 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Locked buffers: 0 space 0 Total buffers: 2042 space 0 db:1:pho1> ps pid ppid pgrp uid state wmesg wchan cmd 23408 1 1 0 D ufs 0xfffff8001b2747d0 getty 23407 1 1 0 D ufs 0xfffff8001b2747d0 getty 23406 1 1 0 D ufs 0xfffff8001b2747d0 getty 23405 1 1 0 D ufs 0xfffff8001b2747d0 getty 23404 1 1 0 D ufs 0xfffff8001b2747d0 getty 23403 1 1 0 D ufs 0xfffff8001b2747d0 getty 23402 1 1 0 D ufs 0xfffff8001b2747d0 getty 23401 1 1 0 D ufs 0xfffff8001b2747d0 getty 23400 1 1 0 D ufs 0xfffff8001b2747d0 getty 23324 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23323 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23322 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23321 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23320 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23319 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23318 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23317 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23316 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23315 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23314 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23313 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23312 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23311 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23310 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23309 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23308 23284 784 0 D+ vmwait 0xffffffff819a0e40 swap 23307 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23306 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23305 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23304 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23303 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23302 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23301 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23300 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23299 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23298 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23297 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23296 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23295 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23294 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23293 23284 784 0 D+ vmwait 0xffffffff819a0e40 swap 23292 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23291 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23290 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23289 23284 784 0 DW+ pfault 0xffffffff819a0e40 swap 23288 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23287 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23286 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23285 23284 784 0 D+ ufs 0xfffff8001b2747d0 swap 23284 23280 784 0 ?W+ swap 23280 22297 784 0 D+ vmwait 0xffffffff819a0e40 swap 22352 22308 22352 2007 ?W+ fdgrowtable 22308 22297 784 0 ?W+ su 22297 22295 784 0 ?W+ sh 22295 784 784 0 ?W+ sh 1102 1100 1102 0 ?W+ bash 1100 1099 1100 0 ?W+ csh 1099 1098 1099 2006 ?W+ su 1098 1097 1098 2006 ?Ws+ bash 1097 1 1089 2006 D ufs 0xfffff8001b2747d0 sshd 784 781 784 0 ?W+ sh 781 779 781 0 ?W+ bash 779 778 779 0 ?W+ csh 778 777 778 2006 ?W+ su 777 1 777 2006 ?Ws+ bash 773 722 773 0 Ds ufs 0xfffff8001b2747d0 sshd 769 1 769 0 ?Ws+ login 726 1 726 0 ?Ws cron 722 1 722 0 Ss select 0xfffff80003ef1ac0 sshd 720 1 720 65 Ds ufs 0xfffff8001b2747d0 dhclient 569 1 569 0 Ds ufs 0xfffff8001b2747d0 casperd 528 1 528 0 Ds ufs 0xfffff8001b2747d0 rpcbind 33 0 0 0 DL vlruwt 0xfffff8001b382a70 [vnlru] 32 0 0 0 DL syncer 0xffffffff81ee9c00 [syncer] 31 0 0 0 DL (threaded) [bufdaemon] 100128 D qsleep 0xffffffff81ee90a8 [bufdaemon] 100131 D bufspac 0xffffffff81963738 [bufspacedaemon-0] 100132 D - 0xffffffff8196ba40 [bufspacedaemon-1] 100134 D - 0xffffffff81973d00 [bufspacedaemon-2] 100136 D - 0xffffffff8197bfc0 [bufspacedaemon-3] 100137 D - 0xffffffff81984280 [bufspacedaemon-4] 100138 D - 0xffffffff8198c540 [bufspacedaemon-5] 100139 D - 0xffffffff81994800 [bufspacedaemon-6] 100150 D sdflush 0xfffff8001be126e8 [/ worker] 100155 D sdflush 0xfffff8001be7c8e8 [/tmp worker] 30 0 0 0 DL psleep 0xffffffff81ef5618 [vmdaemon] 29 0 0 0 DL (threaded) [pagedaemon] 100126 D pwait 0xffffffff81d4057b [dom0] 100129 D grbmaw 0xfffff8084ad12508 [laundry: dom0] 100135 D umarcl 0xffffffff80dcf2f0 [uma] 28 0 0 0 DL idle 0xfffff8001b28d000 [enc_daemon1] 27 0 0 0 DL idle 0xfffff8001b28d538 [enc_daemon0] 26 0 0 0 DL - 0xffffffff81c147c8 [rand_harvestq] 25 0 0 0 DL waiting 0xffffffff81f630c0 [sctp_iterator] 24 0 0 0 DL - 0xffffffff81ee8aa4 [soaiod4] 23 0 0 0 DL - 0xffffffff81ee8aa4 [soaiod3] 22 0 0 0 DL - 0xffffffff81ee8aa4 [soaiod2] 21 0 0 0 DL - 0xffffffff81ee8aa4 [soaiod1] 20 0 0 0 DL (threaded) [usb] 100094 D - 0xfffffe009c7e1460 [usbus0] 100095 D - 0xfffffe009c7e14b8 [usbus0] 100096 D - 0xfffffe009c7e1510 [usbus0] 100097 D - 0xfffffe009c7e1568 [usbus0] 100098 D - 0xfffffe009c7e15c0 [usbus0] 100100 D - 0xfffffe009a5eed10 [usbus1] 100101 D - 0xfffffe009a5eed68 [usbus1] 100102 D - 0xfffffe009a5eedc0 [usbus1] 100103 D - 0xfffffe009a5eee18 [usbus1] 100104 D - 0xfffffe009a5eee70 [usbus1] 100105 D - 0xfffffe009ca62d10 [usbus2] 100106 D - 0xfffffe009ca62d68 [usbus2] 100107 D - 0xfffffe009ca62dc0 [usbus2] 100108 D - 0xfffffe009ca62e18 [usbus2] 100109 D - 0xfffffe009ca62e70 [usbus2] 19 0 0 0 DL (threaded) [cam] 100089 D - 0xffffffff81aefac0 [doneq0] 100090 D - 0xffffffff81aefb40 [doneq1] 100091 D - 0xffffffff81aefbc0 [doneq2] 100123 D - 0xffffffff81aef988 [scanner] 18 0 0 0 DL crypto_ 0xfffff80003ad2450 [crypto returns 11] 17 0 0 0 DL crypto_ 0xfffff80003ad23f0 [crypto returns 10] 16 0 0 0 DL crypto_ 0xfffff80003ad2390 [crypto returns 9] 15 0 0 0 DL crypto_ 0xfffff80003ad2330 [crypto returns 8] 14 0 0 0 DL crypto_ 0xfffff80003ad22d0 [crypto returns 7] 9 0 0 0 DL crypto_ 0xfffff80003ad2270 [crypto returns 6] 8 0 0 0 DL crypto_ 0xfffff80003ad2210 [crypto returns 5] 7 0 0 0 DL crypto_ 0xfffff80003ad21b0 [crypto returns 4] 6 0 0 0 DL crypto_ 0xfffff80003ad2150 [crypto returns 3] 5 0 0 0 DL crypto_ 0xfffff80003ad20f0 [crypto returns 2] 4 0 0 0 DL crypto_ 0xfffff80003ad2090 [crypto returns 1] 3 0 0 0 DL crypto_ 0xfffff80003ad2030 [crypto returns 0] 2 0 0 0 DL crypto_ 0xffffffff81eefe58 [crypto] 13 0 0 0 DL (threaded) [geom] 100060 D - 0xffffffff81f3fb20 [g_event] 100061 D - 0xffffffff81f3fb28 [g_up] 100062 D - 0xffffffff81f3fb30 [g_down] 12 0 0 0 WL (threaded) [intr] 100015 I [swi6: task queue] 100016 I [swi6: Giant taskq] 100018 I [swi5: fast taskq] 100046 I [swi4: clock (0)] 100047 I [swi4: clock (1)] 100048 I [swi4: clock (2)] 100049 I [swi4: clock (3)] 100050 I [swi4: clock (4)] 100051 I [swi4: clock (5)] 100052 I [swi4: clock (6)] 100053 I [swi4: clock (7)] 100054 I [swi4: clock (8)] 100055 I [swi4: clock (9)] 100056 I [swi4: clock (10)] 100057 I [swi4: clock (11)] 100058 I [swi1: netisr 0] 100059 I [swi3: vm] 100092 I [irq264: ahci0] 100093 I [irq19: xhci0] 100099 I [irq18: ehci0 ehci1] 100110 I [irq279: ahci1] 100111 I [swi0: uart uart] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 Run CPU 10 [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffff80003934538 [init] 10 0 0 0 DL audit_w 0xffffffff81f63da8 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D vmwait 0xffffffff819a0e40 [swapper] 100017 D - 0xfffff80003970600 [thread taskq] 100019 D - 0xfffff80003970100 [kqueue_ctx taskq] 100020 D - 0xfffff8000396fe00 [aiod_kick taskq] 100021 D - 0xfffff8000397e000 [if_config_tqg_0] 100022 D - 0xfffff8000397de00 [softirq_0] 100023 D - 0xfffff8000397dd00 [softirq_1] 100024 D - 0xfffff8000397dc00 [softirq_2] 100025 D - 0xfffff8000397db00 [softirq_3] 100026 D - 0xfffff8000397da00 [softirq_4] 100027 D - 0xfffff8000397d900 [softirq_5] 100028 D - 0xfffff8000397d800 [softirq_6] 100029 D - 0xfffff8000397d700 [softirq_7] 100030 D - 0xfffff8000397d600 [softirq_8] 100031 D - 0xfffff8000397d500 [softirq_9] 100032 D - 0xfffff8000397d400 [softirq_10] 100033 D - 0xfffff8000397d300 [softirq_11] 100034 D - 0xfffff8000397d200 [if_io_tqg_0] 100035 D - 0xfffff8000397d100 [if_io_tqg_1] 100036 D - 0xfffff8000397d000 [if_io_tqg_2] 100037 D - 0xfffff8000397ce00 [if_io_tqg_3] 100038 D - 0xfffff8000397cd00 [if_io_tqg_4] 100039 D - 0xfffff8000397cc00 [if_io_tqg_5] 100040 D - 0xfffff8000397cb00 [if_io_tqg_6] 100041 D - 0xfffff8000397ca00 [if_io_tqg_7] 100042 D - 0xfffff8000397c900 [if_io_tqg_8] 100043 D - 0xfffff8000397c800 [if_io_tqg_9] 100044 D - 0xfffff8000397c700 [if_io_tqg_10] 100045 D - 0xfffff8000397c600 [if_io_tqg_11] 100063 D - 0xfffff80003a0a100 [firmware taskq] 100064 D - 0xfffff80003a09e00 [crypto_0] 100065 D - 0xfffff80003a09e00 [crypto_1] 100066 D - 0xfffff80003a09e00 [crypto_2] 100067 D - 0xfffff80003a09e00 [crypto_3] 100068 D - 0xfffff80003a09e00 [crypto_4] 100069 D - 0xfffff80003a09e00 [crypto_5] 100070 D - 0xfffff80003a09e00 [crypto_6] 100071 D - 0xfffff80003a09e00 [crypto_7] 100072 D - 0xfffff80003a09e00 [crypto_8] 100073 D - 0xfffff80003a09e00 [crypto_9] 100074 D - 0xfffff80003a09e00 [crypto_10] 100075 D - 0xfffff80003a09e00 [crypto_11] 100112 D - 0xfffff8000c0c3c00 [mca taskq] 100118 D - 0xfffff8001b39b700 [acpi_task_0] 100119 D - 0xfffff8001b39b700 [acpi_task_1] 100120 D - 0xfffff8001b39b700 [acpi_task_2] 100122 D - 0xfffff80003971300 [CAM taskq] 22296 784 784 0 Z+ tee 776 773 773 2006 Z sshd 770 769 770 0 Z+ csh 1103 1102 1103 0 Z+ top 23399 22352 22352 2007 Z+ fdgrowtable 23398 22352 22352 2007 Z+ fdgrowtable db:0:pho> allt Tracing command getty pid 23408 tid 100200 td 0xfffff8001bdb8000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef350b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef350e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef35120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef35150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef351b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef35230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef352d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef35320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef35350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef353c0 vget() at vget+0x98/frame 0xfffffe009ef35400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef354e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef35540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef35570 lookup() at lookup+0x692/frame 0xfffffe009ef35610 namei() at namei+0x53b/frame 0xfffffe009ef356d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef35810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef35980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef35ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef35ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23407 tid 100140 td 0xfffff8001bad3560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee090b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee090e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee09120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee09150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee091b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee09230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee092d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee09320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee09350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee093c0 vget() at vget+0x98/frame 0xfffffe009ee09400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ee094e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ee09540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ee09570 lookup() at lookup+0x692/frame 0xfffffe009ee09610 namei() at namei+0x53b/frame 0xfffffe009ee096d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ee09810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ee09980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee09ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee09ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23406 tid 100187 td 0xfffff8001bdf2000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eef40b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eef40e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eef4120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eef4150 sleeplk() at sleeplk+0x149/frame 0xfffffe009eef41b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eef4230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eef42d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eef4320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eef4350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eef43c0 vget() at vget+0x98/frame 0xfffffe009eef4400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009eef44e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009eef4540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009eef4570 lookup() at lookup+0x692/frame 0xfffffe009eef4610 namei() at namei+0x53b/frame 0xfffffe009eef46d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009eef4810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009eef4980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009eef4ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009eef4ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23405 tid 100157 td 0xfffff8001bdf2560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee5e0b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee5e0e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee5e120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee5e150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee5e1b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee5e230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee5e2d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee5e320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee5e350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee5e3c0 vget() at vget+0x98/frame 0xfffffe009ee5e400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ee5e4e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ee5e540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ee5e570 lookup() at lookup+0x692/frame 0xfffffe009ee5e610 namei() at namei+0x53b/frame 0xfffffe009ee5e6d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ee5e810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ee5e980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee5eab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee5eab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23404 tid 100195 td 0xfffff8001bdf1560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef1c0b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef1c0e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef1c120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef1c150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef1c1b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef1c230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef1c2d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef1c320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef1c350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef1c3c0 vget() at vget+0x98/frame 0xfffffe009ef1c400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef1c4e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef1c540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef1c570 lookup() at lookup+0x692/frame 0xfffffe009ef1c610 namei() at namei+0x53b/frame 0xfffffe009ef1c6d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef1c810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef1c980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef1cab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef1cab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23403 tid 100196 td 0xfffff8001bdf1000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef210b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef210e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef21120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef21150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef211b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef21230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef212d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef21320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef21350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef213c0 vget() at vget+0x98/frame 0xfffffe009ef21400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef214e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef21540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef21570 lookup() at lookup+0x692/frame 0xfffffe009ef21610 namei() at namei+0x53b/frame 0xfffffe009ef216d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef21810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef21980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef21ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef21ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23402 tid 100197 td 0xfffff8001bdf0560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef260b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef260e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef26120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef26150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef261b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef26230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef262d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef26320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef26350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef263c0 vget() at vget+0x98/frame 0xfffffe009ef26400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef264e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef26540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef26570 lookup() at lookup+0x692/frame 0xfffffe009ef26610 namei() at namei+0x53b/frame 0xfffffe009ef266d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef26810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef26980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef26ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef26ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23401 tid 100198 td 0xfffff8001bdf0000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef2b0b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef2b0e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef2b120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef2b150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef2b1b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef2b230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef2b2d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef2b320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef2b350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef2b3c0 vget() at vget+0x98/frame 0xfffffe009ef2b400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef2b4e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef2b540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef2b570 lookup() at lookup+0x692/frame 0xfffffe009ef2b610 namei() at namei+0x53b/frame 0xfffffe009ef2b6d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef2b810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef2b980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef2bab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef2bab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command getty pid 23400 tid 100199 td 0xfffff8001bdb8560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef300b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef300e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef30120 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef30150 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef301b0 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef30230 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef302d0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef30320 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef30350 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef303c0 vget() at vget+0x98/frame 0xfffffe009ef30400 cache_lookup() at cache_lookup+0x6fa/frame 0xfffffe009ef304e0 vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfffffe009ef30540 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xf7/frame 0xfffffe009ef30570 lookup() at lookup+0x692/frame 0xfffffe009ef30610 namei() at namei+0x53b/frame 0xfffffe009ef306d0 vn_open_cred() at vn_open_cred+0x207/frame 0xfffffe009ef30810 kern_openat() at kern_openat+0x20c/frame 0xfffffe009ef30980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ef30ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ef30ab0 --- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x800221eea, rsp = 0x7fffffffd228, rbp = 0x7fffffffd310 --- Tracing command swap pid 23324 tid 100495 td 0xfffff806f18a6000 sched_switch() at sched_switch+0x942/frame 0xfffffe00af0e0490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00af0e04c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00af0e0500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00af0e0530 sleeplk() at sleeplk+0x149/frame 0xfffffe00af0e0590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00af0e0610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00af0e06b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00af0e0700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00af0e0730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00af0e07a0 vget() at vget+0x98/frame 0xfffffe00af0e07e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00af0e0910 vm_fault() at vm_fault+0x75/frame 0xfffffe00af0e0950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00af0e09a0 trap() at trap+0x3be/frame 0xfffffe00af0e0ab0 calltrap() at calltrap+0x8/frame 0xfffffe00af0e0ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23323 tid 100491 td 0xfffff806f1d60000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23322 tid 100498 td 0xfffff80452908560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23321 tid 100497 td 0xfffff80452909000 sched_switch() at sched_switch+0x942/frame 0xfffffe00af0ea490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00af0ea4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00af0ea500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00af0ea530 sleeplk() at sleeplk+0x149/frame 0xfffffe00af0ea590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00af0ea610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00af0ea6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00af0ea700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00af0ea730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00af0ea7a0 vget() at vget+0x98/frame 0xfffffe00af0ea7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00af0ea910 vm_fault() at vm_fault+0x75/frame 0xfffffe00af0ea950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00af0ea9a0 trap() at trap+0x3be/frame 0xfffffe00af0eaab0 calltrap() at calltrap+0x8/frame 0xfffffe00af0eaab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23320 tid 100307 td 0xfffff801f3ab6560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a0f4e490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a0f4e4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a0f4e500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a0f4e530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a0f4e590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a0f4e610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a0f4e6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a0f4e700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a0f4e730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a0f4e7a0 vget() at vget+0x98/frame 0xfffffe00a0f4e7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a0f4e910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a0f4e950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a0f4e9a0 trap() at trap+0x3be/frame 0xfffffe00a0f4eab0 calltrap() at calltrap+0x8/frame 0xfffffe00a0f4eab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23319 tid 100223 td 0xfffff8001bad1000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23318 tid 100222 td 0xfffff8001bad1560 sched_switch() at sched_switch+0x942/frame 0xfffffe009efa3490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009efa34c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009efa3500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009efa3530 sleeplk() at sleeplk+0x149/frame 0xfffffe009efa3590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009efa3610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009efa36b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009efa3700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009efa3730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009efa37a0 vget() at vget+0x98/frame 0xfffffe009efa37e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009efa3910 vm_fault() at vm_fault+0x75/frame 0xfffffe009efa3950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009efa39a0 trap() at trap+0x3be/frame 0xfffffe009efa3ab0 calltrap() at calltrap+0x8/frame 0xfffffe009efa3ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23317 tid 100166 td 0xfffff8001bdb1560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee8b490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee8b4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee8b500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee8b530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee8b590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee8b610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee8b6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee8b700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee8b730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee8b7a0 vget() at vget+0x98/frame 0xfffffe009ee8b7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee8b910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee8b950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee8b9a0 trap() at trap+0x3be/frame 0xfffffe009ee8bab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee8bab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23316 tid 100208 td 0xfffff8001bff1560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23315 tid 100192 td 0xfffff8006aaed000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23314 tid 100190 td 0xfffff8006aaee000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef03490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef034c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef03500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef03530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef03590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef03610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef036b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef03700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef03730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef037a0 vget() at vget+0x98/frame 0xfffffe009ef037e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ef03910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ef03950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ef039a0 trap() at trap+0x3be/frame 0xfffffe009ef03ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ef03ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23313 tid 100457 td 0xfffff807565d6000 sched_switch() at sched_switch+0x942/frame 0xfffffe00af02c490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00af02c4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00af02c500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00af02c530 sleeplk() at sleeplk+0x149/frame 0xfffffe00af02c590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00af02c610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00af02c6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00af02c700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00af02c730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00af02c7a0 vget() at vget+0x98/frame 0xfffffe00af02c7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00af02c910 vm_fault() at vm_fault+0x75/frame 0xfffffe00af02c950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00af02c9a0 trap() at trap+0x3be/frame 0xfffffe00af02cab0 calltrap() at calltrap+0x8/frame 0xfffffe00af02cab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23312 tid 100456 td 0xfffff8016ea3f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00af027490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00af0274c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00af027500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00af027530 sleeplk() at sleeplk+0x149/frame 0xfffffe00af027590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00af027610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00af0276b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00af027700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00af027730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00af0277a0 vget() at vget+0x98/frame 0xfffffe00af0277e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00af027910 vm_fault() at vm_fault+0x75/frame 0xfffffe00af027950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00af0279a0 trap() at trap+0x3be/frame 0xfffffe00af027ab0 calltrap() at calltrap+0x8/frame 0xfffffe00af027ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23311 tid 100455 td 0xfffff8016ea40000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23310 tid 100217 td 0xfffff8001ba7c000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23309 tid 100144 td 0xfffff8001ba6d560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee1d490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee1d4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee1d500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee1d530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee1d590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee1d610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee1d6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee1d700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee1d730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee1d7a0 vget() at vget+0x98/frame 0xfffffe009ee1d7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee1d910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee1d950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee1d9a0 trap() at trap+0x3be/frame 0xfffffe009ee1dab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee1dab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23308 tid 100169 td 0xfffff8001ba6c560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee9a540 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee9a570 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee9a5b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee9a5e0 _sleep() at _sleep+0x367/frame 0xfffffe009ee9a690 vm_wait_doms() at vm_wait_doms+0x85/frame 0xfffffe009ee9a6b0 _pmap_allocpte() at _pmap_allocpte+0x1e5/frame 0xfffffe009ee9a720 pmap_enter() at pmap_enter+0x532/frame 0xfffffe009ee9a7e0 vm_fault_hold() at vm_fault_hold+0x1cd4/frame 0xfffffe009ee9a910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee9a950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee9a9a0 trap() at trap+0x3be/frame 0xfffffe009ee9aab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee9aab0 --- trap 0xc, rip = 0x401d69, rsp = 0x7fffffffe570, rbp = 0x7fffffffe590 --- Tracing command swap pid 23307 tid 100189 td 0xfffff8000419b000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23306 tid 100423 td 0xfffff80003c5a000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a1194490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a11944c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a1194500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a1194530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a1194590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a1194610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a11946b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a1194700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a1194730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a11947a0 vget() at vget+0x98/frame 0xfffffe00a11947e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a1194910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a1194950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a11949a0 trap() at trap+0x3be/frame 0xfffffe00a1194ab0 calltrap() at calltrap+0x8/frame 0xfffffe00a1194ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23305 tid 100147 td 0xfffff8001ba7e000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee2c490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee2c4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee2c500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee2c530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee2c590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee2c610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee2c6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee2c700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee2c730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee2c7a0 vget() at vget+0x98/frame 0xfffffe009ee2c7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee2c910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee2c950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee2c9a0 trap() at trap+0x3be/frame 0xfffffe009ee2cab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee2cab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23304 tid 100167 td 0xfffff8001ba7d560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee90490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee904c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee90500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee90530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee90590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee90610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee906b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee90700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee90730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee907a0 vget() at vget+0x98/frame 0xfffffe009ee907e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee90910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee90950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee909a0 trap() at trap+0x3be/frame 0xfffffe009ee90ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee90ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23303 tid 100287 td 0xfffff801f39d7560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a0eea490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a0eea4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a0eea500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a0eea530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a0eea590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a0eea610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a0eea6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a0eea700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a0eea730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a0eea7a0 vget() at vget+0x98/frame 0xfffffe00a0eea7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a0eea910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a0eea950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a0eea9a0 trap() at trap+0x3be/frame 0xfffffe00a0eeaab0 calltrap() at calltrap+0x8/frame 0xfffffe00a0eeaab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23302 tid 100288 td 0xfffff801f39d7000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a0eef490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a0eef4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a0eef500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a0eef530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a0eef590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a0eef610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a0eef6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a0eef700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a0eef730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a0eef7a0 vget() at vget+0x98/frame 0xfffffe00a0eef7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a0eef910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a0eef950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a0eef9a0 trap() at trap+0x3be/frame 0xfffffe00a0eefab0 calltrap() at calltrap+0x8/frame 0xfffffe00a0eefab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23301 tid 100164 td 0xfffff8001ba66560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee81490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee814c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee81500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee81530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee81590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee81610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee816b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee81700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee81730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee817a0 vget() at vget+0x98/frame 0xfffffe009ee817e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee81910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee81950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee819a0 trap() at trap+0x3be/frame 0xfffffe009ee81ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee81ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23300 tid 100165 td 0xfffff8001ba66000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee86490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee864c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee86500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee86530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee86590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee86610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee866b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee86700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee86730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee867a0 vget() at vget+0x98/frame 0xfffffe009ee867e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee86910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee86950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee869a0 trap() at trap+0x3be/frame 0xfffffe009ee86ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee86ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23299 tid 100212 td 0xfffff8016e006560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef71490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef714c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef71500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef71530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef71590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef71610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef716b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef71700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef71730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef717a0 vget() at vget+0x98/frame 0xfffffe009ef717e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ef71910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ef71950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ef719a0 trap() at trap+0x3be/frame 0xfffffe009ef71ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ef71ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23298 tid 100424 td 0xfffff8006aaee560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23297 tid 100401 td 0xfffff8024194b560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a1126490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a11264c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a1126500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a1126530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a1126590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a1126610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a11266b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a1126700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a1126730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a11267a0 vget() at vget+0x98/frame 0xfffffe00a11267e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a1126910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a1126950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a11269a0 trap() at trap+0x3be/frame 0xfffffe00a1126ab0 calltrap() at calltrap+0x8/frame 0xfffffe00a1126ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23296 tid 100402 td 0xfffff8024194b000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a112b490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a112b4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a112b500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a112b530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a112b590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a112b610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a112b6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a112b700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a112b730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a112b7a0 vget() at vget+0x98/frame 0xfffffe00a112b7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a112b910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a112b950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a112b9a0 trap() at trap+0x3be/frame 0xfffffe00a112bab0 calltrap() at calltrap+0x8/frame 0xfffffe00a112bab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23295 tid 100206 td 0xfffff8016e007000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef53490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef534c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef53500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef53530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef53590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef53610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef536b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef53700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef53730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef537a0 vget() at vget+0x98/frame 0xfffffe009ef537e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ef53910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ef53950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ef539a0 trap() at trap+0x3be/frame 0xfffffe009ef53ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ef53ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23294 tid 100185 td 0xfffff80003c5b560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eeea490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eeea4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eeea500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eeea530 sleeplk() at sleeplk+0x149/frame 0xfffffe009eeea590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eeea610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eeea6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eeea700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eeea730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eeea7a0 vget() at vget+0x98/frame 0xfffffe009eeea7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009eeea910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eeea950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eeea9a0 trap() at trap+0x3be/frame 0xfffffe009eeeaab0 calltrap() at calltrap+0x8/frame 0xfffffe009eeeaab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23293 tid 100182 td 0xfffff8001bcbb560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eedb540 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eedb570 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eedb5b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eedb5e0 _sleep() at _sleep+0x367/frame 0xfffffe009eedb690 vm_wait_doms() at vm_wait_doms+0x85/frame 0xfffffe009eedb6b0 _pmap_allocpte() at _pmap_allocpte+0x1e5/frame 0xfffffe009eedb720 pmap_enter() at pmap_enter+0x532/frame 0xfffffe009eedb7e0 vm_fault_hold() at vm_fault_hold+0x1cd4/frame 0xfffffe009eedb910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eedb950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eedb9a0 trap() at trap+0x3be/frame 0xfffffe009eedbab0 calltrap() at calltrap+0x8/frame 0xfffffe009eedbab0 --- trap 0xc, rip = 0x8008292c4, rsp = 0x7fffffffe218, rbp = 0x7fffffffe260 --- Tracing command swap pid 23292 tid 100426 td 0xfffff8006a369000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23291 tid 100181 td 0xfffff8001bcff000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eed6490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eed64c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eed6500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eed6530 sleeplk() at sleeplk+0x149/frame 0xfffffe009eed6590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eed6610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eed66b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eed6700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eed6730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eed67a0 vget() at vget+0x98/frame 0xfffffe009eed67e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009eed6910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eed6950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eed69a0 trap() at trap+0x3be/frame 0xfffffe009eed6ab0 calltrap() at calltrap+0x8/frame 0xfffffe009eed6ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23290 tid 100159 td 0xfffff8001bd00560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee68490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee684c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee68500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee68530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee68590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee68610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee686b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee68700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee68730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee687a0 vget() at vget+0x98/frame 0xfffffe009ee687e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee68910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee68950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee689a0 trap() at trap+0x3be/frame 0xfffffe009ee68ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee68ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23289 tid 100213 td 0xfffff8001be2c560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23288 tid 100158 td 0xfffff8001bad2560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee63490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee634c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee63500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee63530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee63590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee63610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee636b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee63700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee63730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee637a0 vget() at vget+0x98/frame 0xfffffe009ee637e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee63910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee63950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee639a0 trap() at trap+0x3be/frame 0xfffffe009ee63ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee63ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23287 tid 100414 td 0xfffff80241af6000 sched_switch() at sched_switch+0x942/frame 0xfffffe00a1167490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a11674c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a1167500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a1167530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a1167590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a1167610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a11676b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a1167700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a1167730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a11677a0 vget() at vget+0x98/frame 0xfffffe00a11677e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a1167910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a1167950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a11679a0 trap() at trap+0x3be/frame 0xfffffe00a1167ab0 calltrap() at calltrap+0x8/frame 0xfffffe00a1167ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23286 tid 100415 td 0xfffff80241a32560 sched_switch() at sched_switch+0x942/frame 0xfffffe00a116c490 mi_switch() at mi_switch+0x18c/frame 0xfffffe00a116c4c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00a116c500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00a116c530 sleeplk() at sleeplk+0x149/frame 0xfffffe00a116c590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe00a116c610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe00a116c6b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe00a116c700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe00a116c730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe00a116c7a0 vget() at vget+0x98/frame 0xfffffe00a116c7e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe00a116c910 vm_fault() at vm_fault+0x75/frame 0xfffffe00a116c950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00a116c9a0 trap() at trap+0x3be/frame 0xfffffe00a116cab0 calltrap() at calltrap+0x8/frame 0xfffffe00a116cab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23285 tid 100142 td 0xfffff8000419d000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee13490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee134c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee13500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee13530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee13590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee13610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee136b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee13700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee13730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee137a0 vget() at vget+0x98/frame 0xfffffe009ee137e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee13910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee13950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee139a0 trap() at trap+0x3be/frame 0xfffffe009ee13ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee13ab0 --- trap 0xc, rip = 0x8006dc000, rsp = 0x7fffffffdfa8, rbp = 0x7fffffffe440 --- Tracing command swap pid 23284 tid 100221 td 0xfffff8001bad2000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 23280 tid 100496 td 0xfffff80756572560 sched_switch() at sched_switch+0x942/frame 0xfffffe00af0e5540 mi_switch() at mi_switch+0x18c/frame 0xfffffe00af0e5570 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00af0e55b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00af0e55e0 _sleep() at _sleep+0x367/frame 0xfffffe00af0e5690 vm_wait_doms() at vm_wait_doms+0x85/frame 0xfffffe00af0e56b0 _pmap_allocpte() at _pmap_allocpte+0x1e5/frame 0xfffffe00af0e5720 pmap_enter() at pmap_enter+0x532/frame 0xfffffe00af0e57e0 vm_fault_hold() at vm_fault_hold+0x1cd4/frame 0xfffffe00af0e5910 vm_fault() at vm_fault+0x75/frame 0xfffffe00af0e5950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe00af0e59a0 trap() at trap+0x3be/frame 0xfffffe00af0e5ab0 calltrap() at calltrap+0x8/frame 0xfffffe00af0e5ab0 --- trap 0xc, rip = 0x8007b59ca, rsp = 0x7fffffffe5a8, rbp = 0x7fffffffe5f0 --- Tracing command fdgrowtable pid 22352 tid 100413 td 0xfffff80241af6560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 22308 tid 100403 td 0xfffff802416b4560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 22297 tid 100494 td 0xfffff806f18a6560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 22295 tid 100289 td 0xfffff801f38fc560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 1102 tid 100218 td 0xfffff8016ea42000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command csh pid 1100 tid 100163 td 0xfffff8001be2d560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 1099 tid 100184 td 0xfffff8001bcbb000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 1098 tid 100214 td 0xfffff8016e006000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 1097 tid 100168 td 0xfffff8001ba7d000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee95490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee954c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee95500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ee95530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ee95590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ee95610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ee956b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ee95700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ee95730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ee957a0 vget() at vget+0x98/frame 0xfffffe009ee957e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ee95910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ee95950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ee959a0 trap() at trap+0x3be/frame 0xfffffe009ee95ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ee95ab0 --- trap 0xc, rip = 0x800728ab0, rsp = 0x7fffffffa3a8, rbp = 0x7fffffffa440 --- Tracing command sh pid 784 tid 100145 td 0xfffff8001ba6d000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 781 tid 100202 td 0xfffff8016e007560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command csh pid 779 tid 100153 td 0xfffff80003c5c000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 778 tid 100178 td 0xfffff8006a36a560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 777 tid 100170 td 0xfffff8001bdb1000 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 773 tid 100191 td 0xfffff8006aaed560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ef08490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ef084c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ef08500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009ef08530 sleeplk() at sleeplk+0x149/frame 0xfffffe009ef08590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009ef08610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009ef086b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009ef08700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009ef08730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009ef087a0 vget() at vget+0x98/frame 0xfffffe009ef087e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009ef08910 vm_fault() at vm_fault+0x75/frame 0xfffffe009ef08950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009ef089a0 trap() at trap+0x3be/frame 0xfffffe009ef08ab0 calltrap() at calltrap+0x8/frame 0xfffffe009ef08ab0 --- trap 0xc, rip = 0x8008337c0, rsp = 0x7fffffffe4c8, rbp = 0x7fffffffe500 --- Tracing command login pid 769 tid 100148 td 0xfffff8001bdb2560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cron pid 726 tid 100180 td 0xfffff8000419b560 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0078959f10 kdb_reenter() at kdb_reenter+0x2f/frame 0xfffffe0078959f20 trap() at trap+0x5c/frame 0xfffffe007895a030 calltrap() at calltrap+0x8/frame 0xfffffe007895a030 --- trap 0xc, rip = 0xffffffff80f46396, rsp = 0xfffffe007895a100, rbp = 0xfffffe007895a110 --- db_trace_thread() at db_trace_thread+0x26/frame 0xfffffe007895a110 _db_stack_trace_all() at _db_stack_trace_all+0xf4/frame 0xfffffe007895a1b0 db_command() at db_command+0x279/frame 0xfffffe007895a280 db_script_exec() at db_script_exec+0x188/frame 0xfffffe007895a320 db_command() at db_command+0x279/frame 0xfffffe007895a3f0 db_command_loop() at db_command_loop+0x64/frame 0xfffffe007895a400 db_trap() at db_trap+0xef/frame 0xfffffe007895a490 kdb_trap() at kdb_trap+0x1c3/frame 0xfffffe007895a500 trap() at trap+0x328/frame 0xfffffe007895a610 calltrap() at calltrap+0x8/frame 0xfffffe007895a610 --- trap 0x3, rip = 0xffffffff80af4366, rsp = 0xfffffe007895a6e0, rbp = 0xfffffe007895a700 --- kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 722 tid 100143 td 0xfffff8001bd02000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee185b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee185e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee18620 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe009ee18680 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe009ee186b0 _cv_wait_sig() at _cv_wait_sig+0x1fe/frame 0xfffffe009ee18710 seltdwait() at seltdwait+0x8d/frame 0xfffffe009ee18750 kern_select() at kern_select+0x886/frame 0xfffffe009ee18940 sys_select() at sys_select+0x56/frame 0xfffffe009ee18980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe009ee18ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009ee18ab0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800874cda, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dhclient pid 720 tid 100172 td 0xfffff8001bff2560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eea9490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eea94c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eea9500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eea9530 sleeplk() at sleeplk+0x149/frame 0xfffffe009eea9590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eea9610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eea96b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eea9700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eea9730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eea97a0 vget() at vget+0x98/frame 0xfffffe009eea97e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009eea9910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eea9950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eea99a0 trap() at trap+0x3be/frame 0xfffffe009eea9ab0 calltrap() at calltrap+0x8/frame 0xfffffe009eea9ab0 --- trap 0xc, rip = 0x8003be2bc, rsp = 0x7fffffffebe0, rbp = 0x7fffffffec00 --- Tracing command casperd pid 569 tid 100175 td 0xfffff8006a36b560 sched_switch() at sched_switch+0x942/frame 0xfffffe009eeb8490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eeb84c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eeb8500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eeb8530 sleeplk() at sleeplk+0x149/frame 0xfffffe009eeb8590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eeb8610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eeb86b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eeb8700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eeb8730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eeb87a0 vget() at vget+0x98/frame 0xfffffe009eeb87e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009eeb8910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eeb8950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eeb89a0 trap() at trap+0x3be/frame 0xfffffe009eeb8ab0 calltrap() at calltrap+0x8/frame 0xfffffe009eeb8ab0 --- trap 0xc, rip = 0x800704000, rsp = 0x7fffffffe1a8, rbp = 0x7fffffffe640 --- Tracing command rpcbind pid 528 tid 100171 td 0xfffff8001bd00000 sched_switch() at sched_switch+0x942/frame 0xfffffe009eea4490 mi_switch() at mi_switch+0x18c/frame 0xfffffe009eea44c0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009eea4500 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe009eea4530 sleeplk() at sleeplk+0x149/frame 0xfffffe009eea4590 lockmgr_slock_hard() at lockmgr_slock_hard+0x2d4/frame 0xfffffe009eea4610 __lockmgr_args() at __lockmgr_args+0x547/frame 0xfffffe009eea46b0 ffs_lock() at ffs_lock+0xaf/frame 0xfffffe009eea4700 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xe1/frame 0xfffffe009eea4730 _vn_lock() at _vn_lock+0x66/frame 0xfffffe009eea47a0 vget() at vget+0x98/frame 0xfffffe009eea47e0 vm_fault_hold() at vm_fault_hold+0xaf8/frame 0xfffffe009eea4910 vm_fault() at vm_fault+0x75/frame 0xfffffe009eea4950 trap_pfault() at trap_pfault+0x171/frame 0xfffffe009eea49a0 trap() at trap+0x3be/frame 0xfffffe009eea4ab0 calltrap() at calltrap+0x8/frame 0xfffffe009eea4ab0 --- trap 0xc, rip = 0x8003a4780, rsp = 0x7fffffffca58, rbp = 0x7fffffffec70 --- Tracing command vnlru pid 33 tid 100133 td 0xfffff8001ba68000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005e6870 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005e68a0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005e68e0 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005e6920 _sleep() at _sleep+0x307/frame 0xfffffe00005e69d0 vnlru_proc() at vnlru_proc+0xa4/frame 0xfffffe00005e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 32 tid 100130 td 0xfffff8001ba68560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005d28c0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005d28f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005d2930 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005d2970 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x221/frame 0xfffffe00005d29e0 sched_sync() at sched_sync+0x609/frame 0xfffffe00005d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100128 td 0xfffff8001ba6a000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005cd8c0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005cd8f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005cd930 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005cd970 _sleep() at _sleep+0x307/frame 0xfffffe00005cda20 buf_daemon() at buf_daemon+0xe8/frame 0xfffffe00005cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100131 td 0xfffff8001b2a4000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005d78b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005d78e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005d7920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005d7960 _sleep() at _sleep+0x307/frame 0xfffffe00005d7a10 bufspace_daemon() at bufspace_daemon+0x20e/frame 0xfffffe00005d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100132 td 0xfffff8001b293560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005e18b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005e18e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005e1920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005e1960 _sleep() at _sleep+0x307/frame 0xfffffe00005e1a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100134 td 0xfffff8001b293000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005eb8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005eb8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005eb920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005eb960 _sleep() at _sleep+0x307/frame 0xfffffe00005eba10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100136 td 0xfffff8001ba80000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005f58b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005f58e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005f5920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005f5960 _sleep() at _sleep+0x307/frame 0xfffffe00005f5a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100137 td 0xfffff8001ba7f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005fa8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005fa8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005fa920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005fa960 _sleep() at _sleep+0x307/frame 0xfffffe00005faa10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100138 td 0xfffff8001ba7f000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005ff8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005ff8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005ff920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005ff960 _sleep() at _sleep+0x307/frame 0xfffffe00005ffa10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe00005ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100139 td 0xfffff8001ba7e560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee048b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee048e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee04920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee04960 _sleep() at _sleep+0x307/frame 0xfffffe009ee04a10 bufspace_daemon() at bufspace_daemon+0x2d6/frame 0xfffffe009ee04a70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee04ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee04ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100150 td 0xfffff8001bad3000 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee3b8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee3b910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee3b950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee3b990 _sleep() at _sleep+0x307/frame 0xfffffe009ee3ba40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe009ee3ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee3bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee3bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100155 td 0xfffff8001bd01560 sched_switch() at sched_switch+0x942/frame 0xfffffe009ee548e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe009ee54910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe009ee54950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe009ee54990 _sleep() at _sleep+0x307/frame 0xfffffe009ee54a40 softdep_flush() at softdep_flush+0x17f/frame 0xfffffe009ee54a70 fork_exit() at fork_exit+0x84/frame 0xfffffe009ee54ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009ee54ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 30 tid 100127 td 0xfffff8001b2a7560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005c88a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005c88d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005c8910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005c8940 _sleep() at _sleep+0x367/frame 0xfffffe00005c89f0 vm_daemon() at vm_daemon+0x85/frame 0xfffffe00005c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100126 td 0xfffff8001b38e000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005c37b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005c37e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005c3820 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005c3860 _sleep() at _sleep+0x307/frame 0xfffffe00005c3910 pause_sbt() at pause_sbt+0x144/frame 0xfffffe00005c3960 vm_pageout_worker() at vm_pageout_worker+0x242/frame 0xfffffe00005c3a50 vm_pageout() at vm_pageout+0x133/frame 0xfffffe00005c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100129 td 0xfffff8001ba6e560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005dbd90 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005dbdc0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005dbe00 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005dbe30 _sleep() at _sleep+0x367/frame 0xfffffe00005dbee0 vm_page_busy_sleep() at vm_page_busy_sleep+0xc5/frame 0xfffffe00005dbf10 vm_page_grab_pages() at vm_page_grab_pages+0x3d4/frame 0xfffffe00005dbfb0 allocbuf() at allocbuf+0x33a/frame 0xfffffe00005dc020 getblk() at getblk+0x819/frame 0xfffffe00005dc0b0 cluster_read() at cluster_read+0xb7/frame 0xfffffe00005dc190 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1ade/frame 0xfffffe00005dc340 ffs_write() at ffs_write+0x20f/frame 0xfffffe00005dc3e0 VOP_WRITE_APV() at VOP_WRITE_APV+0x17a/frame 0xfffffe00005dc4f0 vnode_pager_generic_putpages() at vnode_pager_generic_putpages+0x581/frame 0xfffffe00005dc5e0 VOP_PUTPAGES_APV() at VOP_PUTPAGES_APV+0xfd/frame 0xfffffe00005dc610 vnode_pager_putpages() at vnode_pager_putpages+0x7d/frame 0xfffffe00005dc680 vm_pageout_flush() at vm_pageout_flush+0x144/frame 0xfffffe00005dc710 vm_pageout_cluster() at vm_pageout_cluster+0x3fa/frame 0xfffffe00005dc990 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0x797/frame 0xfffffe00005dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100135 td 0xfffff8001ba6e000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005f08e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005f0910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005f0950 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005f0990 _sleep() at _sleep+0x307/frame 0xfffffe00005f0a40 uma_reclaim_worker() at uma_reclaim_worker+0xda/frame 0xfffffe00005f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 28 tid 100125 td 0xfffff8001b2a4560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005be8a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005be8d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005be910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005be940 _sleep() at _sleep+0x367/frame 0xfffffe00005be9f0 enc_daemon() at enc_daemon+0x1ca/frame 0xfffffe00005bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 27 tid 100124 td 0xfffff8001b2a5000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005b98a0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005b98d0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005b9910 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005b9940 _sleep() at _sleep+0x367/frame 0xfffffe00005b99f0 enc_daemon() at enc_daemon+0x1ca/frame 0xfffffe00005b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 26 tid 100121 td 0xfffff8001b396560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005aa8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005aa8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005aa920 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe00005aa960 _sleep() at _sleep+0x307/frame 0xfffffe00005aaa10 random_kthread() at random_kthread+0x57/frame 0xfffffe00005aaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005aaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005aaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sctp_iterator pid 25 tid 100117 td 0xfffff8001b398560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000596900 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000596930 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000596970 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005969a0 _sleep() at _sleep+0x367/frame 0xfffffe0000596a50 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe0000596a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000596ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000596ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 24 tid 100116 td 0xfffff8001b2a5560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005908d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000590900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000590940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000590980 _sleep() at _sleep+0x307/frame 0xfffffe0000590a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000590a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000590ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000590ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 23 tid 100115 td 0xfffff8001b2a6000 sched_switch() at sched_switch+0x942/frame 0xfffffe000058b8d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000058b900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000058b940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe000058b980 _sleep() at _sleep+0x307/frame 0xfffffe000058ba30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe000058ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000058bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000058bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 22 tid 100114 td 0xfffff8001b2a6560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005868d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000586900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000586940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000586980 _sleep() at _sleep+0x307/frame 0xfffffe0000586a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000586a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000586ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000586ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 21 tid 100113 td 0xfffff8001b2a7000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005818d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000581900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000581940 sleepq_timedwait() at sleepq_timedwait+0x50/frame 0xfffffe0000581980 _sleep() at _sleep+0x307/frame 0xfffffe0000581a30 soaio_kproc_loop() at soaio_kproc_loop+0x196/frame 0xfffffe0000581a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000581ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000581ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100094 td 0xfffff8000401a560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000522930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000522960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005229a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005229d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000522a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000522a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000522ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000522ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100095 td 0xfffff8000401a000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000527930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000527960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005279a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005279d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000527a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000527a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000527ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000527ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100096 td 0xfffff80004019560 sched_switch() at sched_switch+0x942/frame 0xfffffe000052c930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000052c960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000052c9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000052c9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000052ca30 usb_process() at usb_process+0x12b/frame 0xfffffe000052ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000052cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000052cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100097 td 0xfffff80004019000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000531930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000531960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005319a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005319d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000531a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000531a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000531ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000531ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100098 td 0xfffff80004011560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000536930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000536960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005369a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005369d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000536a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000536a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000536ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000536ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100100 td 0xfffff80003c5d560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000540930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000540960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005409a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005409d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000540a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000540a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000540ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000540ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100101 td 0xfffff8000419f000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000545930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000545960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005459a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005459d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000545a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000545a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000545ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000545ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100102 td 0xfffff8000419e560 sched_switch() at sched_switch+0x942/frame 0xfffffe000054a930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000054a960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000054a9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000054a9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000054aa30 usb_process() at usb_process+0x12b/frame 0xfffffe000054aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000054aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000054aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100103 td 0xfffff8000419e000 sched_switch() at sched_switch+0x942/frame 0xfffffe000054f930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000054f960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000054f9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000054f9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000054fa30 usb_process() at usb_process+0x12b/frame 0xfffffe000054fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000054fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000054fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100104 td 0xfffff8000419d560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000554930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000554960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005549a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005549d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000554a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000554a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000554ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000554ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100105 td 0xfffff8000c089560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000559930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000559960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005599a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005599d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000559a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000559a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000559ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000559ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100106 td 0xfffff8000c089000 sched_switch() at sched_switch+0x942/frame 0xfffffe000055e930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000055e960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000055e9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000055e9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000055ea30 usb_process() at usb_process+0x12b/frame 0xfffffe000055ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000055eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000055eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100107 td 0xfffff8000c088560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000563930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000563960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005639a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005639d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000563a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000563a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000563ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000563ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100108 td 0xfffff8000c088000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000568930 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000568960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005689a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005689d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0000568a30 usb_process() at usb_process+0x12b/frame 0xfffffe0000568a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000568ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000568ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 20 tid 100109 td 0xfffff8000c087560 sched_switch() at sched_switch+0x942/frame 0xfffffe000056d930 mi_switch() at mi_switch+0x18c/frame 0xfffffe000056d960 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000056d9a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000056d9d0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe000056da30 usb_process() at usb_process+0x12b/frame 0xfffffe000056da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000056dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000056dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100089 td 0xfffff80003ad8000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005098d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000509900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000509940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000509970 _sleep() at _sleep+0x367/frame 0xfffffe0000509a20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0000509a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000509ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000509ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100090 td 0xfffff80003aca560 sched_switch() at sched_switch+0x942/frame 0xfffffe000050e8d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000050e900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000050e940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000050e970 _sleep() at _sleep+0x367/frame 0xfffffe000050ea20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe000050ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000050eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000050eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100091 td 0xfffff80003c5d000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005138d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000513900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000513940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000513970 _sleep() at _sleep+0x367/frame 0xfffffe0000513a20 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe0000513a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000513ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000513ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 19 tid 100123 td 0xfffff8001b38e560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005b48c0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005b48f0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005b4930 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005b4960 _sleep() at _sleep+0x367/frame 0xfffffe00005b4a10 xpt_scanner_thread() at xpt_scanner_thread+0x93/frame 0xfffffe00005b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 11 pid 18 tid 100088 td 0xfffff80003ad8560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005048b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005048e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000504920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000504950 _sleep() at _sleep+0x367/frame 0xfffffe0000504a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe0000504a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000504ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000504ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 10 pid 17 tid 100087 td 0xfffff80003ad9000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004ff8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004ff8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004ff920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004ff950 _sleep() at _sleep+0x367/frame 0xfffffe00004ffa00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 9 pid 16 tid 100086 td 0xfffff80003ad9560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004fa8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004fa8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004fa920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004fa950 _sleep() at _sleep+0x367/frame 0xfffffe00004faa00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 8 pid 15 tid 100085 td 0xfffff80003ada000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004f58b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004f58e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004f5920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004f5950 _sleep() at _sleep+0x367/frame 0xfffffe00004f5a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 7 pid 14 tid 100084 td 0xfffff80003ada560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004f08b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004f08e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004f0920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004f0950 _sleep() at _sleep+0x367/frame 0xfffffe00004f0a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 6 pid 9 tid 100083 td 0xfffff80003adb000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004eb8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004eb8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004eb920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004eb950 _sleep() at _sleep+0x367/frame 0xfffffe00004eba00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 5 pid 8 tid 100082 td 0xfffff80003adb560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004e68b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004e68e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004e6920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004e6950 _sleep() at _sleep+0x367/frame 0xfffffe00004e6a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 4 pid 7 tid 100081 td 0xfffff80003ac6000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004e18b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004e18e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004e1920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004e1950 _sleep() at _sleep+0x367/frame 0xfffffe00004e1a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 3 pid 6 tid 100080 td 0xfffff80003ac6560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004dc8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004dc8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004dc920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004dc950 _sleep() at _sleep+0x367/frame 0xfffffe00004dca00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 2 pid 5 tid 100079 td 0xfffff80003ac7000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004d78b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004d78e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004d7920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004d7950 _sleep() at _sleep+0x367/frame 0xfffffe00004d7a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 1 pid 4 tid 100078 td 0xfffff80003ac7560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004d28b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004d28e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004d2920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004d2950 _sleep() at _sleep+0x367/frame 0xfffffe00004d2a00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 0 pid 3 tid 100077 td 0xfffff80003ac8000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004cd8b0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004cd8e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004cd920 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004cd950 _sleep() at _sleep+0x367/frame 0xfffffe00004cda00 crypto_ret_proc() at crypto_ret_proc+0x238/frame 0xfffffe00004cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100076 td 0xfffff80003ac8560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004c88e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004c8910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004c8950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004c8980 _sleep() at _sleep+0x367/frame 0xfffffe00004c8a30 crypto_proc() at crypto_proc+0x2ba/frame 0xfffffe00004c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100060 td 0xfffff80003a2d560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004788e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000478910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000478950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000478980 _sleep() at _sleep+0x367/frame 0xfffffe0000478a30 g_run_events() at g_run_events+0x4f/frame 0xfffffe0000478a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000478ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000478ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100061 td 0xfffff80003a2d000 sched_switch() at sched_switch+0x942/frame 0xfffffe000047d8e0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000047d910 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000047d950 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000047d980 _sleep() at _sleep+0x367/frame 0xfffffe000047da30 g_io_schedule_up() at g_io_schedule_up+0xa5/frame 0xfffffe000047da60 g_up_procbody() at g_up_procbody+0x6d/frame 0xfffffe000047da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000047dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000047dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100062 td 0xfffff80003a2c560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004828d0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000482900 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000482940 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000482970 _sleep() at _sleep+0x367/frame 0xfffffe0000482a20 g_io_schedule_down() at g_io_schedule_down+0x14c/frame 0xfffffe0000482a60 g_down_procbody() at g_down_procbody+0x6d/frame 0xfffffe0000482a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000482ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000482ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100015 td 0xfffff80003947000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789969f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078996a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0078996a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078996ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078996ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100016 td 0xfffff80003946560 sched_switch() at sched_switch+0x942/frame 0xfffffe007899b9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe007899ba20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe007899ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe007899bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007899bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xfffff80003939560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100046 td 0xfffff80003a24560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004329f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000432a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000432a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000432ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000432ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100047 td 0xfffff80003a24000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100048 td 0xfffff80003a23560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100049 td 0xfffff80003a23000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100050 td 0xfffff80003a22560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xfffff80003a22000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xfffff80003a16560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100053 td 0xfffff80003a16000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xfffff80003a00560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100055 td 0xfffff80003a30000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100056 td 0xfffff80003a2f560 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100057 td 0xfffff80003a2f000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100058 td 0xfffff80003a2e560 sched_switch() at sched_switch+0x942/frame 0xfffffe000046e9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000046ea20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe000046ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000046eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000046eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffff80003a2e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100092 td 0xfffff8000401b560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005189f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000518a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000518a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000518ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000518ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100093 td 0xfffff8000401b000 sched_switch() at sched_switch+0x942/frame 0xfffffe000051d9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000051da20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe000051da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000051dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000051dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100099 td 0xfffff80004011000 sched_switch() at sched_switch+0x942/frame 0xfffffe000053b9f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000053ba20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe000053ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000053bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000053bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100110 td 0xfffff8000c087000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100111 td 0xfffff8000c07f560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005779f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000577a20 ithread_loop() at ithread_loop+0x1f1/frame 0xfffffe0000577a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000577ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000577ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff80003938000 (CPU 0) kdb_alt_break_internal() at kdb_alt_break_internal+0x106/frame 0xfffffe007895a700 kdb_alt_break() at kdb_alt_break+0xb/frame 0xfffffe007895a710 uart_intr_rxready() at uart_intr_rxready+0x9a/frame 0xfffffe007895a740 uart_intr() at uart_intr+0x138/frame 0xfffffe007895a780 intr_event_handle() at intr_event_handle+0xbc/frame 0xfffffe007895a7d0 intr_execute_handlers() at intr_execute_handlers+0x58/frame 0xfffffe007895a800 lapic_handle_intr() at lapic_handle_intr+0x6d/frame 0xfffffe007895a840 apic_isr1_u() at apic_isr1_u+0xc1/frame 0xfffffe007895a910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895a960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895a980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895a9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff80003937560 (CPU 1) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00007ffdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00007ffe10 trap() at trap+0x49/frame 0xfffffe00007fff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00007fff20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe007895f910, rbp = 0xfffffe007895f910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007895f910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007895f960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007895f980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007895f9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007895fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe007895fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007895fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff80003937000 (CPU 2) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071b2df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071b2e10 trap() at trap+0x49/frame 0xfffffe00071b2f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071b2f20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078964910, rbp = 0xfffffe0078964910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078964910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078964960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078964980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789649a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078964a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078964ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078964ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff80003936560 (CPU 3) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071bbdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071bbe10 trap() at trap+0x49/frame 0xfffffe00071bbf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071bbf20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078969910, rbp = 0xfffffe0078969910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078969910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078969960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078969980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789699a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078969a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078969ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078969ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff80003936000 (CPU 4) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071c4df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071c4e10 trap() at trap+0x49/frame 0xfffffe00071c4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071c4f20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe007896e910, rbp = 0xfffffe007896e910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007896e910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007896e960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007896e980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007896e9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007896ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe007896eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007896eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff80003935560 (CPU 5) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071cddf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071cde10 trap() at trap+0x49/frame 0xfffffe00071cdf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071cdf20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078973910, rbp = 0xfffffe0078973910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078973910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078973960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078973980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789739a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078973a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078973ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078973ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff80003935000 (CPU 6) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071d6df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071d6e10 trap() at trap+0x49/frame 0xfffffe00071d6f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071d6f20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078978910, rbp = 0xfffffe0078978910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078978910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078978960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078978980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789789a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078978a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078978ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078978ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff80003949560 (CPU 7) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071dfdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071dfe10 trap() at trap+0x49/frame 0xfffffe00071dff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071dff20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe007897d910, rbp = 0xfffffe007897d910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007897d910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007897d960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007897d980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007897d9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007897da70 fork_exit() at fork_exit+0x84/frame 0xfffffe007897dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007897dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff80003949000 (CPU 8) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071e8df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071e8e10 trap() at trap+0x49/frame 0xfffffe00071e8f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071e8f20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078982910, rbp = 0xfffffe0078982910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078982910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078982960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078982980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789829a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078982a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078982ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078982ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff80003948560 (CPU 9) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071f1df0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071f1e10 trap() at trap+0x49/frame 0xfffffe00071f1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071f1f20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078987910, rbp = 0xfffffe0078987910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078987910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078987960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078987980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789879a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078987a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078987ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078987ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff80003948000 (CPU 10) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071fadf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071fae10 trap() at trap+0x49/frame 0xfffffe00071faf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071faf20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe007898c910, rbp = 0xfffffe007898c910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe007898c910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe007898c960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe007898c980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe007898c9a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe007898ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe007898cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe007898cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff80003947560 (CPU 11) cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00071ffdf0 ipi_nmi_handler() at ipi_nmi_handler+0x44/frame 0xfffffe00071ffe10 trap() at trap+0x49/frame 0xfffffe00071fff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00071fff20 --- trap 0x13, rip = 0xffffffff810c7d76, rsp = 0xfffffe0078991910, rbp = 0xfffffe0078991910 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe0078991910 acpi_cpu_idle() at acpi_cpu_idle+0x2ee/frame 0xfffffe0078991960 cpu_idle_acpi() at cpu_idle_acpi+0x3f/frame 0xfffffe0078991980 cpu_idle() at cpu_idle+0x8f/frame 0xfffffe00789919a0 sched_idletd() at sched_idletd+0x517/frame 0xfffffe0078991a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078991ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078991ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff80003938560 sched_switch() at sched_switch+0x942/frame 0xfffffe0078955550 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078955580 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789555c0 sleepq_catch_signals() at sleepq_catch_signals+0x300/frame 0xfffffe0078955620 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0078955650 _sleep() at _sleep+0x359/frame 0xfffffe0078955700 kern_wait6() at kern_wait6+0x4c4/frame 0xfffffe0078955790 sys_wait4() at sys_wait4+0x78/frame 0xfffffe0078955980 amd64_syscall() at amd64_syscall+0x7ab/frame 0xfffffe0078955ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0078955ab0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2a71ca, rsp = 0x7fffffffe788, rbp = 0x7fffffffe7f0 --- Tracing command audit pid 10 tid 100001 td 0xfffff80003939000 sched_switch() at sched_switch+0x942/frame 0xfffffe0078950910 mi_switch() at mi_switch+0x18c/frame 0xfffffe0078950940 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0078950980 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789509b0 _cv_wait() at _cv_wait+0x203/frame 0xfffffe0078950a10 audit_worker() at audit_worker+0x93/frame 0xfffffe0078950a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0078950ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0078950ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81f400a0 sched_switch() at sched_switch+0x942/frame 0xffffffff824f49b0 mi_switch() at mi_switch+0x18c/frame 0xffffffff824f49e0 sleepq_switch() at sleepq_switch+0x10d/frame 0xffffffff824f4a20 sleepq_wait() at sleepq_wait+0x43/frame 0xffffffff824f4a50 _sleep() at _sleep+0x367/frame 0xffffffff824f4b00 vm_wait_min() at vm_wait_min+0x64/frame 0xffffffff824f4b20 swapper() at swapper+0x25/frame 0xffffffff824f4b70 btext() at btext+0x2c Tracing command kernel pid 0 tid 100017 td 0xfffff80003946000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789a08f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789a0920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789a0960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789a0990 _sleep() at _sleep+0x367/frame 0xfffffe00789a0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00789a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffff80003976000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789aa8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789aa920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789aa960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789aa990 _sleep() at _sleep+0x367/frame 0xfffffe00789aaa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00789aaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789aaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789aaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffff80003975560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789af990 _sleep() at _sleep+0x367/frame 0xfffffe00789afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00789afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffff80003975000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789b4920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789b4950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789b4990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789b49c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789b4a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffff80003974560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789b9920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789b9950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789b9990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789b99c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789b9a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffff80003974000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789be920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789be950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789be990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789be9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789bea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffff80003973560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789c3920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789c3950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789c3990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789c39c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789c3a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffff80003973000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789c8920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789c8950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789c8990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789c89c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789c8a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789c8a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789c8ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789c8ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffff80003972560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789cd920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789cd950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789cd990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789cd9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789cda40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789cda70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789cdab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789cdab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffff80003972000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789d2920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789d2950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789d2990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789d29c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789d2a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789d2a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffff8000398a560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789d7920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789d7950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789d7990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789d79c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789d7a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789d7a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789d7ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789d7ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffff8000398a000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789dc920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789dc950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789dc990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789dc9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789dca40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789dca70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789dcab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789dcab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffff80003985560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789e1920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789e1950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789e1990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789e19c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789e1a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789e1a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789e1ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789e1ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffff80003985000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789e6920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789e6950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789e6990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789e69c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789e6a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789e6a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789e6ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789e6ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffff80003984560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789eb920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789eb950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789eb990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789eb9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789eba40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789eba70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789ebab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789ebab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffff80003984000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789f0920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789f0950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789f0990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789f09c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789f0a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789f0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789f0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789f0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffff80003983560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789f5920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789f5950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789f5990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789f59c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789f5a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789f5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789f5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789f5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffff80003983000 sched_switch() at sched_switch+0x942/frame 0xfffffe00789fa920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789fa950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789fa990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789fa9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789faa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789faa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789faab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789faab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffff80003976560 sched_switch() at sched_switch+0x942/frame 0xfffffe00789ff920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00789ff950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00789ff990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00789ff9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00789ffa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe00789ffa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00789ffab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00789ffab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffff80003a00000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000405920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000405950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000405990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004059c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000405a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000405a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000405ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000405ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffff800039ff560 sched_switch() at sched_switch+0x942/frame 0xfffffe000040a920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000040a950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000040a990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000040a9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000040aa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000040aa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000040aab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000040aab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffff800039ff000 sched_switch() at sched_switch+0x942/frame 0xfffffe000040f920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000040f950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000040f990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000040f9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000040fa40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000040fa70 fork_exit() at fork_exit+0x84/frame 0xfffffe000040fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000040fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffff800039fe560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000414920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000414950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000414990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004149c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000414a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000414a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000414ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000414ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffff800039fe000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000419920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000419950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000419990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004199c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000419a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000419a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000419ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000419ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffff800039fd560 sched_switch() at sched_switch+0x942/frame 0xfffffe000041e920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000041e950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000041e990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000041e9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000041ea40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000041ea70 fork_exit() at fork_exit+0x84/frame 0xfffffe000041eab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000041eab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffff800039fd000 sched_switch() at sched_switch+0x942/frame 0xfffffe0000423920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000423950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000423990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004239c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000423a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000423a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000423ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000423ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffff800039fc560 sched_switch() at sched_switch+0x942/frame 0xfffffe0000428920 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000428950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000428990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004289c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe0000428a40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe0000428a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000428ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000428ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffff800039fc000 sched_switch() at sched_switch+0x942/frame 0xfffffe000042d920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000042d950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000042d990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000042d9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000042da40 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xb1/frame 0xfffffe000042da70 fork_exit() at fork_exit+0x84/frame 0xfffffe000042dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000042dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffff80003a2c000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004878f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000487920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000487960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000487990 _sleep() at _sleep+0x367/frame 0xfffffe0000487a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000487a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000487ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000487ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffff80003ac5560 sched_switch() at sched_switch+0x942/frame 0xfffffe000048c8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000048c920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000048c960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000048c990 _sleep() at _sleep+0x367/frame 0xfffffe000048ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe000048ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000048cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000048cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffff80003ac5000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004918f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000491920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000491960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000491990 _sleep() at _sleep+0x367/frame 0xfffffe0000491a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000491a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000491ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000491ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffff80003ac4560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004968f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe0000496920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe0000496960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0000496990 _sleep() at _sleep+0x367/frame 0xfffffe0000496a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe0000496a70 fork_exit() at fork_exit+0x84/frame 0xfffffe0000496ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000496ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffff80003ac4000 sched_switch() at sched_switch+0x942/frame 0xfffffe000049b8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe000049b920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000049b960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000049b990 _sleep() at _sleep+0x367/frame 0xfffffe000049ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe000049ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000049bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000049bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffff80003ac3560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004a08f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004a0920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004a0960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004a0990 _sleep() at _sleep+0x367/frame 0xfffffe00004a0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffff80003ac3000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004a58f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004a5920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004a5960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004a5990 _sleep() at _sleep+0x367/frame 0xfffffe00004a5a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004a5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004a5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004a5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffff80003ac2560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004aa8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004aa920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004aa960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004aa990 _sleep() at _sleep+0x367/frame 0xfffffe00004aaa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004aaa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004aaab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004aaab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffff80003ac2000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004af990 _sleep() at _sleep+0x367/frame 0xfffffe00004afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffff80003a30560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004b48f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004b4920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004b4960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004b4990 _sleep() at _sleep+0x367/frame 0xfffffe00004b4a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004b4a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004b4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004b4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffff80003aca000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004b98f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004b9920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004b9960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004b9990 _sleep() at _sleep+0x367/frame 0xfffffe00004b9a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004b9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004b9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004b9ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffff80003ac9560 sched_switch() at sched_switch+0x942/frame 0xfffffe00004be8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004be920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004be960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004be990 _sleep() at _sleep+0x367/frame 0xfffffe00004bea40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004bea70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004beab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004beab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffff80003ac9000 sched_switch() at sched_switch+0x942/frame 0xfffffe00004c38f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00004c3920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00004c3960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00004c3990 _sleep() at _sleep+0x367/frame 0xfffffe00004c3a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00004c3a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00004c3ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004c3ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xfffff8000c07f000 sched_switch() at sched_switch+0x942/frame 0xfffffe000057c920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000057c950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000057c990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000057c9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000057ca40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000057ca70 fork_exit() at fork_exit+0x84/frame 0xfffffe000057cab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000057cab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100118 td 0xfffff8001b398000 sched_switch() at sched_switch+0x942/frame 0xfffffe000059b920 mi_switch() at mi_switch+0x18c/frame 0xfffffe000059b950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe000059b990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe000059b9c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe000059ba40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe000059ba70 fork_exit() at fork_exit+0x84/frame 0xfffffe000059bab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000059bab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100119 td 0xfffff8001b397560 sched_switch() at sched_switch+0x942/frame 0xfffffe00005a0920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005a0950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005a0990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005a09c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00005a0a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00005a0a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005a0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a0ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100120 td 0xfffff8001b397000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005a5920 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005a5950 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005a5990 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005a59c0 msleep_spin_sbt() at msleep_spin_sbt+0x1eb/frame 0xfffffe00005a5a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00005a5a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005a5ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a5ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100122 td 0xfffff8001b396000 sched_switch() at sched_switch+0x942/frame 0xfffffe00005af8f0 mi_switch() at mi_switch+0x18c/frame 0xfffffe00005af920 sleepq_switch() at sleepq_switch+0x10d/frame 0xfffffe00005af960 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00005af990 _sleep() at _sleep+0x367/frame 0xfffffe00005afa40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe1/frame 0xfffffe00005afa70 fork_exit() at fork_exit+0x84/frame 0xfffffe00005afab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005afab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:pho> dump Dumping 1298 out of 32625 MB:..2%..12%..21%..31%..41%..51%..61%..71%..81%..92% Dump complete db> reset Index: /usr/src/sys/kern/vfs_bio.c =================================================================== --- /usr/src/sys/kern/vfs_bio.c (revision 330873) +++ /usr/src/sys/kern/vfs_bio.c (working copy) @@ -50,6 +50,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -100,6 +101,52 @@ struct buf_ops buf_ops_bio = { .bop_bdflush = bufbdflush, }; +struct bufqueue { + struct mtx_padalign bq_lock; + TAILQ_HEAD(, buf) bq_queue; + uint8_t bq_index; + uint16_t bq_subqueue; + int bq_len; +} __aligned(CACHE_LINE_SIZE); + +#define BQ_LOCKPTR(bq) (&(bq)->bq_lock) +#define BQ_LOCK(bq) mtx_lock(BQ_LOCKPTR((bq))) +#define BQ_UNLOCK(bq) mtx_unlock(BQ_LOCKPTR((bq))) +#define BQ_ASSERT_LOCKED(bq) mtx_assert(BQ_LOCKPTR((bq)), MA_OWNED) + +struct bufdomain { + struct bufqueue bd_subq[MAXCPU + 1]; /* Per-cpu sub queues + global */ + struct bufqueue bd_dirtyq; + struct bufqueue *bd_cleanq; + struct mtx_padalign bd_run_lock; + /* Constants */ + long bd_maxbufspace; + long bd_hibufspace; + long bd_lobufspace; + long bd_bufspacethresh; + int bd_hifreebuffers; + int bd_lofreebuffers; + int bd_hidirtybuffers; + int bd_lodirtybuffers; + int bd_dirtybufthresh; + int bd_lim; + /* atomics */ + int bd_wanted; + int __aligned(CACHE_LINE_SIZE) bd_numdirtybuffers; + int __aligned(CACHE_LINE_SIZE) bd_running; + long __aligned(CACHE_LINE_SIZE) bd_bufspace; + int __aligned(CACHE_LINE_SIZE) bd_freebuffers; +} __aligned(CACHE_LINE_SIZE); + +#define BD_LOCKPTR(bd) (&(bd)->bd_cleanq->bq_lock) +#define BD_LOCK(bd) mtx_lock(BD_LOCKPTR((bd))) +#define BD_UNLOCK(bd) mtx_unlock(BD_LOCKPTR((bd))) +#define BD_ASSERT_LOCKED(bd) mtx_assert(BD_LOCKPTR((bd)), MA_OWNED) +#define BD_RUN_LOCKPTR(bd) (&(bd)->bd_run_lock) +#define BD_RUN_LOCK(bd) mtx_lock(BD_RUN_LOCKPTR((bd))) +#define BD_RUN_UNLOCK(bd) mtx_unlock(BD_RUN_LOCKPTR((bd))) +#define BD_DOMAIN(bd) (bd - bdomain) + static struct buf *buf; /* buffer header pool */ extern struct buf *swbuf; /* Swap buffer header pool. */ caddr_t unmapped_buf; @@ -123,8 +170,8 @@ static int vfs_bio_clcheck(struct vnode *vp, int s daddr_t lblkno, daddr_t blkno); static void breada(struct vnode *, daddr_t *, int *, int, struct ucred *, int, void (*)(struct buf *)); -static int buf_flush(struct vnode *vp, int); -static int flushbufqueues(struct vnode *, int, int); +static int buf_flush(struct vnode *vp, struct bufdomain *, int); +static int flushbufqueues(struct vnode *, struct bufdomain *, int, int); static void buf_daemon(void); static __inline void bd_wakeup(void); static int sysctl_runningspace(SYSCTL_HANDLER_ARGS); @@ -133,6 +180,16 @@ static void bufkva_free(struct buf *); static int buf_import(void *, void **, int, int, int); static void buf_release(void *, void **, int); static void maxbcachebuf_adjust(void); +static inline struct bufdomain *bufdomain(struct buf *); +static void bq_remove(struct bufqueue *bq, struct buf *bp); +static void bq_insert(struct bufqueue *bq, struct buf *bp, bool unlock); +static int buf_recycle(struct bufdomain *, bool kva); +static void bq_init(struct bufqueue *bq, int qindex, int cpu, + const char *lockname); +static void bd_init(struct bufdomain *bd); +static int bd_flushall(struct bufdomain *bd); +static int sysctl_bufdomain_long(SYSCTL_HANDLER_ARGS); +static int sysctl_bufdomain_int(SYSCTL_HANDLER_ARGS); static int sysctl_bufspace(SYSCTL_HANDLER_ARGS); int vmiodirenable = TRUE; @@ -147,7 +204,9 @@ static counter_u64_t bufkvaspace; SYSCTL_COUNTER_U64(_vfs, OID_AUTO, bufkvaspace, CTLFLAG_RD, &bufkvaspace, "Kernel virtual memory used for buffers"); static long maxbufspace; -SYSCTL_LONG(_vfs, OID_AUTO, maxbufspace, CTLFLAG_RW, &maxbufspace, 0, +SYSCTL_PROC(_vfs, OID_AUTO, maxbufspace, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &maxbufspace, + __offsetof(struct bufdomain, bd_maxbufspace), sysctl_bufdomain_long, "L", "Maximum allowed value of bufspace (including metadata)"); static long bufmallocspace; SYSCTL_LONG(_vfs, OID_AUTO, bufmallocspace, CTLFLAG_RD, &bufmallocspace, 0, @@ -156,14 +215,20 @@ static long maxbufmallocspace; SYSCTL_LONG(_vfs, OID_AUTO, maxmallocbufspace, CTLFLAG_RW, &maxbufmallocspace, 0, "Maximum amount of malloced memory for buffers"); static long lobufspace; -SYSCTL_LONG(_vfs, OID_AUTO, lobufspace, CTLFLAG_RW, &lobufspace, 0, +SYSCTL_PROC(_vfs, OID_AUTO, lobufspace, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &lobufspace, + __offsetof(struct bufdomain, bd_lobufspace), sysctl_bufdomain_long, "L", "Minimum amount of buffers we want to have"); long hibufspace; -SYSCTL_LONG(_vfs, OID_AUTO, hibufspace, CTLFLAG_RW, &hibufspace, 0, +SYSCTL_PROC(_vfs, OID_AUTO, hibufspace, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &hibufspace, + __offsetof(struct bufdomain, bd_hibufspace), sysctl_bufdomain_long, "L", "Maximum allowed value of bufspace (excluding metadata)"); long bufspacethresh; -SYSCTL_LONG(_vfs, OID_AUTO, bufspacethresh, CTLFLAG_RW, &bufspacethresh, - 0, "Bufspace consumed before waking the daemon to free some"); +SYSCTL_PROC(_vfs, OID_AUTO, bufspacethresh, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &bufspacethresh, + __offsetof(struct bufdomain, bd_bufspacethresh), sysctl_bufdomain_long, "L", + "Bufspace consumed before waking the daemon to free some"); static counter_u64_t buffreekvacnt; SYSCTL_COUNTER_U64(_vfs, OID_AUTO, buffreekvacnt, CTLFLAG_RW, &buffreekvacnt, "Number of times we have freed the KVA space from some buffer"); @@ -190,26 +255,37 @@ SYSCTL_INT(_vfs, OID_AUTO, altbufferflushes, CTLFL static int recursiveflushes; SYSCTL_INT(_vfs, OID_AUTO, recursiveflushes, CTLFLAG_RW, &recursiveflushes, 0, "Number of flushes skipped due to being recursive"); -static int numdirtybuffers; -SYSCTL_INT(_vfs, OID_AUTO, numdirtybuffers, CTLFLAG_RD, &numdirtybuffers, 0, +static int sysctl_numdirtybuffers(SYSCTL_HANDLER_ARGS); +SYSCTL_PROC(_vfs, OID_AUTO, numdirtybuffers, + CTLTYPE_INT|CTLFLAG_MPSAFE|CTLFLAG_RD, NULL, 0, sysctl_numdirtybuffers, "I", "Number of buffers that are dirty (has unwritten changes) at the moment"); static int lodirtybuffers; -SYSCTL_INT(_vfs, OID_AUTO, lodirtybuffers, CTLFLAG_RW, &lodirtybuffers, 0, +SYSCTL_PROC(_vfs, OID_AUTO, lodirtybuffers, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &lodirtybuffers, + __offsetof(struct bufdomain, bd_lodirtybuffers), sysctl_bufdomain_int, "L", "How many buffers we want to have free before bufdaemon can sleep"); static int hidirtybuffers; -SYSCTL_INT(_vfs, OID_AUTO, hidirtybuffers, CTLFLAG_RW, &hidirtybuffers, 0, +SYSCTL_PROC(_vfs, OID_AUTO, hidirtybuffers, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &hidirtybuffers, + __offsetof(struct bufdomain, bd_hidirtybuffers), sysctl_bufdomain_int, "L", "When the number of dirty buffers is considered severe"); int dirtybufthresh; -SYSCTL_INT(_vfs, OID_AUTO, dirtybufthresh, CTLFLAG_RW, &dirtybufthresh, - 0, "Number of bdwrite to bawrite conversions to clear dirty buffers"); +SYSCTL_PROC(_vfs, OID_AUTO, dirtybufthresh, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &dirtybufthresh, + __offsetof(struct bufdomain, bd_dirtybufthresh), sysctl_bufdomain_int, "L", + "Number of bdwrite to bawrite conversions to clear dirty buffers"); static int numfreebuffers; SYSCTL_INT(_vfs, OID_AUTO, numfreebuffers, CTLFLAG_RD, &numfreebuffers, 0, "Number of free buffers"); static int lofreebuffers; -SYSCTL_INT(_vfs, OID_AUTO, lofreebuffers, CTLFLAG_RW, &lofreebuffers, 0, +SYSCTL_PROC(_vfs, OID_AUTO, lofreebuffers, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &lofreebuffers, + __offsetof(struct bufdomain, bd_lofreebuffers), sysctl_bufdomain_int, "L", "Target number of free buffers"); static int hifreebuffers; -SYSCTL_INT(_vfs, OID_AUTO, hifreebuffers, CTLFLAG_RW, &hifreebuffers, 0, +SYSCTL_PROC(_vfs, OID_AUTO, hifreebuffers, + CTLTYPE_LONG|CTLFLAG_MPSAFE|CTLFLAG_RW, &hifreebuffers, + __offsetof(struct bufdomain, bd_hifreebuffers), sysctl_bufdomain_int, "L", "Threshold for clean buffer recycling"); static counter_u64_t getnewbufcalls; SYSCTL_COUNTER_U64(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RD, @@ -294,66 +370,19 @@ static int bdirtywait; #define QUEUE_CLEAN 3 /* non-B_DELWRI buffers */ #define QUEUE_SENTINEL 4 /* not an queue index, but mark for sentinel */ -struct bufqueue { - struct mtx_padalign bq_lock; - TAILQ_HEAD(, buf) bq_queue; - uint8_t bq_index; - uint16_t bq_subqueue; - int bq_len; -} __aligned(CACHE_LINE_SIZE); +/* Maximum number of buffer domains. */ +#define BUF_DOMAINS 8 -#define BQ_LOCKPTR(bq) (&(bq)->bq_lock) -#define BQ_LOCK(bq) mtx_lock(BQ_LOCKPTR((bq))) -#define BQ_UNLOCK(bq) mtx_unlock(BQ_LOCKPTR((bq))) -#define BQ_ASSERT_LOCKED(bq) mtx_assert(BQ_LOCKPTR((bq)), MA_OWNED) +struct bufdomainset bdlodirty; /* Domains > lodirty */ +struct bufdomainset bdhidirty; /* Domains > hidirty */ -struct bufqueue __exclusive_cache_line bqempty; -struct bufqueue __exclusive_cache_line bqdirty; - -struct bufdomain { - struct bufqueue bd_subq[MAXCPU + 1]; /* Per-cpu sub queues + global */ - struct bufqueue *bd_cleanq; - struct mtx_padalign bd_run_lock; - /* Constants */ - long bd_maxbufspace; - long bd_hibufspace; - long bd_lobufspace; - long bd_bufspacethresh; - int bd_hifreebuffers; - int bd_lofreebuffers; - int bd_lim; - /* atomics */ - int bd_wanted; - int __aligned(CACHE_LINE_SIZE) bd_running; - long __aligned(CACHE_LINE_SIZE) bd_bufspace; - int __aligned(CACHE_LINE_SIZE) bd_freebuffers; -} __aligned(CACHE_LINE_SIZE); - -#define BD_LOCKPTR(bd) (&(bd)->bd_cleanq->bq_lock) -#define BD_LOCK(bd) mtx_lock(BD_LOCKPTR((bd))) -#define BD_UNLOCK(bd) mtx_unlock(BD_LOCKPTR((bd))) -#define BD_ASSERT_LOCKED(bd) mtx_assert(BD_LOCKPTR((bd)), MA_OWNED) -#define BD_RUN_LOCKPTR(bd) (&(bd)->bd_run_lock) -#define BD_RUN_LOCK(bd) mtx_lock(BD_RUN_LOCKPTR((bd))) -#define BD_RUN_UNLOCK(bd) mtx_unlock(BD_RUN_LOCKPTR((bd))) -#define BD_DOMAIN(bd) (bd - bdclean) - -/* Maximum number of clean buffer domains. */ -#define CLEAN_DOMAINS 8 - /* Configured number of clean queues. */ -static int __read_mostly clean_domains; +static int __read_mostly buf_domains; -struct bufdomain __exclusive_cache_line bdclean[CLEAN_DOMAINS]; +BITSET_DEFINE(bufdomainset, BUF_DOMAINS); +struct bufdomain __exclusive_cache_line bdomain[BUF_DOMAINS]; +struct bufqueue __exclusive_cache_line bqempty; -static void bq_remove(struct bufqueue *bq, struct buf *bp); -static void bq_insert(struct bufqueue *bq, struct buf *bp, bool unlock); -static int buf_recycle(struct bufdomain *, bool kva); -static void bq_init(struct bufqueue *bq, int qindex, int cpu, - const char *lockname); -static void bd_init(struct bufdomain *bd); -static int bd_flushall(struct bufdomain *bd); - /* * per-cpu empty buffer cache. */ @@ -393,6 +422,44 @@ sysctl_runningspace(SYSCTL_HANDLER_ARGS) return (error); } +static int +sysctl_bufdomain_int(SYSCTL_HANDLER_ARGS) +{ + int error; + int value; + int i; + + value = *(int *)arg1; + error = sysctl_handle_int(oidp, &value, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + *(int *)arg1 = value; + for (i = 0; i < buf_domains; i++) + *(int *)(((uintptr_t)&bdomain[i]) + arg2) = + value / buf_domains; + + return (error); +} + +static int +sysctl_bufdomain_long(SYSCTL_HANDLER_ARGS) +{ + long value; + int error; + int i; + + value = *(long *)arg1; + error = sysctl_handle_long(oidp, &value, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + *(long *)arg1 = value; + for (i = 0; i < buf_domains; i++) + *(long *)(((uintptr_t)&bdomain[i]) + arg2) = + value / buf_domains; + + return (error); +} + #if defined(COMPAT_FREEBSD4) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD7) static int @@ -403,8 +470,8 @@ sysctl_bufspace(SYSCTL_HANDLER_ARGS) int i; lvalue = 0; - for (i = 0; i < clean_domains; i++) - lvalue += bdclean[i].bd_bufspace; + for (i = 0; i < buf_domains; i++) + lvalue += bdomain[i].bd_bufspace; if (sizeof(int) == sizeof(long) || req->oldlen >= sizeof(long)) return (sysctl_handle_long(oidp, &lvalue, 0, req)); if (lvalue > INT_MAX) @@ -421,12 +488,24 @@ sysctl_bufspace(SYSCTL_HANDLER_ARGS) int i; lvalue = 0; - for (i = 0; i < clean_domains; i++) - lvalue += bdclean[i].bd_bufspace; + for (i = 0; i < buf_domains; i++) + lvalue += bdomain[i].bd_bufspace; return (sysctl_handle_long(oidp, &lvalue, 0, req)); } #endif +static int +sysctl_numdirtybuffers(SYSCTL_HANDLER_ARGS) +{ + int value; + int i; + + value = 0; + for (i = 0; i < buf_domains; i++) + value += bdomain[i].bd_numdirtybuffers; + return (sysctl_handle_int(oidp, &value, 0, req)); +} + /* * bdirtywakeup: * @@ -444,6 +523,42 @@ bdirtywakeup(void) } /* + * bd_clear: + * + * Clear a domain from the appropriate bitsets when dirtybuffers + * is decremented. + */ +static void +bd_clear(struct bufdomain *bd) +{ + + mtx_lock(&bdirtylock); + if (bd->bd_numdirtybuffers <= bd->bd_lodirtybuffers) + BIT_CLR(BUF_DOMAINS, BD_DOMAIN(bd), &bdlodirty); + if (bd->bd_numdirtybuffers <= bd->bd_hidirtybuffers) + BIT_CLR(BUF_DOMAINS, BD_DOMAIN(bd), &bdhidirty); + mtx_unlock(&bdirtylock); +} + +/* + * bd_set: + * + * Set a domain in the appropriate bitsets when dirtybuffers + * is incremented. + */ +static void +bd_set(struct bufdomain *bd) +{ + + mtx_lock(&bdirtylock); + if (bd->bd_numdirtybuffers > bd->bd_lodirtybuffers) + BIT_SET(BUF_DOMAINS, BD_DOMAIN(bd), &bdlodirty); + if (bd->bd_numdirtybuffers > bd->bd_hidirtybuffers) + BIT_SET(BUF_DOMAINS, BD_DOMAIN(bd), &bdhidirty); + mtx_unlock(&bdirtylock); +} + +/* * bdirtysub: * * Decrement the numdirtybuffers count by one and wakeup any @@ -450,12 +565,17 @@ bdirtywakeup(void) * threads blocked in bwillwrite(). */ static void -bdirtysub(void) +bdirtysub(struct buf *bp) { + struct bufdomain *bd; + int num; - if (atomic_fetchadd_int(&numdirtybuffers, -1) == - (lodirtybuffers + hidirtybuffers) / 2) + bd = bufdomain(bp); + num = atomic_fetchadd_int(&bd->bd_numdirtybuffers, -1); + if (num == (bd->bd_lodirtybuffers + bd->bd_hidirtybuffers) / 2) bdirtywakeup(); + if (num == bd->bd_lodirtybuffers || num == bd->bd_hidirtybuffers) + bd_clear(bd); } /* @@ -465,16 +585,21 @@ static void * daemon if needed. */ static void -bdirtyadd(void) +bdirtyadd(struct buf *bp) { + struct bufdomain *bd; + int num; /* * Only do the wakeup once as we cross the boundary. The * buf daemon will keep running until the condition clears. */ - if (atomic_fetchadd_int(&numdirtybuffers, 1) == - (lodirtybuffers + hidirtybuffers) / 2) + bd = bufdomain(bp); + num = atomic_fetchadd_int(&bd->bd_numdirtybuffers, 1); + if (num == (bd->bd_lodirtybuffers + bd->bd_hidirtybuffers) / 2) bd_wakeup(); + if (num == bd->bd_lodirtybuffers || num == bd->bd_hidirtybuffers) + bd_set(bd); } /* @@ -539,11 +664,11 @@ bufspace_adjust(struct buf *bp, int bufsize) KASSERT((bp->b_flags & B_MALLOC) == 0, ("bufspace_adjust: malloc buf %p", bp)); - bd = &bdclean[bp->b_domain]; + bd = bufdomain(bp); diff = bufsize - bp->b_bufsize; if (diff < 0) { atomic_subtract_long(&bd->bd_bufspace, -diff); - } else { + } else if (diff > 0) { space = atomic_fetchadd_long(&bd->bd_bufspace, diff); /* Wake up the daemon on the transition. */ if (space < bd->bd_bufspacethresh && @@ -638,7 +763,7 @@ bufspace_wait(struct bufdomain *bd, struct vnode * * recursion. */ td->td_pflags |= TDP_BUFNEED | TDP_NORUNNINGBUF; - fl = buf_flush(vp, flushbufqtarget); + fl = buf_flush(vp, bd, flushbufqtarget); td->td_pflags &= norunbuf; BD_LOCK(bd); if (fl != 0) @@ -700,6 +825,15 @@ bufspace_daemon(void *arg) if (buf_recycle(bd, false) != 0) { if (bd_flushall(bd)) continue; + /* + * Speedup dirty if we've run out of clean + * buffers. This is possible in particular + * because softdep may held many bufs locked + * pending writes to other bufs which are + * marked for delayed write, exhausting + * clean space until they are written. + */ + bd_speedup(); BD_LOCK(bd); if (bd->bd_wanted) { msleep(&bd->bd_wanted, BD_LOCKPTR(bd), @@ -1025,7 +1159,6 @@ bufinit(void) ("maxbcachebuf (%d) must be >= MAXBSIZE (%d)\n", maxbcachebuf, MAXBSIZE)); bq_init(&bqempty, QUEUE_EMPTY, -1, "bufq empty lock"); - bq_init(&bqdirty, QUEUE_DIRTY, -1, "bufq dirty lock"); mtx_init(&rbreqlock, "runningbufspace lock", NULL, MTX_DEF); mtx_init(&bdlock, "buffer daemon lock", NULL, MTX_DEF); mtx_init(&bdirtylock, "dirty buf lock", NULL, MTX_DEF); @@ -1093,7 +1226,6 @@ bufinit(void) */ hidirtybuffers = nbuf / 4 + 20; dirtybufthresh = hidirtybuffers * 9 / 10; - numdirtybuffers = 0; /* * To support extreme low-memory systems, make sure hidirtybuffers * cannot eat up all available buffer space. This occurs when our @@ -1128,22 +1260,26 @@ bufinit(void) * One queue per-256mb up to the max. More queues gives better * concurrency but less accurate LRU. */ - clean_domains = MIN(howmany(maxbufspace, 256*1024*1024), CLEAN_DOMAINS); - for (i = 0 ; i < clean_domains; i++) { + buf_domains = MIN(howmany(maxbufspace, 256*1024*1024), BUF_DOMAINS); + for (i = 0 ; i < buf_domains; i++) { struct bufdomain *bd; - bd = &bdclean[i]; + bd = &bdomain[i]; bd_init(bd); - bd->bd_freebuffers = nbuf / clean_domains; - bd->bd_hifreebuffers = hifreebuffers / clean_domains; - bd->bd_lofreebuffers = lofreebuffers / clean_domains; + bd->bd_freebuffers = nbuf / buf_domains; + bd->bd_hifreebuffers = hifreebuffers / buf_domains; + bd->bd_lofreebuffers = lofreebuffers / buf_domains; bd->bd_bufspace = 0; - bd->bd_maxbufspace = maxbufspace / clean_domains; - bd->bd_hibufspace = hibufspace / clean_domains; - bd->bd_lobufspace = lobufspace / clean_domains; - bd->bd_bufspacethresh = bufspacethresh / clean_domains; + bd->bd_maxbufspace = maxbufspace / buf_domains; + bd->bd_hibufspace = hibufspace / buf_domains; + bd->bd_lobufspace = lobufspace / buf_domains; + bd->bd_bufspacethresh = bufspacethresh / buf_domains; + bd->bd_numdirtybuffers = 0; + bd->bd_hidirtybuffers = hidirtybuffers / buf_domains; + bd->bd_lodirtybuffers = lodirtybuffers / buf_domains; + bd->bd_dirtybufthresh = dirtybufthresh / buf_domains; /* Don't allow more than 2% of bufs in the per-cpu caches. */ - bd->bd_lim = nbuf / clean_domains / 50 / mp_ncpus; + bd->bd_lim = nbuf / buf_domains / 50 / mp_ncpus; } getnewbufcalls = counter_u64_alloc(M_WAITOK); getnewbufrestarts = counter_u64_alloc(M_WAITOK); @@ -1327,6 +1463,13 @@ bpmap_qenter(struct buf *bp) (vm_offset_t)(bp->b_offset & PAGE_MASK)); } +static inline struct bufdomain * +bufdomain(struct buf *bp) +{ + + return (&bdomain[bp->b_domain]); +} + static struct bufqueue * bufqueue(struct buf *bp) { @@ -1339,9 +1482,9 @@ bufqueue(struct buf *bp) case QUEUE_EMPTY: return (&bqempty); case QUEUE_DIRTY: - return (&bqdirty); + return (&bufdomain(bp)->bd_dirtyq); case QUEUE_CLEAN: - return (&bdclean[bp->b_domain].bd_subq[bp->b_subqueue]); + return (&bufdomain(bp)->bd_subq[bp->b_subqueue]); default: break; } @@ -1404,14 +1547,14 @@ binsfree(struct buf *bp, int qindex) bq_remove(bq, bp); BQ_UNLOCK(bq); } + bd = bufdomain(bp); if (qindex == QUEUE_CLEAN) { - bd = &bdclean[bp->b_domain]; if (bd->bd_lim != 0) bq = &bd->bd_subq[PCPU_GET(cpuid)]; else bq = bd->bd_cleanq; } else - bq = &bqdirty; + bq = &bd->bd_dirtyq; bq_insert(bq, bp, true); } @@ -1439,7 +1582,7 @@ buf_free(struct buf *bp) if (!LIST_EMPTY(&bp->b_dep)) buf_deallocate(bp); bufkva_free(bp); - atomic_add_int(&bdclean[bp->b_domain].bd_freebuffers, 1); + atomic_add_int(&bufdomain(bp)->bd_freebuffers, 1); BUF_UNLOCK(bp); uma_zfree(buf_zone, bp); } @@ -1715,9 +1858,10 @@ bd_init(struct bufdomain *bd) int domain; int i; - domain = bd - bdclean; + domain = bd - bdomain; bd->bd_cleanq = &bd->bd_subq[mp_maxid + 1]; bq_init(bd->bd_cleanq, QUEUE_CLEAN, mp_maxid + 1, "bufq clean lock"); + bq_init(&bd->bd_dirtyq, QUEUE_DIRTY, -1, "bufq dirty lock"); for (i = 0; i <= mp_maxid; i++) bq_init(&bd->bd_subq[i], QUEUE_CLEAN, i, "bufq clean subqueue lock"); @@ -1809,7 +1953,7 @@ bq_insert(struct bufqueue *bq, struct buf *bp, boo if (bp->b_qindex != QUEUE_NONE) panic("bq_insert: free buffer %p onto another queue?", bp); - bd = &bdclean[bp->b_domain]; + bd = bufdomain(bp); if (bp->b_flags & B_AGE) { /* Place this buf directly on the real queue. */ if (bq->bq_index == QUEUE_CLEAN) @@ -1926,8 +2070,8 @@ bufkva_reclaim(vmem_t *vmem, int flags) done = false; for (i = 0; i < 5; i++) { - for (q = 0; q < clean_domains; q++) - if (buf_recycle(&bdclean[q], true) != 0) + for (q = 0; q < buf_domains; q++) + if (buf_recycle(&bdomain[q], true) != 0) done = true; if (done) break; @@ -2319,7 +2463,7 @@ bdirty(struct buf *bp) if ((bp->b_flags & B_DELWRI) == 0) { bp->b_flags |= /* XXX B_DONE | */ B_DELWRI; reassignbuf(bp); - bdirtyadd(); + bdirtyadd(bp); } } @@ -2347,7 +2491,7 @@ bundirty(struct buf *bp) if (bp->b_flags & B_DELWRI) { bp->b_flags &= ~B_DELWRI; reassignbuf(bp); - bdirtysub(); + bdirtysub(bp); } /* * Since it is now being written, we can clear its deferred write flag. @@ -2419,9 +2563,9 @@ void bwillwrite(void) { - if (numdirtybuffers >= hidirtybuffers) { + if (buf_dirty_count_severe()) { mtx_lock(&bdirtylock); - while (numdirtybuffers >= hidirtybuffers) { + while (buf_dirty_count_severe()) { bdirtywait = 1; msleep(&bdirtywait, &bdirtylock, (PRIBIO + 4), "flswai", 0); @@ -2437,7 +2581,7 @@ int buf_dirty_count_severe(void) { - return(numdirtybuffers >= hidirtybuffers); + return (!BIT_EMPTY(BUF_DOMAINS, &bdhidirty)); } /* @@ -2522,7 +2666,7 @@ brelse(struct buf *bp) if (!LIST_EMPTY(&bp->b_dep)) buf_deallocate(bp); if (bp->b_flags & B_DELWRI) - bdirtysub(); + bdirtysub(bp); bp->b_flags &= ~(B_DELWRI | B_CACHE); if ((bp->b_flags & B_VMIO) == 0) { allocbuf(bp, 0); @@ -3135,9 +3279,9 @@ getnewbuf(struct vnode *vp, int slpflag, int slpti else metadata = false; if (vp == NULL) - bd = &bdclean[0]; + bd = &bdomain[0]; else - bd = &bdclean[vp->v_bufobj.bo_domain]; + bd = &bdomain[vp->v_bufobj.bo_domain]; counter_u64_add(getnewbufcalls, 1); reserved = false; @@ -3183,11 +3327,11 @@ static struct kproc_desc buf_kp = { SYSINIT(bufdaemon, SI_SUB_KTHREAD_BUF, SI_ORDER_FIRST, kproc_start, &buf_kp); static int -buf_flush(struct vnode *vp, int target) +buf_flush(struct vnode *vp, struct bufdomain *bd, int target) { int flushed; - flushed = flushbufqueues(vp, target, 0); + flushed = flushbufqueues(vp, bd, target, 0); if (flushed == 0) { /* * Could not find any buffers without rollback @@ -3196,7 +3340,7 @@ static int */ if (vp != NULL && target > 2) target /= 2; - flushbufqueues(vp, target, 1); + flushbufqueues(vp, bd, target, 1); } return (flushed); } @@ -3204,6 +3348,8 @@ static int static void buf_daemon() { + struct bufdomain *bd; + int speedupreq; int lodirty; int i; @@ -3216,11 +3362,11 @@ buf_daemon() /* * Start the buf clean daemons as children threads. */ - for (i = 0 ; i < clean_domains; i++) { + for (i = 0 ; i < buf_domains; i++) { int error; error = kthread_add((void (*)(void *))bufspace_daemon, - &bdclean[i], curproc, NULL, 0, 0, "bufspacedaemon-%d", i); + &bdomain[i], curproc, NULL, 0, 0, "bufspacedaemon-%d", i); if (error) panic("error %d spawning bufspace daemon", error); } @@ -3235,20 +3381,30 @@ buf_daemon() mtx_unlock(&bdlock); kproc_suspend_check(bufdaemonproc); - lodirty = lodirtybuffers; - if (bd_speedupreq) { - lodirty = numdirtybuffers / 2; - bd_speedupreq = 0; - } + /* - * Do the flush. Limit the amount of in-transit I/O we - * allow to build up, otherwise we would completely saturate - * the I/O system. + * Save speedupreq for this pass and reset to capture new + * requests. */ - while (numdirtybuffers > lodirty) { - if (buf_flush(NULL, numdirtybuffers - lodirty) == 0) - break; - kern_yield(PRI_USER); + speedupreq = bd_speedupreq; + bd_speedupreq = 0; + + /* + * Flush each domain sequentially according to its level and + * the speedup request. + */ + for (i = 0; i < buf_domains; i++) { + bd = &bdomain[i]; + if (speedupreq) + lodirty = bd->bd_numdirtybuffers / 2; + else + lodirty = bd->bd_lodirtybuffers; + while (bd->bd_numdirtybuffers > lodirty) { + if (buf_flush(NULL, bd, + bd->bd_numdirtybuffers - lodirty) == 0) + break; + kern_yield(PRI_USER); + } } /* @@ -3262,7 +3418,7 @@ buf_daemon() * to avoid endless loops on unlockable buffers. */ mtx_lock(&bdlock); - if (numdirtybuffers <= lodirtybuffers) { + if (!BIT_EMPTY(BUF_DOMAINS, &bdlodirty)) { /* * We reached our low water mark, reset the * request and sleep until we are needed again. @@ -3301,7 +3457,8 @@ SYSCTL_INT(_vfs, OID_AUTO, flushwithdeps, CTLFLAG_ 0, "Number of buffers flushed with dependecies that require rollbacks"); static int -flushbufqueues(struct vnode *lvp, int target, int flushdeps) +flushbufqueues(struct vnode *lvp, struct bufdomain *bd, int target, + int flushdeps) { struct bufqueue *bq; struct buf *sentinel; @@ -3314,7 +3471,7 @@ static int bool unlock; flushed = 0; - bq = &bqdirty; + bq = &bd->bd_dirtyq; bp = NULL; sentinel = malloc(sizeof(struct buf), M_TEMP, M_WAITOK | M_ZERO); sentinel->b_qindex = QUEUE_SENTINEL; @@ -3650,7 +3807,7 @@ bp_unmapped_get_kva(struct buf *bp, daddr_t blkno, panic("GB_NOWAIT_BD and GB_UNMAPPED %p", bp); } counter_u64_add(mappingrestarts, 1); - bufspace_wait(&bdclean[bp->b_domain], bp->b_vp, gbflags, 0, 0); + bufspace_wait(bufdomain(bp), bp->b_vp, gbflags, 0, 0); } has_addr: if (need_mapping) { @@ -3848,7 +4005,7 @@ loop: */ if (flags & GB_NOCREAT) return NULL; - if (bdclean[bo->bo_domain].bd_freebuffers == 0 && + if (bdomain[bo->bo_domain].bd_freebuffers == 0 && TD_IS_IDLETHREAD(curthread)) return NULL; @@ -3905,7 +4062,7 @@ loop: if (gbincore(bo, blkno)) { BO_UNLOCK(bo); bp->b_flags |= B_INVAL; - bufspace_release(&bdclean[bp->b_domain], maxsize); + bufspace_release(bufdomain(bp), maxsize); brelse(bp); goto loop; } @@ -3940,7 +4097,7 @@ loop: } allocbuf(bp, size); - bufspace_release(&bdclean[bp->b_domain], maxsize); + bufspace_release(bufdomain(bp), maxsize); bp->b_flags &= ~B_DONE; } CTR4(KTR_BUF, "getblk(%p, %ld, %d) = %p", vp, (long)blkno, size, bp); @@ -3969,7 +4126,7 @@ geteblk(int size, int flags) return (NULL); } allocbuf(bp, size); - bufspace_release(&bdclean[bp->b_domain], maxsize); + bufspace_release(bufdomain(bp), maxsize); bp->b_flags |= B_INVAL; /* b_dep cleared by getnewbuf() */ BUF_ASSERT_HELD(bp); return (bp); @@ -4838,7 +4995,7 @@ bufobj_init(struct bufobj *bo, void *private) static volatile int bufobj_cleanq; bo->bo_domain = - atomic_fetchadd_int(&bufobj_cleanq, 1) % clean_domains; + atomic_fetchadd_int(&bufobj_cleanq, 1) % buf_domains; rw_init(BO_LOCKPTR(bo), "bufobj interlock"); bo->bo_private = private; TAILQ_INIT(&bo->bo_clean.bv_hd); @@ -5160,6 +5317,7 @@ DB_SHOW_COMMAND(buffer, db_show_buffer) } db_printf("\n"); } + BUF_LOCKPRINTINFO(bp); #if defined(FULL_BUF_TRACKING) db_printf("b_io_tracking: b_io_tcnt = %u\n", bp->b_io_tcnt); @@ -5174,19 +5332,19 @@ DB_SHOW_COMMAND(buffer, db_show_buffer) db_printf("b_io_tracking: %s\n", bp->b_io_tracking); #endif db_printf(" "); - BUF_LOCKPRINTINFO(bp); } DB_SHOW_COMMAND(bufqueues, bufqueues) { struct bufdomain *bd; - int i, j; + struct buf *bp; + long total; + int i, j, cnt; db_printf("bqempty: %d\n", bqempty.bq_len); - db_printf("bqdirty: %d\n", bqdirty.bq_len); - for (i = 0; i < clean_domains; i++) { - bd = &bdclean[i]; + for (i = 0; i < buf_domains; i++) { + bd = &bdomain[i]; db_printf("Buf domain %d\n", i); db_printf("\tfreebufs\t%d\n", bd->bd_freebuffers); db_printf("\tlofreebufs\t%d\n", bd->bd_lofreebuffers); @@ -5198,7 +5356,21 @@ DB_SHOW_COMMAND(bufqueues, bufqueues) db_printf("\tlobufspace\t%ld\n", bd->bd_lobufspace); db_printf("\tbufspacethresh\t%ld\n", bd->bd_bufspacethresh); db_printf("\n"); - db_printf("\tcleanq count\t%d\n", bd->bd_cleanq->bq_len); + db_printf("\tnumdirtybuffers\t%d\n", bd->bd_numdirtybuffers); + db_printf("\tlodirtybuffers\t%d\n", bd->bd_lodirtybuffers); + db_printf("\thidirtybuffers\t%d\n", bd->bd_hidirtybuffers); + db_printf("\tdirtybufthresh\t%d\n", bd->bd_dirtybufthresh); + db_printf("\n"); + total = 0; + TAILQ_FOREACH(bp, &bd->bd_cleanq->bq_queue, b_freelist) + total += bp->b_bufsize; + db_printf("\tcleanq count\t%d (%ld)\n", + bd->bd_cleanq->bq_len, total); + total = 0; + TAILQ_FOREACH(bp, &bd->bd_dirtyq.bq_queue, b_freelist) + total += bp->b_bufsize; + db_printf("\tdirtyq count\t%d (%ld)\n", + bd->bd_dirtyq.bq_len, total); db_printf("\twakeup\t\t%d\n", bd->bd_wanted); db_printf("\tlim\t\t%d\n", bd->bd_lim); db_printf("\tCPU "); @@ -5205,6 +5377,22 @@ DB_SHOW_COMMAND(bufqueues, bufqueues) for (j = 0; j <= mp_maxid; j++) db_printf("%d, ", bd->bd_subq[j].bq_len); db_printf("\n"); + cnt = 0; + total = 0; + for (j = 0; j < nbuf; j++) + if (buf[j].b_domain == i && BUF_ISLOCKED(&buf[j])) { + cnt++; + total += buf[j].b_bufsize; + } + db_printf("\tLocked buffers: %d space %jd\n", cnt, total); + cnt = 0; + total = 0; + for (j = 0; j < nbuf; j++) + if (buf[j].b_domain == i) { + cnt++; + total += buf[j].b_bufsize; + } + db_printf("\tTotal buffers: %d space %jd\n", cnt, total); } } Index: /usr/src/sys/vm/vm_page.c =================================================================== --- /usr/src/sys/vm/vm_page.c (revision 330873) +++ /usr/src/sys/vm/vm_page.c (working copy) @@ -139,7 +139,9 @@ extern int vmem_startup_count(void); struct vm_domain vm_dom[MAXMEMDOM]; struct mtx_padalign __exclusive_cache_line pa_lock[PA_LOCK_COUNT]; + struct mtx_padalign __exclusive_cache_line vm_domainset_lock; +/* The following fields are protected by the domainset lock. */ domainset_t __exclusive_cache_line vm_min_domains; domainset_t __exclusive_cache_line vm_severe_domains; static int vm_min_waiters; @@ -146,7 +148,6 @@ static int vm_min_waiters; static int vm_severe_waiters; static int vm_pageproc_waiters; - /* * bogus page -- for I/O to/from partially complete buffers, * or for paging into sparsely invalid regions. @@ -184,7 +185,6 @@ static void vm_page_insert_radixdone(vm_page_t m, vm_page_t mpred); static int vm_page_reclaim_run(int req_class, int domain, u_long npages, vm_page_t m_run, vm_paddr_t high); -static void vm_domain_free_wakeup(struct vm_domain *); static int vm_domain_alloc_fail(struct vm_domain *vmd, vm_object_t object, int req); @@ -430,6 +430,7 @@ vm_page_domain_init(int domain) MTX_DEF | MTX_DUPOK); } mtx_init(&vmd->vmd_free_mtx, "vm page free queue", NULL, MTX_DEF); + mtx_init(&vmd->vmd_pageout_mtx, "vm pageout lock", NULL, MTX_DEF); snprintf(vmd->vmd_name, sizeof(vmd->vmd_name), "%d", domain); } @@ -731,8 +732,8 @@ vm_page_startup(vm_offset_t vaddr) vmd = VM_DOMAIN(seg->domain); vm_domain_free_lock(vmd); vm_phys_free_contig(m, pagecount); - vm_domain_freecnt_adj(vmd, (int)pagecount); vm_domain_free_unlock(vmd); + vm_domain_freecnt_inc(vmd, pagecount); vm_cnt.v_page_count += (u_int)pagecount; vmd = VM_DOMAIN(seg->domain); @@ -1694,7 +1695,6 @@ vm_page_alloc_domain_after(vm_object_t object, vm_ struct vm_domain *vmd; vm_page_t m; int flags; - u_int free_count; KASSERT((object != NULL) == ((req & VM_ALLOC_NOOBJ) == 0) && (object != NULL || (req & VM_ALLOC_SBUSY) == 0) && @@ -1747,6 +1747,9 @@ again: #endif } } + if (m != NULL) + vm_domain_freecnt_dec(vmd, 1); + vm_domain_free_unlock(vmd); if (m == NULL) { /* * Not allocatable, give up. @@ -1760,15 +1763,7 @@ again: * At this point we had better have found a good page. */ KASSERT(m != NULL, ("missing page")); - free_count = vm_domain_freecnt_adj(vmd, -1); - vm_domain_free_unlock(vmd); - /* - * Don't wakeup too often - wakeup the pageout daemon when - * we would be nearly out of memory. - */ - if (vm_paging_needed(vmd, free_count)) - pagedaemon_wakeup(vmd->vmd_domain); #if VM_NRESERVLEVEL > 0 found: #endif @@ -1804,7 +1799,6 @@ found: if (object != NULL) { if (vm_page_insert_after(m, object, pindex, mpred)) { - pagedaemon_wakeup(domain); if (req & VM_ALLOC_WIRED) { vm_wire_sub(1); m->wire_count = 0; @@ -1961,13 +1955,14 @@ retry: goto retry; #endif } + if (m_ret != NULL) + vm_domain_freecnt_dec(vmd, npages); + vm_domain_free_unlock(vmd); if (m_ret == NULL) { if (vm_domain_alloc_fail(vmd, object, req)) goto again; return (NULL); } - vm_domain_freecnt_adj(vmd, -npages); - vm_domain_free_unlock(vmd); #if VM_NRESERVLEVEL > 0 found: #endif @@ -2006,7 +2001,6 @@ found: m->oflags = oflags; if (object != NULL) { if (vm_page_insert_after(m, object, pindex, mpred)) { - pagedaemon_wakeup(domain); if ((req & VM_ALLOC_WIRED) != 0) vm_wire_sub(npages); KASSERT(m->object == NULL, @@ -2035,9 +2029,6 @@ found: pmap_page_set_memattr(m, memattr); pindex++; } - vmd = VM_DOMAIN(domain); - if (vm_paging_needed(vmd, vmd->vmd_free_count)) - pagedaemon_wakeup(domain); return (m_ret); } @@ -2100,7 +2091,7 @@ vm_page_alloc_freelist_domain(int domain, int free { struct vm_domain *vmd; vm_page_t m; - u_int flags, free_count; + u_int flags; /* * Do not allocate reserved pages unless the req has asked for it. @@ -2111,13 +2102,14 @@ again: if (vm_domain_available(vmd, req, 1)) m = vm_phys_alloc_freelist_pages(domain, freelist, VM_FREEPOOL_DIRECT, 0); + if (m != NULL) + vm_domain_freecnt_dec(vmd, 1); + vm_domain_free_unlock(vmd); if (m == NULL) { if (vm_domain_alloc_fail(vmd, NULL, req)) goto again; return (NULL); } - free_count = vm_domain_freecnt_adj(vmd, -1); - vm_domain_free_unlock(vmd); vm_page_alloc_check(m); /* @@ -2138,8 +2130,6 @@ again: } /* Unmanaged pages don't use "act_count". */ m->oflags = VPO_UNMANAGED; - if (vm_paging_needed(vmd, free_count)) - pagedaemon_wakeup(domain); return (m); } @@ -2539,15 +2529,19 @@ unlock: if (m_mtx != NULL) mtx_unlock(m_mtx); if ((m = SLIST_FIRST(&free)) != NULL) { + int cnt; + vmd = VM_DOMAIN(domain); + cnt = 0; vm_domain_free_lock(vmd); do { MPASS(vm_phys_domain(m) == domain); SLIST_REMOVE_HEAD(&free, plinks.s.ss); vm_page_free_phys(vmd, m); + cnt++; } while ((m = SLIST_FIRST(&free)) != NULL); - vm_domain_free_wakeup(vmd); vm_domain_free_unlock(vmd); + vm_domain_freecnt_inc(vmd, cnt); } return (error); } @@ -2710,7 +2704,7 @@ vm_domain_set(struct vm_domain *vmd) /* * Clear the domain from the appropriate page level domainset. */ -static void +void vm_domain_clear(struct vm_domain *vmd) { @@ -2731,6 +2725,22 @@ vm_domain_clear(struct vm_domain *vmd) wakeup(&vm_severe_domains); } } + + /* + * If pageout daemon needs pages, then tell it that there are + * some free. + */ + if (vmd->vmd_pageout_pages_needed && + vmd->vmd_free_count >= vmd->vmd_pageout_free_min) { + wakeup(&vmd->vmd_pageout_pages_needed); + vmd->vmd_pageout_pages_needed = 0; + } + + /* See comments in vm_wait_doms(). */ + if (vm_pageproc_waiters) { + vm_pageproc_waiters = 0; + wakeup(&vm_pageproc_waiters); + } mtx_unlock(&vm_domainset_lock); } @@ -2769,7 +2779,7 @@ u_int vm_wait_count(void) { - return (vm_severe_waiters + vm_min_waiters); + return (vm_severe_waiters + vm_min_waiters + vm_pageproc_waiters); } static void @@ -2787,9 +2797,8 @@ vm_wait_doms(const domainset_t *wdoms) if (curproc == pageproc) { mtx_lock(&vm_domainset_lock); vm_pageproc_waiters++; - msleep(&vm_pageproc_waiters, &vm_domainset_lock, PVM, + msleep(&vm_pageproc_waiters, &vm_domainset_lock, PVM | PDROP, "pageprocwait", 1); - mtx_unlock(&vm_domainset_lock); } else { /* * XXX Ideally we would wait only until the allocation could @@ -2819,14 +2828,17 @@ vm_wait_domain(int domain) domainset_t wdom; vmd = VM_DOMAIN(domain); - vm_domain_free_assert_locked(vmd); + vm_domain_free_assert_unlocked(vmd); if (curproc == pageproc) { - vmd->vmd_pageout_pages_needed = 1; - msleep(&vmd->vmd_pageout_pages_needed, - vm_domain_free_lockptr(vmd), PDROP | PSWP, "VMWait", 0); + mtx_lock(&vm_domainset_lock); + if (vmd->vmd_free_count < vmd->vmd_pageout_free_min) { + vmd->vmd_pageout_pages_needed = 1; + msleep(&vmd->vmd_pageout_pages_needed, + &vm_domainset_lock, PDROP | PSWP, "VMWait", 0); + } else + mtx_unlock(&vm_domainset_lock); } else { - vm_domain_free_unlock(vmd); if (pageproc == NULL) panic("vm_wait in early boot"); DOMAINSET_ZERO(&wdom); @@ -2876,7 +2888,7 @@ static int vm_domain_alloc_fail(struct vm_domain *vmd, vm_object_t object, int req) { - vm_domain_free_assert_locked(vmd); + vm_domain_free_assert_unlocked(vmd); atomic_add_int(&vmd->vmd_pageout_deficit, max((u_int)req >> VM_ALLOC_COUNT_SHIFT, 1)); @@ -2888,10 +2900,8 @@ vm_domain_alloc_fail(struct vm_domain *vmd, vm_obj VM_OBJECT_WLOCK(object); if (req & VM_ALLOC_WAITOK) return (EAGAIN); - } else { - vm_domain_free_unlock(vmd); - pagedaemon_wakeup(vmd->vmd_domain); } + return (0); } @@ -3063,46 +3073,6 @@ vm_page_activate(vm_page_t m) } /* - * vm_domain_free_wakeup: - * - * Helper routine for vm_page_free_toq(). This routine is called - * when a page is added to the free queues. - * - * The page queues must be locked. - */ -static void -vm_domain_free_wakeup(struct vm_domain *vmd) -{ - - vm_domain_free_assert_locked(vmd); - - /* - * if pageout daemon needs pages, then tell it that there are - * some free. - */ - if (vmd->vmd_pageout_pages_needed && - vmd->vmd_free_count >= vmd->vmd_pageout_free_min) { - wakeup(&vmd->vmd_pageout_pages_needed); - vmd->vmd_pageout_pages_needed = 0; - } - /* - * wakeup processes that are waiting on memory if we hit a - * high water mark. And wakeup scheduler process if we have - * lots of memory. this process will swapin processes. - */ - if ((vmd->vmd_minset && !vm_paging_min(vmd)) || - (vmd->vmd_severeset && !vm_paging_severe(vmd))) - vm_domain_clear(vmd); - - /* See comments in vm_wait(); */ - if (vm_pageproc_waiters) { - vm_pageproc_waiters = 0; - wakeup(&vm_pageproc_waiters); - } - -} - -/* * vm_page_free_prep: * * Prepares the given page to be put on the free list, @@ -3183,7 +3153,8 @@ vm_page_free_prep(vm_page_t m, bool pagequeue_lock /* * Insert the page into the physical memory allocator's free page - * queues. This is the last step to free a page. + * queues. This is the last step to free a page. The caller is + * responsible for adjusting the free page count. */ static void vm_page_free_phys(struct vm_domain *vmd, vm_page_t m) @@ -3191,7 +3162,6 @@ vm_page_free_phys(struct vm_domain *vmd, vm_page_t vm_domain_free_assert_locked(vmd); - vm_domain_freecnt_adj(vmd, 1); #if VM_NRESERVLEVEL > 0 if (!vm_reserv_free_page(m)) #endif @@ -3203,24 +3173,28 @@ vm_page_free_phys_pglist(struct pglist *tq) { struct vm_domain *vmd; vm_page_t m; + int cnt; if (TAILQ_EMPTY(tq)) return; vmd = NULL; + cnt = 0; TAILQ_FOREACH(m, tq, listq) { if (vmd != vm_pagequeue_domain(m)) { if (vmd != NULL) { - vm_domain_free_wakeup(vmd); vm_domain_free_unlock(vmd); + vm_domain_freecnt_inc(vmd, cnt); + cnt = 0; } vmd = vm_pagequeue_domain(m); vm_domain_free_lock(vmd); } vm_page_free_phys(vmd, m); + cnt++; } if (vmd != NULL) { - vm_domain_free_wakeup(vmd); vm_domain_free_unlock(vmd); + vm_domain_freecnt_inc(vmd, cnt); } } @@ -3243,8 +3217,8 @@ vm_page_free_toq(vm_page_t m) vmd = vm_pagequeue_domain(m); vm_domain_free_lock(vmd); vm_page_free_phys(vmd, m); - vm_domain_free_wakeup(vmd); vm_domain_free_unlock(vmd); + vm_domain_freecnt_inc(vmd, 1); } /* Index: /usr/src/sys/vm/vm_pageout.c =================================================================== --- /usr/src/sys/vm/vm_pageout.c (revision 330873) +++ /usr/src/sys/vm/vm_pageout.c (working copy) @@ -1790,7 +1790,13 @@ vm_pageout_worker(void *arg) * The pageout daemon worker is never done, so loop forever. */ while (TRUE) { - vm_domain_free_lock(vmd); + vm_domain_pageout_lock(vmd); + /* + * We need to clear wanted before we check the limits. This + * prevents races with wakers who will check wanted after they + * reach the limit. + */ + atomic_store_int(&vmd->vmd_pageout_wanted, 0); /* * Might the page daemon need to run again? @@ -1801,7 +1807,7 @@ vm_pageout_worker(void *arg) * we have performed a level >= 1 (page reclamation) * scan, then sleep a bit and try again. */ - vm_domain_free_unlock(vmd); + vm_domain_pageout_unlock(vmd); if (pass > 1) pause("pwait", hz / VM_INACT_SCAN_RATE); } else { @@ -1809,12 +1815,18 @@ vm_pageout_worker(void *arg) * No, sleep until the next wakeup or until pages * need to have their reference stats updated. */ - vmd->vmd_pageout_wanted = false; if (mtx_sleep(&vmd->vmd_pageout_wanted, - vm_domain_free_lockptr(vmd), PDROP | PVM, + vm_domain_pageout_lockptr(vmd), PDROP | PVM, "psleep", hz / VM_INACT_SCAN_RATE) == 0) VM_CNT_INC(v_pdwakeups); } + /* Prevent spurious wakeups by ensuring that wanted is set. */ + atomic_store_int(&vmd->vmd_pageout_wanted, 1); + + /* + * Use the controller to calculate how many pages to free in + * this interval. + */ shortage = pidctrl_daemon(&vmd->vmd_pid, vmd->vmd_free_count); if (shortage && pass == 0) pass = 1; @@ -1970,10 +1982,14 @@ pagedaemon_wakeup(int domain) struct vm_domain *vmd; vmd = VM_DOMAIN(domain); - vm_domain_free_assert_unlocked(vmd); + vm_domain_pageout_assert_unlocked(vmd); + if (curproc == pageproc) + return; - if (!vmd->vmd_pageout_wanted && curthread->td_proc != pageproc) { - vmd->vmd_pageout_wanted = true; + if (atomic_fetchadd_int(&vmd->vmd_pageout_wanted, 1) == 0) { + vm_domain_pageout_lock(vmd); + atomic_store_int(&vmd->vmd_pageout_wanted, 1); wakeup(&vmd->vmd_pageout_wanted); + vm_domain_pageout_unlock(vmd); } } Index: /usr/src/sys/vm/vm_pagequeue.h =================================================================== --- /usr/src/sys/vm/vm_pagequeue.h (revision 330873) +++ /usr/src/sys/vm/vm_pagequeue.h (working copy) @@ -76,17 +76,31 @@ struct vm_pagequeue { #include struct sysctl_oid; +/* + * One vm_domain per-numa domain. Contains pagequeues, free page structures, + * and accounting. + * + * Lock Key: + * f vmd_free_mtx + * p vmd_pageout_mtx + * d vm_domainset_lock + * a atomic + * c const after boot +*/ struct vm_domain { struct vm_pagequeue vmd_pagequeues[PQ_COUNT]; struct mtx_padalign vmd_free_mtx; - struct vmem *vmd_kernel_arena; - u_int vmd_domain; /* Domain number. */ - u_int vmd_page_count; - long vmd_segs; /* bitmask of the segments */ + struct mtx_padalign vmd_pageout_mtx; + struct vmem *vmd_kernel_arena; /* (c) per-domain kva arena. */ + u_int vmd_domain; /* (c) Domain number. */ + u_int vmd_page_count; /* (c) Total page count. */ + long vmd_segs; /* (c) bitmask of the segments */ + u_int __aligned(CACHE_LINE_SIZE) vmd_free_count; /* (a,f) free page count */ + u_int vmd_pageout_deficit; /* (a) Estimated number of pages deficit */ + uint8_t vmd_pad[CACHE_LINE_SIZE - (sizeof(u_int) * 2)]; - /* Paging control variables, locked by domain_free_mtx. */ + /* Paging control variables, used within single threaded page daemon. */ struct pidctrl vmd_pid; /* Pageout controller. */ - u_int vmd_free_count; boolean_t vmd_oom; int vmd_oom_seq; int vmd_last_active_scan; @@ -94,11 +108,10 @@ struct vm_domain { struct vm_page vmd_marker; /* marker for pagedaemon private use */ struct vm_page vmd_inacthead; /* marker for LRU-defeating insertions */ - int vmd_pageout_pages_needed; /* page daemon waiting for pages? */ - int vmd_pageout_deficit; /* Estimated number of pages deficit */ - bool vmd_pageout_wanted; /* pageout daemon wait channel */ - bool vmd_minset; /* Are we in vm_min_domains? */ - bool vmd_severeset; /* Are we in vm_severe_domains? */ + int vmd_pageout_wanted; /* (a, p) pageout daemon wait channel */ + int vmd_pageout_pages_needed; /* (d) page daemon waiting for pages? */ + bool vmd_minset; /* (d) Are we in vm_min_domains? */ + bool vmd_severeset; /* (d) Are we in vm_severe_domains? */ int vmd_inactq_scans; enum { VM_LAUNDRY_IDLE = 0, @@ -142,6 +155,17 @@ extern struct vm_domain vm_dom[MAXMEMDOM]; #define vm_domain_free_unlock(d) \ mtx_unlock(vm_domain_free_lockptr((d))) +#define vm_domain_pageout_lockptr(d) \ + (&(d)->vmd_pageout_mtx) +#define vm_domain_pageout_assert_locked(n) \ + mtx_assert(vm_domain_pageout_lockptr((n)), MA_OWNED) +#define vm_domain_pageout_assert_unlocked(n) \ + mtx_assert(vm_domain_pageout_lockptr((n)), MA_NOTOWNED) +#define vm_domain_pageout_lock(d) \ + mtx_lock(vm_domain_pageout_lockptr((d))) +#define vm_domain_pageout_unlock(d) \ + mtx_unlock(vm_domain_pageout_lockptr((d))) + static __inline void vm_pagequeue_cnt_add(struct vm_pagequeue *pq, int addend) { @@ -155,6 +179,7 @@ vm_pagequeue_cnt_add(struct vm_pagequeue *pq, int #define vm_pagequeue_cnt_dec(pq) vm_pagequeue_cnt_add((pq), -1) void vm_domain_set(struct vm_domain *vmd); +void vm_domain_clear(struct vm_domain *vmd); int vm_domain_available(struct vm_domain *vmd, int req, int npages); /* @@ -221,18 +246,40 @@ vm_laundry_target(struct vm_domain *vmd) return (vm_paging_target(vmd)); } -static inline u_int -vm_domain_freecnt_adj(struct vm_domain *vmd, int adj) +void pagedaemon_wakeup(int domain); + +static inline void +vm_domain_freecnt_inc(struct vm_domain *vmd, int adj) { - u_int ret; + u_int old, new; - vm_domain_free_assert_locked(vmd); - ret = vmd->vmd_free_count += adj; - if ((!vmd->vmd_minset && vm_paging_min(vmd)) || - (!vmd->vmd_severeset && vm_paging_severe(vmd))) + old = atomic_fetchadd_int(&vmd->vmd_free_count, adj); + new = old + adj; + /* + * Only update bitsets on transitions. Notice we short-circuit the + * rest of the checks if we're above min already. + */ + if (old < vmd->vmd_free_min && (new >= vmd->vmd_free_min || + (old < vmd->vmd_free_severe && new >= vmd->vmd_free_severe) || + (old < vmd->vmd_pageout_free_min && + new >= vmd->vmd_pageout_free_min))) + vm_domain_clear(vmd); +} + +static inline void +vm_domain_freecnt_dec(struct vm_domain *vmd, int adj) +{ + u_int old, new; + + old = atomic_fetchadd_int(&vmd->vmd_free_count, -adj); + new = old - adj; + KASSERT(new >= 0, ("vm_domain_freecnt_dec: free count underflow")); + if (vm_paging_needed(vmd, new) && !vm_paging_needed(vmd, old)) + pagedaemon_wakeup(vmd->vmd_domain); + /* Only update bitsets on transitions. */ + if ((old >= vmd->vmd_free_min && new < vmd->vmd_free_min) || + (old >= vmd->vmd_free_severe && new < vmd->vmd_free_severe)) vm_domain_set(vmd); - - return (ret); } Index: /usr/src/sys/vm/vm_reserv.c =================================================================== --- /usr/src/sys/vm/vm_reserv.c (revision 330873) +++ /usr/src/sys/vm/vm_reserv.c (working copy) @@ -593,7 +593,7 @@ vm_reserv_extend_contig(int req, vm_object_t objec } for (i = 0; i < npages; i++) vm_reserv_populate(rv, index + i); - vm_domain_freecnt_adj(vmd, -npages); + vm_domain_freecnt_dec(vmd, npages); out: vm_domain_free_unlock(vmd); return (m); @@ -789,7 +789,7 @@ vm_reserv_extend(int req, vm_object_t object, vm_p struct vm_domain *vmd; vm_page_t m, msucc; vm_reserv_t rv; - int index, free_count; + int index; VM_OBJECT_ASSERT_WLOCKED(object); @@ -822,14 +822,10 @@ vm_reserv_extend(int req, vm_object_t object, vm_p m = NULL; if (m != NULL) { vm_reserv_populate(rv, index); - free_count = vm_domain_freecnt_adj(vmd, -1); - } else - free_count = vmd->vmd_free_count; + vm_domain_freecnt_dec(vmd, 1); + } vm_domain_free_unlock(vmd); - if (vm_paging_needed(vmd, free_count)) - pagedaemon_wakeup(domain); - return (m); } -- Test scenario: misc/fdgrowtable.sh what /usr/src/sys/amd64/compile/PHO/kernel.debug /usr/src/sys/amd64/compile/PHO/kernel.debug: ls -l /usr/src/sys/amd64/compile/PHO/kernel.debug /var/crash/vmcore.63 -rwxr-xr-x 1 root wheel 82418248 Mar 13 21:22 /usr/src/sys/amd64/compile/PHO/kernel.debug -rw------- 1 root wheel 1361063936 Mar 14 07:21 /var/crash/vmcore.63 vmstat -N /usr/src/sys/amd64/compile/PHO/kernel.debug -M /var/crash/vmcore.63 -m Type InUse MemUse HighUse Requests Size(s) feeder 7 1K - 7 32 CAM I/O Scheduler 1 1K - 1 128 CAM queue 29 9K - 89 16,32,512 UART 6 5K - 6 16,1024 USB 53 72K - 58 16,128,256,512,1024,2048,4096 USBdev 39 6K - 39 32,64,128,256,512 CAM dev queue 13 1K - 13 64 vtbuf 24 1968K - 46 4096 vt 11 6K - 11 512 DEVFS3 150 38K - 466 256 DEVFS1 127 64K - 241 512 DEVFS 17 1K - 18 16,128 DEVFSP 1 1K - 2 64 NFSD V4client 1 1K - 1 256 NFSD lckfile 1 1K - 1 256 NFSD session 1 1K - 1 1024 pfs_nodes 20 5K - 20 256 GEOM 209 1196K - 9148 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768 raid_data 0 0K - 1752 32,128,256 isadev 7 1K - 7 128 SCSI ENC 46 199K - 10652 16,64,256,2048,32768 ddb_capture 1 64K - 1 65536 cdev 3 1K - 3 256 filedesc 14 113K - 215497 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 sigio 0 0K - 18 64 filecaps 1 1K - 1968 16,32,64 kdtrace 332 62K - 46942 64,256 kenv 146 14K - 160 16,32,64,128,8192 kqueue 112 7K - 23410 64,256 proc-args 131 5K - 9567 16,32,64,128,256 hhook 13 4K - 13 256 ithread 162 30K - 162 32,128,256 KTRACE 100 13K - 100 128 linker 217 37K - 231 16,32,64,128,256,512,1024,2048,4096,8192 lockf 11 2K - 358 64,128 loginclass 3 1K - 3 64 devbuf 17817 34761K - 38504 16,32,64,128,256,512,1024,2048,4096,8192 temp 80 18K - 35803809 16,32,64,128,256,512,1024,2048,4096,16384,32768,65536 module 516 65K - 517 128 mtx_pool 3 84K - 3 8192 osd 3 1K - 9 16,32,64,128,256 pmchooks 1 1K - 1 128 pgrp 21 3K - 162 128 session 11 2K - 122 128 proc 2 128K - 2 65536 subproc 497 641K - 23794 512,4096 cred 37 10K - 1783 256 acpiintr 1 1K - 1 64 acpica 24513 2400K - 359710 16,32,64,128,256,512,1024,2048,4096 acpitask 1 64K - 1 65536 plimit 17 5K - 1802 256 uidinfo 5 17K - 89 128,16384 dumper 1 1K - 1 512 kbdmux 7 22K - 7 16,512,1024,2048,16384 sysctl 0 0K - 3 64 sysctloid 7407 377K - 7550 16,32,64,128 sysctltmp 0 0K - 406 16,64,256,512,1024 LED 40 3K - 40 16,128 tidhash 1 128K - 1 callout 13 4232K - 13 umtx 1040 130K - 1040 128 p1003.1b 1 1K - 1 16 SWAP 1 4372K - 19 bus 1658 181K - 23404 16,32,64,128,256,1024 bus-sc 124 1911K - 17623 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 devstat 8 17K - 8 32,4096 eventhandler 124 11K - 124 64,128 gtaskqueue 78 32K - 78 16,32,256,8192 kobj 354 1416K - 2399 4096 Per-cpu 1 1K - 1 32 rman 307 35K - 665 16,32,128 sbuf 0 0K - 9332 16,32,64,128,256,512,1024,2048,4096,8192,16384 toponodes 32 4K - 32 128 md_disk 0 0K - 25 4096 taskqueue 30 4K - 30 16,32,128,256 terminal 11 3K - 11 256 Unitno 22 2K - 13328 32,64 vmem 3 138K - 11 2048,4096,8192 Witness 1539 3282K - 1539 2048,16384 ioctlops 0 0K - 99 256,512,1024 select 33 5K - 33 128 iov 0 0K - 36134949 16,32,64,128,256,512 msg 4 30K - 4 2048,4096,8192,16384 sem 4 106K - 4 2048,4096 shm 1 32K - 1 32768 tty 16 16K - 16 1024 pts 2 1K - 2 256 mbuf_tag 0 0K - 66 32 shmfd 1 8K - 1 8192 soname 5 1K - 30654 16,32,128 pcb 21 4247K - 691 16,32,128,1024,2048,8192 vfscache 4 8289K - 4 256,32768,65536 cl_savebuf 0 0K - 1341 64 vfs_hash 1 4096K - 1 vnodes 1 1K - 1 256 mount 31 2K - 309 16,32,64,128,256 statfs 0 0K - 625 4096 vnodemarker 0 0K - 5715 1024 chacha20random 1 4K - 1 4096 BPF 7 9K - 11 128,512,4096 ifnet 4 7K - 4 128,2048 ifaddr 50 14K - 51 16,32,64,128,256,512,2048,4096 ether_multi 47 3K - 52 16,32,64 clone 8 1K - 8 128 ipsec 1 1K - 1 256 lltable 15 5K - 15 256,512 iflib 128 793K - 136 64,128,1024,8192,16384,32768 routetbl 26 4K - 29 32,64,128,512 vnet 1 1K - 1 64 vnet_data 1 64K - 1 65536 vnet_data_free 1 1K - 1 32 igmp 3 1K - 3 128 in_multi 2 1K - 3 256 encap_export_host 2 2K - 2 1024 sctp_a_it 0 0K - 4 16 sctp_vrf 1 1K - 1 64 sctp_ifa 5 1K - 5 128 sctp_ifn 2 1K - 2 128 sctp_iter 0 0K - 4 256 tfo_ccache 1 128K - 1 hostcache 1 32K - 1 32768 LRO 24 480K - 24 8192,32768 tcpfunc 1 1K - 1 64 syncache 1 68K - 1 in6_mfilter 1 1K - 1 1024 in6_multi 27 4K - 27 32,256 ip6_moptions 2 1K - 2 32,256 mld 3 1K - 3 128 ip6ndp 5 1K - 6 64,256 inpcbpolicy 13 1K - 1438 32 secasvar 1 1K - 1 1024 sahead 1 1K - 1 1024 ipsecpolicy 2 2K - 2 256,1024 ipsec-saq 2 2K - 2 1024 NLM 0 0K - 1 32 crypto 2 3K - 2 512,2048 rpc 5 10K - 196 16,32,64,128,256,512,1024,4096 audit_evclass 230 8K - 286 32 pagedep 2 1024K - 684 256 inodedep 2 8192K - 3917 512 bmsafemap 2 16K - 4580 256,8192 newblk 2 16384K - 9266213 256 indirdep 0 0K - 3866 128,32768 freefrag 0 0K - 2521 128 freeblks 0 0K - 1608 256 freefile 0 0K - 1099 64 diradd 0 0K - 2488 128 mkdir 0 0K - 208 128 dirrem 0 0K - 1891 128 newdirblk 0 0K - 104 64 freework 2 1K - 7562 32,128 freedep 0 0K - 1425 64 jaddref 0 0K - 1723 128 jremref 0 0K - 1138 128 jnewblk 0 0K - 8970486 128 jfreefrag 0 0K - 2433 128 jseg 0 0K - 10761 128 jsegdep 0 0K - 8975780 64 sbdep 0 0K - 799 64 savedino 0 0K - 1218 256 jblocks 0 0K - 32 128,256 softdep 2 1K - 18 512 ufs_dirhash 15 8K - 797 16,32,64,128,512 ufs_quota 1 4096K - 1 ufs_mount 6 57K - 72 512,4096,8192,32768 vm_pgdata 1 1K - 19 128 UMAHash 7 108K - 26 512,1024,2048,4096,8192,16384,32768,65536 memdesc 1 4K - 1 4096 pci_link 16 2K - 16 64,128 acpisem 31 4K - 31 128 acpidev 183 12K - 183 64 CAM SIM 13 4K - 13 256 CAM XPT 48 5K - 225 16,32,64,128,256,512,1024,2048,65536 apmdev 1 1K - 1 128 madt_table 0 0K - 2 128,4096 entropy 1 1K - 46 32,4096 CAM DEV 16 32K - 35 2048 CAM CCB 0 0K - 2590815 2048 CAM path 19 1K - 112 32 CAM periph 8 2K - 32 16,32,64,128,256 io_apic 2 4K - 2 2048 local_apic 1 16K - 1 16384 MCA 35 9K - 35 128,512 cpus 2 1K - 2 64 msi 24 3K - 24 128 nexusdev 5 1K - 5 16 acpi_perf 12 6K - 12 512 pmc 1130 11809K - 1130 16,128,256,512,1024,4096,8192,65536 vmstat -N /usr/src/sys/amd64/compile/PHO/kernel.debug -M /var/crash/vmcore.63 -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 280, 0, 208, 0, 208, 0, 0 UMA Zones: 1160, 0, 225, 0, 225, 0, 0 UMA Slabs: 112, 0, 6132, 2968, 296480, 0, 0 UMA Hash: 256, 0, 3, 12, 10, 0, 0 4 Bucket: 32, 0, 36, 2196, 47503,6746, 0 6 Bucket: 48, 0, 47, 1115, 10739, 600, 0 8 Bucket: 64, 0, 0, 0, 498548, 829, 0 12 Bucket: 96, 0, 0, 0, 392773,4060, 0 16 Bucket: 128, 0, 0, 0, 2204504,26838, 0 32 Bucket: 256, 0, 0, 0, 2292636,52118, 0 64 Bucket: 512, 0, 789, 443, 1461234,662768, 0 128 Bucket: 1024, 0, 0, 0, 1144898,8640854, 0 256 Bucket: 2048, 0, 0, 0, 864707,20738612, 0 vmem: 1792, 0, 1, 1, 1, 0, 0 vmem btag: 56, 0, 11137, 3134, 17947, 103, 0 VM OBJECT: 256, 0, 2215, 1655, 232939, 0, 0 RADIX NODE: 144, 0, 538739, 1342,65318634, 0, 0 MAP: 232, 0, 3, 65, 3, 0, 0 KMAP ENTRY: 120, 0, 6, 93, 6, 0, 0 MAP ENTRY: 120, 0, 2187, 2367, 774017, 0, 0 VMSPACE: 2512, 0, 73, 223, 23352, 0, 0 fakepg: 104, 0, 0, 152, 36, 0, 0 64 pcpu: 8, 0, 2635, 949, 2905, 0, 0 mt_zone: 16400, 0, 415, 0, 415, 0, 0 16: 16, 0, 24, 3213, 81, 0, 0 16: 16, 0, 472, 1271, 159419, 0, 0 16: 16, 0, 280, 2459, 13998, 0, 0 16: 16, 0, 2496, 2484, 9659, 0, 0 16: 16, 0, 212, 1780, 9367, 0, 0 16: 16, 0, 42, 1452, 208, 0, 0 16: 16, 0, 0, 0, 0, 0, 0 16: 16, 0, 40, 1454, 844, 0, 0 32: 32, 0, 137, 1475, 7615, 0, 0 32: 32, 0, 513, 851, 8731, 0, 0 32: 32, 0, 180, 1308, 10027, 0, 0 32: 32, 0, 2849, 1367, 5199, 0, 0 32: 32, 0, 388, 1348, 9104, 0, 0 32: 32, 0, 16, 1224, 13324, 0, 0 32: 32, 0, 8, 240, 9, 0, 0 32: 32, 0, 32, 588, 338, 0, 0 64: 64, 0, 44, 390, 9663, 0, 0 64: 64, 0, 9317, 417,36129788, 0, 0 64: 64, 0, 649, 715, 2941, 0, 0 64: 64, 0, 683, 867, 9001992, 0, 0 64: 64, 0, 8721, 1013,35794222, 0, 0 64: 64, 0, 134, 734, 224, 0, 0 64: 64, 0, 1, 61, 1204, 0, 0 64: 64, 0, 13, 297, 221, 0, 0 128: 128, 0, 40, 487, 293, 0, 0 128: 128, 0, 14340, 230, 175372, 0, 0 128: 128, 0, 607, 354, 19685, 0, 0 128: 128, 0, 2022, 396, 34422, 0, 0 128: 128, 0, 531, 368, 41817, 0, 0 128: 128, 0, 1689, 481, 2012, 0, 0 128: 128, 0, 1, 30, 8973026, 0, 0 128: 128, 0, 23, 287, 8708, 0, 0 256: 256, 0, 70, 185, 3034, 0, 0 256: 256, 0, 64, 71, 17576, 0, 0 256: 256, 0, 387, 123, 9084, 0, 0 256: 256, 0, 48, 237, 2101, 0, 0 256: 256, 0, 669, 231, 34343, 0, 0 256: 256, 0, 9, 111, 31, 0, 0 256: 256, 0, 2, 28, 9266197, 0, 0 256: 256, 0, 52, 128, 1777, 0, 0 512: 512, 0, 12, 72, 12, 0, 0 512: 512, 0, 23, 33, 52, 0, 0 512: 512, 0, 2, 12, 3975, 0, 0 512: 512, 0, 18, 45, 65, 0, 0 512: 512, 0, 259, 77, 8451, 0, 0 512: 512, 0, 415, 103, 4334, 0, 0 512: 512, 0, 11, 10, 11, 0, 0 512: 512, 0, 2, 12, 2919, 0, 0 1024: 1024, 0, 7, 9, 7, 0, 0 1024: 1024, 0, 44, 20, 6582, 0, 0 1024: 1024, 0, 20, 36, 2645, 0, 0 1024: 1024, 0, 4, 8, 182, 0, 0 1024: 1024, 0, 40, 16, 47348, 0, 0 1024: 1024, 0, 6, 10, 5725, 0, 0 1024: 1024, 0, 1, 3, 1, 0, 0 1024: 1024, 0, 2, 6, 288, 0, 0 2048: 2048, 0, 3, 1, 3, 0, 0 2048: 2048, 0, 1559, 15, 2592396, 0, 0 2048: 2048, 0, 15, 3, 475, 0, 0 2048: 2048, 0, 4, 2, 26, 0, 0 2048: 2048, 0, 91, 7, 8310, 0, 0 2048: 2048, 0, 5, 1, 8, 0, 0 2048: 2048, 0, 1, 1, 4, 0, 0 2048: 2048, 0, 2, 2, 294, 0, 0 4096: 4096, 0, 1025, 0, 1025, 0, 0 4096: 4096, 0, 1, 0, 13, 0, 0 4096: 4096, 0, 7, 0, 1207, 0, 0 4096: 4096, 0, 356, 0, 3048, 0, 0 4096: 4096, 0, 40, 0, 16105, 0, 0 4096: 4096, 0, 133, 0, 23456, 0, 0 4096: 4096, 0, 12, 0, 71, 0, 0 4096: 4096, 0, 0, 0, 132, 0, 0 8192: 8192, 0, 6, 0, 6, 0, 0 8192: 8192, 0, 38, 0, 38, 0, 0 8192: 8192, 0, 12, 0, 1224, 0, 0 8192: 8192, 0, 2, 0, 24, 0, 0 8192: 8192, 0, 14, 0, 15974, 0, 0 8192: 8192, 0, 5, 0, 39, 0, 0 8192: 8192, 0, 1, 0, 3, 0, 0 8192: 8192, 0, 135, 0, 1351, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 6, 0, 6, 0, 0 16384: 16384, 0, 1, 0, 1845, 0, 0 16384: 16384, 0, 1, 0, 23, 0, 0 16384: 16384, 0, 9, 0, 15975, 0, 0 16384: 16384, 0, 1, 0, 3, 0, 0 16384: 16384, 0, 0, 0, 0, 0, 0 16384: 16384, 0, 0, 0, 12, 0, 0 32768: 32768, 0, 7, 0, 2367, 0, 0 32768: 32768, 0, 13, 0, 13, 0, 0 32768: 32768, 0, 0, 0, 2058, 0, 0 32768: 32768, 0, 1, 0, 1, 0, 0 32768: 32768, 0, 12, 0, 15980, 0, 0 32768: 32768, 0, 2, 0, 4, 0, 0 32768: 32768, 0, 0, 0, 0, 0, 0 32768: 32768, 0, 4, 0, 8, 0, 0 65536: 65536, 0, 26, 0, 26, 0, 0 65536: 65536, 0, 1, 0, 45, 0, 0 65536: 65536, 0, 3, 0, 1084, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 1, 0, 16029, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 1, 0, 1, 0, 0 65536: 65536, 0, 0, 0, 0, 0, 0 SLEEPQUEUE: 88, 0, 521, 378, 521, 0, 0 Files: 80, 0, 56, 1855, 89886, 0, 0 filedesc0: 1104, 0, 106, 17, 23409, 0, 0 rl_entry: 40, 0, 346, 2228, 346, 0, 0 TURNSTILE: 136, 0, 521, 459, 521, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 umtx_shm: 88, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1336, 0, 111, 273, 23408, 0, 0 THREAD: 1368, 0, 492, 28, 508, 0, 0 cpuset: 104, 0, 142, 352, 207, 0, 0 domainset: 32, 0, 0, 0, 0, 0, 0 audit_record: 1280, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 13008735, 26, 3382, 14094, 0, 0 mbuf: 256, 13008735, 6222, 3382,179537340, 0, 0 mbuf_cluster: 2048, 2560000, 6169, 49, 197112, 0, 0 mbuf_jumbo_page: 4096, 1016306, 0, 0, 14, 0, 0 mbuf_jumbo_9k: 9216, 903381, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 677536, 0, 0, 0, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 g_bio: 408, 0, 0, 0,17498534, 10, 0 ttyinq: 160, 0, 210, 54, 555, 0, 0 ttyoutq: 256, 0, 109, 71, 290, 0, 0 nvme_request: 128, 0, 0, 0, 0, 0, 0 cryptop: 128, 0, 0, 0, 0, 0, 0 cryptodesc: 72, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 632, 0, 925, 83, 3285, 0, 0 VNODEPOLL: 120, 0, 0, 0, 0, 0, 0 BUF TRIE: 144, 0, 0, 105948, 6371070, 0, 0 S VFS Cache: 108, 0, 922, 1108, 10520, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 0, 0, 0, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 9, 3, 157244, 0, 0 rentr: 24, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 0, 0, 1610, 0, 0 NCLNODE: 744, 0, 0, 0, 0, 0, 0 procdesc: 136, 0, 0, 0, 4, 0, 0 Mountpoints: 2896, 0, 3, 9, 19, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0 AIOP: 32, 0, 0, 0, 0, 0, 0 AIOCB: 752, 0, 0, 0, 0, 0, 0 AIOLIO: 280, 0, 0, 0, 0, 0, 0 pipe: 760, 0, 4, 11, 4583, 0, 0 ksiginfo: 112, 0, 360, 410, 5778, 0, 0 itimer: 352, 0, 0, 0, 0, 0, 0 KNOTE: 160, 0, 0, 0, 0, 0, 0 socket: 856, 1044004, 24, 248, 6015, 0, 0 IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0 unpcb: 232, 1044004, 10, 92, 4517, 0, 0 ipq: 56, 80017, 0, 0, 0, 0, 0 udp_inpcb: 424, 1044009, 7, 20, 269, 0, 0 udpcb: 32, 1044080, 7, 365, 269, 0, 0 tcp_inpcb: 424, 1044009, 6, 30, 1166, 0, 0 tcpcb: 904, 1044004, 6, 10, 1166, 0, 0 tcptw: 88, 27810, 0, 0, 569, 0, 0 syncache: 176, 15378, 0, 0, 571, 0, 0 hostcache: 96, 15375, 0, 0, 1, 0, 0 sackhole: 32, 0, 0, 0, 0, 0, 0 tfo: 4, 0, 0, 0, 0, 0, 0 tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0 tcpreass: 40, 160083, 0, 0, 0, 0, 0 sctp_ep: 1216, 1044006, 0, 0, 0, 0, 0 sctp_asoc: 2408, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 0, 4, 0, 0 sctp_raddr: 736, 80000, 0, 0, 0, 0, 0 sctp_chunk: 152, 400010, 0, 0, 0, 0, 0 sctp_readq: 152, 400010, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400015, 0, 0, 0, 0, 0 sctp_asconf: 40, 400059, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 udplite_inpcb: 424, 1044009, 0, 0, 0, 0, 0 ripcb: 424, 1044009, 0, 0, 3, 0, 0 rtentry: 208, 0, 12, 102, 13, 0, 0 selfd: 64, 0, 37, 707, 105886, 0, 0 swpctrie: 144, 4065255, 1327, 9554, 317353, 1, 0 swblk: 136, 4065249, 8213, 85312, 2886252, 1, 0 FFS inode: 160, 0, 876, 396, 3169, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 876, 264, 3169, 0, 0