Out of free pages, vmcore.126 KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #1 r355490M: Sat Dec 7 20:39:23 CET 2019 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 9.0.0 (tags/RELEASE_900/final 372316) (based on LLVM 9.0.0) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 7 boot pages left startup_alloc from "UMA Zones", 6 boot pages left startup_alloc from "UMA Zones", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "UMA Zones", 3 boot pages left startup_alloc from "UMA Zones", 2 boot pages left startup_alloc from "UMA Zones", 1 boot pages left Entering uma_startup1 with 0 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33164136448 (31627 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 7 2 6 11 5 4 8 3 9 10 Timecounter "TSC-low" frequency 1750033694 Hz quality 1000 random: entropy device external interface WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0. kbd1 at kbdmux0 000.000055 [4336] netmap_init netmap: loaded module [ath_hal] loaded nexus0 vtvga0: cryptosoft0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb433000-0xfb4333ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb432000-0xfb4327ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 p4tcc0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec Expensive timeout(9) function: 0xffffffff804d9790(0xfffffe00d1356000) 0.004917582 s ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> ses0: pass0 in 'Slot 01', SATA Slot: scbus1 target 0 IAP/4/48/0x3ff IAF/3/48/0x67 Expensive timeout(9) function: 0xffffffff80a34d70(0xffffffff818e6608) 0.007403282 s Trying to mount root from ufs:/dev/gpt/work [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. ugen0.1: <0x8086 XHCI root HUB> at usbus0 ugen2.1: at usbus2 ugen1.1: at usbus1 uhub0 on usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1 on usbus1 uhub2 on usbus2 uhub2: on usbus2 uhub1: on usbus1 WARNING: /work was not properly dismounted Interface IP-Address Broadcast ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 . lo0: link state changed to UP Starting Network: lo0 igb0 igb1. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 igb0: flags=8843 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:da inet6 fe80::ec4:7aff:fea8:cdda%igb0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect status: no carrier nd6 options=21 igb1: flags=8802 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Autoloading module: ioat.ko ioat0: mem 0xfb42c000-0xfb42ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0xfb428000-0xfb42bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0xfb424000-0xfb427fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0xfb420000-0xfb423fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0xfb41c000-0xfb41ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0xfb418000-0xfb41bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0xfb414000-0xfb417fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0xfb410000-0xfb413fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8802 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid.ko Autoloading module: ums.ko ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192if_delmulti_locked: detaching ifnet instance 0xfffff80005050000 .168.5.1 bound to 192.168.5.171 -- renewal in 300 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Setting date via ntp. 8 Dec 13:47:05 ntpdate[2058]: step time server 192.168.5.1 offset 0.186204 sec Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. Clearing /tmp (X related). Starting statd. Starting lockd. Recovering vi editor sessions:. fscklock order reversal: 1st 0xfffff80023b9d2e0 ufs (ufs) @ kern/vfs_mount.c:1603 2nd 0xfffff80023aaa558 devfs (devfs) @ ufs/ffs/ffs_softdep.c:1927 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b97b88 at lockmgr_lock_fast_path+0x178 #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80ec3a4d at softdep_flushworklist+0x5d #6 0xffffffff80ee4a36 at ffs_sync+0x406 #7 0xffffffff80c96c76 at dounmount+0x466 #8 0xffffffff80c967be at sys_unmount+0x2ae #9 0xffffffff8106d901 at amd64_syscall+0x2f1 #10 0xffffffff81043dd0 at fast_syscall_common+0x101 ** /dev/gpt/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 269 files, 30826 used, 1234581 free (149 frags, 154304 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Updating motd:. Mounting late filesystems:. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Starting powerd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Sun Dec 8 13:47 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: root Dec 8 13:47:15 mercat1 login[2340]: ROOT LOGIN (root) ON ttyu1 Last login: Sun Dec 8 07:16:51 on ttyu1 FreeBSD 13.0-CURRENT (PHO) #1 r355490M: Sat Dec 7 20:39:23 CET 2019 root@mercat1:~ # Dec 8 13:47:30 mercat1 su[2356]: pho to root on /dev/pts/0 20191208 13:47:47 all (1/679): 1st.sh 20191208 13:47:49 all (2/679): 256m.sh 20191208 13:47:50 all (3/679): audit.sh 20191208 13:47:52 all (4/679): beneath3.sh 20191208 13:47:55 all (5/679): buildworld.sh 20191208 13:47:57 all (6/679): buildworld2.sh 20191208 13:47:58 all (7/679): cluster.sh 20191208 13:48:00 all (8/679): cpuset.sh 20191208 13:48:01 all (9/679): crossmp2.sh 20191208 13:48:02 all (10/679): crossmp6.sh 20191208 13:48:04 all (11/679): datagram2.sh 20191208 13:48:06 all (12/679): datamove.sh freework leak: 20/41. 20191208 13:48:08 all (13/679): datamove3.sh 20191208 13:48:09 all (14/679): devfs2.sh 20191208 13:48:10 all (15/679): devfs5.sh 20191208 13:48:12 all (16/679): dtrace_fault.sh 20191208 13:48:13 all (17/679): dumpfs.sh newblk leak: 288/312. 20191208 13:48:15 all (18/679): dup2.sh 20191208 13:48:16 all (19/679): execi386.sh 20191208 13:48:18 all (20/679): extattr_set_fd.sh 20191208 13:48:20 all (21/679): extattrctl.sh lock order reversal: 1st 0xfffff80040ce0638 ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:119 2nd 0xfffff8078319d068 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1718 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b9816f at lockmgr_xlock_hard+0x6f #3 0xffffffff80b98cfd at __lockmgr_args+0x60d #4 0xffffffff80ee1b41 at ffs_vgetf+0x161 #5 0xffffffff80ef4d6b at ufs_root+0x1b #6 0xffffffff80ca3798 at vfs_cache_root_fallback+0x128 #7 0xffffffff80ca3515 at vfs_cache_root+0x1b5 #8 0xffffffff80eecdcf at ufs_extattr_autostart+0x5f #9 0xffffffff80ee3f26 at ffs_mount+0x1c66 #10 0xffffffff80c95951 at vfs_domount+0x811 #11 0xffffffff80c942f1 at vfs_donmount+0x911 #12 0xffffffff80c939a9 at sys_nmount+0x69 #13 0xffffffff8106d901 at amd64_syscall+0x2f1 #14 0xffffffff81043dd0 at fast_syscall_common+0x101 20191208 13:48:21 all (22/679): fcntl3.sh 20191208 13:48:23 all (23/679): fifo.sh 20191208 13:48:24 all (24/679): forkbomb.sh 20191208 13:48:26 all (25/679): fsck2.sh 20191208 13:48:27 all (26/679): fsck3.sh 20191208 13:48:29 all (27/679): fsck4.sh 20191208 13:48:30 all (28/679): getrandom.sh 20191208 13:48:31 all (29/679): gnop3.sh 20191208 13:48:33 all (30/679): holdcnt0.sh 20191208 13:48:34 all (31/679): holdcnt02.sh 20191208 13:48:36 all (32/679): holdcnt03.sh 20191208 13:48:37 all (33/679): holdcnt04.sh 20191208 13:48:38 all (34/679): holdcnt05.sh 20191208 13:48:40 all (35/679): isofs2.sh 20191208 13:48:41 all (36/679): jail.sh 20191208 13:48:43 all (37/679): jail2.sh 20191208 13:48:44 all (38/679): jail3.sh 20191208 13:48:46 all (39/679): jail4.sh 20191208 13:48:47 all (40/679): kern_umtx_inf_loop.sh 20191208 13:48:49 all (41/679): kevent10.sh 20191208 13:48:50 all (42/679): kevent3.sh 20191208 13:48:52 all (43/679): ldt.sh 20191208 13:48:53 all (44/679): ldt2.sh 20191208 13:48:55 all (45/679): libMicro.sh 20191208 13:48:56 all (46/679): linux.sh 20191208 13:48:58 all (47/679): lockd.sh 20191208 13:48:59 all (48/679): lockf.sh 20191208 13:49:00 all (49/679): lookup_shared.sh 20191208 13:49:02 all (50/679): mac.sh 20191208 13:49:03 all (51/679): mac_chkexec.sh 20191208 13:49:05 all (52/679): machipc.sh 20191208 13:49:06 all (53/679): machipc2.sh 20191208 13:49:07 all (54/679): maxmemdom.sh 20191208 13:49:09 all (55/679): maxproc.sh 20191208 13:49:10 all (56/679): md4.sh 20191208 13:49:12 all (57/679): md5.sh 20191208 13:49:13 all (58/679): md6.sh 20191208 13:49:15 all (59/679): mdconfig.sh 20191208 13:49:16 all (60/679): mdconfig3.sh 20191208 13:49:18 all (61/679): memguard4.sh 20191208 13:49:19 all (62/679): mkfifo5.sh 20191208 13:49:21 all (63/679): mkfifo7.sh mount leak: 16/47 20191208 13:49:22 all (64/679): mlockall5.sh 20191208 13:49:24 all (65/679): mmap.sh 20191208 13:49:25 all (66/679): mmap12.sh 20191208 13:49:27 all (67/679): mmap16.sh 20191208 13:49:28 all (68/679): mmap17.sh 20191208 13:49:30 all (69/679): mmap20.sh 20191208 13:49:31 all (70/679): mmap24.sh 20191208 13:49:33 all (71/679): mmap30.sh 20191208 13:49:34 all (72/679): mmap33.sh 20191208 13:49:36 all (73/679): mmap35.sh 20191208 13:49:37 all (74/679): mmap36.sh 20191208 13:49:38 all (75/679): mount.sh 20191208 13:49:40 all (76/679): mprotect.sh 20191208 13:49:41 all (77/679): msdos2.sh 20191208 13:49:43 all (78/679): msdos3.sh 20191208 13:49:44 all (79/679): msdos7.sh lock order reversal: 1st 0xfffff8078350c558 msdosfs (msdosfs) @ kern/vfs_syscalls.c:3424 2nd 0xfffff8015015a068 devfs (devfs) @ fs/msdosfs/msdosfs_vnops.c:853 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b97b88 at lockmgr_lock_fast_path+0x178 #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80a84d19 at msdosfs_fsync+0x49 #6 0xffffffff810e985a at VOP_FSYNC_APV+0xaa #7 0xffffffff80cae519 at kern_fsync+0x199 #8 0xffffffff8106d901 at amd64_syscall+0x2f1 #9 0xffffffff81043dd0 at fast_syscall_common+0x101 20191208 13:49:46 all (80/679): msdos8.sh 20191208 13:49:47 all (81/679): msetdomain.sh 20191208 13:49:49 all (82/679): nanosleep.sh 20191208 13:49:50 all (83/679): nbufkv.sh 20191208 13:49:52 all (84/679): nfs.sh 20191208 13:49:53 all (85/679): nfs12.sh 20191208 13:49:54 all (86/679): nfs14.sh 20191208 13:49:56 all (87/679): nfs15.sh 20191208 13:49:57 all (88/679): nfs15lockd.sh 20191208 13:49:59 all (89/679): nfs15lockd3.sh 20191208 13:50:00 all (90/679): nfs3.sh 20191208 13:50:01 all (91/679): nfs4.sh 20191208 13:50:03 all (92/679): nfs7.sh 20191208 13:50:04 all (93/679): nfs8.sh 20191208 13:50:06 all (94/679): nfs9.sh 20191208 13:50:07 all (95/679): nfs_halfpage.sh 20191208 13:50:08 all (96/679): nfs_halfpage2.sh 20191208 13:50:10 all (97/679): nfsdelegation.sh 20191208 13:50:11 all (98/679): nfsdepth.sh 20191208 13:50:13 all (99/679): nfsrename.sh 20191208 13:50:14 all (100/679): nfssillyrename.sh 20191208 13:50:15 all (101/679): nullfs13.sh 20191208 13:50:17 all (102/679): nullfs14.sh 20191208 13:50:18 all (103/679): nullfs15.sh 20191208 13:50:20 all (104/679): nullfs21.sh 20191208 13:50:21 all (105/679): nullfs23.sh 20191208 13:50:23 all (106/679): nullfs4.sh 20191208 13:50:24 all (107/679): nullfs6.sh 20191208 13:50:26 all (108/679): nullfs7.sh 20191208 13:50:27 all (109/679): nullfs9.sh 20191208 13:50:29 all (110/679): numa.sh 20191208 13:50:30 all (111/679): oom.sh 20191208 13:50:31 all (112/679): open.sh 20191208 13:50:33 all (113/679): pagefault.sh 20191208 13:50:34 all (114/679): parallelmount2.sh 20191208 13:50:36 all (115/679): pathconf.sh 20191208 13:50:37 all (116/679): pathconf2.sh 20191208 13:50:38 all (117/679): pcatch.sh 20191208 13:50:40 all (118/679): pcatch2.sh 20191208 13:50:41 all (119/679): ping.sh 20191208 13:50:43 all (120/679): pkru.sh 20191208 13:50:44 all (121/679): pkru2.sh 20191208 13:50:45 all (122/679): pmc4.sh 20191208 13:50:47 all (123/679): pmc5.sh 20191208 13:50:49 all (124/679): pmc8.sh 20191208 13:50:50 all (125/679): poll.sh 20191208 13:50:51 all (126/679): posix_fadvise.sh 20191208 13:50:53 all (127/679): posix_fadvise2.sh 20191208 13:50:54 all (128/679): procfs5.sh 20191208 13:50:56 all (129/679): procfs6.sh 20191208 13:50:57 all (130/679): procstat.sh 20191208 13:50:59 all (131/679): pthread9.sh 20191208 13:51:00 all (132/679): ptrace.sh 20191208 13:51:02 all (133/679): ptrace3.sh 20191208 13:51:03 all (134/679): pts2.sh 20191208 13:51:05 all (135/679): quota5.sh 20191208 13:51:06 all (136/679): rename13.sh 20191208 13:51:07 all (137/679): rename4.sh 20191208 13:51:09 all (138/679): revoke.sh 20191208 13:51:11 all (139/679): routetbl.sh 20191208 13:51:12 all (140/679): ruby.sh 20191208 13:51:14 all (141/679): rwlock_ronly.sh 20191208 13:51:15 all (142/679): schedfuzz.sh 20191208 13:51:16 all (143/679): seekdir.sh 20191208 13:51:18 all (144/679): segnp.sh 20191208 13:51:20 all (145/679): segregs.sh 20191208 13:51:21 all (146/679): select3.sh 20191208 13:51:22 all (147/679): sendfile2.sh lock order reversal: 1st 0xfffff80023f20040 filedesc structure (filedesc structure) @ kern/sys_generic.c:1516 2nd 0xfffff807831867d0 ufs (ufs) @ kern/vfs_vnops.c:1552 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b9816f at lockmgr_xlock_hard+0x6f #3 0xffffffff80b98cfd at __lockmgr_args+0x60d #4 0xffffffff80ee629c at ffs_lock+0x7c #5 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #6 0xffffffff80cb29c5 at _vn_lock+0x65 #7 0xffffffff80cb146a at vn_poll+0x3a #8 0xffffffff80c389a9 at kern_poll+0x419 #9 0xffffffff80c38580 at sys_poll+0x50 #10 0xffffffff8106d901 at amd64_syscall+0x2f1 #11 0xffffffff81043dd0 at fast_syscall_common+0x101 20191208 13:51:24 all (148/679): sendfile4.sh 20191208 13:51:26 all (149/679): setuid.sh 20191208 13:51:27 all (150/679): shm_super.sh 20191208 13:51:29 all (151/679): sigaltstack.sh 20191208 13:51:30 all (152/679): sigreturn.sh 20191208 13:51:32 all (153/679): sigstop.sh 20191208 13:51:33 all (154/679): snap.sh lock order reversal: 1st 0xfffffe0037ee8d18 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:283 2nd 0xfffff80023478830 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:2327 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b9816f at lockmgr_xlock_hard+0x6f #3 0xffffffff80b98cfd at __lockmgr_args+0x60d #4 0xffffffff80ec2ae6 at ffs_copyonwrite+0x166 #5 0xffffffff80ee530c at ffs_geom_strategy+0x8c #6 0xffffffff80efabdc at ufs_strategy+0x14c #7 0xffffffff810eaf12 at VOP_STRATEGY_APV+0xb2 #8 0xffffffff80c79d90 at bufstrategy+0x40 #9 0xffffffff80c79c2a at bufwrite+0x24a #10 0xffffffff80ee711d at ffs_syncvnode+0x2bd #11 0xffffffff80ee6142 at ffs_fsync+0x22 #12 0xffffffff810e985a at VOP_FSYNC_APV+0xaa #13 0xffffffff80efc9f2 at VOP_FSYNC+0x42 #14 0xffffffff80ef77e4 at ufs_remove+0x114 #15 0xffffffff810e99ee at VOP_REMOVE_APV+0xce #16 0xffffffff80cab1ce at kern_funlinkat+0x2be #17 0xffffffff80caaf08 at sys_unlink+0x28 lock order reversal: 1st 0xfffff80023478830 snaplk (snaplk) @ kern/vfs_subr.c:3012 2nd 0xfffff807839e5068 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1650 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b9816f at lockmgr_xlock_hard+0x6f #3 0xffffffff80b98cfd at __lockmgr_args+0x60d #4 0xffffffff80ec104d at ffs_snapremove+0xed #5 0xffffffff80ebb72a at ffs_truncate+0x26a #6 0xffffffff80eeee2c at ufs_inactive+0x2ec #7 0xffffffff810ea6ea at VOP_INACTIVE_APV+0xaa #8 0xffffffff80c9fb64 at vinactive+0xf4 #9 0xffffffff80ca00ea at vputx+0x2aa #10 0xffffffff80cab226 at kern_funlinkat+0x316 #11 0xffffffff80caaf08 at sys_unlink+0x28 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 20191208 13:51:35 all (155/679): snap12.sh 20191208 13:51:37 all (156/679): snap2-1.sh 20191208 13:51:38 all (157/679): snap2.sh 20191208 13:51:40 all (158/679): sndstat.sh 20191208 13:51:41 all (159/679): socketpair.sh 20191208 13:51:43 all (160/679): socketpair2.sh 20191208 13:51:44 all (161/679): suj2.sh 20191208 13:51:46 all (162/679): suj32.sh 20191208 13:51:47 all (163/679): suj35.sh Dec 8 13:51:48 mercat1 kernel: pid 60188 (mkdir), uid 0 inumber 255 on /mnt: out of inodes 20191208 13:51:49 all (164/679): suj7.sh Failed to find journal. Use tunefs to create one Failed to start journal: 2 20191208 13:51:50 all (165/679): tar.sh 20191208 13:51:52 all (166/679): temp.sh 20191208 13:51:53 all (167/679): tmpfs15.sh 20191208 13:51:54 all (168/679): tmpfs4.sh 20191208 13:51:56 all (169/679): tmpfs6.sh 20191208 13:51:58 all (170/679): trim5.sh 20191208 13:51:59 all (171/679): trim6.sh 20191208 13:52:01 all (172/679): trim7.sh 20191208 13:52:02 all (173/679): trim8.sh 20191208 13:52:04 all (174/679): truncate4.sh 20191208 13:52:05 all (175/679): truncate6.sh 20191208 13:52:07 all (176/679): tvnlru.sh 20191208 13:52:08 all (177/679): ufssuspend.sh 20191208 13:52:10 all (178/679): uma_zalloc_arg.sh 20191208 13:52:11 all (179/679): unlink.sh 20191208 13:52:13 all (180/679): unload.sh 20191208 13:52:14 all (181/679): unload2.sh 20191208 13:52:16 all (182/679): vmio.sh 20191208 13:52:17 all (183/679): watchman.sh 20191208 13:52:18 all (184/679): write.sh 20191208 13:52:20 all (185/679): datagram3.sh 20191208 13:52:21 all (186/679): sendfile6.sh 20191208 13:52:23 all (187/679): suj8.sh Dec 8 13:52:24 mercat1 kernel: pid 66828 (dd), uid 0 inumber 4 on /mnt: filesystem full Dec 8 13:52:25 mercat1 kernel: pid 66844 (dd), uid 0 inumber 5 on /mnt: filesystem full 20191208 13:52:25 all (188/679): tmpfs7.sh 20191208 13:52:27 all (189/679): vfork.sh 20191208 13:52:28 all (190/679): beneath2.sh 20191208 13:52:30 all (191/679): contigmalloc3.sh 20191208 13:52:33 all (192/679): datagram.sh 20191208 13:52:35 all (193/679): ext2fs3.sh 20191208 13:52:38 all (194/679): extattr.sh 20191208 13:52:40 all (195/679): fcntl.sh 20191208 13:52:42 all (196/679): fpclone2.sh 20191208 13:52:44 all (197/679): geomleak.sh 20191208 13:52:46 all (198/679): gnop5.sh GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. GEOM_NOP: Device md10.nop created. GEOM_NOP: Device md10.nop removed. 20191208 13:52:48 all (199/679): isofs.sh 20191208 13:52:50 all (200/679): killpg.sh 20191208 13:52:53 all (201/679): md9.sh 20191208 13:52:55 all (202/679): mmap25.sh 20191208 13:52:57 all (203/679): mmap26.sh 20191208 13:53:00 all (204/679): mmap28.sh Dec 8 13:53:02 mercat1 kernel: Failed to fully fault in a core file segment at VA 0x800e00000 with size 0x3a97000 to be written at offset 0x86b000 for process mmap28 20191208 13:53:02 all (205/679): mmap4.sh vnode_pager_putpages: I/O error 28 0xfffff80542ae54f0: tag ufs, type VREG usecount 0, writecount 0, refcount 3203 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0xfffff8016bdad630 ref 0 pages 25608 cleanbuf 3200 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff8016bf196e0 (pid 74317, mmap4, tid 101380) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80c9fff2 at vputx+0x1b2 #6 0xffffffff80f1349d at vm_map_process_deferred+0xad #7 0xffffffff80f1a07b at vm_map_remove+0xeb #8 0xffffffff80f12db5 at vmspace_exit+0xd5 #9 0xffffffff80b7daf2 at exit1+0x5a2 #10 0xffffffff80b7d54d at sys_sys_exit+0xd #11 0xffffffff8106d901 at amd64_syscall+0x2f1 #12 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=104857600, extsize 0 generation=cca237ef, uid=0, gid=0, flags=0x0 ino 4, on dev md10a vnode_pager_putpages: residual I/O 32768 at 9760 0xfffff80542ae54f0: tag ufs, type VREG usecount 0, writecount 0, refcount 3203 flags (VI_ACTIVE|VI_DOINGINACT) v_object 0xfffff8016bdad630 ref 0 pages 25608 cleanbuf 3200 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff8016bf196e0 (pid 74317, mmap4, tid 101380) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80c9fff2 at vputx+0x1b2 #6 0xffffffff80f1349d at vm_map_process_deferred+0xad #7 0xffffffff80f1a07b at vm_map_remove+0xeb #8 0xffffffff80f12db5 at vmspace_exit+0xd5 #9 0xffffffff80b7daf2 at exit1+0x5a2 #10 0xffffffff80b7d54d at sys_sys_exit+0xd Dec 8 13:53:04 #11 0xffffffff8106d901 at amd64_syscall+0x2f1 #mercat1 kernel: 12 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=104857600, extsize 0 generation=cca237ef, uid=0, gid=0, flags=0x0 ino 4, on dev md10a pid 74317 (mmap4), uid 0 inumber 4 on /mnt: filesystem full vnode_pager_putpages: zero-length write at 40075264 resid 131072 0xfffff80542ae54f0: tag ufs, type VREG usecount 1, writecount 0, refcount 3203 flags (VI_ACTIVE) v_object 0xfffff8016bdad630 ref 0 pages 25608 cleanbuf 3201 dirtybuf 0 lock type ufs: EXCL by thread 0xfffff8078301e000 (pid 74325, umount, tid 100589) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80c9f649 at vget_finish+0x89 #6 0xffffffff80ca19d4 at vfs_msync+0x154 #7 0xffffffff80c96be5 at dounmount+0x3d5 #8 0xffffffff80c967be at sys_unmount+0x2ae #9 0xffffffff8106d901 at amd64_syscall+0x2f1 #10 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=104857600, extsize 0 generation=cca237ef, uid=0, gid=0, flags=0x0 ino 4, on dev md10a Dec 8 13:53:05 mercat1 kernel: pid 74325 (umount), uid 0 inumber 4 on /mnt: filesystem full 20191208 13:53:05 all (206/679): mmap8.sh 20191208 13:53:08 all (207/679): mmap9.sh 20191208 13:53:11 all (208/679): mountro2.sh 20191208 13:53:12 all (209/679): mountro3.sh newblk leak: 1743/2055. 20191208 13:53:14 all (210/679): msdos9.sh 20191208 13:53:17 all (211/679): newfs5.sh 20191208 13:53:20 all (212/679): nullfs10.sh 20191208 13:53:22 all (213/679): nullfs12.sh 20191208 13:53:24 all (214/679): nullfs16.sh 20191208 13:53:27 all (215/679): nullfs3.sh 20191208 13:53:29 all (216/679): openlock.sh 20191208 13:53:32 all (217/679): overcommit.sh Dec 8 13:55:57 mercat1 kernel: pid 84234 (rw), uid 2007 inumber 173135 on /mnt: filesystem full Dec 8 13:55:58 mercat1 kernel: pid 84196 (rw), uid 2007 inumber 166290 on /mnt: filesystem full Dec 8 13:57:51 mercat1 kernel: pid 85486 (rw), uid 2007 inumber 10024 on /mnt: filesystem full Dec 8 13:57:59 mercat1 kernel: pid 85537 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 13:58:14 mercat1 kernel: pid 85557 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 13:58:26 mercat1 kernel: pid 85562 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 13:58:39 mercat1 kernel: pid 85579 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 13:58:54 mercat1 kernel: pid 85585 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 14:00:13 mercat1 kernel: pid 87464 (rw), uid 2007 inumber 81279 on /mnt: filesystem full Dec 8 14:02:49 mercat1 kernel: pid 93627 (rw), uid 2007 inumber 64397 on /mnt: filesystem full Dec 8 14:02:55 mercat1 kernel: pid 93590 (swap), jid 0, uid 2007, was killed: out of swap space Dec 8 14:03:54 mercat1 kernel: pid 93606 (rw), uid 2007 inumber 251040 on /mnt: filesystem full 20191208 14:04:12 all (218/679): procfs2.sh 20191208 14:04:15 all (219/679): ptrace11.sh 20191208 14:04:20 all (220/679): ptrace8.sh 20191208 14:04:22 all (221/679): quota9.sh lock order reversal: 1st 0xfffff807830ec888 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:759 2nd 0xfffff8016bf43000 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:765 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80ba3a94 at __mtx_lock_flags+0x94 #3 0xffffffff80ef394d at quotaoff_inchange+0xbd #4 0xffffffff80ee195d at ffs_flushfiles+0x12d #5 0xffffffff80ec3d0d at softdep_flushfiles+0xad #6 0xffffffff80ee4127 at ffs_unmount+0xd7 #7 0xffffffff80c96c65 at dounmount+0x455 #8 0xffffffff80c967be at sys_unmount+0x2ae #9 0xffffffff8106d901 at amd64_syscall+0x2f1 #10 0xffffffff81043dd0 at fast_syscall_common+0x101 20191208 14:04:25 all (222/679): random.sh 20191208 14:04:27 all (223/679): rename6.sh 20191208 14:04:30 all (224/679): scp.sh 20191208 14:04:33 all (225/679): sendfile.sh 20191208 14:04:35 all (226/679): snap10.sh 20191208 14:04:37 all (227/679): spin.sh 20191208 14:04:40 all (228/679): trim2.sh 20191208 14:04:42 all (229/679): umtx_suspend.sh 20191208 14:04:44 all (230/679): vm_fault_dontneed.sh 20191208 14:04:47 all (231/679): truncate5.sh 20191208 14:04:50 all (232/679): contigmalloc2.sh 20191208 14:04:58 all (233/679): datamove4.sh 20191208 14:05:01 all (234/679): dup.sh 20191208 14:05:05 all (235/679): extattr3.sh 20191208 14:05:08 all (236/679): kevent2.sh 20191208 14:05:11 all (237/679): kevent8.sh 20191208 14:05:15 all (238/679): mlockall7.sh 20191208 14:05:19 all (239/679): mmap23.sh 20191208 14:05:23 all (240/679): pthread.sh 20191208 14:05:26 all (241/679): tmpfs5.sh 20191208 14:05:30 all (242/679): udp.sh 20191208 14:05:33 all (243/679): unix_socket.sh 20191208 14:05:37 all (244/679): devfd.sh lock order reversal: 1st 0xfffff80040b58a48 syncer (syncer) @ kern/vfs_subr.c:2347 2nd 0xfffff806cc0f7cc0 ufs (ufs) @ kern/vfs_subr.c:2761 stack backtrace: #0 0xffffffff80c324e1 at witness_debugger+0x71 #1 0xffffffff80c32256 at witness_checkorder+0xad6 #2 0xffffffff80b9816f at lockmgr_xlock_hard+0x6f #3 0xffffffff80b98cfd at __lockmgr_args+0x60d #4 0xffffffff80ee629c at ffs_lock+0x7c #5 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #6 0xffffffff80cb29c5 at _vn_lock+0x65 #7 0xffffffff80c9f649 at vget_finish+0x89 #8 0xffffffff80ca19d4 at vfs_msync+0x154 #9 0xffffffff80ca73ce at sync_fsync+0x10e #10 0xffffffff810e985a at VOP_FSYNC_APV+0xaa #11 0xffffffff80ca5342 at sched_sync+0x3d2 #12 0xffffffff80b83000 at fork_exit+0x80 #13 0xffffffff810444ee at fork_trampoline+0xe 20191208 14:05:41 all (245/679): linger3.sh 20191208 14:05:46 all (246/679): mkfifo6.sh 20191208 14:05:51 all (247/679): mmap13.sh 20191208 14:05:55 all (248/679): mountro4.sh 20191208 14:05:59 all (249/679): procfs3.sh 20191208 14:06:04 all (250/679): ptrace2.sh 20191208 14:06:09 all (251/679): rename11.sh 20191208 14:06:14 all (252/679): truncate7.sh 20191208 14:06:18 all (253/679): datamove5.sh 20191208 14:06:23 all (254/679): growfs.sh g_access(1000): provider ufsid/5decf550bc6ce0ab has error 6 set g_access(1000): provider ufsid/5decf550bc6ce0ab has error 6 set g_access(1000): provider ufsid/5decf550bc6ce0ab has error 6 set g_access(1000): provider gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda has error 6 set g_access(1000): provider gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda has error 6 set g_access(1000): provider gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda has error 6 set g_dev_taste: make_dev_p() failed (gp->name=gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5decf550bc6ce0ab, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5decf550bc6ce0ab, error=17) g_dev_taste: make_dev_p() failed (gp->name=gptid/899adbd1-19bb-11ea-8e2e-0cc47aa8cdda, error=17) g_dev_taste: make_dev_p() failed (gp->name=ufsid/5decf550bc6ce0ab, error=17) GEOM leak: 14/87 20191208 14:06:29 all (255/679): procfs.sh 20191208 14:06:34 all (256/679): pshared.sh 20191208 14:06:40 all (257/679): symlink4.sh 20191208 14:06:45 all (258/679): tmpfs10.sh 20191208 14:06:50 all (259/679): truncate3.sh 20191208 14:06:55 all (260/679): fork.sh 20191208 14:07:02 all (261/679): linger2.sh Dec 8 14:07:05 mercat1 kernel: pid 56729 (linger2), uid 2007 inumber 888 on /mnt: filesystem full 20191208 14:07:09 all (262/679): linger4.sh Dec 8 14:07:14 mercat1 kernel: pid 58036 (linger4), uid 2007 inumber 234265 on /mnt: out of inodes 20191208 14:07:16 all (263/679): mlockall4.sh 20191208 14:07:23 all (264/679): ptrace9.sh 20191208 14:07:29 all (265/679): sparse.sh 20191208 14:07:31 all (266/679): symlink2.sh 20191208 14:07:39 all (267/679): sysctl.sh 20191208 14:07:46 all (268/679): execve.sh 20191208 14:07:58 all (269/679): quota11.sh 20191208 14:08:07 all (270/679): ffs_blkfree.sh newblk leak: 26629/28684. 20191208 14:08:16 all (271/679): md8.sh 20191208 14:08:25 all (272/679): tmpfs8.sh 20191208 14:08:35 all (273/679): compare.sh 20191208 14:08:45 all (274/679): rename8.sh 20191208 14:08:57 all (275/679): thr.sh witness_lock_list_get: witness exhausted 20191208 14:09:07 all (276/679): proccontrol.sh 20191208 14:09:18 all (277/679): fsck5.sh 20191208 14:09:28 all (278/679): nfs15lockd2.sh 20191208 14:09:40 all (279/679): posix_fadvise3.sh 20191208 14:09:51 all (280/679): rdgsbase.sh 20191208 14:10:03 all (281/679): recursiveflushes.sh 20191208 14:10:14 all (282/679): rename9.sh 20191208 14:10:24 all (283/679): umountf4.sh freework leak: 156/197. 20191208 14:10:35 all (284/679): pmc7.sh 20191208 14:10:47 all (285/679): shm_open.sh 20191208 14:11:00 all (286/679): tmpfs14.sh 20191208 14:11:12 all (287/679): umountf6.sh 20191208 14:11:23 all (288/679): mprotect2.sh 20191208 14:11:34 all (289/679): pipe2.sh 20191208 14:11:47 all (290/679): sendfile_shm.sh 20191208 14:12:00 all (291/679): umountf11.sh 20191208 14:12:13 all (292/679): umountf12.sh 20191208 14:12:27 all (293/679): tmpfs19.sh 20191208 14:12:39 all (294/679): rdwr.sh 20191208 14:12:55 all (295/679): suj10.sh 20191208 14:13:11 all (296/679): kevent6.sh 20191208 14:13:27 all (297/679): radix.sh Dec 8 14:13:36 mercat1 kernel: pid 62053 (radix), jid 0, uid 0, was killed: out of swap space 20191208 14:13:42 all (298/679): mountu.sh 20191208 14:13:58 all (299/679): suj14.sh Dec 8 14:14:12 mercat1 kernel: pid 66021 (cp), uid 0 inumber 40665 on /mnt: filesystem full Dec 8 14:14:13 mercat1 kernel: pid 66041 (cp), uid 0 inumber 40695 on /mnt: filesystem full Dec 8 14:14:14 mercat1 kernel: pid 67218 (mkdir), uid 0 inumber 40696 on /mnt: filesystem full 20191208 14:14:15 all (300/679): sem.sh 20191208 14:14:33 all (301/679): rename7.sh 20191208 14:14:52 all (302/679): core4.sh Dec 8 14:14:54 mercat1 kernel: Failed to write core file for process core4 (error 5) 20191208 14:15:14 all (303/679): mkfifo3.sh Dec 8 14:15:14 mercat1 syslogd: last message repeated 59 times 20191208 14:15:35 all (304/679): mountro6.sh 20191208 14:15:57 all (305/679): multicast.sh 20191208 14:16:18 all (306/679): pdfork.sh if_delmulti_locked: detaching ifnet instance 0xfffff80005050000 20191208 14:16:40 all (307/679): f_offset.sh 20191208 14:17:01 all (308/679): sendfile5.sh 20191208 14:17:23 all (309/679): mmap7.sh 20191208 14:17:44 all (310/679): mmap5.sh 20191208 14:18:36 all (311/679): sendmsg2.sh 20191208 14:18:58 all (312/679): core2.sh 20191208 14:19:25 all (313/679): nullfs22.sh 20191208 14:19:50 all (314/679): swap4.sh Dec 8 14:20:03 mercat1 kernel: pid 79897 (swap4), jid 0, uid 0, was killed: out of swap space Dec 8 14:20:08 mercat1 kernel: pid 79894 (swap4), jid 0, uid 0, was killed: out of swap space 20191208 14:20:08 all (315/679): pread.sh 20191208 14:20:34 all (316/679): suj33.sh fsync: giving up on dirty (error = 35) 0xfffff806fa1cf278: tag devfs, type VCHR usecount 1, writecount 0, refcount 293 rdev 0xfffff80005151800 flags (VI_ACTIVE) v_object 0xfffff807138fa630 ref 0 pages 20411 cleanbuf 290 dirtybuf 1 lock type devfs: EXCL by thread 0xfffff803f27426e0 (pid 87384, umount, tid 103892) #0 0xffffffff80b97bf0 at lockmgr_lock_fast_path+0x1e0 #1 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #2 0xffffffff80cb29c5 at _vn_lock+0x65 #3 0xffffffff80ec3a4d at softdep_flushworklist+0x5d #4 0xffffffff80ee4a36 at ffs_sync+0x406 #5 0xffffffff80c96c76 at dounmount+0x466 #6 0xffffffff80c967be at sys_unmount+0x2ae #7 0xffffffff8106d901 at amd64_syscall+0x2f1 #8 0xffffffff81043dd0 at fast_syscall_common+0x101 dev md10a fsync: giving up on dirty (error = 35) 0xfffff806fa1cf278: tag devfs, type VCHR usecount 1, writecount 0, refcount 293 rdev 0xfffff80005151800 flags (VI_ACTIVE) v_object 0xfffff807138fa630 ref 0 pages 20411 cleanbuf 290 dirtybuf 1 lock type devfs: EXCL by thread 0xfffff803f27426e0 (pid 87384, umount, tid 103892) #0 0xffffffff80b97bf0 at lockmgr_lock_fast_path+0x1e0 #1 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #2 0xffffffff80cb29c5 at _vn_lock+0x65 #3 0xffffffff80ee48fd at ffs_sync+0x2cd #4 0xffffffff80c96c76 at dounmount+0x466 #5 0xffffffff80c967be at sys_unmount+0x2ae #6 0xffffffff8106d901 at amd64_syscall+0x2f1 #7 0xffffffff81043dd0 at fast_syscall_common+0x101 dev md10a 20191208 14:22:39 all (317/679): credleak.sh 20191208 14:23:06 all (318/679): namecache2.sh 20191208 14:23:33 all (319/679): sendfile3.sh 20191208 14:24:02 all (320/679): rename3.sh 20191208 14:24:30 all (321/679): msdos5.sh GEOM leak: 19/106 20191208 14:25:01 all (322/679): mmap37.sh 20191208 14:25:33 all (323/679): split.sh newblk leak: 39835/68519. 20191208 14:26:06 all (324/679): fifo3.sh 20191208 14:26:40 all (325/679): alternativeFlushPath.sh Dec 8 14:32:40 mercat1 kernel: pid 77698 (alternativeFlushPat), uid 0 inumber 561793 on /tmp: out of inodes 20191208 14:46:42 all (326/679): vnodes.sh 20191208 14:47:15 all (327/679): tcp4.sh Expensive timeout(9) function: 0xffffffff80e03900(0xfffff802820c0000) 0.009123594 s Expensive timeout(9) function: 0xffffffff80e028b0(0xfffff806556d9bb8) 0.015298032 s Expensive timeout(9) function: 0xffffffff80e03900(0xfffff8017293c7d0) 0.015647904 s 20191208 14:47:51 all (328/679): ufsbench.sh 20191208 14:48:36 all (329/679): crossmp7.sh ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Warning: memory type solaris leaked memory on destroy (4 allocations, 2112 bytes leaked). 20191208 14:49:21 all (330/679): suj6.sh 20191208 14:49:42 all (331/679): beneath.sh 20191208 14:50:29 all (332/679): quota8.sh freework leak: 496/693. mount leak: 12/59 GEOM leak: 4/110 20191208 14:51:43 all (333/679): dd.sh interrupt total irq3: uart1 1597 irq4: uart0 5 irq9: acpi0 4 irq18: ehci0 ehci1 8061 irq19: xhci0 5076 cpu0:timer 6942619 cpu1:timer 7023506 cpu2:timer 7080546 cpu3:timer 7296905 cpu4:timer 7332770 cpu5:timer 7945477 cpu6:timer 7356366 cpu7:timer 7395542 cpu8:timer 7207813 cpu9:timer 7114647 cpu10:timer 7366658 cpu11:timer 7177214 irq56: ahci0 2092040 irq57: igb0:rxq0 17566 irq58: igb0:rxq1 126 irq59: igb0:rxq2 446 irq60: igb0:rxq3 15 irq61: igb0:rxq4 366 irq62: igb0:rxq5 81 irq63: igb0:aq 2 Total 89365493 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1bd/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: watchdog timeout [ thread pid 20 tid 100092 ] Stopped at kdb_enter+0x37: movq $0,0x1087286(%rip) db> ~ run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 20 tid 100092 td 0xfffff80003d2d6e0 kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x972e00 curthread = 0xfffff807c08a5000: pid 1373 tid 102278 "swap" curpcb = 0xfffff807c08a54f0 fpcurthread = 0xfffff807c08a5000: pid 1373 "swap" idlethread = 0xfffff800038aa000: tid 100003 "idle: cpu0" self = 0xffffffff82410000 curpmap = 0xfffff806c9046130 tssp = 0xffffffff82410384 rsp0 = 0xfffffe00e56a4cc0 kcr3 = 0x800000038b139222 ucr3 = 0x8000000249c9aa22 scr3 = 0x1a61bdf18 gs32p = 0xffffffff82410404 ldt = 0xffffffff82410444 tss = 0xffffffff82410434 tlb gen = 10523031 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00bc7ebe00 curthread = 0xfffff80003a72000: pid 12 tid 100049 "swi4: clock (0)" curpcb = 0xfffff80003a724f0 fpcurthread = none idlethread = 0xfffff800038a96e0: tid 100004 "idle: cpu1" self = 0xffffffff82411000 curpmap = 0xffffffff81d9d200 tssp = 0xffffffff82411384 rsp0 = 0xfffffe003e409cc0 kcr3 = 0x8000000002302005 ucr3 = 0xffffffffffffffff scr3 = 0x1a7220e9e gs32p = 0xffffffff82411404 ldt = 0xffffffff82411444 tss = 0xffffffff82411434 tlb gen = 10869036 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00bc7f7e00 curthread = 0xfffff803f22d46e0: pid 1375 tid 103743 "swap" curpcb = 0xfffff803f22d4bd0 fpcurthread = 0xfffff803f22d46e0: pid 1375 "swap" idlethread = 0xfffff800038a9000: tid 100005 "idle: cpu2" self = 0xffffffff82412000 curpmap = 0xfffff8043f31a130 tssp = 0xffffffff82412384 rsp0 = 0xfffffe033f654cc0 kcr3 = 0x80000005ec475714 ucr3 = 0x80000001a7220f14 scr3 = 0x249f27f17 gs32p = 0xffffffff82412404 ldt = 0xffffffff82412444 tss = 0xffffffff82412434 tlb gen = 10869037 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00bc803e00 curthread = 0xfffff80003d2d6e0: pid 20 tid 100092 "doneq1" curpcb = 0xfffff80003d2dbd0 fpcurthread = none idlethread = 0xfffff800038b96e0: tid 100006 "idle: cpu3" self = 0xffffffff82413000 curpmap = 0xffffffff81d9d200 tssp = 0xffffffff82413384 rsp0 = 0xfffffe003e540cc0 kcr3 = 0x8000000002302004 ucr3 = 0xffffffffffffffff scr3 = 0x1bad77f9c gs32p = 0xffffffff82413404 ldt = 0xffffffff82413444 tss = 0xffffffff82413434 tlb gen = 10312270 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00bc80fe00 curthread = 0xfffff803f26e5000: pid 1379 tid 105206 "swap" curpcb = 0xfffff803f26e54f0 fpcurthread = 0xfffff803f26e5000: pid 1379 "swap" idlethread = 0xfffff800038b9000: tid 100007 "idle: cpu4" self = 0xffffffff82414000 curpmap = 0xfffff807c099b130 tssp = 0xffffffff82414384 rsp0 = 0xfffffe0339436cc0 kcr3 = 0x800000082726c652 ucr3 = 0x80000005ec95ce52 scr3 = 0x43f483e4d gs32p = 0xffffffff82414404 ldt = 0xffffffff82414444 tss = 0xffffffff82414434 tlb gen = 10869039 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00bc81be00 curthread = 0xfffff806be50c000: pid 1378 tid 104383 "swap" curpcb = 0xfffff806be50c4f0 fpcurthread = 0xfffff806be50c000: pid 1378 "swap" idlethread = 0xfffff800038b86e0: tid 100008 "idle: cpu5" self = 0xffffffff82415000 curpmap = 0xfffff80047eea130 tssp = 0xffffffff82415384 rsp0 = 0xfffffe033ab2bcc0 kcr3 = 0x80000005b033d187 ucr3 = 0x80000002964fe987 scr3 = 0x3bf9f9980 gs32p = 0xffffffff82415404 ldt = 0xffffffff82415444 tss = 0xffffffff82415434 tlb gen = 10384509 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00bc827e00 curthread = 0xfffff80023a976e0: pid 2140 tid 100146 "watchdogd" curpcb = 0xfffff80023a97bd0 fpcurthread = 0xfffff80023a976e0: pid 2140 "watchdogd" idlethread = 0xfffff800038b8000: tid 100009 "idle: cpu6" self = 0xffffffff82416000 curpmap = 0xfffff8002391e130 tssp = 0xffffffff82416384 rsp0 = 0xfffffe00e1bd8cc0 kcr3 = 0x8000000040f6e009 ucr3 = 0x8000000040fcd809 scr3 = 0x6f637192e gs32p = 0xffffffff82416404 ldt = 0xffffffff82416444 tss = 0xffffffff82416434 tlb gen = 10869034 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00bc833e00 curthread = 0xfffff8002322b000: pid 30 tid 100128 "dom0" curpcb = 0xfffff8002322b4f0 fpcurthread = none idlethread = 0xfffff800038ab6e0: tid 100010 "idle: cpu7" self = 0xffffffff82417000 curpmap = 0xffffffff81d9d200 tssp = 0xffffffff82417384 rsp0 = 0xfffffe00d3224cc0 kcr3 = 0x8000000002302003 ucr3 = 0xffffffffffffffff scr3 = 0x447c4bf97 gs32p = 0xffffffff82417404 ldt = 0xffffffff82417444 tss = 0xffffffff82417434 tlb gen = 10336267 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00bc83fe00 curthread = 0xfffff80077b396e0: pid 1371 tid 102427 "swap" curpcb = 0xfffff80077b39bd0 fpcurthread = 0xfffff80077b396e0: pid 1371 "swap" idlethread = 0xfffff800038bc000: tid 100011 "idle: cpu8" self = 0xffffffff82418000 curpmap = 0xfffff805f94fd130 tssp = 0xffffffff82418384 rsp0 = 0xfffffe00e57adcc0 kcr3 = 0x8000000065fa915d ucr3 = 0x800000043f48395d scr3 = 0x38b7da955 gs32p = 0xffffffff82418404 ldt = 0xffffffff82418444 tss = 0xffffffff82418434 tlb gen = 10869030 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00bc84be00 curthread = 0xffffffff81d9c9c0: pid 0 tid 100000 "swapper" curpcb = 0xffffffff81d9ceb0 fpcurthread = none idlethread = 0xfffff800038bb6e0: tid 100012 "idle: cpu9" self = 0xffffffff82419000 curpmap = 0xffffffff81d9d200 tssp = 0xffffffff82419384 rsp0 = 0xffffffff822fbe00 kcr3 = 0x8000000002302042 ucr3 = 0xffffffffffffffff scr3 = 0x6f6371f99 gs32p = 0xffffffff82419404 ldt = 0xffffffff82419444 tss = 0xffffffff82419434 tlb gen = 10869035 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00bc857e00 curthread = 0xfffff806be2936e0: pid 1372 tid 105077 "swap" curpcb = 0xfffff806be293bd0 fpcurthread = 0xfffff806be2936e0: pid 1372 "swap" idlethread = 0xfffff800038bb000: tid 100013 "idle: cpu10" self = 0xffffffff8241a000 curpmap = 0xfffff807c0378130 tssp = 0xffffffff8241a384 rsp0 = 0xfffffe033afadcc0 kcr3 = 0x800000074dc27105 ucr3 = 0x8000000249f27905 scr3 = 0x43f338911 gs32p = 0xffffffff8241a404 ldt = 0xffffffff8241a444 tss = 0xffffffff8241a434 tlb gen = 10869038 curvnet = 0 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00bc863e00 curthread = 0xfffff80023a97000: pid 2250 tid 100156 "ntpd" curpcb = 0xfffff80023a974f0 fpcurthread = 0xfffff80023a97000: pid 2250 "ntpd" idlethread = 0xfffff800038ba6e0: tid 100014 "idle: cpu11" self = 0xffffffff8241b000 curpmap = 0xfffff80023aba130 tssp = 0xffffffff8241b384 rsp0 = 0xfffffe00e2e0ecc0 kcr3 = 0x8000000150265607 ucr3 = 0x80000001502aae07 scr3 = 0x6f6371df0 gs32p = 0xffffffff8241b404 ldt = 0xffffffff8241b444 tss = 0xffffffff8241b434 tlb gen = 10804846 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffff8002380d4f0: tag ufs, type VREG usecount 2, writecount 0, refcount 3 flags (VI_ACTIVE) v_object 0xfffff8002380b948 ref 32 pages 29 cleanbuf 1 dirtybuf 0 lock type ufs: EXCL by thread 0xfffff800234de000 (pid 33, syncer, tid 100140) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80c9f649 at vget_finish+0x89 #6 0xffffffff80ee4b71 at ffs_sync+0x541 #7 0xffffffff80ca73de at sync_fsync+0x11e #8 0xffffffff810e985a at VOP_FSYNC_APV+0xaa #9 0xffffffff80ca5342 at sched_sync+0x3d2 #10 0xffffffff80b83000 at fork_exit+0x80 #11 0xffffffff810444ee at fork_trampoline+0xe nlink=1, effnlink=1, size=395368, extsize 0 generation=12a980aa, uid=0, gid=0, flags=0x0 ino 56339877, on dev gpt/work vnode 0xfffff80023bac768: tag syncer, type VNON usecount 1, writecount 0, refcount 2 flags (VI_ACTIVE) lock type syncer: EXCL by thread 0xfffff800234de000 (pid 33, syncer, tid 100140) #0 0xffffffff80b97bf0 at lockmgr_lock_fast_path+0x1e0 #1 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #2 0xffffffff80cb29c5 at _vn_lock+0x65 #3 0xffffffff80ca5314 at sched_sync+0x3a4 #4 0xffffffff80b83000 at fork_exit+0x80 #5 0xffffffff810444ee at fork_trampoline+0xe vnode 0xfffff80783121000: tag ufs, type VREG usecount 71, writecount 0, refcount 6 flags (VV_NOSYNC|VI_ACTIVE) v_object 0xfffff8016bcfc738 ref 0 pages 48 cleanbuf 3 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff80023aa86e0 (pid 1975, dhclient, tid 100157) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80b9245f at ktr_writerequest+0x1cf #6 0xffffffff80b8fc07 at ktr_submitrequest+0x47 #7 0xffffffff80c6c854 at kern_sendit+0x294 #8 0xffffffff80c6cb85 at sendit+0x1d5 #9 0xffffffff80c6c99d at sys_sendto+0x4d #10 0xffffffff8106d901 at amd64_syscall+0x2f1 #11 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=0, effnlink=0, size=761193626, extsize 0 generation=9cf25d5c, uid=0, gid=0, flags=0x0 ino 196, on dev gpt/tmp vnode 0xfffff802b9822000: tag ufs, type VREG usecount 1, writecount 1, refcount 2876 flags (VI_ACTIVE) v_object 0xfffff8057bc04840 ref 0 pages 22984 cleanbuf 2870 dirtybuf 4 lock type ufs: EXCL by thread 0xfffff803f2f14000 (pid 1669, dd, tid 105200) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=94109696, extsize 0 generation=233e4052, uid=0, gid=0, flags=0x0 ino 561798, on dev gpt/tmp vnode 0xfffff804e46334f0: tag ufs, type VREG usecount 1, writecount 1, refcount 3698 flags (VI_ACTIVE) v_object 0xfffff804c1aab630 ref 0 pages 35198 cleanbuf 3691 dirtybuf 5 lock type ufs: EXCL by thread 0xfffff803f2b456e0 (pid 1668, dd, tid 101246) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=144048128, extsize 0 generation=c12c71a0, uid=0, gid=0, flags=0x0 ino 561799, on dev gpt/tmp vnode 0xfffff802778ff768: tag ufs, type VREG usecount 1, writecount 1, refcount 2258 flags (VI_ACTIVE) v_object 0xfffff80581f38948 ref 0 pages 22800 cleanbuf 2253 dirtybuf 3 lock type ufs: EXCL by thread 0xfffff803f2b45000 (pid 1667, dd, tid 101247) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=93356032, extsize 0 generation=c610c4c4, uid=0, gid=0, flags=0x0 ino 561800, on dev gpt/tmp vnode 0xfffff803d69d89e0: tag ufs, type VREG usecount 1, writecount 1, refcount 2369 flags (VI_ACTIVE) v_object 0xfffff801df24e318 ref 0 pages 18928 cleanbuf 2366 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff803f2231000 (pid 1671, dd, tid 105234) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=77496320, extsize 0 generation=abf51198, uid=0, gid=0, flags=0x0 ino 561801, on dev gpt/tmp vnode 0xfffff8018c9259e0: tag ufs, type VREG usecount 1, writecount 1, refcount 2873 flags (VI_ACTIVE) v_object 0xfffff80793ba5738 ref 0 pages 28416 cleanbuf 2870 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff806be541000 (pid 1672, dd, tid 101262) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=116359168, extsize 0 generation=ee3502a4, uid=0, gid=0, flags=0x0 ino 561802, on dev gpt/tmp vnode 0xfffff80048f1d000: tag ufs, type VREG usecount 1, writecount 1, refcount 2034 flags (VI_ACTIVE) v_object 0xfffff8031c6c3000 ref 0 pages 20376 cleanbuf 2031 dirtybuf 1 lock type ufs: EXCL by thread 0xfffff806be723000 (pid 1670, dd, tid 105197) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=83427328, extsize 0 generation=e07b4cc9, uid=0, gid=0, flags=0x0 ino 561803, on dev gpt/tmp vnode 0xfffff803e019ec58: tag ufs, type VREG usecount 1, writecount 1, refcount 3486 flags (VI_ACTIVE) v_object 0xfffff80718a5f948 ref 0 pages 33976 cleanbuf 3481 dirtybuf 3 lock type ufs: EXCL by thread 0xfffff806bea2a000 (pid 1673, dd, tid 102569) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=139067392, extsize 0 generation=353e0496, uid=0, gid=0, flags=0x0 ino 561804, on dev gpt/tmp vnode 0xfffff801c426b768: tag ufs, type VREG usecount 1, writecount 1, refcount 2709 flags (VI_ACTIVE) v_object 0xfffff802fc547528 ref 0 pages 21648 cleanbuf 2705 dirtybuf 2 lock type ufs: EXCL by thread 0xfffff806bea43000 (pid 1674, dd, tid 102552) #0 0xffffffff80b9850a at lockmgr_xlock_hard+0x40a #1 0xffffffff80b98cfd at __lockmgr_args+0x60d #2 0xffffffff80ee629c at ffs_lock+0x7c #3 0xffffffff810eab02 at VOP_LOCK1_APV+0x92 #4 0xffffffff80cb29c5 at _vn_lock+0x65 #5 0xffffffff80cb6bec at vn_write+0x1cc #6 0xffffffff80cb6683 at vn_io_fault_doio+0x43 #7 0xffffffff80cb316c at vn_io_fault1+0x16c #8 0xffffffff80cb1052 at vn_io_fault+0x182 #9 0xffffffff80c36d41 at dofilewrite+0x81 #10 0xffffffff80c36989 at kern_writev+0x49 #11 0xffffffff80c36933 at sys_write+0x83 #12 0xffffffff8106d901 at amd64_syscall+0x2f1 #13 0xffffffff81043dd0 at fast_syscall_common+0x101 nlink=1, effnlink=1, size=88637440, extsize 0 generation=93f1cbb7, uid=0, gid=0, flags=0x0 ino 561805, on dev gpt/tmp db:1:pho1> show mount 0xfffff8002377f000 /dev/gpt/work on / (ufs) 0xfffff800234b8000 devfs on /dev (devfs) 0xfffff80150267000 /dev/gpt/tmp on /tmp (ufs) 0xfffff807835c2000 procfs on /proc (procfs) More info: show mount db:1:pho1> show bufqueues bqempty: 61409 Buf domain 0 freebufs 7945 lofreebufs 219 hifreebufs 329 bufspace 235450368 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 10 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 6905 (226205696) dirtyq count 10 (299008) wakeup 0 lim 25 CPU 12, 4, 14, 9, 15, 14, 24, 23, 23, 20, 11, 7, Locked buffers: 100 space 3211264 Total buffers: 18584 space 235384832 Buf domain 1 freebufs 9885 lofreebufs 219 hifreebufs 329 bufspace 21852160 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 5246 (21835776) dirtyq count 0 (0) wakeup 0 lim 25 CPU 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, Locked buffers: 0 space 0 Total buffers: 5395 space 21852160 Buf domain 2 freebufs 3975 lofreebufs 219 hifreebufs 329 bufspace 187142144 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 5 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 11133 (186372096) dirtyq count 5 (163840) wakeup 0 lim 25 CPU 0, 0, 0, 1, 0, 2, 15, 0, 2, 0, 0, 0, Locked buffers: 1 space 32768 Total buffers: 11377 space 187142144 Buf domain 3 freebufs 9808 lofreebufs 219 hifreebufs 329 bufspace 22732800 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 5320 (22663168) dirtyq count 0 (0) wakeup 0 lim 25 CPU 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, Locked buffers: 0 space 0 Total buffers: 5640 space 22732800 Buf domain 4 freebufs 12687 lofreebufs 219 hifreebufs 329 bufspace 78213120 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 1 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 2344 (74899456) dirtyq count 1 (32768) wakeup 0 lim 25 CPU 2, 6, 6, 15, 8, 14, 13, 3, 4, 21, 1, 7, Locked buffers: 2 space 32768 Total buffers: 7440 space 78180352 Buf domain 5 freebufs 7942 lofreebufs 219 hifreebufs 329 bufspace 235433984 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 11 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 6980 (228519936) dirtyq count 11 (360448) wakeup 0 lim 25 CPU 19, 5, 1, 19, 21, 18, 4, 21, 3, 6, 11, 10, Locked buffers: 65 space 2031616 Total buffers: 35369 space 235335680 Buf domain 6 freebufs 9547 lofreebufs 219 hifreebufs 329 bufspace 182960128 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 6 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 5402 (176939008) dirtyq count 6 (196608) wakeup 0 lim 25 CPU 0, 21, 20, 13, 14, 7, 9, 15, 19, 18, 4, 3, Locked buffers: 37 space 1146880 Total buffers: 22126 space 182894592 db:1:pho1> show page vm_cnt.v_free_count: 272 vm_cnt.v_inactive_count: 27571 vm_cnt.v_active_count: 7148106 vm_cnt.v_laundry_count: 39115 vm_cnt.v_wire_count: 908567 vm_cnt.v_free_reserved: 10644 vm_cnt.v_free_min: 51264 vm_cnt.v_free_target: 173124 vm_cnt.v_inactive_target: 259686 db:1:pho1> show pageq pq_free 272 dom 0 page_cnt 8124093 free 272 pq_act 7148106 pq_inact 27571 pq_laund 39115 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000000 | 000000 05 (000128K) | 000000 | 000000 04 (000064K) | 000000 | 000000 03 (000032K) | 000000 | 000000 02 (000016K) | 000000 | 000000 01 (000008K) | 000000 | 000000 00 (000004K) | 000000 | 000000 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000000 | 000000 05 (000128K) | 000000 | 000000 04 (000064K) | 000000 | 000000 03 (000032K) | 000000 | 000000 02 (000016K) | 000000 | 000000 01 (000008K) | 000000 | 000000 00 (000004K) | 000000 | 000000 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Total Mem XFree socket 904 54 939650 951658 0 13 849492416 0 VNODE 632 616344 4164 60031181 0 13 392161056 0 VM OBJECT 264 604287 20478 65855571 0 29 164937960 0 FFS2 dinode 256 616226 4534 56832737 0 61 158914560 0 FFS inode 160 616226 4624 56832791 0 61 99336000 0 Files 80 155 941395 74209697 0 125 75324000 0 RADIX NODE 144 507619 272 29175140 0 61 73136304 0 S VFS Cache 108 621905 395 14792848 0 125 67208400 0 128 128 15717 203050 167275290 0 125 28002176 0 mbuf_cluster 2048 11528 86 156612 0 5 23785472 0 BUF TRIE 144 3099 102849 9147058 0 61 15256512 0 swblk 136 9450 99532 326852 0 61 14821552 0 THREAD 1760 4220 3380 373806 0 5 13376000 0 512 512 4136 16448 92847538 0 29 10539008 0 VMSPACE 2552 68 3969 1000760 0 3 10302424 0 mbuf 256 15667 11625 102916569 0 253 6986752 0 vmem btag 56 49129 72481 340928 0 253 6810160 0 256 256 25552 263 29234924 0 61 6608640 0 L VFS Cache 328 19236 312 32787 0 29 6411744 0 UMA Slabs 104 9869 45290 3416978022 0 125 5736536 0 PROC 1328 101 4003 1000996 0 9 5450112 0 128 128 14676 1258 2959854 0 126 2039552 0 MAP ENTRY 112 2278 12982 60393830 0 125 1709120 0 4096 4096 368 0 175171 0 1 1507328 0 128 128 9847 1282 128095480 0 125 1424512 0 swpctrie 144 1060 6905 28664 0 61 1146960 0 TURNSTILE 136 7601 399 7601 0 61 1088000 0 g_bio 408 719 1810 38416150 0 34 1031832 0 pbuf 984 0 870 0 0 0 856080 0 32768 32768 26 0 2697 0 1 851968 0 8192 8192 103 0 181 0 1 843776 0 DIRHASH 1024 631 105 1346105 0 13 753664 0 SLEEPQUEUE 88 7601 490 7601 0 125 712008 0 64 64 9419 1555 172552663 0 253 702336 0 64 64 8726 1690 2725680 0 253 666624 0 ksiginfo 112 4079 1591 126876 0 125 635040 0 16384 16384 37 0 37 0 1 606208 0 4096 4096 132 0 1032873 0 1 540672 0 512 512 938 118 1664196 0 29 540672 0 256 256 755 595 3106603 0 61 345600 0 Mountpoints 3008 4 110 20766 0 3 342912 0 64 64 2533 2427 76417433 0 253 317440 0 1024 1024 233 35 2666154 0 13 274432 0 32 32 6912 1588 373282 0 253 272000 0 32768 32768 8 0 408 0 1 262144 0 256 Bucket 2048 72 49 2097103622 0 253 247808 0 16 16 12050 1894 2027031 0 253 223104 0 16384 16384 13 0 2676 0 1 212992 0 2048 2048 96 8 723 0 5 212992 0 64 Bucket 512 121 293 1841555 0 29 211968 0 filedesc0 1104 101 82 1000997 0 9 202032 0 65536 65536 3 0 52 0 1 196608 0 65536 65536 3 0 33 0 1 196608 0 65536 65536 3 0 2158 0 1 196608 0 selfd 64 312 2540 8065677 0 253 182528 0 4096 4096 43 0 324 0 1 176128 0 UMA Zones 1968 86 1 270 0 5 171216 0 rl_entry 40 2640 1360 2640 0 253 160000 0 fakepg 104 0 1520 2447 0 125 158080 0 128 128 618 467 2506821 0 125 138880 0 64 64 31 2077 22948295 0 253 134912 0 16 Bucket 128 57 996 6487369 0 127 134784 0 256 256 389 121 3440323 0 61 130560 0 clpbuf 984 0 128 1185870 92397 5 125952 0 8 Bucket 64 40 1881 576278 0 253 122944 0 32 Bucket 256 20 444 2264871 0 61 118784 0 128 128 375 462 18617 0 125 107136 0 2048 2048 51 1 2100539 0 5 106496 0 32 32 955 2170 46564 0 253 100000 0 16384 16384 6 0 182 0 1 98304 0 512 512 131 61 6315 0 29 98304 0 ttyinq 160 330 245 5190 0 61 92000 0 ttyoutq 256 173 157 2762 0 61 84480 0 256 256 70 260 904141 0 61 84480 0 64 64 631 671 3575471 0 253 83328 0 8192 8192 10 0 1670 0 1 81920 0 256 256 90 210 53512303 0 61 76800 0 128 Bucket 1024 16 56 1025932 0 44 73728 0 32 32 78 2172 39898 0 253 72000 0 512 512 65 71 1049579 0 29 69632 0 256 256 63 207 19618545 0 61 69120 0 65536 65536 1 0 69 0 1 65536 0 65536 65536 1 0 102 0 1 65536 0 65536 65536 1 0 15 0 1 65536 0 65536 65536 1 0 1810 0 1 65536 0 16384 16384 4 0 10 0 1 65536 0 32 32 41 1959 4480855 0 253 64000 0 256 256 41 184 888233 0 61 57600 0 1024 1024 32 24 3411 0 13 57344 0 512 512 28 84 470373 0 29 57344 0 32 32 168 1582 24841 0 253 56000 0 16 16 748 2738 1850376 0 253 55776 0 128 128 19 384 208 0 125 51584 0 16384 16384 3 0 59958 0 1 49152 0 8192 8192 6 0 2246 0 1 49152 0 16 16 39 2949 2592367 0 253 47808 0 128 128 82 290 9331722 0 125 47616 0 16 16 294 2445 20505 0 253 43824 0 64 64 73 609 1116242 0 253 43648 0 cpuset 104 27 376 27 0 125 41912 0 8192 8192 5 0 5 0 1 40960 0 32 32 129 1121 3594711 0 253 40000 0 16 16 368 2122 4406283 0 253 39840 0 unpcb 256 19 131 945656 0 61 38400 0 64 pcpu 8 3047 1561 3897 0 253 36864 0 16 16 339 1902 110443 0 253 35856 0 64 64 24 533 4316 0 253 35648 0 32768 32768 1 0 70 0 1 32768 0 32768 32768 1 0 21 0 1 32768 0 32768 32768 1 0 1 0 1 32768 0 8192 8192 4 0 281290 0 1 32768 0 8192 8192 4 0 2798 0 1 32768 0 mt_stats_zone 64 431 81 547 0 253 32768 0 tcpcb 1000 12 20 5508 0 13 32000 0 16 16 33 1959 115 0 253 31872 0 64 64 69 427 158727 0 253 31744 0 tcp_inpcb 488 12 52 5508 0 29 31232 0 udp_inpcb 488 20 44 456 0 29 31232 0 KMAP ENTRY 112 18 259 213 0 125 31024 0 pipe 760 10 30 721218 0 13 30400 0 udpcb 32 20 855 456 0 253 28000 0 32 32 36 839 14615088 0 253 28000 0 128 128 19 198 2978430 0 125 27776 0 512 512 24 24 6467661 0 29 24576 0 32 32 122 628 204569 0 253 24000 0 16 16 19 1475 2852306 0 253 23904 0 256 256 14 76 13186761 0 61 23040 0 2048 2048 5 5 21 0 5 20480 0 mt_zone 24 431 399 547 0 253 19920 0 rtentry 208 12 83 15 0 61 19760 0 UMA Kegs 216 75 3 259 0 61 16848 0 16384 16384 1 0 5 0 1 16384 0 4096 4096 4 0 1079791 0 1 16384 0 2048 2048 4 4 401485 0 5 16384 0 1024 1024 4 12 1486476 0 13 16384 0 1024 1024 5 11 3421 0 13 16384 0 512 512 11 21 5897 0 29 16384 0 MAP 232 2 65 2 0 61 15544 0 2048 2048 4 2 4004277 0 5 12288 0 1024 1024 5 7 3110615 0 13 12288 0 1024 1024 8 4 144 0 13 12288 0 int pcpu 4 456 2616 456 0 253 12288 0 8192 8192 1 0 6 0 1 8192 0 2048 2048 2 2 12738546 0 5 8192 0 2048 2048 3 1 667 0 5 8192 0 1024 1024 3 5 1990 0 13 8192 0 procdesc 136 2 56 37 0 61 7888 0 mbuf_packet 256 23 0 12070312 0 253 5888 0 epoch_record pcpu 256 4 12 4 0 61 4096 0 4096 4096 1 0 1066673 0 1 4096 0 4096 4096 1 0 501904 0 1 4096 0 4096 4096 1 0 3 0 1 4096 0 2048 2048 1 1 1 0 5 4096 0 512 512 2 6 915 0 29 4096 0 hostcache 96 1 40 1 0 125 3936 0 swwbuf 984 4 0 178138 0 1 3936 0 ripcb 488 1 7 15 0 29 3904 0 UMA Hash 256 1 11 2 0 61 3072 0 vmem 1856 0 1 0 0 5 1856 0 FFS1 dinode 128 0 0 11 0 125 0 0 udplite_inpcb 488 0 0 0 0 29 0 0 sctp_asconf_ack 48 0 0 0 0 253 0 0 sctp_asconf 40 0 0 0 0 253 0 0 sctp_stream_msg_out 112 0 0 0 0 125 0 0 sctp_readq 152 0 0 0 0 61 0 0 sctp_chunk 152 0 0 0 0 61 0 0 sctp_raddr 736 0 0 0 0 13 0 0 sctp_laddr 48 0 0 4 0 253 0 0 sctp_asoc 2408 0 0 0 0 3 0 0 sctp_ep 1280 0 0 0 0 9 0 0 tcp_log_node 120 0 0 0 0 125 0 0 tcp_log_bucket 144 0 0 0 0 61 0 0 tcp_log 408 0 0 0 0 29 0 0 tcpreass 48 0 0 12026 0 253 0 0 tfo_ccache_entries 80 0 0 0 0 125 0 0 tfo 4 0 0 0 0 253 0 0 sackhole 32 0 0 16559 0 253 0 0 syncache 168 0 0 1466 0 61 0 0 tcptw 88 0 0 110 0 125 0 0 ipq 56 0 0 0 0 253 0 0 IPsec SA lft_c 16 0 0 0 0 253 0 0 KNOTE 160 0 0 8368748 0 61 0 0 itimer 352 0 0 0 0 29 0 0 AIOLIO 280 0 0 0 0 29 0 0 AIOCB 752 0 0 0 0 13 0 0 AIOP 32 0 0 0 0 253 0 0 AIO 208 0 0 0 0 61 0 0 NCLNODE 744 0 0 0 0 13 0 0 LTS VFS Cache 368 0 0 0 0 29 0 0 STS VFS Cache 148 0 0 0 0 61 0 0 rentr 24 0 0 12711063 0 253 0 0 NAMEI 1024 0 0 224182047 0 13 0 0 VNODEPOLL 120 0 0 281 0 125 0 0 vtnet_tx_hdr 24 0 0 0 0 253 0 0 crypto_session 24 0 0 0 0 253 0 0 cryptodesc 120 0 0 0 0 125 0 0 cryptop 128 0 0 0 0 125 0 0 nvme_request 128 0 0 0 0 125 0 0 FPU_save_area 832 0 0 0 0 13 0 0 DMAR_MAP_ENTRY 120 0 0 0 0 125 0 0 mbuf_extpgs 256 0 0 0 0 61 0 0 mbuf_jumbo_16k 16384 0 0 0 0 1 0 0 mbuf_jumbo_9k 9216 0 0 0 0 1 0 0 mbuf_jumbo_page 4096 0 0 1030919 0 1 0 0 audit_record 1280 0 0 0 0 9 0 0 domainset 40 0 0 0 0 253 0 0 MAC labels 40 0 0 0 0 253 0 0 vnpbuf 984 0 0 2484 0 13 0 0 mdpbuf 984 0 0 49369 0 1 0 0 nfspbuf 984 0 0 0 0 5 0 0 swrbuf 984 0 0 48910 0 5 0 0 umtx_shm 88 0 0 35 0 125 0 0 umtx pi 96 0 0 2 0 125 0 0 rangeset pctrie nodes 144 0 0 0 0 61 0 0 65536 65536 0 0 0 0 1 0 0 32768 32768 0 0 338811 0 1 0 0 32768 32768 0 0 2324 0 1 0 0 32768 32768 0 0 4 0 1 0 0 16384 16384 0 0 30845 0 1 0 0 16384 16384 0 0 1 0 1 0 0 8192 8192 0 0 3105 0 1 0 0 4096 4096 0 0 47 0 1 0 0 1024 1024 0 0 1209005 0 13 0 0 12 Bucket 96 0 0 511597 0 132 0 0 6 Bucket 48 0 0 412282 0 253 0 0 4 Bucket 32 0 0 639159 0 253 0 0 db:0:pho> show umacache Zone Size Used Free Requests Bucket debugnet_mbuf_packet 2048 0 0 0 0 debugnet_mbuf_cluster 2048 -6144 0 0 0 debugnet_mbuf 256 -10240 0 0 0 kstack_cache 16384 4093 0 373678 1 vm pgcache 4096 567447 0 28780327 253 vm pgcache 4096 2068530 0 108047483 253 buf free cache 984 44142 380 8845973 13 buffer arena-65 65536 0 0 218 1 buffer arena-61 61440 0 0 0 1 buffer arena-57 57344 0 0 0 1 buffer arena-53 53248 0 0 0 1 buffer arena-49 49152 0 0 0 1 buffer arena-45 45056 0 0 0 1 buffer arena-40 40960 0 0 0 1 buffer arena-36 36864 0 0 0 1 buffer arena-32 32768 21751 0 2599801 1 buffer arena-28 28672 0 0 0 1 buffer arena-24 24576 0 0 0 1 buffer arena-20 20480 0 0 0 1 buffer arena-16 16384 3 0 561517 1 buffer arena-12 12288 0 0 0 1 buffer arena-81 8192 0 0 0 1 buffer arena-40 4096 0 0 30 1 db:0:pho> acttrace Tracing command swap pid 1379 tid 105206 td 0xfffff803f26e5000 (CPU 4) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e197dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e197df0 trap() at trap+0x61/frame 0xfffffe003e197f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e197f20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe0339436720, rbp = 0xfffffe0339436730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe0339436730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe03394367e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe0339436830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe0339436890 vm_fault() at vm_fault+0x165c/frame 0xfffffe0339436a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe0339436a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe0339436ac0 trap() at trap+0x470/frame 0xfffffe0339436bf0 calltrap() at calltrap+0x8/frame 0xfffffe0339436bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1378 tid 104383 td 0xfffff806be50c000 (CPU 5) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1a3dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1a3df0 trap() at trap+0x61/frame 0xfffffe003e1a3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1a3f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe033ab2b720, rbp = 0xfffffe033ab2b730 --- lock_delay() at lock_delay+0x32/frame 0xfffffe033ab2b730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033ab2b7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033ab2b830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033ab2b890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033ab2ba00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033ab2ba40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033ab2bac0 trap() at trap+0x470/frame 0xfffffe033ab2bbf0 calltrap() at calltrap+0x8/frame 0xfffffe033ab2bbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1375 tid 103743 td 0xfffff803f22d46e0 (CPU 2) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e17fdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e17fdf0 trap() at trap+0x61/frame 0xfffffe003e17ff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e17ff20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe033f654720, rbp = 0xfffffe033f654730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe033f654730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033f6547e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033f654830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033f654890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033f654a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033f654a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033f654ac0 trap() at trap+0x470/frame 0xfffffe033f654bf0 calltrap() at calltrap+0x8/frame 0xfffffe033f654bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1373 tid 102278 td 0xfffff807c08a5000 (CPU 0) cpustop_handler() at cpustop_handler+0x2a/frame 0xffffffff81d84f40 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff81d84f60 trap() at trap+0x61/frame 0xffffffff81d85090 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81d85090 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe00e56a4720, rbp = 0xfffffe00e56a4730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe00e56a4730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e56a47e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e56a4830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e56a4890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e56a4a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e56a4a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e56a4ac0 trap() at trap+0x470/frame 0xfffffe00e56a4bf0 calltrap() at calltrap+0x8/frame 0xfffffe00e56a4bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1372 tid 105077 td 0xfffff806be2936e0 (CPU 10) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1dfdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1dfdf0 trap() at trap+0x61/frame 0xfffffe003e1dff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1dff20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe033afad720, rbp = 0xfffffe033afad730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe033afad730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033afad7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033afad830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033afad890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033afada00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033afada40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033afadac0 trap() at trap+0x470/frame 0xfffffe033afadbf0 calltrap() at calltrap+0x8/frame 0xfffffe033afadbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1371 tid 102427 td 0xfffff80077b396e0 (CPU 8) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1c7dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1c7df0 trap() at trap+0x61/frame 0xfffffe003e1c7f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1c7f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe00e57ad720, rbp = 0xfffffe00e57ad730 --- lock_delay() at lock_delay+0x32/frame 0xfffffe00e57ad730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e57ad7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e57ad830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e57ad890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e57ada00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e57ada40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e57adac0 trap() at trap+0x470/frame 0xfffffe00e57adbf0 calltrap() at calltrap+0x8/frame 0xfffffe00e57adbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command ntpd pid 2250 tid 100156 td 0xfffff80023a97000 (CPU 11) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1ebdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1ebdf0 trap() at trap+0x61/frame 0xfffffe003e1ebf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1ebf20 --- trap 0x13, rip = 0xffffffff80f054a6, rsp = 0xfffffe00e2e0e630, rbp = 0xfffffe00e2e0e660 --- uma_zalloc_arg() at uma_zalloc_arg+0xa6/frame 0xfffffe00e2e0e660 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e2e0e6f0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e2e0e730 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00e2e0e7b0 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00e2e0e7e0 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00e2e0e840 zone_import() at zone_import+0x177/frame 0xfffffe00e2e0e900 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00e2e0e990 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e2e0e9d0 cache_free() at cache_free+0x27b/frame 0xfffffe00e2e0ea20 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe00e2e0ea70 sigqueue_get() at sigqueue_get+0x15c/frame 0xfffffe00e2e0ead0 postsig() at postsig+0x129/frame 0xfffffe00e2e0eba0 ast() at ast+0x4eb/frame 0xfffffe00e2e0ebf0 doreti_ast() at doreti_ast+0x1f/frame 0x7fffffffdfc0 Tracing command watchdogd pid 2140 tid 100146 td 0xfffff80023a976e0 (CPU 6) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1afdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1afdf0 trap() at trap+0x61/frame 0xfffffe003e1aff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1aff20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe00e1bd8290, rbp = 0xfffffe00e1bd82a0 --- lock_delay() at lock_delay+0x32/frame 0xfffffe00e1bd82a0 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e1bd8350 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e1bd83a0 cache_alloc() at cache_alloc+0x128/frame 0xfffffe00e1bd8430 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8470 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd8500 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8540 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd85d0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8610 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00e1bd8690 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00e1bd86c0 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00e1bd8720 zone_import() at zone_import+0x177/frame 0xfffffe00e1bd87e0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00e1bd8870 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd88b0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd8940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8980 fork1() at fork1+0x1c1/frame 0xfffffe00e1bd8a60 sys_vfork() at sys_vfork+0x4e/frame 0xfffffe00e1bd8ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1bd8bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1bd8bf0 --- syscall (66, FreeBSD ELF64, sys_vfork), rip = 0x8003a0e6d, rsp = 0x7fffffffeac0, rbp = 0x7fffffffeb70 --- Tracing command pagedaemon pid 30 tid 100128 td 0xfffff8002322b000 (CPU 7) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1bbdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1bbdf0 trap() at trap+0x61/frame 0xfffffe003e1bbf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1bbf20 --- trap 0x13, rip = 0xffffffff80ba40bf, rsp = 0xfffffe00d3224130, rbp = 0xfffffe00d3224150 --- __mtx_unlock_flags() at __mtx_unlock_flags+0x9f/frame 0xfffffe00d3224150 zone_alloc_item_locked() at zone_alloc_item_locked+0x75/frame 0xfffffe00d32241b0 uma_zalloc_arg() at uma_zalloc_arg+0x10c/frame 0xfffffe00d32241f0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00d3224280 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d32242c0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00d3224350 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d3224390 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00d3224410 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00d3224440 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00d32244a0 zone_import() at zone_import+0x177/frame 0xfffffe00d3224560 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00d32245f0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d3224630 cache_free() at cache_free+0x27b/frame 0xfffffe00d3224680 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe00d32246d0 free() at free+0x91/frame 0xfffffe00d3224700 ufsdirhash_destroy() at ufsdirhash_destroy+0x10a/frame 0xfffffe00d3224740 ufsdirhash_lowmem() at ufsdirhash_lowmem+0xa4/frame 0xfffffe00d3224770 vm_pageout_worker() at vm_pageout_worker+0x3c0/frame 0xfffffe00d3224b70 vm_pageout() at vm_pageout+0x193/frame 0xfffffe00d3224bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3224bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3224bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100092 td 0xfffff80003d2d6e0 (CPU 3) kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100049 td 0xfffff80003a72000 (CPU 1) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e173dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e173df0 trap() at trap+0x61/frame 0xfffffe003e173f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e173f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe003e409940, rbp = 0xfffffe003e409950 --- lock_delay() at lock_delay+0x32/frame 0xfffffe003e409950 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe003e409a00 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e409a50 softclock_call_cc() at softclock_call_cc+0x113/frame 0xfffffe003e409b20 softclock() at softclock+0x7c/frame 0xfffffe003e409b50 ithread_loop() at ithread_loop+0x1c6/frame 0xfffffe003e409bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e409bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e409bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81d9c9c0 (CPU 9) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1d3dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1d3df0 trap() at trap+0x61/frame 0xfffffe003e1d3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1d3f20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xffffffff822fbc40, rbp = 0xffffffff822fbc50 --- lock_delay() at lock_delay+0x30/frame 0xffffffff822fbc50 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xffffffff822fbd00 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xffffffff822fbd50 swapper() at swapper+0x15f/frame 0xffffffff822fbda0 mi_startup() at mi_startup+0x2ed/frame 0xffffffff822fbdf0 btext() at btext+0x2c db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 1697 1695 10083 0 S piperd 0xfffff800238982f8 sed 1696 1695 10083 0 ?W sh 1695 1393 10083 0 SW wait 0xfffff80077bef530 sh 1674 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1673 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1672 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1671 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1670 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1669 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1668 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1667 1356 2359 0 D+ vmwait 0xffffffff81847f40 dd 1393 10141 10083 0 S piperd 0xfffff80023898000 sh 1382 1368 2359 0 R+ swap 1381 1368 2359 0 ?W+ swap 1380 1368 2359 0 R+ swap 1379 1368 2359 0 R+ CPU 4 swap 1378 1368 2359 0 R+ CPU 5 swap 1377 1368 2359 0 R+ swap 1376 1368 2359 0 R+ swap 1375 1368 2359 0 R+ CPU 2 swap 1374 1368 2359 0 R+ swap 1373 1368 2359 0 R+ CPU 0 swap 1372 1368 2359 0 R+ CPU 10 swap 1371 1368 2359 0 RL+ CPU 8 swap 1370 1368 2359 0 R+ swap 1369 1368 2359 0 ?W+ swap 1368 1361 2359 0 ?W+ swap 1361 1356 2359 0 S+ wait 0xfffff804b2ae4530 swap 1356 1354 2359 0 SW+ pause 0xfffff805873d2b08 sh 1355 2359 2359 0 S+ piperd 0xfffff800413178e8 tee 1354 2359 2359 0 SW+ wait 0xfffff806b7107530 sh 10144 10083 10083 0 S piperd 0xfffff80023d18be0 sh 10142 10083 10083 0 S piperd 0xfffff8003c5962f8 awk 10141 10083 10083 0 SW wait 0xfffff80023b71530 sh 10083 1 10083 0 SWs wait 0xfffff8016bcf6000 sh 2359 2358 2359 0 SW+ wait 0xfffff80023b50530 sh 2358 2356 2358 0 S+ select 0xfffff80023483ec0 bash 2356 2355 2356 0 SW+ pause 0xfffff800234ad0a8 csh 2355 2349 2355 2006 SW+ wait 0xfffff801504d0000 su 2349 2348 2349 2006 SWs+ wait 0xfffff801504d0530 bash 2348 2345 2345 2006 S select 0xfffff80023da8f40 sshd 2345 2281 2345 0 Ss select 0xfffff8002312f3c0 sshd 2341 2340 2341 0 S+ ttyin 0xfffff8000514ecb0 csh 2340 1 2340 0 SWs+ wait 0xfffff801504d0a60 login 2339 1 2339 0 Ss+ ttyin 0xfffff8000514b8b0 getty 2338 1 2338 0 Ss+ ttyin 0xfffff800232590b0 getty 2337 1 2337 0 Ss+ ttyin 0xfffff800232330b0 getty 2336 1 2336 0 Ss+ ttyin 0xfffff800232338b0 getty 2335 1 2335 0 Ss+ ttyin 0xfffff8002323e0b0 getty 2334 1 2334 0 Ss+ ttyin 0xfffff8002323e8b0 getty 2333 1 2333 0 Ss+ ttyin 0xfffff8002323d0b0 getty 2332 1 2332 0 Ss+ ttyin 0xfffff8002323d8b0 getty 2331 1 2331 0 Ss+ ttyin 0xfffff8002323c0b0 getty 2285 1 2285 0 ?Ws cron 2281 1 2281 0 Ss select 0xfffff80003a2fac0 sshd 2253 1 2253 0 Rs powerd 2250 1 2250 123 Rs (threaded) ntpd 100156 Run CPU 11 ntpd 100243 S usem 0xfffff801360c8e00 ntpd 2189 1 2189 0 Rs NLM: master 2186 1 2186 0 Ss select 0xfffff80005049840 rpc.statd 2140 1 2140 0 Rs CPU 6 watchdogd 2061 1 2061 0 Ss select 0xfffff8002312fcc0 rpcbind 2051 1 2051 0 Ss select 0xfffff8002312f540 syslogd 1980 1 1980 0 Ss select 0xfffff8003c607ec0 devd 1975 1 1975 65 Ds getblk 0xfffffe003ca520d8 dhclient 1928 1 1928 0 Ss select 0xfffff80040699740 dhclient 1925 1 1925 0 Ds ktrace_ 0xffffffff81c7a198 dhclient 34 0 0 0 RL [vnlru] 33 0 0 0 DL getblk 0xfffffe003a966a18 [syncer] 32 0 0 0 RL (threaded) [bufdaemon] 100131 RunQ [bufdaemon] 100133 D - 0xffffffff8180a980 [bufspacedaemon-0] 100134 D - 0xffffffff81812c40 [bufspacedaemon-1] 100135 D - 0xffffffff8181af00 [bufspacedaemon-2] 100136 D - 0xffffffff818231c0 [bufspacedaemon-3] 100137 D - 0xffffffff8182b480 [bufspacedaemon-4] 100138 D - 0xffffffff81833740 [bufspacedaemon-5] 100139 D - 0xffffffff8183ba00 [bufspacedaemon-6] 100153 D sdflush 0xfffff80023c3b6e8 [/ worker] 100197 D sdflush 0xfffff80023cbdae8 [/tmp worker] 31 0 0 0 RL [vmdaemon] 30 0 0 0 RL (threaded) [pagedaemon] 100128 Run CPU 7 [dom0] 100130 D swbufa 0xffffffff81d6cf44 [laundry: dom0] 100132 RunQ [uma] 29 0 0 0 DL idle 0xfffff80023420a60 [enc_daemon1] 28 0 0 0 DL idle 0xfffff80023334a60 [enc_daemon0] 27 0 0 0 RL [rand_harvestq] 26 0 0 0 DL waiting 0xffffffff81de3860 [sctp_iterator] 25 0 0 0 DL - 0xffffffff81d555dc [soaiod4] 24 0 0 0 DL - 0xffffffff81d555dc [soaiod3] 23 0 0 0 DL - 0xffffffff81d555dc [soaiod2] 22 0 0 0 DL - 0xffffffff81d555dc [soaiod1] 21 0 0 0 DL (threaded) [usb] 100096 D - 0xfffffe00d1584460 [usbus0] 100097 D - 0xfffffe00d15844b8 [usbus0] 100098 D - 0xfffffe00d1584510 [usbus0] 100099 D - 0xfffffe00d1584568 [usbus0] 100100 D - 0xfffffe00d15845c0 [usbus0] 100102 D - 0xfffffe0037b66d10 [usbus1] 100103 D - 0xfffffe0037b66d68 [usbus1] 100104 D - 0xfffffe0037b66dc0 [usbus1] 100105 D - 0xfffffe0037b66e18 [usbus1] 100106 D - 0xfffffe0037b66e70 [usbus1] 100107 D - 0xfffffe00d16abd10 [usbus2] 100108 D - 0xfffffe00d16abd68 [usbus2] 100109 D - 0xfffffe00d16abdc0 [usbus2] 100110 D - 0xfffffe00d16abe18 [usbus2] 100111 D - 0xfffffe00d16abe70 [usbus2] 20 0 0 0 RL (threaded) [cam] 100091 D - 0xffffffff819b72c0 [doneq0] 100092 Run CPU 3 [doneq1] 100093 D - 0xffffffff819b73c0 [doneq2] 100125 D - 0xffffffff819b7188 [scanner] 19 0 0 0 DL crypto_ 0xfffff80003bce450 [crypto returns 11] 18 0 0 0 DL crypto_ 0xfffff80003bce3f0 [crypto returns 10] 17 0 0 0 DL crypto_ 0xfffff80003bce390 [crypto returns 9] 16 0 0 0 DL crypto_ 0xfffff80003bce330 [crypto returns 8] 15 0 0 0 DL crypto_ 0xfffff80003bce2d0 [crypto returns 7] 9 0 0 0 DL crypto_ 0xfffff80003bce270 [crypto returns 6] 8 0 0 0 DL crypto_ 0xfffff80003bce210 [crypto returns 5] 7 0 0 0 DL crypto_ 0xfffff80003bce1b0 [crypto returns 4] 6 0 0 0 DL crypto_ 0xfffff80003bce150 [crypto returns 3] 5 0 0 0 DL crypto_ 0xfffff80003bce0f0 [crypto returns 2] 4 0 0 0 DL crypto_ 0xfffff80003bce090 [crypto returns 1] 3 0 0 0 DL crypto_ 0xfffff80003bce030 [crypto returns 0] 2 0 0 0 DL crypto_ 0xffffffff81d6b6b8 [crypto] 14 0 0 0 DL seqstat 0xfffff80003a89888 [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100062 D - 0xffffffff81d9c8c8 [g_event] 100063 D - 0xffffffff81d9c8d8 [g_up] 100064 D - 0xffffffff81d9c8d0 [g_down] 12 0 0 0 RL (threaded) [intr] 100017 I [swi6: task queue] 100018 I [swi6: Giant taskq] 100020 I [swi5: fast taskq] 100047 I [swi3: vm] 100048 I [swi1: netisr 0] 100049 Run CPU 1 [swi4: clock (0)] 100050 I [swi4: clock (1)] 100051 I [swi4: clock (2)] 100052 I [swi4: clock (3)] 100053 I [swi4: clock (4)] 100054 I [swi4: clock (5)] 100055 I [swi4: clock (6)] 100056 I [swi4: clock (7)] 100057 I [swi4: clock (8)] 100058 I [swi4: clock (9)] 100059 I [swi4: clock (10)] 100060 I [swi4: clock (11)] 100094 I [irq56: ahci0] 100095 I [irq19: xhci0] 100101 I [irq18: ehci0 ehci1] 100112 I [irq71: ahci1] 100113 I [swi0: uart uart] 100177 I [irq72: ioat0] 100178 I [irq73: ioat1] 100179 I [irq74: ioat2] 100180 I [irq75: ioat3] 100181 I [irq76: ioat4] 100182 I [irq77: ioat5] 100183 I [irq78: ioat6] 100184 I [irq79: ioat7] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 100007 CanRun [idle: cpu4] 100008 CanRun [idle: cpu5] 100009 CanRun [idle: cpu6] 100010 CanRun [idle: cpu7] 100011 CanRun [idle: cpu8] 100012 CanRun [idle: cpu9] 100013 CanRun [idle: cpu10] 100014 CanRun [idle: cpu11] 1 0 1 0 SLs wait 0xfffff800038a8530 [init] 10 0 0 0 DL audit_w 0xffffffff81de44f0 [audit] 0 0 0 0 RLs (threaded) [kernel] 100000 Run CPU 9 [swapper] 100015 D - 0xfffff800038ee000 [kqueue_ctx taskq] 100016 D - 0xfffff800038ede00 [aiod_kick taskq] 100019 D - 0xfffff800038edb00 [thread taskq] 100021 D - 0xfffff800038d3000 [config_0] 100022 D - 0xfffff800038d2e00 [if_config_tqg_0] 100023 D - 0xfffff800038d2d00 [softirq_0] 100024 D - 0xfffff800038d2c00 [softirq_1] 100025 D - 0xfffff800038d2b00 [softirq_2] 100026 D - 0xfffff800038d2a00 [softirq_3] 100027 D - 0xfffff800038d2900 [softirq_4] 100028 D - 0xfffff800038d2800 [softirq_5] 100029 D - 0xfffff800038d2700 [softirq_6] 100030 D - 0xfffff800038d2600 [softirq_7] 100031 D - 0xfffff800038d2500 [softirq_8] 100032 D - 0xfffff800038d2400 [softirq_9] 100033 D - 0xfffff800038d2300 [softirq_10] 100034 D - 0xfffff800038d2200 [softirq_11] 100035 D - 0xfffff800038d2100 [if_io_tqg_0] 100036 D - 0xfffff800038d2000 [if_io_tqg_1] 100037 D - 0xfffff800038d1e00 [if_io_tqg_2] 100038 D - 0xfffff800038d1d00 [if_io_tqg_3] 100039 D - 0xfffff800038d1c00 [if_io_tqg_4] 100040 D - 0xfffff800038d1b00 [if_io_tqg_5] 100041 D - 0xfffff800038d1a00 [if_io_tqg_6] 100042 D - 0xfffff800038d1900 [if_io_tqg_7] 100043 D - 0xfffff800038d1800 [if_io_tqg_8] 100044 D - 0xfffff800038d1700 [if_io_tqg_9] 100045 D - 0xfffff800038d1600 [if_io_tqg_10] 100046 D - 0xfffff800038d1500 [if_io_tqg_11] 100061 D - 0xfffff80003b15100 [firmware taskq] 100066 D - 0xfffff80003b15000 [crypto_0] 100067 D - 0xfffff80003b15000 [crypto_1] 100068 D - 0xfffff80003b15000 [crypto_2] 100069 D - 0xfffff80003b15000 [crypto_3] 100070 D - 0xfffff80003b15000 [crypto_4] 100071 D - 0xfffff80003b15000 [crypto_5] 100072 D - 0xfffff80003b15000 [crypto_6] 100073 D - 0xfffff80003b15000 [crypto_7] 100074 D - 0xfffff80003b15000 [crypto_8] 100075 D - 0xfffff80003b15000 [crypto_9] 100076 D - 0xfffff80003b15000 [crypto_10] 100077 D - 0xfffff80003b15000 [crypto_11] 100114 D - 0xfffff8000518d200 [mca taskq] 100121 D - 0xfffff80003b16d00 [acpi_task_0] 100122 D - 0xfffff80003b16d00 [acpi_task_1] 100123 D - 0xfffff80003b16d00 [acpi_task_2] 100124 D - 0xfffff80003b16e00 [CAM taskq] 6001 1 2359 0 Z+ pdfork db:0:pho> allt Tracing command sed pid 1697 tid 105207 td 0xfffff803f2c9e6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe033943b7d0 mi_switch() at mi_switch+0x182/frame 0xfffffe033943b810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033943b850 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe033943b8b0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe033943b8e0 _sleep() at _sleep+0x2c3/frame 0xfffffe033943b980 pipe_read() at pipe_read+0x428/frame 0xfffffe033943b9e0 dofileread() at dofileread+0x81/frame 0xfffffe033943ba30 kern_readv() at kern_readv+0x49/frame 0xfffffe033943ba70 sys_read() at sys_read+0x83/frame 0xfffffe033943bac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe033943bbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe033943bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80041fb1a, rsp = 0x7fffffffe468, rbp = 0x7fffffffe480 --- Tracing command sh pid 1696 tid 105215 td 0xfffff803f2c9e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339463700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe0339463708 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 1695 tid 102100 td 0xfffff80077c72000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e8126690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e8126698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command dd pid 1674 tid 102552 td 0xfffff806bea43000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033e330040 mi_switch() at mi_switch+0x182/frame 0xfffffe033e330080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033e3300c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe033e3300f0 _sleep() at _sleep+0x2cd/frame 0xfffffe033e330190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe033e3301b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe033e3301e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe033e330290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe033e3302f0 getblkx() at getblkx+0x719/frame 0xfffffe033e3303c0 getblk() at getblk+0x22/frame 0xfffffe033e3303f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe033e330590 ffs_write() at ffs_write+0x217/frame 0xfffffe033e330630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe033e330740 vn_write() at vn_write+0x27a/frame 0xfffffe033e3307c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe033e330820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe033e330970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe033e3309e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe033e330a30 kern_writev() at kern_writev+0x49/frame 0xfffffe033e330a70 sys_write() at sys_write+0x83/frame 0xfffffe033e330ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe033e330bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe033e330bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1673 tid 102569 td 0xfffff806bea2a000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033e640040 mi_switch() at mi_switch+0x182/frame 0xfffffe033e640080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033e6400c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe033e6400f0 _sleep() at _sleep+0x2cd/frame 0xfffffe033e640190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe033e6401b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe033e6401e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe033e640290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe033e6402f0 getblkx() at getblkx+0x719/frame 0xfffffe033e6403c0 getblk() at getblk+0x22/frame 0xfffffe033e6403f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe033e640590 ffs_write() at ffs_write+0x217/frame 0xfffffe033e640630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe033e640740 vn_write() at vn_write+0x27a/frame 0xfffffe033e6407c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe033e640820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe033e640970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe033e6409e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe033e640a30 kern_writev() at kern_writev+0x49/frame 0xfffffe033e640a70 sys_write() at sys_write+0x83/frame 0xfffffe033e640ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe033e640bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe033e640bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1672 tid 101262 td 0xfffff806be541000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339481040 mi_switch() at mi_switch+0x182/frame 0xfffffe0339481080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394810c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe03394810f0 _sleep() at _sleep+0x2cd/frame 0xfffffe0339481190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe03394811b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe03394811e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe0339481290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe03394812f0 getblkx() at getblkx+0x719/frame 0xfffffe03394813c0 getblk() at getblk+0x22/frame 0xfffffe03394813f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe0339481590 ffs_write() at ffs_write+0x217/frame 0xfffffe0339481630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe0339481740 vn_write() at vn_write+0x27a/frame 0xfffffe03394817c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0339481820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe0339481970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe03394819e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0339481a30 kern_writev() at kern_writev+0x49/frame 0xfffffe0339481a70 sys_write() at sys_write+0x83/frame 0xfffffe0339481ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339481bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339481bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1671 tid 105234 td 0xfffff803f2231000 sched_switch() at sched_switch+0x38f/frame 0xfffffe03394c2040 mi_switch() at mi_switch+0x182/frame 0xfffffe03394c2080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394c20c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe03394c20f0 _sleep() at _sleep+0x2cd/frame 0xfffffe03394c2190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe03394c21b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe03394c21e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe03394c2290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe03394c22f0 getblkx() at getblkx+0x719/frame 0xfffffe03394c23c0 getblk() at getblk+0x22/frame 0xfffffe03394c23f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe03394c2590 ffs_write() at ffs_write+0x217/frame 0xfffffe03394c2630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe03394c2740 vn_write() at vn_write+0x27a/frame 0xfffffe03394c27c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe03394c2820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe03394c2970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe03394c29e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe03394c2a30 kern_writev() at kern_writev+0x49/frame 0xfffffe03394c2a70 sys_write() at sys_write+0x83/frame 0xfffffe03394c2ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe03394c2bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe03394c2bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1670 tid 105197 td 0xfffff806be723000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339409040 mi_switch() at mi_switch+0x182/frame 0xfffffe0339409080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394090c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe03394090f0 _sleep() at _sleep+0x2cd/frame 0xfffffe0339409190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe03394091b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe03394091e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe0339409290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe03394092f0 getblkx() at getblkx+0x719/frame 0xfffffe03394093c0 getblk() at getblk+0x22/frame 0xfffffe03394093f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe0339409590 ffs_write() at ffs_write+0x217/frame 0xfffffe0339409630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe0339409740 vn_write() at vn_write+0x27a/frame 0xfffffe03394097c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0339409820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe0339409970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe03394099e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0339409a30 kern_writev() at kern_writev+0x49/frame 0xfffffe0339409a70 sys_write() at sys_write+0x83/frame 0xfffffe0339409ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339409bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339409bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1669 tid 105200 td 0xfffff803f2f14000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339418040 mi_switch() at mi_switch+0x182/frame 0xfffffe0339418080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394180c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe03394180f0 _sleep() at _sleep+0x2cd/frame 0xfffffe0339418190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe03394181b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe03394181e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe0339418290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe03394182f0 getblkx() at getblkx+0x719/frame 0xfffffe03394183c0 getblk() at getblk+0x22/frame 0xfffffe03394183f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe0339418590 ffs_write() at ffs_write+0x217/frame 0xfffffe0339418630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe0339418740 vn_write() at vn_write+0x27a/frame 0xfffffe03394187c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0339418820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe0339418970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe03394189e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0339418a30 kern_writev() at kern_writev+0x49/frame 0xfffffe0339418a70 sys_write() at sys_write+0x83/frame 0xfffffe0339418ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339418bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339418bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1668 tid 101246 td 0xfffff803f2b456e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe033949a040 mi_switch() at mi_switch+0x182/frame 0xfffffe033949a080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033949a0c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe033949a0f0 _sleep() at _sleep+0x2cd/frame 0xfffffe033949a190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe033949a1b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe033949a1e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe033949a290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe033949a2f0 getblkx() at getblkx+0x719/frame 0xfffffe033949a3c0 getblk() at getblk+0x22/frame 0xfffffe033949a3f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe033949a590 ffs_write() at ffs_write+0x217/frame 0xfffffe033949a630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe033949a740 vn_write() at vn_write+0x27a/frame 0xfffffe033949a7c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe033949a820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe033949a970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe033949a9e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe033949aa30 kern_writev() at kern_writev+0x49/frame 0xfffffe033949aa70 sys_write() at sys_write+0x83/frame 0xfffffe033949aac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe033949abf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe033949abf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command dd pid 1667 tid 101247 td 0xfffff803f2b45000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339490040 mi_switch() at mi_switch+0x182/frame 0xfffffe0339490080 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394900c0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe03394900f0 _sleep() at _sleep+0x2cd/frame 0xfffffe0339490190 vm_wait_doms() at vm_wait_doms+0xcd/frame 0xfffffe03394901b0 vm_domainset_iter_page() at vm_domainset_iter_page+0xa4/frame 0xfffffe03394901e0 vm_page_grab_pages() at vm_page_grab_pages+0x288/frame 0xfffffe0339490290 allocbuf() at allocbuf+0x2f9/frame 0xfffffe03394902f0 getblkx() at getblkx+0x719/frame 0xfffffe03394903c0 getblk() at getblk+0x22/frame 0xfffffe03394903f0 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1775/frame 0xfffffe0339490590 ffs_write() at ffs_write+0x217/frame 0xfffffe0339490630 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe0339490740 vn_write() at vn_write+0x27a/frame 0xfffffe03394907c0 vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe0339490820 vn_io_fault1() at vn_io_fault1+0x16c/frame 0xfffffe0339490970 vn_io_fault() at vn_io_fault+0x182/frame 0xfffffe03394909e0 dofilewrite() at dofilewrite+0x81/frame 0xfffffe0339490a30 kern_writev() at kern_writev+0x49/frame 0xfffffe0339490a70 sys_write() at sys_write+0x83/frame 0xfffffe0339490ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339490bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339490bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800434ada, rsp = 0x7fffffffe698, rbp = 0x7fffffffe6d0 --- Tracing command sh pid 1393 tid 104260 td 0xfffff806be22b000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033a8727d0 mi_switch() at mi_switch+0x182/frame 0xfffffe033a872810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033a872850 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe033a8728b0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe033a8728e0 _sleep() at _sleep+0x2c3/frame 0xfffffe033a872980 pipe_read() at pipe_read+0x428/frame 0xfffffe033a8729e0 dofileread() at dofileread+0x81/frame 0xfffffe033a872a30 kern_readv() at kern_readv+0x49/frame 0xfffffe033a872a70 sys_read() at sys_read+0x83/frame 0xfffffe033a872ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe033a872bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe033a872bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800476b1a, rsp = 0x7fffffffdfe8, rbp = 0x7fffffffe130 --- Tracing command swap pid 1382 tid 105232 td 0xfffff806be27b6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe03394b8700 mi_switch() at mi_switch+0x182/frame 0xfffffe03394b8740 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe03394b8780 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe03394b87c0 _sleep() at _sleep+0x26c/frame 0xfffffe03394b8860 vm_waitpfault() at vm_waitpfault+0x79/frame 0xfffffe03394b8890 vm_fault() at vm_fault+0xa10/frame 0xfffffe03394b8a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe03394b8a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe03394b8ac0 trap() at trap+0x470/frame 0xfffffe03394b8bf0 calltrap() at calltrap+0x8/frame 0xfffffe03394b8bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1381 tid 103153 td 0xfffff80270a8b000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033ea3b700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe033ea3b708 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 1380 tid 102577 td 0xfffff806bea2d000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033e63b700 mi_switch() at mi_switch+0x182/frame 0xfffffe033e63b740 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033e63b780 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe033e63b7c0 _sleep() at _sleep+0x26c/frame 0xfffffe033e63b860 vm_waitpfault() at vm_waitpfault+0x79/frame 0xfffffe033e63b890 vm_fault() at vm_fault+0xa10/frame 0xfffffe033e63ba00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033e63ba40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033e63bac0 trap() at trap+0x470/frame 0xfffffe033e63bbf0 calltrap() at calltrap+0x8/frame 0xfffffe033e63bbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1379 tid 105206 td 0xfffff803f26e5000 (CPU 4) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e197dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e197df0 trap() at trap+0x61/frame 0xfffffe003e197f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e197f20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe0339436720, rbp = 0xfffffe0339436730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe0339436730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe03394367e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe0339436830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe0339436890 vm_fault() at vm_fault+0x165c/frame 0xfffffe0339436a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe0339436a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe0339436ac0 trap() at trap+0x470/frame 0xfffffe0339436bf0 calltrap() at calltrap+0x8/frame 0xfffffe0339436bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1378 tid 104383 td 0xfffff806be50c000 (CPU 5) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1a3dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1a3df0 trap() at trap+0x61/frame 0xfffffe003e1a3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1a3f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe033ab2b720, rbp = 0xfffffe033ab2b730 --- lock_delay() at lock_delay+0x32/frame 0xfffffe033ab2b730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033ab2b7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033ab2b830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033ab2b890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033ab2ba00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033ab2ba40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033ab2bac0 trap() at trap+0x470/frame 0xfffffe033ab2bbf0 calltrap() at calltrap+0x8/frame 0xfffffe033ab2bbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1377 tid 105213 td 0xfffff806a37d2000 sched_switch() at sched_switch+0x38f/frame 0xfffffe03394595b0 mi_switch() at mi_switch+0x182/frame 0xfffffe03394595f0 sched_preempt() at sched_preempt+0x97/frame 0xfffffe0339459620 ipi_bitmap_handler() at ipi_bitmap_handler+0x64/frame 0xfffffe0339459650 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0xb3/frame 0xfffffe0339459650 --- interrupt, rip = 0xffffffff80c119f0, rsp = 0xfffffe0339459720, rbp = 0xfffffe0339459730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe0339459730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe03394597e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe0339459830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe0339459890 vm_fault() at vm_fault+0x165c/frame 0xfffffe0339459a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe0339459a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe0339459ac0 trap() at trap+0x470/frame 0xfffffe0339459bf0 calltrap() at calltrap+0x8/frame 0xfffffe0339459bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1376 tid 100659 td 0xfffff807836466e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e4c415b0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e4c415f0 sched_preempt() at sched_preempt+0x97/frame 0xfffffe00e4c41620 ipi_bitmap_handler() at ipi_bitmap_handler+0x64/frame 0xfffffe00e4c41650 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0xb3/frame 0xfffffe00e4c41650 --- interrupt, rip = 0xffffffff80c119f0, rsp = 0xfffffe00e4c41720, rbp = 0xfffffe00e4c41730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe00e4c41730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e4c417e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e4c41830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e4c41890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e4c41a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e4c41a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e4c41ac0 trap() at trap+0x470/frame 0xfffffe00e4c41bf0 calltrap() at calltrap+0x8/frame 0xfffffe00e4c41bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1375 tid 103743 td 0xfffff803f22d46e0 (CPU 2) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e17fdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e17fdf0 trap() at trap+0x61/frame 0xfffffe003e17ff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e17ff20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe033f654720, rbp = 0xfffffe033f654730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe033f654730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033f6547e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033f654830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033f654890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033f654a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033f654a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033f654ac0 trap() at trap+0x470/frame 0xfffffe033f654bf0 calltrap() at calltrap+0x8/frame 0xfffffe033f654bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1374 tid 100588 td 0xfffff80783072000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e4c505b0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e4c505f0 sched_preempt() at sched_preempt+0x97/frame 0xfffffe00e4c50620 ipi_bitmap_handler() at ipi_bitmap_handler+0x64/frame 0xfffffe00e4c50650 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0xb3/frame 0xfffffe00e4c50650 --- interrupt, rip = 0xffffffff80c119f0, rsp = 0xfffffe00e4c50720, rbp = 0xfffffe00e4c50730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe00e4c50730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e4c507e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e4c50830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e4c50890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e4c50a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e4c50a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e4c50ac0 trap() at trap+0x470/frame 0xfffffe00e4c50bf0 calltrap() at calltrap+0x8/frame 0xfffffe00e4c50bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1373 tid 102278 td 0xfffff807c08a5000 (CPU 0) cpustop_handler() at cpustop_handler+0x2a/frame 0xffffffff81d84f40 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff81d84f60 trap() at trap+0x61/frame 0xffffffff81d85090 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81d85090 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe00e56a4720, rbp = 0xfffffe00e56a4730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe00e56a4730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e56a47e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e56a4830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e56a4890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e56a4a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e56a4a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e56a4ac0 trap() at trap+0x470/frame 0xfffffe00e56a4bf0 calltrap() at calltrap+0x8/frame 0xfffffe00e56a4bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1372 tid 105077 td 0xfffff806be2936e0 (CPU 10) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1dfdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1dfdf0 trap() at trap+0x61/frame 0xfffffe003e1dff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1dff20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xfffffe033afad720, rbp = 0xfffffe033afad730 --- lock_delay() at lock_delay+0x30/frame 0xfffffe033afad730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe033afad7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe033afad830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe033afad890 vm_fault() at vm_fault+0x165c/frame 0xfffffe033afada00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033afada40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033afadac0 trap() at trap+0x470/frame 0xfffffe033afadbf0 calltrap() at calltrap+0x8/frame 0xfffffe033afadbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1371 tid 102427 td 0xfffff80077b396e0 (CPU 8) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1c7dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1c7df0 trap() at trap+0x61/frame 0xfffffe003e1c7f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1c7f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe00e57ad720, rbp = 0xfffffe00e57ad730 --- lock_delay() at lock_delay+0x32/frame 0xfffffe00e57ad730 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e57ad7e0 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e57ad830 vm_pageout_oom() at vm_pageout_oom+0xed/frame 0xfffffe00e57ad890 vm_fault() at vm_fault+0x165c/frame 0xfffffe00e57ada00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe00e57ada40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe00e57adac0 trap() at trap+0x470/frame 0xfffffe00e57adbf0 calltrap() at calltrap+0x8/frame 0xfffffe00e57adbf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1370 tid 103900 td 0xfffff802707f6000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033f944700 mi_switch() at mi_switch+0x182/frame 0xfffffe033f944740 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe033f944780 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe033f9447c0 _sleep() at _sleep+0x26c/frame 0xfffffe033f944860 vm_waitpfault() at vm_waitpfault+0x79/frame 0xfffffe033f944890 vm_fault() at vm_fault+0xa10/frame 0xfffffe033f944a00 vm_fault_trap() at vm_fault_trap+0x6e/frame 0xfffffe033f944a40 trap_pfault() at trap_pfault+0x1f3/frame 0xfffffe033f944ac0 trap() at trap+0x470/frame 0xfffffe033f944bf0 calltrap() at calltrap+0x8/frame 0xfffffe033f944bf0 --- trap 0xc, rip = 0x8003a3a2f, rsp = 0x7fffffffe5c0, rbp = 0x7fffffffe5d0 --- Tracing command swap pid 1369 tid 105242 td 0xfffff8057ce626e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe03394ea700 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe03394ea708 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 1368 tid 102099 td 0xfffff80077c726e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e8121690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e8121698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command swap pid 1361 tid 101263 td 0xfffff803f22316e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339486690 mi_switch() at mi_switch+0x182/frame 0xfffffe03394866d0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0339486710 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe0339486770 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe03394867a0 _sleep() at _sleep+0x2c3/frame 0xfffffe0339486840 kern_wait6() at kern_wait6+0x482/frame 0xfffffe03394868d0 sys_wait4() at sys_wait4+0x78/frame 0xfffffe0339486ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339486bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339486bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8003c391a, rsp = 0x7fffffffe638, rbp = 0x7fffffffe750 --- Tracing command sh pid 1356 tid 103392 td 0xfffff8057ce0c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe033eef4880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe033eef4888 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command tee pid 1355 tid 102101 td 0xfffff80077b226e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e812b7d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e812b810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e812b850 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e812b8b0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e812b8e0 _sleep() at _sleep+0x2c3/frame 0xfffffe00e812b980 pipe_read() at pipe_read+0x428/frame 0xfffffe00e812b9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e812ba30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e812ba70 sys_read() at sys_read+0x83/frame 0xfffffe00e812bac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e812bbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e812bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800417b1a, rsp = 0x7fffffffe738, rbp = 0x7fffffffe780 --- Tracing command sh pid 1354 tid 105199 td 0xfffff803f2f146e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe0339413690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe0339413698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 10144 tid 100260 td 0xfffff8016bdeb6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e4a817d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e4a81810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e4a81850 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e4a818b0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e4a818e0 _sleep() at _sleep+0x2c3/frame 0xfffffe00e4a81980 pipe_read() at pipe_read+0x428/frame 0xfffffe00e4a819e0 dofileread() at dofileread+0x81/frame 0xfffffe00e4a81a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e4a81a70 sys_read() at sys_read+0x83/frame 0xfffffe00e4a81ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e4a81bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e4a81bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800476b1a, rsp = 0x7fffffffe2c8, rbp = 0x7fffffffe3c0 --- Tracing command awk pid 10142 tid 100194 td 0xfffff80023e56000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f2c7d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f2c810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f2c850 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f2c8b0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f2c8e0 _sleep() at _sleep+0x2c3/frame 0xfffffe00e2f2c980 pipe_read() at pipe_read+0x428/frame 0xfffffe00e2f2c9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f2ca30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f2ca70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f2cac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f2cbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f2cbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80047bb1a, rsp = 0x7fffffffe3a8, rbp = 0x7fffffffe3c0 --- Tracing command sh pid 10141 tid 100267 td 0xfffff800406886e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e4aa4690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e4aa4698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 10083 tid 100278 td 0xfffff8016be5e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e4adb690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e4adb698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 2359 tid 100168 td 0xfffff80023db36e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e6e690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e2e6e698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 2358 tid 100214 td 0xfffff801505b0000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f9c6b0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f9c6f0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f9c730 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f9c790 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f9c7c0 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f9c820 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e2f9c860 kern_select() at kern_select+0x962/frame 0xfffffe00e2f9ca40 sys_pselect() at sys_pselect+0x127/frame 0xfffffe00e2f9cac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f9cbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f9cbf0 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x800517a3a, rsp = 0x7fffffffd6c8, rbp = 0x7fffffffd790 --- Tracing command csh pid 2356 tid 100144 td 0xfffff800234d7000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1bce880 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e1bce888 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 2355 tid 100213 td 0xfffff801505b06e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f97690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e2f97698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 2349 tid 100212 td 0xfffff801505cb000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f92690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e2f92698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 2348 tid 100145 td 0xfffff80003a52000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1bd36f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e1bd3730 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e1bd3770 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e1bd37d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e1bd3800 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e1bd3860 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e1bd38a0 kern_select() at kern_select+0x962/frame 0xfffffe00e1bd3a80 sys_select() at sys_select+0x56/frame 0xfffffe00e1bd3ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1bd3bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1bd3bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8008f5afa, rsp = 0x7fffffffa608, rbp = 0x7fffffffa640 --- Tracing command sshd pid 2345 tid 100176 td 0xfffff80023c80000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e96770 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e967b0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e967f0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2e96850 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2e96880 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2e968e0 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e2e96920 kern_poll() at kern_poll+0x48b/frame 0xfffffe00e2e96aa0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e2e96ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e96bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e96bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80089913a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6a0 --- Tracing command csh pid 2341 tid 100154 td 0xfffff80023c7f000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e04750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e04790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e047d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2e04830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2e04860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2e048c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2e048f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2e04950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2e04980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2e049e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2e04a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2e04a70 sys_read() at sys_read+0x83/frame 0xfffffe00e2e04ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e04bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e04bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004fbb1a, rsp = 0x7fffffffe278, rbp = 0x7fffffffe2a0 --- Tracing command login pid 2340 tid 100210 td 0xfffff80040fd76e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f88690 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e2f88698 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command getty pid 2339 tid 100209 td 0xfffff80040f6a000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f83750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f83790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f837d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f83830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f83860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f838c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f838f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f83950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f83980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f839e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f83a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f83a70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f83ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f83bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f83bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2338 tid 100208 td 0xfffff80040f6a6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f7e750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f7e790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f7e7d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f7e830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f7e860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f7e8c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f7e8f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f7e950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f7e980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f7e9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f7ea30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f7ea70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f7eac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f7ebf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f7ebf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2337 tid 100207 td 0xfffff80150452000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f79750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f79790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f797d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f79830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f79860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f798c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f798f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f79950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f79980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f799e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f79a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f79a70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f79ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f79bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f79bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2336 tid 100206 td 0xfffff801504526e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f74750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f74790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f747d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f74830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f74860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f748c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f748f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f74950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f74980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f749e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f74a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f74a70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f74ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f74bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f74bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2335 tid 100205 td 0xfffff80040fba6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f6f750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f6f790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f6f7d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f6f830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f6f860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f6f8c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f6f8f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f6f950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f6f980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f6f9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f6fa30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f6fa70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f6fac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f6fbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f6fbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2334 tid 100204 td 0xfffff8015010c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f6a750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f6a790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f6a7d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f6a830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f6a860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f6a8c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f6a8f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f6a950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f6a980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f6a9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f6aa30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f6aa70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f6aac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f6abf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f6abf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2333 tid 100203 td 0xfffff8015010c6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f65750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f65790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f657d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f65830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f65860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f658c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f658f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f65950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f65980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f659e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f65a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f65a70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f65ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f65bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f65bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2332 tid 100201 td 0xfffff8015010b000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f5b750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f5b790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f5b7d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f5b830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2f5b860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2f5b8c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e2f5b8f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e2f5b950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e2f5b980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e2f5b9e0 dofileread() at dofileread+0x81/frame 0xfffffe00e2f5ba30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e2f5ba70 sys_read() at sys_read+0x83/frame 0xfffffe00e2f5bac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f5bbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f5bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command getty pid 2331 tid 100142 td 0xfffff8000511a6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1bc4750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e1bc4790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e1bc47d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e1bc4830 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e1bc4860 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e1bc48c0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e1bc48f0 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e1bc4950 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e1bc4980 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e1bc49e0 dofileread() at dofileread+0x81/frame 0xfffffe00e1bc4a30 kern_readv() at kern_readv+0x49/frame 0xfffffe00e1bc4a70 sys_read() at sys_read+0x83/frame 0xfffffe00e1bc4ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1bc4bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1bc4bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800433b1a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 --- Tracing command cron pid 2285 tid 100202 td 0xfffff80023aa8000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f60830 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fa10 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fa20 trap() at trap+0x73/frame 0xfffffe003e53fb50 calltrap() at calltrap+0x8/frame 0xfffffe003e53fb50 --- trap 0xc, rip = 0xffffffff810407b0, rsp = 0xfffffe003e53fc20, rbp = 0xfffffe003e53fcb0 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x32/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e2f60838 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fca0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fcb0 db_get_value() at db_get_value+0x4c/frame 0xfffffe003e53fcf0 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe003e53fd80 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe003e53fda0 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 2281 tid 100186 td 0xfffff8003c347000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2ef86f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2ef8730 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2ef8770 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2ef87d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2ef8800 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2ef8860 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e2ef88a0 kern_select() at kern_select+0x962/frame 0xfffffe00e2ef8a80 sys_select() at sys_select+0x56/frame 0xfffffe00e2ef8ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2ef8bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2ef8bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8008f5afa, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe730 --- Tracing command powerd pid 2253 tid 100198 td 0xfffff80023f7e6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f4c6d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f4c710 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f4c750 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f4c7b0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00e2f4c7f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e2f4c860 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e2f4c8a0 kern_select() at kern_select+0x962/frame 0xfffffe00e2f4ca80 sys_select() at sys_select+0x56/frame 0xfffffe00e2f4cac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f4cbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f4cbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800430afa, rsp = 0x7fffffffec08, rbp = 0x7fffffffed50 --- Tracing command ntpd pid 2250 tid 100156 td 0xfffff80023a97000 (CPU 11) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1ebdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1ebdf0 trap() at trap+0x61/frame 0xfffffe003e1ebf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1ebf20 --- trap 0x13, rip = 0xffffffff80f054a6, rsp = 0xfffffe00e2e0e630, rbp = 0xfffffe00e2e0e660 --- uma_zalloc_arg() at uma_zalloc_arg+0xa6/frame 0xfffffe00e2e0e660 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e2e0e6f0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e2e0e730 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00e2e0e7b0 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00e2e0e7e0 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00e2e0e840 zone_import() at zone_import+0x177/frame 0xfffffe00e2e0e900 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00e2e0e990 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e2e0e9d0 cache_free() at cache_free+0x27b/frame 0xfffffe00e2e0ea20 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe00e2e0ea70 sigqueue_get() at sigqueue_get+0x15c/frame 0xfffffe00e2e0ead0 postsig() at postsig+0x129/frame 0xfffffe00e2e0eba0 ast() at ast+0x4eb/frame 0xfffffe00e2e0ebf0 doreti_ast() at doreti_ast+0x1f/frame 0x7fffffffdfc0 Tracing command ntpd pid 2250 tid 100243 td 0xfffff806be5416e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe03394317e0 mi_switch() at mi_switch+0x182/frame 0xfffffe0339431820 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0339431860 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe03394318c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe03394318f0 _sleep() at _sleep+0x2c3/frame 0xfffffe0339431990 umtxq_sleep() at umtxq_sleep+0x19e/frame 0xfffffe03394319f0 do_sem2_wait() at do_sem2_wait+0x422/frame 0xfffffe0339431a70 __umtx_op_sem2_wait() at __umtx_op_sem2_wait+0x76/frame 0xfffffe0339431ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe0339431bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0339431bf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80071df7a, rsp = 0x7fffdfffded8, rbp = 0x7fffdfffdf50 --- Tracing command rpc.lockd pid 2189 tid 100163 td 0xfffff80023db2000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e3d610 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e3d650 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e3d690 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2e3d6f0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00e2e3d730 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e2e3d7a0 svc_run_internal() at svc_run_internal+0xabe/frame 0xfffffe00e2e3d8e0 svc_run() at svc_run+0x253/frame 0xfffffe00e2e3d940 sys_nlm_syscall() at sys_nlm_syscall+0x745/frame 0xfffffe00e2e3dac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e3dbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e3dbf0 --- syscall (154, FreeBSD ELF64, sys_nlm_syscall), rip = 0x8002f54aa, rsp = 0x7fffffffec88, rbp = 0x7fffffffed50 --- Tracing command rpc.statd pid 2186 tid 100193 td 0xfffff8015010b6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f276d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f27710 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f27750 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2f277b0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00e2f277f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e2f27860 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e2f278a0 kern_select() at kern_select+0x962/frame 0xfffffe00e2f27a80 sys_select() at sys_select+0x56/frame 0xfffffe00e2f27ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2f27bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2f27bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800427afa, rsp = 0x7fffffffeb38, rbp = 0x7fffffffec80 --- Tracing command watchdogd pid 2140 tid 100146 td 0xfffff80023a976e0 (CPU 6) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1afdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1afdf0 trap() at trap+0x61/frame 0xfffffe003e1aff20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1aff20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe00e1bd8290, rbp = 0xfffffe00e1bd82a0 --- lock_delay() at lock_delay+0x32/frame 0xfffffe00e1bd82a0 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe00e1bd8350 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe00e1bd83a0 cache_alloc() at cache_alloc+0x128/frame 0xfffffe00e1bd8430 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8470 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd8500 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8540 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd85d0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8610 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00e1bd8690 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00e1bd86c0 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00e1bd8720 zone_import() at zone_import+0x177/frame 0xfffffe00e1bd87e0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00e1bd8870 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd88b0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00e1bd8940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00e1bd8980 fork1() at fork1+0x1c1/frame 0xfffffe00e1bd8a60 sys_vfork() at sys_vfork+0x4e/frame 0xfffffe00e1bd8ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1bd8bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1bd8bf0 --- syscall (66, FreeBSD ELF64, sys_vfork), rip = 0x8003a0e6d, rsp = 0x7fffffffeac0, rbp = 0x7fffffffeb70 --- Tracing command rpcbind pid 2061 tid 100166 td 0xfffff80023f7e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e64750 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e64790 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e647d0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2e64830 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00e2e64870 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e2e648e0 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e2e64920 kern_poll() at kern_poll+0x48b/frame 0xfffffe00e2e64aa0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e2e64ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e64bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e64bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8003cf13a, rsp = 0x7fffffffcac8, rbp = 0x7fffffffece0 --- Tracing command syslogd pid 2051 tid 100174 td 0xfffff80023ac46e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e8c6f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e8c730 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e8c770 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e2e8c7d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e2e8c800 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e2e8c860 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e2e8c8a0 kern_select() at kern_select+0x962/frame 0xfffffe00e2e8ca80 sys_select() at sys_select+0x56/frame 0xfffffe00e2e8cac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e8cbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e8cbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800438afa, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffed50 --- Tracing command devd pid 1980 tid 100148 td 0xfffff80023b5b6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1be26d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e1be2710 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e1be2750 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e1be27b0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00e1be27f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e1be2860 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e1be28a0 kern_select() at kern_select+0x962/frame 0xfffffe00e1be2a80 sys_select() at sys_select+0x56/frame 0xfffffe00e1be2ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1be2bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1be2bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x2f5caa, rsp = 0x7fffffffcbe8, rbp = 0x7fffffffed70 --- Tracing command dhclient pid 1975 tid 100157 td 0xfffff80023aa86e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e13130 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e13170 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e131b0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00e2e131e0 sleeplk() at sleeplk+0x107/frame 0xfffffe00e2e13240 lockmgr_xlock_hard() at lockmgr_xlock_hard+0x225/frame 0xfffffe00e2e13300 __lockmgr_args() at __lockmgr_args+0x60d/frame 0xfffffe00e2e133c0 getblkx() at getblkx+0x15f/frame 0xfffffe00e2e134a0 breadn_flags() at breadn_flags+0x44/frame 0xfffffe00e2e13510 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x1900/frame 0xfffffe00e2e136e0 ffs_write() at ffs_write+0x217/frame 0xfffffe00e2e13780 VOP_WRITE_APV() at VOP_WRITE_APV+0x113/frame 0xfffffe00e2e13890 ktr_writerequest() at ktr_writerequest+0x211/frame 0xfffffe00e2e13970 ktr_submitrequest() at ktr_submitrequest+0x47/frame 0xfffffe00e2e13990 kern_sendit() at kern_sendit+0x294/frame 0xfffffe00e2e13a20 sendit() at sendit+0x1d5/frame 0xfffffe00e2e13a70 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe00e2e13ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e13bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e13bf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x80043708a, rsp = 0x7fffffffe0e8, rbp = 0x7fffffffe1b0 --- Tracing command dhclient pid 1928 tid 100150 td 0xfffff800050366e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1bec770 mi_switch() at mi_switch+0x182/frame 0xfffffe00e1bec7b0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e1bec7f0 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00e1bec850 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00e1bec880 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e1bec8e0 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e1bec920 kern_poll() at kern_poll+0x48b/frame 0xfffffe00e1becaa0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e1becac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e1becbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e1becbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8003f213a, rsp = 0x7fffffffecc8, rbp = 0x7fffffffecf0 --- Tracing command dhclient pid 1925 tid 100172 td 0xfffff80023c806e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2e827d0 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2e82810 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2e82850 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00e2e82880 _sx_xlock_hard() at _sx_xlock_hard+0x448/frame 0xfffffe00e2e82940 _sx_xlock() at _sx_xlock+0xbf/frame 0xfffffe00e2e82980 ktr_submitrequest() at ktr_submitrequest+0x34/frame 0xfffffe00e2e829a0 kern_recvit() at kern_recvit+0x247/frame 0xfffffe00e2e82a50 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe00e2e82ac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00e2e82bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00e2e82bf0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x8003f211a, rsp = 0x7fffffffe908, rbp = 0x7fffffffe9d0 --- Tracing command vnlru pid 34 tid 100141 td 0xfffff800234d76e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00da9f39b0 mi_switch() at mi_switch+0x182/frame 0xfffffe00da9f39f0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00da9f3a30 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00da9f3a70 _sleep() at _sleep+0x26c/frame 0xfffffe00da9f3b10 vnlru_proc() at vnlru_proc+0x149/frame 0xfffffe00da9f3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00da9f3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00da9f3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command syncer pid 33 tid 100140 td 0xfffff800234de000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00da9ee5a0 mi_switch() at mi_switch+0x182/frame 0xfffffe00da9ee5e0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00da9ee620 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00da9ee650 sleeplk() at sleeplk+0x107/frame 0xfffffe00da9ee6b0 lockmgr_xlock_hard() at lockmgr_xlock_hard+0x225/frame 0xfffffe00da9ee770 __lockmgr_args() at __lockmgr_args+0x60d/frame 0xfffffe00da9ee830 getblkx() at getblkx+0x15f/frame 0xfffffe00da9ee910 breadn_flags() at breadn_flags+0x44/frame 0xfffffe00da9ee980 ffs_update() at ffs_update+0x138/frame 0xfffffe00da9eea10 ffs_sync() at ffs_sync+0x556/frame 0xfffffe00da9eeac0 sync_fsync() at sync_fsync+0x11e/frame 0xfffffe00da9eeb00 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xaa/frame 0xfffffe00da9eeb20 sched_sync() at sched_sync+0x3d2/frame 0xfffffe00da9eebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00da9eebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00da9eebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100131 td 0xfffff800234de6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d323fa00 mi_switch() at mi_switch+0x182/frame 0xfffffe00d323fa40 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d323fa80 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d323fac0 _sleep() at _sleep+0x26c/frame 0xfffffe00d323fb60 buf_daemon() at buf_daemon+0x112/frame 0xfffffe00d323fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d323fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d323fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100133 td 0xfffff800233336e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d33299f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3329a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d3329a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d3329ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00d3329b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00d3329bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3329bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3329bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100134 td 0xfffff80023333000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d33ee9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d33eea30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d33eea70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d33eeab0 _sleep() at _sleep+0x26c/frame 0xfffffe00d33eeb50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00d33eebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d33eebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d33eebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100135 td 0xfffff800232316e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d33f39f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d33f3a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d33f3a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d33f3ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00d33f3b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00d33f3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d33f3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d33f3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100136 td 0xfffff80023530000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d33f89f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d33f8a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d33f8a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d33f8ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00d33f8b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00d33f8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d33f8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d33f8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100137 td 0xfffff800235316e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d33fd9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d33fda30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d33fda70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00d33fdab0 _sleep() at _sleep+0x26c/frame 0xfffffe00d33fdb50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00d33fdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d33fdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d33fdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100138 td 0xfffff80023531000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00da8e49f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00da8e4a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00da8e4a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00da8e4ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00da8e4b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00da8e4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00da8e4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00da8e4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100139 td 0xfffff800234876e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00da9e99f0 mi_switch() at mi_switch+0x182/frame 0xfffffe00da9e9a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00da9e9a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00da9e9ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00da9e9b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00da9e9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00da9e9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00da9e9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100153 td 0xfffff800051186e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e1bfba20 mi_switch() at mi_switch+0x182/frame 0xfffffe00e1bfba60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e1bfbaa0 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00e1bfbae0 _sleep() at _sleep+0x26c/frame 0xfffffe00e1bfbb80 softdep_flush() at softdep_flush+0x175/frame 0xfffffe00e1bfbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e1bfbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e1bfbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 32 tid 100197 td 0xfffff80040fbb6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00e2f47a20 mi_switch() at mi_switch+0x182/frame 0xfffffe00e2f47a60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00e2f47aa0 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe00e2f47ae0 _sleep() at _sleep+0x26c/frame 0xfffffe00e2f47b80 softdep_flush() at softdep_flush+0x175/frame 0xfffffe00e2f47bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e2f47bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e2f47bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 31 tid 100129 td 0xfffff800234dd000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d3229a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3229ab0 sched_bind() at sched_bind+0xd8/frame 0xfffffe00d3229ae0 vm_page_pqbatch_drain() at vm_page_pqbatch_drain+0x80/frame 0xfffffe00d3229b40 vm_daemon() at vm_daemon+0xc1/frame 0xfffffe00d3229bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3229bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3229bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100128 td 0xfffff8002322b000 (CPU 7) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1bbdd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1bbdf0 trap() at trap+0x61/frame 0xfffffe003e1bbf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1bbf20 --- trap 0x13, rip = 0xffffffff80ba40bf, rsp = 0xfffffe00d3224130, rbp = 0xfffffe00d3224150 --- __mtx_unlock_flags() at __mtx_unlock_flags+0x9f/frame 0xfffffe00d3224150 zone_alloc_item_locked() at zone_alloc_item_locked+0x75/frame 0xfffffe00d32241b0 uma_zalloc_arg() at uma_zalloc_arg+0x10c/frame 0xfffffe00d32241f0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00d3224280 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d32242c0 cache_alloc() at cache_alloc+0x422/frame 0xfffffe00d3224350 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d3224390 vm_page_alloc_domain_after() at vm_page_alloc_domain_after+0x176/frame 0xfffffe00d3224410 uma_small_alloc() at uma_small_alloc+0x6e/frame 0xfffffe00d3224440 keg_alloc_slab() at keg_alloc_slab+0x119/frame 0xfffffe00d32244a0 zone_import() at zone_import+0x177/frame 0xfffffe00d3224560 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe00d32245f0 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe00d3224630 cache_free() at cache_free+0x27b/frame 0xfffffe00d3224680 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe00d32246d0 free() at free+0x91/frame 0xfffffe00d3224700 ufsdirhash_destroy() at ufsdirhash_destroy+0x10a/frame 0xfffffe00d3224740 ufsdirhash_lowmem() at ufsdirhash_lowmem+0xa4/frame 0xfffffe00d3224770 vm_pageout_worker() at vm_pageout_worker+0x3c0/frame 0xfffffe00d3224b70 vm_pageout() at vm_pageout+0x193/frame 0xfffffe00d3224bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3224bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3224bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100130 td 0xfffff800235076e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d323a420 mi_switch() at mi_switch+0x182/frame 0xfffffe00d323a460 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d323a4a0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00d323a4d0 _sleep() at _sleep+0x2cd/frame 0xfffffe00d323a570 swap_pager_putpages() at swap_pager_putpages+0x1f6/frame 0xfffffe00d323a610 vm_pageout_flush() at vm_pageout_flush+0xf5/frame 0xfffffe00d323a6c0 vm_pageout_cluster() at vm_pageout_cluster+0x491/frame 0xfffffe00d323a930 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0x95a/frame 0xfffffe00d323abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d323abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d323abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 30 tid 100132 td 0xfffff80023507000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d3244ab0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3244af0 sched_bind() at sched_bind+0xd8/frame 0xfffffe00d3244b20 pcpu_cache_drain_safe() at pcpu_cache_drain_safe+0x21d/frame 0xfffffe00d3244b50 uma_reclaim() at uma_reclaim+0x120/frame 0xfffffe00d3244b80 uma_reclaim_worker() at uma_reclaim_worker+0x30/frame 0xfffffe00d3244bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3244bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3244bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 29 tid 100127 td 0xfffff8002335c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d32139e0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3213a20 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d3213a60 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00d3213a90 _sleep() at _sleep+0x2cd/frame 0xfffffe00d3213b30 enc_daemon() at enc_daemon+0xf6/frame 0xfffffe00d3213bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3213bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3213bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 28 tid 100126 td 0xfffff8002335c6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d320e9e0 mi_switch() at mi_switch+0x182/frame 0xfffffe00d320ea20 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d320ea60 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00d320ea90 _sleep() at _sleep+0x2cd/frame 0xfffffe00d320eb30 enc_daemon() at enc_daemon+0xf6/frame 0xfffffe00d320ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d320ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d320ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 27 tid 100120 td 0xfffff80003d2e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5f09f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5f0a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5f0a70 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe003e5f0ab0 _sleep() at _sleep+0x26c/frame 0xfffffe003e5f0b50 random_kthread() at random_kthread+0x67/frame 0xfffffe003e5f0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5f0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5f0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sctp_iterator pid 26 tid 100119 td 0xfffff80003d2e6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5dfa40 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5dfa80 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5dfac0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5dfaf0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e5dfb90 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe003e5dfbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5dfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5dfbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 25 tid 100118 td 0xfffff80023148000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5cea10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5cea50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5cea90 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe003e5cead0 _sleep() at _sleep+0x26c/frame 0xfffffe003e5ceb70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe003e5cebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5cebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5cebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 24 tid 100117 td 0xfffff800231486e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5c9a10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5c9a50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5c9a90 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe003e5c9ad0 _sleep() at _sleep+0x26c/frame 0xfffffe003e5c9b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe003e5c9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5c9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5c9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 23 tid 100116 td 0xfffff80023158000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5c4a10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5c4a50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5c4a90 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe003e5c4ad0 _sleep() at _sleep+0x26c/frame 0xfffffe003e5c4b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe003e5c4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5c4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5c4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 22 tid 100115 td 0xfffff800231586e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5bfa10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5bfa50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5bfa90 sleepq_timedwait() at sleepq_timedwait+0x4f/frame 0xfffffe003e5bfad0 _sleep() at _sleep+0x26c/frame 0xfffffe003e5bfb70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe003e5bfbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5bfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5bfbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100096 td 0xfffff800040b9000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e554a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e554aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e554ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e554b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e554b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e554bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e554bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e554bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100097 td 0xfffff800040b16e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e559a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e559aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e559ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e559b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e559b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e559bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e559bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e559bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100098 td 0xfffff800040b1000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e55ea60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e55eaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e55eae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e55eb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e55eb70 usb_process() at usb_process+0xc3/frame 0xfffffe003e55ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e55ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e55ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100099 td 0xfffff800050146e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e563a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e563aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e563ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e563b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e563b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e563bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e563bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e563bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100100 td 0xfffff80005014000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e568a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e568aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e568ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e568b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e568b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e568bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e568bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e568bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100102 td 0xfffff80005015000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e572a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e572aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e572ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e572b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e572b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e572bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e572bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e572bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100103 td 0xfffff800040b86e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e577a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e577aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e577ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e577b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e577b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e577bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e577bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e577bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100104 td 0xfffff80005034000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e57ca60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e57caa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e57cae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e57cb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e57cb70 usb_process() at usb_process+0xc3/frame 0xfffffe003e57cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e57cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e57cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100105 td 0xfffff800050356e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e581a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e581aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e581ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e581b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e581b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e581bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e581bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e581bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100106 td 0xfffff80005035000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e586a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e586aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e586ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e586b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e586b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e586bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e586bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e586bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100107 td 0xfffff8000510e6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e597a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e597aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e597ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e597b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e597b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e597bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e597bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e597bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100108 td 0xfffff8000510e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e59ca60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e59caa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e59cae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e59cb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e59cb70 usb_process() at usb_process+0xc3/frame 0xfffffe003e59cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e59cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e59cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100109 td 0xfffff800051066e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5a1a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5a1aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5a1ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5a1b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e5a1b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e5a1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5a1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5a1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100110 td 0xfffff80005106000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5a6a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5a6aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5a6ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5a6b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e5a6b70 usb_process() at usb_process+0xc3/frame 0xfffffe003e5a6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5a6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5a6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100111 td 0xfffff800050346e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5aba60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5abaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5abae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5abb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e5abb70 usb_process() at usb_process+0xc3/frame 0xfffffe003e5abbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5abbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5abbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100091 td 0xfffff80003bdd000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e53ba10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e53ba50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e53ba90 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e53bac0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e53bb60 xpt_done_td() at xpt_done_td+0x8b/frame 0xfffffe003e53bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e53bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e53bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100092 td 0xfffff80003d2d6e0 (CPU 3) kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100093 td 0xfffff80003d2d000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e545a10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e545a50 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e545a90 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e545ac0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e545b60 xpt_done_td() at xpt_done_td+0x8b/frame 0xfffffe003e545bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e545bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e545bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100125 td 0xfffff8002322b6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d3209a00 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3209a40 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d3209a80 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00d3209ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00d3209b50 xpt_scanner_thread() at xpt_scanner_thread+0x92/frame 0xfffffe00d3209bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3209bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3209bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 11 pid 19 tid 100090 td 0xfffff80003bdd6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5369f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e536a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e536a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e536aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e536b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e536bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e536bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e536bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 10 pid 18 tid 100089 td 0xfffff80003bdc000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5319f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e531a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e531a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e531aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e531b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e531bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e531bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e531bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 9 pid 17 tid 100088 td 0xfffff80003bdc6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e52c9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e52ca30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e52ca70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e52caa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e52cb40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e52cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e52cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e52cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 8 pid 16 tid 100087 td 0xfffff80003bdb000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5279f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e527a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e527a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e527aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e527b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e527bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e527bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e527bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 7 pid 15 tid 100086 td 0xfffff80003bca6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5229f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e522a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e522a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e522aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e522b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e522bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e522bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e522bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 6 pid 9 tid 100085 td 0xfffff80003bd2000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e51d9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e51da30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e51da70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e51daa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e51db40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e51dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e51dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e51dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 5 pid 8 tid 100084 td 0xfffff80003bd26e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5189f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e518a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e518a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e518aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e518b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e518bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e518bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e518bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 4 pid 7 tid 100083 td 0xfffff80003bd1000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5139f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e513a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e513a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e513aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e513b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e513bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e513bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e513bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 3 pid 6 tid 100082 td 0xfffff80003bd16e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e50e9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e50ea30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e50ea70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e50eaa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e50eb40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e50ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e50ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e50ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 2 pid 5 tid 100081 td 0xfffff80003bc4000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5099f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e509a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e509a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e509aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e509b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e509bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e509bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e509bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 1 pid 4 tid 100080 td 0xfffff80003bc46e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5049f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e504a30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e504a70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e504aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e504b40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e504bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e504bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e504bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 0 pid 3 tid 100079 td 0xfffff80003bc3000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4ff9f0 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4ffa30 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4ffa70 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4ffaa0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4ffb40 crypto_ret_proc() at crypto_ret_proc+0x23d/frame 0xfffffe003e4ffbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4ffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4ffbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100078 td 0xfffff80003bc36e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4faa20 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4faa60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4faaa0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4faad0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4fab70 crypto_proc() at crypto_proc+0x2bf/frame 0xfffffe003e4fabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4fabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4fabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sequencer 00 pid 14 tid 100065 td 0xfffff80003b36000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4b9a20 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4b9a60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4b9aa0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4b9ad0 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe003e4b9b30 seq_eventthread() at seq_eventthread+0x8c/frame 0xfffffe003e4b9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4b9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4b9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100062 td 0xfffff80003b356e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e49ea20 mi_switch() at mi_switch+0x182/frame 0xfffffe003e49ea60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e49eaa0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e49ead0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e49eb70 g_run_events() at g_run_events+0x47d/frame 0xfffffe003e49ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e49ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e49ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100063 td 0xfffff80003b35000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4a3a20 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4a3a60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4a3aa0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4a3ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4a3b70 g_io_schedule_up() at g_io_schedule_up+0x4c/frame 0xfffffe003e4a3ba0 g_up_procbody() at g_up_procbody+0x5c/frame 0xfffffe003e4a3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4a3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4a3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100064 td 0xfffff80003b366e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4a8a20 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4a8a60 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4a8aa0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4a8ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4a8b70 g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xfffffe003e4a8ba0 g_down_procbody() at g_down_procbody+0x5c/frame 0xfffffe003e4a8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4a8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4a8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100017 td 0xfffff800038f0000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addb8b10 mi_switch() at mi_switch+0x182/frame 0xfffffe00addb8b50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe00addb8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addb8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addb8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xfffff800038ef6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addbdb10 mi_switch() at mi_switch+0x182/frame 0xfffffe00addbdb50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe00addbdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addbdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addbdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100020 td 0xfffff800038bc6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100047 td 0xfffff80003a71000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100048 td 0xfffff80003a726e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e404b10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e404b50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe003e404bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e404bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e404bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100049 td 0xfffff80003a72000 (CPU 1) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e173dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e173df0 trap() at trap+0x61/frame 0xfffffe003e173f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e173f20 --- trap 0x13, rip = 0xffffffff80c119f2, rsp = 0xfffffe003e409940, rbp = 0xfffffe003e409950 --- lock_delay() at lock_delay+0x32/frame 0xfffffe003e409950 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xfffffe003e409a00 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e409a50 softclock_call_cc() at softclock_call_cc+0x113/frame 0xfffffe003e409b20 softclock() at softclock+0x7c/frame 0xfffffe003e409b50 ithread_loop() at ithread_loop+0x1c6/frame 0xfffffe003e409bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e409bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e409bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100050 td 0xfffff80003a6a6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xfffff80003a6a000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xfffff80003a7d6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100053 td 0xfffff80003a7d000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xfffff80003a7e6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100055 td 0xfffff80003a7e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100056 td 0xfffff80003a716e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100057 td 0xfffff80003a82000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100058 td 0xfffff80003a7b6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xfffff80003a7b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100060 td 0xfffff80003a7c6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100094 td 0xfffff800040b8000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e54ab10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e54ab50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe003e54abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e54abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e54abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100095 td 0xfffff800040b96e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e54fb10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e54fb50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe003e54fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e54fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e54fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100101 td 0xfffff800050156e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e56db10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e56db50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe003e56dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e56dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e56dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100112 td 0xfffff80005118000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100113 td 0xfffff800051196e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5b5b10 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5b5b50 ithread_loop() at ithread_loop+0x39e/frame 0xfffffe003e5b5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5b5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5b5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100177 td 0xfffff80040688000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100178 td 0xfffff800406896e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100179 td 0xfffff80040689000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100180 td 0xfffff8004067e6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100181 td 0xfffff8004067e000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100182 td 0xfffff8004069b6e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100183 td 0xfffff8004069b000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100184 td 0xfffff800406906e0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xfffff800038aa000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add72a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add72ab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add72ad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add72bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add72bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add72bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff800038a96e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add77a90 mi_switch() at mi_switch+0x182/frame 0xfffffe00add77ad0 sched_idletd() at sched_idletd+0x459/frame 0xfffffe00add77bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add77bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add77bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffff800038a9000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add7ca70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add7cab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add7cad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add7cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add7cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add7cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffff800038b96e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add81a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add81ab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add81ad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add81bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add81bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add81bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffff800038b9000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add86a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add86ab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add86ad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add86bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add86bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add86bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffff800038b86e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add8ba70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add8bab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add8bad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add8bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add8bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add8bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffff800038b8000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add90a90 mi_switch() at mi_switch+0x182/frame 0xfffffe00add90ad0 sched_idletd() at sched_idletd+0x459/frame 0xfffffe00add90bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add90bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add90bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffff800038ab6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add95a90 mi_switch() at mi_switch+0x182/frame 0xfffffe00add95ad0 sched_idletd() at sched_idletd+0x459/frame 0xfffffe00add95bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add95bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add95bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffff800038bc000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add9aa70 mi_switch() at mi_switch+0x182/frame 0xfffffe00add9aab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00add9aad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00add9abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add9abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add9abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffff800038bb6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add9fa90 mi_switch() at mi_switch+0x182/frame 0xfffffe00add9fad0 sched_idletd() at sched_idletd+0x459/frame 0xfffffe00add9fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add9fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add9fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffff800038bb000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00adda4a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00adda4ab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00adda4ad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00adda4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adda4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adda4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffff800038ba6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00adda9a70 mi_switch() at mi_switch+0x182/frame 0xfffffe00adda9ab0 critical_exit_preempt() at critical_exit_preempt+0x63/frame 0xfffffe00adda9ad0 sched_idletd() at sched_idletd+0x401/frame 0xfffffe00adda9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adda9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adda9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff800038aa6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add6d690 mi_switch() at mi_switch+0x182/frame 0xfffffe00add6d6d0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00add6d710 sleepq_catch_signals() at sleepq_catch_signals+0x415/frame 0xfffffe00add6d770 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00add6d7a0 _sleep() at _sleep+0x2c3/frame 0xfffffe00add6d840 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00add6d8d0 sys_wait4() at sys_wait4+0x78/frame 0xfffffe00add6dac0 amd64_syscall() at amd64_syscall+0x2f1/frame 0xfffffe00add6dbf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00add6dbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2b0c2a, rsp = 0x7fffffffe808, rbp = 0x7fffffffe870 --- Tracing command audit pid 10 tid 100001 td 0xfffff800038ab000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00add68a40 mi_switch() at mi_switch+0x182/frame 0xfffffe00add68a80 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00add68ac0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00add68af0 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00add68b50 audit_worker() at audit_worker+0x9f/frame 0xfffffe00add68bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00add68bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00add68bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81d9c9c0 (CPU 9) cpustop_handler() at cpustop_handler+0x2a/frame 0xfffffe003e1d3dd0 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xfffffe003e1d3df0 trap() at trap+0x61/frame 0xfffffe003e1d3f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003e1d3f20 --- trap 0x13, rip = 0xffffffff80c119f0, rsp = 0xffffffff822fbc40, rbp = 0xffffffff822fbc50 --- lock_delay() at lock_delay+0x30/frame 0xffffffff822fbc50 __mtx_lock_sleep() at __mtx_lock_sleep+0x108/frame 0xffffffff822fbd00 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xffffffff822fbd50 swapper() at swapper+0x15f/frame 0xffffffff822fbda0 mi_startup() at mi_startup+0x2ed/frame 0xffffffff822fbdf0 btext() at btext+0x2c Tracing command kernel pid 0 tid 100015 td 0xfffff800038ba000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addaea30 mi_switch() at mi_switch+0x182/frame 0xfffffe00addaea70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00addaeab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00addaeae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00addaeb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00addaebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addaebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addaebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffff800038f06e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addb3a30 mi_switch() at mi_switch+0x182/frame 0xfffffe00addb3a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00addb3ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00addb3ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00addb3b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00addb3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addb3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addb3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffff800038ef000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addc2a30 mi_switch() at mi_switch+0x182/frame 0xfffffe00addc2a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00addc2ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00addc2ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00addc2b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00addc2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addc2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addc2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffff800038f6000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00addcca60 mi_switch() at mi_switch+0x182/frame 0xfffffe00addccaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00addccae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00addccb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00addccb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00addccbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00addccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00addccbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffff800038f56e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe0037781a60 mi_switch() at mi_switch+0x182/frame 0xfffffe0037781aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0037781ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0037781b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe0037781b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe0037781bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037781bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037781bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffff800038f5000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0037786a60 mi_switch() at mi_switch+0x182/frame 0xfffffe0037786aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0037786ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0037786b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe0037786b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe0037786bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037786bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037786bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffff800038f46e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003778ba60 mi_switch() at mi_switch+0x182/frame 0xfffffe003778baa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003778bae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003778bb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003778bb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe003778bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003778bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003778bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffff800038f4000 sched_switch() at sched_switch+0x38f/frame 0xfffffe0037790a60 mi_switch() at mi_switch+0x182/frame 0xfffffe0037790aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0037790ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0037790b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe0037790b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe0037790bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037790bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037790bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffff800039286e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe0037795a60 mi_switch() at mi_switch+0x182/frame 0xfffffe0037795aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe0037795ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe0037795b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe0037795b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe0037795bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe0037795bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0037795bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffff80003928000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003779aa60 mi_switch() at mi_switch+0x182/frame 0xfffffe003779aaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003779aae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003779ab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003779ab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe003779abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003779abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003779abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffff800039276e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003779fa60 mi_switch() at mi_switch+0x182/frame 0xfffffe003779faa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003779fae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003779fb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003779fb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe003779fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003779fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003779fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffff80003927000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377a4a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377a4aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377a4ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377a4b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377a4b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377a4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377a4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377a4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffff800038f66e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377a9a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377a9aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377a9ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377a9b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377a9b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377a9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377a9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377a9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffff8000392d000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377aea60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377aeaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377aeae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377aeb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377aeb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377aebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377aebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377aebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffff8000392c6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377b3a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377b3aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377b3ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377b3b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377b3b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377b3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffff8000392c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377b8a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377b8aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377b8ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377b8b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377b8b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377b8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377b8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffff8000392b6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377bda60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377bdaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377bdae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377bdb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377bdb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377bdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377bdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377bdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffff8000392b000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377c2a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377c2aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377c2ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377c2b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377c2b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377c2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377c2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffff80003a496e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377c7a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377c7aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377c7ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377c7b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377c7b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377c7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377c7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffff80003a49000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377cca60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377ccaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377ccae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377ccb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377ccb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377ccbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377ccbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffff80003a4a6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377d1a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377d1aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377d1ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377d1b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377d1b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377d1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377d1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377d1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffff80003a4a000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377d6a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377d6aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377d6ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377d6b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377d6b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377d6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377d6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377d6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffff8000392d6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377dba60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377dbaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377dbae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377dbb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377dbb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100041 td 0xfffff80003a4c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377e0a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377e0aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377e0ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377e0b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377e0b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377e0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377e0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377e0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xfffff80003a4d6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377e5a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377e5aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377e5ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377e5b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377e5b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377e5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377e5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377e5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffff80003a4d000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377eaa60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377eaaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377eaae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377eab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377eab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377eabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377eabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377eabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffff80003a4e6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377efa60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377efaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377efae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377efb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377efb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377efbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377efbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377efbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xfffff80003a4e000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377f4a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377f4aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377f4ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377f4b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377f4b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377f4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xfffff80003a526e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe00377f9a60 mi_switch() at mi_switch+0x182/frame 0xfffffe00377f9aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00377f9ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00377f9b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00377f9b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xf1/frame 0xfffffe00377f9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377f9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377f9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100061 td 0xfffff80003a7c000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e499a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e499a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e499ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e499ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e499b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e499bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e499bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e499bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xfffff80003a826e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4bea30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4bea70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4beab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4beae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4beb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4bebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4bebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4bebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffff80003bc1000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4c3a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4c3a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4c3ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4c3ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4c3b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4c3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4c3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4c3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffff80003bc26e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4c8a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4c8a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4c8ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4c8ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4c8b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4c8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4c8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4c8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffff80003bc2000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4cda30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4cda70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4cdab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4cdae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4cdb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4cdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4cdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4cdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffff80003bbb6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4d2a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4d2a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4d2ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4d2ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4d2b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4d2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4d2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4d2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffff80003bbb000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4d7a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4d7a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4d7ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4d7ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4d7b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4d7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4d7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4d7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffff80003bbf6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4dca30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4dca70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4dcab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4dcae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4dcb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4dcbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4dcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4dcbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffff80003bbf000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4e1a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4e1a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4e1ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4e1ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4e1b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4e1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4e1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4e1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffff80003bc06e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4e6a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4e6a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4e6ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4e6ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4e6b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4e6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4e6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4e6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffff80003bc0000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4eba30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4eba70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4ebab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4ebae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4ebb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4ebbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4ebbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4ebbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100076 td 0xfffff80003bc16e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4f0a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4f0a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4f0ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4f0ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4f0b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4f0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4f0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4f0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xfffff80003bca000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e4f5a30 mi_switch() at mi_switch+0x182/frame 0xfffffe003e4f5a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e4f5ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e4f5ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe003e4f5b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe003e4f5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e4f5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e4f5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100114 td 0xfffff80005119000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5baa60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5baaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5baae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5bab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003e5bab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe003e5babb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5babf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5babf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100121 td 0xfffff80003bdb6e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5f5a60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5f5aa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5f5ae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5f5b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003e5f5b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe003e5f5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5f5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5f5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100122 td 0xfffff80023231000 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5faa60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5faaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5faae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5fab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003e5fab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe003e5fabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5fabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5fabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100123 td 0xfffff800232326e0 sched_switch() at sched_switch+0x38f/frame 0xfffffe003e5ffa60 mi_switch() at mi_switch+0x182/frame 0xfffffe003e5ffaa0 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe003e5ffae0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe003e5ffb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe003e5ffb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xf1/frame 0xfffffe003e5ffbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e5ffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e5ffbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100124 td 0xfffff80023232000 sched_switch() at sched_switch+0x38f/frame 0xfffffe00d3204a30 mi_switch() at mi_switch+0x182/frame 0xfffffe00d3204a70 sleepq_switch() at sleepq_switch+0x10e/frame 0xfffffe00d3204ab0 sleepq_wait() at sleepq_wait+0x43/frame 0xfffffe00d3204ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00d3204b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xb1/frame 0xfffffe00d3204bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00d3204bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d3204bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:pho> dump Dumping 3727 out of 32626 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete db:0:pho> show allchains chain 1: thread 105207 (pid 1697, sed) sleeping on 0xfffff800238982f8 "piperd" chain 2: thread 102100 (pid 1695, sh) sleeping on 0xfffff80077bef530 "wait" chain 3: thread 102552 (pid 1674, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 4: thread 102569 (pid 1673, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 5: thread 101262 (pid 1672, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 6: thread 105234 (pid 1671, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 7: thread 105197 (pid 1670, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 8: thread 105200 (pid 1669, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 9: thread 101246 (pid 1668, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 10: thread 101247 (pid 1667, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 11: thread 104260 (pid 1393, sh) sleeping on 0xfffff80023898000 "piperd" chain 12: thread 101263 (pid 1361, swap) sleeping on 0xfffff804b2ae4530 "wait" chain 13: thread 103392 (pid 1356, sh) sleeping on 0xfffff805873d2b08 "pause" chain 14: thread 102101 (pid 1355, tee) sleeping on 0xfffff800413178e8 "piperd" chain 15: thread 105199 (pid 1354, sh) sleeping on 0xfffff806b7107530 "wait" chain 16: thread 100260 (pid 10144, sh) sleeping on 0xfffff80023d18be0 "piperd" chain 17: thread 100194 (pid 10142, awk) sleeping on 0xfffff8003c5962f8 "piperd" chain 18: thread 100267 (pid 10141, sh) sleeping on 0xfffff80023b71530 "wait" chain 19: thread 100278 (pid 10083, sh) sleeping on 0xfffff8016bcf6000 "wait" chain 20: thread 100168 (pid 2359, sh) sleeping on 0xfffff80023b50530 "wait" chain 21: thread 100214 (pid 2358, bash) sleeping on 0xfffff80023483ec0 "select" chain 22: thread 100144 (pid 2356, csh) sleeping on 0xfffff800234ad0a8 "pause" chain 23: thread 100213 (pid 2355, su) sleeping on 0xfffff801504d0000 "wait" chain 24: thread 100212 (pid 2349, bash) sleeping on 0xfffff801504d0530 "wait" chain 25: thread 100145 (pid 2348, sshd) sleeping on 0xfffff80023da8f40 "select" chain 26: thread 100176 (pid 2345, sshd) sleeping on 0xfffff8002312f3c0 "select" chain 27: thread 100154 (pid 2341, csh) sleeping on 0xfffff8000514ecb0 "ttyin" chain 28: thread 100210 (pid 2340, login) sleeping on 0xfffff801504d0a60 "wait" chain 29: thread 100209 (pid 2339, getty) sleeping on 0xfffff8000514b8b0 "ttyin" chain 30: thread 100208 (pid 2338, getty) sleeping on 0xfffff800232590b0 "ttyin" chain 31: thread 100207 (pid 2337, getty) sleeping on 0xfffff800232330b0 "ttyin" chain 32: thread 100206 (pid 2336, getty) sleeping on 0xfffff800232338b0 "ttyin" chain 33: thread 100205 (pid 2335, getty) sleeping on 0xfffff8002323e0b0 "ttyin" chain 34: thread 100204 (pid 2334, getty) sleeping on 0xfffff8002323e8b0 "ttyin" chain 35: thread 100203 (pid 2333, getty) sleeping on 0xfffff8002323d0b0 "ttyin" chain 36: thread 100201 (pid 2332, getty) sleeping on 0xfffff8002323d8b0 "ttyin" chain 37: thread 100142 (pid 2331, getty) sleeping on 0xfffff8002323c0b0 "ttyin" chain 38: thread 100186 (pid 2281, sshd) sleeping on 0xfffff80003a2fac0 "select" chain 39: thread 100243 (pid 2250, ntpd) sleeping on 0xfffff801360c8e00 "usem" chain 40: thread 100193 (pid 2186, rpc.statd) sleeping on 0xfffff80005049840 "select" chain 41: thread 100166 (pid 2061, rpcbind) sleeping on 0xfffff8002312fcc0 "select" chain 42: thread 100174 (pid 2051, syslogd) sleeping on 0xfffff8002312f540 "select" chain 43: thread 100148 (pid 1980, devd) sleeping on 0xfffff8003c607ec0 "select" chain 44: thread 100157 (pid 1975, dhclient) blocked on lockmgr bufwaitEXCL KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fbe0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fbf0 trap() at trap+0x73/frame 0xfffffe003e53fd20 calltrap() at calltrap+0x8/frame 0xfffffe003e53fd20 --- trap 0xc, rip = 0xffffffff80c29e10, rsp = 0xfffffe003e53fdf0, rbp = 0xfffffe003e53fe10 --- print_lockchain() at print_lockchain+0x30/frame 0xfffffe003e53fe10 db_show_allchains() at db_show_allchains+0x97/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Script command 'show allchains' returned error db:0:pho> chain 1: thread 105207 (pid 1697, sed) sleeping on 0xfffff800238982f8 "piperd" chain 2: thread 102100 (pid 1695, sh) sleeping on 0xfffff80077bef530 "wait" chain 3: thread 102552 (pid 1674, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 4: thread 102569 (pid 1673, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 5: thread 101262 (pid 1672, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 6: thread 105234 (pid 1671, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 7: thread 105197 (pid 1670, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 8: thread 105200 (pid 1669, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 9: thread 101246 (pid 1668, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 10: thread 101247 (pid 1667, dd) sleeping on 0xffffffff81847f40 "vmwait" chain 11: thread 104260 (pid 1393, sh) sleeping on 0xfffff80023898000 "piperd" chain 12: thread 101263 (pid 1361, swap) sleeping on 0xfffff804b2ae4530 "wait" chain 13: thread 103392 (pid 1356, sh) sleeping on 0xfffff805873d2b08 "pause" chain 14: thread 102101 (pid 1355, tee) sleeping on 0xfffff800413178e8 "piperd" chain 15: thread 105199 (pid 1354, sh) sleeping on 0xfffff806b7107530 "wait" chain 16: thread 100260 (pid 10144, sh) sleeping on 0xfffff80023d18be0 "piperd" chain 17: thread 100194 (pid 10142, awk) sleeping on 0xfffff8003c5962f8 "piperd" chain 18: thread 100267 (pid 10141, sh) sleeping on 0xfffff80023b71530 "wait" chain 19: thread 100278 (pid 10083, sh) sleeping on 0xfffff8016bcf6000 "wait" chain 20: thread 100168 (pid 2359, sh) sleeping on 0xfffff80023b50530 "wait" chain 21: thread 100214 (pid 2358, bash) sleeping on 0xfffff80023483ec0 "select" chain 22: thread 100144 (pid 2356, csh) sleeping on 0xfffff800234ad0a8 "pause" chain 23: thread 100213 (pid 2355, su) sleeping on 0xfffff801504d0000 "wait" chain 24: thread 100212 (pid 2349, bash) sleeping on 0xfffff801504d0530 "wait" chain 25: thread 100145 (pid 2348, sshd) sleeping on 0xfffff80023da8f40 "select" chain 26: thread 100176 (pid 2345, sshd) sleeping on 0xfffff8002312f3c0 "select" chain 27: thread 100154 (pid 2341, csh) sleeping on 0xfffff8000514ecb0 "ttyin" chain 28: thread 100210 (pid 2340, login) sleeping on 0xfffff801504d0a60 "wait" chain 29: thread 100209 (pid 2339, getty) sleeping on 0xfffff8000514b8b0 "ttyin" chain 30: thread 100208 (pid 2338, getty) sleeping on 0xfffff800232590b0 "ttyin" chain 31: thread 100207 (pid 2337, getty) sleeping on 0xfffff800232330b0 "ttyin" chain 32: thread 100206 (pid 2336, getty) sleeping on 0xfffff800232338b0 "ttyin" chain 33: thread 100205 (pid 2335, getty) sleeping on 0xfffff8002323e0b0 "ttyin" chain 34: thread 100204 (pid 2334, getty) sleeping on 0xfffff8002323e8b0 "ttyin" chain 35: thread 100203 (pid 2333, getty) sleeping on 0xfffff8002323d0b0 "ttyin" chain 36: thread 100201 (pid 2332, getty) sleeping on 0xfffff8002323d8b0 "ttyin" chain 37: thread 100142 (pid 2331, getty) sleeping on 0xfffff8002323c0b0 "ttyin" chain 38: thread 100186 (pid 2281, sshd) sleeping on 0xfffff80003a2fac0 "select" chain 39: thread 100243 (pid 2250, ntpd) sleeping on 0xfffff801360c8e00 "usem" chain 40: thread 100193 (pid 2186, rpc.statd) sleeping on 0xfffff80005049840 "select" chain 41: thread 100166 (pid 2061, rpcbind) sleeping on 0xfffff8002312fcc0 "select" chain 42: thread 100174 (pid 2051, syslogd) sleeping on 0xfffff8002312f540 "select" chain 43: thread 100148 (pid 1980, devd) sleeping on 0xfffff8003c607ec0 "select" chain 44: thread 100157 (pid 1975, dhclient) blocked on lockmgr bufwaitEXCL KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe003e53fbe0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe003e53fbf0 trap() at trap+0x73/frame 0xfffffe003e53fd20 calltrap() at calltrap+0x8/frame 0xfffffe003e53fd20 --- trap 0xc, rip = 0xffffffff80c29e10, rsp = 0xfffffe003e53fdf0, rbp = 0xfffffe003e53fe10 --- print_lockchain() at print_lockchain+0x30/frame 0xfffffe003e53fe10 db_show_allchains() at db_show_allchains+0x97/frame 0xfffffe003e53fe40 db_command() at db_command+0x25c/frame 0xfffffe003e53ff10 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe003e53ffb0 db_command() at db_command+0x25c/frame 0xfffffe003e540080 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe003e540090 db_trap() at db_trap+0xe8/frame 0xfffffe003e540120 kdb_trap() at kdb_trap+0x157/frame 0xfffffe003e5401a0 trap() at trap+0x7d4/frame 0xfffffe003e5402d0 calltrap() at calltrap+0x8/frame 0xfffffe003e5402d0 --- trap 0x3, rip = 0xffffffff80c103d7, rsp = 0xfffffe003e5403a0, rbp = 0xfffffe003e5403b0 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe003e5403b0 hardclock() at hardclock+0x1d0/frame 0xfffffe003e540420 handleevents() at handleevents+0xb4/frame 0xfffffe003e540470 timercb() at timercb+0x196/frame 0xfffffe003e5404d0 lapic_handle_timer() at lapic_handle_timer+0xa0/frame 0xfffffe003e540510 Xtimerint() at Xtimerint+0xb1/frame 0xfffffe003e540510 --- interrupt, rip = 0xffffffff80ba3c6c, rsp = 0xfffffe003e5405e0, rbp = 0xfffffe003e540680 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x10c/frame 0xfffffe003e540680 __mtx_lock_flags() at __mtx_lock_flags+0xea/frame 0xfffffe003e5406d0 item_ctor() at item_ctor+0x15d/frame 0xfffffe003e540730 zone_alloc_item_locked() at zone_alloc_item_locked+0x275/frame 0xfffffe003e540790 keg_alloc_slab() at keg_alloc_slab+0xc3/frame 0xfffffe003e5407f0 zone_import() at zone_import+0x177/frame 0xfffffe003e5408b0 cache_alloc() at cache_alloc+0x47a/frame 0xfffffe003e540940 uma_zalloc_arg() at uma_zalloc_arg+0xc4/frame 0xfffffe003e540980 cache_free() at cache_free+0x27b/frame 0xfffffe003e5409d0 uma_zfree_arg() at uma_zfree_arg+0x1e1/frame 0xfffffe003e540a20 g_std_done() at g_std_done+0x2c/frame 0xfffffe003e540a40 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540a90 g_io_deliver() at g_io_deliver+0x265/frame 0xfffffe003e540ae0 g_disk_done() at g_disk_done+0x106/frame 0xfffffe003e540b20 xpt_done_process() at xpt_done_process+0x384/frame 0xfffffe003e540b60 xpt_done_td() at xpt_done_td+0xf5/frame 0xfffffe003e540bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe003e540bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003e540bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Script command '' returned error db> (kgdb) tid 100130 [Switching to thread 138 (Thread 100130)]#0 sched_switch (td=0xfffff800235076e0, newtd=0xfffff80783072000, flags=) at ../../../kern/sched_ule.c:2165 2165 cpuid = PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=0xfffff800235076e0, newtd=0xfffff80783072000, flags=) at ../../../kern/sched_ule.c:2165 #1 0xffffffff80bd0d82 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:524 #2 0xffffffff80c1fd3e in sleepq_switch (wchan=0xffffffff81d6cf44, pri=0x54) at ../../../kern/subr_sleepqueue.c:633 #3 0xffffffff80c1fbe3 in sleepq_wait (wchan=0xffffffff81d6cf44, pri=0x54) at ../../../kern/subr_sleepqueue.c:712 #4 0xffffffff80bd042d in _sleep (ident=0xffffffff81d6cf44, lock=, priority=, wmesg=0xffffffff811b54a6 "swbufa", sbt=0x0, pr=0x0, flags=0x100) at ../../../kern/kern_synch.c:220 #5 0xffffffff80eff486 in swap_pager_putpages (object=, ma=0xfffffe00d323a780, count=0xb, flags=, rtvals=0xfffffe00d323a620) at ../../../vm/swap_pager.c:1419 #6 0xffffffff80f2dbd5 in vm_pageout_flush (mc=0xfffffe00d323a780, count=0xb, flags=0x4, mreq=0x0, prunlen=0x0, eio=0x0) at vm_pager.h:135 #7 0xffffffff80f318c1 in vm_pageout_cluster (m=) at ../../../vm/vm_pageout.c:432 #8 0xffffffff80f3094a in vm_pageout_laundry_worker (arg=) at ../../../vm/vm_pageout.c:690 #9 0xffffffff80b83000 in fork_exit (callout=0xffffffff80f2fff0 , arg=0x0, frame=0xfffffe00d323ac00) at ../../../kern/kern_fork.c:1059 #10 0xffffffff810444ee in fork_trampoline () at ../../../amd64/amd64/exception.S:1080 #11 0x0000000000000000 in ?? () (kgdb) f 5 #5 0xffffffff80eff486 in swap_pager_putpages (object=, ma=0xfffffe00d323a780, count=0xb, flags=, rtvals=0xfffffe00d323a620) at ../../../vm/swap_pager.c:1419 1419 msleep(&nsw_wcount_async, &swbuf_mtx, PVM, (kgdb) l 1414 * request and assign the swap space. 1415 */ 1416 if (async) { 1417 mtx_lock(&swbuf_mtx); 1418 while (nsw_wcount_async == 0) 1419 msleep(&nsw_wcount_async, &swbuf_mtx, PVM, 1420 "swbufa", 0); 1421 nsw_wcount_async--; 1422 mtx_unlock(&swbuf_mtx); 1423 } (kgdb) info loc addr = i = 0x0 async = 0x1 blk = 0x1fdc5 n = 0xb j = bp = mreq = n_free = s_free = (kgdb) p/d nsw_wcount_async $1 = 0 (kgdb) Index: src/sys/ufs/ufs/ufs_extern.h =================================================================== --- src/sys/ufs/ufs/ufs_extern.h (revision 355490) +++ src/sys/ufs/ufs/ufs_extern.h (working copy) @@ -79,6 +79,7 @@ int ufs_inactive(struct vop_inactive_args *); int ufs_init(struct vfsconf *); void ufs_itimes(struct vnode *vp); int ufs_lookup(struct vop_cachedlookup_args *); +int ufs_need_inactive(struct vop_need_inactive_args *); int ufs_readdir(struct vop_readdir_args *); int ufs_reclaim(struct vop_reclaim_args *); void ffs_snapgone(struct inode *); Index: src/sys/ufs/ufs/ufs_inode.c =================================================================== --- src/sys/ufs/ufs/ufs_inode.c (revision 355490) +++ src/sys/ufs/ufs/ufs_inode.c (working copy) @@ -63,6 +63,40 @@ __FBSDID("$FreeBSD$"); #include #endif +int +ufs_need_inactive(ap) + struct vop_need_inactive_args *ap; +{ + struct vnode *vp; + struct inode *ip; +#ifdef QUOTA + int i; +#endif + + vp = ap->a_vp; + ip = VTOI(vp); + if (UFS_RDONLY(ip)) + return (0); + if (ip->i_mode == 0 || ip->i_nlink <= 0 || + (ip->i_effnlink == 0 && DOINGSOFTDEP(vp)) || + (ip->i_flag & (IN_ACCESS | IN_CHANGE | IN_MODIFIED | + IN_UPDATE)) != 0 || + (ip->i_effnlink <= 0 && (ip->i_size != 0 || (I_IS_UFS2(ip) && + ip->i_din2->di_extsize != 0)))) + return (1); +#ifdef QUOTA + for (i = 0; i < MAXQUOTAS; i++) { + if (ip->i_dquot[i] != NULL) + return (1); + } +#endif + /* + * No need to check ufs_gjournal_close() condition since we + * return 1 if only i_nlink <= 0. + */ + return (0); +} + /* * Last reference to an inode. If necessary, write or delete it. */ Index: src/sys/ufs/ufs/ufs_vnops.c =================================================================== --- src/sys/ufs/ufs/ufs_vnops.c (revision 355490) +++ src/sys/ufs/ufs/ufs_vnops.c (working copy) @@ -2742,6 +2742,7 @@ struct vop_vector ufs_vnodeops = { .vop_markatime = ufs_markatime, .vop_mkdir = ufs_mkdir, .vop_mknod = ufs_mknod, + .vop_need_inactive = ufs_need_inactive, .vop_open = ufs_open, .vop_pathconf = ufs_pathconf, .vop_poll = vop_stdpoll, -- Test scenario: misc/dd.sh