panic: mutex unp owned at ../../../kern/uipc_usrreq.c:804, vmcore.225 KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #1 r365247M: Wed Sep 2 17:12:18 CEST 2020 pho@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33165787136 (31629 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 hardware threads random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 ioapic1 irqs 24-47 Launching APs: 1 6 10 11 5 4 2 3 7 8 9 Timecounter "TSC-low" frequency 1750032972 Hz quality 1000 random: entropy device external interface WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0. kbd1 at kbdmux0 000.000052 [4346] netmap_init netmap: loaded module [ath_hal] loaded nexus0 vtvga0: cryptosoft0: acpi0: acpi0: Power Button (fixed) cpu0: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: on acpi0 pci0: on pcib0 pci0: at device 11.1 (no driver attached) pci0: at device 11.2 (no driver attached) pci0: at device 16.1 (no driver attached) pci0: at device 16.6 (no driver attached) pci0: at device 18.1 (no driver attached) acpi_syscontainer0: on acpi0 acpi_syscontainer1: on acpi0 acpi_syscontainer2: on acpi0 acpi_syscontainer3: on acpi0 apei0: on acpi0 pcib1: port 0xcf8-0xcff on acpi0 pci1: on pcib1 pcib2: irq 26 at device 1.0 on pci1 pci2: on pcib2 pcib3: irq 32 at device 2.0 on pci1 pci3: on pcib3 pci3: at device 0.0 (no driver attached) pci3: at device 0.1 (no driver attached) pcib4: irq 40 at device 3.0 on pci1 pci4: on pcib4 pci1: at device 17.0 (no driver attached) ahci0: port 0xf110-0xf117,0xf100-0xf103,0xf0f0-0xf0f7,0xf0e0-0xf0e3,0xf020-0xf03f mem 0xfb438000-0xfb4387ff irq 16 at device 17.4 on pci1 ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahciem0: on ahci0 xhci0: mem 0xfb400000-0xfb40ffff irq 19 at device 20.0 on pci1 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci1: at device 22.0 (no driver attached) pci1: at device 22.1 (no driver attached) ehci0: mem 0xfb434000-0xfb4343ff irq 18 at device 26.0 on pci1 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib5: irq 16 at device 28.0 on pci1 pci5: on pcib5 pcib6: irq 18 at device 28.2 on pci1 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 vgapci0: port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7 vgapci0: Boot video device pcib8: irq 16 at device 28.4 on pci1 pci8: on pcib8 igb0: port 0xd020-0xd03f mem 0xfb320000-0xfb33ffff,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: Ethernet address: 0c:c4:7a:a8:cd:da igb0: netmap queues/slots: TX 6/1024, RX 6/1024 igb1: port 0xd000-0xd01f mem 0xfb300000-0xfb31ffff,0xfb340000-0xfb343fff irq 17 at device 0.1 on pci8 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb1: Using 6 RX queues 6 TX queues igb1: Using MSI-X interrupts with 7 vectors igb1: Ethernet address: 0c:c4:7a:a8:cd:db igb1: netmap queues/slots: TX 6/1024, RX 6/1024 ehci1: mem 0xfb433000-0xfb4333ff irq 18 at device 29.0 on pci1 usbus2: EHCI version 1.0 usbus2 on ehci1 usbus2: 480Mbps High Speed USB v2.0 isab0: at device 31.0 on pci1 isa0: on isab0 ahci1: port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xfb432000-0xfb4327ff irq 16 at device 31.2 on pci1 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: at channel 0 on ahci1 ahcich5: at channel 1 on ahci1 ahcich6: at channel 2 on ahci1 ahcich7: at channel 3 on ahci1 ahcich8: at channel 4 on ahci1 ahcich9: at channel 5 on ahci1 ahciem1: on ahci1 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: console (115200,n,8,1) orm0: at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 est0: on cpu0 p4tcc0: on cpu0 NULL mp in getnewvnode(9), tag crossmp Timecounters tick every 1.000 msec Expensive timeout(9) function: 0xffffffff804df9d0(0xfffffe00da8ae000) 0.005001760 s ses0 at ahciem0 bus 0 scbus4 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus11 target 0 lun 0 ses1: SEMB S-E-S 2.00 device ses1: SEMB SES Device ses0: (none) in 'Slot 01', SATA Slot: scbus1 target 0 ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ACS-2 ATA SATA 3.x device ada0: Serial Number WD-WCC3F3KDDP98 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) ada0: quirks=0x1<4K> Trying to mount root from ufs:/dev/gpt/work [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. ugen1.1: at usbus1 ugen2.1: at usbus2 uhub0 on usbus1 uhub0: on usbus1 ugen0.1: <0x8086 XHCI root HUB> at usbus0 uhub1 on usbus2 uhub1: on usbus2 Expensive timeout(9) function: 0xffffffff80a32470(0xffffffff818e6888) 0.007394560 s uhub2 on usbus0 uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 WARNING: / was not properly dismounted Interface IP-Address Broadcast Setting hostuuid: 00000000-0000-0000-0000-002590c99976. Setting hostid: 0x90b83241. Starting file system checks: uhub0: 2 ports with 2 removable, self powered WARNING: / was not properly dismounted uhub1: 2 ports with 2 removable, self powered . ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 . lo0: link state changed to UP NKLOCAL> igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Starting devd. igb0: link state changed to UP Autoloading module: ioat.ko ioat0: mem 0xfb42c000-0xfb42ffff irq 31 at device 4.0 on pci1 ioat0: Capabilities: 2f7 ioat1: mem 0xfb428000-0xfb42bfff irq 39 at device 4.1 on pci1 ioat1: Capabilities: 2f7 ioat2: mem 0xfb424000-0xfb427fff irq 31 at device 4.2 on pci1 ioat2: Capabilities: f7 ioat3: mem 0xfb420000-0xfb423fff irq 39 at device 4.3 on pci1 ioat3: Capabilities: f7 ioat4: mem 0xfb41c000-0xfb41ffff irq 31 at device 4.4 on pci1 ioat4: Capabilities: f7 ioat5: mem 0xfb418000-0xfb41bfff irq 39 at device 4.5 on pci1 ioat5: Capabilities: f7 ioat6: mem 0xfb414000-0xfb417fff irq 31 at device 4.6 on pci1 ioat6: Capabilities: f7 ioat7: mem 0xfb410000-0xfb413fff irq 39 at device 4.7 on pci1 ioat7: Capabilities: f7 Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Autoloading module: ioat.ko Starting Network: igb1. igb1: flags=8822 metric 0 mtu 1500 options=4e507bb ether 0c:c4:7a:a8:cd:db media: Ethernet autoselect status: no carrier nd6 options=29 Autoloading module: uhid.ko Autoloading module: ums.ko ums0 on uhub3 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Starting dhclient. DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPACK from 192.168.5.1 if_delmulti_locked: detaching ifnet instance 0xfffff80039052000 bound to 192.168.5.171 -- renewal in 300 seconds. add host 127.0.0.1: gateway lo0 fib 0: route already in table add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Setting date via ntp. 2 Sep 17:20:35 ntpdate[2066]: step time server 192.168.5.1 offset -0.564381 sec Starting rpcbind. NFS access cache time=60 Updating CPU Microcode... CPU: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz (3500.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff Features2=0x7ffefbff AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0x37ab Structured Extended Features3=0x9c000400 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. Starting watchdogd. No core dumps found. Clearing /tmp (X related). Starting statd. Starting lockd. Recovering vi editor sessions:. fscklock order reversal: 1st 0xfffff8004598d2f8 ufs (ufs, lockmgr) @ kern/vfs_mount.c:1711 2nd 0xfffff80045650808 devfs (devfs, lockmgr) @ kern/vfs_subr.c:3118 lock order devfs -> ufs established at: #0 0xffffffff80c44b8c at witness_checkorder+0x46c #1 0xffffffff80ba7ea2 at lockmgr_lock_flags+0x172 #2 0xffffffff80e983bd at ffs_lock+0x8d #3 0xffffffff810d66c0 at VOP_LOCK1_APV+0x40 #4 0xffffffff80cd2c54 at _vn_lock+0x54 #5 0xffffffff80cb2090 at vfs_domount+0xe80 #6 0xffffffff80cb0451 at vfs_donmount+0x871 #7 0xffffffff80cb4b67 at kernel_mount+0x57 #8 0xffffffff80cb7261 at parse_mount+0x4a1 #9 0xffffffff80cb5789 at vfs_mountroot+0x589 #10 0xffffffff80b69df8 at start_init+0x28 #11 0xffffffff80b932d0 at fork_exit+0x80 #12 0xffffffff80ff8cce at fork_trampoline+0xe lock order ufs -> devfs attempted at: #0 0xffffffff80c454d8 at witness_checkorder+0xdb8 #1 0xffffffff80ba7ea2 at lockmgr_lock_flags+0x172 #2 0xffffffff810d66c0 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cd2c54 at _vn_lock+0x54 #4 0xffffffff80cbbf24 at vput_final+0x114 #5 0xffffffff80e9604f at ffs_unmount+0x3af #6 0xffffffff80cb317d at dounmount+0x43d #7 0xffffffff80cb2ce9 at kern_unmount+0x329 #8 0xffffffff81024d19 at amd64_syscall+0x159 #9 0xffffffff80ff859e at fast_syscall_common+0xf8 ** /dev/gpt/tmp ** Last Mounted on /tmp ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 535 files, 178038 used, 1087369 free (105 frags, 135908 blocks, 0.0% fragmentation) ***** FILE SYSTEM IS CLEAN ***** Updating motd:. Mounting late filesystems:. Security policy loaded: MAC/ntpd (mac_ntpd) Starting ntpd. Starting powerd. Configuring vt: blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting cron. sysctl: unknown oid 'debug.fail_point.pmap_allocpte_page_alloc' at line 12 Wed Sep 2 17:20 FreeBSD/amd64 (mercat1.netperf.freebsd.org) (ttyu1) login: root Sep 2 17:21:50 mercat1 login[2425]: ROOT LOGIN (root) ON ttyu1 Last login: Wed Sep 2 15:50:09 on ttyu1 FreeBSD 13.0-CURRENT (PHO) #1 r365247M: Wed Sep 2 17:12:18 CEST 2020 root@mercat1:~ # Sep 2 17:24:13 mercat1 su[2454]: pho to root on /dev/pts/0 20200902 17:24:44 all (1/49): callout_reset_on.sh Expensive timeout(9) function: 0xffffffff80bf0d70(0xfffff80549b22520) 0.008018823 s 20200902 17:26:16 all (2/49): callout_reset_on2.sh Expensive timeout(9) function: 0xffffffff80db5850(0xfffffe00f042a8a0) 2.122995933 s Expensive timeout(9) function: 0xffffffff80c6b560(0) 5.015999266 s Expensive timeout(9) function: 0xffffffff80c6b630(0) 154.070991918 s 20200902 17:36:19 all (3/49): datagram.sh stress2: newblk leak: 2/6. 20200902 17:36:22 all (4/49): datagram2.sh 20200902 17:36:23 all (5/49): datagram3.sh 20200902 17:36:25 all (6/49): jumbo.sh 20200902 17:37:26 all (7/49): kevent7.sh lock order reversal: (sleepable after non-sleepable) 1st 0xfffff80003962808 seqflq (seqflq, sleep mutex) @ dev/sound/midi/sequencer.c:757 2nd 0xffffffff81ae5ce0 midistat lock (midistat lock, sx) @ dev/sound/midi/midi.c:1481 lock order seqflq -> midistat lock attempted at: #0 0xffffffff80c454d8 at witness_checkorder+0xdb8 #1 0xffffffff80be1687 at _sx_xlock+0x67 #2 0xffffffff80980b52 at midimapper_open+0x22 #3 0xffffffff80983a32 at mseq_open+0xf2 #4 0xffffffff80a7861f at devfs_open+0x12f #5 0xffffffff810d3b36 at VOP_OPEN_APV+0x56 #6 0xffffffff80cd29ee at vn_open_vnode+0x18e #7 0xffffffff80cd2385 at vn_open_cred+0x395 #8 0xffffffff80cc972a at kern_openat+0x25a #9 0xffffffff81024d19 at amd64_syscall+0x159 #10 0xffffffff80ff859e at fast_syscall_common+0xf8 20200902 17:42:33 all (8/49): multicast.sh stress2: freework leak: 1/6. stress2: newblk leak: 2/6. 20200902 17:42:55 all (9/49): overflow3.sh if_delmulti_locked: detaching ifnet instance 0xfffff80039052000 20200902 17:44:57 all (10/49): sctp.sh 20200902 17:46:05 all (11/49): sendfile.sh stress2: newblk leak: 2/6. 20200902 17:46:07 all (12/49): sendfile10.sh 20200902 17:48:09 all (13/49): sendfile11.sh GEOM_ELI: Device md10.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software Sep 2 17:49:42 mercat1 kernel: pid 58495 (swap), jid 0, uid 0, was killed: out of swap space GEOM_ELI: md10 has been killed. GEOM_ELI: Device md10.eli destroyed. stress2: freework leak: 1/6. 20200902 17:49:45 all (14/49): sendfile12.sh 20200902 17:50:55 all (15/49): sendfile13.sh 20200902 17:56:31 all (16/49): sendfile14.sh 20200902 18:06:58 all (17/49): sendfile15.sh 20200902 18:12:05 all (18/49): sendfile16.sh 20200902 18:17:12 all (19/49): sendfile17.sh 20200902 18:22:14 all (20/49): sendfile18.sh GEOM_NOP: Device md10.nop created. lock order reversal: 1st 0xfffff8031a0eb600 unp (unp, sleep mutex) @ kern/uipc_usrreq.c:1221 2nd 0xffffffff81d98d80 unp_link_rwlock (unp_link_rwlock, rw) @ kern/uipc_usrreq.c:1240 lock order unp_link_rwlock -> unp established at: #0 0xffffffff80c44b8c at witness_checkorder+0x46c #1 0xffffffff80bb47f4 at __mtx_lock_flags+0x94 #2 0xffffffff80c847fd at uipc_peeraddr+0xdd #3 0xffffffff80c80c02 at kern_getpeername+0xc2 #4 0xffffffff80c80d8e at sys_getpeername+0x3e #5 0xffffffff81024d19 at amd64_syscall+0x159 #6 0xffffffff80ff859e at fast_syscall_common+0xf8 lock order unp -> unp_link_rwlock attempted at: #0 0xffffffff80c454d8 at witness_checkorder+0xdb8 #1 0xffffffff80bd2d42 at __rw_rlock_int+0x82 #2 0xffffffff80c85964 at uipc_ready+0x264 #3 0xffffffff80bd5c9c at sendfile_iodone+0x1ac #4 0xffffffff80ef060a at vnode_pager_generic_getpages_done_async+0x3a #5 0xffffffff80c95b2c at bufdone+0x6c #6 0xffffffff80b18bde at g_io_deliver+0x25e #7 0xffffffff80b18bde at g_io_deliver+0x25e #8 0xffffffff806ecc69 at md_kthread+0x259 #9 0xffffffff80b932d0 at fork_exit+0x80 #10 0xffffffff80ff8cce at fork_trampoline+0xe panic: mutex unp owned at ../../../kern/uipc_usrreq.c:804 cpuid = 10 time = 1599063736 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db800 vpanic() at vpanic+0x182/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 3100 tid 100372 ] Stopped at kdb_enter+0x37: movq $0,0x10b7396(%rip) db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> dump Dumping 1400 out of 32628 MB:..2%..11%..21%..31%..42%..51%..61%..71%..82%..91% Dump complete db:1:pho1> bt Tracing pid 3100 tid 100372 td 0xfffffe00dff06a00 kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:1:pho1> show allpcpu Current CPU: 10 cpuid = 0 dynamic pcpu = 0x797b00 curthread = 0xfffffe00ad7d9300: pid 11 tid 100003 critnest 1 "idle: cpu0" curpcb = 0xfffffe00ad7d9810 fpcurthread = none idlethread = 0xfffffe00ad7d9300: tid 100003 "idle: cpu0" self = 0xffffffff82210000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82210384 rsp0 = 0xfffffe00ad546cc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x733b65ef3 gs32p = 0xffffffff82210404 ldt = 0xffffffff82210444 tss = 0xffffffff82210434 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0xfffffe00bc247b00 curthread = 0xfffffe00ad7d8c00: pid 11 tid 100004 critnest 1 "idle: cpu1" curpcb = 0xfffffe00ad7d9110 fpcurthread = none idlethread = 0xfffffe00ad7d8c00: tid 100004 "idle: cpu1" self = 0xffffffff82211000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82211384 rsp0 = 0xfffffe00ad54bcc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x31b5ba8e1 gs32p = 0xffffffff82211404 ldt = 0xffffffff82211444 tss = 0xffffffff82211434 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0xfffffe00bc253b00 curthread = 0xfffffe00ad7d8500: pid 11 tid 100005 critnest 1 "idle: cpu2" curpcb = 0xfffffe00ad7d8a10 fpcurthread = none idlethread = 0xfffffe00ad7d8500: tid 100005 "idle: cpu2" self = 0xffffffff82212000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82212384 rsp0 = 0xfffffe00ad550cc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x2d1c4bd2 gs32p = 0xffffffff82212404 ldt = 0xffffffff82212444 tss = 0xffffffff82212434 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0xfffffe00bc25fb00 curthread = 0xfffffe00ad7d7e00: pid 11 tid 100006 critnest 1 "idle: cpu3" curpcb = 0xfffffe00ad7d8310 fpcurthread = none idlethread = 0xfffffe00ad7d7e00: tid 100006 "idle: cpu3" self = 0xffffffff82213000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82213384 rsp0 = 0xfffffe00ad555cc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x31bdc9abc gs32p = 0xffffffff82213404 ldt = 0xffffffff82213444 tss = 0xffffffff82213434 curvnet = 0 spin locks held: cpuid = 4 dynamic pcpu = 0xfffffe00bc26bb00 curthread = 0xfffffe00ad7d7700: pid 11 tid 100007 critnest 1 "idle: cpu4" curpcb = 0xfffffe00ad7d7c10 fpcurthread = none idlethread = 0xfffffe00ad7d7700: tid 100007 "idle: cpu4" self = 0xffffffff82214000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82214384 rsp0 = 0xfffffe00ad55acc0 kcr3 = 0x8000000002128004 ucr3 = 0xffffffffffffffff scr3 = 0x2a3bf7b39 gs32p = 0xffffffff82214404 ldt = 0xffffffff82214444 tss = 0xffffffff82214434 curvnet = 0 spin locks held: cpuid = 5 dynamic pcpu = 0xfffffe00bc277b00 curthread = 0xfffffe00ad7d7000: pid 11 tid 100008 critnest 1 "idle: cpu5" curpcb = 0xfffffe00ad7d7510 fpcurthread = none idlethread = 0xfffffe00ad7d7000: tid 100008 "idle: cpu5" self = 0xffffffff82215000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82215384 rsp0 = 0xfffffe00ad55fcc0 kcr3 = 0x8000000002128005 ucr3 = 0xffffffffffffffff scr3 = 0x766031b35 gs32p = 0xffffffff82215404 ldt = 0xffffffff82215444 tss = 0xffffffff82215434 curvnet = 0 spin locks held: cpuid = 6 dynamic pcpu = 0xfffffe00bc283b00 curthread = 0xfffffe00ad7e1a00: pid 11 tid 100009 critnest 1 "idle: cpu6" curpcb = 0xfffffe00ad7e1f10 fpcurthread = none idlethread = 0xfffffe00ad7e1a00: tid 100009 "idle: cpu6" self = 0xffffffff82216000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82216384 rsp0 = 0xfffffe00ad564cc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x20b2f879 gs32p = 0xffffffff82216404 ldt = 0xffffffff82216444 tss = 0xffffffff82216434 curvnet = 0 spin locks held: cpuid = 7 dynamic pcpu = 0xfffffe00bc28fb00 curthread = 0xfffffe00ad7e1300: pid 11 tid 100010 critnest 1 "idle: cpu7" curpcb = 0xfffffe00ad7e1810 fpcurthread = none idlethread = 0xfffffe00ad7e1300: tid 100010 "idle: cpu7" self = 0xffffffff82217000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82217384 rsp0 = 0xfffffe00ad569cc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x530de2ac3 gs32p = 0xffffffff82217404 ldt = 0xffffffff82217444 tss = 0xffffffff82217434 curvnet = 0 spin locks held: cpuid = 8 dynamic pcpu = 0xfffffe00bc29bb00 curthread = 0xfffffe00ad7e0c00: pid 11 tid 100011 critnest 1 "idle: cpu8" curpcb = 0xfffffe00ad7e1110 fpcurthread = none idlethread = 0xfffffe00ad7e0c00: tid 100011 "idle: cpu8" self = 0xffffffff82218000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82218384 rsp0 = 0xfffffe00ad56ecc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x1d772c9c8 gs32p = 0xffffffff82218404 ldt = 0xffffffff82218444 tss = 0xffffffff82218434 curvnet = 0 spin locks held: cpuid = 9 dynamic pcpu = 0xfffffe00bc2a7b00 curthread = 0xfffffe00ad7e0500: pid 11 tid 100012 critnest 1 "idle: cpu9" curpcb = 0xfffffe00ad7e0a10 fpcurthread = none idlethread = 0xfffffe00ad7e0500: tid 100012 "idle: cpu9" self = 0xffffffff82219000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff82219384 rsp0 = 0xfffffe00ad573cc0 kcr3 = 0x8000000002128006 ucr3 = 0xffffffffffffffff scr3 = 0x61ab51d43 gs32p = 0xffffffff82219404 ldt = 0xffffffff82219444 tss = 0xffffffff82219434 curvnet = 0 spin locks held: cpuid = 10 dynamic pcpu = 0xfffffe00bc2b3b00 curthread = 0xfffffe00dff06a00: pid 3100 tid 100372 critnest 1 "md10" curpcb = 0xfffffe00dff06f10 fpcurthread = none idlethread = 0xfffffe00ad7dfe00: tid 100013 "idle: cpu10" self = 0xffffffff8221a000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff8221a384 rsp0 = 0xfffffe00f06dbcc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x502394870 gs32p = 0xffffffff8221a404 ldt = 0xffffffff8221a444 tss = 0xffffffff8221a434 curvnet = 0xfffff800037b6040 spin locks held: cpuid = 11 dynamic pcpu = 0xfffffe00bc2bfb00 curthread = 0xfffffe00ad7df700: pid 11 tid 100014 critnest 1 "idle: cpu11" curpcb = 0xfffffe00ad7dfc10 fpcurthread = none idlethread = 0xfffffe00ad7df700: tid 100014 "idle: cpu11" self = 0xffffffff8221b000 curpmap = 0xffffffff81c8d060 tssp = 0xffffffff8221b384 rsp0 = 0xfffffe00ad57dcc0 kcr3 = 0x8000000002128002 ucr3 = 0xffffffffffffffff scr3 = 0x822714e0f gs32p = 0xffffffff8221b404 ldt = 0xffffffff8221b444 tss = 0xffffffff8221b434 curvnet = 0 spin locks held: db:1:pho1> show alllocks Process 3100 (md10) thread 0xfffffe00dff06a00 (100372) exclusive sleep mutex unp (unp) r = 0 (0xfffff8031a0eb600) locked @ kern/uipc_usrreq.c:1221 Process 2059 (syslogd) thread 0xfffffe00dfff3300 (100194) exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe003c82dc68) locked @ kern/vfs_bio.c:3870 exclusive lockmgr ufs (ufs) r = 0 (0xfffff8031a862a90) locked @ kern/vfs_syscalls.c:3468 db:1:pho1> show lockedvnods Locked vnodes vnode 0xfffff8031a862a20: type VREG usecount 1, writecount 1, refcount 4 seqc users 0 hold count flags () flags (VIRF_PGREAD|VMP_LAZYLIST) v_object 0xfffff8004595a948 ref 0 pages 16 cleanbuf 2 dirtybuf 0 lock type ufs: EXCL by thread 0xfffffe00dfff3300 (pid 2059, syslogd, tid 100194) #0 0xffffffff80ba7f05 at lockmgr_lock_flags+0x1d5 #1 0xffffffff80e983bd at ffs_lock+0x8d #2 0xffffffff810d66c0 at VOP_LOCK1_APV+0x40 #3 0xffffffff80cd2c54 at _vn_lock+0x54 #4 0xffffffff80cce662 at kern_fsync+0xf2 #5 0xffffffff81024d19 at amd64_syscall+0x159 #6 0xffffffff80ff859e at fast_syscall_common+0xf8 nlink=1, effnlink=1, size=682241, extsize 0 generation=1931733, uid=0, gid=0, flags=0x0 ino 9951830, on dev gpt/work db:1:pho1> show mount 0xfffffe00dfef5700 /dev/gpt/work on / (ufs) 0xfffffe00dfef6280 devfs on /dev (devfs) 0xfffffe00eff0d280 /dev/gpt/tmp on /tmp (ufs) 0xfffffe00dffe8280 /dev/md10.nop on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 103670 Buf domain 0 freebufs 15066 lofreebufs 219 hifreebufs 329 bufspace 1716224 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 29 (880640) dirtyq count 0 (0) wakeup 0 lim 25 CPU 3, 3, 2, 5, 2, 7, 4, 0, 5, 1, 4, 2, Locked buffers: 0 space 0 Total buffers: 23867 space 1716224 Buf domain 1 freebufs 14541 lofreebufs 219 hifreebufs 329 bufspace 18841600 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 1 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 476 (15548416) dirtyq count 1 (32768) wakeup 0 lim 25 CPU 5, 4, 22, 3, 6, 12, 7, 15, 18, 1, 4, 18, Locked buffers: 0 space 0 Total buffers: 20596 space 18841600 Buf domain 2 freebufs 15083 lofreebufs 219 hifreebufs 329 bufspace 1093632 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 7 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 7 (229376) wakeup 0 lim 25 CPU 7, 2, 2, 7, 3, 2, 1, 3, 7, 2, 4, 3, Locked buffers: 0 space 0 Total buffers: 7226 space 1093632 Buf domain 3 freebufs 15074 lofreebufs 219 hifreebufs 329 bufspace 1277952 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 0 lim 25 CPU 1, 7, 2, 7, 3, 5, 6, 2, 8, 2, 10, 6, Locked buffers: 0 space 0 Total buffers: 24130 space 1277952 Buf domain 4 freebufs 15104 lofreebufs 219 hifreebufs 329 bufspace 446464 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 0 (0) dirtyq count 0 (0) wakeup 0 lim 25 CPU 4, 3, 1, 2, 0, 4, 1, 6, 5, 1, 1, 1, Locked buffers: 0 space 0 Total buffers: 7306 space 446464 Buf domain 5 freebufs 13908 lofreebufs 219 hifreebufs 329 bufspace 39415808 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 0 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 1127 (36724736) dirtyq count 0 (0) wakeup 0 lim 25 CPU 4, 4, 4, 9, 9, 13, 8, 4, 8, 9, 4, 22, Locked buffers: 0 space 0 Total buffers: 15500 space 39415808 Buf domain 6 freebufs 15013 lofreebufs 219 hifreebufs 329 bufspace 3358720 maxbufspace 247939072 hibufspace 247845449 lobufspace 235453176 bufspacethresh 241649312 numdirtybuffers 32 lodirtybuffers 1893 hidirtybuffers 3786 dirtybufthresh 3407 cleanq count 44 (1253376) dirtyq count 32 (970752) wakeup 0 lim 25 CPU 3, 2, 7, 1, 3, 5, 1, 0, 7, 6, 5, 4, Locked buffers: 1 space 32768 Total buffers: 7264 space 3358720 db:1:pho1> show page vm_cnt.v_free_count: 7977592 vm_cnt.v_inactive_count: 340 vm_cnt.v_active_count: 1342 vm_cnt.v_laundry_count: 13 vm_cnt.v_wire_count: 138269 vm_cnt.v_free_reserved: 10644 vm_cnt.v_free_min: 51267 vm_cnt.v_free_target: 173136 vm_cnt.v_inactive_target: 259704 db:1:pho1> show pageq pq_free 7977592 dom 0 page_cnt 8124610 free 7977592 pq_act 1342 pq_inact 340 pq_laund 13 pq_unsw 0 db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000661 | 000000 11 (008192K) | 000875 | 000000 10 (004096K) | 001371 | 000000 09 (002048K) | 001454 | 000000 08 (001024K) | 001810 | 000043 07 (000512K) | 002109 | 000040 06 (000256K) | 003062 | 000066 05 (000128K) | 003671 | 000126 04 (000064K) | 004891 | 000057 03 (000032K) | 007058 | 000334 02 (000016K) | 009170 | 001369 01 (000008K) | 013916 | 002652 00 (000004K) | 021249 | 004069 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 (016384K) | 000000 | 000000 11 (008192K) | 000000 | 000000 10 (004096K) | 000000 | 000000 09 (002048K) | 000000 | 000000 08 (001024K) | 000000 | 000000 07 (000512K) | 000000 | 000000 06 (000256K) | 000001 | 000000 05 (000128K) | 000001 | 000000 04 (000064K) | 000002 | 000000 03 (000032K) | 000000 | 000000 02 (000016K) | 000001 | 000000 01 (000008K) | 000000 | 000000 00 (000004K) | 000000 | 000000 db:0:pho> show uma Zone Size Used Free Requests Sleeps Bucket Total Mem XFree RADIX NODE 144 6916 535162 78055526 0 254 78059232 0 mbuf_cluster 2048 14336 32 42529 0 8 29425664 0 BUF TRIE 144 326 105626 2490948 0 62 15257088 0 swblk 136 1498 58764 828474 0 62 8195632 0 VNODE 648 10373 325 19519 0 16 6932304 0 mbuf 256 16428 5026 709425393 0 254 5492224 0 socket 960 53 3759 3362917 0 17 3659520 0 VM OBJECT 264 3315 7725 36316617 0 54 2914560 0 256 256 10340 745 176318 0 62 2837760 0 FFS2 dinode 256 10188 687 19281 0 62 2784000 0 THREAD 1792 984 177 14072 0 8 2080512 0 128 128 14450 1267 972701 0 126 2011776 0 VMSPACE 2536 42 720 3600168 0 14 1932432 0 FFS inode 160 10188 1112 19281 0 62 1808000 0 4096 4096 77 324 3600225 0 9 1642496 0 128 128 10730 1143 10883 0 126 1519744 0 S VFS Cache 104 10031 4360 34748 0 126 1496664 0 4096 4096 336 15 1475 0 2 1437696 0 256 Bucket 2048 577 73 3054641 0 8 1331200 0 UMA Slabs 0 112 10051 1751 73700 0 126 1321824 0 32768 32768 12 26 376 0 1 1245184 0 PROC 1312 76 788 3600224 0 10 1133568 0 vmem btag 56 12429 7215 24328 0 254 1100064 0 DEVCTL 1024 0 1000 964 0 16 1024000 0 MAP ENTRY 96 1658 7960 83046573 0 135 923328 0 64 64 9099 4005 6085672 0 254 838656 0 64 64 9494 2854 14276528 0 254 790272 0 65536 65536 5 7 71 0 1 786432 0 swpctrie 144 226 5206 149857 0 62 782208 0 pbuf 984 0 761 0 0 0 748824 0 64 Bucket 512 950 434 610077 0 30 708608 0 8192 8192 86 0 337 0 1 704512 0 128 128 3334 1533 242310 0 126 622976 0 filedesc0 1080 77 462 3600225 0 16 582120 0 mbuf_packet 256 25 2026 5301561 0 254 525056 0 256 256 936 1059 7707420 0 62 510720 0 512 512 886 66 899 0 30 487424 0 128 Bucket 1024 264 207 1156968 0 16 482304 0 UMA Zones 2048 229 0 229 0 8 468992 0 32768 32768 12 0 1899 0 1 393216 0 32768 32768 12 0 13 0 1 393216 0 1024 1024 161 191 1229 0 16 360448 0 ksiginfo 112 838 2294 4369722 0 126 350784 0 2048 2048 18 146 459244 0 8 335872 0 65536 65536 3 2 2236 0 1 327680 0 65536 65536 0 5 116 0 1 327680 0 32 32 6816 3138 328210 0 254 318528 0 128 128 769 1525 445524 0 126 293632 0 64 64 3248 1288 414816 0 254 290304 0 8192 8192 19 15 2466 0 1 278528 0 unpcb 256 21 1059 3169585 0 62 276480 0 32 Bucket 256 142 923 781855 0 62 272640 0 64 64 136 3959 18895076 0 254 262080 0 1024 1024 38 210 475646 0 16 253952 0 16 16 13842 1908 894167 0 254 252000 0 16384 16384 15 0 168233 0 1 245760 0 4096 4096 47 12 90501 0 2 241664 0 g_bio 408 4 576 3496926 0 30 236640 0 64 64 42 3612 353521 0 254 233856 0 Files 80 127 2673 7919416 0 126 224000 0 pipe 760 8 277 197551 0 16 216600 0 NAMEI 1024 0 208 5593630 0 16 212992 0 TURNSTILE 136 1162 392 1162 0 62 211344 0 4096 4096 30 20 1656 0 2 204800 0 1024 1024 8 192 81911 0 16 204800 0 65536 65536 3 0 3 0 1 196608 0 65536 65536 3 0 3 0 1 196608 0 16 Bucket 144 349 967 2205395 0 62 189504 0 512 512 95 273 9743 0 30 188416 0 4 Bucket 48 5 3859 1265146 0 254 185472 0 256 256 79 626 16444 0 62 180480 0 2048 2048 88 0 89 0 8 180224 0 8 Bucket 80 86 2064 536085 0 126 172000 0 32768 32768 3 2 2830 0 1 163840 0 16384 16384 10 0 1636248 0 1 163840 0 udp_inpcb 488 20 300 412 0 30 156160 0 2048 2048 13 57 1123 0 8 143360 0 PWD 32 18 4266 26352 0 254 137088 0 32 32 133 4151 3570079 0 254 137088 0 SLEEPQUEUE 88 1162 342 1162 0 126 132352 0 65536 65536 2 0 13 0 1 131072 0 256 256 227 253 2634 0 62 122880 0 256 256 43 437 2411 0 62 122880 0 fakepg 104 0 1170 84 0 126 121680 0 2 Bucket 32 180 3600 665883 0 254 120960 0 128 128 40 890 876 0 126 119040 0 rl_entry 40 888 1940 888 0 254 113120 0 selfd 64 185 1516 46008 0 254 108864 0 32 32 300 3102 24757 0 254 108864 0 32 32 578 2824 29023 0 254 108864 0 8192 8192 13 0 45 0 1 106496 0 512 512 23 185 7928 0 30 106496 0 128 128 41 765 3205094 0 126 103168 0 DIRHASH 1024 30 66 388 0 16 98304 0 512 512 1 183 125 0 30 94208 0 ttyinq 160 330 245 540 0 62 92000 0 32 32 216 2556 9880 0 254 88704 0 16384 16384 0 5 29 0 1 81920 0 UMA Kegs 384 199 7 199 0 30 79104 0 tcp_inpcb 488 9 151 32796 0 30 78080 0 ttyoutq 256 173 127 282 0 62 76800 0 udpcb 32 20 2374 412 0 254 76608 0 8192 8192 9 0 9 0 1 73728 0 32 32 182 2086 1484 0 254 72576 0 4096 4096 16 1 16 0 2 69632 0 1024 1024 18 50 32 0 16 69632 0 512 512 113 23 181 0 30 69632 0 64 64 152 919 661 0 254 68544 0 32 32 38 2104 33247 0 254 68544 0 128 128 18 509 88 0 126 67456 0 128 128 142 354 641 0 126 63488 0 1024 1024 49 11 53 0 16 61440 0 512 512 42 78 640221 0 30 61440 0 16 16 472 3278 4460971 0 254 60000 0 Mountpoints 2944 4 16 17 0 4 58880 0 8192 8192 4 3 18 0 1 57344 0 tcpcb 1040 9 46 32796 0 8 57200 0 64 64 28 791 1395 0 254 52416 0 16 16 395 2855 2716 0 254 52000 0 2048 2048 17 7 17 0 8 49152 0 16 16 435 2565 20053 0 254 48000 0 mt_stats_zone 64 439 265 439 0 254 45056 0 64 pcpu 8 3247 1873 3248 0 254 40960 0 swrbuf 984 0 41 98003 0 2 40344 0 1024 1024 2 34 21 0 16 36864 0 512 512 24 48 25 0 30 36864 0 KMAP ENTRY 96 28 347 29 0 126 36000 0 procdesc 136 1 260 88 0 62 35496 0 cpuset 104 27 314 27 0 126 35464 0 32768 32768 1 0 1 0 1 32768 0 512 512 30 34 46 0 30 32768 0 256 256 22 98 160650 0 62 30720 0 4096 4096 3 4 329 0 2 28672 0 rentr 24 0 1169 15 0 254 28056 0 mt_zone 24 439 730 439 0 254 28056 0 16 16 9 1741 133 0 254 28000 0 rtentry 176 12 126 15 0 62 24288 0 32 32 10 746 10 0 254 24192 0 16 16 45 1455 222 0 254 24000 0 16 16 32 1468 34 0 254 24000 0 routing nhops 256 7 83 15 0 62 23040 0 256 256 25 65 307 0 62 23040 0 1024 1024 13 7 16 0 16 20480 0 16384 16384 1 0 1 0 1 16384 0 16384 16384 1 0 1 0 1 16384 0 8192 8192 2 0 2 0 1 16384 0 8192 8192 2 0 2 0 1 16384 0 2048 2048 4 2 5643 0 8 12288 0 256 256 18 27 98 0 62 11520 0 4096 4096 2 0 269 0 2 8192 0 1024 1024 7 1 31 0 16 8192 0 tcptw 88 0 92 1195 0 126 8096 0 vnpbuf 984 1 7 160477 0 8 7872 0 FPU_save_area 832 1 8 1 0 16 7488 0 epoch_record pcpu 256 4 12 4 0 62 4096 0 2048 2048 1 1 1 0 8 4096 0 int pcpu 4 82 942 82 0 254 4096 0 hostcache 96 1 41 1 0 126 4032 0 64 64 1 62 2 0 254 4032 0 ripcb 488 1 7 3 0 30 3904 0 UMA Slabs 1 176 11 11 11 0 62 3872 0 L VFS Cache 320 6 6 6 0 30 3840 0 vmem 1856 1 1 1 0 8 3712 0 SMR CPU 32 2 93 2 0 254 3040 0 SMR SHARED 24 2 93 2 0 254 2280 0 clpbuf 984 0 2 947612 0 2 1968 0 sctp_asconf_ack 48 0 0 0 0 254 0 0 sctp_asconf 40 0 0 0 0 254 0 0 sctp_stream_msg_out 112 0 0 80000 0 126 0 0 sctp_readq 152 0 0 80000 0 62 0 0 sctp_chunk 152 0 0 320000 0 62 0 0 sctp_raddr 736 0 0 320000 0 16 0 0 sctp_laddr 48 0 0 0 0 254 0 0 sctp_asoc 2288 0 0 160000 0 4 0 0 sctp_ep 1280 0 0 160100 0 8 0 0 FFS1 dinode 128 0 0 0 0 126 0 0 IPsec SA lft_c 16 0 0 0 0 254 0 0 udplite_inpcb 488 0 0 0 0 30 0 0 tcp_log_node 120 0 0 0 0 126 0 0 tcp_log_bucket 176 0 0 0 0 62 0 0 tcp_log 416 0 0 0 0 30 0 0 tcpreass 48 0 0 7406 0 254 0 0 tfo_ccache_entries 80 0 0 0 0 126 0 0 tfo 4 0 0 0 0 254 0 0 sackhole 32 0 0 10034 0 254 0 0 syncache 168 0 0 9092 0 62 0 0 ipq 56 0 0 0 0 254 0 0 KNOTE 160 0 0 2355 0 62 0 0 itimer 352 0 0 0 0 30 0 0 AIOLIO 280 0 0 0 0 30 0 0 AIOCB 752 0 0 0 0 16 0 0 AIOP 32 0 0 0 0 254 0 0 AIO 208 0 0 0 0 62 0 0 TMPFS node 232 0 0 0 0 62 0 0 TMPFS dirent 64 0 0 0 0 254 0 0 NCLNODE 744 0 0 0 0 16 0 0 LTS VFS Cache 360 0 0 0 0 30 0 0 STS VFS Cache 144 0 0 0 0 62 0 0 VNODEPOLL 120 0 0 0 0 126 0 0 vtnet_tx_hdr 24 0 0 0 0 254 0 0 crypto_session 72 0 0 13 0 126 0 0 cryptop 280 0 0 144684 0 30 0 0 nvme_request 128 0 0 0 0 126 0 0 IOMMU_MAP_ENTRY 120 0 0 0 0 126 0 0 mbuf_jumbo_16k 16384 0 0 0 0 1 0 0 mbuf_jumbo_9k 9216 0 0 0 0 1 0 0 mbuf_jumbo_page 4096 0 0 10491965 0 2 0 0 audit_record 1280 0 0 0 0 8 0 0 domainset 40 0 0 0 0 254 0 0 MAC labels 40 0 0 0 0 254 0 0 mdpbuf 984 0 0 0 0 0 0 0 nfspbuf 984 0 0 0 0 2 0 0 swwbuf 984 0 0 183187 0 0 0 0 umtx_shm 88 0 0 0 0 126 0 0 umtx pi 96 0 0 0 0 126 0 0 rangeset pctrie nodes 144 0 0 0 0 62 0 0 kenv 258 0 0 19248 0 30 0 0 65536 65536 0 0 0 0 1 0 0 65536 65536 0 0 0 0 1 0 0 32768 32768 0 0 0 0 1 0 0 32768 32768 0 0 38 0 1 0 0 32768 32768 0 0 0 0 1 0 0 16384 16384 0 0 0 0 1 0 0 16384 16384 0 0 16 0 1 0 0 16384 16384 0 0 0 0 1 0 0 8192 8192 0 0 0 0 1 0 0 4096 4096 0 0 0 0 2 0 0 2048 2048 0 0 0 0 8 0 0 2048 2048 0 0 0 0 8 0 0 16 16 0 0 4 0 254 0 0 UMA Hash 256 0 0 0 0 62 0 0 db:0:pho> show umacache Zone Size Used Free Requests Bucket debugnet_mbuf_packet 2048 0 0 0 0 debugnet_mbuf_cluster 2048 -6144 0 0 0 debugnet_mbuf 256 -10240 0 0 0 vm pgcache 4096 45853 3615 72943093 81 vm pgcache 4096 8435 3276 224202799 145 buf free cache 984 2142 119 3771295 16 buffer arena-65 65536 0 0 0 1 buffer arena-61 61440 0 0 0 1 buffer arena-57 57344 0 0 0 1 buffer arena-53 53248 0 0 0 1 buffer arena-49 49152 0 0 0 1 buffer arena-45 45056 0 0 0 1 buffer arena-40 40960 0 0 0 1 buffer arena-36 36864 0 0 0 1 buffer arena-32 32768 124 17 9733 1 buffer arena-28 28672 0 0 0 1 buffer arena-24 24576 0 0 0 1 buffer arena-20 20480 0 0 0 1 buffer arena-16 16384 5 6 533 1 buffer arena-12 12288 0 0 0 1 buffer arena-81 8192 0 0 0 1 buffer arena-40 4096 0 0 0 2 kstack_cache 16384 984 24 14072 1 db:0:pho> acttrace Tracing command md10 pid 3100 tid 100372 td 0xfffffe00dff06a00 (CPU 10) kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00ad7d9300 (CPU 0) cpustop_handler() at cpustop_handler+0x26/frame 0xffffffff81de70a0 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81de70b0 trap() at trap+0x56/frame 0xffffffff81de71c0 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81de71c0 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad546a50, rbp = 0xfffffe00ad546a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad546a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad546aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad546ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad546ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad546bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad546bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad546bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00ad7d8c00 (CPU 1) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbd0e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbd0e10 trap() at trap+0x56/frame 0xfffffe003dbd0f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbd0f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad54ba50, rbp = 0xfffffe00ad54ba50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad54ba50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad54baa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad54bac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad54bae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad54bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad54bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad54bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00ad7d8500 (CPU 2) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbdce00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbdce10 trap() at trap+0x56/frame 0xfffffe003dbdcf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbdcf20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad550a50, rbp = 0xfffffe00ad550a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad550a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad550aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad550ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad550ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad550bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad550bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad550bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00ad7d7e00 (CPU 3) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbe8e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbe8e10 trap() at trap+0x56/frame 0xfffffe003dbe8f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbe8f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad555a50, rbp = 0xfffffe00ad555a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad555a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad555aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad555ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad555ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad555bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad555bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad555bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00ad7d7700 (CPU 4) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbf4e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbf4e10 trap() at trap+0x56/frame 0xfffffe003dbf4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbf4f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad55aa50, rbp = 0xfffffe00ad55aa50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad55aa50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad55aaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad55aac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad55aae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad55abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad55abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad55abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffffe00ad7d7000 (CPU 5) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc00e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc00e10 trap() at trap+0x56/frame 0xfffffe003dc00f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc00f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad55fa50, rbp = 0xfffffe00ad55fa50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad55fa50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad55faa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad55fac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad55fae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad55fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad55fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad55fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00ad7e1a00 (CPU 6) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc0ce00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc0ce10 trap() at trap+0x56/frame 0xfffffe003dc0cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc0cf20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad564a50, rbp = 0xfffffe00ad564a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad564a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad564aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad564ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad564ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad564bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad564bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad564bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffffe00ad7e1300 (CPU 7) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc18e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc18e10 trap() at trap+0x56/frame 0xfffffe003dc18f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc18f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad569a50, rbp = 0xfffffe00ad569a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad569a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad569aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad569ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad569ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad569bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad569bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad569bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00ad7e0c00 (CPU 8) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc24e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc24e10 trap() at trap+0x56/frame 0xfffffe003dc24f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc24f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad56ea50, rbp = 0xfffffe00ad56ea50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad56ea50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad56eaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad56eac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad56eae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad56ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad56ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad56ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffffe00ad7e0500 (CPU 9) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc30e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc30e10 trap() at trap+0x56/frame 0xfffffe003dc30f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc30f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad573a50, rbp = 0xfffffe00ad573a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad573a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad573aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad573ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad573ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad573bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad573bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad573bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00ad7df700 (CPU 11) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc48e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc48e10 trap() at trap+0x56/frame 0xfffffe003dc48f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc48f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad57da50, rbp = 0xfffffe00ad57da50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad57da50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad57daa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad57dac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad57dae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad57dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad57dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad57dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 3111 3080 2461 0 S+ nanslp 0xffffffff81cacdc3 sendfile18 3100 0 0 0 RL CPU 10 [md10] 3080 3078 2461 0 S+ wait 0xfffff80388885520 sh 3079 2463 2461 0 S+ piperd 0xfffff803883fb5f0 tee 3078 2463 2461 0 S+ wait 0xfffff805490de000 sh 2457 2746 2696 0 S nanslp 0xffffffff81cacdca sleep 55463 0 0 0 DL waiting 0xffffffff823aff98 [sctp_iterator] 2749 2696 2696 0 S piperd 0xfffff80045b325f0 sh 2747 2696 2696 0 S piperd 0xfffff80045b32be0 awk 2746 2696 2696 0 S wait 0xfffff80045a4e000 sh 2696 1 2696 0 SWs wait 0xfffff800456fba40 sh 2463 2461 2461 0 S+ wait 0xfffff800456faa40 sh 2461 2456 2461 0 SW+ wait 0xfffff80045a28520 sh 2456 2454 2456 0 S+ select 0xfffff8004553f0c0 bash 2454 2453 2454 0 SW+ pause 0xfffff80045a320a8 csh 2453 2446 2453 2006 SW+ wait 0xfffff8004565f520 su 2446 2445 2446 2006 SWs+ wait 0xfffff80045c14520 bash 2445 2442 2442 2006 S select 0xfffff8003909dc40 sshd 2442 2363 2442 0 Ss select 0xfffff8004553f140 sshd 2428 2425 2428 0 S+ ttyin 0xfffff800392228b0 csh 2425 1 2425 0 SWs+ wait 0xfffff800458f3000 login 2424 1 2424 0 Ss+ ttyin 0xfffff80039222cb0 getty 2423 1 2423 0 Ss+ ttyin 0xfffff8004531d0b0 getty 2422 1 2422 0 Ss+ ttyin 0xfffff8004531d4b0 getty 2421 1 2421 0 Ss+ ttyin 0xfffff8004531d8b0 getty 2420 1 2420 0 Ss+ ttyin 0xfffff8004531dcb0 getty 2419 1 2419 0 Ss+ ttyin 0xfffff8004531c0b0 getty 2418 1 2418 0 Ss+ ttyin 0xfffff8004531c4b0 getty 2417 1 2417 0 Ss+ ttyin 0xfffff8004531c8b0 getty 2416 1 2416 0 Ss+ ttyin 0xfffff8004531ccb0 getty 2367 1 2367 0 Ss nanslp 0xffffffff81cacdc9 cron 2363 1 2363 0 Ss select 0xfffff80045b5c240 sshd 2335 1 2335 0 Ss select 0xfffff8000407c640 powerd 2332 1 2332 123 Ss (threaded) ntpd 100193 S select 0xfffff8004553f1c0 ntpd 101108 S usem 0xfffff805be3c2900 ntpd 2267 1 2267 0 Ss rpcsvc 0xfffff80045705960 NLM: master 2264 1 2264 0 Ss select 0xfffff8031a0084c0 rpc.statd 2218 1 2218 0 Ss nanslp 0xffffffff81cacdc0 watchdogd 2069 1 2069 0 Ss select 0xfffff800040fda40 rpcbind 2059 1 2059 0 Ds biowr 0xfffffe003c82dbe8 syslogd 1988 1 1988 0 Ss select 0xfffff8003908e0c0 devd 1983 1 1983 65 Ss select 0xfffff80003f9c9c0 dhclient 1924 1 1924 0 Ss select 0xfffff800043fc440 dhclient 1921 1 1921 0 Ss select 0xfffff8004594e640 dhclient 33 0 0 0 DL syncer 0xffffffff81d99ba8 [syncer] 32 0 0 0 DL vlruwt 0xfffff8004530b520 [vnlru] 31 0 0 0 DL (threaded) [bufdaemon] 100131 D qsleep 0xffffffff81d98ef0 [bufdaemon] 100132 D - 0xffffffff8180aa00 [bufspacedaemon-0] 100133 D - 0xffffffff81812cc0 [bufspacedaemon-1] 100134 D - 0xffffffff8181af80 [bufspacedaemon-2] 100135 D - 0xffffffff81823240 [bufspacedaemon-3] 100136 D - 0xffffffff8182b500 [bufspacedaemon-4] 100137 D - 0xffffffff818337c0 [bufspacedaemon-5] 100138 D - 0xffffffff8183ba80 [bufspacedaemon-6] 100154 D sdflush 0xfffff80045a75ce8 [/ worker] 100212 D sdflush 0xfffff803886dbce8 [/tmp worker] 101184 D sdflush 0xfffff8076654c4e8 [/mnt worker] 30 0 0 0 DL psleep 0xffffffff81dbe048 [vmdaemon] 29 0 0 0 DL (threaded) [pagedaemon] 100129 D psleep 0xffffffff81db24f8 [dom0] 100139 D launds 0xffffffff81db2504 [laundry: dom0] 100140 D umarcl 0xffffffff80eb7ae0 [uma] 28 0 0 0 DL idle 0xfffff80003bb4000 [enc_daemon1] 27 0 0 0 DL idle 0xfffff80003bb4520 [enc_daemon0] 26 0 0 0 DL - 0xffffffff81ae0988 [rand_harvestq] 25 0 0 0 DL - 0xffffffff81d9882c [soaiod4] 24 0 0 0 DL - 0xffffffff81d9882c [soaiod3] 23 0 0 0 DL - 0xffffffff81d9882c [soaiod2] 22 0 0 0 DL - 0xffffffff81d9882c [soaiod1] 21 0 0 0 DL (threaded) [usb] 100098 D - 0xfffffe00dabfc440 [usbus0] 100099 D - 0xfffffe00dabfc498 [usbus0] 100100 D - 0xfffffe00dabfc4f0 [usbus0] 100101 D - 0xfffffe00dabfc548 [usbus0] 100102 D - 0xfffffe00dabfc5a0 [usbus0] 100104 D - 0xfffffe00db0a0cf0 [usbus1] 100105 D - 0xfffffe00db0a0d48 [usbus1] 100106 D - 0xfffffe00db0a0da0 [usbus1] 100107 D - 0xfffffe00db0a0df8 [usbus1] 100108 D - 0xfffffe00db0a0e50 [usbus1] 100109 D - 0xfffffe00dc073cf0 [usbus2] 100110 D - 0xfffffe00dc073d48 [usbus2] 100111 D - 0xfffffe00dc073da0 [usbus2] 100112 D - 0xfffffe00dc073df8 [usbus2] 100113 D - 0xfffffe00dc073e50 [usbus2] 20 0 0 0 RL (threaded) [cam] 100092 D - 0xffffffff819b86c0 [doneq0] 100093 RunQ [doneq1] 100094 D - 0xffffffff819b87c0 [doneq2] 100126 D - 0xffffffff819b8590 [scanner] 19 0 0 0 DL crypto_ 0xfffff80003ba6c50 [crypto returns 11] 18 0 0 0 DL crypto_ 0xfffff80003ba6bf0 [crypto returns 10] 17 0 0 0 DL crypto_ 0xfffff80003ba6b90 [crypto returns 9] 16 0 0 0 DL crypto_ 0xfffff80003ba6b30 [crypto returns 8] 15 0 0 0 DL crypto_ 0xfffff80003ba6ad0 [crypto returns 7] 9 0 0 0 DL crypto_ 0xfffff80003ba6a70 [crypto returns 6] 8 0 0 0 DL crypto_ 0xfffff80003ba6a10 [crypto returns 5] 7 0 0 0 DL crypto_ 0xfffff80003ba69b0 [crypto returns 4] 6 0 0 0 DL crypto_ 0xfffff80003ba6950 [crypto returns 3] 5 0 0 0 DL crypto_ 0xfffff80003ba68f0 [crypto returns 2] 4 0 0 0 DL crypto_ 0xfffff80003ba6890 [crypto returns 1] 3 0 0 0 DL crypto_ 0xfffff80003ba6830 [crypto returns 0] 2 0 0 0 DL crypto_ 0xffffffff81dafa20 [crypto] 14 0 0 0 DL seqstat 0xfffff80003962888 [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100063 D - 0xffffffff81c8bd50 [g_event] 100064 D - 0xffffffff81c8bd58 [g_up] 100065 D - 0xffffffff81c8bd60 [g_down] 12 0 0 0 WL (threaded) [intr] 100041 I [swi6: task queue] 100042 I [swi6: Giant taskq] 100045 I [swi5: fast taskq] 100048 I [swi4: clock (0)] 100049 I [swi4: clock (1)] 100050 I [swi4: clock (2)] 100051 I [swi4: clock (3)] 100052 I [swi4: clock (4)] 100053 I [swi4: clock (5)] 100054 I [swi4: clock (6)] 100055 I [swi4: clock (7)] 100056 I [swi4: clock (8)] 100057 I [swi4: clock (9)] 100058 I [swi4: clock (10)] 100059 I [swi4: clock (11)] 100060 I [swi1: netisr 0] 100061 I [swi3: vm] 100095 I [swi4: apei] 100096 I [irq56: ahci0] 100097 I [irq19: xhci0] 100103 I [irq18: ehci0 ehci1] 100114 I [irq71: ahci1] 100115 I [swi0: uart uart] 100176 I [irq72: ioat0] 100177 I [irq73: ioat1] 100178 I [irq74: ioat2] 100179 I [irq75: ioat3] 100180 I [irq76: ioat4] 100181 I [irq77: ioat5] 100182 I [irq78: ioat6] 100183 I [irq79: ioat7] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 Run CPU 6 [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 100011 Run CPU 8 [idle: cpu8] 100012 Run CPU 9 [idle: cpu9] 100013 CanRun [idle: cpu10] 100014 Run CPU 11 [idle: cpu11] 1 0 1 0 SLs wait 0xfffff800038df000 [init] 10 0 0 0 DL audit_w 0xffffffff81daff08 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff81c8c2f0 [swapper] 100015 D - 0xfffff80003917100 [softirq_0] 100016 D - 0xfffff80003917000 [softirq_1] 100017 D - 0xfffff80003914e00 [softirq_2] 100018 D - 0xfffff80003914d00 [softirq_3] 100019 D - 0xfffff80003914c00 [softirq_4] 100020 D - 0xfffff80003914b00 [softirq_5] 100021 D - 0xfffff80003914a00 [softirq_6] 100022 D - 0xfffff80003914900 [softirq_7] 100023 D - 0xfffff80003914800 [softirq_8] 100024 D - 0xfffff80003914700 [softirq_9] 100025 D - 0xfffff80003914600 [softirq_10] 100026 D - 0xfffff80003914500 [softirq_11] 100027 D - 0xfffff80003914400 [if_io_tqg_0] 100028 D - 0xfffff80003914300 [if_io_tqg_1] 100029 D - 0xfffff80003914200 [if_io_tqg_2] 100030 D - 0xfffff80003914100 [if_io_tqg_3] 100031 D - 0xfffff80003914000 [if_io_tqg_4] 100032 D - 0xfffff80003915e00 [if_io_tqg_5] 100033 D - 0xfffff80003915d00 [if_io_tqg_6] 100034 D - 0xfffff80003915c00 [if_io_tqg_7] 100035 D - 0xfffff80003915b00 [if_io_tqg_8] 100036 D - 0xfffff80003915a00 [if_io_tqg_9] 100037 D - 0xfffff80003915900 [if_io_tqg_10] 100038 D - 0xfffff80003915800 [if_io_tqg_11] 100039 D - 0xfffff80003915700 [if_config_tqg_0] 100040 D - 0xfffff80003920100 [kqueue_ctx taskq] 100043 D - 0xfffff80003921d00 [thread taskq] 100044 D - 0xfffff80003921c00 [inm_free taskq] 100046 D - 0xfffff80003921a00 [aiod_kick taskq] 100047 D - 0xfffff80003921900 [in6m_free taskq] 100062 D - 0xfffff80003921800 [firmware taskq] 100067 D - 0xfffff80003921700 [crypto_0] 100068 D - 0xfffff80003921700 [crypto_1] 100069 D - 0xfffff80003921700 [crypto_2] 100070 D - 0xfffff80003921700 [crypto_3] 100071 D - 0xfffff80003921700 [crypto_4] 100072 D - 0xfffff80003921700 [crypto_5] 100073 D - 0xfffff80003921700 [crypto_6] 100074 D - 0xfffff80003921700 [crypto_7] 100075 D - 0xfffff80003921700 [crypto_8] 100076 D - 0xfffff80003921700 [crypto_9] 100077 D - 0xfffff80003921700 [crypto_10] 100078 D - 0xfffff80003921700 [crypto_11] 100116 D - 0xfffff800390b3100 [mca taskq] 100121 D - 0xfffff80045273100 [acpi_task_0] 100122 D - 0xfffff80045273100 [acpi_task_1] 100123 D - 0xfffff80045273100 [acpi_task_2] 100125 D - 0xfffff80003921500 [CAM taskq] db:0:pho> allt Tracing command sendfile18 pid 3111 tid 100556 td 0xfffffe00f0571a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f0a6e870 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0a6e890 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0a6e8d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0a6e930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00f0a6e970 _sleep() at _sleep+0x258/frame 0xfffffe00f0a6ea10 kern_clock_nanosleep() at kern_clock_nanosleep+0x1e2/frame 0xfffffe00f0a6ea90 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe00f0a6ead0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0a6ebf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0a6ebf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8003b93ba, rsp = 0x7fffffffe618, rbp = 0x7fffffffe660 --- Tracing command md10 pid 3100 tid 100372 td 0xfffffe00dff06a00 (CPU 10) kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 3080 tid 100426 td 0xfffffe00dff10300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f07e46e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f07e4700 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f07e4740 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f07e47a0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f07e47b0 _sleep() at _sleep+0x2c3/frame 0xfffffe00f07e4850 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00f07e48e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe00f07e4ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f07e4bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f07e4bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80041a25a, rsp = 0x7fffffffe278, rbp = 0x7fffffffe330 --- Tracing command tee pid 3079 tid 100219 td 0xfffffe00ee421800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f025b840 mi_switch() at mi_switch+0x155/frame 0xfffffe00f025b860 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f025b8a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f025b900 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f025b910 _sleep() at _sleep+0x2c3/frame 0xfffffe00f025b9b0 pipe_read() at pipe_read+0x518/frame 0xfffffe00f025ba10 dofileread() at dofileread+0x81/frame 0xfffffe00f025ba60 sys_read() at sys_read+0xbc/frame 0xfffffe00f025bad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f025bbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f025bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004290da, rsp = 0x7fffffffe648, rbp = 0x7fffffffe6a0 --- Tracing command sh pid 3078 tid 100867 td 0xfffffe00f0d1e700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f12826e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f1282700 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f1282740 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f12827a0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f12827b0 _sleep() at _sleep+0x2c3/frame 0xfffffe00f1282850 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00f12828e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe00f1282ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f1282bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f1282bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80041a25a, rsp = 0x7fffffffdda8, rbp = 0x7fffffffde60 --- Tracing command sleep pid 2457 tid 100608 td 0xfffffe00f05d3000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f0b72870 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0b72890 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0b728d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0b72930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00f0b72970 _sleep() at _sleep+0x258/frame 0xfffffe00f0b72a10 kern_clock_nanosleep() at kern_clock_nanosleep+0x1e2/frame 0xfffffe00f0b72a90 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe00f0b72ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0b72bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0b72bf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8003ba3ba, rsp = 0x7fffffffe658, rbp = 0x7fffffffe6a0 --- Tracing command sctp_iterator pid 55463 tid 100504 td 0xfffffe00f0490100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f096aa90 mi_switch() at mi_switch+0x155/frame 0xfffffe00f096aab0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f096aaf0 _sleep() at _sleep+0x2cd/frame 0xfffffe00f096ab90 sctp_iterator_thread() at sctp_iterator_thread+0x59/frame 0xfffffe00f096abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f096abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f096abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 2749 tid 100186 td 0xfffffe00dff04700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e656e840 mi_switch() at mi_switch+0x155/frame 0xfffffe00e656e860 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e656e8a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e656e900 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e656e910 _sleep() at _sleep+0x2c3/frame 0xfffffe00e656e9b0 pipe_read() at pipe_read+0x518/frame 0xfffffe00e656ea10 dofileread() at dofileread+0x81/frame 0xfffffe00e656ea60 sys_read() at sys_read+0xbc/frame 0xfffffe00e656ead0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e656ebf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e656ebf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004890da, rsp = 0x7fffffffddb8, rbp = 0x7fffffffe2e0 --- Tracing command awk pid 2747 tid 100184 td 0xfffffe00e8450a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6564840 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6564860 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65648a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6564900 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e6564910 _sleep() at _sleep+0x2c3/frame 0xfffffe00e65649b0 pipe_read() at pipe_read+0x518/frame 0xfffffe00e6564a10 dofileread() at dofileread+0x81/frame 0xfffffe00e6564a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e6564ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6564bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6564bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80048d0da, rsp = 0x7fffffffe2c8, rbp = 0x7fffffffe2e0 --- Tracing command sh pid 2746 tid 100204 td 0xfffffe00dffdb100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02046e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0204700 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0204740 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f02047a0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f02047b0 _sleep() at _sleep+0x2c3/frame 0xfffffe00f0204850 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00f02048e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe00f0204ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0204bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0204bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80041a25a, rsp = 0x7fffffffe1e8, rbp = 0x7fffffffe2a0 --- Tracing command sh pid 2696 tid 100158 td 0xfffffe00dff0f500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64a66e0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e64a66e8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sh pid 2463 tid 100225 td 0xfffffe00dff11100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02746e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0274700 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0274740 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f02747a0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f02747b0 _sleep() at _sleep+0x2c3/frame 0xfffffe00f0274850 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00f02748e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe00f0274ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0274bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0274bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80041a25a, rsp = 0x7fffffffe0c8, rbp = 0x7fffffffe180 --- Tracing command sh pid 2461 tid 100226 td 0xfffffe00efe23a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f026f6e0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00f026f6e8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 2456 tid 100209 td 0xfffffe00efe21e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f021d700 mi_switch() at mi_switch+0x155/frame 0xfffffe00f021d720 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f021d760 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f021d7c0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f021d7d0 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00f021d830 seltdwait() at seltdwait+0x8d/frame 0xfffffe00f021d870 kern_select() at kern_select+0x8f0/frame 0xfffffe00f021da50 sys_pselect() at sys_pselect+0x127/frame 0xfffffe00f021dad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f021dbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f021dbf0 --- syscall (522, FreeBSD ELF64, sys_pselect), rip = 0x80054337a, rsp = 0x7fffffffd588, rbp = 0x7fffffffd650 --- Tracing command csh pid 2454 tid 100153 td 0xfffffe00dffd8e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e648d8d0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e648d8d8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command su pid 2453 tid 100188 td 0xfffffe00ee41e000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e65786e0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00e65786e8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bash pid 2446 tid 100215 td 0xfffffe00efe24100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02476e0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00f02476e8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sshd pid 2445 tid 100167 td 0xfffffe00dffd8700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64d3740 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64d3760 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64d37a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64d3800 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e64d3810 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64d3870 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e64d38b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e64d3a90 sys_select() at sys_select+0x56/frame 0xfffffe00e64d3ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64d3bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64d3bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8008f10ba, rsp = 0x7fffffffa4f8, rbp = 0x7fffffffa530 --- Tracing command sshd pid 2442 tid 100208 td 0xfffffe00efe45000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02187b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f02187d0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0218810 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0218870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f0218880 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00f02188e0 seltdwait() at seltdwait+0x8d/frame 0xfffffe00f0218920 kern_poll() at kern_poll+0x4b7/frame 0xfffffe00f0218ab0 sys_poll() at sys_poll+0x50/frame 0xfffffe00f0218ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0218bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0218bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80089481a, rsp = 0x7fffffffe558, rbp = 0x7fffffffe590 --- Tracing command csh pid 2428 tid 100151 td 0xfffffe00dfef7000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64837c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64837e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6483820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6483880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e6483890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64838f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e6483920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e6483980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e64839b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e6483a10 dofileread() at dofileread+0x81/frame 0xfffffe00e6483a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e6483ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6483bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6483bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80051f0da, rsp = 0x7fffffffe198, rbp = 0x7fffffffe1c0 --- Tracing command login pid 2425 tid 100223 td 0xfffffe00dff11800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f026a6e0 KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06daeb0 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06daec0 trap() at trap+0x6c/frame 0xfffffe00f06dafd0 calltrap() at calltrap+0x8/frame 0xfffffe00f06dafd0 --- trap 0xc, rip = 0xffffffff80ff4d60, rsp = 0xfffffe00f06db0a0, rbp = 0xfffffe00f06db130 --- db_read_bytes() at db_read_bytes+0x90/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x32/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- *** error reading from address fffffe00f026a6e8 *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00f06db120 kdb_reenter() at kdb_reenter+0x2c/frame 0xfffffe00f06db130 db_get_value() at db_get_value+0x4c/frame 0xfffffe00f06db170 db_backtrace() at db_backtrace+0x23e/frame 0xfffffe00f06db200 db_trace_thread() at db_trace_thread+0x3b/frame 0xfffffe00f06db220 _db_stack_trace_all() at _db_stack_trace_all+0xb6/frame 0xfffffe00f06db2c0 db_command() at db_command+0x25c/frame 0xfffffe00f06db390 db_script_exec() at db_script_exec+0x1a8/frame 0xfffffe00f06db430 db_command() at db_command+0x25c/frame 0xfffffe00f06db500 db_command_loop() at db_command_loop+0x4d/frame 0xfffffe00f06db510 db_trap() at db_trap+0xe8/frame 0xfffffe00f06db5a0 kdb_trap() at kdb_trap+0x154/frame 0xfffffe00f06db610 trap() at trap+0x7b8/frame 0xfffffe00f06db720 calltrap() at calltrap+0x8/frame 0xfffffe00f06db720 --- trap 0x3, rip = 0xffffffff80c222e7, rsp = 0xfffffe00f06db7f0, rbp = 0xfffffe00f06db800 --- kdb_enter() at kdb_enter+0x37/frame 0xfffffe00f06db800 vpanic() at vpanic+0x19e/frame 0xfffffe00f06db850 panic() at panic+0x43/frame 0xfffffe00f06db8b0 __mtx_assert() at __mtx_assert+0xc2/frame 0xfffffe00f06db8c0 uipc_detach() at uipc_detach+0x3ad/frame 0xfffffe00f06db990 sofree() at sofree+0x2f2/frame 0xfffffe00f06db9e0 sendfile_iodone() at sendfile_iodone+0x1fe/frame 0xfffffe00f06dba20 vnode_pager_generic_getpages_done_async() at vnode_pager_generic_getpages_done_async+0x3a/frame 0xfffffe00f06dba40 bufdone() at bufdone+0x6c/frame 0xfffffe00f06dbab0 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb00 g_io_deliver() at g_io_deliver+0x25e/frame 0xfffffe00f06dbb50 md_kthread() at md_kthread+0x259/frame 0xfffffe00f06dbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f06dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f06dbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command getty pid 2424 tid 100222 td 0xfffffe00dffc2000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02657c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f02657e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0265820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0265880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f0265890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00f02658f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00f0265920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00f0265980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00f02659b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00f0265a10 dofileread() at dofileread+0x81/frame 0xfffffe00f0265a60 sys_read() at sys_read+0xbc/frame 0xfffffe00f0265ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0265bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0265bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2423 tid 100221 td 0xfffffe00dffc2700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f02607c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f02607e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0260820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0260880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f0260890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00f02608f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00f0260920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00f0260980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00f02609b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00f0260a10 dofileread() at dofileread+0x81/frame 0xfffffe00f0260a60 sys_read() at sys_read+0xbc/frame 0xfffffe00f0260ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0260bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0260bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2422 tid 100220 td 0xfffffe00dff01500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64b07c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64b07e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64b0820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64b0880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e64b0890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64b08f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e64b0920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e64b0980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e64b09b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e64b0a10 dofileread() at dofileread+0x81/frame 0xfffffe00e64b0a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e64b0ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64b0bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64b0bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2421 tid 100157 td 0xfffffe00dffc2e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64a17c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64a17e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64a1820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64a1880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e64a1890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64a18f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e64a1920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e64a1980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e64a19b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e64a1a10 dofileread() at dofileread+0x81/frame 0xfffffe00e64a1a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e64a1ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64a1bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64a1bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2420 tid 100166 td 0xfffffe00dff04e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64ce7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64ce7e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64ce820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64ce880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e64ce890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64ce8f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e64ce920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e64ce980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e64ce9b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e64cea10 dofileread() at dofileread+0x81/frame 0xfffffe00e64cea60 sys_read() at sys_read+0xbc/frame 0xfffffe00e64cead0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64cebf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64cebf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2419 tid 100195 td 0xfffffe00efd5fc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e65a77c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e65a77e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65a7820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e65a7880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e65a7890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e65a78f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e65a7920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e65a7980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e65a79b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e65a7a10 dofileread() at dofileread+0x81/frame 0xfffffe00e65a7a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e65a7ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e65a7bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e65a7bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2418 tid 100152 td 0xfffffe00dfd5fc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64887c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64887e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6488820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6488880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e6488890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64888f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e6488920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e6488980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e64889b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e6488a10 dofileread() at dofileread+0x81/frame 0xfffffe00e6488a60 sys_read() at sys_read+0xbc/frame 0xfffffe00e6488ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6488bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6488bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2417 tid 100213 td 0xfffffe00efe24800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f023d7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00f023d7e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f023d820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f023d880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00f023d890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00f023d8f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00f023d920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00f023d980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00f023d9b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00f023da10 dofileread() at dofileread+0x81/frame 0xfffffe00f023da60 sys_read() at sys_read+0xbc/frame 0xfffffe00f023dad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f023dbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f023dbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command getty pid 2416 tid 100143 td 0xfffffe00dff06300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e645b7c0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e645b7e0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e645b820 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e645b880 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e645b890 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e645b8f0 tty_wait() at tty_wait+0x42/frame 0xfffffe00e645b920 ttydisc_read() at ttydisc_read+0x20a/frame 0xfffffe00e645b980 ttydev_read() at ttydev_read+0x4b/frame 0xfffffe00e645b9b0 devfs_read_f() at devfs_read_f+0xde/frame 0xfffffe00e645ba10 dofileread() at dofileread+0x81/frame 0xfffffe00e645ba60 sys_read() at sys_read+0xbc/frame 0xfffffe00e645bad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e645bbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e645bbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8004460da, rsp = 0x7fffffffeb78, rbp = 0x7fffffffebc0 --- Tracing command cron pid 2367 tid 100170 td 0xfffffe00dffdb800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64ee870 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64ee890 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64ee8d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64ee930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e64ee970 _sleep() at _sleep+0x258/frame 0xfffffe00e64eea10 kern_clock_nanosleep() at kern_clock_nanosleep+0x1e2/frame 0xfffffe00e64eea90 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe00e64eead0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64eebf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64eebf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8003eb3ba, rsp = 0x7fffffffea58, rbp = 0x7fffffffec40 --- Tracing command sshd pid 2363 tid 100201 td 0xfffffe00efe22c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e65f5740 mi_switch() at mi_switch+0x155/frame 0xfffffe00e65f5760 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65f57a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e65f5800 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e65f5810 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e65f5870 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e65f58b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e65f5a90 sys_select() at sys_select+0x56/frame 0xfffffe00e65f5ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e65f5bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e65f5bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8008f10ba, rsp = 0x7fffffffe5e8, rbp = 0x7fffffffe620 --- Tracing command powerd pid 2335 tid 100198 td 0xfffffe00dfff1e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e65e6700 mi_switch() at mi_switch+0x155/frame 0xfffffe00e65e6720 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65e6760 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e65e67c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e65e6800 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e65e6870 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e65e68b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e65e6a90 sys_select() at sys_select+0x56/frame 0xfffffe00e65e6ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e65e6bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e65e6bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8004430ba, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec50 --- Tracing command ntpd pid 2332 tid 100193 td 0xfffffe00efd60300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6591740 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6591760 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65917a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6591800 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e6591810 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e6591870 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e65918b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e6591a90 sys_select() at sys_select+0x56/frame 0xfffffe00e6591ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6591bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6591bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8008200ba, rsp = 0x7fffffffde78, rbp = 0x7fffffffdeb0 --- Tracing command ntpd pid 2332 tid 101108 td 0xfffffe00f0dc7100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe0037762820 mi_switch() at mi_switch+0x155/frame 0xfffffe0037762840 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe0037762880 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00377628e0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00377628f0 _sleep() at _sleep+0x2c3/frame 0xfffffe0037762990 umtxq_sleep() at umtxq_sleep+0x19e/frame 0xfffffe00377629f0 do_sem2_wait() at do_sem2_wait+0x38f/frame 0xfffffe0037762a80 __umtx_op_sem2_wait() at __umtx_op_sem2_wait+0x76/frame 0xfffffe0037762ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe0037762bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0037762bf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80070a2ea, rsp = 0x7fffdff3aee8, rbp = 0x7fffdff3af60 --- Tracing command rpc.lockd pid 2267 tid 100210 td 0xfffffe00e844e700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f0222640 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0222660 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f02226a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0222700 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00f0222740 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00f02227b0 svc_run_internal() at svc_run_internal+0xafe/frame 0xfffffe00f02228f0 svc_run() at svc_run+0x253/frame 0xfffffe00f0222950 sys_nlm_syscall() at sys_nlm_syscall+0x74d/frame 0xfffffe00f0222ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0222bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0222bf0 --- syscall (154, FreeBSD ELF64, sys_nlm_syscall), rip = 0x8002fb4ba, rsp = 0x7fffffffeb78, rbp = 0x7fffffffec40 --- Tracing command rpc.statd pid 2264 tid 100185 td 0xfffffe00e8450300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6569700 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6569720 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6569760 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e65697c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e6569800 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e6569870 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e65698b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e6569a90 sys_select() at sys_select+0x56/frame 0xfffffe00e6569ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6569bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6569bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8004390ba, rsp = 0x7fffffffea18, rbp = 0x7fffffffeb60 --- Tracing command watchdogd pid 2218 tid 100207 td 0xfffffe00efe45700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f0213870 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0213890 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f02138d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00f0213930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00f0213970 _sleep() at _sleep+0x258/frame 0xfffffe00f0213a10 kern_clock_nanosleep() at kern_clock_nanosleep+0x1e2/frame 0xfffffe00f0213a90 sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe00f0213ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00f0213bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00f0213bf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8003d43ba, rsp = 0x7fffffffea18, rbp = 0x7fffffffea60 --- Tracing command rpcbind pid 2069 tid 100156 td 0xfffffe00dffde000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e649c770 mi_switch() at mi_switch+0x155/frame 0xfffffe00e649c790 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e649c7d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e649c830 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e649c870 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e649c8e0 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e649c920 kern_poll() at kern_poll+0x4b7/frame 0xfffffe00e649cab0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e649cad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e649cbf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e649cbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8003e181a, rsp = 0x7fffffffc9b8, rbp = 0x7fffffffebd0 --- Tracing command syslogd pid 2059 tid 100194 td 0xfffffe00dfff3300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e65a27a0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e65a27c0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65a2800 _sleep() at _sleep+0x2cd/frame 0xfffffe00e65a28a0 bufwait() at bufwait+0xc4/frame 0xfffffe00e65a28e0 bufwrite() at bufwrite+0x25a/frame 0xfffffe00e65a2930 ffs_update() at ffs_update+0x2c2/frame 0xfffffe00e65a2990 ffs_syncvnode() at ffs_syncvnode+0x44a/frame 0xfffffe00e65a2a10 ffs_fsync() at ffs_fsync+0x1f/frame 0xfffffe00e65a2a40 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x59/frame 0xfffffe00e65a2a60 kern_fsync() at kern_fsync+0x178/frame 0xfffffe00e65a2ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e65a2bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e65a2bf0 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x80044a12a, rsp = 0x7fffffffe6c8, rbp = 0x7fffffffec40 --- Tracing command devd pid 1988 tid 100150 td 0xfffffe00dfd60300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e647e700 mi_switch() at mi_switch+0x155/frame 0xfffffe00e647e720 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e647e760 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e647e7c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e647e800 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e647e870 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e647e8b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e647ea90 sys_select() at sys_select+0x56/frame 0xfffffe00e647ead0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e647ebf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e647ebf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x302c0a, rsp = 0x7fffffffcac8, rbp = 0x7fffffffec60 --- Tracing command dhclient pid 1983 tid 100187 td 0xfffffe00ee41e700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6573770 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6573790 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e65737d0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6573830 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12/frame 0xfffffe00e6573870 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x184/frame 0xfffffe00e65738e0 seltdwait() at seltdwait+0x6b/frame 0xfffffe00e6573920 kern_poll() at kern_poll+0x4b7/frame 0xfffffe00e6573ab0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e6573ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6573bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6573bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80040681a, rsp = 0x7fffffffeba8, rbp = 0x7fffffffebe0 --- Tracing command dhclient pid 1924 tid 100169 td 0xfffffe00dff00e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64e97b0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64e97d0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64e9810 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e64e9870 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e64e9880 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e64e98e0 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e64e9920 kern_poll() at kern_poll+0x4b7/frame 0xfffffe00e64e9ab0 sys_poll() at sys_poll+0x50/frame 0xfffffe00e64e9ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e64e9bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e64e9bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80040681a, rsp = 0x7fffffffebb8, rbp = 0x7fffffffebe0 --- Tracing command dhclient pid 1921 tid 100155 td 0xfffffe00dff05c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6497740 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6497760 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e64977a0 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00e6497800 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00e6497810 _cv_wait_sig() at _cv_wait_sig+0x161/frame 0xfffffe00e6497870 seltdwait() at seltdwait+0x8d/frame 0xfffffe00e64978b0 kern_select() at kern_select+0x8f0/frame 0xfffffe00e6497a90 sys_select() at sys_select+0x56/frame 0xfffffe00e6497ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00e6497bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00e6497bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8004630ba, rsp = 0x7fffffffe968, rbp = 0x7fffffffea20 --- Tracing command syncer pid 33 tid 100142 td 0xfffffe00dfd60a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6456a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6456a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6456a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e6456ab0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x184/frame 0xfffffe00e6456b20 sched_sync() at sched_sync+0x68b/frame 0xfffffe00e6456bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e6456bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e6456bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vnlru pid 32 tid 100141 td 0xfffffe00dfd61100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e64519d0 mi_switch() at mi_switch+0x155/frame 0xfffffe00e64519f0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6451a30 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e6451a70 _sleep() at _sleep+0x26c/frame 0xfffffe00e6451b10 vnlru_proc() at vnlru_proc+0xaac/frame 0xfffffe00e6451bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e6451bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e6451bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100131 td 0xfffffe00dfd61800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de89ea20 mi_switch() at mi_switch+0x155/frame 0xfffffe00de89ea40 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de89ea80 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de89eac0 _sleep() at _sleep+0x26c/frame 0xfffffe00de89eb60 buf_daemon() at buf_daemon+0x112/frame 0xfffffe00de89ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de89ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de89ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100132 td 0xfffffe00dfefa100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0544a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0544a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0544a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0544ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0544b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0544bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0544bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0544bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100133 td 0xfffffe00dfef9a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0769a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0769a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0769a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0769ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0769b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0769bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0769bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0769bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100134 td 0xfffffe00dfef9300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e090ea10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e090ea30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e090ea70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e090eab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e090eb50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e090ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e090ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e090ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100135 td 0xfffffe00dfef8c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0a33a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0a33a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0a33a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0a33ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0a33b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0a33bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0a33bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0a33bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100136 td 0xfffffe00dfef8500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0bf8a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0bf8a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0bf8a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0bf8ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0bf8b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0bf8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0bf8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0bf8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100137 td 0xfffffe00dfef7e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0d1da10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0d1da30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0d1da70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0d1dab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0d1db50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0d1dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0d1dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0d1dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100138 td 0xfffffe00dfef7700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e0e42a10 mi_switch() at mi_switch+0x155/frame 0xfffffe00e0e42a30 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e0e42a70 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e0e42ab0 _sleep() at _sleep+0x26c/frame 0xfffffe00e0e42b50 bufspace_daemon() at bufspace_daemon+0x2d9/frame 0xfffffe00e0e42bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e0e42bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e0e42bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100154 td 0xfffffe00dffde700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e6492a40 mi_switch() at mi_switch+0x155/frame 0xfffffe00e6492a60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e6492aa0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e6492ae0 _sleep() at _sleep+0x26c/frame 0xfffffe00e6492b80 softdep_flush() at softdep_flush+0x16e/frame 0xfffffe00e6492bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e6492bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e6492bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 100212 td 0xfffffe00dff07800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00f0238a40 mi_switch() at mi_switch+0x155/frame 0xfffffe00f0238a60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00f0238aa0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00f0238ae0 _sleep() at _sleep+0x26c/frame 0xfffffe00f0238b80 softdep_flush() at softdep_flush+0x16e/frame 0xfffffe00f0238bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00f0238bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f0238bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command bufdaemon pid 31 tid 101184 td 0xfffffe00f0d8aa00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00377cea40 mi_switch() at mi_switch+0x155/frame 0xfffffe00377cea60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00377ceaa0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00377ceae0 _sleep() at _sleep+0x26c/frame 0xfffffe00377ceb80 softdep_flush() at softdep_flush+0x16e/frame 0xfffffe00377cebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00377cebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00377cebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command vmdaemon pid 30 tid 100130 td 0xfffffe00de7dce00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de899a40 mi_switch() at mi_switch+0x155/frame 0xfffffe00de899a60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de899aa0 _sleep() at _sleep+0x2cd/frame 0xfffffe00de899b40 vm_daemon() at vm_daemon+0x82/frame 0xfffffe00de899bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de899bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de899bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100129 td 0xfffffe00de7dd500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de894770 mi_switch() at mi_switch+0x155/frame 0xfffffe00de894790 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de8947d0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de894810 _sleep() at _sleep+0x26c/frame 0xfffffe00de8948b0 vm_pageout_worker() at vm_pageout_worker+0x28c/frame 0xfffffe00de894b70 vm_pageout() at vm_pageout+0x21e/frame 0xfffffe00de894bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de894bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de894bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100139 td 0xfffffe00dff02a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e1c27830 mi_switch() at mi_switch+0x155/frame 0xfffffe00e1c27850 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e1c27890 _sleep() at _sleep+0x2cd/frame 0xfffffe00e1c27930 vm_pageout_laundry_worker() at vm_pageout_laundry_worker+0xc0e/frame 0xfffffe00e1c27bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e1c27bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e1c27bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pagedaemon pid 29 tid 100140 td 0xfffffe00dff02300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00e1c4ca40 mi_switch() at mi_switch+0x155/frame 0xfffffe00e1c4ca60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00e1c4caa0 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00e1c4cae0 _sleep() at _sleep+0x26c/frame 0xfffffe00e1c4cb80 uma_reclaim_worker() at uma_reclaim_worker+0xb8/frame 0xfffffe00e1c4cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00e1c4cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e1c4cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon1 pid 28 tid 100128 td 0xfffffe003dbfd100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de88fa40 mi_switch() at mi_switch+0x155/frame 0xfffffe00de88fa60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de88faa0 _sleep() at _sleep+0x2cd/frame 0xfffffe00de88fb40 enc_daemon() at enc_daemon+0xe4/frame 0xfffffe00de88fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de88fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de88fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command enc_daemon0 pid 27 tid 100127 td 0xfffffe003dbfd800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de88aa40 mi_switch() at mi_switch+0x155/frame 0xfffffe00de88aa60 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de88aaa0 _sleep() at _sleep+0x2cd/frame 0xfffffe00de88ab40 enc_daemon() at enc_daemon+0xe4/frame 0xfffffe00de88abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de88abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de88abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command rand_harvestq pid 26 tid 100124 td 0xfffffe00de7dea00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de87b9f0 mi_switch() at mi_switch+0x155/frame 0xfffffe00de87ba10 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de87ba50 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de87ba90 _sleep() at _sleep+0x26c/frame 0xfffffe00de87bb30 random_kthread() at random_kthread+0x67/frame 0xfffffe00de87bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de87bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de87bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod4 pid 25 tid 100120 td 0xfffffe00dc203100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de85ba30 mi_switch() at mi_switch+0x155/frame 0xfffffe00de85ba50 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de85ba90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de85bad0 _sleep() at _sleep+0x26c/frame 0xfffffe00de85bb70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00de85bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de85bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de85bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod3 pid 24 tid 100119 td 0xfffffe00dc203800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de856a30 mi_switch() at mi_switch+0x155/frame 0xfffffe00de856a50 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de856a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de856ad0 _sleep() at _sleep+0x26c/frame 0xfffffe00de856b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00de856bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de856bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de856bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod2 pid 23 tid 100118 td 0xfffffe00de7dc000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de851a30 mi_switch() at mi_switch+0x155/frame 0xfffffe00de851a50 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de851a90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de851ad0 _sleep() at _sleep+0x26c/frame 0xfffffe00de851b70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00de851bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de851bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de851bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command soaiod1 pid 22 tid 100117 td 0xfffffe00de7dc700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de84ca30 mi_switch() at mi_switch+0x155/frame 0xfffffe00de84ca50 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de84ca90 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xfffffe00de84cad0 _sleep() at _sleep+0x26c/frame 0xfffffe00de84cb70 soaio_kproc_loop() at soaio_kproc_loop+0x168/frame 0xfffffe00de84cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de84cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de84cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100098 td 0xfffffe00d69d6700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb81ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb81ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb81b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb81b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb81bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb81bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb81bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100099 td 0xfffffe00d69d6000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb86ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb86ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb86b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb86b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb86bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb86bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb86bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100100 td 0xfffffe003dbf1800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb8bab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb8bad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb8bb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb8bb70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb8bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb8bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb8bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100101 td 0xfffffe003dbf1100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb90ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb90ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb90b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb90b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb90bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb90bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb90bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100102 td 0xfffffe003dbf0a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb95ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb95ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb95b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb95b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb95bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb95bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb95bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100104 td 0xfffffe00db22de00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb9fab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb9fad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb9fb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adb9fb70 usb_process() at usb_process+0xc3/frame 0xfffffe00adb9fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb9fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb9fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100105 td 0xfffffe00db22d700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adba4ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adba4ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adba4b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adba4b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adba4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adba4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adba4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100106 td 0xfffffe00db22d000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adba9ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adba9ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adba9b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adba9b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adba9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adba9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adba9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100107 td 0xfffffe00d69d9800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbaeab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbaead0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbaeb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbaeb70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbaebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbaebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbaebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100108 td 0xfffffe00d69d9100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbb3ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbb3ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbb3b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbb3b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbb3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbb3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbb3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100109 td 0xfffffe00dc200e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbc4ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbc4ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbc4b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbc4b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbc4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbc4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbc4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100110 td 0xfffffe00dc200700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbc9ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbc9ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbc9b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbc9b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbc9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbc9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbc9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100111 td 0xfffffe00dc200000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbceab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbcead0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbceb10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbceb70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbcebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbcebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbcebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100112 td 0xfffffe00db230800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbd3ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbd3ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbd3b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbd3b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbd3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbd3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbd3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command usb pid 21 tid 100113 td 0xfffffe00db230100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbd8ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbd8ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbd8b10 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adbd8b70 usb_process() at usb_process+0xc3/frame 0xfffffe00adbd8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbd8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbd8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100092 td 0xfffffe003dbeee00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb63a60 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb63a80 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb63ac0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb63b60 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00adb63bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb63bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb63bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100093 td 0xfffffe003dbee700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb68a60 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb68a80 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb68ac0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb68b60 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00adb68bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb68bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb68bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100094 td 0xfffffe003dbee000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb6da60 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb6da80 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb6dac0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb6db60 xpt_done_td() at xpt_done_td+0x7b/frame 0xfffffe00adb6dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb6dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb6dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command cam pid 20 tid 100126 td 0xfffffe00de7ddc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de885a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00de885a90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de885ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe00de885b70 xpt_scanner_thread() at xpt_scanner_thread+0x68/frame 0xfffffe00de885bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de885bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de885bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 11 pid 19 tid 100091 td 0xfffffe003dbef500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb5ea50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb5ea70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb5eab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb5eb50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb5ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb5ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb5ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 10 pid 18 tid 100090 td 0xfffffe003dbefc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb59a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb59a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb59ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb59b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb59bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb59bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb59bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 9 pid 17 tid 100089 td 0xfffffe003dbf0300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb54a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb54a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb54ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb54b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb54bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb54bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb54bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 8 pid 16 tid 100088 td 0xfffffe003dc15800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb4fa50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb4fa70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb4fab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb4fb50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb4fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb4fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb4fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 7 pid 15 tid 100087 td 0xfffffe003dbfa000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb4aa50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb4aa70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb4aab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb4ab50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb4abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb4abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb4abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 6 pid 9 tid 100086 td 0xfffffe003dbfa700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb45a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb45a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb45ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb45b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb45bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb45bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb45bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 5 pid 8 tid 100085 td 0xfffffe003dbfae00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb40a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb40a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb40ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb40b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb40bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb40bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb40bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 4 pid 7 tid 100084 td 0xfffffe003dbfb500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb3ba50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb3ba70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb3bab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb3bb50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb3bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb3bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb3bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 3 pid 6 tid 100083 td 0xfffffe003dbfbc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb36a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb36a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb36ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb36b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb36bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb36bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb36bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 2 pid 5 tid 100082 td 0xfffffe003dbfc300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb31a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb31a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb31ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb31b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb31bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb31bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb31bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 1 pid 4 tid 100081 td 0xfffffe003dbfca00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb2ca50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb2ca70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb2cab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb2cb50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb2cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb2cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb2cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto returns 0 pid 3 tid 100080 td 0xfffffe003dc12000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb27a50 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb27a70 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb27ab0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb27b50 crypto_ret_proc() at crypto_ret_proc+0x21f/frame 0xfffffe00adb27bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb27bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb27bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command crypto pid 2 tid 100079 td 0xfffffe003dc12700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb22a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb22a90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb22ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb22b70 crypto_proc() at crypto_proc+0x25f/frame 0xfffffe00adb22bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb22bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb22bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command sequencer 00 pid 14 tid 100066 td 0xfffffe003dc39100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adae1a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00adae1a90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adae1ad0 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00adae1b30 seq_eventthread() at seq_eventthread+0x91/frame 0xfffffe00adae1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adae1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adae1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100063 td 0xfffffe003dc43500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adac6a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00adac6a90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adac6ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adac6b70 g_run_events() at g_run_events+0x64/frame 0xfffffe00adac6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adac6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adac6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100064 td 0xfffffe003dc42e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adacba70 mi_switch() at mi_switch+0x155/frame 0xfffffe00adacba90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adacbad0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adacbb70 g_io_schedule_up() at g_io_schedule_up+0x3c/frame 0xfffffe00adacbba0 g_up_procbody() at g_up_procbody+0x5c/frame 0xfffffe00adacbbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adacbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adacbbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command geom pid 13 tid 100065 td 0xfffffe003dc39800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adad0a70 mi_switch() at mi_switch+0x155/frame 0xfffffe00adad0a90 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adad0ad0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adad0b70 g_io_schedule_down() at g_io_schedule_down+0x4c/frame 0xfffffe00adad0ba0 g_down_procbody() at g_down_procbody+0x5c/frame 0xfffffe00adad0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adad0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adad0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100041 td 0xfffffe00ad870e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada04b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada04b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00ada04bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada04bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada04bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100042 td 0xfffffe00ad870700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada09b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada09b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00ada09bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada09bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada09bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100045 td 0xfffffe00ad86d100 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100048 td 0xfffffe00ad86bc00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada57b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada57b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00ada57bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada57bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada57bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100049 td 0xfffffe003dc42700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100050 td 0xfffffe003dc42000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xfffffe00ad873800 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xfffffe00ad873100 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100053 td 0xfffffe00ad872a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xfffffe00ad872300 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100055 td 0xfffffe00ad871c00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100056 td 0xfffffe00ad871500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100057 td 0xfffffe003dc36000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100058 td 0xfffffe003dc45800 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100059 td 0xfffffe003dc45100 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100060 td 0xfffffe003dc44a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada93b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada93b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00ada93bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada93bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada93bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100061 td 0xfffffe003dc44300 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100095 td 0xfffffe00d69d7c00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100096 td 0xfffffe00d69d7500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb77b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb77b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00adb77bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb77bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb77bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100097 td 0xfffffe00d69d6e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb7cb00 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb7cb20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00adb7cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb7cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb7cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100103 td 0xfffffe00db22e500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb9ab00 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb9ab20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00adb9abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb9abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb9abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100114 td 0xfffffe00db22fa00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100115 td 0xfffffe00db22f300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbe2b00 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbe2b20 ithread_loop() at ithread_loop+0x99/frame 0xfffffe00adbe2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbe2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbe2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command intr pid 12 tid 100176 td 0xfffffe00dff00700 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100177 td 0xfffffe00dff00000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100178 td 0xfffffe00dfefa800 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100179 td 0xfffffe00ee420a00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100180 td 0xfffffe00ee420300 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100181 td 0xfffffe00ee41fc00 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100182 td 0xfffffe00ee41f500 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100183 td 0xfffffe00ee41ee00 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xfffffe00ad7d9300 (CPU 0) cpustop_handler() at cpustop_handler+0x26/frame 0xffffffff81de70a0 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xffffffff81de70b0 trap() at trap+0x56/frame 0xffffffff81de71c0 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81de71c0 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad546a50, rbp = 0xfffffe00ad546a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad546a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad546aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad546ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad546ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad546bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad546bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad546bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00ad7d8c00 (CPU 1) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbd0e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbd0e10 trap() at trap+0x56/frame 0xfffffe003dbd0f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbd0f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad54ba50, rbp = 0xfffffe00ad54ba50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad54ba50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad54baa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad54bac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad54bae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad54bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad54bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad54bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe00ad7d8500 (CPU 2) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbdce00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbdce10 trap() at trap+0x56/frame 0xfffffe003dbdcf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbdcf20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad550a50, rbp = 0xfffffe00ad550a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad550a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad550aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad550ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad550ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad550bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad550bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad550bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xfffffe00ad7d7e00 (CPU 3) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbe8e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbe8e10 trap() at trap+0x56/frame 0xfffffe003dbe8f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbe8f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad555a50, rbp = 0xfffffe00ad555a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad555a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad555aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad555ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad555ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad555bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad555bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad555bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xfffffe00ad7d7700 (CPU 4) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dbf4e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dbf4e10 trap() at trap+0x56/frame 0xfffffe003dbf4f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dbf4f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad55aa50, rbp = 0xfffffe00ad55aa50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad55aa50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad55aaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad55aac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad55aae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad55abb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad55abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad55abf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xfffffe00ad7d7000 (CPU 5) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc00e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc00e10 trap() at trap+0x56/frame 0xfffffe003dc00f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc00f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad55fa50, rbp = 0xfffffe00ad55fa50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad55fa50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad55faa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad55fac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad55fae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad55fbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad55fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad55fbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xfffffe00ad7e1a00 (CPU 6) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc0ce00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc0ce10 trap() at trap+0x56/frame 0xfffffe003dc0cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc0cf20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad564a50, rbp = 0xfffffe00ad564a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad564a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad564aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad564ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad564ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad564bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad564bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad564bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xfffffe00ad7e1300 (CPU 7) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc18e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc18e10 trap() at trap+0x56/frame 0xfffffe003dc18f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc18f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad569a50, rbp = 0xfffffe00ad569a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad569a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad569aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad569ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad569ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad569bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad569bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad569bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xfffffe00ad7e0c00 (CPU 8) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc24e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc24e10 trap() at trap+0x56/frame 0xfffffe003dc24f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc24f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad56ea50, rbp = 0xfffffe00ad56ea50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad56ea50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad56eaa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad56eac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad56eae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad56ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad56ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad56ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xfffffe00ad7e0500 (CPU 9) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc30e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc30e10 trap() at trap+0x56/frame 0xfffffe003dc30f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc30f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad573a50, rbp = 0xfffffe00ad573a50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad573a50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad573aa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad573ac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad573ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad573bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad573bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad573bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xfffffe00ad7dfe00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad578ac0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad578ae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad578bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad578bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad578bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xfffffe00ad7df700 (CPU 11) cpustop_handler() at cpustop_handler+0x26/frame 0xfffffe003dc48e00 ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe003dc48e10 trap() at trap+0x56/frame 0xfffffe003dc48f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe003dc48f20 --- trap 0x13, rip = 0xffffffff80fe1596, rsp = 0xfffffe00ad57da50, rbp = 0xfffffe00ad57da50 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe00ad57da50 acpi_cpu_idle() at acpi_cpu_idle+0x232/frame 0xfffffe00ad57daa0 cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe00ad57dac0 cpu_idle() at cpu_idle+0x9f/frame 0xfffffe00ad57dae0 sched_idletd() at sched_idletd+0x3d1/frame 0xfffffe00ad57dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad57dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad57dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00ad7d9a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5416e0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad541700 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad541740 sleepq_catch_signals() at sleepq_catch_signals+0x3f2/frame 0xfffffe00ad5417a0 sleepq_wait_sig() at sleepq_wait_sig+0x9/frame 0xfffffe00ad5417b0 _sleep() at _sleep+0x2c3/frame 0xfffffe00ad541850 kern_wait6() at kern_wait6+0x482/frame 0xfffffe00ad5418e0 sys_wait4() at sys_wait4+0x7b/frame 0xfffffe00ad541ad0 amd64_syscall() at amd64_syscall+0x159/frame 0xfffffe00ad541bf0 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00ad541bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x2bda3a, rsp = 0x7fffffffe728, rbp = 0x7fffffffe790 --- Tracing command audit pid 10 tid 100001 td 0xfffffe00ad7da100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad53ca90 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad53cab0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad53caf0 _cv_wait() at _cv_wait+0x15a/frame 0xfffffe00ad53cb50 audit_worker() at audit_worker+0x9f/frame 0xfffffe00ad53cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad53cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad53cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81c8c820 sched_switch() at sched_switch+0x5b2/frame 0xffffffff82121ad0 mi_switch() at mi_switch+0x155/frame 0xffffffff82121af0 sleepq_switch() at sleepq_switch+0x11a/frame 0xffffffff82121b30 sleepq_timedwait() at sleepq_timedwait+0x4b/frame 0xffffffff82121b70 _sleep() at _sleep+0x26c/frame 0xffffffff82121c10 swapper() at swapper+0x75/frame 0xffffffff82121c60 mi_startup() at mi_startup+0x2ed/frame 0xffffffff82121cb0 btext() at btext+0x2c Tracing command kernel pid 0 tid 100015 td 0xfffffe00ad7df000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad582ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad582ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad582b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad582b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad582bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad582bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad582bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe00ad7da800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad587ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad587ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad587b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad587b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad587bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad587bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad587bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xfffffe00ad862300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad58cab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad58cad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad58cb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad58cb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad58cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad58cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad58cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe00ad861c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad591ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad591ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad591b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad591b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad591bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad591bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad591bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffffe00ad861500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad596ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad596ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad596b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad596b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad596bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad596bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad596bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe00ad860e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad59bab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad59bad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad59bb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad59bb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad59bbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad59bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad59bbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xfffffe00ad860700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5a0ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5a0ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5a0b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5a0b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5a0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5a0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5a0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffffe00ad860000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5a5ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5a5ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5a5b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5a5b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5a5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5a5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5a5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffffe00ad7e2800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5aaab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5aaad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5aab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5aab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5aabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5aabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5aabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00ad7e2100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5afab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5afad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5afb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5afb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5afbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5afbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5afbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00ad865c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5b4ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5b4ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5b4b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5b4b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5b4bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5b4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5b4bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00ad865500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5b9ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5b9ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5b9b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5b9b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5b9bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5b9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5b9bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xfffffe00ad864e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5beab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5bead0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5beb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5beb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5bebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5bebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5bebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xfffffe00ad864700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5c3ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5c3ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5c3b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5c3b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5c3bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5c3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5c3bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xfffffe00ad864000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5c8ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5c8ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5c8b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5c8b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5c8bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5c8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5c8bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xfffffe00ad863800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5cdab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5cdad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5cdb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5cdb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5cdbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5cdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5cdbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xfffffe00ad863100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5d2ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5d2ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5d2b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5d2b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5d2bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5d2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5d2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100032 td 0xfffffe00ad862a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5d7ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5d7ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5d7b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5d7b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5d7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5d7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5d7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xfffffe00ad86b500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5dcab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5dcad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5dcb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5dcb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5dcbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5dcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5dcbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100034 td 0xfffffe00ad86ae00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5e1ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5e1ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5e1b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5e1b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5e1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5e1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5e1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100035 td 0xfffffe00ad86a700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5e6ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5e6ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5e6b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5e6b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5e6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5e6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5e6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xfffffe00ad86a000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5ebab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5ebad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5ebb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5ebb80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5ebbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5ebbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5ebbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100037 td 0xfffffe00ad867800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5f0ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5f0ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5f0b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5f0b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5f0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5f0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5f0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100038 td 0xfffffe00ad867100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5f5ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5f5ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5f5b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5f5b80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5f5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5f5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5f5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100039 td 0xfffffe00ad866a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5faab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5faad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5fab10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00ad5fab80 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xef/frame 0xfffffe00ad5fabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5fabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5fabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xfffffe00ad866300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ad5ffa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00ad5ffaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ad5ffae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00ad5ffb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00ad5ffbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ad5ffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ad5ffbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100043 td 0xfffffe00ad870000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada0ea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada0eaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ada0eae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00ada0eb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00ada0ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada0ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada0ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xfffffe00ad86d800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada13a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada13aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ada13ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00ada13b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00ada13bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada13bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada13bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xfffffe00ad86ca00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada1da80 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada1daa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ada1dae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00ada1db80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00ada1dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada1dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada1dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100047 td 0xfffffe00ad86c300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00ada22a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00ada22aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00ada22ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00ada22b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00ada22bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00ada22bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ada22bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100062 td 0xfffffe003dc43c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adac1a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adac1aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adac1ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adac1b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adac1bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adac1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adac1bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xfffffe003dc38a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adae6a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adae6aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adae6ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adae6b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adae6bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adae6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adae6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffffe003dc38300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adaeba80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adaebaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adaebae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adaebb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adaebbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adaebbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adaebbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100069 td 0xfffffe003dc37c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adaf0a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adaf0aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adaf0ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adaf0b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adaf0bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adaf0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adaf0bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100070 td 0xfffffe003dc37500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adaf5a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adaf5aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adaf5ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adaf5b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adaf5bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adaf5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adaf5bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100071 td 0xfffffe003dc36e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adafaa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adafaaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adafaae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adafab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adafabb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adafabf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adafabf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xfffffe003dc36700 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adaffa80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adaffaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adaffae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adaffb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adaffbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adaffbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adaffbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100073 td 0xfffffe003dc15100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb04a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb04aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb04ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb04b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb04bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb04bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb04bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100074 td 0xfffffe003dc14a00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb09a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb09aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb09ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb09b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb09bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb09bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb09bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100075 td 0xfffffe003dc14300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb0ea80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb0eaa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb0eae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb0eb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb0ebb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb0ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb0ebf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100076 td 0xfffffe003dc13c00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb13a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb13aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb13ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb13b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb13bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb13bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb13bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100077 td 0xfffffe003dc13500 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb18a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb18aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb18ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb18b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb18bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb18bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb18bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100078 td 0xfffffe003dc12e00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adb1da80 mi_switch() at mi_switch+0x155/frame 0xfffffe00adb1daa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adb1dae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00adb1db80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00adb1dbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adb1dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adb1dbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100116 td 0xfffffe00db22ec00 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00adbe7ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00adbe7ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00adbe7b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00adbe7b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xef/frame 0xfffffe00adbe7bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00adbe7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00adbe7bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100121 td 0xfffffe00dfd5e000 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de86cab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00de86cad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de86cb10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00de86cb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xef/frame 0xfffffe00de86cbb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de86cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de86cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100122 td 0xfffffe00de7df800 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de871ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00de871ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de871b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00de871b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xef/frame 0xfffffe00de871bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de871bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de871bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100123 td 0xfffffe00de7df100 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de876ab0 mi_switch() at mi_switch+0x155/frame 0xfffffe00de876ad0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de876b10 msleep_spin_sbt() at msleep_spin_sbt+0x152/frame 0xfffffe00de876b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xef/frame 0xfffffe00de876bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de876bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de876bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command kernel pid 0 tid 100125 td 0xfffffe00de7de300 sched_switch() at sched_switch+0x5b2/frame 0xfffffe00de880a80 mi_switch() at mi_switch+0x155/frame 0xfffffe00de880aa0 sleepq_switch() at sleepq_switch+0x11a/frame 0xfffffe00de880ae0 _sleep() at _sleep+0x2cd/frame 0xfffffe00de880b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7/frame 0xfffffe00de880bb0 fork_exit() at fork_exit+0x80/frame 0xfffffe00de880bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00de880bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=0x0) at ../../../kern/kern_shutdown.c:394 #2 0xffffffff8049e61a in db_dump (dummy=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:575 #3 0xffffffff8049e3dc in db_command (last_cmdp=, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:482 #4 0xffffffff804a33e8 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #5 0xffffffff8049e3dc in db_command (last_cmdp=, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:482 #6 0xffffffff804a33e8 in db_script_exec (scriptname=, warnifnotfound=) at ../../../ddb/db_script.c:304 #7 0xffffffff8049e3dc in db_command (last_cmdp=, cmd_table=, dopager=0x1) at ../../../ddb/db_command.c:482 #8 0xffffffff8049e14d in db_command_loop () at ../../../ddb/db_command.c:535 #9 0xffffffff804a13f8 in db_trap (type=, code=) at ../../../ddb/db_main.c:270 #10 0xffffffff80c22b54 in kdb_trap (type=0x3, code=0x0, tf=) at ../../../kern/subr_kdb.c:699 #11 0xffffffff81023eb8 in trap (frame=0xfffffe00f06db730) at ../../../amd64/amd64/trap.c:576 #12 #13 kdb_enter (why=0xffffffff811f0d15 "panic", msg=) at ../../../kern/subr_kdb.c:486 #14 0xffffffff80bd78be in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:902 #15 0xffffffff80bd7653 in panic (fmt=0xffffffff81c8dc48 "'4\033\201\377\377\377\377") at ../../../kern/kern_shutdown.c:839 #16 0xffffffff80bb4f32 in __mtx_assert (c=, what=, file=0xfffffe00f06db6f0 "", line=0x80) at ../../../kern/kern_mutex.c:1099 #17 0xffffffff80c8404d in uipc_detach (so=0xfffff8006051e000) at ../../../kern/uipc_usrreq.c:804 #18 0xffffffff80c76cc2 in sofree (so=0xfffff8006051e000) at ../../../kern/uipc_socket.c:1142 #19 0xffffffff80bd5cee in sendfile_iodone (arg=0xfffff800455aed40, pa=, count=, error=) at ../../../kern/kern_sendfile.c:403 #20 0xffffffff80ef060a in vnode_pager_generic_getpages_done_async (bp=0xfffff8004552b800) at ../../../vm/vnode_pager.c:1110 #21 0xffffffff80c95b2c in bufdone (bp=0xfffff8004552b800) at ../../../kern/vfs_bio.c:4435 #22 0xffffffff80b18bde in g_io_deliver (bp=0xfffff8028c30d7f8, error=0x0) at ../../../geom/geom_io.c:695 #23 0xffffffff80b18bde in g_io_deliver (bp=0xfffff8028c30d660, error=0x0) at ../../../geom/geom_io.c:695 #24 0xffffffff806ecc69 in md_kthread (arg=0xfffff804cadf6000) at ../../../dev/md/md.c:1248 #25 0xffffffff80b932d0 in fork_exit (callout=0xffffffff806eca10 , arg=0xfffff804cadf6000, frame=0xfffffe00f06dbc00) at ../../../kern/kern_fork.c:1052 #26 (kgdb) f 17 #17 0xffffffff80c8404d in uipc_detach (so=0xfffff8006051e000) at ../../../kern/uipc_usrreq.c:804 804 UNP_PCB_UNLOCK_ASSERT(unp); (kgdb) l 799 LIST_REMOVE(unp, unp_dead); 800 unp->unp_gencnt = ++unp_gencnt; 801 --unp_count; 802 UNP_LINK_WUNLOCK(); 803 804 UNP_PCB_UNLOCK_ASSERT(unp); 805 restart: 806 if ((vp = unp->unp_vnode) != NULL) { 807 vplock = mtx_pool_find(mtxpool_sleep, vp); 808 mtx_lock(vplock); (kgdb) info loc unp = 0xfffff8031a0eb600 unp2 = 0xffffffff80c81770 vplock = 0x0 vp = 0x0 local_unp_rights = 0xfffff800 (kgdb) p *so $1 = {so_lock = {lock_object = {lo_name = 0xffffffff8111f003 "socket", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xfffff8087fd82b00}, mtx_lock = 0x0}, so_count = 0x0, so_rdsel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff80c75b70 , kl_unlock = 0xffffffff80c75bb0 , kl_assert_locked = 0xffffffff80c75bf0 , kl_assert_unlocked = 0xffffffff80c75c40 , kl_lockarg = 0xfffff8006051e000, kl_autodestroy = 0x0}, si_mtx = 0x0}, so_wrsel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff80c75c80 , kl_unlock = 0xffffffff80c75cc0 , kl_assert_locked = 0xffffffff80c75d00 , kl_assert_unlocked = 0xffffffff80c75d50 , kl_lockarg = 0xfffff8006051e000, kl_autodestroy = 0x0}, si_mtx = 0x0}, so_type = 0x1, so_options = 0x0, so_linger = 0x0, so_state = 0x2001, so_pcb = 0xfffff8031a0eb600, so_vnet = 0xfffff800037b6040, so_proto = 0xffffffff8191eee0 , so_timeo = 0x0, so_error = 0x0, so_sigio = 0x0, so_cred = 0xfffff80003f77800, so_label = 0x0, so_gencnt = 0x66a094, so_emuldata = 0x0, so_dtor = 0x0, osd = {osd_nslots = 0x0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, so_fibnum = 0x0, so_user_cookie = 0x0, so_ts_clock = 0x0, so_max_pacing_rate = 0x0, {{so_rcv = {sb_mtx = {lock_object = {lo_name = 0xffffffff81193166 "so_rcv", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd74800}, mtx_lock = 0x0}, sb_sx = {lock_object = { lo_name = 0xffffffff811ebe6d "so_rcv_sx", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xfffff8087fd82c00}, sx_lock = 0x1}, sb_sel = 0xfffff8006051e028, sb_state = 0x20, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0x0, sb_acc = 0x0, sb_ccc = 0x0, sb_hiwat = 0x0, sb_mbcnt = 0x0, sb_mcnt = 0x0, sb_ccnt = 0x0, sb_mbmax = 0x0, sb_ctl = 0x0, sb_tlscc = 0x0, sb_tlsdcc = 0x0, sb_lowat = 0x1, sb_timeo = 0x0, sb_tls_seqno = 0x0, sb_tls_info = 0x0, sb_mtls = 0x0, sb_mtlstail = 0x0, sb_flags = 0x0, sb_upcall = 0x0, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff8006051e240}, sb_aiotask = {ta_link = {stqe_next = 0x0}, ta_pending = 0x0, ta_priority = 0x0, ta_flags = 0x0, ta_func = 0xffffffff80c539c0 , ta_context = 0xfffff8006051e000}}, so_snd = { sb_mtx = {lock_object = {lo_name = 0xffffffff811a8597 "so_snd", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xfffff8087fd74780}, mtx_lock = 0x0}, sb_sx = {lock_object = {lo_name = 0xffffffff81218333 "so_snd_sx", lo_flags = 0x2330000, lo_data = 0x0, lo_witness = 0xfffff8087fd82b80}, sx_lock = 0x1}, sb_sel = 0xfffff8006051e078, sb_state = 0x10, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0x0, sb_acc = 0x0, sb_ccc = 0x0, sb_hiwat = 0x2000, sb_mbcnt = 0x0, sb_mcnt = 0x0, sb_ccnt = 0x0, sb_mbmax = 0x10000, sb_ctl = 0x0, sb_tlscc = 0x0, sb_tlsdcc = 0x0, sb_lowat = 0x1000, sb_timeo = 0x0, sb_tls_seqno = 0x0, sb_tls_info = 0x0, sb_mtls = 0x0, sb_mtlstail = 0x0, sb_flags = 0x0, sb_upcall = 0x0, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff8006051e358}, sb_aiotask = { ta_link = {stqe_next = 0x0}, ta_pending = 0x0, ta_priority = 0x0, ta_flags = 0x0, ta_func = 0xffffffff80c541d0 , ta_context = 0xfffff8006051e000}}, so_list = {tqe_next = 0x0, tqe_prev = 0x0}, so_listen = 0x0, so_qstate = SQ_NONE, so_peerlabel = 0x0, so_oobmark = 0x0, so_ktls_rx_list = {stqe_next = 0x0}}, {sol_incomp = {tqh_first = 0xffffffff81193166, tqh_last = 0x1030000}, sol_comp = {tqh_first = 0xfffff8087fd74800, tqh_last = 0x0}, sol_qlen = 0x811ebe6d, sol_incqlen = 0xffffffff, sol_qlimit = 0x2330000, sol_accept_filter = 0xfffff8087fd82c00, sol_accept_filter_arg = 0x1, sol_accept_filter_str = 0xfffff8006051e028 "", sol_upcall = 0x20, sol_upcallarg = 0x0, sol_sbrcv_lowat = 0x0, sol_sbsnd_lowat = 0x0, sol_sbrcv_hiwat = 0x0, sol_sbsnd_hiwat = 0x0, sol_sbrcv_flags = 0x0, sol_sbsnd_flags = 0x0, sol_sbrcv_timeo = 0x0, sol_sbsnd_timeo = 0x0, sol_lastover = {tv_sec = 0x0, tv_usec = 0x0}, sol_overcount = 0x0}}} (kgdb) info loc unp = 0xfffff8031a0eb600 unp2 = 0xffffffff80c81770 vplock = 0x0 vp = 0x0 local_unp_rights = 0xfffff800 (kgdb) p *unp $2 = {unp_mtx = {lock_object = {lo_name = 0xffffffff8110f568 "unp", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0xfffff8087fd74f80}, mtx_lock = 0xfffffe00dff06a00}, unp_conn = 0x0, unp_refcount = 0x1, unp_flags = 0x1, unp_gcflag = 0x2, unp_addr = 0x0, unp_socket = 0xfffff8006051e000, unp_vnode = 0x0, unp_peercred = {cr_version = 0x0, cr_uid = 0x0, cr_ngroups = 0x3, cr_groups = {0x0, 0x0, 0x5, 0x0 }, {_cr_unused1 = 0xc27, cr_pid = 0xc27}}, unp_reflink = {le_next = 0x0, le_prev = 0x0}, unp_link = {le_next = 0xffffffffffffffff, le_prev = 0xffffffffffffffff}, unp_refs = {lh_first = 0x0}, unp_gencnt = 0x60ba4e, unp_file = 0x0, unp_msgcount = 0x0, unp_gcrefs = 0x0, unp_ino = 0x305d30, unp_dead = {le_next = 0x0, le_prev = 0x0}} (kgdb) Index: src/sys/kern/uipc_usrreq.c =================================================================== --- src/sys/kern/uipc_usrreq.c (revision 365247) +++ src/sys/kern/uipc_usrreq.c (working copy) @@ -66,12 +66,12 @@ __FBSDID("$FreeBSD$"); #include #include #include -#include /* XXX must be before */ #include #include #include #include #include +#include #include #include #include @@ -106,9 +106,7 @@ __FBSDID("$FreeBSD$"); MALLOC_DECLARE(M_FILECAPS); /* - * Locking key: - * (l) Locked using list lock - * (g) Locked using linkage lock + * See unpcb.h for the locking key. */ static uma_zone_t unp_zone; @@ -196,41 +194,32 @@ SYSCTL_INT(_net_local, OID_AUTO, deferred, CTLFLAG /* * Locking and synchronization: * - * Three types of locks exist in the local domain socket implementation: a - * a global linkage rwlock, the mtxpool lock, and per-unpcb mutexes. - * The linkage lock protects the socket count, global generation number, - * and stream/datagram global lists. + * Several types of locks exist in the lock domain socket implementation: + * - a global linkage lock + * - a global connection list lock + * - the mtxpool lock + * - per-unpcb mutexes * - * The mtxpool lock protects the vnode from being modified while referenced. - * Lock ordering requires that it be acquired before any unpcb locks. + * The linkage lock protects the global socket lists, the generation number + * counter and garbage collector state. * - * The unpcb lock (unp_mtx) protects all fields in the unpcb. Of particular - * note is that this includes the unp_conn field. So long as the unpcb lock - * is held the reference to the unpcb pointed to by unp_conn is valid. If we - * require that the unpcb pointed to by unp_conn remain live in cases where - * we need to drop the unp_mtx as when we need to acquire the lock for a - * second unpcb the caller must first acquire an additional reference on the - * second unpcb and then revalidate any state (typically check that unp_conn - * is non-NULL) upon requiring the initial unpcb lock. The lock ordering - * between unpcbs is the conventional ascending address order. Two helper - * routines exist for this: + * The connection list lock protects the list of referring sockets in a datagram + * socket PCB. This lock is also overloaded to protect a global list of + * sockets whose buffers contain socket references in the form of SCM_RIGHTS + * messages. To avoid recursion, such references are released by a dedicated + * thread. * - * - unp_pcb_lock2(unp, unp2) - which just acquires the two locks in the - * safe ordering. + * The mtxpool lock protects the vnode from being modified while referenced. + * Lock ordering rules require that it be acquired before any PCB locks. * - * - unp_pcb_owned_lock2(unp, unp2, freed) - the lock for unp is held - * when called. If unp is unlocked and unp2 is subsequently freed - * freed will be set to 1. + * The unpcb lock (unp_mtx) protects the most commonly referenced fields in the + * unpcb. This includes the unp_conn field, which either links two connected + * PCBs together (for connected socket types) or points at the destination + * socket (for connectionless socket types). The operations of creating or + * destroying a connection therefore involve locking multiple PCBs. To avoid + * lock order reversals, in some cases this involves dropping a PCB lock and + * using a reference counter to maintain liveness. * - * The helper routines for references are: - * - * - unp_pcb_hold(unp): Can be called any time we currently hold a valid - * reference to unp. - * - * - unp_pcb_rele(unp): The caller must hold the unp lock. If we are - * releasing the last reference, detach must have been called thus - * unp->unp_socket be NULL. - * * UNIX domain sockets each have an unpcb hung off of their so_pcb pointer, * allocated in pru_attach() and freed in pru_detach(). The validity of that * pointer is an invariant, so no lock is required to dereference the so_pcb @@ -324,35 +313,46 @@ static void unp_process_defers(void * __unused, in static void unp_pcb_hold(struct unpcb *unp) { - MPASS(unp->unp_refcount); - refcount_acquire(&unp->unp_refcount); + u_int old __unused; + + old = refcount_acquire(&unp->unp_refcount); + KASSERT(old > 0, ("%s: unpcb %p has no references", __func__, unp)); } -static int +static __result_use_check bool unp_pcb_rele(struct unpcb *unp) { - int freed; + bool ret; UNP_PCB_LOCK_ASSERT(unp); - MPASS(unp->unp_refcount); - if ((freed = refcount_release(&unp->unp_refcount))) { - /* we got here with having detached? */ - MPASS(unp->unp_socket == NULL); + + if ((ret = refcount_release(&unp->unp_refcount))) { UNP_PCB_UNLOCK(unp); UNP_PCB_LOCK_DESTROY(unp); uma_zfree(unp_zone, unp); } - return (freed); + return (ret); } static void -unp_pcb_lock2(struct unpcb *unp, struct unpcb *unp2) +unp_pcb_rele_notlast(struct unpcb *unp) { - MPASS(unp != unp2); + bool ret __unused; + + ret = refcount_release(&unp->unp_refcount); + KASSERT(!ret, ("%s: unpcb %p has no references", __func__, unp)); +} + +static void +unp_pcb_lock_pair(struct unpcb *unp, struct unpcb *unp2) +{ UNP_PCB_UNLOCK_ASSERT(unp); UNP_PCB_UNLOCK_ASSERT(unp2); - if ((uintptr_t)unp2 > (uintptr_t)unp) { + + if (unp == unp2) { UNP_PCB_LOCK(unp); + } else if ((uintptr_t)unp2 > (uintptr_t)unp) { + UNP_PCB_LOCK(unp); UNP_PCB_LOCK(unp2); } else { UNP_PCB_LOCK(unp2); @@ -360,35 +360,56 @@ static void } } -static __noinline void -unp_pcb_owned_lock2_slowpath(struct unpcb *unp, struct unpcb **unp2p, - int *freed) +/* + * Try to lock the connected peer of an already locked socket. In some cases + * this requires that we unlock the current socket. The UNP_CONNECTING flag is + * used to block concurrent connection attempts while the lock is dropped. The + * caller must be careful to revalidate PCB state. + */ +static struct unpcb * +unp_pcb_lock_peer(struct unpcb *unp) { struct unpcb *unp2; - unp2 = *unp2p; + UNP_PCB_LOCK_ASSERT(unp); + unp2 = unp->unp_conn; + if (__predict_false(unp2 == NULL)) + return (NULL); + if (__predict_false(unp == unp2)) + return (unp); + + KASSERT((unp->unp_flags & UNP_CONNECTING) == 0, + ("%s: socket %p is connecting", __func__, unp)); + UNP_PCB_UNLOCK_ASSERT(unp2); + + if (__predict_true(UNP_PCB_TRYLOCK(unp2))) + return (unp2); + if ((uintptr_t)unp2 > (uintptr_t)unp) { + UNP_PCB_LOCK(unp2); + return (unp2); + } + unp->unp_flags |= UNP_CONNECTING; unp_pcb_hold(unp2); UNP_PCB_UNLOCK(unp); + UNP_PCB_LOCK(unp2); UNP_PCB_LOCK(unp); - *freed = unp_pcb_rele(unp2); - if (*freed) - *unp2p = NULL; + KASSERT(unp->unp_conn == unp2 || unp->unp_conn == NULL, + ("%s: socket %p was reconnected", __func__, unp)); + KASSERT((unp->unp_flags & UNP_CONNECTING) != 0, + ("%s: socket %p has UNP_CONNECTING clear", __func__, unp)); + unp->unp_flags &= ~UNP_CONNECTING; + if (unp_pcb_rele(unp2)) { + /* unp2 is unlocked. */ + return (NULL); + } + if (unp->unp_conn == NULL) { + UNP_PCB_UNLOCK(unp2); + return (NULL); + } + return (unp2); } -#define unp_pcb_owned_lock2(unp, unp2, freed) do { \ - freed = 0; \ - UNP_PCB_LOCK_ASSERT(unp); \ - UNP_PCB_UNLOCK_ASSERT(unp2); \ - MPASS((unp) != (unp2)); \ - if (__predict_true(UNP_PCB_TRYLOCK(unp2))) \ - break; \ - else if ((uintptr_t)(unp2) > (uintptr_t)(unp)) \ - UNP_PCB_LOCK(unp2); \ - else \ - unp_pcb_owned_lock2_slowpath((unp), &(unp2), &freed); \ -} while (0) - /* * Definitions of protocols supported in the LOCAL domain. */ @@ -470,18 +491,16 @@ uipc_accept(struct socket *so, struct sockaddr **n KASSERT(unp != NULL, ("uipc_accept: unp == NULL")); *nam = malloc(sizeof(struct sockaddr_un), M_SONAME, M_WAITOK); - UNP_LINK_RLOCK(); - unp2 = unp->unp_conn; - if (unp2 != NULL && unp2->unp_addr != NULL) { - UNP_PCB_LOCK(unp2); - sa = (struct sockaddr *) unp2->unp_addr; - bcopy(sa, *nam, sa->sa_len); + UNP_PCB_LOCK(unp); + unp2 = unp_pcb_lock_peer(unp); + if (unp2 != NULL && unp2->unp_addr != NULL) + sa = (struct sockaddr *)unp2->unp_addr; + else + sa = &sun_noname; + bcopy(sa, *nam, sa->sa_len); + if (unp2 != NULL) UNP_PCB_UNLOCK(unp2); - } else { - sa = &sun_noname; - bcopy(sa, *nam, sa->sa_len); - } - UNP_LINK_RUNLOCK(); + UNP_PCB_UNLOCK(unp); return (0); } @@ -525,7 +544,7 @@ uipc_attach(struct socket *so, int proto, struct t UNP_PCB_LOCK_INIT(unp); unp->unp_socket = so; so->so_pcb = unp; - unp->unp_refcount = 1; + refcount_init(&unp->unp_refcount, 1); if ((locked = UNP_LINK_WOWNED()) == false) UNP_LINK_WLOCK(); @@ -702,7 +721,7 @@ uipc_close(struct socket *so) struct unpcb *unp, *unp2; struct vnode *vp = NULL; struct mtx *vplock; - int freed; + unp = sotounpcb(so); KASSERT(unp != NULL, ("uipc_close: unp == NULL")); @@ -720,18 +739,9 @@ uipc_close(struct socket *so) VOP_UNP_DETACH(vp); unp->unp_vnode = NULL; } - unp2 = unp->unp_conn; - unp_pcb_hold(unp); - if (__predict_false(unp == unp2)) { + if ((unp2 = unp_pcb_lock_peer(unp)) != NULL) unp_disconnect(unp, unp2); - } else if (unp2 != NULL) { - unp_pcb_hold(unp2); - unp_pcb_owned_lock2(unp, unp2, freed); - unp_disconnect(unp, unp2); - if (unp_pcb_rele(unp2) == 0) - UNP_PCB_UNLOCK(unp2); - } - if (unp_pcb_rele(unp) == 0) + else UNP_PCB_UNLOCK(unp); if (vp) { mtx_unlock(vplock); @@ -749,10 +759,7 @@ uipc_connect2(struct socket *so1, struct socket *s KASSERT(unp != NULL, ("uipc_connect2: unp == NULL")); unp2 = so2->so_pcb; KASSERT(unp2 != NULL, ("uipc_connect2: unp2 == NULL")); - if (unp != unp2) - unp_pcb_lock2(unp, unp2); - else - UNP_PCB_LOCK(unp); + unp_pcb_lock_pair(unp, unp2); error = unp_connect2(so1, so2, PRU_CONNECT2); if (unp != unp2) UNP_PCB_UNLOCK(unp2); @@ -766,7 +773,7 @@ uipc_detach(struct socket *so) struct unpcb *unp, *unp2; struct mtx *vplock; struct vnode *vp; - int freeunp, local_unp_rights; + int local_unp_rights; unp = sotounpcb(so); KASSERT(unp != NULL, ("uipc_detach: unp == NULL")); @@ -773,7 +780,6 @@ uipc_detach(struct socket *so) vp = NULL; vplock = NULL; - local_unp_rights = 0; SOCK_LOCK(so); if (!SOLISTENING(so)) { @@ -812,24 +818,11 @@ uipc_detach(struct socket *so) VOP_UNP_DETACH(vp); unp->unp_vnode = NULL; } - if (__predict_false(unp == unp->unp_conn)) { - unp_disconnect(unp, unp); - unp2 = NULL; - } else { - if ((unp2 = unp->unp_conn) != NULL) { - unp_pcb_owned_lock2(unp, unp2, freeunp); - if (freeunp) - unp2 = NULL; - } - unp_pcb_hold(unp); - if (unp2 != NULL) { - unp_pcb_hold(unp2); - unp_disconnect(unp, unp2); - if (unp_pcb_rele(unp2) == 0) - UNP_PCB_UNLOCK(unp2); - } - } - UNP_PCB_UNLOCK(unp); + if ((unp2 = unp_pcb_lock_peer(unp)) != NULL) + unp_disconnect(unp, unp2); + else + UNP_PCB_UNLOCK(unp); + UNP_REF_LIST_LOCK(); while (!LIST_EMPTY(&unp->unp_refs)) { struct unpcb *ref = LIST_FIRST(&unp->unp_refs); @@ -842,11 +835,9 @@ uipc_detach(struct socket *so) unp_drop(ref); UNP_REF_LIST_LOCK(); } + UNP_REF_LIST_UNLOCK(); - UNP_REF_LIST_UNLOCK(); UNP_PCB_LOCK(unp); - freeunp = unp_pcb_rele(unp); - MPASS(freeunp == 0); local_unp_rights = unp_rights; unp->unp_socket->so_pcb = NULL; unp->unp_socket = NULL; @@ -866,30 +857,15 @@ static int uipc_disconnect(struct socket *so) { struct unpcb *unp, *unp2; - int freed; unp = sotounpcb(so); KASSERT(unp != NULL, ("uipc_disconnect: unp == NULL")); UNP_PCB_LOCK(unp); - if ((unp2 = unp->unp_conn) == NULL) { + if ((unp2 = unp_pcb_lock_peer(unp)) != NULL) + unp_disconnect(unp, unp2); + else UNP_PCB_UNLOCK(unp); - return (0); - } - if (__predict_true(unp != unp2)) { - unp_pcb_owned_lock2(unp, unp2, freed); - if (__predict_false(freed)) { - UNP_PCB_UNLOCK(unp); - return (0); - } - unp_pcb_hold(unp2); - } - unp_pcb_hold(unp); - unp_disconnect(unp, unp2); - if (unp_pcb_rele(unp) == 0) - UNP_PCB_UNLOCK(unp); - if ((unp != unp2) && unp_pcb_rele(unp2) == 0) - UNP_PCB_UNLOCK(unp2); return (0); } @@ -1008,27 +984,6 @@ uipc_rcvd(struct socket *so, int flags) } static int -connect_internal(struct socket *so, struct sockaddr *nam, struct thread *td) -{ - int error; - struct unpcb *unp; - - unp = so->so_pcb; - if (unp->unp_conn != NULL) - return (EISCONN); - error = unp_connect(so, nam, td); - if (error) - return (error); - UNP_PCB_LOCK(unp); - if (unp->unp_conn == NULL) { - UNP_PCB_UNLOCK(unp); - if (error == 0) - error = ENOTCONN; - } - return (error); -} - -static int uipc_send(struct socket *so, int flags, struct mbuf *m, struct sockaddr *nam, struct mbuf *control, struct thread *td) { @@ -1058,57 +1013,26 @@ uipc_send(struct socket *so, int flags, struct mbu const struct sockaddr *from; if (nam != NULL) { - /* - * We return with UNP_PCB_LOCK_HELD so we know that - * the reference is live if the pointer is valid. - */ - if ((error = connect_internal(so, nam, td))) + error = unp_connect(so, nam, td); + if (error != 0) break; - MPASS(unp->unp_conn != NULL); - unp2 = unp->unp_conn; - } else { - UNP_PCB_LOCK(unp); + } + UNP_PCB_LOCK(unp); - /* - * Because connect() and send() are non-atomic in a sendto() - * with a target address, it's possible that the socket will - * have disconnected before the send() can run. In that case - * return the slightly counter-intuitive but otherwise - * correct error that the socket is not connected. - */ - if ((unp2 = unp->unp_conn) == NULL) { - UNP_PCB_UNLOCK(unp); - error = ENOTCONN; - break; - } - } - if (__predict_false(unp == unp2)) { - if (unp->unp_socket == NULL) { - error = ENOTCONN; - break; - } - goto connect_self; - } - unp_pcb_owned_lock2(unp, unp2, freed); - if (__predict_false(freed)) { - UNP_PCB_UNLOCK(unp); - error = ENOTCONN; - break; - } /* - * The socket referencing unp2 may have been closed - * or unp may have been disconnected if the unp lock - * was dropped to acquire unp2. + * Because connect() and send() are non-atomic in a sendto() + * with a target address, it's possible that the socket will + * have disconnected before the send() can run. In that case + * return the slightly counter-intuitive but otherwise + * correct error that the socket is not connected. */ - if (__predict_false(unp->unp_conn == NULL) || - unp2->unp_socket == NULL) { + unp2 = unp_pcb_lock_peer(unp); + if (unp2 == NULL) { UNP_PCB_UNLOCK(unp); - if (unp_pcb_rele(unp2) == 0) - UNP_PCB_UNLOCK(unp2); error = ENOTCONN; break; } - connect_self: + if (unp2->unp_flags & UNP_WANTCRED) control = unp_addsockcred(td, control); if (unp->unp_addr != NULL) @@ -1126,11 +1050,13 @@ uipc_send(struct socket *so, int flags, struct mbu SOCKBUF_UNLOCK(&so2->so_rcv); error = ENOBUFS; } - if (nam != NULL) + if (nam != NULL) { unp_disconnect(unp, unp2); - if (__predict_true(unp != unp2)) - UNP_PCB_UNLOCK(unp2); - UNP_PCB_UNLOCK(unp); + } else { + UNP_PCB_UNLOCK(unp); + if (unp != unp2) + UNP_PCB_UNLOCK(unp2); + } break; } @@ -1138,7 +1064,7 @@ uipc_send(struct socket *so, int flags, struct mbu case SOCK_STREAM: if ((so->so_state & SS_ISCONNECTED) == 0) { if (nam != NULL) { - error = connect_internal(so, nam, td); + error = unp_connect(so, nam, td); if (error != 0) break; } else { @@ -1145,29 +1071,20 @@ uipc_send(struct socket *so, int flags, struct mbu error = ENOTCONN; break; } - } else { - UNP_PCB_LOCK(unp); } - if ((unp2 = unp->unp_conn) == NULL) { + UNP_PCB_LOCK(unp); + if ((unp2 = unp_pcb_lock_peer(unp)) == NULL) { UNP_PCB_UNLOCK(unp); error = ENOTCONN; break; } else if (so->so_snd.sb_state & SBS_CANTSENDMORE) { UNP_PCB_UNLOCK(unp); + UNP_PCB_UNLOCK(unp2); error = EPIPE; break; - } else if ((unp2 = unp->unp_conn) == NULL) { - UNP_PCB_UNLOCK(unp); - error = ENOTCONN; - break; } - unp_pcb_owned_lock2(unp, unp2, freed); UNP_PCB_UNLOCK(unp); - if (__predict_false(freed)) { - error = ENOTCONN; - break; - } if ((so2 = unp2->unp_socket) == NULL) { UNP_PCB_UNLOCK(unp2); error = ENOTCONN; @@ -1302,30 +1219,18 @@ uipc_ready(struct socket *so, struct mbuf *m, int ("%s: unexpected socket type for %p", __func__, so)); UNP_PCB_LOCK(unp); - if ((unp2 = unp->unp_conn) == NULL) { + if ((unp2 = unp_pcb_lock_peer(unp)) != NULL) { UNP_PCB_UNLOCK(unp); - goto search; + so2 = unp2->unp_socket; + SOCKBUF_LOCK(&so2->so_rcv); + if ((error = sbready(&so2->so_rcv, m, count)) == 0) + sorwakeup_locked(so2); + else + SOCKBUF_UNLOCK(&so2->so_rcv); + UNP_PCB_UNLOCK(unp2); + return (error); } - if (unp != unp2) { - if (UNP_PCB_TRYLOCK(unp2) == 0) { - unp_pcb_hold(unp2); - UNP_PCB_UNLOCK(unp); - UNP_PCB_LOCK(unp2); - if (unp_pcb_rele(unp2)) - goto search; - } else - UNP_PCB_UNLOCK(unp); - } - so2 = unp2->unp_socket; - SOCKBUF_LOCK(&so2->so_rcv); - if ((error = sbready(&so2->so_rcv, m, count)) == 0) - sorwakeup_locked(so2); - else - SOCKBUF_UNLOCK(&so2->so_rcv); - UNP_PCB_UNLOCK(unp2); - return (error); -search: /* * The receiving socket has been disconnected, but may still be valid. * In this case, the now-ready mbufs are still present in its socket @@ -1568,7 +1473,8 @@ static int unp_connectat(int fd, struct socket *so, struct sockaddr *nam, struct thread *td) { - struct sockaddr_un *soun = (struct sockaddr_un *)nam; + struct mtx *vplock; + struct sockaddr_un *soun; struct vnode *vp; struct socket *so2; struct unpcb *unp, *unp2, *unp3; @@ -1576,8 +1482,8 @@ unp_connectat(int fd, struct socket *so, struct so char buf[SOCK_MAXADDRLEN]; struct sockaddr *sa; cap_rights_t rights; - int error, len, freed; - struct mtx *vplock; + int error, len; + bool connreq; if (nam->sa_family != AF_UNIX) return (EAFNOSUPPORT); @@ -1586,19 +1492,28 @@ unp_connectat(int fd, struct socket *so, struct so len = nam->sa_len - offsetof(struct sockaddr_un, sun_path); if (len <= 0) return (EINVAL); + soun = (struct sockaddr_un *)nam; bcopy(soun->sun_path, buf, len); buf[len] = 0; unp = sotounpcb(so); UNP_PCB_LOCK(unp); - if (unp->unp_flags & UNP_CONNECTING) { + if (unp->unp_conn != NULL) { UNP_PCB_UNLOCK(unp); + return (EISCONN); + } + if ((unp->unp_flags & UNP_CONNECTING) != 0) { + UNP_PCB_UNLOCK(unp); return (EALREADY); } unp->unp_flags |= UNP_CONNECTING; UNP_PCB_UNLOCK(unp); - sa = malloc(sizeof(struct sockaddr_un), M_SONAME, M_WAITOK); + connreq = (so->so_proto->pr_flags & PR_CONNREQUIRED) != 0; + if (connreq) + sa = malloc(sizeof(struct sockaddr_un), M_SONAME, M_WAITOK); + else + sa = NULL; NDINIT_ATRIGHTS(&nd, LOOKUP, FOLLOW | LOCKSHARED | LOCKLEAF, UIO_SYSSPACE, buf, fd, cap_rights_init(&rights, CAP_CONNECTAT), td); error = namei(&nd); @@ -1639,7 +1554,7 @@ unp_connectat(int fd, struct socket *so, struct so error = EPROTOTYPE; goto bad2; } - if (so->so_proto->pr_flags & PR_CONNREQUIRED) { + if (connreq) { if (so2->so_options & SO_ACCEPTCONN) { CURVNET_SET(so2->so_vnet); so2 = sonewconn(so2, 0); @@ -1651,7 +1566,7 @@ unp_connectat(int fd, struct socket *so, struct so goto bad2; } unp3 = sotounpcb(so2); - unp_pcb_lock2(unp2, unp3); + unp_pcb_lock_pair(unp2, unp3); if (unp2->unp_addr != NULL) { bcopy(unp2->unp_addr, sa, unp2->unp_addr->sun_len); unp3->unp_addr = (struct sockaddr_un *) sa; @@ -1662,21 +1577,18 @@ unp_connectat(int fd, struct socket *so, struct so UNP_PCB_UNLOCK(unp2); unp2 = unp3; - unp_pcb_owned_lock2(unp2, unp, freed); - if (__predict_false(freed)) { - UNP_PCB_UNLOCK(unp2); - error = ECONNREFUSED; - goto bad2; - } + + /* + * It is safe to block on the PCB lock here since unp2 is + * nascent and cannot be connected to any other sockets. + */ + UNP_PCB_LOCK(unp); #ifdef MAC mac_socketpeer_set_from_socket(so, so2); mac_socketpeer_set_from_socket(so2, so); #endif } else { - if (unp == unp2) - UNP_PCB_LOCK(unp); - else - unp_pcb_lock2(unp, unp2); + unp_pcb_lock_pair(unp, unp2); } KASSERT(unp2 != NULL && so2 != NULL && unp2->unp_socket == so2 && sotounpcb(so2) == unp2, @@ -1693,6 +1605,8 @@ bad: } free(sa, M_SONAME); UNP_PCB_LOCK(unp); + KASSERT((unp->unp_flags & UNP_CONNECTING) != 0, + ("%s: unp %p has UNP_CONNECTING clear", __func__, unp)); unp->unp_flags &= ~UNP_CONNECTING; UNP_PCB_UNLOCK(unp); return (error); @@ -1732,6 +1646,8 @@ unp_connect2(struct socket *so, struct socket *so2 UNP_PCB_LOCK_ASSERT(unp); UNP_PCB_LOCK_ASSERT(unp2); + KASSERT(unp->unp_conn == NULL, + ("%s: socket %p is already connected", __func__, unp)); if (so2->so_type != so->so_type) return (EPROTOTYPE); @@ -1748,6 +1664,8 @@ unp_connect2(struct socket *so, struct socket *so2 case SOCK_STREAM: case SOCK_SEQPACKET: + KASSERT(unp2->unp_conn == NULL, + ("%s: socket %p is already connected", __func__, unp2)); unp2->unp_conn = unp; if (req == PRU_CONNECT && ((unp->unp_flags | unp2->unp_flags) & UNP_CONNWAIT)) @@ -1767,17 +1685,15 @@ static void unp_disconnect(struct unpcb *unp, struct unpcb *unp2) { struct socket *so, *so2; - int freed __unused; +#ifdef INVARIANTS + struct unpcb *unptmp; +#endif - KASSERT(unp2 != NULL, ("unp_disconnect: unp2 == NULL")); - UNP_PCB_LOCK_ASSERT(unp); UNP_PCB_LOCK_ASSERT(unp2); + KASSERT(unp->unp_conn == unp2, + ("%s: unpcb %p is not connected to %p", __func__, unp, unp2)); - if (unp->unp_conn == NULL && unp2->unp_conn == NULL) - return; - - MPASS(unp->unp_conn == unp2); unp->unp_conn = NULL; so = unp->unp_socket; so2 = unp2->unp_socket; @@ -1784,6 +1700,14 @@ unp_disconnect(struct unpcb *unp, struct unpcb *un switch (unp->unp_socket->so_type) { case SOCK_DGRAM: UNP_REF_LIST_LOCK(); +#ifdef INVARIANTS + LIST_FOREACH(unptmp, &unp2->unp_refs, unp_reflink) { + if (unptmp == unp) + break; + } + KASSERT(unptmp != NULL, + ("%s: %p not found in reflist of %p", __func__, unp, unp2)); +#endif LIST_REMOVE(unp, unp_reflink); UNP_REF_LIST_UNLOCK(); if (so) { @@ -1803,10 +1727,17 @@ unp_disconnect(struct unpcb *unp, struct unpcb *un soisdisconnected(so2); break; } - freed = unp_pcb_rele(unp); - MPASS(freed == 0); - freed = unp_pcb_rele(unp2); - MPASS(freed == 0); + + if (unp == unp2) { + unp_pcb_rele_notlast(unp); + if (!unp_pcb_rele(unp)) + UNP_PCB_UNLOCK(unp); + } else { + if (!unp_pcb_rele(unp)) + UNP_PCB_UNLOCK(unp); + if (!unp_pcb_rele(unp2)) + UNP_PCB_UNLOCK(unp2); + } } /* @@ -1902,9 +1833,10 @@ unp_pcblist(SYSCTL_HANDLER_ARGS) for (i = 0; i < n; i++) { unp = unp_list[i]; UNP_PCB_LOCK(unp); - freeunp = unp_pcb_rele(unp); + if (unp_pcb_rele(unp)) + continue; - if (freeunp == 0 && unp->unp_gencnt <= gencnt) { + if (unp->unp_gencnt <= gencnt) { xu->xu_len = sizeof *xu; xu->xu_unpp = (uintptr_t)unp; /* @@ -1931,8 +1863,9 @@ unp_pcblist(SYSCTL_HANDLER_ARGS) sotoxsocket(unp->unp_socket, &xu->xu_socket); UNP_PCB_UNLOCK(unp); error = SYSCTL_OUT(req, xu, sizeof *xu); - } else if (freeunp == 0) + } else { UNP_PCB_UNLOCK(unp); + } } free(xu, M_TEMP); if (!error) { @@ -1987,7 +1920,6 @@ unp_drop(struct unpcb *unp) { struct socket *so = unp->unp_socket; struct unpcb *unp2; - int freed; /* * Regardless of whether the socket's peer dropped the connection @@ -1994,23 +1926,17 @@ unp_drop(struct unpcb *unp) * with this socket by aborting or disconnecting, POSIX requires * that ECONNRESET is returned. */ - /* acquire a reference so that unp isn't freed from underneath us */ UNP_PCB_LOCK(unp); if (so) so->so_error = ECONNRESET; - unp2 = unp->unp_conn; - if (unp2 == unp) { + if ((unp2 = unp_pcb_lock_peer(unp)) != NULL) { + /* Last reference dropped in unp_disconnect(). */ + unp_pcb_rele_notlast(unp); unp_disconnect(unp, unp2); - } else if (unp2 != NULL) { - unp_pcb_hold(unp2); - unp_pcb_owned_lock2(unp, unp2, freed); - unp_disconnect(unp, unp2); - if (unp_pcb_rele(unp2) == 0) - UNP_PCB_UNLOCK(unp2); + } else if (!unp_pcb_rele(unp)) { + UNP_PCB_UNLOCK(unp); } - if (unp_pcb_rele(unp) == 0) - UNP_PCB_UNLOCK(unp); } static void @@ -2148,18 +2074,44 @@ unp_zone_change(void *tag) uma_zone_set_max(unp_zone, maxsockets); } +#ifdef INVARIANTS static void +unp_zdtor(void *mem, int size __unused, void *arg __unused) +{ + struct unpcb *unp; + + unp = mem; + + KASSERT(LIST_EMPTY(&unp->unp_refs), + ("%s: unpcb %p has lingering refs", __func__, unp)); + KASSERT(unp->unp_socket == NULL, + ("%s: unpcb %p has socket backpointer", __func__, unp)); + KASSERT(unp->unp_vnode == NULL, + ("%s: unpcb %p has vnode references", __func__, unp)); + KASSERT(unp->unp_conn == NULL, + ("%s: unpcb %p is still connected", __func__, unp)); + KASSERT(unp->unp_addr == NULL, + ("%s: unpcb %p has leaked addr", __func__, unp)); +} +#endif + +static void unp_init(void) { + uma_dtor dtor; #ifdef VIMAGE if (!IS_DEFAULT_VNET(curvnet)) return; #endif - unp_zone = uma_zcreate("unpcb", sizeof(struct unpcb), NULL, NULL, + +#ifdef INVARIANTS + dtor = unp_zdtor; +#else + dtor = NULL; +#endif + unp_zone = uma_zcreate("unpcb", sizeof(struct unpcb), NULL, dtor, NULL, NULL, UMA_ALIGN_CACHE, 0); - if (unp_zone == NULL) - panic("unp_init"); uma_zone_set_max(unp_zone, maxsockets); uma_zone_set_warning(unp_zone, "kern.ipc.maxsockets limit reached"); EVENTHANDLER_REGISTER(maxsockets_change, unp_zone_change, Index: src/sys/sys/unpcb.h =================================================================== --- src/sys/sys/unpcb.h (revision 365247) +++ src/sys/sys/unpcb.h (working copy) @@ -65,30 +65,37 @@ typedef uint64_t unp_gen_t; * Stream sockets keep copies of receive sockbuf sb_cc and sb_mbcnt * so that changes in the sockbuf may be computed to modify * back pressure on the sender accordingly. + * + * Locking key: + * (a) Atomic + * (c) Constant + * (g) Locked using linkage lock + * (l) Locked using list lock + * (p) Locked using pcb lock */ LIST_HEAD(unp_head, unpcb); struct unpcb { /* Cache line 1 */ - struct mtx unp_mtx; /* mutex */ - struct unpcb *unp_conn; /* control block of connected socket */ - volatile u_int unp_refcount; - short unp_flags; /* flags */ - short unp_gcflag; /* Garbage collector flags. */ - struct sockaddr_un *unp_addr; /* bound address of socket */ - struct socket *unp_socket; /* pointer back to socket */ + struct mtx unp_mtx; /* PCB mutex */ + struct unpcb *unp_conn; /* (p) connected socket */ + volatile u_int unp_refcount; /* (a, p) atomic refcount */ + short unp_flags; /* (p) PCB flags */ + short unp_gcflag; /* (g) Garbage collector flags */ + struct sockaddr_un *unp_addr; /* (p) bound address of socket */ + struct socket *unp_socket; /* (c) pointer back to socket */ /* Cache line 2 */ - struct vnode *unp_vnode; /* if associated with file */ - struct xucred unp_peercred; /* peer credentials, if applicable */ - LIST_ENTRY(unpcb) unp_reflink; /* link in unp_refs list */ - LIST_ENTRY(unpcb) unp_link; /* glue on list of all PCBs */ - struct unp_head unp_refs; /* referencing socket linked list */ - unp_gen_t unp_gencnt; /* generation count of this instance */ - struct file *unp_file; /* back-pointer to file for gc. */ - u_int unp_msgcount; /* references from message queue */ - u_int unp_gcrefs; /* garbage collector refcount */ - ino_t unp_ino; /* fake inode number */ - LIST_ENTRY(unpcb) unp_dead; /* link in dead list */ + struct vnode *unp_vnode; /* (p) associated file if applicable */ + struct xucred unp_peercred; /* (p) peer credentials if applicable */ + LIST_ENTRY(unpcb) unp_reflink; /* (l) link in unp_refs list */ + LIST_ENTRY(unpcb) unp_link; /* (g) glue on list of all PCBs */ + struct unp_head unp_refs; /* (l) referencing socket linked list */ + unp_gen_t unp_gencnt; /* (g) generation count of this item */ + struct file *unp_file; /* (g) back-pointer to file for gc */ + u_int unp_msgcount; /* (g) references from message queue */ + u_int unp_gcrefs; /* (g) garbage collector refcount */ + ino_t unp_ino; /* (g) fake inode number */ + LIST_ENTRY(unpcb) unp_dead; /* (g) link in dead list */ } __aligned(CACHE_LINE_SIZE); /* -- Test scenario: misc/sendfile18.sh