GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r247849+0939d49: Fri Mar 8 19:53:18 CET 2013 pho@x4.osted.lan:/var/tmp/deviant2/sys/amd64/compile/PHO amd64 gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 0x10 Model = 0x2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8098992128 (7723 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <060508 APIC1048> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 random device not loaded; using insecure entropy ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <060508 RSDT1048> on motherboard acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7f00000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xc000-0xc0ff mem 0xf0000000-0xf7ffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq 18 at device 5.0 on pci1 hdac0: mem 0xfe9e8000-0xfe9ebfff irq 19 at device 5.2 on pci1 pcib2: at device 7.0 on pci0 pci2: on pcib2 re0: port 0xd800-0xd8ff mem 0xfeaff000-0xfeafffff irq 19 at device 0.0 on pci2 re0: Using 1 MSI message re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:21:85:15:33:7d ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfe7ff800-0xfe7ffbff irq 22 at device 18.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfe7fe000-0xfe7fefff irq 16 at device 19.0 on pci0 usbus0 on ohci0 ohci1: mem 0xfe7fd000-0xfe7fdfff irq 17 at device 19.1 on pci0 usbus1 on ohci1 ohci2: mem 0xfe7fc000-0xfe7fcfff irq 18 at device 19.2 on pci0 usbus2 on ohci2 ohci3: mem 0xfe7fb000-0xfe7fbfff irq 17 at device 19.3 on pci0 usbus3 on ohci3 ohci4: mem 0xfe7fa000-0xfe7fafff irq 18 at device 19.4 on pci0 usbus4 on ohci4 ehci0: mem 0xfe7ff000-0xfe7ff0ff irq 19 at device 19.5 on pci0 ehci0: AMD SB600/700 quirk applied usbus5: EHCI version 1.0 usbus5 on ehci0 pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 hdac1: mem 0xfe7f4000-0xfe7f7fff irq 16 at device 20.2 on pci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib3: at device 20.4 on pci0 pci3: on pcib3 pci3: at device 6.0 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range acpi_throttle0: on cpu0 acpi_throttle0: CLK_VAL field overlaps THT_EN bit device_attach: acpi_throttle0 attach returned 6 hwpstate0: on cpu0 Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 3 on hdaa0 hdacc1: at cad 3 on hdac1 hdaa1: at nid 1 on hdacc1 pcm1: at nid 20,22,21,23 and 24,26 on hdaa1 pcm2: at nid 27 and 25 on hdaa1 pcm3: at nid 30 on hdaa1 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 Expensive timeout(9) function: 0xffffffff803828e0(0xfffffe0005440a00) 0.008920127 s (aprobe0:ahcich0:0:15:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 (aprobe0:ahcich0:0:15:0): CAM status: Command timeout (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 cd0 at ata0 bus 0 scbus4 target 1 lun 0 cd0: <_NEC DVD_RW ND-4550A 1.06> Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 1800021174 Hz quality 800 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 Root mount waiting for: usbus5 uhub5: 10 ports with 10 removable, self powered Trying to mount root from ufs:/dev/ufs/root [rw]... Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ufs/root: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/root: clean, 447890 free (914 frags, 55872 blocks, 0.1% fragmentation) /dev/ufs/home: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/home: clean, 113264 free (5200 frags, 13508 blocks, 0.5% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 56209555 free (1355 frags, 7026025 blocks, 0.0% fragmentation) /dev/ufs/usr: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/usr: clean, 4272769 free (173337 frags, 512429 blocks, 1.7% fragmentation) /dev/ufs/var: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ufs/var: clean, 7961242 free (46074 frags, 989396 blocks, 0.2% fragmentation) Mounting local file systems:. Setting hostname: x4.osted.lan. re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::221:85ff:fe15:337d%re0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (none) status: no carrier Starting devd. add net default: gateway 192.168.1.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/kde4/lib /usr/local/lib/compat/pkg /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. savecore: unable to read from bounds, using 0 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device Mar 8 20:03:55 x4 savecore: couldn't find media and/or sector size of /var/crash: Inappropriate ioctl for device savecore: unable to read from bounds, using 0 No core dumps found. Additional ABI support: linux. Starting rpcbind. NFS access cache time=60 lock order reversal: 1st 0xffffff81e6e42498 bufwait (bufwait) @ kern/vfs_bio.c:3046 2nd 0xfffffe0005403600 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82309413c0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230941480 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82309414a0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230941550 _sx_xlock() at _sx_xlock+0x74/frame 0xffffff8230941580 ufsdirhash_acquire() at ufsdirhash_acquire+0x44/frame 0xffffff82309415a0 ufsdirhash_add() at ufsdirhash_add+0x19/frame 0xffffff82309415d0 ufs_direnter() at ufs_direnter+0x6c1/frame 0xffffff82309416a0 ufs_mkdir() at ufs_mkdir+0x50e/frame 0xffffff8230941890 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xc9/frame 0xffffff82309418c0 kern_mkdirat() at kern_mkdirat+0x230/frame 0xffffff8230941ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230941bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230941bf0 --- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092532a, rsp = 0x7fffffffd788, rbp = 0x801006050 --- Clearing /tmp (X related). Starting mountd. Starting nfsd. Recovering vi editor sessions:. Updating motd:. Starting ntpd. Configuring syscons: keymap blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting default moused. Expensive timeout(9) function: 0xffffffff80721740(0xffffffff8159e6c0) 0.013890871 s Starting inetd. Fri Mar 8 20:03:59 CET 2013 FreeBSD/amd64 (x4.osted.lan) (console) login: Mar 8 20:19:40 x4 su: pho to root on /dev/pts/1 20130308 20:19:47 all: kinfo3.sh Expensive timeout(9) function: 0xffffffff8076e780(0xffffff80007e2320) 0.018449526 s 20130308 20:35:01 all: sysctl.sh 20130308 20:35:12 all: kinfo.sh lock order reversal: 1st 0xfffffe0007990040 filedesc structure (filedesc structure) @ kern/kern_descrip.c:795 2nd 0xfffffe0007d3a068 ufs (ufs) @ kern/vfs_subr.c:4401 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff823092d620 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff823092d6e0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff823092d700 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff823092d7b0 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff823092d8a0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823092d8f0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823092d920 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823092d9a0 knlist_remove_kq() at knlist_remove_kq+0x175/frame 0xffffff823092d9d0 knote_fdclose() at knote_fdclose+0x187/frame 0xffffff823092da20 closefp() at closefp+0x6d/frame 0xffffff823092da60 do_dup() at do_dup+0x25d/frame 0xffffff823092dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823092dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823092dbf0 --- syscall (90, FreeBSD ELF64, sys_dup2), rip = 0x8008bde6a, rsp = 0x7fffffffd998, rbp = 0x800b8eb80 --- 20130308 21:05:14 all: mmap4.sh vnode_pager_putpages: I/O error 28 vnode_pager_putpages: residual I/O 32768 at 9776 Mar 8 21:05:15 x4 kernel: pid 66032 (mmap4), uid 0 inumber 4 on /mnt: filesystem full lock order reversal: 1st 0xfffffe0007d58c98 ufs (ufs) @ kern/vfs_mount.c:1236 2nd 0xfffffe009c32e2d8 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1410 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230abd4a0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230abd560 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230abd580 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230abd630 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff8230abd720 vop_stdlock() at vop_stdlock+0x39/frame 0xffffff8230abd740 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8230abd770 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8230abd7f0 ffs_flushfiles() at ffs_flushfiles+0x114/frame 0xffffff8230abd870 softdep_flushfiles() at softdep_flushfiles+0x54/frame 0xffffff8230abd8e0 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff8230abd950 dounmount() at dounmount+0x349/frame 0xffffff8230abd9d0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff8230abdad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230abdbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230abdbf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffceb8, rbp = 0x801006ce8 --- 20130308 21:05:17 all: namecache.sh 20130308 21:18:28 all: rename5.sh 20130308 21:19:07 all: extattrctl.sh lock order reversal: 1st 0xfffffe0007845838 ufs_extattr_sx (ufs_extattr_sx) @ ufs/ufs/ufs_extattr.c:114 2nd 0xfffffe011027e2d8 ufs (ufs) @ ufs/ffs/ffs_vfsops.c:1709 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230a811b0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230a81270 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230a81290 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230a81340 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff8230a81430 ffs_vgetf() at ffs_vgetf+0x163/frame 0xffffff8230a814c0 ufs_root() at ufs_root+0x42/frame 0xffffff8230a81500 ufs_extattr_autostart() at ufs_extattr_autostart+0x89/frame 0xffffff8230a81560 ffs_mount() at ffs_mount+0x21b7/frame 0xffffff8230a81770 vfs_donmount() at vfs_donmount+0x9cc/frame 0xffffff8230a81a90 sys_nmount() at sys_nmount+0x66/frame 0xffffff8230a81ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230a81bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230a81bf0 --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800a97dfa, rsp = 0x7fffffffc698, rbp = 0x7fffffffc6b0 --- 20130308 21:19:08 all: gjournal3.sh GEOM_JOURNAL: Journal 1030327831: md5 contains data. GEOM_JOURNAL: Journal 1030327831: md5 contains journal. GEOM_JOURNAL: Journal md5 clean. GEOM_JOURNAL: BIO_FLUSH not supported by md5. lock order reversal: 1st 0xfffffe009cd60430 snaplk (snaplk) @ ufs/ufs/ufs_vnops.c:968 2nd 0xfffffe009cdfc548 ufs (ufs) @ ufs/ffs/ffs_snapshot.c:1627 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230c2a270 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230c2a330 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230c2a350 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230c2a400 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff8230c2a4f0 ffs_snapremove() at ffs_snapremove+0xe2/frame 0xffffff8230c2a570 ffs_truncate() at ffs_truncate+0xd35/frame 0xffffff8230c2a7c0 ufs_inactive() at ufs_inactive+0x28c/frame 0xffffff8230c2a800 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xc9/frame 0xffffff8230c2a830 vinactive() at vinactive+0xb2/frame 0xffffff8230c2a890 vputx() at vputx+0x375/frame 0xffffff8230c2a8f0 kern_unlinkat() at kern_unlinkat+0x1a1/frame 0xffffff8230c2aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230c2abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230c2abf0 --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x8009249da, rsp = 0x7fffffffd5e8, rbp = 0 --- lock order reversal: 1st 0xffffff81e6fc9998 bufwait (bufwait) @ kern/vfs_bio.c:3046 2nd 0xfffffe009cd60430 snaplk (snaplk) @ ufs/ffs/ffs_snapshot.c:1763 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230d1efa0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230d1f060 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230d1f080 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230d1f130 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff8230d1f220 ffs_snapblkfree() at ffs_snapblkfree+0xaa/frame 0xffffff8230d1f2d0 ffs_blkfree() at ffs_blkfree+0x179/frame 0xffffff8230d1f330 ffs_realloccg() at ffs_realloccg+0xb01/frame 0xffffff8230d1f450 ffs_balloc_ufs2() at ffs_balloc_ufs2+0x111e/frame 0xffffff8230d1f620 ffs_write() at ffs_write+0x227/frame 0xffffff8230d1f6c0 VOP_WRITE_APV() at VOP_WRITE_APV+0x135/frame 0xffffff8230d1f7d0 vn_write() at vn_write+0x281/frame 0xffffff8230d1f860 vn_io_fault() at vn_io_fault+0x196/frame 0xffffff8230d1f9f0 dofilewrite() at dofilewrite+0x85/frame 0xffffff8230d1fa40 kern_writev() at kern_writev+0x6c/frame 0xffffff8230d1fa80 sys_write() at sys_write+0x64/frame 0xffffff8230d1fad0 ia32_syscall() at ia32_syscall+0x35f/frame 0xffffff8230d1fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x91/frame 0xffffff8230d1fbf0 --- syscall (4, FreeBSD ELF32, sys_write), rip = 0x2819249f, rsp = 0xffffb688, rbp = 0xffffc730 --- 20130308 21:21:10 all: rename2.sh 20130308 21:56:11 all: revoke.sh 20130308 21:56:12 all: linger.sh lock order reversal: 1st 0xfffffe009cc3d2d8 ufs (ufs) @ kern/vfs_lookup.c:517 2nd 0xffffff81e6f89798 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xfffffe009cc03548 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230cdded0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230cddf90 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230cddfb0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230cde060 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff8230cde150 ffs_lock() at ffs_lock+0x9b/frame 0xffffff8230cde1a0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8230cde1d0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8230cde250 vget() at vget+0x63/frame 0xffffff8230cde2a0 vfs_hash_get() at vfs_hash_get+0xd5/frame 0xffffff8230cde2f0 ffs_vgetf() at ffs_vgetf+0x48/frame 0xffffff8230cde380 softdep_sync_buf() at softdep_sync_buf+0x5ec/frame 0xffffff8230cde470 ffs_syncvnode() at ffs_syncvnode+0x311/frame 0xffffff8230cde4f0 ffs_fsync() at ffs_fsync+0x54/frame 0xffffff8230cde530 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff8230cde560 ufs_direnter() at ufs_direnter+0x980/frame 0xffffff8230cde630 ufs_makeinode() at ufs_makeinode+0x359/frame 0xffffff8230cde7f0 VOP_CREATE_APV() at VOP_CREATE_APV+0xc6/frame 0xffffff8230cde820 vn_open_cred() at vn_open_cred+0x2e6/frame 0xffffff8230cde970 kern_openat() at kern_openat+0x20d/frame 0xffffff8230cdead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230cdebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230cdebf0 --- syscall (5, FreeBSD ELF64, sys_open), rip = 0x80092cb9a, rsp = 0x7fffffffd5e8, rbp = 0 --- 20130308 22:14:52 all: umountf2.sh 20130308 22:23:14 all: truncate2.sh 20130308 22:27:19 all: fullpath.sh 20130308 22:28:20 all: quota4.sh lock order reversal: 1st 0xfffffe01af5faa28 syncer (syncer) @ kern/vfs_subr.c:1795 2nd 0xfffffe01b1fb12d8 ufs (ufs) @ kern/vfs_subr.c:2176 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff82308d85d0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82308d8690 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff82308d86b0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff82308d8760 __lockmgr_args() at __lockmgr_args+0x1158/frame 0xffffff82308d8850 ffs_lock() at ffs_lock+0x9b/frame 0xffffff82308d88a0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff82308d88d0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff82308d8950 vget() at vget+0x63/frame 0xffffff82308d89a0 qsync() at qsync+0x107/frame 0xffffff82308d89f0 ffs_sync() at ffs_sync+0x4a1/frame 0xffffff82308d8ab0 sync_fsync() at sync_fsync+0x12f/frame 0xffffff82308d8ae0 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff82308d8b10 sched_sync() at sched_sync+0x335/frame 0xffffff82308d8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308d8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308d8bf0 --- trap 0, rip = 0, rsp = 0xffffff82308d8cb0, rbp = 0 --- lock order reversal: 1st 0xfffffe0007ab4288 FFS (FFS Lock) @ ufs/ufs/ufs_quota.c:714 2nd 0xfffffe00078ec3c8 struct mount mtx (struct mount mtx) @ ufs/ufs/ufs_quota.c:720 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230e285f0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230e286b0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230e286d0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230e28780 __mtx_lock_flags() at __mtx_lock_flags+0x94/frame 0xffffff8230e287b0 quotaoff_inchange() at quotaoff_inchange+0x8a/frame 0xffffff8230e287f0 ffs_flushfiles() at ffs_flushfiles+0x92/frame 0xffffff8230e28870 softdep_flushfiles() at softdep_flushfiles+0x54/frame 0xffffff8230e288e0 ffs_unmount() at ffs_unmount+0x1d1/frame 0xffffff8230e28950 dounmount() at dounmount+0x349/frame 0xffffff8230e289d0 sys_unmount() at sys_unmount+0x38e/frame 0xffffff8230e28ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230e28bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230e28bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x8008841fa, rsp = 0x7fffffffcea8, rbp = 0x801006ce8 --- 20130308 22:30:01 all: nullfs4.sh 20130308 22:30:02 all: socketpair.sh kern.maxfiles limit exceeded by uid 0, please see tuning(7). 20130308 22:30:07 all: md.sh Mar 8 22:30:09 x4 kernel: pid 30614 (rw), uid 0 inumber 134 on /mnt: filesystem full Mar 8 22:30:10 x4 kernel: pid 30611 (rw), uid 0 inumber 5 on /mnt: filesystem full Mar 8 22:30:15 x4 kernel: pid 30612 (rw), uid 0 inumber 129 on /mnt: filesystem full : : Mar 8 22:56:13 x4 kernel: pid 33369 (rw), uid 0 inumber 143 on /mnt: filesystem full Mar 8 22:56:16 x4 kernel: pid 33368 (rw), uid 0 inumber 136 on /mnt: filesystem full Mar 8 22:56:17 x4 kernel: pid 33373 (rw), uid 0 inumber 5 on /mnt: filesystem full Mar 8 22:56:18 x4 kernel: pid 33362 (rw), uid 0 inumber 133 on /mnt: filesystem full 20130308 22:58:09 all: vnodes.sh 20130308 22:59:24 all: rot.sh 20130308 23:23:18 all: mmap2.sh 20130308 23:42:56 all: zfs2.sh ZFS filesystem version: 5 ZFS storage pool version: features support (5000) 20130308 23:56:53 all: jail2.sh 20130308 23:56:56 all: sem.sh 20130308 23:57:14 all: umountf.sh 20130308 23:57:53 all: flock_open_close.sh 20130309 00:05:01 all: umountf6.sh Mar 9 00:05:13 x4 mountd[745]: can't delete exports for /mnt15: Invalid argument Mar 9 00:05:13 x4 mountd[745]: can't delete exports for /mnt13: Invalid argument Mar 9 00:05:13 x4 mountd[745]: can't delete exports for /mnt12: Invalid argument Mar 9 00:05:13 x4 mountd[745]: can't delete exports for /mnt14: Invalid argument : : Mar 9 00:08:38 x4 mountd[745]: can't delete exports for /mnt14: Invalid argument Mar 9 00:08:40 x4 mountd[745]: can't delete exports for /mnt7: Invalid argument Mar 9 00:08:40 x4 mountd[745]: can't delete exports for /mnt14: Invalid argument Mar 9 00:08:40 x4 mountd[745]: can't delete exports for /mnt10: Invalid argument Mar 9 00:08:40 x4 mountd[745]: can't delete exports for /mnt13: Invalid argument Mar 9 00:08:40 x4 mountd[745]: can't delete exports for /mnt11: Invalid argument 20130309 00:08:41 all: nullfs5.sh 20130309 00:20:13 all: zfs5.sh lock order reversal: 1st 0xfffffe019c7d7548 zfs (zfs) @ /var/tmp/deviant2/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1754 2nd 0xfffffe01a9f94a28 ufs (ufs) @ kern/vfs_vnops.c:461 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xffffff8230f859f0 kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff8230f85ab0 _witness_debugger() at _witness_debugger+0x2c/frame 0xffffff8230f85ad0 witness_checkorder() at witness_checkorder+0x82d/frame 0xffffff8230f85b80 __lockmgr_args() at __lockmgr_args+0x1496/frame 0xffffff8230f85c70 ffs_lock() at ffs_lock+0x9b/frame 0xffffff8230f85cc0 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff8230f85cf0 _vn_lock() at _vn_lock+0x8e/frame 0xffffff8230f85d70 vn_rdwr() at vn_rdwr+0xdb/frame 0xffffff8230f85e40 vdev_file_io_start() at vdev_file_io_start+0xd3/frame 0xffffff8230f85ed0 zio_vdev_io_start() at zio_vdev_io_start+0x1d4/frame 0xffffff8230f85f20 zio_execute() at zio_execute+0xc3/frame 0xffffff8230f85f60 vdev_mirror_io_start() at vdev_mirror_io_start+0x174/frame 0xffffff8230f85fe0 zio_vdev_io_start() at zio_vdev_io_start+0x212/frame 0xffffff8230f86030 zio_execute() at zio_execute+0xc3/frame 0xffffff8230f86070 arc_read() at arc_read+0x71b/frame 0xffffff8230f86120 dbuf_read() at dbuf_read+0x438/frame 0xffffff8230f861e0 dmu_buf_hold() at dmu_buf_hold+0xe0/frame 0xffffff8230f86230 zap_lockdir() at zap_lockdir+0x58/frame 0xffffff8230f862c0 zap_lookup_norm() at zap_lookup_norm+0x45/frame 0xffffff8230f86320 zap_lookup() at zap_lookup+0x2e/frame 0xffffff8230f86350 zfs_dirent_lock() at zfs_dirent_lock+0x4ff/frame 0xffffff8230f86400 zfs_dirlook() at zfs_dirlook+0xd6/frame 0xffffff8230f86480 zfs_lookup() at zfs_lookup+0x275/frame 0xffffff8230f86510 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81/frame 0xffffff8230f86660 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xc6/frame 0xffffff8230f86690 vfs_cache_lookup() at vfs_cache_lookup+0xfb/frame 0xffffff8230f86700 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xc6/frame 0xffffff8230f86730 lookup() at lookup+0x36e/frame 0xffffff8230f867c0 namei() at namei+0x410/frame 0xffffff8230f86870 kern_statat_vnhook() at kern_statat_vnhook+0xbc/frame 0xffffff8230f86a10 kern_statat() at kern_statat+0x15/frame 0xffffff8230f86a30 sys_lstat() at sys_lstat+0x2a/frame 0xffffff8230f86ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230f86bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230f86bf0 --- syscall (190, FreeBSD ELF64, sys_lstat), rip = 0x8017b92ea, rsp = 0x7fffffffc408, rbp = 0x802c84080 --- 20130309 00:30:57 all: quota5.sh 20130309 00:30:57 all: linger4.sh Mar 9 00:33:45 x4 kernel: pid 76160 (linger4), uid 1004 inumber 40 on /mnt: out of inodes 20130309 00:33:52 all: sndstat.sh 20130309 00:33:52 all: kevent4.sh 20130309 00:35:55 all: ffs_syncvnode.sh 20130309 01:11:47 all: msdos4.sh 20130309 01:32:12 all: suj2.sh 20130309 01:32:13 all: rename9.sh 20130309 01:32:59 all: nfs7.sh Mar 9 01:47:51 x4 kernel: pid 6612 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:48:41 x4 kernel: pid 7589 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:49:47 x4 kernel: pid 7591 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:49:58 x4 kernel: pid 7592 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:49:59 x4 kernel: pid 7593 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:00 x4 kernel: pid 7586 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:00 x4 kernel: pid 7596 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:02 x4 kernel: pid 7582 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:03 x4 kernel: pid 7590 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:06 x4 kernel: pid 7588 (lockf), uid 0, was killed: exceeded maximum CPU limit Mar 9 01:50:09 x4 kernel: pid 7584 (lockf), uid 0, was killed: exceeded maximum CPU limit 20130309 01:50:34 all: rename8.sh 20130309 01:51:24 all: umountf5.sh 20130309 01:51:56 all: jail3.sh 20130309 01:51:57 all: newfs4.sh 20130309 01:55:06 all: tmpfs5.sh 20130309 01:55:29 all: mac.sh 20130309 01:55:30 all: fragments.sh Mar 9 01:55:39 x4 kernel: pid 21283 (fragments), uid 1004 inumber 83872 on /mnt: filesystem full Mar 9 01:55:39 x4 kernel: pid 21283 (fragments), uid 1004 inumber 65554 on /mnt: filesystem full Mar 9 01:55:40 x4 kernel: pid 21303 (fragments), uid 1004 inumber 6086 on /mnt: filesystem full Mar 9 01:55:41 x4 kernel: pid 21302 (fragments), uid 1004 inumber 6091 on /mnt: filesystem full Mar 9 01:55:42 x4 kernel: pid 21300 (fragments), uid 1004 inumber 6089 on /mnt: filesystem full : : Mar 9 02:27:07 x4 kernel: pid 24178 (fragments), uid 1004 inumber 6145 on /mnt: filesystem full Mar 9 02:27:08 x4 kernel: pid 24177 (fragments), uid 1004 inumber 6091 on /mnt: filesystem full Mar 9 02:27:09 x4 kernel: pid 24177 (fragments), uid 1004 inumber 6089 on /mnt: filesystem full Mar 9 02:27:10 x4 kernel: pid 24182 (fragments), uid 1004 inumber 6128 on /mnt: filesystem full Mar 9 02:27:11 x4 kernel: pid 24182 (fragments), uid 1004 inumber 6090 on /mnt: filesystem full 20130309 02:27:14 all: snap6.sh Mar 9 02:31:31 x4 kernel: pid 24690 (dd), uid 0 inumber 7 on /var: filesystem full Mar 9 02:31:34 x4 kernel: pid 24689 (mksnap_ffs), uid 0 inumber 11 on /var: filesystem full 20130309 02:31:38 all: suj12.sh GEOM_NOP: Device md5.nop created. GEOM_NOP: Device md5.nop removed. 20130309 02:52:12 all: symlink2.sh 20130309 02:52:20 all: zfs3.sh 20130309 03:04:35 all: rdwr.sh 20130309 03:04:37 all: graid3.sh GEOM_RAID3: Device raid3/data launched (3/3). GEOM_RAID3: Device data: provider raid3/data destroyed. GEOM_RAID3: Device data destroyed. 20130309 03:25:57 all: zfs.sh 20130309 03:59:07 all: fpclone.sh 20130309 04:01:12 all: devfs.sh Mar 9 04:01:13 x4 mountd[745]: can't delete exports for /mnt10: Invalid argument Mar 9 04:01:13 x4 mountd[745]: can't delete exports for /mnt5: Invalid argument Mar 9 04:01:13 x4 mountd[745]: can't delete exports for /mnt4: Invalid argument Mar 9 04:01:13 x4 mountd[745]: can't delete exports for /mnt7: Invalid argument Mar 9 04:01:13 x4 mountd[745]: can't delete exports for /mnt6: Invalid argument Mar 9 04:01:14 x4 mountd[745]: can't delete exports for /mnt3: Invalid argument Mar 9 04:01:14 x4 mountd[745]: can't delete exports for /mnt10: Invalid argument Mar 9 04:01:14 x4 mountd[745]: can't delete exports for /mnt2: Invalid argument Mar 9 04:01:14 x4 mountd[745]: can't delete exports for /mnt7: Invalid argument : : Mar 9 04:01:31 x4 mountd[745]: can't delete exports for /mnt4: Invalid argument Mar 9 04:01:34 x4 mountd[745]: can't delete exports for /mnt3: Invalid argument Mar 9 04:01:36 x4 mountd[745]: can't delete exports for /mnt5: Invalid argument Mar 9 04:01:39 x4 last message repeated 2 times Mar 9 04:01:39 x4 mountd[745]: can't delete exports for /mnt3: Invalid argument 20130309 04:01:42 all: sendfile.sh 20130309 04:08:28 all: symlink.sh Expensive timeout(9) function: 0xffffffff8076e780(0xffffff80007df320) 0.188873657 s 20130309 04:29:31 all: inversion.sh 20130309 04:59:32 all: quota8.sh Mar 9 05:01:31 x4 kernel: pid 97452 (rw), uid 1004 inumber 68226 on /mnt: filesystem full 20130309 05:02:41 all: union.sh Mar 9 05:02:42 x4 mountd[745]: can't delete exports for /home/pho/stress2/misc/mp1: Invalid argument 20130309 05:13:01 all: md3.sh 20130309 05:23:17 all: extattr_set_fd.sh 20130309 05:23:19 all: quota9.sh 20130309 05:23:22 all: gbde.sh Mar 9 05:31:37 x4 kernel: pid 7107 (openat), uid 1004 inumber 63802 on /mnt: out of inodes 20130309 05:45:02 all: tmpfs2.sh Mar 9 05:45:03 x4 mountd[745]: can't delete exports for /mnt13: Invalid argument Mar 9 05:45:03 x4 mountd[745]: can't delete exports for /mnt0: Invalid argument Mar 9 05:45:04 x4 mountd[745]: can't delete exports for /mnt10: Invalid argument Mar 9 05:45:05 x4 mountd[745]: can't delete exports for /mnt13: Invalid argument Mar 9 05:45:05 x4 mountd[745]: can't delete exports for /mnt1: Invalid argument : : Mar 9 05:49:16 x4 mountd[745]: can't delete exports for /mnt12: Invalid argument Mar 9 05:49:16 x4 mountd[745]: can't delete exports for /mnt12: Invalid argument Mar 9 05:49:16 x4 mountd[745]: can't delete exports for /mnt14: Invalid argument Mar 9 05:49:16 x4 mountd[745]: can't delete exports for /mnt6: Invalid argument 20130309 05:49:17 all: nullfs3.sh 20130309 05:49:18 all: procfs2.sh 20130309 05:49:24 all: mountro.sh fsync: giving up on dirty 0xfffffe00b2ac59c0: tag devfs, type VCHR usecount 1, writecount 0, refcount 133 mountedhere 0xfffffe0063ce1800 flags (VI_ACTIVE) v_object 0xfffffe01a93ad1e0 ref 0 pages 1006 lock type devfs: EXCL by thread 0xfffffe009c7d2920 (pid 82334, umount, tid 100394) #0 0xffffffff808c5f95 at __lockmgr_args+0x6e5 #1 0xffffffff80978619 at vop_stdlock+0x39 #2 0xffffffff80d282fb at VOP_LOCK1_APV+0xab #3 0xffffffff80998e3e at _vn_lock+0x8e #4 0xffffffff80b35af1 at softdep_flushworklist+0x61 #5 0xffffffff80b39b8e at ffs_sync+0x3ce #6 0xffffffff80980d48 at dounmount+0x318 #7 0xffffffff8098150e at sys_unmount+0x38e #8 0xffffffff80c89403 at amd64_syscall+0x2d3 #9 0xffffffff80c73577 at Xfast_syscall+0xf7 dev md5a fsync: giving up on dirty 0xfffffe00b2ac59c0: tag devfs, type VCHR usecount 1, writecount 0, refcount 134 mountedhere 0xfffffe0063ce1800 flags (VI_ACTIVE) v_object 0xfffffe01a93ad1e0 ref 0 pages 1014 lock type devfs: EXCL by thread 0xfffffe009c7d2920 (pid 82334, umount, tid 100394) #0 0xffffffff808c5f95 at __lockmgr_args+0x6e5 #1 0xffffffff80978619 at vop_stdlock+0x39 #2 0xffffffff80d282fb at VOP_LOCK1_APV+0xab #3 0xffffffff80998e3e at _vn_lock+0x8e #4 0xffffffff80b39a34 at ffs_sync+0x274 #5 0xffffffff80980d48 at dounmount+0x318 #6 0xffffffff8098150e at sys_unmount+0x38e #7 0xffffffff80c89403 at amd64_syscall+0x2d3 #8 0xffffffff80c73577 at Xfast_syscall+0xf7 dev md5a 20130309 05:51:26 all: unload2.sh 20130309 05:51:28 all: tmpfs.sh 20130309 06:03:05 all: sendfile2.sh 20130309 06:03:06 all: tmpfs4.sh 20130309 06:03:07 all: core.sh 20130309 06:06:00 all: kinfo2.sh 20130309 06:36:02 all: tmpfs3.sh 20130309 06:46:09 all: suj3.sh cryptosoft0: on motherboard GEOM_ELI: Device md5.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software GEOM_ELI: md5 has been killed. GEOM_ELI: Device md5.eli destroyed. 20130309 06:52:25 all: rename6.sh 20130309 06:52:30 all: suj13.sh 20130309 07:13:50 all: nullfs6.sh 20130309 07:13:50 all: mountro3.sh 20130309 07:13:52 all: nfs5.sh 20130309 07:17:38 all: libMicro.sh 20130309 07:51:01 all: suj29.sh 20130309 07:57:14 all: tmpfs7.sh 20130309 07:57:15 all: nullfs12.sh 20130309 07:57:16 all: suj8.sh Mar 9 07:57:17 x4 kernel: pid 84246 (dd), uid 0 inumber 4 on /mnt: filesystem full Mar 9 07:57:19 x4 kernel: pid 84267 (dd), uid 0 inumber 5 on /mnt: filesystem full 20130309 07:57:21 all: nullfs11.sh 20130309 08:01:27 all: mlockall2.sh 20130309 09:53:00 all: rdwr.sh 20130309 09:53:01 all: suj4.sh 20130309 10:53:11 all: truncate2.sh 20130309 10:57:15 all: kinfo.sh 20130309 11:27:17 all: rename2.sh 20130309 12:02:13 all: nullfs.sh 20130309 12:06:04 all: fpclone.sh 20130309 12:08:08 all: tmpfs3.sh 20130309 12:18:14 all: rename5.sh 20130309 12:18:55 all: suj14.sh Mar 9 12:20:14 x4 kernel: pid 86355 (cp), uid 0 inumber 50924 on /mnt: filesystem full Mar 9 12:20:15 x4 kernel: pid 86355 (cp), uid 0 inumber 50940 on /mnt: filesystem full Mar 9 12:20:16 x4 kernel: pid 86355 (cp), uid 0 inumber 50950 on /mnt: filesystem full Mar 9 12:20:17 x4 kernel: pid 86355 (cp), uid 0 inumber 50960 on /mnt: filesystem full Mar 9 12:20:18 x4 kernel: pid 86360 (mkdir), uid 0 inumber 50966 on /mnt: filesystem full Mar 9 12:20:19 x4 kernel: pid 86370 (mkdir), uid 0 inumber 50966 on /mnt: filesystem full Mar 9 12:20:20 x4 kernel: pid 86395 (mkdir), uid 0 inumber 50966 on /mnt: filesystem full Mar 9 12:20:21 x4 kernel: pid 86405 (mkdir), uid 0 inumber 50966 on /mnt: filesystem full 20130309 12:20:26 all: extattr_set_fd.sh 20130309 12:20:27 all: nullfs5.sh Expensive timeout(9) function: 0xffffffff80c9b720(0xffffffff8158c860) 0.233103876 s 20130309 12:31:06 all: procfs4.sh 20130309 12:40:15 all: tmpfs5.sh 20130309 12:40:38 all: sndstat.sh 20130309 12:40:38 all: fpclone2.sh 20130309 12:40:43 all: altbufferflushes.sh Mar 9 12:46:59 x4 kernel: pid 37357 (dd), uid 0 inumber 5416967 on /var: filesystem full Mar 9 12:53:28 x4 kernel: pid 37970 (dd), uid 0 inumber 6 on /var: filesystem full Mar 9 12:53:30 x4 kernel: pid 37970 (dd), uid 0 inumber 5416966 on /var: filesystem full /var: got error 28 while accessing filesystem panic: softdep_deallocate_dependencies: unrecovered I/O error cpuid = 2 KDB: enter: panic [ thread pid 614 tid 100082 ] Stopped at kdb_enter+0x3b: movq $0,0xa9d892(%rip) db> run pho db:0:pho> bt Tracing pid 614 tid 100082 td 0xfffffe0007828490 kdb_enter() at kdb_enter+0x3b/frame 0xffffff8230905650 vpanic() at vpanic+0xe1/frame 0xffffff8230905690 panic() at panic+0x67/frame 0xffffff8230905770 softdep_deallocate_dependencies() at softdep_deallocate_dependencies+0x72/frame 0xffffff8230905790 brelse() at brelse+0x83/frame 0xffffff8230905810 bufdone() at bufdone+0x78/frame 0xffffff8230905830 ufs_strategy() at ufs_strategy+0x45/frame 0xffffff8230905860 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0xd1/frame 0xffffff8230905890 bufstrategy() at bufstrategy+0x4b/frame 0xffffff82309058d0 bufwrite() at bufwrite+0x125/frame 0xffffff8230905900 vfs_bio_awrite() at vfs_bio_awrite+0xa3/frame 0xffffff8230905980 ffs_syncvnode() at ffs_syncvnode+0x34a/frame 0xffffff8230905a00 ffs_fsync() at ffs_fsync+0x54/frame 0xffffff8230905a40 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff8230905a70 sys_fsync() at sys_fsync+0x150/frame 0xffffff8230905ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230905bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230905bf0 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x800b4c38a, rsp = 0x7fffffffd118, rbp = 0x80142c108 --- db:0:bt> show allpcpu Current CPU: 2 cpuid = 0 dynamic pcpu = 0x5da400 curthread = 0xfffffe000521b490: pid 13 "g_up" curpcb = 0xffffff8000294cc0 fpcurthread = none idlethread = 0xfffffe00051f6490: tid 100003 "idle: cpu0" curpmap = 0xffffffff813647d0 tssp = 0xffffffff81576280 commontssp = 0xffffffff81576280 rsp0 = 0xffffff8000294cc0 gs32p = 0xffffffff815743b8 ldt = 0xffffffff815743f8 tss = 0xffffffff815743e8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807ef17400 curthread = 0xfffffe00051f6000: pid 11 "idle: cpu1" curpcb = 0xffffff8000262cc0 fpcurthread = none idlethread = 0xfffffe00051f6000: tid 100004 "idle: cpu1" curpmap = 0xffffffff813647d0 tssp = 0xffffffff815762e8 commontssp = 0xffffffff815762e8 rsp0 = 0xffffff8000262cc0 gs32p = 0xffffffff81574420 ldt = 0xffffffff81574460 tss = 0xffffffff81574450 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807ef1e400 curthread = 0xfffffe0007828490: pid 614 "syslogd" curpcb = 0xffffff8230905cc0 fpcurthread = none idlethread = 0xfffffe0005203920: tid 100005 "idle: cpu2" curpmap = 0xfffffe0007a71440 tssp = 0xffffffff81576350 commontssp = 0xffffffff81576350 rsp0 = 0xffffff8230905cc0 gs32p = 0xffffffff81574488 ldt = 0xffffffff815744c8 tss = 0xffffffff815744b8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807ef25400 curthread = 0xfffffe0005203490: pid 11 "idle: cpu3" curpcb = 0xffffff800026ccc0 fpcurthread = none idlethread = 0xfffffe0005203490: tid 100006 "idle: cpu3" curpmap = 0xffffffff813647d0 tssp = 0xffffffff815763b8 commontssp = 0xffffffff815763b8 rsp0 = 0xffffff800026ccc0 gs32p = 0xffffffff815744f0 ldt = 0xffffffff81574530 tss = 0xffffffff81574520 spin locks held: db:0:allpcpu> show alllocks Process 37970 (dd) thread 0xfffffe008ee74490 (102541) exclusive lockmgr ufs (ufs) r = 0 (0xfffffe011f21a7b8) locked @ kern/vfs_subr.c:2176 Process 614 (syslogd) thread 0xfffffe0007828490 (100082) exclusive lockmgr ufs (ufs) r = 0 (0xfffffe01edbb9c98) locked @ kern/vfs_syscalls.c:3437 Process 13 (geom) thread 0xfffffe000521b490 (100014) shared rw pmap pv list (pmap pv list) r = 0 (0xffffffff8158c3a0) locked @ amd64/amd64/pmap.c:4410 shared rw pmap pv global (pmap pv global) r = 0 (0xffffffff8158bd80) locked @ amd64/amd64/pmap.c:4408 exclusive sleep mutex vm page (vm page) r = 0 (0xffffffff8156f980) locked @ kern/vfs_bio.c:1812 exclusive sleep mutex vm object (vm object) r = 0 (0xfffffe00079b5b40) locked @ kern/vfs_bio.c:1804 db:0:alllocks> show lockedvnods Locked vnodes 0xfffffe01edbb9c30: tag ufs, type VREG usecount 2, writecount 1, refcount 4 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffffe01a7b26c30 ref 0 pages 1 lock type ufs: EXCL by thread 0xfffffe0007828490 (pid 614, syslogd, tid 100082) with shared waiters pending #0 0xffffffff808c5f95 at __lockmgr_args+0x6e5 #1 0xffffffff80b3e43b at ffs_lock+0x9b #2 0xffffffff80d282fb at VOP_LOCK1_APV+0xab #3 0xffffffff80998e3e at _vn_lock+0x8e #4 0xffffffff809966e0 at sys_fsync+0xc0 #5 0xffffffff80c89403 at amd64_syscall+0x2d3 #6 0xffffffff80c73577 at Xfast_syscall+0xf7 ino 2896944, on dev ufs/var 0xfffffe011f21a750: tag ufs, type VREG usecount 1, writecount 1, refcount 51328 mountedhere 0 flags (VI_ACTIVE) v_object 0xfffffe01a9a904b0 ref 0 pages 805756 lock type ufs: EXCL by thread 0xfffffe008ee74490 (pid 37970, dd, tid 102541) #0 0xffffffff808c5f95 at __lockmgr_args+0x6e5 #1 0xffffffff80b3e43b at ffs_lock+0x9b #2 0xffffffff80d282fb at VOP_LOCK1_APV+0xab #3 0xffffffff80998e3e at _vn_lock+0x8e #4 0xffffffff80999dd5 at vn_write+0x105 #5 0xffffffff80997ea6 at vn_io_fault+0x196 #6 0xffffffff80938055 at dofilewrite+0x85 #7 0xffffffff80939e5c at kern_writev+0x6c #8 0xffffffff80939f74 at sys_write+0x64 #9 0xffffffff80c89403 at amd64_syscall+0x2d3 #10 0xffffffff80c73577 at Xfast_syscall+0xf7 ino 5416966, on dev ufs/var db:0:lockedvnods> show mount 0xfffffe00078ecb58 /dev/ufs/root on / (ufs) 0xfffffe00078ed000 devfs on /dev (devfs) 0xfffffe0007aa4790 /dev/ufs/home on /home (ufs) 0xfffffe0007aa43c8 /dev/label/tmp on /tmp (ufs) 0xfffffe0007aa4000 /dev/ufs/usr on /usr (ufs) 0xfffffe00078edb58 /dev/ufs/var on /var (ufs) 0xfffffe00078ed790 procfs on /proc (procfs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 38095 1130 1127 1001 S nanslp 0xffffffff8136bdc8 sleep 37971 36879 58875 0 S+ piperd 0xfffffe01e80c32e8 egrep 37970 36879 58875 0 D+ bo_wwait 0xfffffe011f21a948 dd 36879 59414 58875 0 S+ wait 0xfffffe01cba4c970 sh 60372 60371 60372 1001 Ss+ ttyin 0xfffffe00305264a8 bash 60371 60369 60369 1001 S select 0xfffffe005e612bc0 sshd 60369 941 60369 0 Ss select 0xfffffe01510d1b40 sshd 59414 58875 58875 0 S+ wait 0xfffffe003ef74970 sh 58875 1174 58875 0 S+ wait 0xfffffe009ca3c970 sh 59543 13737 59543 1001 S+ kqread 0xfffffe01f1572800 tail 13737 13400 13737 1001 Ss+ wait 0xfffffe009c720000 bash 13400 9653 9653 1001 S select 0xfffffe005e619140 sshd 9653 941 9653 0 Ss select 0xfffffe0089ac31c0 sshd 24410 0 0 0 DL crypto_r 0xffffffff81a40360 [crypto returns] 24409 0 0 0 DL crypto_w 0xffffffff81a40320 [crypto] 40250 0 0 0 DL (threaded) [zfskern] 100256 D l2arc_fe 0xffffffff81bd1540 [l2arc_feed_thread] 101505 D arc_recl 0xffffffff81bc16a0 [arc_reclaim_thread] 1174 1171 1174 0 S+ wait 0xfffffe009c43b000 bash 1171 1155 1171 0 S+ pause 0xfffffe009c46a560 csh 1155 1150 1155 1001 S+ wait 0xfffffe0007ebf970 su 1150 1149 1150 1001 Ss+ wait 0xfffffe009c401000 bash 1149 1147 1147 1001 S select 0xfffffe00078e40c0 sshd 1147 941 1147 0 Ss select 0xfffffe009c49e940 sshd 1131 1127 1127 1001 S piperd 0xfffffe0007ed18b8 awk 1130 1127 1127 1001 S wait 0xfffffe00079ec970 sh 1129 1125 1129 1001 Ss+ select 0xfffffe009c30acc0 top 1128 1126 1128 1001 Ds ufs 0xfffffe01edbb9c98 tail 1127 1124 1127 1001 Ss wait 0xfffffe00079c9000 sh 1126 1119 1119 1001 S select 0xfffffe000789eac0 sshd 1125 1120 1120 1001 S select 0xfffffe00078e4040 sshd 1124 1118 1118 1001 S select 0xfffffe000789d2c0 sshd 1120 941 1120 0 Ss select 0xfffffe00078e5cc0 sshd 1119 941 1119 0 Ss select 0xfffffe009c30ad40 sshd 1118 941 1118 0 Ss select 0xfffffe000789d140 sshd 1070 1 1070 0 Ss+ ttyin 0xfffffe00052dc4a8 getty 1069 1 1069 0 Ss+ ttyin 0xfffffe00052dc8a8 getty 1068 1 1068 0 Ss+ ttyin 0xfffffe00052dcca8 getty 1067 1 1067 0 Ss+ ttyin 0xfffffe00070570a8 getty 1066 1 1066 0 Ss+ ttyin 0xfffffe00070574a8 getty 1065 1 1065 0 Ss+ ttyin 0xfffffe00070578a8 getty 1064 1 1064 0 Ss+ ttyin 0xfffffe00052db0a8 getty 1063 1 1063 0 Ss+ ttyin 0xfffffe00052db4a8 getty 1062 1 1062 0 Ss+ ttyin 0xfffffe00052db8a8 getty 1023 1 1023 0 Ss select 0xfffffe00078e5c40 inetd 990 1 990 0 Ss select 0xfffffe000787d2c0 moused 970 1 970 0 Ss nanslp 0xffffffff8136bdc8 watchdogd 960 1 960 0 Ss nanslp 0xffffffff8136bdc8 cron 953 1 953 25 Ss pause 0xfffffe0007aaca18 sendmail 949 1 949 0 Ss select 0xfffffe000789edc0 sendmail 941 1 941 0 Ss select 0xfffffe000787d340 sshd 850 1 850 0 Ss select 0xfffffe000787edc0 ntpd 755 754 754 0 S (threaded) nfsd 100109 S rpcsvc 0xfffffe0007af0020 nfsd: service 100108 S rpcsvc 0xfffffe0007af00a0 nfsd: service 100107 S rpcsvc 0xfffffe0007af0120 nfsd: service 100101 S rpcsvc 0xfffffe0007b57d20 nfsd: master 754 1 754 0 Ss select 0xfffffe00078e4240 nfsd 745 1 745 0 Ss select 0xfffffe00078e4340 mountd 640 1 640 0 Ss select 0xfffffe00078e5dc0 rpcbind 614 1 614 0 Rs CPU 2 syslogd 433 1 433 0 Ss select 0xfffffe000789d840 devd 17 0 0 0 DL sdflush 0xffffffff81569460 [softdepflush] 16 0 0 0 DL syncer 0xffffffff8155da00 [syncer] 9 0 0 0 DL vlruwt 0xfffffe00070b7000 [vnlru] 8 0 0 0 DL psleep 0xffffffff8155d420 [bufdaemon] 7 0 0 0 DL pgzero 0xffffffff8157323c [pagezero] 6 0 0 0 DL psleep 0xffffffff815723f0 [vmdaemon] 5 0 0 0 DL psleep 0xffffffff815723cc [pagedaemon] 4 0 0 0 DL ccb_scan 0xffffffff81329a60 [xpt_thrd] 3 0 0 0 DL waiting_ 0xffffffff81563720 [sctp_iterator] 2 0 0 0 DL - 0xfffffe0007025048 [fdc0] 15 0 0 0 DL (threaded) [usb] 100058 D - 0xffffff80007f0e18 [usbus5] 100057 D - 0xffffff80007f0dc0 [usbus5] 100056 D - 0xffffff80007f0d68 [usbus5] 100055 D - 0xffffff80007f0d10 [usbus5] 100053 D - 0xffffff80007e8460 [usbus4] 100052 D - 0xffffff80007e8408 [usbus4] 100051 D - 0xffffff80007e83b0 [usbus4] 100050 D - 0xffffff80007e8358 [usbus4] 100049 D - 0xffffff80007e5460 [usbus3] 100048 D - 0xffffff80007e5408 [usbus3] 100047 D - 0xffffff80007e53b0 [usbus3] 100046 D - 0xffffff80007e5358 [usbus3] 100045 D - 0xffffff80007e2460 [usbus2] 100044 D - 0xffffff80007e2408 [usbus2] 100043 D - 0xffffff80007e23b0 [usbus2] 100042 D - 0xffffff80007e2358 [usbus2] 100040 D - 0xffffff80007df460 [usbus1] 100039 D - 0xffffff80007df408 [usbus1] 100038 D - 0xffffff80007df3b0 [usbus1] 100037 D - 0xffffff80007df358 [usbus1] 100035 D - 0xffffff80007dc460 [usbus0] 100034 D - 0xffffff80007dc408 [usbus0] 100033 D - 0xffffff80007dc3b0 [usbus0] 100032 D - 0xffffff80007dc358 [usbus0] 14 0 0 0 RL CPU 255 [yarrow] 13 0 0 0 RL (threaded) [geom] 100015 D - 0xffffffff81363a50 [g_down] 100014 Run CPU 0 [g_up] 100013 D - 0xffffffff81363a38 [g_event] 12 0 0 0 RL (threaded) [intr] 100063 I [irq12: psm0] 100062 I [irq1: atkbd0] 100060 I [swi0: uart] 100059 I [irq14: ata0] 100054 I [irq19: ehci0] 100041 I [irq18: ohci2 ohci4] 100036 I [irq17: ohci1 ohci3] 100031 I [irq16: hdac1 ohci0] 100030 I [irq22: ahci0] 100029 I [irq257: re0] 100028 I [irq256: hdac0] 100027 I [swi2: cambio] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100012 I [swi1: netisr 0] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 RunQ [swi4: clock] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 Run CPU 3 [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe00051f4970 [init] 10 0 0 0 DL audit_wo 0xffffffff815683d0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100255 D - 0xfffffe0007b04400 [system_taskq_3] 100211 D - 0xfffffe0007b04400 [system_taskq_2] 100191 D - 0xfffffe0007b04400 [system_taskq_1] 100176 D - 0xfffffe0007b04400 [system_taskq_0] 100065 D - 0xffffffff8136aa04 [deadlkres] 100064 D - 0xfffffe0007064b80 [mca taskq] 100026 D - 0xfffffe00051d3000 [acpi_task_2] 100025 D - 0xfffffe00051d3000 [acpi_task_1] 100024 D - 0xfffffe00051d3000 [acpi_task_0] 100023 D - 0xfffffe00051d3080 [kqueue taskq] 100020 D - 0xfffffe00051d3200 [thread taskq] 100018 D - 0xfffffe00051d3300 [ffs_trim taskq] 100016 D - 0xfffffe00051d3a80 [firmware taskq] 100000 D sched 0xffffffff81363d60 [swapper] db:0:ps> allt Tracing command sleep pid 38095 tid 100246 td 0xfffffe009c842920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230b1c860 mi_switch() at mi_switch+0x238/frame 0xffffff8230b1c8b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230b1c8f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230b1c950 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff8230b1c980 _sleep() at _sleep+0x3bb/frame 0xffffff8230b1ca10 kern_nanosleep() at kern_nanosleep+0xc6/frame 0xffffff8230b1ca70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff8230b1cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230b1cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230b1cbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800913baa, rsp = 0x7fffffffdb48, rbp = 0x7fffffffdbd0 --- Tracing command egrep pid 37971 tid 100166 td 0xfffffe009c6aa920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230aa9800 mi_switch() at mi_switch+0x238/frame 0xffffff8230aa9850 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230aa9890 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230aa98f0 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230aa9910 _sleep() at _sleep+0x375/frame 0xffffff8230aa99a0 pipe_read() at pipe_read+0x44c/frame 0xffffff8230aa99f0 dofileread() at dofileread+0xa1/frame 0xffffff8230aa9a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230aa9a80 sys_read() at sys_read+0x64/frame 0xffffff8230aa9ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230aa9bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230aa9bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800f7e92a, rsp = 0x7fffffffd6f8, rbp = 0x1 --- Tracing command dd pid 37970 tid 102541 td 0xfffffe008ee74490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82325151c0 mi_switch() at mi_switch+0x238/frame 0xffffff8232515210 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8232515250 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8232515280 _sleep() at _sleep+0x3eb/frame 0xffffff8232515310 bufobj_wwait() at bufobj_wwait+0x82/frame 0xffffff8232515350 ffs_syncvnode() at ffs_syncvnode+0x183/frame 0xffffff82325153d0 ffs_truncate() at ffs_truncate+0x10ef/frame 0xffffff8232515620 ffs_write() at ffs_write+0x64e/frame 0xffffff82325156c0 VOP_WRITE_APV() at VOP_WRITE_APV+0x135/frame 0xffffff82325157d0 vn_write() at vn_write+0x281/frame 0xffffff8232515860 vn_io_fault() at vn_io_fault+0x196/frame 0xffffff82325159f0 dofilewrite() at dofilewrite+0x85/frame 0xffffff8232515a40 kern_writev() at kern_writev+0x6c/frame 0xffffff8232515a80 sys_write() at sys_write+0x64/frame 0xffffff8232515ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232515bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232515bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80093890a, rsp = 0x7fffffffd878, rbp = 0x100000 --- Tracing command sh pid 36879 tid 100774 td 0xfffffe000b94d000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8231b06670 mi_switch() at mi_switch+0x238/frame 0xffffff8231b066c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8231b06700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8231b06760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8231b06780 _sleep() at _sleep+0x375/frame 0xffffff8231b06810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff8231b068b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8231b06a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8231b06ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8231b06bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8231b06bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd4c8, rbp = 0x1 --- Tracing command bash pid 60372 tid 102471 td 0xfffffe00d604c490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82323b7710 mi_switch() at mi_switch+0x238/frame 0xffffff82323b7760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82323b77a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82323b7800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82323b7820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82323b7880 tty_wait() at tty_wait+0x4c/frame 0xffffff82323b78b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff82323b7950 ttydev_read() at ttydev_read+0x95/frame 0xffffff82323b7980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82323b79f0 dofileread() at dofileread+0xa1/frame 0xffffff82323b7a40 kern_readv() at kern_readv+0x6c/frame 0xffffff82323b7a80 sys_read() at sys_read+0x64/frame 0xffffff82323b7ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82323b7bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82323b7bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80113b92a, rsp = 0x7fffffffc8c8, rbp = 0x7fffffffc8e7 --- Tracing command sshd pid 60371 tid 102410 td 0xfffffe00c1374920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82322866a0 mi_switch() at mi_switch+0x238/frame 0xffffff82322866f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8232286730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8232286790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82322867b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8232286810 seltdwait() at seltdwait+0xdd/frame 0xffffff8232286850 kern_select() at kern_select+0x68c/frame 0xffffff8232286a80 sys_select() at sys_select+0x5d/frame 0xffffff8232286ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8232286bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8232286bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 60369 tid 100293 td 0xfffffe000b9e8920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823154d780 mi_switch() at mi_switch+0x238/frame 0xffffff823154d7d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823154d810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823154d870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823154d890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823154d8f0 seltdwait() at seltdwait+0xdd/frame 0xffffff823154d930 sys_poll() at sys_poll+0x236/frame 0xffffff823154dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823154dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823154dbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command sh pid 59414 tid 100426 td 0xfffffe0057993920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8231773670 mi_switch() at mi_switch+0x238/frame 0xffffff82317736c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8231773700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8231773760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8231773780 _sleep() at _sleep+0x375/frame 0xffffff8231773810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff82317738b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8231773a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8231773ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8231773bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8231773bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffc1f8, rbp = 0x1 --- Tracing command sh pid 58875 tid 101313 td 0xfffffe009c972490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230d15670 mi_switch() at mi_switch+0x238/frame 0xffffff8230d156c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230d15700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230d15760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230d15780 _sleep() at _sleep+0x375/frame 0xffffff8230d15810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff8230d158b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8230d15a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8230d15ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230d15bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230d15bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd538, rbp = 0x1 --- Tracing command tail pid 59543 tid 100413 td 0xfffffe01467e4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823173c690 mi_switch() at mi_switch+0x238/frame 0xffffff823173c6e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823173c720 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823173c780 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823173c7a0 _sleep() at _sleep+0x375/frame 0xffffff823173c830 kern_kevent() at kern_kevent+0x365/frame 0xffffff823173ca10 sys_kevent() at sys_kevent+0x90/frame 0xffffff823173cad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823173cbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823173cbf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800918b2a, rsp = 0x7fffffffd7e8, rbp = 0x8010060a8 --- Tracing command bash pid 13737 tid 101286 td 0xfffffe009c7a5920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230c8e670 mi_switch() at mi_switch+0x238/frame 0xffffff8230c8e6c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230c8e700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230c8e760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230c8e780 _sleep() at _sleep+0x375/frame 0xffffff8230c8e810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff8230c8e8b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8230c8ea10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8230c8ead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230c8ebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230c8ebf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010a12aa, rsp = 0x7fffffffd8c8, rbp = 0x8018ae3e0 --- Tracing command sshd pid 13400 tid 100423 td 0xfffffe000b998000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82317696a0 mi_switch() at mi_switch+0x238/frame 0xffffff82317696f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8231769730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8231769790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82317697b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8231769810 seltdwait() at seltdwait+0xdd/frame 0xffffff8231769850 kern_select() at kern_select+0x68c/frame 0xffffff8231769a80 sys_select() at sys_select+0x5d/frame 0xffffff8231769ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8231769bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8231769bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 9653 tid 102424 td 0xfffffe011b5e4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82322cc780 mi_switch() at mi_switch+0x238/frame 0xffffff82322cc7d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82322cc810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82322cc870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82322cc890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82322cc8f0 seltdwait() at seltdwait+0xdd/frame 0xffffff82322cc930 sys_poll() at sys_poll+0x236/frame 0xffffff82322ccad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82322ccbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82322ccbf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command crypto returns pid 24410 tid 101453 td 0xfffffe0114150490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230fd1a00 mi_switch() at mi_switch+0x238/frame 0xffffff8230fd1a50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230fd1a90 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230fd1ac0 _sleep() at _sleep+0x3eb/frame 0xffffff8230fd1b50 crypto_ret_proc() at crypto_ret_proc+0x1a4/frame 0xffffff8230fd1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230fd1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230fd1bf0 --- trap 0, rip = 0, rsp = 0xffffff8230fd1cb0, rbp = 0 --- Tracing command crypto pid 24409 tid 101484 td 0xfffffe009c972920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823106ca20 mi_switch() at mi_switch+0x238/frame 0xffffff823106ca70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823106cab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823106cae0 _sleep() at _sleep+0x3eb/frame 0xffffff823106cb70 crypto_proc() at crypto_proc+0x1d7/frame 0xffffff823106cba0 fork_exit() at fork_exit+0x139/frame 0xffffff823106cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823106cbf0 --- trap 0, rip = 0, rsp = 0xffffff823106ccb0, rbp = 0 --- Tracing command zfskern pid 40250 tid 100256 td 0xfffffe01467c1490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8231210910 mi_switch() at mi_switch+0x238/frame 0xffffff8231210960 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82312109a0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82312109d0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x1a4/frame 0xffffff8231210a40 l2arc_feed_thread() at l2arc_feed_thread+0x1a9/frame 0xffffff8231210ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8231210bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8231210bf0 --- trap 0, rip = 0, rsp = 0xffffff8231210cb0, rbp = 0 --- Tracing command zfskern pid 40250 tid 101505 td 0xfffffe009c7c5920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82310d59b0 mi_switch() at mi_switch+0x238/frame 0xffffff82310d5a00 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82310d5a40 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82310d5a70 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x1a4/frame 0xffffff82310d5ae0 arc_reclaim_thread() at arc_reclaim_thread+0x2ae/frame 0xffffff82310d5ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82310d5bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82310d5bf0 --- trap 0, rip = 0, rsp = 0xffffff82310d5cb0, rbp = 0 --- Tracing command bash pid 1174 tid 100133 td 0xfffffe0007ec8920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230a04670 mi_switch() at mi_switch+0x238/frame 0xffffff8230a046c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230a04700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230a04760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230a04780 _sleep() at _sleep+0x375/frame 0xffffff8230a04810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff8230a048b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8230a04a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8230a04ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230a04bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230a04bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010a12aa, rsp = 0x7fffffffd678, rbp = 0x8018c3180 --- Tracing command csh pid 1171 tid 100135 td 0xfffffe0007ec8000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230a0e8a0 mi_switch() at mi_switch+0x238/frame 0xffffff8230a0e8f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230a0e930 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230a0e990 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230a0e9b0 _sleep() at _sleep+0x375/frame 0xffffff8230a0ea40 kern_sigsuspend() at kern_sigsuspend+0xb4/frame 0xffffff8230a0eaa0 sys_sigsuspend() at sys_sigsuspend+0x34/frame 0xffffff8230a0ead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230a0ebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230a0ebf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x800d4509a, rsp = 0x7fffffffcda8, rbp = 0x801838400 --- Tracing command su pid 1155 tid 100110 td 0xfffffe000796e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230991670 mi_switch() at mi_switch+0x238/frame 0xffffff82309916c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230991700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230991760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230991780 _sleep() at _sleep+0x375/frame 0xffffff8230991810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff82309918b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8230991a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8230991ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230991bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230991bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800ed42aa, rsp = 0x7fffffffd478, rbp = 0x493 --- Tracing command bash pid 1150 tid 100132 td 0xfffffe009c465000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309ff670 mi_switch() at mi_switch+0x238/frame 0xffffff82309ff6c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309ff700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309ff760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309ff780 _sleep() at _sleep+0x375/frame 0xffffff82309ff810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff82309ff8b0 kern_wait() at kern_wait+0x9c/frame 0xffffff82309ffa10 sys_wait4() at sys_wait4+0x35/frame 0xffffff82309ffad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309ffbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309ffbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8010a12aa, rsp = 0x7fffffffd8c8, rbp = 0x80192fd40 --- Tracing command sshd pid 1149 tid 100131 td 0xfffffe009c465490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309fa6a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309fa6f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309fa730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309fa790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309fa7b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309fa810 seltdwait() at seltdwait+0xdd/frame 0xffffff82309fa850 kern_select() at kern_select+0x68c/frame 0xffffff82309faa80 sys_select() at sys_select+0x5d/frame 0xffffff82309faad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309fabf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309fabf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1147 tid 100129 td 0xfffffe009c470000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309f0780 mi_switch() at mi_switch+0x238/frame 0xffffff82309f07d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309f0810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309f0870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309f0890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309f08f0 seltdwait() at seltdwait+0xdd/frame 0xffffff82309f0930 sys_poll() at sys_poll+0x236/frame 0xffffff82309f0ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309f0bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309f0bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command awk pid 1131 tid 100102 td 0xfffffe00079d3490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230969800 mi_switch() at mi_switch+0x238/frame 0xffffff8230969850 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230969890 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309698f0 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230969910 _sleep() at _sleep+0x375/frame 0xffffff82309699a0 pipe_read() at pipe_read+0x44c/frame 0xffffff82309699f0 dofileread() at dofileread+0xa1/frame 0xffffff8230969a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230969a80 sys_read() at sys_read+0x64/frame 0xffffff8230969ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230969bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230969bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7792a, rsp = 0x7fffffffd848, rbp = 0x800db5d60 --- Tracing command sh pid 1130 tid 100098 td 0xfffffe0007a91920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230955670 mi_switch() at mi_switch+0x238/frame 0xffffff82309556c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230955700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230955760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230955780 _sleep() at _sleep+0x375/frame 0xffffff8230955810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff82309558b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8230955a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8230955ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230955bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230955bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd678, rbp = 0x1 --- Tracing command top pid 1129 tid 100095 td 0xfffffe00079d4490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230946680 mi_switch() at mi_switch+0x238/frame 0xffffff82309466d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230946710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230946770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff82309467a0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff8230946810 seltdwait() at seltdwait+0x77/frame 0xffffff8230946850 kern_select() at kern_select+0x68c/frame 0xffffff8230946a80 sys_select() at sys_select+0x5d/frame 0xffffff8230946ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230946bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230946bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fb98aa, rsp = 0x7fffffffd988, rbp = 0xe --- Tracing command tail pid 1128 tid 100090 td 0xfffffe0007a93920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823092d530 mi_switch() at mi_switch+0x238/frame 0xffffff823092d580 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823092d5c0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823092d5f0 __lockmgr_args() at __lockmgr_args+0x105f/frame 0xffffff823092d6e0 ffs_lock() at ffs_lock+0x9b/frame 0xffffff823092d730 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xab/frame 0xffffff823092d760 _vn_lock() at _vn_lock+0x8e/frame 0xffffff823092d7e0 vn_read() at vn_read+0x8d/frame 0xffffff823092d860 vn_io_fault() at vn_io_fault+0x196/frame 0xffffff823092d9f0 dofileread() at dofileread+0xa1/frame 0xffffff823092da40 kern_readv() at kern_readv+0x6c/frame 0xffffff823092da80 sys_read() at sys_read+0x64/frame 0xffffff823092dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823092dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823092dbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80093792a, rsp = 0x7fffffffd988, rbp = 0x801006020 --- Tracing command sh pid 1127 tid 100096 td 0xfffffe00079d4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823094b670 mi_switch() at mi_switch+0x238/frame 0xffffff823094b6c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823094b700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823094b760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823094b780 _sleep() at _sleep+0x375/frame 0xffffff823094b810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff823094b8b0 kern_wait() at kern_wait+0x9c/frame 0xffffff823094ba10 sys_wait4() at sys_wait4+0x35/frame 0xffffff823094bad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823094bbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823094bbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d302aa, rsp = 0x7fffffffd938, rbp = 0x1 --- Tracing command sshd pid 1126 tid 100116 td 0xfffffe0007eca490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309af6a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309af6f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309af730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309af790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309af7b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309af810 seltdwait() at seltdwait+0xdd/frame 0xffffff82309af850 kern_select() at kern_select+0x68c/frame 0xffffff82309afa80 sys_select() at sys_select+0x5d/frame 0xffffff82309afad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309afbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309afbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1125 tid 100081 td 0xfffffe0007828920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309006a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309006f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230900730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230900790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309007b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230900810 seltdwait() at seltdwait+0xdd/frame 0xffffff8230900850 kern_select() at kern_select+0x68c/frame 0xffffff8230900a80 sys_select() at sys_select+0x5d/frame 0xffffff8230900ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230900bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230900bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1124 tid 100099 td 0xfffffe0007a91490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823095a6a0 mi_switch() at mi_switch+0x238/frame 0xffffff823095a6f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823095a730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823095a790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823095a7b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823095a810 seltdwait() at seltdwait+0xdd/frame 0xffffff823095a850 kern_select() at kern_select+0x68c/frame 0xffffff823095aa80 sys_select() at sys_select+0x5d/frame 0xffffff823095aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823095abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823095abf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcc58, rbp = 0x7fffffffcce0 --- Tracing command sshd pid 1120 tid 100086 td 0xfffffe0007828000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230919780 mi_switch() at mi_switch+0x238/frame 0xffffff82309197d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230919810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230919870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230919890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309198f0 seltdwait() at seltdwait+0xdd/frame 0xffffff8230919930 sys_poll() at sys_poll+0x236/frame 0xffffff8230919ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230919bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230919bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command sshd pid 1119 tid 100118 td 0xfffffe0007ec9920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309b9780 mi_switch() at mi_switch+0x238/frame 0xffffff82309b97d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309b9810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309b9870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309b9890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309b98f0 seltdwait() at seltdwait+0xdd/frame 0xffffff82309b9930 sys_poll() at sys_poll+0x236/frame 0xffffff82309b9ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309b9bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309b9bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command sshd pid 1118 tid 100115 td 0xfffffe000782b490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309aa780 mi_switch() at mi_switch+0x238/frame 0xffffff82309aa7d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309aa810 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309aa870 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309aa890 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309aa8f0 seltdwait() at seltdwait+0xdd/frame 0xffffff82309aa930 sys_poll() at sys_poll+0x236/frame 0xffffff82309aaad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309aabf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309aabf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8024f950a, rsp = 0x7fffffffccf8, rbp = 0x803c22190 --- Tracing command getty pid 1070 tid 100111 td 0xfffffe0007968920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230996710 mi_switch() at mi_switch+0x238/frame 0xffffff8230996760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309967a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230996800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230996820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230996880 tty_wait() at tty_wait+0x4c/frame 0xffffff82309968b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff8230996950 ttydev_read() at ttydev_read+0x95/frame 0xffffff8230996980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82309969f0 dofileread() at dofileread+0xa1/frame 0xffffff8230996a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230996a80 sys_read() at sys_read+0x64/frame 0xffffff8230996ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230996bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230996bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1069 tid 100088 td 0xfffffe000796e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230923710 mi_switch() at mi_switch+0x238/frame 0xffffff8230923760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309237a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230923800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230923820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230923880 tty_wait() at tty_wait+0x4c/frame 0xffffff82309238b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff8230923950 ttydev_read() at ttydev_read+0x95/frame 0xffffff8230923980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82309239f0 dofileread() at dofileread+0xa1/frame 0xffffff8230923a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230923a80 sys_read() at sys_read+0x64/frame 0xffffff8230923ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230923bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230923bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1068 tid 100077 td 0xfffffe0007970490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308ec710 mi_switch() at mi_switch+0x238/frame 0xffffff82308ec760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308ec7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82308ec800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82308ec820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82308ec880 tty_wait() at tty_wait+0x4c/frame 0xffffff82308ec8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff82308ec950 ttydev_read() at ttydev_read+0x95/frame 0xffffff82308ec980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82308ec9f0 dofileread() at dofileread+0xa1/frame 0xffffff82308eca40 kern_readv() at kern_readv+0x6c/frame 0xffffff82308eca80 sys_read() at sys_read+0x64/frame 0xffffff82308ecad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82308ecbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82308ecbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1067 tid 100083 td 0xfffffe000796f920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823090a710 mi_switch() at mi_switch+0x238/frame 0xffffff823090a760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823090a7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823090a800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823090a820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823090a880 tty_wait() at tty_wait+0x4c/frame 0xffffff823090a8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823090a950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823090a980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823090a9f0 dofileread() at dofileread+0xa1/frame 0xffffff823090aa40 kern_readv() at kern_readv+0x6c/frame 0xffffff823090aa80 sys_read() at sys_read+0x64/frame 0xffffff823090aad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823090abf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823090abf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1066 tid 100091 td 0xfffffe0007a93490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230932710 mi_switch() at mi_switch+0x238/frame 0xffffff8230932760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309327a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230932800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230932820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230932880 tty_wait() at tty_wait+0x4c/frame 0xffffff82309328b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff8230932950 ttydev_read() at ttydev_read+0x95/frame 0xffffff8230932980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82309329f0 dofileread() at dofileread+0xa1/frame 0xffffff8230932a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230932a80 sys_read() at sys_read+0x64/frame 0xffffff8230932ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230932bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230932bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1065 tid 100094 td 0xfffffe0007a92490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230941710 mi_switch() at mi_switch+0x238/frame 0xffffff8230941760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309417a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230941800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8230941820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230941880 tty_wait() at tty_wait+0x4c/frame 0xffffff82309418b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff8230941950 ttydev_read() at ttydev_read+0x95/frame 0xffffff8230941980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82309419f0 dofileread() at dofileread+0xa1/frame 0xffffff8230941a40 kern_readv() at kern_readv+0x6c/frame 0xffffff8230941a80 sys_read() at sys_read+0x64/frame 0xffffff8230941ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230941bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230941bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1064 tid 100100 td 0xfffffe0007a91000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823095f710 mi_switch() at mi_switch+0x238/frame 0xffffff823095f760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823095f7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823095f800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823095f820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823095f880 tty_wait() at tty_wait+0x4c/frame 0xffffff823095f8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823095f950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823095f980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823095f9f0 dofileread() at dofileread+0xa1/frame 0xffffff823095fa40 kern_readv() at kern_readv+0x6c/frame 0xffffff823095fa80 sys_read() at sys_read+0x64/frame 0xffffff823095fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823095fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823095fbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1063 tid 100106 td 0xfffffe00079d2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823097d710 mi_switch() at mi_switch+0x238/frame 0xffffff823097d760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823097d7a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823097d800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823097d820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823097d880 tty_wait() at tty_wait+0x4c/frame 0xffffff823097d8b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff823097d950 ttydev_read() at ttydev_read+0x95/frame 0xffffff823097d980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff823097d9f0 dofileread() at dofileread+0xa1/frame 0xffffff823097da40 kern_readv() at kern_readv+0x6c/frame 0xffffff823097da80 sys_read() at sys_read+0x64/frame 0xffffff823097dad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823097dbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823097dbf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command getty pid 1062 tid 100075 td 0xfffffe0007970920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308e2710 mi_switch() at mi_switch+0x238/frame 0xffffff82308e2760 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308e27a0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82308e2800 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82308e2820 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82308e2880 tty_wait() at tty_wait+0x4c/frame 0xffffff82308e28b0 ttydisc_read() at ttydisc_read+0x38e/frame 0xffffff82308e2950 ttydev_read() at ttydev_read+0x95/frame 0xffffff82308e2980 devfs_read_f() at devfs_read_f+0x90/frame 0xffffff82308e29f0 dofileread() at dofileread+0xa1/frame 0xffffff82308e2a40 kern_readv() at kern_readv+0x6c/frame 0xffffff82308e2a80 sys_read() at sys_read+0x64/frame 0xffffff82308e2ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82308e2bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82308e2bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b4b92a, rsp = 0x7fffffffdc98, rbp = 0 --- Tracing command inetd pid 1023 tid 100080 td 0xfffffe000782a000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308fb6a0 mi_switch() at mi_switch+0x238/frame 0xffffff82308fb6f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308fb730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82308fb790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82308fb7b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82308fb810 seltdwait() at seltdwait+0xdd/frame 0xffffff82308fb850 kern_select() at kern_select+0x68c/frame 0xffffff82308fba80 sys_select() at sys_select+0x5d/frame 0xffffff82308fbad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82308fbbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82308fbbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800f608aa, rsp = 0x7fffffffcd68, rbp = 0x1 --- Tracing command moused pid 990 tid 100092 td 0xfffffe0007a93000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309376a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309376f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230937730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230937790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309377b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230937810 seltdwait() at seltdwait+0xdd/frame 0xffffff8230937850 kern_select() at kern_select+0x68c/frame 0xffffff8230937a80 sys_select() at sys_select+0x5d/frame 0xffffff8230937ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230937bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230937bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d728aa, rsp = 0x7fffffffd858, rbp = 0x7fffffffdf31 --- Tracing command watchdogd pid 970 tid 100112 td 0xfffffe0007968490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823099b860 mi_switch() at mi_switch+0x238/frame 0xffffff823099b8b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823099b8f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823099b950 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823099b980 _sleep() at _sleep+0x3bb/frame 0xffffff823099ba10 kern_nanosleep() at kern_nanosleep+0xc6/frame 0xffffff823099ba70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff823099bad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823099bbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823099bbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b27baa, rsp = 0x7fffffffdba8, rbp = 0x7fffffffdbf0 --- Tracing command cron pid 960 tid 100117 td 0xfffffe0007eca000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309b4860 mi_switch() at mi_switch+0x238/frame 0xffffff82309b48b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309b48f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309b4950 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff82309b4980 _sleep() at _sleep+0x3bb/frame 0xffffff82309b4a10 kern_nanosleep() at kern_nanosleep+0xc6/frame 0xffffff82309b4a70 sys_nanosleep() at sys_nanosleep+0x6e/frame 0xffffff82309b4ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309b4bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309b4bf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d39baa, rsp = 0x7fffffffdac8, rbp = 0x3c --- Tracing command sendmail pid 953 tid 100119 td 0xfffffe0007ec9490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309be8a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309be8f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309be930 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309be990 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309be9b0 _sleep() at _sleep+0x375/frame 0xffffff82309bea40 kern_sigsuspend() at kern_sigsuspend+0xb4/frame 0xffffff82309beaa0 sys_sigsuspend() at sys_sigsuspend+0x34/frame 0xffffff82309bead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309bebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309bebf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80139609a, rsp = 0x7fffffffbda8, rbp = 0x1 --- Tracing command sendmail pid 949 tid 100089 td 0xfffffe0007a70000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230928680 mi_switch() at mi_switch+0x238/frame 0xffffff82309286d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230928710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230928770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff82309287a0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff8230928810 seltdwait() at seltdwait+0x77/frame 0xffffff8230928850 kern_select() at kern_select+0x68c/frame 0xffffff8230928a80 sys_select() at sys_select+0x5d/frame 0xffffff8230928ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230928bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230928bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8014498aa, rsp = 0x7fffffffb218, rbp = 0x7fffffffb2b0 --- Tracing command sshd pid 941 tid 100120 td 0xfffffe0007ec9000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309c36a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309c36f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309c3730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309c3790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309c37b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff82309c3810 seltdwait() at seltdwait+0xdd/frame 0xffffff82309c3850 kern_select() at kern_select+0x68c/frame 0xffffff82309c3a80 sys_select() at sys_select+0x5d/frame 0xffffff82309c3ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff82309c3bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82309c3bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80255f8aa, rsp = 0x7fffffffcd78, rbp = 0x2 --- Tracing command ntpd pid 850 tid 100105 td 0xfffffe00079d2490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309786a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309786f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230978730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230978790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309787b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230978810 seltdwait() at seltdwait+0xdd/frame 0xffffff8230978850 kern_select() at kern_select+0x68c/frame 0xffffff8230978a80 sys_select() at sys_select+0x5d/frame 0xffffff8230978ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230978bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230978bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8013a98aa, rsp = 0x7fffffffdbc8, rbp = 0x7fffffffdd08 --- Tracing command nfsd pid 755 tid 100109 td 0xfffffe0007a90000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823098c8d0 mi_switch() at mi_switch+0x238/frame 0xffffff823098c920 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823098c960 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823098c9c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823098c9f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff823098ca60 svc_run_internal() at svc_run_internal+0x8a8/frame 0xffffff823098cb90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff823098cba0 fork_exit() at fork_exit+0x139/frame 0xffffff823098cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823098cbf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 755 tid 100108 td 0xfffffe0007a90490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309878d0 mi_switch() at mi_switch+0x238/frame 0xffffff8230987920 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230987960 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309879c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff82309879f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff8230987a60 svc_run_internal() at svc_run_internal+0x8a8/frame 0xffffff8230987b90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff8230987ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230987bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230987bf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 755 tid 100107 td 0xfffffe0007a90920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309828d0 mi_switch() at mi_switch+0x238/frame 0xffffff8230982920 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230982960 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff82309829c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff82309829f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff8230982a60 svc_run_internal() at svc_run_internal+0x8a8/frame 0xffffff8230982b90 svc_thread_start() at svc_thread_start+0xb/frame 0xffffff8230982ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230982bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230982bf0 --- trap 0xc, rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 755 tid 100101 td 0xfffffe00079d3920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230964040 mi_switch() at mi_switch+0x238/frame 0xffffff8230964090 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309640d0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230964130 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff8230964160 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff82309641d0 svc_run_internal() at svc_run_internal+0x8a8/frame 0xffffff8230964300 svc_run() at svc_run+0x94/frame 0xffffff8230964330 nfsrvd_nfsd() at nfsrvd_nfsd+0x1c7/frame 0xffffff8230964480 nfssvc_nfsd() at nfssvc_nfsd+0x9b/frame 0xffffff8230964ab0 sys_nfssvc() at sys_nfssvc+0xb0/frame 0xffffff8230964ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230964bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230964bf0 --- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x800885cfa, rsp = 0x7fffffffd678, rbp = 0x5 --- Tracing command nfsd pid 754 tid 100097 td 0xfffffe0007a92000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82309506a0 mi_switch() at mi_switch+0x238/frame 0xffffff82309506f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230950730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230950790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff82309507b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff8230950810 seltdwait() at seltdwait+0xdd/frame 0xffffff8230950850 kern_select() at kern_select+0x68c/frame 0xffffff8230950a80 sys_select() at sys_select+0x5d/frame 0xffffff8230950ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230950bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230950bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8009388aa, rsp = 0x7fffffffd928, rbp = 0x7 --- Tracing command mountd pid 745 tid 100084 td 0xfffffe000796f490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823090f6a0 mi_switch() at mi_switch+0x238/frame 0xffffff823090f6f0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823090f730 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823090f790 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff823090f7b0 _cv_wait_sig() at _cv_wait_sig+0x181/frame 0xffffff823090f810 seltdwait() at seltdwait+0xdd/frame 0xffffff823090f850 kern_select() at kern_select+0x68c/frame 0xffffff823090fa80 sys_select() at sys_select+0x5d/frame 0xffffff823090fad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823090fbf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823090fbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b508aa, rsp = 0x7fffffffdb78, rbp = 0x801419060 --- Tracing command rpcbind pid 640 tid 100085 td 0xfffffe000796f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230914760 mi_switch() at mi_switch+0x238/frame 0xffffff82309147b0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82309147f0 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8230914850 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff8230914880 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff82309148f0 seltdwait() at seltdwait+0x77/frame 0xffffff8230914930 sys_poll() at sys_poll+0x236/frame 0xffffff8230914ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230914bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230914bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800cf350a, rsp = 0x7fffffffba48, rbp = 0x80141b020 --- Tracing command syslogd pid 614 tid 100082 td 0xfffffe0007828490 kdb_enter() at kdb_enter+0x3b/frame 0xffffff8230905650 vpanic() at vpanic+0xe1/frame 0xffffff8230905690 panic() at panic+0x67/frame 0xffffff8230905770 softdep_deallocate_dependencies() at softdep_deallocate_dependencies+0x72/frame 0xffffff8230905790 brelse() at brelse+0x83/frame 0xffffff8230905810 bufdone() at bufdone+0x78/frame 0xffffff8230905830 ufs_strategy() at ufs_strategy+0x45/frame 0xffffff8230905860 VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0xd1/frame 0xffffff8230905890 bufstrategy() at bufstrategy+0x4b/frame 0xffffff82309058d0 bufwrite() at bufwrite+0x125/frame 0xffffff8230905900 vfs_bio_awrite() at vfs_bio_awrite+0xa3/frame 0xffffff8230905980 ffs_syncvnode() at ffs_syncvnode+0x34a/frame 0xffffff8230905a00 ffs_fsync() at ffs_fsync+0x54/frame 0xffffff8230905a40 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xc9/frame 0xffffff8230905a70 sys_fsync() at sys_fsync+0x150/frame 0xffffff8230905ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8230905bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8230905bf0 --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x800b4c38a, rsp = 0x7fffffffd118, rbp = 0x80142c108 --- Tracing command devd pid 433 tid 100087 td 0xfffffe000796e920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823091e680 mi_switch() at mi_switch+0x238/frame 0xffffff823091e6d0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823091e710 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff823091e770 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19/frame 0xffffff823091e7a0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1a4/frame 0xffffff823091e810 seltdwait() at seltdwait+0x77/frame 0xffffff823091e850 kern_select() at kern_select+0x68c/frame 0xffffff823091ea80 sys_select() at sys_select+0x5d/frame 0xffffff823091ead0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff823091ebf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff823091ebf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x442aaa, rsp = 0x7fffffffd828, rbp = 0x10 --- Tracing command softdepflush pid 17 tid 100074 td 0xfffffe000782a920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308dda10 mi_switch() at mi_switch+0x238/frame 0xffffff82308dda60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308ddaa0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308ddad0 _sleep() at _sleep+0x294/frame 0xffffff82308ddb60 softdep_flush() at softdep_flush+0x21f/frame 0xffffff82308ddba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308ddbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308ddbf0 --- trap 0, rip = 0, rsp = 0xffffff82308ddcb0, rbp = 0 --- Tracing command syncer pid 16 tid 100073 td 0xfffffe000782b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308d89e0 mi_switch() at mi_switch+0x238/frame 0xffffff82308d8a30 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308d8a70 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308d8aa0 _cv_timedwait_sbt() at _cv_timedwait_sbt+0x1a4/frame 0xffffff82308d8b10 sched_sync() at sched_sync+0x4ff/frame 0xffffff82308d8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308d8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308d8bf0 --- trap 0, rip = 0, rsp = 0xffffff82308d8cb0, rbp = 0 --- Tracing command vnlru pid 9 tid 100072 td 0xfffffe000703c000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308d39c0 mi_switch() at mi_switch+0x238/frame 0xffffff82308d3a10 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308d3a50 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308d3a80 _sleep() at _sleep+0x294/frame 0xffffff82308d3b10 vnlru_proc() at vnlru_proc+0x55e/frame 0xffffff82308d3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308d3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308d3bf0 --- trap 0, rip = 0, rsp = 0xffffff82308d3cb0, rbp = 0 --- Tracing command bufdaemon pid 8 tid 100071 td 0xfffffe000703c490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308cea30 mi_switch() at mi_switch+0x238/frame 0xffffff82308cea80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308ceac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308ceaf0 _sleep() at _sleep+0x294/frame 0xffffff82308ceb80 buf_daemon() at buf_daemon+0x1ba/frame 0xffffff82308ceba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308cebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308cebf0 --- trap 0, rip = 0, rsp = 0xffffff82308cecb0, rbp = 0 --- Tracing command pagezero pid 7 tid 100070 td 0xfffffe000703c920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308c9a30 mi_switch() at mi_switch+0x238/frame 0xffffff82308c9a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308c9ac0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308c9af0 _sleep() at _sleep+0x294/frame 0xffffff82308c9b80 vm_pagezero() at vm_pagezero+0x88/frame 0xffffff82308c9ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308c9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308c9bf0 --- trap 0, rip = 0, rsp = 0xffffff82308c9cb0, rbp = 0 --- Tracing command vmdaemon pid 6 tid 100069 td 0xfffffe000703e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308c49d0 mi_switch() at mi_switch+0x238/frame 0xffffff82308c4a20 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308c4a60 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82308c4a90 _sleep() at _sleep+0x3eb/frame 0xffffff82308c4b20 vm_daemon() at vm_daemon+0x57/frame 0xffffff82308c4ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308c4bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308c4bf0 --- trap 0, rip = 0, rsp = 0xffffff82308c4cb0, rbp = 0 --- Tracing command pagedaemon pid 5 tid 100068 td 0xfffffe000703e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82308bf930 mi_switch() at mi_switch+0x238/frame 0xffffff82308bf980 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82308bf9c0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff82308bf9f0 _sleep() at _sleep+0x294/frame 0xffffff82308bfa80 vm_pageout() at vm_pageout+0xbbf/frame 0xffffff82308bfba0 fork_exit() at fork_exit+0x139/frame 0xffffff82308bfbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82308bfbf0 --- trap 0, rip = 0, rsp = 0xffffff82308bfcb0, rbp = 0 --- Tracing command xpt_thrd pid 4 tid 100067 td 0xfffffe000703e920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823073aa20 mi_switch() at mi_switch+0x238/frame 0xffffff823073aa70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823073aab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823073aae0 _sleep() at _sleep+0x3eb/frame 0xffffff823073ab70 xpt_scanner_thread() at xpt_scanner_thread+0xf2/frame 0xffffff823073aba0 fork_exit() at fork_exit+0x139/frame 0xffffff823073abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823073abf0 --- trap 0, rip = 0, rsp = 0xffffff823073acb0, rbp = 0 --- Tracing command sctp_iterator pid 3 tid 100066 td 0xfffffe0007040000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230735a30 mi_switch() at mi_switch+0x238/frame 0xffffff8230735a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230735ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230735af0 _sleep() at _sleep+0x3eb/frame 0xffffff8230735b80 sctp_iterator_thread() at sctp_iterator_thread+0x4b/frame 0xffffff8230735ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230735bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230735bf0 --- trap 0, rip = 0, rsp = 0xffffff8230735cb0, rbp = 0 --- Tracing command fdc0 pid 2 tid 100061 td 0xfffffe0007041920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230717990 mi_switch() at mi_switch+0x238/frame 0xffffff82307179e0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230717a20 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff8230717a50 _sleep() at _sleep+0x294/frame 0xffffff8230717ae0 fdc_thread() at fdc_thread+0x846/frame 0xffffff8230717ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230717bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230717bf0 --- trap 0, rip = 0, rsp = 0xffffff8230717cb0, rbp = 0 --- Tracing command usb pid 15 tid 100058 td 0xfffffe0007016000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82306d1a50 mi_switch() at mi_switch+0x238/frame 0xffffff82306d1aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82306d1ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82306d1b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82306d1b70 usb_process() at usb_process+0x172/frame 0xffffff82306d1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82306d1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82306d1bf0 --- trap 0, rip = 0, rsp = 0xffffff82306d1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100057 td 0xfffffe0007016490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82306cca50 mi_switch() at mi_switch+0x238/frame 0xffffff82306ccaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82306ccae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82306ccb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82306ccb70 usb_process() at usb_process+0x172/frame 0xffffff82306ccba0 fork_exit() at fork_exit+0x139/frame 0xffffff82306ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82306ccbf0 --- trap 0, rip = 0, rsp = 0xffffff82306cccb0, rbp = 0 --- Tracing command usb pid 15 tid 100056 td 0xfffffe0007016920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82306c7a50 mi_switch() at mi_switch+0x238/frame 0xffffff82306c7aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82306c7ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82306c7b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82306c7b70 usb_process() at usb_process+0x172/frame 0xffffff82306c7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82306c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82306c7bf0 --- trap 0, rip = 0, rsp = 0xffffff82306c7cb0, rbp = 0 --- Tracing command usb pid 15 tid 100055 td 0xfffffe0007017000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82306c2a50 mi_switch() at mi_switch+0x238/frame 0xffffff82306c2aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82306c2ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82306c2b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82306c2b70 usb_process() at usb_process+0x172/frame 0xffffff82306c2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82306c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82306c2bf0 --- trap 0, rip = 0, rsp = 0xffffff82306c2cb0, rbp = 0 --- Tracing command usb pid 15 tid 100053 td 0xfffffe0007017920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230535a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230535aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230535ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230535b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230535b70 usb_process() at usb_process+0x172/frame 0xffffff8230535ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230535bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230535bf0 --- trap 0, rip = 0, rsp = 0xffffff8230535cb0, rbp = 0 --- Tracing command usb pid 15 tid 100052 td 0xfffffe0007018000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230530a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230530aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230530ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230530b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230530b70 usb_process() at usb_process+0x172/frame 0xffffff8230530ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230530bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230530bf0 --- trap 0, rip = 0, rsp = 0xffffff8230530cb0, rbp = 0 --- Tracing command usb pid 15 tid 100051 td 0xfffffe0007018490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823052ba50 mi_switch() at mi_switch+0x238/frame 0xffffff823052baa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823052bae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823052bb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823052bb70 usb_process() at usb_process+0x172/frame 0xffffff823052bba0 fork_exit() at fork_exit+0x139/frame 0xffffff823052bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823052bbf0 --- trap 0, rip = 0, rsp = 0xffffff823052bcb0, rbp = 0 --- Tracing command usb pid 15 tid 100050 td 0xfffffe0007018920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230526a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230526aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230526ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230526b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230526b70 usb_process() at usb_process+0x172/frame 0xffffff8230526ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230526bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230526bf0 --- trap 0, rip = 0, rsp = 0xffffff8230526cb0, rbp = 0 --- Tracing command usb pid 15 tid 100049 td 0xfffffe0007009000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82304dda50 mi_switch() at mi_switch+0x238/frame 0xffffff82304ddaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82304ddae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82304ddb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82304ddb70 usb_process() at usb_process+0x172/frame 0xffffff82304ddba0 fork_exit() at fork_exit+0x139/frame 0xffffff82304ddbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82304ddbf0 --- trap 0, rip = 0, rsp = 0xffffff82304ddcb0, rbp = 0 --- Tracing command usb pid 15 tid 100048 td 0xfffffe0007009490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82304d8a50 mi_switch() at mi_switch+0x238/frame 0xffffff82304d8aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82304d8ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82304d8b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82304d8b70 usb_process() at usb_process+0x172/frame 0xffffff82304d8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82304d8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82304d8bf0 --- trap 0, rip = 0, rsp = 0xffffff82304d8cb0, rbp = 0 --- Tracing command usb pid 15 tid 100047 td 0xfffffe0007009920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82304d3a50 mi_switch() at mi_switch+0x238/frame 0xffffff82304d3aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82304d3ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82304d3b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82304d3b70 usb_process() at usb_process+0x172/frame 0xffffff82304d3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82304d3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82304d3bf0 --- trap 0, rip = 0, rsp = 0xffffff82304d3cb0, rbp = 0 --- Tracing command usb pid 15 tid 100046 td 0xfffffe000700a000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82304cea50 mi_switch() at mi_switch+0x238/frame 0xffffff82304ceaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82304ceae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82304ceb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82304ceb70 usb_process() at usb_process+0x172/frame 0xffffff82304ceba0 fork_exit() at fork_exit+0x139/frame 0xffffff82304cebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82304cebf0 --- trap 0, rip = 0, rsp = 0xffffff82304cecb0, rbp = 0 --- Tracing command usb pid 15 tid 100045 td 0xfffffe000700a490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230485a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230485aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230485ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230485b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230485b70 usb_process() at usb_process+0x172/frame 0xffffff8230485ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230485bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230485bf0 --- trap 0, rip = 0, rsp = 0xffffff8230485cb0, rbp = 0 --- Tracing command usb pid 15 tid 100044 td 0xfffffe000700a920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230480a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230480aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230480ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230480b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230480b70 usb_process() at usb_process+0x172/frame 0xffffff8230480ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230480bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230480bf0 --- trap 0, rip = 0, rsp = 0xffffff8230480cb0, rbp = 0 --- Tracing command usb pid 15 tid 100043 td 0xfffffe000700e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff823047ba50 mi_switch() at mi_switch+0x238/frame 0xffffff823047baa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823047bae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823047bb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823047bb70 usb_process() at usb_process+0x172/frame 0xffffff823047bba0 fork_exit() at fork_exit+0x139/frame 0xffffff823047bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823047bbf0 --- trap 0, rip = 0, rsp = 0xffffff823047bcb0, rbp = 0 --- Tracing command usb pid 15 tid 100042 td 0xfffffe000700e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230476a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230476aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230476ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230476b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230476b70 usb_process() at usb_process+0x172/frame 0xffffff8230476ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230476bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230476bf0 --- trap 0, rip = 0, rsp = 0xffffff8230476cb0, rbp = 0 --- Tracing command usb pid 15 tid 100040 td 0xfffffe000700f000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230428a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230428aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230428ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230428b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230428b70 usb_process() at usb_process+0x172/frame 0xffffff8230428ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230428bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230428bf0 --- trap 0, rip = 0, rsp = 0xffffff8230428cb0, rbp = 0 --- Tracing command usb pid 15 tid 100039 td 0xfffffe0007003000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230423a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230423aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230423ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230423b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230423b70 usb_process() at usb_process+0x172/frame 0xffffff8230423ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230423bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230423bf0 --- trap 0, rip = 0, rsp = 0xffffff8230423cb0, rbp = 0 --- Tracing command usb pid 15 tid 100038 td 0xfffffe0007003490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823041ea50 mi_switch() at mi_switch+0x238/frame 0xffffff823041eaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823041eae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823041eb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff823041eb70 usb_process() at usb_process+0x172/frame 0xffffff823041eba0 fork_exit() at fork_exit+0x139/frame 0xffffff823041ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823041ebf0 --- trap 0, rip = 0, rsp = 0xffffff823041ecb0, rbp = 0 --- Tracing command usb pid 15 tid 100037 td 0xfffffe0007003920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230419a50 mi_switch() at mi_switch+0x238/frame 0xffffff8230419aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8230419ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8230419b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8230419b70 usb_process() at usb_process+0x172/frame 0xffffff8230419ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230419bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230419bf0 --- trap 0, rip = 0, rsp = 0xffffff8230419cb0, rbp = 0 --- Tracing command usb pid 15 tid 100035 td 0xfffffe0007004490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82303cba50 mi_switch() at mi_switch+0x238/frame 0xffffff82303cbaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82303cbae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82303cbb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82303cbb70 usb_process() at usb_process+0x172/frame 0xffffff82303cbba0 fork_exit() at fork_exit+0x139/frame 0xffffff82303cbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82303cbbf0 --- trap 0, rip = 0, rsp = 0xffffff82303cbcb0, rbp = 0 --- Tracing command usb pid 15 tid 100034 td 0xfffffe0007004920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82303c6a50 mi_switch() at mi_switch+0x238/frame 0xffffff82303c6aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82303c6ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82303c6b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82303c6b70 usb_process() at usb_process+0x172/frame 0xffffff82303c6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82303c6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82303c6bf0 --- trap 0, rip = 0, rsp = 0xffffff82303c6cb0, rbp = 0 --- Tracing command usb pid 15 tid 100033 td 0xfffffe0007005000 sched_switch() at sched_switch+0x1b4/frame 0xffffff82303c1a50 mi_switch() at mi_switch+0x238/frame 0xffffff82303c1aa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82303c1ae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82303c1b10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82303c1b70 usb_process() at usb_process+0x172/frame 0xffffff82303c1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82303c1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82303c1bf0 --- trap 0, rip = 0, rsp = 0xffffff82303c1cb0, rbp = 0 --- Tracing command usb pid 15 tid 100032 td 0xfffffe0007005490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82303bca50 mi_switch() at mi_switch+0x238/frame 0xffffff82303bcaa0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82303bcae0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82303bcb10 _cv_wait() at _cv_wait+0x17d/frame 0xffffff82303bcb70 usb_process() at usb_process+0x172/frame 0xffffff82303bcba0 fork_exit() at fork_exit+0x139/frame 0xffffff82303bcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82303bcbf0 --- trap 0, rip = 0, rsp = 0xffffff82303bccb0, rbp = 0 --- Tracing command yarrow pid 14 tid 100017 td 0xfffffe000522c920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a3a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002a3a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002a3ab0 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff80002a3ae0 msleep_spin_sbt() at msleep_spin_sbt+0x17f/frame 0xffffff80002a3b60 random_kthread() at random_kthread+0x190/frame 0xffffff80002a3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a3bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a3cb0, rbp = 0 --- Tracing command geom pid 13 tid 100015 td 0xfffffe000521b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002999e0 mi_switch() at mi_switch+0x238/frame 0xffffff8000299a30 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000299a70 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000299aa0 _sleep() at _sleep+0x3eb/frame 0xffffff8000299b30 g_io_schedule_down() at g_io_schedule_down+0x24b/frame 0xffffff8000299b90 g_down_procbody() at g_down_procbody+0x7c/frame 0xffffff8000299ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000299bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000299bf0 --- trap 0, rip = 0, rsp = 0xffffff8000299cb0, rbp = 0 --- Tracing command geom pid 13 tid 100014 td 0xfffffe000521b490 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffffff81583a60 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffffff81583a80 trap() at trap+0x325/frame 0xffffffff81583c80 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff81583c80 --- trap 0x13, rip = 0xffffffff80932681, rsp = 0xffffffff81583d40, rbp = 0xffffff8000294850 --- find_instance() at find_instance+0x61/frame 0xffffff8000294850 witness_assert() at witness_assert+0xd7/frame 0xffffff80002948a0 _rw_runlock_cookie() at _rw_runlock_cookie+0x4f/frame 0xffffff80002948d0 pmap_page_is_mapped() at pmap_page_is_mapped+0x8f/frame 0xffffff8000294900 vm_page_free_toq() at vm_page_free_toq+0x45/frame 0xffffff8000294920 vfs_vmio_release() at vfs_vmio_release+0x26e/frame 0xffffff8000294950 brelse() at brelse+0x6cb/frame 0xffffff80002949d0 softdep_disk_write_complete() at softdep_disk_write_complete+0x12e/frame 0xffffff8000294aa0 bufdone_finish() at bufdone_finish+0x2c/frame 0xffffff8000294b10 bufdone() at bufdone+0x78/frame 0xffffff8000294b30 biodone() at biodone+0x8c/frame 0xffffff8000294b70 g_io_schedule_up() at g_io_schedule_up+0xa8/frame 0xffffff8000294b90 g_up_procbody() at g_up_procbody+0x7c/frame 0xffffff8000294ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000294bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000294bf0 --- trap 0, rip = 0, rsp = 0xffffff8000294cb0, rbp = 0 --- Tracing command geom pid 13 tid 100013 td 0xfffffe000521b920 sched_switch() at sched_switch+0x1b4/frame 0xffffff800028fa10 mi_switch() at mi_switch+0x238/frame 0xffffff800028fa60 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800028faa0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800028fad0 _sleep() at _sleep+0x3eb/frame 0xffffff800028fb60 g_run_events() at g_run_events+0x45f/frame 0xffffff800028fba0 fork_exit() at fork_exit+0x139/frame 0xffffff800028fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800028fbf0 --- trap 0, rip = 0, rsp = 0xffffff800028fcb0, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xfffffe0007041000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xfffffe0007041490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823071cb00 mi_switch() at mi_switch+0x238/frame 0xffffff823071cb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff823071cba0 fork_exit() at fork_exit+0x139/frame 0xffffff823071cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823071cbf0 --- trap 0, rip = 0, rsp = 0xffffff823071ccb0, rbp = 0 --- Tracing command intr pid 12 tid 100060 td 0xfffffe000700f490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230708b00 mi_switch() at mi_switch+0x238/frame 0xffffff8230708b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8230708ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230708bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230708bf0 --- trap 0, rip = 0, rsp = 0xffffff8230708cb0, rbp = 0 --- Tracing command intr pid 12 tid 100059 td 0xfffffe000700f920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82306f9b00 mi_switch() at mi_switch+0x238/frame 0xffffff82306f9b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff82306f9ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82306f9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82306f9bf0 --- trap 0, rip = 0, rsp = 0xffffff82306f9cb0, rbp = 0 --- Tracing command intr pid 12 tid 100054 td 0xfffffe0007017490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100041 td 0xfffffe000700e920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230471b00 mi_switch() at mi_switch+0x238/frame 0xffffff8230471b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8230471ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230471bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230471bf0 --- trap 0, rip = 0, rsp = 0xffffff8230471cb0, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xfffffe0007004000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8230414b00 mi_switch() at mi_switch+0x238/frame 0xffffff8230414b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8230414ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8230414bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8230414bf0 --- trap 0, rip = 0, rsp = 0xffffff8230414cb0, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xfffffe0007005920 sched_switch() at sched_switch+0x1b4/frame 0xffffff82303b7b00 mi_switch() at mi_switch+0x238/frame 0xffffff82303b7b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff82303b7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff82303b7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82303b7bf0 --- trap 0, rip = 0, rsp = 0xffffff82303b7cb0, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xfffffe000522e490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003ecb00 mi_switch() at mi_switch+0x238/frame 0xffffff80003ecb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80003ecba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003ecbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003ecbf0 --- trap 0, rip = 0, rsp = 0xffffff80003eccb0, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xfffffe000522e920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80003e7b00 mi_switch() at mi_switch+0x238/frame 0xffffff80003e7b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80003e7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80003e7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80003e7bf0 --- trap 0, rip = 0, rsp = 0xffffff80003e7cb0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xfffffe00053c2000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002dbb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002dbb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002dbba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002dbbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002dbbf0 --- trap 0, rip = 0, rsp = 0xffffff80002dbcb0, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xfffffe00053c2490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002d6b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002d6b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002d6ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002d6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d6bf0 --- trap 0, rip = 0, rsp = 0xffffff80002d6cb0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xfffffe000522b000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002bdb00 mi_switch() at mi_switch+0x238/frame 0xffffff80002bdb50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002bdba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002bdbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bdbf0 --- trap 0, rip = 0, rsp = 0xffffff80002bdcb0, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xfffffe000522b490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b8b00 mi_switch() at mi_switch+0x238/frame 0xffffff80002b8b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff80002b8ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b8bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b8cb0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xfffffe000522c000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xfffffe000521c000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800028ab00 mi_switch() at mi_switch+0x238/frame 0xffffff800028ab50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff800028aba0 fork_exit() at fork_exit+0x139/frame 0xffffff800028abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800028abf0 --- trap 0, rip = 0, rsp = 0xffffff800028acb0, rbp = 0 --- Tracing command intr pid 12 tid 100011 td 0xfffffe000521c490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100010 td 0xfffffe000521c920 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100009 td 0xfffffe00051f7490 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100008 td 0xfffffe00051f7920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000276b00 mi_switch() at mi_switch+0x238/frame 0xffffff8000276b50 ithread_loop() at ithread_loop+0x23e/frame 0xffffff8000276ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000276bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000276bf0 --- trap 0, rip = 0, rsp = 0xffffff8000276cb0, rbp = 0 --- Tracing command intr pid 12 tid 100007 td 0xfffffe0005203000 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100006 td 0xfffffe0005203490 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800024cd00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800024cd20 trap() at trap+0x325/frame 0xffffff800024cf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800024cf20 --- trap 0x13, rip = 0xffffffff8090e757, rsp = 0xffffff800024cfe0, rbp = 0xffffff800026ca10 --- sched_clock() at sched_clock+0x117/frame 0xffffff800026ca10 statclock_cnt() at statclock_cnt+0xea/frame 0xffffff800026ca60 handleevents() at handleevents+0x26f/frame 0xffffff800026cab0 cpu_activeclock() at cpu_activeclock+0x64/frame 0xffffff800026caf0 cpu_idle() at cpu_idle+0x7b/frame 0xffffff800026cb10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff800026cba0 fork_exit() at fork_exit+0x139/frame 0xffffff800026cbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800026cbf0 --- trap 0, rip = 0, rsp = 0xffffff800026ccb0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xfffffe0005203920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000267aa0 mi_switch() at mi_switch+0x238/frame 0xffffff8000267af0 critical_exit() at critical_exit+0xb0/frame 0xffffff8000267b10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff8000267ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000267bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000267bf0 --- trap 0, rip = 0, rsp = 0xffffff8000267cb0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffffe00051f6000 cpustop_handler() at cpustop_handler+0x2c/frame 0xffffff800023ed00 ipi_nmi_handler() at ipi_nmi_handler+0x3d/frame 0xffffff800023ed20 trap() at trap+0x325/frame 0xffffff800023ef20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffff800023ef20 --- trap 0x13, rip = 0xffffffff80c70079, rsp = 0xffffff800023efe0, rbp = 0xffffff8000262aa0 --- ctx_switch_xsave() at ctx_switch_xsave+0x31/frame 0xffffff8000262aa0 mi_switch() at mi_switch+0x238/frame 0xffffff8000262af0 critical_exit() at critical_exit+0xb0/frame 0xffffff8000262b10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff8000262ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000262bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000262bf0 --- trap 0, rip = 0, rsp = 0xffffff8000262cb0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffffe00051f6490 sched_switch() at sched_switch+0x1b4/frame 0xffffff800025daa0 mi_switch() at mi_switch+0x238/frame 0xffffff800025daf0 critical_exit() at critical_exit+0xb0/frame 0xffffff800025db10 sched_idletd() at sched_idletd+0x15e/frame 0xffffff800025dba0 fork_exit() at fork_exit+0x139/frame 0xffffff800025dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800025dbf0 --- trap 0, rip = 0, rsp = 0xffffff800025dcb0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffffe00051f6920 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000258670 mi_switch() at mi_switch+0x238/frame 0xffffff80002586c0 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000258700 sleepq_catch_signals() at sleepq_catch_signals+0x2c6/frame 0xffffff8000258760 sleepq_wait_sig() at sleepq_wait_sig+0x16/frame 0xffffff8000258780 _sleep() at _sleep+0x375/frame 0xffffff8000258810 kern_wait6() at kern_wait6+0x5fb/frame 0xffffff80002588b0 kern_wait() at kern_wait+0x9c/frame 0xffffff8000258a10 sys_wait4() at sys_wait4+0x35/frame 0xffffff8000258ad0 amd64_syscall() at amd64_syscall+0x2d3/frame 0xffffff8000258bf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff8000258bf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41242a, rsp = 0x7fffffffd798, rbp = 0x89 --- Tracing command audit pid 10 tid 100001 td 0xfffffe00051f7000 sched_switch() at sched_switch+0x1b4/frame 0xffffff8000253a00 mi_switch() at mi_switch+0x238/frame 0xffffff8000253a50 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8000253a90 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8000253ac0 _cv_wait() at _cv_wait+0x17d/frame 0xffffff8000253b20 audit_worker() at audit_worker+0x77/frame 0xffffff8000253ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8000253bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8000253bf0 --- trap 0, rip = 0, rsp = 0xffffff8000253cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100255 td 0xfffffe0049715920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823120ba20 mi_switch() at mi_switch+0x238/frame 0xffffff823120ba70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823120bab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823120bae0 _sleep() at _sleep+0x3eb/frame 0xffffff823120bb70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff823120bba0 fork_exit() at fork_exit+0x139/frame 0xffffff823120bbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823120bbf0 --- trap 0, rip = 0, rsp = 0xffffff823120bcb0, rbp = 0 --- Tracing command kernel pid 0 tid 100211 td 0xfffffe01d4dfe490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8231206a20 mi_switch() at mi_switch+0x238/frame 0xffffff8231206a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8231206ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8231206ae0 _sleep() at _sleep+0x3eb/frame 0xffffff8231206b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff8231206ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8231206bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8231206bf0 --- trap 0, rip = 0, rsp = 0xffffff8231206cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100191 td 0xfffffe0146264490 sched_switch() at sched_switch+0x1b4/frame 0xffffff8231201a20 mi_switch() at mi_switch+0x238/frame 0xffffff8231201a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff8231201ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff8231201ae0 _sleep() at _sleep+0x3eb/frame 0xffffff8231201b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff8231201ba0 fork_exit() at fork_exit+0x139/frame 0xffffff8231201bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff8231201bf0 --- trap 0, rip = 0, rsp = 0xffffff8231201cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100176 td 0xfffffe004952f490 sched_switch() at sched_switch+0x1b4/frame 0xffffff82311fca20 mi_switch() at mi_switch+0x238/frame 0xffffff82311fca70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff82311fcab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff82311fcae0 _sleep() at _sleep+0x3eb/frame 0xffffff82311fcb70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff82311fcba0 fork_exit() at fork_exit+0x139/frame 0xffffff82311fcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff82311fcbf0 --- trap 0, rip = 0, rsp = 0xffffff82311fccb0, rbp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xfffffe0007040490 sched_switch() at sched_switch+0x1b4/frame 0xffffff823072f9d0 mi_switch() at mi_switch+0x238/frame 0xffffff823072fa20 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823072fa60 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffff823072fa90 _sleep() at _sleep+0x294/frame 0xffffff823072fb20 pause_sbt() at pause_sbt+0xb6/frame 0xffffff823072fb60 deadlkres() at deadlkres+0x2e5/frame 0xffffff823072fba0 fork_exit() at fork_exit+0x139/frame 0xffffff823072fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823072fbf0 --- trap 0, rip = 0, rsp = 0xffffff823072fcb0, rbp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xfffffe0007040920 sched_switch() at sched_switch+0x1b4/frame 0xffffff823072aa30 mi_switch() at mi_switch+0x238/frame 0xffffff823072aa80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff823072aac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff823072aaf0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff823072ab70 taskqueue_thread_loop() at taskqueue_thread_loop+0x78/frame 0xffffff823072aba0 fork_exit() at fork_exit+0x139/frame 0xffffff823072abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff823072abf0 --- trap 0, rip = 0, rsp = 0xffffff823072acb0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffffe00053c2920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002d1a30 mi_switch() at mi_switch+0x238/frame 0xffffff80002d1a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002d1ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002d1af0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002d1b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x78/frame 0xffffff80002d1ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002d1bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002d1bf0 --- trap 0, rip = 0, rsp = 0xffffff80002d1cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xfffffe00053c4000 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002cca30 mi_switch() at mi_switch+0x238/frame 0xffffff80002cca80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002ccac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002ccaf0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002ccb70 taskqueue_thread_loop() at taskqueue_thread_loop+0x78/frame 0xffffff80002ccba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002ccbf0 --- trap 0, rip = 0, rsp = 0xffffff80002cccb0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffffe00053c4490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002c7a30 mi_switch() at mi_switch+0x238/frame 0xffffff80002c7a80 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002c7ac0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002c7af0 msleep_spin_sbt() at msleep_spin_sbt+0x22d/frame 0xffffff80002c7b70 taskqueue_thread_loop() at taskqueue_thread_loop+0x78/frame 0xffffff80002c7ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002c7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002c7bf0 --- trap 0, rip = 0, rsp = 0xffffff80002c7cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100023 td 0xfffffe00053c4920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002c2a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002c2a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002c2ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002c2ae0 _sleep() at _sleep+0x3eb/frame 0xffffff80002c2b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff80002c2ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002c2bf0 --- trap 0, rip = 0, rsp = 0xffffff80002c2cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffffe000522b920 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002b3a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002b3a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002b3ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002b3ae0 _sleep() at _sleep+0x3eb/frame 0xffffff80002b3b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff80002b3ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002b3bf0 --- trap 0, rip = 0, rsp = 0xffffff80002b3cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffffe000522c490 sched_switch() at sched_switch+0x1b4/frame 0xffffff80002a9a20 mi_switch() at mi_switch+0x238/frame 0xffffff80002a9a70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff80002a9ab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff80002a9ae0 _sleep() at _sleep+0x3eb/frame 0xffffff80002a9b70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff80002a9ba0 fork_exit() at fork_exit+0x139/frame 0xffffff80002a9bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002a9bf0 --- trap 0, rip = 0, rsp = 0xffffff80002a9cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xfffffe000522e000 sched_switch() at sched_switch+0x1b4/frame 0xffffff800029ea20 mi_switch() at mi_switch+0x238/frame 0xffffff800029ea70 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffff800029eab0 sleepq_wait() at sleepq_wait+0x4d/frame 0xffffff800029eae0 _sleep() at _sleep+0x3eb/frame 0xffffff800029eb70 taskqueue_thread_loop() at taskqueue_thread_loop+0xd6/frame 0xffffff800029eba0 fork_exit() at fork_exit+0x139/frame 0xffffff800029ebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff800029ebf0 --- trap 0, rip = 0, rsp = 0xffffff800029ecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff81364220 sched_switch() at sched_switch+0x1b4/frame 0xffffffff81901af0 mi_switch() at mi_switch+0x238/frame 0xffffffff81901b40 sleepq_switch() at sleepq_switch+0xfe/frame 0xffffffff81901b80 sleepq_timedwait() at sleepq_timedwait+0x4d/frame 0xffffffff81901bb0 _sleep() at _sleep+0x294/frame 0xffffffff81901c40 scheduler() at scheduler+0x2c4/frame 0xffffffff81901c90 mi_startup() at mi_startup+0x77/frame 0xffffffff81901cb0 btext() at btext+0x2c db:0:allt> call doadump Dumping 1885 out of 8040 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Dump complete = 0 db:0:doadump> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 2 (kgdb) bt #0 doadump (textdump=0x51f5000) at ../../../kern/kern_shutdown.c:263 #1 0xffffffff80343edc in db_fncall (dummy1=, dummy2=, dummy3=, dummy4=) at ../../../ddb/db_command.c:578 #2 0xffffffff8034418d in db_command (last_cmdp=0xffffffff8132b660, cmd_table=, dopager=0x0) at ../../../ddb/db_command.c:449 #3 0xffffffff80348d23 in db_script_exec (scriptname=0xffffffff8132bf00 "doadump", warnifnotfound=0x1) at ../../../ddb/db_script.c:302 #4 0xffffffff80344211 in db_command (last_cmdp=0xffffffff8132b660, cmd_table=, dopager=0x1) at ../../../ddb/db_command.c:449 #5 0xffffffff80344460 in db_command_loop () at ../../../ddb/db_command.c:502 #6 0xffffffff803465d9 in db_trap (type=, code=) at ../../../ddb/db_main.c:231 #7 0xffffffff8091df68 in kdb_trap (type=0x3, code=0x0, tf=0xffffff8230905580) at ../../../kern/subr_kdb.c:654 #8 0xffffffff80c8a62d in trap (frame=0xffffff8230905580) at ../../../amd64/amd64/trap.c:579 #9 0xffffffff80c73293 in calltrap () at ../../../amd64/amd64/exception.S:228 #10 0xffffffff8091d9eb in kdb_enter (why=0xffffffff80eff6ce "panic", msg=0x80
) at cpufunc.h:63 #11 0xffffffff808e2711 in vpanic (fmt=, ap=) at ../../../kern/kern_shutdown.c:746 #12 0xffffffff808e2817 in panic (fmt=) at ../../../kern/kern_shutdown.c:682 #13 0xffffffff80b27b22 in softdep_deallocate_dependencies (bp=) at ../../../ufs/ffs/ffs_softdep.c:13550 #14 0xffffffff8096f893 in brelse (bp=0xffffff81e73ddf00) at buf.h:430 #15 0xffffffff809723f8 in bufdone (bp=0xffffff81e73ddf00) at ../../../kern/vfs_bio.c:3745 #16 0xffffffff80b4c675 in ufs_strategy (ap=) at ../../../ufs/ufs/ufs_vnops.c:2344 #17 0xffffffff80d27fc1 in VOP_STRATEGY_APV (vop=0xffffffff812fac20, a=0xffffff82309058a0) at vnode_if.c:2276 #18 0xffffffff8096c46b in bufstrategy (bo=, bp=0xffffff81e73ddf00) at vnode_if.h:940 #19 0xffffffff809729c5 in bufwrite (bp=0xffffff81e73ddf00) at buf.h:409 #20 0xffffffff8096db23 in vfs_bio_awrite (bp=0xffffff81e73ddf00) at buf.h:397 #21 0xffffffff80b3e82a in ffs_syncvnode (vp=0xfffffe01edbb9c30, waitfor=0x1, flags=0x0) at ../../../ufs/ffs/ffs_vnops.c:289 #22 0xffffffff80b3eaa4 in ffs_fsync (ap=0xffffff8230905a80) at ../../../ufs/ffs/ffs_vnops.c:188 #23 0xffffffff80d25a39 in VOP_FSYNC_APV (vop=0xffffffff812fa140, a=0xffffff8230905a80) at vnode_if.c:1330 #24 0xffffffff80996770 in sys_fsync (td=0xfffffe0007828490, uap=) at vnode_if.h:549 #25 0xffffffff80c89403 in amd64_syscall (td=0xfffffe0007828490, traced=0x0) at subr_syscall.c:134 #26 0xffffffff80c73577 in Xfast_syscall () at ../../../amd64/amd64/exception.S:387 #27 0x0000000800b4c38a in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 14 #14 0xffffffff8096f893 in brelse (bp=0xffffff81e73ddf00) at buf.h:430 430 (*bioops.io_deallocate)(bp); (kgdb) l 425 426 static __inline void 427 buf_deallocate(struct buf *bp) 428 { 429 if (bioops.io_deallocate) 430 (*bioops.io_deallocate)(bp); 431 } 432 433 static __inline int 434 buf_countdeps(struct buf *bp, int i) (kgdb) info loc No locals. (kgdb) f 15 #15 0xffffffff809723f8 in bufdone (bp=0xffffff81e73ddf00) at ../../../kern/vfs_bio.c:3745 3745 bufdone_finish(bp); (kgdb) p *bp $1 = {b_bufobj = 0xfffffe01edbb9d98, b_bcount = 0x1000, b_caller1 = 0x0, b_data = 0xffffff8000219000
, b_error = 0x1c, b_iocmd = 0x2, b_ioflags = 0x23, b_iooffset = 0x5871a8800, b_resid = 0x0, b_iodone = 0, b_blkno = 0x2c38d44, b_offset = 0x0, b_bobufs = {tqe_next = 0x0, tqe_prev = 0xfffffe01edbb9de8}, b_left = 0x0, b_right = 0x0, b_vflags = 0x1, b_freelist = {tqe_next = 0x0, tqe_prev = 0xffffff81e7cb3678}, b_qindex = 0x2, b_flags = 0xa0022824, b_xflags = 0x2, b_lock = {lock_object = {lo_name = 0xffffffff80f0e297 "bufwait", lo_flags = 0x5730000, lo_data = 0x0, lo_witness = 0xffffff80006cd300}, lk_lock = 0xfffffffffffffff0, lk_exslpfail = 0x0, lk_timo = 0x0, lk_pri = 0x60, lk_stack = {depth = 0x9, pcs = {0xffffffff808c5610, 0xffffffff80972a39, 0xffffffff8096db23, 0xffffffff80b3e82a, 0xffffffff80b3eaa4, 0xffffffff80d25a39, 0xffffffff80996770, 0xffffffff80c89403, 0xffffffff80c73577, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, b_bufsize = 0x1000, b_runningbufspace = 0x0, b_kvabase = 0xffffff8000219000
, b_kvaalloc = 0x0, b_kvasize = 0x4000, b_lblkno = 0x0, b_vp = 0xfffffe01edbb9c30, b_dirtyoff = 0x0, b_dirtyend = 0x0, b_rcred = 0x0, b_wcred = 0x0, b_saveaddr = 0xffffff81ec3dc000, b_pager = {pg_reqpage = 0x0}, b_cluster = {cluster_head = {tqh_first = 0xffffff81e7b46800, tqh_last = 0xffffff81e6dd89c0}, cluster_entry = {tqe_next = 0xffffff81e7b46800, tqe_prev = 0xffffff81e6dd89c0}}, b_pages = {0xfffffe021b51ca98, 0x0 }, b_npages = 0x1, b_dep = {lh_first = 0xfffffe01e348fe00}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0, b_fsprivate3 = 0x0, b_pin_count = 0x0} (kgdb) f 21 #21 0xffffffff80b3e82a in ffs_syncvnode (vp=0xfffffe01edbb9c30, waitfor=0x1, flags=0x0) at ../../../ufs/ffs/ffs_vnops.c:289 289 (void) vfs_bio_awrite(bp); (kgdb) p *vp $2 = {v_tag = 0xffffffff80eecc87 "ufs", v_op = 0xffffffff812fa140, v_data = 0xfffffe018261fdc8, v_mount = 0xfffffe00078edb58, v_nmntvnodes = {tqe_next = 0xfffffe0007a4c9c0, tqe_prev = 0xfffffe0062f51c50}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xffffff80008d9778}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0xfffffe008099a850, tqh_last = 0xfffffe008099a870}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff80eecc87 "ufs", lo_flags = 0x57b0000, lo_data = 0x0, lo_witness = 0xffffff80006d1e80}, lk_lock = 0xfffffe0007828492, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x60, lk_stack = {depth = 0x7, pcs = {0xffffffff808c5f95, 0xffffffff80b3e43b, 0xffffffff80d282fb, 0xffffffff80998e3e, 0xffffffff809966e0, 0xffffffff80c89403, 0xffffffff80c73577, 0x0 }}}, v_interlock = {lock_object = { lo_name = 0xffffffff80f083ba "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff80006c9600}, mtx_lock = 0x4}, v_vnlock = 0xfffffe01edbb9c98, v_actfreelist = {tqe_next = 0xfffffe0062f51c30, tqe_prev = 0xfffffe0007d398a8}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xffffffff80f11a15 "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff80006cfe80}, mtx_lock = 0x4}, bo_ops = 0xffffffff812b3de0, bo_object = 0xfffffe01a7b26c30, bo_synclist = {le_next = 0x0, le_prev = 0xfffffe0007012b58}, bo_private = 0xfffffe01edbb9c30, __bo_vnode = 0xfffffe01edbb9c30, bo_clean = {bv_hd = {tqh_first = 0xffffff81e73ddf00, tqh_last = 0xffffff81e73ddf50}, bv_root = 0xffffff81e73ddf00, bv_cnt = 0x1}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffffe01edbb9e08}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x1, bo_flag = 0x0, bo_bsize = 0x4000}, v_pollinfo = 0xfffffe009ca01070, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0xfffffe00079aac30, tqh_last = 0xfffffe00079aac30}, rl_currdep = 0x0}, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_holdcnt = 0x4, v_usecount = 0x2, v_iflag = 0x200, v_vflag = 0x0, v_writecount = 0x1, v_hash = 0x2c3430, v_type = VREG} (kgdb) -- Test scenario: altbufferflushes.sh + /var full