211136M from 2010-08-10 07:56:56 +0000, vmcore.6 GDB: current port: uart KDB: debugger backends: ddb gdb KDB: current backend: ddb Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0: Tue Aug 10 11:41:14 CEST 2010 test@worm2.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2211.38-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 18119393280 (17280 MB) avail memory = 16531009536 (15765 MB) Event timer "LAPIC" frequency 0 Hz quality 500 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 4 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 ioapic0: Changing APIC ID to 16 ioapic1: Changing APIC ID to 17 ioapic2: Changing APIC ID to 18 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 40-46 on motherboard ioapic2 irqs 47-53 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of e0000000, 10000000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, c7f00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xdfebf000-0xdfebffff irq 21 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ehci0: mem 0xdfebec00-0xdfebecff irq 22 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 4.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xc480-0xc487,0xc400-0xc403,0xc080-0xc087,0xc000-0xc003,0xbc00-0xbc0f mem 0xdfebd000-0xdfebdfff irq 23 at device 5.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0xb880-0xb887,0xb800-0xb803,0xb480-0xb487,0xb400-0xb403,0xb080-0xb08f mem 0xdfebc000-0xdfebcfff irq 20 at device 5.1 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] atapci3: port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f mem 0xdfebb000-0xdfebbfff irq 21 at device 5.2 on pci0 atapci3: [ITHREAD] ata6: on atapci3 ata6: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] pcib1: at device 6.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd0ff mem 0xd0000000-0xd7ffffff,0xdfff0000-0xdfffffff irq 16 at device 1.0 on pci1 pcib2: at device 10.0 on pci0 pci2: on pcib2 pcib3: at device 11.0 on pci0 pci3: on pcib3 pcib4: at device 12.0 on pci0 pci4: on pcib4 pcib5: at device 13.0 on pci0 pci5: on pcib5 pcib6: at device 14.0 on pci0 pci6: on pcib6 pcib7: at device 15.0 on pci0 pci7: on pcib7 pcib8: on acpi0 pci8: on pcib8 pcib9: at device 1.0 on pci8 pci9: on pcib9 pcib10: at device 2.0 on pci8 pci10: on pcib10 em0: port 0xec00-0xec3f mem 0xcffe0000-0xcfffffff irq 47 at device 3.0 on pci10 em0: [FILTER] em0: Ethernet address: 00:30:48:bf:aa:aa em1: port 0xe880-0xe8bf mem 0xcffa0000-0xcffbffff irq 48 at device 3.1 on pci10 em1: [FILTER] em1: Ethernet address: 00:30:48:bf:aa:ab acpi_button0: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 attimer0: [FILTER] Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: [FILTER] Event timer "RTC" frequency 32768 Hz quality 0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x90 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 hpet0: iomem 0xfed00000-0xfed00fff on acpi0 Timecounter "HPET" frequency 25000000 Hz quality 900 hpet0: [FILTER] Event timer "HPET" frequency 25000000 Hz quality 450 Event timer "HPET1" frequency 25000000 Hz quality 440 Event timer "HPET2" frequency 25000000 Hz quality 440 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 hwpstate0: on cpu0 Starting kernel event timers: LAPIC @ 1000Hz, HPET @ 127Hz Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 acd0: DVDROM at ata0-slave UDMA33 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xffffffff805f5e20(0xffffff0007206000) 0.011935874 s ugen1.1: at usbus1 uhub1: on usbus1 ad6: 152627MB at ata3-master UDMA100 SATA 1.5Gb/s SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #13 Launched! WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Root mount waiting for: usbus1 usbus0 uhub0: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 ugen1.2: at usbus1 ums0: on usbus1 ums0: 3 buttons and [Z] coordinates ID=0 ukbd0: on usbus1 kbd2 at ukbd0 Trying to mount root from nfs: NFS ROOT: 192.168.5.1:/zoo/worm2/ ugen0.2: <(null)> at usbus0 (disconnected) uhub_reattach_port: could not allocate new device em0: link state changed to UP Interface em0 IP-Address 192.168.5.121 Broadcast 192.168.5.255 Setting hostuuid: 00020003-0004-0005-0006-000700080009. Setting hostid: 0x81f4ec68. No suitable dump device was found. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/worm2 Mounting local file systems:. Setting hostname: worm2.netperf.freebsd.org. Starting Network: lo0 em0 em1 plip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=209b ether 00:30:48:bf:aa:aa inet 192.168.5.121 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:febf:aaaa%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8802 metric 0 mtu 1500 options=209b ether 00:30:48:bf:aa:ab media: Ethernet autoselect status: no carrier plip0: flags=8810 metric 0 mtu 1500 Starting devd. Starting Network: em1. em1: flags=8802 metric 0 mtu 1500 options=209b ether 00:30:48:bf:aa:ab media: Ethernet autoselect status: no carrier Starting Network: plip0. plip0: flags=8810 metric 0 mtu 1500 Starting ums0 moused. DHCPDISCOVER on em0 to 255.255.255.255 port 67 interval 8 DHCPOFFER from 192.168.5.1 DHCPREQUEST on em0 to 255.255.255.255 port 67 DHCPACK from 192.168.5.1 ifaddr cache = 0xffffff000783c200 is deleted warning bit=24568 set, but no fle found warning bit=24568 set, but no fle found bound to 192.168.5.121 -- renewal in 300 seconds. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Waiting 30s for the default route interface: ............................. Creating and/or trimming log files. Starting syslogd. /etc/rc: WARNING: Dump device does not exist. Savecore not run. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting statd. Starting lockd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Fri Aug 20 09:29:12 UTC 2010 FreeBSD/amd64 (worm2.netperf.freebsd.org) (ttyu0) login: root Aug 20 09:54:33 worm2 login: ROOT LOGIN (root) ON ttyu0 Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 9.0-CURRENT (PHO) #0: Tue Aug 10 11:41:14 CEST 2010 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. worm2# ~/test/keepalive.sh Fri Aug 20 11:54:41 CEST 2010 11:54AM up 26 mins, 1 user, load averages: 0.00, 0.00, 0.00 Aug 20 11:54:48 worm2 su: test to root on /dev/pts/0 lock order reversal: 1st 0xffffff0007d1fcf0 nfs (nfs) @ kern/vfs_mount.c:1201 2nd 0xffffff01342e3588 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1250 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x807 __lockmgr_args() at __lockmgr_args+0xd63 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 ffs_flushfiles() at ffs_flushfiles+0xc5 softdep_flushfiles() at softdep_flushfiles+0x63 ffs_unmount() at ffs_unmount+0x27a dounmount() at dounmount+0x2e6 unmount() at unmount+0x27e syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a3a6c, rsp = 0x7fffffffe308, rbp = 0x800c08360 --- lock order reversal: 1st 0xffffff814aa8cc08 bufwait (bufwait) @ kern/vfs_bio.c:2616 2nd 0xffffff0007a0f200 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:283 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x807 _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x8b4 ufs_makeinode() at ufs_makeinode+0x250 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x181 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (5, FreeBSD ELF64, open), rip = 0x800848e7c, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe7b0 --- lock order reversal: 1st 0xffffff0134333098 ufs (ufs) @ kern/vfs_lookup.c:502 2nd 0xffffff814aa8c910 bufwait (bufwait) @ ufs/ffs/ffs_softdep.c:11291 3rd 0xffffff01349b0cf0 ufs (ufs) @ kern/vfs_subr.c:2101 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x807 __lockmgr_args() at __lockmgr_args+0xd63 ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b vfs_hash_get() at vfs_hash_get+0xd5 ffs_vgetf() at ffs_vgetf+0x48 softdep_sync_metadata() at softdep_sync_metadata+0x5dc ffs_syncvnode() at ffs_syncvnode+0x213 ffs_fsync() at ffs_fsync+0x43 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 ufs_direnter() at ufs_direnter+0x325 ufs_makeinode() at ufs_makeinode+0x250 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x46b kern_openat() at kern_openat+0x181 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (5, FreeBSD ELF64, open), rip = 0x800848e7c, rsp = 0x7fffffffe6f8, rbp = 0x7fffffffe7b0 --- Fri Aug 20 12:04:41 CEST 2010 12:04PM up 36 mins, 2 users, load averages: 7.87, 6.35, 3.04 Fri Aug 20 12:15:46 CEST 2010 12:15PM up 47 mins, 2 users, load averages: 10.42, 11.42, 8.11 Fri Aug 20 12:25:56 CEST 2010 12:25PM up 58 mins, 2 users, load averages: 7.49, 11.61, 9.49 Fri Aug 20 12:35:58 CEST 2010 12:35PM up 1:08, 2 users, load averages: 3.99, 8.36, 8.95 Fri Aug 20 12:46:03 CEST 2010 12:46PM up 1:18, 2 users, load averages: 0.04, 2.08, 5.38 Fri Aug 20 12:56:22 CEST 2010 12:56PM up 1:28, 2 users, load averages: 10.36, 7.40, 6.43 Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex vfs hash (vfs hash) r = 0 (0xffffffff80e694e0) locked @ kern/vfs_hash.c:114 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_warn() at witness_warn+0x2c4 trap() at trap+0x2fe calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff80642920, rsp = 0xffffff81d000d540, rbp = 0xffffff81d000d5a0 --- vfs_hash_index() at vfs_hash_index ffs_vgetf() at ffs_vgetf+0x239 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Fatal trap 12: page fault while in kernel mode cpuid = 13; apic id = 0d fault virtual address = 0x2a8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80642920 stack pointer = 0x28:0xffffff81d000d540 frame pointer = 0x28:0xffffff81d000d5a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2476 (creat) [ thread pid 2476 tid 100292 ] Stopped at vfs_hash_index: addl 0x2a8(%rdi),%esi db> lock order reversal: (Giant after non-sleepable) 1st 0xffffffff80e694e0 vfs hash (vfs hash) @ kern/vfs_hash.c:114 2nd 0xffffffff80ca6a40 Giant (Giant) @ dev/usb/input/ukbd.c:1697 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x807 _mtx_lock_flags() at _mtx_lock_flags+0x79 ukbd_poll() at ukbd_poll+0x44 kbdmux_poll() at kbdmux_poll+0x3f sc_cngetc() at sc_cngetc+0xec cncheckc() at cncheckc+0x4a cngetc() at cngetc+0x1c db_readline() at db_readline+0x77 db_read_line() at db_read_line+0x15 db_command_loop() at db_command_loop+0x38 db_trap() at db_trap+0x89 kdb_trap() at kdb_trap+0xfe trap_fatal() at trap_fatal+0x29d trap() at trap+0x315 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff80642920, rsp = 0xffffff81d000d540, rbp = 0xffffff81d000d5a0 --- vfs_hash_index() at vfs_hash_index ffs_vgetf() at ffs_vgetf+0x239 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- bt Tracing pid 2476 tid 100292 td 0xffffff009c1be440 vfs_hash_index() at vfs_hash_index ffs_vgetf() at ffs_vgetf+0x239 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- db> set $lines 0 db> show allpcpu Current CPU: 13 cpuid = 0 dynamic pcpu = 0x492200 curthread = 0xffffff000760f440: pid 8 "pagedaemon" curpcb = 0xffffff81cd822d00 fpcurthread = none idlethread = 0xffffff0007039440: tid 100018 "idle: cpu0" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87780 commontssp = 0xffffffff80e87780 rsp0 = 0xffffff81cd822d00 gs32p = 0xffffffff80e865b8 ldt = 0xffffffff80e865f8 tss = 0xffffffff80e865e8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f3d9200 curthread = 0xffffff0007039880: pid 11 "idle: cpu1" curpcb = 0xffffff80000f0d00 fpcurthread = none idlethread = 0xffffff0007039880: tid 100017 "idle: cpu1" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e877e8 commontssp = 0xffffffff80e877e8 rsp0 = 0xffffff80000f0d00 gs32p = 0xffffffff80e86620 ldt = 0xffffffff80e86660 tss = 0xffffffff80e86650 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f3e0200 curthread = 0xffffff000703b000: pid 11 "idle: cpu2" curpcb = 0xffffff80000ebd00 fpcurthread = none idlethread = 0xffffff000703b000: tid 100016 "idle: cpu2" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87850 commontssp = 0xffffffff80e87850 rsp0 = 0xffffff80000ebd00 gs32p = 0xffffffff80e86688 ldt = 0xffffffff80e866c8 tss = 0xffffffff80e866b8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f3e7200 curthread = 0xffffff0007035000: pid 11 "idle: cpu3" curpcb = 0xffffff80000e6d00 fpcurthread = none idlethread = 0xffffff0007035000: tid 100015 "idle: cpu3" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e878b8 commontssp = 0xffffffff80e878b8 rsp0 = 0xffffff80000e6d00 gs32p = 0xffffffff80e866f0 ldt = 0xffffffff80e86730 tss = 0xffffffff80e86720 spin locks held: cpuid = 4 dynamic pcpu = 0xffffff807f3ee200 curthread = 0xffffff000760e000: pid 17 "vnlru" curpcb = 0xffffff81cd836d00 fpcurthread = none idlethread = 0xffffff0007035440: tid 100014 "idle: cpu4" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87920 commontssp = 0xffffffff80e87920 rsp0 = 0xffffff81cd836d00 gs32p = 0xffffffff80e86758 ldt = 0xffffffff80e86798 tss = 0xffffffff80e86788 spin locks held: cpuid = 5 dynamic pcpu = 0xffffff807f3f5200 curthread = 0xffffff02bdf09440: pid 2464 "creat" curpcb = 0xffffff81d01cad00 fpcurthread = none idlethread = 0xffffff0007035880: tid 100013 "idle: cpu5" curpmap = 0xffffff01344bc5e0 tssp = 0xffffffff80e87988 commontssp = 0xffffffff80e87988 rsp0 = 0xffffff81d01cad00 gs32p = 0xffffffff80e867c0 ldt = 0xffffffff80e86800 tss = 0xffffffff80e867f0 spin locks held: cpuid = 6 dynamic pcpu = 0xffffff807f3fc200 curthread = 0xffffff0095e29000: pid 2484 "creat" curpcb = 0xffffff81d034bd00 fpcurthread = none idlethread = 0xffffff0007036000: tid 100012 "idle: cpu6" curpmap = 0xffffff01340bfa90 tssp = 0xffffffff80e879f0 commontssp = 0xffffffff80e879f0 rsp0 = 0xffffff81d034bd00 gs32p = 0xffffffff80e86828 ldt = 0xffffffff80e86868 tss = 0xffffffff80e86858 spin locks held: cpuid = 7 dynamic pcpu = 0xffffff807f403200 curthread = 0xffffff000760c440: pid 19 "softdepflush" curpcb = 0xffffff81cd840d00 fpcurthread = none idlethread = 0xffffff0007036440: tid 100011 "idle: cpu7" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87a58 commontssp = 0xffffffff80e87a58 rsp0 = 0xffffff81cd840d00 gs32p = 0xffffffff80e86890 ldt = 0xffffffff80e868d0 tss = 0xffffffff80e868c0 spin locks held: cpuid = 8 dynamic pcpu = 0xffffff807f40a200 curthread = 0xffffff0007036880: pid 11 "idle: cpu8" curpcb = 0xffffff80000cdd00 fpcurthread = none idlethread = 0xffffff0007036880: tid 100010 "idle: cpu8" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87ac0 commontssp = 0xffffffff80e87ac0 rsp0 = 0xffffff80000cdd00 gs32p = 0xffffffff80e868f8 ldt = 0xffffffff80e86938 tss = 0xffffffff80e86928 spin locks held: cpuid = 9 dynamic pcpu = 0xffffff807f411200 curthread = 0xffffff0095df8000: pid 2495 "creat" curpcb = 0xffffff81d0215d00 fpcurthread = none idlethread = 0xffffff000702b440: tid 100009 "idle: cpu9" curpmap = 0xffffff027ae7f900 tssp = 0xffffffff80e87b28 commontssp = 0xffffffff80e87b28 rsp0 = 0xffffff81d0215d00 gs32p = 0xffffffff80e86960 ldt = 0xffffffff80e869a0 tss = 0xffffffff80e86990 spin locks held: cpuid = 10 dynamic pcpu = 0xffffff807f418200 curthread = 0xffffff000702b880: pid 11 "idle: cpu10" curpcb = 0xffffff80000c3d00 fpcurthread = none idlethread = 0xffffff000702b880: tid 100008 "idle: cpu10" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87b90 commontssp = 0xffffffff80e87b90 rsp0 = 0xffffff80000c3d00 gs32p = 0xffffffff80e869c8 ldt = 0xffffffff80e86a08 tss = 0xffffffff80e869f8 spin locks held: cpuid = 11 dynamic pcpu = 0xffffff807f41f200 curthread = 0xffffff0007034000: pid 11 "idle: cpu11" curpcb = 0xffffff80000bed00 fpcurthread = none idlethread = 0xffffff0007034000: tid 100007 "idle: cpu11" curpmap = 0xffffffff80ca44b0 tssp = 0xffffffff80e87bf8 commontssp = 0xffffffff80e87bf8 rsp0 = 0xffffff80000bed00 gs32p = 0xffffffff80e86a30 ldt = 0xffffffff80e86a70 tss = 0xffffffff80e86a60 spin locks held: cpuid = 12 dynamic pcpu = 0xffffff807f426200 curthread = 0xffffff0007a35880: pid 2494 "creat" curpcb = 0xffffff81cfcc0d00 fpcurthread = none idlethread = 0xffffff0007034440: tid 100006 "idle: cpu12" curpmap = 0xffffff027af135e0 tssp = 0xffffffff80e87c60 commontssp = 0xffffffff80e87c60 rsp0 = 0xffffff81cfcc0d00 gs32p = 0xffffffff80e86a98 ldt = 0xffffffff80e86ad8 tss = 0xffffffff80e86ac8 spin locks held: cpuid = 13 dynamic pcpu = 0xffffff807f42d200 curthread = 0xffffff009c1be440: pid 2476 "creat" curpcb = 0xffffff81d000dd00 fpcurthread = none idlethread = 0xffffff0007034880: tid 100005 "idle: cpu13" curpmap = 0xffffff027ae7fdb0 tssp = 0xffffffff80e87cc8 commontssp = 0xffffffff80e87cc8 rsp0 = 0xffffff81d000dd00 gs32p = 0xffffffff80e86b00 ldt = 0xffffffff80e86b40 tss = 0xffffffff80e86b30 spin locks held: cpuid = 14 dynamic pcpu = 0xffffff807f434200 curthread = 0xffffff019d7d5000: pid 2479 "creat" curpcb = 0xffffff81cff6dd00 fpcurthread = none idlethread = 0xffffff000702a000: tid 100004 "idle: cpu14" curpmap = 0xffffff000702edb0 tssp = 0xffffffff80e87d30 commontssp = 0xffffffff80e87d30 rsp0 = 0xffffff81cff6dd00 gs32p = 0xffffffff80e86b68 ldt = 0xffffffff80e86ba8 tss = 0xffffffff80e86b98 spin locks held: cpuid = 15 dynamic pcpu = 0xffffff807f43b200 curthread = 0xffffff025e8ab440: pid 2456 "creat" curpcb = 0xffffff81d01d9d00 fpcurthread = none idlethread = 0xffffff000702a440: tid 100003 "idle: cpu15" curpmap = 0xffffff03e40fd450 tssp = 0xffffffff80e87d98 commontssp = 0xffffffff80e87d98 rsp0 = 0xffffff81d01d9d00 gs32p = 0xffffffff80e86bd0 ldt = 0xffffffff80e86c10 tss = 0xffffffff80e86c00 spin locks held: db> show alllocks Process 2495 (creat) thread 0xffffff0095df8000 (100396) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0188e7da78) locked @ kern/vfs_lookup.c:502 Process 2494 (creat) thread 0xffffff0007a35880 (100123) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0381989cf0) locked @ ufs/ffs/ffs_vfsops.c:1521 exclusive lockmgr ufs (ufs) r = 0 (0xffffff018bf73310) locked @ kern/vfs_lookup.c:502 Process 2484 (creat) thread 0xffffff0095e29000 (100458) exclusive lockmgr ufs (ufs) r = 0 (0xffffff014fc53098) locked @ kern/vfs_lookup.c:502 Process 2479 (creat) thread 0xffffff019d7d5000 (100260) exclusive lockmgr ufs (ufs) r = 0 (0xffffff02dac4d310) locked @ kern/vfs_lookup.c:502 Process 2476 (creat) thread 0xffffff009c1be440 (100292) exclusive sleep mutex vfs hash (vfs hash) r = 0 (0xffffffff80e694e0) locked @ kern/vfs_hash.c:114 exclusive lockmgr ufs (ufs) r = 0 (0xffffff01854c2588) locked @ ufs/ffs/ffs_vfsops.c:1521 exclusive lockmgr ufs (ufs) r = 0 (0xffffff018b291310) locked @ kern/vfs_lookup.c:502 Process 2468 (swap) thread 0xffffff0030787440 (100475) shared sx user map (user map) r = 0 (0xffffff01a1e98398) locked @ vm/vm_map.c:3525 Process 2466 (swap) thread 0xffffff024925a880 (100474) shared sx user map (user map) r = 0 (0xffffff014fb77b68) locked @ vm/vm_map.c:3525 Process 2464 (creat) thread 0xffffff02bdf09440 (100381) exclusive lockmgr ufs (ufs) r = 0 (0xffffff038197d098) locked @ kern/vfs_lookup.c:502 Process 2462 (swap) thread 0xffffff014f7a9000 (100473) shared sx user map (user map) r = 0 (0xffffff014fb77cf8) locked @ vm/vm_map.c:3525 Process 2461 (swap) thread 0xffffff014f7a9440 (100472) shared sx user map (user map) r = 0 (0xffffff0132e46078) locked @ vm/vm_map.c:3525 Process 2456 (creat) thread 0xffffff025e8ab440 (100384) exclusive lockmgr ufs (ufs) r = 0 (0xffffff015bb6f800) locked @ kern/vfs_lookup.c:502 Process 1600 (sshd) thread 0xffffff0007b8a880 (100108) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff01340513a0) locked @ kern/uipc_sockbuf.c:148 Process 19 (softdepflush) thread 0xffffff000760c440 (100081) exclusive lockmgr ufs (ufs) r = 0 (0xffffff01891c0a78) locked @ ufs/ffs/ffs_vfsops.c:1521 Process 17 (vnlru) thread 0xffffff000760e000 (100079) exclusive lockmgr ufs (ufs) r = 0 (0xffffff021b76d800) locked @ kern/vfs_subr.c:898 Process 8 (pagedaemon) thread 0xffffff000760f440 (100075) exclusive sleep mutex pmap (pmap) r = 0 (0xffffff027ae7fdb0) locked @ amd64/amd64/pmap.c:4390 exclusive sleep mutex vm object (standard object) r = 0 (0xffffff0029cec288) locked @ vm/vm_pageout.c:811 exclusive sleep mutex page lock (page lock) r = 0 (0xffffffff80e80600) locked @ vm/vm_pageout.c:292 exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 1 (0xffffffff80e81b80) locked @ amd64/amd64/pmap.c:4352 db> show lockedvnods Locked vnodes 0xffffff015bb6f768: tag ufs, type VDIR usecount 2, writecount 0, refcount 24 mountedhere 0 flags () v_object 0xffffff018cc07948 ref 0 pages 80 lock type ufs: EXCL by thread 0xffffff025e8ab440 (pid 2456) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553217, on dev ad6s1d 0xffffff038197d000: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff03886b85e8 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff02bdf09440 (pid 2464) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553219, on dev ad6s1d 0xffffff018b291278: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff0187c69000 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff009c1be440 (pid 2476) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553220, on dev ad6s1d 0xffffff02dac4d278: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff015b8546c0 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff019d7d5000 (pid 2479) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553221, on dev ad6s1d 0xffffff014fc53000: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff038872a870 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff0095e29000 (pid 2484) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553227, on dev ad6s1d 0xffffff018bf73278: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff018ce93948 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff0007a35880 (pid 2494) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553230, on dev ad6s1d 0xffffff0188e7d9e0: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xffffff01854b0ca8 ref 0 pages 72 lock type ufs: EXCL by thread 0xffffff0095df8000 (pid 2495) #0 0xffffffff805a1801 at __lockmgr_args+0x531 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff8065c267 at _vn_lock+0x57 #4 0xffffffff806439c6 at lookup+0xe6 #5 0xffffffff80644c61 at namei+0x481 #6 0xffffffff80654388 at kern_unlinkat+0x128 #7 0xffffffff805f8faa at syscallenter+0x1aa #8 0xffffffff808b2e30 at syscall+0x60 #9 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12553233, on dev ad6s1d 0xffffff021b76d768: tag ufs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) lock type ufs: EXCL by thread 0xffffff000760e000 (pid 17) #0 0xffffffff805a1a64 at __lockmgr_args+0x794 #1 0xffffffff807f9e1c at ffs_lock+0x9c #2 0xffffffff8090d4bb at VOP_LOCK1_APV+0x9b #3 0xffffffff806513ab at vnlru_free+0x20b #4 0xffffffff80651cae at vnlru_proc+0x49e #5 0xffffffff8058f47a at fork_exit+0x12a #6 0xffffffff8089d4fe at fork_trampoline+0xe ino 12727239, on dev ad6s1d 0xffffff01891c09e0: tag ufs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED|VI_DOINGINACT) lock type ufs: EXCL by thread 0xffffff000760c440 (pid 19) #0 0xffffffff805a1a64 at __lockmgr_args+0x794 #1 0xffffffff807f520f at ffs_vgetf+0x1bf #2 0xffffffff807e89b9 at process_worklist_item+0x139 #3 0xffffffff807eefdb at softdep_process_worklist+0x8b #4 0xffffffff807ef3c1 at softdep_flush+0xc1 #5 0xffffffff8058f47a at fork_exit+0x12a #6 0xffffffff8089d4fe at fork_trampoline+0xe ino 13919232, on dev ad6s1d 0xffffff01854c24f0: tag ufs, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff009c1be440 (pid 2476) #0 0xffffffff805a1a64 at __lockmgr_args+0x794 #1 0xffffffff807f520f at ffs_vgetf+0x1bf #2 0xffffffff808003ab at ufs_lookup_ino+0xd8b #3 0xffffffff8090c02f at VOP_CACHEDLOOKUP_APV+0xaf #4 0xffffffff8063cfb0 at vfs_cache_lookup+0xf0 #5 0xffffffff8090e397 at VOP_LOOKUP_APV+0xb7 #6 0xffffffff80643cb3 at lookup+0x3d3 #7 0xffffffff80644c61 at namei+0x481 #8 0xffffffff80654388 at kern_unlinkat+0x128 #9 0xffffffff805f8faa at syscallenter+0x1aa #10 0xffffffff808b2e30 at syscall+0x60 #11 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 13285045, on dev ad6s1d 0xffffff0381989c58: tag ufs, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff0007a35880 (pid 2494) #0 0xffffffff805a1a64 at __lockmgr_args+0x794 #1 0xffffffff807f520f at ffs_vgetf+0x1bf #2 0xffffffff808003ab at ufs_lookup_ino+0xd8b #3 0xffffffff8090c02f at VOP_CACHEDLOOKUP_APV+0xaf #4 0xffffffff8063cfb0 at vfs_cache_lookup+0xf0 #5 0xffffffff8090e397 at VOP_LOOKUP_APV+0xb7 #6 0xffffffff80643cb3 at lookup+0x3d3 #7 0xffffffff80644c61 at namei+0x481 #8 0xffffffff80654388 at kern_unlinkat+0x128 #9 0xffffffff805f8faa at syscallenter+0x1aa #10 0xffffffff808b2e30 at syscall+0x60 #11 0xffffffff8089d302 at Xfast_syscall+0xe2 ino 12629944, on dev ad6s1d db> ps pid ppid pgrp uid state wmesg wchan cmd 2513 1596 1596 0 S+ nanslp 0xffffffff80ca76e8 sleep 2495 2455 1657 1001 R+ CPU 9 creat 2494 2455 1657 1001 R+ CPU 12 creat 2484 2455 1657 1001 R+ CPU 6 creat 2482 2455 1657 1001 R+ creat 2479 2455 1657 1001 R+ CPU 14 creat 2476 2455 1657 1001 R+ CPU 13 creat 2468 2457 1657 1001 LL+ *vm page 0xffffff01a6057000 swap 2466 2457 1657 1001 LL+ *vm page 0xffffff01a6057000 swap 2464 2455 1657 1001 R+ CPU 5 creat 2462 2457 1657 1001 LL+ *vm page 0xffffff01a6057000 swap 2461 2457 1657 1001 LL+ *vm page 0xffffff01a6057000 swap 2457 2452 1657 1001 S+ wait 0xffffff01343658c0 swap 2456 2455 1657 1001 R+ CPU 15 creat 2455 2453 1657 1001 S+ wait 0xffffff027ae6b000 creat 2453 1667 1657 1001 S+ wait 0xffffff021b9ae460 creat 2452 1667 1657 1001 S+ wait 0xffffff021b9ae8c0 swap 1667 1666 1657 1001 S+ wait 0xffffff01344058c0 run 1666 1665 1657 1001 SW+ wait 0xffffff0134406000 run 1665 1657 1657 1001 S+ nanslp 0xffffffff80ca76e8 run 1657 1605 1657 1001 SW+ wait 0xffffff0007bb6000 sh 1605 1604 1605 1001 SWs+ wait 0xffffff0007bbe8c0 sh 1604 1600 1600 1001 S select 0xffffff000785d9c0 sshd 1600 1451 1600 0 SWs sbwait 0xffffff01340513ec sshd 1596 1592 1596 0 S+ wait 0xffffff0007d73000 sh 1592 1540 1592 0 SW+ pause 0xffffff01340b10a0 csh 1541 1 1 0 S ttydcd 0xffffff000713e4e8 getty 1540 1 1540 0 SWs+ wait 0xffffff0007962000 login 1539 1 1539 0 Ss+ ttyin 0xffffff00071404a8 getty 1538 1 1538 0 Ss+ ttyin 0xffffff00071408a8 getty 1537 1 1537 0 Ss+ ttyin 0xffffff0007140ca8 getty 1536 1 1536 0 Ss+ ttyin 0xffffff000754b0a8 getty 1470 1 1470 0 Ss nanslp 0xffffffff80ca76e8 cron 1463 1 1463 25 SWs pause 0xffffff0007dad0a0 sendmail 1459 1 1459 0 Ss select 0xffffff00072a9d40 sendmail 1451 1 1451 0 Ss select 0xffffff000785dac0 sshd 1323 1 1323 0 Ss rpcsvc 0xffffff0007e60020 NLM: master 1316 1 1316 0 Ss select 0xffffff00072a9e40 rpc.statd 1243 0 0 0 DL mdwait 0xffffff00076c8800 [md1] 1188 1 1188 0 Ss select 0xffffff0007845140 rpcbind 1157 1 1157 0 Ss select 0xffffff000785dbc0 syslogd 882 1 882 0 Ss select 0xffffff00072aa040 devd 881 1 881 65 Ss select 0xffffff00072aa0c0 dhclient 865 1 865 0 Ss select 0xffffff00072aa140 dhclient 822 1 822 0 Ss select 0xffffff00072aa1c0 moused 103 0 0 0 DL mdwait 0xffffff00076c9000 [md0] 20 0 0 0 DL flowclea 0xffffffff80e69a30 [flowcleaner] 19 0 0 0 RL CPU 7 [softdepflush] 18 0 0 0 DL syncer 0xffffffff80e69780 [syncer] 17 0 0 0 RL CPU 4 [vnlru] 16 0 0 0 DL psleep 0xffffffff80e692a8 [bufdaemon] 15 0 0 0 DL pgzero 0xffffffff80e829bc [pagezero] 9 0 0 0 DL psleep 0xffffffff80e81c68 [vmdaemon] 8 0 0 0 RL CPU 0 [pagedaemon] 7 0 0 0 DL ccb_scan 0xffffffff80c6f960 [xpt_thrd] 6 0 0 0 DL waiting_ 0xffffffff80e6c5c0 [sctp_iterator] 5 0 0 0 DL - 0xffffff0007549248 [fdc0] 14 0 0 0 DL (threaded) [usb] 100061 D - 0xffffff80009dbdd0 [usbus1] 100060 D - 0xffffff80009dbd78 [usbus1] 100059 D - 0xffffff80009dbd20 [usbus1] 100058 D - 0xffffff80009dbcc8 [usbus1] 100056 D - 0xffffff80009d3460 [usbus0] 100055 D - 0xffffff80009d3408 [usbus0] 100054 D - 0xffffff80009d33b0 [usbus0] 100053 D - 0xffffff80009d3358 [usbus0] 13 0 0 0 DL - 0xffffffff80ca73e4 [yarrow] 4 0 0 0 DL - 0xffffffff80ca39c8 [g_down] 3 0 0 0 DL - 0xffffffff80ca39c0 [g_up] 2 0 0 0 DL - 0xffffffff80ca39b0 [g_event] 12 0 0 0 WL (threaded) [intr] 100071 I [irq1: atkbd0] 100070 I [irq7: ppc0] 100068 I [swi0: uart uart] 100065 I [irq20: atapci2] 100064 I [irq23: atapci1] 100063 I [irq15: ata1] 100062 I [irq14: ata0] 100057 I [irq22: ehci0] 100052 I [irq21: ohci0+] 100051 I [irq9: acpi0] 100050 I [swi5: +] 100048 I [swi2: cambio] 100044 I [swi6: task queue] 100043 I [swi6: Giant taskq] 100036 I [swi3: vm] 100035 I [swi1: netisr 0] 100034 I [swi4: clock] 100033 I [swi4: clock] 100032 I [swi4: clock] 100031 I [swi4: clock] 100030 I [swi4: clock] 100029 I [swi4: clock] 100028 I [swi4: clock] 100027 I [swi4: clock] 100026 I [swi4: clock] 100025 I [swi4: clock] 100024 I [swi4: clock] 100023 I [swi4: clock] 100022 I [swi4: clock] 100021 I [swi4: clock] 100020 I [swi4: clock] 100019 I [swi4: clock] 11 0 0 0 RL (threaded) [idle] 100018 CanRun [idle: cpu0] 100017 Run CPU 1 [idle: cpu1] 100016 Run CPU 2 [idle: cpu2] 100015 Run CPU 3 [idle: cpu3] 100014 CanRun [idle: cpu4] 100013 CanRun [idle: cpu5] 100012 CanRun [idle: cpu6] 100011 CanRun [idle: cpu7] 100010 Run CPU 8 [idle: cpu8] 100009 CanRun [idle: cpu9] 100008 Run CPU 10 [idle: cpu10] 100007 Run CPU 11 [idle: cpu11] 100006 CanRun [idle: cpu12] 100005 CanRun [idle: cpu13] 100004 CanRun [idle: cpu14] 100003 CanRun [idle: cpu15] 1 0 1 0 SLs wait 0xffffff00070288c0 [init] 10 0 0 0 DL audit_wo 0xffffffff80e77c50 [audit] 0 0 0 0 DLs (threaded) [kernel] 100072 D deadlkre 0xffffffff80ca73e4 [deadlkres] 100067 D - 0xffffff0007441a80 [em1 taskq] 100066 D - 0xffffff0007441d80 [em0 taskq] 100049 D - 0xffffff00071ca300 [kqueue taskq] 100047 D - 0xffffff00071ca380 [acpi_task_2] 100046 D - 0xffffff00071ca380 [acpi_task_1] 100045 D - 0xffffff00071ca380 [acpi_task_0] 100042 D - 0xffffff00071ca500 [thread taskq] 100040 D - 0xffffff0007037800 [firmware taskq] 100000 D sched 0xffffffff80ca3b00 [swapper] 2460 2455 1657 1001 Z+ creat 2491 2455 1657 1001 Z+ creat 2472 2455 1657 1001 Z+ creat 2486 2455 1657 1001 Z+ creat 2458 2455 1657 1001 Z+ creat 2470 2455 1657 1001 Z+ creat 2496 2455 1657 1001 Z+ creat 2493 2455 1657 1001 Z+ creat 2454 1667 1657 1001 Z+ mkdir 2489 2455 1657 1001 Z+ creat db> allt Tracing command sleep pid 2513 tid 100215 td 0xffffff01a605d440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24d kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80072939c, rsp = 0x7fffffffeb88, rbp = 0x7fffffffebe0 --- Tracing command creat pid 2495 tid 100396 td 0xffffff0095df8000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e2, rsp = 0xffffff800006ffe0, rbp = 0xffffff81d0215510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x102 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_get() at vfs_hash_get+0x4c ffs_vgetf() at ffs_vgetf+0x48 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command creat pid 2494 tid 100123 td 0xffffff0007a35880 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e2, rsp = 0xffffff8000084fe0, rbp = 0xffffff81cfcc0510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x102 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_insert() at vfs_hash_insert+0x54 ffs_vgetf() at ffs_vgetf+0x239 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command creat pid 2484 tid 100458 td 0xffffff0095e29000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94f3, rsp = 0xffffff800005afe0, rbp = 0xffffff81d034b510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x113 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_get() at vfs_hash_get+0x4c ffs_vgetf() at ffs_vgetf+0x48 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command creat pid 2482 tid 100409 td 0xffffff010903f880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ast() at ast+0x2af doreti_ast() at doreti_ast+0x1f Tracing command creat pid 2479 tid 100260 td 0xffffff019d7d5000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e2, rsp = 0xffffff8000092fe0, rbp = 0xffffff81cff6d510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x102 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_get() at vfs_hash_get+0x4c ffs_vgetf() at ffs_vgetf+0x48 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command creat pid 2476 tid 100292 td 0xffffff009c1be440 vfs_hash_index() at vfs_hash_index ffs_vgetf() at ffs_vgetf+0x239 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command swap pid 2468 tid 100475 td 0xffffff0030787440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xd4 pmap_enter() at pmap_enter+0x94 vm_fault() at vm_fault+0x1519 trap_pfault() at trap_pfault+0x118 trap() at trap+0x512 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x800857714, rsp = 0x7fffffffe658, rbp = 0x7fffffffe7c0 --- Tracing command swap pid 2466 tid 100474 td 0xffffff024925a880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xd4 pmap_enter() at pmap_enter+0x94 vm_fault() at vm_fault+0x1519 trap_pfault() at trap_pfault+0x118 trap() at trap+0x512 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x800857714, rsp = 0x7fffffffe658, rbp = 0x7fffffffe7c0 --- Tracing command creat pid 2464 tid 100381 td 0xffffff02bdf09440 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e6, rsp = 0xffffff8000053fe0, rbp = 0xffffff81d01ca510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x106 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_get() at vfs_hash_get+0x4c ffs_vgetf() at ffs_vgetf+0x48 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command swap pid 2462 tid 100473 td 0xffffff014f7a9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xd4 pmap_enter() at pmap_enter+0x94 vm_fault() at vm_fault+0x1519 trap_pfault() at trap_pfault+0x118 trap() at trap+0x512 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x800857714, rsp = 0x7fffffffe658, rbp = 0x7fffffffe7c0 --- Tracing command swap pid 2461 tid 100472 td 0xffffff014f7a9440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xd4 pmap_enter() at pmap_enter+0x94 vm_fault() at vm_fault+0x1519 trap_pfault() at trap_pfault+0x118 trap() at trap+0x512 calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x800857714, rsp = 0x7fffffffe658, rbp = 0x7fffffffe7c0 --- Tracing command swap pid 2457 tid 100180 td 0xffffff0134443440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007ca81c, rsp = 0x7fffffffe7e8, rbp = 0x7fffffffe820 --- Tracing command creat pid 2456 tid 100384 td 0xffffff025e8ab440 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e6, rsp = 0xffffff8000099fe0, rbp = 0xffffff81d01d9510 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x106 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_get() at vfs_hash_get+0x4c ffs_vgetf() at ffs_vgetf+0x48 ufs_lookup_ino() at ufs_lookup_ino+0xd8b VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x481 kern_unlinkat() at kern_unlinkat+0x128 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (10, FreeBSD ELF64, unlink), rip = 0x80084679c, rsp = 0x7fffffffe708, rbp = 0x7fffffffe7b0 --- Tracing command creat pid 2455 tid 100460 td 0xffffff027ae78000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb81c, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe810 --- Tracing command creat pid 2453 tid 100428 td 0xffffff0126dc8880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb81c, rsp = 0x7fffffffe808, rbp = 0x7fffffffe840 --- Tracing command swap pid 2452 tid 100427 td 0xffffff01ff7ec000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007ca81c, rsp = 0x7fffffffe818, rbp = 0x7fffffffe850 --- Tracing command run pid 1667 tid 100173 td 0xffffff0007e97880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8007cb81c, rsp = 0x7fffffffe2b8, rbp = 0x7fffffffe770 --- Tracing command run pid 1666 tid 100168 td 0xffffff0007e84000 Tracing command run pid 1665 tid 100176 td 0xffffff0007e97440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24d kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80083d39c, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe800 --- Tracing command sh pid 1657 tid 100107 td 0xffffff0007b8b000 Tracing command sh pid 1605 tid 100129 td 0xffffff0007abe440 Tracing command sshd pid 1604 tid 100163 td 0xffffff0007bdf880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x8016eaf8c, rsp = 0x7fffffffe388, rbp = 0x7fffffffe410 --- Tracing command sshd pid 1600 tid 100108 td 0xffffff0007b8a880 Tracing command sh pid 1596 tid 100128 td 0xffffff0007abe880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x80093c81c, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe978 --- Tracing command csh pid 1592 tid 100161 td 0xffffff01340bc880 Tracing command getty pid 1541 tid 100159 td 0xffffff000796b880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydev_open() at ttydev_open+0x343 devfs_open() at devfs_open+0x1ac VOP_OPEN_APV() at VOP_OPEN_APV+0xaf vn_open_cred() at vn_open_cred+0x316 kern_openat() at kern_openat+0x181 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (5, FreeBSD ELF64, open), rip = 0x800848e7c, rsp = 0x7fffffffecd8, rbp = 0x5086e0 --- Tracing command login pid 1540 tid 100158 td 0xffffff000796c000 Tracing command getty pid 1539 tid 100157 td 0xffffff000796c440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (3, FreeBSD ELF64, read), rip = 0x80085900c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1538 tid 100156 td 0xffffff000796c880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (3, FreeBSD ELF64, read), rip = 0x80085900c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1537 tid 100155 td 0xffffff000796d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (3, FreeBSD ELF64, read), rip = 0x80085900c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 1536 tid 100154 td 0xffffff000796d440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x88 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (3, FreeBSD ELF64, read), rip = 0x80085900c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command cron pid 1470 tid 100086 td 0xffffff0007a2e440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x24d kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80094839c, rsp = 0x7fffffffeb38, rbp = 0x3c --- Tracing command sendmail pid 1463 tid 100137 td 0xffffff0007e87880 Tracing command sendmail pid 1459 tid 100153 td 0xffffff0134088880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x800df1f8c, rsp = 0x7fffffffcd88, rbp = 0x7fffffffce20 --- Tracing command sshd pid 1451 tid 100096 td 0xffffff0007a2c880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x8016eaf8c, rsp = 0x7fffffffe4a8, rbp = 0x2 --- Tracing command rpc.lockd pid 1323 tid 100102 td 0xffffff000760b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x81a svc_run() at svc_run+0x8f nlm_syscall() at nlm_syscall+0x865 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (154, FreeBSD ELF64, nlm_syscall), rip = 0x8008c43dc, rsp = 0x7fffffffec38, rbp = 0x7fffffffed60 --- Tracing command rpc.statd pid 1316 tid 100140 td 0xffffff0007e84880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x800851f8c, rsp = 0x7fffffffeb58, rbp = 0 --- Tracing command md1 pid 1243 tid 100131 td 0xffffff0007abb880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 md_kthread() at md_kthread+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cfce8cf0, rbp = 0 --- Tracing command rpcbind pid 1188 tid 100124 td 0xffffff0007a35440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80091024c, rsp = 0x7fffffffca88, rbp = 0x800c5b0c0 --- Tracing command syslogd pid 1157 tid 100135 td 0xffffff0007e79440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x80085bf8c, rsp = 0x7fffffffe158, rbp = 0x7fffffffe53a --- Tracing command devd pid 882 tid 100106 td 0xffffff0007b8b440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x4474bc, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 --- Tracing command dhclient pid 881 tid 100091 td 0xffffff0007ac1000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80070024c, rsp = 0x7fffffffecd8, rbp = 0x45d30 --- Tracing command dhclient pid 865 tid 100092 td 0xffffff0007ac0880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac poll() at poll+0x2b6 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80070024c, rsp = 0x7fffffffece8, rbp = 0x4 --- Tracing command moused pid 822 tid 100117 td 0xffffff0007bdd440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x640 select() at select+0x5d syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (93, FreeBSD ELF64, select), rip = 0x80097bf8c, rsp = 0x7fffffffe878, rbp = 0x7fffffffed10 --- Tracing command md0 pid 103 tid 100103 td 0xffffff0007abf440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 md_kthread() at md_kthread+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cfc5ccf0, rbp = 0 --- Tracing command flowcleaner pid 20 tid 100082 td 0xffffff000760c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xe8 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd845cf0, rbp = 0 --- Tracing command softdepflush pid 19 tid 100081 td 0xffffff000760c440 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e2, rsp = 0xffffff8000061fe0, rbp = 0xffffff81cd840890 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x102 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_remove() at vfs_hash_remove+0x26 ufs_reclaim() at ufs_reclaim+0x78 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vrecycle() at vrecycle+0x58 ufs_inactive() at ufs_inactive+0x60 VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2f8 process_worklist_item() at process_worklist_item+0x258 softdep_process_worklist() at softdep_process_worklist+0x8b softdep_flush() at softdep_flush+0xc1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd840cf0, rbp = 0 --- Tracing command syncer pid 18 tid 100080 td 0xffffff000760c880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4de fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd83bcf0, rbp = 0 --- Tracing command vnlru pid 17 tid 100079 td 0xffffff000760e000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805a94e6, rsp = 0xffffff800004cfe0, rbp = 0xffffff81cd8369c0 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x106 _mtx_lock_flags() at _mtx_lock_flags+0xd4 vfs_hash_remove() at vfs_hash_remove+0x26 ufs_reclaim() at ufs_reclaim+0x78 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 vnlru_proc() at vnlru_proc+0x49e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd836cf0, rbp = 0 --- Tracing command bufdaemon pid 16 tid 100078 td 0xffffff000760e440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 buf_daemon() at buf_daemon+0x18d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd831cf0, rbp = 0 --- Tracing command pagezero pid 15 tid 100077 td 0xffffff000760e880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd82ccf0, rbp = 0 --- Tracing command vmdaemon pid 9 tid 100076 td 0xffffff000760f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd827cf0, rbp = 0 --- Tracing command pagedaemon pid 8 tid 100075 td 0xffffff000760f440 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80906612, rsp = 0xffffffff80e8f540, rbp = 0xffffff81cd822850 --- lapic_ipi_vectored() at lapic_ipi_vectored+0x82 timer2cb() at timer2cb+0xca hpet_intr_single() at hpet_intr_single+0x4f hpet_intr() at hpet_intr+0x8a intr_event_handle() at intr_event_handle+0x7a intr_execute_handlers() at intr_execute_handlers+0x5f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa5 --- interrupt, rip = 0xffffffff808a3463, rsp = 0xffffff81cd822a20, rbp = 0xffffff81cd822a40 --- spinlock_exit() at spinlock_exit+0x33 pmap_invalidate_page() at pmap_invalidate_page+0x6c pmap_ts_referenced() at pmap_ts_referenced+0x27e vm_pageout() at vm_pageout+0xc03 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd822cf0, rbp = 0 --- Tracing command xpt_thrd pid 7 tid 100074 td 0xffffff000760f880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd81dcf0, rbp = 0 --- Tracing command sctp_iterator pid 6 tid 100073 td 0xffffff0007610000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 sctp_iterator_thread() at sctp_iterator_thread+0x49 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd818cf0, rbp = 0 --- Tracing command fdc0 pid 5 tid 100069 td 0xffffff0007404000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 fdc_thread() at fdc_thread+0x74a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd800cf0, rbp = 0 --- Tracing command usb pid 14 tid 100061 td 0xffffff0007407880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6b6cf0, rbp = 0 --- Tracing command usb pid 14 tid 100060 td 0xffffff000706e440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6b1cf0, rbp = 0 --- Tracing command usb pid 14 tid 100059 td 0xffffff000706e880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6accf0, rbp = 0 --- Tracing command usb pid 14 tid 100058 td 0xffffff0007205000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6a7cf0, rbp = 0 --- Tracing command usb pid 14 tid 100056 td 0xffffff0007205880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001f7cf0, rbp = 0 --- Tracing command usb pid 14 tid 100055 td 0xffffff0007206000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001f2cf0, rbp = 0 --- Tracing command usb pid 14 tid 100054 td 0xffffff0007206440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001edcf0, rbp = 0 --- Tracing command usb pid 14 tid 100053 td 0xffffff0007206880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001e8cf0, rbp = 0 --- Tracing command yarrow pid 13 tid 100041 td 0xffffff000706c880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000168cf0, rbp = 0 --- Tracing command g_down pid 4 tid 100039 td 0xffffff0007056000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 g_io_schedule_down() at g_io_schedule_down+0x228 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800015ecf0, rbp = 0 --- Tracing command g_up pid 3 tid 100038 td 0xffffff0007056440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 g_io_schedule_up() at g_io_schedule_up+0x10a g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000159cf0, rbp = 0 --- Tracing command g_event pid 2 tid 100037 td 0xffffff0007056880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000154cf0, rbp = 0 --- Tracing command intr pid 12 tid 100071 td 0xffffff0007401440 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100070 td 0xffffff0007401880 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100068 td 0xffffff0007404440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd7f1cf0, rbp = 0 --- Tracing command intr pid 12 tid 100065 td 0xffffff0007406440 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100064 td 0xffffff0007406880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd70bcf0, rbp = 0 --- Tracing command intr pid 12 tid 100063 td 0xffffff0007407000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100062 td 0xffffff0007407440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6decf0, rbp = 0 --- Tracing command intr pid 12 tid 100057 td 0xffffff0007205440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd6a2cf0, rbp = 0 --- Tracing command intr pid 12 tid 100052 td 0xffffff0007208000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80001e3cf0, rbp = 0 --- Tracing command intr pid 12 tid 100051 td 0xffffff0007208440 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100050 td 0xffffff0007208880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000195cf0, rbp = 0 --- Tracing command intr pid 12 tid 100048 td 0xffffff0007069440 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100044 td 0xffffff000706a880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000177cf0, rbp = 0 --- Tracing command intr pid 12 tid 100043 td 0xffffff000706c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000172cf0, rbp = 0 --- Tracing command intr pid 12 tid 100036 td 0xffffff0007058000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100035 td 0xffffff0007058440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800014acf0, rbp = 0 --- Tracing command intr pid 12 tid 100034 td 0xffffff0007058880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000145cf0, rbp = 0 --- Tracing command intr pid 12 tid 100033 td 0xffffff0007059000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000140cf0, rbp = 0 --- Tracing command intr pid 12 tid 100032 td 0xffffff0007059440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800013bcf0, rbp = 0 --- Tracing command intr pid 12 tid 100031 td 0xffffff0007059880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000136cf0, rbp = 0 --- Tracing command intr pid 12 tid 100030 td 0xffffff000703b440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000131cf0, rbp = 0 --- Tracing command intr pid 12 tid 100029 td 0xffffff000703b880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800012ccf0, rbp = 0 --- Tracing command intr pid 12 tid 100028 td 0xffffff0007050000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000127cf0, rbp = 0 --- Tracing command intr pid 12 tid 100027 td 0xffffff0007050440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000122cf0, rbp = 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff0007050880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800011dcf0, rbp = 0 --- Tracing command intr pid 12 tid 100025 td 0xffffff0007051000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000118cf0, rbp = 0 --- Tracing command intr pid 12 tid 100024 td 0xffffff0007051440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000113cf0, rbp = 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff0007051880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800010ecf0, rbp = 0 --- Tracing command intr pid 12 tid 100022 td 0xffffff0007038000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000109cf0, rbp = 0 --- Tracing command intr pid 12 tid 100021 td 0xffffff0007038440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000104cf0, rbp = 0 --- Tracing command intr pid 12 tid 100020 td 0xffffff0007038880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000ffcf0, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xffffff0007039000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b ithread_loop() at ithread_loop+0x23c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000facf0, rbp = 0 --- Tracing command idle pid 11 tid 100018 td 0xffffff0007039440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000f5b10, rbp = 0xffffff80000f5b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000f5cf0, rbp = 0 --- Tracing command idle pid 11 tid 100017 td 0xffffff0007039880 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff8000037fe0, rbp = 0xffffff80000f0b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000f0cf0, rbp = 0 --- Tracing command idle pid 11 tid 100016 td 0xffffff000703b000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff800003efe0, rbp = 0xffffff80000ebb20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000ebcf0, rbp = 0 --- Tracing command idle pid 11 tid 100015 td 0xffffff0007035000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff8000045fe0, rbp = 0xffffff80000e6b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000e6cf0, rbp = 0 --- Tracing command idle pid 11 tid 100014 td 0xffffff0007035440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000e1b10, rbp = 0xffffff80000e1b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000e1cf0, rbp = 0 --- Tracing command idle pid 11 tid 100013 td 0xffffff0007035880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000dcb10, rbp = 0xffffff80000dcb20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000dccf0, rbp = 0 --- Tracing command idle pid 11 tid 100012 td 0xffffff0007036000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff805a9bd7, rsp = 0xffffff80000d7b00, rbp = 0xffffff80000d7b60 --- _thread_lock_flags() at _thread_lock_flags+0xd7 sched_idletd() at sched_idletd+0xef fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000d7cf0, rbp = 0 --- Tracing command idle pid 11 tid 100011 td 0xffffff0007036440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000d2b10, rbp = 0xffffff80000d2b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000d2cf0, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xffffff0007036880 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff8000068fe0, rbp = 0xffffff80000cdb20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000cdcf0, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xffffff000702b440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff801cf780, rsp = 0xffffff80000c8aa0, rbp = 0xffffff80000c8af0 --- AcpiHwReadPort() at AcpiHwReadPort+0xc0 AcpiHwRead() at AcpiHwRead+0x62 acpi_cpu_idle() at acpi_cpu_idle+0x224 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000c8cf0, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xffffff000702b880 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff8000076fe0, rbp = 0xffffff80000c3b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000c3cf0, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xffffff0007034000 cpustop_handler() at cpustop_handler+0x3c ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x1a8 nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80893f06, rsp = 0xffffff800007dfe0, rbp = 0xffffff80000beb20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000becf0, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xffffff0007034440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000b9b10, rbp = 0xffffff80000b9b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b9cf0, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff0007034880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000b4b10, rbp = 0xffffff80000b4b20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b4cf0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff000702a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000afb10, rbp = 0xffffff80000afb20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000afcf0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff000702a440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x5e Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9c --- interrupt, rip = 0xffffffff80893f06, rsp = 0xffffff80000aab10, rbp = 0xffffff80000aab20 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x214 sched_idletd() at sched_idletd+0x128 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000aacf0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff000702a880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2a6 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x31a kern_wait() at kern_wait+0x3e9 wait4() at wait4+0x35 syscallenter() at syscallenter+0x1aa syscall() at syscall+0x60 Xfast_syscall() at Xfast_syscall+0xe2 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40cafc, rsp = 0x7fffffffe838, rbp = 0x401d80 --- Tracing command audit pid 10 tid 100001 td 0xffffff000702b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a0cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100072 td 0xffffff0007401000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 deadlkres() at deadlkres+0x29b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd813cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100067 td 0xffffff0007404880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd7eccf0, rbp = 0 --- Tracing command kernel pid 0 tid 100066 td 0xffffff0007406000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81cd7e7cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100049 td 0xffffff0007069000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000190cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100047 td 0xffffff0007069880 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000186cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xffffff000706a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000181cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100045 td 0xffffff000706a440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x75 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800017ccf0, rbp = 0 --- Tracing command kernel pid 0 tid 100042 td 0xffffff000706c440 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800016dcf0, rbp = 0 --- Tracing command kernel pid 0 tid 100040 td 0xffffff000706e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x369 taskqueue_thread_loop() at taskqueue_thread_loop+0xc7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000163cf0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80ca3f60 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21b sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x352 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> call doadump Physical memory: 16367 MB Dumping 2319 MB: 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Dump complete = 0 db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 13 (kgdb) bt #0 doadump () at ../../../kern/kern_shutdown.c:248 #1 0xffffffff801e389c in db_fncall (dummy1=Variable "dummy1" is not available. ) at ../../../ddb/db_command.c:548 #2 0xffffffff801e3bd1 in db_command (last_cmdp=0xffffffff80c71080, cmd_table=Variable "cmd_table" is not available. ) at ../../../ddb/db_command.c:445 #3 0xffffffff801e3e20 in db_command_loop () at ../../../ddb/db_command.c:498 #4 0xffffffff801e5e89 in db_trap (type=Variable "type" is not available. ) at ../../../ddb/db_main.c:229 #5 0xffffffff805ebcbe in kdb_trap (type=0xc, code=0x0, tf=0xffffff81d000d490) at ../../../kern/subr_kdb.c:535 #6 0xffffffff808b2a5d in trap_fatal (frame=0xffffff81d000d490, eva=Variable "eva" is not available. ) at ../../../amd64/amd64/trap.c:777 #7 0xffffffff808b3245 in trap (frame=0xffffff81d000d490) at ../../../amd64/amd64/trap.c:297 #8 0xffffffff8089d024 in calltrap () at ../../../amd64/amd64/exception.S:224 #9 0xffffffff80642920 in extattr_delete_fd () #10 0xffffffff807f5289 in ffs_vgetf (mp=0x0, ino=0x854c24f0, flags=0xcab6b5, vpp=0x0, ffs_flags=Variable "ffs_flags" is not available. ) at ../../../ufs/ffs/ffs_vfsops.c:1531 #11 0xffffffff808003ab in ufs_lookup_ino (vdp=0xffffff018b291278, vpp=0xffffff81d000d990, cnp=0xffffff81d000d9b8, dd_ino=0x0) at ../../../ufs/ufs/ufs_lookup.c:608 #12 0xffffffff8090c02f in VOP_CACHEDLOOKUP_APV (vop=0xffffffff80c358e0, a=0xffffff81d000d760) at vnode_if.c:187 #13 0xffffffff8063cfb0 in vfs_cache_lookup (ap=Variable "ap" is not available. ) at vnode_if.h:80 #14 0xffffffff8090e397 in VOP_LOOKUP_APV (vop=0xffffffff80c358e0, a=0xffffff81d000d820) at vnode_if.c:123 #15 0xffffffff80643cb3 in lookup (ndp=0xffffff81d000d960) at vnode_if.h:54 #16 0xffffffff80644c61 in namei (ndp=0xffffff81d000d960) at ../../../kern/vfs_lookup.c:269 #17 0xffffffff80654388 in kern_unlinkat (td=0xffffff009c1be440, fd=0xffffff9c, path=0x7fffffffe720
, pathseg=UIO_USERSPACE, oldinum=0x0) at ../../../kern/vfs_syscalls.c:1836 #18 0xffffffff805f8faa in syscallenter (td=0xffffff009c1be440, sa=0xffffff81d000dba0) at ../../../kern/subr_trap.c:319 #19 0xffffffff808b2e30 in syscall (frame=0xffffff81d000dc40) at ../../../amd64/amd64/trap.c:887 #20 0xffffffff8089d302 in Xfast_syscall () at ../../../amd64/amd64/exception.S:377 #21 0x000000080084679c in ?? () (kgdb) f 9 #9 0xffffffff80642920 in extattr_delete_fd () (kgdb) l 377 call syscall 378 movq PCPU(CURPCB),%rax 379 andq $~PCB_FULLCTX,PCB_FLAGS(%rax) 380 MEXITCOUNT 381 jmp doreti 382 383 /* 384 * Here for CYA insurance, in case a "syscall" instruction gets 385 * issued from 32 bit compatability mode. MSR_CSTAR has to point 386 * to *something* if EFER_SCE is enabled. (kgdb) up #10 0xffffffff807f5289 in ffs_vgetf (mp=0x0, ino=0x854c24f0, flags=0xcab6b5, vpp=0x0, ffs_flags=Variable "ffs_flags" is not available. ) at ../../../ufs/ffs/ffs_vfsops.c:1531 1531 error = vfs_hash_insert(vp, ino, flags, curthread, vpp, NULL, NULL); (kgdb) l 1526 uma_zfree(uma_inode, ip); 1527 *vpp = NULL; 1528 return (error); 1529 } 1530 vp->v_vflag &= ~VV_FORCEINSMQ; 1531 error = vfs_hash_insert(vp, ino, flags, curthread, vpp, NULL, NULL); 1532 if (error || *vpp != NULL) 1533 return (error); 1534 1535 /* Read in the disk contents for the inode, copy into the inode. */ (kgdb) info loc fs = (struct fs *) 0xffffff0007558000 ip = (struct inode *) 0xffffff0186a11bd0 ump = (struct ufsmount *) 0xffffff0007bc9600 bp = Variable "bp" is not available. (kgdb) p *vp $4 = {v_type = VNON, v_tag = 0xffffffff809c471d "ufs", v_op = 0xffffffff80c350a0, v_data = 0xffffff0186a11bd0, v_mount = 0xffffff0134545390, v_nmntvnodes = {tqe_next = 0xffffff0381989c58, tqe_prev = 0xffffff018ba32518}, v_un = { vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0, vu_yield = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_hash = 0xcab6b5, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xffffff01854c2550}, v_cache_dd = 0x0, v_cstart = 0x0, v_lasta = 0x0, v_lastw = 0x0, v_clen = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff809c471d "ufs", lo_flags = 0x57b0008, lo_data = 0x0, lo_witness = 0xffffff8000223380}, lk_lock = 0xffffff009c1be440, lk_exslpfail = 0x0, lk_timo = 0x33, lk_pri = 0x50, lk_stack = {depth = 0xc, pcs = {0xffffffff805a1a64, 0xffffffff807f520f, 0xffffffff808003ab, 0xffffffff8090c02f, 0xffffffff8063cfb0, 0xffffffff8090e397, 0xffffffff80643cb3, 0xffffffff80644c61, 0xffffffff80654388, 0xffffffff805f8faa, 0xffffffff808b2e30, 0xffffffff8089d302, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}, v_interlock = {lock_object = { lo_name = 0xffffffff809d5432 "vnode interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff800021b600}, mtx_lock = 0x4}, v_vnlock = 0xffffff01854c2588, v_holdcnt = 0x1, v_usecount = 0x1, v_iflag = 0x0, v_vflag = 0x0, v_writecount = 0x0, v_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xffffffff809de57b "bufobj interlock", lo_flags = 0x1030000, lo_data = 0x0, lo_witness = 0xffffff8000221380}, mtx_lock = 0x4}, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff01854c26c8}, bv_root = 0x0, bv_cnt = 0x0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xffffff01854c26e8}, bv_root = 0x0, bv_cnt = 0x0}, bo_numoutput = 0x0, bo_flag = 0x0, bo_ops = 0xffffffff80c178c0, bo_bsize = 0x4000, bo_object = 0x0, bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xffffff01854c24f0, __bo_vnode = 0xffffff01854c24f0}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) p vpp $5 = (struct vnode **) 0x0 (kgdb) l *0xffffffff80642920 0xffffffff80642920 is in vfs_hash_index (../../../kern/vfs_hash.c:59). 54 /* Must be SI_ORDER_SECOND so desiredvnodes is available */ 55 SYSINIT(vfs_hash, SI_SUB_VFS, SI_ORDER_SECOND, vfs_hashinit, NULL); 56 57 static struct vfs_hash_head * 58 vfs_hash_index(const struct mount *mp, u_int hash) 59 { 60 61 return(&vfs_hash_tbl[(hash + mp->mnt_hashseed) & vfs_hash_mask]); 62 } 63 (kgdb) -- Test scenario: run.sh vfs.cfg