KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0: Sat Apr 24 10:13:16 UTC 2010 test@worm2.netperf.freebsd.org:/usr/src/sys/amd64/compile/NODEBUG amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Quad-Core AMD Opteron(tm) Processor 8354 (2211.35-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Family = 10 Model = 2 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 18119393280 (17280 MB) avail memory = 16547627008 (15781 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 4 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 cpu12 (AP): APIC ID: 12 cpu13 (AP): APIC ID: 13 cpu14 (AP): APIC ID: 14 cpu15 (AP): APIC ID: 15 ioapic0: Changing APIC ID to 16 ioapic1: Changing APIC ID to 17 ioapic2: Changing APIC ID to 18 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 40-46 on motherboard ioapic2 irqs 47-53 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of e0000000, 10000000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, c7f00000 (3) failed Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 25000000 Hz quality 900 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xdeebf000-0xdeebffff irq 21 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ehci0: mem 0xdeebec00-0xdeebecff irq 22 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 4.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xc480-0xc487,0xc400-0xc403,0xc080-0xc087,0xc000-0xc003,0xbc00-0xbc0f mem 0xdeebd000-0xdeebdfff irq 23 at device 5.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0xb880-0xb887,0xb800-0xb803,0xb480-0xb487,0xb400-0xb403,0xb080-0xb08f mem 0xdeebc000-0xdeebcfff irq 20 at device 5.1 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] atapci3: port 0xb000-0xb007,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa48f mem 0xdeebb000-0xdeebbfff irq 21 at device 5.2 on pci0 atapci3: [ITHREAD] ata6: on atapci3 ata6: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] pcib1: at device 6.0 on pci0 pci1: on pcib1 vgapci0: port 0xd000-0xd0ff mem 0xd0000000-0xd7ffffff,0xdeff0000-0xdeffffff irq 16 at device 1.0 on pci1 pcib2: at device 10.0 on pci0 pci2: on pcib2 cxgbc0: mem 0xdffff000-0xdfffffff,0xdf000000-0xdf7fffff,0xdfffe000-0xdfffefff irq 17 at device 0.0 on pci2 prep adapter failed cxgbc0: no msi message to release device_attach: cxgbc0 attach returned 19 pcib3: at device 11.0 on pci0 pci3: on pcib3 pcib4: at device 12.0 on pci0 pci4: on pcib4 pcib5: at device 13.0 on pci0 pci5: on pcib5 pcib6: at device 14.0 on pci0 pci6: on pcib6 pcib7: at device 15.0 on pci0 pci7: on pcib7 pcib8: on acpi0 pci8: on pcib8 pcib9: at device 1.0 on pci8 pci9: on pcib9 pcib10: at device 2.0 on pci8 pci10: on pcib10 em0: port 0xec00-0xec3f mem 0xcffe0000-0xcfffffff irq 47 at device 3.0 on pci10 em0: [FILTER] em0: Ethernet address: 00:30:48:bf:aa:aa em1: port 0xe880-0xe8bf mem 0xcffa0000-0xcffbffff irq 48 at device 3.1 on pci10 em1: [FILTER] em1: Ethernet address: 00:30:48:bf:aa:ab acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 hwpstate0: on cpu0 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 uhub0: 10 ports with 10 removable, self powered acd0: DVDROM at ata0-slave UDMA33 SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #13 Launched! WARNING: WITNESS option enabled, expect reduced performance. Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 ugen1.2: at usbus1 ums0: on usbus1 ums0: 3 buttons and [Z] coordinates ID=0 ukbd0: on usbus1 kbd2 at ukbd0 Trying to mount root from nfs: NFS ROOT: 192.168.5.1:/zoo/worm2/ em0: link state changed to UP Interface em0 IP-Address 192.168.5.121 Broadcast 192.168.5.255 Setting hostuuid: 00020003-0004-0005-0006-000700080009. Setting hostid: 0x81f4ec68. No suitable dump device was found. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/worm2 Mounting local file systems:. Setting hostname: worm2.netperf.freebsd.org. Starting Network: lo0 em0 em1 plip0. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=389b ether 00:30:48:bf:aa:aa inet 192.168.5.121 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:febf:aaaa%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8802 metric 0 mtu 1500 options=389b ether 00:30:48:bf:aa:ab media: Ethernet autoselect status: no carrier plip0: flags=8810 metric 0 mtu 1500 Starting devd. Starting Network: em1. em1: flags=8802 metric 0 mtu 1500 options=389b ether 00:30:48:bf:aa:ab media: Ethernet autoselect status: no carrier Starting Network: plip0. plip0: flags=8810 metric 0 mtu 1500 Starting ums0 moused. DHCPDISCOVER on em0 to 255.255.255.255 port 67 interval 6 DHCPOFFER from 192.168.5.1 DHCPREQUEST on em0 to 255.255.255.255 port 67 DHCPACK from 192.168.5.1 bound to 192.168.5.121 -- renewal in 300 seconds. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Waiting 30s for the default route interface: ............................. Creating and/or trimming log files. Starting syslogd. /etc/rc: WARNING: Dump device does not exist. Savecore not run. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting statd. Starting lockd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Sat Apr 24 10:32:13 UTC 2010 FreeBSD/amd64 (worm2.netperf.freebsd.org) (ttyu0) login: Apr 24 11:03:58 worm2 su: test to root on /dev/pts/0 lock order reversal: 1st 0xffffff814aa868d8 bufwait (bufwait) @ kern/vfs_bio.c:2559 2nd 0xffffff0006bfb400 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x33 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x889 ufs_makeinode() at ufs_makeinode+0x239 VOP_CREATE_APV() at VOP_CREATE_APV+0x8d vn_open_cred() at vn_open_cred+0x46a kern_openat() at kern_openat+0x179 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800848e7c, rsp = 0x7fffffffe628, rbp = 0x7fffffffe6e0 --- lock order reversal: 1st 0xffffff0006ae6bd8 nfs (nfs) @ kern/vfs_mount.c:1201 2nd 0xffffff00bd1899f8 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1232 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd11 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x47 ffs_flushfiles() at ffs_flushfiles+0xb5 ffs_unmount() at ffs_unmount+0x67 dounmount() at dounmount+0x2e6 unmount() at unmount+0x27e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a3a6c, rsp = 0x7fffffffe0d8, rbp = 0x800c08360 --- Apr 24 11:31:59 worm2 kernel: pid 2425 (cp), uid 0 inumber 2374 on /mnt: filesystem full Apr 24 11:31:59 worm2 kernel: pid 2427 (cp), uid 0 inumber 448614 on /mnt: filesystem full Apr 24 12:42:47 worm2 kernel: pid 49216 (cp), uid 0 inumber 567408 on /mnt: filesystem full Apr 24 12:42:47 worm2 kernel: pid 49214 (cp), uid 0 inumber 356496 on /mnt: filesystem full lock order reversal: 1st 0xffffff02aafbd818 ufs (ufs) @ kern/vfs_vnops.c:611 2nd 0xffffff814bcf9fb8 bufwait (bufwait) @ kern/vfs_bio.c:1835 3rd 0xffffff029ca7fbd8 ufs (ufs) @ kern/vfs_subr.c:2091 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd8e ffs_lock() at ffs_lock+0x8c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x47 vget() at vget+0x7b vm_fault() at vm_fault+0x1606 trap_pfault() at trap_pfault+0x118 trap() at trap+0x37f calltrap() at calltrap+0x8 --- trap 0xc, rip = 0xffffffff808944ed, rsp = 0xffffff81cf4077c0, rbp = 0xffffff81cf407840 --- copyin() at copyin+0x3d ffs_write() at ffs_write+0x32d VOP_WRITE_APV() at VOP_WRITE_APV+0x103 vn_write() at vn_write+0x2a2 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800748fec, rsp = 0x7fffffffe6c8, rbp = 0x80053e000 --- ZFS filesystem version 3 ZFS storage pool version 14 Apr 24 15:29:57 worm2 kernel: pid 20363 (fstool), uid 0 inumber 1718930 on /mnt: out of inodes Apr 24 16:01:08 worm2 kernel: pid 20590 (fstool), uid 0 inumber 1860391 on /mnt: out of inodes Apr 24 16:24:56 worm2 kernel: pid 21133 (cp), uid 0 inumber 430271 on /mnt: filesystem full Apr 24 16:24:56 worm2 kernel: pid 21135 (cp), uid 0 inumber 518672 on /mnt: filesystem full panic: bad pte cpuid = 0 KDB: enter: panic [ thread pid 46100 tid 100443 ] Stopped at kdb_enter+0x3d: movq $0,0x6b9a50(%rip) bt Tracing pid 46100 tid 100443 td 0xffffff012cb62720 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b pmap_remove_pages() at pmap_remove_pages+0x632 vmspace_exit() at vmspace_exit+0x9a exit1() at exit1+0x376 sys_exit() at sys_exit+0xe syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x8006f6f3c, rsp = 0x7fffffffe698, rbp = 0 --- db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x48b300 curthread = 0xffffff012cb62720: pid 46100 "cp" curpcb = 0xffffff81cf3c1d40 fpcurthread = none idlethread = 0xffffff0002fcbab0: pid 11 "idle: cpu0" curpmap = 0 tssp = 0xffffffff80e7ed80 commontssp = 0xffffffff80e7ed80 rsp0 = 0xffffff81cf3c1d40 gs32p = 0xffffffff80e7dbb8 ldt = 0xffffffff80e7dbf8 tss = 0xffffffff80e7dbe8 spin locks held: cpuid = 1 dynamic pcpu = 0xffffff807f40c300 curthread = 0xffffff0002fcc000: pid 11 "idle: cpu1" --More-- curpcb = 0xffffff80000ffd40 fpcurthread = none idlethread = 0xffffff0002fcc000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff80e7ede8 commontssp = 0xffffffff80e7ede8 rsp0 = 0xffffff80000ffd40 gs32p = 0xffffffff80e7dc20 ldt = 0xffffffff80e7dc60 tss = 0xffffffff80e7dc50 spin locks held: cpuid = 2 dynamic pcpu = 0xffffff807f413300 curthread = 0xffffff0002fcc390: pid 11 "idle: cpu2" curpcb = 0xffffff80000fad40 fpcurthread = none idlethread = 0xffffff0002fcc390: pid 11 "idle: cpu2" curpmap = 0 tssp = 0xffffffff80e7ee50 --More-- commontssp = 0xffffffff80e7ee50 rsp0 = 0xffffff80000fad40 gs32p = 0xffffffff80e7dc88 ldt = 0xffffffff80e7dcc8 tss = 0xffffffff80e7dcb8 spin locks held: cpuid = 3 dynamic pcpu = 0xffffff807f41a300 curthread = 0xffffff0002fcc720: pid 11 "idle: cpu3" curpcb = 0xffffff80000f5d40 fpcurthread = none idlethread = 0xffffff0002fcc720: pid 11 "idle: cpu3" curpmap = 0 tssp = 0xffffffff80e7eeb8 commontssp = 0xffffffff80e7eeb8 rsp0 = 0xffffff80000f5d40 gs32p = 0xffffffff80e7dcf0 ldt = 0xffffffff80e7dd30 tss = 0xffffffff80e7dd20 --More-- spin locks held: cpuid = 4 dynamic pcpu = 0xffffff807f421300 curthread = 0xffffff0002fccab0: pid 11 "idle: cpu4" curpcb = 0xffffff80000f0d40 fpcurthread = none idlethread = 0xffffff0002fccab0: pid 11 "idle: cpu4" curpmap = 0 tssp = 0xffffffff80e7ef20 commontssp = 0xffffffff80e7ef20 rsp0 = 0xffffff80000f0d40 gs32p = 0xffffffff80e7dd58 ldt = 0xffffffff80e7dd98 tss = 0xffffffff80e7dd88 spin locks held: cpuid = 5 dynamic pcpu = 0xffffff807f428300 curthread = 0xffffff0002fcd000: pid 11 "idle: cpu5" --More-- curpcb = 0xffffff80000ebd40 fpcurthread = none idlethread = 0xffffff0002fcd000: pid 11 "idle: cpu5" curpmap = 0 tssp = 0xffffffff80e7ef88 commontssp = 0xffffffff80e7ef88 rsp0 = 0xffffff80000ebd40 gs32p = 0xffffffff80e7ddc0 ldt = 0xffffffff80e7de00 tss = 0xffffffff80e7ddf0 spin locks held: cpuid = 6 dynamic pcpu = 0xffffff807f42f300 curthread = 0xffffff0002fcd390: pid 11 "idle: cpu6" curpcb = 0xffffff80000e6d40 fpcurthread = none idlethread = 0xffffff0002fcd390: pid 11 "idle: cpu6" curpmap = 0 tssp = 0xffffffff80e7eff0 --More-- commontssp = 0xffffffff80e7eff0 rsp0 = 0xffffff80000e6d40 gs32p = 0xffffffff80e7de28 ldt = 0xffffffff80e7de68 tss = 0xffffffff80e7de58 spin locks held: cpuid = 7 dynamic pcpu = 0xffffff807f436300 curthread = 0xffffff0002fc1390: pid 11 "idle: cpu7" curpcb = 0xffffff80000e1d40 fpcurthread = none idlethread = 0xffffff0002fc1390: pid 11 "idle: cpu7" curpmap = 0 tssp = 0xffffffff80e7f058 commontssp = 0xffffffff80e7f058 rsp0 = 0xffffff80000e1d40 gs32p = 0xffffffff80e7de90 ldt = 0xffffffff80e7ded0 tss = 0xffffffff80e7dec0 --More-- spin locks held: cpuid = 8 dynamic pcpu = 0xffffff807f43d300 curthread = 0xffffff0002fc1720: pid 11 "idle: cpu8" curpcb = 0xffffff80000dcd40 fpcurthread = none idlethread = 0xffffff0002fc1720: pid 11 "idle: cpu8" curpmap = 0 tssp = 0xffffffff80e7f0c0 commontssp = 0xffffffff80e7f0c0 rsp0 = 0xffffff80000dcd40 gs32p = 0xffffffff80e7def8 ldt = 0xffffffff80e7df38 tss = 0xffffffff80e7df28 spin locks held: cpuid = 9 dynamic pcpu = 0xffffff807f444300 curthread = 0xffffff0002fc1ab0: pid 11 "idle: cpu9" --More-- curpcb = 0xffffff80000d7d40 fpcurthread = none idlethread = 0xffffff0002fc1ab0: pid 11 "idle: cpu9" curpmap = 0 tssp = 0xffffffff80e7f128 commontssp = 0xffffffff80e7f128 rsp0 = 0xffffff80000d7d40 gs32p = 0xffffffff80e7df60 ldt = 0xffffffff80e7dfa0 tss = 0xffffffff80e7df90 spin locks held: cpuid = 10 dynamic pcpu = 0xffffff807f44b300 curthread = 0xffffff0002fcb000: pid 11 "idle: cpu10" curpcb = 0xffffff80000d2d40 fpcurthread = none idlethread = 0xffffff0002fcb000: pid 11 "idle: cpu10" curpmap = 0 tssp = 0xffffffff80e7f190 --More-- commontssp = 0xffffffff80e7f190 rsp0 = 0xffffff80000d2d40 gs32p = 0xffffffff80e7dfc8 ldt = 0xffffffff80e7e008 tss = 0xffffffff80e7dff8 spin locks held: cpuid = 11 dynamic pcpu = 0xffffff807f452300 curthread = 0xffffff0002fcb390: pid 11 "idle: cpu11" curpcb = 0xffffff80000cdd40 fpcurthread = none idlethread = 0xffffff0002fcb390: pid 11 "idle: cpu11" curpmap = 0 tssp = 0xffffffff80e7f1f8 commontssp = 0xffffffff80e7f1f8 rsp0 = 0xffffff80000cdd40 gs32p = 0xffffffff80e7e030 ldt = 0xffffffff80e7e070 tss = 0xffffffff80e7e060 --More-- spin locks held: cpuid = 12 dynamic pcpu = 0xffffff807f459300 curthread = 0xffffff0002fcb720: pid 11 "idle: cpu12" curpcb = 0xffffff80000c8d40 fpcurthread = none idlethread = 0xffffff0002fcb720: pid 11 "idle: cpu12" curpmap = 0 tssp = 0xffffffff80e7f260 commontssp = 0xffffffff80e7f260 rsp0 = 0xffffff80000c8d40 gs32p = 0xffffffff80e7e098 ldt = 0xffffffff80e7e0d8 tss = 0xffffffff80e7e0c8 spin locks held: cpuid = 13 dynamic pcpu = 0xffffff807f460300 curthread = 0xffffff0002fbf000: pid 11 "idle: cpu13" --More-- curpcb = 0xffffff80000c3d40 fpcurthread = none idlethread = 0xffffff0002fbf000: pid 11 "idle: cpu13" curpmap = 0 tssp = 0xffffffff80e7f2c8 commontssp = 0xffffffff80e7f2c8 rsp0 = 0xffffff80000c3d40 gs32p = 0xffffffff80e7e100 ldt = 0xffffffff80e7e140 tss = 0xffffffff80e7e130 spin locks held: cpuid = 14 dynamic pcpu = 0xffffff807f467300 curthread = 0xffffff0002fbf390: pid 11 "idle: cpu14" curpcb = 0xffffff80000bed40 fpcurthread = none idlethread = 0xffffff0002fbf390: pid 11 "idle: cpu14" curpmap = 0 tssp = 0xffffffff80e7f330 --More-- commontssp = 0xffffffff80e7f330 rsp0 = 0xffffff80000bed40 gs32p = 0xffffffff80e7e168 ldt = 0xffffffff80e7e1a8 tss = 0xffffffff80e7e198 spin locks held: cpuid = 15 dynamic pcpu = 0xffffff807f46e300 curthread = 0xffffff0002fbf720: pid 11 "idle: cpu15" curpcb = 0xffffff80000b9d40 fpcurthread = none idlethread = 0xffffff0002fbf720: pid 11 "idle: cpu15" curpmap = 0 tssp = 0xffffffff80e7f398 commontssp = 0xffffffff80e7f398 rsp0 = 0xffffff80000b9d40 gs32p = 0xffffffff80e7e1d0 ldt = 0xffffffff80e7e210 tss = 0xffffffff80e7e200 --More-- spin locks held: db> set $lines 0 db> show locks exclusive sleep mutex pmap (pmap) r = 0 (0xffffff0121dd3750) locked @ amd64/amd64/pmap.c:4002 exclusive sleep mutex vm page queue mutex (vm page queue mutex) r = 0 (0xffffffff80e45500) locked @ amd64/amd64/pmap.c:4001 db> ps pid ppid pgrp uid state wmesg wchan cmd 46100 44447 20504 0 RE+ CPU 0 cp 44447 44433 20504 0 S+ wait 0xffffff0006e3f460 sh 44446 44433 20504 0 S+ pipewr 0xffffff012cb525b0 find 44442 0 0 0 DL mdwait 0xffffff03982a3800 [md5] 44433 20504 20504 0 S+ wait 0xffffff0121dcb8c0 sh 44300 0 0 0 SL - 0xffffffff80e3c6e0 [nfsiod 0] 20504 1629 20504 0 S+ wait 0xffffff022ae24000 sh 95670 0 0 0 DL (threaded) [zfskern] 100639 D l2arc_fe 0xffffffff8130a6c0 [l2arc_feed_thread] 100261 D arc_recl 0xffffffff812fa820 [arc_reclaim_thread] 1640 1639 1640 1001 Ss+ ttyin 0xffffff00064964a8 sh 1639 1636 1636 1001 S select 0xffffff00bd219740 sshd 1636 1452 1636 0 Ss sbwait 0xffffff0006e04144 sshd 1634 0 0 0 DL mdwait 0xffffff00bd268000 [md4] 1629 1625 1629 0 S+ wait 0xffffff00bd1ff000 sh 1625 1624 1625 0 S+ pause 0xffffff0006de1960 csh 1624 1623 1624 1001 S+ wait 0xffffff0006de3000 su 1623 1622 1623 1001 Ss+ wait 0xffffff00069c3460 sh 1622 1617 1617 1001 S select 0xffffff000671bb40 sshd 1617 1452 1617 0 Ss sbwait 0xffffff0006e043ec sshd 1542 1 1 0 S ttydcd 0xffffff000642d8e8 getty 1541 1 1541 0 Ss+ ttyin 0xffffff000642d4a8 getty 1540 1 1540 0 Ss+ ttyin 0xffffff000647aca8 getty 1539 1 1539 0 Ss+ ttyin 0xffffff000647b0a8 getty 1538 1 1538 0 Ss+ ttyin 0xffffff000647b4a8 getty 1537 1 1537 0 Ss+ ttyin 0xffffff000647b8a8 getty 1471 1 1471 0 Ss nanslp 0xffffffff80c742c8 cron 1464 1 1464 25 Ss pause 0xffffff0006de20a0 sendmail 1460 1 1460 0 Ss select 0xffffff00068616c0 sendmail 1452 1 1452 0 Ss select 0xffffff0006861a40 sshd 1322 1 1322 0 Ss rpcsvc 0xffffff00065ad3a0 NLM: master 1315 1 1315 0 Ss select 0xffffff0006b55d40 rpc.statd 1242 0 0 0 DL mdwait 0xffffff0006519000 [md1] 1187 1 1187 0 Ss select 0xffffff00065ad0c0 rpcbind 1156 1 1156 0 Ss select 0xffffff0006b62cc0 syslogd 881 1 881 0 Ss select 0xffffff00068ec440 devd 880 1 880 65 Ss select 0xffffff0006b555c0 dhclient 864 1 864 0 Ss select 0xffffff000671bdc0 dhclient 821 1 821 0 Ss select 0xffffff00068ec140 moused 102 0 0 0 DL mdwait 0xffffff0006a8c800 [md0] 19 0 0 0 DL flowclea 0xffffffff80e35550 [flowcleaner] 18 0 0 0 DL sdflush 0xffffffff80e444d8 [softdepflush] 17 0 0 0 DL vlruwt 0xffffff00065308c0 [vnlru] 16 0 0 0 DL syncer 0xffffffff80e352a0 [syncer] 15 0 0 0 DL psleep 0xffffffff80e34dc8 [bufdaemon] 9 0 0 0 DL pgzero 0xffffffff80e4624c [pagezero] 8 0 0 0 DL psleep 0xffffffff80e455e8 [vmdaemon] 7 0 0 0 DL psleep 0xffffffff80e455ac [pagedaemon] 6 0 0 0 DL ccb_scan 0xffffffff80c3c760 [xpt_thrd] 5 0 0 0 DL - 0xffffff0006459248 [fdc0] 14 0 0 0 DL (threaded) [usb] 100061 D - 0xffffff80009dcdd0 [usbus1] 100060 D - 0xffffff80009dcd78 [usbus1] 100059 D - 0xffffff80009dcd20 [usbus1] 100058 D - 0xffffff80009dccc8 [usbus1] 100056 D - 0xffffff80009d4460 [usbus0] 100055 D - 0xffffff80009d4408 [usbus0] 100054 D - 0xffffff80009d43b0 [usbus0] 100053 D - 0xffffff80009d4358 [usbus0] 13 0 0 0 DL - 0xffffffff80c73fa4 [yarrow] 4 0 0 0 DL - 0xffffffff80c70768 [g_down] 3 0 0 0 DL - 0xffffffff80c70760 [g_up] 2 0 0 0 DL - 0xffffffff80c70750 [g_event] 12 0 0 0 WL (threaded) [intr] 100071 I [irq1: atkbd0] 100070 I [irq7: ppc0] 100068 I [swi0: uart uart] 100065 I [irq20: atapci2] 100064 I [irq23: atapci1] 100063 I [irq15: ata1] 100062 I [irq14: ata0] 100057 I [irq22: ehci0] 100052 I [irq21: ohci0+] 100051 I [irq9: acpi0] 100050 I [swi6: task queue] 100049 I [swi6: Giant taskq] 100047 I [swi5: +] 100042 I [swi2: cambio] 100036 I [swi4: clock] 100035 I [swi4: clock] 100034 I [swi4: clock] 100033 I [swi4: clock] 100032 I [swi4: clock] 100031 I [swi4: clock] 100030 I [swi4: clock] 100029 I [swi4: clock] 100028 I [swi4: clock] 100027 I [swi4: clock] 100026 I [swi4: clock] 100025 I [swi4: clock] 100024 I [swi4: clock] 100023 I [swi4: clock] 100022 I [swi4: clock] 100021 I [swi4: clock] 100020 I [swi3: vm] 100019 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100018 CanRun [idle: cpu0] 100017 Run CPU 1 [idle: cpu1] 100016 Run CPU 2 [idle: cpu2] 100015 Run CPU 3 [idle: cpu3] 100014 Run CPU 4 [idle: cpu4] 100013 Run CPU 5 [idle: cpu5] 100012 Run CPU 6 [idle: cpu6] 100011 Run CPU 7 [idle: cpu7] 100010 Run CPU 8 [idle: cpu8] 100009 Run CPU 9 [idle: cpu9] 100008 Run CPU 10 [idle: cpu10] 100007 Run CPU 11 [idle: cpu11] 100006 Run CPU 12 [idle: cpu12] 100005 Run CPU 13 [idle: cpu13] 100004 Run CPU 14 [idle: cpu14] 100003 Run CPU 15 [idle: cpu15] 1 0 1 0 SLs wait 0xffffff0002fbc8c0 [init] 10 0 0 0 DL audit_wo 0xffffffff80e43810 [audit] 0 0 0 0 DLs (threaded) [kernel] 100072 D deadlkre 0xffffffff80c73fa4 [deadlkres] 100067 D - 0xffffff0006424880 [em1 taskq] 100066 D - 0xffffff0006429080 [em0 taskq] 100048 D - 0xffffff000617bd80 [thread taskq] 100046 D - 0xffffff0006160380 [kqueue taskq] 100045 D - 0xffffff0006160400 [acpi_task_2] 100044 D - 0xffffff0006160400 [acpi_task_1] 100043 D - 0xffffff0006160400 [acpi_task_0] 100040 D - 0xffffff0002fe8580 [firmware taskq] 100000 D sched 0xffffffff80c708a0 [swapper] db> show mount 0xffffff000663b5f0 192.168.5.1:/zoo/worm2/ on / (nfs) 0xffffff0006622000 devfs on /dev (devfs) 0xffffff0006ab8000 /dev/md0 on /var (ufs) 0xffffff0006621be0 /dev/md1 on /tmp (ufs) 0xffffff00066212f8 /dev/md5a on /mnt (ufs) More info: show mount db> reset -- Test scenario: t5.sh