HEAD r222559, vmcore.25 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0: Wed Jun 1 15:53:02 CEST 2011 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.05-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2084655104 (1988 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pcib5: failed to allocate initial I/O port window: 0-0xfff pcib5: failed to allocate initial memory window: 0-0xfffff pcib5: failed to allocate initial prefetch window: 0-0xfffff pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: on atapci0 ahci0: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc09eb730(0xc57165c0) 0.011015366 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc0521860(0xc5737c80) 0.020265424 s ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-7cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! Timecounter "TSC" frequency 2128045488 Hz quality 800 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered Root mount waiting for: usbus4 Root mount waiting for: usbus4 Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface em0 IP-Address 192.168.5.109 Broadcast 192.168.5.255 Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0d: clean, 112672236 free (1484 frags, 14083844 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:68 inet 192.168.5.109 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:fe8e:d068%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 1 Jun 18:11:18 ntpdate[1387]: step time server 192.168.5.1 offset -1.447137 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Starting inetd. Wed Jun 1 18:11:19 UTC 2011 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: root Jun 1 18:13:50 leopard3 login: ROOT LOGIN (root) ON ttyu0 FreeBSD 9.0-CURRENT (PHO) #0: Wed Jun 1 15:53:02 CEST 2011 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. leopard3# ~test/keepalive.sh Wed Jun 1 20:13:59 CEST 2011 8:13PM up 3 mins, 1 user, load averages: 0.25, 0.37, 0.18 lock order reversal: 1st 0xdef14aa0 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 2nd 0xc623b400 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0e9181e,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c09e4f6b,c0e952bb,c55442a0,c5548610,ed96e894,...) at kdb_backtrace+0x2a _witness_debugger(c0e952bb,c623b400,c0ec485a,c5548610,c0ec44db,...) at _witness_debugger+0x25 witness_checkorder(c623b400,9,c0ec44db,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c623b400,0,c0ec44db,11c,c5ff91d0,...) at _sx_xlock+0x85 ufsdirhash_acquire(def14a40,ed96e9ec,1f4,dff4c018,ed96e964,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c5ff91d0,ed96e9ec,18,ed96e950,ed96e954,...) at ufsdirhash_add+0x13 ufs_direnter(c617f414,c6233c3c,ed96e9ec,ed96ebd0,def2cb60,...) at ufs_direnter+0x749 ufs_mkdir(ed96ebf8,c0ee2a5d,0,0,ed96eb3c,...) at ufs_mkdir+0x916 VOP_MKDIR_APV(c0fc2d00,ed96ebf8,ed96ebd0,ed96eb3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c61842e0,ffffff9c,804d360,0,1f8,...) at kern_mkdirat+0x225 kern_mkdir(c61842e0,804d360,0,1f8,ed96ec7c,...) at kern_mkdir+0x2e mkdir(c61842e0,ed96ecec,ed96ed28,c0e93ae6,0,...) at mkdir+0x29 syscallenter(c61842e0,ed96ece4,ed96ece4,0,0,...) at syscallenter+0x263 syscall(ed96ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28187b33, esp = 0xbfbfe77c, ebp = 0xbfbfe818 --- Wed Jun 1 20:23:59 CEST 2011 8:23PM up 13 mins, 2 users, load averages: 0.30, 2.72, 2.85 Wed Jun 1 20:34:02 CEST 2011 8:34PM up 23 mins, 2 users, load averages: 1.34, 0.73, 1.56 Wed Jun 1 20:44:02 CEST 2011 8:44PM up 33 mins, 2 users, load averages: 0.66, 0.45, 0.94 Wed Jun 1 20:54:03 CEST 2011 8:54PM up 43 mins, 2 users, load averages: 0.34, 0.30, 0.56 Wed Jun 1 21:04:16 CEST 2011 9:04PM up 53 mins, 2 users, load averages: 21.94, 13.74, 7.07 Wed Jun 1 21:14:17 CEST 2011 9:14PM up 1:03, 2 users, load averages: 101.22, 63.93, 33.63 Wed Jun 1 21:24:33 CEST 2011 9:24PM up 1:14, 2 users, load averages: 0.95, 19.47, 27.35 Wed Jun 1 21:34:35 CEST 2011 9:34PM up 1:24, 2 users, load averages: 16.74, 15.00, 19.39 lock order reversal: 1st 0xc6048460 so_snd_sx (so_snd_sx) @ /mnt/pho/src/sys/kern/uipc_sockbuf.c:148 2nd 0xdeee87c0 bufwait (bufwait) @ /mnt/pho/src/sys/vm/vm_pager.c:309 KDB: stack backtrace: db_trace_self_wrapper(c0e9181e,2f6d762f,705f6d76,72656761,333a632e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c09e4f6b,c0e952bb,c5547f28,c55442a0,ed9bf69c,...) at kdb_backtrace+0x2a _witness_debugger(c0e952bb,deee87c0,c0e9b11c,c55442a0,c0ec8f9e,...) at _witness_debugger+0x25 witness_checkorder(deee87c0,9,c0ec8f9e,135,0,...) at witness_checkorder+0x839 __lockmgr_args(deee87c0,80000,0,0,0,...) at __lockmgr_args+0x814 initpbuf(c11984ec,0,c0ec8f9e,160,0,...) at initpbuf+0xe0 getpbuf(c1196bb0,0,c0ec5942,486,0,...) at getpbuf+0x12f swap_pager_getpages(c62cee58,ed9bf8c8,3,0,ed9bf94c,...) at swap_pager_getpages+0x1bd vm_fault_hold(c603d740,28475000,1,0,0,...) at vm_fault_hold+0x104f vm_fault(c603d740,28475000,1,0,28475020,...) at vm_fault+0x2f trap_pfault(5,0,c0edc4a6,ed9bfa00,c6254834,...) at trap_pfault+0x10d trap(ed9bfa64) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0cd4202, esp = 0xed9bfaa4, ebp = 0xed9bfad8 --- copyin(c6589d18,20,ed9bfc28,1,0,...) at copyin+0x32 m_uiotombuf(ed9bfc28,2,a800,0,0,...) at m_uiotombuf+0xf6 sosend_generic(c6048340,0,ed9bfc28,0,0,...) at sosend_generic+0x432 sosend(c6048340,0,ed9bfc28,0,0,...) at sosend+0x3f soo_write(c605a700,ed9bfc28,c57ff580,0,c5eab2e0,...) at soo_write+0x63 dofilewrite(ed9bfc28,ffffffff,ffffffff,0,c605a700,...) at dofilewrite+0x95 kern_writev(c5eab2e0,3,ed9bfc28,ed9bfc48,1,...) at kern_writev+0x58 write(c5eab2e0,ed9bfcec,ed9bfd28,c0e93ae6,0,...) at write+0x4f syscallenter(c5eab2e0,ed9bfce4,ed9bfce4,0,c1026f80,...) at syscallenter+0x263 syscall(ed9bfd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x28194da3, esp = 0xbfbfe8fc, ebp = 0xbfbfe948 --- Expensive timeout(9) function: 0xc0647080(0xc56bae00) 0.082897127 s Wed Jun 1 21:45:34 CEST 2011 9:45PM up 1:35, 2 users, load averages: 0.40, 7.79, 14.30 Wed Jun 1 21:55:42 CEST 2011 9:55PM up 1:45, 2 users, load averages: 0.69, 3.97, 8.98 tvotohz: negative time difference -16842863 sec -432657 usec Wed Jun 1 22:05:45 CEST 2011 10:05PM up 1:55, 2 users, load averages: 12.12, 11.31, 10.14 Wed Jun 1 22:15:46 CEST 2011 10:15PM up 2:05, 2 users, load averages: 3.76, 10.45, 11.22 Wed Jun 1 22:25:46 CEST 2011 10:25PM up 2:15, 2 users, load averages: 10.23, 10.15, 11.34 Wed Jun 1 22:37:06 CEST 2011 10:37PM up 2:26, 2 users, load averages: 74.32, 376.82, 283.33 Wed Jun 1 22:47:16 CEST 2011 10:47PM up 2:36, 2 users, load averages: 0.25, 50.11, 139.43 Wed Jun 1 22:57:19 CEST 2011 10:57PM up 2:46, 2 users, load averages: 0.10, 6.98, 69.05 Wed Jun 1 23:07:22 CEST 2011 11:07PM up 2:56, 2 users, load averages: 5.59, 7.97, 37.34 Wed Jun 1 23:17:26 CEST 2011 11:17PM up 3:06, 2 users, load averages: 12.04, 13.02, 23.93 Wed Jun 1 23:27:26 CEST 2011 11:27PM up 3:16, 2 users, load averages: 64.51, 31.17, 25.38 Wed Jun 1 23:38:23 CEST 2011 11:38PM up 3:27, 2 users, load averages: 4.47, 39.73, 41.59 Wed Jun 1 23:48:30 CEST 2011 11:48PM up 3:38, 2 users, load averages: 0.25, 5.30, 20.26 Wed Jun 1 23:58:37 CEST 2011 11:58PM up 3:48, 2 users, load averages: 2.75, 9.56, 15.50 Thu Jun 2 00:08:45 CEST 2011 12:08AM up 3:58, 2 users, load averages: 119.38, 337.16, 166.66 Thu Jun 2 00:18:45 CEST 2011 12:18AM up 4:08, 2 users, load averages: 1.55, 49.68, 85.68 Thu Jun 2 00:28:45 CEST 2011 12:28AM up 4:18, 2 users, load averages: 21.54, 20.26, 49.28 Thu Jun 2 00:38:57 CEST 2011 12:38AM up 4:28, 2 users, load averages: 23.24, 20.79, 34.35 Thu Jun 2 00:48:57 CEST 2011 12:48AM up 4:38, 2 users, load averages: 27.48, 10.42, 19.95 Thu Jun 2 01:00:08 CEST 2011 1:00AM up 4:49, 2 users, load averages: 45.52, 281.32, 266.86 Thu Jun 2 01:10:16 CEST 2011 1:10AM up 4:59, 2 users, load averages: 0.17, 37.93, 131.94 Thu Jun 2 01:20:19 CEST 2011 1:20AM up 5:09, 2 users, load averages: 0.00, 5.21, 65.59 Thu Jun 2 01:30:26 CEST 2011 1:30AM up 5:19, 2 users, load averages: 0.29, 0.88, 31.96 Thu Jun 2 01:40:42 CEST 2011 1:40AM up 5:30, 2 users, load averages: 1.04, 1.71, 16.48 Thu Jun 2 01:50:44 CEST 2011 1:50AM up 5:40, 2 users, load averages: 87.17, 26.08, 18.17 Thu Jun 2 02:01:28 CEST 2011 2:01AM up 5:50, 2 users, load averages: 5.15, 30.84, 33.55 Thu Jun 2 02:11:35 CEST 2011 2:11AM up 6:01, 2 users, load averages: 0.08, 4.28, 16.64 Thu Jun 2 02:21:35 CEST 2011 2:21AM up 6:11, 2 users, load averages: 6.13, 10.56, 14.04 Thu Jun 2 02:31:36 CEST 2011 2:31AM up 6:21, 2 users, load averages: 15.52, 220.44, 181.40 Thu Jun 2 02:41:40 CEST 2011 2:41AM up 6:31, 2 users, load averages: 0.19, 32.42, 91.66 Thu Jun 2 02:51:43 CEST 2011 2:51AM up 6:41, 2 users, load averages: 8.96, 15.40, 51.61 Thu Jun 2 03:01:44 CEST 2011 3:01AM up 6:51, 2 users, load averages: 5.87, 9.69, 30.35 Thu Jun 2 03:11:46 CEST 2011 3:11AM up 7:01, 2 users, load averages: 28.75, 18.59, 24.35 witness_lock_list_get: witness exhausted Thu Jun 2 03:21:47 CEST 2011 3:21AM up 7:11, 2 users, load averages: 28.32, 307.24, 216.27 Expensive timeout(9) function: 0xc0b3eb10(0xc5ac3568) 0.287668125 s Thu Jun 2 03:32:42 CEST 2011 3:32AM up 7:22, 2 users, load averages: 0.75, 34.53, 100.51 Thu Jun 2 03:42:49 CEST 2011 3:42AM up 7:32, 2 users, load averages: 0.11, 4.78, 49.74 Thu Jun 2 03:52:57 CEST 2011 3:52AM up 7:42, 2 users, load averages: 0.12, 0.70, 24.44 Thu Jun 2 04:02:57 CEST 2011 4:02AM up 7:52, 2 users, load averages: 0.10, 2.39, 13.99 Thu Jun 2 04:12:57 CEST 2011 4:12AM up 8:02, 2 users, load averages: 0.97, 2.50, 8.17 Thu Jun 2 04:22:58 CEST 2011 4:22AM up 8:12, 2 users, load averages: 96.37, 56.55, 30.84 Thu Jun 2 04:33:43 CEST 2011 4:33AM up 8:23, 2 users, load averages: 1.30, 16.51, 24.89 Thu Jun 2 04:43:48 CEST 2011 4:43AM up 8:33, 2 users, load averages: 24.20, 11.62, 15.93 Thu Jun 2 04:55:07 CEST 2011 4:55AM up 8:44, 2 users, load averages: 0.49, 6.67, 12.48 Thu Jun 2 05:05:11 CEST 2011 5:05AM up 8:54, 2 users, load averages: 13.41, 12.80, 12.68 Thu Jun 2 05:15:11 CEST 2011 5:15AM up 9:04, 2 users, load averages: 17.13, 11.17, 10.43 Thu Jun 2 05:25:33 CEST 2011 5:25AM up 9:15, 2 users, load averages: 19.64, 13.87, 12.12 Thu Jun 2 05:35:47 CEST 2011 5:35AM up 9:25, 2 users, load averages: 16.70, 13.87, 11.75 Thu Jun 2 05:45:48 CEST 2011 5:45AM up 9:35, 2 users, load averages: 188.56, 222.57, 123.99 Thu Jun 2 05:56:26 CEST 2011 5:56AM up 9:45, 2 users, load averages: 0.53, 32.73, 63.63 Thu Jun 2 06:06:26 CEST 2011 6:06AM up 9:55, 2 users, load averages: 0.13, 4.64, 31.58 Thu Jun 2 06:16:27 CEST 2011 6:16AM up 10:05, 2 users, load averages: 0.30, 1.05, 15.95 Thu Jun 2 06:26:53 CEST 2011 6:26AM up 10:16, 2 users, load averages: 13.69, 7.21, 10.61 Thu Jun 2 06:36:54 CEST 2011 6:36AM up 10:26, 2 users, load averages: 16.75, 11.78, 12.89 Thu Jun 2 06:46:54 CEST 2011 6:46AM up 10:36, 2 users, load averages: 16.22, 51.72, 42.81 Thu Jun 2 06:57:41 CEST 2011 6:57AM up 10:47, 2 users, load averages: 0.01, 7.54, 21.68 Thu Jun 2 07:07:42 CEST 2011 7:07AM up 10:57, 2 users, load averages: 7.34, 6.98, 14.08 Thu Jun 2 07:18:50 CEST 2011 7:18AM up 11:08, 2 users, load averages: 0.58, 7.58, 12.35 Thu Jun 2 07:28:51 CEST 2011 7:28AM up 11:18, 2 users, load averages: 16.08, 6.85, 8.44 Thu Jun 2 07:38:51 CEST 2011 7:38AM up 11:28, 2 users, load averages: 1.32, 3.01, 6.20 Thu Jun 2 07:48:51 CEST 2011 7:48AM up 11:38, 2 users, load averages: 19.79, 14.20, 10.77 Thu Jun 2 07:58:54 CEST 2011 7:58AM up 11:48, 2 users, load averages: 11.51, 16.28, 13.92 Thu Jun 2 08:08:56 CEST 2011 8:08AM up 11:58, 2 users, load averages: 730.14, 475.13, 193.07 Thu Jun 2 08:20:31 CEST 2011 8:20AM up 12:10, 2 users, load averages: 26.07, 236.28, 267.34 Thu Jun 2 08:30:42 CEST 2011 8:30AM up 12:20, 2 users, load averages: 0.21, 32.41, 132.92 Thu Jun 2 08:40:49 CEST 2011 8:40AM up 12:30, 2 users, load averages: 0.21, 4.71, 65.52 Thu Jun 2 08:50:49 CEST 2011 8:50AM up 12:40, 2 users, load averages: 0.64, 1.13, 32.27 Thu Jun 2 09:02:31 CEST 2011 9:02AM up 12:51, 2 users, load averages: 21.71, 22.23, 25.56 Thu Jun 2 09:12:34 CEST 2011 9:12AM up 13:02, 2 users, load averages: 72.42, 44.02, 31.74 Thu Jun 2 09:22:36 CEST 2011 9:22AM up 13:12, 2 users, load averages: 7.23, 29.68, 34.62 Thu Jun 2 09:32:38 CEST 2011 9:32AM up 13:22, 2 users, load averages: 32.78, 17.51, 22.56 Thu Jun 2 09:43:30 CEST 2011 9:43AM up 13:33, 2 users, load averages: 0.63, 6.84, 14.48 Thu Jun 2 09:53:37 CEST 2011 9:53AM up 13:43, 2 users, load averages: 12.18, 11.52, 12.51 kern.ipc.maxpipekva exceeded; see tuning(7) kern.ipc.maxpipekva exceeded; see tuning(7) kern.ipc.maxpipekva exceeded; see tuning(7) /home/test/keepalive.sh: Pipe call failed: Cannot allocate memory /home/test/keepalive.sh: Pipe call failed: Cannot allocate memory Connection to zoo.freebsd.org closed. $ ssh -Y zoo.freebsd.org Thu Jun 2 11:33:46 CEST 2011 11:33AM up 15:23, 2 users, load averages: 0.00, 0.00, 0.01 Thu Jun 2 11:43:46 CEST 2011 11:43AM up 15:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 11:53:46 CEST 2011 11:53AM up 15:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 12:03:46 CEST 2011 12:03PM up 15:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 12:13:46 CEST 2011 12:13PM up 16:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 12:23:46 CEST 2011 12:23PM up 16:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 12:33:46 CEST 2011 12:33PM up 16:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 12:43:46 CEST 2011 12:43PM up 16:33, 2 users, load averages: 0.03, 0.01, 0.00 Thu Jun 2 12:53:46 CEST 2011 12:53PM up 16:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:03:46 CEST 2011 1:03PM up 16:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:13:46 CEST 2011 1:13PM up 17:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:23:46 CEST 2011 1:23PM up 17:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:33:46 CEST 2011 1:33PM up 17:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:43:46 CEST 2011 1:43PM up 17:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 13:53:46 CEST 2011 1:53PM up 17:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:03:46 CEST 2011 2:03PM up 17:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:13:46 CEST 2011 2:13PM up 18:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:23:46 CEST 2011 2:23PM up 18:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:33:46 CEST 2011 2:33PM up 18:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:43:46 CEST 2011 2:43PM up 18:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 14:53:46 CEST 2011 2:53PM up 18:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:03:46 CEST 2011 3:03PM up 18:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:13:46 CEST 2011 3:13PM up 19:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:23:46 CEST 2011 3:23PM up 19:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:33:46 CEST 2011 3:33PM up 19:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:43:46 CEST 2011 3:43PM up 19:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 15:53:46 CEST 2011 3:53PM up 19:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:03:46 CEST 2011 4:03PM up 19:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:13:46 CEST 2011 4:13PM up 20:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:23:46 CEST 2011 4:23PM up 20:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:33:47 CEST 2011 4:33PM up 20:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:43:47 CEST 2011 4:43PM up 20:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 16:53:47 CEST 2011 4:53PM up 20:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 17:03:47 CEST 2011 5:03PM up 20:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 17:13:47 CEST 2011 5:13PM up 21:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 17:23:47 CEST 2011 5:23PM up 21:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 17:33:47 CEST 2011 5:33PM up 21:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 17:43:47 CEST 2011 5:43PM up 21:33, 2 users, load averages: 0.08, 0.02, 0.01 Thu Jun 2 17:53:47 CEST 2011 5:53PM up 21:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:03:47 CEST 2011 6:03PM up 21:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:13:47 CEST 2011 6:13PM up 22:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:23:47 CEST 2011 6:23PM up 22:13, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:33:47 CEST 2011 6:33PM up 22:23, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:43:47 CEST 2011 6:43PM up 22:33, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 18:53:47 CEST 2011 6:53PM up 22:43, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 19:03:47 CEST 2011 7:03PM up 22:53, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 19:13:47 CEST 2011 7:13PM up 23:03, 2 users, load averages: 0.00, 0.00, 0.00 Thu Jun 2 19:26:12 CEST 2011 7:26PM up 23:15, 2 users, load averages: 10.61, 7.13, 3.11 Thu Jun 2 19:36:19 CEST 2011 7:36PM up 23:25, 2 users, load averages: 0.35, 24.37, 25.19 Thu Jun 2 19:46:22 CEST 2011 7:46PM up 23:35, 2 users, load averages: 0.49, 3.51, 12.51 Thu Jun 2 19:56:22 CEST 2011 7:56PM up 23:45, 2 users, load averages: 0.18, 0.53, 6.22 Thu Jun 2 20:06:26 CEST 2011 8:06PM up 23:55, 2 users, load averages: 0.29, 0.32, 3.18 Thu Jun 2 20:18:09 CEST 2011 8:18PM up 1 day, 7 mins, 2 users, load averages: 3.08, 9.28, 8.15 Thu Jun 2 20:28:16 CEST 2011 8:28PM up 1 day, 17 mins, 2 users, load averages: 93.62, 44.75, 22.16 Thu Jun 2 20:39:19 CEST 2011 8:39PM up 1 day, 28 mins, 2 users, load averages: 1.57, 21.13, 26.52 Thu Jun 2 20:49:23 CEST 2011 8:49PM up 1 day, 38 mins, 2 users, load averages: 14.13, 13.11, 17.94 Thu Jun 2 21:00:34 CEST 2011 9:00PM up 1 day, 50 mins, 2 users, load averages: 0.12, 7.31, 13.87 Thu Jun 2 21:10:38 CEST 2011 9:10PM up 1 day, 1 hr, 2 users, load averages: 0.71, 6.61, 11.02 Thu Jun 2 21:20:38 CEST 2011 9:20PM up 1 day, 1:10, 2 users, load averages: 20.37, 10.95, 9.70 tvotohz: negative time difference -1006622708 sec -615425 usec Thu Jun 2 21:30:59 CEST 2011 9:30PM up 1 day, 1:20, 2 users, load averages: 12.56, 16.81, 13.92 Thu Jun 2 21:40:59 CEST 2011 9:40PM up 1 day, 1:30, 2 users, load averages: 11.43, 13.43, 12.76 Thu Jun 2 21:51:03 CEST 2011 9:51PM up 1 day, 1:40, 2 users, load averages: 711.16, 323.12, 148.26 Thu Jun 2 22:01:05 CEST 2011 10:01PM up 1 day, 1:50, 2 users, load averages: 0.41, 114.25, 155.66 Thu Jun 2 22:11:05 CEST 2011 10:11PM up 1 day, 2 hrs, 2 users, load averages: 0.05, 15.77, 77.46 Thu Jun 2 22:21:05 CEST 2011 10:21PM up 1 day, 2:10, 2 users, load averages: 0.42, 2.41, 38.21 Thu Jun 2 22:31:09 CEST 2011 10:31PM up 1 day, 2:20, 2 users, load averages: 0.14, 0.51, 18.98 Thu Jun 2 22:41:21 CEST 2011 10:41PM up 1 day, 2:30, 2 users, load averages: 15.54, 12.57, 15.06 Thu Jun 2 22:51:40 CEST 2011 10:51PM up 1 day, 2:41, 2 users, load averages: 99.75, 37.84, 21.69 Thu Jun 2 23:02:13 CEST 2011 11:02PM up 1 day, 2:51, 2 users, load averages: 2.19, 35.62, 35.93 Thu Jun 2 23:12:19 CEST 2011 11:12PM up 1 day, 3:01, 2 users, load averages: 0.17, 6.63, 19.64 Thu Jun 2 23:22:37 CEST 2011 11:22PM up 1 day, 3:12, 2 users, load averages: 29.52, 12.28, 14.32 Thu Jun 2 23:34:08 CEST 2011 11:34PM up 1 day, 3:23, 2 users, load averages: 2.03, 6.76, 10.77 Thu Jun 2 23:44:12 CEST 2011 11:44PM up 1 day, 3:33, 2 users, load averages: 11.15, 8.98, 9.24 Thu Jun 2 23:54:13 CEST 2011 11:54PM up 1 day, 3:43, 2 users, load averages: 10.35, 7.02, 7.12 Fri Jun 3 00:04:13 CEST 2011 12:04AM up 1 day, 3:53, 2 users, load averages: 8.56, 13.14, 11.09 Fri Jun 3 00:14:15 CEST 2011 12:14AM up 1 day, 4:03, 2 users, load averages: 16.21, 17.44, 13.95 Fri Jun 3 00:24:16 CEST 2011 12:24AM up 1 day, 4:13, 2 users, load averages: 201.45, 604.99, 298.05 Fri Jun 3 00:34:40 CEST 2011 12:34AM up 1 day, 4:24, 2 users, load averages: 0.31, 79.11, 147.35 Fri Jun 3 00:44:40 CEST 2011 12:44AM up 1 day, 4:34, 2 users, load averages: 0.48, 10.94, 72.63 Fri Jun 3 00:54:43 CEST 2011 12:54AM up 1 day, 4:44, 2 users, load averages: 0.09, 1.69, 36.03 Fri Jun 3 01:05:06 CEST 2011 1:05AM up 1 day, 4:54, 2 users, load averages: 11.95, 3.50, 18.83 Fri Jun 3 01:15:17 CEST 2011 1:15AM up 1 day, 5:04, 2 users, load averages: 2.19, 5.12, 12.06 Fri Jun 3 01:26:32 CEST 2011 1:26AM up 1 day, 5:16, 2 users, load averages: 14.43, 45.99, 38.50 Fri Jun 3 01:36:38 CEST 2011 1:36AM up 1 day, 5:26, 2 users, load averages: 0.11, 6.27, 19.07 Fri Jun 3 01:47:08 CEST 2011 1:47AM up 1 day, 5:36, 2 users, load averages: 14.62, 12.43, 16.34 Fri Jun 3 01:57:11 CEST 2011 1:57AM up 1 day, 5:46, 2 users, load averages: 20.95, 205.76, 141.66 Fri Jun 3 02:07:11 CEST 2011 2:07AM up 1 day, 5:56, 2 users, load averages: 0.52, 30.89, 72.76 Fri Jun 3 02:17:20 CEST 2011 2:17AM up 1 day, 6:06, 2 users, load averages: 15.11, 19.19, 44.57 Fri Jun 3 02:27:21 CEST 2011 2:27AM up 1 day, 6:16, 2 users, load averages: 9.10, 12.47, 27.87 Fri Jun 3 02:37:25 CEST 2011 2:37AM up 1 day, 6:26, 2 users, load averages: 385.10, 397.08, 168.15 Fri Jun 3 02:48:11 CEST 2011 2:48AM up 1 day, 6:37, 2 users, load averages: 69.88, 60.87, 328.18 Fri Jun 3 02:58:16 CEST 2011 2:58AM up 1 day, 6:47, 2 users, load averages: 0.37, 8.16, 160.35 Fri Jun 3 03:08:16 CEST 2011 3:08AM up 1 day, 6:57, 2 users, load averages: 0.06, 1.28, 79.32 Fri Jun 3 03:18:23 CEST 2011 3:18AM up 1 day, 7:07, 2 users, load averages: 0.22, 0.31, 39.73 Fri Jun 3 03:28:54 CEST 2011 3:30AM up 1 day, 7:19, 2 users, load averages: 30.35, 12.53, 22.55 Fri Jun 3 03:40:17 CEST 2011 3:40AM up 1 day, 7:29, 2 users, load averages: 102.10, 64.01, 40.78 Fri Jun 3 03:50:20 CEST 2011 3:50AM up 1 day, 7:39, 2 users, load averages: 1.86, 20.77, 31.21 Fri Jun 3 04:00:20 CEST 2011 4:00AM up 1 day, 7:49, 2 users, load averages: 3.95, 5.13, 16.50 Fri Jun 3 04:10:46 CEST 2011 4:10AM up 1 day, 8 hrs, 2 users, load averages: 9.97, 8.98, 13.46 Fri Jun 3 04:20:46 CEST 2011 4:20AM up 1 day, 8:10, 2 users, load averages: 18.51, 27.50, 28.33 Fri Jun 3 04:30:46 CEST 2011 4:30AM up 1 day, 8:20, 2 users, load averages: 14.61, 12.38, 18.40 Read from remote host zoo.freebsd.org: Operation timed out Connection to zoo.freebsd.org closed. $ ssh -Y zoo.freebsd.org $ console leopard3 [Enter `^Ec?' for help] (panic: _mtx_lock_sleep: recursed on non-recursive mutex process lock @ /mnt/pho/src/sys/i386/i386/trap.c:854) db> run pho db:0:pho> bt Tracing pid 1675 tid 100100 td 0xc5feab80 kdb_enter(c0e8d86b,c0e8d86b,c0e8c015,df3fb06c,0,...) at kdb_enter+0x3a panic(c0e8c015,c0e8c0a9,c0edc3c8,356,c5fe3088,...) at panic+0x134 _mtx_lock_sleep(c5fe3088,c5feab80,0,c0edc3c8,356,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c5fe3088,0,c0edc3c8,356,6e7765c2,...) at _mtx_lock_flags+0xf7 trap_pfault(5,0,c0edc4a6,c5feab80,c5fe3000,...) at trap_pfault+0xc3 trap(df3fb18c) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc09f6e3f, esp = 0xdf3fb1cc, ebp = 0xdf3fb200 --- witness_unlock(c5fe3088,8,c0edc3c8,358,c5fe3088,...) at witness_unlock+0xaf _mtx_unlock_flags(c5fe3088,0,c0edc3c8,358,6e7765c2,...) at _mtx_unlock_flags+0xc3 trap_pfault(5,0,c0edc4a6,c0e7d292,c5fe3000,...) at trap_pfault+0xea trap(df3fb304) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc09f6e3f, esp = 0xdf3fb344, ebp = 0xdf3fb378 --- witness_unlock(c1018c44,0,c0e92018,db,0,...) at witness_unlock+0xaf _sx_sunlock(c1018c44,c0e92018,db,a,df3fb3fc,0,4,6,0,c5af28ec,c64d7e00,c5af28ec,c64d7e00,df3fb3fc) at _sx_sunlock+0xa0 tprintf(c5fe3000,6,c0e7d27d,c5ab7cd8,c0ebadc6,...) at tprintf+0x1cd nfs_msg(0,0,c0e7d234,3c0,c5feab80,...) at nfs_msg+0x67 nfs_feedback(4,4,df3fb520,c5b94000,df3fb5cc,...) at nfs_feedback+0x153 clnt_reconnect_call(c5a5d600,df3fb4f4,4,c5b94000,df3fb5cc,...) at clnt_reconnect_call+0x787 newnfs_request(df3fb5cc,c5a9b800,0,c5a9b910,c5925ae0,c5feab80,c5eda600,186a3,3,0,1,0) at newnfs_request+0xb7a nfscl_request(df3fb5cc,c5925ae0,c5feab80,c5eda600,0,...) at nfscl_request+0x82 nfsrpc_accessrpc(c5925ae0,3f,c5eda600,c5feab80,df3fb6a0,...) at nfsrpc_accessrpc+0x3f3 nfs34_access_otw(c5eda600,df3fb780,c0e7e4e8,18f,c5925ae0,...) at nfs34_access_otw+0x50 nfs_access(df3fb974,c0ee2656,c5a9b800,c5a9b800,df3fbb58,...) at nfs_access+0x1c8 VOP_ACCESS_APV(c0f9a1e0,df3fb974,c0e7e4e8,3eb,c60028ec,...) at VOP_ACCESS_APV+0xc5 nfs_lookup(df3fba20,c0e9dd63,c0fdd980,200000,df3fbb2c,...) at nfs_lookup+0x16c VOP_LOOKUP_APV(c0f9a1e0,df3fba20,df3fbb58,1f5,df3fbb44,...) at VOP_LOOKUP_APV+0xe5 lookup(df3fbb2c,c0e9c41d,ee,c9,c5fe3000,...) at lookup+0x67b namei(df3fbb2c,df3fbacc,60,0,c5feab80,...) at namei+0x592 kern_statat_vnhook(c5feab80,0,ffffff9c,804ef73,0,...) at kern_statat_vnhook+0x72 kern_statat(c5feab80,0,ffffff9c,804ef73,0,...) at kern_statat+0x3c kern_stat(c5feab80,804ef73,0,df3fbbe8,60,...) at kern_stat+0x36 stat(c5feab80,df3fbcec,df3fbd28,c0e93ae6,0,...) at stat+0x2f syscallenter(c5feab80,df3fbce4,df3fbce4,0,0,...) at syscallenter+0x263 syscall(df3fbd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (188, FreeBSD ELF32, stat), eip = 0x28191b13, esp = 0xbfbfe97c, ebp = 0xbfbfec78 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x446b80 curthread = 0xc5feab80: pid 1675 "cron" curpcb = 0xdf3fbd80 fpcurthread = none idlethread = 0xc55a48a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x4228b80 curthread = 0xc55a4b80: pid 1 "init" curpcb = 0xc520dd80 fpcurthread = none idlethread = 0xc55a45c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Script command 'show alllocks' returned error db:0:alllocks> show lockedvnods Locked vnodes 0xc5925ae0: 0xc5925ae0: tag newnfs, type VDIR tag newnfs, type VDIR usecount 62, writecount 0, refcount 63 mountedhere 0 usecount 62, writecount 0, refcount 63 mountedhere 0 flags (VV_ROOT) flags (VV_ROOT) v_object 0xc5f07660 ref 0 pages 0 v_object 0xc5f07660 ref 0 pages 0 lock type newnfs: SHARED (count 1) lock type newnfs: SHARED (count 1) #0 0xc09940a5 at __lockmgr_args+0x5a5 #0 0xc09940a5 at __lockmgr_args+0x5a5 #1 0xc08ff6fd at nfs_lock1+0xcd #1 0xc08ff6fd at nfs_lock1+0xcd #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #3 0xc0a54438 at _vn_lock+0x78 #3 0xc0a54438 at _vn_lock+0x78 #4 0xc0a392e6 at lookup+0x106 #4 0xc0a392e6 at lookup+0x106 #5 0xc0a3a6c2 at namei+0x592 #5 0xc0a3a6c2 at namei+0x592 #6 0xc0a4b9d2 at kern_statat_vnhook+0x72 #6 0xc0a4b9d2 at kern_statat_vnhook+0x72 #7 0xc0a4bb2c at kern_statat+0x3c #7 0xc0a4bb2c at kern_statat+0x3c #8 0xc0a4bc76 at kern_stat+0x36 #8 0xc0a4bc76 at kern_stat+0x36 #9 0xc0a4bd1f at stat+0x2f #9 0xc0a4bd1f at stat+0x2f #10 0xc09ef113 at syscallenter+0x263 #10 0xc09ef113 at syscallenter+0x263 #11 0xc0cd645f at syscall+0x4f #11 0xc0cd645f at syscall+0x4f #12 0xc0cbf921 at Xint0x80_syscall+0x21 #12 0xc0cbf921 at Xint0x80_syscall+0x21 fileid 19359850 fsid 0x600ff01 fileid 19359850 fsid 0x600ff01 0xc59256cc: 0xc59256cc: tag newnfs, type VREG tag newnfs, type VREG usecount 5, writecount 0, refcount 6 mountedhere 0 usecount 5, writecount 0, refcount 6 mountedhere 0 flags (VV_TEXT) flags (VV_TEXT) v_object 0xc159de58 ref 3 pages 10 v_object 0xc159de58 ref 3 pages 10 lock type newnfs: SHARED (count 1) lock type newnfs: SHARED (count 1) #0 0xc09940a5 at __lockmgr_args+0x5a5 #0 0xc09940a5 at __lockmgr_args+0x5a5 #1 0xc08ff6fd at nfs_lock1+0xcd #1 0xc08ff6fd at nfs_lock1+0xcd #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #3 0xc0a54438 at _vn_lock+0x78 #3 0xc0a54438 at _vn_lock+0x78 #4 0xc0a4756b at vget+0xbb #4 0xc0a4756b at vget+0xbb #5 0xc0c033cb at vm_fault_hold+0xacb #5 0xc0c033cb at vm_fault_hold+0xacb #6 0xc0c04b1f at vm_fault+0x2f #6 0xc0c04b1f at vm_fault+0x2f #7 0xc0cd631d at trap_pfault+0x10d #7 0xc0cd631d at trap_pfault+0x10d #8 0xc0cd6809 at trap+0x2c9 #8 0xc0cd6809 at trap+0x2c9 #9 0xc0cbf8bc at calltrap+0x6 #9 0xc0cbf8bc at calltrap+0x6 fileid 19394949 fsid 0x600ff01 fileid 19394949 fsid 0x600ff01 0xc592515c: 0xc592515c: tag newnfs, type VDIR tag newnfs, type VDIR usecount 6, writecount 0, refcount 7 mountedhere 0 usecount 6, writecount 0, refcount 7 mountedhere 0 flags () flags () v_object 0xc6088ee0 ref 0 pages 0 v_object 0xc6088ee0 ref 0 pages 0 lock type newnfs: SHARED (count 6) lock type newnfs: SHARED (count 6) #0 0xc09940a5 at __lockmgr_args+0x5a5 #0 0xc09940a5 at __lockmgr_args+0x5a5 #1 0xc08ff6fd at nfs_lock1+0xcd #1 0xc08ff6fd at nfs_lock1+0xcd #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #3 0xc0a54438 at _vn_lock+0x78 #3 0xc0a54438 at _vn_lock+0x78 #4 0xc0a4756b at vget+0xbb #4 0xc0a4756b at vget+0xbb #5 0xc0a31ceb at cache_lookup+0x67b #5 0xc0a31ceb at cache_lookup+0x67b #6 0xc0905653 at nfs_lookup+0x193 #6 0xc0905653 at nfs_lookup+0x193 #7 0xc0cf9d85 at VOP_LOOKUP_APV+0xe5 #7 0xc0cf9d85 at VOP_LOOKUP_APV+0xe5 #8 0xc0a3985b at lookup+0x67b #8 0xc0a3985b at lookup+0x67b #9 0xc0a3a6c2 at namei+0x592 #9 0xc0a3a6c2 at namei+0x592 #10 0xc097b837 at kern_execve+0x347 #10 0xc097b837 at kern_execve+0x347 #11 0xc097ca3c at execve+0x4c #11 0xc097ca3c at execve+0x4c #12 0xc09ef113 at syscallenter+0x263 #12 0xc09ef113 at syscallenter+0x263 #13 0xc0cd645f at syscall+0x4f #13 0xc0cd645f at syscall+0x4f #14 0xc0cbf921 at Xint0x80_syscall+0x21 #14 0xc0cbf921 at Xint0x80_syscall+0x21 fileid 20778016 fsid 0x600ff01 fileid 20778016 fsid 0x600ff01 0xc5925000: 0xc5925000: tag newnfs, type VREG tag newnfs, type VREG usecount 9, writecount 0, refcount 15 mountedhere 0 usecount 9, writecount 0, refcount 15 mountedhere 0 flags (VV_TEXT) flags (VV_TEXT) v_object 0xc159d880 ref 4 pages 27 v_object 0xc159d880 ref 4 pages 27 lock type newnfs: EXCL by thread 0xc6ffe2e0 (pid 82665) lock type newnfs: EXCL by thread 0xc6ffe2e0 (pid 82665) with exclusive waiters pending with exclusive waiters pending #0 0xc09946fe at __lockmgr_args+0xbfe #0 0xc09946fe at __lockmgr_args+0xbfe #1 0xc08ff6fd at nfs_lock1+0xcd #1 0xc08ff6fd at nfs_lock1+0xcd #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #2 0xc0cf8c05 at VOP_LOCK1_APV+0xb5 #3 0xc0a54438 at _vn_lock+0x78 #3 0xc0a54438 at _vn_lock+0x78 #4 0xc0a4756b at vget+0xbb #4 0xc0a4756b at vget+0xbb #5 0xc0a31ceb at cache_lookup+0x67b #5 0xc0a31ceb at cache_lookup+0x67b #6 0xc0905653 at nfs_lookup+0x193 #6 0xc0905653 at nfs_lookup+0x193 #7 0xc0cf9d85 at VOP_LOOKUP_APV+0xe5 #7 0xc0cf9d85 at VOP_LOOKUP_APV+0xe5 #8 0xc0a3985b at lookup+0x67b #8 0xc0a3985b at lookup+0x67b #9 0xc0a3a6c2 at namei+0x592 #9 0xc0a3a6c2 at namei+0x592 #10 0xc097b837 at kern_execve+0x347 #10 0xc097b837 at kern_execve+0x347 #11 0xc097ca3c at execve+0x4c #11 0xc097ca3c at execve+0x4c #12 0xc09ef113 at syscallenter+0x263 #12 0xc09ef113 at syscallenter+0x263 #13 0xc0cd645f at syscall+0x4f #13 0xc0cd645f at syscall+0x4f #14 0xc0cbf921 at Xint0x80_syscall+0x21 #14 0xc0cbf921 at Xint0x80_syscall+0x21 fileid 20775449 fsid 0x600ff01 fileid 20775449 fsid 0x600ff01 db:0:lockedvnods> show mount 0xc5ab7b50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc5ab8000 devfs on /dev (devfs) 0xc5ab787c /dev/ada0d on /work (ufs) 0xc5ab75a8 /dev/md0 on /var (ufs) 0xc5ab72d4 /dev/md1 on /tmp (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 82669 81694 49799 1001 D+ newnfs 0xc5925058 run 82668 1 49799 1001 D+ newnfs 0xc5925058 syscall 82667 1 49799 1001 D+ newnfs 0xc5925058 syscall 82666 1 49799 1001 D+ newnfs 0xc5925058 syscall 82665 1 49799 1001 R+ syscall 82664 1 49799 1001 D+ newnfs 0xc5925058 syscall 82328 1757 1757 0 SW+ nanslp 0xc101a324 sleep 81694 81693 49799 1001 S+ wait 0xc73b0af0 run 81693 81692 49799 1001 S+ wait 0xc5b5caf0 run 81692 81686 49799 1001 S+ wait 0xc5ecc578 run 81686 49799 49799 1001 SW+ wait 0xc7f04834 sh 49799 48872 49799 1001 SW+ wait 0xc6a182bc sh 48872 48871 48872 1001 SWs+ wait 0xc6276834 sh 48871 48866 48866 1001 S select 0xc7004ce4 sshd 48866 1664 48866 0 SWs sbwait 0xc76920b8 sshd 1757 1752 1757 0 SW+ wait 0xc5a6d2bc sh 1752 1751 1752 0 SW+ pause 0xc5fe2058 csh 1751 1 1751 0 SWs+ wait 0xc5ea9578 login 1750 1 1750 0 Ss+ ttyin 0xc56bbe70 getty 1749 1 1749 0 Ss+ ttyin 0xc584e070 getty 1748 1 1748 0 Ss+ ttyin 0xc584e270 getty 1747 1 1747 0 Ss+ ttyin 0xc584e470 getty 1746 1 1746 0 Ss+ ttyin 0xc56ba070 getty 1745 1 1745 0 Ss+ ttyin 0xc56ba270 getty 1744 1 1744 0 Ss+ ttyin 0xc56ba470 getty 1743 1 1743 0 Ss+ ttyin 0xc56ba670 getty 1705 1 1705 0 Ss select 0xc6043de4 inetd 1675 1 1675 0 RLs CPU 0 cron 1664 1 1664 0 Ss select 0xc6044064 sshd 1523 1 1523 0 Ss select 0xc5eb0b64 rpc.statd 1450 0 0 0 DL mdwait 0xc5ffe800 [md1] 1395 1 1395 0 Ss select 0xc6043e64 rpcbind 1365 1 1365 0 Ss select 0xc6044024 syslogd 1186 1 1186 0 Ss select 0xc5717ce4 devd 99 0 0 0 DL mdwait 0xc574a000 [md0] 17 0 0 0 DL sdflush 0xc1196a20 [softdepflush] 16 0 0 0 DL vlruwt 0xc5a6b834 [vnlru] 9 0 0 0 DL syncer 0xc11904f4 [syncer] 8 0 0 0 DL psleep 0xc1190188 [bufdaemon] 7 0 0 0 DL pgzero 0xc11988dc [pagezero] 6 0 0 0 DL psleep 0xc11984c8 [vmdaemon] 5 0 0 0 DL psleep 0xc1198490 [pagedaemon] 4 0 0 0 DL ccb_scan 0xc0fdf1d4 [xpt_thrd] 3 0 0 0 DL waiting_ 0xc1191ff8 [sctp_iterator] 2 0 0 0 DL - 0xc573ba3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100049 D - 0xc57d8d34 [usbus4] 100048 D - 0xc57d8d04 [usbus4] 100047 D - 0xc57d8cd4 [usbus4] 100046 D - 0xc57d8ca4 [usbus4] 100045 D - 0xc57c0dac [usbus3] 100044 D - 0xc57c0d7c [usbus3] 100043 D - 0xc57c0d4c [usbus3] 100042 D - 0xc57c0d1c [usbus3] 100040 D - 0xc57aedac [usbus2] 100039 D - 0xc57aed7c [usbus2] 100038 D - 0xc57aed4c [usbus2] 100037 D - 0xc57aed1c [usbus2] 100035 D - 0xc5797dac [usbus1] 100034 D - 0xc5797d7c [usbus1] 100033 D - 0xc5797d4c [usbus1] 100032 D - 0xc5797d1c [usbus1] 100030 D - 0xc5785dac [usbus0] 100029 D - 0xc5785d7c [usbus0] 100028 D - 0xc5785d4c [usbus0] 100027 D - 0xc5785d1c [usbus0] 14 0 0 0 DL - 0xc1019124 [yarrow] 13 0 0 0 DL (threaded) [geom] 100011 D - 0xc1016d48 [g_down] 100010 D - 0xc1016d44 [g_up] 100009 D - 0xc1016d3c [g_event] 12 0 0 0 RL (threaded) [intr] 100054 I [irq1: atkbd0] 100052 I [swi0: uart uart] 100051 I [irq258: ahci0] 100050 I [irq14: ata0] 100041 I [irq16: uhci3] 100036 I [irq18: uhci2] 100031 I [irq19: uhci1] 100026 I [irq23: uhci0 ehci0] 100023 I [swi2: cambio] 100021 I [swi6: task queue] 100020 I [swi6: Giant taskq] 100017 I [swi5: +] 100008 RunQ [swi4: clock] 100007 I [swi4: clock] 100006 I [swi1: netisr 0] 100005 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 RLs CPU 1 [init] 10 0 0 0 DL audit_wo 0xc1196220 [audit] 0 0 0 0 RLs (threaded) [kernel] 100065 D - 0xc5a4f0c0 [mca taskq] 100055 D - 0xc1019124 [deadlkres] 100025 D - 0xc57691c0 [em1 taskq] 100024 RunQ [em0 taskq] 100022 D - 0xc56b1500 [kqueue taskq] 100019 D - 0xc56b15c0 [ffs_trim taskq] 100018 D - 0xc56b1600 [thread taskq] 100016 D - 0xc56b1680 [acpi_task_2] 100015 D - 0xc56b1680 [acpi_task_1] 100014 D - 0xc56b1680 [acpi_task_0] 100012 D - 0xc555eb40 [firmware taskq] 100000 D sched 0xc1016e20 [swapper] db:0:ps> allt Tracing command run pid 82669 tid 100198 td 0xc62b65c0 sched_switch(c62b65c0,0,104,191,1e65c6d8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c62b65c0,0,c0e93140,268,60,...) at sleepq_switch+0x162 sleepq_wait(c5925058,60,c0e7ea23,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c5925058,80100,c59250c4,0,0,...) at __lockmgr_args+0xb1f nfs_lock1(eda946dc,c09f674b,c0e9b8b0,80100,c5925000,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c0f9a1e0,eda946dc,c0e8bdc7,c0fddd40,c5925000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c5925000,80100,c0e9cfea,856,0,...) at _vn_lock+0x78 vget(c5925000,80100,c62b65c0,244,0,...) at vget+0xbb cache_lookup(c592515c,eda94b90,eda94ba4,3eb,c09f6dea,...) at cache_lookup+0x67b nfs_lookup(eda949d4,c0e9dd63,c0fdd980,80000,eda94b78,...) at nfs_lookup+0x193 VOP_LOOKUP_APV(c0f9a1e0,eda949d4,eda94ba4,1f5,eda94b90,...) at VOP_LOOKUP_APV+0xe5 lookup(eda94b78,c0e9c41d,ee,c9,c62d6578,...) at lookup+0x67b namei(eda94b78,0,c0e87d40,16d,c5d3d870,...) at namei+0x592 kern_execve(c62b65c0,eda94c24,0,bfbfe330,bfbfeb78,e8625000,e8625008,e862504f,e8625338,e8625000,0,3fcd0,3,26,0) at kern_execve+0x347 execve(c62b65c0,eda94cec,281a8df0,1,0,...) at execve+0x4c syscallenter(c62b65c0,eda94ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(eda94d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe31c, ebp = 0xbfbfe378 --- Tracing command syscall pid 82668 tid 100231 td 0xc5b978a0 sched_switch(c5b978a0,0,104,191,d948d0d8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c5b978a0,0,c0e93140,268,60,...) at sleepq_switch+0x162 sleepq_wait(c5925058,60,c0e7ea23,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c5925058,80100,c59250c4,0,0,...) at __lockmgr_args+0xb1f nfs_lock1(edb1a6dc,c09f674b,c0e9b8b0,80100,c5925000,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c0f9a1e0,edb1a6dc,c0e8bdc7,c0fddd40,c5925000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c5925000,80100,c0e9cfea,856,0,...) at _vn_lock+0x78 vget(c5925000,80100,c5b978a0,244,0,...) at vget+0xbb cache_lookup(c592515c,edb1ab90,edb1aba4,3eb,c0e8bdc7,...) at cache_lookup+0x67b nfs_lookup(edb1a9d4,c0e9dd63,c0fdd980,80000,edb1ab78,...) at nfs_lookup+0x193 VOP_LOOKUP_APV(c0f9a1e0,edb1a9d4,edb1aba4,1f5,edb1ab90,...) at VOP_LOOKUP_APV+0xe5 lookup(edb1ab78,c0e9c41d,ee,c9,c59fe2bc,...) at lookup+0x67b namei(edb1ab78,0,c0e87d40,16d,c62de300,...) at namei+0x592 kern_execve(c5b978a0,edb1ac24,0,bfbfe880,bfbfeba8,e85e4000,e85e4008,e85e4033,e85e431c,e85e4000,0,3fcec,3,26,0) at kern_execve+0x347 execve(c5b978a0,edb1acec,281a8df0,1,0,...) at execve+0x4c syscallenter(c5b978a0,edb1ace4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(edb1ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe86c, ebp = 0xbfbfe8c8 --- Tracing command syscall pid 82667 tid 769640 td 0xc6958000 sched_switch(c6958000,0,104,191,ae211cb0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c6958000,0,c0e93140,268,60,...) at sleepq_switch+0x162 sleepq_wait(c5925058,60,c0e7ea23,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c5925058,80100,c59250c4,0,0,...) at __lockmgr_args+0xb1f nfs_lock1(edc766dc,c09f674b,c0e9b8b0,80100,c5925000,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c0f9a1e0,edc766dc,c0e8bdc7,c0fddd40,c5925000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c5925000,80100,c0e9cfea,856,0,...) at _vn_lock+0x78 vget(c5925000,80100,c6958000,244,0,...) at vget+0xbb cache_lookup(c592515c,edc76b90,edc76ba4,3eb,c09f6dea,...) at cache_lookup+0x67b nfs_lookup(edc769d4,c0e9dd63,c0fdd980,80000,edc76b78,...) at nfs_lookup+0x193 VOP_LOOKUP_APV(c0f9a1e0,edc769d4,edc76ba4,1f5,edc76b90,...) at VOP_LOOKUP_APV+0xe5 lookup(edc76b78,c0e9c41d,ee,c9,c6a29834,...) at lookup+0x67b namei(edc76b78,0,c0e87d40,16d,c6327870,...) at namei+0x592 kern_execve(c6958000,edc76c24,0,bfbfe880,bfbfeba8,e8562000,e8562008,e8562033,e856231c,e8562000,0,3fcec,3,26,0) at kern_execve+0x347 execve(c6958000,edc76cec,281a8df0,1,0,...) at execve+0x4c syscallenter(c6958000,edc76ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(edc76d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe86c, ebp = 0xbfbfe8c8 --- Tracing command syscall pid 82666 tid 19305685 td 0xc688ab80 sched_switch(c688ab80,0,104,191,ae1d9618,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c688ab80,0,c0e93140,268,60,...) at sleepq_switch+0x162 sleepq_wait(c5925058,60,c0e7ea23,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c5925058,80100,c59250c4,0,0,...) at __lockmgr_args+0xb1f nfs_lock1(ede6d6dc,c09f674b,c0e9b8b0,80100,c5925000,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c0f9a1e0,ede6d6dc,c0e8bdc7,c0fddd40,c5925000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c5925000,80100,c0e9cfea,856,0,...) at _vn_lock+0x78 vget(c5925000,80100,c688ab80,244,0,...) at vget+0xbb cache_lookup(c592515c,ede6db90,ede6dba4,3eb,c09f6dea,...) at cache_lookup+0x67b nfs_lookup(ede6d9d4,c0e9dd63,c0fdd980,80000,ede6db78,...) at nfs_lookup+0x193 VOP_LOOKUP_APV(c0f9a1e0,ede6d9d4,ede6dba4,1f5,ede6db90,...) at VOP_LOOKUP_APV+0xe5 lookup(ede6db78,c0e9c41d,ee,c9,c5b91578,...) at lookup+0x67b namei(ede6db78,0,c0e87d40,16d,c603da40,...) at namei+0x592 kern_execve(c688ab80,ede6dc24,0,bfbfe880,bfbfeba8,e8521000,e8521008,e8521033,e852131c,e8521000,0,3fcec,3,26,0) at kern_execve+0x347 execve(c688ab80,ede6dcec,281a8df0,1,0,...) at execve+0x4c syscallenter(c688ab80,ede6dce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(ede6dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe86c, ebp = 0xbfbfe8c8 --- Tracing command syscall pid 82665 tid 19305670 td 0xc6ffe2e0 sched_switch(c6ffe2e0,0,104,191,ae185a48,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c6ffe2e0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c5c43600,0,c0e7d2ab,0,0,...) at sleepq_timedwait+0x6b _sleep(c5c43600,c5a4e600,0,c0e7d2ab,493e1,...) at _sleep+0x342 clnt_vc_call(c5a5d580,ede404cc,1,c6774600,ede40590,...) at clnt_vc_call+0x815 clnt_reconnect_call(c5a5d600,ede404cc,1,c6774600,ede40590,...) at clnt_reconnect_call+0x670 newnfs_request(ede40590,c5a9b800,0,c5a9b910,c5925000,c6ffe2e0,c5c5ad80,186a3,3,0,1,0) at newnfs_request+0xb7a nfscl_request(ede40590,c5925000,c6ffe2e0,c5c5ad80,0,...) at nfscl_request+0x82 nfsrpc_getattr(c5925000,c5c5ad80,c6ffe2e0,ede4064c,0,...) at nfsrpc_getattr+0xd0 nfs_getattr(ede40928,c0ee25c9,c5d592e8,c5d592e8,ede40ba4,...) at nfs_getattr+0x1f1 VOP_GETATTR_APV(c0f9a1e0,ede40928,c0e7e4e8,409,c09f6dea,...) at VOP_GETATTR_APV+0xc5 nfs_lookup(ede409d4,c0e9dd63,c0fdd980,80000,ede40b78,...) at nfs_lookup+0x275 VOP_LOOKUP_APV(c0f9a1e0,ede409d4,ede40ba4,1f5,ede40b90,...) at VOP_LOOKUP_APV+0xe5 lookup(ede40b78,c0e9c41d,ee,c9,c5bdc578,...) at lookup+0x67b namei(ede40b78,0,c0e87d40,16d,c603dde0,...) at namei+0x592 kern_execve(c6ffe2e0,ede40c24,0,bfbfe880,bfbfeba8,e84e0000,e84e0008,e84e0033,e84e031c,e84e0000,0,3fcec,3,26,0) at kern_execve+0x347 execve(c6ffe2e0,ede40cec,281a8df0,1,0,...) at execve+0x4c syscallenter(c6ffe2e0,ede40ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(ede40d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe86c, ebp = 0xbfbfe8c8 --- Tracing command syscall pid 82664 tid 26284170 td 0xc749d2e0 sched_switch(c749d2e0,0,104,191,d2880e28,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c749d2e0,0,c0e93140,268,60,...) at sleepq_switch+0x162 sleepq_wait(c5925058,60,c0e7ea23,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c5925058,80100,c59250c4,0,0,...) at __lockmgr_args+0xb1f nfs_lock1(edf246dc,c09f674b,c0e9b8b0,80100,c5925000,...) at nfs_lock1+0xcd VOP_LOCK1_APV(c0f9a1e0,edf246dc,c0e8bdc7,c0fddd40,c5925000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c5925000,80100,c0e9cfea,856,0,...) at _vn_lock+0x78 vget(c5925000,80100,c749d2e0,244,0,...) at vget+0xbb cache_lookup(c592515c,edf24b90,edf24ba4,3eb,c09f6dea,...) at cache_lookup+0x67b nfs_lookup(edf249d4,c0e9dd63,c0fdd980,80000,edf24b78,...) at nfs_lookup+0x193 VOP_LOOKUP_APV(c0f9a1e0,edf249d4,edf24ba4,1f5,edf24b90,...) at VOP_LOOKUP_APV+0xe5 lookup(edf24b78,c0e9c41d,ee,c9,c7f04af0,...) at lookup+0x67b namei(edf24b78,0,c0e87d40,16d,c62de3e8,...) at namei+0x592 kern_execve(c749d2e0,edf24c24,0,bfbfe880,bfbfeba8,e85a3000,e85a3008,e85a3033,e85a331c,e85a3000,0,3fcec,3,26,0) at kern_execve+0x347 execve(c749d2e0,edf24cec,281a8df0,1,0,...) at execve+0x4c syscallenter(c749d2e0,edf24ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(edf24d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (59, FreeBSD ELF32, execve), eip = 0x28103fbb, esp = 0xbfbfe86c, ebp = 0xbfbfe8c8 --- Tracing command sleep pid 82328 tid 100221 td 0xc62d92e0 Tracing command run pid 81694 tid 100252 td 0xc73a6b80 sched_switch(c73a6b80,0,104,191,e41ad310,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c73a6b80,0,c0e93140,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c73b0af0,6c,c0e95f7a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c73b0af0,c73b0b78,16c,c0e95f7a,0,...) at _sleep+0x35c kern_wait(c73a6b80,142ed,edb78c44,0,0,...) at kern_wait+0xbe6 wait4(c73a6b80,edb78cec,edb78d28,c0e93ae6,0,...) at wait4+0x3b syscallenter(c73a6b80,edb78ce4,edb78ce4,fffffffe,0,...) at syscallenter+0x263 syscall(edb78d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28103ffb, esp = 0xbfbfe37c, ebp = 0xbfbfe428 --- Tracing command run pid 81693 tid 19305688 td 0xc66f35c0 sched_switch(c66f35c0,0,104,191,e41a4198,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c66f35c0,0,c0e93140,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5b5caf0,6c,c0e95f7a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5b5caf0,c5b5cb78,16c,c0e95f7a,0,...) at _sleep+0x35c kern_wait(c66f35c0,13f1e,ede76c44,0,0,...) at kern_wait+0xbe6 wait4(c66f35c0,ede76cec,28103ff9,1,0,...) at wait4+0x3b syscallenter(c66f35c0,ede76ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(ede76d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28103ffb, esp = 0xbfbfe96c, ebp = 0xbfbfe988 --- Tracing command run pid 81692 tid 100082 td 0xc5f0bb80 sched_switch(c5f0bb80,0,104,191,e061ded0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,6c,...) at mi_switch+0x1eb sleepq_switch(c5f0bb80,0,c0e93140,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5ecc578,6c,c0e95f7a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5ecc578,c5ecc600,16c,c0e95f7a,0,...) at _sleep+0x35c kern_wait(c5f0bb80,13f1d,df3b1c44,0,0,...) at kern_wait+0xbe6 wait4(c5f0bb80,df3b1cec,281a8b08,1,0,...) at wait4+0x3b syscallenter(c5f0bb80,df3b1ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(df3b1d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28103ffb, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command sh pid 81686 tid 26284173 td 0xc749db80 Tracing command sh pid 49799 tid 773538 td 0xc5d415c0 Tracing command sh pid 48872 tid 100166 td 0xc6288000 Tracing command sshd pid 48871 tid 100291 td 0xc60108a0 sched_switch(c60108a0,0,104,191,94f38998,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c60108a0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c099c5ba,c7004cd0,0,c0e8bdc7,c60108a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c7004ce4,0,edc15a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c7004ce4,c7004cd0,c0e957cd,627,c5e3b038,...) at _cv_wait_sig+0x243 seltdwait(c5e3b038,58,c5c5a100,c60108a0,0,...) at seltdwait+0xa2 kern_select(c60108a0,a,288060b8,288060cc,0,0,20,c6391834,c6391834) at kern_select+0x504 select(c60108a0,edc15cec,edc15d28,c0e93ae6,0,...) at select+0x66 syscallenter(c60108a0,edc15ce4,edc15ce4,0,c1027600,...) at syscallenter+0x263 syscall(edc15d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284b0d43, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 48866 tid 100219 td 0xc62d98a0 Tracing command sh pid 1757 tid 100111 td 0xc5fe78a0 Tracing command csh pid 1752 tid 100105 td 0xc5fe9b80 Tracing command login pid 1751 tid 100074 td 0xc5eb15c0 Tracing command getty pid 1750 tid 100103 td 0xc5fea2e0 sched_switch(c5fea2e0,0,104,191,7c529c20,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5fea2e0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5fea2e0,c5fea2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56bbe70,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56bbe70,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56bbe00,c56bbe70,ed922c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56bbe00,ed922c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c584d200,ed922c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e27150,ed922c28,c5585e00,0,c5fea2e0,...) at devfs_read_f+0x7e dofileread(ed922c28,ffffffff,ffffffff,0,c5e27150,...) at dofileread+0x9e kern_readv(c5fea2e0,0,ed922c28,ed922c48,1,...) at kern_readv+0x58 read(c5fea2e0,ed922cec,ed922d28,c0e93ae6,0,...) at read+0x4f syscallenter(c5fea2e0,ed922ce4,ed922ce4,0,0,...) at syscallenter+0x263 syscall(ed922d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1749 tid 100108 td 0xc5fe92e0 sched_switch(c5fe92e0,0,104,191,7c519310,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5fe92e0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5fe92e0,c5fe92e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c584e070,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c584e070,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c584e000,c584e070,ed931c28,1,0,...) at tty_wait+0x71 ttydisc_read(c584e000,ed931c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c584d300,ed931c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e27850,ed931c28,c5585e00,0,c5fe92e0,...) at devfs_read_f+0x7e dofileread(ed931c28,ffffffff,ffffffff,0,c5e27850,...) at dofileread+0x9e kern_readv(c5fe92e0,0,ed931c28,ed931c48,1,...) at kern_readv+0x58 read(c5fe92e0,ed931cec,ed931d28,c0e93ae6,0,...) at read+0x4f syscallenter(c5fe92e0,ed931ce4,ed931ce4,0,0,...) at syscallenter+0x263 syscall(ed931d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1748 tid 100085 td 0xc5f0b5c0 sched_switch(c5f0b5c0,0,104,191,7c3472a8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5f0b5c0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5f0b5c0,c5f0b5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c584e270,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c584e270,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c584e200,c584e270,df3bdc28,1,0,...) at tty_wait+0x71 ttydisc_read(c584e200,df3bdc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c584d400,df3bdc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e27690,df3bdc28,c5585e00,0,c5f0b5c0,...) at devfs_read_f+0x7e dofileread(df3bdc28,ffffffff,ffffffff,0,c5e27690,...) at dofileread+0x9e kern_readv(c5f0b5c0,0,df3bdc28,df3bdc48,1,...) at kern_readv+0x58 read(c5f0b5c0,df3bdcec,df3bdd28,c0e93ae6,0,...) at read+0x4f syscallenter(c5f0b5c0,df3bdce4,df3bdce4,0,c1026f80,...) at syscallenter+0x263 syscall(df3bdd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1747 tid 100073 td 0xc5eb18a0 sched_switch(c5eb18a0,0,104,191,7c22f8d8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5eb18a0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5eb18a0,c5eb18a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c584e470,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c584e470,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c584e400,c584e470,df38dc28,1,0,...) at tty_wait+0x71 ttydisc_read(c584e400,df38dc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c584d500,df38dc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e27a10,df38dc28,c5585e00,0,c5eb18a0,...) at devfs_read_f+0x7e dofileread(df38dc28,ffffffff,ffffffff,0,c5e27a10,...) at dofileread+0x9e kern_readv(c5eb18a0,0,df38dc28,df38dc48,1,...) at kern_readv+0x58 read(c5eb18a0,df38dcec,df38dd28,c0e93ae6,0,...) at read+0x4f syscallenter(c5eb18a0,df38dce4,df38dce4,0,0,...) at syscallenter+0x263 syscall(df38dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1746 tid 100093 td 0xc5f0a000 sched_switch(c5f0a000,0,104,191,7be91370,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5f0a000,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5f0a000,c5f0a000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56ba070,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56ba070,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56ba000,c56ba070,df3dac28,1,0,...) at tty_wait+0x71 ttydisc_read(c56ba000,df3dac28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c584d600,df3dac28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e27460,df3dac28,c5585e00,0,c5f0a000,...) at devfs_read_f+0x7e dofileread(df3dac28,ffffffff,ffffffff,0,c5e27460,...) at dofileread+0x9e kern_readv(c5f0a000,0,df3dac28,df3dac48,1,...) at kern_readv+0x58 read(c5f0a000,df3dacec,df3dad28,c0e93ae6,0,...) at read+0x4f syscallenter(c5f0a000,df3dace4,df3dace4,0,0,...) at syscallenter+0x263 syscall(df3dad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1745 tid 100092 td 0xc5f0a2e0 sched_switch(c5f0a2e0,0,104,191,7c4f5b48,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5f0a2e0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5f0a2e0,c5f0a2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56ba270,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56ba270,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56ba200,c56ba270,df3d7c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56ba200,df3d7c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c57ea500,df3d7c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e26230,df3d7c28,c5585e00,0,c5f0a2e0,...) at devfs_read_f+0x7e dofileread(df3d7c28,ffffffff,ffffffff,0,c5e26230,...) at dofileread+0x9e kern_readv(c5f0a2e0,0,df3d7c28,df3d7c48,1,...) at kern_readv+0x58 read(c5f0a2e0,df3d7cec,df3d7d28,c0e93ae6,0,...) at read+0x4f syscallenter(c5f0a2e0,df3d7ce4,df3d7ce4,0,0,...) at syscallenter+0x263 syscall(df3d7d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1744 tid 100080 td 0xc5eb1000 sched_switch(c5eb1000,0,104,191,7c0cac68,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5eb1000,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5eb1000,c5eb1000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56ba470,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56ba470,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56ba400,c56ba470,df3a9c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56ba400,df3a9c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c57ea600,df3a9c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e3b578,df3a9c28,c5585e00,0,c5eb1000,...) at devfs_read_f+0x7e dofileread(df3a9c28,ffffffff,ffffffff,0,c5e3b578,...) at dofileread+0x9e kern_readv(c5eb1000,0,df3a9c28,df3a9c48,1,...) at kern_readv+0x58 read(c5eb1000,df3a9cec,df3a9d28,c0e93ae6,0,...) at read+0x4f syscallenter(c5eb1000,df3a9ce4,df3a9ce4,0,0,...) at syscallenter+0x263 syscall(df3a9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1743 tid 100068 td 0xc5a705c0 sched_switch(c5a705c0,0,104,191,7b361b10,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5a705c0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0e93140,160,0,c5a705c0,c5a705c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56ba670,0,c0e97efa,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56ba670,c1018a10,c0e97870,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56ba600,c56ba670,df379c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56ba600,df379c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c57ea800,df379c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e26498,df379c28,c5585e00,0,c5a705c0,...) at devfs_read_f+0x7e dofileread(df379c28,ffffffff,ffffffff,0,c5e26498,...) at dofileread+0x9e kern_readv(c5a705c0,0,df379c28,df379c48,1,...) at kern_readv+0x58 read(c5a705c0,df379cec,df379d28,c0e93ae6,0,...) at read+0x4f syscallenter(c5a705c0,df379ce4,df379ce4,0,0,...) at syscallenter+0x263 syscall(df379d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 1705 tid 100104 td 0xc5fea000 sched_switch(c5fea000,0,104,191,6d9a7338,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5fea000,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c099c5ba,c6043dd0,0,c0e8bdc7,c5fea000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6043de4,0,ed925a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6043de4,c6043dd0,c0e957cd,627,c5e3b3f0,...) at _cv_wait_sig+0x243 seltdwait(c5e3b3f0,58,c5585e00,c5fea000,c5fea0b0,...) at seltdwait+0xa2 kern_select(c5fea000,8,bfbfdfa0,0,0,0,20,c5fe22bc,c5fe22bc) at kern_select+0x504 select(c5fea000,ed925cec,ed925d28,c0e93ae6,0,...) at select+0x66 syscallenter(c5fea000,ed925ce4,ed925ce4,0,0,...) at syscallenter+0x263 syscall(ed925d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a8d43, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command cron pid 1675 tid 100100 td 0xc5feab80 kdb_enter(c0e8d86b,c0e8d86b,c0e8c015,df3fb06c,0,...) at kdb_enter+0x3a panic(c0e8c015,c0e8c0a9,c0edc3c8,356,c5fe3088,...) at panic+0x134 _mtx_lock_sleep(c5fe3088,c5feab80,0,c0edc3c8,356,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c5fe3088,0,c0edc3c8,356,6e7765c2,...) at _mtx_lock_flags+0xf7 trap_pfault(5,0,c0edc4a6,c5feab80,c5fe3000,...) at trap_pfault+0xc3 trap(df3fb18c) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc09f6e3f, esp = 0xdf3fb1cc, ebp = 0xdf3fb200 --- witness_unlock(c5fe3088,8,c0edc3c8,358,c5fe3088,...) at witness_unlock+0xaf _mtx_unlock_flags(c5fe3088,0,c0edc3c8,358,6e7765c2,...) at _mtx_unlock_flags+0xc3 trap_pfault(5,0,c0edc4a6,c0e7d292,c5fe3000,...) at trap_pfault+0xea trap(df3fb304) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc09f6e3f, esp = 0xdf3fb344, ebp = 0xdf3fb378 --- witness_unlock(c1018c44,0,c0e92018,db,0,...) at witness_unlock+0xaf _sx_sunlock(c1018c44,c0e92018,db,a,df3fb3fc,0,4,6,0,c5af28ec,c64d7e00,c5af28ec,c64d7e00,df3fb3fc) at _sx_sunlock+0xa0 tprintf(c5fe3000,6,c0e7d27d,c5ab7cd8,c0ebadc6,...) at tprintf+0x1cd nfs_msg(0,0,c0e7d234,3c0,c5feab80,...) at nfs_msg+0x67 nfs_feedback(4,4,df3fb520,c5b94000,df3fb5cc,...) at nfs_feedback+0x153 clnt_reconnect_call(c5a5d600,df3fb4f4,4,c5b94000,df3fb5cc,...) at clnt_reconnect_call+0x787 newnfs_request(df3fb5cc,c5a9b800,0,c5a9b910,c5925ae0,c5feab80,c5eda600,186a3,3,0,1,0) at newnfs_request+0xb7a nfscl_request(df3fb5cc,c5925ae0,c5feab80,c5eda600,0,...) at nfscl_request+0x82 nfsrpc_accessrpc(c5925ae0,3f,c5eda600,c5feab80,df3fb6a0,...) at nfsrpc_accessrpc+0x3f3 nfs34_access_otw(c5eda600,df3fb780,c0e7e4e8,18f,c5925ae0,...) at nfs34_access_otw+0x50 nfs_access(df3fb974,c0ee2656,c5a9b800,c5a9b800,df3fbb58,...) at nfs_access+0x1c8 VOP_ACCESS_APV(c0f9a1e0,df3fb974,c0e7e4e8,3eb,c60028ec,...) at VOP_ACCESS_APV+0xc5 nfs_lookup(df3fba20,c0e9dd63,c0fdd980,200000,df3fbb2c,...) at nfs_lookup+0x16c VOP_LOOKUP_APV(c0f9a1e0,df3fba20,df3fbb58,1f5,df3fbb44,...) at VOP_LOOKUP_APV+0xe5 lookup(df3fbb2c,c0e9c41d,ee,c9,c5fe3000,...) at lookup+0x67b namei(df3fbb2c,df3fbacc,60,0,c5feab80,...) at namei+0x592 kern_statat_vnhook(c5feab80,0,ffffff9c,804ef73,0,...) at kern_statat_vnhook+0x72 kern_statat(c5feab80,0,ffffff9c,804ef73,0,...) at kern_statat+0x3c kern_stat(c5feab80,804ef73,0,df3fbbe8,60,...) at kern_stat+0x36 stat(c5feab80,df3fbcec,df3fbd28,c0e93ae6,0,...) at stat+0x2f syscallenter(c5feab80,df3fbce4,df3fbce4,0,0,...) at syscallenter+0x263 syscall(df3fbd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (188, FreeBSD ELF32, stat), eip = 0x28191b13, esp = 0xbfbfe97c, ebp = 0xbfbfec78 --- Tracing command sshd pid 1664 tid 100083 td 0xc5eabb80 sched_switch(c5eabb80,0,104,191,8e9cfb30,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5eabb80,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c099c5ba,c6044050,0,c0e8bdc7,c5eabb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6044064,0,df3b5a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6044064,c6044050,c0e957cd,627,c5e27188,...) at _cv_wait_sig+0x243 seltdwait(c5e27188,58,c5eda900,c5eabb80,f33,...) at seltdwait+0xa2 kern_select(c5eabb80,5,2880c0b0,0,0,0,20,c5a6daf0,c5a6daf0) at kern_select+0x504 select(c5eabb80,df3b5cec,2880d070,1,0,...) at select+0x66 syscallenter(c5eabb80,df3b5ce4,c0cd6cbd,c1018a10,0,...) at syscallenter+0x263 syscall(df3b5d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284b0d43, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command rpc.statd pid 1523 tid 100109 td 0xc5fe9000 sched_switch(c5fe9000,0,104,191,e00c8638,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5fe9000,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ed934a14,c099c5ba,c5eb0b50,0,c5fe9000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c5eb0b64,0,ed934a4c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5eb0b64,c5eb0b50,7531,627,c5e3bd58,...) at _cv_timedwait_sig+0x252 seltdwait(ed934bf8,ed934c00,c5585e00,c5fe9000,f33,...) at seltdwait+0x8a kern_select(c5fe9000,8,bfbfed14,0,0,ed934c40,20,1e,0) at kern_select+0x504 select(c5fe9000,ed934cec,ed934d28,c0e93ae6,0,...) at select+0x66 syscallenter(c5fe9000,ed934ce4,ed934ce4,0,0,...) at syscallenter+0x263 syscall(ed934d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x2818cd43, esp = 0xbfbfec6c, ebp = 0xbfbfeda8 --- Tracing command md1 pid 1450 tid 100076 td 0xc5a6e2e0 sched_switch(c5a6e2e0,0,104,191,e4dd0550,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c5a6e2e0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5ffe800,5c,c0e38f61,0,0,...) at sleepq_wait+0x63 _sleep(c5ffe800,c5ffe820,25c,c0e38f61,0,...) at _sleep+0x372 md_kthread(c5ffe800,df399d28,c0e880a8,390,c5ece2bc,...) at md_kthread+0x11a fork_exit(c072ab50,c5ffe800,df399d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf399d60, ebp = 0 --- Tracing command rpcbind pid 1395 tid 100110 td 0xc5fe7b80 sched_switch(c5fe7b80,0,104,191,e199080,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5fe7b80,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ed95fa6c,c099c5ba,c6043e50,0,c5fe7b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c6043e64,0,ed95faa4,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c6043e64,c6043e50,7531,627,ed95fb5c,...) at _cv_timedwait_sig+0x252 seltdwait(ed95fc2c,ed95fc34,535,c5fe7b80,ed95fb2c,...) at seltdwait+0x8a poll(c5fe7b80,ed95fcec,ed95fd28,c0e93ae6,0,...) at poll+0x300 syscallenter(c5fe7b80,ed95fce4,ed95fce4,0,0,...) at syscallenter+0x263 syscall(ed95fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813ff4b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1365 tid 100091 td 0xc5f0a5c0 sched_switch(c5f0a5c0,0,104,191,aca3f228,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5f0a5c0,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c099c5ba,c6044010,0,c0e8bdc7,c5f0a5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6044024,0,df3d4a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6044024,c6044010,c0e957cd,627,c5e26508,...) at _cv_wait_sig+0x243 seltdwait(c5e26508,58,c5585e00,c5f0a5c0,0,...) at seltdwait+0xa2 kern_select(c5f0a5c0,9,284760ac,0,0,0,20,c5f1b2bc,c5f1b2bc) at kern_select+0x504 select(c5f0a5c0,df3d4cec,df3d4d28,c0e93ae6,0,...) at select+0x66 syscallenter(c5f0a5c0,df3d4ce4,df3d4ce4,0,c09f6dea,...) at syscallenter+0x263 syscall(df3d4d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28199d43, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1186 tid 100077 td 0xc5a6e000 sched_switch(c5a6e000,0,104,191,661b1960,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5a6e000,0,c0e93140,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c099c5ba,c5717cd0,0,c0e8bdc7,c5a6e000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5717ce4,0,df39da4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5717ce4,c5717cd0,c0e957cd,627,c5e27e00,...) at _cv_wait_sig+0x243 seltdwait(c5e27e00,58,c5585e00,c5a6e000,c0ec5f24,...) at seltdwait+0xa2 kern_select(c5a6e000,5,bfbfe960,0,0,0,20,c5ece000,c5ece000) at kern_select+0x504 select(c5a6e000,df39dcec,df39dd28,c0e93ae6,0,...) at select+0x66 syscallenter(c5a6e000,df39dce4,df39dce4,0,0,...) at syscallenter+0x263 syscall(df39dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x808ba43, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command md0 pid 99 tid 100107 td 0xc5fe95c0 sched_switch(c5fe95c0,0,104,191,58fbfad8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c5fe95c0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c574a000,5c,c0e38f61,0,0,...) at sleepq_wait+0x63 _sleep(c574a000,c574a020,25c,c0e38f61,0,...) at _sleep+0x372 md_kthread(c574a000,ed92ed28,c0e880a8,390,c5f1c834,...) at md_kthread+0x11a fork_exit(c072ab50,c574a000,ed92ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed92ed60, ebp = 0 --- Tracing command softdepflush pid 17 tid 100064 td 0xc5a71000 sched_switch(c5a71000,0,104,191,9fcd9e50,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c5a71000,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1196a20,54,c0ec3129,0,0,...) at sleepq_timedwait+0x6b _sleep(c1196a20,c11969c4,54,c0ec3129,3e8,...) at _sleep+0x342 softdep_flush(0,df36bd28,c0e880a8,390,c5a6b578,...) at softdep_flush+0x25a fork_exit(c0bda5d0,0,df36bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf36bd60, ebp = 0 --- Tracing command vnlru pid 16 tid 100063 td 0xc57d22e0 sched_switch(c57d22e0,0,104,191,88d88250,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,60,...) at mi_switch+0x1eb sleepq_switch(c57d22e0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c5a6b834,60,c0e9df64,0,0,...) at sleepq_timedwait+0x6b _sleep(c5a6b834,c11904b4,260,c0e9df64,3e8,...) at _sleep+0x342 vnlru_proc(0,df368d28,c0e880a8,390,c5a6b834,...) at vnlru_proc+0xe7 fork_exit(c0a48dd0,0,df368d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf368d60, ebp = 0 --- Tracing command syncer pid 9 tid 100062 td 0xc57d25c0 sched_switch(c57d25c0,0,104,191,8adffde8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d25c0,0,c0e93140,28b,c57d25c0,...) at sleepq_switch+0x162 sleepq_timedwait(c11904f4,0,df365c78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c11904f4,c11904e0,3e8,76a,4e20,...) at _cv_timedwait+0x252 sched_sync(0,df365d28,c0e880a8,390,c5a6baf0,...) at sched_sync+0x595 fork_exit(c0a46d70,0,df365d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf365d60, ebp = 0 --- Tracing command bufdaemon pid 8 tid 100061 td 0xc57d28a0 sched_switch(c57d28a0,0,104,191,b8690700,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c57d28a0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1190188,54,c0e9af3a,0,0,...) at sleepq_timedwait+0x6b _sleep(c1190188,c1190190,54,c0e9af3a,3e8,...) at _sleep+0x342 buf_daemon(0,df362d28,c0e880a8,390,c5a6d000,...) at buf_daemon+0x175 fork_exit(c0a2e4e0,0,df362d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf362d60, ebp = 0 --- Tracing command pagezero pid 7 tid 100060 td 0xc57d2b80 sched_switch(c57d2b80,0,104,191,d22cae80,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d2b80,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c11988dc,0,c0ec9a70,0,0,...) at sleepq_timedwait+0x6b _sleep(c11988dc,c1197380,0,c0ec9a70,493e0,...) at _sleep+0x342 vm_pagezero(0,df35fd28,c0e880a8,390,c55a32bc,...) at vm_pagezero+0xdc fork_exit(c0c1ec60,0,df35fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf35fd60, ebp = 0 --- Tracing command vmdaemon pid 6 tid 100059 td 0xc581f000 sched_switch(c581f000,0,104,191,cf150d58,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,74,...) at mi_switch+0x1eb sleepq_switch(c581f000,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11984c8,74,c0e9af3a,0,0,...) at sleepq_wait+0x63 _sleep(c11984c8,c11984cc,74,c0e9af3a,0,...) at _sleep+0x372 vm_daemon(0,df35cd28,c0e880a8,390,c55a3578,...) at vm_daemon+0x59 fork_exit(c0c18820,0,df35cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf35cd60, ebp = 0 --- Tracing command pagedaemon pid 5 tid 100058 td 0xc581f2e0 sched_switch(c581f2e0,0,104,191,86db05a8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c581f2e0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1198490,54,c0e9af3a,0,0,...) at sleepq_timedwait+0x6b _sleep(c1198490,c1197380,54,c0e9af3a,1388,...) at _sleep+0x342 vm_pageout(0,df359d28,c0e880a8,390,c55a3834,...) at vm_pageout+0x2c7 fork_exit(c0c19840,0,df359d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf359d60, ebp = 0 --- Tracing command xpt_thrd pid 4 tid 100057 td 0xc581f5c0 sched_switch(c581f5c0,0,104,191,30d638a8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c581f5c0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0fdf1d4,5c,c0cfdbea,0,0,...) at sleepq_wait+0x63 _sleep(c0fdf1d4,c0fdf218,5c,c0cfdbea,0,...) at _sleep+0x372 xpt_scanner_thread(0,df356d28,c0e880a8,390,c55a3af0,...) at xpt_scanner_thread+0x47 fork_exit(c048bee0,0,df356d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf356d60, ebp = 0 --- Tracing command sctp_iterator pid 3 tid 100056 td 0xc581f8a0 sched_switch(c581f8a0,0,104,191,c6924268,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c581f8a0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1191ff8,0,c0eb133f,0,0,...) at sleepq_wait+0x63 _sleep(c1191ff8,c1191fc0,0,c0eb133f,0,...) at _sleep+0x372 sctp_iterator_thread(0,df353d28,c0e880a8,390,c5793000,...) at sctp_iterator_thread+0x5c fork_exit(c0ad83f0,0,df353d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf353d60, ebp = 0 --- Tracing command fdc0 pid 2 tid 100053 td 0xc58212e0 sched_switch(c58212e0,0,104,191,b56438,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c58212e0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c573ba3c,5c,c0e8071a,0,0,...) at sleepq_timedwait+0x6b _sleep(c573ba3c,c573baf0,5c,c0e8071a,3e8,...) at _sleep+0x342 fdc_thread(c573ba00,df34ad28,c0e880a8,390,c57932bc,...) at fdc_thread+0x27d fork_exit(c0c9dde0,c573ba00,df34ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf34ad60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc57cf5c0 sched_switch(c57cf5c0,0,104,191,10192e48,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57cf5c0,0,c0e93140,268,c57cf5c0,...) at sleepq_switch+0x162 sleepq_wait(c57d8d34,0,c5314cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57d8d34,c57d8dd4,c0e5ea9e,6b,c57d8d3c,...) at _cv_wait+0x243 usb_process(c57d8d2c,c5314d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57d8d2c,c5314d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5314d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc57cf8a0 sched_switch(c57cf8a0,0,104,191,3e8503e8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57cf8a0,0,c0e93140,268,c57cf8a0,...) at sleepq_switch+0x162 sleepq_wait(c57d8d04,0,c5311cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57d8d04,c57d8dd4,c0e5ea9e,6b,c57d8d0c,...) at _cv_wait+0x243 usb_process(c57d8cfc,c5311d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57d8cfc,c5311d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5311d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc57cfb80 sched_switch(c57cfb80,0,104,191,f9ae680,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57cfb80,0,c0e93140,268,c57cfb80,...) at sleepq_switch+0x162 sleepq_wait(c57d8cd4,0,c530ecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57d8cd4,c57d8dd4,c0e5ea9e,6b,c57d8cdc,...) at _cv_wait+0x243 usb_process(c57d8ccc,c530ed28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57d8ccc,c530ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc530ed60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc57d0000 sched_switch(c57d0000,0,104,191,f9ad1c8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d0000,0,c0e93140,268,c57d0000,...) at sleepq_switch+0x162 sleepq_wait(c57d8ca4,0,c530bcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57d8ca4,c57d8dd4,c0e5ea9e,6b,c57d8cac,...) at _cv_wait+0x243 usb_process(c57d8c9c,c530bd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57d8c9c,c530bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc530bd60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc57d02e0 sched_switch(c57d02e0,0,104,191,f9abb18,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d02e0,0,c0e93140,268,c57d02e0,...) at sleepq_switch+0x162 sleepq_wait(c57c0dac,0,c5305cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57c0dac,c57c0e4c,c0e5ea9e,6b,c57c0db4,...) at _cv_wait+0x243 usb_process(c57c0da4,c5305d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57c0da4,c5305d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5305d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc57d05c0 sched_switch(c57d05c0,0,104,191,c4b7b448,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d05c0,0,c0e93140,268,c57d05c0,...) at sleepq_switch+0x162 sleepq_wait(c57c0d7c,0,c5302cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57c0d7c,c57c0e4c,c0e5ea9e,6b,c57c0d84,...) at _cv_wait+0x243 usb_process(c57c0d74,c5302d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57c0d74,c5302d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5302d60, ebp = 0 --- Tracing command usb pid 15 tid 100043 td 0xc57d08a0 sched_switch(c57d08a0,0,104,191,8892c88,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d08a0,0,c0e93140,268,c57d08a0,...) at sleepq_switch+0x162 sleepq_wait(c57c0d4c,0,c52ffcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57c0d4c,c57c0e4c,c0e5ea9e,6b,c57c0d54,...) at _cv_wait+0x243 usb_process(c57c0d44,c52ffd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57c0d44,c52ffd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52ffd60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc57d0b80 sched_switch(c57d0b80,0,104,191,8891aa0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57d0b80,0,c0e93140,268,c57d0b80,...) at sleepq_switch+0x162 sleepq_wait(c57c0d1c,0,c52fccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57c0d1c,c57c0e4c,c0e5ea9e,6b,c57c0d24,...) at _cv_wait+0x243 usb_process(c57c0d14,c52fcd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57c0d14,c52fcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52fcd60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc57a8000 sched_switch(c57a8000,0,104,191,8890258,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57a8000,0,c0e93140,268,c57a8000,...) at sleepq_switch+0x162 sleepq_wait(c57aedac,0,c52f5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57aedac,c57aee4c,c0e5ea9e,6b,c57aedb4,...) at _cv_wait+0x243 usb_process(c57aeda4,c52f5d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57aeda4,c52f5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52f5d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc57a82e0 sched_switch(c57a82e0,0,104,191,c4b76260,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57a82e0,0,c0e93140,268,c57a82e0,...) at sleepq_switch+0x162 sleepq_wait(c57aed7c,0,c52f2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57aed7c,c57aee4c,c0e5ea9e,6b,c57aed84,...) at _cv_wait+0x243 usb_process(c57aed74,c52f2d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57aed74,c52f2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52f2d60, ebp = 0 --- Tracing command usb pid 15 tid 100038 td 0xc57a85c0 sched_switch(c57a85c0,0,104,191,80ed940,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57a85c0,0,c0e93140,268,c57a85c0,...) at sleepq_switch+0x162 sleepq_wait(c57aed4c,0,c52efcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57aed4c,c57aee4c,c0e5ea9e,6b,c57aed54,...) at _cv_wait+0x243 usb_process(c57aed44,c52efd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57aed44,c52efd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52efd60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc57a88a0 sched_switch(c57a88a0,0,104,191,80ec660,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57a88a0,0,c0e93140,268,c57a88a0,...) at sleepq_switch+0x162 sleepq_wait(c57aed1c,0,c52eccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57aed1c,c57aee4c,c0e5ea9e,6b,c57aed24,...) at _cv_wait+0x243 usb_process(c57aed14,c52ecd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c57aed14,c52ecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52ecd60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc57aa000 sched_switch(c57aa000,0,104,191,80eac18,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57aa000,0,c0e93140,268,c57aa000,...) at sleepq_switch+0x162 sleepq_wait(c5797dac,0,c52e5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5797dac,c5797e4c,c0e5ea9e,6b,c5797db4,...) at _cv_wait+0x243 usb_process(c5797da4,c52e5d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5797da4,c52e5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52e5d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc57aa2e0 sched_switch(c57aa2e0,0,104,191,c4b6de00,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57aa2e0,0,c0e93140,268,c57aa2e0,...) at sleepq_switch+0x162 sleepq_wait(c5797d7c,0,c52e2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5797d7c,c5797e4c,c0e5ea9e,6b,c5797d84,...) at _cv_wait+0x243 usb_process(c5797d74,c52e2d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5797d74,c52e2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52e2d60, ebp = 0 --- Tracing command usb pid 15 tid 100033 td 0xc57aa5c0 sched_switch(c57aa5c0,0,104,191,794b258,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57aa5c0,0,c0e93140,268,c57aa5c0,...) at sleepq_switch+0x162 sleepq_wait(c5797d4c,0,c52dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5797d4c,c5797e4c,c0e5ea9e,6b,c5797d54,...) at _cv_wait+0x243 usb_process(c5797d44,c52dfd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5797d44,c52dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52dfd60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc57aa8a0 sched_switch(c57aa8a0,0,104,191,794a110,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57aa8a0,0,c0e93140,268,c57aa8a0,...) at sleepq_switch+0x162 sleepq_wait(c5797d1c,0,c52dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5797d1c,c5797e4c,c0e5ea9e,6b,c5797d24,...) at _cv_wait+0x243 usb_process(c5797d14,c52dcd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5797d14,c52dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52dcd60, ebp = 0 --- Tracing command usb pid 15 tid 100030 td 0xc57162e0 sched_switch(c57162e0,0,104,191,7948830,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57162e0,0,c0e93140,268,c57162e0,...) at sleepq_switch+0x162 sleepq_wait(c5785dac,0,c52d5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5785dac,c5785e4c,c0e5ea9e,6b,c5785db4,...) at _cv_wait+0x243 usb_process(c5785da4,c52d5d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5785da4,c52d5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52d5d60, ebp = 0 --- Tracing command usb pid 15 tid 100029 td 0xc57165c0 sched_switch(c57165c0,0,104,191,c4b705b0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57165c0,0,c0e93140,268,c57165c0,...) at sleepq_switch+0x162 sleepq_wait(c5785d7c,0,c52d2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5785d7c,c5785e4c,c0e5ea9e,6b,c5785d84,...) at _cv_wait+0x243 usb_process(c5785d74,c52d2d28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5785d74,c52d2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52d2d60, ebp = 0 --- Tracing command usb pid 15 tid 100028 td 0xc57168a0 sched_switch(c57168a0,0,104,191,718c9e8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57168a0,0,c0e93140,268,c57168a0,...) at sleepq_switch+0x162 sleepq_wait(c5785d4c,0,c52cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5785d4c,c5785e4c,c0e5ea9e,6b,c5785d54,...) at _cv_wait+0x243 usb_process(c5785d44,c52cfd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5785d44,c52cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52cfd60, ebp = 0 --- Tracing command usb pid 15 tid 100027 td 0xc5716b80 sched_switch(c5716b80,0,104,191,718b660,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5716b80,0,c0e93140,268,c5716b80,...) at sleepq_switch+0x162 sleepq_wait(c5785d1c,0,c52cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5785d1c,c5785e4c,c0e5ea9e,6b,c5785d24,...) at _cv_wait+0x243 usb_process(c5785d14,c52ccd28,c0e880a8,390,c5793578,...) at usb_process+0x193 fork_exit(c0853a60,c5785d14,c52ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52ccd60, ebp = 0 --- Tracing command yarrow pid 14 tid 100013 td 0xc55a52e0 sched_switch(c55a52e0,0,104,191,f5ac5c58,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c55a52e0,0,c0e93140,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1019124,0,c0e8071a,2,0,...) at sleepq_timedwait+0x6b _sleep(c1019124,0,0,c0e8071a,64,...) at _sleep+0x342 pause(c0e8071a,64,c0e4d24e,111,0,...) at pause+0x47 random_kthread(0,c5236d28,c0e880a8,390,c55a2000,...) at random_kthread+0x1ef fork_exit(c07bea70,0,c5236d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5236d60, ebp = 0 --- Tracing command geom pid 13 tid 100011 td 0xc55a58a0 sched_switch(c55a58a0,0,104,191,e41277a0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c55a58a0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1016d48,5c,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c1016d48,c1016ca8,25c,c0e8071a,0,...) at _sleep+0x372 g_io_schedule_down(c55a58a0,0,c0e822c5,6c,c522cd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c522cd28,c0e880a8,390,c55a22bc,...) at g_down_procbody+0x9d fork_exit(c0942160,0,c522cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc522cd60, ebp = 0 --- Tracing command geom pid 13 tid 100010 td 0xc55a5b80 sched_switch(c55a5b80,0,104,191,e416bd80,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c55a5b80,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1016d44,5c,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c1016d44,c1016cc8,25c,c0e8071a,0,...) at _sleep+0x372 g_io_schedule_up(c55a5b80,0,c0e822c5,5f,c5229d14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c5229d28,c0e880a8,390,c55a22bc,...) at g_up_procbody+0x9d fork_exit(c0942200,0,c5229d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5229d60, ebp = 0 --- Tracing command geom pid 13 tid 100009 td 0xc55eb000 sched_switch(c55eb000,0,104,191,eabe03d8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,5c,...) at mi_switch+0x1eb sleepq_switch(c55eb000,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c1016d3c,5c,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c1016d3c,c1016c6c,25c,c0e8071a,0,...) at _sleep+0x372 g_run_events(c1026f80,0,c0e822c5,79,c5226d14,...) at g_run_events+0x526 g_event_procbody(0,c5226d28,c0e880a8,390,c55a22bc,...) at g_event_procbody+0x90 fork_exit(c09422a0,0,c5226d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5226d60, ebp = 0 --- Tracing command intr pid 12 tid 100054 td 0xc5821000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xc58215c0 sched_switch(c58215c0,0,109,191,6266d3b8,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c58215c0,...) at mi_switch+0x1eb ithread_loop(c55a0a50,df33dd28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0a50,df33dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf33dd60, ebp = 0 --- Tracing command intr pid 12 tid 100051 td 0xc57cf000 sched_switch(c57cf000,0,109,191,e4161d98,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c57cf000,...) at mi_switch+0x1eb ithread_loop(c55a0ae0,c533ed28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0ae0,c533ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc533ed60, ebp = 0 --- Tracing command intr pid 12 tid 100050 td 0xc57cf2e0 sched_switch(c57cf2e0,0,109,191,925393b8,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c57cf2e0,...) at mi_switch+0x1eb ithread_loop(c55a0b10,c533ad28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0b10,c533ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc533ad60, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc57d2000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xc57a8b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xc57aab80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100026 td 0xc5718000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc57188a0 sched_switch(c57188a0,0,109,191,e41659b0,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c57188a0,...) at mi_switch+0x1eb ithread_loop(c55a0b90,c5254d28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0b90,c5254d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5254d60, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc55eb8a0 sched_switch(c55eb8a0,0,109,191,62b04988,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c55eb8a0,...) at mi_switch+0x1eb ithread_loop(c55a0bd0,c524ed28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0bd0,c524ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc524ed60, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc55ebb80 sched_switch(c55ebb80,0,109,191,64c141f8,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c55ebb80,...) at mi_switch+0x1eb ithread_loop(c55a0be0,c524bd28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0be0,c524bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc524bd60, ebp = 0 --- Tracing command intr pid 12 tid 100017 td 0xc57145c0 sched_switch(c57145c0,0,109,191,624827a0,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c57145c0,...) at mi_switch+0x1eb ithread_loop(c55a0bf0,c5242d28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0bf0,c5242d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5242d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc55eb2e0 sched_switch(c55eb2e0,0,109,191,b54b60,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c55eb2e0,...) at mi_switch+0x1eb ithread_loop(c55a0c00,c5223d28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0c00,c5223d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5223d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc55eb5c0 sched_switch(c55eb5c0,0,109,191,137bbc8,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c55eb5c0,...) at mi_switch+0x1eb ithread_loop(c55a0c10,c5220d28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0c10,c5220d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5220d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc55a4000 sched_switch(c55a4000,0,109,191,e931e328,...) at sched_switch+0x3bc mi_switch(109,0,c0e883e5,554,c55a4000,...) at mi_switch+0x1eb ithread_loop(c55a0c20,c521dd28,c0e880a8,390,c55a2578,...) at ithread_loop+0x1fb fork_exit(c0983400,c55a0c20,c521dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc521dd60, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc55a42e0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100004 td 0xc55a45c0 sched_switch(c55a45c0,0,608,18c,137a560,...) at sched_switch+0x3bc mi_switch(608,0,c0e8e09f,cd,ffffffff,...) at mi_switch+0x1eb critical_exit(0,ffffffff,c1027600,1,c5214ce8,...) at critical_exit+0xa8 cpu_idle(0,c5214cc4,c0e8f8e3,3bb,c55a45c0,...) at cpu_idle+0x9e sched_idletd(0,c5214d28,c0e880a8,390,c55a2834,...) at sched_idletd+0x263 fork_exit(c09d2230,0,c5214d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5214d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc55a48a0 sched_switch(c55a48a0,0,608,18c,136e260,...) at sched_switch+0x3bc mi_switch(608,0,c0e8e09f,cd,ffffffff,...) at mi_switch+0x1eb critical_exit(0,ffffffff,c1026f80,1,c5211ce8,...) at critical_exit+0xa8 cpu_idle(0,c5211cc4,c0e8f8e3,3bb,c55a48a0,...) at cpu_idle+0x9e sched_idletd(0,c5211d28,c0e880a8,390,c55a2834,...) at sched_idletd+0x263 fork_exit(c09d2230,0,c5211d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5211d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc55a4b80 cpustop_handler(2,c520d078,c0cd6576,c5207c80,0,...) at cpustop_handler+0x34 ipi_nmi_handler(c5207c80,0,c0edddf0,f4,c55a2af0,...) at ipi_nmi_handler+0x2f trap(c520d084) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0cbfc50, esp = 0xc520d0c4, ebp = 0xc520d0f4 --- Xapic_isr1(c0e8c015,c0e8c0a9,c0edc3c8,356,c55a2b78,...) at Xapic_isr1 _mtx_lock_sleep(c55a2b78,c55a4b80,0,c0edc3c8,356,...) at _mtx_lock_sleep+0x4a _mtx_lock_flags(c55a2b78,0,c0edc3c8,356,6e7765c2,...) at _mtx_lock_flags+0xf7 trap_pfault(5,0,c0edc4a6,c5537800,c55a2af0,...) at trap_pfault+0xc3 trap(c520d220) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc09f6e3f, esp = 0xc520d260, ebp = 0xc520d294 --- witness_unlock(c55a2b78,8,c0edc3c8,358,c55a2b78,...) at witness_unlock+0xaf _mtx_unlock_flags(c55a2b78,0,c0edc3c8,358,75f,...) at _mtx_unlock_flags+0xc3 trap_pfault(5,0,c0edc4a6,c099c533,c55a2af0,...) at trap_pfault+0xea trap(c520d398) at trap+0x4c5 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0a58388, esp = 0xc520d3d8, ebp = 0xc520d3d8 --- strlen(75f,c520d448,c520d468,c0ace864,c5a46000,...) at strlen+0x8 msgbuf_addstr(c1488fc8,6,75f,1,c520d4ec,...) at msgbuf_addstr+0x25 msglogstr(c15b5308,9fd,10003e4,c5abec6c,105b5308,...) at msglogstr+0x28 putbuf(c55a4b80,c101a370,c0ebadd4,c09e5470,c0e7d292,...) at putbuf+0x195 putchar(a,c520d648,0,0,c55a4dd0,...) at putchar+0x8b kvprintf(c0e7d27d,c09e5470,c520d648,a,c520d688,...) at kvprintf+0x9e tprintf(c55a2af0,6,c0e7d27d,c5ab7cd8,c0ebadc6,...) at tprintf+0x16d nfs_msg(0,0,c0e7d234,3c0,c55a4b80,...) at nfs_msg+0x67 nfs_feedback(4,6,c520d7ac,c8e10700,c520d894,...) at nfs_feedback+0x153 clnt_reconnect_call(c5a5d600,c520d780,6,c8e10700,c520d894,...) at clnt_reconnect_call+0x787 newnfs_request(c520d894,c5a9b800,0,c5a9b910,c59256cc,c55a4b80,c5585e00,186a3,3,0,1,0) at newnfs_request+0xb7a nfscl_request(c520d894,c59256cc,c55a4b80,c5585e00,0,...) at nfscl_request+0x82 nfsrpc_read(c59256cc,c520da28,c5585e00,c55a4b80,c520d950,...) at nfsrpc_read+0x4f9 ncl_readrpc(c59256cc,c520da28,c5585e00,90,c4,...) at ncl_readrpc+0x4b ncl_getpages(c520da94,c0ee2357,c59256cc,c59256cc,c159de58,...) at ncl_getpages+0x389 VOP_GETPAGES_APV(c0f9a1e0,c520da94,c0ec9ac4,297,c0fddea0,...) at VOP_GETPAGES_APV+0xd6 vnode_pager_getpages(c159de58,c520db8c,8,5,c520dc10,...) at vnode_pager_getpages+0xb4 vm_fault_hold(c55a6e80,8055000,1,0,0,...) at vm_fault_hold+0x104f vm_fault(c55a6e80,8055000,1,0,80551d3,...) at vm_fault+0x2f trap_pfault(5,0,c0edc4a6,30d,c55a2af0,...) at trap_pfault+0x10d trap(c520dd28) at trap+0x2c9 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0x80551d3, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc55a5000 sched_switch(c55a5000,0,104,191,71471d8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c55a5000,0,c0e93140,268,c55a5000,...) at sleepq_switch+0x162 sleepq_wait(c1196220,0,c520ac8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c1196220,c1196204,c0ebddcf,194,0,...) at _cv_wait+0x243 audit_worker(0,c520ad28,c0e880a8,390,c55a3000,...) at audit_worker+0x84 fork_exit(c0b9f4c0,0,c520ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc520ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100065 td 0xc5a70b80 sched_switch(c5a70b80,0,104,191,ab65a3a8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5a70b80,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5a4f0c0,0,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c5a4f0c0,c5a4f0d8,0,c0e8071a,0,...) at _sleep+0x372 taskqueue_thread_loop(c11edca0,df36ed28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c09eed50,c11edca0,df36ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf36ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100055 td 0xc581fb80 sched_switch(c581fb80,0,104,191,97387648,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c581fb80,0,c0e93140,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c1019124,0,c0e8071a,2,0,...) at sleepq_timedwait+0x6b _sleep(c1019124,0,0,c0e8071a,bb8,...) at _sleep+0x342 pause(c0e8071a,bb8,124,122,c1016ea8,...) at pause+0x47 deadlkres(0,df350d28,c0e880a8,390,c1016e20,...) at deadlkres+0x324 fork_exit(c0963c60,0,df350d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf350d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc57182e0 sched_switch(c57182e0,0,104,191,7131390,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57182e0,0,c0e93140,268,c10170e0,...) at sleepq_switch+0x162 sleepq_wait(c57691c0,0,c0e8e5d9,c0e8071a,0,...) at sleepq_wait+0x63 msleep_spin(c57691c0,c57691d8,c0e8071a,0,c0e8bdc7,...) at msleep_spin+0x21d taskqueue_thread_loop(c5771568,c52c5d28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0x8f fork_exit(c09eed50,c5771568,c52c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52c5d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc57185c0 sched_switch(c57185c0,0,104,191,b86f31c8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57185c0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5753140,0,c0e8e5d9,c0e8071a,0,...) at sleepq_wait+0x63 msleep_spin(c5753140,c5753158,c0e8071a,0,c0e8bdc7,...) at msleep_spin+0x21d taskqueue_thread_loop(c5759568,c529ad28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0x8f fork_exit(c09eed50,c5759568,c529ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc529ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc5718b80 sched_switch(c5718b80,0,104,191,30d833a0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5718b80,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b1500,0,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c56b1500,c56b1518,0,c0e8071a,0,...) at _sleep+0x372 taskqueue_thread_loop(c10177b8,c5251d28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c09eed50,c10177b8,c5251d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5251d60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc5714000 sched_switch(c5714000,0,104,191,30d82208,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5714000,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b15c0,0,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c56b15c0,c56b15d8,0,c0e8071a,0,...) at _sleep+0x372 taskqueue_thread_loop(c1196804,c5248d28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c09eed50,c1196804,c5248d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5248d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc57142e0 sched_switch(c57142e0,0,104,191,b92779c0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57142e0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b1600,0,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c56b1600,c56b1618,0,c0e8071a,0,...) at _sleep+0x372 taskqueue_thread_loop(c1035d08,c5245d28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c09eed50,c1035d08,c5245d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5245d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc57148a0 sched_switch(c57148a0,0,104,191,30d7eb28,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c57148a0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b1680,0,c0e8e5d9,c0e8071a,0,...) at sleepq_wait+0x63 msleep_spin(c56b1680,c56b1698,c0e8071a,0,c0e8bdc7,...) at msleep_spin+0x21d taskqueue_thread_loop(c0fe20b8,c523fd28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0x8f fork_exit(c09eed50,c0fe20b8,c523fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc523fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc5714b80 sched_switch(c5714b80,0,104,191,30d7dda8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5714b80,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b1680,0,c0e8e5d9,c0e8071a,0,...) at sleepq_wait+0x63 msleep_spin(c56b1680,c56b1698,c0e8071a,0,c0e8bdc7,...) at msleep_spin+0x21d taskqueue_thread_loop(c0fe20b8,c523cd28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0x8f fork_exit(c09eed50,c0fe20b8,c523cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc523cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc5716000 sched_switch(c5716000,0,104,191,30d7cd58,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c5716000,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c56b1680,0,c0e8e5d9,c0e8071a,0,...) at sleepq_wait+0x63 msleep_spin(c56b1680,c56b1698,c0e8071a,0,c0e8bdc7,...) at msleep_spin+0x21d taskqueue_thread_loop(c0fe20b8,c5239d28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0x8f fork_exit(c09eed50,c0fe20b8,c5239d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5239d60, ebp = 0 --- Tracing command kernel pid 0 tid 100012 td 0xc55a55c0 sched_switch(c55a55c0,0,104,191,749cf4e8,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,0,...) at mi_switch+0x1eb sleepq_switch(c55a55c0,0,c0e93140,268,0,...) at sleepq_switch+0x162 sleepq_wait(c555eb40,0,c0e8071a,0,0,...) at sleepq_wait+0x63 _sleep(c555eb40,c555eb58,0,c0e8071a,0,...) at _sleep+0x372 taskqueue_thread_loop(c1034720,c522fd28,c0e880a8,390,c1016e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c09eed50,c1034720,c522fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc522fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc10170e0 sched_switch(c10170e0,0,104,191,c97075b0,...) at sched_switch+0x3bc mi_switch(104,0,c0e93140,1f3,54,...) at mi_switch+0x1eb sleepq_switch(c10170e0,0,c0e93140,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c1016e20,54,c0e901ff,0,0,...) at sleepq_timedwait+0x6b _sleep(c1016e20,0,54,c0e901ff,2710,...) at _sleep+0x342 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2030 MB Physical memory: 2030 MB Dumping 149 MB:Dumping 149 MB: 134 134 118 118 102 102 86 86 70 70 54 54 38 38 22 22 6 6 Dump complete Dump complete = 0xf db:0:doadump> reset (kgdb) bt #0 doadump () at pcpu.h:244 #1 0xc04de0f9 in db_fncall (dummy1=-1063385158, dummy2=0, dummy3=-1, dummy4=0xdf3fad04 "\030­?ß") at /mnt/pho/src/sys/ddb/db_command.c:548 #2 0xc04de52f in db_command (last_cmdp=0xc0fe027c, cmd_table=0x0, dopager=0) at /mnt/pho/src/sys/ddb/db_command.c:445 #3 0xc04de5e4 in db_command_script (command=0xc0fe1188 "call doadump") at /mnt/pho/src/sys/ddb/db_command.c:516 #4 0xc04e28b0 in db_script_exec (scriptname=0xc0fe0ae0 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at /mnt/pho/src/sys/ddb/db_script.c:302 #5 0xc04e2941 in db_run_cmd (addr=-1057051072, have_addr=0, count=-1067160775, modif=0xdf3fae3c "") at /mnt/pho/src/sys/ddb/db_script.c:375 #6 0xc04de4f1 in db_command (last_cmdp=0xc0fe027c, cmd_table=0x0, dopager=1) at /mnt/pho/src/sys/ddb/db_command.c:445 #7 0xc04de64a in db_command_loop () at /mnt/pho/src/sys/ddb/db_command.c:498 #8 0xc04e05cd in db_trap (type=3, code=0) at /mnt/pho/src/sys/ddb/db_main.c:229 #9 0xc09e0932 in kdb_trap (type=3, code=0, tf=0xdf3fafec) at /mnt/pho/src/sys/kern/subr_kdb.c:533 #10 0xc0cd6bab in trap (frame=0xdf3fafec) at /mnt/pho/src/sys/i386/i386/trap.c:719 #11 0xc0cbf8bc in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #12 0xc09e07ba in kdb_enter (why=0xc0e8d86b "panic", msg=0xc0e8d86b "panic") at cpufunc.h:71 #13 0xc09abd84 in panic (fmt=0xc0e8c015 "_mtx_lock_sleep: recursed on non-recursive mutex %s @ %s:%d\n") at /mnt/pho/src/sys/kern/kern_shutdown.c:584 #14 0xc099c72a in _mtx_lock_sleep (m=0xc5fe3088, tid=3321801600, opts=0, file=0xc0edc3c8 "/mnt/pho/src/sys/i386/i386/trap.c", line=854) at /mnt/pho/src/sys/kern/kern_mutex.c:341 #15 0xc099c947 in _mtx_lock_flags (m=0xc5fe3088, opts=0, file=0xc0edc3c8 "/mnt/pho/src/sys/i386/i386/trap.c", line=854) at /mnt/pho/src/sys/kern/kern_mutex.c:203 #16 0xc0cd62d3 in trap_pfault (frame=0xdf3fb18c, usermode=0, eva=1853318594) at /mnt/pho/src/sys/i386/i386/trap.c:854 #17 0xc0cd6a05 in trap (frame=0xdf3fb18c) at /mnt/pho/src/sys/i386/i386/trap.c:555 #18 0xc0cbf8bc in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #19 0xc09f6e3f in witness_unlock (lock=0xc5fe3088, flags=8, file=0xc0edc3c8 "/mnt/pho/src/sys/i386/i386/trap.c", line=856) at /mnt/pho/src/sys/kern/subr_witness.c:1499 #20 0xc099c513 in _mtx_unlock_flags (m=0xc5fe3088, opts=0, file=0xc0edc3c8 "/mnt/pho/src/sys/i386/i386/trap.c", line=856) at /mnt/pho/src/sys/kern/kern_mutex.c:220 #21 0xc0cd62fa in trap_pfault (frame=0xdf3fb304, usermode=0, eva=1853318594) at /mnt/pho/src/sys/i386/i386/trap.c:856 #22 0xc0cd6a05 in trap (frame=0xdf3fb304) at /mnt/pho/src/sys/i386/i386/trap.c:555 #23 0xc0cbf8bc in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #24 0xc09f6e3f in witness_unlock (lock=0xc1018c44, flags=0, file=0xc0e92018 "/mnt/pho/src/sys/kern/subr_prf.c", line=219) at /mnt/pho/src/sys/kern/subr_witness.c:1499 #25 0xc09b43a0 in _sx_sunlock (sx=0xc1018c44, file=0xc0e92018 "/mnt/pho/src/sys/kern/subr_prf.c", line=219) at /mnt/pho/src/sys/kern/kern_sx.c:338 #26 0xc09e57dd in tprintf (p=0xc5fe3000, pri=6, fmt=0xc0e7d27d "newnfs server %s: %s\n") at /mnt/pho/src/sys/kern/subr_prf.c:219 #27 0xc08d3657 in nfs_msg (td=Variable "td" is not available. ) at /mnt/pho/src/sys/fs/nfs/nfs_commonkrpc.c:934 #28 0xc08d37c3 in nfs_feedback (type=4, proc=4, arg=0xdf3fb520) at /mnt/pho/src/sys/fs/nfs/nfs_commonkrpc.c:961 #29 0xc0b88547 in clnt_reconnect_call (cl=0xc5a5d600, ext=0xdf3fb4f4, proc=4, args=0xc5b94000, resultsp=0xdf3fb5cc, utimeout={tv_sec = 300, tv_usec = 0}) at /mnt/pho/src/sys/rpc/clnt_rc.c:322 #30 0xc08d4b3a in newnfs_request (nd=0xdf3fb5cc, nmp=0xc5a9b800, clp=0x0, nrp=0xc5a9b910, vp=0xc5925ae0, td=0xc5feab80, cred=0xc5eda600, prog=100003, vers=3, retsum=0x0, toplevel=1, xidp=0x0) at /mnt/pho/src/sys/fs/nfs/nfs_commonkrpc.c:605 #31 0xc090a842 in nfscl_request (nd=0xdf3fb5cc, vp=0xc5925ae0, p=0xc5feab80, cred=0xc5eda600, stuff=0x0) at /mnt/pho/src/sys/fs/nfsclient/nfs_clport.c:829 #32 0xc08eea13 in nfsrpc_accessrpc (vp=0xc5925ae0, mode=63, cred=0xc5eda600, p=0xc5feab80, nap=0xdf3fb6a0, attrflagp=0xdf3fb728, rmodep=0xdf3fb724, stuff=0x0) at /mnt/pho/src/sys/fs/nfsclient/nfs_clrpcops.c:177 #33 0xc0900f40 in nfs34_access_otw (vp=0xc5925ae0, wmode=Variable "wmode" is not available. ) at /mnt/pho/src/sys/fs/nfsclient/nfs_clvnops.c:276 #34 0xc09052e8 in nfs_access (ap=0xdf3fb974) at /mnt/pho/src/sys/fs/nfsclient/nfs_clvnops.c:405 #35 0xc0cf7c15 in VOP_ACCESS_APV (vop=0xc0f9a1e0, a=0xdf3fb974) at vnode_if.c:571 #36 0xc090562c in nfs_lookup (ap=0xdf3fba20) at vnode_if.h:254 #37 0xc0cf9d85 in VOP_LOOKUP_APV (vop=0xc0f9a1e0, a=0xdf3fba20) at vnode_if.c:123 #38 0xc0a3985b in lookup (ndp=0xdf3fbb2c) at vnode_if.h:54 #39 0xc0a3a6c2 in namei (ndp=0xdf3fbb2c) at /mnt/pho/src/sys/kern/vfs_lookup.c:268 #40 0xc0a4b9d2 in kern_statat_vnhook (td=0xc5feab80, flag=0, fd=-100, path=0x804ef73
, pathseg=UIO_USERSPACE, sbp=0xdf3fbbe8, hook=0) at /mnt/pho/src/sys/kern/vfs_syscalls.c:2352 #41 0xc0a4bb2c in kern_statat (td=0xc5feab80, flag=0, fd=-100, path=0x804ef73
, pathseg=UIO_USERSPACE, sbp=0xdf3fbbe8) at /mnt/pho/src/sys/kern/vfs_syscalls.c:2333 #42 0xc0a4bc76 in kern_stat (td=0xc5feab80, path=0x804ef73
, pathseg=UIO_USERSPACE, sbp=0xdf3fbbe8) at /mnt/pho/src/sys/kern/vfs_syscalls.c:2325 #43 0xc0a4bd1f in stat (td=0xc5feab80, uap=0xdf3fbcec) at /mnt/pho/src/sys/kern/vfs_syscalls.c:2294 #44 0xc09ef113 in syscallenter (td=0xc5feab80, sa=0xdf3fbce4) at /mnt/pho/src/sys/kern/subr_trap.c:344 #45 0xc0cd645f in syscall (frame=0xdf3fbd28) at /mnt/pho/src/sys/i386/i386/trap.c:1082 #46 0xc0cbf921 in Xint0x80_syscall () at /mnt/pho/src/sys/i386/i386/exception.s:266 #47 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 14 #14 0xc099c72a in _mtx_lock_sleep (m=0xc5fe3088, tid=3321801600, opts=0, file=0xc0edc3c8 "/mnt/pho/src/sys/i386/i386/trap.c", line=854) at /mnt/pho/src/sys/kern/kern_mutex.c:341 341 KASSERT((m->lock_object.lo_flags & LO_RECURSABLE) != 0, (kgdb) p *m $1 = {lock_object = {lo_name = 0xc0e8c0a9 "process lock", lo_flags = 21168128, lo_data = 0, lo_witness = 0xc5540340}, mtx_lock = 3321801600} (kgdb) -- Test scenario: run.sh -a