GENERIC HEAD 220307M, vmcore.8 KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #0: Mon Apr 4 18:00:04 CEST 2011 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.05-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2084786176 (1988 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: on atapci0 atapci1: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 atapci1: AHCI called from vendor specific driver atapci1: AHCI v1.10 controller with 4 3Gbps ports, PM not supported ata2: on atapci1 ata3: on atapci1 ata4: on atapci1 ata5: on atapci1 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc093ace0(0xc57065c0) 0.011039671 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc093ace0(0xc57c38a0) 0.101067593 s acd0: CDROM at ata0-master UDMA33 ad4: 238475MB at ata2-master UDMA100 SATA 3Gb/s SMP: AP CPU #1 Launched! WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered Root mount waiting for: usbus4 Root mount waiting for: usbus4 Root mount waiting for: usbus4 uhub4: 8 ports with 8 removable, self powered Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface em0 IP-Address 192.168.5.109 Broadcast 192.168.5.255 Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ad4d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ad4d: clean, 114623864 free (1352 frags, 14327814 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:68 inet 192.168.5.109 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:fe8e:d068%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8802 metric 0 mtu 1500 options=219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 4 Apr 16:22:01 ntpdate[1389]: step time server 192.168.5.1 offset -1.357034 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Starting inetd. Mon Apr 4 16:22:02 UTC 2011 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: root Apr 4 16:22:04 leopard3 login: ROOT LOGIN (root) ON ttyu0 FreeBSD 9.0-CURRENT (PHO) #0: Mon Apr 4 18:00:04 CEST 2011 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. leopard3# ~test/keepalive.sh Mon Apr 4 18:22:09 CEST 2011 6:22PM up 23 secs, 1 user, load averages: 2.31, 0.62, 0.23 Apr 4 18:22:17 leopard3 su: test to root on /dev/pts/0 lock order reversal: 1st 0xdef145c0 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 2nd 0xc5e56000 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0dea23e,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0934dcb,c0dedb2b,c5534238,c55380c8,df3887e8,...) at kdb_backtrace+0x2a _witness_debugger(c0dedb2b,c5e56000,c0e1e986,c55380c8,c0e1e607,...) at _witness_debugger+0x25 witness_checkorder(c5e56000,9,c0e1e607,11c,0,...) at witness_checkorder+0x839 _sx_xlock(c5e56000,0,c0e1e607,11c,c5f827b4,...) at _sx_xlock+0x85 ufsdirhash_acquire(def14560,dff5550c,f4,dff5551c,df3888b8,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c5f827b4,df388914,151c,df3888a4,df3888a8,...) at ufsdirhash_add+0x13 ufs_direnter(c6144570,c614e6cc,df388914,df388ba4,0,...) at ufs_direnter+0x749 ufs_makeinode(df388ba4,c0f1b160,df388b00,df388a5c,c0c66045,...) at ufs_makeinode+0x5c5 ufs_create(df388b00,c0e3c90d,0,0,df388b78,...) at ufs_create+0x30 VOP_CREATE_APV(c0f1abe0,df388b00,df388ba4,df388a98,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(df388b78,df388c2c,1ed,0,c5576180,...) at vn_open_cred+0x205 vn_open(df388b78,df388c2c,1ed,c5e03c78,0,...) at vn_open+0x3b kern_openat(c5e8c5c0,ffffff9c,28830140,0,202,...) at kern_openat+0x12f kern_open(c5e8c5c0,28830140,0,201,1ed,...) at kern_open+0x35 open(c5e8c5c0,df388cec,df388d28,c0dec356,0,...) at open+0x30 syscallenter(c5e8c5c0,df388ce4,df388ce4,2,0,...) at syscallenter+0x263 syscall(df388d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, open), eip = 0x28344b73, esp = 0xbfbfe34c, ebp = 0xbfbfec58 --- Apr 4 18:24:05 leopard3 su: test to root on /dev/pts/0 lock order reversal: 1st 0xc61bb058 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 2nd 0xdef1b5e0 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:11486 3rd 0xc61a41b4 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2127 KDB: stack backtrace: db_trace_self_wrapper(c0dea23e,75735f73,632e7262,3231323a,a0d37,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0934dcb,c0dedb44,c5534238,c5537f28,ed92e334,...) at kdb_backtrace+0x2a _witness_debugger(c0dedb44,c61a41b4,c0ddd004,c5537f28,c0df590b,...) at _witness_debugger+0x25 witness_checkorder(c61a41b4,9,c0df590b,84f,0,...) at witness_checkorder+0x839 __lockmgr_args(c61a41b4,80100,c61a4220,0,0,...) at __lockmgr_args+0x814 ffs_lock(ed92e458,c094592b,c0df4c26,80100,c61a415c,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c0f1abe0,ed92e458,c604e670,c0f35c00,c61a415c,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c61a415c,80100,c0df590b,84f,4,...) at _vn_lock+0x78 vget(c61a415c,80100,c604e5c0,50,0,...) at vget+0xbb vfs_hash_get(c5a98b50,5,80000,c604e5c0,ed92e5ac,...) at vfs_hash_get+0xed ffs_vgetf(c5a98b50,5,80000,ed92e5ac,1,...) at ffs_vgetf+0x49 softdep_sync_metadata(c61bb000,0,c0e1e0f5,144,0,...) at softdep_sync_metadata+0xca2 ffs_syncvnode(c61bb000,1,c5534238,c10b4a48,ed92e640,...) at ffs_syncvnode+0x3e2 ffs_truncate(c61bb000,400,0,880,c5a99480,...) at ffs_truncate+0x8ae ufs_direnter(c61bb000,c61a415c,ed92e914,ed92eba4,0,...) at ufs_direnter+0x924 ufs_makeinode(ed92eba4,c0f1b160,ed92eb00,ed92ea5c,c0c66045,...) at ufs_makeinode+0x5c5 ufs_create(ed92eb00,c0e3c90d,0,0,ed92eb78,...) at ufs_create+0x30 VOP_CREATE_APV(c0f1abe0,ed92eb00,ed92eba4,ed92ea98,0,...) at VOP_CREATE_APV+0xc5 vn_open_cred(ed92eb78,ed92ec2c,1a4,0,c5a99480,...) at vn_open_cred+0x205 vn_open(ed92eb78,ed92ec2c,1a4,c5e02230,0,...) at vn_open+0x3b kern_openat(c604e5c0,ffffff9c,8048be7,0,201,...) at kern_openat+0x12f kern_open(c604e5c0,8048be7,0,200,1a4,...) at kern_open+0x35 open(c604e5c0,ed92ecec,28172b6c,1,0,...) at open+0x30 syscallenter(c604e5c0,ed92ece4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(ed92ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (5, FreeBSD ELF32, open), eip = 0x28172b73, esp = 0xbfbfe9bc, ebp = 0xbfbfe9d8 --- Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex Softdep Lock (Softdep Lock) r = 0 (0xc10edc9c) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1874 KDB: stack backtrace: db_trace_self_wrapper(c0dea23e,3cc,1,c55317f8,c5537f28,...) at db_trace_self_wrapper+0x26 kdb_backtrace(752,2,ffffffff,c10b46dc,df354864,...) at kdb_backtrace+0x2a _witness_debugger(c0ded0d7,df354878,4,1,0,...) at _witness_debugger+0x25 witness_warn(5,0,c0e36246,c09467b6,c5e89af0,...) at witness_warn+0x1fe trap(df354904) at trap+0x195 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc0b4414e, esp = 0xdf354944, ebp = 0xdf354964 --- softdep_revert_mkdir(c631cae0,c6a289f8,df3549ec,df354bd0,df1433c0,...) at softdep_revert_mkdir+0x4e ufs_mkdir(df354bf8,c0e3c747,0,0,df354b3c,...) at ufs_mkdir+0x9b9 VOP_MKDIR_APV(c0f1abe0,df354bf8,df354bd0,df354b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e8f8a0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e8f8a0,bfbfe92c,0,1e0,df354c7c,...) at kern_mkdir+0x2e mkdir(c5e8f8a0,df354cec,df354d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e8f8a0,df354ce4,df354ce4,0,0,...) at syscallenter+0x263 syscall(df354d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x28 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0b4414e stack pointer = 0x28:0xdf354944 frame pointer = 0x28:0xdf354964 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 1826 (suj10) [ thread pid 1826 tid 100084 ] Stopped at softdep_revert_mkdir+0x4e: movl 0x28(%ebx),%eax db> run pho db:0:pho> bt Tracing pid 1826 tid 100084 td 0xc5e8f8a0 softdep_revert_mkdir(c631cae0,c6a289f8,df3549ec,df354bd0,df1433c0,...) at softdep_revert_mkdir+0x4e ufs_mkdir(df354bf8,c0e3c747,0,0,df354b3c,...) at ufs_mkdir+0x9b9 VOP_MKDIR_APV(c0f1abe0,df354bf8,df354bd0,df354b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e8f8a0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e8f8a0,bfbfe92c,0,1e0,df354c7c,...) at kern_mkdir+0x2e mkdir(c5e8f8a0,df354cec,df354d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e8f8a0,df354ce4,df354ce4,0,0,...) at syscallenter+0x263 syscall(df354d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x4eed00 curthread = 0xc5e8f8a0: pid 1826 "suj10" curpcb = 0xdf354d80 fpcurthread = none idlethread = 0xc55975c0: tid 100004 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x42b0d00 curthread = 0xc55e02e0: pid 12 "swi4: clock" curpcb = 0xc5203d80 fpcurthread = none idlethread = 0xc55978a0: tid 100003 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 1829 (suj10) thread 0xc5e2a8a0 (100072) exclusive lockmgr bufwait (bufwait) r = 0 (0xdf133500) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:1911 exclusive lockmgr ufs (ufs) r = 0 (0xc61a4058) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1827 (suj10) thread 0xc5e8f2e0 (100087) exclusive lockmgr ufs (ufs) r = 0 (0xc63b1058) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1826 (suj10) thread 0xc5e8f8a0 (100084) exclusive sleep mutex Softdep Lock (Softdep Lock) r = 0 (0xc10edc9c) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:1874 exclusive lockmgr ufs (ufs) r = 0 (0xc698bdf0) locked @ /mnt/pho/src/sys/ufs/ufs/ufs_lookup.c:972 exclusive lockmgr ufs (ufs) r = 0 (0xc63319dc) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1825 (suj10) thread 0xc5a4b000 (100077) exclusive lockmgr ufs (ufs) r = 0 (0xc6d85b38) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1521 exclusive lockmgr ufs (ufs) r = 0 (0xc61bcb38) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1824 (suj10) thread 0xc5e2a5c0 (100073) exclusive lockmgr bufwait (bufwait) r = 0 (0xdef6eda0) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 exclusive lockmgr bufwait (bufwait) r = 0 (0xdf1a21e0) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:1911 exclusive lockmgr ufs (ufs) r = 0 (0xc6df6b38) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1521 exclusive lockmgr ufs (ufs) r = 0 (0xc6362724) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1823 (suj10) thread 0xc5e8d000 (100093) exclusive lockmgr bufwait (bufwait) r = 0 (0xdf16bc80) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:1911 exclusive lockmgr ufs (ufs) r = 0 (0xc61ca9dc) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1821 (suj10) thread 0xc604e8a0 (100125) exclusive lockmgr bufwait (bufwait) r = 0 (0xdefa9c20) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:2658 shared sx dirhash (dirhash) r = 0 (0xc5e57c00) locked @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:234 exclusive lockmgr ufs (ufs) r = 0 (0xc6258724) locked @ /mnt/pho/src/sys/kern/vfs_lookup.c:501 Process 1780 (sshd) thread 0xc5e8f5c0 (100086) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc612c08c) locked @ /mnt/pho/src/sys/kern/uipc_sockbuf.c:148 db:0:alllocks> show lockedvnods Locked vnodes 0xc61ca984: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xc61a8110 ref 0 pages 72 lock type ufs: EXCL by thread 0xc5e8d000 (pid 1823) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 70656, on dev md5a 0xc61a4000: tag ufs, type VDIR usecount 2, writecount 0, refcount 21 mountedhere 0 flags () v_object 0xc61a8088 ref 0 pages 68 lock type ufs: EXCL by thread 0xc5e2a8a0 (pid 1829) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 47104, on dev md5a 0xc61bcae0: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xc6253cc0 ref 0 pages 72 lock type ufs: EXCL by thread 0xc5a4b000 (pid 1825) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 94208, on dev md5a 0xc62586cc: tag ufs, type VDIR usecount 2, writecount 0, refcount 21 mountedhere 0 flags () v_object 0xc61a8000 ref 0 pages 68 lock type ufs: EXCL by thread 0xc604e8a0 (pid 1821) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 23552, on dev md5a 0xc63626cc: tag ufs, type VDIR usecount 2, writecount 0, refcount 20 mountedhere 0 flags () v_object 0xc6192dd0 ref 0 pages 68 lock type ufs: EXCL by thread 0xc5e2a5c0 (pid 1824) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 29, on dev md5a 0xc6331984: tag ufs, type VDIR usecount 2, writecount 0, refcount 22 mountedhere 0 flags () v_object 0xc6253990 ref 0 pages 72 lock type ufs: EXCL by thread 0xc5e8f8a0 (pid 1826) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 35, on dev md5a 0xc63b1000: tag ufs, type VDIR usecount 2, writecount 0, refcount 19 mountedhere 0 flags () v_object 0xc63acbb0 ref 0 pages 60 lock type ufs: EXCL by thread 0xc5e8f2e0 (pid 1827) #0 0xc08e4795 at __lockmgr_args+0x5a5 #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0987996 at lookup+0x106 #5 0xc0988d72 at namei+0x592 #6 0xc0999601 at kern_mkdirat+0x71 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 23617, on dev md5a 0xc698bd98: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type ufs: EXCL by thread 0xc5e8f8a0 (pid 1826) #0 0xc08e4dee at __lockmgr_args+0xbfe #1 0xc0b53061 at ffs_lock+0xa1 #2 0xc0c65075 at VOP_LOCK1_APV+0xb5 #3 0xc09a2808 at _vn_lock+0x78 #4 0xc0b5b8a7 at ufs_direnter+0x377 #5 0xc0b65f36 at ufs_mkdir+0x916 #6 0xc0c654f5 at VOP_MKDIR_APV+0xc5 #7 0xc09997b5 at kern_mkdirat+0x225 #8 0xc099987e at kern_mkdir+0x2e #9 0xc09998a9 at mkdir+0x29 #10 0xc093e303 at syscallenter+0x263 #11 0xc0c42ebf at syscall+0x4f #12 0xc0c2c591 at Xint0x80_syscall+0x21 ino 111968, on dev md5a 0xc6d85ae0: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xc5a4b000 (pid 1825) #0 0xc08e4dee at __lockmgr_args+0xbfe #1 0xc0b4d799 at ffs_vgetf+0x189 #2 0xc0b4dbbe at ffs_vget+0x2e #3 0xc0b262d0 at ffs_valloc+0x500 #4 0xc0b656c9 at ufs_mkdir+0xa9 #5 0xc0c654f5 at VOP_MKDIR_APV+0xc5 #6 0xc09997b5 at kern_mkdirat+0x225 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 17957, on dev md5a 0xc6df6ae0: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type ufs: EXCL by thread 0xc5e2a5c0 (pid 1824) #0 0xc08e4dee at __lockmgr_args+0xbfe #1 0xc0b4d799 at ffs_vgetf+0x189 #2 0xc0b4dbbe at ffs_vget+0x2e #3 0xc0b262d0 at ffs_valloc+0x500 #4 0xc0b656c9 at ufs_mkdir+0xa9 #5 0xc0c654f5 at VOP_MKDIR_APV+0xc5 #6 0xc09997b5 at kern_mkdirat+0x225 #7 0xc099987e at kern_mkdir+0x2e #8 0xc09998a9 at mkdir+0x29 #9 0xc093e303 at syscallenter+0x263 #10 0xc0c42ebf at syscall+0x4f #11 0xc0c2c591 at Xint0x80_syscall+0x21 ino 135724, on dev md5a db:0:lockedvnods> show mount 0xc5a97b50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc5a98000 devfs on /dev (devfs) 0xc5f6d5a8 /dev/ad4d on /work (ufs) 0xc5f6d2d4 /dev/md0 on /var (ufs) 0xc5f6d000 /dev/md1 on /tmp (ufs) 0xc5a98b50 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 1830 1820 1820 1001 R+ suj10 1829 1820 1820 1001 RL+ suj10 1828 1820 1820 1001 R+ suj10 1827 1820 1820 1001 R+ suj10 1826 1820 1820 1001 R+ CPU 0 suj10 1825 1820 1820 1001 R+ suj10 1824 1820 1820 1001 R+ suj10 1823 1820 1820 1001 RL+ suj10 1822 1820 1820 1001 R+ suj10 1821 1820 1820 1001 R+ suj10 1820 1819 1820 1001 S+ wait 0xc61a7000 suj10 1819 1796 1796 0 S+ wait 0xc61a5af0 su 1813 0 0 0 DL mdwait 0xc5f70800 [md5] 1796 1792 1796 0 S+ wait 0xc5e892bc sh 1792 1791 1792 0 Ss+ pause 0xc5ec05d0 csh 1791 1790 1791 0 S+ select 0xc5a9a864 script 1790 1786 1790 0 S+ wait 0xc5a41af0 sh 1786 1785 1786 0 S+ pause 0xc5e39b48 csh 1785 1784 1785 1001 S+ wait 0xc5ec0834 su 1784 1783 1784 1001 Ss+ wait 0xc5e8a578 sh 1783 1780 1780 1001 S select 0xc5a9a064 sshd 1780 1666 1780 0 Ss sbwait 0xc612c0b8 sshd 1761 1758 1758 0 S+ nanslp 0xc0f6c304 sleep 1758 1754 1758 0 S+ wait 0xc5e38af0 sh 1754 1753 1754 0 S+ pause 0xc5a41314 csh 1753 1 1753 0 Ss+ wait 0xc603d834 login 1752 1 1752 0 Ss+ ttyin 0xc56e0670 getty 1751 1 1751 0 Ss+ ttyin 0xc56e0870 getty 1750 1 1750 0 Ss+ ttyin 0xc56e0a70 getty 1749 1 1749 0 Ss+ ttyin 0xc56e0c70 getty 1748 1 1748 0 Ss+ ttyin 0xc56e0e70 getty 1747 1 1747 0 Ss+ ttyin 0xc5814070 getty 1746 1 1746 0 Ss+ ttyin 0xc5814270 getty 1745 1 1745 0 Ss+ ttyin 0xc5814470 getty 1707 1 1707 0 Ss select 0xc5a3f3e4 inetd 1677 1 1677 0 Ss nanslp 0xc0f6c304 cron 1666 1 1666 0 Ss select 0xc5a3fce4 sshd 1525 1 1525 0 Ss select 0xc5a3f064 rpc.statd 1452 0 0 0 DL mdwait 0xc5f7e000 [md1] 1397 1 1397 0 Ss select 0xc59aad64 rpcbind 1367 1 1367 0 Ss select 0xc59aaba4 syslogd 1188 1 1188 0 Ss select 0xc5a3fc24 devd 101 0 0 0 DL mdwait 0xc5f7f000 [md0] 21 0 0 0 SL - 0xc10e7ba0 [nfsiod 0] 19 0 0 0 DL sdflush 0xc10edce0 [softdepflush] 18 0 0 0 DL syncer 0xc10e2294 [syncer] 17 0 0 0 DL vlruwt 0xc5a40578 [vnlru] 16 0 0 0 DL psleep 0xc10e1f28 [bufdaemon] 15 0 0 0 DL pgzero 0xc10efb5c [pagezero] 9 0 0 0 DL psleep 0xc10ef748 [vmdaemon] 8 0 0 0 DL psleep 0xc10ef710 [pagedaemon] 7 0 0 0 DL ccb_scan 0xc0f37054 [xpt_thrd] 6 0 0 0 DL waiting_ 0xc10e3d38 [sctp_iterator] 5 0 0 0 DL - 0xc572823c [fdc0] 14 0 0 0 DL (threaded) [usb] 100049 D - 0xc57cbd34 [usbus4] 100048 D - 0xc57cbd04 [usbus4] 100047 D - 0xc57cbcd4 [usbus4] 100046 D - 0xc57cbca4 [usbus4] 100045 D - 0xc57b4dac [usbus3] 100044 D - 0xc57b4d7c [usbus3] 100043 D - 0xc57b4d4c [usbus3] 100042 D - 0xc57b4d1c [usbus3] 100040 D - 0xc57a1dac [usbus2] 100039 D - 0xc57a1d7c [usbus2] 100038 D - 0xc57a1d4c [usbus2] 100037 D - 0xc57a1d1c [usbus2] 100035 D - 0xc5788dac [usbus1] 100034 D - 0xc5788d7c [usbus1] 100033 D - 0xc5788d4c [usbus1] 100032 D - 0xc5788d1c [usbus1] 100030 D - 0xc5776dac [usbus0] 100029 D - 0xc5776d7c [usbus0] 100028 D - 0xc5776d4c [usbus0] 100027 D - 0xc5776d1c [usbus0] 13 0 0 0 DL - 0xc0f6b104 [yarrow] 4 0 0 0 RL [g_down] 3 0 0 0 DL - 0xc0f68d40 [g_up] 2 0 0 0 DL - 0xc0f68d38 [g_event] 12 0 0 0 RL (threaded) [intr] 100053 I [irq1: atkbd0] 100051 I [swi0: uart uart] 100050 I [irq14: ata0] 100041 I [irq16: uhci3] 100036 I [irq18: uhci2] 100031 I [irq19: uhci1+] 100026 I [irq23: uhci0 ehci0] 100023 I [swi6: Giant taskq] 100021 I [swi5: +] 100018 I [swi2: cambio] 100014 I [swi6: task queue] 100008 Run CPU 1 [swi4: clock] 100007 I [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu0] 100003 CanRun [idle: cpu1] 1 0 1 0 SLs wait 0xc5595af0 [init] 10 0 0 0 DL audit_wo 0xc10ed500 [audit] 0 0 0 0 DLs (threaded) [kernel] 100064 D - 0xc5566b40 [mca taskq] 100054 D - 0xc0f6b104 [deadlkres] 100025 D - 0xc5566bc0 [em1 taskq] 100024 D - 0xc5566c00 [em0 taskq] 100022 D - 0xc5566c80 [thread taskq] 100020 D - 0xc5566d00 [ffs_trim taskq] 100019 D - 0xc5566d40 [kqueue taskq] 100017 D - 0xc5566d80 [acpi_task_2] 100016 D - 0xc5566d80 [acpi_task_1] 100015 D - 0xc5566d80 [acpi_task_0] 100009 D - 0xc5567040 [firmware taskq] 100000 D sched 0xc0f68e20 [swapper] db:0:ps> allt Tracing command suj10 pid 1830 tid 100110 td 0xc5f5f2e0 sched_switch(c5f5f2e0,0,207,18c,acc69c10,...) at sched_switch+0x3bc mi_switch(207,0,c0dec30d,d4,c1144480,...) at mi_switch+0x219 ast(df3dbd28) at ast+0x2b6 doreti_ast() at doreti_ast+0x17 Tracing command suj10 pid 1829 tid 100072 td 0xc5e2a8a0 sched_switch(c5e2a8a0,0,104,191,a938f140,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c5e2a8a0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(df1334a0,5c,c0df3223,0,0,...) at sleepq_wait+0x63 _sleep(df1334a0,c554b36c,5c,c0df3223,0,...) at _sleep+0x372 bwait(df1334a0,5c,c0df3223,df1334a0,df323590,...) at bwait+0x6f bufwait(df1334a0,df1334a0,df,df1334a0,e53cbfe0,...) at bufwait+0x48 bufwrite(df1334a0,0,c0e1daac,767,c9e388c0) at bufwrite+0x165 ffs_bufwrite(df1334a0,0,c0e1ad8c,b40,c0e1ad8c,...) at ffs_bufwrite+0x290 softdep_process_journal(1fa1,0,c5e2a8a0,df034c60,c61bb2b8,...) at softdep_process_journal+0x728 jwait(c10edc9c,0,c0e1ad8c,1fa1,0,...) at jwait+0x27 softdep_disk_io_initiation(df034c60,c272dd50,0,df034c60,c61a4000,...) at softdep_disk_io_initiation+0x1858 ffs_geom_strategy(c61bb3b0,df034c60,df3237d8,3,c5e2a8a0,...) at ffs_geom_strategy+0x14f ufs_strategy(df323814,c0e3c608,c5e2a8a0,df034c60,4,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c0f1abe0,df323814,c0df2e7b,c0f35c60,c61a4000,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(c61a40f8,df034c60,3a6,df034cc0,df034c60,...) at bufstrategy+0x61 bufwrite(df034c60,df034cc0,df3238b0,c0b52cc5,df034c60,...) at bufwrite+0x159 bawrite(df034c60,0,c0e1e0f5,ff,0,...) at bawrite+0x5c ffs_syncvnode(c61a4000,1,c09960df,c0f1abe0,c61a40f8,...) at ffs_syncvnode+0x2a5 ffs_fsync(df323940,c0e3be9c,c601be80,df323940,c61a4000,...) at ffs_fsync+0x27 VOP_FSYNC_APV(c0f1abe0,df323940,40000,0,158f9,...) at VOP_FSYNC_APV+0xc5 ufs_direnter(c61a4000,c698bc3c,df3239ec,df323bd0,df0a3600,...) at ufs_direnter+0x351 ufs_mkdir(df323bf8,c0e3c747,0,0,df323b3c,...) at ufs_mkdir+0x916 VOP_MKDIR_APV(c0f1abe0,df323bf8,df323bd0,df323b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e2a8a0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e2a8a0,bfbfe92c,0,1e0,df323c7c,...) at kern_mkdir+0x2e mkdir(c5e2a8a0,df323cec,df323d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e2a8a0,df323ce4,df323ce4,0,c0f78d80,...) at syscallenter+0x263 syscall(df323d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1828 tid 100117 td 0xc60512e0 sched_switch(c60512e0,0,207,18c,ac441910,...) at sched_switch+0x3bc mi_switch(207,0,c0dec30d,d4,c1144480,...) at mi_switch+0x219 ast(df3f0d28) at ast+0x2b6 doreti_ast() at doreti_ast+0x17 Tracing command suj10 pid 1827 tid 100087 td 0xc5e8f2e0 sched_switch(c5e8f2e0,0,103,18c,ad679a80,...) at sched_switch+0x3bc mi_switch(103,0,c0dec428,2e1,4c8,...) at mi_switch+0x219 turnstile_wait(c5550a80,c5e2a5c0,0,1ac,c61bb3b0,...) at turnstile_wait+0x4a4 _mtx_lock_sleep(c61bb3b0,c5e8f2e0,0,c0df2e7b,a54,...) at _mtx_lock_sleep+0x14d _mtx_lock_flags(c61bb3b0,0,c0df2e7b,a54,60,...) at _mtx_lock_flags+0xf7 getblk(c61bb2b8,c0,0,4000,0,...) at getblk+0xe3 breadn(c61bb2b8,c0,0,4000,0,...) at breadn+0x44 bread(c61bb2b8,c0,0,4000,0,...) at bread+0x4c ffs_nodealloccg(c636ce0c,0,0,0,41e0,...) at ffs_nodealloccg+0x113 ffs_hashalloc(0,0,41e0,0,c0b265c0,...) at ffs_hashalloc+0x85 ffs_valloc(c63b1000,41e0,c5a99480,df3609e8,c5a99480,...) at ffs_valloc+0x4d3 ufs_mkdir(df360bf8,c0e3c747,0,0,df360b3c,...) at ufs_mkdir+0xa9 VOP_MKDIR_APV(c0f1abe0,df360bf8,df360bd0,df360b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e8f2e0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e8f2e0,bfbfe92c,0,1e0,df360c7c,...) at kern_mkdir+0x2e mkdir(c5e8f2e0,df360cec,df360d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e8f2e0,df360ce4,df360ce4,0,0,...) at syscallenter+0x263 syscall(df360d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1826 tid 100084 td 0xc5e8f8a0 softdep_revert_mkdir(c631cae0,c6a289f8,df3549ec,df354bd0,df1433c0,...) at softdep_revert_mkdir+0x4e ufs_mkdir(df354bf8,c0e3c747,0,0,df354b3c,...) at ufs_mkdir+0x9b9 VOP_MKDIR_APV(c0f1abe0,df354bf8,df354bd0,df354b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e8f8a0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e8f8a0,bfbfe92c,0,1e0,df354c7c,...) at kern_mkdir+0x2e mkdir(c5e8f8a0,df354cec,df354d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e8f8a0,df354ce4,df354ce4,0,0,...) at syscallenter+0x263 syscall(df354d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1825 tid 100077 td 0xc5a4b000 sched_switch(c5a4b000,0,103,18c,ad67aff8,...) at sched_switch+0x3bc mi_switch(103,0,c0dec428,2e1,4c8,...) at mi_switch+0x219 turnstile_wait(c5550a80,c5e2a5c0,0,1ac,c61bb3b0,...) at turnstile_wait+0x4a4 _mtx_lock_sleep(c61bb3b0,c5a4b000,0,c0df2e7b,a54,...) at _mtx_lock_sleep+0x14d _mtx_lock_flags(c61bb3b0,0,c0df2e7b,a54,60,...) at _mtx_lock_flags+0xf7 getblk(c61bb2b8,c0,0,4000,0,...) at getblk+0xe3 breadn(c61bb2b8,c0,0,4000,0,...) at breadn+0x44 bread(c61bb2b8,c0,0,4000,0,...) at bread+0x4c ffs_alloccg(c6db6074,0,8,0,800,...) at ffs_alloccg+0x123 ffs_hashalloc(8,0,800,800,c0b276a0,...) at ffs_hashalloc+0x85 ffs_alloc(c6db6074,0,0,8,0,...) at ffs_alloc+0x2f2 ffs_balloc_ufs2(c6d85ae0,0,0,200,c5a99480,...) at ffs_balloc_ufs2+0x110c ufs_mkdir(df337bf8,c0e3c747,0,0,df337b3c,...) at ufs_mkdir+0x6da VOP_MKDIR_APV(c0f1abe0,df337bf8,df337bd0,df337b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5a4b000,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5a4b000,bfbfe92c,0,1e0,df337c7c,...) at kern_mkdir+0x2e mkdir(c5a4b000,df337cec,df337d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5a4b000,df337ce4,df337ce4,0,0,...) at syscallenter+0x263 syscall(df337d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1824 tid 100073 td 0xc5e2a5c0 sched_switch(c5e2a5c0,0,602,18c,c71e3ad8,...) at sched_switch+0x3bc mi_switch(602,0,c0de6aff,cd,c5e2a5c0,...) at mi_switch+0x219 critical_exit(c1144480,0,1,c10edc9c,0,...) at critical_exit+0xa8 lapic_handle_timer(df3277e4) at lapic_handle_timer+0x95 Xtimerint() at Xtimerint+0x20 --- interrupt, eip = 0xc08ece6c, esp = 0xdf327824, ebp = 0xdf327840 --- _mtx_lock_sleep(c10edc9c,c5e2a5c0,0,c0e1ad8c,445,...) at _mtx_lock_sleep+0x9c _mtx_lock_flags(c10edc9c,0,c0e1ad8c,445,c5a98b50,...) at _mtx_lock_flags+0xf7 workitem_alloc(18,c0f18b68,2,c6288400,e,...) at workitem_alloc+0x39 softdep_setup_directory_add(def6ed40,c631a15c,3ef38,0,2122c,...) at softdep_setup_directory_add+0x1d3 ufs_direnter(c63626cc,c6df6ae0,df3279ec,df327bd0,df1a2180,...) at ufs_direnter+0x7db ufs_mkdir(df327bf8,c0e3c747,0,0,df327b3c,...) at ufs_mkdir+0x916 VOP_MKDIR_APV(c0f1abe0,df327bf8,df327bd0,df327b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e2a5c0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e2a5c0,bfbfe92c,0,1e0,df327c7c,...) at kern_mkdir+0x2e mkdir(c5e2a5c0,df327cec,df327d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e2a5c0,df327ce4,df327ce4,0,c0f78d80,...) at syscallenter+0x263 syscall(df327d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1823 tid 100093 td 0xc5e8d000 sched_switch(c5e8d000,0,103,18c,ad677e20,...) at sched_switch+0x3bc mi_switch(103,0,c0dec428,2e1,4c8,...) at mi_switch+0x219 turnstile_wait(c5550a80,c5e2a5c0,0,1ac,c61bb3b0,...) at turnstile_wait+0x4a4 _mtx_lock_sleep(c61bb3b0,c5e8d000,0,c0df590b,661,...) at _mtx_lock_sleep+0x14d _mtx_lock_flags(c61bb3b0,0,c0df590b,661,c61bb2b8,...) at _mtx_lock_flags+0xf7 brelvp(df16bc20,0,c09785b5,c554b150,0,...) at brelvp+0x54 brelse(df16bc20,df16bc20,df,df16bc20,e5c7bfe0,...) at brelse+0x186 bufwrite(df16bc20,0,c0e1daac,767,c8ff2740) at bufwrite+0x16f ffs_bufwrite(df16bc20,0,c0e1ad8c,b40,c0e1ad8c,...) at ffs_bufwrite+0x290 softdep_process_journal(1fa1,0,c5e8d000,df0f8960,c61bb2b8,...) at softdep_process_journal+0x728 jwait(c10edc9c,0,c0e1ad8c,1fa1,0,...) at jwait+0x27 softdep_disk_io_initiation(df0f8960,c2646e88,0,df0f8960,c61ca984,...) at softdep_disk_io_initiation+0x1858 ffs_geom_strategy(c61bb3b0,df0f8960,df3737d8,3,c5e8d000,...) at ffs_geom_strategy+0x14f ufs_strategy(df373814,c0e3c608,c5e8d000,df0f8960,4,...) at ufs_strategy+0xd3 VOP_STRATEGY_APV(c0f1abe0,df373814,c0df2e7b,c0f35c60,c61ca984,...) at VOP_STRATEGY_APV+0xe5 bufstrategy(c61caa7c,df0f8960,3a6,df0f89c0,df0f8960,...) at bufstrategy+0x61 bufwrite(df0f8960,df0f89c0,df3738b0,c0b52cc5,df0f8960,...) at bufwrite+0x159 bawrite(df0f8960,0,c0e1e0f5,ff,0,...) at bawrite+0x5c ffs_syncvnode(c61ca984,1,c09960df,c0f1abe0,c61caa7c,...) at ffs_syncvnode+0x2a5 ffs_fsync(df373940,c0e3be9c,c5f8232c,df373940,c61ca984,...) at ffs_fsync+0x27 VOP_FSYNC_APV(c0f1abe0,df373940,44000,0,1b5c4,...) at VOP_FSYNC_APV+0xc5 ufs_direnter(c61ca984,c6c55570,df3739ec,df373bd0,df1928e0,...) at ufs_direnter+0x351 ufs_mkdir(df373bf8,c0e3c747,0,0,df373b3c,...) at ufs_mkdir+0x916 VOP_MKDIR_APV(c0f1abe0,df373bf8,df373bd0,df373b3c,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c5e8d000,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x225 kern_mkdir(c5e8d000,bfbfe92c,0,1e0,df373c7c,...) at kern_mkdir+0x2e mkdir(c5e8d000,df373cec,df373d28,c0dec356,0,...) at mkdir+0x29 syscallenter(c5e8d000,df373ce4,df373ce4,0,c0f79400,...) at syscallenter+0x263 syscall(df373d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1822 tid 100081 td 0xc58058a0 sched_switch(c58058a0,0,207,18c,abbc4cb8,...) at sched_switch+0x3bc mi_switch(207,0,c0dec30d,d4,c1144480,...) at mi_switch+0x219 ast(df347d28) at ast+0x2b6 doreti_ast() at doreti_ast+0x17 Tracing command suj10 pid 1821 tid 100125 td 0xc604e8a0 sched_switch(c604e8a0,0,103,18c,ad66f5e0,...) at sched_switch+0x3bc mi_switch(103,0,c0dec428,2e1,2e8,...) at mi_switch+0x219 turnstile_wait(c5551400,c5e2a5c0,0,1ac,c10e1f74,...) at turnstile_wait+0x4a4 _mtx_lock_sleep(c10e1f74,c604e8a0,0,c0df2e7b,58b,...) at _mtx_lock_sleep+0x14d _mtx_lock_flags(c10e1f74,0,c0df2e7b,58b,10,...) at _mtx_lock_flags+0xf7 brelse(defa9bc0,42000,0,ed92a904,ed92a900,...) at brelse+0x50f ufsdirhash_findfree(c62bb740,18,ed92a990,c604e8a0,ed92a944,...) at ufsdirhash_findfree+0x1a1 ufs_lookup_ino(c62586cc,ed92abbc,ed92abd0,0,ed92a9dc,...) at ufs_lookup_ino+0x1e2 ufs_lookup(ed92a9fc,c0e3c104,c62586cc,c62586cc,ed92abd0,...) at ufs_lookup+0x2a VOP_CACHEDLOOKUP_APV(c0f1abe0,ed92a9fc,ed92abd0,ed92abbc,c5a99480,...) at VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(ed92aa90,c0df6684,ed92aba4,200000,ed92aba4,...) at vfs_cache_lookup+0xd6 VOP_LOOKUP_APV(c0f1abe0,ed92aa90,c0df4d1d,2a6,ed92abbc,...) at VOP_LOOKUP_APV+0xe5 lookup(ed92aba4,c0df4d1d,ee,c9,ffffff9c,...) at lookup+0x67b namei(ed92aba4,ffffffdf,ed92abd0,ed92ab3c,0,...) at namei+0x592 kern_mkdirat(c604e8a0,ffffff9c,bfbfe92c,0,1e0,...) at kern_mkdirat+0x71 kern_mkdir(c604e8a0,bfbfe92c,0,1e0,ed92ac7c,...) at kern_mkdir+0x2e mkdir(c604e8a0,ed92acec,ed92ad28,c0dec356,0,...) at mkdir+0x29 syscallenter(c604e8a0,ed92ace4,ed92ace4,0,0,...) at syscallenter+0x263 syscall(ed92ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, mkdir), eip = 0x28172b33, esp = 0xbfbfe90c, ebp = 0xbfbfe9b8 --- Tracing command suj10 pid 1820 tid 100126 td 0xc604e5c0 sched_switch(c604e5c0,0,104,191,53a5b620,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c604e5c0,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c61a7000,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c61a7000,c61a7088,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c604e5c0,ffffffff,ed92ec44,0,0,...) at kern_wait+0xbe6 wait4(c604e5c0,ed92ecec,28195b08,1,0,...) at wait4+0x3b syscallenter(c604e5c0,ed92ece4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(ed92ed28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f0ffb, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command su pid 1819 tid 100127 td 0xc604e2e0 sched_switch(c604e2e0,0,104,191,3034bd80,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c604e2e0,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c61a5af0,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c61a5af0,c61a5b78,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c604e2e0,71c,ed932c44,2,0,...) at kern_wait+0xbe6 wait4(c604e2e0,ed932cec,281c8b08,1,0,...) at wait4+0x3b syscallenter(c604e2e0,ed932ce4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(ed932d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28123ffb, esp = 0xbfbfe47c, ebp = 0xbfbfe498 --- Tracing command md5 pid 1813 tid 100134 td 0xc61bf2e0 sched_switch(c61bf2e0,0,104,191,ab9a1c58,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c61bf2e0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5f70800,5c,c0d9c102,0,0,...) at sleepq_wait+0x63 _sleep(c5f70800,c5f70820,25c,c0d9c102,0,...) at _sleep+0x372 md_kthread(c5f70800,ed94ed28,c0de0b08,390,c603e578,...) at md_kthread+0x11a fork_exit(c070c750,c5f70800,ed94ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xed94ed60, ebp = 0 --- Tracing command sh pid 1796 tid 100088 td 0xc5e8f000 sched_switch(c5e8f000,0,104,191,2e1a2450,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5e8f000,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e892bc,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5e892bc,c5e89344,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5e8f000,ffffffff,df364c44,2,0,...) at kern_wait+0xbe6 wait4(c5e8f000,df364cec,df364d28,c0dec356,0,...) at wait4+0x3b syscallenter(c5e8f000,df364ce4,df364ce4,0,0,...) at syscallenter+0x263 syscall(df364d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28161ffb, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command csh pid 1792 tid 100100 td 0xc5f60b80 sched_switch(c5f60b80,0,104,191,adc27970,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,74,...) at mi_switch+0x219 sleepq_switch(c5f60b80,0,c0deba40,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5ec05d0,74,c0c8fe0e,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5ec05d0,c5ec0600,174,c0c8fe0e,0,...) at _sleep+0x35c kern_sigsuspend(c5f60b80,0,0,0,0,...) at kern_sigsuspend+0xba sigsuspend(c5f60b80,df394cec,df394d28,c0dec356,0,...) at sigsuspend+0x4d syscallenter(c5f60b80,df394ce4,df394ce4,0,0,...) at syscallenter+0x263 syscall(df394d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817d8fb, esp = 0xbfbfe63c, ebp = 0xbfbfe6a8 --- Tracing command script pid 1791 tid 100091 td 0xc5e8d5c0 sched_switch(c5e8d5c0,0,104,191,2ca37570,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e8d5c0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(df36da14,c08eccaa,c5a9a850,0,c5e8d5c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c5a9a864,0,df36da4c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5a9a864,c5a9a850,7531,627,c5e02bd0,...) at _cv_timedwait_sig+0x252 seltdwait(df36dbf8,df36dc00,c5576480,c5e8d5c0,c0ddc24a,...) at seltdwait+0x8a kern_select(c5e8d5c0,5,bfbfe2c4,0,0,df36dc40,20,1e,0) at kern_select+0x504 select(c5e8d5c0,df36dcec,df36dd28,c0dec356,0,...) at select+0x66 syscallenter(c5e8d5c0,df36dce4,df36dce4,0,0,...) at syscallenter+0x263 syscall(df36dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28193d43, esp = 0xbfbfe28c, ebp = 0xbfbfebc8 --- Tracing command sh pid 1790 tid 100082 td 0xc5e27b80 sched_switch(c5e27b80,0,104,191,db9b2988,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5e27b80,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a41af0,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5a41af0,c5a41b78,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5e27b80,ffffffff,df34bc44,2,0,...) at kern_wait+0xbe6 wait4(c5e27b80,df34bcec,28206b08,1,0,...) at wait4+0x3b syscallenter(c5e27b80,df34bce4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(df34bd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28161ffb, esp = 0xbfbfe98c, ebp = 0xbfbfe9a8 --- Tracing command csh pid 1786 tid 100068 td 0xc5a4c5c0 sched_switch(c5a4c5c0,0,104,191,416c6878,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,74,...) at mi_switch+0x219 sleepq_switch(c5a4c5c0,0,c0deba40,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e39b48,74,c0c8fe0e,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5e39b48,c5e39b78,174,c0c8fe0e,0,...) at _sleep+0x35c kern_sigsuspend(c5a4c5c0,0,0,0,0,...) at kern_sigsuspend+0xba sigsuspend(c5a4c5c0,df313cec,df313d28,c0dec356,0,...) at sigsuspend+0x4d syscallenter(c5a4c5c0,df313ce4,df313ce4,0,0,...) at syscallenter+0x263 syscall(df313d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817d8fb, esp = 0xbfbfe72c, ebp = 0xbfbfe798 --- Tracing command su pid 1785 tid 100099 td 0xc5f62000 sched_switch(c5f62000,0,104,191,554de110,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5f62000,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5ec0834,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5ec0834,c5ec08bc,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5f62000,6fa,df391c44,2,0,...) at kern_wait+0xbe6 wait4(c5f62000,df391cec,281d1b08,1,0,...) at wait4+0x3b syscallenter(c5f62000,df391ce4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(df391d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2812cffb, esp = 0xbfbfe76c, ebp = 0xbfbfe788 --- Tracing command sh pid 1784 tid 100105 td 0xc5f5fb80 sched_switch(c5f5fb80,0,104,191,522c50e0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5f5fb80,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e8a578,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5e8a578,c5e8a600,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5f5fb80,ffffffff,df3a3c44,2,0,...) at kern_wait+0xbe6 wait4(c5f5fb80,df3a3cec,28206b08,1,0,...) at wait4+0x3b syscallenter(c5f5fb80,df3a3ce4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(df3a3d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28161ffb, esp = 0xbfbfeaac, ebp = 0xbfbfeac8 --- Tracing command sshd pid 1783 tid 100090 td 0xc5e8d8a0 sched_switch(c5e8d8a0,0,104,191,2ca741e8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e8d8a0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c08eccaa,c5a9a050,0,c0de4827,c5e8d8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a9a064,0,df36aa4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a9a064,c5a9a050,c0dee05b,627,c5e03bd0,...) at _cv_wait_sig+0x243 seltdwait(c5e03bd0,58,c5576880,c5e8d8a0,246,...) at seltdwait+0xa2 kern_select(c5e8d8a0,a,288060b8,288060cc,0,0,20,c5e88af0,c5e88af0) at kern_select+0x504 select(c5e8d8a0,df36acec,df36ad28,c0dec356,0,...) at select+0x66 syscallenter(c5e8d8a0,df36ace4,df36ace4,0,c0f79400,...) at syscallenter+0x263 syscall(df36ad28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284b0d43, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 1780 tid 100086 td 0xc5e8f5c0 sched_switch(c5e8f5c0,0,104,191,8c267570,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,68,...) at mi_switch+0x219 sleepq_switch(c5e8f5c0,0,c0deba40,1a7,68,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c612c0b8,68,c0df1f51,100,0,...) at sleepq_wait_sig+0x17 _sleep(c612c0b8,c612c078,168,c0df1f51,0) at _sleep+0x35c sbwait(c612c054,4,c0df2029,5ec,c612c078,...) at sbwait+0x76 soreceive_generic(c612c000,0,df35cc28,0,0,...) at soreceive_generic+0x3f0 soreceive(c612c000,0,df35cc28,0,0,0) at soreceive+0x38 soo_read(c5e03af0,df35cc28,c5576600,0,c5e8f5c0,...) at soo_read+0x4e dofileread(df35cc28,ffffffff,ffffffff,0,c5e03af0,...) at dofileread+0x9e kern_readv(c5e8f5c0,5,df35cc28,df35cc48,1,...) at kern_readv+0x58 read(c5e8f5c0,df35ccec,df35cd28,c0dec356,0,...) at read+0x4f syscallenter(c5e8f5c0,df35cce4,df35cce4,0,0,...) at syscallenter+0x263 syscall(df35cd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x284b0dc3, esp = 0xbfbfdd6c, ebp = 0xbfbfdda8 --- Tracing command sleep pid 1761 tid 100109 td 0xc5f5f5c0 sched_switch(c5f5f5c0,0,104,191,1ad85620,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5f5f5c0,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(927c1,c093ace0,c5f5f5c0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c0f6c304,6c,c0de783f,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c0f6c304,0,16c,c0de783f,927c1,...) at _sleep+0x328 kern_nanosleep(c5f5f5c0,df3d8c34,df3d8c3c,258,0,...) at kern_nanosleep+0xc1 nanosleep(c5f5f5c0,df3d8cec,df3d8d28,c0dec356,0,...) at nanosleep+0x6f syscallenter(c5f5f5c0,df3d8ce4,df3d8ce4,0,0,...) at syscallenter+0x263 syscall(df3d8d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281660b7, esp = 0xbfbfec2c, ebp = 0xbfbfec78 --- Tracing command sh pid 1758 tid 100075 td 0xc5a4b5c0 sched_switch(c5a4b5c0,0,104,191,1aaa41d8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5a4b5c0,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5e38af0,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5e38af0,c5e38b78,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5a4b5c0,ffffffff,df32fc44,2,0,...) at kern_wait+0xbe6 wait4(c5a4b5c0,df32fcec,df32fd28,c0dec356,0,...) at wait4+0x3b syscallenter(c5a4b5c0,df32fce4,df32fce4,0,0,...) at syscallenter+0x263 syscall(df32fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28161ffb, esp = 0xbfbfe9dc, ebp = 0xbfbfe9f8 --- Tracing command csh pid 1754 tid 100108 td 0xc5e272e0 sched_switch(c5e272e0,0,104,191,168594c8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,74,...) at mi_switch+0x219 sleepq_switch(c5e272e0,0,c0deba40,1a7,74,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a41314,74,c0c8fe0e,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5a41314,c5a41344,174,c0c8fe0e,0,...) at _sleep+0x35c kern_sigsuspend(c5e272e0,0,0,0,0,...) at kern_sigsuspend+0xba sigsuspend(c5e272e0,df3d5cec,df3d5d28,c0dec356,0,...) at sigsuspend+0x4d syscallenter(c5e272e0,df3d5ce4,df3d5ce4,0,0,...) at syscallenter+0x263 syscall(df3d5d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (4, FreeBSD ELF32, write), eip = 0x2817d8fb, esp = 0xbfbfe7ec, ebp = 0xbfbfe858 --- Tracing command login pid 1753 tid 100111 td 0xc5f5f000 sched_switch(c5f5f000,0,104,191,cbac9a98,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5f5f000,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c603d834,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c603d834,c603d8bc,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5f5f000,6da,df3dec44,0,0,...) at kern_wait+0xbe6 wait4(c5f5f000,df3decec,281cab08,1,0,...) at wait4+0x3b syscallenter(c5f5f000,df3dece4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(df3ded28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28125ffb, esp = 0xbfbfecec, ebp = 0xbfbfed08 --- Tracing command getty pid 1752 tid 100067 td 0xc5a4c8a0 sched_switch(c5a4c8a0,0,104,191,a2cdec50,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5a4c8a0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5a4c8a0,c5a4c8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56e0670,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56e0670,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56e0600,c56e0670,df30fc28,1,0,...) at tty_wait+0x71 ttydisc_read(c56e0600,df30fc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721900,df30fc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e03380,df30fc28,c5575e80,0,c5a4c8a0,...) at devfs_read_f+0x7e dofileread(df30fc28,ffffffff,ffffffff,0,c5e03380,...) at dofileread+0x9e kern_readv(c5a4c8a0,0,df30fc28,df30fc48,1,...) at kern_readv+0x58 read(c5a4c8a0,df30fcec,df30fd28,c0dec356,0,...) at read+0x4f syscallenter(c5a4c8a0,df30fce4,df30fce4,0,0,...) at syscallenter+0x263 syscall(df30fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1751 tid 100121 td 0xc604f5c0 sched_switch(c604f5c0,0,104,191,a34a77c8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c604f5c0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c604f5c0,c604f5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56e0870,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56e0870,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56e0800,c56e0870,df3fcc28,1,0,...) at tty_wait+0x71 ttydisc_read(c56e0800,df3fcc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721a00,df3fcc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c60140a8,df3fcc28,c5575e80,0,c604f5c0,...) at devfs_read_f+0x7e dofileread(df3fcc28,ffffffff,ffffffff,0,c60140a8,...) at dofileread+0x9e kern_readv(c604f5c0,0,df3fcc28,df3fcc48,1,...) at kern_readv+0x58 read(c604f5c0,df3fccec,df3fcd28,c0dec356,0,...) at read+0x4f syscallenter(c604f5c0,df3fcce4,df3fcce4,0,0,...) at syscallenter+0x263 syscall(df3fcd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1750 tid 100071 td 0xc5a4bb80 sched_switch(c5a4bb80,0,104,191,a302f128,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5a4bb80,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5a4bb80,c5a4bb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56e0a70,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56e0a70,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56e0a00,c56e0a70,df31fc28,1,0,...) at tty_wait+0x71 ttydisc_read(c56e0a00,df31fc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721b00,df31fc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e18850,df31fc28,c5575e80,0,c5a4bb80,...) at devfs_read_f+0x7e dofileread(df31fc28,ffffffff,ffffffff,0,c5e18850,...) at dofileread+0x9e kern_readv(c5a4bb80,0,df31fc28,df31fc48,1,...) at kern_readv+0x58 read(c5a4bb80,df31fcec,df31fd28,c0dec356,0,...) at read+0x4f syscallenter(c5a4bb80,df31fce4,df31fce4,0,0,...) at syscallenter+0x263 syscall(df31fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1749 tid 100120 td 0xc604f8a0 sched_switch(c604f8a0,0,104,191,a339b338,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c604f8a0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c604f8a0,c604f8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56e0c70,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56e0c70,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56e0c00,c56e0c70,df3f9c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56e0c00,df3f9c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721c00,df3f9c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e185b0,df3f9c28,c5575e80,0,c604f8a0,...) at devfs_read_f+0x7e dofileread(df3f9c28,ffffffff,ffffffff,0,c5e185b0,...) at dofileread+0x9e kern_readv(c604f8a0,0,df3f9c28,df3f9c48,1,...) at kern_readv+0x58 read(c604f8a0,df3f9cec,df3f9d28,c0dec356,0,...) at read+0x4f syscallenter(c604f8a0,df3f9ce4,df3f9ce4,0,0,...) at syscallenter+0x263 syscall(df3f9d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1748 tid 100083 td 0xc5e8fb80 sched_switch(c5e8fb80,0,104,191,a3180b40,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e8fb80,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5e8fb80,c5e8fb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c56e0e70,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c56e0e70,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c56e0e00,c56e0e70,df350c28,1,0,...) at tty_wait+0x71 ttydisc_read(c56e0e00,df350c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721d00,df350c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e189d8,df350c28,c5575e80,0,c5e8fb80,...) at devfs_read_f+0x7e dofileread(df350c28,ffffffff,ffffffff,0,c5e189d8,...) at dofileread+0x9e kern_readv(c5e8fb80,0,df350c28,df350c48,1,...) at kern_readv+0x58 read(c5e8fb80,df350cec,df350d28,c0dec356,0,...) at read+0x4f syscallenter(c5e8fb80,df350ce4,df350ce4,0,0,...) at syscallenter+0x263 syscall(df350d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1747 tid 100116 td 0xc5f5e000 sched_switch(c5f5e000,0,104,191,a349f010,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5f5e000,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5f5e000,c5f5e000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5814070,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5814070,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c5814000,c5814070,df3edc28,1,0,...) at tty_wait+0x71 ttydisc_read(c5814000,df3edc28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5721e00,df3edc28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c6014000,df3edc28,c5575e80,0,c5f5e000,...) at devfs_read_f+0x7e dofileread(df3edc28,ffffffff,ffffffff,0,c6014000,...) at dofileread+0x9e kern_readv(c5f5e000,0,df3edc28,df3edc48,1,...) at kern_readv+0x58 read(c5f5e000,df3edcec,df3edd28,c0dec356,0,...) at read+0x4f syscallenter(c5f5e000,df3edce4,df3edce4,0,0,...) at syscallenter+0x263 syscall(df3edd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1746 tid 100115 td 0xc5f5e2e0 sched_switch(c5f5e2e0,0,104,191,a3320938,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5f5e2e0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5f5e2e0,c5f5e2e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5814270,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5814270,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c5814200,c5814270,df3eac28,1,0,...) at tty_wait+0x71 ttydisc_read(c5814200,df3eac28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5800000,df3eac28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e181c0,df3eac28,c5575e80,0,c5f5e2e0,...) at devfs_read_f+0x7e dofileread(df3eac28,ffffffff,ffffffff,0,c5e181c0,...) at dofileread+0x9e kern_readv(c5f5e2e0,0,df3eac28,df3eac48,1,...) at kern_readv+0x58 read(c5f5e2e0,df3eacec,df3ead28,c0dec356,0,...) at read+0x4f syscallenter(c5f5e2e0,df3eace4,df3eace4,0,0,...) at syscallenter+0x263 syscall(df3ead28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1745 tid 100114 td 0xc5f5e5c0 sched_switch(c5f5e5c0,0,104,191,a2baea28,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5f5e5c0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,c5f5e5c0,c5f5e5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5814470,0,c0df077e,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5814470,c0f6a9f0,c0df00f4,514,0,...) at _cv_wait_sig+0x243 tty_wait(c5814400,c5814470,df3e7c28,1,0,...) at tty_wait+0x71 ttydisc_read(c5814400,df3e7c28,0,9f,0,...) at ttydisc_read+0x22c ttydev_read(c5800100,df3e7c28,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5e18d20,df3e7c28,c5575e80,0,c5f5e5c0,...) at devfs_read_f+0x7e dofileread(df3e7c28,ffffffff,ffffffff,0,c5e18d20,...) at dofileread+0x9e kern_readv(c5f5e5c0,0,df3e7c28,df3e7c48,1,...) at kern_readv+0x58 read(c5f5e5c0,df3e7cec,df3e7d28,c0dec356,0,...) at read+0x4f syscallenter(c5f5e5c0,df3e7ce4,df3e7ce4,0,0,...) at syscallenter+0x263 syscall(df3e7d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 1707 tid 100119 td 0xc604fb80 sched_switch(c604fb80,0,104,191,92ef3890,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c604fb80,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c08eccaa,c5a3f3d0,0,c0de4827,c604fb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3f3e4,0,df3f6a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3f3e4,c5a3f3d0,c0dee05b,627,c5e182d8,...) at _cv_wait_sig+0x243 seltdwait(c5e182d8,58,c5575e80,c604fb80,c09459cc,...) at seltdwait+0xa2 kern_select(c604fb80,8,bfbfdfa0,0,0,0,20,c603b000,c603b000) at kern_select+0x504 select(c604fb80,df3f6cec,df3f6d28,c0dec356,0,...) at select+0x66 syscallenter(c604fb80,df3f6ce4,df3f6ce4,0,0,...) at syscallenter+0x263 syscall(df3f6d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x281a8d43, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command cron pid 1677 tid 100069 td 0xc5a4c2e0 sched_switch(c5a4c2e0,0,104,191,ff932858,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5a4c2e0,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c093ace0,c5a4c2e0,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c0f6c304,6c,c0de783f,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c0f6c304,0,16c,c0de783f,ea61,...) at _sleep+0x328 kern_nanosleep(c5a4c2e0,df317c34,df317c3c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c5a4c2e0,df317cec,df317d28,c0dec356,0,...) at nanosleep+0x6f syscallenter(c5a4c2e0,df317ce4,df317ce4,0,0,...) at syscallenter+0x263 syscall(df317d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x281850b7, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sshd pid 1666 tid 100103 td 0xc5f602e0 sched_switch(c5f602e0,0,104,191,8aeb1bd8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5f602e0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c08eccaa,c5a3fcd0,0,c0de4827,c5f602e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3fce4,0,df39da4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3fce4,c5a3fcd0,c0dee05b,627,c5e029a0,...) at _cv_wait_sig+0x243 seltdwait(c5e029a0,58,c5575500,c5f602e0,efe,...) at seltdwait+0xa2 kern_select(c5f602e0,5,2880c0b0,0,0,0,20,c5e8aaf0,c5e8aaf0) at kern_select+0x504 select(c5f602e0,df39dcec,2880d070,1,0,...) at select+0x66 syscallenter(c5f602e0,df39dce4,c0c4371d,c0f6a9f0,0,...) at syscallenter+0x263 syscall(df39dd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x284b0d43, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command rpc.statd pid 1525 tid 100079 td 0xc5e2a000 sched_switch(c5e2a000,0,104,191,896bf1e0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e2a000,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(df33fa14,c08eccaa,c5a3f050,0,c5e2a000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c5a3f064,0,df33fa4c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5a3f064,c5a3f050,7531,627,c5e02690,...) at _cv_timedwait_sig+0x252 seltdwait(df33fbf8,df33fc00,c5575e80,c5e2a000,c0997fd0,...) at seltdwait+0x8a kern_select(c5e2a000,8,bfbfed14,0,0,df33fc40,20,1e,0) at kern_select+0x504 select(c5e2a000,df33fcec,df33fd28,c0dec356,0,...) at select+0x66 syscallenter(c5e2a000,df33fce4,df33fce4,0,0,...) at syscallenter+0x263 syscall(df33fd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x2818cd43, esp = 0xbfbfec6c, ebp = 0xbfbfeda8 --- Tracing command md1 pid 1452 tid 100094 td 0xc5e8cb80 sched_switch(c5e8cb80,0,104,191,193b2850,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c5e8cb80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5f7e000,5c,c0d9c102,0,0,...) at sleepq_wait+0x63 _sleep(c5f7e000,c5f7e020,25c,c0d9c102,0,...) at _sleep+0x372 md_kthread(c5f7e000,df377d28,c0de0b08,390,c5e88000,...) at md_kthread+0x11a fork_exit(c070c750,c5f7e000,df377d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf377d60, ebp = 0 --- Tracing command rpcbind pid 1397 tid 100107 td 0xc5e275c0 sched_switch(c5e275c0,0,104,191,896d27c8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e275c0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(df3d2a6c,c08eccaa,c59aad50,0,c5e275c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c59aad64,0,df3d2aa4,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c59aad64,c59aad50,7531,627,df3d2b5c,...) at _cv_timedwait_sig+0x252 seltdwait(df3d2c2c,df3d2c34,535,c5e275c0,df3d2b2c,...) at seltdwait+0x8a poll(c5e275c0,df3d2cec,df3d2d28,c0dec356,0,...) at poll+0x300 syscallenter(c5e275c0,df3d2ce4,df3d2ce4,0,c0f78d80,...) at syscallenter+0x263 syscall(df3d2d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813ff4b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1367 tid 100089 td 0xc5e8db80 sched_switch(c5e8db80,0,104,191,30d07808,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e8db80,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c08eccaa,c59aab90,0,c0de4827,c5e8db80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c59aaba4,0,df367a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59aaba4,c59aab90,c0dee05b,627,c5e033b8,...) at _cv_wait_sig+0x243 seltdwait(c5e033b8,58,c5575e80,c5e8db80,0,...) at seltdwait+0xa2 kern_select(c5e8db80,9,284760ac,0,0,0,20,c5e89000,c5e89000) at kern_select+0x504 select(c5e8db80,df367cec,df367d28,c0dec356,0,...) at select+0x66 syscallenter(c5e8db80,df367ce4,df367ce4,0,246,...) at syscallenter+0x263 syscall(df367d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x28199d43, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1188 tid 100095 td 0xc5e8c8a0 sched_switch(c5e8c8a0,0,104,191,2d3e1700,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5e8c8a0,0,c0deba40,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c08eccaa,c5a3fc10,0,c0de4827,c5e8c8a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5a3fc24,0,df385a4c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5a3fc24,c5a3fc10,c0dee05b,627,c5e03ce8,...) at _cv_wait_sig+0x243 seltdwait(c5e03ce8,58,c5575e80,c5e8c8a0,c0e20070,...) at seltdwait+0xa2 kern_select(c5e8c8a0,5,bfbfe960,0,0,0,20,c5ec1578,c5ec1578) at kern_select+0x504 select(c5e8c8a0,df385cec,df385d28,c0dec356,0,...) at select+0x66 syscallenter(c5e8c8a0,df385ce4,df385ce4,0,0,...) at syscallenter+0x263 syscall(df385d28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, select), eip = 0x808ba43, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command md0 pid 101 tid 100098 td 0xc5e8c000 sched_switch(c5e8c000,0,104,191,e569348,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c5e8c000,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5f7f000,5c,c0d9c102,0,0,...) at sleepq_wait+0x63 _sleep(c5f7f000,c5f7f020,25c,c0d9c102,0,...) at _sleep+0x372 md_kthread(c5f7f000,df38ed28,c0de0b08,390,c5ec0af0,...) at md_kthread+0x11a fork_exit(c070c750,c5f7f000,df38ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf38ed60, ebp = 0 --- Tracing command nfsiod 0 pid 21 tid 100066 td 0xc5e2ab80 sched_switch(c5e2ab80,0,104,191,1a6ba8a8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5e2ab80,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(1d4c0,c093ace0,c5e2ab80,0,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c10e7ba0,6c,c0dd969a,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c10e7ba0,c10e7b8c,16c,c0dd969a,1d4c0,...) at _sleep+0x328 nfssvc_iod(c10e7540,df30cd28,c0de0b08,390,c5e25af0,...) at nfssvc_iod+0xd7 fork_exit(c0acbe20,c10e7540,df30cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf30cd60, ebp = 0 --- Tracing command softdepflush pid 19 tid 100063 td 0xc57c52e0 sched_switch(c57c52e0,0,104,191,4896b378,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,54,...) at mi_switch+0x219 sleepq_switch(c57c52e0,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10edce0,54,c0e1cca7,0,0,...) at sleepq_timedwait+0x6b _sleep(c10edce0,c10edc9c,54,c0e1cca7,3e8,...) at _sleep+0x342 softdep_flush(0,df301d28,c0de0b08,390,c5a40000,...) at softdep_flush+0x25a fork_exit(c0b472f0,0,df301d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf301d60, ebp = 0 --- Tracing command syncer pid 18 tid 100062 td 0xc57c55c0 sched_switch(c57c55c0,0,104,191,9630e370,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c55c0,0,c0deba40,28b,c57c55c0,...) at sleepq_switch+0x162 sleepq_timedwait(c10e2294,0,df2fec78,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c10e2294,c10e2280,3e8,763,4e20,...) at _cv_timedwait+0x252 sched_sync(0,df2fed28,c0de0b08,390,c5a402bc,...) at sched_sync+0x585 fork_exit(c0995550,0,df2fed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2fed60, ebp = 0 --- Tracing command vnlru pid 17 tid 100061 td 0xc57c58a0 sched_switch(c57c58a0,0,104,191,84330560,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,60,...) at mi_switch+0x219 sleepq_switch(c57c58a0,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c5a40578,60,c0df6885,0,0,...) at sleepq_timedwait+0x6b _sleep(c5a40578,c10e2254,260,c0df6885,3e8,...) at _sleep+0x342 vnlru_proc(0,df2fbd28,c0de0b08,390,c5a40578,...) at vnlru_proc+0xe7 fork_exit(c09975a0,0,df2fbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2fbd60, ebp = 0 --- Tracing command bufdaemon pid 16 tid 100060 td 0xc57c5b80 sched_switch(c57c5b80,0,104,191,ab6d56c0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,54,...) at mi_switch+0x219 sleepq_switch(c57c5b80,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10e1f28,54,c0df383d,0,0,...) at sleepq_timedwait+0x6b _sleep(c10e1f28,c10e1f30,54,c0df383d,3e8,...) at _sleep+0x342 buf_daemon(0,df2f8d28,c0de0b08,390,c5a40834,...) at buf_daemon+0x175 fork_exit(c097d010,0,df2f8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2f8d60, ebp = 0 --- Tracing command pagezero pid 15 tid 100059 td 0xc5802000 sched_switch(c5802000,0,104,191,40f5ce20,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5802000,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10efb5c,0,c0e23b74,0,0,...) at sleepq_timedwait+0x6b _sleep(c10efb5c,c10ee600,0,c0e23b74,493e0,...) at _sleep+0x342 vm_pagezero(0,df2f5d28,c0de0b08,390,c5a40af0,...) at vm_pagezero+0xdc fork_exit(c0b8d940,0,df2f5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2f5d60, ebp = 0 --- Tracing command vmdaemon pid 9 tid 100058 td 0xc58022e0 sched_switch(c58022e0,0,104,191,40f25408,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,74,...) at mi_switch+0x219 sleepq_switch(c58022e0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10ef748,74,c0df383d,0,0,...) at sleepq_wait+0x63 _sleep(c10ef748,c10ef74c,74,c0df383d,0,...) at _sleep+0x372 vm_daemon(0,df2f2d28,c0de0b08,390,c5a41000,...) at vm_daemon+0x59 fork_exit(c0b87720,0,df2f2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2f2d60, ebp = 0 --- Tracing command pagedaemon pid 8 tid 100057 td 0xc58025c0 sched_switch(c58025c0,0,104,191,870cb330,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,54,...) at mi_switch+0x219 sleepq_switch(c58025c0,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c10ef710,54,c0df383d,0,0,...) at sleepq_timedwait+0x6b _sleep(c10ef710,c10ee600,54,c0df383d,1388,...) at _sleep+0x342 vm_pageout(0,df2efd28,c0de0b08,390,c55962bc,...) at vm_pageout+0x2c7 fork_exit(c0b887f0,0,df2efd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2efd60, ebp = 0 --- Tracing command xpt_thrd pid 7 tid 100056 td 0xc58028a0 sched_switch(c58028a0,0,104,191,27c9ffe0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c58028a0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0f37054,5c,c0c6a06a,0,0,...) at sleepq_wait+0x63 _sleep(c0f37054,c0f37098,5c,c0c6a06a,0,...) at _sleep+0x372 xpt_scanner_thread(0,df2ecd28,c0de0b08,390,c5596578,...) at xpt_scanner_thread+0x47 fork_exit(c0487a00,0,df2ecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2ecd60, ebp = 0 --- Tracing command sctp_iterator pid 6 tid 100055 td 0xc5802b80 sched_switch(c5802b80,0,104,191,b44b1e8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5802b80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10e3d38,0,c0e09a1f,0,0,...) at sleepq_wait+0x63 _sleep(c10e3d38,c10e3d00,0,c0e09a1f,0,...) at _sleep+0x372 sctp_iterator_thread(0,df2e9d28,c0de0b08,390,c5596834,...) at sctp_iterator_thread+0x5c fork_exit(c0a25d60,0,df2e9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2e9d60, ebp = 0 --- Tracing command fdc0 pid 5 tid 100052 td 0xc58055c0 sched_switch(c58055c0,0,104,191,6c47eea8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c58055c0,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c572823c,5c,c0dd969a,0,0,...) at sleepq_timedwait+0x6b _sleep(c572823c,c57282f0,5c,c0dd969a,3e8,...) at _sleep+0x342 fdc_thread(c5728200,df2e0d28,c0de0b08,390,c5596af0,...) at fdc_thread+0x27d fork_exit(c0c0c690,c5728200,df2e0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2e0d60, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc57c35c0 sched_switch(c57c35c0,0,104,191,27c3ebb8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c35c0,0,c0deba40,268,c57c35c0,...) at sleepq_switch+0x162 sleepq_wait(c57cbd34,0,c52f4cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57cbd34,c57cbdd4,c0dc01be,6b,c57cbd3c,...) at _cv_wait+0x243 usb_process(c57cbd2c,c52f4d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57cbd2c,c52f4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52f4d60, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc57c38a0 sched_switch(c57c38a0,0,104,191,37b33468,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c38a0,0,c0deba40,268,c57c38a0,...) at sleepq_switch+0x162 sleepq_wait(c57cbd04,0,c52f1cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57cbd04,c57cbdd4,c0dc01be,6b,c57cbd0c,...) at _cv_wait+0x243 usb_process(c57cbcfc,c52f1d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57cbcfc,c52f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52f1d60, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc57c3b80 sched_switch(c57c3b80,0,104,191,274367b0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c3b80,0,c0deba40,268,c57c3b80,...) at sleepq_switch+0x162 sleepq_wait(c57cbcd4,0,c52eecac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57cbcd4,c57cbdd4,c0dc01be,6b,c57cbcdc,...) at _cv_wait+0x243 usb_process(c57cbccc,c52eed28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57cbccc,c52eed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52eed60, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc57c4000 sched_switch(c57c4000,0,104,191,27435378,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c4000,0,c0deba40,268,c57c4000,...) at sleepq_switch+0x162 sleepq_wait(c57cbca4,0,c52ebcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57cbca4,c57cbdd4,c0dc01be,6b,c57cbcac,...) at _cv_wait+0x243 usb_process(c57cbc9c,c52ebd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57cbc9c,c52ebd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52ebd60, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc57c42e0 sched_switch(c57c42e0,0,104,191,27433dc8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c42e0,0,c0deba40,268,c57c42e0,...) at sleepq_switch+0x162 sleepq_wait(c57b4dac,0,c52e5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57b4dac,c57b4e4c,c0dc01be,6b,c57b4db4,...) at _cv_wait+0x243 usb_process(c57b4da4,c52e5d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57b4da4,c52e5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52e5d60, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc57c45c0 sched_switch(c57c45c0,0,104,191,a11a2850,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c45c0,0,c0deba40,268,c57c45c0,...) at sleepq_switch+0x162 sleepq_wait(c57b4d7c,0,c52e2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57b4d7c,c57b4e4c,c0dc01be,6b,c57b4d84,...) at _cv_wait+0x243 usb_process(c57b4d74,c52e2d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57b4d74,c52e2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52e2d60, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc57c48a0 sched_switch(c57c48a0,0,104,191,20204a18,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c48a0,0,c0deba40,268,c57c48a0,...) at sleepq_switch+0x162 sleepq_wait(c57b4d4c,0,c52dfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57b4d4c,c57b4e4c,c0dc01be,6b,c57b4d54,...) at _cv_wait+0x243 usb_process(c57b4d44,c52dfd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57b4d44,c52dfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52dfd60, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc57c4b80 sched_switch(c57c4b80,0,104,191,20203820,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57c4b80,0,c0deba40,268,c57c4b80,...) at sleepq_switch+0x162 sleepq_wait(c57b4d1c,0,c52dccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57b4d1c,c57b4e4c,c0dc01be,6b,c57b4d24,...) at _cv_wait+0x243 usb_process(c57b4d14,c52dcd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57b4d14,c52dcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52dcd60, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc5798000 sched_switch(c5798000,0,104,191,202022f8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5798000,0,c0deba40,268,c5798000,...) at sleepq_switch+0x162 sleepq_wait(c57a1dac,0,c52d5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57a1dac,c57a1e4c,c0dc01be,6b,c57a1db4,...) at _cv_wait+0x243 usb_process(c57a1da4,c52d5d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57a1da4,c52d5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52d5d60, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc57982e0 sched_switch(c57982e0,0,104,191,a1bc7670,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57982e0,0,c0deba40,268,c57982e0,...) at sleepq_switch+0x162 sleepq_wait(c57a1d7c,0,c52d2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57a1d7c,c57a1e4c,c0dc01be,6b,c57a1d84,...) at _cv_wait+0x243 usb_process(c57a1d74,c52d2d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57a1d74,c52d2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52d2d60, ebp = 0 --- Tracing command usb pid 14 tid 100038 td 0xc57985c0 sched_switch(c57985c0,0,104,191,1fa46318,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57985c0,0,c0deba40,268,c57985c0,...) at sleepq_switch+0x162 sleepq_wait(c57a1d4c,0,c52cfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57a1d4c,c57a1e4c,c0dc01be,6b,c57a1d54,...) at _cv_wait+0x243 usb_process(c57a1d44,c52cfd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57a1d44,c52cfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52cfd60, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc57988a0 sched_switch(c57988a0,0,104,191,1fa44fe8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57988a0,0,c0deba40,268,c57988a0,...) at sleepq_switch+0x162 sleepq_wait(c57a1d1c,0,c52cccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c57a1d1c,c57a1e4c,c0dc01be,6b,c57a1d24,...) at _cv_wait+0x243 usb_process(c57a1d14,c52ccd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c57a1d14,c52ccd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52ccd60, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc579b000 sched_switch(c579b000,0,104,191,1fa43bd8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c579b000,0,c0deba40,268,c579b000,...) at sleepq_switch+0x162 sleepq_wait(c5788dac,0,c52c5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5788dac,c5788e4c,c0dc01be,6b,c5788db4,...) at _cv_wait+0x243 usb_process(c5788da4,c52c5d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5788da4,c52c5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52c5d60, ebp = 0 --- Tracing command usb pid 14 tid 100034 td 0xc579b2e0 sched_switch(c579b2e0,0,104,191,9dad7390,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c579b2e0,0,c0deba40,268,c579b2e0,...) at sleepq_switch+0x162 sleepq_wait(c5788d7c,0,c52c2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5788d7c,c5788e4c,c0dc01be,6b,c5788d84,...) at _cv_wait+0x243 usb_process(c5788d74,c52c2d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5788d74,c52c2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52c2d60, ebp = 0 --- Tracing command usb pid 14 tid 100033 td 0xc579b5c0 sched_switch(c579b5c0,0,104,191,1f287598,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c579b5c0,0,c0deba40,268,c579b5c0,...) at sleepq_switch+0x162 sleepq_wait(c5788d4c,0,c52bfcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5788d4c,c5788e4c,c0dc01be,6b,c5788d54,...) at _cv_wait+0x243 usb_process(c5788d44,c52bfd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5788d44,c52bfd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52bfd60, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc579b8a0 sched_switch(c579b8a0,0,104,191,1f2862a0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c579b8a0,0,c0deba40,268,c579b8a0,...) at sleepq_switch+0x162 sleepq_wait(c5788d1c,0,c52bccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5788d1c,c5788e4c,c0dc01be,6b,c5788d24,...) at _cv_wait+0x243 usb_process(c5788d14,c52bcd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5788d14,c52bcd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52bcd60, ebp = 0 --- Tracing command usb pid 14 tid 100030 td 0xc57062e0 sched_switch(c57062e0,0,104,191,1f2848a0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57062e0,0,c0deba40,268,c57062e0,...) at sleepq_switch+0x162 sleepq_wait(c5776dac,0,c52b5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5776dac,c5776e4c,c0dc01be,6b,c5776db4,...) at _cv_wait+0x243 usb_process(c5776da4,c52b5d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5776da4,c52b5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52b5d60, ebp = 0 --- Tracing command usb pid 14 tid 100029 td 0xc57065c0 sched_switch(c57065c0,0,104,191,9b241c38,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57065c0,0,c0deba40,268,c57065c0,...) at sleepq_switch+0x162 sleepq_wait(c5776d7c,0,c52b2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5776d7c,c5776e4c,c0dc01be,6b,c5776d84,...) at _cv_wait+0x243 usb_process(c5776d74,c52b2d28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5776d74,c52b2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52b2d60, ebp = 0 --- Tracing command usb pid 14 tid 100028 td 0xc57068a0 sched_switch(c57068a0,0,104,191,1ea65d48,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57068a0,0,c0deba40,268,c57068a0,...) at sleepq_switch+0x162 sleepq_wait(c5776d4c,0,c52afcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5776d4c,c5776e4c,c0dc01be,6b,c5776d54,...) at _cv_wait+0x243 usb_process(c5776d44,c52afd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5776d44,c52afd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52afd60, ebp = 0 --- Tracing command usb pid 14 tid 100027 td 0xc5706b80 sched_switch(c5706b80,0,104,191,1ea64998,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5706b80,0,c0deba40,268,c5706b80,...) at sleepq_switch+0x162 sleepq_wait(c5776d1c,0,c52accac,1,0,...) at sleepq_wait+0x63 _cv_wait(c5776d1c,c5776e4c,c0dc01be,6b,c5776d24,...) at _cv_wait+0x243 usb_process(c5776d14,c52acd28,c0de0b08,390,c5625000,...) at usb_process+0x193 fork_exit(c081fb50,c5776d14,c52acd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52acd60, ebp = 0 --- Tracing command yarrow pid 13 tid 100013 td 0xc559c2e0 sched_switch(c559c2e0,0,104,191,ba6b90b0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c559c2e0,0,c0deba40,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c0f6b104,0,c0dd969a,2,0,...) at sleepq_timedwait+0x6b _sleep(c0f6b104,0,0,c0dd969a,64,...) at _sleep+0x342 pause(c0dd969a,64,c0daecae,111,0,...) at pause+0x47 random_kthread(0,c5212d28,c0de0b08,390,c56252bc,...) at random_kthread+0x1ef fork_exit(c078f960,0,c5212d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5212d60, ebp = 0 --- Tracing command g_down pid 4 tid 100012 td 0xc559c5c0 sched_switch(c559c5c0,0,104,191,ab995f90,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c559c5c0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0f68d44,5c,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c0f68d44,c0f68ca8,25c,c0dd969a,0,...) at _sleep+0x372 g_io_schedule_down(c559c5c0,0,c0ddb235,74,0,...) at g_io_schedule_down+0x5b g_down_procbody(0,c520fd28,c0de0b08,390,c5625578,...) at g_down_procbody+0x8d fork_exit(c08939f0,0,c520fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc520fd60, ebp = 0 --- Tracing command g_up pid 3 tid 100011 td 0xc559c8a0 sched_switch(c559c8a0,0,104,191,abb8e6e0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c559c8a0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0f68d40,5c,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c0f68d40,c0f68cc8,25c,c0dd969a,0,...) at _sleep+0x372 g_io_schedule_up(c559c8a0,0,c0ddb235,5d,0,...) at g_io_schedule_up+0x134 g_up_procbody(0,c520cd28,c0de0b08,390,c5595000,...) at g_up_procbody+0x8d fork_exit(c0893a80,0,c520cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc520cd60, ebp = 0 --- Tracing command g_event pid 2 tid 100010 td 0xc559cb80 sched_switch(c559cb80,0,104,191,2d3bf1e0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,5c,...) at mi_switch+0x219 sleepq_switch(c559cb80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c0f68d38,5c,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c0f68d38,c0f68c6c,25c,c0dd969a,0,...) at _sleep+0x372 g_run_events(c0f78d80,0,c0ddb235,8b) at g_run_events+0x526 g_event_procbody(0,c5209d28,c0de0b08,390,c55952bc,...) at g_event_procbody+0x8a fork_exit(c0893b10,0,c5209d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5209d60, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc58052e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100051 td 0xc57c3000 sched_switch(c57c3000,0,109,191,562e8a40,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c57c3000,...) at mi_switch+0x219 ithread_loop(c555b700,df2d3d28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b700,df2d3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2d3d60, ebp = 0 --- Tracing command intr pid 12 tid 100050 td 0xc57c32e0 sched_switch(c57c32e0,0,109,191,aac10bb0,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c57c32e0,...) at mi_switch+0x219 ithread_loop(c555b710,c531ad28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b710,c531ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc531ad60, ebp = 0 --- Tracing command intr pid 12 tid 100041 td 0xc57c5000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xc5798b80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xc579bb80 sched_switch(c579bb80,0,109,191,193e7450,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c579bb80,...) at mi_switch+0x219 ithread_loop(c555b740,c52b9d28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b740,c52b9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52b9d60, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc5708000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc57088a0 sched_switch(c57088a0,0,109,191,fde50d68,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c57088a0,...) at mi_switch+0x219 ithread_loop(c555b760,c5234d28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b760,c5234d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5234d60, ebp = 0 --- Tracing command intr pid 12 tid 100021 td 0xc55e08a0 sched_switch(c55e08a0,0,109,191,5446f0c0,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c55e08a0,...) at mi_switch+0x219 ithread_loop(c555b770,c522ed28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b770,c522ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc522ed60, ebp = 0 --- Tracing command intr pid 12 tid 100018 td 0xc57052e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100014 td 0xc5706000 sched_switch(c5706000,0,109,191,54aeb920,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c5706000,...) at mi_switch+0x219 ithread_loop(c555b790,c5219d28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b790,c5219d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5219d60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc55e02e0 cpustop_handler(2,c5203b50,c0c42fd6,a,3,...) at cpustop_handler+0x34 ipi_nmi_handler(a,3,4,2,c5595578,...) at ipi_nmi_handler+0x2f trap(c5203b5c) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc07f3f5f, esp = 0xc5203b9c, ebp = 0xc5203bbc --- ns8250_bus_ipend(c557fe00,13cb280,c0c35e7b,c557fe80,c557fe58,...) at ns8250_bus_ipend+0x5f uart_intr(c557fe00,c0c4c93d,c5203c14,109,0,...) at uart_intr+0x35b intr_event_handle(c5593d00,c5203c58,c55e02e0,c55e02e0,c55deb00,...) at intr_event_handle+0x76 intr_execute_handlers(c556dcd0,c5203c58,1,c5203c9c,c0c2c8f5,...) at intr_execute_handlers+0x49 lapic_handle_intr(31,c5203c58) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0c35e7b, esp = 0xc5203c98, ebp = 0xc5203c9c --- spinlock_exit(c0f79400,4,c0de4827,109) at spinlock_exit+0x2b _mtx_unlock_spin_flags(c0f79400,0,c0de0e45,557,c55e02e0,...) at _mtx_unlock_spin_flags+0xf4 ithread_loop(c555b7a0,c5203d28,c0de0b08,390,c5595578,...) at ithread_loop+0x25f fork_exit(c08d3d60,c555b7a0,c5203d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5203d60, ebp = 0 --- Tracing command intr pid 12 tid 100007 td 0xc55e05c0 sched_switch(c55e05c0,0,109,191,acc481e0,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c55e05c0,...) at mi_switch+0x219 ithread_loop(c555b7b0,c5200d28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b7b0,c5200d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5200d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc5597000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100005 td 0xc55972e0 sched_switch(c55972e0,0,109,191,bb6dada0,...) at sched_switch+0x3bc mi_switch(109,0,c0de0e45,54d,c55972e0,...) at mi_switch+0x219 ithread_loop(c555b7d0,c51fad28,c0de0b08,390,c5595578,...) at ithread_loop+0x1fb fork_exit(c08d3d60,c555b7d0,c51fad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc51fad60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc55975c0 sched_switch(c55975c0,0,108,18c,ab056b40,...) at sched_switch+0x3bc mi_switch(108,0,c0de8303,3b9,c55975c0,...) at mi_switch+0x219 sched_idletd(0,c51f4d28,c0de0b08,390,c5595834,...) at sched_idletd+0x19f fork_exit(c09225c0,0,c51f4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc51f4d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc55978a0 sched_switch(c55978a0,0,108,18c,a8f92430,...) at sched_switch+0x3bc mi_switch(108,0,c0de8303,3b9,c55978a0,...) at mi_switch+0x219 sched_idletd(0,c51f1d28,c0de0b08,390,c5595834,...) at sched_idletd+0x19f fork_exit(c09225c0,0,c51f1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc51f1d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc5597b80 sched_switch(c5597b80,0,104,191,d79f2e88,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,6c,...) at mi_switch+0x219 sleepq_switch(c5597b80,0,c0deba40,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0deba40,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c5595af0,6c,c0dee80a,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5595af0,c5595b78,16c,c0dee80a,0,...) at _sleep+0x35c kern_wait(c5597b80,ffffffff,c51edc44,0,0,...) at kern_wait+0xbe6 wait4(c5597b80,c51edcec,c51edd28,c0dec356,0,...) at wait4+0x3b syscallenter(c5597b80,c51edce4,c51edce4,0,0,...) at syscallenter+0x263 syscall(c51edd28) at syscall+0x4f Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x80551d3, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc559c000 sched_switch(c559c000,0,104,191,1ea2c298,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c559c000,0,c0deba40,268,c559c000,...) at sleepq_switch+0x162 sleepq_wait(c10ed500,0,c51eac8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c10ed500,c10ed4e4,c0e17faf,194,0,...) at _cv_wait+0x243 audit_worker(0,c51ead28,c0de0b08,390,c5596000,...) at audit_worker+0x84 fork_exit(c0b0fb30,0,c51ead28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc51ead60, ebp = 0 --- Tracing command kernel pid 0 tid 100064 td 0xc5a4d000 sched_switch(c5a4d000,0,104,191,3f44be68,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5a4d000,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566b40,0,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c5566b40,c5566b58,0,c0dd969a,0,...) at _sleep+0x372 taskqueue_thread_loop(c1144520,df304d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c093df40,c1144520,df304d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf304d60, ebp = 0 --- Tracing command kernel pid 0 tid 100054 td 0xc5805000 sched_switch(c5805000,0,104,191,6b9f6210,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5805000,0,c0deba40,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c0f6b104,0,c0dd969a,2,0,...) at sleepq_timedwait+0x6b _sleep(c0f6b104,0,0,c0dd969a,bb8,...) at _sleep+0x342 pause(c0dd969a,bb8,120,11e,c0f68ea8,...) at pause+0x47 deadlkres(0,df2e6d28,c0de0b08,390,c0f68e20,...) at deadlkres+0x302 fork_exit(c08b4de0,0,df2e6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xdf2e6d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc57082e0 sched_switch(c57082e0,0,104,191,1e9dcd48,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57082e0,0,c0deba40,268,c0f690e0,...) at sleepq_switch+0x162 sleepq_wait(c5566bc0,0,c0de7039,c0dd969a,0,...) at sleepq_wait+0x63 msleep_spin(c5566bc0,c5566bd8,c0dd969a,0,c0de4827,...) at msleep_spin+0x21d taskqueue_thread_loop(c5764568,c52a5d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0x8f fork_exit(c093df40,c5764568,c52a5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc52a5d60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc57085c0 sched_switch(c57085c0,0,104,191,9cd103e0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57085c0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566c00,0,c0de7039,c0dd969a,0,...) at sleepq_wait+0x63 msleep_spin(c5566c00,c5566c18,c0dd969a,0,c0de4827,...) at msleep_spin+0x21d taskqueue_thread_loop(c5751568,c527ad28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0x8f fork_exit(c093df40,c5751568,c527ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc527ad60, ebp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xc5708b80 sched_switch(c5708b80,0,104,191,62fa9740,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5708b80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566c80,0,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c5566c80,c5566c98,0,c0dd969a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0f87aa8,c5231d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c093df40,c0f87aa8,c5231d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5231d60, ebp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xc55e0b80 sched_switch(c55e0b80,0,104,191,27e0c2f0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c55e0b80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566d00,0,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c5566d00,c5566d18,0,c0dd969a,0,...) at _sleep+0x372 taskqueue_thread_loop(c10edae4,c522bd28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c093df40,c10edae4,c522bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc522bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc5705000 sched_switch(c5705000,0,104,191,27e0afa8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5705000,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566d40,0,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c5566d40,c5566d58,0,c0dd969a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0f69798,c5228d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c093df40,c0f69798,c5228d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5228d60, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc57055c0 sched_switch(c57055c0,0,104,191,27e09ef0,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57055c0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566d80,0,c0de7039,c0dd969a,0,...) at sleepq_wait+0x63 msleep_spin(c5566d80,c5566d98,c0dd969a,0,c0de4827,...) at msleep_spin+0x21d taskqueue_thread_loop(c0f39f38,c5222d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0x8f fork_exit(c093df40,c0f39f38,c5222d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5222d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc57058a0 sched_switch(c57058a0,0,104,191,27e08f18,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c57058a0,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566d80,0,c0de7039,c0dd969a,0,...) at sleepq_wait+0x63 msleep_spin(c5566d80,c5566d98,c0dd969a,0,c0de4827,...) at msleep_spin+0x21d taskqueue_thread_loop(c0f39f38,c521fd28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0x8f fork_exit(c093df40,c0f39f38,c521fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc521fd60, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc5705b80 sched_switch(c5705b80,0,104,191,27e07fe8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c5705b80,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5566d80,0,c0de7039,c0dd969a,0,...) at sleepq_wait+0x63 msleep_spin(c5566d80,c5566d98,c0dd969a,0,c0de4827,...) at msleep_spin+0x21d taskqueue_thread_loop(c0f39f38,c521cd28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0x8f fork_exit(c093df40,c0f39f38,c521cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc521cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100009 td 0xc55e0000 sched_switch(c55e0000,0,104,191,6ecf6ee8,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,0,...) at mi_switch+0x219 sleepq_switch(c55e0000,0,c0deba40,268,0,...) at sleepq_switch+0x162 sleepq_wait(c5567040,0,c0dd969a,0,0,...) at sleepq_wait+0x63 _sleep(c5567040,c5567058,0,c0dd969a,0,...) at _sleep+0x372 taskqueue_thread_loop(c0f86520,c5206d28,c0de0b08,390,c0f68e20,...) at taskqueue_thread_loop+0xb5 fork_exit(c093df40,c0f86520,c5206d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5206d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0f690e0 sched_switch(c0f690e0,0,104,191,870db320,...) at sched_switch+0x3bc mi_switch(104,0,c0deba40,1f3,54,...) at mi_switch+0x219 sleepq_switch(c0f690e0,0,c0deba40,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c0f68e20,54,c0de8c1f,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f68e20,0,54,c0de8c1f,2710,...) at _sleep+0x342 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2030 MB Dumping 199 MB: 184 168 152 136 120 104 88 72 56 40 24 8 Dump complete = 0xf db:0:doadump> reset (kgdb) bt #0 doadump () at pcpu.h:244 #1 0xc04d8ef9 in db_fncall (dummy1=-1061928626, dummy2=0, dummy3=-1, dummy4=0xdf3545d8 "ìE5ß") at /mnt/pho/src/sys/ddb/db_command.c:548 #2 0xc04d932f in db_command (last_cmdp=0xc0f380fc, cmd_table=0x0, dopager=0) at /mnt/pho/src/sys/ddb/db_command.c:445 #3 0xc04d93e4 in db_command_script (command=0xc0f39008 "call doadump") at /mnt/pho/src/sys/ddb/db_command.c:516 #4 0xc04dd650 in db_script_exec (scriptname=0xc0f38960 "doadump", warnifnotfound=Variable "warnifnotfound" is not available. ) at /mnt/pho/src/sys/ddb/db_script.c:302 #5 0xc04dd6e1 in db_run_cmd (addr=-1057739232, have_addr=0, count=-1067149719, modif=0xdf354710 "") at /mnt/pho/src/sys/ddb/db_script.c:375 #6 0xc04d92f1 in db_command (last_cmdp=0xc0f380fc, cmd_table=0x0, dopager=1) at /mnt/pho/src/sys/ddb/db_command.c:445 #7 0xc04d944a in db_command_loop () at /mnt/pho/src/sys/ddb/db_command.c:498 #8 0xc04db36d in db_trap (type=12, code=0) at /mnt/pho/src/sys/ddb/db_main.c:229 #9 0xc0930bc2 in kdb_trap (type=12, code=0, tf=0xdf354904) at /mnt/pho/src/sys/kern/subr_kdb.c:533 #10 0xc0c42c1f in trap_fatal (frame=0xdf354904, eva=40) at /mnt/pho/src/sys/i386/i386/trap.c:956 #11 0xc0c43143 in trap (frame=0xdf354904) at /mnt/pho/src/sys/i386/i386/trap.c:347 #12 0xc0c2c52c in calltrap () at /mnt/pho/src/sys/i386/i386/exception.s:168 #13 0xc0b4414e in softdep_revert_mkdir (dp=0xc631cae0, ip=0xc6a289f8) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:4123 #14 0xc0b65fd9 in ufs_mkdir (ap=0xdf354bf8) at /mnt/pho/src/sys/ufs/ufs/ufs_vnops.c:1980 #15 0xc0c654f5 in VOP_MKDIR_APV (vop=0xc0f1b160, a=0xdf354bf8) at vnode_if.c:1534 #16 0xc09997b5 in kern_mkdirat (td=0xc5e8f8a0, fd=-100, path=0xbfbfe92c
, segflg=UIO_USERSPACE, mode=Variable "mode" is not available. ) at vnode_if.h:665 #17 0xc099987e in kern_mkdir (td=0xc5e8f8a0, path=0xbfbfe92c
, segflg=UIO_USERSPACE, mode=480) at /mnt/pho/src/sys/kern/vfs_syscalls.c:3727 #18 0xc09998a9 in mkdir (td=0xc5e8f8a0, uap=0xdf354cec) at /mnt/pho/src/sys/kern/vfs_syscalls.c:3706 #19 0xc093e303 in syscallenter (td=0xc5e8f8a0, sa=0xdf354ce4) at /mnt/pho/src/sys/kern/subr_trap.c:344 #20 0xc0c42ebf in syscall (frame=0xdf354d28) at /mnt/pho/src/sys/i386/i386/trap.c:1080 #21 0xc0c2c591 in Xint0x80_syscall () at /mnt/pho/src/sys/i386/i386/exception.s:266 #22 0x00000033 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) f 13 #13 0xc0b4414e in softdep_revert_mkdir (dp=0xc631cae0, ip=0xc6a289f8) at /mnt/pho/src/sys/ufs/ffs/ffs_softdep.c:4123 4123 jaddref = (struct jaddref *)TAILQ_LAST(&inodedep->id_inoreflst, (kgdb) l 4118 dvp = ITOV(dp); 4119 4120 ACQUIRE_LOCK(&lk); 4121 inodedep = inodedep_lookup_ip(dp); 4122 if (DOINGSUJ(dvp)) { 4123 jaddref = (struct jaddref *)TAILQ_LAST(&inodedep->id_inoreflst, 4124 inoreflst); 4125 KASSERT(jaddref->ja_parent == ip->i_number, 4126 ("softdep_revert_mkdir: dotdot addref parent mismatch")); 4127 cancel_jaddref(jaddref, inodedep, &inodedep->id_inowait); (kgdb) p inodedep->id_inoreflst $1 = {tqh_first = 0x0, tqh_last = 0xc97dba3c} (kgdb) p *dp $2 = {i_nextsnap = {tqe_next = 0x0, tqe_prev = 0x0}, i_vnode = 0xc6331984, i_ump = 0xc5580600, i_flag = 2, i_dev = 0xc6002a00, i_number = 35, i_effnlink = 11425, i_fs = 0xc5f6f000, i_dquot = {0x0, 0x0}, i_count = 0, i_endoff = 278528, i_diroff = 0, i_offset = 278528, i_un = {dirhash = 0xcabae600, snapblklist = 0xcabae600}, i_ea_area = 0x0, i_ea_len = 0, i_ea_error = 0, i_ea_refs = 0, i_mode = 16864, i_nlink = 11425, i_size = 279040, i_flags = 0, i_gen = 1925633162, i_uid = 1001, i_gid = 0, dinode_u = {din1 = 0xc6343900, din2 = 0xc6343900}} (kgdb) p *ip $3 = {i_nextsnap = {tqe_next = 0x0, tqe_prev = 0x0}, i_vnode = 0xc698bd98, i_ump = 0xc5580600, i_flag = 2, i_dev = 0xc6002a00, i_number = 111968, i_effnlink = 0, i_fs = 0xc5f6f000, i_dquot = {0x0, 0x0}, i_count = 0, i_endoff = 0, i_diroff = 0, i_offset = 0, i_un = {dirhash = 0x0, snapblklist = 0x0}, i_ea_area = 0x0, i_ea_len = 0, i_ea_error = 0, i_ea_refs = 0, i_mode = 16864, i_nlink = 0, i_size = 512, i_flags = 0, i_gen = 639086888, i_uid = 1001, i_gid = 0, dinode_u = { din1 = 0xc69c4100, din2 = 0xc69c4100}} (kgdb) info loc inodedep = (struct inodedep *) 0xc97dba00 jaddref = (struct jaddref *) 0x0 dotaddref = (struct jaddref *) 0xdf1083a0 dvp = (struct vnode *) 0xc6331984 (kgdb) -- Test scenario: suj10.sh