r231644M KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0: Tue Feb 14 09:04:21 CET 2012 test@leopard3.netperf.freebsd.org:/mnt/pho/obj/mnt/pho/src/sys/PHO i386 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. CPU: Intel(R) Xeon(R) CPU 3050 @ 2.13GHz (2128.04-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f2 Family = 6 Model = f Stepping = 2 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2080350208 (1983 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 0.0 on pci1 pci2: on pcib2 pcib3: at device 0.2 on pci1 pci3: on pcib3 pcib4: irq 16 at device 3.0 on pci0 pci10: on pcib4 pci10: at device 0.0 (no driver attached) pcib5: irq 17 at device 28.0 on pci0 pcib5: failed to allocate initial I/O port window: 0-0xfff pcib5: failed to allocate initial memory window: 0-0xfffff pcib5: failed to allocate initial prefetch window: 0-0xfffff pci14: on pcib5 pcib6: irq 17 at device 28.4 on pci0 pci18: on pcib6 em0: port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci18 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:8e:d0:68 pcib7: irq 16 at device 28.5 on pci0 pci19: on pcib7 em1: port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci19 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:8e:d0:69 uhci0: port 0x3000-0x301f irq 23 at device 29.0 on pci0 usbus0: on uhci0 uhci1: port 0x3020-0x303f irq 19 at device 29.1 on pci0 usbus1: on uhci1 uhci2: port 0x3040-0x305f irq 18 at device 29.2 on pci0 usbus2: on uhci2 uhci3: port 0x3060-0x307f irq 16 at device 29.3 on pci0 usbus3: on uhci3 ehci0: mem 0xe0000000-0xe00003ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4: on ehci0 pcib8: at device 30.0 on pci0 pci20: on pcib8 vgapci0: port 0x7000-0x70ff mem 0xe8000000-0xefffffff,0xe0500000-0xe050ffff irq 16 at device 4.0 on pci20 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0 ata0: at channel 0 on atapci0 ahci0: port 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem 0xe0000400-0xe00007ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcbfff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: parallel port not found. ctl: CAM Target Layer loaded est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 828082806000828 device_attach: est1 attach returned 6 p4tcc1: on cpu1 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xc0a9c4e0(0xc63a8b80) 0.011069004 s ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 Expensive timeout(9) function: 0xc0554be0(0xc634c600) 0.020083558 s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 8 ports with 8 removable, self powered ada0 at ahcich0 bus 0 scbus1 target 0 lun 0 ada0: ATA-7cd0 at ata0 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 16625336 Hz quality 1000 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from nfs: []... NFS ROOT: 192.168.5.1:/zoo/leopard3 em0: link state changed to UP Interface em0 IP-Address 192.168.5.109 Broadcast 192.168.5.255 Setting hostuuid: 64a93476-27b1-3004-c299-0030488ed164. Setting hostid: 0x7acc54f1. Starting ddb. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/ada0d: 63810 files, 6817718 used, 109407408 free (1504 frags, 13675738 blocks, 0.0% fragmentation) mount_nfs: can't update /var/db/mounttab for 192.168.5.1:/zoo/leopard3 Mounting local file systems:. Setting hostname: leopard3.netperf.freebsd.org. Starting Network: lo0 em0 em1. lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 nd6 options=21 em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:68 inet 192.168.5.109 netmask 0xffffff00 broadcast 192.168.5.255 inet6 fe80::230:48ff:fe8e:d068%em0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting devd. Starting Network: em1. em1: flags=8c02 metric 0 mtu 1500 options=4219b ether 00:30:48:8e:d0:69 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 Creating and/or trimming log files. Starting syslogd. No core dumps found. ELF ldconfig path: /lib /usr/lib /usr/lib/compat a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting date via ntp. 15 Feb 08:44:46 ntpdate[1390]: step time server 192.168.5.1 offset -0.727771 sec Starting rpcbind. NFS access cache time=60 Clearing /tmp (X related). Starting mountd. Feb 15 08:44:47 leopard3 mountd[1518]: can't delete exports for /dev: Invalid argument mount option is unknown Starting nfsd. Starting statd. Updating motd:. Configuring syscons: blanktime. Starting sshd. Starting cron. Local package initialization: watchdogd. Starting inetd. Wed Feb 15 08:44:48 UTC 2012 FreeBSD/i386 (leopard3.netperf.freebsd.org) (ttyu0) login: root Feb 15 08:46:33 leopard3 login: ROOT LOGIN (root) ON ttyu0 FreeBSD 10.0-CURRENT (PHO) #0: Tue Feb 14 09:04:21 CET 2012 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc distribution has been installed, they're also available formatted in /usr/share/doc. If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. You may also use sysinstall(8) to re-enter the installation and configuration utility. Edit /etc/motd to change this login announcement. leopard3# ~test/keepalive.sh Wed Feb 15 09:46:40 CET 2012 9:46AM up 4 mins, 1 user, load averages: 0.06, 0.32, 0.19 Feb 15 09:46:55 leopard3 su: test to root on /dev/pts/0 Feb 15 08:47:02 leopard3 mountd[1519]: can't delete exports for /dev: Invalid argument mount option is unknown lock order reversal: 1st 0xdfd19a40 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2652 2nd 0xc8dbd400 dirhash (dirhash) @ /mnt/pho/src/sys/ufs/ufs/ufs_dirhash.c:284 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366752f,7366752f,7366752f,7269645f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273f00,11c,ee82a878,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c8dbd400,c0fc6912,c615ba88,c0fc6593,...) at _witness_debugger+0x25 witness_checkorder(c8dbd400,9,c0fc6593,11c,0,...) at witness_checkorder+0x86f _sx_xlock(c8dbd400,0,c0fc6593,11c,c8e8cd98,...) at _sx_xlock+0xa0 ufsdirhash_acquire(dfd199e0,c8e8cd98,ee82a9f4,e0c94028,ee82a948,...) at ufsdirhash_acquire+0x48 ufsdirhash_add(c8e8cd98,ee82a9f4,28,ee82a934,ee82a938,...) at ufsdirhash_add+0x13 ufs_direnter(c909315c,c916b414,ee82a9f4,ee82ab84,dfd21a40,...) at ufs_direnter+0x7a9 ufs_mkdir(ee82ac14,c0fe7ce5,0,0,ee82abac,...) at ufs_mkdir+0xa6f VOP_MKDIR_APV(c10ddc40,ee82ac14,ee82ab84,ee82abac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c8f745c0,ffffff9c,bfbfeda5,0,1f8,...) at kern_mkdirat+0x291 kern_mkdir(c8f745c0,bfbfeda5,0,1f8,ee82ad1c,...) at kern_mkdir+0x2e sys_mkdir(c8f745c0,ee82acec,c0fe0b42,ee82ad80,257,...) at sys_mkdir+0x29 syscall(ee82ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28185b33, esp = 0xbfbfe8cc, ebp = 0xbfbfe968 --- lock order reversal: 1st 0xc90931b4 ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 2nd 0xdfd19a40 bufwait (bufwait) @ /mnt/pho/src/sys/ufs/ffs/ffs_vnops.c:262 3rd 0xc916b46c ufs (ufs) @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,75735f73,632e7262,3631323a,6f000a37,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96319,c1273ee0,877,ee82a3c8,...) at kdb_backtrace+0x2a _witness_debugger(c0f96319,c916b46c,c0f84bd4,c615b8e8,c0f9e3d4,...) at _witness_debugger+0x25 witness_checkorder(c916b46c,9,c0f9e3d4,877,0,...) at witness_checkorder+0x86f __lockmgr_args(c916b46c,80100,c916b4d8,0,0,...) at __lockmgr_args+0x834 ffs_lock(ee82a4ec,0,c0f95525,80100,c916b414,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c10ddc40,ee82a4ec,c61541e0,c10eec60,c916b414,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c916b414,80100,c0f9e3d4,877,4,...) at _vn_lock+0x78 vget(c916b414,80100,c8f745c0,50,0,...) at vget+0xbb vfs_hash_get(c91685a8,b800,80000,c8f745c0,ee82a638,...) at vfs_hash_get+0xed ffs_vgetf(c91685a8,b800,80000,ee82a638,1,...) at ffs_vgetf+0x49 softdep_sync_buf(c909315c,dfd199e0,1,108,0,...) at softdep_sync_buf+0x4a3 ffs_syncvnode(c909315c,1,bfbfe8cc,bfbfe968,bfbfe968,...) at ffs_syncvnode+0x24c ffs_truncate(c909315c,400,0,880,c8925a00,...) at ffs_truncate+0x827 ufs_direnter(c909315c,c916b414,ee82a9f4,ee82ab84,dfd21a40,...) at ufs_direnter+0x9a1 ufs_mkdir(ee82ac14,c0fe7ce5,0,0,ee82abac,...) at ufs_mkdir+0xa6f VOP_MKDIR_APV(c10ddc40,ee82ac14,ee82ab84,ee82abac,0,...) at VOP_MKDIR_APV+0xc5 kern_mkdirat(c8f745c0,ffffff9c,bfbfeda5,0,1f8,...) at kern_mkdirat+0x291 kern_mkdir(c8f745c0,bfbfeda5,0,1f8,ee82ad1c,...) at kern_mkdir+0x2e sys_mkdir(c8f745c0,ee82acec,c0fe0b42,ee82ad80,257,...) at sys_mkdir+0x29 syscall(ee82ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (136, FreeBSD ELF32, sys_mkdir), eip = 0x28185b33, esp = 0xbfbfe8cc, ebp = 0xbfbfe968 --- lock order reversal: 1st 0xdfd21420 bufwait (bufwait) @ /mnt/pho/src/sys/kern/vfs_bio.c:2652 2nd 0xc8e84e1c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:2299 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,662f7366,662f7366,735f7366,7370616e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273c78,8fb,ee83c950,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c8e84e1c,c0fc249d,c615bc28,c0fc2416,...) at _witness_debugger+0x25 witness_checkorder(c8e84e1c,9,c0fc2416,8fb,0,...) at witness_checkorder+0x86f __lockmgr_args(c8e84e1c,80900,c90934d8,0,0,...) at __lockmgr_args+0x834 ffs_copyonwrite(c9093414,dfd213c0,dfd213c0,0,0,...) at ffs_copyonwrite+0x2c2 ffs_geom_strategy(c909350c,dfd213c0,0,dfd213c0,0,...) at ffs_geom_strategy+0xeb bufwrite(dfd213c0,0,c0fc58d0,7de,0) at bufwrite+0x159 ffs_bufwrite(dfd213c0,c9106800,100,4000,0,...) at ffs_bufwrite+0x290 ffs_update(c916b414,2,c0fc6059,130,0,...) at ffs_update+0x495 ffs_syncvnode(c916b414,2,c8f732e0,5db,c0f8caeb,...) at ffs_syncvnode+0x442 ffs_sync(c91685a8,2,2,9b,c91685a8,...) at ffs_sync+0x29f sys_sync(c8f732e0,ee83ccec,c,c,286,...) at sys_sync+0x110 syscall(ee83cd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (36, FreeBSD ELF32, sys_sync), eip = 0x280dcabf, esp = 0xbfbfe95c, ebp = 0xbfbfe9c8 --- lock order reversal: 1st 0xc8eb9724 newnfs (newnfs) @ /mnt/pho/src/sys/kern/vfs_mount.c:1249 2nd 0xc910e310 devfs (devfs) @ /mnt/pho/src/sys/ufs/ffs/ffs_vfsops.c:1376 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366662f,7366662f,7366765f,2e73706f,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273c78,560,ee83c9b8,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c910e310,c0f7c90f,c615b2d0,c0fc58d0,...) at _witness_debugger+0x25 witness_checkorder(c910e310,9,c0fc58d0,560,c910e37c,...) at witness_checkorder+0x86f __lockmgr_args(c910e310,80400,c910e37c,0,0,...) at __lockmgr_args+0x834 vop_stdlock(ee83cad8,c8f73390,c0fc6a99,80400,c910e2b8,...) at vop_stdlock+0x65 VOP_LOCK1_APV(c10b38c0,ee83cad8,c910d15c,c10eec60,c910e2b8,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c910e2b8,80400,c0fc58d0,560,c8e83e00,...) at _vn_lock+0x78 ffs_flushfiles(c917a000,0,c8f732e0,4,c0f8caeb,...) at ffs_flushfiles+0x11c ffs_unmount(c917a000,8000000,c0f9da27,51a,c6030000,...) at ffs_unmount+0x1d6 dounmount(c917a000,8000000,c8f732e0,49c,0,...) at dounmount+0x419 sys_unmount(c8f732e0,ee83ccec,c,c,206,...) at sys_unmount+0x421 syscall(ee83cd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (22, FreeBSD ELF32, sys_unmount), eip = 0x280dcb3f, esp = 0xbfbfe21c, ebp = 0xbfbfe2e8 --- lock order reversal: 1st 0xc8e84e1c snaplk (snaplk) @ /mnt/pho/src/sys/ufs/ufs/ufs_vnops.c:965 2nd 0xc916b5c8 ufs (ufs) @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1629 KDB: stack backtrace: db_trace_self_wrapper(c0f927fb,7366662f,616e735f,6f687370,3a632e74,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0a95cab,c0f96300,c1273c68,65d,ee7da704,...) at kdb_backtrace+0x2a _witness_debugger(c0f96300,c916b5c8,c0f84bd4,c615b8e8,c0fc2416,...) at _witness_debugger+0x25 witness_checkorder(c916b5c8,9,c0fc2416,65d,0,...) at witness_checkorder+0x86f __lockmgr_args(c916b5c8,80000,0,0,0,...) at __lockmgr_args+0x834 ffs_snapremove(c916b570,c0fc2354,c10c0b00,c0f9b8da,dfd198a0,...) at ffs_snapremove+0x11f ffs_truncate(c916b570,0,0,c00,0,...) at ffs_truncate+0x607 ufs_inactive(ee7daadc,c0fe7478,c916b634,c916b634,c916b570,...) at ufs_inactive+0x1fb VOP_INACTIVE_APV(c10ddc40,ee7daadc,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c10ddc40,ee7dab10,c0f9e3d4,8c6,0,...) at vinactive+0xb1 vputx(ee7dac3c,c0b003e3,c916b570,ffffffdf,c916b570,...) at vputx+0x36c vput(c916b570,ffffffdf,c916b570,ee7dab94,2c00,...) at vput+0x10 kern_unlinkat(c8dd0000,ffffff9c,bfbfebba,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(c8dd0000,bfbfebba,0,ee7dad1c,c0dc08c7,...) at kern_unlink+0x2f sys_unlink(c8dd0000,ee7dacec,c0fe0b42,c0f9701a,246,...) at sys_unlink+0x22 syscall(ee7dad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x281720af, esp = 0xbfbfe91c, ebp = 0xbfbfe998 --- fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 30 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 106 lock type devfs: EXCL by thread 0xc90c22e0 (pid 2043) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 33 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 118 lock type devfs: EXCL by thread 0xc8f74000 (pid 2047) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 33 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 118 lock type devfs: EXCL by thread 0xc8f74000 (pid 2047) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 33 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 118 lock type devfs: EXCL by thread 0xc8dd4000 (pid 2051) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 37 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 134 lock type devfs: EXCL by thread 0xc8da12e0 (pid 2055) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 37 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 134 lock type devfs: EXCL by thread 0xc8dd0000 (pid 2059) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 50 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 186 lock type devfs: EXCL by thread 0xc8dd22e0 (pid 2073) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 50 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 186 lock type devfs: EXCL by thread 0xc90c25c0 (pid 2077) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc8f74000 (pid 2091) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc6411b80 (pid 2125) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc8dd4000 (pid 2139) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc93035c0 (pid 2153) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc90c25c0 (pid 2187) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc8dd22e0 (pid 2251) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc9302b80 (pid 2275) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc88f5b80 (pid 2299) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc93022e0 (pid 2413) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 51 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 190 lock type devfs: EXCL by thread 0xc90c52e0 (pid 2427) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 55 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 206 lock type devfs: EXCL by thread 0xc6411b80 (pid 2461) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 83 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 318 lock type devfs: EXCL by thread 0xc8f732e0 (pid 2465) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 83 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 318 lock type devfs: EXCL by thread 0xc9303b80 (pid 2469) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 83 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 318 lock type devfs: EXCL by thread 0xc9303b80 (pid 2469) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 92 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 354 lock type devfs: EXCL by thread 0xc93035c0 (pid 2472) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 113 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 438 lock type devfs: EXCL by thread 0xc8cc5b80 (pid 2477) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 113 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 438 lock type devfs: EXCL by thread 0xc8cc5b80 (pid 2477) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc9093414: tag devfs, type VCHR usecount 1, writecount 0, refcount 262 mountedhere 0xc8be6e00 flags () v_object 0xc9113a20 ref 0 pages 1034 lock type devfs: EXCL by thread 0xc8f732e0 (pid 2480) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9adc7 at softdep_flushworklist+0x47 #5 0xc0c9fe2d at ffs_sync+0x32d #6 0xc0b093b2 at vfs_write_suspend+0x122 #7 0xc0c83e0f at ffs_snapshot+0xcef #8 0xc0ca23d7 at ffs_mount+0x1d17 #9 0xc0af18ab at vfs_donmount+0x12eb #10 0xc0af1f4f at sys_nmount+0x7f #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a : : fsync: giving up on dirty 0xc8954828: tag devfs, type VCHR usecount 1, writecount 0, refcount 737 mountedhere 0xcad45100 flags () v_object 0xcb9c6360 ref 0 pages 2930 lock type devfs: EXCL by thread 0xc8dcf2e0 (pid 45104) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a fsync: giving up on dirty 0xc8954828: tag devfs, type VCHR usecount 1, writecount 0, refcount 737 mountedhere 0xcad45100 flags () v_object 0xcb9c6360 ref 0 pages 2930 lock type devfs: EXCL by thread 0xc8ce7b80 (pid 45158) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ae76f5 at vop_stdlock+0x65 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0c9fef7 at ffs_sync+0x3f7 #5 0xc0b093b2 at vfs_write_suspend+0x122 #6 0xc0c83e0f at ffs_snapshot+0xcef #7 0xc0ca23d7 at ffs_mount+0x1d17 #8 0xc0af18ab at vfs_donmount+0x12eb #9 0xc0af1f4f at sys_nmount+0x7f #10 0xc0dc08c7 at syscall+0x2c7 #11 0xc0da9bf1 at Xint0x80_syscall+0x21 dev md5a Feb 15 13:21:23 leopard3 last message repeated 96 times Wed Feb 15 14:29:54 CET 2012 2:29PM up 4:48, 2 users, load averages: 0.04, 4.87, 10.66 Wed Feb 15 14:39:54 CET 2012 2:39PM up 4:58, 2 users, load averages: 0.00, 0.63, 5.21 Wed Feb 15 14:49:54 CET 2012 2:49PM up 5:08, 2 users, load averages: 0.00, 0.07, 2.57 Wed Feb 15 14:59:54 CET 2012 2:59PM up 5:18, 2 users, load averages: 0.00, 0.00, 1.24 Wed Feb 15 15:09:54 CET 2012 3:09PM up 5:28, 2 users, load averages: 0.00, 0.00, 0.60 Wed Feb 15 15:19:54 CET 2012 3:19PM up 5:38, 2 users, load averages: 0.00, 0.00, 0.28 KDB: enter: Break to debugger [ thread pid 11 tid 100003 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> run pho db:0:pho> bt Tracing pid 11 tid 100003 td 0xc61ad8a0 kdb_enter(c0e02aae,c0f927c6,c642d000,c5eb2b74,c0a916e3,...) at kdb_enter+0x3a kdb_alt_break_internal(c5eb2ba4,c08c927e,2,c642d058,c0da5555,...) at kdb_alt_break_internal+0x9a kdb_alt_break(2,c642d058,c0da5555,c642d080,c642d058,...) at kdb_alt_break+0x13 uart_intr(c642d000,0,c12d64e0,c5eb2be8,0,...) at uart_intr+0x1ae intr_event_handle(c61f2d80,c5eb2c10,c5eb2c34,0,c634d800,...) at intr_event_handle+0x76 intr_execute_handlers(c61f10d0,c5eb2c10,0,c5eb2c50,c0da9f55,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,c5eb2c10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0da5555, esp = 0xc5eb2c50, ebp = 0xc5eb2c50 --- acpi_cpu_c1(c5eb2c6c,c10f07b0,0,301,5005,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(ffffffff,c5eb2ca0,c0db37e9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1138640,1,c5eb2ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c5eb2cc4,c0f90852,3c0,c61ad8a0,...) at cpu_idle+0x89 sched_idletd(0,c5eb2d28,c0f88d50,3db,c61ab84c,...) at sched_idletd+0x263 fork_exit(c0a82d70,0,c5eb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb2d60, ebp = 0 --- db:0:bt> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x735b80 curthread = 0xc61ad8a0: pid 11 "idle: cpu0" curpcb = 0xc5eb2d80 fpcurthread = none idlethread = 0xc61ad8a0: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 dynamic pcpu = 0x4db8b80 curthread = 0xc61ad5c0: pid 11 "idle: cpu1" curpcb = 0xc5eb5d80 fpcurthread = none idlethread = 0xc61ad5c0: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db:0:allpcpu> show alllocks Process 45181 (ls) thread 0xc8ce7b80 (100408) exclusive lockmgr ufs (ufs) r = 0 (0xcb8dab38) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 Process 45179 (rm) thread 0xc8ce72e0 (100411) exclusive lockmgr ufs (ufs) r = 0 (0xc8fb25c8) locked @ /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1629 Process 45059 (mkdir) thread 0xc8cc5b80 (100069) exclusive lockmgr ufs (ufs) r = 0 (0xc9926058) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 Process 45053 (mkdir) thread 0xc8c7b5c0 (100398) exclusive lockmgr ufs (ufs) r = 0 (0xc8b285c8) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 Process 45051 (mkdir) thread 0xc8fad8a0 (100334) exclusive lockmgr ufs (ufs) r = 0 (0xc9711058) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 Process 45043 (mkdir) thread 0xc93515c0 (100456) exclusive lockmgr ufs (ufs) r = 0 (0xc98c3c94) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2279 Process 1783 (sshd) thread 0xc8da05c0 (100098) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xc8f05bec) locked @ /mnt/pho/src/sys/kern/uipc_sockbuf.c:148 Process 18 (softdepflush) thread 0xc88f65c0 (100067) exclusive lockmgr ufs (ufs) r = 0 (0xca8c7724) locked @ /mnt/pho/src/sys/kern/vfs_subr.c:2167 Process 16 (syncer) thread 0xc88f6b80 (100065) exclusive lockmgr bufwait (bufwait) r = 0 (0xdfe33e20) locked @ /mnt/pho/src/sys/kern/vfs_bio.c:2652 db:0:alllocks> show lockedvnods Locked vnodes 0xcb8daae0: tag ufs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VV_ROOT|VI_DOINGINACT) v_object 0xcb870870 ref 0 pages 0 lock type ufs: EXCL by thread 0xc8ce7b80 (pid 45181) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afdaf6 at vputx+0x266 #5 0xc0afe340 at vrele+0x10 #6 0xc0b06f45 at sys_fchdir+0x395 #7 0xc0dc08c7 at syscall+0x2c7 #8 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 2, on dev md5a 0xc8fb2570: tag ufs, type VREG usecount 0, writecount 0, refcount 83 mountedhere 0 flags (VV_SYSTEM|VI_DOINGINACT) v_object 0xca3f3b40 ref 0 pages 212 lock type ufs: EXCL by thread 0xc8ce72e0 (pid 45179) with exclusive waiters pending #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0c8189f at ffs_snapremove+0x11f #2 0xc0c7cf67 at ffs_truncate+0x607 #3 0xc0cacc1b at ufs_inactive+0x1fb #4 0xc0de17e5 at VOP_INACTIVE_APV+0xc5 #5 0xc0afb0b1 at vinactive+0xb1 #6 0xc0afdbfc at vputx+0x36c #7 0xc0afdc70 at vput+0x10 #8 0xc0b003e3 at kern_unlinkat+0x2a3 #9 0xc0b0045f at kern_unlink+0x2f #10 0xc0b00492 at sys_unlink+0x22 #11 0xc0dc08c7 at syscall+0x2c7 #12 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 6, on dev md5a 0xc9711000: tag ufs, type VDIR usecount 0, writecount 0, refcount 3 mountedhere 0 flags (VI_DOINGINACT) v_object 0xcb519d80 ref 0 pages 1 lock type ufs: EXCL by thread 0xc8fad8a0 (pid 45051) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afdaf6 at vputx+0x266 #5 0xc0afe340 at vrele+0x10 #6 0xc0b052b0 at kern_chdir+0x220 #7 0xc0b052e2 at sys_chdir+0x22 #8 0xc0dc08c7 at syscall+0x2c7 #9 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 28, on dev md5a 0xc8b28570: tag ufs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT) v_object 0xcae2ae10 ref 0 pages 0 lock type ufs: EXCL by thread 0xc8c7b5c0 (pid 45053) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afdaf6 at vputx+0x266 #5 0xc0afe340 at vrele+0x10 #6 0xc0b052b0 at kern_chdir+0x220 #7 0xc0b052e2 at sys_chdir+0x22 #8 0xc0dc08c7 at syscall+0x2c7 #9 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 27, on dev md5a 0xc98c3c3c: tag ufs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT) v_object 0xcb0423f0 ref 0 pages 0 lock type ufs: EXCL by thread 0xc93515c0 (pid 45043) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afdaf6 at vputx+0x266 #5 0xc0afe340 at vrele+0x10 #6 0xc0b052b0 at kern_chdir+0x220 #7 0xc0b052e2 at sys_chdir+0x22 #8 0xc0dc08c7 at syscall+0x2c7 #9 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 14, on dev md5a 0xc9926000: tag ufs, type VDIR usecount 0, writecount 0, refcount 3 mountedhere 0 flags (VI_DOINGINACT) v_object 0xc9170ea0 ref 0 pages 1 lock type ufs: EXCL by thread 0xc8cc5b80 (pid 45059) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afdaf6 at vputx+0x266 #5 0xc0afe340 at vrele+0x10 #6 0xc0b052b0 at kern_chdir+0x220 #7 0xc0b052e2 at sys_chdir+0x22 #8 0xc0dc08c7 at syscall+0x2c7 #9 0xc0da9bf1 at Xint0x80_syscall+0x21 ino 62, on dev md5a 0xca8c76cc: tag ufs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags () v_object 0xcb58a7e0 ref 0 pages 0 lock type ufs: EXCL by thread 0xc88f65c0 (pid 18) #0 0xc0a4395e at __lockmgr_args+0xc1e #1 0xc0ca5bf1 at ffs_lock+0xa1 #2 0xc0de3c25 at VOP_LOCK1_APV+0xb5 #3 0xc0b097b8 at _vn_lock+0x78 #4 0xc0afc26b at vget+0xbb #5 0xc0aebc7d at vfs_hash_get+0xed #6 0xc0c9ee19 at ffs_vgetf+0x49 #7 0xc0c8b8d0 at handle_workitem_remove+0x60 #8 0xc0c8b134 at process_worklist_item+0x2b4 #9 0xc0c9ac68 at softdep_process_worklist+0x88 #10 0xc0c9b240 at softdep_flush+0x2b0 #11 0xc0a2e438 at fork_exit+0xb8 #12 0xc0da9c04 at fork_trampoline+0x8 ino 61, on dev md5a db:0:lockedvnods> show mount 0xc8923b50 192.168.5.1:/zoo/leopard3 on / (nfs) 0xc8924000 devfs on /dev (devfs) 0xc892387c /dev/ada0d on /work (ufs) 0xc89235a8 procfs on /proc (procfs) 0xc89232d4 /dev/md0 on /var (ufs) 0xc8923000 /dev/md1 on /tmp (ufs) 0xc9299b50 /dev/md5a on /mnt (ufs) More info: show mount db:0:mount> ps pid ppid pgrp uid state wmesg wchan cmd 45303 1779 1779 0 S+ nanslp 0xc112b9e4 sleep 45181 45180 1696 0 D getblk 0xdfe33e20 ls 45180 1696 1696 0 S wait 0xc9bf52c4 sh 45179 12493 1797 0 D+ getblk 0xdfe33e20 rm 45059 45041 12521 1001 D+ getblk 0xdfe33e20 mkdir 45053 45041 12521 1001 D+ getblk 0xdfe33e20 mkdir 45051 45041 12521 1001 D+ getblk 0xdfe33e20 mkdir 45043 45041 12521 1001 D+ getblk 0xdfe33e20 mkdir 45041 45017 12521 1001 S+ wait 0xc8f72588 mkdir 45017 12524 12521 1001 S+ wait 0xc8dcb2c4 mkdir 12524 12523 12521 1001 S+ wait 0xc9895000 run 12523 12521 12521 1001 S+ wait 0xc8c8bb10 run 12521 12511 12521 1001 S+ wait 0xc9097b10 run 12511 12493 1797 0 SW+ wait 0xc8ba82c4 su 12506 0 0 0 DL mdwait 0xc9254800 [md5] 12493 1797 1797 0 SW+ wait 0xc989684c sh 1797 1796 1797 0 SW+ wait 0xc8e80000 sh 1796 1792 1796 0 SW+ wait 0xc8dc82c4 sh 1792 1791 1792 0 SW+ pause 0xc90b031c csh 1791 1788 1791 1001 SW+ wait 0xc90b0588 su 1788 1787 1788 1001 SWs+ wait 0xc8e8084c sh 1787 1783 1783 1001 S select 0xc8f64024 sshd 1783 1675 1783 0 SWs sbwait 0xc8f05c18 sshd 1779 1775 1779 0 S+ wait 0xc8f71588 sh 1775 1770 1775 0 SW+ pause 0xc8dcbb68 csh 1770 1 1770 0 SWs+ wait 0xc88ed2c4 login 1769 1 1769 0 Ss+ ttyin 0xc6179a70 getty 1768 1 1768 0 Ss+ ttyin 0xc6179c70 getty 1767 1 1767 0 Ss+ ttyin 0xc6179e70 getty 1766 1 1766 0 Ss+ ttyin 0xc6452070 getty 1765 1 1765 0 Ss+ ttyin 0xc6452270 getty 1764 1 1764 0 Ss+ ttyin 0xc6452470 getty 1763 1 1763 0 Ss+ ttyin 0xc6178070 getty 1762 1 1762 0 Ss+ ttyin 0xc6178470 getty 1724 1 1724 0 Ss select 0xc88fc1a4 inetd 1696 1 1696 0 Ss wait 0xc8d78b10 watchdogd 1686 1 1686 0 Ss nanslp 0xc112b9e4 cron 1675 1 1675 0 Ss select 0xc8f64a24 sshd 1536 1 1536 0 Ss select 0xc884fea4 rpc.statd 1529 1528 1528 0 S (threaded) nfsd 100116 S rpcsvc 0xc6366850 nfsd: service 100115 S rpcsvc 0xc6366c90 nfsd: service 100114 S rpcsvc 0xc6366790 nfsd: service 100102 S rpcsvc 0xc63667d0 nfsd: master 1528 1 1528 0 Ss select 0xc88fc224 nfsd 1519 1 1519 0 Ss select 0xc88fc024 mountd 1453 0 0 0 DL mdwait 0xc8e92800 [md1] 1398 1 1398 0 Ss select 0xc6455da4 rpcbind 1368 1 1368 0 Ss select 0xc884fde4 syslogd 1189 1 1189 0 Ss select 0xc884f864 devd 102 0 0 0 DL mdwait 0xc8e93000 [md0] 18 0 0 0 DL getblk 0xdfe33e20 [softdepflush] 17 0 0 0 DL vlruwt 0xc88ec588 [vnlru] 16 0 0 0 DL ufs 0xc8fb25c8 [syncer] 9 0 0 0 DL psleep 0xc12a1848 [bufdaemon] 8 0 0 0 DL pgzero 0xc12a9d5c [pagezero] 7 0 0 0 DL psleep 0xc12a9948 [vmdaemon] 6 0 0 0 DL psleep 0xc12a9910 [pagedaemon] 5 0 0 0 DL ccb_scan 0xc10f01d4 [xpt_thrd] 4 0 0 0 DL waiting_ 0xc12a36d8 [sctp_iterator] 3 0 0 0 DL ctl_work 0xc645a000 [ctl_thrd] 2 0 0 0 DL - 0xc634ee3c [fdc0] 15 0 0 0 DL (threaded) [usb] 100051 D - 0xc63ebd34 [usbus4] 100050 D - 0xc63ebd04 [usbus4] 100049 D - 0xc63ebcd4 [usbus4] 100048 D - 0xc63ebca4 [usbus4] 100047 D - 0xc63d7dac [usbus3] 100046 D - 0xc63d7d7c [usbus3] 100045 D - 0xc63d7d4c [usbus3] 100044 D - 0xc63d7d1c [usbus3] 100042 D - 0xc63c0dac [usbus2] 100041 D - 0xc63c0d7c [usbus2] 100040 D - 0xc63c0d4c [usbus2] 100039 D - 0xc63c0d1c [usbus2] 100037 D - 0xc63aedac [usbus1] 100036 D - 0xc63aed7c [usbus1] 100035 D - 0xc63aed4c [usbus1] 100034 D - 0xc63aed1c [usbus1] 100032 D - 0xc6395dac [usbus0] 100031 D - 0xc6395d7c [usbus0] 100030 D - 0xc6395d4c [usbus0] 100029 D - 0xc6395d1c [usbus0] 14 0 0 0 DL - 0xc112a7e4 [yarrow] 13 0 0 0 DL (threaded) [geom] 100011 D - 0xc11283e8 [g_down] 100010 D - 0xc11283e4 [g_up] 100009 D - 0xc11283dc [g_event] 12 0 0 0 WL (threaded) [intr] 100056 I [irq1: atkbd0] 100054 I [swi0: uart uart] 100053 I [irq258: ahci0] 100052 I [irq14: ata0] 100043 I [irq16: uhci3] 100038 I [irq18: uhci2] 100033 I [irq19: uhci1] 100028 I [irq23: uhci0 ehci0] 100023 I [swi6: task queue] 100022 I [swi6: Giant taskq] 100020 I [swi5: +] 100014 I [swi2: cambio] 100008 I [swi3: vm] 100007 I [swi4: clock] 100006 I [swi4: clock] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xc61abb10 [init] 10 0 0 0 DL audit_wo 0xc12a75e0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100068 D - 0xc64258c0 [mca taskq] 100058 D - 0xc112a7e4 [deadlkres] 100027 D - 0xc6301840 [em1 txq] 100026 D - 0xc6301880 [em1 que] 100025 D - 0xc6301940 [em0 txq] 100024 D - 0xc6301980 [em0 que] 100021 D - 0xc6301e80 [thread taskq] 100019 D - 0xc6302040 [acpi_task_2] 100018 D - 0xc6302040 [acpi_task_1] 100017 D - 0xc6302040 [acpi_task_0] 100016 D - 0xc6302080 [kqueue taskq] 100015 D - 0xc63020c0 [ffs_trim taskq] 100012 D - 0xc6191e80 [firmware taskq] 100000 D sched 0xc11284c0 [swapper] 45018 12524 12521 1001 Z+ swap 45046 45041 12521 1001 Z+ mkdir 45057 45041 12521 1001 Z+ mkdir 45045 45041 12521 1001 Z+ mkdir 45044 45041 12521 1001 Z+ mkdir 45054 45041 12521 1001 Z+ mkdir 45049 45041 12521 1001 Z+ mkdir 45056 45041 12521 1001 Z+ mkdir 45055 45041 12521 1001 Z+ mkdir 45058 45041 12521 1001 Z+ mkdir 45048 45041 12521 1001 Z+ mkdir 45052 45041 12521 1001 Z+ mkdir 45050 45041 12521 1001 Z+ mkdir 45060 45041 12521 1001 Z+ mkdir 45047 45041 12521 1001 Z+ mkdir db:0:ps> allt Tracing command sleep pid 45303 tid 100299 td 0xc919a8a0 sched_switch(c919a8a0,0,104,1a3,7678f500,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c919a8a0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(927c1,c0a9c4e0,c919a8a0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c112b9e4,6c,c0f8fcbf,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c112b9e4,0,16c,c0f8fcbf,927c1,...) at _sleep+0x34f kern_nanosleep(c919a8a0,eeaccc50,eeaccc58,258,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c919a8a0,eeacccec,c0fe0b42,c0f979e0,246,...) at sys_nanosleep+0x6f syscall(eeaccd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281660b7, esp = 0xbfbfec2c, ebp = 0xbfbfec78 --- Tracing command ls pid 45181 tid 100408 td 0xc8ce7b80 sched_switch(c8ce7b80,0,104,1a3,9ff67b60,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c8ce7b80,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(cb8daae0,0,1,eec88b6c,c1273f90,...) at ffs_update+0x1ab ufs_inactive(eec88bc4,c0fe7478,cb8daba4,cb8daba4,cb8daae0,...) at ufs_inactive+0x37a VOP_INACTIVE_APV(c10ddc40,eec88bc4,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(cb8daba4,0,c0f9e3d4,8e8,0,...) at vinactive+0xb1 vputx(eec88c6c,c0b06f45,cb8daae0,c0f9f9e7,319,...) at vputx+0x36c vrele(cb8daae0,c0f9f9e7,319,316,eec88c58,...) at vrele+0x10 sys_fchdir(c8ce7b80,eec88cec,c0fe0b42,c0f7ef01,246,...) at sys_fchdir+0x395 syscall(eec88d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (13, FreeBSD ELF32, sys_fchdir), eip = 0x28130bff, esp = 0xbfbfe5fc, ebp = 0xbfbfe668 --- Tracing command sh pid 45180 tid 100229 td 0xc956e8a0 sched_switch(c956e8a0,0,104,1a3,9bd6b468,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c956e8a0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c9bf52c4,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c9bf52c4,c9bf534c,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c956e8a0,ffffffff,ee9dac60,2,0,...) at kern_wait+0x626 sys_wait4(c956e8a0,ee9dacec,c,c,212,...) at sys_wait4+0x3b syscall(ee9dad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- Tracing command rm pid 45179 tid 100411 td 0xc8ce72e0 sched_switch(c8ce72e0,0,104,1a3,df4a6db8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c8ce72e0,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(c8fb2570,1,c0fc6059,146,c0adb63b,...) at ffs_update+0x1ab ffs_syncvnode(c8fb2570,1,0,4000,c6151e00,...) at ffs_syncvnode+0x4d5 ffs_snapremove(c8fb2570,c0fc2354,c1146a54,0,c0f941e0,...) at ffs_snapremove+0x6d3 ffs_truncate(c8fb2570,0,0,c00,0,...) at ffs_truncate+0x607 ufs_inactive(eec91adc,c0fe7478,c8fb2634,c8fb2634,c8fb2570,...) at ufs_inactive+0x1fb VOP_INACTIVE_APV(c10ddc40,eec91adc,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c10ddc40,eec91b10,c0f9e3d4,8c6,0,...) at vinactive+0xb1 vputx(eec91c3c,c0b003e3,c8fb2570,ffffffdf,c8fb2570,...) at vputx+0x36c vput(c8fb2570,ffffffdf,c8fb2570,eec91b94,bc0,...) at vput+0x10 kern_unlinkat(c8ce72e0,ffffff9c,bfbfebba,0,0,...) at kern_unlinkat+0x2a3 kern_unlink(c8ce72e0,bfbfebba,0,eec91d1c,c0dc08c7,...) at kern_unlink+0x2f sys_unlink(c8ce72e0,eec91cec,c0fe0b42,c0f9701a,246,...) at sys_unlink+0x22 syscall(eec91d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (10, FreeBSD ELF32, sys_unlink), eip = 0x281720af, esp = 0xbfbfe91c, ebp = 0xbfbfe998 --- Tracing command mkdir pid 45059 tid 100069 td 0xc8cc5b80 sched_switch(c8cc5b80,0,104,1a3,f7cdf370,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c8cc5b80,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(c9926000,0,1,ee739b08,c1272eb8,...) at ffs_update+0x1ab ufs_inactive(ee739b60,c0fe7478,c99260c4,c99260c4,c9926000,...) at ufs_inactive+0x37a VOP_INACTIVE_APV(c10ddc40,ee739b60,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c99260c4,0,c0f9e3d4,8e8,0,...) at vinactive+0xb1 vputx(ee739c58,c0b052b0,c9926000,c0f9f9e7,34d,...) at vputx+0x36c vrele(c9926000,c0f9f9e7,34d,34a,c9714700,...) at vrele+0x10 kern_chdir(c8cc5b80,804b1b8,0,ee739d1c,c0dc08c7,...) at kern_chdir+0x220 sys_chdir(c8cc5b80,ee739cec,c1138640,ee739c88,246,...) at sys_chdir+0x22 syscall(ee739d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (12, FreeBSD ELF32, sys_chdir), eip = 0x280eec1f, esp = 0xbfbfadbc, ebp = 0xbfbfb1f8 --- Tracing command mkdir pid 45053 tid 100398 td 0xc8c7b5c0 sched_switch(c8c7b5c0,0,104,1a3,e18d3b08,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c8c7b5c0,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(c8b28570,0,1,eec6ab08,c1272af0,...) at ffs_update+0x1ab ufs_inactive(eec6ab60,c0fe7478,c8b28634,c8b28634,c8b28570,...) at ufs_inactive+0x37a VOP_INACTIVE_APV(c10ddc40,eec6ab60,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c8b28634,0,c0f9e3d4,8e8,0,...) at vinactive+0xb1 vputx(eec6ac58,c0b052b0,c8b28570,c0f9f9e7,34d,...) at vputx+0x36c vrele(c8b28570,c0f9f9e7,34d,34a,c943e200,...) at vrele+0x10 kern_chdir(c8c7b5c0,804b1b8,0,eec6ad1c,c0dc08c7,...) at kern_chdir+0x220 sys_chdir(c8c7b5c0,eec6acec,c0fe0b42,c0fe8015,246,...) at sys_chdir+0x22 syscall(eec6ad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (12, FreeBSD ELF32, sys_chdir), eip = 0x280eec1f, esp = 0xbfbfbebc, ebp = 0xbfbfc2f8 --- Tracing command mkdir pid 45051 tid 100334 td 0xc8fad8a0 sched_switch(c8fad8a0,0,104,1a3,e65d6eb8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c8fad8a0,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(c9711000,0,1,eebaab08,c1273388,...) at ffs_update+0x1ab ufs_inactive(eebaab60,c0fe7478,c97110c4,c97110c4,c9711000,...) at ufs_inactive+0x37a VOP_INACTIVE_APV(c10ddc40,eebaab60,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c97110c4,0,c0f9e3d4,8e8,0,...) at vinactive+0xb1 vputx(eebaac58,c0b052b0,c9711000,c0f9f9e7,34d,...) at vputx+0x36c vrele(c9711000,c0f9f9e7,34d,34a,c943ed00,...) at vrele+0x10 kern_chdir(c8fad8a0,804b1b8,0,eebaad1c,c0dc08c7,...) at kern_chdir+0x220 sys_chdir(c8fad8a0,eebaacec,c0fe0b42,c0fe8015,246,...) at sys_chdir+0x22 syscall(eebaad28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (12, FreeBSD ELF32, sys_chdir), eip = 0x280eec1f, esp = 0xbfbf987c, ebp = 0xbfbf9cb8 --- Tracing command mkdir pid 45043 tid 100456 td 0xc93515c0 sched_switch(c93515c0,0,104,1a3,5da80e70,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c93515c0,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(c98c3c3c,0,1,eed3eb08,c1272d58,...) at ffs_update+0x1ab ufs_inactive(eed3eb60,c0fe7478,c98c3d00,c98c3d00,c98c3c3c,...) at ufs_inactive+0x37a VOP_INACTIVE_APV(c10ddc40,eed3eb60,c0f9e3d4,96f,c10eec20,...) at VOP_INACTIVE_APV+0xc5 vinactive(c98c3d00,0,c0f9e3d4,8e8,0,...) at vinactive+0xb1 vputx(eed3ec58,c0b052b0,c98c3c3c,c0f9f9e7,34d,...) at vputx+0x36c vrele(c98c3c3c,c0f9f9e7,34d,34a,c94dd700,...) at vrele+0x10 kern_chdir(c93515c0,804b1b8,0,eed3ed1c,c0dc08c7,...) at kern_chdir+0x220 sys_chdir(c93515c0,eed3ecec,c0fe0b42,c0fe8015,246,...) at sys_chdir+0x22 syscall(eed3ed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (12, FreeBSD ELF32, sys_chdir), eip = 0x280eec1f, esp = 0xbfbfa53c, ebp = 0xbfbfa978 --- Tracing command mkdir pid 45041 tid 100142 td 0xc90c22e0 sched_switch(c90c22e0,0,104,1a3,f4ee07f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c90c22e0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f72588,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8f72588,c8f72610,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c90c22e0,aff3,ee866c60,0,0,...) at kern_wait+0x626 sys_wait4(c90c22e0,ee866cec,c,c,212,...) at sys_wait4+0x3b syscall(ee866d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28104ffb, esp = 0xbfbfe90c, ebp = 0xbfbfe928 --- Tracing command mkdir pid 45017 tid 100113 td 0xc8dcf8a0 sched_switch(c8dcf8a0,0,104,1a3,2e5de2e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8dcf8a0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8dcb2c4,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8dcb2c4,c8dcb34c,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c8dcf8a0,aff1,ee7e0c60,0,0,...) at kern_wait+0x626 sys_wait4(c8dcf8a0,ee7e0cec,c,c,212,...) at sys_wait4+0x3b syscall(ee7e0d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28104ffb, esp = 0xbfbfe90c, ebp = 0xbfbfe928 --- Tracing command run pid 12524 tid 100452 td 0xc939ab80 sched_switch(c939ab80,0,104,1a3,f8a07f80,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c939ab80,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c9895000,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c9895000,c9895088,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c939ab80,afd9,eed0cc60,0,0,...) at kern_wait+0x626 sys_wait4(c939ab80,eed0ccec,c,c,212,...) at sys_wait4+0x3b syscall(eed0cd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28103ffb, esp = 0xbfbfe38c, ebp = 0xbfbfe3a8 --- Tracing command run pid 12523 tid 100266 td 0xc8c9b5c0 sched_switch(c8c9b5c0,0,104,1a3,f7896780,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8c9b5c0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8c8bb10,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8c8bb10,c8c8bb98,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c8c9b5c0,30ec,eea69c60,0,0,...) at kern_wait+0x626 sys_wait4(c8c9b5c0,eea69cec,c,c,212,...) at sys_wait4+0x3b syscall(eea69d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28103ffb, esp = 0xbfbfe8ac, ebp = 0xbfbfe8c8 --- Tracing command run pid 12521 tid 100287 td 0xc917d8a0 sched_switch(c917d8a0,0,104,1a3,f44536c0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c917d8a0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c9097b10,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c9097b10,c9097b98,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c917d8a0,30eb,eeaa8c60,0,0,...) at kern_wait+0x626 sys_wait4(c917d8a0,eeaa8cec,c,c,212,...) at sys_wait4+0x3b syscall(eeaa8d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28103ffb, esp = 0xbfbfe8ac, ebp = 0xbfbfe8c8 --- Tracing command su pid 12511 tid 100392 td 0xc8c888a0 Tracing command md5 pid 12506 tid 100293 td 0xc8c9e5c0 sched_switch(c8c9e5c0,0,104,1a3,69caa318,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8c9e5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c9254800,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c9254800,c9254820,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c9254800,eeabad28,c0f88d50,3db,c908384c,...) at md_kthread+0x11a fork_exit(c0769790,c9254800,eeabad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xeeabad60, ebp = 0 --- Tracing command sh pid 12493 tid 100444 td 0xc989b5c0 Tracing command sh pid 1797 tid 100104 td 0xc8dd25c0 Tracing command sh pid 1796 tid 100100 td 0xc8da0000 Tracing command csh pid 1792 tid 100133 td 0xc8dd48a0 Tracing command su pid 1791 tid 100132 td 0xc8dd4b80 Tracing command sh pid 1788 tid 100129 td 0xc8f735c0 Tracing command sshd pid 1787 tid 100096 td 0xc8da0b80 sched_switch(c8da0b80,0,104,1a3,62226118,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8da0b80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c8f64010,0,c0f8caeb,c8da0b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f64024,0,ee7a9a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c8f64024,c8f64010,c0f96978,6a4,c8cd1cb0,...) at _cv_wait_sig+0x260 seltdwait(c8cd1cb0,58,c61a9500,c8da0b80,ee7a9acc,...) at seltdwait+0xa2 kern_select(c8da0b80,a,288060b8,288060cc,0,0,20,c8dca000,0) at kern_select+0x655 sys_select(c8da0b80,ee7a9cec,c0fe0b42,c0f5c807,206,...) at sys_select+0x66 syscall(ee7a9d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe19c, ebp = 0xbfbfe1e8 --- Tracing command sshd pid 1783 tid 100098 td 0xc8da05c0 Tracing command sh pid 1779 tid 100120 td 0xc8f752e0 sched_switch(c8f752e0,0,104,1a3,76227260,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8f752e0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f71588,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8f71588,c8f71610,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c8f752e0,ffffffff,ee81ec60,2,0,...) at kern_wait+0x626 sys_wait4(c8f752e0,ee81ecec,c0fe0b42,c0f9713e,212,...) at sys_wait4+0x3b syscall(ee81ed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x28161ffb, esp = 0xbfbfe9dc, ebp = 0xbfbfe9f8 --- Tracing command csh pid 1775 tid 100110 td 0xc8dd02e0 Tracing command login pid 1770 tid 100080 td 0xc8cc02e0 Tracing command getty pid 1769 tid 100089 td 0xc8da22e0 sched_switch(c8da22e0,0,104,1a3,96693278,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8da22e0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8da22e0,c8da22e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6179a70,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6179a70,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6179a00,c6179a70,ee78dc44,1,0,...) at tty_wait+0x71 ttydisc_read(c6179a00,ee78dc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c6453a00,ee78dc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8ce29a0,ee78dc44,c61a8e00,0,c8da22e0,...) at devfs_read_f+0x7e dofileread(ee78dc44,ffffffff,ffffffff,0,ee78dc20,...) at dofileread+0x9e kern_readv(c8da22e0,0,ee78dc44,ee78dc64,1,...) at kern_readv+0x68 sys_read(c8da22e0,ee78dcec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee78dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1768 tid 100070 td 0xc88f6000 sched_switch(c88f6000,0,104,1a3,9612ee90,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f6000,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c88f6000,c88f6000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6179c70,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6179c70,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6179c00,c6179c70,ee73dc44,1,0,...) at tty_wait+0x71 ttydisc_read(c6179c00,ee73dc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c6453b00,ee73dc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd02d8,ee73dc44,c61a8e00,0,c88f6000,...) at devfs_read_f+0x7e dofileread(ee73dc44,ffffffff,ffffffff,0,ee73dc20,...) at dofileread+0x9e kern_readv(c88f6000,0,ee73dc44,ee73dc64,1,...) at kern_readv+0x68 sys_read(c88f6000,ee73dcec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee73dd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1767 tid 100112 td 0xc8dcfb80 sched_switch(c8dcfb80,0,104,1a3,968894c0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dcfb80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8dcfb80,c8dcfb80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6179e70,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6179e70,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6179e00,c6179e70,ee7ddc44,1,0,...) at tty_wait+0x71 ttydisc_read(c6179e00,ee7ddc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c6453c00,ee7ddc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd0a80,ee7ddc44,c61a8e00,0,c8dcfb80,...) at devfs_read_f+0x7e dofileread(ee7ddc44,ffffffff,ffffffff,0,ee7ddc20,...) at dofileread+0x9e kern_readv(c8dcfb80,0,ee7ddc44,ee7ddc64,1,...) at kern_readv+0x68 sys_read(c8dcfb80,ee7ddcec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee7ddd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1766 tid 100087 td 0xc8da28a0 sched_switch(c8da28a0,0,104,1a3,96848eb8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8da28a0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8da28a0,c8da28a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6452070,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6452070,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6452000,c6452070,ee784c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6452000,ee784c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c63fd200,ee784c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8ce2e38,ee784c44,c61a8e00,0,c8da28a0,...) at devfs_read_f+0x7e dofileread(ee784c44,ffffffff,ffffffff,0,ee784c20,...) at dofileread+0x9e kern_readv(c8da28a0,0,ee784c44,ee784c64,1,...) at kern_readv+0x68 sys_read(c8da28a0,ee784cec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee784d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1765 tid 100107 td 0xc8dd0b80 sched_switch(c8dd0b80,0,104,1a3,964ce918,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dd0b80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8dd0b80,c8dd0b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6452270,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6452270,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6452200,c6452270,ee7cec44,1,0,...) at tty_wait+0x71 ttydisc_read(c6452200,ee7cec44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c63fcc00,ee7cec44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8ce2cb0,ee7cec44,c61a8e00,0,c8dd0b80,...) at devfs_read_f+0x7e dofileread(ee7cec44,ffffffff,ffffffff,0,ee7cec20,...) at dofileread+0x9e kern_readv(c8dd0b80,0,ee7cec44,ee7cec64,1,...) at kern_readv+0x68 sys_read(c8dd0b80,ee7cecec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee7ced28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1764 tid 100083 td 0xc88f88a0 sched_switch(c88f88a0,0,104,1a3,96961350,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f88a0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c88f88a0,c88f88a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6452470,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6452470,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6452400,c6452470,ee772c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6452400,ee772c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c63fcd00,ee772c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd1a80,ee772c44,c61a8e00,0,c88f88a0,...) at devfs_read_f+0x7e dofileread(ee772c44,ffffffff,ffffffff,0,ee772c20,...) at dofileread+0x9e kern_readv(c88f88a0,0,ee772c44,ee772c64,1,...) at kern_readv+0x68 sys_read(c88f88a0,ee772cec,c0fe0b42,ee772d80,246,...) at sys_read+0x4f syscall(ee772d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1763 tid 100106 td 0xc8dd2000 sched_switch(c8dd2000,0,104,1a3,9654b7e0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dd2000,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8dd2000,c8dd2000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6178070,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6178070,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6178000,c6178070,ee7cbc44,1,0,...) at tty_wait+0x71 ttydisc_read(c6178000,ee7cbc44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c63fce00,ee7cbc44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8cd1af0,ee7cbc44,c61a8e00,0,c8dd2000,...) at devfs_read_f+0x7e dofileread(ee7cbc44,ffffffff,ffffffff,0,ee7cbc20,...) at dofileread+0x9e kern_readv(c8dd2000,0,ee7cbc44,ee7cbc64,1,...) at kern_readv+0x68 sys_read(c8dd2000,ee7cbcec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee7cbd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command getty pid 1762 tid 100108 td 0xc8dd08a0 sched_switch(c8dd08a0,0,104,1a3,9657f4b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dd08a0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,c8dd08a0,c8dd08a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c6178470,0,c0f990b0,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c6178470,c112a090,c0f98a70,527,0,...) at _cv_wait_sig+0x260 tty_wait(c6178400,c6178470,ee7d1c44,1,0,...) at tty_wait+0x71 ttydisc_read(c6178400,ee7d1c44,0,a1,0,...) at ttydisc_read+0x22c ttydev_read(c63fd100,ee7d1c44,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c8ce2d20,ee7d1c44,c61a8e00,0,c8dd08a0,...) at devfs_read_f+0x7e dofileread(ee7d1c44,ffffffff,ffffffff,0,ee7d1c20,...) at dofileread+0x9e kern_readv(c8dd08a0,0,ee7d1c44,ee7d1c64,1,...) at kern_readv+0x68 sys_read(c8dd08a0,ee7d1cec,c0fe0b42,c0df31ae,246,...) at sys_read+0x4f syscall(ee7d1d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x28196dc3, esp = 0xbfbfed8c, ebp = 0xbfbfedb8 --- Tracing command inetd pid 1724 tid 100122 td 0xc8f74b80 sched_switch(c8f74b80,0,104,1a3,839fb030,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8f74b80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c88fc190,0,c0f8caeb,c8f74b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88fc1a4,0,ee824a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88fc1a4,c88fc190,c0f96978,6a4,c8ce2888,...) at _cv_wait_sig+0x260 seltdwait(c8ce2888,58,c61a8e00,c8f74b80,c8f74c30,...) at seltdwait+0xa2 kern_select(c8f74b80,8,bfbfdfa0,0,0,0,20,c8f71000,4) at kern_select+0x655 sys_select(c8f74b80,ee824cec,c0fe0b42,c0f971ab,202,...) at sys_select+0x66 syscall(ee824d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x281a8d43, esp = 0xbfbfdf5c, ebp = 0xbfbfede8 --- Tracing command watchdogd pid 1696 tid 100086 td 0xc8da2b80 sched_switch(c8da2b80,0,104,1a3,91fb2088,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8da2b80,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8d78b10,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c8d78b10,c8d78b98,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c8da2b80,b07c,ee77fc60,0,0,...) at kern_wait+0x626 sys_wait4(c8da2b80,ee77fcec,c,c,202,...) at sys_wait4+0x3b syscall(ee77fd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x2811cffb, esp = 0xbfbfec3c, ebp = 0xbfbfece8 --- Tracing command cron pid 1686 tid 100128 td 0xc8f738a0 sched_switch(c8f738a0,0,104,1a3,c1f0acc8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c8f738a0,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(ea61,c0a9c4e0,c8f738a0,1,100,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c112b9e4,6c,c0f8fcbf,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c112b9e4,0,16c,c0f8fcbf,ea61,...) at _sleep+0x34f kern_nanosleep(c8f738a0,ee836c50,ee836c58,3c,0,...) at kern_nanosleep+0xc1 sys_nanosleep(c8f738a0,ee836cec,c0fe0b42,c0f9760c,206,...) at sys_nanosleep+0x6f syscall(ee836d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (240, FreeBSD ELF32, sys_nanosleep), eip = 0x281850b7, esp = 0xbfbfec4c, ebp = 0xbfbfec78 --- Tracing command sshd pid 1675 tid 100117 td 0xc8dcf5c0 sched_switch(c8dcf5c0,0,104,1a3,ff089d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dcf5c0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c8f64a10,0,c0f8caeb,c8dcf5c0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c8f64a24,0,ee815a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c8f64a24,c8f64a10,c0f96978,6a4,c8cd19d8,...) at _cv_wait_sig+0x260 seltdwait(c8cd19d8,58,c61a9480,c8dcf5c0,0,...) at seltdwait+0xa2 kern_select(c8dcf5c0,7,2880c0b0,0,0,0,20,c8f72000,a) at kern_select+0x655 sys_select(c8dcf5c0,ee815cec,c0fe0b42,c0f979f2,202,...) at sys_select+0x66 syscall(ee815d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x284b0d43, esp = 0xbfbfe25c, ebp = 0xbfbfedf8 --- Tracing command rpc.statd pid 1536 tid 100077 td 0xc88f58a0 sched_switch(c88f58a0,0,104,1a3,3b685ff0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f58a0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee758a20,c0a4b8fa,c884fe90,0,c88f58a0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c884fea4,0,ee758a58,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c884fea4,c884fe90,7531,6a4,c8cd1038,...) at _cv_timedwait_sig+0x270 seltdwait(ee758c10,ee758c18,c61a8e00,c88f58a0,c61b0000,...) at seltdwait+0x8a kern_select(c88f58a0,8,bfbfed14,0,0,ee758c5c,20,1e,0) at kern_select+0x655 sys_select(c88f58a0,ee758cec,c0fe0b42,c0f9729c,202,...) at sys_select+0x66 syscall(ee758d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2818cd43, esp = 0xbfbfec6c, ebp = 0xbfbfeda8 --- Tracing command nfsd pid 1529 tid 100116 td 0xc88f5000 sched_switch(c88f5000,0,104,1a3,724270f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f5000,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee812be0,c0a4b8fa,c6433d80,0,c88f5000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c6366850,0,ee812c18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c6366850,c6433d80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee812d14,c0a2e438,c6433d80,ee812d28,c0f88d50,...) at svc_run_internal+0x356 svc_thread_start(c6433d80,ee812d28,c0f88d50,3db,c8e80588,...) at svc_thread_start+0x10 fork_exit(c0c4b4e0,c6433d80,ee812d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0x16, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1529 tid 100115 td 0xc88f52e0 sched_switch(c88f52e0,0,104,1a3,5bfe6d68,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f52e0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee80fbe0,c0a4b8fa,c6433d80,0,c88f52e0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c6366c90,0,ee80fc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c6366c90,c6433d80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee80fd14,c0a2e438,c6433d80,ee80fd28,c0f88d50,...) at svc_run_internal+0x356 svc_thread_start(c6433d80,ee80fd28,c0f88d50,3db,c8e80588,...) at svc_thread_start+0x10 fork_exit(c0c4b4e0,c6433d80,ee80fd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0x16, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1529 tid 100114 td 0xc88f55c0 sched_switch(c88f55c0,0,104,1a3,5bfe58e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f55c0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee80cbe0,c0a4b8fa,c6433d80,0,c88f55c0,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c6366790,0,ee80cc18,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c6366790,c6433d80,1388,3b1,0,...) at _cv_timedwait_sig+0x270 svc_run_internal(ee80cd14,c0a2e438,c6433d80,ee80cd28,c0f88d50,...) at svc_run_internal+0x356 svc_thread_start(c6433d80,ee80cd28,c0f88d50,3db,c8e80588,...) at svc_thread_start+0x10 fork_exit(c0c4b4e0,c6433d80,ee80cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0x804c72e, eip = 0x16, esp = 0x33, ebp = 0 --- Tracing command nfsd pid 1529 tid 100102 td 0xc8dd2b80 sched_switch(c8dd2b80,0,104,1a3,359858a8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dd2b80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee7bfa40,c0a4b8fa,c6433d80,0,c8dd2b80,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c63667d0,0,ee7bfa78,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c63667d0,c6433d80,1388,3b1,20,...) at _cv_timedwait_sig+0x270 svc_run_internal(c8dd2d30,14,c0fbde62,c8f1f9cf,ee7bfc2c,...) at svc_run_internal+0x356 svc_run(c6433d80,0,c8f1fe17,1bf,c0a67a28,...) at svc_run+0x7b nfssvc_nfsd(bfbfe870,ee7bfc2c,c,c61a8e00,ee7bfc40,...) at nfssvc_nfsd+0x175 nfssvc_nfsserver(c8dd2b80,ee7bfcec,bfbfe870,c8dd2b80,ee7bfd1c,...) at nfssvc_nfsserver+0x55 sys_nfssvc(c8dd2b80,ee7bfcec,c0fe0b42,c0f9756b,202,...) at sys_nfssvc+0x89 syscall(ee7bfd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (155, FreeBSD ELF32, sys_nfssvc), eip = 0x280dd4fb, esp = 0xbfbfe82c, ebp = 0xbfbfea98 --- Tracing command nfsd pid 1528 tid 100081 td 0xc8cc0000 sched_switch(c8cc0000,0,104,1a3,1981380,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8cc0000,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c88fc210,0,c0f8caeb,c8cc0000,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88fc224,0,ee768a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88fc224,c88fc210,c0f96978,6a4,c8cd0038,...) at _cv_wait_sig+0x260 seltdwait(c8cd0038,58,c61a8e00,c8cc0000,c61af9f8,...) at seltdwait+0xa2 kern_select(c8cc0000,5,bfbfecb0,0,0,0,20,c8d79b10,1) at kern_select+0x655 sys_select(c8cc0000,ee768cec,c,c,202,...) at sys_select+0x66 syscall(ee768d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28184d43, esp = 0xbfbfea9c, ebp = 0xbfbfede8 --- Tracing command mountd pid 1519 tid 100103 td 0xc8dd28a0 sched_switch(c8dd28a0,0,104,1a3,c8413048,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8dd28a0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c88fc010,0,c0f8caeb,c8dd28a0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c88fc024,0,ee7c2a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c88fc024,c88fc010,c0f96978,6a4,c8cd1dc8,...) at _cv_wait_sig+0x260 seltdwait(c8cd1dc8,58,c61a8e00,c8dd28a0,478,...) at seltdwait+0xa2 kern_select(c8dd28a0,a,bfbfed4c,0,0,0,20,c8e802c4,bfbfed4c) at kern_select+0x655 sys_select(c8dd28a0,ee7c2cec,c0fe0b42,c0f7ef01,206,...) at sys_select+0x66 syscall(ee7c2d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x2819ad43, esp = 0xbfbfed1c, ebp = 0xbfbfede8 --- Tracing command md1 pid 1453 tid 100071 td 0xc8cc58a0 sched_switch(c8cc58a0,0,104,1a3,8b36a2d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8cc58a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8e92800,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c8e92800,c8e92820,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c8e92800,ee740d28,c0f88d50,3db,c8cbf84c,...) at md_kthread+0x11a fork_exit(c0769790,c8e92800,ee740d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee740d60, ebp = 0 --- Tracing command rpcbind pid 1398 tid 100074 td 0xc8cc5000 sched_switch(c8cc5000,0,104,1a3,d78cbeb8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8cc5000,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(ee74ca84,c0a4b8fa,c6455d90,0,c8cc5000,...) at sleepq_catch_signals+0xf9 sleepq_timedwait_sig(c6455da4,0,ee74cabc,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c6455da4,c6455d90,7531,6a4,ee74cb74,...) at _cv_timedwait_sig+0x270 seltdwait(ee74cc44,ee74cc4c,5a1,c8cc5000,c0dc8158,...) at seltdwait+0x8a sys_poll(c8cc5000,ee74ccec,c0fe0b42,c0f9729c,202,...) at sys_poll+0x321 syscall(ee74cd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (209, FreeBSD ELF32, sys_poll), eip = 0x2813ff4b, esp = 0xbfbfcbac, ebp = 0xbfbfed78 --- Tracing command syslogd pid 1368 tid 100075 td 0xc8cc0b80 sched_switch(c8cc0b80,0,104,1a3,c6bc7430,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c8cc0b80,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c884fdd0,0,c0f8caeb,c8cc0b80,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c884fde4,0,ee750a58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c884fde4,c884fdd0,c0f96978,6a4,c8cd09d8,...) at _cv_wait_sig+0x260 seltdwait(c8cd09d8,58,c61a8e00,c8cc0b80,0,...) at seltdwait+0xa2 kern_select(c8cc0b80,9,284760ac,0,0,0,20,c88edb10,0) at kern_select+0x655 sys_select(c8cc0b80,ee750cec,c0fe0b42,c0f9714e,246,...) at sys_select+0x66 syscall(ee750d28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x28199d43, esp = 0xbfbfe29c, ebp = 0xbfbfede8 --- Tracing command devd pid 1189 tid 100085 td 0xc88f82e0 sched_switch(c88f82e0,0,104,1a3,fcb1e88,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f82e0,0,c0f941e0,1a7,0,...) at sleepq_switch+0x162 sleepq_catch_signals(c0a4b8fa,c884f850,0,c0f8caeb,c88f82e0,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c884f864,0,ee77ba58,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c884f864,c884f850,c0f96978,6a4,c8cd0498,...) at _cv_wait_sig+0x260 seltdwait(c8cd0498,58,c61a8e00,c88f82e0,c1274b98,...) at seltdwait+0xa2 kern_select(c88f82e0,5,bfbfe960,0,0,0,20,c8d79000,35) at kern_select+0x655 sys_select(c88f82e0,ee77bcec,c0fe0b42,c0dee45f,206,...) at sys_select+0x66 syscall(ee77bd28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (93, FreeBSD ELF32, sys_select), eip = 0x808ba43, esp = 0xbfbfe92c, ebp = 0xbfbfee08 --- Tracing command md0 pid 102 tid 100109 td 0xc8dd05c0 sched_switch(c8dd05c0,0,104,1a3,705c058,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c8dd05c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c8e93000,5c,c0f2a96a,0,0,...) at sleepq_wait+0x63 _sleep(c8e93000,c8e93020,25c,c0f2a96a,0,...) at _sleep+0x39c md_kthread(c8e93000,ee7d4d28,c0f88d50,3db,c8e7f000,...) at md_kthread+0x11a fork_exit(c0769790,c8e93000,ee7d4d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xee7d4d60, ebp = 0 --- Tracing command softdepflush pid 18 tid 100067 td 0xc88f65c0 sched_switch(c88f65c0,0,104,1a3,e01c49c0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c88f65c0,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(dfe33e20,60,c0f868d4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(dfe33e20,81900,c8954920,c0f868d4,60,...) at __lockmgr_args+0xb3f getblk(c8954828,e0,0,4000,0,...) at getblk+0x167 breadn_flags(c8954828,e0,0,4000,0,...) at breadn_flags+0x43 ffs_update(ca8c76cc,0,c0fc29eb,249b,1,...) at ffs_update+0x1ab handle_workitem_remove(0,ec546c6c,2,6bb,ec546c70,...) at handle_workitem_remove+0x478 process_worklist_item(c12a7e60,0,c0fc29eb,5ea,c88f65c0,...) at process_worklist_item+0x2b4 softdep_process_worklist(c9299b50,0,c0fc29eb,556,3e8,...) at softdep_process_worklist+0x88 softdep_flush(0,ec546d28,c0f88d50,3db,c88ec2c4,...) at softdep_flush+0x2b0 fork_exit(c0c9af90,0,ec546d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec546d60, ebp = 0 --- Tracing command vnlru pid 17 tid 100066 td 0xc88f68a0 sched_switch(c88f68a0,0,104,1a3,7cd88040,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c88f68a0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c88ec588,60,c0f9f34d,0,0,...) at sleepq_timedwait+0x6b _sleep(c88ec588,c12a1b74,260,c0f9f34d,3e8,...) at _sleep+0x36a vnlru_proc(0,ec543d28,c0f88d50,3db,c88ec588,...) at vnlru_proc+0xe7 fork_exit(c0afd240,0,ec543d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec543d60, ebp = 0 --- Tracing command syncer pid 16 tid 100065 td 0xc88f6b80 sched_switch(c88f6b80,0,104,1a3,dbc41ee8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,60,...) at mi_switch+0x1ff sleepq_switch(c88f6b80,0,c0f941e0,268,60,...) at sleepq_switch+0x162 sleepq_wait(c8fb25c8,60,c0f84bd4,4,0,...) at sleepq_wait+0x63 __lockmgr_args(c8fb25c8,80400,c8fb2634,0,0,...) at __lockmgr_args+0xb3f ffs_lock(ec540af8,c0ae7d65,c8954920,80400,c8fb2570,...) at ffs_lock+0xa1 VOP_LOCK1_APV(c10ddc40,ec540af8,0,c10eec60,c8fb2570,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c8fb2570,80400,c0fc22f6,78,0,...) at _vn_lock+0x78 ffs_update(c8fb2570,3,c0fc6059,130,0,...) at ffs_update+0x359 ffs_syncvnode(c8fb2570,3,c0afb3af,c10ddc40,c8fb2668,...) at ffs_syncvnode+0x442 ffs_fsync(ec540c6c,c0fe7843,c8fb2634,ec540c6c,c8fb2570,...) at ffs_fsync+0x27 VOP_FSYNC_APV(c10ddc40,ec540c6c,c0f9e3d4,6e7,c88f6b80,...) at VOP_FSYNC_APV+0xc5 sync_vnode(c12a1bb4,c12a1ba0,3e8,78b,4e20,...) at sync_vnode+0x16b sched_sync(0,ec540d28,c0f88d50,3db,c88ec84c,...) at sched_sync+0x282 fork_exit(c0afc5b0,0,ec540d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec540d60, ebp = 0 --- Tracing command bufdaemon pid 9 tid 100064 td 0xc88f8000 sched_switch(c88f8000,0,104,1a3,8842e9b0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c88f8000,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a1848,54,c0f9c229,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a1848,c12a1850,54,c0f9c229,3e8,...) at _sleep+0x36a buf_daemon(0,ec53dd28,c0f88d50,3db,c88ecb10,...) at buf_daemon+0x175 fork_exit(c0ae0e10,0,ec53dd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53dd60, ebp = 0 --- Tracing command pagezero pid 8 tid 100063 td 0xc63d32e0 sched_switch(c63d32e0,0,104,1a3,5323d180,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d32e0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a9d5c,0,c0fcc066,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a9d5c,c12a8800,0,c0fcc066,493e0,...) at _sleep+0x36a vm_pagezero(0,ec53ad28,c0f88d50,3db,c88ed000,...) at vm_pagezero+0xdc fork_exit(c0ce2320,0,ec53ad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec53ad60, ebp = 0 --- Tracing command vmdaemon pid 7 tid 100062 td 0xc63d35c0 sched_switch(c63d35c0,0,104,1a3,2d0b2738,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,74,...) at mi_switch+0x1ff sleepq_switch(c63d35c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a9948,74,c0f9c229,0,0,...) at sleepq_wait+0x63 _sleep(c12a9948,c12a994c,74,c0f9c229,0,...) at _sleep+0x39c vm_daemon(0,ec537d28,c0f88d50,3db,c61ac2c4,...) at vm_daemon+0x59 fork_exit(c0cdbc30,0,ec537d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec537d60, ebp = 0 --- Tracing command pagedaemon pid 6 tid 100061 td 0xc63d38a0 sched_switch(c63d38a0,0,104,1a3,72426820,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c63d38a0,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c12a9910,54,c0f9c229,0,0,...) at sleepq_timedwait+0x6b _sleep(c12a9910,c12a8800,54,c0f9c229,1388,...) at _sleep+0x36a vm_pageout(0,ec534d28,c0f88d50,3db,c61ac588,...) at vm_pageout+0x2c7 fork_exit(c0cdccf0,0,ec534d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec534d60, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100060 td 0xc63d3b80 sched_switch(c63d3b80,0,104,1a3,6d00f848,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c63d3b80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c10f01d4,5c,c0de8c57,0,0,...) at sleepq_wait+0x63 _sleep(c10f01d4,c10f0218,5c,c0de8c57,0,...) at _sleep+0x39c xpt_scanner_thread(0,ec531d28,c0f88d50,3db,c61ac84c,...) at xpt_scanner_thread+0x47 fork_exit(c049d320,0,ec531d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec531d60, ebp = 0 --- Tracing command sctp_iterator pid 4 tid 100059 td 0xc640e000 sched_switch(c640e000,0,104,1a3,98c64448,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640e000,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c12a36d8,0,c0fb2bbf,0,0,...) at sleepq_wait+0x63 _sleep(c12a36d8,c12a36a0,0,c0fb2bbf,0,...) at _sleep+0x39c sctp_iterator_thread(0,ec52ed28,c0f88d50,3db,c61acb10,...) at sctp_iterator_thread+0x5c fork_exit(c0b8ea60,0,ec52ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52ed60, ebp = 0 --- Tracing command ctl_thrd pid 3 tid 100057 td 0xc640e5c0 sched_switch(c640e5c0,0,104,1a3,4c471850,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640e5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c645a000,5c,c0df368a,0,0,...) at sleepq_wait+0x63 _sleep(c645a000,c645a000,5c,c0df368a,0,...) at _sleep+0x39c ctl_work_thread(c645a000,ec528d28,c0f88d50,3db,c63a4000,...) at ctl_work_thread+0x22f6 fork_exit(c04c6720,c645a000,ec528d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec528d60, ebp = 0 --- Tracing command fdc0 pid 2 tid 100055 td 0xc640eb80 sched_switch(c640eb80,0,104,1a3,8a8b5f40,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c640eb80,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c634ee3c,5c,c0f80f1a,0,0,...) at sleepq_timedwait+0x6b _sleep(c634ee3c,c634eef0,5c,c0f80f1a,3e8,...) at _sleep+0x36a fdc_thread(c634ee00,ec522d28,c0f88d50,3db,c63a42c4,...) at fdc_thread+0x27d fork_exit(c0d61570,c634ee00,ec522d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec522d60, ebp = 0 --- Tracing command usb pid 15 tid 100051 td 0xc63d0000 sched_switch(c63d0000,0,104,1a3,4c418560,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d0000,0,c0f941e0,268,c63d0000,...) at sleepq_switch+0x162 sleepq_wait(c63ebd34,0,c5fbbcac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebd34,c63ebe4c,c0f672f1,74,c63ebd3c,...) at _cv_wait+0x260 usb_process(c63ebd2c,c5fbbd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebd2c,c5fbbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fbbd60, ebp = 0 --- Tracing command usb pid 15 tid 100050 td 0xc63d02e0 sched_switch(c63d02e0,0,104,1a3,c0dea9a8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d02e0,0,c0f941e0,268,c63d02e0,...) at sleepq_switch+0x162 sleepq_wait(c63ebd04,0,c5fb8cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebd04,c63ebe4c,c0f672f1,74,c63ebd0c,...) at _cv_wait+0x260 usb_process(c63ebcfc,c5fb8d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebcfc,c5fb8d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb8d60, ebp = 0 --- Tracing command usb pid 15 tid 100049 td 0xc63d05c0 sched_switch(c63d05c0,0,104,1a3,4bc3ec88,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d05c0,0,c0f941e0,268,c63d05c0,...) at sleepq_switch+0x162 sleepq_wait(c63ebcd4,0,c5fb5cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebcd4,c63ebe4c,c0f672f1,74,c63ebcdc,...) at _cv_wait+0x260 usb_process(c63ebccc,c5fb5d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebccc,c5fb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb5d60, ebp = 0 --- Tracing command usb pid 15 tid 100048 td 0xc63d08a0 sched_switch(c63d08a0,0,104,1a3,4bc3d4b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d08a0,0,c0f941e0,268,c63d08a0,...) at sleepq_switch+0x162 sleepq_wait(c63ebca4,0,c5fb2cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63ebca4,c63ebe4c,c0f672f1,74,c63ebcac,...) at _cv_wait+0x260 usb_process(c63ebc9c,c5fb2d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63ebc9c,c5fb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fb2d60, ebp = 0 --- Tracing command usb pid 15 tid 100047 td 0xc63d0b80 sched_switch(c63d0b80,0,104,1a3,4bc3bf20,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d0b80,0,c0f941e0,268,c63d0b80,...) at sleepq_switch+0x162 sleepq_wait(c63d7dac,0,c5faccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7dac,c63d7ec4,c0f672f1,74,c63d7db4,...) at _cv_wait+0x260 usb_process(c63d7da4,c5facd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7da4,c5facd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5facd60, ebp = 0 --- Tracing command usb pid 15 tid 100046 td 0xc63d2000 sched_switch(c63d2000,0,104,1a3,b10c0330,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d2000,0,c0f941e0,268,c63d2000,...) at sleepq_switch+0x162 sleepq_wait(c63d7d7c,0,c5fa9cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d7c,c63d7ec4,c0f672f1,74,c63d7d84,...) at _cv_wait+0x260 usb_process(c63d7d74,c5fa9d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d74,c5fa9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa9d60, ebp = 0 --- Tracing command usb pid 15 tid 100045 td 0xc63d22e0 sched_switch(c63d22e0,0,104,1a3,44abbec8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d22e0,0,c0f941e0,268,c63d22e0,...) at sleepq_switch+0x162 sleepq_wait(c63d7d4c,0,c5fa6cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d4c,c63d7ec4,c0f672f1,74,c63d7d54,...) at _cv_wait+0x260 usb_process(c63d7d44,c5fa6d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d44,c5fa6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa6d60, ebp = 0 --- Tracing command usb pid 15 tid 100044 td 0xc63d25c0 sched_switch(c63d25c0,0,104,1a3,44abad08,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d25c0,0,c0f941e0,268,c63d25c0,...) at sleepq_switch+0x162 sleepq_wait(c63d7d1c,0,c5fa3cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63d7d1c,c63d7ec4,c0f672f1,74,c63d7d24,...) at _cv_wait+0x260 usb_process(c63d7d14,c5fa3d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63d7d14,c5fa3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fa3d60, ebp = 0 --- Tracing command usb pid 15 tid 100042 td 0xc63d2b80 sched_switch(c63d2b80,0,104,1a3,44ab92b8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d2b80,0,c0f941e0,268,c63d2b80,...) at sleepq_switch+0x162 sleepq_wait(c63c0dac,0,c5f9ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0dac,c63c0ec4,c0f672f1,74,c63c0db4,...) at _cv_wait+0x260 usb_process(c63c0da4,c5f9cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0da4,c5f9cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f9cd60, ebp = 0 --- Tracing command usb pid 15 tid 100041 td 0xc63d3000 sched_switch(c63d3000,0,104,1a3,7a0e80f0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63d3000,0,c0f941e0,268,c63d3000,...) at sleepq_switch+0x162 sleepq_wait(c63c0d7c,0,c5f99cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d7c,c63c0ec4,c0f672f1,74,c63c0d84,...) at _cv_wait+0x260 usb_process(c63c0d74,c5f99d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d74,c5f99d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f99d60, ebp = 0 --- Tracing command usb pid 15 tid 100040 td 0xc63a6000 sched_switch(c63a6000,0,104,1a3,44311f50,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a6000,0,c0f941e0,268,c63a6000,...) at sleepq_switch+0x162 sleepq_wait(c63c0d4c,0,c5f96cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d4c,c63c0ec4,c0f672f1,74,c63c0d54,...) at _cv_wait+0x260 usb_process(c63c0d44,c5f96d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d44,c5f96d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f96d60, ebp = 0 --- Tracing command usb pid 15 tid 100039 td 0xc63a62e0 sched_switch(c63a62e0,0,104,1a3,44310b60,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a62e0,0,c0f941e0,268,c63a62e0,...) at sleepq_switch+0x162 sleepq_wait(c63c0d1c,0,c5f93cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63c0d1c,c63c0ec4,c0f672f1,74,c63c0d24,...) at _cv_wait+0x260 usb_process(c63c0d14,c5f93d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63c0d14,c5f93d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f93d60, ebp = 0 --- Tracing command usb pid 15 tid 100037 td 0xc63a68a0 sched_switch(c63a68a0,0,104,1a3,4430f338,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a68a0,0,c0f941e0,268,c63a68a0,...) at sleepq_switch+0x162 sleepq_wait(c63aedac,0,c5f8ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aedac,c63aeec4,c0f672f1,74,c63aedb4,...) at _cv_wait+0x260 usb_process(c63aeda4,c5f8cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aeda4,c5f8cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f8cd60, ebp = 0 --- Tracing command usb pid 15 tid 100036 td 0xc63a6b80 sched_switch(c63a6b80,0,104,1a3,7a0e59d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a6b80,0,c0f941e0,268,c63a6b80,...) at sleepq_switch+0x162 sleepq_wait(c63aed7c,0,c5f89cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed7c,c63aeec4,c0f672f1,74,c63aed84,...) at _cv_wait+0x260 usb_process(c63aed74,c5f89d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed74,c5f89d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f89d60, ebp = 0 --- Tracing command usb pid 15 tid 100035 td 0xc63a8000 sched_switch(c63a8000,0,104,1a3,43b6c608,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a8000,0,c0f941e0,268,c63a8000,...) at sleepq_switch+0x162 sleepq_wait(c63aed4c,0,c5f86cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed4c,c63aeec4,c0f672f1,74,c63aed54,...) at _cv_wait+0x260 usb_process(c63aed44,c5f86d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed44,c5f86d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f86d60, ebp = 0 --- Tracing command usb pid 15 tid 100034 td 0xc63a82e0 sched_switch(c63a82e0,0,104,1a3,43b6b178,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a82e0,0,c0f941e0,268,c63a82e0,...) at sleepq_switch+0x162 sleepq_wait(c63aed1c,0,c5f83cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c63aed1c,c63aeec4,c0f672f1,74,c63aed24,...) at _cv_wait+0x260 usb_process(c63aed14,c5f83d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c63aed14,c5f83d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f83d60, ebp = 0 --- Tracing command usb pid 15 tid 100032 td 0xc63a88a0 sched_switch(c63a88a0,0,104,1a3,43b699a8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a88a0,0,c0f941e0,268,c63a88a0,...) at sleepq_switch+0x162 sleepq_wait(c6395dac,0,c5f7ccac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395dac,c6395ec4,c0f672f1,74,c6395db4,...) at _cv_wait+0x260 usb_process(c6395da4,c5f7cd28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395da4,c5f7cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f7cd60, ebp = 0 --- Tracing command usb pid 15 tid 100031 td 0xc63a8b80 sched_switch(c63a8b80,0,104,1a3,c0de51e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c63a8b80,0,c0f941e0,268,c63a8b80,...) at sleepq_switch+0x162 sleepq_wait(c6395d7c,0,c5f79cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d7c,c6395ec4,c0f672f1,74,c6395d84,...) at _cv_wait+0x260 usb_process(c6395d74,c5f79d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d74,c5f79d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f79d60, ebp = 0 --- Tracing command usb pid 15 tid 100030 td 0xc632b2e0 sched_switch(c632b2e0,0,104,1a3,433a8c10,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632b2e0,0,c0f941e0,268,c632b2e0,...) at sleepq_switch+0x162 sleepq_wait(c6395d4c,0,c5f76cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d4c,c6395ec4,c0f672f1,74,c6395d54,...) at _cv_wait+0x260 usb_process(c6395d44,c5f76d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d44,c5f76d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f76d60, ebp = 0 --- Tracing command usb pid 15 tid 100029 td 0xc632b5c0 sched_switch(c632b5c0,0,104,1a3,433a77e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632b5c0,0,c0f941e0,268,c632b5c0,...) at sleepq_switch+0x162 sleepq_wait(c6395d1c,0,c5f73cac,1,0,...) at sleepq_wait+0x63 _cv_wait(c6395d1c,c6395ec4,c0f672f1,74,c6395d24,...) at _cv_wait+0x260 usb_process(c6395d14,c5f73d28,c0f88d50,3db,c63a4588,...) at usb_process+0x1a3 fork_exit(c08fda80,c6395d14,c5f73d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f73d60, ebp = 0 --- Tracing command yarrow pid 14 tid 100013 td 0xc61ae2e0 sched_switch(c61ae2e0,0,104,1a3,9f7927d0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae2e0,0,c0f941e0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c112a7e4,0,c0f80f1a,2,0,...) at sleepq_timedwait+0x6b _sleep(c112a7e4,0,0,c0f80f1a,64) at _sleep+0x36a pause(c0f80f1a,64,c0f42f30,111,0,...) at pause+0xa0 random_kthread(0,c5ed7d28,c0f88d50,3db,c61ab000,...) at random_kthread+0x1ef fork_exit(c0805cf0,0,c5ed7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed7d60, ebp = 0 --- Tracing command geom pid 13 tid 100011 td 0xc61ae8a0 sched_switch(c61ae8a0,0,104,1a3,7059260,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61ae8a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283e8,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283e8,c1128348,25c,c0f80f1a,0,...) at _sleep+0x39c g_io_schedule_down(c61ae8a0,0,c0f82ac1,6c,c5ecdd14,...) at g_io_schedule_down+0x5b g_down_procbody(0,c5ecdd28,c0f88d50,3db,c61ab2c4,...) at g_down_procbody+0x9d fork_exit(c09ef410,0,c5ecdd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecdd60, ebp = 0 --- Tracing command geom pid 13 tid 100010 td 0xc61aeb80 sched_switch(c61aeb80,0,104,1a3,705e350,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61aeb80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283e4,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283e4,c1128368,25c,c0f80f1a,0,...) at _sleep+0x39c g_io_schedule_up(c61aeb80,0,c0f82ac1,5f,c5ecad14,...) at g_io_schedule_up+0x134 g_up_procbody(0,c5ecad28,c0f88d50,3db,c61ab2c4,...) at g_up_procbody+0x9d fork_exit(c09ef4b0,0,c5ecad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ecad60, ebp = 0 --- Tracing command geom pid 13 tid 100009 td 0xc61f9000 sched_switch(c61f9000,0,104,1a3,3c4596e8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,5c,...) at mi_switch+0x1ff sleepq_switch(c61f9000,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c11283dc,5c,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c11283dc,c112830c,25c,c0f80f1a,0,...) at _sleep+0x39c g_run_events(c1138640,0,c0f82ac1,79,c5ec7d14,...) at g_run_events+0x54b g_event_procbody(0,c5ec7d28,c0f88d50,3db,c61ab2c4,...) at g_event_procbody+0x90 fork_exit(c09ef550,0,c5ec7d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec7d60, ebp = 0 --- Tracing command intr pid 12 tid 100056 td 0xc640e8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100054 td 0xc6411000 sched_switch(c6411000,0,109,1a3,4261880,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c6411000,...) at mi_switch+0x1ff ithread_loop(c61f86e0,ec515d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f86e0,ec515d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec515d60, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc64112e0 sched_switch(c64112e0,0,109,1a3,178c6c20,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c64112e0,...) at mi_switch+0x1ff ithread_loop(c61f86f0,c5fe5d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f86f0,c5fe5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe5d60, ebp = 0 --- Tracing command intr pid 12 tid 100052 td 0xc64115c0 sched_switch(c64115c0,0,109,1a3,467194c0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c64115c0,...) at mi_switch+0x1ff ithread_loop(c61f8700,c5fe1d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8700,c5fe1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5fe1d60, ebp = 0 --- Tracing command intr pid 12 tid 100043 td 0xc63d28a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xc63a65c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xc63a85c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xc632b8a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc632c8a0 sched_switch(c632c8a0,0,109,1a3,b107fbd0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632c8a0,...) at mi_switch+0x1ff ithread_loop(c61f8750,c5ef5d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8750,c5ef5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ef5d60, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc632cb80 sched_switch(c632cb80,0,109,1a3,fc8e310,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632cb80,...) at mi_switch+0x1ff ithread_loop(c61f8760,c5ef2d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8760,c5ef2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ef2d60, ebp = 0 --- Tracing command intr pid 12 tid 100020 td 0xc61f9b80 sched_switch(c61f9b80,0,109,1a3,b09f73f0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61f9b80,...) at mi_switch+0x1ff ithread_loop(c61f8770,c5eecd28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8770,c5eecd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eecd60, ebp = 0 --- Tracing command intr pid 12 tid 100014 td 0xc632b000 sched_switch(c632b000,0,109,1a3,178c98a0,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c632b000,...) at mi_switch+0x1ff ithread_loop(c61f8780,c5edad28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f8780,c5edad28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5edad60, ebp = 0 --- Tracing command intr pid 12 tid 100008 td 0xc61f92e0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100007 td 0xc61f95c0 sched_switch(c61f95c0,0,109,1a3,95f5b248,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61f95c0,...) at mi_switch+0x1ff ithread_loop(c61f87d0,c5ec1d28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f87d0,c5ec1d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ec1d60, ebp = 0 --- Tracing command intr pid 12 tid 100006 td 0xc61ad000 sched_switch(c61ad000,0,109,1a3,a01b67a8,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61ad000,...) at mi_switch+0x1ff ithread_loop(c61f87e0,c5ebed28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f87e0,c5ebed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebed60, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc61ad2e0 sched_switch(c61ad2e0,0,109,1a3,4c71e710,...) at sched_switch+0x3ec mi_switch(109,0,c0f8908b,554,c61ad2e0,...) at mi_switch+0x1ff ithread_loop(c61f87f0,c5ebbd28,c0f88d50,3db,c61ab588,...) at ithread_loop+0x1fb fork_exit(c0a31f50,c61f87f0,c5ebbd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ebbd60, ebp = 0 --- Tracing command idle pid 11 tid 100004 td 0xc61ad5c0 cpustop_handler(1,13,c5eb5c04,c0dc0cc6,f5,...) at cpustop_handler+0x28 ipi_nmi_handler(f5,0,0,c12d680c,c61ab84c,...) at ipi_nmi_handler+0x47 trap(c5eb5c10) at trap+0x36 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0da5555, esp = 0xc5eb5c50, ebp = 0xc5eb5c50 --- acpi_cpu_c1(c5eb5c6c,c10f07b0,1,301,5005,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(ffffffff,c5eb5ca0,c0db37e9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1138cc0,1,c5eb5ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c5eb5cc4,c0f90852,3c0,c61ad5c0,...) at cpu_idle+0x89 sched_idletd(0,c5eb5d28,c0f88d50,3db,c61ab84c,...) at sched_idletd+0x263 fork_exit(c0a82d70,0,c5eb5d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb5d60, ebp = 0 --- Tracing command idle pid 11 tid 100003 td 0xc61ad8a0 kdb_enter(c0e02aae,c0f927c6,c642d000,c5eb2b74,c0a916e3,...) at kdb_enter+0x3a kdb_alt_break_internal(c5eb2ba4,c08c927e,2,c642d058,c0da5555,...) at kdb_alt_break_internal+0x9a kdb_alt_break(2,c642d058,c0da5555,c642d080,c642d058,...) at kdb_alt_break+0x13 uart_intr(c642d000,0,c12d64e0,c5eb2be8,0,...) at uart_intr+0x1ae intr_event_handle(c61f2d80,c5eb2c10,c5eb2c34,0,c634d800,...) at intr_event_handle+0x76 intr_execute_handlers(c61f10d0,c5eb2c10,0,c5eb2c50,c0da9f55,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,c5eb2c10) at lapic_handle_intr+0x36 Xapic_isr1() at Xapic_isr1+0x35 --- interrupt, eip = 0xc0da5555, esp = 0xc5eb2c50, ebp = 0xc5eb2c50 --- acpi_cpu_c1(c5eb2c6c,c10f07b0,0,301,5005,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(ffffffff,c5eb2ca0,c0db37e9,0,ffffffff,...) at acpi_cpu_idle+0x122 cpu_idle_acpi(0,ffffffff,c1138640,1,c5eb2ce8,...) at cpu_idle_acpi+0x2f cpu_idle(0,c5eb2cc4,c0f90852,3c0,c61ad8a0,...) at cpu_idle+0x89 sched_idletd(0,c5eb2d28,c0f88d50,3db,c61ab84c,...) at sched_idletd+0x263 fork_exit(c0a82d70,0,c5eb2d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eb2d60, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc61adb80 sched_switch(c61adb80,0,104,1a3,5814fe78,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,6c,...) at mi_switch+0x1ff sleepq_switch(c61adb80,0,c0f941e0,1a7,6c,...) at sleepq_switch+0x162 sleepq_catch_signals(c0f941e0,160,0,100,100,...) at sleepq_catch_signals+0xf9 sleepq_wait_sig(c61abb10,6c,c0f9715f,100,0,...) at sleepq_wait_sig+0x17 _sleep(c61abb10,c61abb98,16c,c0f9715f,0,...) at _sleep+0x385 kern_wait(c61adb80,ffffffff,c5eaec60,0,0,...) at kern_wait+0x626 sys_wait4(c61adb80,c5eaecec,c0fe0b42,c0f96f96,286,...) at sys_wait4+0x3b syscall(c5eaed28) at syscall+0x2c7 Xint0x80_syscall() at Xint0x80_syscall+0x21 --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x80551d3, esp = 0xbfbfe8bc, ebp = 0xbfbfe8d8 --- Tracing command audit pid 10 tid 100001 td 0xc61ae000 sched_switch(c61ae000,0,104,1a3,4335fd78,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae000,0,c0f941e0,268,c61ae000,...) at sleepq_switch+0x162 sleepq_wait(c12a75e0,0,c5eabc8c,1,0,...) at sleepq_wait+0x63 _cv_wait(c12a75e0,c12a75c4,c0fbf92f,194,0,...) at _cv_wait+0x260 audit_worker(0,c5eabd28,c0f88d50,3db,c61ac000,...) at audit_worker+0x84 fork_exit(c0c5b830,0,c5eabd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eabd60, ebp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xc88f62e0 sched_switch(c88f62e0,0,104,1a3,ccbfff18,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c88f62e0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c64258c0,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c64258c0,c64258d8,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1308e60,ec549d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1308e60,ec549d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec549d60, ebp = 0 --- Tracing command kernel pid 0 tid 100058 td 0xc640e2e0 sched_switch(c640e2e0,0,104,1a3,95f6b7e0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c640e2e0,0,c0f941e0,28b,2,...) at sleepq_switch+0x162 sleepq_timedwait(c112a7e4,0,c0f80f1a,2,0,...) at sleepq_timedwait+0x6b _sleep(c112a7e4,0,0,c0f80f1a,bb8) at _sleep+0x36a pause(c0f80f1a,bb8,124,122,c1128548,...) at pause+0xa0 deadlkres(0,ec52bd28,c0f88d50,3db,c11284c0,...) at deadlkres+0x324 fork_exit(c0a11720,0,ec52bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xec52bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100027 td 0xc632bb80 sched_switch(c632bb80,0,104,1a3,4333ff28,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632bb80,0,c0f941e0,268,c632bb80,...) at sleepq_switch+0x162 sleepq_wait(c6301840,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301840,c6301858,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c634876c,c5f6cd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c634876c,c5f6cd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f6cd60, ebp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xc632c000 sched_switch(c632c000,0,104,1a3,4333ed48,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c000,0,c0f941e0,268,c632c000,...) at sleepq_switch+0x162 sleepq_wait(c6301880,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301880,c6301898,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c6383544,c5f69d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c6383544,c5f69d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f69d60, ebp = 0 --- Tracing command kernel pid 0 tid 100025 td 0xc632c2e0 sched_switch(c632c2e0,0,104,1a3,4333db90,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c2e0,0,c0f941e0,268,c632c2e0,...) at sleepq_switch+0x162 sleepq_wait(c6301940,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301940,c6301958,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c6348c6c,c5f3ed28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c6348c6c,c5f3ed28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3ed60, ebp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xc632c5c0 sched_switch(c632c5c0,0,104,1a3,73bb2c50,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632c5c0,0,c0f941e0,268,c632c5c0,...) at sleepq_switch+0x162 sleepq_wait(c6301980,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6301980,c6301998,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c636f544,c5f3bd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c636f544,c5f3bd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5f3bd60, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc61f98a0 sched_switch(c61f98a0,0,104,1a3,e14fe9f8,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61f98a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6301e80,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6301e80,c6301e98,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c11473ac,c5eefd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c11473ac,c5eefd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eefd60, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc632a000 sched_switch(c632a000,0,104,1a3,6d025b08,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a000,0,c0f941e0,268,c632a000,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee9d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee9d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee9d60, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc632a2e0 sched_switch(c632a2e0,0,104,1a3,6d024d90,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a2e0,0,c0f941e0,268,c632a2e0,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee6d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee6d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee6d60, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc632a5c0 sched_switch(c632a5c0,0,104,1a3,6d023f08,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a5c0,0,c0f941e0,268,c632a5c0,...) at sleepq_switch+0x162 sleepq_wait(c6302040,0,c0f8f499,c0f80f1a,0,...) at sleepq_wait+0x63 msleep_spin(c6302040,c6302058,c0f80f1a,0,c0f8caeb,...) at msleep_spin+0x259 taskqueue_thread_loop(c10f3438,c5ee3d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0x8f fork_exit(c0a9fbe0,c10f3438,c5ee3d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee3d60, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc632a8a0 sched_switch(c632a8a0,0,104,1a3,6d015fa0,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632a8a0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6302080,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6302080,c6302098,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1128e38,c5ee0d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1128e38,c5ee0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ee0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc632ab80 sched_switch(c632ab80,0,104,1a3,6d014d28,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c632ab80,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c63020c0,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c63020c0,c63020d8,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c12a7bc4,c5eddd28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c12a7bc4,c5eddd28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5eddd60, ebp = 0 --- Tracing command kernel pid 0 tid 100012 td 0xc61ae5c0 sched_switch(c61ae5c0,0,104,1a3,d6e68b50,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,0,...) at mi_switch+0x1ff sleepq_switch(c61ae5c0,0,c0f941e0,268,0,...) at sleepq_switch+0x162 sleepq_wait(c6191e80,0,c0f80f1a,0,0,...) at sleepq_wait+0x63 _sleep(c6191e80,c6191e98,0,c0f80f1a,0,...) at _sleep+0x39c taskqueue_thread_loop(c1145dc0,c5ed0d28,c0f88d50,3db,c11284c0,...) at taskqueue_thread_loop+0xb5 fork_exit(c0a9fbe0,c1145dc0,c5ed0d28) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5ed0d60, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc1128790 sched_switch(c1128790,0,104,1a3,54334730,...) at sched_switch+0x3ec mi_switch(104,0,c0f941e0,1f3,54,...) at mi_switch+0x1ff sleepq_switch(c1128790,0,c0f941e0,28b,0,...) at sleepq_switch+0x162 sleepq_timedwait(c11284c0,54,c0f9116c,0,0,...) at sleepq_timedwait+0x6b _sleep(c11284c0,0,54,c0f9116c,2710,...) at _sleep+0x36a scheduler(0,181ec00,181ec00,181e000,1825000,...) at scheduler+0x240 mi_startup() at mi_startup+0xac begin() at begin+0x2c db:0:allt> call doadump Physical memory: 2026 MB Dumping 186 MB: 171 155 139 123 107 91 75 59 43 27 11 Dump complete = 0 db:0:doadump> reset (kgdb) proc 45179 [Switching to thread 116 (Thread 100411)]#6 0xc0ae2393 in breadn_flags (vp=0xc8954828, blkno=Unhandled dwarf expression opcode 0x93 ) at /mnt/pho/src/sys/kern/vfs_bio.c:841 841 *bpp = bp = getblk(vp, blkno, size, 0, 0, flags); (kgdb) bt #0 sched_switch (td=0xc8ce72e0, newtd=0xc8c46b80, flags=260) at /mnt/pho/src/sys/kern/sched_ule.c:1860 #1 0xc0a655af in mi_switch (flags=260, newtd=0x0) at /mnt/pho/src/sys/kern/kern_synch.c:468 #2 0xc0a9ca42 in sleepq_switch (wchan=0xdfe33e20, pri=96) at /mnt/pho/src/sys/kern/subr_sleepqueue.c:538 #3 0xc0a9d773 in sleepq_wait (wchan=0xdfe33e20, pri=96) at /mnt/pho/src/sys/kern/subr_sleepqueue.c:617 #4 0xc0a4387f in __lockmgr_args (lk=0xdfe33e20, flags=530688, ilk=0xc8954920, wmesg=0xc0f868d4 "getblk", pri=96, timo=0, file=0xc0f9b8da "/mnt/pho/src/sys/kern/vfs_bio.c", line=2652) at /mnt/pho/src/sys/kern/kern_lock.c:219 #5 0xc0ae1b27 in getblk (vp=0xc8954828, blkno=224, size=16384, slpflag=0, slptimeo=0, flags=0) at lockmgr.h:97 #6 0xc0ae2393 in breadn_flags (vp=0xc8954828, blkno=Unhandled dwarf expression opcode 0x93 ) at /mnt/pho/src/sys/kern/vfs_bio.c:841 #7 0xc0c7bf5b in ffs_update (vp=0xc8fb2570, waitfor=1) at /mnt/pho/src/sys/ufs/ffs/ffs_inode.c:107 #8 0xc0ca6185 in ffs_syncvnode (vp=0xc8fb2570, waitfor=1) at /mnt/pho/src/sys/ufs/ffs/ffs_vnops.c:327 #9 0xc0c81e53 in ffs_snapremove (vp=0xc8fb2570) at /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1709 #10 0xc0c7cf67 in ffs_truncate (vp=0xc8fb2570, length=0, flags=3072, cred=0x0, td=0xc8ce72e0) at /mnt/pho/src/sys/ufs/ffs/ffs_inode.c:278 #11 0xc0cacc1b in ufs_inactive (ap=0xeec91adc) at /mnt/pho/src/sys/ufs/ufs/ufs_inode.c:124 #12 0xc0de17e5 in VOP_INACTIVE_APV (vop=0xc10de1c0, a=0xeec91adc) at vnode_if.c:1863 #13 0xc0afb0b1 in vinactive (vp=0xc8fb2570, td=0xc8ce72e0) at vnode_if.h:807 #14 0xc0afdbfc in vputx (vp=0xc8fb2570, func=2) at /mnt/pho/src/sys/kern/vfs_subr.c:2298 #15 0xc0afdc70 in vput (vp=0xc8fb2570) at /mnt/pho/src/sys/kern/vfs_subr.c:2325 #16 0xc0b003e3 in kern_unlinkat (td=0xc8ce72e0, fd=-100, path=0xbfbfebba
, pathseg=UIO_USERSPACE, oldinum=0) at /mnt/pho/src/sys/kern/vfs_syscalls.c:1966 #17 0xc0b0045f in kern_unlink (td=0xc8ce72e0, path=0xbfbfebba
, pathseg=UIO_USERSPACE) at /mnt/pho/src/sys/kern/vfs_syscalls.c:1898 #18 0xc0b00492 in sys_unlink (td=0xc8ce72e0, uap=0xeec91cec) at /mnt/pho/src/sys/kern/vfs_syscalls.c:1868 #19 0xc0dc08c7 in syscall (frame=0xeec91d28) at subr_syscall.c:135 #20 0xc0da9bf1 in Xint0x80_syscall () at /mnt/pho/src/sys/i386/i386/exception.s:266 #21 0x00000033 in ?? () (kgdb) f 9 #9 0xc0c81e53 in ffs_snapremove (vp=0xc8fb2570) at /mnt/pho/src/sys/ufs/ffs/ffs_snapshot.c:1709 1709 ffs_syncvnode(vp, MNT_WAIT); (kgdb) l 1704 * The dirtied indirects must be written out before 1705 * softdep_setup_freeblocks() is called. Otherwise indir_trunc() 1706 * may find indirect pointers using the magic BLK_* values. 1707 */ 1708 if (DOINGSOFTDEP(vp)) 1709 ffs_syncvnode(vp, MNT_WAIT); 1710 #ifdef QUOTA 1711 /* 1712 * Reenable disk quotas for ex-snapshot file. 1713 */ (kgdb) p *vp $1 = {v_type = VREG, v_tag = 0xc0f84bd4 "ufs", v_op = 0xc10ddc40, v_data = 0xc90c89f8, v_mount = 0xc9299b50, v_nmntvnodes = {tqe_next = 0xcb37d570, tqe_prev = 0xc9d3adac}, v_un = { vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0xc87259d4}, v_hash = 6, v_cache_src = {lh_first = 0x0}, v_cache_dst = { tqh_first = 0x0, tqh_last = 0xc8fb25a0}, v_cache_dd = 0x0, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, v_lock = {lock_object = {lo_name = 0xc0f84bd4 "ufs", lo_flags = 91947008, lo_data = 0, lo_witness = 0xc615b8e8}, lk_lock = 3368973028, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96, lk_stack = {depth = 13, pcs = {3231988062, 3234338975, 3234320231, 3234515995, 3235780581, 3232739505, 3232750588, 3232750704, 3232760803, 3232760927, 3232760978, 3235645639, 3235552241, 0, 0, 0, 0, 0}}}, v_interlock = {lock_object = { lo_name = 0xc0f964d8 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc61541e0}, mtx_lock = 4}, v_vnlock = 0xc8fb25c8, v_holdcnt = 83, v_usecount = 0, v_iflag = 2048, v_vflag = 128, v_writecount = 0, v_freelist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_mtx = {lock_object = {lo_name = 0xc0f9f322 "bufobj interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xc6159ce0}, mtx_lock = 4}, bo_clean = {bv_hd = {tqh_first = 0xdfe1c660, tqh_last = 0xdfd1dff8}, bv_root = 0xdfd7cbe0, bv_cnt = 70}, bo_dirty = {bv_hd = {tqh_first = 0xdfddf900, tqh_last = 0xdfe598d8}, bv_root = 0xdfe62940, bv_cnt = 10}, bo_numoutput = 0, bo_flag = 1, bo_ops = 0xc10caec0, bo_bsize = 16384, bo_object = 0xca3f3b40, bo_synclist = {le_next = 0xc8954a7c, le_prev = 0xc8eceac4}, bo_private = 0xc8fb2570, __bo_vnode = 0xc8fb2570}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0} (kgdb) $ svn diff -x -p sys Index: sys/ufs/ufs/inode.h =================================================================== --- sys/ufs/ufs/inode.h (revision 231644) +++ sys/ufs/ufs/inode.h (working copy) @@ -158,6 +158,7 @@ struct inode { #define SHORTLINK(ip) \ (((ip)->i_ump->um_fstype == UFS1) ? \ (caddr_t)(ip)->i_din1->di_db : (caddr_t)(ip)->i_din2->di_db) +#define IS_SNAPSHOT(ip) ((ip)->i_flags & SF_SNAPSHOT) /* * Structure used to pass around logical block paths generated by Index: sys/ufs/ffs/ffs_softdep.c =================================================================== --- sys/ufs/ffs/ffs_softdep.c (revision 231644) +++ sys/ufs/ffs/ffs_softdep.c (working copy) @@ -2827,7 +2827,12 @@ softdep_prealloc(vp, waitok) { struct ufsmount *ump; - if (DOINGSUJ(vp) == 0) + /* + * Nothing to do if we are not running journaled soft updates. + * If we currently hold the snapshot lock, we must avoid handling + * other resources that could cause deadlock. + */ + if (DOINGSUJ(vp) == 0 || IS_SNAPSHOT(VTOI(vp))) return (0); ump = VFSTOUFS(vp->v_mount); ACQUIRE_LOCK(&lk); @@ -2873,7 +2878,12 @@ softdep_prelink(dvp, vp) ump = VFSTOUFS(dvp->v_mount); mtx_assert(&lk, MA_OWNED); - if (journal_space(ump, 0)) + /* + * Nothing to do if we have sufficient journal space. + * If we currently hold the snapshot lock, we must avoid + * handling other resources that could cause deadlock. + */ + if (journal_space(ump, 0) || (vp && IS_SNAPSHOT(VTOI(vp)))) return; stat_journal_low++; FREE_LOCK(&lk); @@ -4304,11 +4314,15 @@ inodedep_lookup_ip(ip) struct inode *ip; { struct inodedep *inodedep; + int dflags; KASSERT(ip->i_nlink >= ip->i_effnlink, ("inodedep_lookup_ip: bad delta")); - (void) inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, - DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, + &inodedep); inodedep->id_nlinkdelta = ip->i_nlink - ip->i_effnlink; return (inodedep); @@ -4696,7 +4710,7 @@ softdep_setup_inomapdep(bp, ip, newinum, mode) * the cylinder group map from which it was allocated. */ ACQUIRE_LOCK(&lk); - if ((inodedep_lookup(mp, newinum, DEPALLOC|NODELAY, &inodedep))) + if ((inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep))) panic("softdep_setup_inomapdep: dependency %p for new" "inode already exists", inodedep); bmsafemap = bmsafemap_lookup(mp, bp, ino_to_cg(fs, newinum)); @@ -5437,6 +5451,7 @@ softdep_setup_allocindir_page(ip, lbn, bp, ptrno, struct allocindir *aip; struct pagedep *pagedep; struct mount *mp; + int dflags; if (lbn != nbp->b_lblkno) panic("softdep_setup_allocindir_page: lbn %jd != lblkno %jd", @@ -5444,7 +5459,10 @@ softdep_setup_allocindir_page(ip, lbn, bp, ptrno, ASSERT_VOP_LOCKED(ITOV(ip), "softdep_setup_allocindir_page"); mp = UFSTOVFS(ip->i_ump); aip = newallocindir(ip, ptrno, newblkno, oldblkno, lbn); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); /* * If we are allocating a directory page, then we must * allocate an associated pagedep to track additions and @@ -5474,11 +5492,15 @@ softdep_setup_allocindir_meta(nbp, ip, bp, ptrno, struct inodedep *inodedep; struct allocindir *aip; ufs_lbn_t lbn; + int dflags; lbn = nbp->b_lblkno; ASSERT_VOP_LOCKED(ITOV(ip), "softdep_setup_allocindir_meta"); aip = newallocindir(ip, ptrno, newblkno, 0, lbn); - inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, &inodedep); WORKLIST_INSERT(&nbp->b_dep, &aip->ai_block.nb_list); if (setup_allocindir_phase2(bp, ip, inodedep, aip, lbn)) panic("softdep_setup_allocindir_meta: Block already existed"); @@ -6085,11 +6107,7 @@ softdep_journal_freeblocks(ip, cred, length, flags struct mount *mp; ufs2_daddr_t extblocks, datablocks; ufs_lbn_t tmpval, lbn, lastlbn; - int frags; - int lastoff, iboff; - int allocblock; - int error, i; - int needj; + int frags, lastoff, iboff, allocblock, needj, dflags, error, i; fs = ip->i_fs; mp = UFSTOVFS(ip->i_ump); @@ -6107,7 +6125,10 @@ softdep_journal_freeblocks(ip, cred, length, flags * we don't need to journal the block frees. The canceled journals * for the allocations will suffice. */ - inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & (UNLINKED | DEPCOMPLETE)) == UNLINKED && length == 0) needj = 0; @@ -6232,7 +6253,7 @@ softdep_journal_freeblocks(ip, cred, length, flags *((struct ufs2_dinode *)bp->b_data + ino_to_fsbo(fs, ip->i_number)) = *ip->i_din2; ACQUIRE_LOCK(&lk); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & IOSTARTED) != 0) panic("softdep_setup_freeblocks: inode busy"); /* @@ -6310,7 +6331,7 @@ softdep_journal_freeblocks(ip, cred, length, flags } ACQUIRE_LOCK(&lk); - inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + inodedep_lookup(mp, ip->i_number, dflags, &inodedep); TAILQ_INSERT_TAIL(&inodedep->id_freeblklst, freeblks, fb_next); freeblks->fb_state |= DEPCOMPLETE | ONDEPLIST; /* @@ -6398,7 +6419,7 @@ softdep_setup_freeblocks(ip, length, flags) struct fs *fs; ufs2_daddr_t extblocks, datablocks; struct mount *mp; - int i, delay, error; + int i, delay, error, dflags; ufs_lbn_t tmpval; ufs_lbn_t lbn; @@ -6463,7 +6484,10 @@ softdep_setup_freeblocks(ip, length, flags) * Find and eliminate any inode dependencies. */ ACQUIRE_LOCK(&lk); - (void) inodedep_lookup(mp, ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + (void) inodedep_lookup(mp, ip->i_number, dflags, &inodedep); if ((inodedep->id_state & IOSTARTED) != 0) panic("softdep_setup_freeblocks: inode busy"); /* @@ -8029,7 +8053,7 @@ softdep_setup_directory_add(bp, dp, diroffset, new dap->da_pagedep = pagedep; LIST_INSERT_HEAD(&pagedep->pd_diraddhd[DIRADDHASH(offset)], dap, da_pdlist); - inodedep_lookup(mp, newinum, DEPALLOC, &inodedep); + inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep); /* * If we're journaling, link the diradd into the jaddref so it * may be completed after the journal entry is written. Otherwise, @@ -8631,8 +8655,7 @@ newdirrem(bp, dp, ip, isrmdir, prevdirremp) * the number of freefile and freeblks structures. */ ACQUIRE_LOCK(&lk); - if (!(ip->i_flags & SF_SNAPSHOT) && - dep_current[D_DIRREM] > max_softdeps / 2) + if (!IS_SNAPSHOT(ip) && dep_current[D_DIRREM] > max_softdeps / 2) (void) request_cleanup(ITOV(dp)->v_mount, FLUSH_BLOCKS); FREE_LOCK(&lk); dirrem = malloc(sizeof(struct dirrem), @@ -8866,11 +8889,11 @@ softdep_setup_directory_change(bp, dp, ip, newinum /* * Lookup the jaddref for this journal entry. We must finish * initializing it and make the diradd write dependent on it. - * If we're not journaling Put it on the id_bufwait list if the inode - * is not yet written. If it is written, do the post-inode write - * processing to put it on the id_pendinghd list. + * If we're not journaling, put it on the id_bufwait list if the + * inode is not yet written. If it is written, do the post-inode + * write processing to put it on the id_pendinghd list. */ - inodedep_lookup(mp, newinum, DEPALLOC, &inodedep); + inodedep_lookup(mp, newinum, DEPALLOC | NODELAY, &inodedep); if (MOUNTEDSUJ(mp)) { jaddref = (struct jaddref *)TAILQ_LAST(&inodedep->id_inoreflst, inoreflst); @@ -8912,9 +8935,13 @@ softdep_change_linkcnt(ip) struct inode *ip; /* the inode with the increased link count */ { struct inodedep *inodedep; + int dflags; ACQUIRE_LOCK(&lk); - inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, DEPALLOC, &inodedep); + dflags = DEPALLOC; + if (IS_SNAPSHOT(ip)) + dflags |= NODELAY; + inodedep_lookup(UFSTOVFS(ip->i_ump), ip->i_number, dflags, &inodedep); if (ip->i_nlink < ip->i_effnlink) panic("softdep_change_linkcnt: bad delta"); inodedep->id_nlinkdelta = ip->i_nlink - ip->i_effnlink; @@ -12544,22 +12571,25 @@ softdep_request_cleanup(fs, vp, cred, resource) ufs2_daddr_t needed; int error; - mp = vp->v_mount; - ump = VFSTOUFS(mp); - mtx_assert(UFS_MTX(ump), MA_OWNED); - if (resource == FLUSH_BLOCKS_WAIT) - stat_cleanup_blkrequests += 1; - else - stat_cleanup_inorequests += 1; - /* * If we are being called because of a process doing a * copy-on-write, then it is not safe to process any * worklist items as we will recurse into the copyonwrite * routine. This will result in an incoherent snapshot. + * If the vnode that we hold is a snapshot, we must avoid + * handling other resources that could cause deadlock. */ - if (curthread->td_pflags & TDP_COWINPROGRESS) + if ((curthread->td_pflags & TDP_COWINPROGRESS) || IS_SNAPSHOT(VTOI(vp))) return (0); + + if (resource == FLUSH_BLOCKS_WAIT) + stat_cleanup_blkrequests += 1; + else + stat_cleanup_inorequests += 1; + + mp = vp->v_mount; + ump = VFSTOUFS(mp); + mtx_assert(UFS_MTX(ump), MA_OWNED); UFS_UNLOCK(ump); error = ffs_update(vp, 1); if (error != 0) { Index: sys/ufs/ffs/ffs_vnops.c =================================================================== --- sys/ufs/ffs/ffs_vnops.c (revision 231644) +++ sys/ufs/ffs/ffs_vnops.c (working copy) @@ -216,8 +216,10 @@ ffs_syncvnode(struct vnode *vp, int waitfor) struct buf *bp; struct buf *nbp; ufs_lbn_t lbn; - int error, wait, passes; + int error, wait, passes, noupdate; + noupdate = waitfor & NO_INO_UPDT; + waitfor &= ~NO_INO_UPDT; ip = VTOI(vp); ip->i_flag &= ~IN_NEEDSYNC; bo = &vp->v_bufobj; @@ -300,7 +302,10 @@ next: } if (waitfor != MNT_WAIT) { BO_UNLOCK(bo); - return (ffs_update(vp, waitfor)); + if (noupdate) + return (0); + else + return (ffs_update(vp, waitfor)); } /* Drain IO to see if we're done. */ bufobj_wwait(bo, 0, 0); @@ -317,7 +322,7 @@ next: */ if (bo->bo_dirty.bv_cnt > 0) { /* Write the inode after sync passes to flush deps. */ - if (wait && DOINGSOFTDEP(vp)) { + if (wait && DOINGSOFTDEP(vp) && noupdate == 0) { BO_UNLOCK(bo); ffs_update(vp, MNT_WAIT); BO_LOCK(bo); @@ -332,7 +337,9 @@ next: #endif } BO_UNLOCK(bo); - error = ffs_update(vp, MNT_WAIT); + error = 0; + if (noupdate == 0) + error = ffs_update(vp, MNT_WAIT); if (DOINGSUJ(vp)) softdep_journal_fsync(VTOI(vp)); return (error); Index: sys/ufs/ffs/ffs_extern.h =================================================================== --- sys/ufs/ffs/ffs_extern.h (revision 231644) +++ sys/ufs/ffs/ffs_extern.h (working copy) @@ -167,6 +167,13 @@ void softdep_freework(struct workhead *); #define FLUSH_INODES_WAIT 2 #define FLUSH_BLOCKS 3 #define FLUSH_BLOCKS_WAIT 4 +/* + * Flag to ffs_syncinode() to request flushing of data only, + * but skip the ffs_update() on the inode itself. Used to avoid + * deadlock when flushing snapshot inodes while holding snaplk. + * Avoid bit conflicts with MNT_WAIT values in sys/mount.h + */ +#define NO_INO_UPDT 0x10 int ffs_rdonly(struct inode *); Index: sys/ufs/ffs/ffs_inode.c =================================================================== --- sys/ufs/ffs/ffs_inode.c (revision 231644) +++ sys/ufs/ffs/ffs_inode.c (working copy) @@ -81,7 +81,7 @@ ffs_update(vp, waitfor) struct fs *fs; struct buf *bp; struct inode *ip; - int error; + int flags, error; ASSERT_VOP_ELOCKED(vp, "ffs_update"); ufs_itimes(vp); @@ -92,11 +92,32 @@ ffs_update(vp, waitfor) fs = ip->i_fs; if (fs->fs_ronly && ip->i_ump->um_fsckpid == 0) return (0); - error = bread(ip->i_devvp, fsbtodb(fs, ino_to_fsba(fs, ip->i_number)), - (int)fs->fs_bsize, NOCRED, &bp); - if (error) { - brelse(bp); - return (error); + /* + * If we are updating a snapshot and another process is currently + * writing the buffer containing the inode for this snapshot then + * a deadlock can occur when it tries to check the snapshot to see + * if that block needs to be copied. Thus when updating a snapshot + * we check to see if the buffer is already locked, and if it is + * we drop the snapshot lock until the buffer has been written + * and is available to us. + */ + flags = 0; + if (IS_SNAPSHOT(ip)) + flags = GB_LOCK_NOWAIT; + error = breadn_flags(ip->i_devvp, + fsbtodb(fs, ino_to_fsba(fs, ip->i_number)), + (int) fs->fs_bsize, 0, 0, 0, NOCRED, flags, &bp); + if (error != 0) { + if (error != EBUSY) { + brelse(bp); + return (error); + } + KASSERT((IS_SNAPSHOT(ip)), ("EBUSY from non-snapshot")); + VOP_UNLOCK(vp, 0); + (void) bread(ip->i_devvp, + fsbtodb(fs, ino_to_fsba(fs, ip->i_number)), + (int) fs->fs_bsize, NOCRED, &bp); + vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); } if (DOINGSOFTDEP(vp)) softdep_update_inodeblock(ip, bp, waitfor); Index: sys/ufs/ffs/ffs_snapshot.c =================================================================== --- sys/ufs/ffs/ffs_snapshot.c (revision 231644) +++ sys/ufs/ffs/ffs_snapshot.c (working copy) @@ -203,6 +203,7 @@ ffs_snapshot(mp, snapfile) ufs2_daddr_t numblks, blkno, *blkp, *snapblklist; int error, cg, snaploc; int i, size, len, loc; + ufs2_daddr_t blockno; uint64_t flag; struct timespec starttime = {0, 0}, endtime; char saved_nice = 0; @@ -225,18 +226,10 @@ ffs_snapshot(mp, snapfile) ump = VFSTOUFS(mp); fs = ump->um_fs; sn = NULL; - /* - * At the moment, journaled soft updates cannot support - * taking snapshots. - */ - if (MOUNTEDSUJ(mp)) { - vfs_mount_error(mp, "%s: Snapshots are not yet supported when " - "running with journaled soft updates", fs->fs_fsmnt); - return (EOPNOTSUPP); - } MNT_ILOCK(mp); flag = mp->mnt_flag; MNT_IUNLOCK(mp); + /* * Need to serialize access to snapshot code per filesystem. */ @@ -815,21 +808,26 @@ out1: if (space != NULL) free(space, M_UFSMNT); /* - * If another process is currently writing the buffer containing - * the inode for this snapshot then a deadlock can occur. Drop - * the snapshot lock until the buffer has been written. + * Preallocate all the direct blocks in the snapshot inode so + * that we never have to write the inode itself to commit an + * update to the contents of the snapshot. Note that once + * created, the size of the snapshot will never change, so + * there will never be a need to write the inode except to + * update the non-integrity-critical time fields and + * allocated-block count. */ - VREF(vp); /* Protect against ffs_snapgone() */ - VOP_UNLOCK(vp, 0); - (void) bread(ip->i_devvp, - fsbtodb(fs, ino_to_fsba(fs, ip->i_number)), - (int) fs->fs_bsize, NOCRED, &nbp); - brelse(nbp); - vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - if (ip->i_effnlink == 0) - error = ENOENT; /* Snapshot file unlinked */ - else - vrele(vp); /* Drop extra reference */ + for (blockno = 0; blockno < NDADDR; blockno++) { + if (DIP(ip, i_db[blockno]) != 0) + continue; + error = UFS_BALLOC(vp, lblktosize(fs, blockno), + fs->fs_bsize, KERNCRED, BA_CLRBUF, &bp); + if (error) + break; + error = readblock(vp, bp, blockno); + bawrite(bp); + if (error != 0) + break; + } done: free(copy_fs->fs_csp, M_UFSMNT); free(copy_fs, M_UFSMNT); @@ -1902,7 +1900,7 @@ retry: bawrite(cbp); if ((vtype == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); continue; } /* @@ -1913,7 +1911,7 @@ retry: bawrite(cbp); if ((vtype == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); break; } savedcbp = cbp; @@ -1931,7 +1929,7 @@ retry: bawrite(savedcbp); if ((vtype == VDIR || dopersistence) && VTOI(vp)->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); } /* * If we have been unable to allocate a block in which to do @@ -2395,7 +2393,7 @@ ffs_copyonwrite(devvp, bp) bawrite(cbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; continue; @@ -2408,7 +2406,7 @@ ffs_copyonwrite(devvp, bp) bawrite(cbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && ip->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; break; @@ -2428,7 +2426,7 @@ ffs_copyonwrite(devvp, bp) bawrite(savedcbp); if ((devvp == bp->b_vp || bp->b_vp->v_type == VDIR || dopersistence) && VTOI(vp)->i_effnlink > 0) - (void) ffs_syncvnode(vp, MNT_WAIT); + (void) ffs_syncvnode(vp, MNT_WAIT|NO_INO_UPDT); else launched_async_io = 1; } @@ -2478,7 +2476,7 @@ ffs_sync_snap(mp, waitfor) } TAILQ_FOREACH(ip, &sn->sn_head, i_nextsnap) { vp = ITOV(ip); - ffs_syncvnode(vp, waitfor); + ffs_syncvnode(vp, waitfor|NO_INO_UPDT); } lockmgr(&sn->sn_lock, LK_RELEASE, NULL); } Index: sys/kern/vfs_bio.c =================================================================== --- sys/kern/vfs_bio.c (revision 231644) +++ sys/kern/vfs_bio.c (working copy) @@ -782,22 +782,7 @@ bremfreel(struct buf *bp) } } - /* - * Get a buffer with the specified data. Look in the cache first. We - * must clear BIO_ERROR and B_INVAL prior to initiating I/O. If B_CACHE - * is set, the buffer is valid and we do not have to do anything ( see - * getblk() ). This is really just a special case of breadn(). - */ -int -bread(struct vnode * vp, daddr_t blkno, int size, struct ucred * cred, - struct buf **bpp) -{ - - return (breadn(vp, blkno, size, 0, 0, 0, cred, bpp)); -} - -/* * Attempt to initiate asynchronous I/O on read-ahead blocks. We must * clear BIO_ERROR and B_INVAL prior to initiating I/O . If B_CACHE is set, * the buffer is valid and we do not have to do anything. @@ -834,19 +819,28 @@ breada(struct vnode * vp, daddr_t * rablkno, int * } /* - * Operates like bread, but also starts asynchronous I/O on - * read-ahead blocks. + * Entry point for bread() and breadn() via #defines in sys/buf.h. + * + * Get a buffer with the specified data. Look in the cache first. We + * must clear BIO_ERROR and B_INVAL prior to initiating I/O. If B_CACHE + * is set, the buffer is valid and we do not have to do anything, see + * getblk(). Also starts asynchronous I/O on read-ahead blocks. */ int -breadn(struct vnode * vp, daddr_t blkno, int size, - daddr_t * rablkno, int *rabsize, - int cnt, struct ucred * cred, struct buf **bpp) +breadn_flags(struct vnode * vp, daddr_t blkno, int size, + daddr_t * rablkno, int *rabsize, int cnt, + struct ucred * cred, int flags, struct buf **bpp) { struct buf *bp; int rv = 0, readwait = 0; CTR3(KTR_BUF, "breadn(%p, %jd, %d)", vp, blkno, size); - *bpp = bp = getblk(vp, blkno, size, 0, 0, 0); + /* + * Can only return NULL if GB_LOCK_NOWAIT flag is specified. + */ + *bpp = bp = getblk(vp, blkno, size, 0, 0, flags); + if (bp == NULL) + return (EBUSY); /* if not found in cache, do some I/O */ if ((bp->b_flags & B_CACHE) == 0) { Index: sys/sys/buf.h =================================================================== --- sys/sys/buf.h (revision 231644) +++ sys/sys/buf.h (working copy) @@ -479,10 +479,13 @@ void bwillwrite(void); int buf_dirty_count_severe(void); void bremfree(struct buf *); void bremfreef(struct buf *); /* XXX Force bremfree, only for nfs. */ -int bread(struct vnode *, daddr_t, int, struct ucred *, struct buf **); +#define bread(vp, blkno, size, cred, bpp) \ + breadn_flags(vp, blkno, size, 0, 0, 0, cred, 0, bpp) +#define breadn(vp, blkno, size, rablkno, rabsize, cnt, cred, bpp) \ + breadn_flags(vp, blkno, size, rablkno, rabsize, cnt, cred, 0, bpp) +int breadn_flags(struct vnode *, daddr_t, int, daddr_t *, int *, int, + struct ucred *, int, struct buf **); void breada(struct vnode *, daddr_t *, int *, int, struct ucred *); -int breadn(struct vnode *, daddr_t, int, daddr_t *, int *, int, - struct ucred *, struct buf **); void bdwrite(struct buf *); void bawrite(struct buf *); void bdirty(struct buf *); -- Test scenario: suj26.sh # ./all.sh suj26.sh Local config: leopard3 20120215 09:47:02 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 10:17:05 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 10:47:40 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 11:18:26 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 11:49:06 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 12:20:06 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 12:51:19 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 13:24:13 all: suj26.sh Local config: leopard3 Local config: leopard3 20120215 13:55:28 all: suj26.sh Local config: leopard3 load: 0.00 cmd: rm 45179 [getblk] 1639.08r 0.00u 0.00s 0% 116k load: 0.00 cmd: rm 45179 [getblk] 2892.51r 0.00u 0.00s 0% 116k load: 0.00 cmd: rm 45179 [getblk] 3734.09r 0.00u 0.00s 0% 116k