GENERIC HEAD from 2019-07-09 07:24:18 UTC, r349856M, vmcore.1143 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2019 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-CURRENT #0 r349856M: Tue Jul 9 23:23:58 CEST 2019 pho@x4.osted.lan:/usr/src/sys/i386/compile/PHO i386 FreeBSD clang version 8.0.1 (branches/release_80 363030) (based on LLVM 8.0.1) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Entering uma_startup with 9 boot pages configured startup_alloc from "UMA Kegs", 8 boot pages left startup_alloc from "UMA Zones", 7 boot pages left startup_alloc from "UMA Hash", 6 boot pages left startup_alloc from "vmem btag", 5 boot pages left startup_alloc from "UMA Zones", 4 boot pages left startup_alloc from "MAP", 3 boot pages left startup_alloc from "KMAP ENTRY", 2 boot pages left startup_alloc from "vmem", 1 boot pages left Entering uma_startup2 with 0 boot pages left VT(vga): resolution 640x480 CPU: AMD Phenom(tm) 9150e Quad-Core Processor (1800.02-MHz 686-class CPU) Origin="AuthenticAMD" Id=0x100f23 Family=0x10 Model=0x2 Stepping=3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff SVM: NP,NAsids=64 TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8261382144 (7878 MB) : Trying to mount root from ufs:/dev/ada0s1a [rw]... WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered Setting hostuuid: 00000000-0000-0000-0000-00218515337d. Setting hostid: 0x6b64ac17. warning: total configured swap (4194304 pages) exceeds maximum recommended amount (2009920 pages). warning: increase kern.maxswzone or reduce amount of swap. Starting file system checks: /dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1a: clean, 605281 free (2633 frags, 75331 blocks, 0.3% fragmentation) /dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1f: clean, 762227 free (10891 frags, 93917 blocks, 1.1% fragmentation) /dev/label/tmp: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/label/tmp: clean, 28205842 free (36442 frags, 3521175 blocks, 0.0% fragmentation) /dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1d: clean, 1282478 free (296470 frags, 123251 blocks, 2.9% fragmentation) /dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/ada0s1e: clean, 2985673 free (159945 frags, 353216 blocks, 0.6% fragmentation) Mounting local filesystems:. ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/arduino/lib /usr/local/lib/compat /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc7 /usr/local/lib/gcc8 /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.28/mach/CORE /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/qt5 /usr/local/llvm37/lib /usr/local/llvm39/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Setting hostname: x4.osted.lan. Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED Feeding entropy: . uhub3: 10 ports with 10 removable, self powered lo0: link state changed to UP re0: link state changed to DOWN Starting Network: lo0 re0. lo0: flags=8049 metric 0 mtu 16384 options=680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21 re0: flags=8843 metric 0 mtu 1500 options=8209b ether 00:21:85:15:33:7d inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (none) status: no carrier nd6 options=29 Starting devd. re0: link state changed to UP add host 127.0.0.1: gateway lo0 fib 0: route already in table add net default: gateway 192.168.1.1 add host ::1: gateway lo0 fib 0: route already in table add net fe80::: gateway ::1 add net ff02::: gateway ::1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 Creating and/or trimming log files. Starting syslogd. Starting rpcbind. NFS access cache time=60 Starting watchdogd. No core dumps found. lock order reversal: 1st 0x11c583c8 bufwait (bufwait) @ kern/vfs_bio.c:3904 2nd 0x2167a000 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:289 stack backtrace: #0 0x102fd01 at witness_debugger+0x61 #1 0x102fad9 at witness_checkorder+0x8a9 #2 0xfd6d2d at _sx_xlock+0x4d #3 0x12a3dd4 at ufsdirhash_move+0x44 #4 0x12a5f41 at ufs_direnter+0x2e1 #5 0x12adb0c at ufs_mkdir+0x78c #6 0x156201d at VOP_MKDIR_APV+0x6d #7 0x109ab93 at kern_mkdirat+0x173 #8 0x109aa19 at sys_mkdir+0x19 #9 0x1531999 at syscall+0x2d9 #10 0xffc033b7 at _end+0xfdc3b0bf Clearing /tmp (X related). Starting nfsuserd. Starting mountd. Starting nfsd. Starting statd. Starting lockd. Recovering vi editor sessions:. Updating motd:. Mounting late filesystems:. Starting ntpd. Starting powerd. Starting smartd. Jul 9 23:41:13 x4 smartd[626]: Device: /dev/ada0, WARNING: A firmware update for this drive may be available, Jul 9 23:41:13 x4 smartd[626]: see the following Seagate web pages: Jul 9 23:41:13 x4 smartd[626]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en Jul 9 23:41:13 x4 smartd[626]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en Configuring vt: keymap blanktime. Performing sanity check on sshd configuration. Starting sshd. Starting sendmail_submit. Starting sendmail_msp_queue. Starting cron. Local package initialization: backup. Starting nfscbd. Starting default mousedmoused: unable to open /dev/psm0: No such file or directory . Starting inetd. Tue Jul 9 23:41 FreeBSD/i386 (x4.osted.lan) (ttyu0) login: Jul 9 23:45:57 x4 su[931]: pho to root on /dev/pts/0 20190709 23:46:10 all (1/1): syscall4.sh lock order reversal: (sleepable after non-sleepable) 1st 0x23dfcb88 pfs_node (pfs_node) @ fs/pseudofs/pseudofs_internal.h:105 2nd 0x11a70650 pidhash (pidhash) @ kern/kern_proc.c:429 stack backtrace: #0 0x102fd01 at witness_debugger+0x61 #1 0x102fad9 at witness_checkorder+0x8a9 #2 0xfd7d69 at _sx_slock_int+0x49 #3 0xfb520c at pfind+0x5c #4 0xf1e3aa at pfs_readdir+0x13a #5 0x15624b9 at VOP_READDIR_APV+0x69 #6 0x109b319 at kern_getdirentries+0x1c9 #7 0x109b555 at sys_getdirentries+0x25 #8 0x1531999 at syscall+0x2d9 #9 0xffc033b7 at exit_tag+0xce3fa387 root Password: Last login: Tue Jul 9 17:12:00 on ttyu0 Jul 9 23:47:50 x4 login[774]: ROOT LOGIN (root) ON ttyu0 FreeBSD 13.0-CURRENT (PHO) #0 r349856M: Tue Jul 9 23:23:58 CEST 2019 You have new mail. root@x4:~ # pgrep syscall4 | xargs ps -lHp UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 65534 3692 1 0 20 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 umtxqb T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 umtxqb T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 umtxqb T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 umtxqb T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 65534 3692 1 0 52 0 42988 4728 - T - 0:00.00 /tmp/syscall4 root@x4:~ # init 1 Stopping inetd. Stopping nfscbd. Waiting for PIDS: 715. Shutting down local packages:. Stopping cron. Stopping sshd. Waiting for PIDS: 668. Stopping smartd. Waiting for PIDS: 628. Stopping powerd. Stopping ntpd. Stopping lockd. Waiting for PIDS: 563. Stopping statd. Stopping nfsd. Waiting for PIDS: 556 559. Stopping mountd. Waiting for PIDS: 550. Stopping nfsuserd. kill: 540: No such process kill: 541: No such process kill: 542: No such process kill: 543: No such process Stopping watchdogd. Stopping rpcbind. if_delmulti_locked: detaching ifnet instance 0x23ec8800 Stopping devd. Writing entropy file:. Writing early boot entropy file:. Terminated . Jul 9 23:48:21 x4 syslogd: exiting on signal 15 2019-07-09T23:48:41.228621+02:00 x4.osted.lan init 1 - - some processes would not die; ps axl advised Enter full pathname of shell or RETURN for /bin/sh: # ps auxwwl USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root 11 382.3 0.0 0 64 - RNL 23:41 22:30.66 [idle] 0 0 0 155 0 - root 0 0.0 0.0 0 384 - DLs 23:41 0:00.03 [kernel] 0 0 0 -16 0 swapin root 1 0.0 0.0 3596 828 - SLs 23:41 0:00.03 /sbin/init 0 0 0 20 0 wait root 2 0.0 0.0 0 16 - DL 23:41 0:00.00 [crypto] 0 0 0 -16 0 crypto_w root 3 0.0 0.0 0 16 - DL 23:41 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 4 0.0 0.0 0 16 - DL 23:41 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 5 0.0 0.0 0 16 - DL 23:41 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 6 0.0 0.0 0 16 - DL 23:41 0:00.00 [crypto returns 0 0 0 -16 0 crypto_r root 7 0.0 0.0 0 32 - DL 23:41 0:00.29 [cam] 0 0 0 -16 0 - root 8 0.0 0.0 0 16 - DL 23:41 0:00.00 [fdc0] 0 0 0 -16 0 - root 9 0.0 0.0 0 16 - DL 23:41 0:00.00 [soaiod1] 0 0 0 -16 0 - root 10 0.0 0.0 0 16 - DL 23:41 0:00.00 [audit] 0 0 0 -16 0 audit_wo root 12 0.0 0.0 0 304 - WL 23:41 0:02.07 [intr] 0 0 0 -52 0 - root 13 0.0 0.0 0 48 - DL 23:41 0:00.06 [geom] 0 0 0 -8 0 - root 14 0.0 0.0 0 16 - DL 23:41 0:00.00 [sequencer 00] 0 0 0 -16 0 seqstate root 15 0.0 0.0 0 480 - DL 23:41 0:00.05 [usb] 0 0 0 -68 0 - root 16 0.0 0.0 0 16 - DL 23:41 0:00.00 [soaiod2] 0 0 0 -16 0 - root 17 0.0 0.0 0 16 - DL 23:41 0:00.00 [soaiod3] 0 0 0 -16 0 - root 18 0.0 0.0 0 16 - DL 23:41 0:00.00 [soaiod4] 0 0 0 -16 0 - root 19 0.0 0.0 0 16 - DL 23:41 0:00.00 [sctp_iterator] 0 0 0 -16 0 waiting_ root 20 0.0 0.0 0 16 - DL 23:41 0:00.04 [rand_harvestq] 0 0 0 -16 0 - root 21 0.0 0.0 0 48 - DL 23:41 0:00.25 [pagedaemon] 0 0 0 -16 0 psleep root 22 0.0 0.0 0 16 - DL 23:41 0:00.00 [vmdaemon] 0 0 0 -16 0 psleep root 23 0.0 0.0 0 112 - DL 23:41 0:00.11 [bufdaemon] 0 0 0 -16 0 qsleep root 24 0.0 0.0 0 16 - DL 23:41 0:00.00 [vnlru] 0 0 0 -16 0 vlruwt root 25 0.0 0.0 0 16 - DL 23:41 0:00.09 [syncer] 0 0 0 16 0 syncer root 2468 0.0 0.0 0 16 - DL 23:46 0:00.01 [md10] 0 0 0 -8 0 mdwait root 2531 0.0 0.0 0 16 - DL 23:46 0:00.00 [aiod1] 0 0 0 -16 0 aiordy root 2532 0.0 0.0 0 16 - DL 23:46 0:00.00 [aiod2] 0 0 0 -16 0 aiordy root 2533 0.0 0.0 0 16 - DL 23:46 0:00.00 [aiod3] 0 0 0 -16 0 aiordy root 2534 0.0 0.0 0 16 - DL 23:46 0:00.00 [aiod4] 0 0 0 -16 0 aiordy nobody 3692 0.0 0.1 42988 4728 - T 23:46 0:00.01 /tmp/syscall4 65534 1 0 20 0 - root 8269 0.0 0.0 5340 2744 u0 Ss 23:48 0:00.01 -sh (sh) 0 1 0 20 0 wait root 8271 0.0 0.0 5016 2616 u0 R+ 23:48 0:00.00 ps auxwwl 0 8269 0 20 0 - # # KDB: enter: Break to debugger [ thread pid 11 tid 100006 ] Stopped at kdb_alt_break_internal+0xeb: movl $0,kdb_why db> run pho db:0:pho> set $lines 20000 db:0:pho> run pho1 db:1:pho1> bt Tracing pid 11 tid 100006 td 0x11fd6a80 kdb_alt_break_internal(0) at kdb_alt_break_internal+0xeb/frame 0x1b6e3b00 kdb_alt_break(2,23d6088c) at kdb_alt_break+0x10/frame 0x1b6e3b0c uart_intr_rxready(23d60800,86a,0,0,11fd6a80,...) at uart_intr_rxready+0x82/frame 0x1b6e3b28 uart_intr(23d60800) at uart_intr+0x10e/frame 0x1b6e3b48 intr_event_handle(11ad2c00,1b6e3bbc) at intr_event_handle+0xaf/frame 0x1b6e3b68 intr_execute_handlers(11ffe138,1b6e3bbc,3) at intr_execute_handlers+0xc7/frame 0x1b6e3b94 lapic_handle_intr(32,1b6e3bbc,8,28,28,...) at lapic_handle_intr+0x55/frame 0x1b6e3bac Xapic_isr1() at 0xffc04208/frame 0x1b6e3bac --- interrupt, eip = 0x15466a5, esp = 0x1b6e3bfc, ebp = 0x1b6e3bfc --- acpi_cpu_c1(0,3,df,99f6c3b2,11e6ddc0,...) at acpi_cpu_c1+0x5/frame 0x1b6e3bfc acpi_cpu_idle(51a6ebb6,0) at acpi_cpu_idle+0x230/frame 0x1b6e3c2c cpu_idle_acpi(51a6ebb6,0) at cpu_idle_acpi+0x35/frame 0x1b6e3c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6e3c5c sched_idletd(0,1b6e3ce8) at sched_idletd+0x3ee/frame 0x1b6e3cb4 fork_exit(ffb570,0,1b6e3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6e3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6e3cd4 --- trap 0, eip = 0, esp = 0x1b6e3d20, ebp = 0 --- (null)() at 0 db:1:pho1> show allpcpu Current CPU: 3 cpuid = 0 dynamic pcpu = 0x8bcb80 curthread = 0x11fd7700: pid 11 tid 100003 "idle: cpu0" curpcb = 0x1b6d4d40 fpcurthread = none idlethread = 0x11fd7700: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 trampstk = 0xffc07ff0 kesp0 = 0x1b6d4d30 common_tssp = 0xffc01260 tlb gen = 60143 curvnet = 0 spin locks held: cpuid = 1 dynamic pcpu = 0x10221b80 curthread = 0x11fd7380: pid 11 tid 100004 "idle: cpu1" curpcb = 0x1b6d9d40 fpcurthread = none idlethread = 0x11fd7380: tid 100004 "idle: cpu1" APIC ID = 1 currentldt = 0x50 trampstk = 0xffc0aff0 kesp0 = 0x1b6d9d30 common_tssp = 0xffc012c8 tlb gen = 60143 curvnet = 0 spin locks held: cpuid = 2 dynamic pcpu = 0x10226b80 curthread = 0x11fd7000: pid 11 tid 100005 "idle: cpu2" curpcb = 0x1b6ded40 fpcurthread = none idlethread = 0x11fd7000: tid 100005 "idle: cpu2" APIC ID = 2 currentldt = 0x50 trampstk = 0xffc0bff0 kesp0 = 0x1b6ded30 common_tssp = 0xffc01330 tlb gen = 60143 curvnet = 0 spin locks held: cpuid = 3 dynamic pcpu = 0x1022bb80 curthread = 0x11fd6a80: pid 11 tid 100006 "idle: cpu3" curpcb = 0x1b6e3d40 fpcurthread = none idlethread = 0x11fd6a80: tid 100006 "idle: cpu3" APIC ID = 3 currentldt = 0x50 trampstk = 0xffc0dff0 kesp0 = 0x1b6e3d30 common_tssp = 0xffc01398 tlb gen = 60139 curvnet = 0 spin locks held: db:1:pho1> show alllocks db:1:pho1> show lockedvnods Locked vnodes db:1:pho1> show allchains chain 1: thread 100110 (pid 8269, sh) sleeping on 0x23d63074 "ttyin" chain 2: thread 100293 (pid 3692, calls) sleeping on 0x1ed73d8 "umtxqb" chain 3: thread 100332 (pid 3692, calls) sleeping on 0x1ed73d8 "umtxqb" chain 4: thread 100349 (pid 3692, calls) sleeping on 0x1ed73d8 "umtxqb" chain 5: thread 100481 (pid 3692, calls) sleeping on 0x1ed73d8 "umtxqb" chain 6: thread 100259 (pid 2534, aiod4) sleeping on 0x3154a354 "aiordy" chain 7: thread 100233 (pid 2533, aiod3) sleeping on 0x3153e000 "aiordy" chain 8: thread 100176 (pid 2532, aiod2) sleeping on 0x23a706a8 "aiordy" chain 9: thread 100166 (pid 2531, aiod1) sleeping on 0x23a75000 "aiordy" chain 10: thread 100183 (pid 2468, md10) sleeping on 0x31557000 "mdwait" chain 11: thread 100104 (pid 25, syncer) sleeping on 0x1f5bde4 "syncer" chain 12: thread 100103 (pid 24, vnlru) sleeping on 0x23fed6a8 "vlruwt" chain 13: thread 100099 (pid 23, bufdaemon) sleeping on 0x1f5b77c "qsleep" chain 14: thread 100100 (pid 23, bufspacedaemon-0) sleeping on 0x1bfbe80 "-" chain 15: thread 100115 (pid 23, /home worker) sleeping on 0x23aa2884 "sdflush" chain 16: thread 100116 (pid 23, /tmp worker) sleeping on 0x24465684 "sdflush" chain 17: thread 100117 (pid 23, /usr worker) sleeping on 0x23a9fa84 "sdflush" chain 18: thread 100118 (pid 23, /var worker) sleeping on 0x2496c484 "sdflush" chain 19: thread 100246 (pid 23, /mnt worker) sleeping on 0x11b73684 "sdflush" chain 20: thread 100098 (pid 22, vmdaemon) sleeping on 0x1f71238 "psleep" chain 21: thread 100097 (pid 21, dom0) sleeping on 0x1fa57dc "psleep" chain 22: thread 100101 (pid 21, laundry: dom0) sleeping on 0x1fa57e8 "launds" chain 23: thread 100102 (pid 21, uma) sleeping on 0x12b9720 "umarcl" chain 24: thread 100094 (pid 20, rand_harvestq) sleeping on 0x1d32c80 "-" chain 25: thread 100090 (pid 19, sctp_iterator) sleeping on 0x1fa48bc "waiting_for_work" chain 26: thread 100089 (pid 18, soaiod4) sleeping on 0x1f5b3a8 "-" chain 27: thread 100088 (pid 17, soaiod3) sleeping on 0x1f5b3a8 "-" chain 28: thread 100087 (pid 16, soaiod2) sleeping on 0x1f5b3a8 "-" chain 29: thread 100086 (pid 9, soaiod1) sleeping on 0x1f5b3a8 "-" chain 30: thread 100083 (pid 8, fdc0) sleeping on 0x216b483c "-" chain 31: thread 100048 (pid 15, usbus0) sleeping on 0x216faacc "-" chain 32: thread 100049 (pid 15, usbus0) sleeping on 0x216faafc "-" chain 33: thread 100050 (pid 15, usbus0) sleeping on 0x216fab2c "-" chain 34: thread 100051 (pid 15, usbus0) sleeping on 0x216fab5c "-" chain 35: thread 100052 (pid 15, usbus0) sleeping on 0x216fab8c "-" chain 36: thread 100054 (pid 15, usbus1) sleeping on 0x23a7bacc "-" chain 37: thread 100055 (pid 15, usbus1) sleeping on 0x23a7bafc "-" chain 38: thread 100056 (pid 15, usbus1) sleeping on 0x23a7bb2c "-" chain 39: thread 100057 (pid 15, usbus1) sleeping on 0x23a7bb5c "-" chain 40: thread 100058 (pid 15, usbus1) sleeping on 0x23a7bb8c "-" chain 41: thread 100060 (pid 15, usbus2) sleeping on 0x23ad3acc "-" chain 42: thread 100061 (pid 15, usbus2) sleeping on 0x23ad3afc "-" chain 43: thread 100062 (pid 15, usbus2) sleeping on 0x23ad3b2c "-" chain 44: thread 100063 (pid 15, usbus2) sleeping on 0x23ad3b5c "-" chain 45: thread 100064 (pid 15, usbus2) sleeping on 0x23ad3b8c "-" chain 46: thread 100065 (pid 15, usbus3) sleeping on 0x23b19acc "-" chain 47: thread 100066 (pid 15, usbus3) sleeping on 0x23b19afc "-" chain 48: thread 100067 (pid 15, usbus3) sleeping on 0x23b19b2c "-" chain 49: thread 100068 (pid 15, usbus3) sleeping on 0x23b19b5c "-" chain 50: thread 100069 (pid 15, usbus3) sleeping on 0x23b19b8c "-" chain 51: thread 100070 (pid 15, usbus4) sleeping on 0x23b62acc "-" chain 52: thread 100071 (pid 15, usbus4) sleeping on 0x23b62afc "-" chain 53: thread 100072 (pid 15, usbus4) sleeping on 0x23b62b2c "-" chain 54: thread 100073 (pid 15, usbus4) sleeping on 0x23b62b5c "-" chain 55: thread 100074 (pid 15, usbus4) sleeping on 0x23b62b8c "-" chain 56: thread 100076 (pid 15, usbus5) sleeping on 0x216f4ca4 "-" chain 57: thread 100077 (pid 15, usbus5) sleeping on 0x216f4cd4 "-" chain 58: thread 100078 (pid 15, usbus5) sleeping on 0x216f4d04 "-" chain 59: thread 100079 (pid 15, usbus5) sleeping on 0x216f4d34 "-" chain 60: thread 100080 (pid 15, usbus5) sleeping on 0x216f4d64 "-" chain 61: thread 100043 (pid 7, doneq0) sleeping on 0x1c6c140 "-" chain 62: thread 100096 (pid 7, scanner) sleeping on 0x1c6c06c "-" chain 63: thread 100042 (pid 6, crypto returns 3) sleeping on 0x11ad0dc4 "crypto_ret_wait" chain 64: thread 100041 (pid 5, crypto returns 2) sleeping on 0x11ad0d8c "crypto_ret_wait" chain 65: thread 100040 (pid 4, crypto returns 1) sleeping on 0x11ad0d54 "crypto_ret_wait" chain 66: thread 100039 (pid 3, crypto returns 0) sleeping on 0x11ad0d1c "crypto_ret_wait" chain 67: thread 100038 (pid 2, crypto) sleeping on 0x1f6fff8 "crypto_wait" chain 68: thread 100033 (pid 14, sequencer 00) sleeping on 0x11b9e84c "seqstate" chain 69: thread 100030 (pid 13, g_event) sleeping on 0x1f94868 "-" chain 70: thread 100031 (pid 13, g_up) sleeping on 0x1f94870 "-" chain 71: thread 100032 (pid 13, g_down) sleeping on 0x1f94864 "-" chain 72: thread 100002 (pid 1, init) sleeping on 0x11fce9fc "wait" chain 73: thread 100001 (pid 10, audit) sleeping on 0x1fa50a0 "audit_worker_cv" chain 74: thread 100000 (pid 0, swapper) sleeping on 0x1f94888 "swapin" chain 75: thread 100007 (pid 0, aiod_kick taskq) sleeping on 0x11f60f00 "-" chain 76: thread 100010 (pid 0, thread taskq) sleeping on 0x11f60d80 "-" chain 77: thread 100012 (pid 0, config_0) sleeping on 0x11fc2c00 "-" chain 78: thread 100013 (pid 0, kqueue_ctx taskq) sleeping on 0x11f60c80 "-" chain 79: thread 100014 (pid 0, if_config_tqg_0) sleeping on 0x11fc2b80 "-" chain 80: thread 100015 (pid 0, if_io_tqg_0) sleeping on 0x11fc2b00 "-" chain 81: thread 100016 (pid 0, if_io_tqg_1) sleeping on 0x11fc2a80 "-" chain 82: thread 100017 (pid 0, if_io_tqg_2) sleeping on 0x11fc2a00 "-" chain 83: thread 100018 (pid 0, if_io_tqg_3) sleeping on 0x11fc2980 "-" chain 84: thread 100019 (pid 0, softirq_0) sleeping on 0x11fc2900 "-" chain 85: thread 100020 (pid 0, softirq_1) sleeping on 0x11fc2880 "-" chain 86: thread 100021 (pid 0, softirq_2) sleeping on 0x11fc2800 "-" chain 87: thread 100022 (pid 0, softirq_3) sleeping on 0x11fc2780 "-" chain 88: thread 100029 (pid 0, firmware taskq) sleeping on 0x11f60c00 "-" chain 89: thread 100034 (pid 0, crypto_0) sleeping on 0x11f60b00 "-" chain 90: thread 100035 (pid 0, crypto_1) sleeping on 0x11f60b00 "-" chain 91: thread 100036 (pid 0, crypto_2) sleeping on 0x11f60b00 "-" chain 92: thread 100037 (pid 0, crypto_3) sleeping on 0x11f60b00 "-" chain 93: thread 100085 (pid 0, mca taskq) sleeping on 0x23d67000 "-" chain 94: thread 100091 (pid 0, acpi_task_0) sleeping on 0x23eb6d00 "-" chain 95: thread 100092 (pid 0, acpi_task_1) sleeping on 0x23eb6d00 "-" chain 96: thread 100093 (pid 0, acpi_task_2) sleeping on 0x23eb6d00 "-" chain 97: thread 100095 (pid 0, CAM taskq) sleeping on 0x11f60a80 "-" db:1:pho1> show mount 0x24989000 /dev/ada0s1a on / (ufs) 0x247fd000 devfs on /dev (devfs) 0x2d391000 /dev/ada0s1f on /home (ufs) 0x247fc000 /dev/label/tmp on /tmp (ufs) 0x247fa000 /dev/ada0s1d on /usr (ufs) 0x2d3e2000 /dev/ada0s1e on /var (ufs) 0x2d3e1000 procfs on /proc (procfs) 0x24988000 /dev/md10a on /mnt (ufs) More info: show mount db:1:pho1> show bufqueues bqempty: 182 Buf domain 0 freebufs 466 lofreebufs 311 hifreebufs 466 bufspace 80308224 maxbufspace 94683136 hibufspace 94027776 lobufspace 89326372 bufspacethresh 91677074 numdirtybuffers 7 lodirtybuffers 732 hidirtybuffers 1464 dirtybufthresh 1317 cleanq count 5260 (79505408) dirtyq count 7 (98304) wakeup 0 lim 28 CPU 0, 1, 21, 24, Locked buffers: 0 space 0 Total buffers: 5597 space 80308224 db:0:pho> ps pid ppid pgrp uid state wmesg wchan cmd 8269 1 8269 0 Ss+ ttyin 0x23d63074 sh 3692 1 3459 65534 T (threaded) syscall4 100241 s syscall4 100250 s test 100253 s calls 100255 s calls 100257 s calls 100258 s calls 100274 s calls 100278 s calls 100282 s calls 100286 s calls 100287 s calls 100289 s calls 100291 s calls 100293 D umtxqb 0x1ed73d8 calls 100295 s calls 100299 s calls 100302 s calls 100303 s calls 100304 s calls 100305 s calls 100313 s calls 100319 s calls 100324 s calls 100325 s calls 100327 s syscall4 100332 D umtxqb 0x1ed73d8 calls 100334 s calls 100336 s syscall4 100338 s calls 100340 s calls 100342 s syscall4 100344 s calls 100347 s syscall4 100349 D umtxqb 0x1ed73d8 calls 100351 s syscall4 100352 s syscall4 100357 s syscall4 100361 s calls 100367 s syscall4 100368 s syscall4 100373 s syscall4 100377 s calls 100381 s syscall4 100423 s syscall4 100424 s syscall4 100425 s calls 100430 s syscall4 100432 s syscall4 100434 s syscall4 100481 D umtxqb 0x1ed73d8 calls 101292 s syscall4 101343 s syscall4 2534 0 0 0 DL aiordy 0x3154a354 [aiod4] 2533 0 0 0 DL aiordy 0x3153e000 [aiod3] 2532 0 0 0 DL aiordy 0x23a706a8 [aiod2] 2531 0 0 0 DL aiordy 0x23a75000 [aiod1] 2468 0 0 0 DL mdwait 0x31557000 [md10] 25 0 0 0 DL syncer 0x1f5bde4 [syncer] 24 0 0 0 DL vlruwt 0x23fed6a8 [vnlru] 23 0 0 0 DL (threaded) [bufdaemon] 100099 D qsleep 0x1f5b77c [bufdaemon] 100100 D - 0x1bfbe80 [bufspacedaemon-0] 100115 D sdflush 0x23aa2884 [/home worker] 100116 D sdflush 0x24465684 [/tmp worker] 100117 D sdflush 0x23a9fa84 [/usr worker] 100118 D sdflush 0x2496c484 [/var worker] 100246 D sdflush 0x11b73684 [/mnt worker] 22 0 0 0 DL psleep 0x1f71238 [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100097 D psleep 0x1fa57dc [dom0] 100101 D launds 0x1fa57e8 [laundry: dom0] 100102 D umarcl 0x12b9720 [uma] 20 0 0 0 DL - 0x1d32c80 [rand_harvestq] 19 0 0 0 DL waiting 0x1fa48bc [sctp_iterator] 18 0 0 0 DL - 0x1f5b3a8 [soaiod4] 17 0 0 0 DL - 0x1f5b3a8 [soaiod3] 16 0 0 0 DL - 0x1f5b3a8 [soaiod2] 9 0 0 0 DL - 0x1f5b3a8 [soaiod1] 8 0 0 0 DL - 0x216b483c [fdc0] 15 0 0 0 DL (threaded) [usb] 100048 D - 0x216faacc [usbus0] 100049 D - 0x216faafc [usbus0] 100050 D - 0x216fab2c [usbus0] 100051 D - 0x216fab5c [usbus0] 100052 D - 0x216fab8c [usbus0] 100054 D - 0x23a7bacc [usbus1] 100055 D - 0x23a7bafc [usbus1] 100056 D - 0x23a7bb2c [usbus1] 100057 D - 0x23a7bb5c [usbus1] 100058 D - 0x23a7bb8c [usbus1] 100060 D - 0x23ad3acc [usbus2] 100061 D - 0x23ad3afc [usbus2] 100062 D - 0x23ad3b2c [usbus2] 100063 D - 0x23ad3b5c [usbus2] 100064 D - 0x23ad3b8c [usbus2] 100065 D - 0x23b19acc [usbus3] 100066 D - 0x23b19afc [usbus3] 100067 D - 0x23b19b2c [usbus3] 100068 D - 0x23b19b5c [usbus3] 100069 D - 0x23b19b8c [usbus3] 100070 D - 0x23b62acc [usbus4] 100071 D - 0x23b62afc [usbus4] 100072 D - 0x23b62b2c [usbus4] 100073 D - 0x23b62b5c [usbus4] 100074 D - 0x23b62b8c [usbus4] 100076 D - 0x216f4ca4 [usbus5] 100077 D - 0x216f4cd4 [usbus5] 100078 D - 0x216f4d04 [usbus5] 100079 D - 0x216f4d34 [usbus5] 100080 D - 0x216f4d64 [usbus5] 7 0 0 0 DL (threaded) [cam] 100043 D - 0x1c6c140 [doneq0] 100096 D - 0x1c6c06c [scanner] 6 0 0 0 DL crypto_ 0x11ad0dc4 [crypto returns 3] 5 0 0 0 DL crypto_ 0x11ad0d8c [crypto returns 2] 4 0 0 0 DL crypto_ 0x11ad0d54 [crypto returns 1] 3 0 0 0 DL crypto_ 0x11ad0d1c [crypto returns 0] 2 0 0 0 DL crypto_ 0x1f6fff8 [crypto] 14 0 0 0 DL seqstat 0x11b9e84c [sequencer 00] 13 0 0 0 DL (threaded) [geom] 100030 D - 0x1f94868 [g_event] 100031 D - 0x1f94870 [g_up] 100032 D - 0x1f94864 [g_down] 12 0 0 0 WL (threaded) [intr] 100008 I [swi6: task queue] 100009 I [swi6: Giant taskq] 100011 I [swi5: fast taskq] 100023 I [swi4: clock (0)] 100024 I [swi4: clock (1)] 100025 I [swi4: clock (2)] 100026 I [swi4: clock (3)] 100027 I [swi1: netisr 0] 100028 I [swi3: vm] 100044 I [irq24: hdac0] 100045 I [irq25: re0] 100046 I [irq22: ahci0] 100047 I [irq16: hdac1 ohci0] 100053 I [irq17: ohci1 ohci3] 100059 I [irq18: ohci2 ohci4] 100075 I [irq19: ehci0] 100081 I [irq14: ata0] 100082 I [swi0: uart] 100084 I [irq1: atkbd0] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0x11fce9fc [init] 10 0 0 0 DL audit_w 0x1fa50a0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0x1f94888 [swapper] 100007 D - 0x11f60f00 [aiod_kick taskq] 100010 D - 0x11f60d80 [thread taskq] 100012 D - 0x11fc2c00 [config_0] 100013 D - 0x11f60c80 [kqueue_ctx taskq] 100014 D - 0x11fc2b80 [if_config_tqg_0] 100015 D - 0x11fc2b00 [if_io_tqg_0] 100016 D - 0x11fc2a80 [if_io_tqg_1] 100017 D - 0x11fc2a00 [if_io_tqg_2] 100018 D - 0x11fc2980 [if_io_tqg_3] 100019 D - 0x11fc2900 [softirq_0] 100020 D - 0x11fc2880 [softirq_1] 100021 D - 0x11fc2800 [softirq_2] 100022 D - 0x11fc2780 [softirq_3] 100029 D - 0x11f60c00 [firmware taskq] 100034 D - 0x11f60b00 [crypto_0] 100035 D - 0x11f60b00 [crypto_1] 100036 D - 0x11f60b00 [crypto_2] 100037 D - 0x11f60b00 [crypto_3] 100085 D - 0x23d67000 [mca taskq] 100091 D - 0x23eb6d00 [acpi_task_0] 100092 D - 0x23eb6d00 [acpi_task_1] 100093 D - 0x23eb6d00 [acpi_task_2] 100095 D - 0x11f60a80 [CAM taskq] db:0:pho> show freepages DOMAIN: 0 FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 002037 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000002 05 (000128K) | 000008 04 (000064K) | 000029 03 (000032K) | 000105 02 (000016K) | 000387 01 (000008K) | 001320 00 (000004K) | 003151 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 001663 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000000 05 (000128K) | 000002 04 (000064K) | 000006 03 (000032K) | 000027 02 (000016K) | 000134 01 (000008K) | 000417 00 (000004K) | 000641 FREE LIST 2: ORDER (SIZE) | NUMBER | POOL 0 -- -- -- -- 09 (002048K) | 000000 08 (001024K) | 000000 07 (000512K) | 000000 06 (000256K) | 000001 05 (000128K) | 000001 04 (000064K) | 000001 03 (000032K) | 000002 02 (000016K) | 000003 01 (000008K) | 000003 00 (000004K) | 000001 db:0:pho> acttrace Tracing command idle pid 11 tid 100003 td 0x11fd7700 (CPU 0) cpustop_handler(0,1b6d4bbc,1b6d4bb0,15306b2,0,...) at cpustop_handler+0x27/frame 0x1b6d4ae8 ipi_nmi_handler(0,1b6d4b28,11fd7700,24fedc0,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6d4af8 trap(1b6d4bbc,8,28,28,2167a800,...) at trap+0xc2/frame 0x1b6d4bb0 calltrap() at 0xffc0316d/frame 0x1b6d4bb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6d4bfc, ebp = 0x1b6d4bfc --- acpi_cpu_c1(0,0,df,9a80b4dd,24fedc0,...) at acpi_cpu_c1+0x5/frame 0x1b6d4bfc acpi_cpu_idle(984b1dd,0) at acpi_cpu_idle+0x230/frame 0x1b6d4c2c cpu_idle_acpi(984b1dd,0) at cpu_idle_acpi+0x35/frame 0x1b6d4c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6d4c5c sched_idletd(0,1b6d4ce8) at sched_idletd+0x3ee/frame 0x1b6d4cb4 fork_exit(ffb570,0,1b6d4ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6d4cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6d4cd4 --- trap 0, eip = 0, esp = 0x1b6d4d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0x11fd7380 (CPU 1) cpustop_handler(1,1b6d9bbc,1b6d9bb0,15306b2,0,...) at cpustop_handler+0x27/frame 0x1b6d9ae8 ipi_nmi_handler(0,1b6d9b28,11fd7380,1eeefdc,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6d9af8 trap(1b6d9bbc,8,28,28,2167a600,...) at trap+0xc2/frame 0x1b6d9bb0 calltrap() at 0xffc0316d/frame 0x1b6d9bb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6d9bfc, ebp = 0x1b6d9bfc --- acpi_cpu_c1(0,1,df,99f7eb71,11e63dc0,...) at acpi_cpu_c1+0x5/frame 0x1b6d9bfc acpi_cpu_idle(1b1cfd3e,0) at acpi_cpu_idle+0x230/frame 0x1b6d9c2c cpu_idle_acpi(1b1cfd3e,0) at cpu_idle_acpi+0x35/frame 0x1b6d9c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6d9c5c sched_idletd(0,1b6d9ce8) at sched_idletd+0x3ee/frame 0x1b6d9cb4 fork_exit(ffb570,0,1b6d9ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6d9cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6d9cd4 --- trap 0, eip = 0, esp = 0x1b6d9d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0x11fd7000 (CPU 2) cpustop_handler(2,1b6debbc,1b6debb0,15306b2,1b6deb14,...) at cpustop_handler+0x27/frame 0x1b6deae8 ipi_nmi_handler(1b6deb14,fd6830,11fd7000,23fefa80,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6deaf8 trap(1b6debbc,8,28,28,2167a400,...) at trap+0xc2/frame 0x1b6debb0 calltrap() at 0xffc0316d/frame 0x1b6debb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6debfc, ebp = 0x1b6debfc --- acpi_cpu_c1(0,2,df,99f7c12a,11e68dc0,...) at acpi_cpu_c1+0x5/frame 0x1b6debfc acpi_cpu_idle(1b1d645a,0) at acpi_cpu_idle+0x230/frame 0x1b6dec2c cpu_idle_acpi(1b1d645a,0) at cpu_idle_acpi+0x35/frame 0x1b6dec40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6dec5c sched_idletd(0,1b6dece8) at sched_idletd+0x3ee/frame 0x1b6decb4 fork_exit(ffb570,0,1b6dece8,0,0,...) at fork_exit+0x6c/frame 0x1b6decd4 fork_trampoline() at 0xffc033ca/frame 0x1b6decd4 --- trap 0, eip = 0, esp = 0x1b6ded20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0x11fd6a80 (CPU 3) kdb_alt_break_internal(0) at kdb_alt_break_internal+0xeb/frame 0x1b6e3b00 kdb_alt_break(2,23d6088c) at kdb_alt_break+0x10/frame 0x1b6e3b0c uart_intr_rxready(23d60800,86a,0,0,11fd6a80,...) at uart_intr_rxready+0x82/frame 0x1b6e3b28 uart_intr(23d60800) at uart_intr+0x10e/frame 0x1b6e3b48 intr_event_handle(11ad2c00,1b6e3bbc) at intr_event_handle+0xaf/frame 0x1b6e3b68 intr_execute_handlers(11ffe138,1b6e3bbc,3) at intr_execute_handlers+0xc7/frame 0x1b6e3b94 lapic_handle_intr(32,1b6e3bbc,8,28,28,...) at lapic_handle_intr+0x55/frame 0x1b6e3bac Xapic_isr1() at 0xffc04208/frame 0x1b6e3bac --- interrupt, eip = 0x15466a5, esp = 0x1b6e3bfc, ebp = 0x1b6e3bfc --- acpi_cpu_c1(0,3,df,99f6c3b2,11e6ddc0,...) at acpi_cpu_c1+0x5/frame 0x1b6e3bfc acpi_cpu_idle(51a6ebb6,0) at acpi_cpu_idle+0x230/frame 0x1b6e3c2c cpu_idle_acpi(51a6ebb6,0) at cpu_idle_acpi+0x35/frame 0x1b6e3c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6e3c5c sched_idletd(0,1b6e3ce8) at sched_idletd+0x3ee/frame 0x1b6e3cb4 fork_exit(ffb570,0,1b6e3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6e3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6e3cd4 --- trap 0, eip = 0, esp = 0x1b6e3d20, ebp = 0 --- (null)() at 0 db:0:pho> allt Tracing command sh pid 8269 tid 100110 td 0x24892700 sched_switch(24892700,0,104,197,2d0cf9a0,...) at sched_switch+0x598/frame 0x2d0cf968 mi_switch(104,0) at mi_switch+0x516/frame 0x2d0cfa2c sleepq_switch(0,23d63004,8,249b19fc,1efdc94,...) at sleepq_switch+0xe1/frame 0x2d0cfa50 sleepq_catch_signals(23d63074,24892700,23d63004,2d0cfac8,f72668,...) at sleepq_catch_signals+0x3b7/frame 0x2d0cfa7c sleepq_wait_sig(23d63074,0) at sleepq_wait_sig+0x11/frame 0x2d0cfa90 _cv_wait_sig(23d63074,23d63004) at _cv_wait_sig+0x138/frame 0x2d0cfac8 tty_wait(23d63000,23d63074) at tty_wait+0x35/frame 0x2d0cfae0 ttydisc_read(23d63000,2d0cfbd8,0) at ttydisc_read+0x1b9/frame 0x2d0cfb14 ttydev_read(11e65c00,2d0cfbd8,0) at ttydev_read+0x3d/frame 0x2d0cfb30 devfs_read_f(2d21b0e0,2d0cfbd8,11fa0b00,0,24892700) at devfs_read_f+0xaa/frame 0x2d0cfb60 dofileread(2d21b0e0,2d0cfbd8,ffffffff,ffffffff,0) at dofileread+0x6d/frame 0x2d0cfb90 kern_readv(24892700,0,2d0cfbd8) at kern_readv+0x3b/frame 0x2d0cfbbc sys_read(24892700,24892988) at sys_read+0x48/frame 0x2d0cfc08 syscall(2d0cfce8,3b,3b,3b,ffbfec50,...) at syscall+0x2d9/frame 0x2d0cfcdc Xint0x80_syscall() at 0xffc033b7/frame 0x2d0cfcdc --- syscall (3, FreeBSD ELF32, sys_read), eip = 0x20606ecb, esp = 0xffbfeb18, ebp = 0xffbfeb2c --- __bss_end() at 0x20606ecb Tracing command syscall4 pid 3692 tid 100241 td 0x31a96700 sched_switch(31a96700,0,20a,197,1030a05,...) at sched_switch+0x598/frame 0x3172abb8 mi_switch(20a,0) at mi_switch+0x516/frame 0x3172ac78 thread_suspend_check(0) at thread_suspend_check+0x83/frame 0x3172aca4 ast(3172ace8,3b,3b,3b,0,...) at ast+0x5a8/frame 0x3172acdc exit_tag(0,0,0,0,0,...) at 0xffc055ad/frame 0xffbfe644 (null)() at 0 Tracing command syscall4 pid 3692 tid 100250 td 0x31a9da80 sched_switch(31a9da80,0,20a,197,1030a05,...) at sched_switch+0x598/frame 0x2e360bb8 mi_switch(20a,0) at mi_switch+0x516/frame 0x2e360c78 thread_suspend_check(0) at thread_suspend_check+0x83/frame 0x2e360ca4 ast(2e360ce8,3b,3b,3b,8001879a,...) at ast+0x5a8/frame 0x2e360cdc KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(e6217b,1bd8c30,0,167bbfa,10b,...) at db_trace_self_wrapper+0x2a/frame 0x1b6e34a0 kdb_backtrace(1b6e35bc,15306c4,1591b26,248,11fd6a80,...) at kdb_backtrace+0x2e/frame 0x1b6e3500 kdb_reenter(1591b26,248,11fd6a80,a6e3534,11fce6a8,...) at kdb_reenter+0x28/frame 0x1b6e3508 trap(1b6e35c8,8,28,28,0,...) at trap+0xd4/frame 0x1b6e35bc calltrap() at 0xffc0316d/frame 0x1b6e35bc --- trap 0xc, eip = 0x150d4e0, esp = 0x1b6e3608, ebp = 0x1b6e3658 --- db_read_bytes(fbbfdebc,4,1b6e366c) at db_read_bytes+0x40/frame 0x1b6e3658 db_get_value(fbbfdebc,4,0) at db_get_value+0x21/frame 0x1b6e3680 db_backtrace(2e360bb8,ff91b8,2e360b58,ffffffff) at db_backtrace+0x2d3/frame 0x1b6e3720 db_trace_thread(31a9da80,ffffffff) at db_trace_thread+0x37/frame 0x1b6e3740 _db_stack_trace_all(101242b,0,ffffffff,1b6e37cc) at _db_stack_trace_all+0xc4/frame 0x1b6e37a0 db_command(1c6eb32) at db_command+0x26f/frame 0x1b6e3854 db_script_exec(bd2210,0,1f75c28,1b6e38e0) at db_script_exec+0x166/frame 0x1b6e38b4 db_command(1b6e39b8,a0463a,bd3011,2a64,11fd6a80,...) at db_command+0x26f/frame 0x1b6e3968 db_command_loop(bd3011,2a64,11fd6a80,1b6e3970,1b6e39b8,...) at db_command_loop+0x52/frame 0x1b6e3970 db_trap(3,0) at db_trap+0xfa/frame 0x1b6e39b8 kdb_trap(3,0,1b6e3ab8) at kdb_trap+0x12a/frame 0x1b6e39ec trap(1b6e3ab8,8,28,28,44,...) at trap+0x3a6/frame 0x1b6e3aac calltrap() at 0xffc0316d/frame 0x1b6e3aac --- trap 0x3, eip = 0x101242b, esp = 0x1b6e3af8, ebp = 0x1b6e3b00 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0xeb/frame 0x1b6e3b00 kdb_alt_break(2,23d6088c) at kdb_alt_break+0x10/frame 0x1b6e3b0c uart_intr_rxready(23d60800,86a,0,0,11fd6a80,...) at uart_intr_rxready+0x82/frame 0x1b6e3b28 uart_intr(23d60800) at uart_intr+0x10e/frame 0x1b6e3b48 intr_event_handle(11ad2c00,1b6e3bbc) at intr_event_handle+0xaf/frame 0x1b6e3b68 intr_execute_handlers(11ffe138,1b6e3bbc,3) at intr_execute_handlers+0xc7/frame 0x1b6e3b94 lapic_handle_intr(32,1b6e3bbc,8,28,28,...) at lapic_handle_intr+0x55/frame 0x1b6e3bac Xapic_isr1() at 0xffc04208/frame 0x1b6e3bac --- interrupt, eip = 0x15466a5, esp = 0x1b6e3bfc, ebp = 0x1b6e3bfc --- acpi_cpu_c1(0,3,df,99f6c3b2,11e6ddc0,...) at acpi_cpu_c1+0x5/frame 0x1b6e3bfc acpi_cpu_idle(51a6ebb6,0) at acpi_cpu_idle+0x230/frame 0x1b6e3c2c cpu_idle_acpi(51a6ebb6,0) at cpu_idle_acpi+0x35/frame 0x1b6e3c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6e3c5c sched_idletd(0,1b6e3ce8) at sched_idletd+0x3ee/frame 0x1b6e3cb4 fork_exit(ffb570,0,1b6e3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6e3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6e3cd4 --- trap 0, eip = 0, esp = 0x1b6e3d20, ebp = 0 --- (null)() at 0 *** error reading from address fbbfdebc *** KDB: reentering KDB: stack backtrace: db_trace_self_wrapper(1b6e3628,1b6e3758,ffc055ad,3933ef26,4,...) at db_trace_self_wrapper+0x2a/frame 0x1b6e3600 kdb_backtrace(1b6e3680,a00c9d,a07337,1c6ed10,3933ef26,...) at kdb_backtrace+0x2e/frame 0x1b6e365c kdb_reenter(a07337,1c6ed10,3933ef26,fbbfdeb8,3154e1a0,...) at kdb_reenter+0x28/frame 0x1b6e3664 db_get_value(fbbfdebc,4,0) at db_get_value+0x3d/frame 0x1b6e3680 db_backtrace(2e360bb8,ff91b8,2e360b58,ffffffff) at db_backtrace+0x2d3/frame 0x1b6e3720 db_trace_thread(31a9da80,ffffffff) at db_trace_thread+0x37/frame 0x1b6e3740 _db_stack_trace_all(101242b,0,ffffffff,1b6e37cc) at _db_stack_trace_all+0xc4/frame 0x1b6e37a0 db_command(1c6eb32) at db_command+0x26f/frame 0x1b6e3854 db_script_exec(bd2210,0,1f75c28,1b6e38e0) at db_script_exec+0x166/frame 0x1b6e38b4 db_command(1b6e39b8,a0463a,bd3011,2a64,11fd6a80,...) at db_command+0x26f/frame 0x1b6e3968 db_command_loop(bd3011,2a64,11fd6a80,1b6e3970,1b6e39b8,...) at db_command_loop+0x52/frame 0x1b6e3970 db_trap(3,0) at db_trap+0xfa/frame 0x1b6e39b8 kdb_trap(3,0,1b6e3ab8) at kdb_trap+0x12a/frame 0x1b6e39ec trap(1b6e3ab8,8,28,28,44,...) at trap+0x3a6/frame 0x1b6e3aac calltrap() at 0xffc0316d/frame 0x1b6e3aac --- trap 0x3, eip = 0x101242b, esp = 0x1b6e3af8, ebp = 0x1b6e3b00 --- kdb_alt_break_internal(0) at kdb_alt_break_internal+0xeb/frame 0x1b6e3b00 kdb_alt_break(2,23d6088c) at kdb_alt_break+0x10/frame 0x1b6e3b0c uart_intr_rxready(23d60800,86a,0,0,11fd6a80,...) at uart_intr_rxready+0x82/frame 0x1b6e3b28 uart_intr(23d60800) at uart_intr+0x10e/frame 0x1b6e3b48 intr_event_handle(11ad2c00,1b6e3bbc) at intr_event_handle+0xaf/frame 0x1b6e3b68 intr_execute_handlers(11ffe138,1b6e3bbc,3) at intr_execute_handlers+0xc7/frame 0x1b6e3b94 lapic_handle_intr(32,1b6e3bbc,8,28,28,...) at lapic_handle_intr+0x55/frame 0x1b6e3bac Xapic_isr1() at 0xffc04208/frame 0x1b6e3bac --- interrupt, eip = 0x15466a5, esp = 0x1b6e3bfc, ebp = 0x1b6e3bfc --- acpi_cpu_c1(0,3,df,99f6c3b2,11e6ddc0,...) at acpi_cpu_c1+0x5/frame 0x1b6e3bfc acpi_cpu_idle(51a6ebb6,0) at acpi_cpu_idle+0x230/frame 0x1b6e3c2c cpu_idle_acpi(51a6ebb6,0) at cpu_idle_acpi+0x35/frame 0x1b6e3c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6e3c5c sched_idletd(0,1b6e3ce8) at sched_idletd+0x3ee/frame 0x1b6e3cb4 fork_exit(ffb570,0,1b6e3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6e3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6e3cd4 --- trap 0, eip = 0, esp = 0x1b6e3d20, ebp = 0 --- (null)() at 0 Tracing command aiod4 pid 2534 tid 100259 td 0x314c6700 sched_switch(314c6700,0,104,197,c,...) at sched_switch+0x598/frame 0x316c1a70 mi_switch(104,0) at mi_switch+0x516/frame 0x316c1b34 sleepq_switch(1f5,314c6700,dffffdd5,1,1,...) at sleepq_switch+0xe1/frame 0x316c1b58 sleepq_timedwait(3154a354,5c,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x316c1b88 _sleep(3154a354,1f5b6f0,5c,15c06ee,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x316c1c70 aio_daemon(4,316c1ce8) at aio_daemon+0x297/frame 0x316c1cb4 fork_exit(106b2c0,4,316c1ce8,0,0,...) at fork_exit+0x6c/frame 0x316c1cd4 fork_trampoline() at 0xffc033ca/frame 0x316c1cd4 --- trap 0, eip = 0, esp = 0x316c1d20, ebp = 0 --- (null)() at 0 Tracing command aiod3 pid 2533 tid 100233 td 0x31454380 sched_switch(31454380,0,104,197,11e6ddc0,...) at sched_switch+0x598/frame 0x3167fa70 mi_switch(104,0) at mi_switch+0x516/frame 0x3167fb34 sleepq_switch(1f5,31454380,dffffdd5,1,2,...) at sleepq_switch+0xe1/frame 0x3167fb58 sleepq_timedwait(3153e000,5c,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x3167fb88 _sleep(3153e000,1f5b6f0,5c,15c06ee,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x3167fc70 aio_daemon(3,3167fce8) at aio_daemon+0x297/frame 0x3167fcb4 fork_exit(106b2c0,3,3167fce8,0,0,...) at fork_exit+0x6c/frame 0x3167fcd4 fork_trampoline() at 0xffc033ca/frame 0x3167fcd4 --- trap 0, eip = 0, esp = 0x3167fd20, ebp = 0 --- (null)() at 0 Tracing command aiod2 pid 2532 tid 100176 td 0x23d87380 sched_switch(23d87380,0,104,197,1fc64f4,...) at sched_switch+0x598/frame 0x2e222a70 mi_switch(104,0) at mi_switch+0x516/frame 0x2e222b34 sleepq_switch(1f5,23d87380,dffffdd5,1,3,...) at sleepq_switch+0xe1/frame 0x2e222b58 sleepq_timedwait(23a706a8,5c,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x2e222b88 _sleep(23a706a8,1f5b6f0,5c,15c06ee,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x2e222c70 aio_daemon(2,2e222ce8) at aio_daemon+0x297/frame 0x2e222cb4 fork_exit(106b2c0,2,2e222ce8,0,0,...) at fork_exit+0x6c/frame 0x2e222cd4 fork_trampoline() at 0xffc033ca/frame 0x2e222cd4 --- trap 0, eip = 0, esp = 0x2e222d20, ebp = 0 --- (null)() at 0 Tracing command aiod1 pid 2531 tid 100166 td 0x23d88000 sched_switch(23d88000,0,104,197,1fc64f4,...) at sched_switch+0x598/frame 0x2dde3a70 mi_switch(104,0) at mi_switch+0x516/frame 0x2dde3b34 sleepq_switch(1f5,23d88000,dffffdd5,1,3,...) at sleepq_switch+0xe1/frame 0x2dde3b58 sleepq_timedwait(23a75000,5c,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x2dde3b88 _sleep(23a75000,1f5b6f0,5c,15c06ee,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x2dde3c70 aio_daemon(1,2dde3ce8) at aio_daemon+0x297/frame 0x2dde3cb4 fork_exit(106b2c0,1,2dde3ce8,0,0,...) at fork_exit+0x6c/frame 0x2dde3cd4 fork_trampoline() at 0xffc033ca/frame 0x2dde3cd4 --- trap 0, eip = 0, esp = 0x2dde3d20, ebp = 0 --- (null)() at 0 Tracing command md10 pid 2468 tid 100183 td 0x2e4c7a80 sched_switch(2e4c7a80,0,104,197,0,...) at sched_switch+0x598/frame 0x2e249a90 mi_switch(104,0) at mi_switch+0x516/frame 0x2e249b50 sleepq_switch(0,2e249b90,161a5e2,2e249c6c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x2e249b74 sleepq_wait(31557000,5c,161a5e2,0,0,...) at sleepq_wait+0x3a/frame 0x2e249b88 _sleep(31557000,31557028,25c,161a5e2,0,0,0,0,100) at _sleep+0x7ec/frame 0x2e249c6c md_kthread(31557000,2e249ce8) at md_kthread+0x1c2/frame 0x2e249cb4 fork_exit(bde510,31557000,2e249ce8,0,0,...) at fork_exit+0x6c/frame 0x2e249cd4 fork_trampoline() at 0xffc033ca/frame 0x2e249cd4 --- trap 0, eip = 0, esp = 0x2e249d20, ebp = 0 --- (null)() at 0 Tracing command syncer pid 25 tid 100104 td 0x23fef380 sched_switch(23fef380,0,104,197,106,...) at sched_switch+0x598/frame 0x271feaf8 mi_switch(104,0) at mi_switch+0x516/frame 0x271febbc sleepq_switch(cb,23fef380,1bee538,23fef380,271fec14,...) at sleepq_switch+0xe1/frame 0x271febe0 sleepq_timedwait(1f5bde4,0) at sleepq_timedwait+0x47/frame 0x271fec10 _cv_timedwait_sbt(1f5bde4,1f5bdc4,fffffed8,0,0,0,100) at _cv_timedwait_sbt+0x157/frame 0x271fec4c sched_sync(0,271fece8) at sched_sync+0x5e3/frame 0x271fecb4 fork_exit(1091830,0,271fece8,0,0,...) at fork_exit+0x6c/frame 0x271fecd4 fork_trampoline() at 0xffc033ca/frame 0x271fecd4 --- trap 0, eip = 0, esp = 0x271fed20, ebp = 0 --- (null)() at 0 Tracing command vnlru pid 24 tid 100103 td 0x23fef700 sched_switch(23fef700,0,104,197,0,...) at sched_switch+0x598/frame 0x271f9a38 mi_switch(104,0) at mi_switch+0x516/frame 0x271f9afc sleepq_switch(1e0,23fef700,fffffed,0,3,...) at sleepq_switch+0xe1/frame 0x271f9b20 sleepq_timedwait(23fed6a8,60,0,0,0,...) at sleepq_timedwait+0x47/frame 0x271f9b50 _sleep(23fed6a8,1f5bd90,260,167d9d8,fffffed8,0,0,0,100) at _sleep+0x79a/frame 0x271f9c34 vnlru_proc(0,271f9ce8) at vnlru_proc+0x8f/frame 0x271f9cb4 fork_exit(1090ff0,0,271f9ce8,0,0,...) at fork_exit+0x6c/frame 0x271f9cd4 fork_trampoline() at 0xffc033ca/frame 0x271f9cd4 --- trap 0, eip = 0, esp = 0x271f9d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100099 td 0x23fefa80 sched_switch(23fefa80,0,104,197,14fedc0,...) at sched_switch+0x598/frame 0x22b26a78 mi_switch(104,0) at mi_switch+0x516/frame 0x22b26b3c sleepq_switch(1df,23fefa80,1999997,0,1,...) at sleepq_switch+0xe1/frame 0x22b26b60 sleepq_timedwait(1f5b77c,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x22b26b90 _sleep(1f5b77c,1bfab80,54,15a40d5,1999997c,0,0,0,100) at _sleep+0x79a/frame 0x22b26c74 buf_daemon(0,22b26ce8) at buf_daemon+0xb1/frame 0x22b26cb4 fork_exit(1072df0,0,22b26ce8,0,0,...) at fork_exit+0x6c/frame 0x22b26cd4 fork_trampoline() at 0xffc033ca/frame 0x22b26cd4 --- trap 0, eip = 0, esp = 0x22b26d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100100 td 0x23d89000 sched_switch(23d89000,0,104,197,287b6acc,...) at sched_switch+0x598/frame 0x287b6a68 mi_switch(104,0) at mi_switch+0x516/frame 0x287b6b2c sleepq_switch(1e0,23d89000,fffffed,0,3,...) at sleepq_switch+0xe1/frame 0x287b6b50 sleepq_timedwait(1bfbe80,5c,0,0,0,...) at sleepq_timedwait+0x47/frame 0x287b6b80 _sleep(1bfbe80,1bfbdc0,25c,15a7839,fffffed8,0,0,0,100) at _sleep+0x79a/frame 0x287b6c68 bufspace_daemon(1bfac80,287b6ce8) at bufspace_daemon+0x2cb/frame 0x287b6cb4 fork_exit(1073120,1bfac80,287b6ce8,0,0,...) at fork_exit+0x6c/frame 0x287b6cd4 fork_trampoline() at 0xffc033ca/frame 0x287b6cd4 --- trap 0, eip = 0, esp = 0x287b6d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100115 td 0x11e6ba80 sched_switch(11e6ba80,0,104,197,11db5c00,...) at sched_switch+0x598/frame 0x2d120a78 mi_switch(104,0) at mi_switch+0x516/frame 0x2d120b3c sleepq_switch(1df,11e6ba80,7fffff6,0,0,...) at sleepq_switch+0xe1/frame 0x2d120b60 sleepq_timedwait(23aa2884,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x2d120b90 _sleep(23aa2884,23aa2800,54,15d4b8d,7fffff6c,0,0,0,100) at _sleep+0x79a/frame 0x2d120c78 softdep_flush(2d391000,2d120ce8) at softdep_flush+0x15c/frame 0x2d120cb4 fork_exit(1281f20,2d391000,2d120ce8,0,0,...) at fork_exit+0x6c/frame 0x2d120cd4 fork_trampoline() at 0xffc033ca/frame 0x2d120cd4 --- trap 0, eip = 0, esp = 0x2d120d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100116 td 0x2d3cd700 sched_switch(2d3cd700,0,104,197,0,...) at sched_switch+0x598/frame 0x2d125a78 mi_switch(104,0) at mi_switch+0x516/frame 0x2d125b3c sleepq_switch(1df,2d3cd700,7fffff6,0,2,...) at sleepq_switch+0xe1/frame 0x2d125b60 sleepq_timedwait(24465684,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x2d125b90 _sleep(24465684,24465600,54,15d4b8d,7fffff6c,0,0,0,100) at _sleep+0x79a/frame 0x2d125c78 softdep_flush(247fc000,2d125ce8) at softdep_flush+0x15c/frame 0x2d125cb4 fork_exit(1281f20,247fc000,2d125ce8,0,0,...) at fork_exit+0x6c/frame 0x2d125cd4 fork_trampoline() at 0xffc033ca/frame 0x2d125cd4 --- trap 0, eip = 0, esp = 0x2d125d20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100117 td 0x2d3cd380 sched_switch(2d3cd380,0,104,197,11e68dc0,...) at sched_switch+0x598/frame 0x2d12aa78 mi_switch(104,0) at mi_switch+0x516/frame 0x2d12ab3c sleepq_switch(1e0,2d3cd380,7fffff6,0,3,...) at sleepq_switch+0xe1/frame 0x2d12ab60 sleepq_timedwait(23a9fa84,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x2d12ab90 _sleep(23a9fa84,23a9fa00,54,15d4b8d,7fffff6c,0,0,0,100) at _sleep+0x79a/frame 0x2d12ac78 softdep_flush(247fa000,2d12ace8) at softdep_flush+0x15c/frame 0x2d12acb4 fork_exit(1281f20,247fa000,2d12ace8,0,0,...) at fork_exit+0x6c/frame 0x2d12acd4 fork_trampoline() at 0xffc033ca/frame 0x2d12acd4 --- trap 0, eip = 0, esp = 0x2d12ad20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100118 td 0x2d3cd000 sched_switch(2d3cd000,0,104,197,11d7d85c,...) at sched_switch+0x598/frame 0x2d12fa78 mi_switch(104,0) at mi_switch+0x516/frame 0x2d12fb3c sleepq_switch(1df,2d3cd000,7fffff6,0,3,...) at sleepq_switch+0xe1/frame 0x2d12fb60 sleepq_timedwait(2496c484,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x2d12fb90 _sleep(2496c484,2496c400,54,15d4b8d,7fffff6c,0,0,0,100) at _sleep+0x79a/frame 0x2d12fc78 softdep_flush(2d3e2000,2d12fce8) at softdep_flush+0x15c/frame 0x2d12fcb4 fork_exit(1281f20,2d3e2000,2d12fce8,0,0,...) at fork_exit+0x6c/frame 0x2d12fcd4 fork_trampoline() at 0xffc033ca/frame 0x2d12fcd4 --- trap 0, eip = 0, esp = 0x2d12fd20, ebp = 0 --- (null)() at 0 Tracing command bufdaemon pid 23 tid 100246 td 0x3152b380 sched_switch(3152b380,0,104,197,0,...) at sched_switch+0x598/frame 0x316bca78 mi_switch(104,0) at mi_switch+0x516/frame 0x316bcb3c sleepq_switch(1e0,3152b380,7fffff6,0,2,...) at sleepq_switch+0xe1/frame 0x316bcb60 sleepq_timedwait(11b73684,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x316bcb90 _sleep(11b73684,11b73600,54,15d4b8d,7fffff6c,0,0,0,100) at _sleep+0x79a/frame 0x316bcc78 softdep_flush(24988000,316bcce8) at softdep_flush+0x15c/frame 0x316bccb4 fork_exit(1281f20,24988000,316bcce8,0,0,...) at fork_exit+0x6c/frame 0x316bccd4 fork_trampoline() at 0xffc033ca/frame 0x316bccd4 --- trap 0, eip = 0, esp = 0x316bcd20, ebp = 0 --- (null)() at 0 Tracing command vmdaemon pid 22 tid 100098 td 0x23ff8000 sched_switch(23ff8000,0,104,197,0,...) at sched_switch+0x598/frame 0x22b1da60 mi_switch(104,0) at mi_switch+0x516/frame 0x22b1db20 sleepq_switch(0,22b1db60,15a40ce,22b1dc3c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x22b1db44 sleepq_wait(1f71238,74,15a40ce,0,0,...) at sleepq_wait+0x3a/frame 0x22b1db58 _sleep(1f71238,1f71224,74,15a40ce,0,0,0,0,100) at _sleep+0x7ec/frame 0x22b1dc3c vm_daemon(0,22b1dce8) at vm_daemon+0xa88/frame 0x22b1dcb4 fork_exit(12e4310,0,22b1dce8,0,0,...) at fork_exit+0x6c/frame 0x22b1dcd4 fork_trampoline() at 0xffc033ca/frame 0x22b1dcd4 --- trap 0, eip = 0, esp = 0x22b1dd20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100097 td 0x23ff8380 sched_switch(23ff8380,0,104,197,1b9,...) at sched_switch+0x598/frame 0x22b18898 mi_switch(104,0) at mi_switch+0x516/frame 0x22b1895c sleepq_switch(1df,23ff8380,1999997,0,2,...) at sleepq_switch+0xe1/frame 0x22b18980 sleepq_timedwait(1fa57dc,54,0,0,0,...) at sleepq_timedwait+0x47/frame 0x22b189b0 _sleep(1fa57dc,1fa5500,254,15a40ce,1999997c,0,0,0,100) at _sleep+0x79a/frame 0x22b18a94 vm_pageout_worker(0,23ff8380,23fee354,23ff8380,23fee354,...) at vm_pageout_worker+0x266/frame 0x22b18c94 vm_pageout(0,22b18ce8) at vm_pageout+0x10a/frame 0x22b18cb4 fork_exit(12d9bd0,0,22b18ce8,0,0,...) at fork_exit+0x6c/frame 0x22b18cd4 fork_trampoline() at 0xffc033ca/frame 0x22b18cd4 --- trap 0, eip = 0, esp = 0x22b18d20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100101 td 0x24894380 sched_switch(24894380,0,104,197,0,...) at sched_switch+0x598/frame 0x287fb938 mi_switch(104,0) at mi_switch+0x516/frame 0x287fb9f8 sleepq_switch(0,287fba38,1692888,287fbb14,fd8f1c,...) at sleepq_switch+0xe1/frame 0x287fba1c sleepq_wait(1fa57e8,54,1692888,0,0,...) at sleepq_wait+0x3a/frame 0x287fba30 _sleep(1fa57e8,1fa5440,54,1692888,0,...) at _sleep+0x7ec/frame 0x287fbb14 vm_pageout_laundry_worker(0,287fbce8) at vm_pageout_laundry_worker+0xd03/frame 0x287fbcb4 fork_exit(12db330,0,287fbce8,0,0,...) at fork_exit+0x6c/frame 0x287fbcd4 fork_trampoline() at 0xffc033ca/frame 0x287fbcd4 --- trap 0, eip = 0, esp = 0x287fbd20, ebp = 0 --- (null)() at 0 Tracing command pagedaemon pid 21 tid 100102 td 0x24894000 sched_switch(24894000,0,104,197,0,...) at sched_switch+0x598/frame 0x271f4a80 mi_switch(104,0) at mi_switch+0x516/frame 0x271f4b44 sleepq_switch(1eff944,24894000,0,100,271f4b98,...) at sleepq_switch+0xe1/frame 0x271f4b68 sleepq_timedwait(12b9720,54,271f4c34,0,0,...) at sleepq_timedwait+0x47/frame 0x271f4b98 _sleep(12b9720,1f70e4c,54,1603be6,fffffed8,0,0,0,100) at _sleep+0x79a/frame 0x271f4c7c uma_reclaim_worker(0,271f4ce8) at uma_reclaim_worker+0x104/frame 0x271f4cb4 fork_exit(12b9df0,0,271f4ce8,0,0,...) at fork_exit+0x6c/frame 0x271f4cd4 fork_trampoline() at 0xffc033ca/frame 0x271f4cd4 --- trap 0, eip = 0, esp = 0x271f4d20, ebp = 0 --- (null)() at 0 Tracing command rand_harvestq pid 20 tid 100094 td 0x23ff9000 sched_switch(23ff9000,0,104,197,0,...) at sched_switch+0x598/frame 0x22b09a58 mi_switch(104,0) at mi_switch+0x516/frame 0x22b09b1c sleepq_switch(1df,23ff9000,ccccccc,0,1,...) at sleepq_switch+0xe1/frame 0x22b09b40 sleepq_timedwait(1d32c80,0,0,0,0,...) at sleepq_timedwait+0x47/frame 0x22b09b70 _sleep(1d32c80,0,0,15a7839,19999999,0,0,0,4) at _sleep+0x79a/frame 0x22b09c54 random_kthread(0,22b09ce8) at random_kthread+0x23e/frame 0x22b09cb4 fork_exit(d43140,0,22b09ce8,0,0,...) at fork_exit+0x6c/frame 0x22b09cd4 fork_trampoline() at 0xffc033ca/frame 0x22b09cd4 --- trap 0, eip = 0, esp = 0x22b09d20, ebp = 0 --- (null)() at 0 Tracing command sctp_iterator pid 19 tid 100090 td 0x23ffa000 sched_switch(23ffa000,0,104,197,0,...) at sched_switch+0x598/frame 0x22af1aa8 mi_switch(104,0) at mi_switch+0x516/frame 0x22af1b68 sleepq_switch(0,22af1ba8,159c483,22af1c88,fd8f1c,...) at sleepq_switch+0xe1/frame 0x22af1b8c sleepq_wait(1fa48bc,0,159c483,0,0,...) at sleepq_wait+0x3a/frame 0x22af1ba0 _sleep(1fa48bc,1fa4884,0,159c483,0,0,0,0,100) at _sleep+0x7ec/frame 0x22af1c88 sctp_iterator_thread(0,22af1ce8) at sctp_iterator_thread+0x43/frame 0x22af1cb4 fork_exit(114b6a0,0,22af1ce8,0,0,...) at fork_exit+0x6c/frame 0x22af1cd4 fork_trampoline() at 0xffc033ca/frame 0x22af1cd4 --- trap 0, eip = 0, esp = 0x22af1d20, ebp = 0 --- (null)() at 0 Tracing command soaiod4 pid 18 tid 100089 td 0x11bb2000 sched_switch(11bb2000,0,104,197,0,...) at sched_switch+0x598/frame 0x22ae7a78 mi_switch(104,0) at mi_switch+0x516/frame 0x22ae7b3c sleepq_switch(1e0,11bb2000,dffffdd5,1,3,...) at sleepq_switch+0xe1/frame 0x22ae7b60 sleepq_timedwait(1f5b3a8,0,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x22ae7b90 _sleep(1f5b3a8,1f5b388,0,15a7839,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x22ae7c78 soaio_kproc_loop(4,22ae7ce8) at soaio_kproc_loop+0x14b/frame 0x22ae7cb4 fork_exit(103c410,4,22ae7ce8,0,0,...) at fork_exit+0x6c/frame 0x22ae7cd4 fork_trampoline() at 0xffc033ca/frame 0x22ae7cd4 --- trap 0, eip = 0, esp = 0x22ae7d20, ebp = 0 --- (null)() at 0 Tracing command soaiod3 pid 17 tid 100088 td 0x11bb2380 sched_switch(11bb2380,0,104,197,2,...) at sched_switch+0x598/frame 0x22ae2a78 mi_switch(104,0) at mi_switch+0x516/frame 0x22ae2b3c sleepq_switch(1e0,11bb2380,dffffdd5,1,0,...) at sleepq_switch+0xe1/frame 0x22ae2b60 sleepq_timedwait(1f5b3a8,0,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x22ae2b90 _sleep(1f5b3a8,1f5b388,0,15a7839,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x22ae2c78 soaio_kproc_loop(3,22ae2ce8) at soaio_kproc_loop+0x14b/frame 0x22ae2cb4 fork_exit(103c410,3,22ae2ce8,0,0,...) at fork_exit+0x6c/frame 0x22ae2cd4 fork_trampoline() at 0xffc033ca/frame 0x22ae2cd4 --- trap 0, eip = 0, esp = 0x22ae2d20, ebp = 0 --- (null)() at 0 Tracing command soaiod2 pid 16 tid 100087 td 0x11bb2700 sched_switch(11bb2700,0,104,197,0,...) at sched_switch+0x598/frame 0x22adda78 mi_switch(104,0) at mi_switch+0x516/frame 0x22addb3c sleepq_switch(1e0,11bb2700,dffffdd5,1,1,...) at sleepq_switch+0xe1/frame 0x22addb60 sleepq_timedwait(1f5b3a8,0,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x22addb90 _sleep(1f5b3a8,1f5b388,0,15a7839,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x22addc78 soaio_kproc_loop(2,22addce8) at soaio_kproc_loop+0x14b/frame 0x22addcb4 fork_exit(103c410,2,22addce8,0,0,...) at fork_exit+0x6c/frame 0x22addcd4 fork_trampoline() at 0xffc033ca/frame 0x22addcd4 --- trap 0, eip = 0, esp = 0x22addd20, ebp = 0 --- (null)() at 0 Tracing command soaiod1 pid 9 tid 100086 td 0x11bb2a80 sched_switch(11bb2a80,0,104,197,0,...) at sched_switch+0x598/frame 0x22ad8a78 mi_switch(104,0) at mi_switch+0x516/frame 0x22ad8b3c sleepq_switch(1e0,11bb2a80,dffffdd5,1,2,...) at sleepq_switch+0xe1/frame 0x22ad8b60 sleepq_timedwait(1f5b3a8,0,1d,0,0,...) at sleepq_timedwait+0x47/frame 0x22ad8b90 _sleep(1f5b3a8,1f5b388,0,15a7839,ffffdd50,1d,0,0,100) at _sleep+0x79a/frame 0x22ad8c78 soaio_kproc_loop(1,22ad8ce8) at soaio_kproc_loop+0x14b/frame 0x22ad8cb4 fork_exit(103c410,1,22ad8ce8,0,0,...) at fork_exit+0x6c/frame 0x22ad8cd4 fork_trampoline() at 0xffc033ca/frame 0x22ad8cd4 --- trap 0, eip = 0, esp = 0x22ad8d20, ebp = 0 --- (null)() at 0 Tracing command fdc0 pid 8 tid 100083 td 0x23d89a80 sched_switch(23d89a80,0,104,197,11e68dec,...) at sched_switch+0x598/frame 0x22ac9a60 mi_switch(104,0) at mi_switch+0x516/frame 0x22ac9b20 sleepq_switch(0,22ac9b60,15a7839,22ac9c3c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x22ac9b44 sleepq_wait(216b483c,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x22ac9b58 _sleep(216b483c,216b48f8,5c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x22ac9c3c fdc_thread(216b4800,22ac9ce8) at fdc_thread+0x10f1/frame 0x22ac9cb4 fork_exit(14a8830,216b4800,22ac9ce8,0,0,...) at fork_exit+0x6c/frame 0x22ac9cd4 fork_trampoline() at 0xffc033ca/frame 0x22ac9cd4 --- trap 0, eip = 0, esp = 0x22ac9d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100048 td 0x21722000 sched_switch(21722000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a13b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a13c24 sleepq_switch(21722000,216facf8,216faacc,22a13c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a13c48 sleepq_wait(216faacc,0) at sleepq_wait+0x3a/frame 0x22a13c5c _cv_wait(216faacc,216facf8) at _cv_wait+0x135/frame 0x22a13c94 usb_process(216faac4,22a13ce8) at usb_process+0x104/frame 0x22a13cb4 fork_exit(e3cfb0,216faac4,22a13ce8,0,0,...) at fork_exit+0x6c/frame 0x22a13cd4 fork_trampoline() at 0xffc033ca/frame 0x22a13cd4 --- trap 0, eip = 0, esp = 0x22a13d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100049 td 0x21721a80 sched_switch(21721a80,0,104,197,0,...) at sched_switch+0x598/frame 0x22a18b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a18c24 sleepq_switch(21721a80,216facf8,216faafc,22a18c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a18c48 sleepq_wait(216faafc,0) at sleepq_wait+0x3a/frame 0x22a18c5c _cv_wait(216faafc,216facf8) at _cv_wait+0x135/frame 0x22a18c94 usb_process(216faaf4,22a18ce8) at usb_process+0x104/frame 0x22a18cb4 fork_exit(e3cfb0,216faaf4,22a18ce8,0,0,...) at fork_exit+0x6c/frame 0x22a18cd4 fork_trampoline() at 0xffc033ca/frame 0x22a18cd4 --- trap 0, eip = 0, esp = 0x22a18d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100050 td 0x21721700 sched_switch(21721700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a1db60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a1dc24 sleepq_switch(21721700,216facf8,216fab2c,22a1dc94,f72175,...) at sleepq_switch+0xe1/frame 0x22a1dc48 sleepq_wait(216fab2c,0) at sleepq_wait+0x3a/frame 0x22a1dc5c _cv_wait(216fab2c,216facf8) at _cv_wait+0x135/frame 0x22a1dc94 usb_process(216fab24,22a1dce8) at usb_process+0x104/frame 0x22a1dcb4 fork_exit(e3cfb0,216fab24,22a1dce8,0,0,...) at fork_exit+0x6c/frame 0x22a1dcd4 fork_trampoline() at 0xffc033ca/frame 0x22a1dcd4 --- trap 0, eip = 0, esp = 0x22a1dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100051 td 0x21721380 sched_switch(21721380,0,104,197,23a98000,...) at sched_switch+0x598/frame 0x22a22b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a22c24 sleepq_switch(21721380,216facf8,216fab5c,22a22c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a22c48 sleepq_wait(216fab5c,0) at sleepq_wait+0x3a/frame 0x22a22c5c _cv_wait(216fab5c,216facf8) at _cv_wait+0x135/frame 0x22a22c94 usb_process(216fab54,22a22ce8) at usb_process+0x104/frame 0x22a22cb4 fork_exit(e3cfb0,216fab54,22a22ce8,0,0,...) at fork_exit+0x6c/frame 0x22a22cd4 fork_trampoline() at 0xffc033ca/frame 0x22a22cd4 --- trap 0, eip = 0, esp = 0x22a22d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100052 td 0x21721000 sched_switch(21721000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a27b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a27c24 sleepq_switch(21721000,216facf8,216fab8c,22a27c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a27c48 sleepq_wait(216fab8c,0) at sleepq_wait+0x3a/frame 0x22a27c5c _cv_wait(216fab8c,216facf8) at _cv_wait+0x135/frame 0x22a27c94 usb_process(216fab84,22a27ce8) at usb_process+0x104/frame 0x22a27cb4 fork_exit(e3cfb0,216fab84,22a27ce8,0,0,...) at fork_exit+0x6c/frame 0x22a27cd4 fork_trampoline() at 0xffc033ca/frame 0x22a27cd4 --- trap 0, eip = 0, esp = 0x22a27d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100054 td 0x21710700 sched_switch(21710700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a31b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a31c24 sleepq_switch(21710700,23a7bcf8,23a7bacc,22a31c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a31c48 sleepq_wait(23a7bacc,0) at sleepq_wait+0x3a/frame 0x22a31c5c _cv_wait(23a7bacc,23a7bcf8) at _cv_wait+0x135/frame 0x22a31c94 usb_process(23a7bac4,22a31ce8) at usb_process+0x104/frame 0x22a31cb4 fork_exit(e3cfb0,23a7bac4,22a31ce8,0,0,...) at fork_exit+0x6c/frame 0x22a31cd4 fork_trampoline() at 0xffc033ca/frame 0x22a31cd4 --- trap 0, eip = 0, esp = 0x22a31d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100055 td 0x21710380 sched_switch(21710380,0,104,197,0,...) at sched_switch+0x598/frame 0x22a36b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a36c24 sleepq_switch(21710380,23a7bcf8,23a7bafc,22a36c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a36c48 sleepq_wait(23a7bafc,0) at sleepq_wait+0x3a/frame 0x22a36c5c _cv_wait(23a7bafc,23a7bcf8) at _cv_wait+0x135/frame 0x22a36c94 usb_process(23a7baf4,22a36ce8) at usb_process+0x104/frame 0x22a36cb4 fork_exit(e3cfb0,23a7baf4,22a36ce8,0,0,...) at fork_exit+0x6c/frame 0x22a36cd4 fork_trampoline() at 0xffc033ca/frame 0x22a36cd4 --- trap 0, eip = 0, esp = 0x22a36d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100056 td 0x21710000 sched_switch(21710000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a3bb60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a3bc24 sleepq_switch(21710000,23a7bcf8,23a7bb2c,22a3bc94,f72175,...) at sleepq_switch+0xe1/frame 0x22a3bc48 sleepq_wait(23a7bb2c,0) at sleepq_wait+0x3a/frame 0x22a3bc5c _cv_wait(23a7bb2c,23a7bcf8) at _cv_wait+0x135/frame 0x22a3bc94 usb_process(23a7bb24,22a3bce8) at usb_process+0x104/frame 0x22a3bcb4 fork_exit(e3cfb0,23a7bb24,22a3bce8,0,0,...) at fork_exit+0x6c/frame 0x22a3bcd4 fork_trampoline() at 0xffc033ca/frame 0x22a3bcd4 --- trap 0, eip = 0, esp = 0x22a3bd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100057 td 0x23ad2380 sched_switch(23ad2380,0,104,197,23a97800,...) at sched_switch+0x598/frame 0x22a40b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a40c24 sleepq_switch(23ad2380,23a7bcf8,23a7bb5c,22a40c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a40c48 sleepq_wait(23a7bb5c,0) at sleepq_wait+0x3a/frame 0x22a40c5c _cv_wait(23a7bb5c,23a7bcf8) at _cv_wait+0x135/frame 0x22a40c94 usb_process(23a7bb54,22a40ce8) at usb_process+0x104/frame 0x22a40cb4 fork_exit(e3cfb0,23a7bb54,22a40ce8,0,0,...) at fork_exit+0x6c/frame 0x22a40cd4 fork_trampoline() at 0xffc033ca/frame 0x22a40cd4 --- trap 0, eip = 0, esp = 0x22a40d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100058 td 0x23ad2000 sched_switch(23ad2000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a45b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a45c24 sleepq_switch(23ad2000,23a7bcf8,23a7bb8c,22a45c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a45c48 sleepq_wait(23a7bb8c,0) at sleepq_wait+0x3a/frame 0x22a45c5c _cv_wait(23a7bb8c,23a7bcf8) at _cv_wait+0x135/frame 0x22a45c94 usb_process(23a7bb84,22a45ce8) at usb_process+0x104/frame 0x22a45cb4 fork_exit(e3cfb0,23a7bb84,22a45ce8,0,0,...) at fork_exit+0x6c/frame 0x22a45cd4 fork_trampoline() at 0xffc033ca/frame 0x22a45cd4 --- trap 0, eip = 0, esp = 0x22a45d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100060 td 0x23ad1700 sched_switch(23ad1700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a4fb60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a4fc24 sleepq_switch(23ad1700,23ad3cf8,23ad3acc,22a4fc94,f72175,...) at sleepq_switch+0xe1/frame 0x22a4fc48 sleepq_wait(23ad3acc,0) at sleepq_wait+0x3a/frame 0x22a4fc5c _cv_wait(23ad3acc,23ad3cf8) at _cv_wait+0x135/frame 0x22a4fc94 usb_process(23ad3ac4,22a4fce8) at usb_process+0x104/frame 0x22a4fcb4 fork_exit(e3cfb0,23ad3ac4,22a4fce8,0,0,...) at fork_exit+0x6c/frame 0x22a4fcd4 fork_trampoline() at 0xffc033ca/frame 0x22a4fcd4 --- trap 0, eip = 0, esp = 0x22a4fd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100061 td 0x23ad1380 sched_switch(23ad1380,0,104,197,0,...) at sched_switch+0x598/frame 0x22a54b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a54c24 sleepq_switch(23ad1380,23ad3cf8,23ad3afc,22a54c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a54c48 sleepq_wait(23ad3afc,0) at sleepq_wait+0x3a/frame 0x22a54c5c _cv_wait(23ad3afc,23ad3cf8) at _cv_wait+0x135/frame 0x22a54c94 usb_process(23ad3af4,22a54ce8) at usb_process+0x104/frame 0x22a54cb4 fork_exit(e3cfb0,23ad3af4,22a54ce8,0,0,...) at fork_exit+0x6c/frame 0x22a54cd4 fork_trampoline() at 0xffc033ca/frame 0x22a54cd4 --- trap 0, eip = 0, esp = 0x22a54d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100062 td 0x23ad1000 sched_switch(23ad1000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a59b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a59c24 sleepq_switch(23ad1000,23ad3cf8,23ad3b2c,22a59c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a59c48 sleepq_wait(23ad3b2c,0) at sleepq_wait+0x3a/frame 0x22a59c5c _cv_wait(23ad3b2c,23ad3cf8) at _cv_wait+0x135/frame 0x22a59c94 usb_process(23ad3b24,22a59ce8) at usb_process+0x104/frame 0x22a59cb4 fork_exit(e3cfb0,23ad3b24,22a59ce8,0,0,...) at fork_exit+0x6c/frame 0x22a59cd4 fork_trampoline() at 0xffc033ca/frame 0x22a59cd4 --- trap 0, eip = 0, esp = 0x22a59d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100063 td 0x23ad0a80 sched_switch(23ad0a80,0,104,197,11bdc000,...) at sched_switch+0x598/frame 0x22a5eb60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a5ec24 sleepq_switch(23ad0a80,23ad3cf8,23ad3b5c,22a5ec94,f72175,...) at sleepq_switch+0xe1/frame 0x22a5ec48 sleepq_wait(23ad3b5c,0) at sleepq_wait+0x3a/frame 0x22a5ec5c _cv_wait(23ad3b5c,23ad3cf8) at _cv_wait+0x135/frame 0x22a5ec94 usb_process(23ad3b54,22a5ece8) at usb_process+0x104/frame 0x22a5ecb4 fork_exit(e3cfb0,23ad3b54,22a5ece8,0,0,...) at fork_exit+0x6c/frame 0x22a5ecd4 fork_trampoline() at 0xffc033ca/frame 0x22a5ecd4 --- trap 0, eip = 0, esp = 0x22a5ed20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100064 td 0x23ad0700 sched_switch(23ad0700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a63b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a63c24 sleepq_switch(23ad0700,23ad3cf8,23ad3b8c,22a63c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a63c48 sleepq_wait(23ad3b8c,0) at sleepq_wait+0x3a/frame 0x22a63c5c _cv_wait(23ad3b8c,23ad3cf8) at _cv_wait+0x135/frame 0x22a63c94 usb_process(23ad3b84,22a63ce8) at usb_process+0x104/frame 0x22a63cb4 fork_exit(e3cfb0,23ad3b84,22a63ce8,0,0,...) at fork_exit+0x6c/frame 0x22a63cd4 fork_trampoline() at 0xffc033ca/frame 0x22a63cd4 --- trap 0, eip = 0, esp = 0x22a63d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100065 td 0x23ad0380 sched_switch(23ad0380,0,104,197,0,...) at sched_switch+0x598/frame 0x22a69b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a69c24 sleepq_switch(23ad0380,23b19cf8,23b19acc,22a69c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a69c48 sleepq_wait(23b19acc,0) at sleepq_wait+0x3a/frame 0x22a69c5c _cv_wait(23b19acc,23b19cf8) at _cv_wait+0x135/frame 0x22a69c94 usb_process(23b19ac4,22a69ce8) at usb_process+0x104/frame 0x22a69cb4 fork_exit(e3cfb0,23b19ac4,22a69ce8,0,0,...) at fork_exit+0x6c/frame 0x22a69cd4 fork_trampoline() at 0xffc033ca/frame 0x22a69cd4 --- trap 0, eip = 0, esp = 0x22a69d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100066 td 0x23ad0000 sched_switch(23ad0000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a6eb60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a6ec24 sleepq_switch(23ad0000,23b19cf8,23b19afc,22a6ec94,f72175,...) at sleepq_switch+0xe1/frame 0x22a6ec48 sleepq_wait(23b19afc,0) at sleepq_wait+0x3a/frame 0x22a6ec5c _cv_wait(23b19afc,23b19cf8) at _cv_wait+0x135/frame 0x22a6ec94 usb_process(23b19af4,22a6ece8) at usb_process+0x104/frame 0x22a6ecb4 fork_exit(e3cfb0,23b19af4,22a6ece8,0,0,...) at fork_exit+0x6c/frame 0x22a6ecd4 fork_trampoline() at 0xffc033ca/frame 0x22a6ecd4 --- trap 0, eip = 0, esp = 0x22a6ed20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100067 td 0x21723a80 sched_switch(21723a80,0,104,197,0,...) at sched_switch+0x598/frame 0x22a73b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a73c24 sleepq_switch(21723a80,23b19cf8,23b19b2c,22a73c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a73c48 sleepq_wait(23b19b2c,0) at sleepq_wait+0x3a/frame 0x22a73c5c _cv_wait(23b19b2c,23b19cf8) at _cv_wait+0x135/frame 0x22a73c94 usb_process(23b19b24,22a73ce8) at usb_process+0x104/frame 0x22a73cb4 fork_exit(e3cfb0,23b19b24,22a73ce8,0,0,...) at fork_exit+0x6c/frame 0x22a73cd4 fork_trampoline() at 0xffc033ca/frame 0x22a73cd4 --- trap 0, eip = 0, esp = 0x22a73d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100068 td 0x21723700 sched_switch(21723700,0,104,197,23b7c800,...) at sched_switch+0x598/frame 0x22a78b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a78c24 sleepq_switch(21723700,23b19cf8,23b19b5c,22a78c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a78c48 sleepq_wait(23b19b5c,0) at sleepq_wait+0x3a/frame 0x22a78c5c _cv_wait(23b19b5c,23b19cf8) at _cv_wait+0x135/frame 0x22a78c94 usb_process(23b19b54,22a78ce8) at usb_process+0x104/frame 0x22a78cb4 fork_exit(e3cfb0,23b19b54,22a78ce8,0,0,...) at fork_exit+0x6c/frame 0x22a78cd4 fork_trampoline() at 0xffc033ca/frame 0x22a78cd4 --- trap 0, eip = 0, esp = 0x22a78d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100069 td 0x21723380 sched_switch(21723380,0,104,197,0,...) at sched_switch+0x598/frame 0x22a7db60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a7dc24 sleepq_switch(21723380,23b19cf8,23b19b8c,22a7dc94,f72175,...) at sleepq_switch+0xe1/frame 0x22a7dc48 sleepq_wait(23b19b8c,0) at sleepq_wait+0x3a/frame 0x22a7dc5c _cv_wait(23b19b8c,23b19cf8) at _cv_wait+0x135/frame 0x22a7dc94 usb_process(23b19b84,22a7dce8) at usb_process+0x104/frame 0x22a7dcb4 fork_exit(e3cfb0,23b19b84,22a7dce8,0,0,...) at fork_exit+0x6c/frame 0x22a7dcd4 fork_trampoline() at 0xffc033ca/frame 0x22a7dcd4 --- trap 0, eip = 0, esp = 0x22a7dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100070 td 0x23bac700 sched_switch(23bac700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a83b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a83c24 sleepq_switch(23bac700,23b62cf8,23b62acc,22a83c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a83c48 sleepq_wait(23b62acc,0) at sleepq_wait+0x3a/frame 0x22a83c5c _cv_wait(23b62acc,23b62cf8) at _cv_wait+0x135/frame 0x22a83c94 usb_process(23b62ac4,22a83ce8) at usb_process+0x104/frame 0x22a83cb4 fork_exit(e3cfb0,23b62ac4,22a83ce8,0,0,...) at fork_exit+0x6c/frame 0x22a83cd4 fork_trampoline() at 0xffc033ca/frame 0x22a83cd4 --- trap 0, eip = 0, esp = 0x22a83d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100071 td 0x23bac380 sched_switch(23bac380,0,104,197,0,...) at sched_switch+0x598/frame 0x22a88b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a88c24 sleepq_switch(23bac380,23b62cf8,23b62afc,22a88c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a88c48 sleepq_wait(23b62afc,0) at sleepq_wait+0x3a/frame 0x22a88c5c _cv_wait(23b62afc,23b62cf8) at _cv_wait+0x135/frame 0x22a88c94 usb_process(23b62af4,22a88ce8) at usb_process+0x104/frame 0x22a88cb4 fork_exit(e3cfb0,23b62af4,22a88ce8,0,0,...) at fork_exit+0x6c/frame 0x22a88cd4 fork_trampoline() at 0xffc033ca/frame 0x22a88cd4 --- trap 0, eip = 0, esp = 0x22a88d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100072 td 0x23bac000 sched_switch(23bac000,0,104,197,0,...) at sched_switch+0x598/frame 0x22a8db60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a8dc24 sleepq_switch(23bac000,23b62cf8,23b62b2c,22a8dc94,f72175,...) at sleepq_switch+0xe1/frame 0x22a8dc48 sleepq_wait(23b62b2c,0) at sleepq_wait+0x3a/frame 0x22a8dc5c _cv_wait(23b62b2c,23b62cf8) at _cv_wait+0x135/frame 0x22a8dc94 usb_process(23b62b24,22a8dce8) at usb_process+0x104/frame 0x22a8dcb4 fork_exit(e3cfb0,23b62b24,22a8dce8,0,0,...) at fork_exit+0x6c/frame 0x22a8dcd4 fork_trampoline() at 0xffc033ca/frame 0x22a8dcd4 --- trap 0, eip = 0, esp = 0x22a8dd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100073 td 0x23baba80 sched_switch(23baba80,0,104,197,23b7c000,...) at sched_switch+0x598/frame 0x22a92b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a92c24 sleepq_switch(23baba80,23b62cf8,23b62b5c,22a92c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a92c48 sleepq_wait(23b62b5c,0) at sleepq_wait+0x3a/frame 0x22a92c5c _cv_wait(23b62b5c,23b62cf8) at _cv_wait+0x135/frame 0x22a92c94 usb_process(23b62b54,22a92ce8) at usb_process+0x104/frame 0x22a92cb4 fork_exit(e3cfb0,23b62b54,22a92ce8,0,0,...) at fork_exit+0x6c/frame 0x22a92cd4 fork_trampoline() at 0xffc033ca/frame 0x22a92cd4 --- trap 0, eip = 0, esp = 0x22a92d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100074 td 0x23bab700 sched_switch(23bab700,0,104,197,0,...) at sched_switch+0x598/frame 0x22a97b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22a97c24 sleepq_switch(23bab700,23b62cf8,23b62b8c,22a97c94,f72175,...) at sleepq_switch+0xe1/frame 0x22a97c48 sleepq_wait(23b62b8c,0) at sleepq_wait+0x3a/frame 0x22a97c5c _cv_wait(23b62b8c,23b62cf8) at _cv_wait+0x135/frame 0x22a97c94 usb_process(23b62b84,22a97ce8) at usb_process+0x104/frame 0x22a97cb4 fork_exit(e3cfb0,23b62b84,22a97ce8,0,0,...) at fork_exit+0x6c/frame 0x22a97cd4 fork_trampoline() at 0xffc033ca/frame 0x22a97cd4 --- trap 0, eip = 0, esp = 0x22a97d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100076 td 0x23bab000 sched_switch(23bab000,0,104,197,0,...) at sched_switch+0x598/frame 0x22aa2b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22aa2c24 sleepq_switch(23bab000,216f4ed0,216f4ca4,22aa2c94,f72175,...) at sleepq_switch+0xe1/frame 0x22aa2c48 sleepq_wait(216f4ca4,0) at sleepq_wait+0x3a/frame 0x22aa2c5c _cv_wait(216f4ca4,216f4ed0) at _cv_wait+0x135/frame 0x22aa2c94 usb_process(216f4c9c,22aa2ce8) at usb_process+0x104/frame 0x22aa2cb4 fork_exit(e3cfb0,216f4c9c,22aa2ce8,0,0,...) at fork_exit+0x6c/frame 0x22aa2cd4 fork_trampoline() at 0xffc033ca/frame 0x22aa2cd4 --- trap 0, eip = 0, esp = 0x22aa2d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100077 td 0x23baaa80 sched_switch(23baaa80,0,104,197,0,...) at sched_switch+0x598/frame 0x22aa7b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22aa7c24 sleepq_switch(23baaa80,216f4ed0,216f4cd4,22aa7c94,f72175,...) at sleepq_switch+0xe1/frame 0x22aa7c48 sleepq_wait(216f4cd4,0) at sleepq_wait+0x3a/frame 0x22aa7c5c _cv_wait(216f4cd4,216f4ed0) at _cv_wait+0x135/frame 0x22aa7c94 usb_process(216f4ccc,22aa7ce8) at usb_process+0x104/frame 0x22aa7cb4 fork_exit(e3cfb0,216f4ccc,22aa7ce8,0,0,...) at fork_exit+0x6c/frame 0x22aa7cd4 fork_trampoline() at 0xffc033ca/frame 0x22aa7cd4 --- trap 0, eip = 0, esp = 0x22aa7d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100078 td 0x23baa700 sched_switch(23baa700,0,104,197,0,...) at sched_switch+0x598/frame 0x22aacb60 mi_switch(104,0) at mi_switch+0x516/frame 0x22aacc24 sleepq_switch(23baa700,216f4ed0,216f4d04,22aacc94,f72175,...) at sleepq_switch+0xe1/frame 0x22aacc48 sleepq_wait(216f4d04,0) at sleepq_wait+0x3a/frame 0x22aacc5c _cv_wait(216f4d04,216f4ed0) at _cv_wait+0x135/frame 0x22aacc94 usb_process(216f4cfc,22aacce8) at usb_process+0x104/frame 0x22aaccb4 fork_exit(e3cfb0,216f4cfc,22aacce8,0,0,...) at fork_exit+0x6c/frame 0x22aaccd4 fork_trampoline() at 0xffc033ca/frame 0x22aaccd4 --- trap 0, eip = 0, esp = 0x22aacd20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100079 td 0x23baa380 sched_switch(23baa380,0,104,197,23b7b800,...) at sched_switch+0x598/frame 0x22ab1b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22ab1c24 sleepq_switch(23baa380,216f4ed0,216f4d34,22ab1c94,f72175,...) at sleepq_switch+0xe1/frame 0x22ab1c48 sleepq_wait(216f4d34,0) at sleepq_wait+0x3a/frame 0x22ab1c5c _cv_wait(216f4d34,216f4ed0) at _cv_wait+0x135/frame 0x22ab1c94 usb_process(216f4d2c,22ab1ce8) at usb_process+0x104/frame 0x22ab1cb4 fork_exit(e3cfb0,216f4d2c,22ab1ce8,0,0,...) at fork_exit+0x6c/frame 0x22ab1cd4 fork_trampoline() at 0xffc033ca/frame 0x22ab1cd4 --- trap 0, eip = 0, esp = 0x22ab1d20, ebp = 0 --- (null)() at 0 Tracing command usb pid 15 tid 100080 td 0x23baa000 sched_switch(23baa000,0,104,197,0,...) at sched_switch+0x598/frame 0x22ab6b60 mi_switch(104,0) at mi_switch+0x516/frame 0x22ab6c24 sleepq_switch(23baa000,216f4ed0,216f4d64,22ab6c94,f72175,...) at sleepq_switch+0xe1/frame 0x22ab6c48 sleepq_wait(216f4d64,0) at sleepq_wait+0x3a/frame 0x22ab6c5c _cv_wait(216f4d64,216f4ed0) at _cv_wait+0x135/frame 0x22ab6c94 usb_process(216f4d5c,22ab6ce8) at usb_process+0x104/frame 0x22ab6cb4 fork_exit(e3cfb0,216f4d5c,22ab6ce8,0,0,...) at fork_exit+0x6c/frame 0x22ab6cd4 fork_trampoline() at 0xffc033ca/frame 0x22ab6cd4 --- trap 0, eip = 0, esp = 0x22ab6d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100043 td 0x11bb3000 sched_switch(11bb3000,0,104,197,2d2340d0,...) at sched_switch+0x598/frame 0x1b7c8a90 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7c8b50 sleepq_switch(0,1b7c8b90,15a7839,1b7c8c6c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7c8b74 sleepq_wait(1c6c140,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b7c8b88 _sleep(1c6c140,1c6c100,5c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7c8c6c xpt_done_td(1c6c100,1b7c8ce8) at xpt_done_td+0x72/frame 0x1b7c8cb4 fork_exit(90a1f0,1c6c100,1b7c8ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7c8cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7c8cd4 --- trap 0, eip = 0, esp = 0x1b7c8d20, ebp = 0 --- (null)() at 0 Tracing command cam pid 7 tid 100096 td 0x23ff8700 sched_switch(23ff8700,0,104,197,0,...) at sched_switch+0x598/frame 0x22b13a88 mi_switch(104,0) at mi_switch+0x516/frame 0x22b13b48 sleepq_switch(0,22b13b88,15a7839,22b13c64,fd8f1c,...) at sleepq_switch+0xe1/frame 0x22b13b6c sleepq_wait(1c6c06c,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x22b13b80 _sleep(1c6c06c,1c6c0cc,5c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x22b13c64 xpt_scanner_thread(0,22b13ce8) at xpt_scanner_thread+0x63/frame 0x22b13cb4 fork_exit(90a490,0,22b13ce8,0,0,...) at fork_exit+0x6c/frame 0x22b13cd4 fork_trampoline() at 0xffc033ca/frame 0x22b13cd4 --- trap 0, eip = 0, esp = 0x22b13d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 3 pid 6 tid 100042 td 0x11bb3380 sched_switch(11bb3380,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7c3a88 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7c3b48 sleepq_switch(0,1b7c3b88,16881c3,1b7c3c64,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7c3b6c sleepq_wait(11ad0dc4,6c,16881c3,0,0,...) at sleepq_wait+0x3a/frame 0x1b7c3b80 _sleep(11ad0dc4,11ad0da8,6c,16881c3,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7c3c64 crypto_ret_proc(11ad0da8,1b7c3ce8) at crypto_ret_proc+0x1e9/frame 0x1b7c3cb4 fork_exit(1233a50,11ad0da8,1b7c3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7c3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7c3cd4 --- trap 0, eip = 0, esp = 0x1b7c3d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 2 pid 5 tid 100041 td 0x11bb3700 sched_switch(11bb3700,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7bea88 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7beb48 sleepq_switch(0,1b7beb88,16881c3,1b7bec64,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7beb6c sleepq_wait(11ad0d8c,6c,16881c3,0,0,...) at sleepq_wait+0x3a/frame 0x1b7beb80 _sleep(11ad0d8c,11ad0d70,6c,16881c3,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7bec64 crypto_ret_proc(11ad0d70,1b7bece8) at crypto_ret_proc+0x1e9/frame 0x1b7becb4 fork_exit(1233a50,11ad0d70,1b7bece8,0,0,...) at fork_exit+0x6c/frame 0x1b7becd4 fork_trampoline() at 0xffc033ca/frame 0x1b7becd4 --- trap 0, eip = 0, esp = 0x1b7bed20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 1 pid 4 tid 100040 td 0x11bb3a80 sched_switch(11bb3a80,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7b9a88 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7b9b48 sleepq_switch(0,1b7b9b88,16881c3,1b7b9c64,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7b9b6c sleepq_wait(11ad0d54,6c,16881c3,0,0,...) at sleepq_wait+0x3a/frame 0x1b7b9b80 _sleep(11ad0d54,11ad0d38,6c,16881c3,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7b9c64 crypto_ret_proc(11ad0d38,1b7b9ce8) at crypto_ret_proc+0x1e9/frame 0x1b7b9cb4 fork_exit(1233a50,11ad0d38,1b7b9ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7b9cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7b9cd4 --- trap 0, eip = 0, esp = 0x1b7b9d20, ebp = 0 --- (null)() at 0 Tracing command crypto returns 0 pid 3 tid 100039 td 0x11ad9700 sched_switch(11ad9700,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7b4a88 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7b4b48 sleepq_switch(0,1b7b4b88,16881c3,1b7b4c64,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7b4b6c sleepq_wait(11ad0d1c,6c,16881c3,0,0,...) at sleepq_wait+0x3a/frame 0x1b7b4b80 _sleep(11ad0d1c,11ad0d00,6c,16881c3,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7b4c64 crypto_ret_proc(11ad0d00,1b7b4ce8) at crypto_ret_proc+0x1e9/frame 0x1b7b4cb4 fork_exit(1233a50,11ad0d00,1b7b4ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7b4cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7b4cd4 --- trap 0, eip = 0, esp = 0x1b7b4d20, ebp = 0 --- (null)() at 0 Tracing command crypto pid 2 tid 100038 td 0x11ad9a80 sched_switch(11ad9a80,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7afa98 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7afb58 sleepq_switch(0,1b7afb98,163cb72,1b7afc74,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7afb7c sleepq_wait(1f6fff8,6c,163cb72,0,0,...) at sleepq_wait+0x3a/frame 0x1b7afb90 _sleep(1f6fff8,1f6ffe0,6c,163cb72,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7afc74 crypto_proc(0,1b7afce8) at crypto_proc+0x250/frame 0x1b7afcb4 fork_exit(12336d0,0,1b7afce8,0,0,...) at fork_exit+0x6c/frame 0x1b7afcd4 fork_trampoline() at 0xffc033ca/frame 0x1b7afcd4 --- trap 0, eip = 0, esp = 0x1b7afd20, ebp = 0 --- (null)() at 0 Tracing command sequencer 00 pid 14 tid 100033 td 0x11e6a000 sched_switch(11e6a000,0,104,197,0,...) at sched_switch+0x598/frame 0x1b796b38 mi_switch(104,0) at mi_switch+0x516/frame 0x1b796bfc sleepq_switch(11e6a000,11b9e804,11b9e84c,1b796c6c,f72175,...) at sleepq_switch+0xe1/frame 0x1b796c20 sleepq_wait(11b9e84c,0) at sleepq_wait+0x3a/frame 0x1b796c34 _cv_wait(11b9e84c,11b9e804) at _cv_wait+0x135/frame 0x1b796c6c seq_eventthread(11b9e800,1b796ce8) at seq_eventthread+0xcb/frame 0x1b796cb4 fork_exit(dc54b0,11b9e800,1b796ce8,0,0,...) at fork_exit+0x6c/frame 0x1b796cd4 fork_trampoline() at 0xffc033ca/frame 0x1b796cd4 --- trap 0, eip = 0, esp = 0x1b796d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100030 td 0x11e6aa80 sched_switch(11e6aa80,0,104,197,1b783b04,...) at sched_switch+0x598/frame 0x1b783aa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b783b60 sleepq_switch(0,1b783ba0,15a7839,1b783c7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b783b84 sleepq_wait(1f94868,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b783b98 _sleep(1f94868,1ec7fec,25c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b783c7c g_run_events(0,1b783ce8) at g_run_events+0x47/frame 0x1b783cb4 fork_exit(f28270,0,1b783ce8,0,0,...) at fork_exit+0x6c/frame 0x1b783cd4 fork_trampoline() at 0xffc033ca/frame 0x1b783cd4 --- trap 0, eip = 0, esp = 0x1b783d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100031 td 0x11e6a700 sched_switch(11e6a700,0,104,197,0,...) at sched_switch+0x598/frame 0x1b788a98 mi_switch(104,0) at mi_switch+0x516/frame 0x1b788b58 sleepq_switch(0,1b788b98,15a7839,1b788c74,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b788b7c sleepq_wait(1f94870,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b788b90 _sleep(1f94870,1ec8034,25c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b788c74 g_io_schedule_up(11e6a700) at g_io_schedule_up+0x8e/frame 0x1b788ca8 g_up_procbody(0,1b788ce8) at g_up_procbody+0x5b/frame 0x1b788cb4 fork_exit(f282c0,0,1b788ce8,0,0,...) at fork_exit+0x6c/frame 0x1b788cd4 fork_trampoline() at 0xffc033ca/frame 0x1b788cd4 --- trap 0, eip = 0, esp = 0x1b788d20, ebp = 0 --- (null)() at 0 Tracing command geom pid 13 tid 100032 td 0x11e6a380 sched_switch(11e6a380,0,104,197,1b78db00,...) at sched_switch+0x598/frame 0x1b78da90 mi_switch(104,0) at mi_switch+0x516/frame 0x1b78db50 sleepq_switch(0,1b78db90,15a7839,1b78dc70,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b78db74 sleepq_wait(1f94864,5c,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b78db88 _sleep(1f94864,1ec8014,25c,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b78dc70 g_io_schedule_down(11e6a380) at g_io_schedule_down+0xa6/frame 0x1b78dca8 g_down_procbody(0,1b78dce8) at g_down_procbody+0x5b/frame 0x1b78dcb4 fork_exit(f28320,0,1b78dce8,0,0,...) at fork_exit+0x6c/frame 0x1b78dcd4 fork_trampoline() at 0xffc033ca/frame 0x1b78dcd4 --- trap 0, eip = 0, esp = 0x1b78dd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100008 td 0x11fd6380 sched_switch(11fd6380,0,109,197,8,...) at sched_switch+0x598/frame 0x1b6f1bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x1b6f1c80 ithread_loop(11ad5020,1b6f1ce8) at ithread_loop+0x2f5/frame 0x1b6f1cb4 fork_exit(f96980,11ad5020,1b6f1ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6f1cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6f1cd4 --- trap 0, eip = 0, esp = 0x1b6f1d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100009 td 0x11fd6000 sched_switch(11fd6000,0,109,197,8,...) at sched_switch+0x598/frame 0x1b6f6bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x1b6f6c80 ithread_loop(11ad5010,1b6f6ce8) at ithread_loop+0x2f5/frame 0x1b6f6cb4 fork_exit(f96980,11ad5010,1b6f6ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6f6cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6f6cd4 --- trap 0, eip = 0, esp = 0x1b6f6d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100011 td 0x11fd0700 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100023 td 0x11ad7000 sched_switch(11ad7000,0,109,197,1fa20c0,...) at sched_switch+0x598/frame 0x1b748bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x1b748c80 ithread_loop(11ad4f90,1b748ce8) at ithread_loop+0x2f5/frame 0x1b748cb4 fork_exit(f96980,11ad4f90,1b748ce8,0,0,...) at fork_exit+0x6c/frame 0x1b748cd4 fork_trampoline() at 0xffc033ca/frame 0x1b748cd4 --- trap 0, eip = 0, esp = 0x1b748d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100024 td 0x11fd8a80 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100025 td 0x11fd8700 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100026 td 0x11fd8380 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100027 td 0x11e6b700 sched_switch(11e6b700,0,109,197,fb0827,...) at sched_switch+0x598/frame 0x1b75cbc0 mi_switch(109,0) at mi_switch+0x516/frame 0x1b75cc80 ithread_loop(11ad4f50,1b75cce8) at ithread_loop+0x2f5/frame 0x1b75ccb4 fork_exit(f96980,11ad4f50,1b75cce8,0,0,...) at fork_exit+0x6c/frame 0x1b75ccd4 fork_trampoline() at 0xffc033ca/frame 0x1b75ccd4 --- trap 0, eip = 0, esp = 0x1b75cd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100028 td 0x11e6b380 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100044 td 0x21723000 sched_switch(21723000,0,109,197,21723000,...) at sched_switch+0x598/frame 0x1b7fabc0 mi_switch(109,0) at mi_switch+0x516/frame 0x1b7fac80 ithread_loop(2170f050,1b7face8) at ithread_loop+0x2f5/frame 0x1b7facb4 fork_exit(f96980,2170f050,1b7face8,0,0,...) at fork_exit+0x6c/frame 0x1b7facd4 fork_trampoline() at 0xffc033ca/frame 0x1b7facd4 --- trap 0, eip = 0, esp = 0x1b7fad20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100045 td 0x21722a80 sched_switch(21722a80,0,109,197,12b7a04,...) at sched_switch+0x598/frame 0x22a04bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22a04c80 ithread_loop(2170f020,22a04ce8) at ithread_loop+0x2f5/frame 0x22a04cb4 fork_exit(f96980,2170f020,22a04ce8,0,0,...) at fork_exit+0x6c/frame 0x22a04cd4 fork_trampoline() at 0xffc033ca/frame 0x22a04cd4 --- trap 0, eip = 0, esp = 0x22a04d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100046 td 0x21722700 sched_switch(21722700,0,109,197,1db,...) at sched_switch+0x598/frame 0x22a09bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22a09c80 ithread_loop(2170f010,22a09ce8) at ithread_loop+0x2f5/frame 0x22a09cb4 fork_exit(f96980,2170f010,22a09ce8,0,0,...) at fork_exit+0x6c/frame 0x22a09cd4 fork_trampoline() at 0xffc033ca/frame 0x22a09cd4 --- trap 0, eip = 0, esp = 0x22a09d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100047 td 0x21722380 sched_switch(21722380,0,109,197,21722380,...) at sched_switch+0x598/frame 0x22a0ebc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22a0ec80 ithread_loop(11ad5ea0,22a0ece8) at ithread_loop+0x2f5/frame 0x22a0ecb4 fork_exit(f96980,11ad5ea0,22a0ece8,0,0,...) at fork_exit+0x6c/frame 0x22a0ecd4 fork_trampoline() at 0xffc033ca/frame 0x22a0ecd4 --- trap 0, eip = 0, esp = 0x22a0ed20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100053 td 0x21710a80 sched_switch(21710a80,0,109,197,fb043a,...) at sched_switch+0x598/frame 0x22a2cbc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22a2cc80 ithread_loop(11ad5e90,22a2cce8) at ithread_loop+0x2f5/frame 0x22a2ccb4 fork_exit(f96980,11ad5e90,22a2cce8,0,0,...) at fork_exit+0x6c/frame 0x22a2ccd4 fork_trampoline() at 0xffc033ca/frame 0x22a2ccd4 --- trap 0, eip = 0, esp = 0x22a2cd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100059 td 0x23ad1a80 sched_switch(23ad1a80,0,109,197,fb043a,...) at sched_switch+0x598/frame 0x22a4abc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22a4ac80 ithread_loop(11ad5e80,22a4ace8) at ithread_loop+0x2f5/frame 0x22a4acb4 fork_exit(f96980,11ad5e80,22a4ace8,0,0,...) at fork_exit+0x6c/frame 0x22a4acd4 fork_trampoline() at 0xffc033ca/frame 0x22a4acd4 --- trap 0, eip = 0, esp = 0x22a4ad20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100075 td 0x23bab380 exit_tag() at 0xffc033c0 Tracing command intr pid 12 tid 100081 td 0x23ad2a80 sched_switch(23ad2a80,0,109,197,22abbbf0,...) at sched_switch+0x598/frame 0x22abbbc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22abbc80 ithread_loop(11ad5e60,22abbce8) at ithread_loop+0x2f5/frame 0x22abbcb4 fork_exit(f96980,11ad5e60,22abbce8,0,0,...) at fork_exit+0x6c/frame 0x22abbcd4 fork_trampoline() at 0xffc033ca/frame 0x22abbcd4 --- trap 0, eip = 0, esp = 0x22abbd20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100082 td 0x23ad2700 sched_switch(23ad2700,0,109,197,1f2cbe8,...) at sched_switch+0x598/frame 0x22ac4bc0 mi_switch(109,0) at mi_switch+0x516/frame 0x22ac4c80 ithread_loop(11ad5e10,22ac4ce8) at ithread_loop+0x2f5/frame 0x22ac4cb4 fork_exit(f96980,11ad5e10,22ac4ce8,0,0,...) at fork_exit+0x6c/frame 0x22ac4cd4 fork_trampoline() at 0xffc033ca/frame 0x22ac4cd4 --- trap 0, eip = 0, esp = 0x22ac4d20, ebp = 0 --- (null)() at 0 Tracing command intr pid 12 tid 100084 td 0x23d89700 exit_tag() at 0xffc033c0 Tracing command idle pid 11 tid 100003 td 0x11fd7700 (CPU 0) cpustop_handler(0,1b6d4bbc,1b6d4bb0,15306b2,0,...) at cpustop_handler+0x27/frame 0x1b6d4ae8 ipi_nmi_handler(0,1b6d4b28,11fd7700,24fedc0,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6d4af8 trap(1b6d4bbc,8,28,28,2167a800,...) at trap+0xc2/frame 0x1b6d4bb0 calltrap() at 0xffc0316d/frame 0x1b6d4bb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6d4bfc, ebp = 0x1b6d4bfc --- acpi_cpu_c1(0,0,df,9a80b4dd,24fedc0,...) at acpi_cpu_c1+0x5/frame 0x1b6d4bfc acpi_cpu_idle(984b1dd,0) at acpi_cpu_idle+0x230/frame 0x1b6d4c2c cpu_idle_acpi(984b1dd,0) at cpu_idle_acpi+0x35/frame 0x1b6d4c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6d4c5c sched_idletd(0,1b6d4ce8) at sched_idletd+0x3ee/frame 0x1b6d4cb4 fork_exit(ffb570,0,1b6d4ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6d4cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6d4cd4 --- trap 0, eip = 0, esp = 0x1b6d4d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100004 td 0x11fd7380 (CPU 1) cpustop_handler(1,1b6d9bbc,1b6d9bb0,15306b2,0,...) at cpustop_handler+0x27/frame 0x1b6d9ae8 ipi_nmi_handler(0,1b6d9b28,11fd7380,1eeefdc,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6d9af8 trap(1b6d9bbc,8,28,28,2167a600,...) at trap+0xc2/frame 0x1b6d9bb0 calltrap() at 0xffc0316d/frame 0x1b6d9bb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6d9bfc, ebp = 0x1b6d9bfc --- acpi_cpu_c1(0,1,df,99f7eb71,11e63dc0,...) at acpi_cpu_c1+0x5/frame 0x1b6d9bfc acpi_cpu_idle(1b1cfd3e,0) at acpi_cpu_idle+0x230/frame 0x1b6d9c2c cpu_idle_acpi(1b1cfd3e,0) at cpu_idle_acpi+0x35/frame 0x1b6d9c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6d9c5c sched_idletd(0,1b6d9ce8) at sched_idletd+0x3ee/frame 0x1b6d9cb4 fork_exit(ffb570,0,1b6d9ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6d9cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6d9cd4 --- trap 0, eip = 0, esp = 0x1b6d9d20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100005 td 0x11fd7000 (CPU 2) cpustop_handler(2,1b6debbc,1b6debb0,15306b2,1b6deb14,...) at cpustop_handler+0x27/frame 0x1b6deae8 ipi_nmi_handler(1b6deb14,fd6830,11fd7000,23fefa80,11fce6a8,...) at ipi_nmi_handler+0x34/frame 0x1b6deaf8 trap(1b6debbc,8,28,28,2167a400,...) at trap+0xc2/frame 0x1b6debb0 calltrap() at 0xffc0316d/frame 0x1b6debb0 --- trap 0x13, eip = 0x15466a5, esp = 0x1b6debfc, ebp = 0x1b6debfc --- acpi_cpu_c1(0,2,df,99f7c12a,11e68dc0,...) at acpi_cpu_c1+0x5/frame 0x1b6debfc acpi_cpu_idle(1b1d645a,0) at acpi_cpu_idle+0x230/frame 0x1b6dec2c cpu_idle_acpi(1b1d645a,0) at cpu_idle_acpi+0x35/frame 0x1b6dec40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6dec5c sched_idletd(0,1b6dece8) at sched_idletd+0x3ee/frame 0x1b6decb4 fork_exit(ffb570,0,1b6dece8,0,0,...) at fork_exit+0x6c/frame 0x1b6decd4 fork_trampoline() at 0xffc033ca/frame 0x1b6decd4 --- trap 0, eip = 0, esp = 0x1b6ded20, ebp = 0 --- (null)() at 0 Tracing command idle pid 11 tid 100006 td 0x11fd6a80 (CPU 3) kdb_alt_break_internal(0) at kdb_alt_break_internal+0xeb/frame 0x1b6e3b00 kdb_alt_break(2,23d6088c) at kdb_alt_break+0x10/frame 0x1b6e3b0c uart_intr_rxready(23d60800,86a,0,0,11fd6a80,...) at uart_intr_rxready+0x82/frame 0x1b6e3b28 uart_intr(23d60800) at uart_intr+0x10e/frame 0x1b6e3b48 intr_event_handle(11ad2c00,1b6e3bbc) at intr_event_handle+0xaf/frame 0x1b6e3b68 intr_execute_handlers(11ffe138,1b6e3bbc,3) at intr_execute_handlers+0xc7/frame 0x1b6e3b94 lapic_handle_intr(32,1b6e3bbc,8,28,28,...) at lapic_handle_intr+0x55/frame 0x1b6e3bac Xapic_isr1() at 0xffc04208/frame 0x1b6e3bac --- interrupt, eip = 0x15466a5, esp = 0x1b6e3bfc, ebp = 0x1b6e3bfc --- acpi_cpu_c1(0,3,df,99f6c3b2,11e6ddc0,...) at acpi_cpu_c1+0x5/frame 0x1b6e3bfc acpi_cpu_idle(51a6ebb6,0) at acpi_cpu_idle+0x230/frame 0x1b6e3c2c cpu_idle_acpi(51a6ebb6,0) at cpu_idle_acpi+0x35/frame 0x1b6e3c40 cpu_idle(0) at cpu_idle+0xa3/frame 0x1b6e3c5c sched_idletd(0,1b6e3ce8) at sched_idletd+0x3ee/frame 0x1b6e3cb4 fork_exit(ffb570,0,1b6e3ce8,0,0,...) at fork_exit+0x6c/frame 0x1b6e3cd4 fork_trampoline() at 0xffc033ca/frame 0x1b6e3cd4 --- trap 0, eip = 0, esp = 0x1b6e3d20, ebp = 0 --- (null)() at 0 Tracing command init pid 1 tid 100002 td 0x11fd7a80 sched_switch(11fd7a80,0,104,197,1fa8194,...) at sched_switch+0x598/frame 0x119fd880 mi_switch(104,0) at mi_switch+0x516/frame 0x119fd944 sleepq_switch(6c,11fd7a80,540,11fce9fc,1efd0d4,...) at sleepq_switch+0xe1/frame 0x119fd968 sleepq_catch_signals(100,119fd9b0,1672f8f,119fda8c,fd8efa,...) at sleepq_catch_signals+0x3b7/frame 0x119fd994 sleepq_wait_sig(11fce9fc,6c,1672f8f,100,0,...) at sleepq_wait_sig+0x11/frame 0x119fd9a8 _sleep(11fce9fc,11fceaa4,36c,1672f8f,0,0,0,0,100) at _sleep+0x7ca/frame 0x119fda8c kern_wait6(11fd7a80,7,0,0,119fdb14,32,0,0) at kern_wait6+0x3bd/frame 0x119fdae8 sys_wait4(11fd7a80,11fd7d08) at sys_wait4+0x78/frame 0x119fdc08 syscall(119fdce8,3b,3b,3b,400d0c,...) at syscall+0x2d9/frame 0x119fdcdc Xint0x80_syscall() at 0xffc033b7/frame 0x119fdcdc --- syscall (7, FreeBSD ELF32, sys_wait4), eip = 0x47f57f, esp = 0xffbfe868, ebp = 0xffbfe87c --- (null)() at 0x47f57f Tracing command audit pid 10 tid 100001 td 0x11fd8000 sched_switch(11fd8000,0,104,197,0,...) at sched_switch+0x598/frame 0x119f8b40 mi_switch(104,0) at mi_switch+0x516/frame 0x119f8c00 sleepq_switch(11fd8000,1fa504c,1fa50a0,119f8c70,f72175,...) at sleepq_switch+0xe1/frame 0x119f8c24 sleepq_wait(1fa50a0,0) at sleepq_wait+0x3a/frame 0x119f8c38 _cv_wait(1fa50a0,1fa504c) at _cv_wait+0x135/frame 0x119f8c70 audit_worker(0,119f8ce8) at audit_worker+0x7f/frame 0x119f8cb4 fork_exit(1252620,0,119f8ce8,0,0,...) at fork_exit+0x6c/frame 0x119f8cd4 fork_trampoline() at 0xffc033ca/frame 0x119f8cd4 --- trap 0, eip = 0, esp = 0x119f8d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100000 td 0x1f94c00 sched_switch(1f94c00,0,104,197,1ec8334,...) at sched_switch+0x598/frame 0x24f9ac8 mi_switch(104,0) at mi_switch+0x516/frame 0x24f9b8c sleepq_switch(1e2,1f94c00,9fffff47,0,2,...) at sleepq_switch+0xe1/frame 0x24f9bb0 sleepq_timedwait(1f94888,54,9,0,0,...) at sleepq_timedwait+0x47/frame 0x24f9be0 _sleep(1f94888,0,54,1675b82,fffff470,9,0,0,100) at _sleep+0x79a/frame 0x24f9cc8 swapper(0,0,0,1c4086c) at swapper+0x6d/frame 0x24f9d14 mi_startup() at mi_startup+0x2b5/frame 0x24f9d38 btext() at btext+0x5f Tracing command kernel pid 0 tid 100007 td 0x11fd6700 sched_switch(11fd6700,0,104,197,0,...) at sched_switch+0x598/frame 0x1b6ecaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b6ecb60 sleepq_switch(0,1b6ecba0,15a7839,1b6ecc7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b6ecb84 sleepq_wait(11f60f00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b6ecb98 _sleep(11f60f00,11f60f1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b6ecc7c taskqueue_thread_loop(1fa4430,1b6ecce8) at taskqueue_thread_loop+0xcb/frame 0x1b6eccb4 fork_exit(1024be0,1fa4430,1b6ecce8,0,0,...) at fork_exit+0x6c/frame 0x1b6eccd4 fork_trampoline() at 0xffc033ca/frame 0x1b6eccd4 --- trap 0, eip = 0, esp = 0x1b6ecd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100010 td 0x11fd0a80 sched_switch(11fd0a80,0,104,197,1b6fbaf0,...) at sched_switch+0x598/frame 0x1b6fbaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b6fbb60 sleepq_switch(0,1b6fbba0,15a7839,1b6fbc7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b6fbb84 sleepq_wait(11f60d80,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b6fbb98 _sleep(11f60d80,11f60d9c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b6fbc7c taskqueue_thread_loop(1fa43e0,1b6fbce8) at taskqueue_thread_loop+0xcb/frame 0x1b6fbcb4 fork_exit(1024be0,1fa43e0,1b6fbce8,0,0,...) at fork_exit+0x6c/frame 0x1b6fbcd4 fork_trampoline() at 0xffc033ca/frame 0x1b6fbcd4 --- trap 0, eip = 0, esp = 0x1b6fbd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100012 td 0x11fd0380 sched_switch(11fd0380,0,104,197,0,...) at sched_switch+0x598/frame 0x1b705ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b705b80 sleepq_switch(1b705bc0,100,15a7839,1b705c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b705ba4 sleepq_wait(11fc2c00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b705bb8 msleep_spin_sbt(11fc2c00,11fc2c1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b705c80 gtaskqueue_thread_loop(11a53404,1b705ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b705cb4 fork_exit(1010f40,11a53404,1b705ce8,0,0,...) at fork_exit+0x6c/frame 0x1b705cd4 fork_trampoline() at 0xffc033ca/frame 0x1b705cd4 --- trap 0, eip = 0, esp = 0x1b705d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100013 td 0x11fd0000 sched_switch(11fd0000,0,104,197,0,...) at sched_switch+0x598/frame 0x1b70aaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b70ab60 sleepq_switch(0,1b70aba0,15a7839,1b70ac7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b70ab84 sleepq_wait(11f60c80,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b70ab98 _sleep(11f60c80,11f60c9c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b70ac7c taskqueue_thread_loop(1f95258,1b70ace8) at taskqueue_thread_loop+0xcb/frame 0x1b70acb4 fork_exit(1024be0,1f95258,1b70ace8,0,0,...) at fork_exit+0x6c/frame 0x1b70acd4 fork_trampoline() at 0xffc033ca/frame 0x1b70acd4 --- trap 0, eip = 0, esp = 0x1b70ad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100014 td 0x11ad9380 sched_switch(11ad9380,0,104,197,0,...) at sched_switch+0x598/frame 0x1b71bac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b71bb80 sleepq_switch(1b71bbc0,100,15a7839,1b71bc80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b71bba4 sleepq_wait(11fc2b80,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b71bbb8 msleep_spin_sbt(11fc2b80,11fc2b9c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b71bc80 gtaskqueue_thread_loop(11a53804,1b71bce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b71bcb4 fork_exit(1010f40,11a53804,1b71bce8,0,0,...) at fork_exit+0x6c/frame 0x1b71bcd4 fork_trampoline() at 0xffc033ca/frame 0x1b71bcd4 --- trap 0, eip = 0, esp = 0x1b71bd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100015 td 0x11ad9000 sched_switch(11ad9000,0,104,197,1ef016c,...) at sched_switch+0x598/frame 0x1b720ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b720b80 sleepq_switch(1b720bc0,100,15a7839,1b720c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b720ba4 sleepq_wait(11fc2b00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b720bb8 msleep_spin_sbt(11fc2b00,11fc2b1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b720c80 gtaskqueue_thread_loop(11a53c04,1b720ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b720cb4 fork_exit(1010f40,11a53c04,1b720ce8,0,0,...) at fork_exit+0x6c/frame 0x1b720cd4 fork_trampoline() at 0xffc033ca/frame 0x1b720cd4 --- trap 0, eip = 0, esp = 0x1b720d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100016 td 0x11ad8a80 sched_switch(11ad8a80,0,104,197,1eef69c,...) at sched_switch+0x598/frame 0x1b725ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b725b80 sleepq_switch(1b725bc0,100,15a7839,1b725c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b725ba4 sleepq_wait(11fc2a80,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b725bb8 msleep_spin_sbt(11fc2a80,11fc2a9c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b725c80 gtaskqueue_thread_loop(11a53c14,1b725ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b725cb4 fork_exit(1010f40,11a53c14,1b725ce8,0,0,...) at fork_exit+0x6c/frame 0x1b725cd4 fork_trampoline() at 0xffc033ca/frame 0x1b725cd4 --- trap 0, eip = 0, esp = 0x1b725d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100017 td 0x11ad8700 sched_switch(11ad8700,0,104,197,1eef3ec,...) at sched_switch+0x598/frame 0x1b72aac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b72ab80 sleepq_switch(1b72abc0,100,15a7839,1b72ac80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b72aba4 sleepq_wait(11fc2a00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b72abb8 msleep_spin_sbt(11fc2a00,11fc2a1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b72ac80 gtaskqueue_thread_loop(11a53c24,1b72ace8) at gtaskqueue_thread_loop+0x9c/frame 0x1b72acb4 fork_exit(1010f40,11a53c24,1b72ace8,0,0,...) at fork_exit+0x6c/frame 0x1b72acd4 fork_trampoline() at 0xffc033ca/frame 0x1b72acd4 --- trap 0, eip = 0, esp = 0x1b72ad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100018 td 0x11ad8380 sched_switch(11ad8380,0,104,197,0,...) at sched_switch+0x598/frame 0x1b72fac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b72fb80 sleepq_switch(1b72fbc0,100,15a7839,1b72fc80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b72fba4 sleepq_wait(11fc2980,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b72fbb8 msleep_spin_sbt(11fc2980,11fc299c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b72fc80 gtaskqueue_thread_loop(11a53c34,1b72fce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b72fcb4 fork_exit(1010f40,11a53c34,1b72fce8,0,0,...) at fork_exit+0x6c/frame 0x1b72fcd4 fork_trampoline() at 0xffc033ca/frame 0x1b72fcd4 --- trap 0, eip = 0, esp = 0x1b72fd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100019 td 0x11ad8000 sched_switch(11ad8000,0,104,197,1eefaac,...) at sched_switch+0x598/frame 0x1b734ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b734b80 sleepq_switch(1b734bc0,100,15a7839,1b734c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b734ba4 sleepq_wait(11fc2900,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b734bb8 msleep_spin_sbt(11fc2900,11fc291c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b734c80 gtaskqueue_thread_loop(11a53004,1b734ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b734cb4 fork_exit(1010f40,11a53004,1b734ce8,0,0,...) at fork_exit+0x6c/frame 0x1b734cd4 fork_trampoline() at 0xffc033ca/frame 0x1b734cd4 --- trap 0, eip = 0, esp = 0x1b734d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100020 td 0x11ad7a80 sched_switch(11ad7a80,0,104,197,1b739afc,...) at sched_switch+0x598/frame 0x1b739ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b739b80 sleepq_switch(1b739bc0,100,15a7839,1b739c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b739ba4 sleepq_wait(11fc2880,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b739bb8 msleep_spin_sbt(11fc2880,11fc289c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b739c80 gtaskqueue_thread_loop(11a53014,1b739ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b739cb4 fork_exit(1010f40,11a53014,1b739ce8,0,0,...) at fork_exit+0x6c/frame 0x1b739cd4 fork_trampoline() at 0xffc033ca/frame 0x1b739cd4 --- trap 0, eip = 0, esp = 0x1b739d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100021 td 0x11ad7700 sched_switch(11ad7700,0,104,197,1,...) at sched_switch+0x598/frame 0x1b73eac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b73eb80 sleepq_switch(1b73ebc0,100,15a7839,1b73ec80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b73eba4 sleepq_wait(11fc2800,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b73ebb8 msleep_spin_sbt(11fc2800,11fc281c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b73ec80 gtaskqueue_thread_loop(11a53024,1b73ece8) at gtaskqueue_thread_loop+0x9c/frame 0x1b73ecb4 fork_exit(1010f40,11a53024,1b73ece8,0,0,...) at fork_exit+0x6c/frame 0x1b73ecd4 fork_trampoline() at 0xffc033ca/frame 0x1b73ecd4 --- trap 0, eip = 0, esp = 0x1b73ed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100022 td 0x11ad7380 sched_switch(11ad7380,0,104,197,1,...) at sched_switch+0x598/frame 0x1b743ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b743b80 sleepq_switch(1b743bc0,100,15a7839,1b743c80,fd959e,...) at sleepq_switch+0xe1/frame 0x1b743ba4 sleepq_wait(11fc2780,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x1b743bb8 msleep_spin_sbt(11fc2780,11fc279c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x1b743c80 gtaskqueue_thread_loop(11a53034,1b743ce8) at gtaskqueue_thread_loop+0x9c/frame 0x1b743cb4 fork_exit(1010f40,11a53034,1b743ce8,0,0,...) at fork_exit+0x6c/frame 0x1b743cd4 fork_trampoline() at 0xffc033ca/frame 0x1b743cd4 --- trap 0, eip = 0, esp = 0x1b743d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100029 td 0x11e6b000 sched_switch(11e6b000,0,104,197,0,...) at sched_switch+0x598/frame 0x1b77eaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b77eb60 sleepq_switch(0,1b77eba0,15a7839,1b77ec7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b77eb84 sleepq_wait(11f60c00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b77eb98 _sleep(11f60c00,11f60c1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b77ec7c taskqueue_thread_loop(1efc20c,1b77ece8) at taskqueue_thread_loop+0xcb/frame 0x1b77ecb4 fork_exit(1024be0,1efc20c,1b77ece8,0,0,...) at fork_exit+0x6c/frame 0x1b77ecd4 fork_trampoline() at 0xffc033ca/frame 0x1b77ecd4 --- trap 0, eip = 0, esp = 0x1b77ed20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100034 td 0x11e69a80 sched_switch(11e69a80,0,104,197,0,...) at sched_switch+0x598/frame 0x1b79baa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b79bb60 sleepq_switch(0,1b79bba0,15a7839,1b79bc7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b79bb84 sleepq_wait(11f60b00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b79bb98 _sleep(11f60b00,11f60b1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b79bc7c taskqueue_thread_loop(1f7009c,1b79bce8) at taskqueue_thread_loop+0xcb/frame 0x1b79bcb4 fork_exit(1024be0,1f7009c,1b79bce8,0,0,...) at fork_exit+0x6c/frame 0x1b79bcd4 fork_trampoline() at 0xffc033ca/frame 0x1b79bcd4 --- trap 0, eip = 0, esp = 0x1b79bd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100035 td 0x11e69700 sched_switch(11e69700,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7a0aa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7a0b60 sleepq_switch(0,1b7a0ba0,15a7839,1b7a0c7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7a0b84 sleepq_wait(11f60b00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b7a0b98 _sleep(11f60b00,11f60b1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7a0c7c taskqueue_thread_loop(1f7009c,1b7a0ce8) at taskqueue_thread_loop+0xcb/frame 0x1b7a0cb4 fork_exit(1024be0,1f7009c,1b7a0ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7a0cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7a0cd4 --- trap 0, eip = 0, esp = 0x1b7a0d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100036 td 0x11e69380 sched_switch(11e69380,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7a5aa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7a5b60 sleepq_switch(0,1b7a5ba0,15a7839,1b7a5c7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7a5b84 sleepq_wait(11f60b00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b7a5b98 _sleep(11f60b00,11f60b1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7a5c7c taskqueue_thread_loop(1f7009c,1b7a5ce8) at taskqueue_thread_loop+0xcb/frame 0x1b7a5cb4 fork_exit(1024be0,1f7009c,1b7a5ce8,0,0,...) at fork_exit+0x6c/frame 0x1b7a5cd4 fork_trampoline() at 0xffc033ca/frame 0x1b7a5cd4 --- trap 0, eip = 0, esp = 0x1b7a5d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100037 td 0x11e69000 sched_switch(11e69000,0,104,197,0,...) at sched_switch+0x598/frame 0x1b7aaaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x1b7aab60 sleepq_switch(0,1b7aaba0,15a7839,1b7aac7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x1b7aab84 sleepq_wait(11f60b00,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x1b7aab98 _sleep(11f60b00,11f60b1c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x1b7aac7c taskqueue_thread_loop(1f7009c,1b7aace8) at taskqueue_thread_loop+0xcb/frame 0x1b7aacb4 fork_exit(1024be0,1f7009c,1b7aace8,0,0,...) at fork_exit+0x6c/frame 0x1b7aacd4 fork_trampoline() at 0xffc033ca/frame 0x1b7aacd4 --- trap 0, eip = 0, esp = 0x1b7aad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100085 td 0x23d89380 sched_switch(23d89380,0,104,197,0,...) at sched_switch+0x598/frame 0x22ad3ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x22ad3b80 sleepq_switch(22ad3bc0,100,15a7839,22ad3c80,fd959e,...) at sleepq_switch+0xe1/frame 0x22ad3ba4 sleepq_wait(23d67000,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x22ad3bb8 msleep_spin_sbt(23d67000,23d6701c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x22ad3c80 taskqueue_thread_loop(1f88d08,22ad3ce8) at taskqueue_thread_loop+0x9c/frame 0x22ad3cb4 fork_exit(1024be0,1f88d08,22ad3ce8,0,0,...) at fork_exit+0x6c/frame 0x22ad3cd4 fork_trampoline() at 0xffc033ca/frame 0x22ad3cd4 --- trap 0, eip = 0, esp = 0x22ad3d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100091 td 0x23ff9a80 sched_switch(23ff9a80,0,104,197,2169d040,...) at sched_switch+0x598/frame 0x22afaac0 mi_switch(104,0) at mi_switch+0x516/frame 0x22afab80 sleepq_switch(22afabc0,100,15a7839,22afac80,fd959e,...) at sleepq_switch+0xe1/frame 0x22afaba4 sleepq_wait(23eb6d00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x22afabb8 msleep_spin_sbt(23eb6d00,23eb6d1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x22afac80 taskqueue_thread_loop(1f8a944,22aface8) at taskqueue_thread_loop+0x9c/frame 0x22afacb4 fork_exit(1024be0,1f8a944,22aface8,0,0,...) at fork_exit+0x6c/frame 0x22afacd4 fork_trampoline() at 0xffc033ca/frame 0x22afacd4 --- trap 0, eip = 0, esp = 0x22afad20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100092 td 0x23ff9700 sched_switch(23ff9700,0,104,197,22affafc,...) at sched_switch+0x598/frame 0x22affac0 mi_switch(104,0) at mi_switch+0x516/frame 0x22affb80 sleepq_switch(22affbc0,100,15a7839,22affc80,fd959e,...) at sleepq_switch+0xe1/frame 0x22affba4 sleepq_wait(23eb6d00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x22affbb8 msleep_spin_sbt(23eb6d00,23eb6d1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x22affc80 taskqueue_thread_loop(1f8a944,22affce8) at taskqueue_thread_loop+0x9c/frame 0x22affcb4 fork_exit(1024be0,1f8a944,22affce8,0,0,...) at fork_exit+0x6c/frame 0x22affcd4 fork_trampoline() at 0xffc033ca/frame 0x22affcd4 --- trap 0, eip = 0, esp = 0x22affd20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100093 td 0x23ff9380 sched_switch(23ff9380,0,104,197,22b04afc,...) at sched_switch+0x598/frame 0x22b04ac0 mi_switch(104,0) at mi_switch+0x516/frame 0x22b04b80 sleepq_switch(22b04bc0,100,15a7839,22b04c80,fd959e,...) at sleepq_switch+0xe1/frame 0x22b04ba4 sleepq_wait(23eb6d00,0,15917af,15a7839,0,...) at sleepq_wait+0x3a/frame 0x22b04bb8 msleep_spin_sbt(23eb6d00,23eb6d1c,15a7839,0,0,0,0,100) at msleep_spin_sbt+0x4be/frame 0x22b04c80 taskqueue_thread_loop(1f8a944,22b04ce8) at taskqueue_thread_loop+0x9c/frame 0x22b04cb4 fork_exit(1024be0,1f8a944,22b04ce8,0,0,...) at fork_exit+0x6c/frame 0x22b04cd4 fork_trampoline() at 0xffc033ca/frame 0x22b04cd4 --- trap 0, eip = 0, esp = 0x22b04d20, ebp = 0 --- (null)() at 0 Tracing command kernel pid 0 tid 100095 td 0x23ff8a80 sched_switch(23ff8a80,0,104,197,0,...) at sched_switch+0x598/frame 0x22b0eaa0 mi_switch(104,0) at mi_switch+0x516/frame 0x22b0eb60 sleepq_switch(0,22b0eba0,15a7839,22b0ec7c,fd8f1c,...) at sleepq_switch+0xe1/frame 0x22b0eb84 sleepq_wait(11f60a80,0,15a7839,0,0,...) at sleepq_wait+0x3a/frame 0x22b0eb98 _sleep(11f60a80,11f60a9c,0,15a7839,0,0,0,0,100) at _sleep+0x7ec/frame 0x22b0ec7c taskqueue_thread_loop(1c6c0f4,22b0ece8) at taskqueue_thread_loop+0xcb/frame 0x22b0ecb4 fork_exit(1024be0,1c6c0f4,22b0ece8,0,0,...) at fork_exit+0x6c/frame 0x22b0ecd4 fork_trampoline() at 0xffc033ca/frame 0x22b0ecd4 --- trap 0, eip = 0, esp = 0x22b0ed20, ebp = 0 --- (null)() at 0 db:0:pho> dump Physical memory: 8034 MB Dumping 346 MB: 331 315 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11 Dump complete db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 (kgdb) tid 100293 (kgdb) bt #0 0x00ff91b8 in sched_switch (td=0x31aa2700, newtd=0x31afc000, flags=0x104) at ../../../kern/sched_ule.c:2127 #1 0x00fda026 in mi_switch (flags=0x104, newtd=0x0) at ../../../kern/kern_synch.c:452 #2 0x01020251 in sleepq_switch (wchan=, pri=0x0) at ../../../kern/subr_sleepqueue.c:625 #3 0x0102012a in sleepq_wait (wchan=0x1ed73d8 , pri=0x0) at ../../../kern/subr_sleepqueue.c:704 #4 0x00fd8f1c in _sleep (ident=0x1ed73d8 , lock=0x1ed73d8 , priority=0x0, wmesg=0x16281c8 "umtxqb", sbt=0x0, pr=0x0, flags=0x100) at ../../../kern/kern_synch.c:217 #5 0x00ff098f in umtxq_busy (key=0x31a99788) at ../../../kern/kern_umtx.c:550 #6 0x00fee95a in do_lock_normal (td=, m=, flags=, timeout=, mode=) at ../../../kern/kern_umtx.c:1154 #7 do_lock_umutex (td=, m=, timeout=0x0, mode=0x2) at ../../../kern/kern_umtx.c:2540 #8 0x00feb80c in __umtx_op_wait_umutex (td=0x31aa2700, uap=0x31aa2988) at ../../../kern/kern_umtx.c:3624 #9 0x01531999 in syscallenter (td=) at ../../../i386/i386/../../kern/subr_syscall.c:135 #10 syscall (frame=0x317acce8) at ../../../i386/i386/trap.c:1160 #11 0xffc033b7 in ?? () #12 0x317acce8 in ?? () Backtrace stopped: Cannot access memory at address 0xfaff1ec4 (kgdb) p/x *(struct umtx_chain *)0x1ed73d8 No struct type named umtx_chain. (kgdb) p *(struct umtxq_chain *)0x1ed73d8 $1 = {uc_lock = {lock_object = {lo_name = 0x15ef12b "umtxql", lo_flags = 0x1430000, lo_data = 0x0, lo_witness = 0x2ded688}, mtx_lock = 0x0}, uc_queue = {{ lh_first = 0x0}, {lh_first = 0x0}}, uc_spare_queue = {lh_first = 0x0}, uc_busy = 0x1, uc_waiters = 0x4, uc_pi_list = {tqh_first = 0x0, tqh_last = 0x1ed7400 }, uc_owner_tid = 0x187ba, uc_owner_stack = {depth = 0x4, pcs = {0xff09b8, 0xfec750, 0x1531999, 0xffc033b7, 0x0 }}} (kgdb) l *0xff09b8 0xff09b8 is in umtxq_busy (../../../kern/kern_umtx.c:556). 551 uc->uc_waiters--; 552 } 553 } 554 uc->uc_busy = 1; 555 uc->uc_owner_tid = curthread->td_tid; 556 stack_save(&uc->uc_owner_stack); 557 } 558 559 /* 560 * Unbusy a chain. (kgdb) l *0xfec750 0xfec750 is in __umtx_op_wake2_umutex (../../../kern/kern_umtx.c:1383). 1378 if ((error = umtx_key_get(m, type, GET_SHARE(flags), &key)) != 0) 1379 return (error); 1380 1381 owner = 0; 1382 umtxq_lock(&key); 1383 umtxq_busy(&key); 1384 count = umtxq_count(&key); 1385 umtxq_unlock(&key); 1386 /* 1387 * Only repair contention bit if there is a waiter, this means the mutex (kgdb) l *0x1531999 0x1531999 is in syscall (../../../i386/i386/../../kern/subr_syscall.c:135). 130 sa->callp->sy_entry != 0)) 131 (*systrace_probe_func)(sa, SYSTRACE_ENTRY, 0); 132 #endif 133 134 AUDIT_SYSCALL_ENTER(sa->code, td); 135 error = (sa->callp->sy_call)(td, sa->args); 136 AUDIT_SYSCALL_EXIT(error, td); 137 138 /* Save the latest error return value. */ 139 if ((td->td_pflags & TDP_NERRNO) == 0) (kgdb) $ svn diff -x -p /usr/src/sys Index: /usr/src/sys/amd64/amd64/support.S =================================================================== --- /usr/src/sys/amd64/amd64/support.S (revision 349856) +++ /usr/src/sys/amd64/amd64/support.S (working copy) @@ -811,6 +811,7 @@ ENTRY(casueword32_nosmap) lock #endif cmpxchgl %ecx,(%rdi) /* new = %ecx */ + setne %cl /* * The old value is in %eax. If the store succeeded it will be the @@ -828,6 +829,7 @@ ENTRY(casueword32_nosmap) */ movl %esi,(%rdx) /* oldp = %rdx */ POP_FRAME_POINTER + movzbl %cl, %eax ret END(casueword32_nosmap) @@ -847,6 +849,7 @@ ENTRY(casueword32_smap) #endif cmpxchgl %ecx,(%rdi) /* new = %ecx */ clac + setne %cl /* * The old value is in %eax. If the store succeeded it will be the @@ -864,6 +867,7 @@ ENTRY(casueword32_smap) */ movl %esi,(%rdx) /* oldp = %rdx */ POP_FRAME_POINTER + movzbl %cl, %eax ret END(casueword32_smap) @@ -886,6 +890,7 @@ ENTRY(casueword_nosmap) lock #endif cmpxchgq %rcx,(%rdi) /* new = %rcx */ + setne %cl /* * The old value is in %rax. If the store succeeded it will be the @@ -897,6 +902,7 @@ ENTRY(casueword_nosmap) movq %rax,PCB_ONFAULT(%r8) movq %rsi,(%rdx) POP_FRAME_POINTER + movzbl %cl, %eax ret END(casueword_nosmap) @@ -916,6 +922,7 @@ ENTRY(casueword_smap) #endif cmpxchgq %rcx,(%rdi) /* new = %rcx */ clac + setne %cl /* * The old value is in %rax. If the store succeeded it will be the @@ -927,6 +934,7 @@ ENTRY(casueword_smap) movq %rax,PCB_ONFAULT(%r8) movq %rsi,(%rdx) POP_FRAME_POINTER + movzbl %cl, %eax ret END(casueword_smap) Index: /usr/src/sys/arm/arm/fusu.S =================================================================== --- /usr/src/sys/arm/arm/fusu.S (revision 349856) +++ /usr/src/sys/arm/arm/fusu.S (working copy) @@ -63,7 +63,7 @@ EENTRY_NP(casueword32) ldr r4, =(VM_MAXUSER_ADDRESS-3) cmp r0, r4 mvncs r0, #0 - bcs 2f + bcs 1f GET_PCB(r6) ldr r6, [r6] @@ -78,12 +78,10 @@ EENTRY_NP(casueword32) str r4, [r6, #PCB_ONFAULT] #if __ARM_ARCH >= 6 -1: + mov r5, #1 ldrex r4, [r0] cmp r4, r1 strexeq r5, r3, [r0] - cmpeq r5, #1 - beq 1b #else ldrt r4, [r0] cmp r4, r1 @@ -92,7 +90,10 @@ EENTRY_NP(casueword32) str r4, [r2] mov r0, #0 str r0, [r6, #PCB_ONFAULT] -2: +#if __ARM_ARCH >= 6 + mov r0, r5 +#endif +1: ldmfd sp!, {r4, r5, r6} RET EEND(casueword32) Index: /usr/src/sys/arm64/arm64/support.S =================================================================== --- /usr/src/sys/arm64/arm64/support.S (revision 349856) +++ /usr/src/sys/arm64/arm64/support.S (working copy) @@ -57,6 +57,7 @@ ENTRY(casueword32) cmp x0, x4 b.cs fsu_fault_nopcb adr x6, fsu_fault /* Load the fault handler */ + mov w5, #1 SET_FAULT_HANDLER(x6, x4) /* And set it */ ENTER_USER_ACCESS(w6, x4) 1: ldxr w4, [x0] /* Load-exclusive the data */ @@ -63,11 +64,10 @@ ENTRY(casueword32) cmp w4, w1 /* Compare */ b.ne 2f /* Not equal, exit */ stxr w5, w3, [x0] /* Store the new data */ - cbnz w5, 1b /* Retry on failure */ 2: EXIT_USER_ACCESS(w6) - SET_FAULT_HANDLER(xzr, x5) /* Reset the fault handler */ + SET_FAULT_HANDLER(xzr, x6) /* Reset the fault handler */ str w4, [x2] /* Store the read data */ - mov x0, #0 /* Success */ + mov w0, w5 /* Result same as store status */ ret /* Return */ END(casueword32) @@ -79,6 +79,7 @@ ENTRY(casueword) cmp x0, x4 b.cs fsu_fault_nopcb adr x6, fsu_fault /* Load the fault handler */ + mov w5, #1 SET_FAULT_HANDLER(x6, x4) /* And set it */ ENTER_USER_ACCESS(w6, x4) 1: ldxr x4, [x0] /* Load-exclusive the data */ @@ -85,11 +86,10 @@ ENTRY(casueword) cmp x4, x1 /* Compare */ b.ne 2f /* Not equal, exit */ stxr w5, x3, [x0] /* Store the new data */ - cbnz w5, 1b /* Retry on failure */ 2: EXIT_USER_ACCESS(w6) - SET_FAULT_HANDLER(xzr, x5) /* Reset the fault handler */ + SET_FAULT_HANDLER(xzr, x6) /* Reset the fault handler */ str x4, [x2] /* Store the read data */ - mov x0, #0 /* Success */ + mov w0, w5 /* Result same as store status */ ret /* Return */ END(casueword) Index: /usr/src/sys/i386/i386/copyout.c =================================================================== --- /usr/src/sys/i386/i386/copyout.c (revision 349856) +++ /usr/src/sys/i386/i386/copyout.c (working copy) @@ -428,6 +428,7 @@ suword32(volatile void *base, int32_t word) struct casueword_arg0 { uint32_t oldval; uint32_t newval; + int res; }; static void @@ -436,7 +437,8 @@ casueword_slow0(vm_offset_t kva, void *arg) struct casueword_arg0 *ca; ca = arg; - atomic_fcmpset_int((u_int *)kva, &ca->oldval, ca->newval); + ca->res = 1 - atomic_fcmpset_int((u_int *)kva, &ca->oldval, + ca->newval); } int @@ -452,7 +454,7 @@ casueword32(volatile uint32_t *base, uint32_t oldv casueword_slow0, &ca); if (res == 0) { *oldvalp = ca.oldval; - return (0); + return (ca.res); } return (-1); } @@ -469,7 +471,7 @@ casueword(volatile u_long *base, u_long oldval, u_ casueword_slow0, &ca); if (res == 0) { *oldvalp = ca.oldval; - return (0); + return (ca.res); } return (-1); } Index: /usr/src/sys/kern/kern_umtx.c =================================================================== --- /usr/src/sys/kern/kern_umtx.c (revision 349856) +++ /usr/src/sys/kern/kern_umtx.c (working copy) @@ -63,6 +63,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include @@ -183,6 +184,8 @@ struct umtxq_chain { u_int length; u_int max_length; #endif + int uc_owner_tid; + struct stack uc_owner_stack; }; #define UMTXQ_LOCKED_ASSERT(uc) mtx_assert(&(uc)->uc_lock, MA_OWNED) @@ -549,6 +552,8 @@ umtxq_busy(struct umtx_key *key) } } uc->uc_busy = 1; + uc->uc_owner_tid = curthread->td_tid; + stack_save(&uc->uc_owner_stack); } /* @@ -690,8 +695,26 @@ umtxq_count_pi(struct umtx_key *key, struct umtx_q return (0); } +/* + * Check for possible stops and suspensions while executing a umtx + * locking operation. + * + * The sleep argument controls whether the function can handle a stop + * request itself or it should return ERESTART and the request is + * proceed at the kernel/user boundary in ast. + * + * Typically, when retrying due to casueword(9) failure (rv == 1), we + * should handle the stop requests there, with exception of cases when + * the thread busied the umtx key, or when functions return + * immediately if umtxq_check_susp() returned non-zero. On the other + * hand, retrying the whole lock operation, we better not stop there + * but delegate the handling to ast. + * + * If the request is for thread termination P_SINGLE_EXIT, we cannot + * handle it at all, and simply return EINTR. + */ static int -umtxq_check_susp(struct thread *td) +umtxq_check_susp(struct thread *td, bool sleep) { struct proc *p; int error; @@ -710,7 +733,7 @@ static int if (p->p_flag & P_SINGLE_EXIT) error = EINTR; else - error = ERESTART; + error = sleep ? thread_suspend_check(0) : ERESTART; } PROC_UNLOCK(p); return (error); @@ -1049,9 +1072,12 @@ do_lock_normal(struct thread *td, struct umutex *m id | UMUTEX_CONTESTED); if (rv == -1) return (EFAULT); - if (owner == UMUTEX_RB_OWNERDEAD) + if (rv == 0) { + MPASS(owner == UMUTEX_RB_OWNERDEAD); return (EOWNERDEAD); /* success */ - rv = umtxq_check_susp(td); + } + MPASS(rv == 1); + rv = umtxq_check_susp(td, false); if (rv != 0) return (rv); continue; @@ -1070,13 +1096,16 @@ do_lock_normal(struct thread *td, struct umutex *m return (EFAULT); /* The acquire succeeded. */ - if (owner == UMUTEX_UNOWNED) + if (rv == 0) { + MPASS(owner == UMUTEX_UNOWNED); return (0); + } /* * If no one owns it but it is contested try * to acquire it. */ + MPASS(rv == 1); if (owner == UMUTEX_CONTESTED) { rv = casueword32(&m->m_owner, UMUTEX_CONTESTED, &owner, @@ -1084,14 +1113,16 @@ do_lock_normal(struct thread *td, struct umutex *m /* The address was invalid. */ if (rv == -1) return (EFAULT); - - if (owner == UMUTEX_CONTESTED) + if (rv == 0) { + MPASS(owner == UMUTEX_CONTESTED); return (0); + } + if (rv == 1) { + rv = umtxq_check_susp(td, false); + if (rv != 0) + return (rv); + } - rv = umtxq_check_susp(td); - if (rv != 0) - return (rv); - /* * If this failed the lock has * changed, restart. @@ -1098,6 +1129,11 @@ do_lock_normal(struct thread *td, struct umutex *m */ continue; } + + /* rv == 1 but not contested, likely store failure */ + rv = umtxq_check_susp(td, false); + if (rv != 0) + return (rv); } if (mode == _UMUTEX_TRY) @@ -1128,14 +1164,21 @@ do_lock_normal(struct thread *td, struct umutex *m rv = casueword32(&m->m_owner, owner, &old, owner | UMUTEX_CONTESTED); - /* The address was invalid. */ - if (rv == -1) { + /* The address was invalid or casueword failed to store. */ + if (rv == -1 || rv == 1) { umtxq_lock(&uq->uq_key); umtxq_remove(uq); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); umtx_key_release(&uq->uq_key); - return (EFAULT); + if (rv == -1) + return (EFAULT); + if (rv == 1) { + rv = umtxq_check_susp(td, false); + if (rv != 0) + return (rv); + } + continue; } /* @@ -1145,15 +1188,15 @@ do_lock_normal(struct thread *td, struct umutex *m */ umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); - if (old == owner) - error = umtxq_sleep(uq, "umtxn", timeout == NULL ? - NULL : &timo); + MPASS(old == owner); + error = umtxq_sleep(uq, "umtxn", timeout == NULL ? + NULL : &timo); umtxq_remove(uq); umtxq_unlock(&uq->uq_key); umtx_key_release(&uq->uq_key); if (error == 0) - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, false); } return (0); @@ -1170,6 +1213,8 @@ do_unlock_normal(struct thread *td, struct umutex int error, count; id = td->td_tid; + +again: /* * Make sure we own this mtx. */ @@ -1185,9 +1230,14 @@ do_unlock_normal(struct thread *td, struct umutex error = casueword32(&m->m_owner, owner, &old, newlock); if (error == -1) return (EFAULT); - if (old == owner) - return (0); - owner = old; + if (error == 1) { + error = umtxq_check_susp(td, false); + if (error != 0) + return (error); + goto again; + } + MPASS(old == owner); + return (0); } /* We should only ever be in here for contested locks */ @@ -1215,8 +1265,14 @@ do_unlock_normal(struct thread *td, struct umutex umtx_key_release(&key); if (error == -1) return (EFAULT); - if (old != owner) - return (EINVAL); + if (error == 1) { + if (old != owner) + return (EINVAL); + error = umtxq_check_susp(td, false); + if (error != 0) + return (error); + goto again; + } return (0); } @@ -1233,6 +1289,7 @@ do_wake_umutex(struct thread *td, struct umutex *m int error; int count; +again: error = fueword32(&m->m_owner, &owner); if (error == -1) return (EFAULT); @@ -1259,14 +1316,26 @@ do_wake_umutex(struct thread *td, struct umutex *m owner != UMUTEX_RB_NOTRECOV) { error = casueword32(&m->m_owner, UMUTEX_CONTESTED, &owner, UMUTEX_UNOWNED); - if (error == -1) + if (error == -1) { error = EFAULT; + } else if (error == 1) { + umtxq_unbusy(&key); + umtxq_unlock(&key); + umtx_key_release(&key); + error = umtxq_check_susp(td, false); + if (error != 0) + return (error); + goto again; + } } umtxq_lock(&key); - if (error == 0 && count != 0 && ((owner & ~UMUTEX_CONTESTED) == 0 || - owner == UMUTEX_RB_OWNERDEAD || owner == UMUTEX_RB_NOTRECOV)) + if (error == 0 && count != 0) { + MPASS((owner & ~UMUTEX_CONTESTED) == 0 || + owner == UMUTEX_RB_OWNERDEAD || + owner == UMUTEX_RB_NOTRECOV); umtxq_signal(&key, 1); + } umtxq_unbusy(&key); umtxq_unlock(&key); umtx_key_release(&key); @@ -1330,12 +1399,12 @@ do_wake2_umutex(struct thread *td, struct umutex * error = EFAULT; break; } - if (old == owner) + if (error == 0) { + MPASS(old == owner); break; + } owner = old; - error = umtxq_check_susp(td); - if (error != 0) - break; + error = umtxq_check_susp(td, true); } } else if (count == 1) { error = fueword32(&m->m_owner, &owner); @@ -1349,12 +1418,12 @@ do_wake2_umutex(struct thread *td, struct umutex * error = EFAULT; break; } - if (old == owner) + if (error == 0) { + MPASS(old == owner); break; + } owner = old; - error = umtxq_check_susp(td); - if (error != 0) - break; + error = umtxq_check_susp(td, true); } } umtxq_lock(&key); @@ -1842,13 +1911,17 @@ do_lock_pi(struct thread *td, struct umutex *m, ui error = EFAULT; break; } - /* The acquire succeeded. */ - if (owner == UMUTEX_UNOWNED) { + if (rv == 0) { + MPASS(owner == UMUTEX_UNOWNED); error = 0; break; } + error = umtxq_check_susp(td, true); + if (error != 0) + break; + if (owner == UMUTEX_RB_NOTRECOV) { error = ENOTRECOVERABLE; break; @@ -1864,36 +1937,37 @@ do_lock_pi(struct thread *td, struct umutex *m, ui error = EFAULT; break; } + if (rv == 1) { + error = umtxq_check_susp(td, true); + if (error != 0) + return (error); + /* + * If this failed the lock could + * changed, restart. + */ + continue; + } - if (owner == old_owner) { - umtxq_lock(&uq->uq_key); - umtxq_busy(&uq->uq_key); - error = umtx_pi_claim(pi, td); - umtxq_unbusy(&uq->uq_key); - umtxq_unlock(&uq->uq_key); - if (error != 0) { - /* - * Since we're going to return an - * error, restore the m_owner to its - * previous, unowned state to avoid - * compounding the problem. - */ - (void)casuword32(&m->m_owner, - id | UMUTEX_CONTESTED, - old_owner); - } - if (error == 0 && - old_owner == UMUTEX_RB_OWNERDEAD) - error = EOWNERDEAD; - break; + MPASS(rv == 0); + MPASS(owner == old_owner); + umtxq_lock(&uq->uq_key); + umtxq_busy(&uq->uq_key); + error = umtx_pi_claim(pi, td); + umtxq_unbusy(&uq->uq_key); + umtxq_unlock(&uq->uq_key); + if (error != 0) { + /* + * Since we're going to return an + * error, restore the m_owner to its + * previous, unowned state to avoid + * compounding the problem. + */ + (void)casuword32(&m->m_owner, + id | UMUTEX_CONTESTED, old_owner); } - - error = umtxq_check_susp(td); - if (error != 0) - break; - - /* If this failed the lock has changed, restart. */ - continue; + if (error == 0 && old_owner == UMUTEX_RB_OWNERDEAD) + error = EOWNERDEAD; + break; } if ((owner & ~UMUTEX_CONTESTED) == id) { @@ -1932,28 +2006,33 @@ do_lock_pi(struct thread *td, struct umutex *m, ui error = EFAULT; break; } + if (rv == 1) { + umtxq_unbusy_unlocked(&uq->uq_key); + error = umtxq_check_susp(td, true); + if (error != 0) + break; - umtxq_lock(&uq->uq_key); - /* - * We set the contested bit, sleep. Otherwise the lock changed - * and we need to retry or we lost a race to the thread - * unlocking the umtx. Note that the UMUTEX_RB_OWNERDEAD - * value for owner is impossible there. - */ - if (old == owner) { - error = umtxq_sleep_pi(uq, pi, - owner & ~UMUTEX_CONTESTED, - "umtxpi", timeout == NULL ? NULL : &timo, - (flags & USYNC_PROCESS_SHARED) != 0); - if (error != 0) - continue; - } else { - umtxq_unbusy(&uq->uq_key); - umtxq_unlock(&uq->uq_key); + /* + * The lock changed and we need to retry or we + * lost a race to the thread unlocking the + * umtx. Note that the UMUTEX_RB_OWNERDEAD + * value for owner is impossible there. + */ + continue; } - error = umtxq_check_susp(td); + umtxq_lock(&uq->uq_key); + + /* We set the contested bit, sleep. */ + MPASS(old == owner); + error = umtxq_sleep_pi(uq, pi, owner & ~UMUTEX_CONTESTED, + "umtxpi", timeout == NULL ? NULL : &timo, + (flags & USYNC_PROCESS_SHARED) != 0); if (error != 0) + continue; + + error = umtxq_check_susp(td, true); + if (error != 0) break; } @@ -1978,6 +2057,8 @@ do_unlock_pi(struct thread *td, struct umutex *m, int count, error, pri; id = td->td_tid; + +usrloop: /* * Make sure we own this mtx. */ @@ -1995,6 +2076,12 @@ do_unlock_pi(struct thread *td, struct umutex *m, error = casueword32(&m->m_owner, owner, &old, new_owner); if (error == -1) return (EFAULT); + if (error == 1) { + error = umtxq_check_susp(td, true); + if (error != 0) + return (error); + goto usrloop; + } if (old == owner) return (0); owner = old; @@ -2074,15 +2161,20 @@ do_unlock_pi(struct thread *td, struct umutex *m, if (count > 1) new_owner |= UMUTEX_CONTESTED; +again: error = casueword32(&m->m_owner, owner, &old, new_owner); - + if (error == 1) { + error = umtxq_check_susp(td, false); + if (error == 0) + goto again; + } umtxq_unbusy_unlocked(&key); umtx_key_release(&key); if (error == -1) return (EFAULT); - if (old != owner) + if (error == 0 && old != owner) return (EINVAL); - return (0); + return (error); } /* @@ -2149,11 +2241,13 @@ do_lock_pp(struct thread *td, struct umutex *m, ui error = EFAULT; break; } - - if (owner == UMUTEX_CONTESTED) { + if (rv == 0) { + MPASS(owner == UMUTEX_CONTESTED); error = 0; break; - } else if (owner == UMUTEX_RB_OWNERDEAD) { + } + /* rv == 1 */ + if (owner == UMUTEX_RB_OWNERDEAD) { rv = casueword32(&m->m_owner, UMUTEX_RB_OWNERDEAD, &owner, id | UMUTEX_CONTESTED); if (rv == -1) { @@ -2160,20 +2254,36 @@ do_lock_pp(struct thread *td, struct umutex *m, ui error = EFAULT; break; } - if (owner == UMUTEX_RB_OWNERDEAD) { + if (rv == 0) { + MPASS(owner == UMUTEX_RB_OWNERDEAD); error = EOWNERDEAD; /* success */ break; } - error = 0; + + /* + * rv == 1, only check for suspension if we + * did not already catched a signal. If we + * get an error from the check, the same + * condition is checked by the umtxq_sleep() + * call below, so we should obliterate the + * error to not skip the last loop iteration. + */ + if (error == 0) { + error = umtxq_check_susp(td, false); + if (error == 0) { + if (try != 0) + error = EBUSY; + else + continue; + } + error = 0; + } } else if (owner == UMUTEX_RB_NOTRECOV) { error = ENOTRECOVERABLE; - break; } - if (try != 0) { + if (try != 0) error = EBUSY; - break; - } /* * If we caught a signal, we have retried and now @@ -2668,11 +2778,12 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw umtx_key_release(&uq->uq_key); return (EFAULT); } - if (oldstate == state) { + if (rv == 0) { + MPASS(oldstate == state); umtx_key_release(&uq->uq_key); return (0); } - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; state = oldstate; @@ -2703,10 +2814,12 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw error = EFAULT; break; } - if (oldstate == state) + if (rv == 0) { + MPASS(oldstate == state); goto sleep; + } state = oldstate; - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; } @@ -2718,7 +2831,7 @@ do_rw_rdlock(struct thread *td, struct urwlock *rw /* state is changed while setting flags, restart */ if (!(state & wrflags)) { umtxq_unbusy_unlocked(&uq->uq_key); - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; continue; @@ -2781,10 +2894,12 @@ sleep: error = EFAULT; break; } - if (oldstate == state) + if (rv == 0) { + MPASS(oldstate == state); break; + } state = oldstate; - error1 = umtxq_check_susp(td); + error1 = umtxq_check_susp(td, true); if (error1 != 0) { if (error == 0) error = error1; @@ -2840,22 +2955,25 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw umtx_key_release(&uq->uq_key); return (EFAULT); } - if (oldstate == state) { + if (rv == 0) { + MPASS(oldstate == state); umtx_key_release(&uq->uq_key); return (0); } state = oldstate; - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; } if (error) { - if (!(state & (URWLOCK_WRITE_OWNER|URWLOCK_WRITE_WAITERS)) && + if ((state & (URWLOCK_WRITE_OWNER | + URWLOCK_WRITE_WAITERS)) == 0 && blocked_readers != 0) { umtxq_lock(&uq->uq_key); umtxq_busy(&uq->uq_key); - umtxq_signal_queue(&uq->uq_key, INT_MAX, UMTX_SHARED_QUEUE); + umtxq_signal_queue(&uq->uq_key, INT_MAX, + UMTX_SHARED_QUEUE); umtxq_unbusy(&uq->uq_key); umtxq_unlock(&uq->uq_key); } @@ -2885,10 +3003,12 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw error = EFAULT; break; } - if (oldstate == state) + if (rv == 0) { + MPASS(oldstate == state); goto sleep; + } state = oldstate; - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; } @@ -2900,7 +3020,7 @@ do_rw_wrlock(struct thread *td, struct urwlock *rw if ((state & URWLOCK_WRITE_OWNER) == 0 && URWLOCK_READER_COUNT(state) == 0) { umtxq_unbusy_unlocked(&uq->uq_key); - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) break; continue; @@ -2958,10 +3078,12 @@ sleep: error = EFAULT; break; } - if (oldstate == state) + if (rv == 0) { + MPASS(oldstate == state); break; + } state = oldstate; - error1 = umtxq_check_susp(td); + error1 = umtxq_check_susp(td, true); /* * We are leaving the URWLOCK_WRITE_WAITERS * behind, but this should not harm the @@ -3021,13 +3143,13 @@ do_rw_unlock(struct thread *td, struct urwlock *rw error = EFAULT; goto out; } - if (oldstate != state) { + if (rv == 1) { state = oldstate; if (!(oldstate & URWLOCK_WRITE_OWNER)) { error = EPERM; goto out; } - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) goto out; } else @@ -3041,13 +3163,13 @@ do_rw_unlock(struct thread *td, struct urwlock *rw error = EFAULT; goto out; } - if (oldstate != state) { + if (rv == 1) { state = oldstate; if (URWLOCK_READER_COUNT(oldstate) == 0) { error = EPERM; goto out; } - error = umtxq_check_susp(td); + error = umtxq_check_susp(td, true); if (error != 0) goto out; } else @@ -3097,7 +3219,7 @@ do_sem_wait(struct thread *td, struct _usem *sem, struct abs_timeout timo; struct umtx_q *uq; uint32_t flags, count, count1; - int error, rv; + int error, rv, rv1; uq = td->td_umtxq; error = fueword32(&sem->_flags, &flags); @@ -3114,15 +3236,24 @@ do_sem_wait(struct thread *td, struct _usem *sem, umtxq_busy(&uq->uq_key); umtxq_insert(uq); umtxq_unlock(&uq->uq_key); +again: rv = casueword32(&sem->_has_waiters, 0, &count1, 1); if (rv == 0) - rv = fueword32(&sem->_count, &count); - if (rv == -1 || count != 0) { + rv1 = fueword32(&sem->_count, &count); + if (rv == -1 || (rv == 0 && (rv1 == -1 || count != 0)) || rv == 1) { umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); umtxq_remove(uq); umtxq_unlock(&uq->uq_key); umtx_key_release(&uq->uq_key); + if (rv == 1) { + rv = umtxq_check_susp(td, true); + if (rv == 0) + goto again; + return (rv); + } + if (rv == 0) + rv = rv1; return (rv == -1 ? EFAULT : 0); } umtxq_lock(&uq->uq_key); @@ -3201,6 +3332,7 @@ do_sem2_wait(struct thread *td, struct _usem2 *sem if (timeout != NULL) abs_timeout_init2(&timo, timeout); +again: umtxq_lock(&uq->uq_key); umtxq_busy(&uq->uq_key); umtxq_insert(uq); @@ -3226,16 +3358,19 @@ do_sem2_wait(struct thread *td, struct _usem2 *sem if (count == USEM_HAS_WAITERS) break; rv = casueword32(&sem->_count, 0, &count, USEM_HAS_WAITERS); - if (rv == -1) { - umtxq_lock(&uq->uq_key); - umtxq_unbusy(&uq->uq_key); - umtxq_remove(uq); - umtxq_unlock(&uq->uq_key); - umtx_key_release(&uq->uq_key); + if (rv == 0) + break; + umtxq_lock(&uq->uq_key); + umtxq_unbusy(&uq->uq_key); + umtxq_remove(uq); + umtxq_unlock(&uq->uq_key); + umtx_key_release(&uq->uq_key); + if (rv == -1) return (EFAULT); - } - if (count == 0) - break; + rv = umtxq_check_susp(td, true); + if (rv != 0) + return (rv); + goto again; } umtxq_lock(&uq->uq_key); umtxq_unbusy(&uq->uq_key); @@ -3288,11 +3423,20 @@ do_sem2_wake(struct thread *td, struct _usem2 *sem if (cnt == 1) { umtxq_unlock(&key); rv = fueword32(&sem->_count, &count); - while (rv != -1 && count & USEM_HAS_WAITERS) + while (rv != -1 && count & USEM_HAS_WAITERS) { rv = casueword32(&sem->_count, count, &count, count & ~USEM_HAS_WAITERS); + if (rv == 1) { + rv = umtxq_check_susp(td, true); + if (rv != 0) + break; + } + } if (rv == -1) error = EFAULT; + else if (rv > 0) { + error = rv; + } umtxq_lock(&key); } Index: /usr/src/sys/mips/mips/support.S =================================================================== --- /usr/src/sys/mips/mips/support.S (revision 349856) +++ /usr/src/sys/mips/mips/support.S (working copy) @@ -336,19 +336,15 @@ XLEAF(casueword) GET_CPU_PCPU(v1) PTR_L v1, PC_CURPCB(v1) PTR_S v0, U_PCB_ONFAULT(v1) -1: + + li v0, 1 move t0, a3 ll t1, 0(a0) - bne a1, t1, 2f + bne a1, t1, 1f nop sc t0, 0(a0) # store word - beqz t0, 1b - nop - j 3f - li v0, 0 -2: - li v0, -1 -3: + xori v0, t0, 1 +1: PTR_S zero, U_PCB_ONFAULT(v1) jr ra sw t1, 0(a2) # unconditionally store old word @@ -363,19 +359,15 @@ XLEAF(casueword) GET_CPU_PCPU(v1) PTR_L v1, PC_CURPCB(v1) PTR_S v0, U_PCB_ONFAULT(v1) -1: + + li v0, 1 move t0, a3 lld t1, 0(a0) - bne a1, t1, 2f + bne a1, t1, 1f nop scd t0, 0(a0) # store double word - beqz t0, 1b - nop - j 3f - li v0, 0 -2: - li v0, -1 -3: + xori v0, t0, 1 +1: PTR_S zero, U_PCB_ONFAULT(v1) jr ra sd t1, 0(a2) # unconditionally store old word Index: /usr/src/sys/powerpc/powerpc/copyinout.c =================================================================== --- /usr/src/sys/powerpc/powerpc/copyinout.c (revision 349856) +++ /usr/src/sys/powerpc/powerpc/copyinout.c (working copy) @@ -433,6 +433,7 @@ casueword32(volatile uint32_t *addr, uint32_t old, pmap_t pm; jmp_buf env; uint32_t *p, val; + int res; td = curthread; pm = &td->td_proc->p_vmspace->vm_pmap; @@ -449,17 +450,19 @@ casueword32(volatile uint32_t *addr, uint32_t old, return (-1); } + res = 0; __asm __volatile ( - "1:\tlwarx %0, 0, %2\n\t" /* load old value */ - "cmplw %3, %0\n\t" /* compare */ - "bne 2f\n\t" /* exit if not equal */ - "stwcx. %4, 0, %2\n\t" /* attempt to store */ - "bne- 1b\n\t" /* spin if failed */ - "b 3f\n\t" /* we've succeeded */ + "lwarx %0, 0, %3\n\t" /* load old value */ + "cmplw %4, %0\n\t" /* compare */ + "bne 1f\n\t" /* exit if not equal */ + "stwcx. %5, 0, %3\n\t" /* attempt to store */ + "bne- 1f\n\t" /* if failed */ + "b 2f\n\t" /* we've succeeded */ + "1:\n\t" + "stwcx. %0, 0, %4\n\t" /* clear reservation (74xx) */ + "li %2, 1\n\t" "2:\n\t" - "stwcx. %0, 0, %2\n\t" /* clear reservation (74xx) */ - "3:\n\t" - : "=&r" (val), "=m" (*p) + : "=&r" (val), "=m" (*p), "=&r" (res) : "r" (p), "r" (old), "r" (new), "m" (*p) : "cr0", "memory"); @@ -466,7 +469,7 @@ casueword32(volatile uint32_t *addr, uint32_t old, td->td_pcb->pcb_onfault = NULL; *oldvalp = val; - return (0); + return (res); } #ifndef __powerpc64__ @@ -485,6 +488,7 @@ casueword(volatile u_long *addr, u_long old, u_lon pmap_t pm; jmp_buf env; u_long *p, val; + int res; td = curthread; pm = &td->td_proc->p_vmspace->vm_pmap; @@ -501,17 +505,19 @@ casueword(volatile u_long *addr, u_long old, u_lon return (-1); } + res = 0; __asm __volatile ( - "1:\tldarx %0, 0, %2\n\t" /* load old value */ - "cmpld %3, %0\n\t" /* compare */ - "bne 2f\n\t" /* exit if not equal */ - "stdcx. %4, 0, %2\n\t" /* attempt to store */ - "bne- 1b\n\t" /* spin if failed */ - "b 3f\n\t" /* we've succeeded */ + "ldarx %0, 0, %3\n\t" /* load old value */ + "cmpld %4, %0\n\t" /* compare */ + "bne 1f\n\t" /* exit if not equal */ + "stdcx. %5, 0, %3\n\t" /* attempt to store */ + "bne- 1f\n\t" /* if failed */ + "b 2f\n\t" /* we've succeeded */ + "1:\n\t" + "stdcx. %0, 0, %3\n\t" /* clear reservation (74xx) */ + "li %2, 1\n\t" "2:\n\t" - "stdcx. %0, 0, %2\n\t" /* clear reservation (74xx) */ - "3:\n\t" - : "=&r" (val), "=m" (*p) + : "=&r" (val), "=m" (*p), "=&r" (res) : "r" (p), "r" (old), "r" (new), "m" (*p) : "cr0", "memory"); @@ -518,6 +524,6 @@ casueword(volatile u_long *addr, u_long old, u_lon td->td_pcb->pcb_onfault = NULL; *oldvalp = val; - return (0); + return (res); } #endif Index: /usr/src/sys/riscv/riscv/support.S =================================================================== --- /usr/src/sys/riscv/riscv/support.S (revision 349856) +++ /usr/src/sys/riscv/riscv/support.S (working copy) @@ -60,14 +60,15 @@ ENTRY(casueword32) la a6, fsu_fault /* Load the fault handler */ SET_FAULT_HANDLER(a6, a4) /* And set it */ ENTER_USER_ACCESS(a4) -1: lr.w a4, 0(a0) /* Load-exclusive the data */ - bne a4, a1, 2f /* If not equal then exit */ + lr.w a4, 0(a0) /* Load-exclusive the data */ + bne a4, a1, 1f /* If not equal then exit */ sc.w a5, a3, 0(a0) /* Store the new data */ - bnez a5, 1b /* Retry on failure */ -2: EXIT_USER_ACCESS(a5) - SET_FAULT_HANDLER(x0, a5) /* Reset the fault handler */ + beqz a5, 2f /* Success */ +1: li a5, 1 /* Normalize failure result */ +2: EXIT_USER_ACCESS(a6) + SET_FAULT_HANDLER(x0, a6) /* Reset the fault handler */ sw a4, 0(a2) /* Store the read data */ - li a0, 0 /* Success */ + mv a0, a5 /* Success indicator */ ret /* Return */ END(casueword32) @@ -80,14 +81,15 @@ ENTRY(casueword) la a6, fsu_fault /* Load the fault handler */ SET_FAULT_HANDLER(a6, a4) /* And set it */ ENTER_USER_ACCESS(a4) -1: lr.d a4, 0(a0) /* Load-exclusive the data */ - bne a4, a1, 2f /* If not equal then exit */ + lr.d a4, 0(a0) /* Load-exclusive the data */ + bne a4, a1, 1f /* If not equal then exit */ sc.d a5, a3, 0(a0) /* Store the new data */ - bnez a5, 1b /* Retry on failure */ -2: EXIT_USER_ACCESS(a5) - SET_FAULT_HANDLER(x0, a5) /* Reset the fault handler */ + beqz a5, 2f /* Success */ +1: li a5, 1 /* Normalize failure result */ +2: EXIT_USER_ACCESS(a6) + SET_FAULT_HANDLER(x0, a6) /* Reset the fault handler */ sd a4, 0(a2) /* Store the read data */ - li a0, 0 /* Success */ + mv a0, a5 /* Success indicator */ ret /* Return */ END(casueword) Index: /usr/src/sys/sparc64/sparc64/support.S =================================================================== --- /usr/src/sys/sparc64/sparc64/support.S (revision 349856) +++ /usr/src/sys/sparc64/sparc64/support.S (working copy) @@ -403,8 +403,7 @@ fs_nofault_begin: .set susword, suword16 .set suword, suword64 - .globl casuword32, casuword, fuptr, suptr - .set casuword, casuword64 + .globl casuword32_int, casuword64_int, fuptr, suptr .set fuptr, fuword64 .set suptr, suword64 @@ -411,20 +410,20 @@ fs_nofault_begin: /* * int32_t casuword32(volatile int32_t *p, int32_t e, int32_t s) */ -ENTRY(casuword32) +ENTRY(casuword32_int) casa [%o0] ASI_AIUP, %o1, %o2 retl mov %o2, %o0 -END(casuword32) +END(casuword32_int) /* * int64_t casuword64(volatile int64_t *p, int64_t e, int64_t s) */ -ENTRY(casuword64) +ENTRY(casuword64_int) casxa [%o0] ASI_AIUP, %o1, %o2 retl mov %o2, %o0 -END(casuword64) +END(casuword64_int) /* * int fuword8(const void *base) Index: /usr/src/sys/sparc64/sparc64/vm_machdep.c =================================================================== --- /usr/src/sys/sparc64/sparc64/vm_machdep.c (revision 349856) +++ /usr/src/sys/sparc64/sparc64/vm_machdep.c (working copy) @@ -462,3 +462,28 @@ sf_buf_unmap(struct sf_buf *sf) pmap_qremove(sf->kva, 1); return (1); } + +uint32_t casuword32_int(volatile uint32_t *base, uint32_t oldval, + uint32_t newval); +uint32_t +casuword32(volatile uint32_t *base, uint32_t oldval, uint32_t newval) +{ + uint32_t ret; + + ret = casuword32_int(base, oldval, newval); + if (ret != -1) + ret = ret != oldval; + return (ret); +} + +u_long casuword64_int(volatile u_long *p, u_long oldval, u_long newval); +u_long +casuword(volatile u_long *p, u_long oldval, u_long newval) +{ + u_long ret; + + ret = casuword64_int(p, oldval, newval); + if (ret != -1L) + ret = ret != oldval; + return (ret); +} -- Test scenario: misc/syscall4.sh