GENERIC RELENG_6 from Sep 22 14:50 UTC, vmcore.86
Ran out of "128 Bucket".
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.2-PRERELEASE #0: Fri Sep 22 17:04:36 CEST 2006
pho@crashbox.osted.lan:/usr/src/sys/i386/compile/PHO
WARNING: WITNESS option enabled, expect reduced performance.
ACPI APIC Table: <A M I OEMAPIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) XEON(TM) CPU 1.80GHz (1799.81-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf24 Stepping = 4
Features=0x3febfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM>
Logical CPUs per core: 2
real memory = 1073676288 (1023 MB)
avail memory = 1032581120 (984 MB)
:
Trying to mount root from ufs:/dev/ad0s1a
fxp0: link state changed to UP
Limiting open port RST response from 250 to 200 packets/sec
KDB: enter: manual escape to debugger
[thread pid 21 tid 100023 ]
Stopped at kdb_enter+0x2b: nop
db> show allpcpu
Current CPU: 0
cpuid = 0
curthread = 0xc5400780: pid 21 "swi6: Giant taskq"
curpcb = 0xe57ead90
fpcurthread = none
idlethread = 0xc52f0a80: pid 13 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:
cpuid = 1
curthread = 0xc52f0900: pid 12 "idle: cpu1"
curpcb = 0xe3d71d90
fpcurthread = none
idlethread = 0xc52f0900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
cpuid = 2
curthread = 0xc52f0780: pid 11 "idle: cpu2"
curpcb = 0xe3d6ed90
fpcurthread = none
idlethread = 0xc52f0780: pid 11 "idle: cpu2"
APIC ID = 6
currentldt = 0x50
spin locks held:
cpuid = 3
curthread = 0xc52f0600: pid 10 "idle: cpu3"
curpcb = 0xe3d6bd90
fpcurthread = none
idlethread = 0xc52f0600: pid 10 "idle: cpu3"
APIC ID = 7
currentldt = 0x50
spin locks held:
db> show alllocks
Process 21 (swi6: Giant taskq) thread 0xc5400780 (100023)
exclusive sleep mutex Giant r = 0 (0xc09d1e40) locked @ kern/kern_intr.c:681
db> ps
pid ppid pgrp uid state wmesg wchan cmd
8471 1006 1005 1001 S+ nanslp 0xc09d272c sleep
7682 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7680 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7678 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7677 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7676 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7675 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7674 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7673 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7672 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7669 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7664 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7659 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7647 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7646 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7645 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7643 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7641 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7614 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7611 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7610 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7604 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7590 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7589 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7588 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7583 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7581 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7580 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7564 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7556 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7555 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7554 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7552 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7549 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7548 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7547 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7545 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7544 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7532 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7505 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7500 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7492 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7489 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7488 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7487 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7483 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7481 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7479 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7469 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7465 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7462 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7439 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7437 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7433 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7430 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7423 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7418 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7416 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7406 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7402 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7397 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7396 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7391 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7388 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7378 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7373 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7372 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7369 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7367 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7364 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7357 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7356 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7353 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7345 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7342 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7336 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7333 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7331 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7322 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7321 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7317 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7310 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7303 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7302 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7300 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7297 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7295 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7292 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7287 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7286 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7284 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7283 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7278 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7277 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7271 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7270 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7263 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7261 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7258 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7249 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7244 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7238 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7237 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7219 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7212 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7210 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7208 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7204 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7203 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7201 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7200 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7198 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7197 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7196 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7193 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7191 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7189 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7183 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7180 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7179 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7175 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7166 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7163 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7162 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7160 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7155 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7154 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7153 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7148 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7145 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7140 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7136 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7135 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7127 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7123 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7114 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7110 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7106 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7101 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7098 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7096 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7090 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7083 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7080 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7078 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7076 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7068 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7066 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7060 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7057 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7044 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7033 1 5671 1001 S+ zonelimi 0xc1853460 tcp
7013 1 5671 1001 S+ zonelimi 0xc1853460 tcp
6923 1 5671 1001 S+ zonelimi 0xc1853460 tcp
6832 1 5671 1001 S+ zonelimi 0xc1853460 tcp
6710 1 5671 1001 S+ zonelimi 0xc1853460 tcp
6709 1 5671 1001 S+ zonelimi 0xc1853460 tcp
6553 1 5671 1001 S+ zonelimi 0xc1853460 tcp
1020 1019 1020 1001 Ss+ ttyin 0xc55f6010 bash
1019 1017 1017 1001 S select 0xc0a1fa84 sshd
1017 847 1017 0 Ss sbwait 0xc57ace90 sshd
1007 1005 1005 1001 S+ piperd 0xc56f8660 awk
1006 1005 1005 1001 S+ wait 0xc569c430 sh
1005 999 1005 1001 S+ wait 0xc56e2000 sh
999 998 999 1001 Ss+ wait 0xc583f218 bash
998 996 996 1001 S select 0xc0a1fa84 sshd
996 847 996 0 Ss sbwait 0xc5926370 sshd
991 1 991 0 Ss+ ttyin 0xc54c0c10 getty
990 1 990 0 Ss+ ttyin 0xc54c3010 getty
989 1 989 0 Ss+ ttyin 0xc54c2010 getty
988 1 988 0 Ss+ ttyin 0xc54c1410 getty
987 1 987 0 Ss+ ttyin 0xc54c1810 getty
986 1 986 0 Ss+ ttyin 0xc54bc010 getty
985 1 985 0 Ss+ ttyin 0xc54c3410 getty
984 1 984 0 Ss+ ttyin 0xc54c2410 getty
967 1 967 0 Ss select 0xc0a1fa84 inetd
960 939 960 1001 Ss+ ttywai 0xc544044c bash
939 883 883 1001 S zonelimi 0xc1853460 sshd
934 1 934 0 Ss select 0xc0a1fa84 moused
917 1 917 0 Ss nanslp 0xc09d272c watchdogd
883 847 883 0 Ss sbwait 0xc57b54d4 sshd
864 1 864 0 Ss nanslp 0xc09d272c cron
858 1 858 25 Ss pause 0xc57ca24c sendmail
854 1 854 0 Ss select 0xc0a1fa84 sendmail
847 1 847 0 Ss select 0xc0a1fa84 sshd
827 1 827 0 Ss select 0xc0a1fa84 ntpd
769 765 765 0 S - 0xc57eae00 nfsd
768 765 765 0 S - 0xc54a0c00 nfsd
767 765 765 0 S - 0xc57eb000 nfsd
766 765 765 0 S - 0xc54a0e00 nfsd
765 1 765 0 Ss select 0xc0a1fa84 nfsd
763 1 763 0 Ss select 0xc0a1fa84 mountd
703 1 703 0 Ss select 0xc0a1fa84 rpcbind
680 1 680 0 Ss select 0xc0a1fa84 syslogd
570 1 570 0 Ss select 0xc0a1fa84 devd
42 0 0 0 SL - 0xe57f6d04 [schedcpu]
41 0 0 0 SL sdflush 0xc0a2d494 [softdepflush]
40 0 0 0 SL syncer 0xc09d24a0 [syncer]
39 0 0 0 SL vlruwt 0xc560b000 [vnlru]
38 0 0 0 SL psleep 0xc0a1ffec [bufdaemon]
37 0 0 0 SL pgzero 0xc0a2e404 [pagezero]
36 0 0 0 SL psleep 0xc0a2df54 [vmdaemon]
35 0 0 0 SL psleep 0xc0a2df10 [pagedaemon]
34 0 0 0 WL [irq7: ppc0]
33 0 0 0 SL - 0xc54a003c [fdc0]
32 0 0 0 WL [swi0: sio]
31 0 0 0 WL [irq12: psm0]
30 0 0 0 WL [irq1: atkbd0]
29 0 0 0 WL [irq15: ata1]
28 0 0 0 WL [irq14: ata0]
27 0 0 0 WL [irq17: fxp0]
26 0 0 0 SL usbtsk 0xc09cf3a4 [usbtask]
25 0 0 0 SL usbevt 0xc5405210 [usb0]
24 0 0 0 WL [irq16: uhci0]
23 0 0 0 SL - 0xc542b400 [em0 taskq]
22 0 0 0 WL [irq9: acpi0]
21 0 0 0 RL CPU 0 [swi6: Giant taskq]
20 0 0 0 WL [swi6: task queue]
9 0 0 0 SL - 0xc5340180 [kqueue taskq]
8 0 0 0 SL - 0xc5340200 [acpi_task_2]
7 0 0 0 SL - 0xc5340200 [acpi_task_1]
6 0 0 0 SL - 0xc5340200 [acpi_task_0]
19 0 0 0 WL [swi2: cambio]
18 0 0 0 WL [swi5: +]
5 0 0 0 SL - 0xc5340500 [thread taskq]
17 0 0 0 SL - 0xc09cd0c0 [yarrow]
4 0 0 0 SL - 0xc09cfbe8 [g_down]
3 0 0 0 SL - 0xc09cfbe4 [g_up]
2 0 0 0 SL - 0xc09cfbdc [g_event]
16 0 0 0 WL [swi1: net]
15 0 0 0 WL [swi3: vm]
14 0 0 0 WL [swi4: clock sio]
13 0 0 0 RL [idle: cpu0]
12 0 0 0 RL CPU 1 [idle: cpu1]
11 0 0 0 RL CPU 2 [idle: cpu2]
10 0 0 0 RL CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xc52f4000 [init]
0 0 0 0 WLs [swapper]
db> where 7682
Tracing pid 7682 tid 101187 td 0xc6989900
sched_switch(c6989900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c1853460,c09d1e00,0,c08ea008,211,...) at sleepq_switch+0xc1
sleepq_wait(c1853460,0,c0902151,8a2,6a0,...) at sleepq_wait+0x46
msleep(c1853460,c1853468,44,c0902495,0) at msleep+0x279
uma_zone_slab(c184fb40,2,c1853468,0,c0902148,8a2) at uma_zone_slab+0xd3
uma_zalloc_internal(c184fb40,cdf24800,2,c1853468,0,...) at uma_zalloc_internal+0x29
uma_zalloc_arg(c184fb40,cdf24800,2) at uma_zalloc_arg+0x31f
mb_zinit_pack(cdf24800,100,2) at mb_zinit_pack+0x18
uma_zalloc_internal(c184fd20,e8ef6bfc,2,c18533c8,0,...) at uma_zalloc_internal+0x90
uma_zalloc_arg(c184fd20,e8ef6bfc,2) at uma_zalloc_arg+0x31f
sosend(c81d16f4,0,e8ef6cbc,0,0,0,c6989900) at sosend+0x3c5
soo_write(c79de7e0,e8ef6cbc,c5be5b00,0,c6989900) at soo_write+0x46
dofilewrite(c6989900,3,c79de7e0,e8ef6cbc,ffffffff,...) at dofilewrite+0x77
kern_writev(c6989900,3,e8ef6cbc,8057000,2000,...) at kern_writev+0x3b
write(c6989900,e8ef6d04) at write+0x45
syscall(2805003b,bfbf003b,bfbf003b,280509c8,bfbfebe4,...) at syscall+0x22f
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2814336f, esp = 0xbfbfea3c, ebp = 0xbfbfea88 ---
db> call doadump
Dumping 1023 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok
Dump complete
= 0xf
db> reset
(kgdb) l *uma_zone_slab+0xd3
0xc07c26d3 is in uma_zone_slab (../../../vm/uma_core.c:2020).
2015 keg->uk_flags |= UMA_ZFLAG_FULL;
2016
2017 if (flags & M_NOWAIT)
2018 break;
2019 else
2020 msleep(keg, &keg->uk_lock, PVM,
2021 "zonelimit", 0);
2022 continue;
2023 }
2024 keg->uk_recurse++;
(kgdb) p *(struct uma_zone *)0xc184fb40
$1 = {uz_name = 0xc08e58ed "mbuf_cluster", uz_lock = 0xc1853468, uz_keg = 0xc1853460, uz_link = {le_next = 0x0,
le_prev = 0xc1853498}, uz_full_bucket = {lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0},
uz_ctor = 0xc0661d74 <mb_ctor_clust>, uz_dtor = 0xc0661e24 <mb_dtor_clust>, uz_init = 0, uz_fini = 0, uz_allocs = 0x279ee,
uz_frees = 0x215ee, uz_fails = 0x0, uz_fills = 0x4, uz_count = 0x80, uz_cpu = {{uc_freebucket = 0xc6c59c48,
uc_allocbucket = 0x0, uc_allocs = 0x0, uc_frees = 0x0}}}
$ vmstat -z | more
ITEM SIZE LIMIT USED FREE REQUESTS
UMA Kegs: 140, 0, 66, 6, 66
UMA Zones: 480, 0, 66, 6, 66
UMA Slabs: 64, 0, 3070, 2122, 3864
UMA RCntSlabs: 104, 0, 12800, 2, 12800
UMA Hash: 128, 0, 3, 27, 6
16 Bucket: 76, 0, 64, 36, 100
32 Bucket: 140, 0, 76, 36, 113
64 Bucket: 268, 0, 78, 20, 143
128 Bucket: 524, 0, 3976, 0, 28677