Adaptive mutex spin: 110287 events in 10.206 seconds (10806 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 57829 52% 52% 0.00 23021 256 Bucket sdt_probe+0x38 24599 22% 75% 0.00 756455 page pv (usermode) 14362 13% 88% 0.00 69864 256 Bucket (usermode) 11430 10% 98% 0.00 143560 page pv sdt_probe+0x38 782 1% 99% 0.00 58108 UPVO entry (usermode) 298 0% 99% 0.00 4106 vm reserv domain sdt_probe+0x38 193 0% 99% 0.00 20603 vm reserv domain (usermode) 167 0% 99% 0.00 31784 UPVO entry sdt_probe+0x38 125 0% 100% 0.00 21423 xhci0 (usermode) 78 0% 100% 0.00 17784 ttymtx (usermode) 69 0% 100% 0.00 5043 vm active pagequeue sdt_probe+0x38 53 0% 100% 0.00 4317 process lock sdt_probe+0x38 47 0% 100% 0.00 19173 xhci0 sdt_probe+0x38 42 0% 100% 0.00 6860 ttymtx sdt_probe+0x38 27 0% 100% 0.00 3830 kqueue (usermode) 23 0% 100% 0.00 46476 vm page free queue (usermode) 16 0% 100% 0.00 4621 vnode interlock sdt_probe+0x38 16 0% 100% 0.00 7809 so_rcv sdt_probe+0x38 13 0% 100% 0.00 19263 vm page free queue sdt_probe+0x38 11 0% 100% 0.00 8489 process lock (usermode) 10 0% 100% 0.00 3635 vm pgcache sdt_probe+0x38 10 0% 100% 0.00 5322 vm active pagequeue (usermode) 9 0% 100% 0.00 6137 USB device mutex sdt_probe+0x38 8 0% 100% 0.00 4055 kqueue sdt_probe+0x38 8 0% 100% 0.00 24822 bge3 sdt_probe+0x38 7 0% 100% 0.00 337809 kernelpmap (usermode) 6 0% 100% 0.00 7553 sellck sdt_probe+0x38 6 0% 100% 0.00 17505 bge3 (usermode) 6 0% 100% 0.00 9917 USB device mutex (usermode) 5 0% 100% 0.00 57251 vnode interlock (usermode) 5 0% 100% 0.00 1728 vm pgcache (usermode) 5 0% 100% 0.00 8758 so_rcv (usermode) 4 0% 100% 0.00 16829 so_snd sdt_probe+0x38 3 0% 100% 0.00 4681 so_snd (usermode) 3 0% 100% 0.00 4806 sellck (usermode) 3 0% 100% 0.00 5564 pipe mutex sdt_probe+0x38 3 0% 100% 0.00 4638 bounce pages lock sdt_probe+0x38 1 0% 100% 0.00 8821 umass2 (usermode) 1 0% 100% 0.00 479 select mtxpool (usermode) 1 0% 100% 0.00 9506 select mtxpool sdt_probe+0x38 1 0% 100% 0.00 1039 pipe mutex (usermode) 1 0% 100% 0.00 10370 CAM device lock sdt_probe+0x38 1 0% 100% 0.00 2156333 bounce pages lock (usermode) ------------------------------------------------------------------------------- Spin lock spin: 379 events in 10.206 seconds (37 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 286 75% 75% 0.00 0 witness lock (usermode) 27 7% 83% 0.00 1137738 smp rendezvous (usermode) 17 4% 87% 0.00 3798 sleepq chain sdt_probe+0x38 14 4% 91% 0.00 0 witness lock sdt_probe+0x38 10 3% 93% 0.00 9506 sleepq chain (usermode) 7 2% 95% 0.00 10765 callout sdt_probe+0x38 6 2% 97% 0.00 6154 et_hw_mtx sdt_probe+0x38 6 2% 98% 0.00 4441 callout (usermode) 2 1% 99% 0.00 36251 smp rendezvous sdt_probe+0x38 1 0% 99% 0.00 6004 sched lock 122 (usermode) 1 0% 99% 0.00 1934 sched lock 127 sdt_probe+0x38 1 0% 100% 0.00 17241 entropy harvest mutex sdt_probe+0x38 1 0% 100% 0.00 24626 sched lock 108 (usermode) ------------------------------------------------------------------------------- Thread lock spin: 135 events in 10.206 seconds (13 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 27 20% 20% 0.00 3322 sched lock 147 sdt_probe+0x38 8 6% 26% 0.00 3593 sched lock 129 sdt_probe+0x38 7 5% 31% 0.00 18906 sched lock 130 sdt_probe+0x38 6 4% 36% 0.00 7207 sched lock 19 sdt_probe+0x38 6 4% 40% 0.00 9943 sched lock 122 sdt_probe+0x38 5 4% 44% 0.00 12348 sched lock 21 sdt_probe+0x38 5 4% 47% 0.00 3873 sched lock 20 sdt_probe+0x38 5 4% 51% 0.00 3067 sched lock 135 sdt_probe+0x38 4 3% 54% 0.00 4238 sched lock 151 sdt_probe+0x38 4 3% 57% 0.00 4315 sched lock 133 sdt_probe+0x38 4 3% 60% 0.00 6825 sched lock 125 sdt_probe+0x38 3 2% 62% 0.00 1680 sched lock 63 sdt_probe+0x38 3 2% 64% 0.00 1592 sched lock 134 sdt_probe+0x38 3 2% 67% 0.00 2533 sched lock 127 sdt_probe+0x38 3 2% 69% 0.00 3119 sched lock 121 sdt_probe+0x38 3 2% 71% 0.00 24167 sched lock 108 sdt_probe+0x38 2 1% 73% 0.00 1729 sched lock 75 sdt_probe+0x38 2 1% 74% 0.00 6866 sched lock 72 sdt_probe+0x38 2 1% 76% 0.00 1206 sched lock 50 sdt_probe+0x38 2 1% 77% 0.00 2094 sched lock 149 sdt_probe+0x38 2 1% 79% 0.00 7047 sched lock 146 sdt_probe+0x38 2 1% 80% 0.00 6960 sched lock 13 sdt_probe+0x38 2 1% 81% 0.00 15513 sched lock 120 sdt_probe+0x38 2 1% 83% 0.00 1617 sched lock 110 sdt_probe+0x38 2 1% 84% 0.00 4846 sched lock 106 sdt_probe+0x38 2 1% 86% 0.00 3731 sched lock 0 sdt_probe+0x38 1 1% 87% 0.00 1621 sched lock 79 sdt_probe+0x38 1 1% 87% 0.00 4830 sched lock 77 sdt_probe+0x38 1 1% 88% 0.00 1631 sched lock 74 sdt_probe+0x38 1 1% 89% 0.00 1060 sched lock 73 sdt_probe+0x38 1 1% 90% 0.00 142714 sched lock 57 sdt_probe+0x38 1 1% 90% 0.00 18432 sched lock 58 sdt_probe+0x38 1 1% 91% 0.00 9574 sched lock 5 sdt_probe+0x38 1 1% 92% 0.00 4821 sched lock 51 sdt_probe+0x38 1 1% 93% 0.00 2616 sched lock 18 sdt_probe+0x38 1 1% 93% 0.00 4711 sched lock 22 sdt_probe+0x38 1 1% 94% 0.00 4733 sched lock 38 sdt_probe+0x38 1 1% 95% 0.00 1463 sched lock 44 sdt_probe+0x38 1 1% 96% 0.00 9000 sched lock 16 sdt_probe+0x38 1 1% 96% 0.00 1523 sched lock 131 sdt_probe+0x38 1 1% 97% 0.00 4625 sched lock 132 sdt_probe+0x38 1 1% 98% 0.00 1627 sched lock 109 sdt_probe+0x38 1 1% 99% 0.00 111299 sched lock 124 sdt_probe+0x38 1 1% 99% 0.00 2574 sched lock 123 sdt_probe+0x38 1 1% 100% 0.00 8846 sched lock 126 sdt_probe+0x38 ------------------------------------------------------------------------------- R/W writer blocked by readers: 1 events in 10.206 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 1 100% 100% 0.00 318112 vmobject (usermode) ------------------------------------------------------------------------------- R/W writer spin on writer: 8 events in 10.206 seconds (1 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 7 88% 88% 0.00 69832 SUrw (usermode) 1 12% 100% 0.00 73412 tcpinp (usermode) ------------------------------------------------------------------------------- R/W writer spin on readers: 5 events in 10.206 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 5 100% 100% 0.00 124563 vmobject (usermode) ------------------------------------------------------------------------------- R/W reader spin on writer: 1 events in 10.206 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 1 100% 100% 0.00 34026 vmobject (usermode) ------------------------------------------------------------------------------- Adaptive mutex hold: 4849061 events in 10.206 seconds (475101 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 2081343 43% 43% 0.00 14558 page pv sdt_probe+0x38 1692036 35% 78% 0.00 24505 pmap sdt_probe+0x38 584424 12% 90% 0.00 1896 256 Bucket sdt_probe+0x38 212717 4% 94% 0.00 2197 UPVO entry sdt_probe+0x38 86370 2% 96% 0.00 2758 vm reserv sdt_probe+0x38 66012 1% 97% 0.00 3214 vnode interlock sdt_probe+0x38 18637 0% 98% 0.00 1932 32 Bucket sdt_probe+0x38 16525 0% 98% 0.00 3968 vm active pagequeue sdt_probe+0x38 10103 0% 98% 0.00 2451 vm reserv domain sdt_probe+0x38 9375 0% 99% 0.00 7447 process lock sdt_probe+0x38 8617 0% 99% 0.00 5815 vm page free queue sdt_probe+0x38 6667 0% 99% 0.00 2058 pwddesc sdt_probe+0x38 5902 0% 99% 0.00 3045 select mtxpool sdt_probe+0x38 5041 0% 99% 0.00 6335 xhci0 sdt_probe+0x38 3496 0% 99% 0.00 8923 ttymtx sdt_probe+0x38 2850 0% 99% 0.00 2700 bounce pages lock sdt_probe+0x38 2803 0% 99% 0.00 2136 vm pgcache sdt_probe+0x38 2443 0% 99% 0.00 2490 FFS sdt_probe+0x38 2303 0% 99% 0.00 3878 global softdep sdt_probe+0x38 2157 0% 99% 0.00 1860 devthrd sdt_probe+0x38 2024 0% 99% 0.00 1956 sigacts sdt_probe+0x38 1777 0% 99% 0.00 12199 kernelpmap sdt_probe+0x38 1654 0% 100% 0.00 5770 so_snd sdt_probe+0x38 1618 0% 100% 0.00 1560 128 Bucket sdt_probe+0x38 1527 0% 100% 0.00 4927 so_rcv sdt_probe+0x38 1370 0% 100% 0.00 2710 struct mount mtx sdt_probe+0x38 1201 0% 100% 0.00 2375 resv obj lock sdt_probe+0x38 1124 0% 100% 0.00 9587 socket sdt_probe+0x38 1046 0% 100% 0.00 4292 kqueue sdt_probe+0x38 998 0% 100% 0.00 2278 XPT topology lock sdt_probe+0x38 966 0% 100% 0.00 6608 sleep mtxpool sdt_probe+0x38 897 0% 100% 0.00 8597 bge3 sdt_probe+0x38 884 0% 100% 0.00 3892 CAM doneq sdt_probe+0x38 865 0% 100% 0.00 7326 ncvn sdt_probe+0x38 852 0% 100% 0.00 10604 sellck sdt_probe+0x38 773 0% 100% 0.00 2442 CAM queue lock sdt_probe+0x38 673 0% 100% 0.00 15168 unp sdt_probe+0x38 603 0% 100% 0.00 19627 CAM device lock sdt_probe+0x38 578 0% 100% 0.00 5736 pipe mutex sdt_probe+0x38 547 0% 100% 0.00 2032 dirhash list sdt_probe+0x38 443 0% 100% 0.00 3270 ncbuc sdt_probe+0x38 432 0% 100% 0.00 10017 process group sdt_probe+0x38 408 0% 100% 0.00 4973 SLB table sdt_probe+0x38 390 0% 100% 0.00 2053 g_vfs sdt_probe+0x38 383 0% 100% 0.00 67929 AHCI channel lock sdt_probe+0x38 312 0% 100% 0.00 2842 reseed mutex sdt_probe+0x38 300 0% 100% 0.00 1750 devstat sdt_probe+0x38 285 0% 100% 0.00 1727 cred sdt_probe+0x38 252 0% 100% 0.00 27554 umass1 sdt_probe+0x38 249 0% 100% 0.00 2513 g_disk_done sdt_probe+0x38 235 0% 100% 0.00 2722 vm inactive pagequeue sdt_probe+0x38 232 0% 100% 0.00 29984 umass2 sdt_probe+0x38 220 0% 100% 0.00 36741 umass0 sdt_probe+0x38 218 0% 100% 0.00 4327 dev mtx sdt_probe+0x38 193 0% 100% 0.00 3071 vm pageout lock sdt_probe+0x38 190 0% 100% 0.00 1956 process_exec sdt_probe+0x38 177 0% 100% 0.00 1935 procid sdt_probe+0x38 168 0% 100% 0.00 2186 jail mutex sdt_probe+0x38 159 0% 100% 0.00 2883 RADIX NODE sdt_probe+0x38 151 0% 100% 0.00 2644 chacha20_mtx sdt_probe+0x38 144 0% 100% 0.00 3311 bufspace daemon run lock sdt_probe+0x38 136 0% 100% 0.00 1904 process_exit sdt_probe+0x38 124 0% 100% 0.00 2571 bufq clean lock sdt_probe+0x38 117 0% 100% 0.00 2040 UMA Slabs 0 sdt_probe+0x38 116 0% 100% 0.00 20102 bufq clean subqueue lock sdt_probe+0x38 99 0% 100% 0.00 1774 thread_dtor sdt_probe+0x38 95 0% 100% 0.00 6161 bufq dirty lock sdt_probe+0x38 88 0% 100% 0.00 2451 DEVCTL sdt_probe+0x38 76 0% 100% 0.00 2333 vnode_list sdt_probe+0x38 75 0% 100% 0.00 43915 USB device mutex sdt_probe+0x38 68 0% 100% 0.00 2582 struct mount vlist mtx sdt_probe+0x38 65 0% 100% 0.00 8502 session sdt_probe+0x38 61 0% 100% 0.00 5395 Syncer mtx sdt_probe+0x38 61 0% 100% 0.00 1761 PWD sdt_probe+0x38 58 0% 100% 0.00 3578 ncnegl sdt_probe+0x38 45 0% 100% 0.00 29432 Giant sdt_probe+0x38 41 0% 100% 0.00 6366 msgbuf lock sdt_probe+0x38 39 0% 100% 0.00 2550 umtxql sdt_probe+0x38 25 0% 100% 0.00 2143 malloc-4096 sdt_probe+0x38 22 0% 100% 0.00 8043 buffer daemon lock sdt_probe+0x38 21 0% 100% 0.00 3293 MAP ENTRY sdt_probe+0x38 19 0% 100% 0.00 3573 g_bio sdt_probe+0x38 16 0% 100% 0.00 2329 VNODE sdt_probe+0x38 15 0% 100% 0.00 6591 vdbatch sdt_probe+0x38 15 0% 100% 0.00 5001 lle_event sdt_probe+0x38 13 0% 100% 0.00 33143 umass3 sdt_probe+0x38 13 0% 100% 0.00 3023 itimer lock sdt_probe+0x38 12 0% 100% 0.00 2439 mld_mtx sdt_probe+0x38 11 0% 100% 0.00 2367 runningbufspace lock sdt_probe+0x38 11 0% 100% 0.00 2591 malloc-32 sdt_probe+0x38 11 0% 100% 0.00 3288 malloc-128 sdt_probe+0x38 11 0% 100% 0.00 2174 dirty buf lock sdt_probe+0x38 10 0% 100% 0.00 2788 ada_ccb sdt_probe+0x38 9 0% 100% 0.00 3186 vm map (system) sdt_probe+0x38 9 0% 100% 0.00 2178 spacqtree sdt_probe+0x38 9 0% 100% 0.00 2116 malloc-512 sdt_probe+0x38 9 0% 100% 0.00 2249 dqhlock sdt_probe+0x38 9 0% 100% 0.00 2229 acqtree sdt_probe+0x38 8 0% 100% 0.00 4150005 vm object_list sdt_probe+0x38 8 0% 100% 0.00 1898 malloc-1024 sdt_probe+0x38 8 0% 100% 0.00 2581 igmp_mtx sdt_probe+0x38 8 0% 100% 0.00 2769 SLB tree node sdt_probe+0x38 7 0% 100% 0.00 1711 itimer sdt_probe+0x38 7 0% 100% 0.00 2904 VM OBJECT sdt_probe+0x38 6 0% 100% 0.00 4292 malloc-16 sdt_probe+0x38 5 0% 100% 0.00 10166 eventhandler sdt_probe+0x38 4 0% 100% 0.00 9060 bufq empty lock sdt_probe+0x38 4 0% 100% 0.00 2829 buffer arena-32 sdt_probe+0x38 3 0% 100% 0.00 2974 VMSPACE sdt_probe+0x38 3 0% 100% 0.00 2122 PROC sdt_probe+0x38 2 0% 100% 0.00 6048 thread sdt_probe+0x38 2 0% 100% 0.00 2651 rib head lock sdt_probe+0x38 2 0% 100% 0.00 2217 pipe sdt_probe+0x38 2 0% 100% 0.00 2910 mbuf sdt_probe+0x38 2 0% 100% 0.00 2524 malloc-384 sdt_probe+0x38 2 0% 100% 0.00 1911 filedesc0 sdt_probe+0x38 2 0% 100% 0.00 2190 da_ccb sdt_probe+0x38 1 0% 100% 0.00 5191 vmem list lock sdt_probe+0x38 1 0% 100% 0.00 2948 netlink lock sdt_probe+0x38 1 0% 100% 0.00 1916 malloc-2048 sdt_probe+0x38 ------------------------------------------------------------------------------- Spin lock hold: 1021056 events in 10.206 seconds (100041 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 763611 75% 75% 0.00 4476 et_hw_mtx sdt_probe+0x38 18765 2% 77% 0.00 1642 witness lock sdt_probe+0x38 12065 1% 78% 0.00 8463 callout sdt_probe+0x38 10886 1% 79% 0.00 2245 sched lock 0 sdt_probe+0x38 10410 1% 80% 0.00 8553 sleepq chain sdt_probe+0x38 10070 1% 81% 0.00 2577 tcsetc sdt_probe+0x38 3165 0% 81% 0.00 639112 smp rendezvous sdt_probe+0x38 1931 0% 81% 0.00 3972 sched lock 147 sdt_probe+0x38 1915 0% 82% 0.00 4031 sched lock 135 sdt_probe+0x38 1901 0% 82% 0.00 3289 sched lock 149 sdt_probe+0x38 1815 0% 82% 0.00 4219 sched lock 106 sdt_probe+0x38 1789 0% 82% 0.00 3850 sched lock 38 sdt_probe+0x38 1765 0% 82% 0.00 3069 sched lock 91 sdt_probe+0x38 1721 0% 82% 0.00 4010 sched lock 130 sdt_probe+0x38 1709 0% 83% 0.00 3778 sched lock 11 sdt_probe+0x38 1685 0% 83% 0.00 3998 sched lock 19 sdt_probe+0x38 1650 0% 83% 0.00 4134 sched lock 20 sdt_probe+0x38 1624 0% 83% 0.00 3636 sched lock 32 sdt_probe+0x38 1623 0% 83% 0.00 4478 sched lock 127 sdt_probe+0x38 1622 0% 83% 0.00 4042 sched lock 131 sdt_probe+0x38 1617 0% 84% 0.00 4170 sched lock 21 sdt_probe+0x38 1543 0% 84% 0.00 4351 sched lock 120 sdt_probe+0x38 1511 0% 84% 0.00 4357 sched lock 150 sdt_probe+0x38 1487 0% 84% 0.00 3053 sched lock 13 sdt_probe+0x38 1482 0% 84% 0.00 4633 sched lock 63 sdt_probe+0x38 1474 0% 84% 0.00 2878 sched lock 117 sdt_probe+0x38 1460 0% 84% 0.00 4520 sched lock 16 sdt_probe+0x38 1459 0% 85% 0.00 3434 sched lock 35 sdt_probe+0x38 1435 0% 85% 0.00 4714 sched lock 146 sdt_probe+0x38 1434 0% 85% 0.00 2940 sched lock 133 sdt_probe+0x38 1412 0% 85% 0.00 2812 sched lock 76 sdt_probe+0x38 1384 0% 85% 0.00 4588 sched lock 129 sdt_probe+0x38 1380 0% 85% 0.00 4606 sched lock 110 sdt_probe+0x38 1366 0% 85% 0.00 3019 sched lock 4 sdt_probe+0x38 1345 0% 86% 0.00 4556 sched lock 104 sdt_probe+0x38 1338 0% 86% 0.00 3031 sched lock 154 sdt_probe+0x38 1332 0% 86% 0.00 4579 sched lock 121 sdt_probe+0x38 1326 0% 86% 0.00 3044 sched lock 68 sdt_probe+0x38 1325 0% 86% 0.00 3058 sched lock 159 sdt_probe+0x38 1322 0% 86% 0.00 4780 sched lock 105 sdt_probe+0x38 1320 0% 86% 0.00 3110 sched lock 51 sdt_probe+0x38 1315 0% 86% 0.00 3251 sched lock 24 sdt_probe+0x38 1315 0% 87% 0.00 3071 sched lock 115 sdt_probe+0x38 1313 0% 87% 0.00 3056 sched lock 65 sdt_probe+0x38 1311 0% 87% 0.00 3110 sched lock 52 sdt_probe+0x38 1304 0% 87% 0.00 3541 sched lock 111 sdt_probe+0x38 1301 0% 87% 0.00 3196 sched lock 41 sdt_probe+0x38 1299 0% 87% 0.00 3196 sched lock 22 sdt_probe+0x38 1298 0% 87% 0.00 3183 sched lock 28 sdt_probe+0x38 1297 0% 87% 0.00 5358 sched lock 151 sdt_probe+0x38 1296 0% 88% 0.00 3149 sched lock 89 sdt_probe+0x38 1293 0% 88% 0.00 2968 sched lock 138 sdt_probe+0x38 1292 0% 88% 0.00 2822 sched lock 59 sdt_probe+0x38 1291 0% 88% 0.00 3190 sched lock 9 sdt_probe+0x38 1291 0% 88% 0.00 4782 sched lock 145 sdt_probe+0x38 1290 0% 88% 0.00 3254 sched lock 40 sdt_probe+0x38 1290 0% 88% 0.00 4843 sched lock 36 sdt_probe+0x38 1282 0% 88% 0.00 2968 sched lock 137 sdt_probe+0x38 1279 0% 89% 0.00 3155 sched lock 81 sdt_probe+0x38 1278 0% 89% 0.00 3072 sched lock 99 sdt_probe+0x38 1278 0% 89% 0.00 3036 sched lock 97 sdt_probe+0x38 1275 0% 89% 0.00 3912 sched lock 126 sdt_probe+0x38 1272 0% 89% 0.00 3776 sched lock 80 sdt_probe+0x38 1271 0% 89% 0.00 4823 sched lock 132 sdt_probe+0x38 1266 0% 89% 0.00 4747 sched lock 124 sdt_probe+0x38 1261 0% 89% 0.00 4945 sched lock 144 sdt_probe+0x38 1237 0% 90% 0.00 4076 sched lock 123 sdt_probe+0x38 1234 0% 90% 0.00 4796 sched lock 134 sdt_probe+0x38 1233 0% 90% 0.00 4902 sched lock 73 sdt_probe+0x38 1218 0% 90% 0.00 4762 sched lock 108 sdt_probe+0x38 1207 0% 90% 0.00 4899 sched lock 5 sdt_probe+0x38 1193 0% 90% 0.00 5031 sched lock 107 sdt_probe+0x38 1191 0% 90% 0.00 5214 sched lock 78 sdt_probe+0x38 1188 0% 90% 0.00 8004 sched lock 143 sdt_probe+0x38 1188 0% 91% 0.00 4918 sched lock 122 sdt_probe+0x38 1180 0% 91% 0.00 4936 sched lock 62 sdt_probe+0x38 1180 0% 91% 0.00 5004 sched lock 109 sdt_probe+0x38 1170 0% 91% 0.00 5634 sched lock 72 sdt_probe+0x38 1165 0% 91% 0.00 4230 sched lock 148 sdt_probe+0x38 1140 0% 91% 0.00 5288 sched lock 75 sdt_probe+0x38 1138 0% 91% 0.00 5102 sched lock 125 sdt_probe+0x38 1135 0% 91% 0.00 5291 sched lock 74 sdt_probe+0x38 1129 0% 91% 0.00 7339 sched lock 118 sdt_probe+0x38 1126 0% 92% 0.00 5068 sched lock 58 sdt_probe+0x38 1122 0% 92% 0.00 5206 sched lock 34 sdt_probe+0x38 1110 0% 92% 0.00 5396 sched lock 77 sdt_probe+0x38 1110 0% 92% 0.00 4935 sched lock 57 sdt_probe+0x38 1097 0% 92% 0.00 5853 sched lock 79 sdt_probe+0x38 1096 0% 92% 0.00 5310 sched lock 56 sdt_probe+0x38 1091 0% 92% 0.00 5011 sched lock 17 sdt_probe+0x38 1081 0% 92% 0.00 5174 sched lock 3 sdt_probe+0x38 1081 0% 92% 0.00 7572 sched lock 113 sdt_probe+0x38 1081 0% 93% 0.00 5323 sched lock 1 sdt_probe+0x38 1080 0% 93% 0.00 7332 sched lock 116 sdt_probe+0x38 1076 0% 93% 0.00 5177 sched lock 2 sdt_probe+0x38 1072 0% 93% 0.00 5317 sched lock 37 sdt_probe+0x38 1068 0% 93% 0.00 6910 sched lock 44 sdt_probe+0x38 1062 0% 93% 0.00 8719 sched lock 103 sdt_probe+0x38 1061 0% 93% 0.00 5184 sched lock 60 sdt_probe+0x38 1061 0% 93% 0.00 5645 sched lock 128 sdt_probe+0x38 1057 0% 93% 0.00 5370 sched lock 6 sdt_probe+0x38 1056 0% 93% 0.00 5656 sched lock 39 sdt_probe+0x38 1054 0% 94% 0.00 8113 sched lock 119 sdt_probe+0x38 1052 0% 94% 0.00 6271 sched lock 153 sdt_probe+0x38 1050 0% 94% 0.00 5143 sched lock 30 sdt_probe+0x38 1049 0% 94% 0.00 5089 sched lock 18 sdt_probe+0x38 1048 0% 94% 0.00 5422 sched lock 33 sdt_probe+0x38 1046 0% 94% 0.00 7118 sched lock 53 sdt_probe+0x38 1045 0% 94% 0.00 5965 sched lock 90 sdt_probe+0x38 1045 0% 94% 0.00 8428 sched lock 100 sdt_probe+0x38 1044 0% 94% 0.00 5444 sched lock 23 sdt_probe+0x38 1044 0% 95% 0.00 8231 sched lock 141 sdt_probe+0x38 1043 0% 95% 0.00 7204 sched lock 50 sdt_probe+0x38 1040 0% 95% 0.00 5153 sched lock 12 sdt_probe+0x38 1039 0% 95% 0.00 5598 sched lock 7 sdt_probe+0x38 1034 0% 95% 0.00 7001 sched lock 87 sdt_probe+0x38 1034 0% 95% 0.00 6447 sched lock 82 sdt_probe+0x38 1034 0% 95% 0.00 8176 sched lock 139 sdt_probe+0x38 1031 0% 95% 0.00 6614 sched lock 84 sdt_probe+0x38 1029 0% 95% 0.00 5321 sched lock 61 sdt_probe+0x38 1028 0% 95% 0.00 6379 sched lock 158 sdt_probe+0x38 1027 0% 96% 0.00 5208 sched lock 29 sdt_probe+0x38 1027 0% 96% 0.00 7761 sched lock 114 sdt_probe+0x38 1026 0% 96% 0.00 8296 sched lock 98 sdt_probe+0x38 1026 0% 96% 0.00 6728 sched lock 152 sdt_probe+0x38 1025 0% 96% 0.00 5244 sched lock 27 sdt_probe+0x38 1023 0% 96% 0.00 5169 sched lock 25 sdt_probe+0x38 1021 0% 96% 0.00 9023 sched lock 136 sdt_probe+0x38 1020 0% 96% 0.00 7652 sched lock 48 sdt_probe+0x38 1020 0% 96% 0.00 8576 sched lock 140 sdt_probe+0x38 1019 0% 96% 0.00 5293 sched lock 14 sdt_probe+0x38 1018 0% 97% 0.00 7633 sched lock 55 sdt_probe+0x38 1016 0% 97% 0.00 6293 sched lock 94 sdt_probe+0x38 1016 0% 97% 0.00 7354 sched lock 54 sdt_probe+0x38 1016 0% 97% 0.00 6460 sched lock 157 sdt_probe+0x38 1016 0% 97% 0.00 6479 sched lock 155 sdt_probe+0x38 1015 0% 97% 0.00 7087 sched lock 43 sdt_probe+0x38 1014 0% 97% 0.00 6237 sched lock 92 sdt_probe+0x38 1014 0% 97% 0.00 6367 sched lock 156 sdt_probe+0x38 1013 0% 97% 0.00 6642 sched lock 86 sdt_probe+0x38 1011 0% 97% 0.00 7588 sched lock 67 sdt_probe+0x38 1011 0% 98% 0.00 7153 sched lock 42 sdt_probe+0x38 1009 0% 98% 0.00 7696 sched lock 70 sdt_probe+0x38 1009 0% 98% 0.00 5254 sched lock 26 sdt_probe+0x38 1009 0% 98% 0.00 5220 sched lock 10 sdt_probe+0x38 1007 0% 98% 0.00 7622 sched lock 69 sdt_probe+0x38 1006 0% 98% 0.00 8915 sched lock 96 sdt_probe+0x38 1006 0% 98% 0.00 6669 sched lock 83 sdt_probe+0x38 1006 0% 98% 0.00 8062 sched lock 64 sdt_probe+0x38 1006 0% 98% 0.00 7552 sched lock 47 sdt_probe+0x38 1006 0% 98% 0.00 7191 sched lock 45 sdt_probe+0x38 1006 0% 98% 0.00 5547 sched lock 31 sdt_probe+0x38 1006 0% 99% 0.00 8239 sched lock 112 sdt_probe+0x38 1006 0% 99% 0.00 8525 sched lock 102 sdt_probe+0x38 1005 0% 99% 0.00 6583 sched lock 85 sdt_probe+0x38 1005 0% 99% 0.00 7280 sched lock 46 sdt_probe+0x38 1005 0% 99% 0.00 8605 sched lock 142 sdt_probe+0x38 1005 0% 99% 0.00 5650 sched lock 15 sdt_probe+0x38 1005 0% 99% 0.00 8543 sched lock 101 sdt_probe+0x38 1004 0% 99% 0.00 6471 sched lock 95 sdt_probe+0x38 1004 0% 99% 0.00 6174 sched lock 93 sdt_probe+0x38 1004 0% 99% 0.00 6400 sched lock 88 sdt_probe+0x38 1004 0% 100% 0.00 7232 sched lock 49 sdt_probe+0x38 1003 0% 100% 0.00 8128 sched lock 71 sdt_probe+0x38 1003 0% 100% 0.00 7738 sched lock 66 sdt_probe+0x38 996 0% 100% 0.00 5459 sched lock 8 sdt_probe+0x38 801 0% 100% 0.00 3619 bge_taskq sdt_probe+0x38 335 0% 100% 0.00 2689 entropy harvest mutex sdt_probe+0x38 100 0% 100% 0.00 8053 process slock sdt_probe+0x38 52 0% 100% 0.00 3907 pstatl sdt_probe+0x38 8 0% 100% 0.00 2136 ntp sdt_probe+0x38 2 0% 100% 0.00 18699 turnstile lock sdt_probe+0x38 2 0% 100% 0.00 13455 turnstile chain sdt_probe+0x38 2 0% 100% 0.00 2194 td_contested sdt_probe+0x38 ------------------------------------------------------------------------------- R/W writer hold: 429723 events in 10.206 seconds (42103 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 421087 98% 98% 0.00 4688 vmobject sdt_probe+0x38 5646 1% 99% 0.00 3541 SUrw sdt_probe+0x38 2506 1% 100% 0.00 3149 bufobj interlock sdt_probe+0x38 275 0% 100% 0.00 41504 tcpinp sdt_probe+0x38 113 0% 100% 0.00 2208 tidhash sdt_probe+0x38 73 0% 100% 0.00 2234 vfs hash sdt_probe+0x38 18 0% 100% 0.00 2468 nd6 list sdt_probe+0x38 5 0% 100% 0.00 39177 lle sdt_probe+0x38 ------------------------------------------------------------------------------- R/W reader hold: 2793266 events in 10.206 seconds (273678 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 2713174 97% 97% 0.00 55118 pte eviction sdt_probe+0x38 79422 3% 100% 0.00 562432 vmobject sdt_probe+0x38 504 0% 100% 0.00 2666 vfs hash sdt_probe+0x38 121 0% 100% 0.00 4839 vnet_rwlock sdt_probe+0x38 39 0% 100% 0.00 3222 tidhash sdt_probe+0x38 6 0% 100% 0.00 48909 udpinp sdt_probe+0x38 ------------------------------------------------------------------------------- SX shared hold: 209242 events in 10.206 seconds (20501 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 206758 99% 99% 0.00 72471 vm map (user) sdt_probe+0x38 1724 1% 100% 0.00 2578 filedesc structure sdt_probe+0x38 547 0% 100% 0.00 14815 dirhash sdt_probe+0x38 168 0% 100% 0.00 2704 sysctl string handler sdt_probe+0x38 36 0% 100% 0.00 321574 killpg racer sdt_probe+0x38 8 0% 100% 0.00 908157 allproc sdt_probe+0x38 1 0% 100% 0.00 5218 pidhash sdt_probe+0x38 ------------------------------------------------------------------------------- SX exclusive hold: 38367 events in 10.206 seconds (3759 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 23774 62% 62% 0.00 2504 filedesc structure sdt_probe+0x38 11544 30% 92% 0.00 3994862 vm map (user) sdt_probe+0x38 1507 4% 96% 0.00 48347 dtrace_lock sdt_probe+0x38 332 1% 97% 0.00 45871 so_snd_sx sdt_probe+0x38 281 1% 98% 0.00 27028 proctree sdt_probe+0x38 203 1% 98% 0.00 22490 so_rcv_sx sdt_probe+0x38 133 0% 98% 0.00 2842 fasttrap_provs.fth_table[i].ftb_mtx sdt_probe+0x38 123 0% 99% 0.00 2423 pidhash sdt_probe+0x38 121 0% 99% 0.00 2543 allproc sdt_probe+0x38 92 0% 99% 0.00 422555 CMDQ lock sdt_probe+0x38 83 0% 100% 0.00 3197 dirhash sdt_probe+0x38 66 0% 100% 0.00 207965 USB suspend and resume SX lock sdt_probe+0x38 39 0% 100% 0.00 342471 USB control transfer SX lock sdt_probe+0x38 33 0% 100% 0.00 20105 killpg racer sdt_probe+0x38 27 0% 100% 0.00 998210 USB config SX lock sdt_probe+0x38 9 0% 100% 0.00 27930 umareclaim sdt_probe+0x38 ------------------------------------------------------------------------------- lockmgr shared hold: 72238 events in 10.206 seconds (7078 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 72238 100% 100% 0.00 16227 ufs sdt_probe+0x38 ------------------------------------------------------------------------------- lockmgr exclusive hold: 7246 events in 10.206 seconds (710 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 6702 92% 92% 0.00 46689 bufwait sdt_probe+0x38 533 7% 100% 0.00 161514 ufs sdt_probe+0x38 11 0% 100% 0.00 357519 rename sdt_probe+0x38 -------------------------------------------------------------------------------