Index: contrib/one-true-awk =================================================================== --- contrib/one-true-awk (.../head) (revision 246685) +++ contrib/one-true-awk (.../projects/calloutng) (revision 246685) Property changes on: contrib/one-true-awk ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/one-true-awk:r236314-246070 Index: contrib/bzip2 =================================================================== --- contrib/bzip2 (.../head) (revision 246685) +++ contrib/bzip2 (.../projects/calloutng) (revision 246685) Property changes on: contrib/bzip2 ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/bzip2:r236314-242500 Index: contrib/tzdata =================================================================== --- contrib/tzdata (.../head) (revision 246685) +++ contrib/tzdata (.../projects/calloutng) (revision 246685) Property changes on: contrib/tzdata ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/tzdata:r236314-243181 Index: contrib/gdtoa =================================================================== --- contrib/gdtoa (.../head) (revision 246685) +++ contrib/gdtoa (.../projects/calloutng) (revision 246685) Property changes on: contrib/gdtoa ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/gdtoa:r236314-243949 Index: contrib/libc++ =================================================================== --- contrib/libc++ (.../head) (revision 246685) +++ contrib/libc++ (.../projects/calloutng) (revision 246685) Property changes on: contrib/libc++ ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libc++:r236314-246684 Index: contrib/top =================================================================== --- contrib/top (.../head) (revision 246685) +++ contrib/top (.../projects/calloutng) (revision 246685) Property changes on: contrib/top ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/top:r236314-244264 Index: contrib/pf =================================================================== --- contrib/pf (.../head) (revision 246685) +++ contrib/pf (.../projects/calloutng) (revision 246685) Property changes on: contrib/pf ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/pf:r236314-242500 Index: contrib/libarchive/libarchive =================================================================== --- contrib/libarchive/libarchive (.../head) (revision 246685) +++ contrib/libarchive/libarchive (.../projects/calloutng) (revision 246685) Property changes on: contrib/libarchive/libarchive ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libarchive/libarchive:r236314-246684 Index: contrib/libarchive/cpio =================================================================== --- contrib/libarchive/cpio (.../head) (revision 246685) +++ contrib/libarchive/cpio (.../projects/calloutng) (revision 246685) Property changes on: contrib/libarchive/cpio ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libarchive/cpio:r236314-239017 Index: contrib/libarchive/libarchive_fe =================================================================== --- contrib/libarchive/libarchive_fe (.../head) (revision 246685) +++ contrib/libarchive/libarchive_fe (.../projects/calloutng) (revision 246685) Property changes on: contrib/libarchive/libarchive_fe ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libarchive/libarchive_fe:r236314-239017 Index: contrib/libarchive/tar =================================================================== --- contrib/libarchive/tar (.../head) (revision 246685) +++ contrib/libarchive/tar (.../projects/calloutng) (revision 246685) Property changes on: contrib/libarchive/tar ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libarchive/tar:r236314-242500 Index: contrib/libarchive =================================================================== --- contrib/libarchive (.../head) (revision 246685) +++ contrib/libarchive (.../projects/calloutng) (revision 246685) Property changes on: contrib/libarchive ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libarchive:r236314-246684 Index: contrib/dtc =================================================================== --- contrib/dtc (.../head) (revision 246685) +++ contrib/dtc (.../projects/calloutng) (revision 246685) Property changes on: contrib/dtc ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/dtc:r236314-243949 Index: contrib/tcpdump =================================================================== --- contrib/tcpdump (.../head) (revision 246685) +++ contrib/tcpdump (.../projects/calloutng) (revision 246685) Property changes on: contrib/tcpdump ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/tcpdump:r236314-246070 Index: contrib/bmake =================================================================== --- contrib/bmake (.../head) (revision 246685) +++ contrib/bmake (.../projects/calloutng) (revision 246685) Property changes on: contrib/bmake ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/bmake:r236314-239165,241146-246684 Index: contrib/file =================================================================== --- contrib/file (.../head) (revision 246685) +++ contrib/file (.../projects/calloutng) (revision 246685) Property changes on: contrib/file ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/file:r236314-246070 Index: contrib/byacc =================================================================== --- contrib/byacc (.../head) (revision 246685) +++ contrib/byacc (.../projects/calloutng) (revision 246685) Property changes on: contrib/byacc ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/byacc:r236314-242500 Index: contrib/ee =================================================================== --- contrib/ee (.../head) (revision 246685) +++ contrib/ee (.../projects/calloutng) (revision 246685) Property changes on: contrib/ee ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/ee:r236314-246070 Index: contrib/gcc =================================================================== --- contrib/gcc (.../head) (revision 246685) +++ contrib/gcc (.../projects/calloutng) (revision 246685) Property changes on: contrib/gcc ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/gcc:r236314-246684 Index: contrib/sendmail =================================================================== --- contrib/sendmail (.../head) (revision 246685) +++ contrib/sendmail (.../projects/calloutng) (revision 246685) Property changes on: contrib/sendmail ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/sendmail:r236314-246070 Index: contrib/libcxxrt =================================================================== --- contrib/libcxxrt (.../head) (revision 246685) +++ contrib/libcxxrt (.../projects/calloutng) (revision 246685) Property changes on: contrib/libcxxrt ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libcxxrt:r236314-246684 Index: contrib/xz =================================================================== --- contrib/xz (.../head) (revision 246685) +++ contrib/xz (.../projects/calloutng) (revision 246685) Property changes on: contrib/xz ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/xz:r236314-244676 Index: contrib/ntp =================================================================== --- contrib/ntp (.../head) (revision 246685) +++ contrib/ntp (.../projects/calloutng) (revision 246685) Property changes on: contrib/ntp ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/ntp:r236314-246070 Index: contrib/openbsm =================================================================== --- contrib/openbsm (.../head) (revision 246685) +++ contrib/openbsm (.../projects/calloutng) (revision 246685) Property changes on: contrib/openbsm ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/openbsm:r236314-244314 Index: contrib/libpcap =================================================================== --- contrib/libpcap (.../head) (revision 246685) +++ contrib/libpcap (.../projects/calloutng) (revision 246685) Property changes on: contrib/libpcap ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libpcap:r236314-246070 Index: contrib/libstdc++ =================================================================== --- contrib/libstdc++ (.../head) (revision 246685) +++ contrib/libstdc++ (.../projects/calloutng) (revision 246685) Property changes on: contrib/libstdc++ ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/libstdc++:r236314-246684 Index: contrib/gdb =================================================================== --- contrib/gdb (.../head) (revision 246685) +++ contrib/gdb (.../projects/calloutng) (revision 246685) Property changes on: contrib/gdb ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/gdb:r236314-246684 Index: contrib/bind9 =================================================================== --- contrib/bind9 (.../head) (revision 246685) +++ contrib/bind9 (.../projects/calloutng) (revision 246685) Property changes on: contrib/bind9 ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/bind9:r236314-244149 Index: contrib/less =================================================================== --- contrib/less (.../head) (revision 246685) +++ contrib/less (.../projects/calloutng) (revision 246685) Property changes on: contrib/less ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/less:r236314-243900 Index: contrib/atf =================================================================== --- contrib/atf (.../head) (revision 246685) +++ contrib/atf (.../projects/calloutng) (revision 246685) Property changes on: contrib/atf ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/atf:r236314-239165,240120-243181 Index: contrib/netcat =================================================================== --- contrib/netcat (.../head) (revision 246685) +++ contrib/netcat (.../projects/calloutng) (revision 246685) Property changes on: contrib/netcat ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/netcat:r236314-242500 Index: contrib/expat =================================================================== --- contrib/expat (.../head) (revision 246685) +++ contrib/expat (.../projects/calloutng) (revision 246685) Property changes on: contrib/expat ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/expat:r236314-246070 Index: contrib/wpa =================================================================== --- contrib/wpa (.../head) (revision 246685) +++ contrib/wpa (.../projects/calloutng) (revision 246685) Property changes on: contrib/wpa ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/wpa:r236314-243900 Index: contrib/binutils =================================================================== --- contrib/binutils (.../head) (revision 246685) +++ contrib/binutils (.../projects/calloutng) (revision 246685) Property changes on: contrib/binutils ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/binutils:r236314-246684 Index: contrib/openresolv =================================================================== --- contrib/openresolv (.../head) (revision 246685) +++ contrib/openresolv (.../projects/calloutng) (revision 246685) Property changes on: contrib/openresolv ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/openresolv:r236314-242500 Index: contrib/compiler-rt =================================================================== --- contrib/compiler-rt (.../head) (revision 246685) +++ contrib/compiler-rt (.../projects/calloutng) (revision 246685) Property changes on: contrib/compiler-rt ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/compiler-rt:r236314-246070 Index: contrib/dialog =================================================================== --- contrib/dialog (.../head) (revision 246685) +++ contrib/dialog (.../projects/calloutng) (revision 246685) Property changes on: contrib/dialog ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/dialog:r236314-246070 Index: contrib/groff =================================================================== --- contrib/groff (.../head) (revision 246685) +++ contrib/groff (.../projects/calloutng) (revision 246685) Property changes on: contrib/groff ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/groff:r236314-242500 Index: contrib/llvm/tools/clang =================================================================== --- contrib/llvm/tools/clang (.../head) (revision 246685) +++ contrib/llvm/tools/clang (.../projects/calloutng) (revision 246685) Property changes on: contrib/llvm/tools/clang ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/llvm/tools/clang:r236314-246684 Index: contrib/llvm =================================================================== --- contrib/llvm (.../head) (revision 246685) +++ contrib/llvm (.../projects/calloutng) (revision 246685) Property changes on: contrib/llvm ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/contrib/llvm:r236314-246684 Index: share/zoneinfo =================================================================== --- share/zoneinfo (.../head) (revision 246685) +++ share/zoneinfo (.../projects/calloutng) (revision 246685) Property changes on: share/zoneinfo ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/share/zoneinfo:r236314-246070 Index: share/man/man4/bhyve.4 =================================================================== --- share/man/man4/bhyve.4 (.../head) (revision 246685) +++ share/man/man4/bhyve.4 (.../projects/calloutng) (revision 246685) Property changes on: share/man/man4/bhyve.4 ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/share/man/man4/bhyve.4:r245652-246070 Index: share/man/man4 =================================================================== --- share/man/man4 (.../head) (revision 246685) +++ share/man/man4 (.../projects/calloutng) (revision 246685) Property changes on: share/man/man4 ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/share/man/man4:r236314-246070 Index: share/man/man9/sleepqueue.9 =================================================================== --- share/man/man9/sleepqueue.9 (.../head) (revision 246685) +++ share/man/man9/sleepqueue.9 (.../projects/calloutng) (revision 246685) @@ -23,7 +23,7 @@ .\" .\" $FreeBSD$ .\" -.Dd January 8, 2010 +.Dd December 21, 2012 .Dt SLEEPQUEUE 9 .Os .Sh NAME @@ -41,6 +41,7 @@ .Nm sleepq_remove , .Nm sleepq_signal , .Nm sleepq_set_timeout , +.Nm sleepq_set_timeout_bt , .Nm sleepq_sleepcnt , .Nm sleepq_timedwait , .Nm sleepq_timedwait_sig , @@ -79,6 +80,9 @@ .Fn sleepq_signal "void *wchan" "int flags" "int pri" "int queue" .Ft void .Fn sleepq_set_timeout "void *wchan" "int timo" +.Ft void +.Fn sleepq_set_timeout_bt "void *wchan" "struct bintime bt" \ +"struct bintime pr" "int flags" .Ft u_int .Fn sleepq_sleepcnt "void *wchan" "int queue" .Ft int @@ -231,6 +235,21 @@ .Fa timo parameter should specify the timeout value in ticks. .Pp +.Fn sleepq_set_timeout_bt +function takes +.Fa bt +argument instead of +.Fa timo . +It allows to specify wakeup time with higher resolution as absolute time +since boot in form of +.Vt struct bintime * , +as returned by +.Fn binuptime +function. +The parameter +.Fa pr +allows to specify wanted absolute event precision. +.Pp The current thread may be marked interruptible by calling .Fn sleepq_catch_signals with Index: share/man/man9/condvar.9 =================================================================== --- share/man/man9/condvar.9 (.../head) (revision 246685) +++ share/man/man9/condvar.9 (.../projects/calloutng) (revision 246685) @@ -26,7 +26,7 @@ .\" .\" $FreeBSD$ .\" -.Dd June 5, 2007 +.Dd December 21, 2012 .Dt CONDVAR 9 .Os .Sh NAME @@ -37,7 +37,9 @@ .Nm cv_wait_sig , .Nm cv_wait_unlock , .Nm cv_timedwait , +.Nm cv_timedwait_bt , .Nm cv_timedwait_sig , +.Nm cv_timedwait_sig_bt , .Nm cv_signal , .Nm cv_broadcast , .Nm cv_broadcastpri , @@ -60,7 +62,13 @@ .Ft int .Fn cv_timedwait "struct cv *cvp" "lock" "int timo" .Ft int +.Fn cv_timedwait_bt "struct cv *cvp" "lock" "struct bintime bt" \ +"struct bintime pr" "int flags" +.Ft int .Fn cv_timedwait_sig "struct cv *cvp" "lock" "int timo" +.Ft int +.Fn cv_timedwait_sig_bt "struct cv *cvp" "lock" "struct bintime bt" \ +"struct bintime pr" "int flags" .Ft void .Fn cv_signal "struct cv *cvp" .Ft void @@ -191,6 +199,23 @@ .Fn cv_signal or .Fn cv_broadcast . +.Pp +.Fn cv_timedwait_bt +and +.Fn cv_timedwait_sig_bt +functions take +.Fa bt +argument instead of +.Fa timo . +It allows to specify unblock time with higher resolution as absolute time +since boot in form of +.Vt struct bintime * , +as returned by +.Fn binuptime +function. +The parameter +.Fa pr +allows to specify wanted absolute event precision. .Sh RETURN VALUES If successful, .Fn cv_wait_sig , Index: share/man/man9/sleep.9 =================================================================== --- share/man/man9/sleep.9 (.../head) (revision 246685) +++ share/man/man9/sleep.9 (.../projects/calloutng) (revision 246685) @@ -25,14 +25,18 @@ .\" .\" $FreeBSD$ .\" -.Dd December 12, 2009 +.Dd December 21, 2012 .Dt SLEEP 9 .Os .Sh NAME .Nm msleep , +.Nm msleep_bt , .Nm msleep_spin , +.Nm msleep_spin_bt , .Nm pause , +.Nm pause_bt , .Nm tsleep , +.Nm tsleep_bt , .Nm wakeup .Nd wait for events .Sh SYNOPSIS @@ -42,11 +46,23 @@ .Ft int .Fn msleep "void *chan" "struct mtx *mtx" "int priority" "const char *wmesg" "int timo" .Ft int +.Fn msleep_bt "void *chan" "struct mtx *mtx" "int priority" \ +"const char *wmesg" "struct bintime bt" "struct bintime pr" "int flags" +.Ft int .Fn msleep_spin "void *chan" "struct mtx *mtx" "const char *wmesg" "int timo" +.Ft int +.Fn msleep_spin_bt "void *chan" "struct mtx *mtx" "const char *wmesg" \ +"struct bintime bt" "struct bintime pr" "int flags" .Ft void .Fn pause "const char *wmesg" "int timo" +.Ft void +.Fn pause_bt "const char *wmesg" "struct bintime bt" "struct bintime pr" \ + "int flags" .Ft int .Fn tsleep "void *chan" "int priority" "const char *wmesg" "int timo" +.Ft int +.Fn tsleep_bt "void *chan" "int priority" "const char *wmesg" \ +"struct bintime bt" "struct bintime pr" "int flags" .Ft void .Fn wakeup "void *chan" .Ft void @@ -148,6 +164,30 @@ then the sleep function will return .Er EWOULDBLOCK . .Pp +The parameter +.Fa flags +allows to pass additional +.Fn callout_reset_bt +flags. +.Pp +.Fn msleep_bt , +.Fn pause_bt +and +.Fn tsleep_bt +functions take +.Fa bt +parameter instead of +.Fa timo . +It allows to specify wakeup time with higher resolution as absolute time +since boot in form of +.Vt struct bintime * , +as returned by +.Fn binuptime +function. +The parameter +.Fa pr +allows to specify wanted absolute event precision. +.Pp Several of the sleep functions including .Fn msleep , .Fn msleep_spin , Index: share/man/man9/timeout.9 =================================================================== --- share/man/man9/timeout.9 (.../head) (revision 246685) +++ share/man/man9/timeout.9 (.../projects/calloutng) (revision 246685) @@ -29,7 +29,7 @@ .\" .\" $FreeBSD$ .\" -.Dd November 1, 2012 +.Dd December 21, 2012 .Dt TIMEOUT 9 .Os .Sh NAME @@ -44,6 +44,9 @@ .Nm callout_reset , .Nm callout_reset_on , .Nm callout_reset_curcpu , +.Nm callout_reset_bt , +.Nm callout_reset_bt_on , +.Nm callout_reset_bt_curcpu , .Nm callout_schedule , .Nm callout_schedule_on , .Nm callout_schedule_curcpu , @@ -82,6 +85,9 @@ .Fn callout_reset_on "struct callout *c" "int ticks" "timeout_t *func" \ "void *arg" "int cpu" .Ft int +.Fn callout_reset_bt_on "struct callout *c" "struct bintime bt" \ +"struct bintime pr" "timeout_t *func" "void *arg" "int cpu" "int flags" +.Ft int .Fn callout_reset_curcpu "struct callout *c" "int ticks" "timeout_t *func" \ "void *arg" .Ft int @@ -326,6 +332,39 @@ .Fn callout_schedule but take an extra parameter specifying the target CPU for the callout. .Pp +The function +.Fn callout_reset_bt_on +allows to get higher time resolution, taking absolute time since boot +in form of struct bintime *, as returned by +.Fn binuptime +or +.Fn getbinuptime +functions, and precision, instead of relative ticks count. +If specified time is in past, it will be silently converted to present +to run handler as soon as possible. +.Pp +The following +.Fa flags +may be specified: +.Bl -tag -width ".Dv C_DIRECT_EXEC" +.It Dv C_ALSOLUTE +Handle the +.Fa bt +argument as absolute time of the event since boot, as returned by +.Fn binuptime +function. +.It Dv C_DIRECT_EXEC +Run handler directly from hardware interrupt context instead of softclock swi. +It is faster, but puts more constraints on handlers. +Handlers may use only spin mutexes for locking, and they must be fast because +they run with absolute priority. +.It Fn C_PREL +Specifies relative event time precision as binary logarithm of time interval +divided by acceptable time deviation: 1 -- 1/2, 2 -- 1/4, etc. +Smaller value allows to aggregate more events in one timer interrupt to +reduce processing overhead and power consumption. +.El +.Pp The functions .Fn callout_reset_curcpu and Index: share/man/man9/Makefile =================================================================== --- share/man/man9/Makefile (.../head) (revision 246685) +++ share/man/man9/Makefile (.../projects/calloutng) (revision 246685) @@ -1196,9 +1196,13 @@ signal.9 SIG_STOPSIGMASK.9 \ signal.9 trapsignal.9 MLINKS+=sleep.9 msleep.9 \ + sleep.9 msleep_bt.9 \ sleep.9 msleep_spin.9 \ + sleep.9 msleep_spin_bt.9 \ sleep.9 pause.9 \ + sleep.9 pause_bt.9 \ sleep.9 tsleep.9 \ + sleep.9 tsleep_bt.9 \ sleep.9 wakeup.9 \ sleep.9 wakeup_one.9 MLINKS+=sleepqueue.9 init_sleepqueues.9 \ @@ -1213,6 +1217,7 @@ sleepqueue.9 sleepq_release.9 \ sleepqueue.9 sleepq_remove.9 \ sleepqueue.9 sleepq_set_timeout.9 \ + sleepqueue.9 sleepq_set_timeout_bt.9 \ sleepqueue.9 sleepq_signal.9 \ sleepqueue.9 sleepq_timedwait.9 \ sleepqueue.9 sleepq_timedwait_sig.9 \ @@ -1335,6 +1340,9 @@ timeout.9 callout_init_rw.9 \ timeout.9 callout_pending.9 \ timeout.9 callout_reset.9 \ + timeout.9 callout_reset_bt.9 \ + timeout.9 callout_reset_on.9 \ + timeout.9 callout_reset_bt_on.9 \ timeout.9 callout_schedule.9 \ timeout.9 callout_stop.9 \ timeout.9 untimeout.9 Index: usr.sbin/jail =================================================================== --- usr.sbin/jail (.../head) (revision 246685) +++ usr.sbin/jail (.../projects/calloutng) (revision 246685) Property changes on: usr.sbin/jail ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.sbin/jail:r236314-242500 Index: usr.sbin/bhyveload =================================================================== --- usr.sbin/bhyveload (.../head) (revision 246685) +++ usr.sbin/bhyveload (.../projects/calloutng) (revision 246685) Property changes on: usr.sbin/bhyveload ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.sbin/bhyveload:r245652-246070 Index: usr.sbin/ndiscvt =================================================================== --- usr.sbin/ndiscvt (.../head) (revision 246685) +++ usr.sbin/ndiscvt (.../projects/calloutng) (revision 246685) Property changes on: usr.sbin/ndiscvt ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.sbin/ndiscvt:r236314-243949 Index: usr.sbin/bhyvectl =================================================================== --- usr.sbin/bhyvectl (.../head) (revision 246685) +++ usr.sbin/bhyvectl (.../projects/calloutng) (revision 246685) Property changes on: usr.sbin/bhyvectl ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.sbin/bhyvectl:r245652-246070 Index: usr.sbin/bhyve =================================================================== --- usr.sbin/bhyve (.../head) (revision 246685) +++ usr.sbin/bhyve (.../projects/calloutng) (revision 246685) Property changes on: usr.sbin/bhyve ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.sbin/bhyve:r245652-246684 Index: lib/libc/stdtime =================================================================== --- lib/libc/stdtime (.../head) (revision 246685) +++ lib/libc/stdtime (.../projects/calloutng) (revision 246685) Property changes on: lib/libc/stdtime ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/lib/libc/stdtime:r236314-237808 Index: lib/libc =================================================================== --- lib/libc (.../head) (revision 246685) +++ lib/libc (.../projects/calloutng) (revision 246685) Property changes on: lib/libc ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/lib/libc:r236314-246684 Index: lib/libz =================================================================== --- lib/libz (.../head) (revision 246685) +++ lib/libz (.../projects/calloutng) (revision 246685) Property changes on: lib/libz ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/lib/libz:r236314-237808 Index: lib/libutil =================================================================== --- lib/libutil (.../head) (revision 246685) +++ lib/libutil (.../projects/calloutng) (revision 246685) Property changes on: lib/libutil ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/lib/libutil:r236314-246070 Index: lib/libvmmapi =================================================================== --- lib/libvmmapi (.../head) (revision 246685) +++ lib/libvmmapi (.../projects/calloutng) (revision 246685) Property changes on: lib/libvmmapi ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/lib/libvmmapi:r245652-246070 Index: lib/libprocstat/zfs.c =================================================================== --- lib/libprocstat/zfs.c (.../head) (revision 246685) +++ lib/libprocstat/zfs.c (.../projects/calloutng) (revision 246685) @@ -35,6 +35,7 @@ #undef lbolt #undef lbolt64 +#undef gethrestime #undef gethrestime_sec #include #include Index: cddl/contrib/opensolaris/cmd/zfs =================================================================== --- cddl/contrib/opensolaris/cmd/zfs (.../head) (revision 246685) +++ cddl/contrib/opensolaris/cmd/zfs (.../projects/calloutng) (revision 246685) Property changes on: cddl/contrib/opensolaris/cmd/zfs ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/cddl/contrib/opensolaris/cmd/zfs:r236314-246684 Index: cddl/contrib/opensolaris/lib/libzfs =================================================================== --- cddl/contrib/opensolaris/lib/libzfs (.../head) (revision 246685) +++ cddl/contrib/opensolaris/lib/libzfs (.../projects/calloutng) (revision 246685) Property changes on: cddl/contrib/opensolaris/lib/libzfs ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/cddl/contrib/opensolaris/lib/libzfs:r236314-246684 Index: cddl/contrib/opensolaris =================================================================== --- cddl/contrib/opensolaris (.../head) (revision 246685) +++ cddl/contrib/opensolaris (.../projects/calloutng) (revision 246685) Property changes on: cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/cddl/contrib/opensolaris:r236314-246684 Index: MAINTAINERS =================================================================== --- MAINTAINERS (.../head) (revision 246685) +++ MAINTAINERS (.../projects/calloutng) (revision 246685) Property changes on: MAINTAINERS ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/MAINTAINERS:r236314-242500 Index: sys/modules/vmm =================================================================== --- sys/modules/vmm (.../head) (revision 246685) +++ sys/modules/vmm (.../projects/calloutng) (revision 246685) Property changes on: sys/modules/vmm ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/modules/vmm:r245652-246070 Index: sys/dev/led/led.c =================================================================== --- sys/dev/led/led.c (.../head) (revision 246685) +++ sys/dev/led/led.c (.../projects/calloutng) (revision 246685) @@ -43,6 +43,7 @@ static struct sx led_sx; static LIST_HEAD(, ledsc) led_list = LIST_HEAD_INITIALIZER(led_list); static struct callout led_ch; +static int blinkers = 0; static MALLOC_DEFINE(M_LED, "LED", "LED driver"); @@ -51,7 +52,6 @@ { struct ledsc *sc; - mtx_lock(&led_mtx); LIST_FOREACH(sc, &led_list, list) { if (sc->ptr == NULL) continue; @@ -61,6 +61,7 @@ } if (*sc->ptr == '.') { sc->ptr = NULL; + blinkers--; continue; } else if (*sc->ptr == 'U' || *sc->ptr == 'u') { if (sc->last_second == time_second) @@ -78,9 +79,8 @@ if (*sc->ptr == '\0') sc->ptr = sc->str; } - mtx_unlock(&led_mtx); - callout_reset(&led_ch, hz / 10, led_timeout, p); - return; + if (blinkers > 0) + callout_reset(&led_ch, hz / 10, led_timeout, p); } static int @@ -92,9 +92,15 @@ sc->spec = *sb; if (*sb != NULL) { sc->str = sbuf_data(*sb); + if (sc->ptr == NULL) { + blinkers++; + callout_reset(&led_ch, hz / 10, led_timeout, NULL); + } sc->ptr = sc->str; } else { sc->str = NULL; + if (sc->ptr != NULL) + blinkers--; sc->ptr = NULL; sc->func(sc->private, state); } @@ -286,8 +292,6 @@ mtx_lock(&led_mtx); sc->dev->si_drv1 = sc; - if (LIST_EMPTY(&led_list)) - callout_reset(&led_ch, hz / 10, led_timeout, NULL); LIST_INSERT_HEAD(&led_list, sc, list); sc->func(sc->private, state != 0); mtx_unlock(&led_mtx); @@ -303,7 +307,8 @@ mtx_lock(&led_mtx); sc = dev->si_drv1; dev->si_drv1 = NULL; - + if (sc->ptr != NULL) + blinkers--; LIST_REMOVE(sc, list); if (LIST_EMPTY(&led_list)) callout_stop(&led_ch); @@ -326,7 +331,7 @@ led_unit = new_unrhdr(0, INT_MAX, NULL); mtx_init(&led_mtx, "LED mtx", NULL, MTX_DEF); sx_init(&led_sx, "LED sx"); - callout_init(&led_ch, CALLOUT_MPSAFE); + callout_init_mtx(&led_ch, &led_mtx, 0); } SYSINIT(leddev, SI_SUB_DRIVERS, SI_ORDER_MIDDLE, led_drvinit, NULL); Index: sys/dev/dcons/dcons_os.c =================================================================== --- sys/dev/dcons/dcons_os.c (.../head) (revision 246685) +++ sys/dev/dcons/dcons_os.c (.../projects/calloutng) (revision 246685) @@ -74,6 +74,10 @@ #define DCONS_POLL_HZ 25 #endif +#ifndef DCONS_POLL_IDLE +#define DCONS_POLL_IDLE 256 +#endif + #ifndef DCONS_BUF_SIZE #define DCONS_BUF_SIZE (16*1024) #endif @@ -90,6 +94,7 @@ static struct dcons_global dg; struct dcons_global *dcons_conf; static int poll_hz = DCONS_POLL_HZ; +static u_int poll_idle = DCONS_POLL_HZ * DCONS_POLL_IDLE; static struct dcons_softc sc[DCONS_NPORT]; @@ -214,14 +219,17 @@ tp = dc->tty; tty_lock(tp); - while ((c = dcons_os_checkc_nopoll(dc)) != -1) + while ((c = dcons_os_checkc_nopoll(dc)) != -1) { ttydisc_rint(tp, c, 0); + poll_idle = 0; + } ttydisc_rint_done(tp); tty_unlock(tp); } - polltime = hz / poll_hz; - if (polltime < 1) - polltime = 1; + poll_idle++; + polltime = hz; + if (poll_idle <= (poll_hz * DCONS_POLL_IDLE)) + polltime /= poll_hz; callout_reset(&dcons_callout, polltime, dcons_timeout, tp); } @@ -368,8 +376,6 @@ dcons_attach_port(DCONS_GDB, "dgdb", DC_GDB); callout_init(&dcons_callout, CALLOUT_MPSAFE); polltime = hz / poll_hz; - if (polltime < 1) - polltime = 1; callout_reset(&dcons_callout, polltime, dcons_timeout, NULL); return(0); } Index: sys/dev/acpica/acpi_cpu.c =================================================================== --- sys/dev/acpica/acpi_cpu.c (.../head) (revision 246685) +++ sys/dev/acpica/acpi_cpu.c (.../projects/calloutng) (revision 246685) @@ -168,7 +168,7 @@ static void acpi_cpu_startup(void *arg); static void acpi_cpu_startup_cx(struct acpi_cpu_softc *sc); static void acpi_cpu_cx_list(struct acpi_cpu_softc *sc); -static void acpi_cpu_idle(void); +static void acpi_cpu_idle(sbintime_t sbt); static void acpi_cpu_notify(ACPI_HANDLE h, UINT32 notify, void *context); static int acpi_cpu_quirks(void); static int acpi_cpu_usage_sysctl(SYSCTL_HANDLER_ARGS); @@ -954,13 +954,13 @@ * interrupts are re-enabled. */ static void -acpi_cpu_idle() +acpi_cpu_idle(sbintime_t sbt) { struct acpi_cpu_softc *sc; struct acpi_cx *cx_next; uint64_t cputicks; uint32_t start_time, end_time; - int bm_active, cx_next_idx, i; + int bm_active, cx_next_idx, i, us; /* * Look up our CPU id to get our softc. If it's NULL, we'll use C1 @@ -980,13 +980,16 @@ } /* Find the lowest state that has small enough latency. */ + us = sc->cpu_prev_sleep; + if (sbt >= 0 && us > sbt / SBT_1US) + us = sbt / SBT_1US; cx_next_idx = 0; if (cpu_disable_deep_sleep) i = min(sc->cpu_cx_lowest, sc->cpu_non_c3); else i = sc->cpu_cx_lowest; for (; i >= 0; i--) { - if (sc->cpu_cx_states[i].trans_lat * 3 <= sc->cpu_prev_sleep) { + if (sc->cpu_cx_states[i].trans_lat * 3 <= us) { cx_next_idx = i; break; } Index: sys/dev/random/randomdev_soft.c =================================================================== --- sys/dev/random/randomdev_soft.c (.../head) (revision 246685) +++ sys/dev/random/randomdev_soft.c (.../projects/calloutng) (revision 246685) @@ -242,10 +242,10 @@ local_count = 0; /* Process until told to stop */ + mtx_lock_spin(&harvest_mtx); for (; random_kthread_control >= 0;) { /* Cycle through all the entropy sources */ - mtx_lock_spin(&harvest_mtx); for (source = RANDOM_START; source < ENTROPYSOURCE; source++) { /* * Drain entropy source records into a thread-local @@ -270,7 +270,6 @@ emptyfifo.count += local_count; local_count = 0; } - mtx_unlock_spin(&harvest_mtx); KASSERT(local_count == 0, ("random_kthread: local_count %d", local_count)); @@ -283,9 +282,11 @@ random_kthread_control = 0; /* Work done, so don't belabour the issue */ - pause("-", hz / 10); + msleep_spin_sbt(&random_kthread_control, &harvest_mtx, + "-", (SBT_1S / 10), 0, C_PREL(1)); } + mtx_unlock_spin(&harvest_mtx); random_set_wakeup_exit(&random_kthread_control); /* NOTREACHED */ Index: sys/dev/bvm =================================================================== --- sys/dev/bvm (.../head) (revision 246685) +++ sys/dev/bvm (.../projects/calloutng) (revision 246685) Property changes on: sys/dev/bvm ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/dev/bvm:r245652-246070 Index: sys/dev/syscons/syscons.c =================================================================== --- sys/dev/syscons/syscons.c (.../head) (revision 246685) +++ sys/dev/syscons/syscons.c (.../projects/calloutng) (revision 246685) @@ -506,6 +506,8 @@ sc = sc_get_softc(unit, flags & SC_KERNEL_CONSOLE); sc->config = flags; + callout_init(&sc->ctimeout, 0); + callout_init(&sc->cblink, 0); scp = sc_get_stat(sc->dev[0]); if (sc_console == NULL) /* sc_console_unit < 0 */ sc_console = scp; @@ -1831,13 +1833,11 @@ scrn_timer(void *arg) { #ifndef PC98 - static int kbd_interval = 0; + static time_t kbd_time_stamp = 0; #endif - struct timeval tv; sc_softc_t *sc; scr_stat *scp; - int again; - int s; + int again, rate; again = (arg != NULL); if (arg != NULL) @@ -1847,18 +1847,18 @@ else return; + /* find the vty to update */ + scp = sc->cur_scp; + /* don't do anything when we are performing some I/O operations */ - if (suspend_in_progress || sc->font_loading_in_progress) { - if (again) - timeout(scrn_timer, sc, hz / 10); - return; - } - s = spltty(); + if (suspend_in_progress || sc->font_loading_in_progress) + goto done; #ifndef PC98 if ((sc->kbd == NULL) && (sc->config & SC_AUTODETECT_KBD)) { /* try to allocate a keyboard automatically */ - if (++kbd_interval >= 25) { + if (kbd_time_stamp != time_uptime) { + kbd_time_stamp = time_uptime; sc->keyboard = sc_allocate_keyboard(sc, -1); if (sc->keyboard >= 0) { sc->kbd = kbd_get_keyboard(sc->keyboard); @@ -1867,25 +1867,20 @@ update_kbd_state(sc->cur_scp, sc->cur_scp->status, LOCK_MASK); } - kbd_interval = 0; } } #endif /* PC98 */ - /* find the vty to update */ - scp = sc->cur_scp; - /* should we stop the screen saver? */ - getmicrouptime(&tv); if (debugger > 0 || panicstr || shutdown_in_progress) sc_touch_scrn_saver(); if (run_scrn_saver) { - if (tv.tv_sec > sc->scrn_time_stamp + scrn_blank_time) + if (time_uptime > sc->scrn_time_stamp + scrn_blank_time) sc->flags |= SC_SCRN_IDLE; else sc->flags &= ~SC_SCRN_IDLE; } else { - sc->scrn_time_stamp = tv.tv_sec; + sc->scrn_time_stamp = time_uptime; sc->flags &= ~SC_SCRN_IDLE; if (scrn_blank_time > 0) run_scrn_saver = TRUE; @@ -1898,12 +1893,8 @@ /* should we just return ? */ if (sc->blink_in_progress || sc->switch_in_progress - || sc->write_in_progress) { - if (again) - timeout(scrn_timer, sc, hz / 10); - splx(s); - return; - } + || sc->write_in_progress) + goto done; /* Update the screen */ scp = sc->cur_scp; /* cur_scp may have changed... */ @@ -1917,9 +1908,19 @@ (*current_saver)(sc, TRUE); #endif - if (again) - timeout(scrn_timer, sc, hz / 25); - splx(s); +done: + if (again) { + /* + * Use reduced "refresh" rate if we are in graphics and that is not a + * graphical screen saver. In such case we just have nothing to do. + */ + if (ISGRAPHSC(scp) && !(sc->flags & SC_SCRN_BLANKED)) + rate = 2; + else + rate = 30; + callout_reset_sbt(&sc->ctimeout, (SBT_1S / rate), 0, + scrn_timer, sc, C_PREL(1)); + } } static int @@ -3863,7 +3864,8 @@ (*scp->rndr->draw)(scp, 0, scp->xsize*scp->ysize, scp->sc->blink_in_progress & 1); scp->sc->blink_in_progress--; - timeout(blink_screen, scp, hz / 10); + callout_reset_sbt(&scp->sc->cblink, (SBT_1S / 15), 0, + blink_screen, scp, C_PREL(0)); } } Index: sys/dev/syscons/syscons.h =================================================================== --- sys/dev/syscons/syscons.h (.../head) (revision 246685) +++ sys/dev/syscons/syscons.h (.../projects/calloutng) (revision 246685) @@ -269,6 +269,8 @@ #ifdef KDB int sc_altbrk; #endif + struct callout ctimeout; + struct callout cblink; } sc_softc_t; /* virtual screen */ Index: sys/conf/NOTES =================================================================== --- sys/conf/NOTES (.../head) (revision 246685) +++ sys/conf/NOTES (.../projects/calloutng) (revision 246685) @@ -259,6 +259,8 @@ # SMP Debugging Options: # +# CALLOUT_PROFILING enables rudimentary profiling of the callwheel data +# structure used as backend in callout(9). # PREEMPTION allows the threads that are in the kernel to be preempted by # higher priority [interrupt] threads. It helps with interactivity # and allows interrupt threads to run sooner rather than waiting. @@ -297,6 +299,9 @@ options MPROF_BUFFERS="1536" options MPROF_HASH_SIZE="1543" +# Profiling for the callout(9) backend. +options CALLOUT_PROFILING + # Profiling for internal hash tables. options SLEEPQUEUE_PROFILING options TURNSTILE_PROFILING Index: sys/conf/options =================================================================== --- sys/conf/options (.../head) (revision 246685) +++ sys/conf/options (.../projects/calloutng) (revision 246685) @@ -68,6 +68,7 @@ ADAPTIVE_LOCKMGRS ALQ AUDIT opt_global.h +CALLOUT_PROFILING CAPABILITIES opt_capsicum.h CAPABILITY_MODE opt_capsicum.h COMPAT_43 opt_compat.h Index: sys/conf =================================================================== --- sys/conf (.../head) (revision 246685) +++ sys/conf (.../projects/calloutng) (revision 246685) Property changes on: sys/conf ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/conf:r236314-246684 Index: sys/x86/isa/clock.c =================================================================== --- sys/x86/isa/clock.c (.../head) (revision 246685) +++ sys/x86/isa/clock.c (.../projects/calloutng) (revision 246685) @@ -125,6 +125,8 @@ static struct attimer_softc *attimer_sc = NULL; static int timer0_period = -2; +static int timer0_mode = 0xffff; +static int timer0_last = 0xffff; /* Values for timerX_state: */ #define RELEASED 0 @@ -404,7 +406,7 @@ static void set_i8254_freq(int mode, uint32_t period) { - int new_count; + int new_count, new_mode; mtx_lock_spin(&clock_lock); if (mode == MODE_STOP) { @@ -423,21 +425,34 @@ timer0_period = (mode == MODE_PERIODIC) ? new_count : -1; switch (mode) { case MODE_STOP: - outb(TIMER_MODE, TIMER_SEL0 | TIMER_INTTC | TIMER_16BIT); + new_mode = TIMER_SEL0 | TIMER_INTTC | TIMER_16BIT; + outb(TIMER_MODE, new_mode); outb(TIMER_CNTR0, 0); outb(TIMER_CNTR0, 0); break; case MODE_PERIODIC: - outb(TIMER_MODE, TIMER_SEL0 | TIMER_RATEGEN | TIMER_16BIT); + new_mode = TIMER_SEL0 | TIMER_RATEGEN | TIMER_16BIT; + outb(TIMER_MODE, new_mode); outb(TIMER_CNTR0, new_count & 0xff); outb(TIMER_CNTR0, new_count >> 8); break; case MODE_ONESHOT: - outb(TIMER_MODE, TIMER_SEL0 | TIMER_INTTC | TIMER_16BIT); + if (new_count < 256 && timer0_last < 256) { + new_mode = TIMER_SEL0 | TIMER_INTTC | TIMER_LSB; + if (new_mode != timer0_mode) + outb(TIMER_MODE, new_mode); + outb(TIMER_CNTR0, new_count & 0xff); + break; + } + new_mode = TIMER_SEL0 | TIMER_INTTC | TIMER_16BIT; + if (new_mode != timer0_mode) + outb(TIMER_MODE, new_mode); outb(TIMER_CNTR0, new_count & 0xff); outb(TIMER_CNTR0, new_count >> 8); break; } + timer0_mode = new_mode; + timer0_last = new_count; out: mtx_unlock_spin(&clock_lock); } @@ -447,6 +462,8 @@ { timer0_period = -2; + timer0_mode = 0xffff; + timer0_last = 0xffff; if (attimer_sc != NULL) set_i8254_freq(attimer_sc->mode, attimer_sc->period); else Index: sys/kern/kern_synch.c =================================================================== --- sys/kern/kern_synch.c (.../head) (revision 246685) +++ sys/kern/kern_synch.c (.../projects/calloutng) (revision 246685) @@ -146,12 +146,12 @@ */ int _sleep(void *ident, struct lock_object *lock, int priority, - const char *wmesg, int timo) + const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags) { struct thread *td; struct proc *p; struct lock_class *class; - int catch, flags, lock_state, pri, rval; + int catch, sleepq_flags, lock_state, pri, rval; WITNESS_SAVE_DECL(lock_witness); td = curthread; @@ -162,7 +162,7 @@ #endif WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, lock, "Sleeping on \"%s\"", wmesg); - KASSERT(timo != 0 || mtx_owned(&Giant) || lock != NULL, + KASSERT(sbt != 0 || mtx_owned(&Giant) || lock != NULL, ("sleeping without a lock")); KASSERT(p != NULL, ("msleep1")); KASSERT(ident != NULL && TD_IS_RUNNING(td), ("msleep")); @@ -199,13 +199,13 @@ sleepq_remove(td, td->td_wchan); if (ident == &pause_wchan) - flags = SLEEPQ_PAUSE; + sleepq_flags = SLEEPQ_PAUSE; else - flags = SLEEPQ_SLEEP; + sleepq_flags = SLEEPQ_SLEEP; if (catch) - flags |= SLEEPQ_INTERRUPTIBLE; + sleepq_flags |= SLEEPQ_INTERRUPTIBLE; if (priority & PBDRY) - flags |= SLEEPQ_STOP_ON_BDRY; + sleepq_flags |= SLEEPQ_STOP_ON_BDRY; sleepq_lock(ident); CTR5(KTR_PROC, "sleep: thread %ld (pid %ld, %s) on %s (%p)", @@ -231,18 +231,18 @@ * stopped, then td will no longer be on a sleep queue upon * return from cursig(). */ - sleepq_add(ident, lock, wmesg, flags, 0); - if (timo) - sleepq_set_timeout(ident, timo); + sleepq_add(ident, lock, wmesg, sleepq_flags, 0); + if (sbt != 0) + sleepq_set_timeout_sbt(ident, sbt, pr, flags); if (lock != NULL && class->lc_flags & LC_SLEEPABLE) { sleepq_release(ident); WITNESS_SAVE(lock, lock_witness); lock_state = class->lc_unlock(lock); sleepq_lock(ident); } - if (timo && catch) + if (sbt != 0 && catch) rval = sleepq_timedwait_sig(ident, pri); - else if (timo) + else if (sbt != 0) rval = sleepq_timedwait(ident, pri); else if (catch) rval = sleepq_wait_sig(ident, pri); @@ -263,7 +263,8 @@ } int -msleep_spin(void *ident, struct mtx *mtx, const char *wmesg, int timo) +msleep_spin_sbt(void *ident, struct mtx *mtx, const char *wmesg, + sbintime_t sbt, sbintime_t pr, int flags) { struct thread *td; struct proc *p; @@ -301,8 +302,8 @@ * We put ourselves on the sleep queue and start our timeout. */ sleepq_add(ident, &mtx->lock_object, wmesg, SLEEPQ_SLEEP, 0); - if (timo) - sleepq_set_timeout(ident, timo); + if (sbt != 0) + sleepq_set_timeout_sbt(ident, sbt, pr, flags); /* * Can't call ktrace with any spin locks held so it can lock the @@ -324,7 +325,7 @@ wmesg); sleepq_lock(ident); #endif - if (timo) + if (sbt != 0) rval = sleepq_timedwait(ident, 0); else { sleepq_wait(ident, 0); @@ -348,28 +349,30 @@ * to a "timo" value of one. */ int -pause(const char *wmesg, int timo) +pause_sbt(const char *wmesg, sbintime_t sbt, sbintime_t pr, int flags) { - KASSERT(timo >= 0, ("pause: timo must be >= 0")); + int sbt_sec; + sbt_sec = sbintime_getsec(sbt); + KASSERT(sbt_sec >= 0, ("pause: timo must be >= 0")); + /* silently convert invalid timeouts */ - if (timo < 1) - timo = 1; + if (sbt == 0) + sbt = tick_sbt; if (cold) { /* - * We delay one HZ at a time to avoid overflowing the + * We delay one second at a time to avoid overflowing the * system specific DELAY() function(s): */ - while (timo >= hz) { + while (sbt_sec > 0) { DELAY(1000000); - timo -= hz; + sbt_sec--; } - if (timo > 0) - DELAY(timo * tick); + DELAY((sbt & 0xffffffff) / SBT_1US); return (0); } - return (tsleep(&pause_wchan, 0, wmesg, timo)); + return (_sleep(&pause_wchan, NULL, 0, wmesg, sbt, pr, flags)); } /* @@ -560,8 +563,9 @@ * random variation to avoid synchronisation with processes that * run at regular intervals. */ - callout_reset(&loadav_callout, hz * 4 + (int)(random() % (hz * 2 + 1)), - loadav, NULL); + callout_reset_sbt(&loadav_callout, + tick_sbt * (hz * 4 + (int)(random() % (hz * 2 + 1))), 0, + loadav, NULL, C_DIRECT_EXEC | C_HARDCLOCK); } /* ARGSUSED */ Index: sys/kern/subr_sleepqueue.c =================================================================== --- sys/kern/subr_sleepqueue.c (.../head) (revision 246685) +++ sys/kern/subr_sleepqueue.c (.../projects/calloutng) (revision 246685) @@ -361,8 +361,10 @@ * sleep queue after timo ticks if the thread has not already been awakened. */ void -sleepq_set_timeout(void *wchan, int timo) +sleepq_set_timeout_sbt(void *wchan, sbintime_t sbt, sbintime_t pr, + int flags) { + struct sleepqueue_chain *sc; struct thread *td; @@ -372,7 +374,8 @@ MPASS(TD_ON_SLEEPQ(td)); MPASS(td->td_sleepqueue == NULL); MPASS(wchan != NULL); - callout_reset_curcpu(&td->td_slpcallout, timo, sleepq_timeout, td); + callout_reset_sbt_on(&td->td_slpcallout, sbt, pr, + sleepq_timeout, td, PCPU_GET(cpuid), flags | C_DIRECT_EXEC); } /* Index: sys/kern/sys_generic.c =================================================================== --- sys/kern/sys_generic.c (.../head) (revision 246685) +++ sys/kern/sys_generic.c (.../projects/calloutng) (revision 246685) @@ -102,7 +102,7 @@ off_t, int); static void doselwakeup(struct selinfo *, int); static void seltdinit(struct thread *); -static int seltdwait(struct thread *, int); +static int seltdwait(struct thread *, sbintime_t, sbintime_t); static void seltdclear(struct thread *); /* @@ -903,8 +903,9 @@ */ fd_mask s_selbits[howmany(2048, NFDBITS)]; fd_mask *ibits[3], *obits[3], *selbits, *sbp; - struct timeval atv, rtv, ttv; - int error, lf, ndu, timo; + sbintime_t asbt, precision, rsbt; + struct timeval rtv; + int error, lf, ndu; u_int nbufbytes, ncpbytes, ncpubytes, nfdbits; if (nd < 0) @@ -995,35 +996,29 @@ if (nbufbytes != 0) bzero(selbits, nbufbytes / 2); + precision = 0; if (tvp != NULL) { - atv = *tvp; - if (itimerfix(&atv)) { + rtv = *tvp; + if (rtv.tv_sec < 0 || rtv.tv_usec < 0 || + rtv.tv_usec >= 1000000) { error = EINVAL; goto done; } - getmicrouptime(&rtv); - timevaladd(&atv, &rtv); - } else { - atv.tv_sec = 0; - atv.tv_usec = 0; - } - timo = 0; + rsbt = timeval2sbintime(rtv); + precision = rsbt; + precision >>= tc_precexp; + if (TIMESEL(&asbt, rsbt)) + asbt += tc_tick_sbt; + asbt += rsbt; + } else + asbt = -1; seltdinit(td); /* Iterate until the timeout expires or descriptors become ready. */ for (;;) { error = selscan(td, ibits, obits, nd); if (error || td->td_retval[0] != 0) break; - if (atv.tv_sec || atv.tv_usec) { - getmicrouptime(&rtv); - if (timevalcmp(&rtv, &atv, >=)) - break; - ttv = atv; - timevalsub(&ttv, &rtv); - timo = ttv.tv_sec > 24 * 60 * 60 ? - 24 * 60 * 60 * hz : tvtohz(&ttv); - } - error = seltdwait(td, timo); + error = seltdwait(td, asbt, precision); if (error) break; error = selrescan(td, ibits, obits); @@ -1255,8 +1250,8 @@ { struct pollfd *bits; struct pollfd smallbits[32]; - struct timeval atv, rtv, ttv; - int error, timo; + sbintime_t asbt, precision, rsbt; + int error; u_int nfds; size_t ni; @@ -1271,36 +1266,27 @@ error = copyin(uap->fds, bits, ni); if (error) goto done; + precision = 0; if (uap->timeout != INFTIM) { - atv.tv_sec = uap->timeout / 1000; - atv.tv_usec = (uap->timeout % 1000) * 1000; - if (itimerfix(&atv)) { + if (uap->timeout < 0) { error = EINVAL; goto done; } - getmicrouptime(&rtv); - timevaladd(&atv, &rtv); - } else { - atv.tv_sec = 0; - atv.tv_usec = 0; - } - timo = 0; + rsbt = SBT_1MS * uap->timeout; + precision = rsbt; + precision >>= tc_precexp; + if (TIMESEL(&asbt, rsbt)) + asbt += tc_tick_sbt; + asbt += rsbt; + } else + asbt = -1; seltdinit(td); /* Iterate until the timeout expires or descriptors become ready. */ for (;;) { error = pollscan(td, bits, nfds); if (error || td->td_retval[0] != 0) break; - if (atv.tv_sec || atv.tv_usec) { - getmicrouptime(&rtv); - if (timevalcmp(&rtv, &atv, >=)) - break; - ttv = atv; - timevalsub(&ttv, &rtv); - timo = ttv.tv_sec > 24 * 60 * 60 ? - 24 * 60 * 60 * hz : tvtohz(&ttv); - } - error = seltdwait(td, timo); + error = seltdwait(td, asbt, precision); if (error) break; error = pollrescan(td); @@ -1642,7 +1628,7 @@ } static int -seltdwait(struct thread *td, int timo) +seltdwait(struct thread *td, sbintime_t sbt, sbintime_t precision) { struct seltd *stp; int error; @@ -1661,8 +1647,11 @@ mtx_unlock(&stp->st_mtx); return (0); } - if (timo > 0) - error = cv_timedwait_sig(&stp->st_wait, &stp->st_mtx, timo); + if (sbt == 0) + error = EWOULDBLOCK; + else if (sbt != -1) + error = cv_timedwait_sig_sbt(&stp->st_wait, &stp->st_mtx, + sbt, precision, C_ABSOLUTE); else error = cv_wait_sig(&stp->st_wait, &stp->st_mtx); mtx_unlock(&stp->st_mtx); Index: sys/kern/kern_clock.c =================================================================== --- sys/kern/kern_clock.c (.../head) (revision 246685) +++ sys/kern/kern_clock.c (.../projects/calloutng) (revision 246685) @@ -425,6 +425,7 @@ void hardclock_cpu(int usermode) { + struct bintime now; struct pstats *pstats; struct thread *td = curthread; struct proc *p = td->td_proc; @@ -459,7 +460,8 @@ if (td->td_intr_frame != NULL) PMC_SOFT_CALL_TF( , , clock, hard, td->td_intr_frame); #endif - callout_tick(); + binuptime(&now); + callout_process(&now); } /* @@ -549,7 +551,6 @@ if (td->td_intr_frame != NULL) PMC_SOFT_CALL_TF( , , clock, hard, td->td_intr_frame); #endif - callout_tick(); /* We are in charge to handle this tick duty. */ if (newticks > 0) { /* Dangerous and no need to call these things concurrently. */ Index: sys/kern/kern_condvar.c =================================================================== --- sys/kern/kern_condvar.c (.../head) (revision 246685) +++ sys/kern/kern_condvar.c (.../projects/calloutng) (revision 246685) @@ -270,12 +270,12 @@ } /* - * Wait on a condition variable for at most timo/hz seconds. Returns 0 if the - * process was resumed by cv_signal or cv_broadcast, EWOULDBLOCK if the timeout - * expires. + * Wait on a condition variable. Returns 0 if the process was resumed by + * cv_signal or cv_broadcast, EWOULDBLOCK if the timeout expires. */ int -_cv_timedwait(struct cv *cvp, struct lock_object *lock, int timo) +_cv_timedwait_sbt(struct cv *cvp, struct lock_object *lock, sbintime_t sbt, + sbintime_t pr, int flags) { WITNESS_SAVE_DECL(lock_witness); struct lock_class *class; @@ -311,7 +311,7 @@ DROP_GIANT(); sleepq_add(cvp, lock, cvp->cv_description, SLEEPQ_CONDVAR, 0); - sleepq_set_timeout(cvp, timo); + sleepq_set_timeout_sbt(cvp, sbt, pr, flags); if (lock != &Giant.lock_object) { if (class->lc_flags & LC_SLEEPABLE) sleepq_release(cvp); @@ -336,13 +336,14 @@ } /* - * Wait on a condition variable for at most timo/hz seconds, allowing - * interruption by signals. Returns 0 if the thread was resumed by cv_signal - * or cv_broadcast, EWOULDBLOCK if the timeout expires, and EINTR or ERESTART if - * a signal was caught. + * Wait on a condition variable allowing interruption by signals. + * Returns 0 if the thread was resumed by cv_signal or cv_broadcast, + * or cv_broadcast, EWOULDBLOCK if the timeout expires, and EINTR + * or ERESTART if a signal was caught. */ int -_cv_timedwait_sig(struct cv *cvp, struct lock_object *lock, int timo) +_cv_timedwait_sig_sbt(struct cv *cvp, struct lock_object *lock, + sbintime_t sbt, sbintime_t pr, int flags) { WITNESS_SAVE_DECL(lock_witness); struct lock_class *class; @@ -379,7 +380,7 @@ sleepq_add(cvp, lock, cvp->cv_description, SLEEPQ_CONDVAR | SLEEPQ_INTERRUPTIBLE, 0); - sleepq_set_timeout(cvp, timo); + sleepq_set_timeout_sbt(cvp, sbt, pr, flags); if (lock != &Giant.lock_object) { if (class->lc_flags & LC_SLEEPABLE) sleepq_release(cvp); Index: sys/kern/kern_time.c =================================================================== --- sys/kern/kern_time.c (.../head) (revision 246685) +++ sys/kern/kern_time.c (.../projects/calloutng) (revision 246685) @@ -43,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -481,38 +482,37 @@ int kern_nanosleep(struct thread *td, struct timespec *rqt, struct timespec *rmt) { - struct timespec ts, ts2, ts3; - struct timeval tv; + struct timespec ts; + sbintime_t sbt, sbtt, prec, tmp; int error; if (rqt->tv_nsec < 0 || rqt->tv_nsec >= 1000000000) return (EINVAL); if (rqt->tv_sec < 0 || (rqt->tv_sec == 0 && rqt->tv_nsec == 0)) return (0); - getnanouptime(&ts); - timespecadd(&ts, rqt); - TIMESPEC_TO_TIMEVAL(&tv, rqt); - for (;;) { - error = tsleep(&nanowait, PWAIT | PCATCH, "nanslp", - tvtohz(&tv)); - getnanouptime(&ts2); - if (error != EWOULDBLOCK) { - if (error == ERESTART) - error = EINTR; - if (rmt != NULL) { - timespecsub(&ts, &ts2); - if (ts.tv_sec < 0) - timespecclear(&ts); - *rmt = ts; - } - return (error); + tmp = timespec2sbintime(*rqt); + prec = tmp; + prec >>= tc_precexp; + if (TIMESEL(&sbt, tmp)) + sbt += tc_tick_sbt; + sbt += tmp; + error = tsleep_sbt(&nanowait, PWAIT | PCATCH, "nanslp", sbt, prec, + C_ABSOLUTE); + if (error != EWOULDBLOCK) { + if (error == ERESTART) + error = EINTR; + TIMESEL(&sbtt, tmp); + if (rmt != NULL) { + ts = sbintime2timespec(sbt - sbtt); + if (ts.tv_sec < 0) + timespecclear(&ts); + *rmt = ts; } - if (timespeccmp(&ts2, &ts, >=)) + if (sbtt >= sbt) return (0); - ts3 = ts; - timespecsub(&ts3, &ts2); - TIMESPEC_TO_TIMEVAL(&tv, &ts3); + return (error); } + return (0); } #ifndef _SYS_SYSPROTO_H_ Index: sys/kern/kern_tc.c =================================================================== --- sys/kern/kern_tc.c (.../head) (revision 246685) +++ sys/kern/kern_tc.c (.../projects/calloutng) (revision 246685) @@ -22,6 +22,7 @@ #include #include +#include #ifdef FFCLOCK #include #include @@ -119,6 +120,21 @@ SYSCTL_INT(_kern_timecounter, OID_AUTO, stepwarnings, CTLFLAG_RW, ×tepwarnings, 0, "Log time steps"); +struct bintime bt_timethreshold; +struct bintime bt_tickthreshold; +sbintime_t sbt_timethreshold; +sbintime_t sbt_tickthreshold; +struct bintime tc_tick_bt; +sbintime_t tc_tick_sbt; +int tc_precexp; +int tc_timepercentage = TC_DEFAULTPERC; +TUNABLE_INT("kern.timecounter.alloweddeviation", &tc_timepercentage); +static int sysctl_kern_timecounter_adjprecision(SYSCTL_HANDLER_ARGS); +SYSCTL_PROC(_kern_timecounter, OID_AUTO, alloweddeviation, + CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_MPSAFE, 0, 0, + sysctl_kern_timecounter_adjprecision, "I", + "Allowed time interval deviation in percents"); + static void tc_windup(void); static void cpu_tick_calibrate(int); @@ -334,6 +350,16 @@ } void +sbinuptime(sbintime_t *sbt) +{ + /* XXX: We need a real implementation, but tomorrow */ + struct bintime bt; + + binuptime(&bt); + *sbt = bintime2sbintime(bt); +} + +void nanouptime(struct timespec *tsp) { struct bintime bt; @@ -391,6 +417,16 @@ } void +getsbinuptime(sbintime_t *sbt) +{ + /* XXX: We need a real implementation, but tomorrow */ + struct bintime bt; + + getbinuptime(&bt); + *sbt = bintime2sbintime(bt); +} + +void getnanouptime(struct timespec *tsp) { struct timehands *th; @@ -883,6 +919,16 @@ } void +sbinuptime(sbintime_t sbt) +{ + /* XXX: We need a real implementation, but tomorrow */ + struct bintime bt; + + binuptime(&bt); + *sbt = bintime2sbintime(bt); +} + +void nanouptime(struct timespec *tsp) { @@ -925,6 +971,16 @@ } void +getsbinuptime(sbintime_t *sbt) +{ + /* XXX: We need a real implementation, but tomorrow */ + struct bintime bt; + + getbinuptime(&bt); + *sbt = bintime2sbintime(bt); +} + +void getnanouptime(struct timespec *tsp) { @@ -1705,10 +1761,47 @@ tc_windup(); } +static void __inline +tc_adjprecision(void) +{ + int t; + + if (tc_timepercentage > 0) { + t = (99 + tc_timepercentage) / tc_timepercentage; + tc_precexp = fls(t + (t >> 1)) - 1; + FREQ2BT(hz / tc_tick, &bt_timethreshold); + FREQ2BT(hz, &bt_tickthreshold); + bintime_shift(&bt_timethreshold, tc_precexp); + bintime_shift(&bt_tickthreshold, tc_precexp); + } else { + tc_precexp = 31; + bt_timethreshold.sec = INT_MAX; + bt_timethreshold.frac = ~(uint64_t)0; + bt_tickthreshold = bt_timethreshold; + } + sbt_timethreshold = bintime2sbintime(bt_timethreshold); + sbt_tickthreshold = bintime2sbintime(bt_tickthreshold); +} + +static int +sysctl_kern_timecounter_adjprecision(SYSCTL_HANDLER_ARGS) +{ + int error, val; + + val = tc_timepercentage; + error = sysctl_handle_int(oidp, &val, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + tc_timepercentage = val; + tc_adjprecision(); + return (0); +} + static void inittimecounter(void *dummy) { u_int p; + int tick_rate; /* * Set the initial timeout to @@ -1722,6 +1815,12 @@ tc_tick = (hz + 500) / 1000; else tc_tick = 1; + tc_adjprecision(); + FREQ2BT(hz, &tick_bt); + tick_sbt = bintime2sbintime(tick_bt); + tick_rate = hz / tc_tick; + FREQ2BT(tick_rate, &tc_tick_bt); + tc_tick_sbt = bintime2sbintime(tc_tick_bt); p = (tc_tick * 1000000) / hz; printf("Timecounters tick every %d.%03u msec\n", p / 1000, p % 1000); Index: sys/kern/subr_param.c =================================================================== --- sys/kern/subr_param.c (.../head) (revision 246685) +++ sys/kern/subr_param.c (.../projects/calloutng) (revision 246685) @@ -81,8 +81,10 @@ static int sysctl_kern_vm_guest(SYSCTL_HANDLER_ARGS); -int hz; -int tick; +int hz; /* system clock's frequency */ +int tick; /* usec per tick (1000000 / hz) */ +struct bintime tick_bt; /* bintime per tick (1s / hz) */ +sbintime_t tick_sbt; int maxusers; /* base tunable */ int maxproc; /* maximum # of processes */ int maxprocperuid; /* max # of procs per user */ @@ -221,6 +223,8 @@ if (hz == -1) hz = vm_guest > VM_GUEST_NO ? HZ_VM : HZ; tick = 1000000 / hz; + FREQ2BT(hz, &tick_bt); + tick_sbt = bintime2sbintime(tick_bt); #ifdef VM_SWZONE_SIZE_MAX maxswzone = VM_SWZONE_SIZE_MAX; Index: sys/kern/kern_event.c =================================================================== --- sys/kern/kern_event.c (.../head) (revision 246685) +++ sys/kern/kern_event.c (.../projects/calloutng) (revision 246685) @@ -517,39 +517,28 @@ * XXX: EVFILT_TIMER should perhaps live in kern_time.c beside the * interval timer support code. */ -static int -timertoticks(intptr_t data) +static __inline sbintime_t +timer2sbintime(intptr_t data) { - struct timeval tv; - int tticks; - tv.tv_sec = data / 1000; - tv.tv_usec = (data % 1000) * 1000; - tticks = tvtohz(&tv); - - return tticks; + return (SBT_1MS * data); } static void filt_timerexpire(void *knx) { - struct knote *kn = knx; struct callout *calloutp; + struct knote *kn; + kn = knx; kn->kn_data++; KNOTE_ACTIVATE(kn, 0); /* XXX - handle locking */ - /* - * timertoticks() uses tvtohz() which always adds 1 to allow - * for the time until the next clock interrupt being strictly - * less than 1 clock tick. We don't want that here since we - * want to appear to be in sync with the clock interrupt even - * when we're delayed. - */ if ((kn->kn_flags & EV_ONESHOT) != EV_ONESHOT) { calloutp = (struct callout *)kn->kn_hook; - callout_reset_curcpu(calloutp, timertoticks(kn->kn_sdata) - 1, - filt_timerexpire, kn); + callout_reset_sbt_on(calloutp, + timer2sbintime(kn->kn_sdata), 0 /* 1ms? */, + filt_timerexpire, kn, PCPU_GET(cpuid), 0); } } @@ -573,8 +562,9 @@ calloutp = malloc(sizeof(*calloutp), M_KQUEUE, M_WAITOK); callout_init(calloutp, CALLOUT_MPSAFE); kn->kn_hook = calloutp; - callout_reset_curcpu(calloutp, timertoticks(kn->kn_sdata), - filt_timerexpire, kn); + callout_reset_sbt_on(calloutp, + timer2sbintime(kn->kn_sdata), 0 /* 1ms? */, + filt_timerexpire, kn, PCPU_GET(cpuid), 0); return (0); } @@ -1319,9 +1309,9 @@ const struct timespec *tsp, struct kevent *keva, struct thread *td) { struct kevent *kevp; - struct timeval atv, rtv, ttv; + sbintime_t asbt, rsbt; struct knote *kn, *marker; - int count, timeout, nkev, error, influx; + int count, nkev, error, influx; int haskqglobal, touch; count = maxevents; @@ -1332,24 +1322,23 @@ if (maxevents == 0) goto done_nl; + rsbt = 0; if (tsp != NULL) { - TIMESPEC_TO_TIMEVAL(&atv, tsp); - if (itimerfix(&atv)) { + if (tsp->tv_sec < 0 || tsp->tv_nsec < 0 || + tsp->tv_nsec > 1000000000) { error = EINVAL; goto done_nl; } - if (tsp->tv_sec == 0 && tsp->tv_nsec == 0) - timeout = -1; - else - timeout = atv.tv_sec > 24 * 60 * 60 ? - 24 * 60 * 60 * hz : tvtohz(&atv); - getmicrouptime(&rtv); - timevaladd(&atv, &rtv); - } else { - atv.tv_sec = 0; - atv.tv_usec = 0; - timeout = 0; - } + if (timespecisset(tsp)) { + rsbt = timespec2sbintime(*tsp); + if (TIMESEL(&asbt, rsbt)) + asbt += tc_tick_sbt; + asbt += rsbt; + rsbt >>= tc_precexp; + } else + asbt = -1; + } else + asbt = 0; marker = knote_alloc(1); if (marker == NULL) { error = ENOMEM; @@ -1357,28 +1346,16 @@ } marker->kn_status = KN_MARKER; KQ_LOCK(kq); - goto start; retry: - if (atv.tv_sec || atv.tv_usec) { - getmicrouptime(&rtv); - if (timevalcmp(&rtv, &atv, >=)) - goto done; - ttv = atv; - timevalsub(&ttv, &rtv); - timeout = ttv.tv_sec > 24 * 60 * 60 ? - 24 * 60 * 60 * hz : tvtohz(&ttv); - } - -start: kevp = keva; if (kq->kq_count == 0) { - if (timeout < 0) { + if (asbt == -1) { error = EWOULDBLOCK; } else { kq->kq_state |= KQ_SLEEP; - error = msleep(kq, &kq->kq_lock, PSOCK | PCATCH, - "kqread", timeout); + error = msleep_sbt(kq, &kq->kq_lock, PSOCK | PCATCH, + "kqread", asbt, rsbt, C_ABSOLUTE); } if (error == 0) goto retry; Index: sys/kern/kern_timeout.c =================================================================== --- sys/kern/kern_timeout.c (.../head) (revision 246685) +++ sys/kern/kern_timeout.c (.../projects/calloutng) (revision 246685) @@ -37,13 +37,16 @@ #include __FBSDID("$FreeBSD$"); +#include "opt_callout_profiling.h" #include "opt_kdtrace.h" +#if defined(__arm__) +#include "opt_timer.h" +#endif #include #include #include #include -#include #include #include #include @@ -55,6 +58,7 @@ #include #include #include +#include #ifdef SMP #include @@ -68,6 +72,7 @@ SDT_PROBE_ARGTYPE(callout_execute, kernel, , callout_end, 0, "struct callout *"); +#ifdef CALLOUT_PROFILING static int avg_depth; SYSCTL_INT(_debug, OID_AUTO, to_avg_depth, CTLFLAG_RD, &avg_depth, 0, "Average number of items examined per softclock call. Units = 1/1000"); @@ -80,6 +85,19 @@ static int avg_mpcalls; SYSCTL_INT(_debug, OID_AUTO, to_avg_mpcalls, CTLFLAG_RD, &avg_mpcalls, 0, "Average number of MP callouts made per softclock call. Units = 1/1000"); +static int avg_depth_dir; +SYSCTL_INT(_debug, OID_AUTO, to_avg_depth_dir, CTLFLAG_RD, &avg_depth_dir, 0, + "Average number of direct callouts examined per callout_process call. " + "Units = 1/1000"); +static int avg_lockcalls_dir; +SYSCTL_INT(_debug, OID_AUTO, to_avg_lockcalls_dir, CTLFLAG_RD, + &avg_lockcalls_dir, 0, "Average number of lock direct callouts made per " + "callout_process call. Units = 1/1000"); +static int avg_mpcalls_dir; +SYSCTL_INT(_debug, OID_AUTO, to_avg_mpcalls_dir, CTLFLAG_RD, &avg_mpcalls_dir, + 0, "Average number of MP direct callouts made per callout_process call. " + "Units = 1/1000"); +#endif /* * TODO: * allocate more timeout table slots when table overflows. @@ -87,58 +105,62 @@ int callwheelsize, callwheelmask; /* - * The callout cpu migration entity represents informations necessary for - * describing the migrating callout to the new callout cpu. + * The callout cpu exec entities represent informations necessary for + * describing the state of callouts currently running on the CPU and the ones + * necessary for migrating callouts to the new callout cpu. In particular, + * the first entry of the array cc_exec_entity holds informations for callout + * running in SWI thread context, while the second one holds informations + * for callout running directly from hardware interrupt context. * The cached informations are very important for deferring migration when * the migrating callout is already running. */ -struct cc_mig_ent { +struct cc_exec { + struct callout *cc_next; + struct callout *cc_curr; #ifdef SMP - void (*ce_migration_func)(void *); - void *ce_migration_arg; - int ce_migration_cpu; - int ce_migration_ticks; + void (*ce_migration_func)(void *); + void *ce_migration_arg; + int ce_migration_cpu; + sbintime_t ce_migration_time; #endif + int cc_cancel; + int cc_waiting; }; - + /* - * There is one struct callout_cpu per cpu, holding all relevant + * There is one struct callou_cpu per cpu, holding all relevant * state for the callout processing thread on the individual CPU. - * In particular: - * cc_ticks is incremented once per tick in callout_cpu(). - * It tracks the global 'ticks' but in a way that the individual - * threads should not worry about races in the order in which - * hardclock() and hardclock_cpu() run on the various CPUs. - * cc_softclock is advanced in callout_cpu() to point to the - * first entry in cc_callwheel that may need handling. In turn, - * a softclock() is scheduled so it can serve the various entries i - * such that cc_softclock <= i <= cc_ticks . - * XXX maybe cc_softclock and cc_ticks should be volatile ? - * - * cc_ticks is also used in callout_reset_cpu() to determine - * when the callout should be served. */ struct callout_cpu { struct mtx_padalign cc_lock; - struct cc_mig_ent cc_migrating_entity; + struct cc_exec cc_exec_entity[2]; struct callout *cc_callout; struct callout_tailq *cc_callwheel; + struct callout_tailq cc_expireq; struct callout_list cc_callfree; - struct callout *cc_next; - struct callout *cc_curr; + sbintime_t cc_firstevent; + sbintime_t cc_lastscan; void *cc_cookie; - int cc_ticks; - int cc_softticks; - int cc_cancel; - int cc_waiting; - int cc_firsttick; }; +#define cc_exec_curr cc_exec_entity[0].cc_curr +#define cc_exec_next cc_exec_entity[0].cc_next +#define cc_exec_cancel cc_exec_entity[0].cc_cancel +#define cc_exec_waiting cc_exec_entity[0].cc_waiting +#define cc_exec_curr_dir cc_exec_entity[1].cc_curr +#define cc_exec_next_dir cc_exec_entity[1].cc_next +#define cc_exec_cancel_dir cc_exec_entity[1].cc_cancel +#define cc_exec_waiting_dir cc_exec_entity[1].cc_waiting + #ifdef SMP -#define cc_migration_func cc_migrating_entity.ce_migration_func -#define cc_migration_arg cc_migrating_entity.ce_migration_arg -#define cc_migration_cpu cc_migrating_entity.ce_migration_cpu -#define cc_migration_ticks cc_migrating_entity.ce_migration_ticks +#define cc_migration_func cc_exec_entity[0].ce_migration_func +#define cc_migration_arg cc_exec_entity[0].ce_migration_arg +#define cc_migration_cpu cc_exec_entity[0].ce_migration_cpu +#define cc_migration_time cc_exec_entity[0].ce_migration_time +#define cc_migration_func_dir cc_exec_entity[1].ce_migration_func +#define cc_migration_arg_dir cc_exec_entity[1].ce_migration_arg +#define cc_migration_cpu_dir cc_exec_entity[1].ce_migration_cpu +#define cc_migration_time_dir cc_exec_entity[1].ce_migration_time struct callout_cpu cc_cpu[MAXCPU]; #define CPUBLOCK MAXCPU @@ -153,40 +175,51 @@ #define CC_UNLOCK(cc) mtx_unlock_spin(&(cc)->cc_lock) #define CC_LOCK_ASSERT(cc) mtx_assert(&(cc)->cc_lock, MA_OWNED) +#define TIME_T_MAX \ + (sizeof(time_t) == (sizeof(int64_t)) ? INT64_MAX : INT32_MAX) + static int timeout_cpu; -void (*callout_new_inserted)(int cpu, int ticks) = NULL; +void (*callout_new_inserted)(int cpu, struct bintime bt, + struct bintime bt_opt) = NULL; +static void +softclock_call_cc(struct callout *c, struct callout_cpu *cc, int *mpcalls, + int *lockcalls, int *gcalls, int direct); static MALLOC_DEFINE(M_CALLOUT, "callout", "Callout datastructures"); /** * Locked by cc_lock: - * cc_curr - If a callout is in progress, it is curr_callout. - * If curr_callout is non-NULL, threads waiting in + * cc_curr - If a callout is in progress, it is cc_curr. + * If cc_curr is non-NULL, threads waiting in * callout_drain() will be woken up as soon as the * relevant callout completes. - * cc_cancel - Changing to 1 with both callout_lock and c_lock held + * cc_cancel - Changing to 1 with both callout_lock and cc_lock held * guarantees that the current callout will not run. * The softclock() function sets this to 0 before it * drops callout_lock to acquire c_lock, and it calls * the handler only if curr_cancelled is still 0 after - * c_lock is successfully acquired. + * cc_lock is successfully acquired. * cc_waiting - If a thread is waiting in callout_drain(), then * callout_wait is nonzero. Set only when - * curr_callout is non-NULL. + * cc_curr is non-NULL. */ /* * Resets the migration entity tied to a specific callout cpu. */ static void -cc_cme_cleanup(struct callout_cpu *cc) +cc_cme_cleanup(struct callout_cpu *cc, int direct) { + cc->cc_exec_entity[direct].cc_curr = NULL; + cc->cc_exec_entity[direct].cc_next = NULL; + cc->cc_exec_entity[direct].cc_cancel = 0; + cc->cc_exec_entity[direct].cc_waiting = 0; #ifdef SMP - cc->cc_migration_cpu = CPUBLOCK; - cc->cc_migration_ticks = 0; - cc->cc_migration_func = NULL; - cc->cc_migration_arg = NULL; + cc->cc_exec_entity[direct].ce_migration_cpu = CPUBLOCK; + cc->cc_exec_entity[direct].ce_migration_time = 0; + cc->cc_exec_entity[direct].ce_migration_func = NULL; + cc->cc_exec_entity[direct].ce_migration_arg = NULL; #endif } @@ -194,18 +227,18 @@ * Checks if migration is requested by a specific callout cpu. */ static int -cc_cme_migrating(struct callout_cpu *cc) +cc_cme_migrating(struct callout_cpu *cc, int direct) { #ifdef SMP - return (cc->cc_migration_cpu != CPUBLOCK); + return (cc->cc_exec_entity[direct].ce_migration_cpu != CPUBLOCK); #else return (0); #endif } /* - * kern_timeout_callwheel_alloc() - kernel low level callwheel initialization + * kern_timeout_callwheel_alloc() - kernel low level callwheel initialization * * This code is called very early in the kernel initialization sequence, * and may be called more then once. @@ -242,7 +275,9 @@ for (i = 0; i < callwheelsize; i++) { TAILQ_INIT(&cc->cc_callwheel[i]); } - cc_cme_cleanup(cc); + TAILQ_INIT(&cc->cc_expireq); + for (i = 0; i < 2; i++) + cc_cme_cleanup(cc, i); if (cc->cc_callout == NULL) return; for (i = 0; i < ncallout; i++) { @@ -330,28 +365,149 @@ SYSINIT(start_softclock, SI_SUB_SOFTINTR, SI_ORDER_FIRST, start_softclock, NULL); +#define CC_HASH_SHIFT 10 + +static inline int +callout_hash(sbintime_t sbt) +{ + + return (int)(sbt >> (32 - CC_HASH_SHIFT)); +} + +static inline int +callout_get_bucket(sbintime_t sbt) +{ + + return callout_hash(sbt) & callwheelmask; +} + void -callout_tick(void) +callout_process(struct bintime *now) { + struct callout *tmp, *tmpn; struct callout_cpu *cc; - int need_softclock; - int bucket; + struct callout_tailq *sc; + uint64_t lookahead; + sbintime_t first, last, max, now_sbt, tmp_max; + int depth_dir, firstb, lastb, mpcalls_dir, nowb, + lockcalls_dir, need_softclock, exit_allowed, exit_wanted; - /* - * Process callouts at a very low cpu priority, so we don't keep the - * relatively high clock interrupt priority any longer than necessary. - */ need_softclock = 0; + depth_dir = 0; + mpcalls_dir = 0; + lockcalls_dir = 0; cc = CC_SELF(); mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET); - cc->cc_firsttick = cc->cc_ticks = ticks; - for (; (cc->cc_softticks - cc->cc_ticks) <= 0; cc->cc_softticks++) { - bucket = cc->cc_softticks & callwheelmask; - if (!TAILQ_EMPTY(&cc->cc_callwheel[bucket])) { - need_softclock = 1; + + /* Compute the buckets of the last scan and present times. */ + firstb = callout_hash(cc->cc_lastscan); + now_sbt = bintime2sbintime(*now); + cc->cc_lastscan = now_sbt; + nowb = callout_hash(now_sbt); + + /* Compute the last bucket and minimum time of the bucket after it. */ + if (nowb == firstb) + lookahead = (SBT_1S / 16); + else if (nowb - firstb == 1) + lookahead = (SBT_1S / 8); + else + lookahead = (SBT_1S / 2); + first = last = now_sbt; + first += (lookahead / 2); + last += lookahead; + last &= (0xffffffffffffffffLLU << (32 - CC_HASH_SHIFT)); + lastb = callout_hash(last) - 1; + max = last; + + /* + * Check if we wrapped around the entire wheel from the last scan. + * In case, we need to scan entirely the wheel for pending callouts. + */ + if (lastb - firstb >= callwheelsize) + lastb = firstb - 1; + if (nowb - firstb >= callwheelsize) + nowb = firstb - 1; + nowb &= callwheelmask; + lastb &= callwheelmask; + firstb &= callwheelmask; + + /* Iterate callwheel from firstb to nowb and then up to lastb. */ + exit_allowed = exit_wanted = 0; + for (;;) { + sc = &cc->cc_callwheel[firstb]; + tmp = TAILQ_FIRST(sc); + while (tmp != NULL) { + /* Run the callout if present time within allowed. */ + if (tmp->c_time <= now_sbt) { + /* + * Consumer told us the callout may be run + * directly from hardware interrupt context. + */ + if (tmp->c_flags & CALLOUT_DIRECT) { + ++depth_dir; + cc->cc_exec_next_dir = + TAILQ_NEXT(tmp, c_links.tqe); + TAILQ_REMOVE(sc, tmp, c_links.tqe); + softclock_call_cc(tmp, cc, + &mpcalls_dir, &lockcalls_dir, + NULL, 1); + tmp = cc->cc_exec_next_dir; + } else { + tmpn = TAILQ_NEXT(tmp, c_links.tqe); + TAILQ_REMOVE(sc, tmp, c_links.tqe); + TAILQ_INSERT_TAIL(&cc->cc_expireq, + tmp, c_links.tqe); + tmp->c_flags |= CALLOUT_PROCESSED; + need_softclock = 1; + tmp = tmpn; + } + continue; + } + /* Skip events from distant future. */ + if (tmp->c_time >= max) + goto next; + /* + * Event minimal time is bigger than present maximal + * time, so it cannot be aggregated. + */ + if (tmp->c_time > last) { + exit_wanted = 1; + goto next; + } + /* Update first and last time, respecting this event. */ + if (tmp->c_time < first) + first = tmp->c_time; + tmp_max = tmp->c_time; + tmp_max += tmp->c_precision; + if (tmp_max < last) + last = tmp_max; +next: + tmp = TAILQ_NEXT(tmp, c_links.tqe); + } + /* Stop if we looked far enough into the future. */ + if (firstb == lastb) break; - } + /* + * Stop if we looked after present time and found + * some event we can't execute at now. + */ + if (firstb == nowb) + exit_allowed = 1; + if (exit_allowed && exit_wanted) + break; + /* Proceed with the next bucket. */ + firstb = (firstb + 1) & callwheelmask; } + cc->cc_exec_next_dir = NULL; + if (callout_new_inserted != NULL) + (*callout_new_inserted)(curcpu, sbintime2bintime(last), + sbintime2bintime(first)); + cc->cc_firstevent = last; +#ifdef CALLOUT_PROFILING + avg_depth_dir += (depth_dir * 1000 - avg_depth_dir) >> 8; + avg_mpcalls_dir += (mpcalls_dir * 1000 - avg_mpcalls_dir) >> 8; + avg_lockcalls_dir += (lockcalls_dir * 1000 - avg_lockcalls_dir) >> 8; +#endif mtx_unlock_spin_flags(&cc->cc_lock, MTX_QUIET); /* * swi_sched acquires the thread lock, so we don't want to call it @@ -361,33 +517,6 @@ swi_sched(cc->cc_cookie, 0); } -int -callout_tickstofirst(int limit) -{ - struct callout_cpu *cc; - struct callout *c; - struct callout_tailq *sc; - int curticks; - int skip = 1; - - cc = CC_SELF(); - mtx_lock_spin_flags(&cc->cc_lock, MTX_QUIET); - curticks = cc->cc_ticks; - while( skip < ncallout && skip < limit ) { - sc = &cc->cc_callwheel[ (curticks+skip) & callwheelmask ]; - /* search scanning ticks */ - TAILQ_FOREACH( c, sc, c_links.tqe ){ - if (c->c_time - curticks <= ncallout) - goto out; - } - skip++; - } -out: - cc->cc_firsttick = curticks + skip; - mtx_unlock_spin_flags(&cc->cc_lock, MTX_QUIET); - return (skip); -} - static struct callout_cpu * callout_lock(struct callout *c) { @@ -413,25 +542,39 @@ } static void -callout_cc_add(struct callout *c, struct callout_cpu *cc, int to_ticks, - void (*func)(void *), void *arg, int cpu) +callout_cc_add(struct callout *c, struct callout_cpu *cc, + sbintime_t sbt, sbintime_t precision, void (*func)(void *), + void *arg, int cpu, int flags) { + sbintime_t last; + int bucket; CC_LOCK_ASSERT(cc); - - if (to_ticks <= 0) - to_ticks = 1; + if (sbt < cc->cc_lastscan) + sbt = cc->cc_lastscan; c->c_arg = arg; c->c_flags |= (CALLOUT_ACTIVE | CALLOUT_PENDING); + if (flags & C_DIRECT_EXEC) + c->c_flags |= CALLOUT_DIRECT; + c->c_flags &= ~CALLOUT_PROCESSED; c->c_func = func; - c->c_time = ticks + to_ticks; - TAILQ_INSERT_TAIL(&cc->cc_callwheel[c->c_time & callwheelmask], - c, c_links.tqe); - if ((c->c_time - cc->cc_firsttick) < 0 && - callout_new_inserted != NULL) { - cc->cc_firsttick = c->c_time; - (*callout_new_inserted)(cpu, - to_ticks + (ticks - cc->cc_ticks)); + c->c_time = sbt; + c->c_precision = precision; + bucket = callout_get_bucket(c->c_time); + CTR3(KTR_CALLOUT, "precision set for %p: %d.%08x", + c, (int)(c->c_precision >> 32), + (u_int)(c->c_precision & 0xffffffff)); + TAILQ_INSERT_TAIL(&cc->cc_callwheel[bucket], c, c_links.tqe); + /* + * Inform the eventtimers(4) subsystem there's a new callout + * that has been inserted, but only if really required. + */ + last = c->c_time + c->c_precision; + if (callout_new_inserted != NULL && ((last < cc->cc_firstevent) || + (cc->cc_firstevent == 0))) { + cc->cc_firstevent = last; + (*callout_new_inserted)(cpu, sbintime2bintime(last), + sbintime2bintime(c->c_time)); } } @@ -447,7 +590,7 @@ static void softclock_call_cc(struct callout *c, struct callout_cpu *cc, int *mpcalls, - int *lockcalls, int *gcalls) + int *lockcalls, int *gcalls, int direct) { void (*c_func)(void *); void *c_arg; @@ -458,7 +601,8 @@ struct callout_cpu *new_cc; void (*new_func)(void *); void *new_arg; - int new_cpu, new_ticks; + int flags, new_cpu; + sbintime_t new_time; #endif #ifdef DIAGNOSTIC struct bintime bt1, bt2; @@ -480,8 +624,8 @@ c->c_flags = CALLOUT_LOCAL_ALLOC; else c->c_flags &= ~CALLOUT_PENDING; - cc->cc_curr = c; - cc->cc_cancel = 0; + cc->cc_exec_entity[direct].cc_curr = c; + cc->cc_exec_entity[direct].cc_cancel = 0; CC_UNLOCK(cc); if (c_lock != NULL) { class->lc_lock(c_lock, sharedlock); @@ -489,14 +633,18 @@ * The callout may have been cancelled * while we switched locks. */ - if (cc->cc_cancel) { + if (cc->cc_exec_entity[direct].cc_cancel) { class->lc_unlock(c_lock); goto skip; } /* The callout cannot be stopped now. */ - cc->cc_cancel = 1; - - if (c_lock == &Giant.lock_object) { + cc->cc_exec_entity[direct].cc_cancel = 1; + /* + * In case we're processing a direct callout we + * can't hold giant because holding a sleep mutex + * from hardware interrupt context is not allowed. + */ + if ((c_lock == &Giant.lock_object) && gcalls != NULL) { (*gcalls)++; CTR3(KTR_CALLOUT, "callout %p func %p arg %p", c, c_func, c_arg); @@ -513,11 +661,13 @@ #ifdef DIAGNOSTIC binuptime(&bt1); #endif - THREAD_NO_SLEEPING(); + if (!direct) + THREAD_NO_SLEEPING(); SDT_PROBE(callout_execute, kernel, , callout_start, c, 0, 0, 0, 0); c_func(c_arg); SDT_PROBE(callout_execute, kernel, , callout_end, c, 0, 0, 0, 0); - THREAD_SLEEPING_OK(); + if (!direct) + THREAD_SLEEPING_OK(); #ifdef DIAGNOSTIC binuptime(&bt2); bintime_sub(&bt2, &bt1); @@ -537,17 +687,17 @@ class->lc_unlock(c_lock); skip: CC_LOCK(cc); - KASSERT(cc->cc_curr == c, ("mishandled cc_curr")); - cc->cc_curr = NULL; - if (cc->cc_waiting) { + KASSERT(cc->cc_exec_entity[direct].cc_curr == c, ("mishandled cc_curr")); + cc->cc_exec_entity[direct].cc_curr = NULL; + if (cc->cc_exec_entity[direct].cc_waiting) { /* * There is someone waiting for the * callout to complete. * If the callout was scheduled for * migration just cancel it. */ - if (cc_cme_migrating(cc)) { - cc_cme_cleanup(cc); + if (cc_cme_migrating(cc, direct)) { + cc_cme_cleanup(cc, direct); /* * It should be assert here that the callout is not @@ -555,11 +705,11 @@ */ c->c_flags &= ~CALLOUT_DFRMIGRATION; } - cc->cc_waiting = 0; + cc->cc_exec_entity[direct].cc_waiting = 0; CC_UNLOCK(cc); - wakeup(&cc->cc_waiting); + wakeup(&cc->cc_exec_entity[direct].cc_waiting); CC_LOCK(cc); - } else if (cc_cme_migrating(cc)) { + } else if (cc_cme_migrating(cc, direct)) { KASSERT((c_flags & CALLOUT_LOCAL_ALLOC) == 0, ("Migrating legacy callout %p", c)); #ifdef SMP @@ -567,11 +717,11 @@ * If the callout was scheduled for * migration just perform it now. */ - new_cpu = cc->cc_migration_cpu; - new_ticks = cc->cc_migration_ticks; - new_func = cc->cc_migration_func; - new_arg = cc->cc_migration_arg; - cc_cme_cleanup(cc); + new_cpu = cc->cc_exec_entity[direct].ce_migration_cpu; + new_time = cc->cc_exec_entity[direct].ce_migration_time; + new_func = cc->cc_exec_entity[direct].ce_migration_func; + new_arg = cc->cc_exec_entity[direct].ce_migration_arg; + cc_cme_cleanup(cc, direct); /* * It should be assert here that the callout is not destroyed @@ -589,8 +739,9 @@ c->c_flags &= ~CALLOUT_DFRMIGRATION; new_cc = callout_cpu_switch(c, cc, new_cpu); - callout_cc_add(c, new_cc, new_ticks, new_func, new_arg, - new_cpu); + flags = (direct) ? C_DIRECT_EXEC : 0; + callout_cc_add(c, new_cc, new_time, c->c_precision, new_func, + new_arg, new_cpu, flags); CC_UNLOCK(new_cc); CC_LOCK(cc); #else @@ -613,7 +764,7 @@ } /* - * The callout mechanism is based on the work of Adam M. Costello and + * The callout mechanism is based on the work of Adam M. Costello and * George Varghese, published in a technical report entitled "Redesigning * the BSD Callout and Timer Facilities" and modified slightly for inclusion * in FreeBSD by Justin T. Gibbs. The original work on the data structures @@ -633,63 +784,25 @@ { struct callout_cpu *cc; struct callout *c; - struct callout_tailq *bucket; - int curticks; - int steps; /* #steps since we last allowed interrupts */ - int depth; - int mpcalls; - int lockcalls; - int gcalls; + int depth, gcalls, lockcalls, mpcalls; -#ifndef MAX_SOFTCLOCK_STEPS -#define MAX_SOFTCLOCK_STEPS 100 /* Maximum allowed value of steps. */ -#endif /* MAX_SOFTCLOCK_STEPS */ - + depth = 0; mpcalls = 0; lockcalls = 0; gcalls = 0; - depth = 0; - steps = 0; cc = (struct callout_cpu *)arg; CC_LOCK(cc); - while (cc->cc_softticks - 1 != cc->cc_ticks) { - /* - * cc_softticks may be modified by hard clock, so cache - * it while we work on a given bucket. - */ - curticks = cc->cc_softticks; - cc->cc_softticks++; - bucket = &cc->cc_callwheel[curticks & callwheelmask]; - c = TAILQ_FIRST(bucket); - while (c != NULL) { - depth++; - if (c->c_time != curticks) { - c = TAILQ_NEXT(c, c_links.tqe); - ++steps; - if (steps >= MAX_SOFTCLOCK_STEPS) { - cc->cc_next = c; - /* Give interrupts a chance. */ - CC_UNLOCK(cc); - ; /* nothing */ - CC_LOCK(cc); - c = cc->cc_next; - steps = 0; - } - } else { - cc->cc_next = TAILQ_NEXT(c, c_links.tqe); - TAILQ_REMOVE(bucket, c, c_links.tqe); - softclock_call_cc(c, cc, &mpcalls, - &lockcalls, &gcalls); - steps = 0; - c = cc->cc_next; - } - } + while ((c = TAILQ_FIRST(&cc->cc_expireq)) != NULL) { + TAILQ_REMOVE(&cc->cc_expireq, c, c_links.tqe); + softclock_call_cc(c, cc, &mpcalls, &lockcalls, &gcalls, 0); + ++depth; } +#ifdef CALLOUT_PROFILING avg_depth += (depth * 1000 - avg_depth) >> 8; avg_mpcalls += (mpcalls * 1000 - avg_mpcalls) >> 8; avg_lockcalls += (lockcalls * 1000 - avg_lockcalls) >> 8; avg_gcalls += (gcalls * 1000 - avg_gcalls) >> 8; - cc->cc_next = NULL; +#endif CC_UNLOCK(cc); } @@ -704,7 +817,7 @@ * Initialize a handle so that using it with untimeout is benign. * * See AT&T BCI Driver Reference Manual for specification. This - * implementation differs from that one in that although an + * implementation differs from that one in that although an * identification value is returned from timeout, the original * arguments to timeout as well as the identifier are used to * identify entries for untimeout. @@ -762,6 +875,10 @@ handle->callout = NULL; } +#ifndef NO_EVENTTIMERS +DPCPU_DECLARE(struct bintime, hardclocktime); +#endif + /* * New interface; clients allocate their own callout structures. * @@ -779,28 +896,68 @@ * callout_deactivate() - marks the callout as having been serviced */ int -callout_reset_on(struct callout *c, int to_ticks, void (*ftn)(void *), - void *arg, int cpu) +callout_reset_sbt_on(struct callout *c, sbintime_t sbt, sbintime_t precision, + void (*ftn)(void *), void *arg, int cpu, int flags) { + sbintime_t to_sbt, pr; + struct bintime to_bt; struct callout_cpu *cc; - int cancelled = 0; + int bucket, cancelled, direct; + cancelled = 0; + if (flags & C_ABSOLUTE) { + to_sbt = sbt; + } else { + if ((flags & C_HARDCLOCK) && (sbt < tick_sbt)) + sbt = tick_sbt; + if ((flags & C_HARDCLOCK) || +#ifdef NO_EVENTTIMERS + sbt >= sbt_timethreshold) { + getsbinuptime(&to_sbt); + /* Add safety belt for the case of hz > 1000. */ + to_sbt += (tc_tick_sbt - tick_sbt); +#else + sbt >= sbt_tickthreshold) { + /* + * Obtain the time of the last hardclock() call on + * this CPU directly from the kern_clocksource.c. + * This value is per-CPU, but it is equal for all + * active ones. + */ + spinlock_enter(); + to_bt = DPCPU_GET(hardclocktime); + to_sbt = bintime2sbintime(to_bt); + spinlock_exit(); +#endif + if ((flags & C_HARDCLOCK) == 0) + to_sbt += tick_sbt; + } else + sbinuptime(&to_sbt); + to_sbt += sbt; + pr = ((C_PRELGET(flags) < 0) ? sbt >> tc_precexp : + sbt >> C_PRELGET(flags)); + if (pr > precision) + precision = pr; + } /* * Don't allow migration of pre-allocated callouts lest they * become unbalanced. */ if (c->c_flags & CALLOUT_LOCAL_ALLOC) cpu = c->c_cpu; + direct = c->c_flags & CALLOUT_DIRECT; cc = callout_lock(c); - if (cc->cc_curr == c) { + if (cc->cc_exec_entity[direct].cc_curr == c) { /* * We're being asked to reschedule a callout which is * currently in progress. If there is a lock then we * can cancel the callout if it has not really started. */ - if (c->c_lock != NULL && !cc->cc_cancel) - cancelled = cc->cc_cancel = 1; - if (cc->cc_waiting) { + if (c->c_lock != NULL && + !cc->cc_exec_entity[direct].cc_cancel) + cancelled = + cc->cc_exec_entity[direct].cc_cancel = 1; + if (cc->cc_exec_entity[direct].cc_waiting) { /* * Someone has called callout_drain to kill this * callout. Don't reschedule. @@ -813,12 +970,15 @@ } } if (c->c_flags & CALLOUT_PENDING) { - if (cc->cc_next == c) { - cc->cc_next = TAILQ_NEXT(c, c_links.tqe); - } - TAILQ_REMOVE(&cc->cc_callwheel[c->c_time & callwheelmask], c, - c_links.tqe); - + if ((c->c_flags & CALLOUT_PROCESSED) == 0) { + if (cc->cc_exec_next_dir == c) + cc->cc_exec_next_dir = TAILQ_NEXT(c, + c_links.tqe); + bucket = callout_get_bucket(c->c_time); + TAILQ_REMOVE(&cc->cc_callwheel[bucket], c, + c_links.tqe); + } else + TAILQ_REMOVE(&cc->cc_expireq, c, c_links.tqe); cancelled = 1; c->c_flags &= ~(CALLOUT_ACTIVE | CALLOUT_PENDING); } @@ -830,15 +990,17 @@ * to a more appropriate moment. */ if (c->c_cpu != cpu) { - if (cc->cc_curr == c) { - cc->cc_migration_cpu = cpu; - cc->cc_migration_ticks = to_ticks; - cc->cc_migration_func = ftn; - cc->cc_migration_arg = arg; + if (cc->cc_exec_entity[direct].cc_curr == c) { + cc->cc_exec_entity[direct].ce_migration_cpu = cpu; + cc->cc_exec_entity[direct].ce_migration_time + = to_sbt; + cc->cc_exec_entity[direct].ce_migration_func = ftn; + cc->cc_exec_entity[direct].ce_migration_arg = arg; c->c_flags |= CALLOUT_DFRMIGRATION; - CTR5(KTR_CALLOUT, - "migration of %p func %p arg %p in %d to %u deferred", - c, c->c_func, c->c_arg, to_ticks, cpu); + CTR6(KTR_CALLOUT, + "migration of %p func %p arg %p in %d.%08x to %u deferred", + c, c->c_func, c->c_arg, (int)(to_sbt >> 32), + (u_int)(to_sbt & 0xffffffff), cpu); CC_UNLOCK(cc); return (cancelled); } @@ -846,9 +1008,10 @@ } #endif - callout_cc_add(c, cc, to_ticks, ftn, arg, cpu); - CTR5(KTR_CALLOUT, "%sscheduled %p func %p arg %p in %d", - cancelled ? "re" : "", c, c->c_func, c->c_arg, to_ticks); + callout_cc_add(c, cc, to_sbt, precision, ftn, arg, cpu, flags); + CTR6(KTR_CALLOUT, "%sscheduled %p func %p arg %p in %d.%08x", + cancelled ? "re" : "", c, c->c_func, c->c_arg,(int)(to_sbt >> 32), + (u_int)(to_sbt & 0xffffffff)); CC_UNLOCK(cc); return (cancelled); @@ -876,7 +1039,7 @@ { struct callout_cpu *cc, *old_cc; struct lock_class *class; - int use_lock, sq_locked; + int bucket, direct, sq_locked, use_lock; /* * Some old subsystems don't hold Giant while running a callout_stop(), @@ -892,7 +1055,7 @@ } } else use_lock = 0; - + direct = c->c_flags & CALLOUT_DIRECT; sq_locked = 0; old_cc = NULL; again: @@ -906,7 +1069,7 @@ if (sq_locked != 0 && cc != old_cc) { #ifdef SMP CC_UNLOCK(cc); - sleepq_release(&old_cc->cc_waiting); + sleepq_release(&old_cc->cc_exec_entity[direct].cc_waiting); sq_locked = 0; old_cc = NULL; goto again; @@ -927,12 +1090,13 @@ * If it wasn't on the queue and it isn't the current * callout, then we can't stop it, so just bail. */ - if (cc->cc_curr != c) { + if (cc->cc_exec_entity[direct].cc_curr != c) { CTR3(KTR_CALLOUT, "failed to stop %p func %p arg %p", c, c->c_func, c->c_arg); CC_UNLOCK(cc); if (sq_locked) - sleepq_release(&cc->cc_waiting); + sleepq_release( + &cc->cc_exec_entity[direct].cc_waiting); return (0); } @@ -943,8 +1107,7 @@ * just wait for the current invocation to * finish. */ - while (cc->cc_curr == c) { - + while (cc->cc_exec_entity[direct].cc_curr == c) { /* * Use direct calls to sleepqueue interface * instead of cv/msleep in order to avoid @@ -964,7 +1127,8 @@ */ if (!sq_locked) { CC_UNLOCK(cc); - sleepq_lock(&cc->cc_waiting); + sleepq_lock( + &cc->cc_exec_entity[direct].cc_waiting); sq_locked = 1; old_cc = cc; goto again; @@ -976,13 +1140,16 @@ * will be packed up, just let softclock() * take care of it. */ - cc->cc_waiting = 1; + cc->cc_exec_entity[direct].cc_waiting = 1; DROP_GIANT(); CC_UNLOCK(cc); - sleepq_add(&cc->cc_waiting, + sleepq_add( + &cc->cc_exec_entity[direct].cc_waiting, &cc->cc_lock.lock_object, "codrain", SLEEPQ_SLEEP, 0); - sleepq_wait(&cc->cc_waiting, 0); + sleepq_wait( + &cc->cc_exec_entity[direct].cc_waiting, + 0); sq_locked = 0; old_cc = NULL; @@ -990,7 +1157,8 @@ PICKUP_GIANT(); CC_LOCK(cc); } - } else if (use_lock && !cc->cc_cancel) { + } else if (use_lock && + !cc->cc_exec_entity[direct].cc_cancel) { /* * The current callout is waiting for its * lock which we hold. Cancel the callout @@ -998,10 +1166,10 @@ * lock, the callout will be skipped in * softclock(). */ - cc->cc_cancel = 1; + cc->cc_exec_entity[direct].cc_cancel = 1; CTR3(KTR_CALLOUT, "cancelled %p func %p arg %p", c, c->c_func, c->c_arg); - KASSERT(!cc_cme_migrating(cc), + KASSERT(!cc_cme_migrating(cc, direct), ("callout wrongly scheduled for migration")); CC_UNLOCK(cc); KASSERT(!sq_locked, ("sleepqueue chain locked")); @@ -1020,16 +1188,19 @@ return (0); } if (sq_locked) - sleepq_release(&cc->cc_waiting); - + sleepq_release(&cc->cc_exec_entity[direct].cc_waiting); c->c_flags &= ~(CALLOUT_ACTIVE | CALLOUT_PENDING); CTR3(KTR_CALLOUT, "cancelled %p func %p arg %p", c, c->c_func, c->c_arg); - if (cc->cc_next == c) - cc->cc_next = TAILQ_NEXT(c, c_links.tqe); - TAILQ_REMOVE(&cc->cc_callwheel[c->c_time & callwheelmask], c, - c_links.tqe); + if ((c->c_flags & CALLOUT_PROCESSED) == 0) { + if (cc->cc_exec_next_dir == c) + cc->cc_exec_next_dir = TAILQ_NEXT(c, c_links.tqe); + bucket = callout_get_bucket(c->c_time); + TAILQ_REMOVE(&cc->cc_callwheel[bucket], c, + c_links.tqe); + } else + TAILQ_REMOVE(&cc->cc_expireq, c, c_links.tqe); callout_cc_del(c, cc); CC_UNLOCK(cc); Index: sys/kern/subr_log.c =================================================================== --- sys/kern/subr_log.c (.../head) (revision 246685) +++ sys/kern/subr_log.c (.../projects/calloutng) (revision 246685) @@ -117,8 +117,8 @@ return (EBUSY); } log_open = 1; - callout_reset(&logsoftc.sc_callout, hz / log_wakeups_per_second, - logtimeout, NULL); + callout_reset_sbt(&logsoftc.sc_callout, + (SBT_1S / log_wakeups_per_second), 0, logtimeout, NULL, C_PREL(1)); mtx_unlock(&msgbuf_lock); fsetown(td->td_proc->p_pid, &logsoftc.sc_sigio); /* signal process only */ @@ -233,22 +233,21 @@ if (!log_open) return; - if (log_wakeups_per_second < 1) { - printf("syslog wakeup is less than one. Adjusting to 1.\n"); - log_wakeups_per_second = 1; - } - if (msgbuftrigger == 0) { - callout_schedule(&logsoftc.sc_callout, - hz / log_wakeups_per_second); - return; - } + if (msgbuftrigger == 0) + goto done; msgbuftrigger = 0; selwakeuppri(&logsoftc.sc_selp, LOG_RDPRI); KNOTE_LOCKED(&logsoftc.sc_selp.si_note, 0); if ((logsoftc.sc_state & LOG_ASYNC) && logsoftc.sc_sigio != NULL) pgsigio(&logsoftc.sc_sigio, SIGIO, 0); cv_broadcastpri(&log_wakeup, LOG_RDPRI); - callout_schedule(&logsoftc.sc_callout, hz / log_wakeups_per_second); +done: + if (log_wakeups_per_second < 1) { + printf("syslog wakeup is less than one. Adjusting to 1.\n"); + log_wakeups_per_second = 1; + } + callout_reset_sbt(&logsoftc.sc_callout, + (SBT_1S / log_wakeups_per_second), 0, logtimeout, NULL, C_PREL(1)); } /*ARGSUSED*/ Index: sys/kern/kern_intr.c =================================================================== --- sys/kern/kern_intr.c (.../head) (revision 246685) +++ sys/kern/kern_intr.c (.../projects/calloutng) (revision 246685) @@ -1103,7 +1103,6 @@ swi_add(struct intr_event **eventp, const char *name, driver_intr_t handler, void *arg, int pri, enum intr_type flags, void **cookiep) { - struct thread *td; struct intr_event *ie; int error; @@ -1125,15 +1124,7 @@ } error = intr_event_add_handler(ie, name, NULL, handler, arg, PI_SWI(pri), flags, cookiep); - if (error) - return (error); - if (pri == SWI_CLOCK) { - td = ie->ie_thread->it_thread; - thread_lock(td); - td->td_flags |= TDF_NOLOAD; - thread_unlock(td); - } - return (0); + return (error); } /* Index: sys/kern/kern_clocksource.c =================================================================== --- sys/kern/kern_clocksource.c (.../head) (revision 246685) +++ sys/kern/kern_clocksource.c (.../projects/calloutng) (revision 246685) @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -46,6 +47,7 @@ #include #include #include +#include #include #include @@ -71,9 +73,8 @@ static void getnextcpuevent(struct bintime *event, int idle); static void getnextevent(struct bintime *event); static int handleevents(struct bintime *now, int fake); -#ifdef SMP -static void cpu_new_callout(int cpu, int ticks); -#endif +static void cpu_new_callout(int cpu, struct bintime bt, + struct bintime bt_opt); static struct mtx et_hw_mtx; @@ -95,7 +96,6 @@ static struct eventtimer *timer = NULL; static struct bintime timerperiod; /* Timer period for periodic mode. */ -static struct bintime hardperiod; /* hardclock() events period. */ static struct bintime statperiod; /* statclock() events period. */ static struct bintime profperiod; /* profclock() events period. */ static struct bintime nexttick; /* Next global timer tick time. */ @@ -135,6 +135,8 @@ struct bintime nexthard; /* Next hardlock() event. */ struct bintime nextstat; /* Next statclock() event. */ struct bintime nextprof; /* Next profclock() event. */ + struct bintime nextcall; /* Next callout event. */ + struct bintime nextcallopt; /* Next optional callout event. */ #ifdef KDTRACE_HOOKS struct bintime nextcyc; /* Next OpenSolaris cyclics event. */ #endif @@ -143,16 +145,8 @@ }; static DPCPU_DEFINE(struct pcpu_state, timerstate); +DPCPU_DEFINE(struct bintime, hardclocktime); -#define FREQ2BT(freq, bt) \ -{ \ - (bt)->sec = 0; \ - (bt)->frac = ((uint64_t)0x8000000000000000 / (freq)) << 1; \ -} -#define BT2FREQ(bt) \ - (((uint64_t)0x8000000000000000 + ((bt)->frac >> 2)) / \ - ((bt)->frac >> 1)) - /* * Timer broadcast IPI handler. */ @@ -180,7 +174,7 @@ static int handleevents(struct bintime *now, int fake) { - struct bintime t; + struct bintime t, *hct; struct trapframe *frame; struct pcpu_state *state; uintfptr_t pc; @@ -205,10 +199,13 @@ runs = 0; while (bintime_cmp(now, &state->nexthard, >=)) { - bintime_addx(&state->nexthard, hardperiod.frac); + bintime_addx(&state->nexthard, tick_bt.frac); runs++; } if (runs) { + hct = DPCPU_PTR(hardclocktime); + *hct = state->nexthard; + bintime_sub(hct, &tick_bt); if ((timer->et_flags & ET_FLAGS_PERCPU) == 0 && bintime_cmp(&state->nexthard, &nexthard, >)) nexthard = state->nexthard; @@ -238,6 +235,12 @@ } } else state->nextprof = state->nextstat; + if (bintime_cmp(now, &state->nextcallopt, >=) && + (state->nextcallopt.sec != -1)) { + state->nextcall.sec = -1; + state->nextcallopt.sec = -1; + callout_process(now); + } #ifdef KDTRACE_HOOKS if (fake == 0 && cyclic_clock_func != NULL && @@ -269,24 +272,28 @@ static void getnextcpuevent(struct bintime *event, int idle) { + struct pcpu_state *state; struct bintime tmp; - struct pcpu_state *state; - int skip; + int hardfreq; state = DPCPU_PTR(timerstate); - /* Handle hardclock() events. */ + /* Handle hardclock() events, skipping some is CPU is idle. */ *event = state->nexthard; if (idle || (!activetick && !profiling && (timer->et_flags & ET_FLAGS_PERCPU) == 0)) { - skip = idle ? 4 : (stathz / 2); - if (curcpu == CPU_FIRST() && tc_min_ticktock_freq > skip) - skip = tc_min_ticktock_freq; - skip = callout_tickstofirst(hz / skip) - 1; - CTR2(KTR_SPARE2, "skip at %d: %d", curcpu, skip); - tmp = hardperiod; - bintime_mul(&tmp, skip); - bintime_add(event, &tmp); + hardfreq = 2; + if (curcpu == CPU_FIRST() && tc_min_ticktock_freq > hardfreq) + hardfreq = tc_min_ticktock_freq; + if (hz > hardfreq) { + tmp = tick_bt; + bintime_mul(&tmp, hz / hardfreq - 1); + bintime_add(event, &tmp); + } } + /* Handle callout events. */ + if (state->nextcall.sec != -1 && + bintime_cmp(event, &state->nextcall, >)) + *event = state->nextcall; if (!idle) { /* If CPU is active - handle other types of events. */ if (bintime_cmp(event, &state->nextstat, >)) *event = state->nextstat; @@ -629,10 +636,10 @@ #ifdef KDTRACE_HOOKS state->nextcyc.sec = -1; #endif + state->nextcall.sec = -1; + state->nextcallopt.sec = -1; } -#ifdef SMP callout_new_inserted = cpu_new_callout; -#endif periodic = want_periodic; /* Grab requested timer or the best of present. */ if (timername[0]) @@ -696,7 +703,7 @@ profhz = round_freq(timer, stathz * 64); } tick = 1000000 / hz; - FREQ2BT(hz, &hardperiod); + FREQ2BT(hz, &tick_bt); FREQ2BT(stathz, &statperiod); FREQ2BT(profhz, &profperiod); ET_LOCK(); @@ -761,7 +768,7 @@ /* * Switch to idle mode (all ticks handled). */ -void +sbintime_t cpu_idleclock(void) { struct bintime now, t; @@ -773,7 +780,7 @@ || curcpu == CPU_FIRST() #endif ) - return; + return (-1); state = DPCPU_PTR(timerstate); if (periodic) now = state->now; @@ -789,6 +796,8 @@ if (!periodic) loadtimer(&now, 0); ET_HW_UNLOCK(state); + bintime_sub(&t, &now); + return (MAX(bintime2sbintime(t), 0)); } /* @@ -856,52 +865,57 @@ } #endif -#ifdef SMP static void -cpu_new_callout(int cpu, int ticks) +cpu_new_callout(int cpu, struct bintime bt, struct bintime bt_opt) { - struct bintime tmp; + struct bintime now; struct pcpu_state *state; - CTR3(KTR_SPARE2, "new co at %d: on %d in %d", - curcpu, cpu, ticks); + CTR6(KTR_SPARE2, "new co at %d: on %d at %d.%08x - %d.%08x", + curcpu, cpu, (int)(bt_opt.sec), (u_int)(bt_opt.frac >> 32), + (int)(bt.sec), (u_int)(bt.frac >> 32)); state = DPCPU_ID_PTR(cpu, timerstate); ET_HW_LOCK(state); - if (state->idle == 0 || busy) { + + /* + * If there is callout time already set earlier -- do nothing. + * This check may appear redundant because we check already in + * callout_process() but this double check guarantees we're safe + * with respect to race conditions between interrupts execution + * and scheduling. + */ + state->nextcallopt = bt_opt; + if (state->nextcall.sec != -1 && + bintime_cmp(&bt, &state->nextcall, >=)) { ET_HW_UNLOCK(state); return; } - /* - * If timer is periodic - just update next event time for target CPU. - * If timer is global - there is chance it is already programmed. - */ - if (periodic || (timer->et_flags & ET_FLAGS_PERCPU) == 0) { - tmp = hardperiod; - bintime_mul(&tmp, ticks - 1); - bintime_add(&tmp, &state->nexthard); - if (bintime_cmp(&tmp, &state->nextevent, <)) - state->nextevent = tmp; - if (periodic || - bintime_cmp(&state->nextevent, &nexttick, >=)) { - ET_HW_UNLOCK(state); - return; - } + state->nextcall = bt; + /* If there is some some other event set earlier -- do nothing. */ + if (bintime_cmp(&state->nextcall, &state->nextevent, >=)) { + ET_HW_UNLOCK(state); + return; } - /* - * Otherwise we have to wake that CPU up, as we can't get present - * bintime to reprogram global timer from here. If timer is per-CPU, - * we by definition can't do it from here. - */ + state->nextevent = state->nextcall; + /* If timer is periodic -- there is nothing to reprogram. */ + if (periodic) { + ET_HW_UNLOCK(state); + return; + } + /* If timer is global or of the current CPU -- reprogram it. */ + if ((timer->et_flags & ET_FLAGS_PERCPU) == 0 || cpu == curcpu) { + binuptime(&now); + loadtimer(&now, 0); + ET_HW_UNLOCK(state); + return; + } + /* Otherwise make other CPU to reprogram it. */ + state->handle = 1; ET_HW_UNLOCK(state); - if (timer->et_flags & ET_FLAGS_PERCPU) { - state->handle = 1; - ipi_cpu(cpu, IPI_HARDCLOCK); - } else { - if (!cpu_idle_wakeup(cpu)) - ipi_cpu(cpu, IPI_AST); - } +#ifdef SMP + ipi_cpu(cpu, IPI_HARDCLOCK); +#endif } -#endif /* * Report or change the active event timers hardware. Index: sys/kern/kern_resource.c =================================================================== --- sys/kern/kern_resource.c (.../head) (revision 246685) +++ sys/kern/kern_resource.c (.../projects/calloutng) (revision 246685) @@ -645,7 +645,8 @@ } } if ((p->p_flag & P_WEXIT) == 0) - callout_reset(&p->p_limco, hz, lim_cb, p); + callout_reset_sbt(&p->p_limco, SBT_1S, 0, + lim_cb, p, C_PREL(1)); } int @@ -697,7 +698,8 @@ case RLIMIT_CPU: if (limp->rlim_cur != RLIM_INFINITY && p->p_cpulimit == RLIM_INFINITY) - callout_reset(&p->p_limco, hz, lim_cb, p); + callout_reset_sbt(&p->p_limco, SBT_1S, 0, + lim_cb, p, C_PREL(1)); p->p_cpulimit = limp->rlim_cur; break; case RLIMIT_DATA: @@ -1137,7 +1139,8 @@ p2->p_limit = lim_hold(p1->p_limit); callout_init_mtx(&p2->p_limco, &p2->p_mtx, 0); if (p1->p_cpulimit != RLIM_INFINITY) - callout_reset(&p2->p_limco, hz, lim_cb, p2); + callout_reset_sbt(&p2->p_limco, SBT_1S, 0, + lim_cb, p2, C_PREL(1) | C_HARDCLOCK); } void Index: sys/contrib/libfdt =================================================================== --- sys/contrib/libfdt (.../head) (revision 246685) +++ sys/contrib/libfdt (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/libfdt ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/libfdt:r236314-239017 Index: sys/contrib/octeon-sdk =================================================================== --- sys/contrib/octeon-sdk (.../head) (revision 246685) +++ sys/contrib/octeon-sdk (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/octeon-sdk ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/octeon-sdk:r236314-246070 Index: sys/contrib/dev/acpica/include =================================================================== --- sys/contrib/dev/acpica/include (.../head) (revision 246685) +++ sys/contrib/dev/acpica/include (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/include ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/include:r236314-246070 Index: sys/contrib/dev/acpica/components/debugger =================================================================== --- sys/contrib/dev/acpica/components/debugger (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/debugger (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/debugger ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/debugger:r236314-246070 Index: sys/contrib/dev/acpica/components/events =================================================================== --- sys/contrib/dev/acpica/components/events (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/events (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/events ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/events:r236314-246070 Index: sys/contrib/dev/acpica/components/executer =================================================================== --- sys/contrib/dev/acpica/components/executer (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/executer (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/executer ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/executer:r236314-246070 Index: sys/contrib/dev/acpica/components/dispatcher =================================================================== --- sys/contrib/dev/acpica/components/dispatcher (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/dispatcher (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/dispatcher ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/dispatcher:r236314-246070 Index: sys/contrib/dev/acpica/components/resources =================================================================== --- sys/contrib/dev/acpica/components/resources (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/resources (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/resources ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/resources:r236314-246070 Index: sys/contrib/dev/acpica/components/tables =================================================================== --- sys/contrib/dev/acpica/components/tables (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/tables (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/tables ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/tables:r236314-246070 Index: sys/contrib/dev/acpica/components/utilities =================================================================== --- sys/contrib/dev/acpica/components/utilities (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/utilities (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/utilities ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/utilities:r236314-246070 Index: sys/contrib/dev/acpica/components/namespace =================================================================== --- sys/contrib/dev/acpica/components/namespace (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/namespace (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/namespace ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/namespace:r236314-246070 Index: sys/contrib/dev/acpica/components/parser =================================================================== --- sys/contrib/dev/acpica/components/parser (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/parser (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/parser ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/parser:r236314-246070 Index: sys/contrib/dev/acpica/components/disassembler =================================================================== --- sys/contrib/dev/acpica/components/disassembler (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/disassembler (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/disassembler ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/disassembler:r236314-246070 Index: sys/contrib/dev/acpica/components/hardware =================================================================== --- sys/contrib/dev/acpica/components/hardware (.../head) (revision 246685) +++ sys/contrib/dev/acpica/components/hardware (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/components/hardware ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/components/hardware:r236314-246070 Index: sys/contrib/dev/acpica/changes.txt =================================================================== --- sys/contrib/dev/acpica/changes.txt (.../head) (revision 246685) +++ sys/contrib/dev/acpica/changes.txt (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/changes.txt ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/changes.txt:r236314-246070 Index: sys/contrib/dev/acpica/os_specific =================================================================== --- sys/contrib/dev/acpica/os_specific (.../head) (revision 246685) +++ sys/contrib/dev/acpica/os_specific (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/os_specific ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/os_specific:r236314-246070 Index: sys/contrib/dev/acpica/common =================================================================== --- sys/contrib/dev/acpica/common (.../head) (revision 246685) +++ sys/contrib/dev/acpica/common (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/common ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/common:r236314-246070 Index: sys/contrib/dev/acpica/compiler =================================================================== --- sys/contrib/dev/acpica/compiler (.../head) (revision 246685) +++ sys/contrib/dev/acpica/compiler (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica/compiler ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica/compiler:r236314-246070 Index: sys/contrib/dev/acpica =================================================================== --- sys/contrib/dev/acpica (.../head) (revision 246685) +++ sys/contrib/dev/acpica (.../projects/calloutng) (revision 246685) Property changes on: sys/contrib/dev/acpica ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/contrib/dev/acpica:r236314-246070 Index: sys/ia64/ia64/machdep.c =================================================================== --- sys/ia64/ia64/machdep.c (.../head) (revision 246685) +++ sys/ia64/ia64/machdep.c (.../projects/calloutng) (revision 246685) @@ -155,7 +155,7 @@ struct msgbuf *msgbufp = NULL; /* Other subsystems (e.g., ACPI) can hook this later. */ -void (*cpu_idle_hook)(void) = NULL; +void (*cpu_idle_hook)(sbintime_t) = NULL; struct kva_md_info kmi; @@ -392,10 +392,11 @@ cpu_idle(int busy) { register_t ie; + sbintime_t sbt = -1; if (!busy) { critical_enter(); - cpu_idleclock(); + sbt = cpu_idleclock(); } ie = intr_disable(); @@ -404,7 +405,7 @@ if (sched_runnable()) ia64_enable_intr(); else if (cpu_idle_hook != NULL) { - (*cpu_idle_hook)(); + (*cpu_idle_hook)(sbt); /* The hook must enable interrupts! */ } else { ia64_call_pal_static(PAL_HALT_LIGHT, 0, 0, 0); Index: sys/boot/powerpc/ofw =================================================================== --- sys/boot/powerpc/ofw (.../head) (revision 246685) +++ sys/boot/powerpc/ofw (.../projects/calloutng) (revision 246685) Property changes on: sys/boot/powerpc/ofw ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/boot/powerpc/ofw:r236314-243181 Index: sys/boot/powerpc/boot1.chrp =================================================================== --- sys/boot/powerpc/boot1.chrp (.../head) (revision 246685) +++ sys/boot/powerpc/boot1.chrp (.../projects/calloutng) (revision 246685) Property changes on: sys/boot/powerpc/boot1.chrp ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/boot/powerpc/boot1.chrp:r236314-243900 Index: sys/boot/i386/efi =================================================================== --- sys/boot/i386/efi (.../head) (revision 246685) +++ sys/boot/i386/efi (.../projects/calloutng) (revision 246685) Property changes on: sys/boot/i386/efi ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/boot/i386/efi:r236314-246070 Index: sys/boot =================================================================== --- sys/boot (.../head) (revision 246685) +++ sys/boot (.../projects/calloutng) (revision 246685) Property changes on: sys/boot ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/boot:r236314-246684 Index: sys/amd64/amd64/machdep.c =================================================================== --- sys/amd64/amd64/machdep.c (.../head) (revision 246685) +++ sys/amd64/amd64/machdep.c (.../projects/calloutng) (revision 246685) @@ -658,7 +658,7 @@ halt(); } -void (*cpu_idle_hook)(void) = NULL; /* ACPI idle hook. */ +void (*cpu_idle_hook)(sbintime_t) = NULL; /* ACPI idle hook. */ static int cpu_ident_amdc1e = 0; /* AMD C1E supported. */ static int idle_mwait = 1; /* Use MONITOR/MWAIT for short idle. */ TUNABLE_INT("machdep.idle_mwait", &idle_mwait); @@ -670,7 +670,7 @@ #define STATE_SLEEPING 0x2 static void -cpu_idle_acpi(int busy) +cpu_idle_acpi(sbintime_t sbt) { int *state; @@ -682,14 +682,14 @@ if (sched_runnable()) enable_intr(); else if (cpu_idle_hook) - cpu_idle_hook(); + cpu_idle_hook(sbt); else __asm __volatile("sti; hlt"); *state = STATE_RUNNING; } static void -cpu_idle_hlt(int busy) +cpu_idle_hlt(sbintime_t sbt) { int *state; @@ -730,7 +730,7 @@ #define MWAIT_C4 0x30 static void -cpu_idle_mwait(int busy) +cpu_idle_mwait(sbintime_t sbt) { int *state; @@ -753,7 +753,7 @@ } static void -cpu_idle_spin(int busy) +cpu_idle_spin(sbintime_t sbt) { int *state; int i; @@ -802,12 +802,13 @@ } } -void (*cpu_idle_fn)(int) = cpu_idle_acpi; +void (*cpu_idle_fn)(sbintime_t) = cpu_idle_acpi; void cpu_idle(int busy) { uint64_t msr; + sbintime_t sbt = -1; CTR2(KTR_SPARE2, "cpu_idle(%d) at %d", busy, curcpu); @@ -825,7 +826,7 @@ /* If we have time - switch timers into idle mode. */ if (!busy) { critical_enter(); - cpu_idleclock(); + sbt = cpu_idleclock(); } /* Apply AMD APIC timer C1E workaround. */ @@ -836,7 +837,7 @@ } /* Call main idle method. */ - cpu_idle_fn(busy); + cpu_idle_fn(sbt); /* Switch timers mack into active mode. */ if (!busy) { Index: sys/amd64/vmm =================================================================== --- sys/amd64/vmm (.../head) (revision 246685) +++ sys/amd64/vmm (.../projects/calloutng) (revision 246685) Property changes on: sys/amd64/vmm ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/amd64/vmm:r245652-246684 Index: sys/amd64/include/vmm_dev.h =================================================================== --- sys/amd64/include/vmm_dev.h (.../head) (revision 246685) +++ sys/amd64/include/vmm_dev.h (.../projects/calloutng) (revision 246685) Property changes on: sys/amd64/include/vmm_dev.h ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/amd64/include/vmm_dev.h:r245652-246070 Index: sys/amd64/include/vmm.h =================================================================== --- sys/amd64/include/vmm.h (.../head) (revision 246685) +++ sys/amd64/include/vmm.h (.../projects/calloutng) (revision 246685) Property changes on: sys/amd64/include/vmm.h ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/amd64/include/vmm.h:r245652-246070 Index: sys/amd64/include/vmm_instruction_emul.h =================================================================== --- sys/amd64/include/vmm_instruction_emul.h (.../head) (revision 246685) +++ sys/amd64/include/vmm_instruction_emul.h (.../projects/calloutng) (revision 246685) Property changes on: sys/amd64/include/vmm_instruction_emul.h ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/amd64/include/vmm_instruction_emul.h:r245652-246684 Index: sys/ofed/include/linux/timer.h =================================================================== --- sys/ofed/include/linux/timer.h (.../head) (revision 246685) +++ sys/ofed/include/linux/timer.h (.../projects/calloutng) (revision 246685) @@ -38,10 +38,9 @@ struct callout timer_callout; void (*function)(unsigned long); unsigned long data; + int expires; }; -#define expires timer_callout.c_time - static inline void _timer_fn(void *context) { @@ -65,13 +64,16 @@ callout_init(&(timer)->timer_callout, CALLOUT_MPSAFE); \ } while (0) -#define mod_timer(timer, expire) \ - callout_reset(&(timer)->timer_callout, (expire) - jiffies, \ - _timer_fn, (timer)) +#define mod_timer(timer, exp) \ +do { \ + (timer)->expires = exp; \ + callout_reset(&(timer)->timer_callout, (exp) - jiffies, \ + _timer_fn, (timer)); \ +} while (0) #define add_timer(timer) \ callout_reset(&(timer)->timer_callout, \ - (timer)->timer_callout.c_time - jiffies, _timer_fn, (timer)) + (timer)->expires - jiffies, _timer_fn, (timer)) #define del_timer(timer) callout_stop(&(timer)->timer_callout) #define del_timer_sync(timer) callout_drain(&(timer)->timer_callout) Index: sys/sys/rwlock.h =================================================================== --- sys/sys/rwlock.h (.../head) (revision 246685) +++ sys/sys/rwlock.h (.../projects/calloutng) (revision 246685) @@ -211,7 +211,8 @@ rw_runlock(rw); \ } while (0) #define rw_sleep(chan, rw, pri, wmesg, timo) \ - _sleep((chan), &(rw)->lock_object, (pri), (wmesg), (timo)) + _sleep((chan), &(rw)->lock_object, (pri), (wmesg), \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) #define rw_initialized(rw) lock_initalized(&(rw)->lock_object) Index: sys/sys/sx.h =================================================================== --- sys/sys/sx.h (.../head) (revision 246685) +++ sys/sys/sx.h (.../projects/calloutng) (revision 246685) @@ -275,7 +275,8 @@ #define sx_unlock(sx) sx_unlock_((sx), LOCK_FILE, LOCK_LINE) #define sx_sleep(chan, sx, pri, wmesg, timo) \ - _sleep((chan), &(sx)->lock_object, (pri), (wmesg), (timo)) + _sleep((chan), &(sx)->lock_object, (pri), (wmesg), \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) /* * Options passed to sx_init_flags(). Index: sys/sys/callout.h =================================================================== --- sys/sys/callout.h (.../head) (revision 246685) +++ sys/sys/callout.h (.../projects/calloutng) (revision 246685) @@ -47,7 +47,17 @@ #define CALLOUT_RETURNUNLOCKED 0x0010 /* handler returns with mtx unlocked */ #define CALLOUT_SHAREDLOCK 0x0020 /* callout lock held in shared mode */ #define CALLOUT_DFRMIGRATION 0x0040 /* callout in deferred migration mode */ +#define CALLOUT_PROCESSED 0x0080 /* callout in wheel or processing list? */ +#define CALLOUT_DIRECT 0x0100 /* allow exec from hw int context */ +#define C_DIRECT_EXEC 0x0001 /* direct execution of callout */ +#define C_PRELBITS 7 +#define C_PRELRANGE ((1 << C_PRELBITS) - 1) +#define C_PREL(x) (((x) + 1) << 1) +#define C_PRELGET(x) (int)((((x) >> 1) & C_PRELRANGE) - 1) +#define C_HARDCLOCK 0x0100 /* align to hardclock() calls */ +#define C_ABSOLUTE 0x0200 /* event time is absolute. */ + struct callout_handle { struct callout *callout; }; @@ -67,7 +77,15 @@ _callout_init_lock((c), ((rw) != NULL) ? &(rw)->lock_object : \ NULL, (flags)) #define callout_pending(c) ((c)->c_flags & CALLOUT_PENDING) -int callout_reset_on(struct callout *, int, void (*)(void *), void *, int); +int callout_reset_sbt_on(struct callout *, sbintime_t, sbintime_t, + void (*)(void *), void *, int, int); +#define callout_reset_sbt(c, sbt, pr, fn, arg, flags) \ + callout_reset_sbt_on((c), (sbt), (pr), (fn), (arg), (c)->c_cpu, flags) +#define callout_reset_sbt_curcpu(c, sbt, pr, fn, arg, flags) \ + callout_reset_sbt_on((c), (sbt), (pr), (fn), (arg), PCPU_GET(cpuid), flags) +#define callout_reset_on(c, to_ticks, fn, arg, cpu) \ + callout_reset_sbt_on((c), (tick_sbt * (to_ticks)), 0, (fn), (arg), \ + (cpu), C_HARDCLOCK) #define callout_reset(c, on_tick, fn, arg) \ callout_reset_on((c), (on_tick), (fn), (arg), (c)->c_cpu) #define callout_reset_curcpu(c, on_tick, fn, arg) \ @@ -78,9 +96,9 @@ callout_schedule_on((c), (on_tick), PCPU_GET(cpuid)) #define callout_stop(c) _callout_stop_safe(c, 0) int _callout_stop_safe(struct callout *, int); -void callout_tick(void); -int callout_tickstofirst(int limit); -extern void (*callout_new_inserted)(int cpu, int ticks); +void callout_process(struct bintime *); +extern void (*callout_new_inserted)(int cpu, struct bintime bt, + struct bintime); #endif Index: sys/sys/_callout.h =================================================================== --- sys/sys/_callout.h (.../head) (revision 246685) +++ sys/sys/_callout.h (.../projects/calloutng) (revision 246685) @@ -39,6 +39,7 @@ #define _SYS__CALLOUT_H #include +#include struct lock_object; @@ -50,7 +51,8 @@ SLIST_ENTRY(callout) sle; TAILQ_ENTRY(callout) tqe; } c_links; - int c_time; /* ticks to the event */ + sbintime_t c_time; /* ticks to the event */ + sbintime_t c_precision; /* delta allowed wrt opt */ void *c_arg; /* function argument */ void (*c_func)(void *); /* function to call */ struct lock_object *c_lock; /* lock to handle */ Index: sys/sys/condvar.h =================================================================== --- sys/sys/condvar.h (.../head) (revision 246685) +++ sys/sys/condvar.h (.../projects/calloutng) (revision 246685) @@ -55,8 +55,10 @@ void _cv_wait(struct cv *cvp, struct lock_object *lock); void _cv_wait_unlock(struct cv *cvp, struct lock_object *lock); int _cv_wait_sig(struct cv *cvp, struct lock_object *lock); -int _cv_timedwait(struct cv *cvp, struct lock_object *lock, int timo); -int _cv_timedwait_sig(struct cv *cvp, struct lock_object *lock, int timo); +int _cv_timedwait_sbt(struct cv *cvp, struct lock_object *lock, + sbintime_t sbt, sbintime_t pr, int flags); +int _cv_timedwait_sig_sbt(struct cv *cvp, struct lock_object *lock, + sbintime_t sbt, sbintime_t pr, int flags); void cv_signal(struct cv *cvp); void cv_broadcastpri(struct cv *cvp, int pri); @@ -68,9 +70,15 @@ #define cv_wait_sig(cvp, lock) \ _cv_wait_sig((cvp), &(lock)->lock_object) #define cv_timedwait(cvp, lock, timo) \ - _cv_timedwait((cvp), &(lock)->lock_object, (timo)) + _cv_timedwait_sbt((cvp), &(lock)->lock_object, \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) +#define cv_timedwait_sbt(cvp, lock, sbt, pr, flags) \ + _cv_timedwait_sbt((cvp), &(lock)->lock_object, (sbt), (pr), (flags)) #define cv_timedwait_sig(cvp, lock, timo) \ - _cv_timedwait_sig((cvp), &(lock)->lock_object, (timo)) + _cv_timedwait_sig_sbt((cvp), &(lock)->lock_object, \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) +#define cv_timedwait_sig_sbt(cvp, lock, sbt, pr, flags) \ + _cv_timedwait_sig_sbt((cvp), &(lock)->lock_object, (sbt), (pr), (flags)) #define cv_broadcast(cvp) cv_broadcastpri(cvp, 0) Index: sys/sys/sleepqueue.h =================================================================== --- sys/sys/sleepqueue.h (.../head) (revision 246685) +++ sys/sys/sleepqueue.h (.../projects/calloutng) (revision 246685) @@ -108,7 +108,10 @@ void sleepq_release(void *wchan); void sleepq_remove(struct thread *td, void *wchan); int sleepq_signal(void *wchan, int flags, int pri, int queue); -void sleepq_set_timeout(void *wchan, int timo); +void sleepq_set_timeout_sbt(void *wchan, sbintime_t sbt, + sbintime_t pr, int flags); +#define sleepq_set_timeout(wchan, timo) \ + sleepq_set_timeout_sbt((wchan), (tick_sbt * (timo)), 0, C_HARDCLOCK) u_int sleepq_sleepcnt(void *wchan, int queue); int sleepq_timedwait(void *wchan, int pri); int sleepq_timedwait_sig(void *wchan, int pri); Index: sys/sys/time.h =================================================================== --- sys/sys/time.h (.../head) (revision 246685) +++ sys/sys/time.h (.../projects/calloutng) (revision 246685) @@ -102,6 +102,21 @@ bt->frac = (p2 << 32) | (p1 & 0xffffffffull); } +static __inline void +bintime_shift(struct bintime *bt, int exp) +{ + + if (exp > 0) { + bt->sec <<= exp; + bt->sec |= bt->frac >> (64 - exp); + bt->frac <<= exp; + } else if (exp < 0) { + bt->frac >>= -exp; + bt->frac |= (uint64_t)bt->sec << (64 + exp); + bt->sec >>= -exp; + } +} + #define bintime_clear(a) ((a)->sec = (a)->frac = 0) #define bintime_isset(a) ((a)->sec || (a)->frac) #define bintime_cmp(a, b, cmp) \ @@ -109,6 +124,45 @@ ((a)->frac cmp (b)->frac) : \ ((a)->sec cmp (b)->sec)) +typedef int64_t sbintime_t; +#define SBT_1S ((sbintime_t)1 << 32) +#define SBT_1M (SBT_1S * 60) +#define SBT_1MS (SBT_1S / 1000) +#define SBT_1US (SBT_1S / 1000000) +#define SBT_1NS (SBT_1S / 1000000000) + +static __inline int +sbintime_getsec(sbintime_t sbt) +{ + + return (int)(sbt >> 32); +} + +static __inline sbintime_t +bintime2sbintime(const struct bintime bt) +{ + + return (((sbintime_t)bt.sec << 32) + (bt.frac >> 32)); +} + +static __inline struct bintime +sbintime2bintime(sbintime_t sbt) +{ + struct bintime bt; + + bt.sec = sbt >> 32; + bt.frac = sbt << 32; + return (bt); + +} + +#ifdef _KERNEL + +extern struct bintime tick_bt; +extern sbintime_t tick_sbt; + +#endif /* KERNEL */ + /*- * Background information: * @@ -156,6 +210,43 @@ /* 18446744073709 = int(2^64 / 1000000) */ bt->frac = tv->tv_usec * (uint64_t)18446744073709LL; } + +static __inline struct timespec +sbintime2timespec(sbintime_t sbt) +{ + struct timespec ts; + + ts.tv_sec = sbt >> 32; + ts.tv_nsec = ((uint64_t)1000000000 * (uint32_t)sbt) >> 32; + return (ts); +} + +static __inline sbintime_t +timespec2sbintime(struct timespec ts) +{ + + return (((sbintime_t)ts.tv_sec << 32) + + (ts.tv_nsec * (((uint64_t)1 << 63) / 500000000) >> 32)); +} + +static __inline struct timeval +sbintime2timeval(sbintime_t sbt) +{ + struct timeval tv; + + tv.tv_sec = sbt >> 32; + tv.tv_usec = ((uint64_t)1000000 * (uint32_t)sbt) >> 32; + return (tv); +} + +static __inline sbintime_t +timeval2sbintime(struct timeval tv) +{ + + return (((sbintime_t)tv.tv_sec << 32) + + (tv.tv_usec * (((uint64_t)1 << 63) / 500000) >> 32)); +} + #endif /* __BSD_VISIBLE */ #ifdef _KERNEL @@ -290,7 +381,15 @@ extern volatile time_t time_second; extern volatile time_t time_uptime; extern struct bintime boottimebin; +extern struct bintime tc_tick_bt; +extern sbintime_t tc_tick_sbt; extern struct timeval boottime; +extern int tc_precexp; +extern int tc_timepercentage; +extern struct bintime bt_timethreshold; +extern struct bintime bt_tickthreshold; +extern sbintime_t sbt_timethreshold; +extern sbintime_t sbt_tickthreshold; /* * Functions for looking at our clock: [get]{bin,nano,micro}[up]time() @@ -314,6 +413,7 @@ */ void binuptime(struct bintime *bt); +void sbinuptime(sbintime_t *sbt); void nanouptime(struct timespec *tsp); void microuptime(struct timeval *tvp); @@ -322,6 +422,7 @@ void microtime(struct timeval *tvp); void getbinuptime(struct bintime *bt); +void getsbinuptime(sbintime_t *sbt); void getnanouptime(struct timespec *tsp); void getmicrouptime(struct timeval *tvp); @@ -337,6 +438,23 @@ void timevaladd(struct timeval *t1, const struct timeval *t2); void timevalsub(struct timeval *t1, const struct timeval *t2); int tvtohz(struct timeval *tv); + +#define TC_DEFAULTPERC 5 + +#define BT2FREQ(bt) \ + (((uint64_t)0x8000000000000000 + ((bt)->frac >> 2)) / \ + ((bt)->frac >> 1)) + +#define FREQ2BT(freq, bt) \ +{ \ + (bt)->sec = 0; \ + (bt)->frac = ((uint64_t)0x8000000000000000 / (freq)) << 1; \ +} + +#define TIMESEL(sbt, sbt2) \ + (((sbt2) >= sbt_timethreshold) ? \ + (getsbinuptime(sbt), 1) : (sbinuptime(sbt), 0)) + #else /* !_KERNEL */ #include Index: sys/sys/proc.h =================================================================== --- sys/sys/proc.h (.../head) (revision 246685) +++ sys/sys/proc.h (.../projects/calloutng) (revision 246685) @@ -908,7 +908,7 @@ void tidhash_remove(struct thread *); void cpu_idle(int); int cpu_idle_wakeup(int); -extern void (*cpu_idle_hook)(void); /* Hook to machdep CPU idler. */ +extern void (*cpu_idle_hook)(sbintime_t); /* Hook to machdep CPU idler. */ void cpu_switch(struct thread *, struct thread *, struct mtx *); void cpu_throw(struct thread *, struct thread *) __dead2; void unsleep(struct thread *); Index: sys/sys/systm.h =================================================================== --- sys/sys/systm.h (.../head) (revision 246685) +++ sys/sys/systm.h (.../projects/calloutng) (revision 246685) @@ -267,7 +267,7 @@ void stopprofclock(struct proc *); void cpu_startprofclock(void); void cpu_stopprofclock(void); -void cpu_idleclock(void); +sbintime_t cpu_idleclock(void); void cpu_activeclock(void); extern int cpu_can_deep_sleep; extern int cpu_disable_deep_sleep; @@ -346,14 +346,27 @@ * less often. */ int _sleep(void *chan, struct lock_object *lock, int pri, const char *wmesg, - int timo) __nonnull(1); + sbintime_t sbt, sbintime_t pr, int flags) __nonnull(1); #define msleep(chan, mtx, pri, wmesg, timo) \ - _sleep((chan), &(mtx)->lock_object, (pri), (wmesg), (timo)) -int msleep_spin(void *chan, struct mtx *mtx, const char *wmesg, int timo) - __nonnull(1); -int pause(const char *wmesg, int timo); + _sleep((chan), &(mtx)->lock_object, (pri), (wmesg), \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) +#define msleep_sbt(chan, mtx, pri, wmesg, bt, pr, flags) \ + _sleep((chan), &(mtx)->lock_object, (pri), (wmesg), (bt), (pr), \ + (flags)) +int msleep_spin_sbt(void *chan, struct mtx *mtx, const char *wmesg, + sbintime_t sbt, sbintime_t pr, int flags) __nonnull(1); +#define msleep_spin(chan, mtx, wmesg, timo) \ + msleep_spin_sbt((chan), (mtx), (wmesg), (tick_sbt * (timo)), \ + 0, C_HARDCLOCK) +int pause_sbt(const char *wmesg, sbintime_t sbt, sbintime_t pr, + int flags); +#define pause(wmesg, timo) \ + pause_sbt((wmesg), (tick_sbt * (timo)), 0, C_HARDCLOCK) #define tsleep(chan, pri, wmesg, timo) \ - _sleep((chan), NULL, (pri), (wmesg), (timo)) + _sleep((chan), NULL, (pri), (wmesg), (tick_sbt * (timo)), \ + 0, C_HARDCLOCK) +#define tsleep_sbt(chan, pri, wmesg, bt, pr, flags) \ + _sleep((chan), NULL, (pri), (wmesg), (bt), (pr), (flags)) void wakeup(void *chan) __nonnull(1); void wakeup_one(void *chan) __nonnull(1); Index: sys/sys/mutex.h =================================================================== --- sys/sys/mutex.h (.../head) (revision 246685) +++ sys/sys/mutex.h (.../projects/calloutng) (revision 246685) @@ -376,7 +376,8 @@ mtx_assert_((m), (what), __FILE__, __LINE__) #define mtx_sleep(chan, mtx, pri, wmesg, timo) \ - _sleep((chan), &(mtx)->lock_object, (pri), (wmesg), (timo)) + _sleep((chan), &(mtx)->lock_object, (pri), (wmesg), \ + (tick_sbt * (timo)), 0, C_HARDCLOCK) #define mtx_initialized(m) lock_initalized(&(m)->lock_object) Index: sys/powerpc/ps3/platform_ps3.c =================================================================== --- sys/powerpc/ps3/platform_ps3.c (.../head) (revision 246685) +++ sys/powerpc/ps3/platform_ps3.c (.../projects/calloutng) (revision 246685) @@ -70,7 +70,7 @@ static struct cpu_group *ps3_smp_topo(platform_t); #endif static void ps3_reset(platform_t); -static void ps3_cpu_idle(void); +static void ps3_cpu_idle(sbintime_t); static platform_method_t ps3_methods[] = { PLATFORMMETHOD(platform_probe, ps3_probe), @@ -245,7 +245,7 @@ } static void -ps3_cpu_idle(void) +ps3_cpu_idle(sbintime_t sbt) { lv1_pause(0); } Index: sys/powerpc/wii/platform_wii.c =================================================================== --- sys/powerpc/wii/platform_wii.c (.../head) (revision 246685) +++ sys/powerpc/wii/platform_wii.c (.../projects/calloutng) (revision 246685) @@ -60,7 +60,7 @@ int *, struct mem_region **, int *); static unsigned long wii_timebase_freq(platform_t, struct cpuref *cpuref); static void wii_reset(platform_t); -static void wii_cpu_idle(void); +static void wii_cpu_idle(sbintime_t); static platform_method_t wii_methods[] = { PLATFORMMETHOD(platform_probe, wii_probe), @@ -155,6 +155,6 @@ } static void -wii_cpu_idle(void) +wii_cpu_idle(sbintime_t sbt) { } Index: sys/powerpc/powerpc/cpu.c =================================================================== --- sys/powerpc/powerpc/cpu.c (.../head) (revision 246685) +++ sys/powerpc/powerpc/cpu.c (.../projects/calloutng) (revision 246685) @@ -79,9 +79,9 @@ static void cpu_booke_setup(int cpuid, uint16_t vers); int powerpc_pow_enabled; -void (*cpu_idle_hook)(void) = NULL; -static void cpu_idle_60x(void); -static void cpu_idle_booke(void); +void (*cpu_idle_hook)(sbintime_t) = NULL; +static void cpu_idle_60x(sbintime_t); +static void cpu_idle_booke(sbintime_t); struct cputab { const char *name; @@ -516,6 +516,7 @@ void cpu_idle(int busy) { + sbintime_t sbt = -1; #ifdef INVARIANTS if ((mfmsr() & PSL_EE) != PSL_EE) { @@ -531,9 +532,9 @@ if (cpu_idle_hook != NULL) { if (!busy) { critical_enter(); - cpu_idleclock(); + sbt = cpu_idleclock(); } - cpu_idle_hook(); + cpu_idle_hook(sbt); if (!busy) { cpu_activeclock(); critical_exit(); @@ -551,7 +552,7 @@ } static void -cpu_idle_60x(void) +cpu_idle_60x(sbintime_t sbt) { register_t msr; uint16_t vers; @@ -586,7 +587,7 @@ } static void -cpu_idle_booke(void) +cpu_idle_booke(sbintime_t sbt) { register_t msr; Index: sys/pc98/pc98/machdep.c =================================================================== --- sys/pc98/pc98/machdep.c (.../head) (revision 246685) +++ sys/pc98/pc98/machdep.c (.../projects/calloutng) (revision 246685) @@ -1145,7 +1145,7 @@ #define STATE_SLEEPING 0x2 static void -cpu_idle_hlt(int busy) +cpu_idle_hlt(sbintime_t sbt) { int *state; @@ -1186,7 +1186,7 @@ #define MWAIT_C4 0x30 static void -cpu_idle_mwait(int busy) +cpu_idle_mwait(sbintime_t sbt) { int *state; @@ -1209,7 +1209,7 @@ } static void -cpu_idle_spin(int busy) +cpu_idle_spin(sbintime_t sbt) { int *state; int i; @@ -1229,11 +1229,12 @@ } } -void (*cpu_idle_fn)(int) = cpu_idle_hlt; +void (*cpu_idle_fn)(sbintime_t) = cpu_idle_hlt; void cpu_idle(int busy) { + sbintime_t sbt = -1; CTR2(KTR_SPARE2, "cpu_idle(%d) at %d", busy, curcpu); @@ -1251,11 +1252,11 @@ /* If we have time - switch timers into idle mode. */ if (!busy) { critical_enter(); - cpu_idleclock(); + sbt = cpu_idleclock(); } /* Call main idle method. */ - cpu_idle_fn(busy); + cpu_idle_fn(sbt); /* Switch timers mack into active mode. */ if (!busy) { Index: sys/i386/conf/XENHVM =================================================================== --- sys/i386/conf/XENHVM (.../head) (revision 246685) +++ sys/i386/conf/XENHVM (.../projects/calloutng) (revision 246685) Property changes on: sys/i386/conf/XENHVM ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/i386/conf/XENHVM:r236314-242500 Index: sys/i386/i386/machdep.c =================================================================== --- sys/i386/i386/machdep.c (.../head) (revision 246685) +++ sys/i386/i386/machdep.c (.../projects/calloutng) (revision 246685) @@ -1220,7 +1220,7 @@ int scheduler_running; static void -cpu_idle_hlt(int busy) +cpu_idle_hlt(sbintime_t sbt) { scheduler_running = 1; @@ -1241,7 +1241,7 @@ #endif -void (*cpu_idle_hook)(void) = NULL; /* ACPI idle hook. */ +void (*cpu_idle_hook)(sbintime_t) = NULL; /* ACPI idle hook. */ static int cpu_ident_amdc1e = 0; /* AMD C1E supported. */ static int idle_mwait = 1; /* Use MONITOR/MWAIT for short idle. */ TUNABLE_INT("machdep.idle_mwait", &idle_mwait); @@ -1253,7 +1253,7 @@ #define STATE_SLEEPING 0x2 static void -cpu_idle_acpi(int busy) +cpu_idle_acpi(sbintime_t sbt) { int *state; @@ -1265,7 +1265,7 @@ if (sched_runnable()) enable_intr(); else if (cpu_idle_hook) - cpu_idle_hook(); + cpu_idle_hook(sbt); else __asm __volatile("sti; hlt"); *state = STATE_RUNNING; @@ -1273,7 +1273,7 @@ #ifndef XEN static void -cpu_idle_hlt(int busy) +cpu_idle_hlt(sbintime_t sbt) { int *state; @@ -1315,7 +1315,7 @@ #define MWAIT_C4 0x30 static void -cpu_idle_mwait(int busy) +cpu_idle_mwait(sbintime_t sbt) { int *state; @@ -1338,7 +1338,7 @@ } static void -cpu_idle_spin(int busy) +cpu_idle_spin(sbintime_t sbt) { int *state; int i; @@ -1388,9 +1388,9 @@ } #ifdef XEN -void (*cpu_idle_fn)(int) = cpu_idle_hlt; +void (*cpu_idle_fn)(sbintime_t) = cpu_idle_hlt; #else -void (*cpu_idle_fn)(int) = cpu_idle_acpi; +void (*cpu_idle_fn)(sbintime_t) = cpu_idle_acpi; #endif void @@ -1399,6 +1399,7 @@ #ifndef XEN uint64_t msr; #endif + sbintime_t sbt = -1; CTR2(KTR_SPARE2, "cpu_idle(%d) at %d", busy, curcpu); @@ -1418,7 +1419,7 @@ /* If we have time - switch timers into idle mode. */ if (!busy) { critical_enter(); - cpu_idleclock(); + sbt = cpu_idleclock(); } #ifndef XEN @@ -1431,7 +1432,7 @@ #endif /* Call main idle method. */ - cpu_idle_fn(busy); + cpu_idle_fn(sbt); /* Switch timers mack into active mode. */ if (!busy) { Index: sys/netinet/tcp_timer.c =================================================================== --- sys/netinet/tcp_timer.c (.../head) (revision 246685) +++ sys/netinet/tcp_timer.c (.../projects/calloutng) (revision 246685) @@ -719,20 +719,24 @@ #define ticks_to_msecs(t) (1000*(t) / hz) void -tcp_timer_to_xtimer(struct tcpcb *tp, struct tcp_timer *timer, struct xtcp_timer *xtimer) +tcp_timer_to_xtimer(struct tcpcb *tp, struct tcp_timer *timer, + struct xtcp_timer *xtimer) { - bzero(xtimer, sizeof(struct xtcp_timer)); + sbintime_t now; + + bzero(xtimer, sizeof(*xtimer)); if (timer == NULL) return; + getsbinuptime(&now); if (callout_active(&timer->tt_delack)) - xtimer->tt_delack = ticks_to_msecs(timer->tt_delack.c_time - ticks); + xtimer->tt_delack = (timer->tt_delack.c_time - now) / SBT_1MS; if (callout_active(&timer->tt_rexmt)) - xtimer->tt_rexmt = ticks_to_msecs(timer->tt_rexmt.c_time - ticks); + xtimer->tt_rexmt = (timer->tt_rexmt.c_time - now) / SBT_1MS; if (callout_active(&timer->tt_persist)) - xtimer->tt_persist = ticks_to_msecs(timer->tt_persist.c_time - ticks); + xtimer->tt_persist = (timer->tt_persist.c_time - now) / SBT_1MS; if (callout_active(&timer->tt_keep)) - xtimer->tt_keep = ticks_to_msecs(timer->tt_keep.c_time - ticks); + xtimer->tt_keep = (timer->tt_keep.c_time - now) / SBT_1MS; if (callout_active(&timer->tt_2msl)) - xtimer->tt_2msl = ticks_to_msecs(timer->tt_2msl.c_time - ticks); + xtimer->tt_2msl = (timer->tt_2msl.c_time - now) / SBT_1MS; xtimer->t_rcvtime = ticks_to_msecs(ticks - tp->t_rcvtime); } Index: sys/cddl/contrib/opensolaris =================================================================== --- sys/cddl/contrib/opensolaris (.../head) (revision 246685) +++ sys/cddl/contrib/opensolaris (.../projects/calloutng) (revision 246685) Property changes on: sys/cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys/cddl/contrib/opensolaris:r236314-246684 Index: sys =================================================================== --- sys (.../head) (revision 246685) +++ sys (.../projects/calloutng) (revision 246685) Property changes on: sys ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sys:r236314-246684 Index: crypto/openssh =================================================================== --- crypto/openssh (.../head) (revision 246685) +++ crypto/openssh (.../projects/calloutng) (revision 246685) Property changes on: crypto/openssh ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/crypto/openssh:r236314-242500 Index: crypto/heimdal =================================================================== --- crypto/heimdal (.../head) (revision 246685) +++ crypto/heimdal (.../projects/calloutng) (revision 246685) Property changes on: crypto/heimdal ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/crypto/heimdal:r236314-246070 Index: crypto/openssl =================================================================== --- crypto/openssl (.../head) (revision 246685) +++ crypto/openssl (.../projects/calloutng) (revision 246685) Property changes on: crypto/openssl ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/crypto/openssl:r236314-246070 Index: usr.bin/calendar =================================================================== --- usr.bin/calendar (.../head) (revision 246685) +++ usr.bin/calendar (.../projects/calloutng) (revision 246685) Property changes on: usr.bin/calendar ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.bin/calendar:r236314-246070 Index: usr.bin/procstat =================================================================== --- usr.bin/procstat (.../head) (revision 246685) +++ usr.bin/procstat (.../projects/calloutng) (revision 246685) Property changes on: usr.bin/procstat ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.bin/procstat:r236314-246684 Index: usr.bin/csup =================================================================== --- usr.bin/csup (.../head) (revision 246685) +++ usr.bin/csup (.../projects/calloutng) (revision 246685) Property changes on: usr.bin/csup ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/usr.bin/csup:r236314-246684 Index: gnu/usr.bin/binutils =================================================================== --- gnu/usr.bin/binutils (.../head) (revision 246685) +++ gnu/usr.bin/binutils (.../projects/calloutng) (revision 246685) Property changes on: gnu/usr.bin/binutils ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/gnu/usr.bin/binutils:r236314-246070 Index: gnu/usr.bin/gdb =================================================================== --- gnu/usr.bin/gdb (.../head) (revision 246685) +++ gnu/usr.bin/gdb (.../projects/calloutng) (revision 246685) Property changes on: gnu/usr.bin/gdb ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/gnu/usr.bin/gdb:r236314-246070 Index: gnu/usr.bin/cc/cc_tools =================================================================== --- gnu/usr.bin/cc/cc_tools (.../head) (revision 246685) +++ gnu/usr.bin/cc/cc_tools (.../projects/calloutng) (revision 246685) Property changes on: gnu/usr.bin/cc/cc_tools ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/gnu/usr.bin/cc/cc_tools:r236314-246070 Index: gnu/lib =================================================================== --- gnu/lib (.../head) (revision 246685) +++ gnu/lib (.../projects/calloutng) (revision 246685) Property changes on: gnu/lib ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/gnu/lib:r236314-246070 Index: sbin/dumpon =================================================================== --- sbin/dumpon (.../head) (revision 246685) +++ sbin/dumpon (.../projects/calloutng) (revision 246685) Property changes on: sbin/dumpon ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sbin/dumpon:r236314-242500 Index: sbin/ipfw =================================================================== --- sbin/ipfw (.../head) (revision 246685) +++ sbin/ipfw (.../projects/calloutng) (revision 246685) Property changes on: sbin/ipfw ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sbin/ipfw:r236314-242500 Index: sbin =================================================================== --- sbin (.../head) (revision 246685) +++ sbin (.../projects/calloutng) (revision 246685) Property changes on: sbin ___________________________________________________________________ Modified: svn:mergeinfo Merged /head/sbin:r236314-246684 Index: . =================================================================== --- . (.../head) (revision 246685) +++ . (.../projects/calloutng) (revision 246685) Property changes on: . ___________________________________________________________________ Modified: svn:mergeinfo Merged /head:r236314-246684