getting access to VSync interrupt

NTDEV
1

Hey guys,

in order to optimize the performance and reliability (tearing etc) of my video renderer, I’d like to get access to the VSync interrupt for informational purposes. Basically what I have in mind is that I would write a little kernel mode driver which connects to the VSync interrupt and then simply sets an event in the ISR routine. The event would then wake up a high-priority user mode thread, which would perform video playback related tasks.

Of course, in an ideal world either the OS or the GPU drivers would allow me to access this kind of functionality through APIs. But since this is not available, I’d like to try to realize it myself. Obviously I can’t write a full GPU driver. Just want to send VSync notifications to my user land video renderer somehow, if it is possible in any way.

From what I can see, I can find out which interrupt vector the GPU uses via setupapi in user land. And I already have code written for kernel land which successfully finds the GPU PDO. So far, so good. So here are my questions:

(1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual driver to connect to the VSync interrupt? I understand that it’s not really strictly how the PnP model is supposed to work because my driver isn’t really a hardware driver. So would doing this risk system stability?

(2) In my ISR routine, I would ideally like to directly call ZwSetEvent. But that doesn’t seem to be possible because ZwSetEvent requires PASSIVE_LEVEL? Is there any other way I can directly set an event for user land from inside my ISR routine?

(3) I understand interrupts can be shared by multiple different hardware devices. In my ISR routine, is there any way for me to find out which hardware the “current” interrupt came from?

(4) From what I can see, every GPU just has one interrupt, even though it might have multiple displays connected. Is there any way for me to find out which display the VSync interrupt originates from? I guess probably not, but I thought I’d ask, just in case…

Thank you!

2

Everything you want to do is way off the reservation and unsupported. Even if it was supported, setting the event does not necessarily immediately wake up the waiting thread, so you would not get the timing you want.

d

debt from my phone

From: xxxxx@gmail.com
Sent: 7/7/2012 2:51 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] getting access to VSync interrupt

Hey guys,

in order to optimize the performance and reliability (tearing etc) of my video renderer, I’d like to get access to the VSync interrupt for informational purposes. Basically what I have in mind is that I would write a little kernel mode driver which connects to the VSync interrupt and then simply sets an event in the ISR routine. The event would then wake up a high-priority user mode thread, which would perform video playback related tasks.

Of course, in an ideal world either the OS or the GPU drivers would allow me to access this kind of functionality through APIs. But since this is not available, I’d like to try to realize it myself. Obviously I can’t write a full GPU driver. Just want to send VSync notifications to my user land video renderer somehow, if it is possible in any way.

From what I can see, I can find out which interrupt vector the GPU uses via setupapi in user land. And I already have code written for kernel land which successfully finds the GPU PDO. So far, so good. So here are my questions:

(1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual driver to connect to the VSync interrupt? I understand that it’s not really strictly how the PnP model is supposed to work because my driver isn’t really a hardware driver. So would doing this risk system stability?

(2) In my ISR routine, I would ideally like to directly call ZwSetEvent. But that doesn’t seem to be possible because ZwSetEvent requires PASSIVE_LEVEL? Is there any other way I can directly set an event for user land from inside my ISR routine?

(3) I understand interrupts can be shared by multiple different hardware devices. In my ISR routine, is there any way for me to find out which hardware the “current” interrupt came from?

(4) From what I can see, every GPU just has one interrupt, even though it might have multiple displays connected. Is there any way for me to find out which display the VSync interrupt originates from? I guess probably not, but I thought I’d ask, just in case…

Thank you!

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

Everything I want to do? Ouch… :wink:

So is there any supported way for me to get some more control over the GPU
in kernel mode? I’ve read that GPU mirror drivers don’t work that well in
Vista+, anymore (they seem to result in desktop composition being turned
off etc). Any other options? Writing a filter/mini/whatever driver? Or am I
totally out of luck? I’m willing to go the “proper” way, if there is one…

FWIW, I don’t need perfect timing. Just looking for something better than a
VSync scanline polling loop with a Sleep(1).

Thanks, Mathias.

2012/7/7 Doron Holan

> Everything you want to do is way off the reservation and unsupported.
> Even if it was supported, setting the event does not necessarily
> immediately wake up the waiting thread, so you would not get the timing you
> want.
>
> d
>
> debt from my phone
> ------------------------------
> From: xxxxx@gmail.com
> Sent: 7/7/2012 2:51 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] getting access to VSync interrupt
>
> Hey guys,
>
> in order to optimize the performance and reliability (tearing etc) of my
> video renderer, I’d like to get access to the VSync interrupt for
> informational purposes. Basically what I have in mind is that I would write
> a little kernel mode driver which connects to the VSync interrupt and then
> simply sets an event in the ISR routine. The event would then wake up a
> high-priority user mode thread, which would perform video playback related
> tasks.
>
> Of course, in an ideal world either the OS or the GPU drivers would allow
> me to access this kind of functionality through APIs. But since this is not
> available, I’d like to try to realize it myself. Obviously I can’t write a
> full GPU driver. Just want to send VSync notifications to my user land
> video renderer somehow, if it is possible in any way.
>
> From what I can see, I can find out which interrupt vector the GPU uses
> via setupapi in user land. And I already have code written for kernel land
> which successfully finds the GPU PDO. So far, so good. So here are my
> questions:
>
> (1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual
> driver to connect to the VSync interrupt? I understand that it’s not really
> strictly how the PnP model is supposed to work because my driver isn’t
> really a hardware driver. So would doing this risk system stability?
>
> (2) In my ISR routine, I would ideally like to directly call ZwSetEvent.
> But that doesn’t seem to be possible because ZwSetEvent requires
> PASSIVE_LEVEL? Is there any other way I can directly set an event for user
> land from inside my ISR routine?
>
> (3) I understand interrupts can be shared by multiple different hardware
> devices. In my ISR routine, is there any way for me to find out which
> hardware the “current” interrupt came from?
>
> (4) From what I can see, every GPU just has one interrupt, even though it
> might have multiple displays connected. Is there any way for me to find out
> which display the VSync interrupt originates from? I guess probably not,
> but I thought I’d ask, just in case…
>
> Thank you!
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

4

You know, to prevent video tearing you’re going about it all wrong. Apps generally use frame buffer page flipping, and the hardware knows how to flip the page at the right moment. You render into the offscreen page, and queue it up for the hardware to flip. This is normal operation for a modern GPU. The standard user mode DirectX/D3D API’s support this, look for the back buffers option on a surface.

Also the interrupt from the GPU is NOT for VSync, it’s for GPU command completion, which timing wise may have nothing to do with VSync. Stop reading hardware docs from 1990, they no longer apply. Also stop reading the Linux source code for video drivers and thinking Windows is the same, it’s not. What your talked about doing has nothing to do with reality on Windows.

Jan

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of madshi
Sent: Saturday, July 07, 2012 9:24 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] getting access to VSync interrupt

Everything I want to do? Ouch… :wink:

So is there any supported way for me to get some more control over the GPU in kernel mode? I’ve read that GPU mirror drivers don’t work that well in Vista+, anymore (they seem to result in desktop composition being turned off etc). Any other options? Writing a filter/mini/whatever driver? Or am I totally out of luck? I’m willing to go the “proper” way, if there is one…

FWIW, I don’t need perfect timing. Just looking for something better than a VSync scanline polling loop with a Sleep(1).

Thanks, Mathias.

2012/7/7 Doron Holan >
Everything you want to do is way off the reservation and unsupported. Even if it was supported, setting the event does not necessarily immediately wake up the waiting thread, so you would not get the timing you want.

d

debt from my phone
________________________________
From: xxxxx@gmail.commailto:xxxxx
Sent: 7/7/2012 2:51 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] getting access to VSync interrupt
Hey guys,

in order to optimize the performance and reliability (tearing etc) of my video renderer, I’d like to get access to the VSync interrupt for informational purposes. Basically what I have in mind is that I would write a little kernel mode driver which connects to the VSync interrupt and then simply sets an event in the ISR routine. The event would then wake up a high-priority user mode thread, which would perform video playback related tasks.

Of course, in an ideal world either the OS or the GPU drivers would allow me to access this kind of functionality through APIs. But since this is not available, I’d like to try to realize it myself. Obviously I can’t write a full GPU driver. Just want to send VSync notifications to my user land video renderer somehow, if it is possible in any way.

From what I can see, I can find out which interrupt vector the GPU uses via setupapi in user land. And I already have code written for kernel land which successfully finds the GPU PDO. So far, so good. So here are my questions:

(1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual driver to connect to the VSync interrupt? I understand that it’s not really strictly how the PnP model is supposed to work because my driver isn’t really a hardware driver. So would doing this risk system stability?

(2) In my ISR routine, I would ideally like to directly call ZwSetEvent. But that doesn’t seem to be possible because ZwSetEvent requires PASSIVE_LEVEL? Is there any other way I can directly set an event for user land from inside my ISR routine?

(3) I understand interrupts can be shared by multiple different hardware devices. In my ISR routine, is there any way for me to find out which hardware the “current” interrupt came from?

(4) From what I can see, every GPU just has one interrupt, even though it might have multiple displays connected. Is there any way for me to find out which display the VSync interrupt originates from? I guess probably not, but I thought I’d ask, just in case…

Thank you!

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx>

5

First, you can’t set an event in an ISR. See the documentation for
KeSetEvent.

Second, I don’t care how high the priority of your thread is, the
liklihood that it will start running before the vertical retrace is
complete is vanishingly small. Even if you use the new Multimedia
scheduler support in Vista+, the highest priority you can run a user
thread is something like 26, and scheduling a user thread is a
very-heavy-duty operation.

Back in the days when I was involved in high-performance video, we used
Vsync to swap frame buffer pointers, and tat was pretty much all we could
do. In the case of video, we would let the application get ahead, but if
it fell behind we could skip N frames to get back into realtime sync. But
this was all done in the ISR, with no user threads involved at all. As
soon as you introduce the concept of scheduling a thread, you are into
random and large delays relative to what you can tolerate. Think of tens
of milliseconds to potentially hundreds of milliseconds latency between
setting the event and the user code running.
joe

Hey guys,

in order to optimize the performance and reliability (tearing etc) of my
video renderer, I’d like to get access to the VSync interrupt for
informational purposes. Basically what I have in mind is that I would
write a little kernel mode driver which connects to the VSync interrupt
and then simply sets an event in the ISR routine. The event would then
wake up a high-priority user mode thread, which would perform video
playback related tasks.

Of course, in an ideal world either the OS or the GPU drivers would allow
me to access this kind of functionality through APIs. But since this is
not available, I’d like to try to realize it myself. Obviously I can’t
write a full GPU driver. Just want to send VSync notifications to my user
land video renderer somehow, if it is possible in any way.

From what I can see, I can find out which interrupt vector the GPU uses
via setupapi in user land. And I already have code written for kernel land
which successfully finds the GPU PDO. So far, so good. So here are my
questions:

(1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual
driver to connect to the VSync interrupt? I understand that it’s not
really strictly how the PnP model is supposed to work because my driver
isn’t really a hardware driver. So would doing this risk system stability?

(2) In my ISR routine, I would ideally like to directly call ZwSetEvent.
But that doesn’t seem to be possible because ZwSetEvent requires
PASSIVE_LEVEL? Is there any other way I can directly set an event for user
land from inside my ISR routine?

(3) I understand interrupts can be shared by multiple different hardware
devices. In my ISR routine, is there any way for me to find out which
hardware the “current” interrupt came from?

(4) From what I can see, every GPU just has one interrupt, even though it
might have multiple displays connected. Is there any way for me to find
out which display the VSync interrupt originates from? I guess probably
not, but I thought I’d ask, just in case…

Thank you!

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

wrote in message news:xxxxx@ntdev…
> the highest priority you can run a user thread is something like 26, and
> scheduling a user thread is a
> very-heavy-duty operation.

31, not 26.

> As soon as you introduce the concept of scheduling a thread, you are into
> random and large delays relative to what you can tolerate. Think of tens
> of milliseconds to potentially hundreds of milliseconds latency between
> setting the event and the user code running.

Microseconds, not milliseconds.

//Daniel

7

wrote in message news:xxxxx@ntdev…
>> As soon as you introduce the concept of scheduling a thread, you are
>> into
>> random and large delays relative to what you can tolerate. Think of tens
>> of milliseconds to potentially hundreds of milliseconds latency between
>> setting the event and the user code running.
>
> Microseconds, not milliseconds.
>
> //Daniel

Worst case, not best case
– pa

8

No, milliseconds. Actual measurement has shown that
interrupt-to-running-in-user-space can be as much as 450 milliseconds.
This is an actual, measured, number. The average was about 225ms.
joe

wrote in message news:xxxxx@ntdev…
>> the highest priority you can run a user thread is something like 26, and
>> scheduling a user thread is a
>> very-heavy-duty operation.
>
> 31, not 26.
>
>
>> As soon as you introduce the concept of scheduling a thread, you are
>> into
>> random and large delays relative to what you can tolerate. Think of
>> tens
>> of milliseconds to potentially hundreds of milliseconds latency between
>> setting the event and the user code running.
>
> Microseconds, not milliseconds.
>
> //Daniel
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

9

Best case was slightly under 100 milliseconds, worst case was 450ms,
average was 225ms. Real numbers, measured on a real system. By someone
who cared a lot about realtime response.

And, the latency is certainly higher than a vertical retrace (do flat
panel displays even have the concept?)
joe

wrote in message news:xxxxx@ntdev…
>>> As soon as you introduce the concept of scheduling a thread, you are
>>> into
>>> random and large delays relative to what you can tolerate. Think of
>>> tens
>>> of milliseconds to potentially hundreds of milliseconds latency between
>>> setting the event and the user code running.
>>
>> Microseconds, not milliseconds.
>>
>> //Daniel
>
> Worst case, not best case
> – pa
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

10

A typical audio interface has a samplerate of 96Khz and a buffer size of 64
frames. How do you think an audio program can render and deliver 1500
buffers per second to the hardware each of which in time ? That leaves about
6ms per buffer. One audio subsystem I know of wakes up a thread for every
single one of those. Whether that’s by an I/O completion or an event should
not matter.

Your numbers are not realistic or you need to be doing something bad such as
hitting hard pagefaults.

//Daniel

wrote in message news:xxxxx@ntdev…
> Best case was slightly under 100 milliseconds, worst case was 450ms,
> average was 225ms. Real numbers, measured on a real system. By someone
> who cared a lot about realtime response.
>
> And, the latency is certainly higher than a vertical retrace (do flat
> panel displays even have the concept?)
> joe
>
>
>> wrote in message news:xxxxx@ntdev…
>>>> As soon as you introduce the concept of scheduling a thread, you are
>>>> into
>>>> random and large delays relative to what you can tolerate. Think of
>>>> tens
>>>> of milliseconds to potentially hundreds of milliseconds latency between
>>>> setting the event and the user code running.
>>>
>>> Microseconds, not milliseconds.
>>>
>>> //Daniel
>>
>> Worst case, not best case
>> – pa
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
>
>

11

>And, the latency is certainly higher than a vertical retrace (do flat panel displays even have the concept?)

They don’t need to have that, but the frame timings might still be kept compatible with CRT monitors.

Anyway, latencies can be somehow kept tight by careful choosing of the thread priorities. sub-miliseconds are definitely possible.

12

I know how page flipping works, I’ve been using it for years. My video
renderer uses D3D9(Ex), and I’m supporting all the various methods like
windowed mode (with and without desktop composition), fullscreen exclusive
mode etc. In windowed mode (without desktop composition) if you ask D3D to
present a frame during the next VSync, D3D goes into a blocking 100% CPU
loop, which polls for the VSync. Nasty. In fullscreen exclusive mode,
things kinda work as expected, but I’m running into all sorts of trouble
with NVidia driver bugs, resulting in presentation glitches being reported
by D3D9Ex and I’m getting non-smooth playback as a result. (No probs with
AMD hardware at the moment.)

So are you saying that the GPU does not use the interrupt for VSync at all,
anymore, in Windows? Microsoft documentation suggests otherwise:

http://msdn.microsoft.com/en-us/library/windows/hardware/gg487377.aspx

“Beginning with Windows Vista Service Pack 1, the operating system can turn
off periodic VSync interrupt counting when the screen is not being
refreshed from new graphics or mouse activity.”

Best regards, Mathias.

2012/7/7 Jan Bottorff

> You know, to prevent video tearing you?re going about it all wrong. Apps
> generally use frame buffer page flipping, and the hardware knows how to
> flip the page at the right moment. You render into the offscreen page, and
> queue it up for the hardware to flip. This is normal operation for a
> modern GPU. The standard user mode DirectX/D3D API?s support this, look for
> the back buffers option on a surface.
>
>
>
> Also the interrupt from the GPU is NOT for VSync, it?s for GPU command
> completion, which timing wise may have nothing to do with VSync. Stop
> reading hardware docs from 1990, they no longer apply. Also stop reading
> the Linux source code for video drivers and thinking Windows is the same,
> it?s not. What your talked about doing has nothing to do with reality on
> Windows.
>
> ****
>
> Jan
>
>
>
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of madshi
> Sent: Saturday, July 07, 2012 9:24 AM
>
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] getting access to VSync interrupt
>
> ****
>
> Everything I want to do? Ouch… :wink:
>
> So is there any supported way for me to get some more control over the GPU
> in kernel mode? I’ve read that GPU mirror drivers don’t work that well in
> Vista+, anymore (they seem to result in desktop composition being turned
> off etc). Any other options? Writing a filter/mini/whatever driver? Or am I
> totally out of luck? I’m willing to go the “proper” way, if there is one…
>
> FWIW, I don’t need perfect timing. Just looking for something better than
> a VSync scanline polling loop with a Sleep(1).
>
> Thanks, Mathias.
>
>
>
> 2012/7/7 Doron Holan
>
> Everything you want to do is way off the reservation and unsupported. Even
> if it was supported, setting the event does not necessarily immediately
> wake up the waiting thread, so you would not get the timing you want.
>
> d
>
> debt from my phone**
> ------------------------------
>
> *From: *xxxxx@gmail.com
> *Sent: *7/7/2012 2:51 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] getting access to VSync interrupt
>
> Hey guys,
>
> in order to optimize the performance and reliability (tearing etc) of my
> video renderer, I’d like to get access to the VSync interrupt for
> informational purposes. Basically what I have in mind is that I would write
> a little kernel mode driver which connects to the VSync interrupt and then
> simply sets an event in the ISR routine. The event would then wake up a
> high-priority user mode thread, which would perform video playback related
> tasks.
>
> Of course, in an ideal world either the OS or the GPU drivers would allow
> me to access this kind of functionality through APIs. But since this is not
> available, I’d like to try to realize it myself. Obviously I can’t write a
> full GPU driver. Just want to send VSync notifications to my user land
> video renderer somehow, if it is possible in any way.
>
> From what I can see, I can find out which interrupt vector the GPU uses
> via setupapi in user land. And I already have code written for kernel land
> which successfully finds the GPU PDO. So far, so good. So here are my
> questions:
>
> (1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual
> driver to connect to the VSync interrupt? I understand that it’s not really
> strictly how the PnP model is supposed to work because my driver isn’t
> really a hardware driver. So would doing this risk system stability?
>
> (2) In my ISR routine, I would ideally like to directly call ZwSetEvent.
> But that doesn’t seem to be possible because ZwSetEvent requires
> PASSIVE_LEVEL? Is there any other way I can directly set an event for user
> land from inside my ISR routine?
>
> (3) I understand interrupts can be shared by multiple different hardware
> devices. In my ISR routine, is there any way for me to find out which
> hardware the “current” interrupt came from?
>
> (4) From what I can see, every GPU just has one interrupt, even though it
> might have multiple displays connected. Is there any way for me to find out
> which display the VSync interrupt originates from? I guess probably not,
> but I thought I’d ask, just in case…
>
> Thank you!
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer ****
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

13

Hi Joe,

swapping frame pointers (or maybe overlay frame pointers)
on VSync would be a dream, of course. That would be step 3,
if it is possible to achieve at all from my own driver. My first 2
steps would be (1) counting the VSyncs. And (2) notifying user
land about them. I guess you did that high-performance video
stuff on another OS and not on Windows?

I know that timing in user land isn’t too reliable, even if you use
the highest possible thread priority. My video rendering code is
working in such a way that I don’t necessarily have to run inside
of the VSync period (although of course that would be great!).
It’s ok if I’m getting the VSync notification a bit too late sometimes.

About the event: What would be the best (performing) way to
notify user land from within an ISR routine? Maybe using
IoQueueWorkItem would work? Of course that would involve
waking up 2 threads until notification reaches user land. I guess
there’s no way to do the wakeup initiation directly from within
the ISR routine?

Thanks, Mathias.

2012/7/8

> First, you can’t set an event in an ISR. See the documentation for
> KeSetEvent.
>
> Second, I don’t care how high the priority of your thread is, the
> liklihood that it will start running before the vertical retrace is
> complete is vanishingly small. Even if you use the new Multimedia
> scheduler support in Vista+, the highest priority you can run a user
> thread is something like 26, and scheduling a user thread is a
> very-heavy-duty operation.
>
> Back in the days when I was involved in high-performance video, we used
> Vsync to swap frame buffer pointers, and tat was pretty much all we could
> do. In the case of video, we would let the application get ahead, but if
> it fell behind we could skip N frames to get back into realtime sync. But
> this was all done in the ISR, with no user threads involved at all. As
> soon as you introduce the concept of scheduling a thread, you are into
> random and large delays relative to what you can tolerate. Think of tens
> of milliseconds to potentially hundreds of milliseconds latency between
> setting the event and the user code running.
> joe
>
> > Hey guys,
> >
> > in order to optimize the performance and reliability (tearing etc) of my
> > video renderer, I’d like to get access to the VSync interrupt for
> > informational purposes. Basically what I have in mind is that I would
> > write a little kernel mode driver which connects to the VSync interrupt
> > and then simply sets an event in the ISR routine. The event would then
> > wake up a high-priority user mode thread, which would perform video
> > playback related tasks.
> >
> > Of course, in an ideal world either the OS or the GPU drivers would allow
> > me to access this kind of functionality through APIs. But since this is
> > not available, I’d like to try to realize it myself. Obviously I can’t
> > write a full GPU driver. Just want to send VSync notifications to my user
> > land video renderer somehow, if it is possible in any way.
> >
> > From what I can see, I can find out which interrupt vector the GPU uses
> > via setupapi in user land. And I already have code written for kernel
> land
> > which successfully finds the GPU PDO. So far, so good. So here are my
> > questions:
> >
> > (1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual
> > driver to connect to the VSync interrupt? I understand that it’s not
> > really strictly how the PnP model is supposed to work because my driver
> > isn’t really a hardware driver. So would doing this risk system
> stability?
> >
> > (2) In my ISR routine, I would ideally like to directly call ZwSetEvent.
> > But that doesn’t seem to be possible because ZwSetEvent requires
> > PASSIVE_LEVEL? Is there any other way I can directly set an event for
> user
> > land from inside my ISR routine?
> >
> > (3) I understand interrupts can be shared by multiple different hardware
> > devices. In my ISR routine, is there any way for me to find out which
> > hardware the “current” interrupt came from?
> >
> > (4) From what I can see, every GPU just has one interrupt, even though it
> > might have multiple displays connected. Is there any way for me to find
> > out which display the VSync interrupt originates from? I guess probably
> > not, but I thought I’d ask, just in case…
> >
> > Thank you!
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

14

Hi Mathias,
I’m not a graphics expert, so there may be something I’m missing, but have you looked at the WaitForVBlank api (http://msdn.microsoft.com/en-us/library/windows/desktop/bb174559(v=vs.85).aspx)? It is a little convoluted to get at it, but I think it does wake up your thread at the refresh rate.

-Neeraj (windows kernel guy)

15

Hi Neeraj,

thanks for the suggestion. Last time I checked, the thread which calls that
API goes into a loop which runs at 100% CPU consumption and polls for the
VSync. This is problematic mainly because of two reasons:

(1) Wasted CPU power, which I would much rather spend on video
decoding/processing.
(2) Calling the API partially blocks D3D, which means that video rendering
(which runs in a different thread) can stall, too.

At least that has been my experience. It also seems to be confirmed here:

http://forums.create.msdn.com/forums/t/42998.aspx

“WaitForVBlank will prevent all other rendering activity in the process
until wait is done.”

I’m doing video presentation and video rendering (which consists of
multiple pixel shader passes) in separate threads. Rendering must continue
to work while presentation waits for the next VSync. No problem in
fullscreen exclusive mode, but windowed mode (with desktop composition
turned off) is problematic. As a result I’ve had to write my own VSync
waiting code, which works reasonably well. But I’d like to improve it even
further, hence my interest in getting access to the VSync interrupt somehow.

Best regards, Mathias.

2012/7/8

> Hi Mathias,
> I’m not a graphics expert, so there may be something I’m missing, but have
> you looked at the WaitForVBlank api (
> http://msdn.microsoft.com/en-us/library/windows/desktop/bb174559(v=vs.85).aspx)?
> It is a little convoluted to get at it, but I think it does wake up your
> thread at the refresh rate.
>
> -Neeraj (windows kernel guy)
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

16

So I looked at the link you posted. All that is saying is there a WDDM driver API for the OS to give power management hints to the GPU driver.

What I’m saying about interrupts is the interrupt from the GPU is owned by the GPU driver, and ONLY the GPU driver knows the correct way to decode the interrupt. So let’s say one of the sources of an interrupt is currently a vsync. To tell the difference between a vsync interrupt and a command completion interrupt, the GPU driver’s ISR likely needs to read a device register (or the interrupt info was already delivered via DMA or part of an MSI-X interrupt). MOST devices have multiple sources of interrupts. YOUR driver would have no way to tell what caused the interrupt, and if YOUR driver touched the GPU registers to try and determine the source, you have likely just screwed up the GPU driver, because reading hardware registers often have side effects like clearing some status. It seems like you’re making some assumption that ANY interrupt MUST be a vsync interrupt, which is almost certainly not the case.

You’re saying everything is fine on AMD video cards? So have you contacted NVidia. It’s easy to join the NVidia developer program and my very limited experience has been they are quite responsive about driver issues. Maybe NVidia can send you a fragment of sample D3D code for their hardware to do what you need to do. I think you would get better suggestions on how to solve you issue in a D3D programming list, or I assume NVidia has a developer community, and someone there might have a suggestion.

Jan

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of madshi
Sent: Saturday, July 07, 2012 11:30 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] getting access to VSync interrupt

I know how page flipping works, I’ve been using it for years. My video renderer uses D3D9(Ex), and I’m supporting all the various methods like windowed mode (with and without desktop composition), fullscreen exclusive mode etc. In windowed mode (without desktop composition) if you ask D3D to present a frame during the next VSync, D3D goes into a blocking 100% CPU loop, which polls for the VSync. Nasty. In fullscreen exclusive mode, things kinda work as expected, but I’m running into all sorts of trouble with NVidia driver bugs, resulting in presentation glitches being reported by D3D9Ex and I’m getting non-smooth playback as a result. (No probs with AMD hardware at the moment.)

So are you saying that the GPU does not use the interrupt for VSync at all, anymore, in Windows? Microsoft documentation suggests otherwise:

http://msdn.microsoft.com/en-us/library/windows/hardware/gg487377.aspx

“Beginning with Windows Vista Service Pack 1, the operating system can turn off periodic VSync interrupt counting when the screen is not being refreshed from new graphics or mouse activity.”

Best regards, Mathias.

2012/7/7 Jan Bottorff >
You know, to prevent video tearing you’re going about it all wrong. Apps generally use frame buffer page flipping, and the hardware knows how to flip the page at the right moment. You render into the offscreen page, and queue it up for the hardware to flip. This is normal operation for a modern GPU. The standard user mode DirectX/D3D API’s support this, look for the back buffers option on a surface.

Also the interrupt from the GPU is NOT for VSync, it’s for GPU command completion, which timing wise may have nothing to do with VSync. Stop reading hardware docs from 1990, they no longer apply. Also stop reading the Linux source code for video drivers and thinking Windows is the same, it’s not. What your talked about doing has nothing to do with reality on Windows.

Jan

From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.commailto:xxxxx] On Behalf Of madshi
Sent: Saturday, July 07, 2012 9:24 AM

To: Windows System Software Devs Interest List
Subject: Re: [ntdev] getting access to VSync interrupt

Everything I want to do? Ouch… :wink:

So is there any supported way for me to get some more control over the GPU in kernel mode? I’ve read that GPU mirror drivers don’t work that well in Vista+, anymore (they seem to result in desktop composition being turned off etc). Any other options? Writing a filter/mini/whatever driver? Or am I totally out of luck? I’m willing to go the “proper” way, if there is one…

FWIW, I don’t need perfect timing. Just looking for something better than a VSync scanline polling loop with a Sleep(1).

Thanks, Mathias.
2012/7/7 Doron Holan >
Everything you want to do is way off the reservation and unsupported. Even if it was supported, setting the event does not necessarily immediately wake up the waiting thread, so you would not get the timing you want.

d

debt from my phone
________________________________
From: xxxxx@gmail.commailto:xxxxx
Sent: 7/7/2012 2:51 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] getting access to VSync interrupt
Hey guys,

in order to optimize the performance and reliability (tearing etc) of my video renderer, I’d like to get access to the VSync interrupt for informational purposes. Basically what I have in mind is that I would write a little kernel mode driver which connects to the VSync interrupt and then simply sets an event in the ISR routine. The event would then wake up a high-priority user mode thread, which would perform video playback related tasks.

Of course, in an ideal world either the OS or the GPU drivers would allow me to access this kind of functionality through APIs. But since this is not available, I’d like to try to realize it myself. Obviously I can’t write a full GPU driver. Just want to send VSync notifications to my user land video renderer somehow, if it is possible in any way.

From what I can see, I can find out which interrupt vector the GPU uses via setupapi in user land. And I already have code written for kernel land which successfully finds the GPU PDO. So far, so good. So here are my questions:

(1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual driver to connect to the VSync interrupt? I understand that it’s not really strictly how the PnP model is supposed to work because my driver isn’t really a hardware driver. So would doing this risk system stability?

(2) In my ISR routine, I would ideally like to directly call ZwSetEvent. But that doesn’t seem to be possible because ZwSetEvent requires PASSIVE_LEVEL? Is there any other way I can directly set an event for user land from inside my ISR routine?

(3) I understand interrupts can be shared by multiple different hardware devices. In my ISR routine, is there any way for me to find out which hardware the “current” interrupt came from?

(4) From what I can see, every GPU just has one interrupt, even though it might have multiple displays connected. Is there any way for me to find out which display the VSync interrupt originates from? I guess probably not, but I thought I’d ask, just in case…

Thank you!

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

17

> And, the latency is certainly higher than a vertical retrace (do flat

panel displays even have the concept?)

The data signalling of VGA and DVI/HDMI still have the concept - the image is rendered left to right, top to bottom so signals are required to mark the start and end of each line (HSync), and the start and end of each frame (VSync).

Tim.

18

You’re right that a GPU interrupt can be something other than a VSync. In
my original post I also mentioned that when using multiple displays there’s
probably no way for me to tell which display the interrupt originates from,
even if I knew it was a VSync interrupt. So there definitely is a problem
for me with properly interpreting the interrupts. But in the end what I’m
looking for is just a better way of synchronizing rendering and
presentation than doing a Sleep(1) polling loop. Even I got false interrupt
notifications, that would still be better than the current solution, as
long as I at least get all the “good” interrupt notifications, too. In user
land I can ask the current VSync scanline position to check if the
interrupt notification I received is useful or not. At the moment I don’t
plan to touch GPU registers, I don’t think I need to.

“Everything is fine” with AMD mostly applies to fullscreen exclusive mode.
This is all a very complicated topic and I didn’t want to bore you with all
the complex details. My video renderer needs to support different
situations. It has to work in windowed mode. It has to work with Aero
turned on and off etc. It also has a fullscreen (exclusive) mode. I need to
make it all work. The most reliable solution is fullscreen exclusive mode.
In that mode AMD works fine, NVidia not so much. I do plan to contact
NVidia about it, hopefully they’ll fix whatever bug their drivers have. But
I need to support windowed mode, too, and for that, frankly, D3D sucks,
pardon my french. Without Aero, if you ask D3D in windowed mode to present
a frame during the next VSync period, the presentation thread goes into a
100% CPU loop and rendering is blocked during the wait. Totally unusable.
With Aero, this is not a problem, but there are different problems. In the
end, no matter what, my rendering code needs to keep track of the VSync to
make sure playback is totally smooth. No way around it. And even if all GPU
driver bugs were fixed, I’d still benefit from getting VSync interrupt
notifications to optimize my rendering and presentation synchronization.
The only case it would not help would be if it would not perform any better
than a Sleep(1) polling loop, and that’s hard for me to believe.

Best regards, Mathias.

2012/7/8 Jan Bottorff

> So I looked at the link you posted. All that is saying is there a WDDM
> driver API for the OS to give power management hints to the GPU driver.
>
>
> **
>
> What I?m saying about interrupts is the interrupt from the GPU is owned by
> the GPU driver, and ONLY the GPU driver knows the correct way to decode the
> interrupt. So let?s say one of the sources of an interrupt is currently a
> vsync. To tell the difference between a vsync interrupt and a command
> completion interrupt, the GPU driver?s ISR likely needs to read a device
> register (or the interrupt info was already delivered via DMA or part of an
> MSI-X interrupt). MOST devices have multiple sources of interrupts. YOUR
> driver would have no way to tell what caused the interrupt, and if YOUR
> driver touched the GPU registers to try and determine the source, you have
> likely just screwed up the GPU driver, because reading hardware registers
> often have side effects like clearing some status. It seems like you?re
> making some assumption that ANY interrupt MUST be a vsync interrupt, which
> is almost certainly not the case.
>
> ****
>
> You?re saying everything is fine on AMD video cards? So have you contacted
> NVidia. It?s easy to join the NVidia developer program and my very limited
> experience has been they are quite responsive about driver issues. Maybe
> NVidia can send you a fragment of sample D3D code for their hardware to do
> what you need to do. I think you would get better suggestions on how to
> solve you issue in a D3D programming list, or I assume NVidia has a
> developer community, and someone there might have a suggestion.
>
>
>
> Jan
>
> ****
>
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of madshi
> Sent: Saturday, July 07, 2012 11:30 PM
>
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] getting access to VSync interrupt
>
> **
>
> I know how page flipping works, I’ve been using it for years. My video
> renderer uses D3D9(Ex), and I’m supporting all the various methods like
> windowed mode (with and without desktop composition), fullscreen exclusive
> mode etc. In windowed mode (without desktop composition) if you ask D3D to
> present a frame during the next VSync, D3D goes into a blocking 100% CPU
> loop, which polls for the VSync. Nasty. In fullscreen exclusive mode,
> things kinda work as expected, but I’m running into all sorts of trouble
> with NVidia driver bugs, resulting in presentation glitches being reported
> by D3D9Ex and I’m getting non-smooth playback as a result. (No probs with
> AMD hardware at the moment.)
>
> So are you saying that the GPU does not use the interrupt for VSync at
> all, anymore, in Windows? Microsoft documentation suggests otherwise:
>
> http://msdn.microsoft.com/en-us/library/windows/hardware/gg487377.aspx
>
> “Beginning with Windows Vista Service Pack 1, the operating system can
> turn off periodic VSync interrupt counting when the screen is not being
> refreshed from new graphics or mouse activity.”
>
> Best regards, Mathias.
>
>
>
>
> 2012/7/7 Jan Bottorff
>
> You know, to prevent video tearing you?re going about it all wrong. Apps
> generally use frame buffer page flipping, and the hardware knows how to
> flip the page at the right moment. You render into the offscreen page, and
> queue it up for the hardware to flip. This is normal operation for a
> modern GPU. The standard user mode DirectX/D3D API?s support this, look for
> the back buffers option on a surface.
>
>
>
> Also the interrupt from the GPU is NOT for VSync, it?s for GPU command
> completion, which timing wise may have nothing to do with VSync. Stop
> reading hardware docs from 1990, they no longer apply. Also stop reading
> the Linux source code for video drivers and thinking Windows is the same,
> it?s not. What your talked about doing has nothing to do with reality on
> Windows.
>
>
>
> Jan
>
>
>
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of madshi
> Sent: Saturday, July 07, 2012 9:24 AM
>
>
> To: Windows System Software Devs Interest List
>
> Subject: Re: [ntdev] getting access to VSync interrupt****
>
>
>
> Everything I want to do? Ouch… :wink:
>
> So is there any supported way for me to get some more control over the GPU
> in kernel mode? I’ve read that GPU mirror drivers don’t work that well in
> Vista+, anymore (they seem to result in desktop composition being turned
> off etc). Any other options? Writing a filter/mini/whatever driver? Or am I
> totally out of luck? I’m willing to go the “proper” way, if there is one…
>
> FWIW, I don’t need perfect timing. Just looking for something better than
> a VSync scanline polling loop with a Sleep(1).
>
> Thanks, Mathias.
>
> 2012/7/7 Doron Holan
>
> Everything you want to do is way off the reservation and unsupported. Even
> if it was supported, setting the event does not necessarily immediately
> wake up the waiting thread, so you would not get the timing you want.
>
> d
>
> debt from my phone
> ------------------------------
>
> From: xxxxx@gmail.com
> Sent: 7/7/2012 2:51 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] getting access to VSync interrupt
>
> Hey guys,
>
> in order to optimize the performance and reliability (tearing etc) of my
> video renderer, I’d like to get access to the VSync interrupt for
> informational purposes. Basically what I have in mind is that I would write
> a little kernel mode driver which connects to the VSync interrupt and then
> simply sets an event in the ISR routine. The event would then wake up a
> high-priority user mode thread, which would perform video playback related
> tasks.
>
> Of course, in an ideal world either the OS or the GPU drivers would allow
> me to access this kind of functionality through APIs. But since this is not
> available, I’d like to try to realize it myself. Obviously I can’t write a
> full GPU driver. Just want to send VSync notifications to my user land
> video renderer somehow, if it is possible in any way.
>
> From what I can see, I can find out which interrupt vector the GPU uses
> via setupapi in user land. And I already have code written for kernel land
> which successfully finds the GPU PDO. So far, so good. So here are my
> questions:
>
> (1) Is it “ok” for me to simply call IoConnectInterrupt(Ex) in my virtual
> driver to connect to the VSync interrupt? I understand that it’s not really
> strictly how the PnP model is supposed to work because my driver isn’t
> really a hardware driver. So would doing this risk system stability?
>
> (2) In my ISR routine, I would ideally like to directly call ZwSetEvent.
> But that doesn’t seem to be possible because ZwSetEvent requires
> PASSIVE_LEVEL? Is there any other way I can directly set an event for user
> land from inside my ISR routine?
>
> (3) I understand interrupts can be shared by multiple different hardware
> devices. In my ISR routine, is there any way for me to find out which
> hardware the “current” interrupt came from?
>
> (4) From what I can see, every GPU just has one interrupt, even though it
> might have multiple displays connected. Is there any way for me to find out
> which display the VSync interrupt originates from? I guess probably not,
> but I thought I’d ask, just in case…
>
> Thank you!
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and
> other seminars visit: http://www.osr.com/seminars To unsubscribe, visit
> the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

19

Why don’t you call Sleep(0). That will give up the processor to other runnable threads, but comes right back if there is no other work. It may APPEAR to consume 100% cpu on the thread, but only if there is no other work. Sleep(1) will often actually be Sleep(16), and will suspend the thread.

Jan

The only case it would not help would be if it would not perform any better than a Sleep(1) polling loop, and that’s hard for me to believe.

20

Unless you set the timer resolution to 1 ms and then it could be a lot closer to a 1 ms acquiesce

d

debt from my phone

From: Jan Bottorff
Sent: 7/8/2012 4:08 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] getting access to VSync interrupt

Why don?t you call Sleep(0). That will give up the processor to other runnable threads, but comes right back if there is no other work. It may APPEAR to consume 100% cpu on the thread, but only if there is no other work. Sleep(1) will often actually be Sleep(16), and will suspend the thread.

Jan

The only case it would not help would be if it would not perform any better than a Sleep(1) polling loop, and that’s hard for me to believe.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer