S3 to S0 state change for PCIe driver

NTDEV
1

Hi,

I have a custom PCIe card which contains four FPGAs (PCIe GEN1 4x) and a PLX PCIe bridge chip (PCIe GEN2 8x). Al FPGAs are connected to the PLX chip. We developed this card and its driver by ourself. The card can be used with a Windows 7, 64 bit and Windows XP, 32 bit system.

The card and driver are working properly in normal operation (state S0, fully powered). The problem I see, is triggered by entering the sleep state (S3) and wake up the system again (to S0). In state S3 the card is totally powered off. After waking up, the driver cannot communicate with the card anymore. The FPGAs (with a PCIe core inside) seems to be working properly (debug LEDs are blinking). If I compare the PCI Configuration registers, I can see the BAR addresses are reset to zero (which is a good reason for communication problems)! Question is, why are the PCI configuration registers (partly) invalid?

Does anyone have an idea what happens here? Is it the responsibility of the driver to rewrite the PCI configuration space when it receives a wake up event (IRP_MN_SET_POWER)? Or should the OS handle this?

Is there a minimum implementation required for supporting S0->S3->S0 state switching? At this moment the power IRPs are just forwarded.

Thanks in advance,

Frank van Eijkelenburg

2

xxxxx@gmail.com wrote:

The card and driver are working properly in normal operation (state S0, fully powered). The problem I see, is triggered by entering the sleep state (S3) and wake up the system again (to S0). In state S3 the card is totally powered off. After waking up, the driver cannot communicate with the card anymore. The FPGAs (with a PCIe core inside) seems to be working properly (debug LEDs are blinking). If I compare the PCI Configuration registers, I can see the BAR addresses are reset to zero (which is a good reason for communication problems)! Question is, why are the PCI configuration registers (partly) invalid?

In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device
responds to various system power states. You need to be telling the
system that you respond to an S3 transition by going into D3, which
means you lose your mind. Then, when you send down your request to
shift back to D09, the system should restore your configuration space.

You can see your current mappings in Device Manager, on the Details tab,
in “Power State Mappings”.

Is there a minimum implementation required for supporting S0->S3->S0 state switching? At this moment the power IRPs are just forwarded.

Hell, yes, there is. When you get a request to transition to S3, you
need to send a request to yourself to transition to D3. Then, when you
get a request to transition back to S0, you need to send a request to
yourself to transition to D0. That gets sent down to the PCI bus
driver, which then realizes “hey, I need to restore this guy”.

This, by the way, is EXACTLY why you should be using KMDF instead of WDM.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

3

As usual, I like Tim’s answer. He’s right. Use KMDF and your immediate problem will certainly go away.

But since you brought it up, we tend to see a lot of FPGA implementations that have a lot of trouble with power management. Many people implementing with FPGAs fail to meet the PCI-spec-mandated 100ms limit on initialization time. If your device’s configuration space doesn’t look exactly like it did before (with respect to device ID, vendor ID, subsystem ID, etc.) 100ms after reapplying power, you’ll see your driver stack torn down as failed.

  • Jake Oshins
    (sometime PCI guy)
    Windows Kernel Team

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Thursday, January 24, 2013 9:56 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

xxxxx@gmail.com wrote:

The card and driver are working properly in normal operation (state S0, fully powered). The problem I see, is triggered by entering the sleep state (S3) and wake up the system again (to S0). In state S3 the card is totally powered off. After waking up, the driver cannot communicate with the card anymore. The FPGAs (with a PCIe core inside) seems to be working properly (debug LEDs are blinking). If I compare the PCI Configuration registers, I can see the BAR addresses are reset to zero (which is a good reason for communication problems)! Question is, why are the PCI configuration registers (partly) invalid?

In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device responds to various system power states. You need to be telling the system that you respond to an S3 transition by going into D3, which means you lose your mind. Then, when you send down your request to shift back to D09, the system should restore your configuration space.

You can see your current mappings in Device Manager, on the Details tab, in “Power State Mappings”.

Is there a minimum implementation required for supporting S0->S3->S0 state switching? At this moment the power IRPs are just forwarded.

Hell, yes, there is. When you get a request to transition to S3, you need to send a request to yourself to transition to D3. Then, when you get a request to transition back to S0, you need to send a request to yourself to transition to D0. That gets sent down to the PCI bus driver, which then realizes “hey, I need to restore this guy”.

This, by the way, is EXACTLY why you should be using KMDF instead of WDM.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

4

Thanks for your responses. I have some experience with WDM drivers, but not
with KMDF driver. In which way is the KMDF implementation really different
from the WDM implementation? And why would it solve my problem? I guess in
the KMDF way of working you also need a certain basic functionality to
implement, since the OS does not know anything about a custom card.

When I look at the pcidrv example that comes with the WDK, I see that
incoming IRPs are queued when the device is going to state D3. Is this
required? In my case the only thing I want is being able to communicate
with my card after getting back to S0. It is not necessary if there was an
application running (which communicates with the card) before going to
state S3 that it automatically continue. In other words, I can make a
requirement to the user that it must close all applications which uses the
card before it goes to state S3.

Is it enough to forward power IRPs to the lower level drivers and fill the
record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES as you
suggested)? Can you point me to an example how I have to react at
IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the
DEVICE_CAPABILITIES structure to tell the system I go to D3 when the system
enters S3 state (is this the DeviceState field of the structure)? For a lot
of fields the documentation indicates it should not written by the driver.

As you can see a lot of questions. I hope you have a little patience with
me. Writing/debugging drivers is not a daily business for me.

Thanks in advance,

Frank van Eijkelenburg

2013/1/24 Jake Oshins

> As usual, I like Tim’s answer. He’s right. Use KMDF and your immediate
> problem will certainly go away.
>
> But since you brought it up, we tend to see a lot of FPGA implementations
> that have a lot of trouble with power management. Many people implementing
> with FPGAs fail to meet the PCI-spec-mandated 100ms limit on initialization
> time. If your device’s configuration space doesn’t look exactly like it
> did before (with respect to device ID, vendor ID, subsystem ID, etc.) 100ms
> after reapplying power, you’ll see your driver stack torn down as failed.
>
> - Jake Oshins
> (sometime PCI guy)
> Windows Kernel Team
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:
> xxxxx@lists.osr.com] On Behalf Of Tim Roberts
> Sent: Thursday, January 24, 2013 9:56 AM
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] S3 to S0 state change for PCIe driver
>
> xxxxx@gmail.com wrote:
> > The card and driver are working properly in normal operation (state S0,
> fully powered). The problem I see, is triggered by entering the sleep state
> (S3) and wake up the system again (to S0). In state S3 the card is totally
> powered off. After waking up, the driver cannot communicate with the card
> anymore. The FPGAs (with a PCIe core inside) seems to be working properly
> (debug LEDs are blinking). If I compare the PCI Configuration registers, I
> can see the BAR addresses are reset to zero (which is a good reason for
> communication problems)! Question is, why are the PCI configuration
> registers (partly) invalid?
>
> In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device responds
> to various system power states. You need to be telling the system that you
> respond to an S3 transition by going into D3, which means you lose your
> mind. Then, when you send down your request to shift back to D09, the
> system should restore your configuration space.
>
> You can see your current mappings in Device Manager, on the Details tab,
> in “Power State Mappings”.
>
>
> > Is there a minimum implementation required for supporting S0->S3->S0
> state switching? At this moment the power IRPs are just forwarded.
>
> Hell, yes, there is. When you get a request to transition to S3, you need
> to send a request to yourself to transition to D3. Then, when you get a
> request to transition back to S0, you need to send a request to yourself to
> transition to D0. That gets sent down to the PCI bus driver, which then
> realizes “hey, I need to restore this guy”.
>
> This, by the way, is EXACTLY why you should be using KMDF instead of WDM.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

5

A typical correct implementation of power management and plug and play in a WDM driver is about 10,000 lines of code. (With that said, most implementations aren’t correct, and clock in around 3500 lines of code.) By using KMDF, you skip the part where you try to do a correct implementation and just delete all that code from your driver.

  • Jake Oshins
    Microsoft

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Frank van Eijkelenburg
Sent: Thursday, January 24, 2013 11:47 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

Thanks for your responses. I have some experience with WDM drivers, but not with KMDF driver. In which way is the KMDF implementation really different from the WDM implementation? And why would it solve my problem? I guess in the KMDF way of working you also need a certain basic functionality to implement, since the OS does not know anything about a custom card.

When I look at the pcidrv example that comes with the WDK, I see that incoming IRPs are queued when the device is going to state D3. Is this required? In my case the only thing I want is being able to communicate with my card after getting back to S0. It is not necessary if there was an application running (which communicates with the card) before going to state S3 that it automatically continue. In other words, I can make a requirement to the user that it must close all applications which uses the card before it goes to state S3.

Is it enough to forward power IRPs to the lower level drivers and fill the record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES as you suggested)? Can you point me to an example how I have to react at IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the DEVICE_CAPABILITIES structure to tell the system I go to D3 when the system enters S3 state (is this the DeviceState field of the structure)? For a lot of fields the documentation indicates it should not written by the driver.

As you can see a lot of questions. I hope you have a little patience with me. Writing/debugging drivers is not a daily business for me.

Thanks in advance,

Frank van Eijkelenburg
2013/1/24 Jake Oshins >
As usual, I like Tim’s answer. He’s right. Use KMDF and your immediate problem will certainly go away.

But since you brought it up, we tend to see a lot of FPGA implementations that have a lot of trouble with power management. Many people implementing with FPGAs fail to meet the PCI-spec-mandated 100ms limit on initialization time. If your device’s configuration space doesn’t look exactly like it did before (with respect to device ID, vendor ID, subsystem ID, etc.) 100ms after reapplying power, you’ll see your driver stack torn down as failed.

- Jake Oshins
(sometime PCI guy)
Windows Kernel Team

-----Original Message-----
From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.commailto:xxxxx] On Behalf Of Tim Roberts
Sent: Thursday, January 24, 2013 9:56 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

xxxxx@gmail.commailto:xxxxx wrote:
> The card and driver are working properly in normal operation (state S0, fully powered). The problem I see, is triggered by entering the sleep state (S3) and wake up the system again (to S0). In state S3 the card is totally powered off. After waking up, the driver cannot communicate with the card anymore. The FPGAs (with a PCIe core inside) seems to be working properly (debug LEDs are blinking). If I compare the PCI Configuration registers, I can see the BAR addresses are reset to zero (which is a good reason for communication problems)! Question is, why are the PCI configuration registers (partly) invalid?

In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device responds to various system power states. You need to be telling the system that you respond to an S3 transition by going into D3, which means you lose your mind. Then, when you send down your request to shift back to D09, the system should restore your configuration space.

You can see your current mappings in Device Manager, on the Details tab, in “Power State Mappings”.

> Is there a minimum implementation required for supporting S0->S3->S0 state switching? At this moment the power IRPs are just forwarded.

Hell, yes, there is. When you get a request to transition to S3, you need to send a request to yourself to transition to D3. Then, when you get a request to transition back to S0, you need to send a request to yourself to transition to D0. That gets sent down to the PCI bus driver, which then realizes “hey, I need to restore this guy”.

This, by the way, is EXACTLY why you should be using KMDF instead of WDM.

–
Tim Roberts, xxxxx@probo.commailto:xxxxx
Providenza & Boekelheide, Inc.

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

6

And you get the correct io queueing behavior. Even though you can say that you expect the application to quiesce during a system power transition, your driver must still be defensive against an ill behaving app (otherwise the app could make the OS crash or cause other nastiness by putting the driver into an unknown state). Furthermore, the app may not get a chance to be notified of a system power state transition. Rather than a complicated app/driver contract, just turn the problem around and say that io is queued during a power transition and then the app’s behavior doesn’t matter.

d

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Jake Oshins
Sent: Thursday, January 24, 2013 2:18 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] S3 to S0 state change for PCIe driver

A typical correct implementation of power management and plug and play in a WDM driver is about 10,000 lines of code. (With that said, most implementations aren’t correct, and clock in around 3500 lines of code.) By using KMDF, you skip the part where you try to do a correct implementation and just delete all that code from your driver.

  • Jake Oshins
    Microsoft

From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.com] On Behalf Of Frank van Eijkelenburg
Sent: Thursday, January 24, 2013 11:47 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

Thanks for your responses. I have some experience with WDM drivers, but not with KMDF driver. In which way is the KMDF implementation really different from the WDM implementation? And why would it solve my problem? I guess in the KMDF way of working you also need a certain basic functionality to implement, since the OS does not know anything about a custom card.

When I look at the pcidrv example that comes with the WDK, I see that incoming IRPs are queued when the device is going to state D3. Is this required? In my case the only thing I want is being able to communicate with my card after getting back to S0. It is not necessary if there was an application running (which communicates with the card) before going to state S3 that it automatically continue. In other words, I can make a requirement to the user that it must close all applications which uses the card before it goes to state S3.

Is it enough to forward power IRPs to the lower level drivers and fill the record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES as you suggested)? Can you point me to an example how I have to react at IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the DEVICE_CAPABILITIES structure to tell the system I go to D3 when the system enters S3 state (is this the DeviceState field of the structure)? For a lot of fields the documentation indicates it should not written by the driver.

As you can see a lot of questions. I hope you have a little patience with me. Writing/debugging drivers is not a daily business for me.

Thanks in advance,

Frank van Eijkelenburg
2013/1/24 Jake Oshins >
As usual, I like Tim’s answer. He’s right. Use KMDF and your immediate problem will certainly go away.

But since you brought it up, we tend to see a lot of FPGA implementations that have a lot of trouble with power management. Many people implementing with FPGAs fail to meet the PCI-spec-mandated 100ms limit on initialization time. If your device’s configuration space doesn’t look exactly like it did before (with respect to device ID, vendor ID, subsystem ID, etc.) 100ms after reapplying power, you’ll see your driver stack torn down as failed.

- Jake Oshins
(sometime PCI guy)
Windows Kernel Team

-----Original Message-----
From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.commailto:xxxxx] On Behalf Of Tim Roberts
Sent: Thursday, January 24, 2013 9:56 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

xxxxx@gmail.commailto:xxxxx wrote:
> The card and driver are working properly in normal operation (state S0, fully powered). The problem I see, is triggered by entering the sleep state (S3) and wake up the system again (to S0). In state S3 the card is totally powered off. After waking up, the driver cannot communicate with the card anymore. The FPGAs (with a PCIe core inside) seems to be working properly (debug LEDs are blinking). If I compare the PCI Configuration registers, I can see the BAR addresses are reset to zero (which is a good reason for communication problems)! Question is, why are the PCI configuration registers (partly) invalid?

In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device responds to various system power states. You need to be telling the system that you respond to an S3 transition by going into D3, which means you lose your mind. Then, when you send down your request to shift back to D09, the system should restore your configuration space.

You can see your current mappings in Device Manager, on the Details tab, in “Power State Mappings”.

> Is there a minimum implementation required for supporting S0->S3->S0 state switching? At this moment the power IRPs are just forwarded.

Hell, yes, there is. When you get a request to transition to S3, you need to send a request to yourself to transition to D3. Then, when you get a request to transition back to S0, you need to send a request to yourself to transition to D0. That gets sent down to the PCI bus driver, which then realizes “hey, I need to restore this guy”.

This, by the way, is EXACTLY why you should be using KMDF instead of WDM.

–
Tim Roberts, xxxxx@probo.commailto:xxxxx
Providenza & Boekelheide, Inc.

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

7

Frank van Eijkelenburg wrote:

Thanks for your responses. I have some experience with WDM drivers,
but not with KMDF driver. In which way is the KMDF implementation
really different from the WDM implementation? And why would it solve
my problem? I guess in the KMDF way of working you also need a certain
basic functionality to implement, since the OS does not know anything
about a custom card.

There has never been a WDM driver that handled power correctly. KMDF
brings you the proper state transitions.

The rule is that, when you get a power state transition request, you
need to do whatever is necessary to survive that transition. If you
have state that will be lost in an S0->S3 transition, you have to save
it. The PCI bus driver can help you with that; if it knows that your
device is making a D0->D3 transition, it will lose its configuration
space. Thus, when it sees that you are making a D3->D0 transition, it
will restore that space.

The KEY is that here is that it is up to the driver to say what D
transitions it is making. So, when you get an S0->S3 transition, if
your device responds to that by going into D3, then YOU need to create a
D0->D3 request and send it to yourself. Then, when you get an S3->S0
request, YOU need to create a D3->D0 request and send it to yourself.
That D3->D0 request gets forwarded down to the PCI bus driver, which
responds by restoring your configuration state.

It does that because it doesn’t KNOW exactly when you go into D3. For
example, a stupid device might go into D3 when it sees a transition to
S1. A device with an onboard battery might not need to go into D3 at
all. It can’t assume.

KMDF handles those transitions automatically, based on your
query_capabilities response.

When I look at the pcidrv example that comes with the WDK, I see that
incoming IRPs are queued when the device is going to state D3. Is this
required?

Can you handle I/O requests while you are in D3, when your device isn’t
responding? Few drivers can do so.

Again, this is something KMDF handles this for you automatically.

In my case the only thing I want is being able to communicate with my
card after getting back to S0. It is not necessary if there was an
application running (which communicates with the card) before going to
state S3 that it automatically continue. In other words, I can make a
requirement to the user that it must close all applications which uses
the card before it goes to state S3.

No, you can’t. Not really. Users don’t follow instructions. Batteries
die at inconvenient times. Lids close. Power timers expire. Systems
need to survive that. When a user’s app crashes after awaking, they
don’t blame your hardware. They blame Windows.

Is it enough to forward power IRPs to the lower level drivers and fill
the record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES
as you suggested)?

No, it’s not enough. You still have to create the D transition requests
yourself. (KMDF will do that.)

Can you point me to an example how I have to react at
IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the
DEVICE_CAPABILITIES structure to tell the system I go to D3 when the
system enters S3 state (is this the DeviceState field of the
structure)? For a lot of fields the documentation indicates it should
not written by the driver.

In the 7600 WDK, look at
src\general\toaster\wdm\func\featured2\power.c. It’s not correct, but
it’s close. You’ll see ToasterQueueCorrespondingDeviceIrp queues up a
device transition request corresponding to the current system transition
notice.

Again, by far your best choice would be to switch to KMDF. Really.

As you can see a lot of questions. I hope you have a little patience
with me. Writing/debugging drivers is not a daily business for me.

Patience is not one of the virtues of this mailing list. A thick skin
is a valuable asset while visiting here.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

8

I appreciate your explanation. I will take a look at the example. I am afraid that there is no budget for porting our existing driver to KMDF. However, I will recommend the KMDF model for the next driver we have to make.

By the way, don’t worry about my skin :wink:

Best regards,

Frank van Eijkelenburg

On 25 jan. 2013, at 19:01, Tim Roberts wrote:

> Frank van Eijkelenburg wrote:
>> Thanks for your responses. I have some experience with WDM drivers,
>> but not with KMDF driver. In which way is the KMDF implementation
>> really different from the WDM implementation? And why would it solve
>> my problem? I guess in the KMDF way of working you also need a certain
>> basic functionality to implement, since the OS does not know anything
>> about a custom card.
>
> There has never been a WDM driver that handled power correctly. KMDF
> brings you the proper state transitions.
>
> The rule is that, when you get a power state transition request, you
> need to do whatever is necessary to survive that transition. If you
> have state that will be lost in an S0->S3 transition, you have to save
> it. The PCI bus driver can help you with that; if it knows that your
> device is making a D0->D3 transition, it will lose its configuration
> space. Thus, when it sees that you are making a D3->D0 transition, it
> will restore that space.
>
> The KEY is that here is that it is up to the driver to say what D
> transitions it is making. So, when you get an S0->S3 transition, if
> your device responds to that by going into D3, then YOU need to create a
> D0->D3 request and send it to yourself. Then, when you get an S3->S0
> request, YOU need to create a D3->D0 request and send it to yourself.
> That D3->D0 request gets forwarded down to the PCI bus driver, which
> responds by restoring your configuration state.
>
> It does that because it doesn’t KNOW exactly when you go into D3. For
> example, a stupid device might go into D3 when it sees a transition to
> S1. A device with an onboard battery might not need to go into D3 at
> all. It can’t assume.
>
> KMDF handles those transitions automatically, based on your
> query_capabilities response.
>
>
>> When I look at the pcidrv example that comes with the WDK, I see that
>> incoming IRPs are queued when the device is going to state D3. Is this
>> required?
>
> Can you handle I/O requests while you are in D3, when your device isn’t
> responding? Few drivers can do so.
>
> Again, this is something KMDF handles this for you automatically.
>
>
>> In my case the only thing I want is being able to communicate with my
>> card after getting back to S0. It is not necessary if there was an
>> application running (which communicates with the card) before going to
>> state S3 that it automatically continue. In other words, I can make a
>> requirement to the user that it must close all applications which uses
>> the card before it goes to state S3.
>
> No, you can’t. Not really. Users don’t follow instructions. Batteries
> die at inconvenient times. Lids close. Power timers expire. Systems
> need to survive that. When a user’s app crashes after awaking, they
> don’t blame your hardware. They blame Windows.
>
>
>> Is it enough to forward power IRPs to the lower level drivers and fill
>> the record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES
>> as you suggested)?
>
> No, it’s not enough. You still have to create the D transition requests
> yourself. (KMDF will do that.)
>
>
>> Can you point me to an example how I have to react at
>> IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the
>> DEVICE_CAPABILITIES structure to tell the system I go to D3 when the
>> system enters S3 state (is this the DeviceState field of the
>> structure)? For a lot of fields the documentation indicates it should
>> not written by the driver.
>
> In the 7600 WDK, look at
> src\general\toaster\wdm\func\featured2\power.c. It’s not correct, but
> it’s close. You’ll see ToasterQueueCorrespondingDeviceIrp queues up a
> device transition request corresponding to the current system transition
> notice.
>
> Again, by far your best choice would be to switch to KMDF. Really.
>
>
>> As you can see a lot of questions. I hope you have a little patience
>> with me. Writing/debugging drivers is not a daily business for me.
>
> Patience is not one of the virtues of this mailing list. A thick skin
> is a valuable asset while visiting here.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

9

It’s clear that you simply don’t believe this. But I’ll say it anyway.

Completing your existing driver with a working power implementation is far more expensive than porting to KMDF. Your budget is going to be blown either way.

  • Jake Oshins

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Friday, January 25, 2013 11:52 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] S3 to S0 state change for PCIe driver

I appreciate your explanation. I will take a look at the example. I am afraid that there is no budget for porting our existing driver to KMDF. However, I will recommend the KMDF model for the next driver we have to make.

By the way, don’t worry about my skin :wink:

Best regards,

Frank van Eijkelenburg

On 25 jan. 2013, at 19:01, Tim Roberts wrote:

> Frank van Eijkelenburg wrote:
>> Thanks for your responses. I have some experience with WDM drivers,
>> but not with KMDF driver. In which way is the KMDF implementation
>> really different from the WDM implementation? And why would it solve
>> my problem? I guess in the KMDF way of working you also need a certain
>> basic functionality to implement, since the OS does not know anything
>> about a custom card.
>
> There has never been a WDM driver that handled power correctly. KMDF
> brings you the proper state transitions.
>
> The rule is that, when you get a power state transition request, you
> need to do whatever is necessary to survive that transition. If you
> have state that will be lost in an S0->S3 transition, you have to save
> it. The PCI bus driver can help you with that; if it knows that your
> device is making a D0->D3 transition, it will lose its configuration
> space. Thus, when it sees that you are making a D3->D0 transition, it
> will restore that space.
>
> The KEY is that here is that it is up to the driver to say what D
> transitions it is making. So, when you get an S0->S3 transition, if
> your device responds to that by going into D3, then YOU need to create a
> D0->D3 request and send it to yourself. Then, when you get an S3->S0
> request, YOU need to create a D3->D0 request and send it to yourself.
> That D3->D0 request gets forwarded down to the PCI bus driver, which
> responds by restoring your configuration state.
>
> It does that because it doesn’t KNOW exactly when you go into D3. For
> example, a stupid device might go into D3 when it sees a transition to
> S1. A device with an onboard battery might not need to go into D3 at
> all. It can’t assume.
>
> KMDF handles those transitions automatically, based on your
> query_capabilities response.
>
>
>> When I look at the pcidrv example that comes with the WDK, I see that
>> incoming IRPs are queued when the device is going to state D3. Is this
>> required?
>
> Can you handle I/O requests while you are in D3, when your device isn’t
> responding? Few drivers can do so.
>
> Again, this is something KMDF handles this for you automatically.
>
>
>> In my case the only thing I want is being able to communicate with my
>> card after getting back to S0. It is not necessary if there was an
>> application running (which communicates with the card) before going to
>> state S3 that it automatically continue. In other words, I can make a
>> requirement to the user that it must close all applications which uses
>> the card before it goes to state S3.
>
> No, you can’t. Not really. Users don’t follow instructions. Batteries
> die at inconvenient times. Lids close. Power timers expire. Systems
> need to survive that. When a user’s app crashes after awaking, they
> don’t blame your hardware. They blame Windows.
>
>
>> Is it enough to forward power IRPs to the lower level drivers and fill
>> the record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES
>> as you suggested)?
>
> No, it’s not enough. You still have to create the D transition requests
> yourself. (KMDF will do that.)
>
>
>> Can you point me to an example how I have to react at
>> IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the
>> DEVICE_CAPABILITIES structure to tell the system I go to D3 when the
>> system enters S3 state (is this the DeviceState field of the
>> structure)? For a lot of fields the documentation indicates it should
>> not written by the driver.
>
> In the 7600 WDK, look at
> src\general\toaster\wdm\func\featured2\power.c. It’s not correct, but
> it’s close. You’ll see ToasterQueueCorrespondingDeviceIrp queues up a
> device transition request corresponding to the current system transition
> notice.
>
> Again, by far your best choice would be to switch to KMDF. Really.
>
>
>> As you can see a lot of questions. I hope you have a little patience
>> with me. Writing/debugging drivers is not a daily business for me.
>
> Patience is not one of the virtues of this mailing list. A thick skin
> is a valuable asset while visiting here.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

10

How do WDM and WDF differ? Well, in one important way with respect to
your problem. WDF handles power management correctly, and WDM does not.
There are lots of other differences, but since your focus is getting power
management right, I’d think that using the only platform that actually
gets power management right would be a compelling reason.

I’m confused about why you should be caring about S-states, when devices
should only care about D-states. My understanding is that before there is
an S-state power reduction, the Power Manager first makes sure all the bus
drivers have handled their bus devices correctly, which means each bus
driver has managed the device power state of each device it controls.
Similarly, when there is an S-state power increase, te Power Manager will
then use te bus drivers to inform each of the devices about the desired
D-state. Maybe someone can clarify my understanding, but ten years ago, I
was doing power management in apps, and my recollection is this was what I
was seeing.

Thanks for your responses. I have some experience with WDM drivers, but
not
with KMDF driver. In which way is the KMDF implementation really different
from the WDM implementation? And why would it solve my problem? I guess in
the KMDF way of working you also need a certain basic functionality to
implement, since the OS does not know anything about a custom card.

But that’s why we write device drivers. The problem became one of
“programming in assembler” as it were; every driver was based on one of
the many incorrect driver examples. Unless the programmer wrote it from
scratch, referring only to the documents (which contradicted the actual
practice), so OF COURSE the kernel doesn’t know about custom cards, but
YOU don’t need to re-implement every single detail of queue management,
handling surprise removal, dealing with proper IRP cancellation, and
thousands of other lines of code that were supposed to be “common” among
all drivers, but weren’t. The difference now is like I would tell my MFC
students in app space: MFC lets you concentrate on the problem you want to
solve instead of worrying about massive irrelevant details whose
implementation must be precisely correct or your app isn’t going to “work
right”.

When I look at the pcidrv example that comes with the WDK, I see that
incoming IRPs are queued when the device is going to state D3. Is this
required? In my case the only thing I want is being able to communicate
with my card after getting back to S0. It is not necessary if there was an
application running (which communicates with the card) before going to
state S3 that it automatically continue. In other words, I can make a
requirement to the user that it must close all applications which uses the
card before it goes to state S3.

The fact that you have to ask this question should be sufficient
explanation about why you should be using KMDF. KMDF will worry about
this for you.

And the idea that the user has a clue when the OS is about to do a state
transition and therefore recognize that the apps should be shut down is at
best a fantasy. I walk away from my machine, leaving it running. While
I’m at dinner, there is a power failure in my neighborhood; the laptop is
left running on its battery. It needs to do an S0-S3 transition. I know
this? I’m four miles away at the pizza shop. So assume that this can
never work (and if your device can’t be installed on a laptop, then
substitute: my server, which has received a notification from its APC UPS
that the battery is low).

Is it enough to forward power IRPs to the lower level drivers and fill the
record with capabilities correctly (in IRP_MN_QUERY_CAPABILITIES as you
suggested)? Can you point me to an example how I have to react at
IRP_MN_QUERY_CAPABILITIES? Which fields should I use in the
DEVICE_CAPABILITIES structure to tell the system I go to D3 when the
system
enters S3 state (is this the DeviceState field of the structure)? For a
lot
of fields the documentation indicates it should not written by the driver.

As you can see a lot of questions. I hope you have a little patience with
me. Writing/debugging drivers is not a daily business for me.

Thanks in advance,

Frank van Eijkelenburg

2013/1/24 Jake Oshins
>
>> As usual, I like Tim’s answer. He’s right. Use KMDF and your immediate
>> problem will certainly go away.
>>
>> But since you brought it up, we tend to see a lot of FPGA
>> implementations
>> that have a lot of trouble with power management. Many people
>> implementing
>> with FPGAs fail to meet the PCI-spec-mandated 100ms limit on
>> initialization
>> time. If your device’s configuration space doesn’t look exactly like it
>> did before (with respect to device ID, vendor ID, subsystem ID, etc.)
>> 100ms
>> after reapplying power, you’ll see your driver stack torn down as
>> failed.
>>
>> - Jake Oshins
>> (sometime PCI guy)
>> Windows Kernel Team
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com [mailto:
>> xxxxx@lists.osr.com] On Behalf Of Tim Roberts
>> Sent: Thursday, January 24, 2013 9:56 AM
>> To: Windows System Software Devs Interest List
>> Subject: Re: [ntdev] S3 to S0 state change for PCIe driver
>>
>> xxxxx@gmail.com wrote:
>> > The card and driver are working properly in normal operation (state
>> S0,
>> fully powered). The problem I see, is triggered by entering the sleep
>> state
>> (S3) and wake up the system again (to S0). In state S3 the card is
>> totally
>> powered off. After waking up, the driver cannot communicate with the
>> card
>> anymore. The FPGAs (with a PCIe core inside) seems to be working
>> properly
>> (debug LEDs are blinking). If I compare the PCI Configuration registers,
>> I
>> can see the BAR addresses are reset to zero (which is a good reason for
>> communication problems)! Question is, why are the PCI configuration
>> registers (partly) invalid?
>>
>> In IRP_MN_QUERY_CAPABILITIES, you tell the system how your device
>> responds
>> to various system power states. You need to be telling the system that
>> you
>> respond to an S3 transition by going into D3, which means you lose your
>> mind. Then, when you send down your request to shift back to D09, the
>> system should restore your configuration space.
>>
>> You can see your current mappings in Device Manager, on the Details tab,
>> in “Power State Mappings”.
>>
>>
>> > Is there a minimum implementation required for supporting S0->S3->S0
>> state switching? At this moment the power IRPs are just forwarded.
>>
>> Hell, yes, there is. When you get a request to transition to S3, you
>> need
>> to send a request to yourself to transition to D3. Then, when you get a
>> request to transition back to S0, you need to send a request to yourself
>> to
>> transition to D0. That gets sent down to the PCI bus driver, which then
>> realizes “hey, I need to restore this guy”.
>>
>> This, by the way, is EXACTLY why you should be using KMDF instead of
>> WDM.
>>
>> –
>> Tim Roberts, xxxxx@probo.com
>> Providenza & Boekelheide, Inc.
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> OSR is HIRING!! See http://www.osr.com/careers
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> OSR is HIRING!! See http://www.osr.com/careers
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer