Re: [ntdev] SRB_FLAGS_BYPASS_LOCKED_QUEUE Question

Note that some modern storage controllers can handle dozens or even hundreds of concurrent requests and excessive serialization can kill performance. Think about a FC controller connected to a SAN or flash array

Sent from Surface Pro

From: Peter Wieland
Sent: ‎Monday‎, ‎March‎ ‎16‎, ‎2015 ‎3‎:‎11‎ ‎PM
To: Windows System Software Devs Interest List

I think it would help you get a better answer if you take a step back and explain what type of device driver you’re building and what you’re trying to do. I’ve tried to parse the last couple of questions and they don’t make tons of sense to me. It seems like you’re probably trying to ask too narrow of a question.

For example, I don’t know if you’re trying to write a storage controller driver (like a storport or scsciport miniport) or a storage device driver (e.g. a disk). You mention there’s a miniport here but I don’t know if you’re writing that, or a driver above it.

Generally the storage controller drivers have two layers of queue within the controller driver. Each LUN has its own queue of requests, and the controller has a queue of requests. The controller decides how many requests to issue from each LUN queue into the controller queue at any given time. Generally that count is either 0 or 1. Once the active request for the LUN has been sent to the controller, the controller driver may decide to admit another request from that LUN.

In the case of SCSIPORT (which is when I last had my hands actively in this code) the LUNs issue one request at a time to the controller queue. Once the miniport has received the request, the LUN will issue another request if the previous request was a tagged command and the next request is also a tagged command, and the number of active tagged requests on the device didn’t exceed what we thought was the LUN’s limit on active tagged commands, and none of the various internal and/or SRB flags which tell us we can’t do tagged commands right now aren’t set.

SRB_FUNCTION_LOCK (or lock_queue – can’t quite remember) is intended to lock the top-level per-LUN queue. It’s used primarily during power or PNP transitions so that the disk driver can stop the flow of requests to the LUN without maintaining its own gate (adding a gate at the disk layer would introduce more interlocked instructions or spinlocks, and they’re going to be done again in the controller driver – why take the extra cost). The expectation was that the drivers in the device stack would only send these command during things like power/PNP transitions, where we could use the transition IRP as the talking-stick. Whoever owns the power IRP is allowed to decide to LOCK, then issue their bypass commands, then they hand the talking-stick to the next driver in the stack. As the talking-stick goes back up, each driver might send more commands (for power-up you do it as the Power IRP completes) and then every driver which sent a LOCK does an UNLOCK.

The purpose of the LOCK is to resolve the inherent race between steady-state IO (read/write) and the transition out of that state without introducing more kernel locks at every layer in the stack. It is not to resolve race conditions between multiple drivers trying to do some sort of state transition – you need another request to serve as the talking-stick there.

If you have multiple drivers in the stack, both sending reads and writes concurrently to the same LUN the queues in the controller driver will be fine. If you need to introduce some specific ordering between read and write requests coming through the driver stack, you will need a per-LUN lower filter driver to impose that ordering.

-p

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Mike Hallard
Sent: Sunday, March 15, 2015 3:51 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] SRB_FLAGS_BYPASS_LOCKED_QUEUE Question

Well basically, I have 2 devices accessing the same shared port (via IoStartPacket), but I believe that because the devices have their own individual queues, it will still mess with the serialization of DriverStartIo? So my assumption is I need to implement my own serialization so that both devices don’t call DriverStartIo at the same time? Am I correct and if so, any suggestions on how to do this? Filter driver is out of the question, because this is a miniport.

On Sun, Mar 15, 2015 at 2:13 PM, wrote:

Other driver can (and do) set the flag which will result in requests being intermingled. You are not understanding what this lock is all about–it’s intended for cleaning out pending requests so the class driver can get back in a good state after a check condition has occurred. If you want to suspend SRBs from all others, you may need a filter driver to do that.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Is backwards compatibility a major issue for a research tool?

Sent from Surface Pro

From: Mike Hallard
Sent: ‎Monday‎, ‎March‎ ‎16‎, ‎2015 ‎4‎:‎10‎ ‎PM
To: Windows System Software Devs Interest List

I did want to refrain from going into details, as what I’m doing is probably considered bad juju, but I’ll give it a try anyway. On windows 7 I have found a way to bypass any malicious filter drivers, object hijacking, and inline hooks that would prevent reading of infected boot sectors: I’m basically implementing my own storage class driver and setting up a parallel miniport instead of risking unhooking the real one (This seems to work flawlessly with ataport & scsiport due to their internal queuing mechanism).

My problem here is Windows XP (Yay, backwards compatibility), which uses IoStartPacket for queuing to the device queue. Having 2 parallel miniport device objects on XP would mean 2 separate device queues, which would violate the serialization of DriverStartIo; and if I queue the request to the original miniport with IoStartPacket (passing my clean miniport’s device object), it will work if the request is sent directly to DriverStartIo (the queue is empty), but if it gets queued, the request is then later processed using the original device object (which results in any miniport hooks getting control of the request).

I know this kind of thing is usually frowned upon, but it’s just a research tool similar to the Hidden FS Readers, aimed at easier dumping of infected boot sectors when performing dynamic malware analysis.

On Mon, Mar 16, 2015 at 7:11 PM, Peter Wieland wrote:

I think it would help you get a better answer if you take a step back and explain what type of device driver you’re building and what you’re trying to do. I’ve tried to parse the last couple of questions and they don’t make tons of sense to me. It seems like you’re probably trying to ask too narrow of a question.

For example, I don’t know if you’re trying to write a storage controller driver (like a storport or scsciport miniport) or a storage device driver (e.g. a disk). You mention there’s a miniport here but I don’t know if you’re writing that, or a driver above it.

Generally the storage controller drivers have two layers of queue within the controller driver. Each LUN has its own queue of requests, and the controller has a queue of requests. The controller decides how many requests to issue from each LUN queue into the controller queue at any given time. Generally that count is either 0 or 1. Once the active request for the LUN has been sent to the controller, the controller driver may decide to admit another request from that LUN.

In the case of SCSIPORT (which is when I last had my hands actively in this code) the LUNs issue one request at a time to the controller queue. Once the miniport has received the request, the LUN will issue another request if the previous request was a tagged command and the next request is also a tagged command, and the number of active tagged requests on the device didn’t exceed what we thought was the LUN’s limit on active tagged commands, and none of the various internal and/or SRB flags which tell us we can’t do tagged commands right now aren’t set.

SRB_FUNCTION_LOCK (or lock_queue – can’t quite remember) is intended to lock the top-level per-LUN queue. It’s used primarily during power or PNP transitions so that the disk driver can stop the flow of requests to the LUN without maintaining its own gate (adding a gate at the disk layer would introduce more interlocked instructions or spinlocks, and they’re going to be done again in the controller driver – why take the extra cost). The expectation was that the drivers in the device stack would only send these command during things like power/PNP transitions, where we could use the transition IRP as the talking-stick. Whoever owns the power IRP is allowed to decide to LOCK, then issue their bypass commands, then they hand the talking-stick to the next driver in the stack. As the talking-stick goes back up, each driver might send more commands (for power-up you do it as the Power IRP completes) and then every driver which sent a LOCK does an UNLOCK.

The purpose of the LOCK is to resolve the inherent race between steady-state IO (read/write) and the transition out of that state without introducing more kernel locks at every layer in the stack. It is not to resolve race conditions between multiple drivers trying to do some sort of state transition – you need another request to serve as the talking-stick there.

If you have multiple drivers in the stack, both sending reads and writes concurrently to the same LUN the queues in the controller driver will be fine. If you need to introduce some specific ordering between read and write requests coming through the driver stack, you will need a per-LUN lower filter driver to impose that ordering.

-p

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Mike Hallard
Sent: Sunday, March 15, 2015 3:51 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] SRB_FLAGS_BYPASS_LOCKED_QUEUE Question

Well basically, I have 2 devices accessing the same shared port (via IoStartPacket), but I believe that because the devices have their own individual queues, it will still mess with the serialization of DriverStartIo? So my assumption is I need to implement my own serialization so that both devices don’t call DriverStartIo at the same time? Am I correct and if so, any suggestions on how to do this? Filter driver is out of the question, because this is a miniport.

On Sun, Mar 15, 2015 at 2:13 PM, wrote:

Other driver can (and do) set the flag which will result in requests being intermingled. You are not understanding what this lock is all about–it’s intended for cleaning out pending requests so the class driver can get back in a good state after a check condition has occurred. If you want to suspend SRBs from all others, you may need a filter driver to do that.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev OSR is HIRING!! See http://www.osr.com/careers For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer