SRB execution and completion order

NTDEV
1

Hi,

more theoretical that practical question. Should storage driver keep the
order of SRBs when completing requests? Example. We have SCSI miniport or
SCSI port driver (does not matter). In pool of worker threads we dequeue
the SRBs (either from own queue in case of SCSI port or from SCSIPORT
queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4. We do
execute them all 4 in parallel. As different SRBs can take different time
to process (“write10” for megabyte block of data will be executed slower
then “test unit ready”) they will have different completion time. Should
storage driver call completion code in the order it got SRBs from queue
(1, 2, 3, 4) or it can call completion code just after SRBs get processed
by lower layer (hardware or software) - 4, 2, 1, 3 for example? We did
second variant (calling completion code after SRB get processed) and I’ve
never seen in any storage driver from DDK any logic to do completion in
the orders SRBs get dequeued. But one of our customers want us to keep the
conpletion order identical to original queue order. Who’s correct?

Thank you very much for help!

Anton Kolomyeytsev

2

no - storage drivers should not need to keep completion order. Anyone
who wants to have guaranteed order of operation or of completion needs
to be issuing the operations one at a time.

-p

-----Original Message-----
From: Anton Kolomyeytsev [mailto:xxxxx@cooldev.com]
Sent: Friday, October 18, 2002 11:06 AM
To: NT Developers Interest List
Subject: [ntdev] SRB execution and completion order

Hi,

more theoretical that practical question. Should storage driver keep the
order of SRBs when completing requests? Example. We have SCSI miniport
or SCSI port driver (does not matter). In pool of worker threads we
dequeue the SRBs (either from own queue in case of SCSI port or from
SCSIPORT queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4.
We do execute them all 4 in parallel. As different SRBs can take
different time to process (“write10” for megabyte block of data will be
executed slower then “test unit ready”) they will have different
completion time. Should storage driver call completion code in the order
it got SRBs from queue (1, 2, 3, 4) or it can call completion code just
after SRBs get processed by lower layer (hardware or software) - 4, 2,
1, 3 for example? We did second variant (calling completion code after
SRB get processed) and I’ve never seen in any storage driver from DDK
any logic to do completion in the orders SRBs get dequeued. But one of
our customers want us to keep the conpletion order identical to original
queue order. Who’s correct?

Thank you very much for help!

Anton Kolomyeytsev

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to %%email.unsub%%

3

I can not imagine someone sending a TUR and an IO before waiting for the
TUR to complete. Sorta breaks the whole idea of a TUR; eh?

Jamey

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Friday, October 18, 2002 11:06 AM
To: NT Developers Interest List
Subject: [ntdev] RE: SRB execution and completion order

no - storage drivers should not need to keep completion order. Anyone
who wants to have guaranteed order of operation or of completion needs
to be issuing the operations one at a time.

-p

-----Original Message-----
From: Anton Kolomyeytsev [mailto:xxxxx@cooldev.com]
Sent: Friday, October 18, 2002 11:06 AM
To: NT Developers Interest List
Subject: [ntdev] SRB execution and completion order

Hi,

more theoretical that practical question. Should storage driver keep the
order of SRBs when completing requests? Example. We have SCSI miniport
or SCSI port driver (does not matter). In pool of worker threads we
dequeue the SRBs (either from own queue in case of SCSI port or from
SCSIPORT queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4.
We do execute them all 4 in parallel. As different SRBs can take
different time to process (“write10” for megabyte block of data will be
executed slower then “test unit ready”) they will have different
completion time. Should storage driver call completion code in the order
it got SRBs from queue (1, 2, 3, 4) or it can call completion code just
after SRBs get processed by lower layer (hardware or software) - 4, 2,
1, 3 for example? We did second variant (calling completion code after
SRB get processed) and I’ve never seen in any storage driver from DDK
any logic to do completion in the orders SRBs get dequeued. But one of
our customers want us to keep the conpletion order identical to original
queue order. Who’s correct?

Thank you very much for help!

Anton Kolomyeytsev

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to %%email.unsub%%

You are currently subscribed to ntdev as: xxxxx@storagecraft.com
To unsubscribe send a blank email to %%email.unsub%%

4

As for IO, you can sort the reads and writes by there location on disk.
Most disks use some form of CSCAN, but doing it in the driver can also
gain some performance.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jamey Kirby
Sent: Friday, October 18, 2002 11:19 AM
To: NT Developers Interest List
Subject: [ntdev] RE: SRB execution and completion order

I can not imagine someone sending a TUR and an IO before waiting for the
TUR to complete. Sorta breaks the whole idea of a TUR; eh?

Jamey

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Friday, October 18, 2002 11:06 AM
To: NT Developers Interest List
Subject: [ntdev] RE: SRB execution and completion order

no - storage drivers should not need to keep completion order. Anyone
who wants to have guaranteed order of operation or of completion needs
to be issuing the operations one at a time.

-p

-----Original Message-----
From: Anton Kolomyeytsev [mailto:xxxxx@cooldev.com]
Sent: Friday, October 18, 2002 11:06 AM
To: NT Developers Interest List
Subject: [ntdev] SRB execution and completion order

Hi,

more theoretical that practical question. Should storage driver keep the
order of SRBs when completing requests? Example. We have SCSI miniport
or SCSI port driver (does not matter). In pool of worker threads we
dequeue the SRBs (either from own queue in case of SCSI port or from
SCSIPORT queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4.
We do execute them all 4 in parallel. As different SRBs can take
different time to process (“write10” for megabyte block of data will be
executed slower then “test unit ready”) they will have different
completion time. Should storage driver call completion code in the order
it got SRBs from queue (1, 2, 3, 4) or it can call completion code just
after SRBs get processed by lower layer (hardware or software) - 4, 2,
1, 3 for example? We did second variant (calling completion code after
SRB get processed) and I’ve never seen in any storage driver from DDK
any logic to do completion in the orders SRBs get dequeued. But one of
our customers want us to keep the conpletion order identical to original
queue order. Who’s correct?

Thank you very much for help!

Anton Kolomyeytsev

You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to %%email.unsub%%

You are currently subscribed to ntdev as: xxxxx@storagecraft.com
To unsubscribe send a blank email to %%email.unsub%%

You are currently subscribed to ntdev as: xxxxx@storagecraft.com
To unsubscribe send a blank email to %%email.unsub%%

5

No ordering requirement is imposed by SCSI spec, especially given the
fact that there is a thing called tagged queue.
Thus, order is not important (but the SCSI sync issues - either N
tagged or 1 untagged request - must be maintained by the port driver,
as must contingent allegiance).

If class driver wants order - it must a) send first SRB b) wait for
its completion c) send second SRB.

Max

----- Original Message -----
From: “Anton Kolomyeytsev”
To: “NT Developers Interest List”
Sent: Friday, October 18, 2002 10:05 PM
Subject: [ntdev] SRB execution and completion order

> Hi,
>
> more theoretical that practical question. Should storage driver keep
the
> order of SRBs when completing requests? Example. We have SCSI
miniport or
> SCSI port driver (does not matter). In pool of worker threads we
dequeue
> the SRBs (either from own queue in case of SCSI port or from
SCSIPORT
> queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4. We
do
> execute them all 4 in parallel. As different SRBs can take different
time
> to process (“write10” for megabyte block of data will be executed
slower
> then “test unit ready”) they will have different completion time.
Should
> storage driver call completion code in the order it got SRBs from
queue
> (1, 2, 3, 4) or it can call completion code just after SRBs get
processed
> by lower layer (hardware or software) - 4, 2, 1, 3 for example? We
did
> second variant (calling completion code after SRB get processed) and
I’ve
> never seen in any storage driver from DDK any logic to do completion
in
> the orders SRBs get dequeued. But one of our customers want us to
keep the
> conpletion order identical to original queue order. Who’s correct?
>
> Thank you very much for help!
>
> Anton Kolomyeytsev
>
> —
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to %%email.unsub%%
>

6

Hi,

thank you all very much for your responses. They really helped.

One more question about this topic. If serie of SRBs were submitted to
storage stack for execution is there any guarantee then SCSI device will
get them all in the order they (SRBs) were submitted? Can application that
sends SRBs be sure this will happed or does it have to send SRBs
one-by-one still?

Thanks!

Anton Kolomyeytsev

P.S. I’ve experimented myself. Seems that SCSIPORT does not alter SCSI
commands order but AFAIK this does not proove anything - can be subject to
change…

No ordering requirement is imposed by SCSI spec, especially given the
fact that there is a thing called tagged queue.
Thus, order is not important (but the SCSI sync issues - either N
tagged or 1 untagged request - must be maintained by the port driver,
as must contingent allegiance).

If class driver wants order - it must a) send first SRB b) wait for
its completion c) send second SRB.

Max

----- Original Message -----
From: “Anton Kolomyeytsev”
> To: “NT Developers Interest List”
> Sent: Friday, October 18, 2002 10:05 PM
> Subject: [ntdev] SRB execution and completion order
>
>
> > Hi,
> >
> > more theoretical that practical question. Should storage driver keep
> the
> > order of SRBs when completing requests? Example. We have SCSI
> miniport or
> > SCSI port driver (does not matter). In pool of worker threads we
> dequeue
> > the SRBs (either from own queue in case of SCSI port or from
> SCSIPORT
> > queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4. We
> do
> > execute them all 4 in parallel. As different SRBs can take different
> time
> > to process (“write10” for megabyte block of data will be executed
> slower
> > then “test unit ready”) they will have different completion time.
> Should
> > storage driver call completion code in the order it got SRBs from
> queue
> > (1, 2, 3, 4) or it can call completion code just after SRBs get
> processed
> > by lower layer (hardware or software) - 4, 2, 1, 3 for example? We
> did
> > second variant (calling completion code after SRB get processed) and
> I’ve
> > never seen in any storage driver from DDK any logic to do completion
> in
> > the orders SRBs get dequeued. But one of our customers want us to
> keep the
> > conpletion order identical to original queue order. Who’s correct?
> >
> > Thank you very much for help!
> >
> > Anton Kolomyeytsev
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to %%email.unsub%%
> >

7

no - there is no guarantee as to what order multiple outstanding
requests will be sent to the device. Requests can be reordered by the
class, port or miniport.

if order of operation is important to the app then it needs to send the
requests one at a time.

-p

-----Original Message-----
From: Anton Kolomyeytsev [mailto:xxxxx@cooldev.com]
Sent: Monday, October 21, 2002 2:05 PM
To: NT Developers Interest List
Subject: [ntdev] Re: SRB execution and completion order

Hi,

thank you all very much for your responses. They really helped.

One more question about this topic. If serie of SRBs were submitted to
storage stack for execution is there any guarantee then SCSI device will
get them all in the order they (SRBs) were submitted? Can application
that sends SRBs be sure this will happed or does it have to send SRBs
one-by-one still?

Thanks!

Anton Kolomyeytsev

P.S. I’ve experimented myself. Seems that SCSIPORT does not alter SCSI
commands order but AFAIK this does not proove anything - can be subject
to change…

No ordering requirement is imposed by SCSI spec, especially given the
fact that there is a thing called tagged queue. Thus, order is not
important (but the SCSI sync issues - either N tagged or 1 untagged
request - must be maintained by the port driver, as must contingent
allegiance).

If class driver wants order - it must a) send first SRB b) wait for
its completion c) send second SRB.

Max

----- Original Message -----
From: “Anton Kolomyeytsev”
> To: “NT Developers Interest List”
> Sent: Friday, October 18, 2002 10:05 PM
> Subject: [ntdev] SRB execution and completion order
>
>
> > Hi,
> >
> > more theoretical that practical question. Should storage driver keep
> the
> > order of SRBs when completing requests? Example. We have SCSI
> miniport or
> > SCSI port driver (does not matter). In pool of worker threads we
> dequeue
> > the SRBs (either from own queue in case of SCSI port or from
> SCSIPORT
> > queue in case of SCSI miniport). SRBs are numbered. 1, 2, 3, 4. We
> do
> > execute them all 4 in parallel. As different SRBs can take different
> time
> > to process (“write10” for megabyte block of data will be executed
> slower
> > then “test unit ready”) they will have different completion time.
> Should
> > storage driver call completion code in the order it got SRBs from
> queue
> > (1, 2, 3, 4) or it can call completion code just after SRBs get
> processed
> > by lower layer (hardware or software) - 4, 2, 1, 3 for example? We
> did
> > second variant (calling completion code after SRB get processed) and
> I’ve
> > never seen in any storage driver from DDK any logic to do completion
> in
> > the orders SRBs get dequeued. But one of our customers want us to
> keep the
> > conpletion order identical to original queue order. Who’s correct?
> >
> > Thank you very much for help!
> >
> > Anton Kolomyeytsev
> >
> > —
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com To
> > unsubscribe send a blank email to %%email.unsub%%
> >

—
You are currently subscribed to ntdev as: xxxxx@microsoft.com To
unsubscribe send a blank email to %%email.unsub%%

8

> One more question about this topic. If serie of SRBs were submitted
to

storage stack for execution is there any guarantee then SCSI device
will
get them all in the order they (SRBs) were submitted? Can
application that
sends SRBs be sure this will happed

No.

or does it have to send SRBs
one-by-one still?

Yes.

Max