IRP cloning and mirrored writes...

Hello,

I am currently porting a pseudo disk driver to NT from Unix of which one
aspect of its operation is to perform a mirrored write. The manner in
which it performs the mirrored write is to clone the original IRP for
each mirror and then issue the cloned IRPs to the primary and secondary
devices. During the mirrored write the original IRP is stalled. After
the cloned requests complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the primary device
the driver seems to work well (as evidenced by a data integrity tool
I’m using).

However, if I instruct the driver to perform mirroring, the data sent
to the primary device is always corrupted (I am currently unable to
verify the secondary as it is not written in the original format).

Here’s is the gist of my IRP clone operation. After calling the
cloning macro a cloned IRP is further modiefied to point to the
correct device by directly setting the DeviceObject field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp() rather
than by IoAllocateIrp().

It’s probably worth admitting that while I have programmed drivers for
Unix for many years I’ve only been working with NT for a month or so.

I would greatly appreciate any insight into how to solve this problem
correctly. I have scoured this list and the web for similar examples
but haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset) \
do { \
IO_STACK_LOCATION *srcstk, *dststk; \
srcstk = IoGetCurrentIrpStackLocation(src); \
dststk = IoGetCurrentIrpStackLocation(dst); \
*dststk = *srcstk; \
dststk->Context = (void *)0; \
(dst)->MdlAddress = IoAllocateMdl( \
MmGetMdlVirtualAddress((src)->MdlAddress), \
MmGetMdlByteCount((src)->MdlAddress), \
FALSE, FALSE, (IRP *)0); \
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress, \
(char *)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0); \
(dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread; \
} while (0)

You can’t just copy the entire stack location, as this contains the
completion routine for the previous driver. Unless you override this,
when the IRP completes it will run the other driver’s completion routine
with the wrong IRP. Look at the IoCopyCurrentIrpStackLocationToNext
macro to see how to copy an IRP.

When you say that the data on the second disk “is not written in the
original format” I’m curious what you mean. Are you modifying the data?
If so remember that the second I/O and the first I/O are using views of
the same physical pages so modifying them for one I/O will modify them
for the other (it will also change the data in the client’s address
space, since that’s another mapping, and corrupt the client - you must
not modify client writes in place).

Why are you making the IRPs by hand rather than going through
IoAllocateIrp? Where is their memory coming from?

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 1:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix
of which one aspect of its operation is to perform a mirrored
write. The manner in which it performs the mirrored write is
to clone the original IRP for each mirror and then issue the
cloned IRPs to the primary and secondary devices. During the
mirrored write the original IRP is stalled. After the cloned
requests complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the
primary device the driver seems to work well (as evidenced by
a data integrity tool I’m using).

However, if I instruct the driver to perform mirroring, the
data sent to the primary device is always corrupted (I am
currently unable to verify the secondary as it is not written
in the original format).

Here’s is the gist of my IRP clone operation. After calling
the cloning macro a cloned IRP is further modiefied to point
to the correct device by directly setting the DeviceObject
field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp()
rather than by IoAllocateIrp().

It’s probably worth admitting that while I have programmed
drivers for Unix for many years I’ve only been working with
NT for a month or so.

I would greatly appreciate any insight into how to solve this
problem correctly. I have scoured this list and the web for
similar examples but haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset)
\
do {
\
IO_STACK_LOCATION *srcstk, *dststk;
\
srcstk = IoGetCurrentIrpStackLocation(src);
\
dststk = IoGetCurrentIrpStackLocation(dst);
\
*dststk = *srcstk;
\
dststk->Context = (void *)0;
\
(dst)->MdlAddress = IoAllocateMdl(
\
MmGetMdlVirtualAddress((src)->MdlAddress),
\
MmGetMdlByteCount((src)->MdlAddress),
\
FALSE, FALSE, (IRP *)0);
\
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress,
\
(char
*)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0);
\
(dst)->Tail.Overlay.Thread =
(src)->Tail.Overlay.Thread; \
} while (0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@windows.microsoft.com To unsubscribe send a blank
email to xxxxx@lists.osr.com

Also, for a PIRP initialized by IoInitializeIrp or IoAllocateIrp, there
is no current stack location. Do you mean to format the next stack
location instead? Unless you are processing the irp internally and
never sending it via IoCallDriver, setting the DeviceObject in the stack
location is not necessary since IoCallDriver will do that for you.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Peter Wieland
Sent: Thursday, January 06, 2005 1:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] IRP cloning and mirrored writes…

You can’t just copy the entire stack location, as this contains the
completion routine for the previous driver. Unless you override this,
when the IRP completes it will run the other driver’s completion routine
with the wrong IRP. Look at the IoCopyCurrentIrpStackLocationToNext
macro to see how to copy an IRP.

When you say that the data on the second disk “is not written in the
original format” I’m curious what you mean. Are you modifying the data?
If so remember that the second I/O and the first I/O are using views of
the same physical pages so modifying them for one I/O will modify them
for the other (it will also change the data in the client’s address
space, since that’s another mapping, and corrupt the client - you must
not modify client writes in place).

Why are you making the IRPs by hand rather than going through
IoAllocateIrp? Where is their memory coming from?

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 1:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix
of which one aspect of its operation is to perform a mirrored
write. The manner in which it performs the mirrored write is
to clone the original IRP for each mirror and then issue the
cloned IRPs to the primary and secondary devices. During the
mirrored write the original IRP is stalled. After the cloned
requests complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the
primary device the driver seems to work well (as evidenced by
a data integrity tool I’m using).

However, if I instruct the driver to perform mirroring, the
data sent to the primary device is always corrupted (I am
currently unable to verify the secondary as it is not written
in the original format).

Here’s is the gist of my IRP clone operation. After calling
the cloning macro a cloned IRP is further modiefied to point
to the correct device by directly setting the DeviceObject
field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp()
rather than by IoAllocateIrp().

It’s probably worth admitting that while I have programmed
drivers for Unix for many years I’ve only been working with
NT for a month or so.

I would greatly appreciate any insight into how to solve this
problem correctly. I have scoured this list and the web for
similar examples but haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset)
\
do {
\
IO_STACK_LOCATION *srcstk, *dststk;
\
srcstk = IoGetCurrentIrpStackLocation(src);
\
dststk = IoGetCurrentIrpStackLocation(dst);
\
*dststk = *srcstk;
\
dststk->Context = (void *)0;
\
(dst)->MdlAddress = IoAllocateMdl(
\
MmGetMdlVirtualAddress((src)->MdlAddress),
\
MmGetMdlByteCount((src)->MdlAddress),
\
FALSE, FALSE, (IRP *)0);
\
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress,
\
(char
*)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0);
\
(dst)->Tail.Overlay.Thread =
(src)->Tail.Overlay.Thread; \
} while (0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@windows.microsoft.com To unsubscribe send a blank
email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Well IoBuildPartialMdl works just fine so, big surprise, you’ve got bugs!

IoGetCurrentIrpStackLocation? The next stack location is found by
IoSetNextIrpStackLocation and then calling IoGetCurrentIrpStackLocation that
is the one that needs setting up.

Why not just use IoAllocateIrp?

Copying stack locations as you do is error prone. See the
IoCopyCurrentIrpStackLocationToNext macro for details. Mostly you don’t need
to setup much more than the MajorFunction and the Parameters.

Make sure your ‘waiting for both Irps to complete asynchronously’ logic is
not a bit on the bogus side. You cannot make any ordering assumptions.

=====================
Mark Roddy

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 4:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix of which one
aspect of its operation is to perform a mirrored write. The manner in which
it performs the mirrored write is to clone the original IRP for each mirror
and then issue the cloned IRPs to the primary and secondary devices. During
the mirrored write the original IRP is stalled. After the cloned requests
complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the primary device the
driver seems to work well (as evidenced by a data integrity tool I’m using).

However, if I instruct the driver to perform mirroring, the data sent to the
primary device is always corrupted (I am currently unable to verify the
secondary as it is not written in the original format).

Here’s is the gist of my IRP clone operation. After calling the cloning
macro a cloned IRP is further modiefied to point to the correct device by
directly setting the DeviceObject field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp() rather than by
IoAllocateIrp().

It’s probably worth admitting that while I have programmed drivers for Unix
for many years I’ve only been working with NT for a month or so.

I would greatly appreciate any insight into how to solve this problem
correctly. I have scoured this list and the web for similar examples but
haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset) \
do { \
IO_STACK_LOCATION *srcstk, *dststk; \
srcstk = IoGetCurrentIrpStackLocation(src); \
dststk = IoGetCurrentIrpStackLocation(dst); \
*dststk = *srcstk; \
dststk->Context = (void *)0; \
(dst)->MdlAddress = IoAllocateMdl( \
MmGetMdlVirtualAddress((src)->MdlAddress), \
MmGetMdlByteCount((src)->MdlAddress), \
FALSE, FALSE, (IRP *)0); \
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress, \
(char *)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0); \
(dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread; \
} while (0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@stratus.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

If you are going to send the PIRP through IoCallDriver(), then

IoSetNextIrpStackLocation
IoGetCurrentIrpStackLocation

Does NOT format the stack location that will be sent to the next device
object. (All you are doing is setting a stack location up that can be
accessible in the completion routine :)). IoGetNextIrpStackLocation sets
up the next stack location for the next device in the call path.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roddy, Mark
Sent: Thursday, January 06, 2005 1:43 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] IRP cloning and mirrored writes…

Well IoBuildPartialMdl works just fine so, big surprise, you’ve got
bugs!

IoGetCurrentIrpStackLocation? The next stack location is found by
IoSetNextIrpStackLocation and then calling IoGetCurrentIrpStackLocation
that
is the one that needs setting up.

Why not just use IoAllocateIrp?

Copying stack locations as you do is error prone. See the
IoCopyCurrentIrpStackLocationToNext macro for details. Mostly you don’t
need
to setup much more than the MajorFunction and the Parameters.

Make sure your ‘waiting for both Irps to complete asynchronously’ logic
is
not a bit on the bogus side. You cannot make any ordering assumptions.

=====================
Mark Roddy

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 4:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix of which one
aspect of its operation is to perform a mirrored write. The manner in
which
it performs the mirrored write is to clone the original IRP for each
mirror
and then issue the cloned IRPs to the primary and secondary devices.
During
the mirrored write the original IRP is stalled. After the cloned
requests
complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the primary device
the
driver seems to work well (as evidenced by a data integrity tool I’m
using).

However, if I instruct the driver to perform mirroring, the data sent to
the
primary device is always corrupted (I am currently unable to verify the
secondary as it is not written in the original format).

Here’s is the gist of my IRP clone operation. After calling the cloning
macro a cloned IRP is further modiefied to point to the correct device
by
directly setting the DeviceObject field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp() rather than
by
IoAllocateIrp().

It’s probably worth admitting that while I have programmed drivers for
Unix
for many years I’ve only been working with NT for a month or so.

I would greatly appreciate any insight into how to solve this problem
correctly. I have scoured this list and the web for similar examples
but
haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset)
\
do {
\
IO_STACK_LOCATION *srcstk, *dststk;
\
srcstk = IoGetCurrentIrpStackLocation(src);
\
dststk = IoGetCurrentIrpStackLocation(dst);
\
*dststk = *srcstk;
\
dststk->Context = (void *)0;
\
(dst)->MdlAddress = IoAllocateMdl(
\
MmGetMdlVirtualAddress((src)->MdlAddress),
\
MmGetMdlByteCount((src)->MdlAddress),
\
FALSE, FALSE, (IRP *)0);
\
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress,
\
(char *)MmGetMdlVirtualAddress((src)->MdlAddress) +
\
(zoffset), (unsigned long)0);
\
(dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread;
\
} while (0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@stratus.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Oops. I was misinterpreting my own code that is setting up my own stack
location so I can replay the IRP as needed. Thanks for the correction.

=====================
Mark Roddy

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Doron Holan
Sent: Thursday, January 06, 2005 4:55 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] IRP cloning and mirrored writes…

If you are going to send the PIRP through IoCallDriver(), then

IoSetNextIrpStackLocation
IoGetCurrentIrpStackLocation

Does NOT format the stack location that will be sent to the next device
object. (All you are doing is setting a stack location up that can be
accessible in the completion routine :)). IoGetNextIrpStackLocation sets up
the next stack location for the next device in the call path.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Roddy, Mark
Sent: Thursday, January 06, 2005 1:43 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] IRP cloning and mirrored writes…

Well IoBuildPartialMdl works just fine so, big surprise, you’ve got bugs!

IoGetCurrentIrpStackLocation? The next stack location is found by
IoSetNextIrpStackLocation and then calling IoGetCurrentIrpStackLocation that
is the one that needs setting up.

Why not just use IoAllocateIrp?

Copying stack locations as you do is error prone. See the
IoCopyCurrentIrpStackLocationToNext macro for details. Mostly you don’t need
to setup much more than the MajorFunction and the Parameters.

Make sure your ‘waiting for both Irps to complete asynchronously’ logic is
not a bit on the bogus side. You cannot make any ordering assumptions.

=====================
Mark Roddy

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 4:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix of which one
aspect of its operation is to perform a mirrored write. The manner in which
it performs the mirrored write is to clone the original IRP for each mirror
and then issue the cloned IRPs to the primary and secondary devices.
During
the mirrored write the original IRP is stalled. After the cloned requests
complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the primary device the
driver seems to work well (as evidenced by a data integrity tool I’m using).

However, if I instruct the driver to perform mirroring, the data sent to the
primary device is always corrupted (I am currently unable to verify the
secondary as it is not written in the original format).

Here’s is the gist of my IRP clone operation. After calling the cloning
macro a cloned IRP is further modiefied to point to the correct device by
directly setting the DeviceObject field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp() rather than by
IoAllocateIrp().

It’s probably worth admitting that while I have programmed drivers for Unix
for many years I’ve only been working with NT for a month or so.

I would greatly appreciate any insight into how to solve this problem
correctly. I have scoured this list and the web for similar examples but
haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset)
\
do {
\
IO_STACK_LOCATION *srcstk, *dststk;
\
srcstk = IoGetCurrentIrpStackLocation(src);
\
dststk = IoGetCurrentIrpStackLocation(dst);
\
*dststk = *srcstk;
\
dststk->Context = (void *)0;
\
(dst)->MdlAddress = IoAllocateMdl(
\
MmGetMdlVirtualAddress((src)->MdlAddress),
\
MmGetMdlByteCount((src)->MdlAddress),
\
FALSE, FALSE, (IRP *)0);
\
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress, \
(char *)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0);
\
(dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread; \ } while
(0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@stratus.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Just a guess… Are you completing the mirrored write before completing the
original write? You may be destroying the user buffer on completion before
the other write has a chance to complete. The solution may be to make a copy
of the incoming write buffer and use that copy for the mirrored write. Or
delay completion of either write until both are complete; which means you
may need a second MDL to represent the data.

Just some thoughts…

Jamey

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Greg Becker
Sent: Thursday, January 06, 2005 4:00 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP cloning and mirrored writes…

Hello,

I am currently porting a pseudo disk driver to NT from Unix of which one
aspect of its operation is to perform a mirrored write. The manner in
which it performs the mirrored write is to clone the original IRP for
each mirror and then issue the cloned IRPs to the primary and secondary
devices. During the mirrored write the original IRP is stalled. After
the cloned requests complete, the original IRP is eventually completed.
FWIW, pretty much everything is done asynchronously.

If I only clone the IRP once and issue the clone to the primary device
the driver seems to work well (as evidenced by a data integrity tool
I’m using).

However, if I instruct the driver to perform mirroring, the data sent
to the primary device is always corrupted (I am currently unable to
verify the secondary as it is not written in the original format).

Here’s is the gist of my IRP clone operation. After calling the
cloning macro a cloned IRP is further modiefied to point to the
correct device by directly setting the DeviceObject field in the IRP.
Also, the IRPs are hand made and set up by IoInitializeIrp() rather
than by IoAllocateIrp().

It’s probably worth admitting that while I have programmed drivers for
Unix for many years I’ve only been working with NT for a month or so.

I would greatly appreciate any insight into how to solve this problem
correctly. I have scoured this list and the web for similar examples
but haven’t found anything relevant thus far.

Thanks!
Greg

#define CLONE_IRP(src, dst, zoffset) \
do { \
IO_STACK_LOCATION *srcstk, *dststk; \
srcstk = IoGetCurrentIrpStackLocation(src); \
dststk = IoGetCurrentIrpStackLocation(dst); \
*dststk = *srcstk; \
dststk->Context = (void *)0; \
(dst)->MdlAddress = IoAllocateMdl( \
MmGetMdlVirtualAddress((src)->MdlAddress), \
MmGetMdlByteCount((src)->MdlAddress), \
FALSE, FALSE, (IRP *)0); \
IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress, \
(char *)MmGetMdlVirtualAddress((src)->MdlAddress) + \
(zoffset), (unsigned long)0); \
(dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread; \
} while (0)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@tfb.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

All,

First, I just want to say thanks for the overwhelming response to my
question! Unfortunately, because I receieve the list in digest form I
didn’t get your responses until this morning. Suffice it to say I’ve
rectified that problem :slight_smile:

Next, it’s worth mentioning that this driver has worked for years on
various Solaris and AIX SMP versions (currently 5.9 and 5.2, respectively),
so it’s reasonbly safe to say that the requisite locking and waiting for all
derivative I/O requests to complete before completing the original request
works correctly as it’s mostly all common code.

Next, let me try to answer all your questions in the order they appear
in the digest. Sorry this is so long, but you all deserve thorough
explanations :slight_smile:

From: “Peter Wieland”
>You can’t just copy the entire stack location, as this contains the
>completion routine for the previous driver. Unless you override this,
>when the IRP completes it will run the other driver’s completion routine
>with the wrong IRP. Look at the IoCopyCurrentIrpStackLocationToNext
>macro to see how to copy an IRP.

I do (and must) override the completion routine because these are my own
private IRPs and I need to retain control of them after they complete.
With regard to cloning, I don’t see how IoCopyCurrentIrpStackLocationToNext()
helps, since I am trying to copy the current stack location from the
source IRP to the destination IRP. That said, prior to sending down the
derivative IRPs, I do call IoCopyCurrentIrpStackLocationToNext() prior
to calling IoSetCompletionRoutine(), after which I call IoCallDriver().
Agreed, this is somewhat inefficient, and I should probably be copying
directly into the next stack location, but having a copy of the parameters
in the derivative IRP has its benefits.

>From: “Peter Wieland”
>When you say that the data on the second disk “is not written in the
>original format” I’m curious what you mean. Are you modifying the data?

No, I have no need to touch the data, but I do change the destination
block address for the clone directed at the secondary device so as to
implement a form of write logging. The data going to the primary,
however, retains the block address of the original IRP.

>From: “Peter Wieland”
>Why are you making the IRPs by hand rather than going through
>IoAllocateIrp? Where is their memory coming from?

Long story short, I am trying to shoehorn the Windows port into a
preexisting I/O request abstraction that requires an embedded fixed
size native I/O request buffer at the head of the structure. I will
be looking at fixing this in the future, but for now I have a fixed
IRP followed by a size stack of 24 locations, allocated from the
non-paged pool via ExAllocatePoolWithTag(NonPagedPool, … Regarding
the fixed stack size, I do check upon opening the lower device object
that my fixed size stack can accomodate the stack size of the lower
device object.

>From: “Doron Holan”
>Also, for a PIRP initialized by IoInitializeIrp or IoAllocateIrp, there
>is no current stack location. Do you mean to format the next stack
>location instead? Unless you are processing the irp internally and
>never sending it via IoCallDriver, setting the DeviceObject in the stack
>location is not necessary since IoCallDriver will do that for you.

Prior to working with any of my private IRPs I initialize as below. I
didn’t realize that IoCallDriver() sets the devobj, so that’s good to
know.

/* nbp == native buf pointer, XX_IRP_STKLOC_MAX == 24 */

IoInitializeIrp((nbp), IoSizeOfIrp(XX_IRP_STKLOC_MAX),
XX_IRP_STKLOC_MAX);
IoSetNextIrpStackLocation((nbp));
bzero(IoGetCurrentIrpStackLocation(nbp), sizeof(IO_STACK_LOCATION));

>From: “Roddy, Mark”
>IoGetCurrentIrpStackLocation? The next stack location is found by
>IoSetNextIrpStackLocation and then calling IoGetCurrentIrpStackLocation that
>is the one that needs setting up.

Right. My IPR init macro calls IoSetNextIrpStackLocation() so that it’s
in the right state by the time the clone macro is invoked.

>From: “Roddy, Mark”
>Make sure your ‘waiting for both Irps to complete asynchronously’ logic is
>not a bit on the bogus side. You cannot make any ordering assumptions.

Right, no such assumptions made,and as I mentioned above it works on
both Solaris and AIX SMP systems, so there is little doubt in my mind
that the overall completion logic is working correctly.

>From: “Doron Holan”
>If you are going to send the PIRP through IoCallDriver(), then
>IoSetNextIrpStackLocation
>IoGetCurrentIrpStackLocation
>Does NOT format the stack location that will be sent to the next device
>object. (All you are doing is setting a stack location up that can be
>accessible in the completion routine :)). IoGetNextIrpStackLocation sets
>up the next stack location for the next device in the call path.

For efficiency, I should probably just go this route, but I am covered by
the fact that my function that actually gives the cloned IRP to
IoCallDriver() calls IoCopyCurrentIrpStackLocationToNext() on it beforehand.

>From: “Jamey Kirby”
>Just a guess… Are you completing the mirrored write before completing the
>original write? You may be destroying the user buffer on completion before
>the other write has a chance to complete. The solution may be to make a copy
>of the incoming write buffer and use that copy for the mirrored write. Or
>delay completion of either write until both are complete; which means you
>may need a second MDL to represent the data.

I do clone the original IRP twice: the first clone goes to the primary
device and the second clone is sent to the secondary device. The original
IRP just hangs out until both clones complete, at which point I am then able
to complete the original IRP. The order in which they are issued typically
is secondary first, primary second, but the order in which they complete
is unknown but correctly accounted for. That said, I did a litte test
wherein I reversed the order and issued the primary first but it had no
effect. Keep in mind, I issue these I/O’s one right after the other so that
they may proceed with as much concurrency as possible.

At any rate, based on some other suggestions, I have updated my cloning
macro (below). But more importantly, it appears that the port was working
fine all along. We are now investigating the test tool itself which seems
to be the source of the problem. I can’t explain why yet, but the tool
was ported to Windows half-native, and half cygwin. Interestingly enough,
I have never ever written a Windows user app, so I am about to embark
on porting the tool so as to eliminate cygwin. Case in point, a coworker
simply introduced a macro to redefine malloc() to use the native malloc
vs cygwin malloc and it elimated all but one problem. The same coworker
also then cobbled together a simpler tool that used only native windows
calls and it worked fine. Lastly, it’s only fair to say that the
half-native portion of the port seemed only to address some of the major
function calls like open, read, and write.

Anyway, thanks again for all the help! I’ll post back if we figure out
what the problem is, but I don’t intend to spend much time disecting
cygwin, if indeed that is the problem and I’m about 90% sure about that
at this point.

Greg

#define SC_NBUF_CLONE(src, dst, zoffset) <br>do { <br> IO_STACK_LOCATION *dststk; <br> SC_ASSERT((src)->MdlAddress); <br> SC_ASSERT(!(dst)->MdlAddress); <br> (dst)->Tail.Overlay.Thread = (src)->Tail.Overlay.Thread; <br> dststk = IoGetCurrentIrpStackLocation(dst); <br> RtlCopyMemory(dststk, IoGetCurrentIrpStackLocation(src), <br> FIELD_OFFSET(IO_STACK_LOCATION, DeviceObject)); <br> dststk->Control = 0; <br> (dst)->MdlAddress = IoAllocateMdl( <br> MmGetMdlVirtualAddress((src)->MdlAddress), <br> MmGetMdlByteCount((src)->MdlAddress), <br> FALSE, FALSE, (IRP *)0); <br> SC_ASSERT((dst)->MdlAddress); <br> IoBuildPartialMdl((src)->MdlAddress, (dst)->MdlAddress, <br> (char *)MmGetMdlVirtualAddress((src)->MdlAddress) + <br> (zoffset), (unsigned long)0); <br>} while (0)