Right value to return in a dispatch function of a hard disk filter driver when MmGetSystemAddressFor

Hello,

I have developed a filter driver for hard disk class and, when I activate
Driver Verifier to test it (enabling the “inject faults” option) I see that,
for example, the function MmGetSystemAddressForMdlSafe fails repeatedly when
I call it in my READ and WRITE dispatch functions to map the the buffer. I’m
completing the IRP in that case with STATUS_INSUFFICIENT_RESOURCES. Some
times, the system shows a message informing me that the lazzy writer was
unable to READ/WRITE data (and the file name) I have not seen any secondary
effect but I would like to know if it is possible to return another status
code or do a different process to avoid this error (in any case the error
message appears only when I run Driver Verifier)

Thanks in advance,

Jose Vicente.

All is OK, DV just fails the lazy writer path, and thus the error.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Jos? Vicente S?nchez Ortega”
To: “Windows System Software Devs Interest List”
Sent: Friday, November 28, 2003 3:35 PM
Subject: [ntdev] Right value to return in a dispatch function of a hard disk
filter driver when MmGetSystemAddressForMdlSafe fails

> Hello,
>
> I have developed a filter driver for hard disk class and, when I activate
> Driver Verifier to test it (enabling the “inject faults” option) I see that,
> for example, the function MmGetSystemAddressForMdlSafe fails repeatedly when
> I call it in my READ and WRITE dispatch functions to map the the buffer. I’m
> completing the IRP in that case with STATUS_INSUFFICIENT_RESOURCES. Some
> times, the system shows a message informing me that the lazzy writer was
> unable to READ/WRITE data (and the file name) I have not seen any secondary
> effect but I would like to know if it is possible to return another status
> code or do a different process to avoid this error (in any case the error
> message appears only when I run Driver Verifier)
>
> Thanks in advance,
>
>
> Jose Vicente.
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

Unfortunately, your filter driver is now causing data corruption. That message means that some data which the user thinks is properly written was not. There is very little chance you can determine what data that is after the fact, and the original writer of that data is likely gone, so it cannot be recreated.

This is why such a huge emphasis on forward progress guarantee was made in the storage stack for Windows XP/WS03. Paging requests are guaranteed never to fail due to insufficient resource in a “stock” Windows XP installation (no 3rd party stuff). The port drivers ensure they can handle a single-page request without resources, the class drivers (disk, cdrom) ensure they can handle any size request without resources (even if they have to pend the IO), and the memory manager retries paging requests that fail with out-of-memory errors as single-page requests.

If you can find a way to prevent disk IO from failing (by pre-allocating in StartDevice() enough memory to handle a single request, and then serializing as needed when out-of-memory occurs). When using the pre-allocated request resources, use a pre-allocated mapping address too:

MmAllocateMappingAddress() allows you to reserve a range of virtual address space to use in out-of-memory conditions when you absolutely must not fail IO, such as disk IO (WinXP and later); use the related functions MmMapLockedPagesWithReservedMapping(), MmUnmapReservedMapping(), and MmFreeMappingAddress() to finish the job. Please use this sparingly, as virtual address space is limited on 32 bit machines.

This does require adding some more logic attached to each request you handle in the completion routine: Was the request using the preallocated resources? If so, you must unmap the reserved mapping and then check if anything else was queued to use the preallocated resources. Of course, checking the queue and leaving the preallocated resources for the next person requires appropriate synchronization (just use a spinlock – you’re already in a low-memory condition, so perf isn’t key).

I encourage you to go that last mile to really make your driver 100% reliable. Data corruption is a very bad thing for customers, and out-of-memory conditions really do occur in the real world. (mostly due to drivers leaking memory, but I’ve seen at least two other causes in heavy loads).

Good luck!
.

-----Original Message-----
From: Nick Ryan [mailto:xxxxx@nryan.com]
Sent: Friday, November 28, 2003 12:14 PM
Subject: Re: Right value to return in a dispatch function of a hard disk filter driver when MmGetSystemAddressForMdlSafe fails

No, you’re doing the correct thing. The lazy writer is attempting to
push out dirty data from the filesystem cache to disk, and the paging
writes are failing because of your driver. Everything is behaving as it
should under this simulated error condition.

Jos? Vicente S?nchez Ortega wrote:

Hello,

I have developed a filter driver for hard disk class and, when I activate
Driver Verifier to test it (enabling the “inject faults” option) I see that,
for example, the function MmGetSystemAddressForMdlSafe fails repeatedly when
I call it in my READ and WRITE dispatch functions to map the the buffer. I’m
completing the IRP in that case with STATUS_INSUFFICIENT_RESOURCES. Some
times, the system shows a message informing me that the lazzy writer was
unable to READ/WRITE data (and the file name) I have not seen any secondary
effect but I would like to know if it is possible to return another status
code or do a different process to avoid this error (in any case the error
message appears only when I run Driver Verifier)

Thanks in advance,

Jose Vicente.


Nick Ryan (MVP for DDK)

Thank you very much for your answer. I use a pre-allocated memory pool in my
driver in order to avoid memory allocation problems in low resource
situations but, as I mentioned, the function which fails is
MmGetSystemAddressForMdlSafe not an allocation function. The problem is that
if this function fails, I don’t know how to map user buffer in order to
manipulate it as I need. I can use the set of functions which you have
mentioned but I still will have a problem in W2K (since my driver must work
in that platform too) In W2K, I can pre-allocate a memory pool too but, how
can I acomplish the task of reserve a virtual address mapping range to
ensure that I will always able to map the MDL I receive?

Best regards,

Jose Vicente.

-----Mensaje original-----
De: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] En nombre de Henry Gabryjelski
Enviado el: s?bado, 29 de noviembre de 2003 19:10
Para: Windows System Software Devs Interest List
Asunto: [ntdev] RE: Right value to return in a dispatch function of a hard
disk filter driver when MmGetSystemAddressForMdlSafe fails

Unfortunately, your filter driver is now causing data corruption. That
message means that some data which the user thinks is properly written was
not. There is very little chance you can determine what data that is after
the fact, and the original writer of that data is likely gone, so it cannot
be recreated.

This is why such a huge emphasis on forward progress guarantee was made in
the storage stack for Windows XP/WS03. Paging requests are guaranteed never
to fail due to insufficient resource in a “stock” Windows XP installation
(no 3rd party stuff). The port drivers ensure they can handle a single-page
request without resources, the class drivers (disk, cdrom) ensure they can
handle any size request without resources (even if they have to pend the
IO), and the memory manager retries paging requests that fail with
out-of-memory errors as single-page requests.

If you can find a way to prevent disk IO from failing (by pre-allocating in
StartDevice() enough memory to handle a single request, and then serializing
as needed when out-of-memory occurs). When using the pre-allocated request
resources, use a pre-allocated mapping address too:

MmAllocateMappingAddress() allows you to reserve a range of virtual address
space to use in out-of-memory conditions when you absolutely must not fail
IO, such as disk IO (WinXP and later); use the related functions
MmMapLockedPagesWithReservedMapping(), MmUnmapReservedMapping(), and
MmFreeMappingAddress() to finish the job. Please use this sparingly, as
virtual address space is limited on 32 bit machines.

This does require adding some more logic attached to each request you handle
in the completion routine: Was the request using the preallocated resources?
If so, you must unmap the reserved mapping and then check if anything else
was queued to use the preallocated resources. Of course, checking the queue
and leaving the preallocated resources for the next person requires
appropriate synchronization (just use a spinlock – you’re already in a
low-memory condition, so perf isn’t key).

I encourage you to go that last mile to really make your driver 100%
reliable. Data corruption is a very bad thing for customers, and
out-of-memory conditions really do occur in the real world. (mostly due to
drivers leaking memory, but I’ve seen at least two other causes in heavy
loads).

Good luck!
.

-----Original Message-----
From: Nick Ryan [mailto:xxxxx@nryan.com]
Sent: Friday, November 28, 2003 12:14 PM
Subject: Re: Right value to return in a dispatch function of a hard disk
filter driver when MmGetSystemAddressForMdlSafe fails

No, you’re doing the correct thing. The lazy writer is attempting to
push out dirty data from the filesystem cache to disk, and the paging
writes are failing because of your driver. Everything is behaving as it
should under this simulated error condition.

Jos? Vicente S?nchez Ortega wrote:

Hello,

I have developed a filter driver for hard disk class and, when I
activate Driver Verifier to test it (enabling the “inject faults”
option) I see that, for example, the function
MmGetSystemAddressForMdlSafe fails repeatedly when I call it in my
READ and WRITE dispatch functions to map the the buffer. I’m
completing the IRP in that case with STATUS_INSUFFICIENT_RESOURCES.
Some times, the system shows a message informing me that the lazzy
writer was unable to READ/WRITE data (and the file name) I have not
seen any secondary effect but I would like to know if it is possible
to return another status code or do a different process to avoid this
error (in any case the error message appears only when I run Driver
Verifier)

Thanks in advance,

Jose Vicente.


Nick Ryan (MVP for DDK)


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@secuware.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

> situations but, as I mentioned, the function which fails is

MmGetSystemAddressForMdlSafe not an allocation function.

It is an allocation function, it allocates the range of system PTEs.

MmAllocateMappingAddress and friends is a way to do pre-allocation of them.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

You are right but for this function (macro) to work properly I need the MDL
thus is not possible to “pre-map” a block a PTEs since I don’t know which
user address I will receive. How can I “pre-map” enough PTEs to ensure that
the mapping won’t fail whenever I need it in low resource situation? (Again
in W2K)

Regards,

Jose Vicente.

-----Mensaje original-----
De: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] En nombre de Maxim S. Shatskih
Enviado el: lunes, 01 de diciembre de 2003 18:25
Para: Windows System Software Devs Interest List
Asunto: [ntdev] RE: Right value to return in a dispatch function of a hard
disk filter driver when MmGetSystemAddressForMdlSafe fails

situations but, as I mentioned, the function which fails is
MmGetSystemAddressForMdlSafe not an allocation function.

It is an allocation function, it allocates the range of system PTEs.

MmAllocateMappingAddress and friends is a way to do pre-allocation of them.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@secuware.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

Do you know the maximum size of the transfer in pages?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Jos? Vicente S?nchez Ortega”
To: “Windows System Software Devs Interest List”
Sent: Monday, December 01, 2003 8:49 PM
Subject: [ntdev] RE: Right value to return in a dispatch function of a hard
disk filter driver when MmGetSystemAddressForMdlSafe fails

> You are right but for this function (macro) to work properly I need the MDL
> thus is not possible to “pre-map” a block a PTEs since I don’t know which
> user address I will receive. How can I “pre-map” enough PTEs to ensure that
> the mapping won’t fail whenever I need it in low resource situation? (Again
> in W2K)
>
> Regards,
>
>
> Jose Vicente.
>
>
> -----Mensaje original-----
> De: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] En nombre de Maxim S. Shatskih
> Enviado el: lunes, 01 de diciembre de 2003 18:25
> Para: Windows System Software Devs Interest List
> Asunto: [ntdev] RE: Right value to return in a dispatch function of a hard
> disk filter driver when MmGetSystemAddressForMdlSafe fails
>
>
> > situations but, as I mentioned, the function which fails is
> > MmGetSystemAddressForMdlSafe not an allocation function.
>
> It is an allocation function, it allocates the range of system PTEs.
>
> MmAllocateMappingAddress and friends is a way to do pre-allocation of them.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@secuware.com To
> unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

It depends. I think the greatest value I have seen is 16 pages.

-----Mensaje original-----
De: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] En nombre de Maxim S. Shatskih
Enviado el: lunes, 01 de diciembre de 2003 19:17
Para: Windows System Software Devs Interest List
Asunto: [ntdev] RE: Right value to return in a dispatch function of a hard
disk filter driver when MmGetSystemAddressForMdlSafe fails

Do you know the maximum size of the transfer in pages?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Jos? Vicente S?nchez Ortega”
To: “Windows System Software Devs Interest List”
Sent: Monday, December 01, 2003 8:49 PM
Subject: [ntdev] RE: Right value to return in a dispatch function of a hard
disk filter driver when MmGetSystemAddressForMdlSafe fails

> You are right but for this function (macro) to work properly I need
> the MDL thus is not possible to “pre-map” a block a PTEs since I don’t
> know which user address I will receive. How can I “pre-map” enough
> PTEs to ensure that the mapping won’t fail whenever I need it in low
> resource situation? (Again in W2K)
>
> Regards,
>
>
> Jose Vicente.
>
>
> -----Mensaje original-----
> De: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] En nombre de Maxim S.
> Shatskih Enviado el: lunes, 01 de diciembre de 2003 18:25
> Para: Windows System Software Devs Interest List
> Asunto: [ntdev] RE: Right value to return in a dispatch function of a
> hard disk filter driver when MmGetSystemAddressForMdlSafe fails
>
>
> > situations but, as I mentioned, the function which fails is
> > MmGetSystemAddressForMdlSafe not an allocation function.
>
> It is an allocation function, it allocates the range of system PTEs.
>
> MmAllocateMappingAddress and friends is a way to do pre-allocation of
> them.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@secuware.com To
> unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com To
> unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@secuware.com To
unsubscribe send a blank email to xxxxx@lists.osr.com