I have a KMDF Volume Class Upper Filter driver for change block tracking. I want to persist some details during volume removal so that I can perform CBT across volume detach attach. I have registered EvtDeviceD0Exit callback function and based on this (https://msdn.microsoft.com/en-us/windows/hardware/drivers/wdf/power-down-and-removal-sequence-for-a-function-or-filter-driver), I thought that it will be called when the system is shutdown. However, with a live debugger attached and breakpoint set for this callback, it does not break into the debugger during reboot (and neither does it perform the operation I have coded it for, just in case there is some live debugger issue causing it to not break at the bp). I am not trying this with a volume having special files. If I offline the volume, or disable the disk on which the volume exists, then I do get the callback.
Is this expected ? If not, any pointers on where to look for problem ?
If yes, what is the correct way to handle the scenario (of shutdown / reboot etc) for persisting details ? Should I call IoRegisterLastChanceShutdownNotification() ? The documentation says “Only one driver in a device stack should register to receive shutdown notification, by calling either IoRegisterShutdownNotification or IoRegisterLastChanceShutdownNotification.”. How can I ensure this ?
as far as i remember, shutdown doesnt call any of the said routines, i
think ur best bet is to do it on volume unmount, if u do not care abt any
io under he fs, then a fs dismount is als oa great time to save ur state.
On Mon, Jan 30, 2017 at 8:33 AM, wrote:
> I have a KMDF Volume Class Upper Filter driver for change block tracking. > I want to persist some details during volume removal so that I can perform > CBT across volume detach attach. I have registered EvtDeviceD0Exit callback > function and based on this (https://msdn.microsoft.com/ > en-us/windows/hardware/drivers/wdf/power-down-and-removal-sequence-for-a- > function-or-filter-driver), I thought that it will be called when the > system is shutdown. However, with a live debugger attached and breakpoint > set for this callback, it does not break into the debugger during reboot > (and neither does it perform the operation I have coded it for, just in > case there is some live debugger issue causing it to not break at the bp). > I am not trying this with a volume having special files. If I offline the > volume, or disable the disk on which the volume exists, then I do get the > callback. > > Is this expected ? If not, any pointers on where to look for problem ? > If yes, what is the correct way to handle the scenario (of shutdown / > reboot etc) for persisting details ? Should I call > IoRegisterLastChanceShutdownNotification() ? The documentation says “Only > one driver in a device stack should register to receive shutdown > notification, by calling either IoRegisterShutdownNotification or > IoRegisterLastChanceShutdownNotification.”. How can I ensure this ? > > — > NTDEV is sponsored by OSR > > Visit the list online at: http:> showlists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> >
Thanks Amit for the suggestion.
To be clear on whether I understand it correctly, do you imply using EvtIoDeviceControl callback and monitor for the last IOCTL_MOUNTDEV_LINK_DELETED ?
IOCTL_MOUNTDEV_LINK_DELETED has to do with drive letters and mount points,
nothing related to shutdown or dismount.
In any case, file systems aren’t dismounted or powered off during shutdown.
They are shut down using IRP_MJ_SHUTDOWN.
There are three different shutdown notifications: before the file system
shuts down (IoRegisterShutdownNotification), to shut down the file system
(IoRegisterFileSystem), after the file system shuts down
(IoRegisterLastChangeShutdownNotification). Once the file system shuts down
you can no longer write to any files on the file system.
So, “I want to write something on shutdown” usually means flushing before
the file system shuts down. If you want to write something after the file
system shuts down then you’ll need to do it by directly writing to the
volume. Note that the file system forwards the IRP_MJ_SHUTDOWN that it
receives to the volume stack, thus you are already being called at
IRP_MJ_SHUTDOWN after the file system is shut down.
Thanks Amit for the suggestion.
To be clear on whether I understand it correctly, do you imply using
EvtIoDeviceControl callback and monitor for the last
IOCTL_MOUNTDEV_LINK_DELETED ?
Thank you Scott for the details.
As per the product requirement, I am right now trying to construct a solution for volumes not having any of the special files. So, I can choose to write the details to the volume having, say, pagefile. I understand that this approach has its set of cons.
Now, under this scenario, for a KMDF volume filter driver, how do I hook into the IRP_MJ_SHUTDOWN flow of events ?
I found the following two MSDN links that seem to suggest that from KMDF filter driver, I will not be able to directly hook into IRP_MJ_SHUTDOWN (or maybe I am misinterpreting) and Amit also mentioned something similar in his response :
https://msdn.microsoft.com/en-us/windows/hardware/drivers/wdf/supporting-pnp-and-power-management-in-software-only-drivers
Other software-only drivers are filter drivers: they reside in a stack of drivers that do access hardware, but the filter drivers do not access hardware. When a filter driver receives an I/O request that specifies a PnP or power management operation, the driver typically just passes the request to the next driver. The framework intercepts these requests and passes them on, so framework-based drivers never see the requests.
IRP_MJ_SHUTDOWN – For control device objects, implement EvtDeviceShutdownNotification (KMDF only)
For all Plug and Play device objects: Not supported; implement EvtDeviceWdmIrpPreprocess (KMDF only).
EvtDeviceWdmIrpPreprocess is indeed called if I call WdfDeviceInitAssignWdmIrpPreprocessCallback() with IRP_MJ_SHUTDOWN as the MajorFunction. Since you have explained that “file system forwards the IRP_MJ_SHUTDOWN that it
receives to the volume stack, thus you are already being called at
IRP_MJ_SHUTDOWN after the file system is shut down”, so, if I am not concerned about writes originating under the filesystem, then, is it correct to intercept IRP_MJ_SHUTDOWN this way (i.e. EvtDeviceWdmIrpPreprocess) ? Or that is erroneous and there is a better way to deal with it (if it is IoRegisterLastChanceShutdownNotification() then please help clear my concern in the last portion of my initial question viz “Only one driver in a device stack…”)
Yes, WdfDeviceInitAssignWdmIrpPreprocessCallback is the right way to get
notified of the the file system’s shutdown notification IRP. Whether you
also or instead want pre-file system or post-file system shutdown
notification (i.e. IoRegisterXxx) will depend on your design and what you’re
ultimately trying to do.
Thank you Scott for the details.
As per the product requirement, I am right now trying to construct a
solution for volumes not having any of the special files. So, I can choose
to write the details to the volume having, say, pagefile. I understand that
this approach has its set of cons.
Now, under this scenario, for a KMDF volume filter driver, how do I hook
into the IRP_MJ_SHUTDOWN flow of events ?
I found the following two MSDN links that seem to suggest that from KMDF
filter driver, I will not be able to directly hook into IRP_MJ_SHUTDOWN (or
maybe I am misinterpreting) and Amit also mentioned something similar in his
response :
https://msdn.microsoft.com/en-us/windows/hardware/drivers/wdf/supporting-pnp-and-power-management-in-software-only-drivers
Other software-only drivers are filter drivers: they reside in a stack of
drivers that do access hardware, but the filter drivers do not access
hardware. When a filter driver receives an I/O request that specifies a PnP
or power management operation, the driver typically just passes the request
to the next driver. The framework intercepts these requests and passes them
on, so framework-based drivers never see the requests.
IRP_MJ_SHUTDOWN – For control device objects, implement
EvtDeviceShutdownNotification (KMDF only)
For all Plug and Play device objects: Not supported; implement
EvtDeviceWdmIrpPreprocess (KMDF only).
EvtDeviceWdmIrpPreprocess is indeed called if I call
WdfDeviceInitAssignWdmIrpPreprocessCallback() with IRP_MJ_SHUTDOWN as the
MajorFunction. Since you have explained that “file system forwards the
IRP_MJ_SHUTDOWN that it
receives to the volume stack, thus you are already being called at
IRP_MJ_SHUTDOWN after the file system is shut down”, so, if I am not
concerned about writes originating under the filesystem, then, is it correct
to intercept IRP_MJ_SHUTDOWN this way (i.e. EvtDeviceWdmIrpPreprocess) ? Or
that is erroneous and there is a better way to deal with it (if it is
IoRegisterLastChanceShutdownNotification() then please help clear my concern
in the last portion of my initial question viz “Only one driver in a device
stack…”)
Thank you Scott for the confirmation.
One thing I am not clear about is whether the KMDF volume filter driver can get the pre-file system shutdown notification. I have called IoRegisterShutdownNotification() which returns success but does not lead to the notification being received. Anyway, that may not be the correct point to persist volume level CBT details since FS flush happens after that and hence there can be further writes onto the volume.
My assumption that if I persist the details onto a volume having one of the special files by using ZwxxxFile also seems to have been wrong since in a case of the system volume and another volume on same disk, I see that the EvtDeviceWdmIrpPreprocess() callback for IRP_MJ_SHUTDOWN is received by the system volume before the other volume and hence ZwxxxFile() calls fail with STATUS_TOO_LATE. I am unable to even access the registry at that time.
So, the only correct way to be able to persist CBT details during system shutdown from a volume filter driver seems to be to create a file of the requisite size and know the extents and then use this detail to send IOs directly to the volume at shutdown time for writing the CBT metadata. The extent details will also be needed when next time the volume is attached so as to read back the persisted details. Is this the correct direction of thought ?
Thank you Scott for the confirmation. I was able to proceed further to write data from my KMDF volume filter driver during system shutdown onto the volume by sending requests directly to the volume (by having figured out the requisite offset, length earlier). However, this does not work for the volume offline case. Where should I do the writes in case of volume offline (or similar scenarios) ?
If the volume is marked as offline during runtime then all I/O sent to the
drive is going to fail. If someone then shuts down there’s not much you’re
going to be able to do if you want your changes stored on the volume. In
general CBT is an optimization, so worst case you need to do a full backup
again on reboot.
Are you seeing this behavior from a specific product or is this strictly a
“what if” situation?
Thank you Scott for the confirmation. I was able to proceed further to write
data from my KMDF volume filter driver during system shutdown onto the
volume by sending requests directly to the volume (by having figured out the
requisite offset, length earlier). However, this does not work for the
volume offline case. Where should I do the writes in case of volume offline
(or similar scenarios) ?
What I was trying to do is to hook into the offline and disable scenarios to write the CBT information to the volume before it really goes away.
I was asking the question from a “what if” perspective. I agree that an explicit user performed volume offline - online or device disable-enable may not be a usual workflow and can be documented to be a case where things will fall back to (optimized) full backup. A resource rebalance case may lead to similar case and again, it will not be a usual case.
consider this:
the best way to be absolutely sure that your meta data is correct and
accurate is to update and commit ur meta data for every write. which means
that for each IO you will also trigger another IO, which is not efficient.
on the other hand, if you dont do it, hen u are in the situation u are
right now.
perhaps there is a more optimal design/algo required here, which carters to
both problems? i dont know what kind of structures you are using to
track/map the volume, but it can be better solved if you *dont* use a
simple bitmap.
On Tue, Feb 14, 2017 at 9:23 PM, wrote:
> Hi Scott, > > What I was trying to do is to hook into the offline and disable scenarios > to write the CBT information to the volume before it really goes away. > > I was asking the question from a “what if” perspective. I agree that an > explicit user performed volume offline - online or device disable-enable > may not be a usual workflow and can be documented to be a case where things > will fall back to (optimized) full backup. A resource rebalance case may > lead to similar case and again, it will not be a usual case. > > > — > NTDEV is sponsored by OSR > > Visit the list online at: http:> showlists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> >
Push the updates off to a system thread for processing, and make sure you
synchronize to your threads queue.
On Wed, Feb 15, 2017 at 7:22 PM Amitrajit B wrote:
> consider this: > the best way to be absolutely sure that your meta data is correct and > accurate is to update and commit ur meta data for every write. which means > that for each IO you will also trigger another IO, which is not efficient. > on the other hand, if you dont do it, hen u are in the situation u are > right now. > > perhaps there is a more optimal design/algo required here, which carters > to both problems? i dont know what kind of structures you are using to > track/map the volume, but it can be better solved if you dont use a > simple bitmap. > > On Tue, Feb 14, 2017 at 9:23 PM, wrote: > > Hi Scott, > > What I was trying to do is to hook into the offline and disable scenarios > to write the CBT information to the volume before it really goes away. > > I was asking the question from a “what if” perspective. I agree that an > explicit user performed volume offline - online or device disable-enable > may not be a usual workflow and can be documented to be a case where things > will fall back to (optimized) full backup. A resource rebalance case may > lead to similar case and again, it will not be a usual case. > > > — > NTDEV is sponsored by OSR > > Visit the list online at: < > http://www.osronline.com/showlists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> > > > > > – > > - ab > — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars > on crash dump analysis, WDF, Windows internals and software drivers! > Details at To unsubscribe, visit the List Server section of OSR Online at</http:>
Constantly checkpointing the changes to disk sounds like a bit of a
nightmare. We’ve never seen enough benefit to warrant the additional
complexity in development, test, and validation (which is the piece that
keeps me personally awake at night).
“Jamey Kirby” wrote in message news:xxxxx@ntdev… Push the updates off to a system thread for processing, and make sure you synchronize to your threads queue.
On Wed, Feb 15, 2017 at 7:22 PM Amitrajit B wrote:
consider this: the best way to be absolutely sure that your meta data is correct and accurate is to update and commit ur meta data for every write. which means that for each IO you will also trigger another IO, which is not efficient. on the other hand, if you dont do it, hen u are in the situation u are right now.
perhaps there is a more optimal design/algo required here, which carters to both problems? i dont know what kind of structures you are using to track/map the volume, but it can be better solved if you dont use a simple bitmap.
On Tue, Feb 14, 2017 at 9:23 PM, wrote: Hi Scott,
What I was trying to do is to hook into the offline and disable scenarios to write the CBT information to the volume before it really goes away.
I was asking the question from a “what if” perspective. I agree that an explicit user performed volume offline - online or device disable-enable may not be a usual workflow and can be documented to be a case where things will fall back to (optimized) full backup. A resource rebalance case may lead to similar case and again, it will not be a usual case.
MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at http:
To unsubscribe, visit the List Server section of OSR Online at http:
–
- ab — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:>
It depends on the requirements. If you are guaranteeing (for some value of
guarantee) no data loss then you have to preserve enough information to
recover from failure somewhere.
Mark Roddy
On Thu, Feb 16, 2017 at 1:47 PM, Scott Noone wrote:
> Constantly checkpointing the changes to disk sounds like a bit of a > nightmare. We’ve never seen enough benefit to warrant the additional > complexity in development, test, and validation (which is the piece that > keeps me personally awake at night). > > -scott > OSR > @OSRDrivers > > “Jamey Kirby” wrote in message news:xxxxx@ntdev. > … > Push the updates off to a system thread for processing, and make sure you > synchronize to your threads queue. > > On Wed, Feb 15, 2017 at 7:22 PM Amitrajit B wrote: > > consider this: > the best way to be absolutely sure that your meta data is correct and > accurate is to update and commit ur meta data for every write. which means > that for each IO you will also trigger another IO, which is not efficient. > on the other hand, if you dont do it, hen u are in the situation u are > right now. > > perhaps there is a more optimal design/algo required here, which carters > to both problems? i dont know what kind of structures you are using to > track/map the volume, but it can be better solved if you dont use a > simple bitmap. > > > On Tue, Feb 14, 2017 at 9:23 PM, wrote: > Hi Scott, > > What I was trying to do is to hook into the offline and disable scenarios > to write the CBT information to the volume before it really goes away. > > I was asking the question from a “what if” perspective. I agree that an > explicit user performed volume offline - online or device disable-enable > may not be a usual workflow and can be documented to be a case where things > will fall back to (optimized) full backup. A resource rebalance case may > lead to similar case and again, it will not be a usual case. > > > > — > NTDEV is sponsored by OSR > > Visit the list online at: http:> lists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> > > > > > > – > > > - ab > — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars > on crash dump analysis, WDF, Windows internals and software drivers! > Details at To unsubscribe, visit the List Server section of OSR Online at > > — > NTDEV is sponsored by OSR > > Visit the list online at: http:> lists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> ></http:></http:></http:></http:>
Not so difficult. No more difficult than maintaining fs consistency during
cache flush
On Thu, Feb 16, 2017, 1:47 PM Scott Noone wrote:
> Constantly checkpointing the changes to disk sounds like a bit of a > nightmare. We’ve never seen enough benefit to warrant the additional > complexity in development, test, and validation (which is the piece that > keeps me personally awake at night). > > -scott > OSR > @OSRDrivers > > “Jamey Kirby” wrote in message news:xxxxx@ntdev. > … > Push the updates off to a system thread for processing, and make sure you > synchronize to your threads queue. > > On Wed, Feb 15, 2017 at 7:22 PM Amitrajit B wrote: > > consider this: > the best way to be absolutely sure that your meta data is correct and > accurate is to update and commit ur meta data for every write. which means > that for each IO you will also trigger another IO, which is not efficient. > on the other hand, if you dont do it, hen u are in the situation u are > right > now. > > perhaps there is a more optimal design/algo required here, which carters to > both problems? i dont know what kind of structures you are using to > track/map the volume, but it can be better solved if you dont use a > simple > bitmap. > > > On Tue, Feb 14, 2017 at 9:23 PM, wrote: > Hi Scott, > > What I was trying to do is to hook into the offline and disable scenarios > to > write the CBT information to the volume before it really goes away. > > I was asking the question from a “what if” perspective. I agree that an > explicit user performed volume offline - online or device disable-enable > may > not be a usual workflow and can be documented to be a case where things > will > fall back to (optimized) full backup. A resource rebalance case may lead to > similar case and again, it will not be a usual case. > > > > — > NTDEV is sponsored by OSR > > Visit the list online at: > http: > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software > drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at > http: > > > > > > – > > > - ab > — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on > crash dump analysis, WDF, Windows internals and software drivers! Details > at > To unsubscribe, visit the List Server section of OSR Online at > > > — > NTDEV is sponsored by OSR > > Visit the list online at: < > http://www.osronline.com/showlists.cfm?list=ntdev> > > MONTHLY seminars on crash dump analysis, WDF, Windows internals and > software drivers! > Details at http: > > To unsubscribe, visit the List Server section of OSR Online at < > http://www.osronline.com/page.cfm?name=ListServer> ></http:></http:></http:></http:>
“Jamey Kirby” wrote in message news:xxxxx@ntdev… Not so difficult. No more difficult than maintaining fs consistency during cache flush
On Thu, Feb 16, 2017, 1:47 PM Scott Noone wrote: Constantly checkpointing the changes to disk sounds like a bit of a nightmare. We’ve never seen enough benefit to warrant the additional complexity in development, test, and validation (which is the piece that keeps me personally awake at night).
“Jamey Kirby” wrote in message news:xxxxx@ntdev… Push the updates off to a system thread for processing, and make sure you synchronize to your threads queue.
On Wed, Feb 15, 2017 at 7:22 PM Amitrajit B wrote:
consider this: the best way to be absolutely sure that your meta data is correct and accurate is to update and commit ur meta data for every write. which means that for each IO you will also trigger another IO, which is not efficient. on the other hand, if you dont do it, hen u are in the situation u are right now.
perhaps there is a more optimal design/algo required here, which carters to both problems? i dont know what kind of structures you are using to track/map the volume, but it can be better solved if you dont use a simple bitmap.
On Tue, Feb 14, 2017 at 9:23 PM, wrote: Hi Scott,
What I was trying to do is to hook into the offline and disable scenarios to write the CBT information to the volume before it really goes away.
I was asking the question from a “what if” perspective. I agree that an explicit user performed volume offline - online or device disable-enable may not be a usual workflow and can be documented to be a case where things will fall back to (optimized) full backup. A resource rebalance case may lead to similar case and again, it will not be a usual case.
MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at http:
To unsubscribe, visit the List Server section of OSR Online at http:
–
- ab — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at
