Data on the volume changed, and the Filter does not know !!!

I have a upper filter driver,and it is used to create a differential using the volume bitmap file, which is created and changed in my filter that is attached to my UpperFilter. Thus, when it is necessary to create a differential image, I take full volume image and my volume bitmap file, which retained all the changes since the last full image. In the bitmap changes are tracked by sector. To read data from the volume using the Volume Shadow Service. problem is that I do not get in the packet filter information for some clusters. I see that the data in these clusters on the volume and the snapshot has changed, but the filter has not received packets IRP_MJ_WRITE for these clusters. Therefore, the differential image is created “broken.” Packages in the filter processes in WriteCompletionRoutine. To determine the initial cluster and the total number of clusters using the field ioStackLocation-> Parameters.Write.ByteOffset and ioStackLocation-> Parameters.Write.Length. problems observed in recent versions of Windows. Specifically test on WIndows 7 and Windows Server 2008 R2 x64. On XP such problems do not watch. Tried to install the filter as LowerFilter, but the bug still remains. Some packages filter does not see. possible to change the data on the volume to pack IRP_MJ_WRITE not created? How can it be that the data on the volume change, and the filter does not know about?

I have a upper filter driver,and it is used to create a differential using the volume bitmap file, which is created and changed in my filter that is attached to my UpperFilter. Thus, when it is necessary to create a differential image, I take full volume image and my volume bitmap file, which retained all the changes since the last full image. In the bitmap changes are tracked by sector. To read data from the volume using the Volume Shadow Service. problem is that I do not get in the packet filter information for some clusters. I see that the data in these clusters on the volume and the snapshot has changed, but the filter has not received packets IRP_MJ_WRITE for these clusters. Therefore, the differential image is created “broken.” Packages in the filter processes in WriteCompletionRoutine. To determine the initial cluster and the total number of clusters using the field ioStackLocation-> Parameters.Write.ByteOffset and ioStackLocation-> Parameters.Write.Length. problems observed in recent versions of Windows. Specifically test on WIndows 7 and Windows Server 2008 R2 x64. On XP such problems do not watch. Tried to install the filter as LowerFilter, but the bug still remains. Some packages filter does not see. possible to change the data on the volume to pack IRP_MJ_WRITE not created? How can it be that the data on the volume change, and the filter does not know about?

the problem description is too generic to make a solid comment, but here
are a few questions I would be looking the answers for, if I were you…

  1. Do these clusters belong to a file? If so is it always the same file or
    differnt files at different times?
  2. It is always possible to write things to the storage device by
    generating ones own IRPs, so is there a driver or program that is openign
    the disk device in your test setup?
  3. You said you did try a volume lower filter as well, which would make me
    assume that making it a lower filter also enables you to see volsnap’s own
    IOs. Volsnap makes many IOs at high IRQL levels, so are you tracking IOs at
    all IRQL levels in your driver?
  4. Does your filter also attach to the snapshot class for volsnap? When a
    snapshot is exposed writers will engage and make cahnges, so the IOS you
    are missing are they those made on teh snapshot device by any chance?
  5. Transactional IO is also another place to look into (since you say that
    this happens only on newer OSs which supports transacrions), in case of
    transactrional IO, the data is writetn into the disk but the MFT is *only*
    updated once the transaction is completed, which means that if the
    transaction is aborted the MFT will not show those IOs at all, so if, in
    your driver/product, you have logic to engage the MFT’s bitmap as well in
    determining the changed sectors, you might need to verify if it works with
    transactions.

Of the top of my head these are the places i would look for, ofcourse, i am
not privy to your design, so ther might be something else more basic which
is the culprit here…

On Fri, Aug 9, 2013 at 9:56 PM, wrote:

> I have a upper filter driver,and it is used to create a differential using
> the volume bitmap file, which is created and changed in my filter that is
> attached to my UpperFilter. Thus, when it is necessary to create a
> differential image, I take full volume image and my volume bitmap file,
> which retained all the changes since the last full image. In the bitmap
> changes are tracked by sector. To read data from the volume using the
> Volume Shadow Service. problem is that I do not get in the packet filter
> information for some clusters. I see that the data in these clusters on the
> volume and the snapshot has changed, but the filter has not received
> packets IRP_MJ_WRITE for these clusters. Therefore, the differential image
> is created “broken.” Packages in the filter processes in
> WriteCompletionRoutine. To determine the initial cluster and the total
> number of clusters using the field ioStackLocation->
> Parameters.Write.ByteOffset and ioStackLocation-> Parameters.Write.Length.
> problems observed in recent versions of Windows. Specifically test on
> WIndows 7 and Windows Server 2008 R2 x64. On XP such problems do not watch.
> Tried to install the filter as LowerFilter, but the bug still remains. Some
> packages filter does not see. possible to change the data on the volume to
> pack IRP_MJ_WRITE not created? How can it be that the data on the volume
> change, and the filter does not know about?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>



- ab

  1. Do these clusters belong to a file? If so is it always the same file or
    differnt files at different times?

Yes these clusters belong to same file which is created on a praticular volume
at a particular time ex: for volume C: creates a volume bitmap file so on.

2.so is there a driver or program that is opening the disk device in your test setup?

Yes there is a driver upper filter (volume) which is opening the file on a volume and track changes.

3.Does your filter also attach to the snapshot class for volsnap?
Yes

Thanks!!!

i dont think you got some of my questions correctly…

#2 I meant, is there another product apart from yours which might we
opening the disk device and writing to it?

Yes these clusters belong to same file which is created on a praticular
volume
at a particular time ex: for volume C: creates a volume bitmap file so on.

So which file is this? Whats it’s name? and which app/service writes it?

On Sat, Aug 10, 2013 at 9:25 PM, wrote:

> 1. Do these clusters belong to a file? If so is it always the same file or
> differnt files at different times?
>
> Yes these clusters belong to same file which is created on a praticular
> volume
> at a particular time ex: for volume C: creates a volume bitmap file so on.
>
> 2.so is there a driver or program that is opening the disk device in your
> test setup?
>
> Yes there is a driver upper filter (volume) which is opening the file on a
> volume and track changes.
>
> 3.Does your filter also attach to the snapshot class for volsnap?
> Yes
>
> Thanks!!!
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>



- ab

#2. on freshly installed win7 i have tested with my driver only there is no other 3rd party tool which can write to the disk !!!

>So which file is this? Whats it’s name? and which app/service writes it?

driver is filtering a volume with the help of that file. this file created by my application and all the writes send by my driver to application. name x.dat file.
my driver is able to update that file but sometimes it gets failed to update properly how to solve it.
some suggestions…

ok, so if I have got he problem correctly, your app and driver have a meta
data file to, lets call i x.dat. This file resides on the volume being
tracked. Some times IO on this file is missed.

If this is the case I have a few questions:

  1. Who does the IO on this file, the user mode application helping your
    driver, or the driver itself?
  2. What are the flags used to open the file?
  3. Are the IOs being missed, those which occur during the backup proess on
    the snapshot volume (that is, the IOs performed by the writers)?

On Sun, Aug 11, 2013 at 11:42 PM, wrote:

> #2. on freshly installed win7 i have tested with my driver only there is
> no other 3rd party tool which can write to the disk !!!
>
> >>So which file is this? Whats it’s name? and which app/service writes it?
>
> driver is filtering a volume with the help of that file. this file created
> by my application and all the writes send by my driver to application. name
> x.dat file.
> my driver is able to update that file but sometimes it gets failed to
> update properly how to solve it.
> some suggestions…
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>



- ab

  1. My Driver alone.
  2. used zwcreate file with ( FILE_READ_DATA | FILE_WRITE_DATA | SYNCHRONIZE,
    FILE_ATTRIBUTE_NORMAL | FILE_ATTRIBUTE_SYSTEM,FILE_OPEN, FILE_NON_DIRECTORY_FILE | FILE_RANDOM_ACCESS | FILE_WRITE_THROUGH |
    FILE_NO_INTERMEDIATE_BUFFERING | FILE_SYNCHRONOUS_IO_NONALERT.
  3. the Ios are missing at the time taking the backup.

thanks for Help!!!