Read contents of a file during boot

Hi,

I am working on an upper volume filter driver based on diskperf.
My driver actually monitors the sectors that are modified and sets the respective blocks in a bitmap. When the system is properly shutdown, the bitmap buffer is copied into a bitmap file.

So now what my requirement is, when the system restarts, I need to copy the contents of this bitmap file into our bitmap buffer before any “Writes” occur to any partition.

So, can anyone tell me how can I achieve this?
Also which function or IOCTL can be used to detect when the partition is actually mounted during boot?

I tried to use “IOCTL_DISK_IS_WRITABLE” IOCTL, after detecting it I am opening the bitmap file using “ZwOpenFile/ZwCreateFile” to read contents from it, but the system hangs over here and does not move on. I am not getting any error in “Windbg” also. What can be the possible cause?

Note: The file is present in the desired location(“\Device\Harddisk1\Partition1\file.txt”), when trying to open it.

Awaiting a positive response.
Thanks in advance.

then you trigger the call to zw* calls, your file systems *must* have
completed mount. If not, you will hang.

For a volume filters, that need to load configuration files *before* FS
mounts, you will have to go below the FS and write your own file reader,
which is capable of understanding disk/FS layouts and fetching the data.

A few ways to proceed:

  1. if you are going to make your meta data file FS accessible, then you
    will have to follow the rules of the FS to store he file. In this case, you
    will have to write the corresponding FS parser to read the file at boot
    time, given that FS code is not always open source, for commercial FS like
    NTFS you will face problems.
  2. If you are going to make your meta data FS agnostic, then you will have
    to store it on the disk outside the FSs reach, in free sectors or
    propitiatory partitions or something similar. This approach also has some
    limitations.
  3. Save the data in the registry, if it is not that huge. Different
    registry hives lead at different times of the boot process, so you will
    need to identify your happy place. Ofcourse this approach has it’s
    limitations as well, but then I don’t know your intent of doing this here,
    and I can only guess.

Hope this helps…

Amit

On Tue, Apr 1, 2014 at 4:37 PM, wrote:

> Hi,
>
> I am working on an upper volume filter driver based on diskperf.
> My driver actually monitors the sectors that are modified and sets the
> respective blocks in a bitmap. When the system is properly shutdown, the
> bitmap buffer is copied into a bitmap file.
>
> So now what my requirement is, when the system restarts, I need to copy
> the contents of this bitmap file into our bitmap buffer before any “Writes”
> occur to any partition.
>
> So, can anyone tell me how can I achieve this?
> Also which function or IOCTL can be used to detect when the partition is
> actually mounted during boot?
>
> I tried to use “IOCTL_DISK_IS_WRITABLE” IOCTL, after detecting it I am
> opening the bitmap file using “ZwOpenFile/ZwCreateFile” to read contents
> from it, but the system hangs over here and does not move on. I am not
> getting any error in “Windbg” also. What can be the possible cause?
>
> Note: The file is present in the desired
> location(“\Device\Harddisk1\Partition1\file.txt”), when trying to open
> it.
>
> Awaiting a positive response.
> Thanks in advance.
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>



- ab

Hi,

Thanks @amitr0 for the reply.
Actually the size of the bitmap file varies from 30Mb-500Mb. So I guess its not possible to store the file in the registry.
If the file is File System accessible then it would be preferred, but the main job is to store the bitmap buffer during proper shutdown at an appropriate location and again read the bitmap buffer from that location when the system restarts and that too if possible, before any “Writes” occurs to any partition.

So what could be the best way to achieve this?

As Amit said, the file system will not be available at this stage in the
boot process and therefore you need a method that does not involve the file
system.

However, there is a fundamental issue with what you’re trying to do: How do
you plan on handling offline modifications to the volume? What’s going to
stop someone from booting into another O/S (or the recovery console) and
making modifications that you won’t track?

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

Hi,

Thanks @amitr0 for the reply.
Actually the size of the bitmap file varies from 30Mb-500Mb. So I guess its
not possible to store the file in the registry.
If the file is File System accessible then it would be preferred, but the
main job is to store the bitmap buffer during proper shutdown at an
appropriate location and again read the bitmap buffer from that location
when the system restarts and that too if possible, before any “Writes”
occurs to any partition.

So what could be the best way to achieve this?

Hi, Thanks @Scott Noone for your reply.

>However, there is a fundamental issue with what you’re trying to do: How do you plan on handling offline modifications to the volume? What’s going to stop someone from booting into another O/S (or the recovery console) and making modifications that you won’t track?

We are not supporting offline modifications to the volume. We will consider it as abnormal activity.

If your driver starts during boot, it can open and read files from inside SystemRoot/Drivers. This happens during INT13 phase. In INT13 phase, any write is unlikely to happen.

The guy states that he can have a 500MB file. A heck of a lot of production
systems keep the system root volume pretty small. Customers I’ve dealt with
over the years would throw out the software and the supplier if you tried
writing a file of that size to the drivers directory.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@broadcom.com
Sent: Tuesday, April 08, 2014 10:08 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Read contents of a file during boot

If your driver starts during boot, it can open and read files from inside
SystemRoot/Drivers. This happens during INT13 phase. In INT13 phase, any
write is unlikely to happen.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

+1
My boot volume is very small. My secondary volumes (on my server) are
measured in multiple terabytes each.

My laptop has a large boot volume: 500 MB. I missed the start of this
thread, so I don’t know the basic purpose of this bitmap, but why would I
want to give between 6% and 100% of my disk over to this driver?

But the issue of “bitmap” raises some interesting questions, such as “why
isn’t it compressed?” and “How much would simple run-length coding
compress it?” and “What about zip compression?” All of which seem to make
more sense than writing out a gigantic uncompressed bitmap.

Next, what is the purpose of this bitmap? (I can’t find the start of the
thread). If it is in any way dealing with data security or data
integrity, then the “abnormal use” scenario could be used by anyone with
physical access to the machine to defeat security, compromise integrity,
or just implement a simple DoS attack. And if safe mode/recovery console
is required, and results in this bitmap getting out-of-sync with whatever
it is trying to represent, you MUST build a recovery mode into whatever
you are delivering. Otherwise, whatever you are building is fragile
software and will probably not survive any form of reality the rest of us
are familiar with.
joe

The guy states that he can have a 500MB file. A heck of a lot of
production
systems keep the system root volume pretty small. Customers I’ve dealt
with
over the years would throw out the software and the supplier if you tried
writing a file of that size to the drivers directory.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@broadcom.com
Sent: Tuesday, April 08, 2014 10:08 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Read contents of a file during boot

If your driver starts during boot, it can open and read files from inside
SystemRoot/Drivers. This happens during INT13 phase. In INT13 phase, any
write is unlikely to happen.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>We are not supporting offline modifications to the volume. We will
consider it as abnormal activity.

that argument might not hold up if you are designing a data protection
software like backupware. the user is supposed to trust your softeware’s
methods to protect the integrity of their drives, and would assume that all
data is correctly backed up, where as in reality you will be missing some
data which made it’s way to the disk because it was connected to a node
which didn’t have your driver running. The problem would only surface up
when the user wants to restore from the backup, and would find the data
either corrupt or completely missing, unfortunately for him it would be too
late to ask for a refund from you, most people take backup several times
but only restore when disaster strikes. And at that point an answer like
‘hey we wrote in the manual that we don’t support disk movement’ would be
the last thing we would want to hear.

the problem compounds because almost all (if not all) busses tese days
support hot removal and disks do move around…

On Tue, Apr 8, 2014 at 7:41 PM, Don Burn wrote:

> The guy states that he can have a 500MB file. A heck of a lot of
> production
> systems keep the system root volume pretty small. Customers I’ve dealt
> with
> over the years would throw out the software and the supplier if you tried
> writing a file of that size to the drivers directory.
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> xxxxx@broadcom.com
> Sent: Tuesday, April 08, 2014 10:08 AM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] Read contents of a file during boot
>
> If your driver starts during boot, it can open and read files from inside
> SystemRoot/Drivers. This happens during INT13 phase. In INT13 phase, any
> write is unlikely to happen.
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>



- ab

Hi,

First of all thanks all of you for your valuable inputs.

@amitr0: The software we are making is a sort of backup software, and we are not supporting hot removal of disk drives or modifications made in a way of which the OS is not aware of.
For eg: booting into repair mode and deleting/adding files.

At the moment I want to know if it is possible to read the file during boot and writing the file during shutdown?

Also @Don Burn has suggested me a solution on the MSDN forum, will try that first and if still I am not successful, will let you know all.

@Joe:

My laptop has a large boot volume: 500 MB. I missed the start of this thread, so I don’t know the basic purpose of this bitmap, but why would I want to give between 6% and 100% of my disk over to this driver?

What we have here is some failure to communicate. Your laptop drive is definitely NOT 500 MB, but some 1000 times greater. 500 MB may not even hold a bare Windows XP installation.

The bitmap supposedly is a bit per disk block (or per the volume allocation unit). As such, it’s up to 1/4096 of the total disk size.

On the unrelated note, Microsoft really needs to rework NTFS to give each file and a directory a modification sequence number. That will make detection and tracking of updates (and snapshotting) totally trivial. And NTFS needs to keep all previous versions of all files and directories (subject to a special FILE_ATTRIBUTE_PRESERVE attribute), which will make many many maintenance and recovery tasks much much easier. It’s not the same as periodic snapshotting. It’s keeping full history. Obviously, %TEMP% will not have PRESERVE attribute.

> @Joe:

>My laptop has a large boot volume: 500 MB. I missed the start of this
> thread, so I don’t know the basic purpose of this bitmap, but why would I
> want to give between 6% and 100% of my disk over to this driver?

What we have here is some failure to communicate. Your laptop drive is
definitely NOT 500 MB, but some 1000 times greater. 500 MB may not even
hold a bare Windows XP installation.

what we have here is not a failure to communicate, but a brain failure on
my part. Still, having to read in a 500MB file at boot time would make
the tedious boot process even longer.

The bitmap supposedly is a bit per disk block (or per the volume
allocation unit). As such, it’s up to 1/4096 of the total disk size.

On the unrelated note, Microsoft really needs to rework NTFS to give each
file and a directory a modification sequence number. That will make
detection and tracking of updates (and snapshotting) totally trivial. And
NTFS needs to keep all previous versions of all files and directories
(subject to a special FILE_ATTRIBUTE_PRESERVE attribute), which will make
many many maintenance and recovery tasks much much easier. It’s not the
same as periodic snapshotting. It’s keeping full history. Obviously,
%TEMP% will not have PRESERVE attribute.

I came from a mainframe background. NTFS, for all its purported features,
is an excellent example of a mid-to-late 1960s file system. Robust and
well-done, but philosophically identical. Our compilers can still only
accept punched cards, even if the images are now kept on disk instead of
cardboard. Why can’t I embed a Visio drawing of my data structures in my
source? A PowerPoint presentation? Put a Word document explaining what is
going on? Last September, I celebrated 50 years as a programmer, and our
representation of a program hasn’t changed at all! Sequential card
images. And it is appalling how many programmers do not understand
asynchronous event-driven programming, and insist on encapsulating
sequentiality in syntax.

NTFS should allow an active file to be read or written. What happens is
that the blocks of the old file are retained for the open handle(s), but
unless there is a file lock in place, the file can be overwritten by a
newer version. When I started using TOPS-20 in, about 1974 (forty years
ago) it had a robust file system that maintained a version count as part
of the file name; if the filename prsented did not include a version
number, the most recent version was opened. This was state-of-the-art on
machines that had gigantic 40MB drives, and a main memory of, perhaps, a
megabyte (if you had enough money). The TOPS-10, based on a 1963 design
for the PDP-6, was only about as powerful as NTFS, and we considered it
antiquated in the mid-1970s.
joe


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Does it have to be strictly “a file” and not, for example “a set of
reserved sectors on the disk”? The set of reserved sectors *could also be
a file*, but reading sectors at boot time is easier than reading a file.

Mark Roddy

On Wed, Apr 9, 2014 at 2:51 AM, wrote:

> Hi,
>
> First of all thanks all of you for your valuable inputs.
>
> @amitr0: The software we are making is a sort of backup software, and we
> are not supporting hot removal of disk drives or modifications made in a
> way of which the OS is not aware of.
> For eg: booting into repair mode and deleting/adding files.
>
> At the moment I want to know if it is possible to read the file during
> boot and writing the file during shutdown?
>
> Also @Don Burn has suggested me a solution on the MSDN forum, will try
> that first and if still I am not successful, will let you know all.
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

> On the unrelated note, Microsoft really needs to rework NTFS to give each file and a directory a

modification sequence number.

What is the need in this if they have VolSnap and VSS?


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> I came from a mainframe background. NTFS, for all its purported features,

is an excellent example of a mid-to-late 1960s file system. Robust and
well-done, but philosophically identical. Our compilers can still only
accept punched cards, even if the images are now kept on disk instead of
cardboard. Why can’t I embed a Visio drawing of my data structures in my
source? A PowerPoint presentation? Put a Word document explaining what is
going on?

And it is very good. Put your Office docs near the source, but not with the source.

Disks still use 512 (well, sometimes 4096) bytes per sector, like in 1980ies. And this is fine.

The file abstraction is classic and should not change. Even NTFS’s streams are evil I think.

Sets of files are a solution, if you need something non-trivial.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> Hi,

First of all thanks all of you for your valuable inputs.

@amitr0: The software we are making is a sort of backup software, and we
are not supporting hot removal of disk drives or modifications made in a
way of which the OS is not aware of.
For eg: booting into repair mode and deleting/adding files.

But when someone does that, for example, to delete a bad device driver (a
very real scenario, and not just for driver developers!) you have to have
some way to get your vision of the disk in sync with reality.
joe

At the moment I want to know if it is possible to read the file during
boot and writing the file during shutdown?

Also @Don Burn has suggested me a solution on the MSDN forum, will try
that first and if still I am not successful, will let you know all.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>Disks still use 512 (well, sometimes 4096) bytes per sector, like in 1980ies. And this is fine.

All disks you can buy today use 4K physical blocks. They emulate 512 bytes, of course.

@Max:

On the unrelated note, Microsoft really needs to rework NTFS to give each file
and a directory a
modification sequence number.

What is the need in this if they have VolSnap and VSS?

The modification seq number can greatly assist those programs that need to find out and scan modified directories and files, such as antiviruses, file indexers, and such. Without that, every time I boot it becomes a thrashfest of media indexing and scanning, and the box (with 8 GB) is less responsive for a few minutes than a 4 MB Windows 95.

>And NTFS needs to keep all previous versions of all files and directories (subject to a special FILE_ATTRIBUTE_PRESERVE attribute), >which will make many many maintenance and recovery tasks much much easier.

I haven’t spent a lot of time looking at it, but I though the newer ReFS was a write journaling file system.

Jan

>The modification seq number can greatly assist those programs that need to find out and scan modified directories and files

Isn’t that exactly what the NTFS USN journal is. http://en.wikipedia.org/wiki/USN_Journal

Jan