Hi All,
I was trying to write some data directly on sectors where one of my friend told me that windows keep some verification information with each sector at last two bytes and i have to take care of that while writing data.
My query is; Is it generic for all file systems. Is there any thing apart from this which needs to be consider while modifying data directly to disk. (from user mode app). Do we have any article/links on this topic(I am already searching it in parallel).
Thanks,
Aditya
Incredibly efficient way to screw up your target system PERMANENTLY and without any hope of recovery - although data-recovery companies may recover data on a disk that got mechanically/electrically/chemically
damaged, they are totally out of luck when it comes to data that got destroyed by deliberate writes…
Anton Bassov
When I wrote the post I was expecting few similar answer. 
But in case I am 100 % sure that the sector to which i am writing is a data sector of a specific target file and my task is to change few bytes in that sector, I can do that safely after knowing all ifs and buts. And that is where the question comes.
I simply want to know the points to consider.
Thanks
Aditya
>
I was trying to write some data directly on sectors where one of my
friend
told me that windows keep some verification information with each
sector
at last two bytes and i have to take care of that while writing data.
Only really old filesystems (amiga ofs is the only one that springs to
mind) stored metadata in the same sector as data. If sectors contain
data only then you can dma them into memory and the data is just there,
ready to use, which greatly improves performance.
My query is; Is it generic for all file systems. Is there any thing
apart
from this which needs to be consider while modifying data directly to
disk. (from user mode app). Do we have any article/links on this
topic(I
am already searching it in parallel).
Points to consider are:
. You will most likely upset any application that is currently using the
file
. Between the time when you figure out which sector you want to write
to, and the time you go to write it, windows may have moved the file and
it might turn out that you are writing over something else.
. You will most likely destroy your filesystem
I could go on…
Why do you want to do this? When asking a question on a mailing list, it
is a really good idea to give some sort of reason for why you want to do
it this way and not the ‘correct’ way, otherwise you will most likely
just get told “don’t do it this way”.
James
On Wed, Jan 7, 2009 at 11:20 AM, wrote:
> Hi All,
>
>
> I was trying to write some data directly on sectors where one of my friend
> told me that windows keep some verification information with each sector at
> last two bytes and i have to take care of that while writing data.
As far as I know this is related to NTFS for sanity checking.
If I remember correctly, these last two bytes are called as Magic Number (or
something like that)
All sectors in a cluster have the same magic number and thus helps in
ensuring that this cluster is corrupted or not.
I don’t see anything like this in FAT.
I think these are the only two filesystems which should concern you.
ext2/3 FS might have some similar functionalities.
Keep reverse engineering!!!
Deepak
>
> My query is; Is it generic for all file systems. Is there any thing apart
> from this which needs to be consider while modifying data directly to disk.
> (from user mode app). Do we have any article/links on this topic(I am
> already searching it in parallel).
>
> Thanks,
> Aditya
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
As far as I know this is related to NTFS for sanity checking. If I remember correctly, these last two bytes are called as Magic Number (or something like that) All sectors in a cluster have the same magic number and thus helps in ensuring that this cluster is corrupted or not. I don’t see anything like this in FAT.
>Thanks Deepak, It helps.
>Thanks for ur inputs.
Why do you want to do this?
>In short It is more like a experiment at analysis level, which will decide our future approach to change a FAT DIRENT information. not included this information initially to reduce the length of post.
Aditya
You may be referring to the boot sector and the master boot record which are
file system independent sectors on disk, which have 0x55, 0xaa at offset 510
and 511 into the sector (even if the sectors are bigger, which they rarely
are).
After that MS publish the spec for FAT filing systems, but I think that NTFS
is still secret, although no doubt available under some sort of NDA. (If
anyone knows of a full public spec for this, please share).
You can experiment from user mode, as raw disk access is permissible under
windows (need to run as admin in Vista). AFAIR you don’t get to see the
master boot record as when you open a “disk” for raw access the first sector
you see is the boot sector for that partition.
This opens disk D: for raw access:
char _devicename = “\\.\D:”;
hSimDisk = CreateFile(_devicename,GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);
This causes Windows to dismount the drive from its filing system, so you are
on your own until you close it.
You can now reverse engineer away and it is especially easy to zap the disk,
often in ways that Windows can’t repair, so expect to reformat it frequently
after you write to it!
M.
>>>>----- Original Message -----
From: xxxxx@gmail.com
To: Windows System Software Devs Interest List
Sent: Wednesday, January 07, 2009 7:46 AM
Subject: RE:[ntdev] Writing data to sector - Points to consider
As far as I know this is related to NTFS for sanity checking. If I remember
correctly, these last two bytes are called as Magic Number (or something
like that) All sectors in a cluster have the same magic number and thus
helps in ensuring that this cluster is corrupted or not. I don’t see
anything like this in FAT.
>Thanks Deepak, It helps.
>Thanks for ur inputs.
Why do you want to do this?
>In short It is more like a experiment at analysis level, which will
>decide our future approach to change a FAT DIRENT information. not
>included this information initially to reduce the length of post.
Aditya
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
> After that MS publish the spec for FAT filing systems, but I think
that
NTFS is still secret, although no doubt available under some sort of
NDA.
(If anyone knows of a full public spec for this, please share).
The ntfs-3g filesystem under Linux is the most robust re-implementation
of ntfs that I know of, and it’s open source so that’s probably the
closest you’ll get to a public spec.
James
Windows does not keep verification data, use FSCTL_GET_RETRIEVAL_POINTERS
and retrieve the blocks on a system then read them raw, you will find all
the data is part of the file, no special data. There are a number of file
system filters that use this call to then be able to manipulate data
directly, none of them worry about special data.
–
Don Burn (MVP, Windows DDK)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply
wrote in message news:xxxxx@ntdev…
> Hi All,
>
>
> I was trying to write some data directly on sectors where one of my friend
> told me that windows keep some verification information with each sector
> at last two bytes and i have to take care of that while writing data.
>
> My query is; Is it generic for all file systems. Is there any thing apart
> from this which needs to be consider while modifying data directly to
> disk. (from user mode app). Do we have any article/links on this topic(I
> am already searching it in parallel).
>
> Thanks,
> Aditya
>
>
> and retrieve the blocks on a system then read them raw
… and somehow protect yourself from defrag, either by prohibiting it
or by detecting it (pause your stuff, refresh chunk list after defrag,
proceed.),
whichever is better in your circumstances.
----- Original Message -----
From: “Don Burn”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Wednesday, January 07, 2009 8:28 AM
Subject: Re:[ntdev] Writing data to sector - Points to consider
> Windows does not keep verification data, use FSCTL_GET_RETRIEVAL_POINTERS
> and retrieve the blocks on a system then read them raw, you will find all
> the data is part of the file, no special data. There are a number of
> file system filters that use this call to then be able to manipulate data
> directly, none of them worry about special data.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
> Remove StopSpam to reply
>
>
>
>
> wrote in message news:xxxxx@ntdev…
>> Hi All,
>>
>>
>> I was trying to write some data directly on sectors where one of my
>> friend told me that windows keep some verification information with each
>> sector at last two bytes and i have to take care of that while writing
>> data.
>>
>> My query is; Is it generic for all file systems. Is there any thing apart
>> from this which needs to be consider while modifying data directly to
>> disk. (from user mode app). Do we have any article/links on this topic(I
>> am already searching it in parallel).
>>
>> Thanks,
>> Aditya
>>
>>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
@Mike
>you don’t get to see the master boot record as when you open a “disk” for raw access the first sector you see is the boot sector for that partition.
We can see MBR from user mode. one has to use \.\PHYSICALDRIVE0 to able to see MBR.
>You may be referring to the boot sector and the master boot record
Well Not this time :-), I was talking about magic bytes that NTFS inserts at last of every sector as mentioned in post by Deepak.
Point noted. I will take care of this, its important. Do we have some method to check if a defrag is going on?
@Don
We have a parser for NTFS mft. I checked the code and found that it manipulates last two bytes of every sector, using fixupsize and fixupoffset value.(it get actual data for the last two bytes from another location).
So i wondered like will it be same for all file system or is it NTFS specific as FAT docs not does mentioned about such thing.
Thank you all for the suggestions.
Aditya
> I was trying to write some data directly on sectors where one of my friend told me that windows keep
some verification information with each sector at last two bytes and i have to take care of that while
writing data.
This is true only about some (not all) NTFS metadata, and these 2 bytes are included to the sector payload as the disk stack sees this. So, even for this, the disk stack is agnostic about this NTFS feature.
Just write the sectors directly with correct alignment, address and size.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
The MFT is a metadata file, so yes there is special stuff. Your original
post did not say you were playing in files that NTFS keeps proprietary, and
subject to change. As Anton said in the first response to your post, this
is a great way to crash systems, but not do much else.
–
Don Burn (MVP, Windows DDK)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
wrote in message news:xxxxx@ntdev…
> @Mike
>>>you don’t get to see the master boot record as when you open a “disk” for
>>>raw access the first sector you see is the boot sector for that
>>>partition.
>
> We can see MBR from user mode. one has to use \.\PHYSICALDRIVE0 to able
> to see MBR.
>
>>>You may be referring to the boot sector and the master boot record
> Well Not this time :-), I was talking about magic bytes that NTFS inserts
> at last of every sector as mentioned in post by Deepak.
>
> @Alex
>
> Point noted. I will take care of this, its important. Do we have some
> method to check if a defrag is going on?
>
> @Don
> We have a parser for NTFS mft. I checked the code and found that it
> manipulates last two bytes of every sector, using fixupsize and
> fixupoffset value.(it get actual data for the last two bytes from another
> location).
>
> So i wondered like will it be same for all file system or is it NTFS
> specific as FAT docs not does mentioned about such thing.
>
> Thank you all for the suggestions.
> Aditya
>
>
>As far as I know this is related to NTFS for sanity checking.
If I remember correctly, these last two bytes are called as Magic Number (or something like that)
“Sequence Number Stride” or so.
All sectors in a cluster have the same magic number and thus helps in ensuring that this cluster is
corrupted or not.
Yes, this is to detect non-atomic multi-sector writes. This exists only for a few metadata streams in NTFS - logfile and IIRC MFT in particular.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
> char _devicename = “\\.\D:”;
hSimDisk = CreateFile(_devicename,GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);This causes Windows to dismount the drive from its filing system
No, it does not. It just takes a lock within the FSD which prevents any other file opens on the volume. Also I think you will need to drop FILE_SHARE_xxx for this effect.
If you really need to remount the FS volume after direct on-disk update, use:
This is how FORMAT and CHKDSK /F worked in XP.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
User mode raw write access:
Yes, sorry, its a long time since I did that, I just checked and for raw
write access from user mode I found this was needed to explicitly dismount
it from Windows…
DWORD result = DeviceIoControl( hSimDisk, // handle to volume
FSCTL_DISMOUNT_VOLUME, // dwIoControlCode
NULL, // lpInBuffer
0, // nInBufferSize
NULL, // lpOutBuffer
0, // nOutBufferSize
&bytesReturned, // number of bytes returned MUST BE PRESENT. BUT
MEANINGLESS APPARENTLY
NULL ); // OVERLAPPED structure CAN BE NULL IF OPENED NON
OVERLAPPED
I never needed to remount it to Windows for my experiments, it was
accessible again after the app exited. (Vista admin and XP). This is useful
for prototyping embedded systems on a PC test bed.
Thanks for additional info and info about NTFS-3g from James.
M…
>>>----- Original Message -----
From: Maxim S. Shatskih
Newsgroups: ntdev
To: Windows System Software Devs Interest List
Sent: Wednesday, January 07, 2009 3:04 PM
Subject: Re:[ntdev] RE:Writing data to sector - Points to consider
char _devicename = “\\.\D:”;
hSimDisk = CreateFile(_devicename,GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, 0, NULL);This causes Windows to dismount the drive from its filing system
No, it does not. It just takes a lock within the FSD which prevents any
other file opens on the volume. Also I think you will need to drop
FILE_SHARE_xxx for this effect.
If you really need to remount the FS volume after direct on-disk update,
use:
This is how FORMAT and CHKDSK /F worked in XP.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
> Point noted. I will take care of this, its important. Do we have some
method to check if a defrag is going on?
I inherited some volume backup (from XIX century, I guess) which
simply checked whether the defrag is running and spit out an
appropriate MsgBox (“stop it” or “don’t do that”),
and I was not supposed to modify this particular
piece of s[oftware].
Sorry.
But I’m sure it’s doable in an intelligent way.
----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Wednesday, January 07, 2009 9:46 AM
Subject: RE:[ntdev] Writing data to sector - Points to consider
> @Mike
>>>you don’t get to see the master boot record as when you open a “disk” for
>>>raw access the first sector you see is the boot sector for that
>>>partition.
>
> We can see MBR from user mode. one has to use \.\PHYSICALDRIVE0 to able
> to see MBR.
>
>>>You may be referring to the boot sector and the master boot record
> Well Not this time :-), I was talking about magic bytes that NTFS inserts
> at last of every sector as mentioned in post by Deepak.
>
> @Alex
>
> Point noted. I will take care of this, its important. Do we have some
> method to check if a defrag is going on?
>
> @Don
> We have a parser for NTFS mft. I checked the code and found that it
> manipulates last two bytes of every sector, using fixupsize and
> fixupoffset value.(it get actual data for the last two bytes from another
> location).
>
> So i wondered like will it be same for all file system or is it NTFS
> specific as FAT docs not does mentioned about such thing.
>
> Thank you all for the suggestions.
> Aditya
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
In my past life, I have written a NTFS parser. Linux’s ntfs was helpful
but not fully sufficient in reading all data structures correctly.
All sectors in a cluster have the same magic number and thus helps in
ensuring that this cluster is corrupted or not.
I am not sure of this.
Harish
-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Wednesday, January 07, 2009 7:01 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Writing data to sector - Points to consider
As far as I know this is related to NTFS for sanity checking.
If I remember correctly, these last two bytes are called as Magic
Number (or something like that)
“Sequence Number Stride” or so.
All sectors in a cluster have the same magic number and thus helps in
ensuring that this cluster is corrupted or not.
Yes, this is to detect non-atomic multi-sector writes. This exists only
for a few metadata streams in NTFS - logfile and IIRC MFT in particular.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
>In my past life, I have written a NTFS parser.
Me too
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
On Wed, Jan 7, 2009 at 11:57 PM, Arora, Harish wrote:
> In my past life, I have written a NTFS parser. Linux’s ntfs was helpful
> but not fully sufficient in reading all data structures correctly.
>
> 1. NTFS’s inode is 1 or more 1k sized mft records. Each mft record has 2
> sectors. A two byte checksum is calculated per sector and this checksum
> is stored somewhere(fixed offset!) in the sector. The original 2 bytes
> at that offset are stored in the last two bytes of the sector. Linux’s
> ntfs source will show you how to work with this.
> 2. only mft records and some other metadata structures contain this 2
> byte fixup. User data is written as it is.
> 4. to safely manipulate any user data on disk, your application has to
> open the raw volume exclusively and then lock it. Then with the same
> file handle you can write the data. This will guarantee that there are
> no other readers|writers of that FS at the time when you are
> manipulating it.
>
It has been long time since I worked on ntfs. I goofed it up a little bit.
Harish is correct.
These two bytes are called as FIXUPS and many important Metadata Records
(like FILE, INDX,
RCRD, RSTR records) use fixups to protect data integrity.
If we take an example of a INDX record, it contains file and directory
information and span over sectors.
last two bytes of each following sector in the record will be stored in
first sector of record and all these sectors will have a common checksum
value as their last two bytes.
For detailed info on NTFS docs, Please follow the link
http://data.linux-ntfs.org/ntfsdoc.html.gz#concept_fixup
> >All sectors in a cluster have the same magic number and thus helps in
> >ensuring that this cluster is corrupted or not.
> I am not sure of this.
All sectors in a cluster which contain above mentioned records.
Regards
Deepak
>
>
> Harish
>
> -----Original Message-----
> From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
> Sent: Wednesday, January 07, 2009 7:01 AM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Writing data to sector - Points to consider
>
> >As far as I know this is related to NTFS for sanity checking.
> >If I remember correctly, these last two bytes are called as Magic
> >Number (or something like that)
>
> “Sequence Number Stride” or so.
>
> >All sectors in a cluster have the same magic number and thus helps in
> >ensuring that this cluster is corrupted or not.
>
> Yes, this is to detect non-atomic multi-sector writes. This exists only
> for a few metadata streams in NTFS - logfile and IIRC MFT in particular.
>
> –
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>