Last sector of partition created for NTFS volume

What I am trying to achieve :

  • Use the NTFS bitmap to read data from a volume (consider no write activity on the volume)
  • Layout this data into a VHD with a NTFS volume
  • Update the volume serial number for the volume thus created on the VHD. This is needed because I see Disk Management related interfaces stuck up for some time for a few cases where the original volume was LDM based (mirror, spanned etc). The volume serial number is seen on the first and last sector of the partition.

What I have observed :
The last sector of the partition created for the volume is not part of the volume itself (considering sector size = cluster size, aligned starting LBA).
Example : Created a 512 MB volume using Disk Management MMC.
fsutil fsinfo ntfsinfo vol: shows :
C:>fsutil fsinfo ntfsinfo p:
NTFS Volume Serial Number : 0xe45accaf5acc8032
NTFS Version : 3.1
LFS Version : 2.0
Number Sectors : 0x00000000000fffff
Total Clusters : 0x00000000000fffff
Free Clusters : 0x00000000000f9e20
Total Reserved : 0x0000000000000000
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 512
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 2
Mft Valid Data Length : 0x0000000000040000
Mft Start Lcn : 0x0000000000055555
Mft2 Start Lcn : 0x0000000000000010
Mft Zone Start : 0x0000000000055740
Mft Zone End : 0x0000000000075560
Resource Manager Identifier : CDED0F2A-95BA-11E6-80CF-00505699EE53

GetFreeDiskSpaceEx() gives the volume size as 536870400 bytes

IOCTL_DISK_GET_LENGTH_INFO gives the volume size as 536870912 bytes which is 1 sector more than the clusters / sectors seen by fsutil info.

As per diskpart detail partition / detail volume also, we can see that partition size is more than volume size.

This last sector has the same data replicated as first sector [confirmed this by creating the volume on a VHD, reading the GPT to get details of partition boundaries, detaching the VHD and then doing reads at concerned offsets within the VHD file]

This seems to be the boot sector as per details given at https://technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx / https://blogs.technet.microsoft.com/askcore/2010/10/08/gpt-in-windows/ (The 2nd link claims that the replicated boot sector will be part of the volume).

Trying to read this last sector using the volume handle does not work (seek fails). Tried using FSCTL_ALLOW_EXTENDED_DASD_IO on the volume handle but that does not help either.

Question:
- Any explanation on why the replicated first sector is not accounted for in the NTFS clusters / sectors data ?
- Any suggestions on how to read the actual last sector using volume handle ?</vol:>

The missing sector is at the start of the disk. The partition starts after
the MBR and the volume starts at the beginning of the partition.

The question becomes, “do you want to close the disk or do you want to clone
the volume?”

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

What I am trying to achieve :

  • Use the NTFS bitmap to read data from a volume (consider no write activity
    on the volume)
  • Layout this data into a VHD with a NTFS volume
  • Update the volume serial number for the volume thus created on the VHD.
    This is needed because I see Disk Management related interfaces stuck up for
    some time for a few cases where the original volume was LDM based (mirror,
    spanned etc). The volume serial number is seen on the first and last sector
    of the partition.

What I have observed :
The last sector of the partition created for the volume is not part of the
volume itself (considering sector size = cluster size, aligned starting
LBA).
Example : Created a 512 MB volume using Disk Management MMC.
fsutil fsinfo ntfsinfo vol: shows :
C:&gt;fsutil fsinfo ntfsinfo p:
NTFS Volume Serial Number : 0xe45accaf5acc8032
NTFS Version : 3.1
LFS Version : 2.0
Number Sectors : 0x00000000000fffff
Total Clusters : 0x00000000000fffff
Free Clusters : 0x00000000000f9e20
Total Reserved : 0x0000000000000000
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 512
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 2
Mft Valid Data Length : 0x0000000000040000
Mft Start Lcn : 0x0000000000055555
Mft2 Start Lcn : 0x0000000000000010
Mft Zone Start : 0x0000000000055740
Mft Zone End : 0x0000000000075560
Resource Manager Identifier : CDED0F2A-95BA-11E6-80CF-00505699EE53

GetFreeDiskSpaceEx() gives the volume size as 536870400 bytes

IOCTL_DISK_GET_LENGTH_INFO gives the volume size as 536870912 bytes which is
1 sector more than the clusters / sectors seen by fsutil info.

As per diskpart detail partition / detail volume also, we can see that
partition size is more than volume size.

This last sector has the same data replicated as first sector [confirmed
this by creating the volume on a VHD, reading the GPT to get details of
partition boundaries, detaching the VHD and then doing reads at concerned
offsets within the VHD file]

This seems to be the boot sector as per details given at
https://technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx /
https://blogs.technet.microsoft.com/askcore/2010/10/08/gpt-in-windows/ (The
2nd link claims that the replicated boot sector will be part of the volume).

Trying to read this last sector using the volume handle does not work (seek
fails). Tried using FSCTL_ALLOW_EXTENDED_DASD_IO on the volume handle but
that does not help either.

Question:
- Any explanation on why the replicated first sector is not accounted for in
the NTFS clusters / sectors data ?
- Any suggestions on how to read the actual last sector using volume handle
?</vol:>

Thanks for the reply Scott.
Apologies in advance for a lengthy post below :

  • I want to clone the volume and not the disk (so that it works for LDM based volumes as well as Simple Volumes)

  • I am looking at data contained within the partition and not the entire disk and there I see partition having 1 extra sector, which is the replica of the first sector of the partition. The 1st sector is part of the NTFS volume but the last sector (replica) is not part of the volume. To substantiate this, I will give some details here :
    a) Created a VHD (fixed size) using Disk Management of 1024 MB
    b) Initialized it with GPT
    c) Created a 256 MB NTFS volume first, with cluster size = sector size = 512 (this was done to avoid causing any confusion in space left over by (Num Sectors * Sector Size) - (Num Clusters * Cluster Size), plus sector size of 512 to avoid any confusion over Advanced Format Disks)
    d) Created a 512 MB NTFS volume (again with sector size = cluster size = 512. I will use this volume in subsequent details, this is to avoid confusion over touching GPT header / footer section as this volume neither starts at the beginning of disk nor ends at the end of the disk)

Diskpart details of the aforesaid disk :

DISKPART> detail disk

Microsoft Virtual Disk
Disk ID: {3CC15E73-9BDF-4D39-9A01-A87EF6000D56}
Type : File Backed Virtual
Status : Online
Path : 0
Target : 0
LUN ID : 4
Location Path : UNAVAILABLE
Current Read-only State : No
Read-only : No
Boot Disk : No
Pagefile Disk : No
Hibernation File Disk : No
Crashdump Disk : No
Clustered Disk : No

Volume ### Ltr Label Fs Type Size Status Info


Volume 13 N New Volume NTFS Partition 256 MB Healthy
Volume 14 P New Volume NTFS Partition 512 MB Healthy

DISKPART> list partition

Partition ### Type Size Offset


Partition 1 Reserved 32 MB 17 KB
Partition 2 Primary 256 MB 32 MB
Partition 3 Primary 512 MB 288 MB

DISKPART> select partition 3

Partition 3 is now the selected partition.

DISKPART> detail partition

Partition 3
Type : ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
Hidden : No
Required: No
Attrib : 0000000000000000
Offset in Bytes: 302055424

Volume ### Ltr Label Fs Type Size Status Info


* Volume 14 P New Volume NTFS Partition 512 MB Healthy

DISKPART> select volume 14

Volume 14 is the selected volume.

DISKPART> detail volume

Disk ### Status Size Free Dyn Gpt


* Disk 8 Online 1024 MB 223 MB *

Read-only : No
Hidden : No
No Default Drive Letter: No
Shadow Copy : No
Offline : No
BitLocker Encrypted : No
Installable : Yes

Volume Capacity : 511 MB
Volume Free Space : 495 MB

DISKPART>


Again, re-iterating what I mentioned earlier, GetFreeDiskSpaceEx() gives the volume size as 536870400 bytes (1048575 sectors) whereas IOCTL_DISK_GET_LENGTH_INFO gives the volume size as 536870912 (1048576 sectors).

fsutil fsino ntfsinfo for this volume is mentioned in the opening post, so avoiding repetition of info. It shows number of sectors for the volume as 0xFFFFF (1048575)

Diskpart also shows “Partition – Size” as 512 MB but “Volume Capacity” as 511 MB (unfortunately, it does not show at byte level)

Going by details given at https://technet.microsoft.com/en-us/library/cc781134(v=ws.10).aspx :

NTFS Boot Sector
The table Boot Sector Sections on an NTFS Volume describes the boot sector of a volume that is formatted with NTFS. When you format an NTFS volume, the format program allocates the first 16 sectors for the boot sector and the bootstrap code.

clusterinfo seems to match what the article says :

C:\Users\Administrator>fsutil volume querycluster p: 0
Cluster 0x0000000000000000 used by -S–D $Boot::$DATA

C:\Users\Administrator>fsutil volume querycluster p: 1
Cluster 0x0000000000000001 used by -S–D $Boot::$DATA


C:\Users\Administrator>fsutil volume querycluster p: 15
Cluster 0x000000000000000f used by -S–D $Boot::$DATA

C:\Users\Administrator>fsutil volume querycluster p: 16
Cluster 0x0000000000000010 used by -S–D $MftMirr::$DATA


The above command also fails for offset 0xFFFFF with invalid parameter indicating that the last sector of partition, beyond the volume boundary, is not part of the volume itself.

I also have an in-house tool for reading GPT header / footer from a VHD file and printing the details. Based on that too, I see partition having 1 more sector than what NTFS says for volume. If you know of any trusted open source GPT header dump tool, please point me to the same so that I can post results of that too and avoid any suspicion over my in-house tool’s correctness.
Dumping the 1st 512 and last 512 of the partition using dd (http://www.chrysocome.net/dd) over the VHD file (based on GPT header details) and then doing a sha1 over the data gives same result.
Dumping the 1st 512 of the volume (again using dd over the volume when VHD is in attached state) gives the same sha1 result.

As I write this, I realize that one experiment indeed is missing from my end - viz creaing a dynamic volume (say spanned over 2 disks) and to see the partition on each of the disks involved and whether there is any replica of first 512 bytes of the volume. Will experiment with that too and post my findings.

The choice to not count the final bootsector within the volume was a
fairly arbitrary implementation detail in NTFS. It could have been
accounted for within the volume as allocated; or it could be excluded
from the volume. I suspect the reason NTFS chose the latter are points
already covered in this thread - if it’s allocated, which file holds the
allocation (does $Boot become fragmented?); and since it’s a sector
which is less than a cluster, how is its location within that file
really described? Excluding it just seemed simpler.

The part in this thread that puzzles me is the claim that
FSCTL_ALLOW_EXTENDED_DASD_IO doesn’t work. The reason it was added is
the exact situation described in this thread. It is used by things like
chkdsk and format which require the ability to read and write boot
sectors outside of the scope of the file system (or previous file
system, in the case of format.) So my advice would be to look carefully
at FSCTL_ALLOW_EXTENDED_DASD_IO, confirm that it is being sent where you
intend it to go (ie., to NTFS) and that the IO to access the backup
bootsector is on the same handle.

  • M

Thank you Malcolm for the explanation.
You are right, there was a glitch in my FSCTL_ALLOW_EXTENDED_DASD_IO code. Rectified it and it allows to read the last sector beyond the NTFS cluster aligned boundary.