VSS Skipping Files in \ProgramData\Microsoft\Crypto\RSA\MachineKeys and others

NTDEV
1

Hi,

My employer makes system backup software, and once in a while we get customers who are unable to complete their PC backup for certain files. We were able to reproduce the issue locally, and it turns out that certain files are skipped during the creation of a VSS snapshot, simply not present when accessed via the VSS snapshot prefix.

Further analysis showed that (this is under Vista and 7) the “Owner” of these skipped files is something weird: when attempting to view the security settings, the dialog states that we do not have access to this information and upon viewing the actual owner, it says to the effect of “Cannot display owner” - though I can (as an elevated admin) change the owner explicitly, and then take full control of the file.

Just for the record, this is not a TrustedInstaller issue.

Now, I have found a workaround for this, but I really hope there is another way since mine is rather obtrusive. What I have found to work is to use GetNamedSecurityInfo to get the owner, store the PSID, use SetNamedSecurityInfo to change the owner to the Administrators group, create the VSS snapshot, back the file up, then change the owner back to the weird PSID that I set aside earlier.

Obviously, this is a very nasty hack (at best)… but it works. And it’s necessary that the VSS snapshot contain all these files. Googling has been of no help, and I cannot see why *some* of the files in the MachineKeys folder would give me this weird owner and others would display “SYSTEM” just fine.

Summary of questions:

  1. Explanation for this weird owner
  2. Why does VSS skip these files, and is there a way to make them available to the VSS snapshot?
  3. Failing that, is temporarily changing the owner of these files a bad idea?

Thanks!

Mahmoud Al-Qudsi

2

Kudos for an exceptionally well formed problem description and series of questions. Seriously. I’m going to send people to this post to learn how to post a problem and ask for help.

Unfortunately, I don’t have any knowledge or experience that would help with your problem… sorry about that. But I *did* want to commend your wonderfully clear post. Thanks for that.

Peter
OSR

3

These files are apparently too valuable for the OS to allow anybody to read them. As you may guess, these are encryption keys. Is Windows Backup able to read them?

4

Peter - Thank you! I truly appreciate the sentiment.

Alex, that’s what I originally thought myself. However, when I discovered that only some of the keys were set to an odd owner on one computer, and all the keys on other PCs were owned by the usual ‘SYSTEM’ … well, it didn’t chime.

When searching for an explanation to this phenomena, I ran into posts by users of Symantec software, Windows Home Server Backup, and others all complaining of similar issues (all these products use VSS, which is basically the only way really, unless you write your own VSS-like provider as I believe Acronis did). I’m actually not aware of anyone even solving the problem via the workaround I described above, I myself highly doubted it would work until I tried it (I had expected GetNamedSecurityInfo to fail seeing as Windows was unable to get the owner when viewing file properties).

Regards,

Mahmoud Al-Qudsi
NeoSmart Technologies

5

> 1) Explanation for this weird owner

What is the SID literally?

  1. Why does VSS skip these files, and is there a way to make them available to the VSS snapshot?

This can occur if some VSS writer deletes them from the snapshot at PostFinalCommit phase. Why? because this writer does not want these files to be backed up, plain and simple.

Use the “no autorecovery” flag when creating the snapshot, this will ban PostFinalCommit at all.

Or exclude the offending writer.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

6

>these are encryption keys. Is Windows Backup able to read them?

Any image sector-wise backup can.

File-based backup is actually obsolete, at least if we are speaking about the volumes with installed OSes.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

7

>products use VSS, which is basically the only way really, unless you write your own VSS-like provider

as I believe Acronis did).

VSS is a must if you want to really correctly commit the writers (like MSSQLServer or Exchange, or even NTDS/AD) and have the writer’s databases in the correct state in the backup image.

Without VSS, when you will boot the restored system, the database apps (writers) will behave the same way as after accidental reboot due to power failure or BSOD, and there are no other ways of solving this except a) VSS OR b) writing lots of app-specific code, and it is IIRC documented only for SQL Server.

For instance, just exclude Hyper-V VSS writer and back up the Hyper-V host with running guests. Then restore it, start the guests and… they will show the recovery options menu with “Start Windows Normally”. There was a joint bug of some backup software titles and MS about 2008 R2 and Hyper-V writer, but it was fixed both sides from the backup vendors and from MS (as a special hotfix not on Windows update).

After years of being unable to correctly back up the running Exchange server’s databases, Acronis TrueImage switched to use VSS and created their own VSS provider.

Now note that Acronis (Symantec Ghost/LiveState, StorageCraft and several other companies) provide sector-level backup software. Such backups are restored by just plain writing the sectors back to the volume and rebooting to this volume. I cannot even imagine why VSS writers must be involved in restore in such a case, since the backup is the complete sector-to-sector snapshot of the whole system (set of volumes).

Now about PostFinalCommit and “no autorecovery” flag.

PostFinalCommit is the phase when the VSS writers are given the ability to post-process the snapshot (which is made temporarily available for writing at this moment) and to delete/update anything the writers do not want to be backed up.

From how I understood the docs, this is mainly to make Jet (and many OS-embedded databases are Jet databases) databases in the backup image to be openable off the read-only media like a DVD.

I cannot understand why such a feature should be honored for sector-wise backups, whose whole purpose is to make a complete snapshot of the whole system, without any postprocessing. Just make a point-in-time copy.

So, sector-wise backups specify “no autorecovery” and are happy. If Jet database in the backup image itself will not be loadable by Jet off read-only media - well, an issue, but a minor one.

Also note that good sector-wise backup products provide you with incremental backups, and, as you can understand, PostFinalCommit is just plain not compatible with sector-wise incrementals. When we give some “side guys” (VSS writers in this case) the ability of post-processing the snapshot (aka backup image - the same data), we lose the incremental chain continuity at this very moment.

So, “no autorecovery” is the way to go.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

8

Maxim, thank you for your replies. I was actually hoping you would chime in :slight_smile:

You’ve given me a lot to think about - I realize that the VSS code we have will need to be revised to take all this into consideration. As a quick test, I attempted to just add the no autorecovery flag to the context initialization parameter, but the same issue persisted.

I would like to point out something though: I don’t believe it’s a post-processing issue with the MachineKeys folder. See, the key in the MachineKeys folder is NOT open by anyone and not in use, just inaccessible due to the weird permissions. Apply my workaround listed in the first post of this thread enables backup of this file, and the file remains present in the VSS snapshot afterwards. It would seem that VSS actually doesn’t have the permission to take this file (though I realize that sounds silly, given how VSS works). But I can find no other explanation for the fact that simply changing the owner (manually or via SetNamedSecurityInfo) lets the file persist in the VSS snapshot, otherwise it’s not there.

As further confirmation, other files on the system do indeed remain vanished from the VSS snapshot even when the owner is forced to the current user - these files however are in use and by products that I’m certain are VSS-aware (Windows Search is one example, with its files in the ProgramData folder being removed from the VSS snapshot).

So it’s not a PostFinalCommit issue with the MachineKeys issue we’re experiencing - it’s actually prevented from ever entering the VSS snapshot index.

9

>

Maxim, thank you for your replies. I was actually hoping you would
chime in :slight_smile:

You’ve given me a lot to think about - I realize that the VSS code we
have
will need to be revised to take all this into consideration. As a
quick test,
I attempted to just add the no autorecovery flag to the context
initialization
parameter, but the same issue persisted.

I would like to point out something though: I don’t believe it’s a
post-
processing issue with the MachineKeys folder. See, the key in the
MachineKeys
folder is NOT open by anyone and not in use, just inaccessible due
to the
weird permissions. Apply my workaround listed in the first post of
this thread
enables backup of this file, and the file remains present in the VSS
snapshot
afterwards. It would seem that VSS actually doesn’t have the
permission to
take this file (though I realize that sounds silly, given how VSS
works). But
I can find no other explanation for the fact that simply changing the
owner
(manually or via SetNamedSecurityInfo) lets the file persist in the
VSS
snapshot, otherwise it’s not there.

As further confirmation, other files on the system do indeed remain
vanished
from the VSS snapshot even when the owner is forced to the current
user -
these files however are in use and by products that I’m certain are
VSS-
aware (Windows Search is one example, with its files in the
ProgramData folder
being removed from the VSS snapshot).

So it’s not a PostFinalCommit issue with the MachineKeys issue we’re
experiencing - it’s actually prevented from ever entering the VSS
snapshot
index.

Are you actually invoking the system state writer when you do your
backup?

James

10

> So it’s not a PostFinalCommit issue with the MachineKeys issue we’re experiencing - it’s actually

prevented from ever entering the VSS snapshot index.

Try to run VSS requester without including any writers. Will the files disappear from the snapshot?

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

11

A lot of ideas here - to be honest, I’m not the one that wrote the VSS code originally; I was however tasked with finding out why these files were being skipped.

It’s going to take me some time to thoroughly read about and understand all the intricacies and components of the VSS subsystem, I thank you all for your help :slight_smile: