Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

Differential backups with Microsoft VSS provider?

Jason_T.Jason_T. Member Posts: 51

Using the in-box Microsoft VSS provider, is there a way to query only what has changed between an 'n' and 'n+1' snapshot? When I do a vssadmin -q I see that all of the snaps my system has taken are marked 'differential'. But my understanding of differential is that it is relative to a full, or base, snapshot. I do not see any such snapshot - every one is marked differential. So I'm not sure if I'm just misunderstanding what differential means in this context or what. Further, even if these are differential in terms of their behind-the-scenes allocation/implementation, is this metadata exposed to the user in any way? I'd like to perform remote backups to sync a series of local VSS snapshots, but obviously do not want to re-backup the entire volume on each subsequent snap if it hasn't changed. I believe 'incremental' is what I'd ideally like to have achieve (though I can't determine whether the ITB provider supports this backup type) but differential would at least be better than doing full backups on each snapshot.

-JT

Comments

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    There is CBT (Change Block Tracking) in more recent OSs for Hyper-V only. You'll need to develop a device driver to do this for generic volumes.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 6,836

    You'll need to develop a device driver to do this for generic volumes

    And, having done this exact project here at OSR multiple times, I can tell you with some confidence that this is not simple. The IOCTLs that you need to know about to successfully interact with VolSnap are not all documented. Sigh...

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    File based or image based backup?

  • Jason_T.Jason_T. Member Posts: 51

    I've built VSS providers in the past and I agree that it's not fun. I was looking at this product below, which claims to provide all changes between two VSS snapshots without a driver, and wondering if perhaps I were missing something basic about VSS capabilities regarding incremental/differential snaps:

    https://hystax.com/windows-changed-blocks-tracker/

    One thought I had was that you could potentially look at the USN stuff for the n and n+1 snaps and see what has changed that way. Of course, to my knowledge, I don't think this would capture any direct volume write activity, only regular FS writes, right?

    I'm needing a complete volume mirroring solution, so purely file-based is likely not feasible. But I am intrigued by that Hystax product and how they could achieve that sans driver, if in fact they do.

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    It maybe one of the most complex drivers to build outside of a full-blown FSD. There is a point where the snapshot transitions between read-only, read-write, and back to read-only. That is where you have to do your magic (GPT attributes are your friend). You really don't need to use any of the undocumented VSS IOCTLs; except the data structure passed to FLUSH_AND_HOLD. I guess you could really fancy up your API using VSS get and set app data IOCTLs, but it is not required.

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    According to their website, they are making diffs between snapshots, not using real-time change tracking:
    "Library reads deltas directly from VSS snapshots, no long API calls or data parsing

    On another note: UrBackup now has a paid option to get CBT.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 6,836

    But I am intrigued by that Hystax product

    Yes, me too. If you're already looking into this, would you be so kind as to report back here what you find?

    And I'd be curious about whether they support both crash-consistent and application-consistent snapshots.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    I guess Hystaxs keeps the previous snapshot around until the next backup. If you lose the snapshot for any reason, you also lose the next incremental.

  • Jamey_KirbyJamey_Kirby Member - All Emails Posts: 401

    They've exaggerated some of the points against having a KM driver.

    They are unsafe – every bug leads to system freeze or blue screen

    Exaggerated

    Doesn’t store data between reboots or can cause system locks if deltas are flushed to a >>file

    Lame. This is just not true. The file systems flush on reboots; right?

    Can cause problems and performance degradation with antivirus software

    Not always. Again, this is a reach.

    Slows down system

    What doesn't? There is no free ride. You are going to pay the price somewhere.

    Using driver is not a native method to track and capture changes on Windows devices

    Diffing snapshots is a unique approach. I'd have to see it in action.

  • Jason_T.Jason_T. Member Posts: 51

    I will report anything I can find, yes. I am still a bit skeptical of their claims, in particular the ones about volume-level backups. I don't see any possible way a user mode component can track raw volume writes. Even if they had reverse engineered the format of the MS storage area (the stuff inside of \System Volume Information) I don't see how they could access that from user mode.

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 6,836

    Doesn’t store data between reboots or can cause system locks if deltas are flushed to a file

    Well, that's entirely correct if the implementer of the driver is stupid.

    Slows down system

    For some measure of "slows down"... As Mr. Kirby said, doesn't everything?

    Using driver is not a native method to track and capture changes on Windows devices

    For some definition of "native method"... In fact, I have no idea what that means.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Jason_T.Jason_T. Member Posts: 51

    Caller can instantly get deltas in a specific file or on a disk with volume-based offsets

    Clearly their marketing team got a bit overzealous on this. We certainly know that regardless of the implementation their product can't provide a list of changed blocks "instantly", at best it would be linearly scaled with the number of changes. Compared to a brute force, full snap-to-snap diff, sure, it should be much better. But this and the comments you have both made are why I'm viewing their claims through a skeptical lens.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!