Fw: bad sectors

NTDEV
1

Thanks for your answers so far.
I probably was not clear enough:
The thread reads then encrypts then writes blocks of x sectors. The read
fails because an unknown number (1-n) of sectors within x is bad.

If, as somebody said, the data and size (ioStatusBlock.Information) should
be considered both unreliable when I recevied the sector fails, then even
the position within x would be unclear, and the data received so far I must
not use to encrypt and write back.

I do not need to encrypt this bad sector/s.
But I am to go on, in some intelligent manner skipping the bad sectors and
finding the next sane sectors (instead of leaving the volume alone). The
disk could contain a single bad sector, but it might as well be near total
corruption.

I heard that past the failure it is better to really go in single sectors
because each repeated hit to bad sectors is probably slower than going in
single sectors steps ?

Regards
Else

----- Forwarded by Else Kluger/Munich/Utimaco/DE on 01/09/2008 12:00 PM

|---------±-------------------------------->
| | Else Kluger |
| | | | e> |
| | Sent by: |
| | bounce-310636-16691@li|
| | sts.osr.com |
| | |
| | |
| | 01/09/2008 09:04 AM |
| | Please respond to |
| | “Windows System |
| | Software Devs Interest|
| | List” |
|---------±-------------------------------->
>-----------------------------------------------------------------------------------------------------------|
| |
| To: “Windows System Software Devs Interest List” |
| cc: |
| Subject: [ntdev] bad sectors |
>-----------------------------------------------------------------------------------------------------------|

Hi all,

I’m writing a sector based disk encryption filter.
If a bad sector is detected (=read request of the sector fails) during the
initial encryption of a volume I currently skip the volume leaving it only
partly encrypted. This seems to be not acceptable and our customers request
to fully encrypt such a volume.
- Can I rely on ioStatusBlock.Information returning me the size of read
bytes in front of the bad sector ?
- Is there an intelligent way / algorithm to do this without risking
further damage to the drive ?

Regards
Else

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

2

So why not then deconstruct your read X to read each sector of X, one sector
at a time? You then will obtain the information missing from your original
read of X regarding which sectors are un readable.

On Jan 9, 2008 6:43 AM, Else Kluger wrote:

> Thanks for your answers so far.
> I probably was not clear enough:
> The thread reads then encrypts then writes blocks of x sectors. The read
> fails because an unknown number (1-n) of sectors within x is bad.
>
> If, as somebody said, the data and size (ioStatusBlock.Information) should
> be considered both unreliable when I recevied the sector fails, then even
> the position within x would be unclear, and the data received so far I
> must
> not use to encrypt and write back.
>
> I do not need to encrypt this bad sector/s.
> But I am to go on, in some intelligent manner skipping the bad sectors and
> finding the next sane sectors (instead of leaving the volume alone). The
> disk could contain a single bad sector, but it might as well be near total
> corruption.
>
> I heard that past the failure it is better to really go in single sectors
> because each repeated hit to bad sectors is probably slower than going in
> single sectors steps ?
>
> Regards
> Else
>
>
> ----- Forwarded by Else Kluger/Munich/Utimaco/DE on 01/09/2008 12:00 PM
> -----
> |---------±-------------------------------->
> | | Else Kluger |
> | | > | | e> |
> | | Sent by: |
> | | bounce-310636-16691@li|
> | | sts.osr.com |
> | | |
> | | |
> | | 01/09/2008 09:04 AM |
> | | Please respond to |
> | | “Windows System |
> | | Software Devs Interest|
> | | List” |
> |---------±-------------------------------->
>
> >-----------------------------------------------------------------------------------------------------------|
> |
> |
> | To: “Windows System Software Devs Interest List” <
> xxxxx@lists.osr.com> |
> | cc:
> |
> | Subject: [ntdev] bad sectors
> |
>
> >-----------------------------------------------------------------------------------------------------------|
>
>
>
>
> Hi all,
>
> I’m writing a sector based disk encryption filter.
> If a bad sector is detected (=read request of the sector fails) during the
> initial encryption of a volume I currently skip the volume leaving it only
> partly encrypted. This seems to be not acceptable and our customers
> request
> to fully encrypt such a volume.
> - Can I rely on ioStatusBlock.Information returning me the size of read
> bytes in front of the bad sector ?
> - Is there an intelligent way / algorithm to do this without risking
> further damage to the drive ?
>
> Regards
> Else
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

–
Mark Roddy

3

Else Kluger wrote:

The thread reads then encrypts then writes blocks of x sectors. The read
fails because an unknown number (1-n) of sectors within x is bad.

If your encryption scheme is block-based, bad sectors in one block will
effectively invalidate all “good” data in the sectors after it.

Therefore if you encounter a block with bad sector/s, you want to (1)
relocate as much salvageable data as possible to a “good” block, (2)
mark the complete old block as unuseable, (3) destroy as much of the
readable data in the bad block as possible.

If the scheme is sector-based, you “just” need to make a best-effort
attempt to encrypt as much as possibly salvageable of the bad sectors
you read (effectively destroying the rest of the data in the sector,
which does not matter because it is “bad” anyway).

If you want to be file system independent, in case of a bad sector
things are really bad, because you cannot easily mark a sector or block
as “bad” for the file system.

Effectively you have to invent your own relocation scheme, e.g.
requiring a spare partition for relocation purposes. Or have the user
accept that you will irretrievably destroy some data. If s/he does not
accept, you would want to reollback (decrypt) everything and give the
user a chance to copy the data to a properly functioning device.

If, as somebody said, the data and size (ioStatusBlock.Information) should
be considered both unreliable when I recevied the sector fails, then even
the position within x would be unclear, and the data received so far I must
not use to encrypt and write back.

Correct. And to not leave the drive completely unuseable, you must
actually roll back the whole encryption process.

Question: What do you do when reading a bad sector in the encrypted
data later on? And how do you check the correctness of the decrypted
data in a block?

I do not need to encrypt this bad sector/s.

Well, if it is a security software you are writing, you should probably
at least try to destroy any possible plaintext data. Encrypting the read
data and a write attempt might serve very well to that purpose.

But I am to go on, in some intelligent manner skipping the bad sectors and
finding the next sane sectors (instead of leaving the volume alone).
> The disk could contain a single bad sector, but it might as well be
> near total corruption.

I heard that past the failure it is better to really go in single sectors
because each repeated hit to bad sectors is probably slower than going in
single sectors steps ?

The OS and hard drive will usually try multiple reads to recover the
sector data before giving up. Depending on your block size it is
probably better to read single sectors if a block read returned an error.

So far my 2 cents.
-H

4

hi,

the big problems here are those, which we encounter from time to time at our
WORM media:

there some written blocks are sometimes readable, but sometimes they are
not. It does not happen very often, but it happens. If it happens during
write (and the direct follow up control read afterwards) in my file system,
we exclude those blocks from the media. After all it is write once! But the
read errors (independently of read retries) sometimes only show up later and
vanish again …

– Reinhard

“Hagen Patzke” wrote in message news:xxxxx@ntdev…
> Else Kluger wrote:
>> The thread reads then encrypts then writes blocks of x sectors. The read
>> fails because an unknown number (1-n) of sectors within x is bad.
>
> If your encryption scheme is block-based, bad sectors in one block will
> effectively invalidate all “good” data in the sectors after it.
>
> Therefore if you encounter a block with bad sector/s, you want to (1)
> relocate as much salvageable data as possible to a “good” block, (2) mark
> the complete old block as unuseable, (3) destroy as much of the readable
> data in the bad block as possible.
>
> If the scheme is sector-based, you “just” need to make a best-effort
> attempt to encrypt as much as possibly salvageable of the bad sectors you
> read (effectively destroying the rest of the data in the sector, which
> does not matter because it is “bad” anyway).
>
> If you want to be file system independent, in case of a bad sector things
> are really bad, because you cannot easily mark a sector or block as “bad”
> for the file system.
>
> Effectively you have to invent your own relocation scheme, e.g. requiring
> a spare partition for relocation purposes. Or have the user accept that
> you will irretrievably destroy some data. If s/he does not accept, you
> would want to reollback (decrypt) everything and give the user a chance to
> copy the data to a properly functioning device.
>
>
>> If, as somebody said, the data and size (ioStatusBlock.Information)
>> should
>> be considered both unreliable when I recevied the sector fails, then even
>> the position within x would be unclear, and the data received so far I
>> must
>> not use to encrypt and write back.
>
> Correct. And to not leave the drive completely unuseable, you must
> actually roll back the whole encryption process.
>
> Question: What do you do when reading a bad sector in the encrypted data
> later on? And how do you check the correctness of the decrypted data in a
> block?
>
>
>> I do not need to encrypt this bad sector/s.
>
> Well, if it is a security software you are writing, you should probably at
> least try to destroy any possible plaintext data. Encrypting the read data
> and a write attempt might serve very well to that purpose.
>
>> But I am to go on, in some intelligent manner skipping the bad sectors
>> and
>> finding the next sane sectors (instead of leaving the volume alone).
> > The disk could contain a single bad sector, but it might as well be
> > near total corruption.
>
>> I heard that past the failure it is better to really go in single sectors
>> because each repeated hit to bad sectors is probably slower than going in
>> single sectors steps ?
>
> The OS and hard drive will usually try multiple reads to recover the
> sector data before giving up. Depending on your block size it is probably
> better to read single sectors if a block read returned an error.
>
> So far my 2 cents.
> -H
>

5

Thanks Hagen and Mark,

you both suggest to use single sector stepping.
I was looking for something faster and more elegant, but in lack of own
ideas I’ll try that.

Hagen,

The thread reads then encrypts then writes blocks of x sectors. The read
fails because an unknown number (1-n) of sectors within x is bad.

> If your encryption scheme is block-based, bad sectors in one block will
> effectively invalidate all “good” data in the sectors after it.

I’m not sure if I understand this right. If you mean the “good” data in the
returned buffer, that’s ok.

Regarding relocation, it’s a volume/disk filter without filesystem contact.
As I said the volume is left partly encrypted in the error case today. With
respect to accessibilty this is no prob. The same driver de/encrypts any
other read/write access and is aware of the threshold.

I do not need to encrypt this bad sector/s.

> Well, if it is a security software you are writing, you should probably
> at least try to destroy any possible plaintext data. Encrypting the read

> data and a write attempt might serve very well to that purpose.

I didnt’t read any data, remember it failed. But I promise I will try and
introduce to write back some random data or zeroes once I have solved the
original problem.

Regards
Else

|---------±-------------------------------->
| | Hagen Patzke |
| | |
| | Sent by: |
| | bounce-310670-16691@li|
| | sts.osr.com |
| | |
| | |
| | 01/09/2008 02:43 PM |
| | Please respond to |
| | “Windows System |
| | Software Devs Interest|
| | List” |
|---------±-------------------------------->
>-----------------------------------------------------------------------------------------------------------|
| |
| To: “Windows System Software Devs Interest List” |
| cc: |
| Subject: Re:[ntdev] Fw: bad sectors |
>-----------------------------------------------------------------------------------------------------------|

Else Kluger wrote:
> The thread reads then encrypts then writes blocks of x sectors. The read
> fails because an unknown number (1-n) of sectors within x is bad.

If your encryption scheme is block-based, bad sectors in one block will
effectively invalidate all “good” data in the sectors after it.

Therefore if you encounter a block with bad sector/s, you want to (1)
relocate as much salvageable data as possible to a “good” block, (2)
mark the complete old block as unuseable, (3) destroy as much of the
readable data in the bad block as possible.

If the scheme is sector-based, you “just” need to make a best-effort
attempt to encrypt as much as possibly salvageable of the bad sectors
you read (effectively destroying the rest of the data in the sector,
which does not matter because it is “bad” anyway).

If you want to be file system independent, in case of a bad sector
things are really bad, because you cannot easily mark a sector or block
as “bad” for the file system.

Effectively you have to invent your own relocation scheme, e.g.
requiring a spare partition for relocation purposes. Or have the user
accept that you will irretrievably destroy some data. If s/he does not
accept, you would want to reollback (decrypt) everything and give the
user a chance to copy the data to a properly functioning device.

> If, as somebody said, the data and size (ioStatusBlock.Information)
should
> be considered both unreliable when I recevied the sector fails, then even
> the position within x would be unclear, and the data received so far I
must
> not use to encrypt and write back.

Correct. And to not leave the drive completely unuseable, you must
actually roll back the whole encryption process.

Question: What do you do when reading a bad sector in the encrypted
data later on? And how do you check the correctness of the decrypted
data in a block?

> I do not need to encrypt this bad sector/s.

Well, if it is a security software you are writing, you should probably
at least try to destroy any possible plaintext data. Encrypting the read
data and a write attempt might serve very well to that purpose.

> But I am to go on, in some intelligent manner skipping the bad sectors
and
> finding the next sane sectors (instead of leaving the volume alone).
> The disk could contain a single bad sector, but it might as well be
> near total corruption.

> I heard that past the failure it is better to really go in single sectors
> because each repeated hit to bad sectors is probably slower than going in
> single sectors steps ?

The OS and hard drive will usually try multiple reads to recover the
sector data before giving up. Depending on your block size it is
probably better to read single sectors if a block read returned an error.

So far my 2 cents.
-H

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

Else Kluger wrote:

you both suggest to use single sector stepping.
I was looking for something faster and more elegant, but in lack of own
ideas I’ll try that.

If you use huge blocks, you could use a divide-and-conquer strategy.

(I.e. if a block_read of e.g. 1024 sectors of 512 bytes each fails, try
to read both 512-sector halves of the block in one go [you may repeat
recursively – if the block size falls below a threshold, switch to
single-sector reads]. THEN encrypt the whole block. THEN write back the
whole encrypted block [you might leave out any defective sectors, of
course, and write ranges of good sector runs instead].)

>> If your encryption scheme is block-based, bad sectors in one block will
>> effectively invalidate all “good” data in the sectors after it.

I’m not sure if I understand this right. If you mean the “good” data in the
returned buffer, that’s ok.

No, I meant in case you would just read-encrypt-write the block and
later try to decrypt it.

Sorry if I was a bit unclear. Let me try to explain it via (a) a
shortcut and then (b) step-by-step:

(a) Shortcut: - analog example -
E.g. with DES-CBC, given you read->encrypt->write a 256-byte block of
data, a persistent “read error” at byte position 59 will render invalid
not only the 8-byte block in which it resides, but everything past it, too.

If this was clear enough, you can skip (b) now. :slight_smile:

(b) BEGIN Step-by-step:
Let’s assume, your encryption scheme is block-based - meaning one
“block” consists of multiple disk sectors, and the sectors are “chained”
like in CBC.
Now let’s assume the first sector S of a given block B is a “bad” one.
Now you read block B that contains the bad sector S.
After this you encrypt block B and write it back to disk.

Now let’s assume that during the next block-read of block B the bad
sector S returns not exactly the same data as you wrote to it.

During decryption of B the output of the defective sector S will - if
you use a chained encryption scheme like CBC - not only itself be
defective, but also destroy the data that follows, resulting in the
whole block containing invalid data.
(b) END

Regarding relocation, it’s a volume/disk filter without filesystem contact.

So you have the “worst case” - you cannot mark a given sector as invalid
in the file system.

Hmmm… either your encryption scheme is sector-based (then it does not
matter much if the de/en/cryption yields invalid data), or you have to
find a clever way to safely en/de/crypt the rest of the block.
E.g. by having your own bad block relocation scheme using a spare
partition, hidden sectors, or something else.

The problem with encryption is always that it (a) has to be as
safe/secure as possible, but (b) must be robust against partial data
loss as well.

As I said the volume is left partly encrypted in the error case today. With
respect to accessibilty this is no prob. The same driver de/encrypts any
other read/write access and is aware of the threshold.

So the threshold is stored at a guaranteed safe place. :slight_smile:

I didnt’t read any data, remember it failed.

The disk is encrypted up to the part where the read error was, and the
rest stays unencrypted. This of course compromises security of the
unencrypted part of the data, and if you don’t point out the fact to the
user during every disk access they might even forget about it.

But I promise I will try and introduce to write back
> some random data or zeroes once I have solved the
original problem.

If you encrypt the “garbage” that was written, and write it back, this
should be random enough. :slight_smile:

Regards,
-H

7

> If, as somebody said, the data and size (ioStatusBlock.Information) should

be considered both unreliable when I recevied the sector fails, then even

This is a true statement

the position within x would be unclear, and the data received so far I
must

This is therefore also true

not use to encrypt and write back.

I do not need to encrypt this bad sector/s.
But I am to go on, in some intelligent manner skipping the bad sectors and
finding the next sane sectors (instead of leaving the volume alone). The
disk could contain a single bad sector, but it might as well be near total
corruption.

I heard that past the failure it is better to really go in single sectors
because each repeated hit to bad sectors is probably slower than going in
single sectors steps ?

Reading a bad sector is almost certainly slower than reading a single
non-bad sector. In the old days it used to be about 3x slower (minimum)
since the sector would be read 3 times before declaring a fail. It could be
much slower than that if the address field of the sector was corrupted as
the drive might decide to home and re-seek the sector in the assumption the
head was mispositioned. This could happen multiple times.

I don’t know what the current bad sector recovery algorithms are on the
usual drives, but I just had a bad sector case on an IDE drive where each
read was taking several seconds before failing. That is certainly slower
than reading a single sector.

Best practice for this sort of thing used to be to read blocks of a
reasonable size (>= 16KB in my view) and encrypt them and write them back,
checking for errors on BOTH the read and the write. When you get a failure,
read that block in smaller increments. You can probably trust the reported
transfer length to be >= the good sector length. So make an initial try of
rounding this down to the next lower sector and see if you can read that
much. If so, fine; if not, fall back to single sector reads. Read upward
until you get a failure, encrypting the good sectors.

Of course there may be readable data on the top of the block. If you assume
bad sectors are rare, you can just start reading this from the highest
sector downwards one sector at a time until you get a failure. Once you get
a failure there is generally no point in seeing if you can read any sectors
between the lowest bad sector and the highest bad sector assuming that the
difference is <= 15 sectors or so.

Note that it will sometimes be possible to end up reading the entire block a
sector at a time where reading it as a larger block fails. Probably very
rare, but it can happen. This shouldn’t cause you any trouble though if it
does happen.

Loren

8

Loren,
finally understand why it’s slower and will take account of your “best
practice” part.

Hagen:
I will talk to our cipher folks about

If your encryption scheme is block-based, bad sectors in one block will
effectively invalidate all “good” data in the sectors after it.

The disk is encrypted up to the part where the read error was, and the
rest stays unencrypted. This of course compromises security of the
unencrypted part of the data, and if you don’t point out the fact to the
user during every disk access they might even forget about it.
Users are and will be reminded about the failure (message box, log event).

In fact all the answers I received were very helpful, thanks again !
Else

|---------±-------------------------------->
| | “Loren Wilton” |
| | | | > |
| | Sent by: |
| | bounce-310699-16691@li|
| | sts.osr.com |
| | |
| | |
| | 01/09/2008 06:13 PM |
| | Please respond to |
| | “Windows System |
| | Software Devs Interest|
| | List” |
|---------±-------------------------------->
>-----------------------------------------------------------------------------------------------------------|
| |
| To: “Windows System Software Devs Interest List” |
| cc: |
| Subject: Re: [ntdev] bad sectors |
>-----------------------------------------------------------------------------------------------------------|

> If, as somebody said, the data and size (ioStatusBlock.Information)
should
> be considered both unreliable when I recevied the sector fails, then even

This is a true statement

> the position within x would be unclear, and the data received so far I
> must

This is therefore also true

> not use to encrypt and write back.
>
> I do not need to encrypt this bad sector/s.
> But I am to go on, in some intelligent manner skipping the bad sectors
and
> finding the next sane sectors (instead of leaving the volume alone). The
> disk could contain a single bad sector, but it might as well be near
total
> corruption.
>
> I heard that past the failure it is better to really go in single sectors
> because each repeated hit to bad sectors is probably slower than going in
> single sectors steps ?

Reading a bad sector is almost certainly slower than reading a single
non-bad sector. In the old days it used to be about 3x slower (minimum)
since the sector would be read 3 times before declaring a fail. It could
be
much slower than that if the address field of the sector was corrupted as
the drive might decide to home and re-seek the sector in the assumption the

head was mispositioned. This could happen multiple times.

I don’t know what the current bad sector recovery algorithms are on the
usual drives, but I just had a bad sector case on an IDE drive where each
read was taking several seconds before failing. That is certainly slower
than reading a single sector.

Best practice for this sort of thing used to be to read blocks of a
reasonable size (>= 16KB in my view) and encrypt them and write them back,
checking for errors on BOTH the read and the write. When you get a
failure,
read that block in smaller increments. You can probably trust the reported

transfer length to be >= the good sector length. So make an initial try of

rounding this down to the next lower sector and see if you can read that
much. If so, fine; if not, fall back to single sector reads. Read upward
until you get a failure, encrypting the good sectors.

Of course there may be readable data on the top of the block. If you
assume
bad sectors are rare, you can just start reading this from the highest
sector downwards one sector at a time until you get a failure. Once you
get
a failure there is generally no point in seeing if you can read any sectors

between the lowest bad sector and the highest bad sector assuming that the
difference is <= 15 sectors or so.

Note that it will sometimes be possible to end up reading the entire block
a
sector at a time where reading it as a larger block fails. Probably very
rare, but it can happen. This shouldn’t cause you any trouble though if it

does happen.

Loren

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer