I’ve been pondering this for awhile now, and I’d like to get some
opinions on what approach is best. This seems like the best place to ask.
Basically I want to have a disk drive that is stored entirely in RAM, but I
don’t want to have to declare its size on bootup, and I don’t want to have
it constantly waste most of the space it isn’t using. I’ve wanted this
countless times when I just want to save a file temporarily but didn’t want
to have to worry about killing it later.
I’ve come up with several ways, most of which aren’t exactly “good” practice
I’m sure.
Take the source for the microsoft ramdisk, and change it so that it
claims to be some large size but in fact is not backed up by ram at all.
Write operations are thrown away and read operations return random data.
This works surprisingly well because the system cache will keep things you
save there around for a pretty long time. Drawbacks are the peculiar
behavior you get after the cache data is thrown away and the fact that your
data lasts for an undefined amount of time.
Implement ms-ramdisk so that instead of allocating data all at once, it
allocates it on demand, perhaps a few sectors at a time. Use some fancy
data structure for accessing the allocated data. This is probably the most
“correct” solution, now that I think about it. Problem is that when you
delete data the space can’t be reclaimed, plus you still have to declare the
max size on bootup.
Implement a device that implements the entire file system stack, all in
one. So it maintains a tree in non-paged pool that is the filesystem, and
writes are done directly to this. The problem is obviously that filesystems
are hard to write. I’ve started this, didn’t get very far. It seems that I
need to support the cache manager, even though that doesn’t make sense
because the data will always be in RAM. I’d prefer to exploit a filesystem
that is already written, but I don’t exactly want to try to take FAT and try
to make this big of a change. This is also good if I did it because
suddenly I have a file system I can fiddle with without the dangers of
making a filter.
Is there another way I’ve missed? Has something like this been done
already? I feel like it would be useful, you get the benefits of a RAM
drive without the drawbacks (constant use of large block of memory that is
mostly empty).
Thanks for taking the time to read all that, it ended up being a little
longer than I’d intended.
What would be the point of a disk that you might not be able to
read back what you wrote? Although you do bring up a point that for a
ramdisk on NT you will have some duplicated data in memory (the ramdisk
copy and the cache copy). If you could somehow determine what pages on
in cache then you could free your ramdisk memory. And then when the
cache goes to drop the pages you would re-allocate the memory. My guess
is that the added complexity probably isn’t worth the space savings.
You could reclaim space by occiasionally asking the FS to tell
you what space it is using by using the defrag support
(FSCTL_VOLUME_BITMAP or whatever extactly it is - check MSDN).
In order to get a run-time variable size drive here are some
ideas:
a) create a new volume, copy the old stuff to it, delete
the old volume. You could probably play around with drive letters to
make it look like the volume went away and then came back as a different
size.
b) see if you can find information about filesystem
support to grow/shrink a volume. I don’t know anything about it, except
I read something about the dynamic disk sets supporting it.
Writing your own FS is a ton of work (usually measured in man-years).
Adding a couple of GBs of RAM to the machine and just wasting unused
ramdisk space would be cheaper and faster. (of course unless you are
planning on deploying this on thousands of machines).
-----Original Message-----
From: Brian [mailto:argus@vt.edu]
Sent: Thursday, April 25, 2002 12:28 PM
To: File Systems Developers
Subject: [ntfsd] A “stretchy” ramdisk in windows: Can it be done?
I’ve been pondering this for awhile now, and I’d like to get some
opinions on what approach is best. This seems like the best place to
ask.
Basically I want to have a disk drive that is stored entirely in RAM,
but I don’t want to have to declare its size on bootup, and I don’t want
to have it constantly waste most of the space it isn’t using. I’ve
wanted this countless times when I just want to save a file temporarily
but didn’t want to have to worry about killing it later.
I’ve come up with several ways, most of which aren’t exactly “good”
practice I’m sure.
Take the source for the microsoft ramdisk, and change it so that it
claims to be some large size but in fact is not backed up by ram at all.
Write operations are thrown away and read operations return random data.
This works surprisingly well because the system cache will keep things
you save there around for a pretty long time. Drawbacks are the
peculiar behavior you get after the cache data is thrown away and the
fact that your data lasts for an undefined amount of time.
Implement ms-ramdisk so that instead of allocating data all at once,
it allocates it on demand, perhaps a few sectors at a time. Use some
fancy data structure for accessing the allocated data. This is probably
the most “correct” solution, now that I think about it. Problem is that
when you delete data the space can’t be reclaimed, plus you still have
to declare the max size on bootup.
Implement a device that implements the entire file system stack, all
in one. So it maintains a tree in non-paged pool that is the
filesystem, and writes are done directly to this. The problem is
obviously that filesystems are hard to write. I’ve started this, didn’t
get very far. It seems that I need to support the cache manager, even
though that doesn’t make sense because the data will always be in RAM.
I’d prefer to exploit a filesystem that is already written, but I don’t
exactly want to try to take FAT and try to make this big of a change.
This is also good if I did it because suddenly I have a file system I
can fiddle with without the dangers of making a filter.
Is there another way I’ve missed? Has something like this been done
already? I feel like it would be useful, you get the benefits of a RAM
drive without the drawbacks (constant use of large block of memory that
is mostly empty).
Thanks for taking the time to read all that, it ended up being a little
longer than I’d intended.
-brian
You are currently subscribed to ntfsd as: xxxxx@microsoft.com To
unsubscribe send a blank email to %%email.unsub%%
The reason the “black hole” drive, as we call it, is useful is because
(for me, anyway) needing to store something for just a little while is
sometimes important. Times like when I want to read a pdf file but have to
download it first. I don’t want to put it somewhere and have it taking up
space, it makes much more sense in that case to save it in the system cache
(which is essentially what the drive does.) In general I find it useful for
all sorts of temporary data. Plus it was really easy to implement =).
Another thing I use it for is my ftp site. I keep IIS up and running
with anonymous write access to FTP turned on. I use this to transfer files
on and off of some annoying lab machines on campus. The interesting thing
is that warez groups apparently constantly scan the internet for such sites.
About once a week or so some group will try to set up shop. The drive is
useful because they will do all the work to upload things, but as soon as
they try to send an iso their directory structure starts to deteriorate. I
think it’s fun to annoy people who try to steal my disk space like that.
-brian
----- Original Message -----
From: “Nathan Nesbit” To: “File Systems Developers” Sent: Thursday, April 25, 2002 5:09 PM Subject: [ntfsd] RE: A “stretchy” ramdisk in windows: Can it be done?
My $.02
1) What would be the point of a disk that you might not be able to read back what you wrote? Although you do bring up a point that for a ramdisk on NT you will have some duplicated data in memory (the ramdisk copy and the cache copy). If you could somehow determine what pages on in cache then you could free your ramdisk memory. And then when the cache goes to drop the pages you would re-allocate the memory. My guess is that the added complexity probably isn’t worth the space savings.
2) You could reclaim space by occiasionally asking the FS to tell you what space it is using by using the defrag support (FSCTL_VOLUME_BITMAP or whatever extactly it is - check MSDN).
In order to get a run-time variable size drive here are some ideas: a) create a new volume, copy the old stuff to it, delete the old volume. You could probably play around with drive letters to make it look like the volume went away and then came back as a different size. b) see if you can find information about filesystem support to grow/shrink a volume. I don’t know anything about it, except I read something about the dynamic disk sets supporting it.
3) Writing your own FS is a ton of work (usually measured in man-years). Adding a couple of GBs of RAM to the machine and just wasting unused ramdisk space would be cheaper and faster. (of course unless you are planning on deploying this on thousands of machines).
-----Original Message----- From: Brian [mailto:argus@vt.edu] Sent: Thursday, April 25, 2002 12:28 PM To: File Systems Developers Subject: [ntfsd] A “stretchy” ramdisk in windows: Can it be done?
I’ve been pondering this for awhile now, and I’d like to get some opinions on what approach is best. This seems like the best place to ask.
Basically I want to have a disk drive that is stored entirely in RAM, but I don’t want to have to declare its size on bootup, and I don’t want to have it constantly waste most of the space it isn’t using. I’ve wanted this countless times when I just want to save a file temporarily but didn’t want to have to worry about killing it later.
I’ve come up with several ways, most of which aren’t exactly “good” practice I’m sure.
1) Take the source for the microsoft ramdisk, and change it so that it claims to be some large size but in fact is not backed up by ram at all. Write operations are thrown away and read operations return random data. This works surprisingly well because the system cache will keep things you save there around for a pretty long time. Drawbacks are the peculiar behavior you get after the cache data is thrown away and the fact that your data lasts for an undefined amount of time.
2) Implement ms-ramdisk so that instead of allocating data all at once, it allocates it on demand, perhaps a few sectors at a time. Use some fancy data structure for accessing the allocated data. This is probably the most “correct” solution, now that I think about it. Problem is that when you delete data the space can’t be reclaimed, plus you still have to declare the max size on bootup.
3) Implement a device that implements the entire file system stack, all in one. So it maintains a tree in non-paged pool that is the filesystem, and writes are done directly to this. The problem is obviously that filesystems are hard to write. I’ve started this, didn’t get very far. It seems that I need to support the cache manager, even though that doesn’t make sense because the data will always be in RAM. I’d prefer to exploit a filesystem that is already written, but I don’t exactly want to try to take FAT and try to make this big of a change. This is also good if I did it because suddenly I have a file system I can fiddle with without the dangers of making a filter.
Is there another way I’ve missed? Has something like this been done already? I feel like it would be useful, you get the benefits of a RAM drive without the drawbacks (constant use of large block of memory that is mostly empty).
Thanks for taking the time to read all that, it ended up being a little longer than I’d intended.
-brian
— You are currently subscribed to ntfsd as: xxxxx@microsoft.com To unsubscribe send a blank email to %%email.unsub%%
— You are currently subscribed to ntfsd as: argus@vt.edu To unsubscribe send a blank email to %%email.unsub%%