RtlCopyMemory RtlMoveMemory

Hi list,

I am working on a file system driver , it heavily uses buffer copy operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the buffer? Here is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer, FileName.Length);

Any pointer to improve performance would be of help.
Rodney.

Hi,
you may not zero whole buffer.

if (FileName.Length < FileName.MaximumLength)
RtlZeroMemory(FileName.Buffer + FileName.Length ,
FileName.MaximumLength - FileName.Length);

If performance is really issue you may not zero at all.

From code you sent ExAllocatePoolWithTag is slowest so you could think about
some kind of memory reusage. There is some function for that (i cannot
remember that) which return you prepared memory blocks.

I dont think you could do RtlCopyMemory faster.

wrote in message news:xxxxx@ntfsd…
> Hi list,
>
> I am working on a file system driver , it heavily uses buffer copy
operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve
performance of the copy operation.
>
> Is there any alternate to RtlCopyMemory API to fastly copy the buffer?
Here is the code snippet…
>
> -----------------
> FileName.Buffer = ExAllocatePoolWithTag(
> PagedPool,
> FileName.MaximumLength,
> MAGIC_TAG
> );
>
> RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
> RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer,
FileName.Length);
> -----------------
>
> Any pointer to improve performance would be of help.
> Rodney.
>

You won’t get any faster than RtlCopyMemory and for the code snippet
concerned the copy count is going to be comparatively small anyway.

If you want to look at performance tweaks to this snippet the
firstly, why bother to zero the allocated string buffer ? If you
want it to be null terminated so it’s printable as a string (dubious
practice, though common), then why not just poke a null on the end
after the copy - assuming the buffer has that space.

Another way to potentially optimise things would be to replace the
ExAllocate with a lookasidelist. Assuming a reasonable flow of
allocating and releasing these buffers you can save on the memory
allocation processing by getting hits in the free list of the lookaside.

None of this is going to speed the overall program execution by much,
though eliminating the ZeroMemory in this snippet could reduce the
CPU cycles for the snippet by 30%-50% or more depending on the
overhead built in to the FileName.MaximumLength.

Mark

At 10:29 15/01/2009, xxxxx@gmail.com wrote:

Hi list,

I am working on a file system driver , it heavily uses buffer copy
operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to
improve performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the
buffer? Here is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer,
IrpSp->FileObject->FileName.Buffer, FileName.Length);

Any pointer to improve performance would be of help.
Rodney.


NTFSD is sponsored by OSR

Yes, lookasidelist is the thing i could not remember. (Look at
ExInitializeNPagedLookasideList)

Zeroing memory could add a bit of security when you pass buffer to user
mode. When you are not passing buffer anywere then there is no reason to do
that but I get used to do that so I usually do that everytime too.

“Mark S. Edwards” wrote in message news:xxxxx@ntfsd…
> You won’t get any faster than RtlCopyMemory and for the code snippet
> concerned the copy count is going to be comparatively small anyway.
>
> If you want to look at performance tweaks to this snippet the
> firstly, why bother to zero the allocated string buffer ? If you
> want it to be null terminated so it’s printable as a string (dubious
> practice, though common), then why not just poke a null on the end
> after the copy - assuming the buffer has that space.
>
> Another way to potentially optimise things would be to replace the
> ExAllocate with a lookasidelist. Assuming a reasonable flow of
> allocating and releasing these buffers you can save on the memory
> allocation processing by getting hits in the free list of the lookaside.
>
> None of this is going to speed the overall program execution by much,
> though eliminating the ZeroMemory in this snippet could reduce the
> CPU cycles for the snippet by 30%-50% or more depending on the
> overhead built in to the FileName.MaximumLength.
>
> Mark
>
>
> At 10:29 15/01/2009, xxxxx@gmail.com wrote:
> >Hi list,
> >
> >I am working on a file system driver , it heavily uses buffer copy
> >operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to
> >improve performance of the copy operation.
> >
> >Is there any alternate to RtlCopyMemory API to fastly copy the
> >buffer? Here is the code snippet…
> >
> >-----------------
> >FileName.Buffer = ExAllocatePoolWithTag(
> > PagedPool,
> > FileName.MaximumLength,
> > MAGIC_TAG
> > );
> >
> > RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
> > RtlCopyMemory(FileName.Buffer,
> > IrpSp->FileObject->FileName.Buffer, FileName.Length);
> >-----------------
> >
> >Any pointer to improve performance would be of help.
> >Rodney.
> >
> >—
> >NTFSD is sponsored by OSR
>
>

Thanks Jan and Mark for the suggestions. Can you share / point to code using ExInitializePagedLookasideList (as a replacement of ExAllocatePoolWithTag)

Rodney.

>

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer,
FileName.Length);

I don’t think I actually agree to this piece of code.
Getting file names, is actually not so heavy buffering.

You should use lookaside lists, this is actually MS suggestion, I don’t see
how you did not find that in documentation.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Thursday, January 15, 2009 12:29 PM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] RtlCopyMemory RtlMoveMemory

Hi list,

I am working on a file system driver , it heavily uses buffer copy
operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve
performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the buffer? Here
is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer,
FileName.Length);

Any pointer to improve performance would be of help.
Rodney.


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

> some kind of memory reusage. There is some function for that (i cannot

remember that) which return you prepared memory blocks.

Lookaside lists.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>

You should use lookaside lists, this is actually MS suggestion, I don’t see
how you did not find that in documentation.

IMO using lookaside lists in working with strings is reasonable if size
of the strings are quite similar.
Or to use the structure like this
struct {
WCHAR StaticBuffer[some_size];
PWCHAR DynamicBuffer;
USHORT DynamicBufferSize;
};
So after the structure is obtained from the lookaside list then either
use StaticBuffer if its size is enough or allocate the necessary buffer
using ExAllocatePoolWithTag and save the pointer in DynamicBuffer.


Best regards,
Vladimir Zinin
xxxxx@eldos.com

YES Vladimir, this is actually the best way.

@ Rodney
Look at the NameLookup library provided in the filesys samples in the WDK.
They do something similar.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vladimir Zinin
Sent: Thursday, January 15, 2009 4:41 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] RtlCopyMemory RtlMoveMemory

You should use lookaside lists, this is actually MS suggestion, I don’t
see
how you did not find that in documentation.

IMO using lookaside lists in working with strings is reasonable if size
of the strings are quite similar.
Or to use the structure like this
struct {
WCHAR StaticBuffer[some_size];
PWCHAR DynamicBuffer;
USHORT DynamicBufferSize;
};
So after the structure is obtained from the lookaside list then either
use StaticBuffer if its size is enough or allocate the necessary buffer
using ExAllocatePoolWithTag and save the pointer in DynamicBuffer.


Best regards,
Vladimir Zinin
xxxxx@eldos.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

A better option is to just not copy more than your initialized data length to user mode.

? S

-----Original Message-----
From: Jan Milan
Sent: Thursday, January 15, 2009 02:59
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] RtlCopyMemory RtlMoveMemory

Yes, lookasidelist is the thing i could not remember. (Look at
ExInitializeNPagedLookasideList)

Zeroing memory could add a bit of security when you pass buffer to user
mode. When you are not passing buffer anywere then there is no reason to do
that but I get used to do that so I usually do that everytime too.

“Mark S. Edwards” wrote in message news:xxxxx@ntfsd…
> You won’t get any faster than RtlCopyMemory and for the code snippet
> concerned the copy count is going to be comparatively small anyway.
>
> If you want to look at performance tweaks to this snippet the
> firstly, why bother to zero the allocated string buffer ? If you
> want it to be null terminated so it’s printable as a string (dubious
> practice, though common), then why not just poke a null on the end
> after the copy - assuming the buffer has that space.
>
> Another way to potentially optimise things would be to replace the
> ExAllocate with a lookasidelist. Assuming a reasonable flow of
> allocating and releasing these buffers you can save on the memory
> allocation processing by getting hits in the free list of the lookaside.
>
> None of this is going to speed the overall program execution by much,
> though eliminating the ZeroMemory in this snippet could reduce the
> CPU cycles for the snippet by 30%-50% or more depending on the
> overhead built in to the FileName.MaximumLength.
>
> Mark
>
>
> At 10:29 15/01/2009, xxxxx@gmail.com wrote:
> >Hi list,
> >
> >I am working on a file system driver , it heavily uses buffer copy
> >operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to
> >improve performance of the copy operation.
> >
> >Is there any alternate to RtlCopyMemory API to fastly copy the
> >buffer? Here is the code snippet…
> >
> >-----------------
> >FileName.Buffer = ExAllocatePoolWithTag(
> > PagedPool,
> > FileName.MaximumLength,
> > MAGIC_TAG
> > );
> >
> > RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
> > RtlCopyMemory(FileName.Buffer,
> > IrpSp->FileObject->FileName.Buffer, FileName.Length);
> >-----------------
> >
> >Any pointer to improve performance would be of help.
> >Rodney.
> >
> >—
> >NTFSD is sponsored by OSR
>
>


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@valhallalegends.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

xxxxx@gmail.com wrote:

Hi list,

I am working on a file system driver , it heavily uses buffer copy operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the buffer? Here is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer, FileName.Length);

Any pointer to improve performance would be of help.
Rodney.

The following “improvement” could save you and your
customers several days (rather than few microseconds):

FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);
if ( !FileName.Buffer )
{
// handle this
} else {
…your code …
}

Regards,
–pa

> So after the structure is obtained from the lookaside list then either

use StaticBuffer if its size is enough or allocate the necessary buffer

Or even simpler:

if( Str->Buffer != (PWSTR)( Str + 1 ) )
ExFreePool(Str->Buffer);
//else
// - the string is allocated from the lookaside together with its buffer


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>A better option is to just not copy more than your initialized data length to user mode.

IO manager will not copy beyound Irp->IoStatus.Information for buffered IO


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

I’m curious: if you are about to overwrite the contents of the buffer with
RtlCopyMemory, why are you bothering to call RtlZeroMemory? If this makes a
difference, your code is erroneous. It means you are depending on a
NUL-terminated string (usually a mistake in the kernel, except for
RtlInitUnicodeString), and in any case, because FileName is a
UNICODE_STRING, it can have a string of FileName.MaximumLength, so if you
copy it to the space you just allocated, such a string would not have a NUL
at the end. So if you think that RtlCopyMemory/RtlMoveMemory makes a
difference in performance, you should eliminate a completely pointless
operation that sets-to-zero data that is about to be overwritten by content.
For that matter, why is your buffer not allocated as
FileName.Buffer = ExAllocatePoolWithTag(PagedPool,
FileName.MaximumLength + sizeof(UNICODE_STRING),
MAGIC_TAG);
Then initialize the UNICODE_STRING at the front of the storage to point to
the rest, and use Rtl functions to copy the UNICODE_STRING. I don’t see why
you are just copying the filename without dealing with either the NUL
character or using a reasonable structure such as UNICODE_STRING.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Thursday, January 15, 2009 11:21 AM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] RtlCopyMemory RtlMoveMemory

xxxxx@gmail.com wrote:

Hi list,

I am working on a file system driver , it heavily uses buffer copy
operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve
performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the buffer?
Here is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer,
IrpSp->FileObject->FileName.Buffer, FileName.Length);

Any pointer to improve performance would be of help.
Rodney.

The following “improvement” could save you and your customers several days
(rather than few microseconds):

FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);
if ( !FileName.Buffer )
{
// handle this
} else {
…your code …
}

Regards,
–pa


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars (including our new fs
mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@flounder.com To
unsubscribe send a blank email to xxxxx@lists.osr.com


This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

Yes. I suspect the OP was describing a scenario in which they instruct the I/O manager to copy a fixed length block and not the actual data length. This is bad practice.

? S

-----Original Message-----
From: Maxim S. Shatskih
Sent: Thursday, January 15, 2009 09:03
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Re:RtlCopyMemory RtlMoveMemory

>A better option is to just not copy more than your initialized data length to user mode.

IO manager will not copy beyound Irp->IoStatus.Information for buffered IO


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

>

The following “improvement” could save you and your
customers several days (rather than few microseconds):

The most constructive answer yet to this topic :slight_smile:

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Pavel A.
Sent: Thursday, January 15, 2009 6:21 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] RtlCopyMemory RtlMoveMemory

xxxxx@gmail.com wrote:

Hi list,

I am working on a file system driver , it heavily uses buffer copy
operations. As of now we replaced RtlMoveMemory by RtlCopyMemory to improve
performance of the copy operation.

Is there any alternate to RtlCopyMemory API to fastly copy the buffer?
Here is the code snippet…


FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);

RtlZeroMemory(FileName.Buffer, FileName.MaximumLength);
RtlCopyMemory(FileName.Buffer, IrpSp->FileObject->FileName.Buffer,
FileName.Length);

Any pointer to improve performance would be of help.
Rodney.

The following “improvement” could save you and your
customers several days (rather than few microseconds):

FileName.Buffer = ExAllocatePoolWithTag(
PagedPool,
FileName.MaximumLength,
MAGIC_TAG
);
if ( !FileName.Buffer )
{
// handle this
} else {
…your code …
}

Regards,
–pa


NTFSD is sponsored by OSR

For our schedule debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

You are currently subscribed to ntfsd as: xxxxx@gmail.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Thanks for the valuable inputs
-Rodney