I’ve implenmented pending the io of files those are in a specified folder.
When pending, the whole system runs slow.
I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
Is there any way makes better performance?
–
No one knows what tomorrow would be,
but I’ll do my best.
It sounds to me like you are serializing all read and write operations
in the system. This is a very bad idea and will degrade system
performance. I recommend that you describe what you are trying to
accomplish so that we can suggest a more efficient way of doing it.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 3:52 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Pending specified file io makes system slow
I’ve implenmented pending the io of files those are in a specified
folder.
When pending, the whole system runs slow.
I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
Is there any way makes better performance?
–
No one knows what tomorrow would be,
but I’ll do my best.
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Thank u.
All my want to do is :
1.When a specified event occurs, I pend CREATE/READ/WRITE io of some defined
files/folder.
2.When another event occurs, process the pending io.
I use interlock to list the pending io. And when the second event occurs,I
create a system thread
to process the io in the list. Use this api:ExInterlockedRemoveHeadList to
get the irp, and call
the corresponding irp dispatch routine which is defined in DriverEntry.
Yes, I am serializing all read and write io of the list.
And there is a question: Must a set a cancel routine for the irp?
Here is some pieces of code :(referenced from ddk sample of FloopyDisk
Driver)
Queuing Irp:
//
// Check if the irp was already canceled
//
if ( (Irp->Cancel ) &&
(IoSetCancelRoutine(Irp, NULL)))
{
//
// Already canceled
//
Irp->IoStatus.Status = STATUS_CANCELLED;
Irp->IoStatus.Information = 0;
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
IoCompleteRequest( Irp, IO_NO_INCREMENT );
Status = STATUS_CANCELLED;
} else {
//
// Queue the Irp
//
Irp->IoStatus.Status = STATUS_PENDING;
IoMarkIrpPending(Irp);
ExInterlockedInsertTailList( &RequestQueue,
&Irp->Tail.Overlay.ListEntry,
&RequestQueueSpinLock);
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
Status = STATUS_PENDING;
}
//////////////////////////////////////////////////////
Processing the Queued Irps:
…
…
while(
(
headOfList =
ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
)
!= NULL)
{
currentIrp = CONTAINING_RECORD( headOfList,
IRP,
Tail.Overlay.ListEntry);
if (IoSetCancelRoutine( currentIrp, NULL)) {
irpSp = IoGetCurrentIrpStackLocation( currentIrp );
} else {
//
// Cancel routine is already running for this IRP.
// Set the IRP field so that it won’t be removed
// in the cancel routine again.
//
currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
currentIrp = NULL;
}
KeReleaseSpinLock(&CancelQueueSpinLock,
oldIrql);
}
if (currentIrp) {
switch ( irpSp->MajorFunction ) {
case IRP_MJ_READ:
(VOID)MyRead(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_WRITE:
(VOID)MyWrite(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_CREATE:
(VOID)MyCreate(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_SET_INFORMATION:
(VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_DEVICE_CONTROL:
(VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_FILE_SYSTEM_CONTROL:
(VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
default:
currentIrp->IoStatus.Information = 0;
currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
}
}
KeAcquireSpinLock(&CancelQueueSpinLock,
&oldIrql);
}
“Neal Christiansen” Wrote news:xxxxx@ntfsd…
It sounds to me like you are serializing all read and write operations
in the system. This is a very bad idea and will degrade system
performance. I recommend that you describe what you are trying to
accomplish so that we can suggest a more efficient way of doing it.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 3:52 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Pending specified file io makes system slow
I’ve implenmented pending the io of files those are in a specified
folder.
When pending, the whole system runs slow.
I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
Is there any way makes better performance?
–
No one knows what tomorrow would be,
but I’ll do my best.
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Yes, you should set cancel routines whenever you pend IO operations.
Based on your description you should use cancel safe queue routines
provided in the kernel. Look at IoCsqInitializeEx() in the latest
IfsKit documentation to see how to use them. If you decide to do your
own cancel processing (which I would not recommend) be sure and read the
section on “Canceling IRPs” in the IFSkit documentation because a lot of
people don’t do it correctly.
It is not safe to pend paging IOs, this will cause deadlocks.
Looking at your code, it is fine to check Irp->Cancel but there is no
reason to call IoSetCancelRoutine() and set a NULL value. You will
never be given an IRP that has a non-null cancel routine. Cancel
routines can only be set while a driver “owns” the IRP. The cancel
routine MUST be removed before passing the IRP to another driver or
completing the IRP. Adding an ASSERT to verify that the field is NULL
would be a reasonable thing to do.
The performance impact of what you are doing is going to be based on
what percentage of IOs you pend and how long you pend them for.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 7:29 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Pending specified file io makes system slow
Thank u.
All my want to do is :
1.When a specified event occurs, I pend CREATE/READ/WRITE io of some
defined
files/folder.
2.When another event occurs, process the pending io.
I use interlock to list the pending io. And when the second event
occurs,I
create a system thread
to process the io in the list. Use this api:ExInterlockedRemoveHeadList
to
get the irp, and call
the corresponding irp dispatch routine which is defined in DriverEntry.
Yes, I am serializing all read and write io of the list.
And there is a question: Must a set a cancel routine for the irp?
Here is some pieces of code :(referenced from ddk sample of FloopyDisk
Driver)
Queuing Irp:
//
// Check if the irp was already canceled
//
if ( (Irp->Cancel ) &&
(IoSetCancelRoutine(Irp, NULL)))
{
//
// Already canceled
//
Irp->IoStatus.Status = STATUS_CANCELLED;
Irp->IoStatus.Information = 0;
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
IoCompleteRequest( Irp, IO_NO_INCREMENT );
Status = STATUS_CANCELLED;
} else {
//
// Queue the Irp
//
Irp->IoStatus.Status = STATUS_PENDING;
IoMarkIrpPending(Irp);
ExInterlockedInsertTailList( &RequestQueue,
&Irp->Tail.Overlay.ListEntry,
&RequestQueueSpinLock);
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
Status = STATUS_PENDING;
}
//////////////////////////////////////////////////////
Processing the Queued Irps:
…
…
while(
(
headOfList =
ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
)
!= NULL)
{
currentIrp = CONTAINING_RECORD( headOfList,
IRP,
Tail.Overlay.ListEntry);
if (IoSetCancelRoutine( currentIrp, NULL)) {
irpSp = IoGetCurrentIrpStackLocation( currentIrp );
} else {
//
// Cancel routine is already running for this IRP.
// Set the IRP field so that it won’t be removed
// in the cancel routine again.
//
currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
currentIrp = NULL;
}
KeReleaseSpinLock(&CancelQueueSpinLock,
oldIrql);
}
if (currentIrp) {
switch ( irpSp->MajorFunction ) {
case IRP_MJ_READ:
(VOID)MyRead(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_WRITE:
(VOID)MyWrite(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_CREATE:
(VOID)MyCreate(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_SET_INFORMATION:
(VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_DEVICE_CONTROL:
(VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_FILE_SYSTEM_CONTROL:
(VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
default:
currentIrp->IoStatus.Information = 0;
currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
}
}
KeAcquireSpinLock(&CancelQueueSpinLock,
&oldIrql);
}
“Neal Christiansen” Wrote
news:xxxxx@ntfsd…
It sounds to me like you are serializing all read and write operations
in the system. This is a very bad idea and will degrade system
performance. I recommend that you describe what you are trying to
accomplish so that we can suggest a more efficient way of doing it.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 3:52 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Pending specified file io makes system slow
I’ve implenmented pending the io of files those are in a specified
folder.
When pending, the whole system runs slow.
I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
Is there any way makes better performance?
–
No one knows what tomorrow would be,
but I’ll do my best.
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
>> I use interlock to list the pending io. And when the second event
> occurs,I create a system thread to process the io in the list
I don’t know if you mean this literally - but if you are really creating
the thread to service each request - you should consider a thread pool
(or maybe just a single thread if you are really serializing). Spinning
up a thread is a somewhat expensive operation and will most definitely
hurt your throughput.
-----Original Message-----
From: Neal Christiansen [mailto:xxxxx@windows.microsoft.com]
Sent: Tuesday, December 30, 2003 3:26 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Pending specified file io makes system slow
Yes, you should set cancel routines whenever you pend IO operations.
Based on your description you should use cancel safe queue routines
provided in the kernel. Look at IoCsqInitializeEx() in the latest
IfsKit documentation to see how to use them. If you decide to do your
own cancel processing (which I would not recommend) be sure and read the
section on “Canceling IRPs” in the IFSkit documentation because a lot of
people don’t do it correctly.
It is not safe to pend paging IOs, this will cause deadlocks.
Looking at your code, it is fine to check Irp->Cancel but there is no
reason to call IoSetCancelRoutine() and set a NULL value. You will
never be given an IRP that has a non-null cancel routine. Cancel
routines can only be set while a driver “owns” the IRP. The cancel
routine MUST be removed before passing the IRP to another driver or
completing the IRP. Adding an ASSERT to verify that the field is NULL
would be a reasonable thing to do.
The performance impact of what you are doing is going to be based on
what percentage of IOs you pend and how long you pend them for.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 7:29 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Pending specified file io makes system slow
Thank u.
All my want to do is :
1.When a specified event occurs, I pend CREATE/READ/WRITE io of some
defined
files/folder.
2.When another event occurs, process the pending io.
I use interlock to list the pending io. And when the second event
occurs,I
create a system thread
to process the io in the list. Use this api:ExInterlockedRemoveHeadList
to
get the irp, and call
the corresponding irp dispatch routine which is defined in DriverEntry.
Yes, I am serializing all read and write io of the list.
And there is a question: Must a set a cancel routine for the irp?
Here is some pieces of code :(referenced from ddk sample of FloopyDisk
Driver)
Queuing Irp:
//
// Check if the irp was already canceled
//
if ( (Irp->Cancel ) &&
(IoSetCancelRoutine(Irp, NULL)))
{
//
// Already canceled
//
Irp->IoStatus.Status = STATUS_CANCELLED;
Irp->IoStatus.Information = 0;
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
IoCompleteRequest( Irp, IO_NO_INCREMENT );
Status = STATUS_CANCELLED;
} else {
//
// Queue the Irp
//
Irp->IoStatus.Status = STATUS_PENDING;
IoMarkIrpPending(Irp);
ExInterlockedInsertTailList( &RequestQueue,
&Irp->Tail.Overlay.ListEntry,
&RequestQueueSpinLock);
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
Status = STATUS_PENDING;
}
//////////////////////////////////////////////////////
Processing the Queued Irps:
…
…
while(
(
headOfList =
ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
)
!= NULL)
{
currentIrp = CONTAINING_RECORD( headOfList,
IRP,
Tail.Overlay.ListEntry);
if (IoSetCancelRoutine( currentIrp, NULL)) {
irpSp = IoGetCurrentIrpStackLocation( currentIrp );
} else {
//
// Cancel routine is already running for this IRP.
// Set the IRP field so that it won’t be removed
// in the cancel routine again.
//
currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
currentIrp = NULL;
}
KeReleaseSpinLock(&CancelQueueSpinLock,
oldIrql);
}
if (currentIrp) {
switch ( irpSp->MajorFunction ) {
case IRP_MJ_READ:
(VOID)MyRead(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_WRITE:
(VOID)MyWrite(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_CREATE:
(VOID)MyCreate(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_SET_INFORMATION:
(VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_DEVICE_CONTROL:
(VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_FILE_SYSTEM_CONTROL:
(VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
default:
currentIrp->IoStatus.Information = 0;
currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
}
}
KeAcquireSpinLock(&CancelQueueSpinLock,
&oldIrql);
}
“Neal Christiansen” Wrote
news:xxxxx@ntfsd…
It sounds to me like you are serializing all read and write operations
in the system. This is a very bad idea and will degrade system
performance. I recommend that you describe what you are trying to
accomplish so that we can suggest a more efficient way of doing it.
Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
rights.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 3:52 AM
To: Windows File Systems Devs Interest List
Subject: [ntfsd] Pending specified file io makes system slow
I’ve implenmented pending the io of files those are in a specified
folder.
When pending, the whole system runs slow.
I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
Is there any way makes better performance?
–
No one knows what tomorrow would be,
but I’ll do my best.
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@exagrid.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
To put it lightly… I remember working on a network application a few
years ago that used a communications library that talked to a remote
server using an SSL connection via the WinInet-provided APIs. Due to a
bug in WinInet, the WinInet connection timeout functionality was not
working properly, so what the author of the library did was to spawn a
new thread for every send/receive request that would watchdog the
request and kill it if it ran past a specified timeout. I should also
mention that this was used as a backend to a network filesystem. Lotsa
fun to performance-tune this one.
Tom Hansen wrote:
>>I use interlock to list the pending io. And when the second event
>>occurs,I create a system thread to process the io in the listI don’t know if you mean this literally - but if you are really creating
the thread to service each request - you should consider a thread pool
(or maybe just a single thread if you are really serializing). Spinning
up a thread is a somewhat expensive operation and will most definitely
hurt your throughput.-----Original Message-----
From: Neal Christiansen [mailto:xxxxx@windows.microsoft.com]
Sent: Tuesday, December 30, 2003 3:26 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Pending specified file io makes system slowYes, you should set cancel routines whenever you pend IO operations.
Based on your description you should use cancel safe queue routines
provided in the kernel. Look at IoCsqInitializeEx() in the latest
IfsKit documentation to see how to use them. If you decide to do your
own cancel processing (which I would not recommend) be sure and read the
section on “Canceling IRPs” in the IFSkit documentation because a lot of
people don’t do it correctly.It is not safe to pend paging IOs, this will cause deadlocks.
Looking at your code, it is fine to check Irp->Cancel but there is no
reason to call IoSetCancelRoutine() and set a NULL value. You will
never be given an IRP that has a non-null cancel routine. Cancel
routines can only be set while a driver “owns” the IRP. The cancel
routine MUST be removed before passing the IRP to another driver or
completing the IRP. Adding an ASSERT to verify that the field is NULL
would be a reasonable thing to do.The performance impact of what you are doing is going to be based on
what percentage of IOs you pend and how long you pend them for.Neal Christiansen
Microsoft File System Filter Group LeadThis posting is provided “AS IS” with no warranties, and confers no
rights.-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ming
Sent: Monday, December 29, 2003 7:29 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Pending specified file io makes system slowThank u.
All my want to do is :
1.When a specified event occurs, I pend CREATE/READ/WRITE io of some
defined
files/folder.
2.When another event occurs, process the pending io.I use interlock to list the pending io. And when the second event
occurs,I
create a system thread
to process the io in the list. Use this api:ExInterlockedRemoveHeadList
to
get the irp, and call
the corresponding irp dispatch routine which is defined in DriverEntry.
Yes, I am serializing all read and write io of the list.
And there is a question: Must a set a cancel routine for the irp?
Here is some pieces of code :(referenced from ddk sample of FloopyDisk
Driver)
Queuing Irp:
//
// Check if the irp was already canceled
//
if ( (Irp->Cancel ) &&
(IoSetCancelRoutine(Irp, NULL)))
{
//
// Already canceled
//
Irp->IoStatus.Status = STATUS_CANCELLED;
Irp->IoStatus.Information = 0;
KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
IoCompleteRequest( Irp, IO_NO_INCREMENT );
Status = STATUS_CANCELLED;} else {
//
// Queue the Irp
//
Irp->IoStatus.Status = STATUS_PENDING;IoMarkIrpPending(Irp);
ExInterlockedInsertTailList( &RequestQueue,
&Irp->Tail.Overlay.ListEntry,
&RequestQueueSpinLock);KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
Status = STATUS_PENDING;
}
//////////////////////////////////////////////////////Processing the Queued Irps:
…
…
while(
(
headOfList =
ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
)!= NULL)
{
currentIrp = CONTAINING_RECORD( headOfList,
IRP,
Tail.Overlay.ListEntry);if (IoSetCancelRoutine( currentIrp, NULL)) {
irpSp = IoGetCurrentIrpStackLocation( currentIrp );
} else {
//
// Cancel routine is already running for this IRP.
// Set the IRP field so that it won’t be removed
// in the cancel routine again.
//
currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
currentIrp = NULL;
}KeReleaseSpinLock(&CancelQueueSpinLock,
oldIrql);
}
if (currentIrp) {switch ( irpSp->MajorFunction ) {
case IRP_MJ_READ:
(VOID)MyRead(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_WRITE:
(VOID)MyWrite(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_CREATE:
(VOID)MyCreate(irpSp->DeviceObject,currentIrp);break;
case IRP_MJ_SET_INFORMATION:
(VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_DEVICE_CONTROL:
(VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
break;
case IRP_MJ_FILE_SYSTEM_CONTROL:
(VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
default:currentIrp->IoStatus.Information = 0;
currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
}
}KeAcquireSpinLock(&CancelQueueSpinLock,
&oldIrql);}
“Neal Christiansen” Wrote
> news:xxxxx@ntfsd…
> It sounds to me like you are serializing all read and write operations
> in the system. This is a very bad idea and will degrade system
> performance. I recommend that you describe what you are trying to
> accomplish so that we can suggest a more efficient way of doing it.
>
> Neal Christiansen
> Microsoft File System Filter Group Lead
>
> This posting is provided “AS IS” with no warranties, and confers no
> rights.
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Ming
> Sent: Monday, December 29, 2003 3:52 AM
> To: Windows File Systems Devs Interest List
> Subject: [ntfsd] Pending specified file io makes system slow
>
> I’ve implenmented pending the io of files those are in a specified
> folder.
> When pending, the whole system runs slow.
> I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
> Is there any way makes better performance?
>
> –
> No one knows what tomorrow would be,
> but I’ll do my best.
>
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@exagrid.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
–
Nick Ryan (MVP for DDK)
I would write the SSL implementation of my own and talk to Berkeley sockets
instead.
Looks like it would save the debugging time and increase the stability of
the product.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: “Nick Ryan”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Wednesday, December 31, 2003 10:46 PM
Subject: Re:[ntfsd] Pending specified file io makes system slow
> To put it lightly… I remember working on a network application a few
> years ago that used a communications library that talked to a remote
> server using an SSL connection via the WinInet-provided APIs. Due to a
> bug in WinInet, the WinInet connection timeout functionality was not
> working properly, so what the author of the library did was to spawn a
> new thread for every send/receive request that would watchdog the
> request and kill it if it ran past a specified timeout. I should also
> mention that this was used as a backend to a network filesystem. Lotsa
> fun to performance-tune this one.
>
> Tom Hansen wrote:
>
> >>>I use interlock to list the pending io. And when the second event
> >>>occurs,I create a system thread to process the io in the list
> >
> >
> > I don’t know if you mean this literally - but if you are really creating
> > the thread to service each request - you should consider a thread pool
> > (or maybe just a single thread if you are really serializing). Spinning
> > up a thread is a somewhat expensive operation and will most definitely
> > hurt your throughput.
> >
> > -----Original Message-----
> > From: Neal Christiansen [mailto:xxxxx@windows.microsoft.com]
> > Sent: Tuesday, December 30, 2003 3:26 PM
> > To: Windows File Systems Devs Interest List
> > Subject: RE: [ntfsd] Pending specified file io makes system slow
> >
> > Yes, you should set cancel routines whenever you pend IO operations.
> >
> > Based on your description you should use cancel safe queue routines
> > provided in the kernel. Look at IoCsqInitializeEx() in the latest
> > IfsKit documentation to see how to use them. If you decide to do your
> > own cancel processing (which I would not recommend) be sure and read the
> > section on “Canceling IRPs” in the IFSkit documentation because a lot of
> > people don’t do it correctly.
> >
> > It is not safe to pend paging IOs, this will cause deadlocks.
> >
> > Looking at your code, it is fine to check Irp->Cancel but there is no
> > reason to call IoSetCancelRoutine() and set a NULL value. You will
> > never be given an IRP that has a non-null cancel routine. Cancel
> > routines can only be set while a driver “owns” the IRP. The cancel
> > routine MUST be removed before passing the IRP to another driver or
> > completing the IRP. Adding an ASSERT to verify that the field is NULL
> > would be a reasonable thing to do.
> >
> > The performance impact of what you are doing is going to be based on
> > what percentage of IOs you pend and how long you pend them for.
> >
> >
> > Neal Christiansen
> > Microsoft File System Filter Group Lead
> >
> > This posting is provided “AS IS” with no warranties, and confers no
> > rights.
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Ming
> > Sent: Monday, December 29, 2003 7:29 PM
> > To: Windows File Systems Devs Interest List
> > Subject: Re:[ntfsd] Pending specified file io makes system slow
> >
> > Thank u.
> > All my want to do is :
> > 1.When a specified event occurs, I pend CREATE/READ/WRITE io of some
> > defined
> > files/folder.
> > 2.When another event occurs, process the pending io.
> >
> > I use interlock to list the pending io. And when the second event
> > occurs,I
> > create a system thread
> > to process the io in the list. Use this api:ExInterlockedRemoveHeadList
> > to
> > get the irp, and call
> > the corresponding irp dispatch routine which is defined in DriverEntry.
> > Yes, I am serializing all read and write io of the list.
> > And there is a question: Must a set a cancel routine for the irp?
> > Here is some pieces of code :(referenced from ddk sample of FloopyDisk
> > Driver)
> > Queuing Irp:
> > //
> > // Check if the irp was already canceled
> > //
> > if ( (Irp->Cancel ) &&
> > (IoSetCancelRoutine(Irp, NULL)))
> > {
> > //
> > // Already canceled
> > //
> > Irp->IoStatus.Status = STATUS_CANCELLED;
> > Irp->IoStatus.Information = 0;
> > KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
> > IoCompleteRequest( Irp, IO_NO_INCREMENT );
> > Status = STATUS_CANCELLED;
> >
> > } else {
> > //
> > // Queue the Irp
> > //
> > Irp->IoStatus.Status = STATUS_PENDING;
> >
> > IoMarkIrpPending(Irp);
> >
> > ExInterlockedInsertTailList( &RequestQueue,
> > &Irp->Tail.Overlay.ListEntry,
> > &RequestQueueSpinLock);
> >
> > KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
> >
> > Status = STATUS_PENDING;
> > }
> > //////////////////////////////////////////////////////
> >
> >
> >
> >
> >
> > Processing the Queued Irps:
> >
> > …
> > …
> > while(
> > (
> > headOfList =
> > ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
> > )
> >
> > != NULL)
> >
> > {
> >
> > currentIrp = CONTAINING_RECORD( headOfList,
> > IRP,
> > Tail.Overlay.ListEntry);
> >
> > if (IoSetCancelRoutine( currentIrp, NULL)) {
> > irpSp = IoGetCurrentIrpStackLocation( currentIrp );
> > } else {
> > //
> > // Cancel routine is already running for this IRP.
> > // Set the IRP field so that it won’t be removed
> > // in the cancel routine again.
> > //
> > currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
> > currentIrp = NULL;
> > }
> >
> > KeReleaseSpinLock(&CancelQueueSpinLock,
> > oldIrql);
> > }
> > if (currentIrp) {
> >
> > switch ( irpSp->MajorFunction ) {
> >
> > case IRP_MJ_READ:
> > (VOID)MyRead(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_WRITE:
> > (VOID)MyWrite(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_CREATE:
> > (VOID)MyCreate(irpSp->DeviceObject,currentIrp);
> >
> > break;
> > case IRP_MJ_SET_INFORMATION:
> > (VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_DEVICE_CONTROL:
> > (VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_FILE_SYSTEM_CONTROL:
> > (VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
> > default:
> >
> > currentIrp->IoStatus.Information = 0;
> > currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
> > IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
> > }
> > }
> >
> >
> > KeAcquireSpinLock(&CancelQueueSpinLock,
> > &oldIrql);
> >
> > }
> >
> > “Neal Christiansen” Wrote
> > news:xxxxx@ntfsd…
> > It sounds to me like you are serializing all read and write operations
> > in the system. This is a very bad idea and will degrade system
> > performance. I recommend that you describe what you are trying to
> > accomplish so that we can suggest a more efficient way of doing it.
> >
> > Neal Christiansen
> > Microsoft File System Filter Group Lead
> >
> > This posting is provided “AS IS” with no warranties, and confers no
> > rights.
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Ming
> > Sent: Monday, December 29, 2003 3:52 AM
> > To: Windows File Systems Devs Interest List
> > Subject: [ntfsd] Pending specified file io makes system slow
> >
> > I’ve implenmented pending the io of files those are in a specified
> > folder.
> > When pending, the whole system runs slow.
> > I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
> > Is there any way makes better performance?
> >
> > –
> > No one knows what tomorrow would be,
> > but I’ll do my best.
> >
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@exagrid.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
>
> –
> Nick Ryan (MVP for DDK)
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
Actually, that’s very bad advice, unless you have the time & money to do
intensive security reviews. Security protocols are notoriously hard to
get right. Casually implementing a security layer is a great way to
introduce extremely painful bugs, whose potential cost (if found and
exploited) is enormous.
Security development is a tricky business. Also, anyone who needs SSL
on Win32 should look at using the existing SSPI version. It exists,
it’s a known, tested, reviewed product, and it works. The API is a bit
weird, since you have to do everything through SSPI, but once you get it
working, it does the job. Oh, and if any bugs are found – they’re
Microsofts, not yours.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 6:45 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slow
I would write the SSL implementation of my own and talk to Berkeley
sockets instead.
Looks like it would save the debugging time and increase the
stability of the product.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Though it is irrelevant, one particular problem I’ve seen using WinInet, is
its
inbuilt caching mechnism. IIRC, there are some APIs to do some separate
caching, but it was way difficult, and did not serve the purpose. OBJECTIVE
WAS :: to have a voice enabled browser along with IE. The voice browser was
for telematics, the little poddy would tell where the hell we are if lost
enroute, where is the nearest BurgerKing …, and you also want that device
tobe offline, so that you can read thru downloaded email, etc. These were
all voice enabled, using xerces, DOM/SOAP, and speech engine… But the
caching APIs were really not very useful, I WENT AND BUILD A URL CACHING
INFRASTRUCTURE MODLUE, not very elegant, but was useful for the first stage
development…
-prokash
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Maxim S. Shatskih
Sent: Wednesday, December 31, 2003 3:45 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slow
I would write the SSL implementation of my own and talk to Berkeley
sockets
instead.
Looks like it would save the debugging time and increase the stability
of
the product.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: “Nick Ryan”
Newsgroups: ntfsd
To: “Windows File Systems Devs Interest List”
Sent: Wednesday, December 31, 2003 10:46 PM
Subject: Re:[ntfsd] Pending specified file io makes system slow
> To put it lightly… I remember working on a network application a few
> years ago that used a communications library that talked to a remote
> server using an SSL connection via the WinInet-provided APIs. Due to a
> bug in WinInet, the WinInet connection timeout functionality was not
> working properly, so what the author of the library did was to spawn a
> new thread for every send/receive request that would watchdog the
> request and kill it if it ran past a specified timeout. I should also
> mention that this was used as a backend to a network filesystem. Lotsa
> fun to performance-tune this one.
>
> Tom Hansen wrote:
>
> >>>I use interlock to list the pending io. And when the second event
> >>>occurs,I create a system thread to process the io in the list
> >
> >
> > I don’t know if you mean this literally - but if you are really creating
> > the thread to service each request - you should consider a thread pool
> > (or maybe just a single thread if you are really serializing). Spinning
> > up a thread is a somewhat expensive operation and will most definitely
> > hurt your throughput.
> >
> > -----Original Message-----
> > From: Neal Christiansen [mailto:xxxxx@windows.microsoft.com]
> > Sent: Tuesday, December 30, 2003 3:26 PM
> > To: Windows File Systems Devs Interest List
> > Subject: RE: [ntfsd] Pending specified file io makes system slow
> >
> > Yes, you should set cancel routines whenever you pend IO operations.
> >
> > Based on your description you should use cancel safe queue routines
> > provided in the kernel. Look at IoCsqInitializeEx() in the latest
> > IfsKit documentation to see how to use them. If you decide to do your
> > own cancel processing (which I would not recommend) be sure and read the
> > section on “Canceling IRPs” in the IFSkit documentation because a lot of
> > people don’t do it correctly.
> >
> > It is not safe to pend paging IOs, this will cause deadlocks.
> >
> > Looking at your code, it is fine to check Irp->Cancel but there is no
> > reason to call IoSetCancelRoutine() and set a NULL value. You will
> > never be given an IRP that has a non-null cancel routine. Cancel
> > routines can only be set while a driver “owns” the IRP. The cancel
> > routine MUST be removed before passing the IRP to another driver or
> > completing the IRP. Adding an ASSERT to verify that the field is NULL
> > would be a reasonable thing to do.
> >
> > The performance impact of what you are doing is going to be based on
> > what percentage of IOs you pend and how long you pend them for.
> >
> >
> > Neal Christiansen
> > Microsoft File System Filter Group Lead
> >
> > This posting is provided “AS IS” with no warranties, and confers no
> > rights.
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Ming
> > Sent: Monday, December 29, 2003 7:29 PM
> > To: Windows File Systems Devs Interest List
> > Subject: Re:[ntfsd] Pending specified file io makes system slow
> >
> > Thank u.
> > All my want to do is :
> > 1.When a specified event occurs, I pend CREATE/READ/WRITE io of some
> > defined
> > files/folder.
> > 2.When another event occurs, process the pending io.
> >
> > I use interlock to list the pending io. And when the second event
> > occurs,I
> > create a system thread
> > to process the io in the list. Use this api:ExInterlockedRemoveHeadList
> > to
> > get the irp, and call
> > the corresponding irp dispatch routine which is defined in DriverEntry.
> > Yes, I am serializing all read and write io of the list.
> > And there is a question: Must a set a cancel routine for the irp?
> > Here is some pieces of code :(referenced from ddk sample of FloopyDisk
> > Driver)
> > Queuing Irp:
> > //
> > // Check if the irp was already canceled
> > //
> > if ( (Irp->Cancel ) &&
> > (IoSetCancelRoutine(Irp, NULL)))
> > {
> > //
> > // Already canceled
> > //
> > Irp->IoStatus.Status = STATUS_CANCELLED;
> > Irp->IoStatus.Information = 0;
> > KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
> > IoCompleteRequest( Irp, IO_NO_INCREMENT );
> > Status = STATUS_CANCELLED;
> >
> > } else {
> > //
> > // Queue the Irp
> > //
> > Irp->IoStatus.Status = STATUS_PENDING;
> >
> > IoMarkIrpPending(Irp);
> >
> > ExInterlockedInsertTailList( &RequestQueue,
> > &Irp->Tail.Overlay.ListEntry,
> > &RequestQueueSpinLock);
> >
> > KeReleaseSpinLock(&CancelQueueSpinLock,oldIrql);
> >
> > Status = STATUS_PENDING;
> > }
> > //////////////////////////////////////////////////////
> >
> >
> >
> >
> >
> > Processing the Queued Irps:
> >
> > …
> > …
> > while(
> > (
> > headOfList =
> > ExInterlockedRemoveHeadList(&RequestQueue,&RequestQueueSpinLock)
> > )
> >
> > != NULL)
> >
> > {
> >
> > currentIrp = CONTAINING_RECORD( headOfList,
> > IRP,
> > Tail.Overlay.ListEntry);
> >
> > if (IoSetCancelRoutine( currentIrp, NULL)) {
> > irpSp = IoGetCurrentIrpStackLocation( currentIrp );
> > } else {
> > //
> > // Cancel routine is already running for this IRP.
> > // Set the IRP field so that it won’t be removed
> > // in the cancel routine again.
> > //
> > currentIrp->Tail.Overlay.ListEntry.Flink = NULL;
> > currentIrp = NULL;
> > }
> >
> > KeReleaseSpinLock(&CancelQueueSpinLock,
> > oldIrql);
> > }
> > if (currentIrp) {
> >
> > switch ( irpSp->MajorFunction ) {
> >
> > case IRP_MJ_READ:
> > (VOID)MyRead(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_WRITE:
> > (VOID)MyWrite(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_CREATE:
> > (VOID)MyCreate(irpSp->DeviceObject,currentIrp);
> >
> > break;
> > case IRP_MJ_SET_INFORMATION:
> > (VOID)MySetInformation(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_DEVICE_CONTROL:
> > (VOID)DevIoCtrl(irpSp->DeviceObject,currentIrp);
> > break;
> > case IRP_MJ_FILE_SYSTEM_CONTROL:
> > (VOID)SfFsControl(irpSp->DeviceObject,currentIrp);
> > default:
> >
> > currentIrp->IoStatus.Information = 0;
> > currentIrp->IoStatus.Status = STATUS_UNSUCCESSFUL;
> > IoCompleteRequest (currentIrp, IO_NO_INCREMENT);
> > }
> > }
> >
> >
> > KeAcquireSpinLock(&CancelQueueSpinLock,
> > &oldIrql);
> >
> > }
> >
> > “Neal Christiansen” Wrote
> > news:xxxxx@ntfsd…
> > It sounds to me like you are serializing all read and write operations
> > in the system. This is a very bad idea and will degrade system
> > performance. I recommend that you describe what you are trying to
> > accomplish so that we can suggest a more efficient way of doing it.
> >
> > Neal Christiansen
> > Microsoft File System Filter Group Lead
> >
> > This posting is provided “AS IS” with no warranties, and confers no
> > rights.
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Ming
> > Sent: Monday, December 29, 2003 3:52 AM
> > To: Windows File Systems Devs Interest List
> > Subject: [ntfsd] Pending specified file io makes system slow
> >
> > I’ve implenmented pending the io of files those are in a specified
> > folder.
> > When pending, the whole system runs slow.
> > I pend MJ_CREATE/READ/WRITE. put the io into a list(provided by system).
> > Is there any way makes better performance?
> >
> > –
> > No one knows what tomorrow would be,
> > but I’ll do my best.
> >
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> > —
> > Questions? First check the IFS FAQ at
> > https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@exagrid.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
>
> –
> Nick Ryan (MVP for DDK)
>
>
> —
> Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
—
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@garlic.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
> Actually, that’s very bad advice, unless you have the time & money to do
intensive security reviews. Security protocols are notoriously hard to
get right.
Take the RFC on SSL. If you really want the sample - take some GPL SSL code. Do
not copy-paste from the code not belonging to you! Just use as a reference (as
I did once with UUENCODE when my task was to write a web-based email system - I
did not steal from GPL, first of all, GPL code was C and mine was C++ based on
MFC’s CString class).
You can even improve the GPL implementation as a side-effect 
Use Windows CryptoAPI for crypto, unless you hit a bug in it. Hitting a bug in
MS’s OS’s module is a VERY rare thing, I would say. No wonder they are
concentrated around IE - in WININET etc. - IE is known to be the only security
disaster among all MS’s products, so, no wonder.
You must also notice that the “newer WININET is free from this bug” is not an
excuse, since you limit the circle of your potential customers to ones with
particular new IE version. The common task in modern commercial software
development is to support the oldest system code base we can (that’s why the
SCI’s main product is still built using NT4 DDK - yes, we have the same BINARY
running on NT OSes from NT4 to w2k3, and that’s why people are still writing
VfW video capture drivers instead of DirectShow ones, even using hard-to-obtain
Win95 DDK. Anyway DirectShow will wrap them around with QCAP.DLL).
Security development is a tricky business. Also, anyone who needs SSL
on Win32 should look at using the existing SSPI version.
And if it is buggy?
Sorry, I HATE buggy 3rd party components. I hate them very, very much, and -
according to my practical experience - the decision of “re-implement!” made
early saves a lot of patching/hacking/OIbscureBugHuntingInNonYourCode time.
Surely, the sane project manager must have a clear answer to the “what NAMELY
do we want to re-implement?” question. Oh yes, the RFCed SSL. Nothing
proprietary or secret. All is open. There are implementations with sources. So
what?
it’s a known, tested, reviewed product, and it works.
It has a bug which is a major PITA for its user (the developer building the
product on top of the feature).
working, it does the job. Oh, and if any bugs are found – they’re
Microsofts, not yours.
OK, tell this to customers, and ask them to upgrade to newer IE. IMHO this is
unprofessional a bit.
“Your product does not work” - this is what the people say. They do not care
the buggy WININET in some old IE - then want the product to work in their
current system setup.
Your own code has the advantage of being independent of what OS add-ons (and
sometimes even what OS core modules) like IE are installed, and what versions
they are.
The advocates of pre-ready components often say this - “OK, if there will be a
bug, this is our bug and not yours”. Fine. Let’s even assume they are
responsible and fix their own bugs swiftly by issuing the new build versions.
But anyway! their component is distributed as a module and is included to lots
of products. So, the user will not necessary wish to upgrade this particular
.DLL file.
The “poor” developer who depend on their component can make an excuse of “well,
upgrade to the FinestThirdPartyDLL.DLL version 4.1.3, our product will not work
with 4.1.0”. So what? And if the person has 4.1.0 and is happy with it?
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
> caching APIs were really not very useful, I WENT AND BUILD A URL CACHING
INFRASTRUCTURE MODLUE, not very elegant, but was useful for the first stage
For me, this is an example of really professional development, development
leadership and project management decision.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
In several projects I’ve used OpenSSL for both SSL communications and
for general cryptographic operations. For applications that require
compatibility with versions of Windows back to NT4 you either can’t rely
on the security and communications APIs that Microsoft provide or they
aren’t even there.
Arlie Davis wrote:
Actually, that’s very bad advice, unless you have the time & money to do
intensive security reviews. Security protocols are notoriously hard to
get right. Casually implementing a security layer is a great way to
introduce extremely painful bugs, whose potential cost (if found and
exploited) is enormous.Security development is a tricky business. Also, anyone who needs SSL
on Win32 should look at using the existing SSPI version. It exists,
it’s a known, tested, reviewed product, and it works. The API is a bit
weird, since you have to do everything through SSPI, but once you get it
working, it does the job. Oh, and if any bugs are found – they’re
Microsofts, not yours.– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 6:45 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slowI would write the SSL implementation of my own and talk to Berkeley
sockets instead.
Looks like it would save the debugging time and increase the
stability of the product.Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
–
Nick Ryan (MVP for DDK)
I know where all the RFCs and such are. I’m saying – if you implement
a security protocol, you are implicitly guaranteeing to its users that
it meets a certain level of security. And typically with security
protocols, there is something at stake – if you fuck up, you may be
financially liable to your users.
I say again. Security protocols should only be implemented by
professionals who know what they are doing, have the time to do it
right, and have the time and resources to vigorously test and
code-review their work. Ask anyone in the security profession –
anything less is just asking for bugs. And bugs in security code often
means total compromise.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 8:03 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slow
Actually, that’s very bad advice, unless you have the time & money to
do intensive security reviews. Security protocols are notoriously
hard to get right.
Take the RFC on SSL. If you really want the sample - take some GPL SSL
code. Do not copy-paste from the code not belonging to you! Just use as
a reference (as I did once with UUENCODE when my task was to write a
web-based email system - I did not steal from GPL, first of all, GPL
code was C and mine was C++ based on MFC’s CString class).
You can even improve the GPL implementation as a side-effect
Use Windows CryptoAPI for crypto, unless you hit a bug in it. Hitting a
bug in MS’s OS’s module is a VERY rare thing, I would say. No wonder
they are concentrated around IE - in WININET etc. - IE is known to be
the only security disaster among all MS’s products, so, no wonder.
You must also notice that the “newer WININET is free from this bug” is
not an excuse, since you limit the circle of your potential customers to
ones with particular new IE version. The common task in modern
commercial software development is to support the oldest system code
base we can (that’s why the SCI’s main product is still built using NT4
DDK - yes, we have the same BINARY running on NT OSes from NT4 to w2k3,
and that’s why people are still writing VfW video capture drivers
instead of DirectShow ones, even using hard-to-obtain Win95 DDK. Anyway
DirectShow will wrap them around with QCAP.DLL).
Security development is a tricky business. Also, anyone who needs SSL
on Win32 should look at using the existing SSPI version.
And if it is buggy?
Sorry, I HATE buggy 3rd party components. I hate them very, very much,
and - according to my practical experience - the decision of
“re-implement!” made early saves a lot of
patching/hacking/OIbscureBugHuntingInNonYourCode time.
Surely, the sane project manager must have a clear answer to the “what
NAMELY do we want to re-implement?” question. Oh yes, the RFCed SSL.
Nothing proprietary or secret. All is open. There are implementations
with sources. So what?
it’s a known, tested, reviewed product, and it works.
It has a bug which is a major PITA for its user (the developer building
the product on top of the feature).
working, it does the job. Oh, and if any bugs are found – they’re
Microsofts, not yours.
OK, tell this to customers, and ask them to upgrade to newer IE. IMHO
this is unprofessional a bit.
“Your product does not work” - this is what the people say. They do not
care the buggy WININET in some old IE - then want the product to work in
their current system setup.
Your own code has the advantage of being independent of what OS add-ons
(and sometimes even what OS core modules) like IE are installed, and
what versions they are.
The advocates of pre-ready components often say this - “OK, if there
will be a bug, this is our bug and not yours”. Fine. Let’s even assume
they are responsible and fix their own bugs swiftly by issuing the new
build versions.
But anyway! their component is distributed as a module and is included
to lots of products. So, the user will not necessary wish to upgrade
this particular .DLL file.
The “poor” developer who depend on their component can make an excuse of
“well, upgrade to the FinestThirdPartyDLL.DLL version 4.1.3, our product
will not work with 4.1.0”. So what? And if the person has 4.1.0 and is
happy with it?
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@sublinear.org To
unsubscribe send a blank email to xxxxx@lists.osr.com
OpenSSL is a fine implementation. (Although – a few full-compromise
bugs were discovered in it recently. So even the *good* implementations
have problems.) I never said otherwise. And if Windows doesn’t provide
it, of course you have to get it elsewhere.
Look, this is a REALLY simple issue. All I’m saying is that you should
ALWAYS look for existing, mature, tested security protocol
implementations rather than develop your own from scratch. If they
don’t exist, or are mind-blowingly expensive, or have some other fatal
flaw – of course you shouldn’t use them. But you should always look
first.
Security is different from normal development. Security involves
adversaries – people who will actively look for holes in your code, and
ways to exploit them. That’s just the very definition of security to
begin with. Didn’t everyone notice how bad a year 2003 was for security
bugs in Microsoft products, at least PR-wise? I don’t think NT is any
less secure than any other major OS of its size and age – security
comes with maturity. But there are people out there who are absolutely
committed to discovering and exploiting security holes. Believe it.
Do you want to be on the harsh end of that? All I’m saying is – THINK
before committing to developing your own security code. Don’t
mindlessly trust someone else’s implementation, of course. But in the
case of SSL on Win32, there is virtually no reason to roll your own
implementation. Either use the Microsoft-supplied one, via SSPI, or use
OpenSSL. Don’t kid yourself that you can get anywhere near the test
coverage (and code review coverage) that both Microsoft’s SSL and
OpenSSL have reached.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Nick Ryan
Sent: Wednesday, December 31, 2003 8:10 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Pending specified file io makes system slow
In several projects I’ve used OpenSSL for both SSL communications and
for general cryptographic operations. For applications that require
compatibility with versions of Windows back to NT4 you either can’t rely
on the security and communications APIs that Microsoft provide or they
aren’t even there.
Arlie Davis wrote:
Actually, that’s very bad advice, unless you have the time & money to
do intensive security reviews. Security protocols are notoriously
hard to get right. Casually implementing a security layer is a great
way to introduce extremely painful bugs, whose potential cost (if
found and
exploited) is enormous.Security development is a tricky business. Also, anyone who needs SSL
on Win32 should look at using the existing SSPI version. It exists,
it’s a known, tested, reviewed product, and it works. The API is a
bit weird, since you have to do everything through SSPI, but once you
get it working, it does the job. Oh, and if any bugs are found –
they’re Microsofts, not yours.– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 6:45 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slowI would write the SSL implementation of my own and talk to
Berkeley sockets instead.
Looks like it would save the debugging time and increase the
stability of the product.Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
–
Nick Ryan (MVP for DDK)
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@sublinear.org To
unsubscribe send a blank email to xxxxx@lists.osr.com
> I say again. Security protocols should only be implemented by
professionals who know what they are doing, have the time to do it
Truth, but - in the context of using the proven-buggy component - this looks a
bit in other light
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
You can hire CounterPane or RSA to review your code, especially if you
use their toolkits.
“Arlie Davis” wrote in message news:xxxxx@ntfsd…
> I know where all the RFCs and such are. I’m saying – if you
implement
> a security protocol, you are implicitly guaranteeing to its users that
> it meets a certain level of security. And typically with security
> protocols, there is something at stake – if you fuck up, you may be
> financially liable to your users.
>
> I say again. Security protocols should only be implemented by
> professionals who know what they are doing, have the time to do it
> right, and have the time and resources to vigorously test and
> code-review their work. Ask anyone in the security profession –
> anything less is just asking for bugs. And bugs in security code
often
> means total compromise.
>
> – arlie
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
> Shatskih
> Sent: Wednesday, December 31, 2003 8:03 PM
> To: Windows File Systems Devs Interest List
> Subject: Re: Re:[ntfsd] Pending specified file io makes system slow
>
>
> > Actually, that’s very bad advice, unless you have the time & money
to
> > do intensive security reviews. Security protocols are notoriously
> > hard to get right.
>
> Take the RFC on SSL. If you really want the sample - take some GPL SSL
> code. Do not copy-paste from the code not belonging to you! Just use
as
> a reference (as I did once with UUENCODE when my task was to write a
> web-based email system - I did not steal from GPL, first of all, GPL
> code was C and mine was C++ based on MFC’s CString class).
>
> You can even improve the GPL implementation as a side-effect
>
> Use Windows CryptoAPI for crypto, unless you hit a bug in it. Hitting
a
> bug in MS’s OS’s module is a VERY rare thing, I would say. No wonder
> they are concentrated around IE - in WININET etc. - IE is known to be
> the only security disaster among all MS’s products, so, no wonder.
>
> You must also notice that the “newer WININET is free from this bug” is
> not an excuse, since you limit the circle of your potential customers
to
> ones with particular new IE version. The common task in modern
> commercial software development is to support the oldest system code
> base we can (that’s why the SCI’s main product is still built using
NT4
> DDK - yes, we have the same BINARY running on NT OSes from NT4 to
w2k3,
> and that’s why people are still writing VfW video capture drivers
> instead of DirectShow ones, even using hard-to-obtain Win95 DDK.
Anyway
> DirectShow will wrap them around with QCAP.DLL).
>
> > Security development is a tricky business. Also, anyone who needs
SSL
>
> > on Win32 should look at using the existing SSPI version.
>
> And if it is buggy?
>
> Sorry, I HATE buggy 3rd party components. I hate them very, very much,
> and - according to my practical experience - the decision of
> “re-implement!” made early saves a lot of
> patching/hacking/OIbscureBugHuntingInNonYourCode time.
>
> Surely, the sane project manager must have a clear answer to the “what
> NAMELY do we want to re-implement?” question. Oh yes, the RFCed SSL.
> Nothing proprietary or secret. All is open. There are implementations
> with sources. So what?
>
> > it’s a known, tested, reviewed product, and it works.
>
> It has a bug which is a major PITA for its user (the developer
building
> the product on top of the feature).
>
> > working, it does the job. Oh, and if any bugs are found – they’re
> > Microsofts, not yours.
>
> OK, tell this to customers, and ask them to upgrade to newer IE. IMHO
> this is unprofessional a bit.
>
> “Your product does not work” - this is what the people say. They do
not
> care the buggy WININET in some old IE - then want the product to work
in
> their current system setup.
>
> Your own code has the advantage of being independent of what OS
add-ons
> (and sometimes even what OS core modules) like IE are installed, and
> what versions they are.
>
> The advocates of pre-ready components often say this - “OK, if there
> will be a bug, this is our bug and not yours”. Fine. Let’s even assume
> they are responsible and fix their own bugs swiftly by issuing the new
> build versions.
>
> But anyway! their component is distributed as a module and is included
> to lots of products. So, the user will not necessary wish to upgrade
> this particular .DLL file.
>
> The “poor” developer who depend on their component can make an excuse
of
> “well, upgrade to the FinestThirdPartyDLL.DLL version 4.1.3, our
product
> will not work with 4.1.0”. So what? And if the person has 4.1.0 and is
> happy with it?
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at
> https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@sublinear.org To
> unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
Tell me how long you expect your implementation to reach the same degree
of quality and bug-free-ness as either Microsoft’s or OpenSSL.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 8:34 PM
To: Windows File Systems Devs Interest List
Subject: Re: Re:[ntfsd] Pending specified file io makes system slow
I say again. Security protocols should only be implemented by
professionals who know what they are doing, have the time to do it
Truth, but - in the context of using the proven-buggy component - this
looks a bit in other light
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@sublinear.org To
unsubscribe send a blank email to xxxxx@lists.osr.com
> Do you want to be on the harsh end of that? All I’m saying is – THINK
before committing to developing your own security code. Don’t
mindlessly trust someone else’s implementation, of course. But in the
OK, I’m in agreement with all of this, but how SSL hand-made implementation
based on Win32 CryptoAPI will be bad?
The security is not touched, it is re-used from MS’s security guys.
Anyway such a solution can be developed only after the proven bugs in WININET.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Every line of code is a potential hole. The more you can work at an
abstract layer, the better. CryptoAPI provides access to cryptographic
algorithms – hashes, ciphers, key generation, etc. Security protocols
like SSL make use of those to provide a higher-level service. Getting
that protocol implementation correct is *not* trivial.
And when you really want to be doing something else – like finishing
your own product – you tend to gloss over details. (Not you
specifically, of course – all developers, me included.)
The SSL implementation available via SSPI is pretty good, and it’s
fairly straightforward to implement. Also bear in mind that if there is
some change to SSL – new algorithms, new states, etc. – you’re covered
if you stick with the platform’s implementation, or even to OpenSSL.
You’re using an API, and someone else is responsible for the
implementation. For example, switching from SSL to TLS, if you are
using SSPI, is really easy.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S.
Shatskih
Sent: Wednesday, December 31, 2003 8:58 PM
To: Windows File Systems Devs Interest List
Subject: Re: [ntfsd] Pending specified file io makes system slow
Do you want to be on the harsh end of that? All I’m saying is –
THINK before committing to developing your own security code. Don’t
mindlessly trust someone else’s implementation, of course. But in the
OK, I’m in agreement with all of this, but how SSL hand-made
implementation based on Win32 CryptoAPI will be bad? The security is not
touched, it is re-used from MS’s security guys.
Anyway such a solution can be developed only after the proven bugs in
WININET.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17
You are currently subscribed to ntfsd as: xxxxx@sublinear.org To
unsubscribe send a blank email to xxxxx@lists.osr.com
If you’re keen to implement your own security protocols, that sounds
like an excellent idea. Review by real security professionals is
invaluable.
– arlie
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of David J. Craig
Sent: Wednesday, December 31, 2003 8:47 PM
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Re:Pending specified file io makes system slow
You can hire CounterPane or RSA to review your code, especially if you
use their toolkits.