When will be queued a packet to the I/O completion port ?

unfortunately i not wait for such result… may be i too bad explain

indeed it DOES matter. While it is true that the Native API gets called eventually, your entire issue (it seems to me) is with how Win32 handles the edge >cases. If you limit your work to the Native NT API (in user mode or kernel mode, “no matter”) then I think you’ll find things much more clear.

NO. you mistake here. this is not win32 issue. nothing is changed if we call direct native api. and handle direct NTSTATUS but not wrong (not rarely) win32 error. may be my mistake that i post both versions. and i agree that Native NT API much more clear , but here problem not in win32 layer.

and here not problem in file system implementations at all too. the problem in I/O manager kernel code - i say this all time. problem in how it handle FAST IO case. at some (LockFile) point I/O manager post completion to IOCP, even if error status returned. at another point (Unlock) - I/O manager -complete request without IOCP (even if STATUS_SUCCESS returned). not driver, but I/O itself ! simply WRK-v1.2 (i paste links) for understand src of problems

Byte range locks are another odd case, and I suspect how they behave will vary from file system to file system.

again - this not file system problem - how it handle request, but I/O manager problem - how it complete request.

and Directory Change Notification here unrelated at all. i nothing write about it, and perfect know how it work and not once use it in async mode, but here no problems.

simply look win

ok. sorry for all

your posts are certianly too long to follow, but let me say what i think your problem is

you need to know when an IOCP notification will be queued and when it will not. This is very important

there is an apparent ambiguity in the documentation re the use of FILE_SKIP_COMPLETION_PORT_ON_SUCCESS. In fact this feature was specifically added to Windows to remove the ambiguity you are worried about

The name FILE_SKIP_COMPLETION_PORT_ON_SUCCESS includes the work SUCCESS. That word has nothing to do with the real meaning of this option. What it means is that if the operation never goes pending and the final state of the call is known directly, then do not queue a completion on the port. This is true regardless of whether the final result of the call was a failure, warning or success

yes, unfortunately i guilty that I could not clearly explain the essence. but this was not question, faster how say already - but I’m trying to draw your attention to a very interesting (as it seems to me) question

you need to know when an IOCP notification will be queued and when it will not. This is very important

absolute agree. and not only i need this. everybody, who do asynchronous I/O programming.

FILE_SKIP_COMPLETION_PORT_ON_SUCCESS - not direct related to problem, and my post not about it. i know how it work, and that it name not does not display the essence (really it prevent IOCP entry if operation completed without pending)

but i about absolute another try say… that I/O manager (not drivers or win32 subsystem) wrong (?) handle FAST IO path what lead to problems

certianly too long

look for concrete comment with 2 absolute concrete question…

if still unclear what i all time try to say, ok, let moderator delete my post at all.

so what about FAST IO bothers you? The IRP can complete immediatly instead of being pended. It can either fail or succeed.

I realize that I have been too sloppy in my terminology. Language barriers increase the need to be technically precise and i have not been

But the point remains the same - what about FAST IO bothers you?

The IRP can complete immediatly instead of being pended. It can either fail or succeed.

True… regardless of whether Turbo I/O is used or an IRP is sent.

I don’t get what Mr. @nektar80 wants…

Peter

but i explain !! are still not clear ?! this is break general rule when will be and no IOCP entry as result of I/O (if fast io used IRP not created, but this is unrelated). so again - how based on returned status from I/O detect will be IOCP notification|callback or no ? we must know this exactly for correct manage resources.

general/semi documented rule is next:

bool IsWillBeIOCPNotification(NTSTATUS status /*returned from api call*/, bool bSkipOnSuccessMode)
{
    if (status == STATUS_PENDING) return true;

    if (NT_ERROR(status)) return false;

    return !bSkipOnSuccessMode;
}

1.) if STATUS_PENDING returned - will be notification. here no questions.

2.) if NT_ERROR(status) - must not be IOCP.

but - i found case, when this rule is beaked. i show concrete example with NtLockFile - when it can return STATUS_INVALID_PARAMETER or STATUS_LOCK_NOT_GRANTED - but will be IOCP entry !!

and this is not related to win32 layer or file system driver. this is related to I/O manager only - look wrk src code - how it complete request after fast io lock. this part still relevant and for windows 10. the same and for FastIoDeviceControl - how I/O manager handle at this point - possible NT_ERROR(status) - and IOCP notification

3.) otherwise based on FILE_SKIP_COMPLETION_PORT_ON_SUCCESS - will be completion or no.

**but ** - look for NtUnlockFile src code (again from wrk - still relevant and in latest win10 at this part) - if driver implement FastIoUnlockSingle and it return TRUE - I/O manager complete request without post IOCP entry (or set event, or APC).

so I/O can returned STATUS_SUCCESS but without IOCP notification (let assume FILE_SKIP_COMPLETION_PORT_ON_SUCCESS not set - for not confuse).

look at I/O manager code for understand !

so i show concrete examples. and “ask” all time about how exactly know - will be or not IOCP notification

@“Peter_Viscarola_(OSR)” - sorry for this topic at all. i not wait such and how i say - i not need help at all. i try draw your attention to this point. but if still unclear… i am sorry. i not want spam you. simply delete my post at all

ок, fortget about win32 layer and FILE_SKIP_COMPLETION_PORT_ON_SUCCESS - it not set.

maximal simply example

struct MY_IRP : public IO_STATUS_BLOCK
{
	//...
};

VOID NTAPI IoCallback(NTSTATUS status, ULONG_PTR Information, PVOID Context)
{
	MY_IRP* Irp = reinterpret_cast<MY_IRP*>(Context);
	//...
	delete Irp;
};

bool WillBeNoCallback(NTSTATUS status); 

	HANDLE hFile;
	RtlSetIoCompletionCallback(hFile, IoCallback, 0);

	if (MY_IRP* Irp = new MY_IRP(..))
	{
		LARGE_INTEGER ByteOffset{x}, Length{y};
		NTSTATUS status = NtLockFile(hFile, 0, 0, Irp, Irp, &ByteOffset, &Length, 'keyX', TRUE, TRUE);
		if (WillBeNoCallback(status)) // if (NT_ERROR(status)) 
		{
			IoCallback(status, 0, Irp);
		}
	}
  • we bind IoCallback to hFile - as result it will be called by system, when IOCP entry
  • every I/O operation require unique IO_STATUS_BLOCK - so i allocate it - MY_IRP* Irp = new MY_IRP(…)
  • now i call asynchronous I/O api (NtLockFile in concrete case) and pass MY_IRP* Irp to it in place ApcContext (for get it back in IoCallback)
  • but MY_IRP* Irp - need be free, when I/O finished. so how, when do this ?
  • native solution (how it seems to me) - do this inside IoCallback.
  • but if IoCallback will be not called (due I/O error) - we need just free MY_IRP* Irp or better - manual call IoCallback(status, 0, Irp);
  • so QUESTION - how detect, based on returned status (and may be IO_STATUS_BLOCK) - will be IoCallback called (as result of IOCP notification) or no
  • so question in implementation of bool WillBeNoCallback(NTSTATUS status);
  • are if (NT_ERROR(status)) - always (forget about FILE_SKIP_COMPLETION_PORT_ON_SUCCESS !! it not set!) correct ?
  • i show that not always !

note - that will be IOCP notification or no - depend only from I/O manager. this not depend from driver, which handle request ! driver can have any implementation of lock/unlock/ ioctl. but not driver post IOCP entry( or queue APC or set user Event). this done exclusive by I/O manager. based on driver return final status.

sorry, really i mistake with NtUnlockFile - this is synchronous api - so never here will be IOCP notification here

but still exist case with IoDeviceControl and IoLock when - api returned NT_ERROR(status) but will be packet queued to IOCP

usually (documented) we need assume that will be no IOCP notification if api return status in range NT_ERROR(status)

as result we can just free IO_STATUS_BLOCK and another resources allocated for call, if we got NT_ERROR(status) (status in [0xC0000000, 0xFFFFFFFF])

but because can be IOCP notification even in this case - the callback binded to this IOCP will be called too, end here we again try free IO_STATUS_BLOCK…

as result double free


poc in Native api code
https://github.com/rbmm/LockFile-Poc/blob/master/NT_Api_poc.cpp
https://github.com/rbmm/LockFile-Poc/blob/master/NT_poc.log

what i say, about from where was error (I/O manager, Fast IO, IRP io) - this of course really no matter.
but i simply try explain - why/how/where was situation when NT_ERROR(status) returned but packet queued to IOCP:
this was in case FastIoDeviceControl or FastIoLock return TRUE with NT_ERROR(status)
really I/O manager queued packet to IOCP in this case (FastIo return TRUE) and not look for final operation status

and result not depend from how drives handle lock or ioctl. only from I/O manager, because this it duty - how complete user request, particularly post or not post entry to IOCP

after more research kernel code - i found that this really bug only inside NtLockFile api:
after FastIoLock return TRUE it not check final status and post entry to IOCP for any status. pseudo code from win10

NTSTATUS NtLockFile(... PVOID ApcContext ...)
{
	PFILE_OBJECT FileObject;

	// code after FastIoLock return TRUE -

	PIO_COMPLETION_CONTEXT CompletionContext = FileObject->CompletionContext;

	if (CompletionContext &&
		ApcContext &&
		!(FileObject->Flags & FO_SKIP_COMPLETION_PORT))
	{
		IoSetIoCompletionEx2( CompletionContext->Port,
			CompletionContext->Key,
			ApcContext,
			status, // not checked !!!
			Information,
			...);
	}
}

look NtLockFile.png

for compare, code from IopXxxControlFile (fixed error in win8, still exist in win7)

NTSTATUS IopXxxControlFile(... PVOID ApcContext ...)
{
	PFILE_OBJECT FileObject;

	NTSTATUS status;
	PVOID Port = 0, Key = 0;

	// code after FastIoDeviceControl 

	PIO_COMPLETION_CONTEXT CompletionContext = FileObject->CompletionContext;

	if (
		CompletionContext && 
		!(FileObject->Flags & FO_SKIP_COMPLETION_PORT) &&
		!NT_ERROR(status) // checked !!!
		)
	{
		Port = CompletionContext->Port, Key = CompletionContext->Key;
	}

	if (Port && ApcContext)
	{
		IoSetIoCompletionEx2( Port, Key, ApcContext,status, Information,...);
	}
}

here added check !NT_ERROR(status)

so if fix error (i believe that this is bug) in ntoskrnl!NtLockFile

bool IsWillBeIOCPNotification(NTSTATUS status /*returned from api call*/, bool FoSkipCompletionPort )
{
    if (status == STATUS_PENDING) return true;

    if (NT_ERROR(status)) return false;

    return !FoSkipCompletionPort;
}

I am curious to know what led you to tumble down this rabbit hole. You started off asking a very general design question, but was this all triggered because your application encountered a problem with the LockFile API? If not, what let you into this investigation?

@Tim_Roberts i am sorry for too bad English and and for the inability to clearly express one’s thought

i am developing general class library for asynchronous I/O and always interesting in this topic. but several days ago, when i play with asynchronous file locking, i random catch design bug - NtLockFile returned STATUS_LOCK_NOT_GRANTED but despite this error was notify the I/O completion port and the associated I/O callback function called. what must not be.

my mistake - that i look initially only to the WRK-v1.2 source code (of course very old already) and i understand why is happens. but when i look to NtDeviceIoControlFile → IopXxxControlFile - i view that here the same mistake in code. so i decide that the same problem must be and with IOCTL if we use asynchronous file and driver implement FastIoDeviceControl and return TRUE from it with NT_ERROR status. this already more serious. but when i test IOCTL in win10 - no error. so i look for binary code (win7, 8.1, 10 ) and understand that error in IopXxxControlFile was fixed in 8.1 how minimum. but MS forget fix NtLockFile. so bug is only here now. easy for fix (only one check !NT_ERROR(status) need add), but still exist. may be MS fix this bug if report it

with my say about NtUnlockFile - i of course hurry and mistake - this is synchronous api, not have ApcContext parameter - so here must not be IOCP notification at all. sorry for this mistake.

bug in NtLockFile really exist, but only here. was early and in ioctl path but fixed long time ago

Dude… it was warnings, directory change notification, Fast I/O for read and write… I already said, I’m totally lost.

I’m not saying the OP hasn’t found some weird edge condition or a bug in some version of Windows handling of some specific IO function code. Heavens knows there are plenty. I’m just saying I have no clue what he’s talking about. The most unfortunate part is that if he can’t explain it to US, he has almost zero chance of explaining it to MSFT in a bug report and thereby getting it fixed.

Peter

@“Peter_Viscarola_(OSR)” about ZwNotifyChangeDirectoryFile - it return STATUS_DATATYPE_MISALIGNMENT if we not pass the DWORD-aligned buffer . this bit confusing, because this is not NT_ERROR status but will be no IOCP notification here, because error from I/O manager. but we can always avoid this error - pass not DWORD-aligned buffer to this api - programmer error. so i mention this situation in vain - it only distracted from the main problem and confused. problem how i final research only in NtLockFile - compare modern IopXxxControlFile and NtLockFile implementation in my cooment and view different - !NT_ERROR(status)

if (FileObject->CompletionContext && ApcContext &&
        !(FileObject->Flags & FO_SKIP_COMPLETION_PORT) &&
        !NT_ERROR(status))

checked in IopXxxControlFile after FastIo return true, but only

if (FileObject->CompletionContext && ApcContext &&
        !(FileObject->Flags & FO_SKIP_COMPLETION_PORT) )

was checked in NtLockFile

early, before win 8 - this error was and in IopXxxControlFile too

when i look wrk source code, how FastIo handled in NtLockFile and NtDeviceIoControlFile - hard was say - are this bug or “by design”. anyway such “design” break general rule - when will be notification to IOCP, so i and asked general question. but after i view, that code code is changed in NtDeviceIoControlFile - check for NT_ERROR(status) added - i understand that was exactly error in xp/2003 and early here. this error is fixed(in win 8 probably) for NtDeviceIoControlFile but MS forget apply same fix for NtLockFile.

we all understand that you are frustrated by the difficulty in explaining the problem that you are looking at. we are also fusterated by our difficulty in understanding what you are trying to say

what is not clear is whether you expect this issue to apply to all kinds of API calls that might result in IOCP completions, or whether you think this is confined to a specific call pattern

I am almost certian that i can explain the correct behaviour to you, since i had exactly this problem before MSFT resolved the problems with Vista / Server 2008 IIRC. But as Peter says it is possible that you have uncovered a specific case that is a bug / flaw

@MBond2 thank, for response

let me summarize.

response for my question: When will be queued a packet to the I/O completion port ?

// status - returned by asynchronous api call 
// (i.e. NtXxx(HANDLE FileHandle,HANDLE Event,PIO_APC_ROUTINE ApcRoutine,PVOID ApcContext,IO_STATUS_BLOCK IoStatusBlock,..)
// bSkipOnSuccess = FileObject->Flags & FO_SKIP_COMPLETION_PORT i.e are we set FILE_SKIP_COMPLETION_PORT_ON_SUCCESS on file

bool Will_be_IOCP_Notification(NTSTATUS status, BOOLEAN bSkipOnSuccess = FALSE)
{
	return (status == STATUS_PENDING) || (!NT_ERROR(status) && !bSkipOnSuccess);
}

I hope there is no objection ?


but i was confused by not fixed bug in NtLockFile and WRK-v1.2 source code. early (including win 7) was bug in IopXxxControlFile too, now fixed

the minimal POC - worked on windows 10

VOID WINAPI IoCompletionNT(
								  _In_    NTSTATUS status,
								  _In_    ULONG_PTR dwNumberOfBytesTransfered,
								  _Inout_ PVOID ApcContext
								  )
{
	WCHAR sz[64];
	swprintf_s(sz, L"(%x %p %p)", status, (void*)dwNumberOfBytesTransfered, ApcContext);
	MessageBoxW(0, sz, L"IoCompletionNT", MB_OK);

	delete ApcContext;
}

void PocLockFile()
{
	static UNICODE_STRING ObjectName = RTL_CONSTANT_STRING(L"\\SystemRoot");
	static OBJECT_ATTRIBUTES oa = { sizeof(oa), 0, &ObjectName };

	HANDLE hFile;
	IO_STATUS_BLOCK iosb;

	if (0 <= NtOpenFile(&hFile, FILE_READ_DATA, &oa, &iosb, FILE_SHARE_VALID_FLAGS, FILE_DIRECTORY_FILE))
	{
		if (0 <= RtlSetIoCompletionCallback(hFile, IoCompletionNT, 0))
		{
			if (IO_STATUS_BLOCK* piosb = new IO_STATUS_BLOCK)
			{
				LARGE_INTEGER ByteOffset{}, Length {1};
				NTSTATUS status = NtLockFile(hFile, 0, 0, piosb, piosb, &ByteOffset, &Length, 'keyX', TRUE, TRUE);
				
				if (!Will_be_IOCP_Notification(status))
				{
					WCHAR sz[64];
					swprintf_s(sz, L"[%x, 0, %p]", status, piosb);
					MessageBoxW(0, sz, L"PocLockFile", MB_OK);
					//delete piosb;
				}
			}
		}
		NtClose(hFile);
	}
}

despite NtLockFile return STATUS_INVALID_PARAMETER here and must not be IOCP notification - it was really

of course call Lock on folder no sense, but show POC. more real case, when we do this on file - https://github.com/rbmm/LockFile-Poc/blob/master/NT_Api_poc.cpp


I mentioned also about ZwNotifyChangeDirectoryFile - which can return STATUS_DATATYPE_MISALIGNMENT (this is not NT_ERROR) but we always must avoid this error, by pass correct aligned buffer

You may have found a bug in this seldom used API

byte range locking is an almost useless feature so it would not be surprising that this is less well tested than other calls

also note that the code you posted has very poor style. At least this SAL is wrong

Inout PVOID ApcContext

@MBond2 - yes, i agree that Inout PVOID ApcContext is bad
but i do copy-paste from minwinbase.h

typedef
VOID
(WINAPI *LPOVERLAPPED_COMPLETION_ROUTINE)(
    _In_    DWORD dwErrorCode,
    _In_    DWORD dwNumberOfBytesTransfered,
    _Inout_ LPOVERLAPPED lpOverlapped
    );

so i take documented callback definition for BindIoCompletionCallback and adjust it for not declared in wdk/sdk headers RtlSetIoCompletionCallback
i do copy-paste and simply not and did not pay attention to SAL here. in this sense Inout LPOVERLAPPED lpOverlapped also wrong - this is in-only pointer to structure. PVOID ApcContext also usual pointer to structure which how minimum containing IO_STATUS_BLOCK. but anyway think SAL here oftopic.

about NtLockFile - yes - this is seldom, but when i first look for wrk source code i decide that same bug exist also in NtDeviceIoControlFile. in case fast io used. this bug here (NtDeviceIoControlFile) really exist yet in windows 7 (tested) but in window 8.1 already fixed

leave SAL and style aside. I mention this for the archives so that someone else won’t copy / paste this code and think it is good

I suggest you open a support case with Microsoft. If they agree that this is a real bug, and it actually has an impact on you, they will likely fix it. It will not be a fast process. Microsoft are still working on an issue I raised in SQL Native Client about 18 months ago - and that is just an unhandled exception leaking from C++ code out through a C API and so does not involve the possibility of breaking changes. your issue does involve the possibility of breaking changes for existing software.