windows service crashes w/o exception code

NTDEV
1

I apologize if this question should have been posted on other blog. Please
advice where to post my question if this is the case.

I have a service running on windows 7 x64 system and crashes intermittently
at customer site. I am able to ask the customer to run a debug version,
attached a visual studio debugger to the process, and collect a crash dump
file when it breaks into the debugger. To my surprise the crash dump file
does not indicate any exception or bug check. It seems to stop because a
breakpoint has been reached, which I did not set. I hope some of you can
shed some light on why this is happening. Below is kernel debugger?s
analysis on the crash dump:

**********************************************

*Microsoft (R) Windows Debugger Version 6.2.9200.20512 AMD64*

*Copyright (c) Microsoft Corporation. All rights reserved.*

*Loading Dump File [D:\temp\64bit\10232013-Crashdump\VldIos_2nd.dmp]*

*User Mini Dump File with Full Memory: Only application data is available*

*WARNING: Path element is empty*

*Symbol search path is:
SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;;D
http::\xxx \IO Service\Debug*

Executable search path is:

Windows 7 Version 7601 (Service Pack 1) MP (16 procs) Free x64

Product: Server, suite: TerminalServer SingleUserTS

Built by: 6.1.7601.18015 (win7sp1_gdr.121129-1432)

Machine Name:

Debug session time: Wed Oct 23 08:59:43.000 2013 (UTC - 8:00)

System Uptime: 2 days 9:27:04.805

Process Uptime: 0 days 5:55:39.000

…

ntdll!NtWaitForSingleObject+0xa:

0000000076d712fa c3 ret*<br><br>*WARNING: Path element is empty*<br><br> ****WARNING: Unable to verify checksum for VldIos.exe*<br><br>*0:001&gt; !analyze -v*<br><br>********************************************************************************* <br><br> **<br>** <br><br> **Exception<br>Analysis** <br><br> **<br>** <br><br> ********************************************************************************* <br><br>*FAULTING_IP:*<br><br>*+0*<br><br>*0000000000000000 ?? ???

EXCEPTION_RECORD: ffffffffffffffff – (.exr 0xffffffffffffffff)

ExceptionAddress: 0000000000000000

* ExceptionCode: 80000003 (Break instruction exception)

 ExceptionFlags: 00000000*

NumberParameters: 0

FAULTING_THREAD: 00000000000005a0

DEFAULT_BUCKET_ID: STATUS_BREAKPOINT

PROCESS_NAME: VldIos.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint
has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651 <%282147483651>) - One or
more arguments are invalid

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

APP: vldios.exe

PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT

BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT

LAST_CONTROL_TRANSFER: from 000007fefcd210dc to 0000000076d712fa

*STACK_TEXT: *

000000000028ef38 000007fefcd210dc : 0000000000000000 0000000076d401d5
0000000000000004 00000000003fa7d0 : ntdll!NtWaitForSingleObject+0xa

000000000028ef40 000007fefef1affb : 00000000ffffffff 000007fefef1344c
0000000000000000 00000000000000d8 : KERNELBASE!WaitForSingleObjectEx+0x79

000000000028efe0 000007fefef19d61 : 00000000003e0c20 00000000000000d8
0000000000000000 0000000000000000 :
sechost!ScSendResponseReceiveControls+0x13b

000000000028f0d0 000007fefef19a51 : 000000000028f218 0000000000000000
0000000000000000 0000000000000000 : sechost!ScDispatcherLoop+0x121

000000000028f1e0 000000014003821f : 0000000000000000 00000000003e8990
000000000028f158 0000000000000000 :
sechost!StartServiceCtrlDispatcherA+0x6d

000000000028f210 000000014001f290 : 0000000000000000 00000001400c7348
000000000028ec28 0000000000000000 : VldIos!InitService+0x8f

000000000028f280 0000000061e7c419 : 000000014013b860 0000000000000000
00000000003137a9 000000000000000a : VldIos!CLhdIOSApp::InitInstance+0x9b0

000000000028f860 000000014007bec1 : 000000013ffe0000 0000000000000000
00000000003137a9 000000000000000a : mfc100d!AfxWinMain+0xc9

000000000028f8f0 0000000140073eb6 : 000000013ffe0000 0000000000000000
00000000003137a9 000000000000000a : VldIos!WinMain+0x31

000000000028f920 0000000140073bae : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : VldIos!__tmainCRTStartup+0x2f6

000000000028fa10 0000000076c1652d : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : VldIos!WinMainCRTStartup+0xe

000000000028fa40 0000000076d4c541 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0xd

000000000028fa70 0000000000000000 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x1d

STACK_COMMAND: ~0s; .ecxr ; kb

FOLLOWUP_IP:

sechost!ScSendResponseReceiveControls+13b

000007fefef1affb 85c0 test eax,eax*<br><br>*SYMBOL_STACK_INDEX: 2*<br><br>*SYMBOL_NAME: sechost!ScSendResponseReceiveControls+13b*<br><br>*FOLLOWUP_NAME: MachineOwner*<br><br>*MODULE_NAME: sechost*<br><br>*IMAGE_NAME: sechost.dll*<br><br>*DEBUG_FLR_IMAGE_TIMESTAMP: 4a5be05e*<br><br>*FAILURE_BUCKET_ID:<br>STATUS_BREAKPOINT_80000003_sechost.dll!ScSendResponseReceiveControls*<br><br>*BUCKET_ID:<br>X64_APPLICATION_FAULT_STATUS_BREAKPOINT_sechost!ScSendResponseReceiveControls+13b*<br><br>*WATSON_STAGEONE_URL:<br>http://watson.microsoft.com/StageOne/VldIos_exe/6_1_0_54321/526180ca/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1<br><http:>*<br><br>*Followup: MachineOwner*<br><br>*0:001&gt; .lastevent*<br><br>*Last event: 1b74.5a0: Break instruction exception - code 80000003<br>(first/second chance not available)*<br><br>* debugger time: Wed Nov 6 12:18:07.287 2013 (UTC - 8:00)*<br><br>*0:001&gt; r*<br><br>*rax=0000000001cc7800 rbx=0000000000000000 rcx=0000000001cc9980*<br><br>*rdx=fefefefefefefefe rsi=00000000ffffffff rdi=0000000000000134*<br><br>*rip=0000000076d712fa rsp=0000000001ccf8a8 rbp=0000000000000000*<br><br>*r8=0000000000000030 r9=0000000000000079 r10=0000000000000000*<br><br>*r11=0000000000000246 r12=0000000000000000 r13=0000000000000000*<br><br>*r14=0000000000000000 r15=0000000000000000*<br><br>*iopl=0 nv up ei pl zr na po nc*<br><br>*cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b<br>efl=00000246*<br><br>*ntdll!NtWaitForSingleObject+0xa:*<br><br>*0000000076d712fa c3 ret

0:001> ub 0000000076d712fa*<br><br>*ntdll!NtMapUserPhysicalPagesScatter:*<br><br>*0000000076d712e0 4c8bd1 mov r10,rcx

0000000076d712e3 b800000000 mov eax,0*<br><br>*0000000076d712e8 0f05 syscall

0000000076d712ea c3 ret*<br><br>*0000000076d712eb 0f1f440000 nop dword ptr [rax+rax]

ntdll!NtWaitForSingleObject:

0000000076d712f0 4c8bd1 mov r10,rcx*<br><br>*0000000076d712f3 b801000000 mov eax,1

0000000076d712f8 0f05 syscall*<br><br>*0:001&gt; ub 000000014003821f

VldIos!InitService+0x60 [d:\xxx\io service\srvcctrl.cpp @ 77]:

00000001400381f0 488905313b1000 mov qword ptr<br>[VldIos!hTerminateEvent (000000014013bd28)],rax

00000001400381f7 48833d293b100000 cmp qword ptr<br>[VldIos!hTerminateEvent (000000014013bd28)],0

00000001400381ff 7464 je VldIos!InitService+0xd5<br>(0000000140038265)

0000000140038201 488d1518400b00 lea rdx,[VldIos!szDelete+0x178<br>(00000001400ec220)]

0000000140038208 488b0d41361000 mov rcx,qword ptr [VldIos!WSMLog<br>(000000014013b850)]

000000014003820f e8229efaff call<br>VldIos!ILT+4145(?DebugOSMLoggingQEAAXPEBDZZ) (000000013ffe2036)

0000000140038214 488d4c2428 lea rcx,[rsp+28h]*<br><br>*0000000140038219 ff15c9eb1000 call qword ptr
[VldIos!_imp_StartServiceCtrlDispatcherA (00000001`40146de8)]

Below is the source codes of my InitService() function, which calls
StartServiceCtrlDispatcher() to start the service, then waits for the
terminate event.

void InitService( DWORD dwStartStatus )

{

dwStartServiceStatus = dwStartStatus;

SERVICE_TABLE_ENTRY serviceTable =

{

{ “”, ( LPSERVICE_MAIN_FUNCTION)ServiceMain },

{ NULL, NULL }

};

if ( ( hTerminateEvent = CreateEvent( NULL, FALSE, FALSE, NULL ) )
!= NULL )

{

LOG_DBG(“InitService() - Starting service control…\n”);

if ( !StartServiceCtrlDispatcher(serviceTable) )

{

::ErrorHandler( “StartServiceCtrlDispatcher Error -
0x%x”, GetLastError() );

::SetEvent( hTerminateEvent );

}

::WaitForSingleObject( hTerminateEvent, INFINITE );

::CloseHandle( hTerminateEvent );

}

else

{

LOG_ERR(“Failed to create terminateevent!\n”);

::ErrorHandler( “CreateTerminateEvent Error.” );

}

}

Please note that the service did not die when it started. It
successfully started and ran for random time before hitting this ?break
point?. I collected 2 separate crash dumps and they pointed to the same
failure point.

Appreciate your guidance/advice. Thanks!

M.</http:></http:>

2

Why do you have to apply the cast to ( LPSERVICE_MAIN_FUNCTION) to your ServiceMain function pointer? Does it compile without the cast?

3

Connect kernel debugger to the SUT. When it hits the breakpoint, have them to issue ‘k’ command to dump the call stack.

4

thanks for the respond.
It compiles fine w/o the cast. The codes (w/ cast) were copied from
windows’ sample codes. Do you suspect the crash may have something to do
with the cast?
thanks.

On Wed, Nov 6, 2013 at 4:57 PM, wrote:

> Why do you have to apply the cast to ( LPSERVICE_MAIN_FUNCTION) to your
> ServiceMain function pointer? Does it compile without the cast?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

5

First note: application CRT has a number of breakpoint instructions
hard-coded into it, and they get hit only when some CRT disaster (such as
garbaged free-pool structures) is discovered.

It does not look like this is what happened to you, because the faulting
IP is 00000000`00000000, which suggests a stack clobber.

The cast is evil, because if the function does not have the right
prototype, it can cause a stack-synch error. This is unlikely since your
service is running native 64-bit, and there is exactly one calling
convention in 64-bit, used everywhere. The 64-bit compiler treats the
attributes __cdecl, __stdcall, __fastcall as noise words and ignores them.
In the 32-bit world, the cast can cause a stack-sync error.

However, in this case, I suspect a buffer overrun on a fixed-size local
variable, which has erased the return address, causing a fault by
attempting to fetch an instruction from the (nonexistent) location 0
(because the RET instruction loaded the value on the stack into the RIP,
and the value on the stack was 0). If true, then it is erroneously
reported as an 80000003 fault. If, on the other hand, it was one of the
CRT INT3 calls that are hardwired, then it would be a heap clobber due to
an overrun of a dynamically-allocated buffer. Make sure that for strings
you do not use strlen() to determine their length, but _tcslen() *
sizeof(TCHAR). Otherwise you are going to allocate buffers half the size
you need, damage the heap, and trigger one of the internal INT3 calls,
which erroneously reports the RIP as 00000000`00000000.

Check carefully every allocation to make sure it is the right size. I
don’t know if App Verifier will work with system services; I’ve never
tried it. I would recommend looking into this. It might find the error
faster than desk checking.

Note that heap damage can go undiscovered for hours, which in “computer
years” means that suns have had time to form and die, the solar system has
moved a significant angular amount across the galaxy because of the
galactic rotation (200,000,000 years), etc. So this is why it may be very
hard to figure out.

Note that you don’t need a debug version to run in order to debug it. A
version which is compiled unoptimized is one solution; another is that you
need to create a .pdb file for the release version. The newer versions of
VS do this automatically; the older ones, for reasons unfathomable to
outsiders, did not create a .pdb file for a release version.

thanks for the respond.
It compiles fine w/o the cast. The codes (w/ cast) were copied from
windows’ sample codes. Do you suspect the crash may have something to do
with the cast?
thanks.

On Wed, Nov 6, 2013 at 4:57 PM, wrote:
>
>> Why do you have to apply the cast to ( LPSERVICE_MAIN_FUNCTION) to your
>> ServiceMain function pointer? Does it compile without the cast?
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>>
>> OSR is HIRING!! See http://www.osr.com/careers
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

6

Mandy Lam wrote:

It compiles fine w/o the cast. The codes (w/ cast) were copied from
windows’ sample codes.

Yes, that’s a bad habit. Every cast is dangerous and should be questioned.

Do you suspect the crash may have something to do with the cast?

If the code also compiles and fails without the cast, then I hope it is
clear the cast is not involved.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

7

Thanks Tim and newcomer (sorry if I get your name wrong) for the valuable
advice.

  1. Thanks for explaining why casting is dangerous. If I understand it
    right, wrong casting may cause stack-sync error when the function returns.
    (e.g. in cdecl calling style, caller cleans up/balance the stack; while in
    fastcall, callee cleans up the stack.) In my case, since I casted the
    ServiceMain function, the problem, if any, should occur upon ServiceMain
    returns. Since my ServiceMain() function does not return until the service
    receives SERVICE_CONTROL_STOP, which is not the case when the crash
    happened, it is unlikely that the casting is what caused the crash (besides
    the fact as newcomer points out that there is exactly one calling
    convention in x64). Do I understand it right?

  2. I understand that if the stack was trashed (because of local
    variable buffer overrun, etc), the return address @ current function may
    becomes 0. When current function returns, RIP gets the wrong null value,
    crash happens and debugger will erroneously reports the RIP as
    00000000`00000000. However from what I can tell the stack content looks
    normal:

*STACK_TEXT: *

*000000000028ef38 000007fefcd210dc : 0000000000000000 0000000076d401d5
0000000000000004 00000000003fa7d0 : ntdll!NtWaitForSingleObject+0xa*

*000000000028ef40 000007fefef1affb : 00000000ffffffff 000007fefef1344c
0000000000000000 00000000000000d8 : KERNELBASE!WaitForSingleObjectEx+0x79*

*000000000028efe0 000007fefef19d61 : 00000000003e0c20 00000000000000d8
0000000000000000 0000000000000000 :
sechost!ScSendResponseReceiveControls+0x13b*

*000000000028f0d0 000007fefef19a51 : 000000000028f218 0000000000000000
0000000000000000 0000000000000000 : sechost!ScDispatcherLoop+0x121*

*000000000028f1e0 000000014003821f : 0000000000000000 00000000003e8990
000000000028f158 0000000000000000 :
sechost!StartServiceCtrlDispatcherA+0x6d*

*000000000028f210 000000014001f290 : 0000000000000000 00000001400c7348
000000000028ec28 0000000000000000 : VldIos!InitService+0x8f*

*000000000028f280 0000000061e7c419 : 000000014013b860 0000000000000000
00000000003137a9 000000000000000a : VldIos!CLhdIOSApp::InitInstance+0x9b0*

*000000000028f860 000000014007bec1 : 000000013ffe0000 0000000000000000
00000000003137a9 000000000000000a : mfc100d!AfxWinMain+0xc9*

*000000000028f8f0 0000000140073eb6 : 000000013ffe0000 0000000000000000
00000000003137a9 000000000000000a : VldIos!WinMain+0x31*

*000000000028f920 0000000140073bae : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : VldIos!__tmainCRTStartup+0x2f6*

*000000000028fa10 0000000076c1652d : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : VldIos!WinMainCRTStartup+0xe*

*000000000028fa40 0000000076d4c541 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0xd*

*000000000028fa70 0000000000000000 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x1d*

*0:000> r*

*rax=000007fefd9889c0 rbx=0000000000000000 rcx=000007fefd8d12b2*

*rdx=000000000028ed58 rsi=00000000ffffffff rdi=00000000000000d8*

*rip=0000000076d712fa rsp=000000000028ef38 rbp=0000000000000000*

* r8=0000000000000000 r9=0000000000000000 r10=000000000000002b*

*r11=0000000000000000 r12=0000000000000000 r13=00000000003e8990*

*r14=0000000000000000 r15=0000000000000000*

*iopl=0 nv up ei pl zr na po nc*

*cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b
efl=00000246*

*0:000> dd 00000000`0028ef38*

*00000000`0028ef38 fcd210dc 000007fe 00000000 00000000*

*00000000`0028ef48 76d401d5 00000000 00000004 00000000*

*00000000`0028ef58 003fa7d0 00000000 0028f080 00000000*

*00000000`0028ef68 00000000 00000000 00000048 00000000*

*00000000`0028ef78 00000001 000007fe 00000000 00000000*

*00000000`0028ef88 00000000 00000000 00000000 00000000*

*00000000`0028ef98 00000000 00000000 00000000 00000000*

*00000000`0028efa8 00000000 00000000 00000000 00000000*

*0:000> u 000007fe`fcd210dc*

*KERNELBASE!WaitForSingleObjectEx+0x79:*

*000007fe`fcd210dc 8bf0 mov esi,eax*

*000007fe`fcd210de 89442420 mov dword ptr [rsp+20h],eax*

*000007fe`fcd210e2 85c0 test eax,eax*

*000007fefcd210e4 0f8812c40100 js KERNELBASE!WaitForSingleObjectEx+0x130 (000007fefcd3d4fc)*

*000007fe`fcd210ea 85db test ebx,ebx*

*000007fefcd210ec 0f8556140000 jne KERNELBASE!WaitForSingleObjectEx+0xae (000007fefcd22548)*

*000007fefcd210f2 eb00 jmp KERNELBASE!WaitForSingleObjectEx+0x13e (000007fefcd210f4)*

*000007fe`fcd210f4 85db test ebx,ebx*

When the crash happens, esp=28ef38. The correct return
address (7fefcd210dc) is stored in the top of stack. The fact that the
debugger is able to re-construct the calling sequences by tracing the stack
all the way back to *ntdll!RtlUserThreadStart * seems to suggest that all
the return addresses within the stack are intact? I guess I do not
understand why the ?!analyze ?v? command indicates that the ?*FAULTING_IP
is **00000000`00000000**?* while the stack content looks normal (and hence
RIP should not have gotten null value) Can you kindly educate me?

Hope my questions make sense. Thank you again.

M.

On Thu, Nov 7, 2013 at 12:26 PM, Tim Roberts wrote:

> Mandy Lam wrote:
> >
> > It compiles fine w/o the cast. The codes (w/ cast) were copied from
> > windows’ sample codes.
>
> Yes, that’s a bad habit. Every cast is dangerous and should be questioned.
>
>
> > Do you suspect the crash may have something to do with the cast?
>
> If the code also compiles and fails without the cast, then I hope it is
> clear the cast is not involved.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

8

It’s possible that the crash actually occurs at a different thread than you see. If you attach the debigger post-fact, it may be different. Inspect all threads.

To catch the very moment of an exception, use a kernel debugger, and make sure that usermode breaks are not disabled (they should not be unless explicitly done).

9

Thank you all for the spot-on advice. I am happy to report that the issue
is found!
As I examine all threads of my process as suggested by grigora,
(fortunately I don’ need a live debugging session to do this), I found one
of the thread hit a “crt_debugger_hook”

*0:010> r*

*rax=0000cfbaaf951aca rbx=0000000000000000 rcx=0000000000000002*

*rdx=00000000c0000417 rsi=0000000000000000 rdi=0000000002a3b3b0*

*rip=000000006e902680 rsp=0000000002a32b98 rbp=0000000000000000*

* r8=0000000000000001 r9=00000000000000f4 r10=0000000000000000*

*r11=0000000000000286 r12=0000000000000000 r13=0000000000000000*

*r14=0000000000000000 r15=0000000000000000*

*iopl=0 nv up ei pl nz ac po cy*

*cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b
efl=00000215*

*MSVCR100D!_crt_debugger_hook:*

*000000006e902680 894c2408 mov dword ptr [rsp+8],ecx ss:0000000002a32ba0=fefefefe*

*0:010> kv*

*Child-SP RetAddr : Args to
Child : Call Site*

*0000000002a32b98 000000006e8f378d : fefefefefefefefe fefefefefefefefe
fefefefefefefefe fefefefefefefefe : MSVCR100D!_crt_debugger_hook*

*0000000002a32ba0 000000006e8f372d : fefefefe00000002 00000000c0000417
0000000000000001 0000000002a33268 : MSVCR100D!call_reportfault+0x3d*

*0000000002a331c0 000000006e8f36f3 : 000000006e947990 000000006e947a08
000000006e9478b0 00000000000000f4 : MSVCR100D!invoke_watson+0x2d*

*0000000002a331f0 000000006e81624a : 000000006e947990 000000006e947a08
000000006e9478b0 00000000000000f4 : MSVCR100D!invalid_parameter+0x83*

*0000000002a33240 000000006e8116b3 : 0000000002a37360 0000000000004000
00000001400d0318 0000000000000000 : MSVCR100D!vsprintf_s_l+0x2aa*

*0000000002a332b0 00000001400237e8 : 0000000002a37360 0000000000004000
00000001400d0318 0000000002a3b400 : MSVCR100D!sprintf_s+0x43*

*0000000002a33300 0000000140023a18 : 0000000000d781c0 0000000000000002
00000001400d0318 0000000002a3b400 :
VldIos!OSMLogging::WriteDiagnostic+0x78 [d:\xxx\common\osmlogging.cpp @
225]*

*0000000002a3b3c0 000000014004e264 : 0000000000d781c0 0000000140107720
0000000004ba0390 0000000060de0088 : VldIos!OSMLogging::Warning+0xb8
[d:\xxx\common\osmlogging.cpp @ 306]*

*…*

This points me to the problem code, which erroneously feed a random DWORD
value to a print statement of string format (%s). Depends on the value of
the DWORD and the content of that random location, the print buffer may
become too small, which causes crt bug check.

I am not able to verify the fix since the customer has already changes
their setup. But I am pretty confident that this is it.
Now I can have a happy weekend. Thank you again!!!

On Thu, Nov 7, 2013 at 7:07 PM, wrote:

> It’s possible that the crash actually occurs at a different thread than
> you see. If you attach the debigger post-fact, it may be different. Inspect
> all threads.
>
> To catch the very moment of an exception, use a kernel debugger, and make
> sure that usermode breaks are not disabled (they should not be unless
> explicitly done).
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>