Contributing some information to the thread as I work for the “customer” We are very grateful for the work OSR has regarding the troubleshooting our mystery. (Although it did take while for us to get organize and start collecting hundreds of BSOD dumps and convince OSR that we needed their help)
Network card,
- M93p’s have an Intel I217-ML They are connected to Cisco 2960s switches on GB ports
Intel drviers, had version 12.11.96.1 for most of the year, updated to 12.12.80.1920 followed by 12.13.17.7 recently. Using default driver settings. Jumbo frames are not enabled.
USB
- The M93p only has physical USB3 ports, (Lenovo hardware maintenance manual list’s an optional USB2 port which we don’t have)
USB info as reported by msinfo32.
Intel(R) USB 3.0 eXtensible Host Controller - 0100 (Microsoft) PCI\VEN_8086&DEV_8C31&SUBSYS_30A317AA&REV_04\3&11583659&0&A0
Intel(R) 8 Series/C220 Series USB EHCI #2 - 8C2D PCI\VEN_8086&DEV_8C2D&SUBSYS_30A317AA&REV_04\3&11583659&0&D0
Intel(R) 8 Series/C220 Series USB EHCI #1 - 8C26 PCI\VEN_8086&DEV_8C26&SUBSYS_30A317AA&REV_04\3&11583659&0&E8
All of our system have at least 2 USB devices at all times : Keyboard and mice.
USB drivers are the Microsoft Windows 8.1 x64 Enterprise drivers :
USBXHCI.SYS
The one critical piece of information Scott has omitted, (as it might lead to conjecture) is that when we recently disabled “Turn off Monitor after Idle” in the windows Power Profile.
The BSOD’s all but stopped !
We did this after OSR observed that the memory scribble BSOD’s were clustered around user logon and power transition events. Our PC’s are on 24/7 with weekly reboots. Users will logoff or stay logged on the end of the day, and prior to our recent changes montior would power off after 15min, User returns in the morning, monitor wakes up. and log’s-on. We are office workers typical person is use the office Office 2013 Suite; Outlook, Word, Excel products all day. Nothing fancy.
All of our M93p are on the High performance profile, we don’t sleep or do any hibernation. Prior to Scott’s recommendation on our Power Settings, We had Turn off Monitor after 15 min of idle, USB low power mode, and HD power-off after idle. These settings have now all been disabled.
After making this change about 1 week ago and rebooting all 1000+ systems the BSOD’s stopped.
Not knowing at time which action we took changed the behavior, we update the NIC and intel Storage drives and power changes all at the same time, 1 week later we re-enabled the Monitor power-off at idle.
Not more than 30 min after making this change we had our 1st BSOD with scribble memory. Upon seeing this change in behavior we reversed course a disabled the Monitor power off again, Reboot all system. Have have only encountered one BSOD since in about 6 days. The “normal” trend line for these systems prior was about 2-6 BSOD per day sometimes hiting peaks of 10-15 unique machines BSODing per day.
The other interesting part of this problem is we have about 100+ Dell E7440 Laptops, these machine run the same image as our M93p. At the image deployment ( SCCM OSD) image different driver package are injected in the image. (Lenovo and Dell both provide drive package for SCCM deployments these always contain out dated drivers that are supposed be vetted and tested.
Suspecting possible bad drivers we now find the most recent drivers for our hardware on the catalog.update.microsoft.com site and download and deploy these version.
Post image we updated the drivers for common hardware ; NIC , Storage, Intel HD iGPU using the same driver for both M93p and the E7440. No single “memory scribble” BSOD has been found on the E7440.
Last part of information regarding our problem. We never knew we has serious issue with BSOD’s unit we started looking for BSOD’s and collecting the dumps. But we always new we had and still have serious issue with Applications from A to Z crashing on our system. About 60-70% of these crashes are buffer over-run’s C000005
Example : Taken from Windows AppCrash events form various PC’s things always crash with Exception code: 0xc0000005.
Although our BSOD have stopped since we made power change, the general user mode crashes have not. Are they getting corrupted by same memory scribble error ?
Date-Time , .ProgramName. module, Exception code
11/24/2015 08:24:11 splwow64.exe KERNELBASE.dll e06d7363
11/24/2015 08:37:40 WINWORD.EXE wwlib.dll c0000005
11/24/2015 08:43:25 CcmExec.exe ntdll.dll c0000005
11/24/2015 08:58:29 AUDIODG.EXE WMALFXGFXDSP.dll c0000005
11/24/2015 08:58:35 splwow64.exe KERNELBASE.dll e06d7363
11/24/2015 09:00:27 wfcrun32.exe ntdll.dll c0000005
11/24/2015 09:12:59 xdcla.exe Profiler.dll c0000135
11/24/2015 09:19:16 WINWORD.EXE mfc100u.dll c0000005
11/24/2015 09:22:30 OUTLOOK.EXE combase.dll c0000005
11/24/2015 09:26:06 lync.exe ntdll.dll c0000005
11/24/2015 09:27:09 IEXPLORE.EXE ntdll.dll c0000005
11/24/2015 09:30:40 OUTLOOK.EXE MSVCR100.dll 40000015
11/24/2015 09:33:23 OUTLOOK.EXE mso.dll c0000602
11/24/2015 09:39:08 splwow64.exe KERNELBASE.dll e06d7363
11/24/2015 09:45:01 Acrobat.exe ntdll.dll c0000374
11/24/2015 09:47:31 OUTLOOK.EXE mso.dll c0000602
11/24/2015 09:51:32 IEXPLORE.EXE MSHTML.dll c0000005
11/24/2015 09:52:52 IEXPLORE.EXE MSHTML.dll c0000005
11/24/2015 09:53:01 IEXPLORE.EXE MSHTML.dll c0000005
11/24/2015 09:55:01 OUTLOOK.EXE unknown c0000005
11/24/2015 09:58:43 ppscanmg.exe KERNELBASE.dll e06d7363
11/24/2015 10:03:15 IEXPLORE.EXE ntdll.dll c0000005
11/24/2015 10:06:20 Acrobat.exe ntdll.dll c0000374
11/24/2015 10:09:12 IEXPLORE.EXE igd10iumd32.dll c0000005
11/24/2015 10:09:24 IEXPLORE.EXE ntdll.dll c0000409
11/24/2015 10:10:14 splwow64.exe KERNELBASE.dll e06d7363
11/24/2015 10:11:04 IEXPLORE.EXE Flash.ocx c0000005
11/24/2015 10:11:24 svchost.exe_Dnscache ntdll.dll c0000008
11/24/2015 10:21:36 OUTLOOK.EXE ntdll.dll c0000374
11/24/2015 10:24:30 IEXPLORE.EXE ntdll.dll c0000005
11/24/2015 10:24:40 OUTLOOK.EXE mso.dll c0000602
11/24/2015 10:31:21 OUTLOOK.EXE ntdll.dll c0000374
11/24/2015 10:31:30 WINWORD.EXE unknown c0000005
11/24/2015 10:34:41 Concordance Image.exe Concordance Image.exe c000041d
11/24/2015 10:34:41 Concordance Image.exe Concordance Image.exe c0000005
Naim