Extending the idea of excluding individual pages, you might try excluding big chunks of memory. I thought there used to be an option to force only memory above 4GB to be used. Ideally you could binary search excluded memory. You potentially could write a little boot start driver that allocated do nothing buffers in specific ranges, testing if you can cause the corruption to only happen in harmless areas.
Jan
On Tue, Nov 24, 2015 at 12:42 PM -0800, “Scott Noone” > wrote:
Thanks for the replies!
That’s an interesting idea, I hadn’t thought of that one.
It’s definitely not always the same page, though the above makes me wonder
if it’s in a specific range of pages. Unfortunately extracting the bad PFN
is more manual than I would prefer so automating this might be difficult. I
have a good sample on which this might work though…
This has been a bit of a dead end only because it’s taking so long to get
the results. I ran !search on one of the more recent dumps this AM at around
7AM ET and it’s still going…I’ll continue to see if it gets me anywhere
though.
I’ll definitely try this if I can identify a range.
I did try !searchpte on several dumps but didn’t find an obvious
culprit…I’ll give this a few more tries on other dumps though to see if I
can catch one in the act.
Verifier has been run (of course) but not the checked images, also a very
good idea.
Thanks again for the ideas! I’ll definitely update if we find the smoking
gun.
-scott
OSR
@OSRDrivers
“Pavel Lebedinsky” wrote in message news:xxxxx@ntdev…
Such corruptions are often caused by firmware issues, like failing to
exclude memory used by devices from the OS memory map.
Some things you can try:
1. Check if there is a specific physical page that gets corrupted more often
than other pages.
2. If corruption appears to hit completely random physical pages, do !search
0000001000400004 and see if there are any other instances of that value,
especially at page offsets FD8/D70. If yes, check these physical pages in
other dumps to see if there is a pattern.
3. If you can identify a suspect physical page, try excluding it using
bcdedit /set badmemorylist.
4. Optionally, do !pfn and !searchpte on the suspect PFN to see what the
page is being used for and where it might be mapped. Check for problems like
a valid PTE pointing to a free/standby page.
5. Try booting with a checked kernel and see if it hits any asserts in the
memory manager.
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: Tuesday, November 24, 2015 7:34 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Corruption Mystery: Any Ideas?
A potentially interesting puzzler for a Tuesday…
We’re looking at a series of crash dumps from a client and am hoping that
this corruption looks familiar to someone. This is an end user installation
and none of our software is running on these machines, it’s strictly a crash
analysis exercise.
The crashes are spread out over 1,000+ “identical” machines. No one machine
crashes with great frequency, but spread out across the install there’s a
few a day. Systems survive anywhere from a few hours to six days before
crashing.
Just looking at the crash codes isn’t helpful, they have about every crash
code you could ever imagine (QUOTA_UNDERRUN??) and it’s blamed on various
modules. However, digging deeper a very consistent pattern emerges.
Specifically, we are consistently seeing one of two values “randomly” appear
in memory:
? 0x0000001000000004<br>? 0x0000001000400004
Interestingly, when the corruption is discovered the value very, very
frequently appears at physical memory page offset 0xFD8 (most common) or
0xD70 (less common).
For example, in one crash the problem was that the MRXSMB20 image file is
corrupted:
3: kd> !chkimg -d mrxsmb20
fffff800826a6fd8-fffff800826a6fdd 6 bytes -
mrxsmb20!Smb2UpdateFileInfoCacheEntry+4c8
[89 7d 18 49 89 45:04 00 00 00 10 00]
fffff800826a6fdf - mrxsmb20!Smb2UpdateFileInfoCacheEntry+4cf (+0x07)
[e8:00]
7 errors : mrxsmb20 (fffff800826a6fd8-fffff800826a6fdf)
Dumping the start of the corrupted range, we see our offset and value:
3: kd> dq fffff800826a6fd8
fffff800826a6fd8 0000001000000004 4c2b894c0000e99c<br>fffff800826a6fe8 ade901b641986d8b 850f02f883fffffd
fffff800826a6ff8 8bc03345fffffbb5 445e15ffce8b49d7<br>fffff800826a7008 fb9f850fc0840002 03fffffefee9ffff
In another crash a pool header is corrupted:
2: kd> !pool ffffc00089adcd70
Pool page ffffc00089adcd70 region is Paged pool …
ffffc00089adcc00 size: 170 previous size: b0 (Free ) MPsc
ffffc00089adcd70 doesn’t look like a valid small pool allocation, checking
to see if the entire page is actually part of a large page allocation…
2: kd> dq ffffc00089adcd70
ffffc00089adcd70 0000001000000004 8d5eb1494b83d33a<br>ffffc00089adcd80 0000000000000000 ffffe000bdecfec0
ffffc00089adcd90 ffffe000bc728860 ffffc000`89adcd98
Due to the fact that the crash appears at random in different virtual
address ranges (paged pool, non-paged pool, code, proto PTEs, working set
lists, etc.), we believe that the corruption must be generated by a device
in the system (or by the platform).
We have tried various things to narrow this further and analyzed the
corruption across hundreds of dump files. The systems are generally idle
when the corruption is discovered, which makes it hard to go back in time
and figure out who might be using the value.
So, my question to you all is: does this LOOK like anything to you? Do those
corruption values hold any meaning to you? What about the offsets of 0xFD8
or 0xD70? I realize it’s not much to go on, but stranger things have
happened…
—
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
