I have a crash happening due to corruption in lookasidelist.
I got the below analysis from microsoft.
Driver1 inserts entries along with Driver2 in lookaside list.
whenever a crash is occurred, we see that a lookaside entry created by driver1 often causing the issue. However, we look in detail, we would find that the corruption would be caused by one of the prev or next nodes which will be created by driver2.
Entry->Prev-Next != Entry and this causes crash.
They say that driver2 doesnt call ExDeleteLookasideListEx correctly. Though they have freed the memory it is referenced in other nodes. The suggestion was to call ExDeleteLookasideListEx.
Couple of basic questions on the above analysis as I am not sure on the suggestion:
crash happens when both drivers are running and my driver needs those lookaside list for its operation. wont calling deletelookasidelist will jeopardize my driver as my driver is still there in the system. I call deletelookasidelist during stop,remove,shutdown,surprise_removal IRPs.
suggestion was to call deletelookasidelistEx during IRP_MJ_CLOSE or IRP_MJ_CLEANUP. but this IRP can come to my driver even when it is running. I believe I cannot delete the lookaside list for these IRPs.
once freed I think the memory is free for other access. In which scenario post freeing the memory the nodes in lookaside points to the freed memory.
I don’t understand that analysis at all. ExDeleteLookasideListEx removes the entire list and all its contents. If you have drivers sharing the list, you can’t remove the entire list while one of them is still using it. Are you actually doing that?
No I am not doing it. This was the same doubt I had.
Also I dont have list shared with another driver. The analysis document says that my driver have reference to the nodes that are freed by my driver.
to remove the reference they had asked to call deletelookasidelist as a suggestion.
. I am sure that deleting is a wrong suggestion. my driver is running and using it.
I am looking into how to fix this issue. what is the other way to do it. I understand that lookaside list kind of doubly list. but we do call freelookasidelist whenever we need to free the node.
The analysis document says
“there are still some nodes in the lookaside list pointing to the freed memory location. This results in a typical Use-After-Free scenario”
Driver1 inserts entries along with Driver2 in lookaside list.
Which is it? One of those statements is false.
“there are still some nodes in the lookaside list pointing to the freed memory location. This results in a typical Use-After-Free scenario”
That would be a bug in the lookaside list functions, wouldn’t it? What (approximately) are you storing in the lookaside list? Does it include pointers to other list members?
Despite the poor description of how the lookaside list is used in your driver, two areas of investigation stand out to me
you are explicitly freeing the memory that you have also returned to the lookaside list. The lookaside list owns the lifetime of the memory once you have freed it back to the list
you are using the memory after you have returned it to the lookaside list, thus trashing the lookaside list entry state
Have you tested your driver(s) which use the lookaside list under Driver Verifier?
Tim,
I store the below in my lookasidelist structure. yes it includes pointers.
pointer to the allocated buffer
pointer to the freed buffer
pooltag
pooltype
Doron,
I suppose returning the memory to lookasidelist means freeing the memory using ExFreeToLookasideListEx. if lookaside list owns the lifetime of memory then how does other driver access the same memory when my driver has not called Deletelookasidelist. Am I missing something here?
the analysis doc says the other driver tries to insert the node into the lookasidelist when crash is seen. Also the node where it tries to access has references in my driver.
Few basic questions as I am a bit confused here.I googled…But did not get the answer.
can 2 drivers share lookaside list?
If yes what type of drivers share lookasidelist. Any examples?
Can two drivers share a look aside list? Yes. But should they? I think that is a bad idea as you run into lifetime issues of the list itself and probably run into DV having problems tracking state when enabled. There are no examples of two drivers sharing a lookaside. The bigger question is why are you designing two drivers to share a lookaside? Why can’t each driver have their own lookaside ? Yes, calling ExFreeToLookasideListEx means the lookaside now owns the memory. It is an internal private decision if the list holds onto the allocation to reuse it later or frees it. That means that while the memory is freed to the lookaside, if it is kept for reuse and a driver incorrectly still references that memory, it can corrupt lookaside state. IIRC one thing that DV can do is force the lookaside to immediately free allocations when ExFreeToLookasideListEx is called. Deletelookasidelist does two things. 1) it removes the lookaside from the kernels list of lookasides. 2) it frees all allocations (waiting for reuse) still in the list. Once you call Deletelookasidelist you can’t use the lookaside anymore. You keep asking how the two drivers can share the memory. Sharing memory has nothing to do with how it’s allocated, so you may have underlying design problems in how these two drivers interact and don’t have a clear way to handle lifetime of allocations to know when they can be freed.
@Doron Can two drivers share a look aside list? Yes. …
Eh?? I had always considered lookaside lists [https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/using-lookaside-lists] as a way to preallocate a bunch of of buffers, always available in case I need some memory and the system is memory constrained without going through the hassle of making my own slab allocator. The example in the literature shows just that, when you need an SRB then there’s always going to be memory available … in fact that’s what I’ll use for some of my drivers that need to be constantly DMA’ing stuff off of hardware of some sort, because it’s always going to be there …
I’m also aware that driver allocated buffers can be pushed up and down the IRP stack so as long as the other driver(s) in the stack knows not to free the memory (which they shouldn’t anyway) then using a lookaside list for that will also work …
I had always considered, however, that the lookaside lists are driver-scope and not system-scope, that I can’t just shuffle them around like some kind of kernel level global atom table [https://docs.microsoft.com/en-us/windows/win32/dataxchg/about-atom-tables] … but are you saying that is not true, that it is a global lookaside list pool and I could use the entries like some kind of kernel level atom table, shuffling them around from driver to peer driver?
From https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/using-lookaside-lists , note the last sentence The operating system maintains state about all paged and nonpaged lookaside lists that are currently being used, dynamically tracking the demand for allocations and deallocations of entries in all lists, and available system pool for new entries. When demand for allocations is high, the operating system increases the number of entries it holds in each lookaside list. When demand falls again, it frees surplus lookaside entries back to system pool. Can a lookaside list be shared between two drivers? TECHNICALLY , yes. Is a good architectural choice? ABSOLUTELY no. I never said there was a global lookaside pool. When you initialize a lookaside list, your driver local list is inserted into private list in the kernel that tracks all active lookaside lists. Deleting the lookaside removed it from the kernels tracking list. This thread is confusing sharing memory with how it’s allocated. They are two separate topics. When sharing memory between two drivers, the method in which the buffer was allocated should be opaque. It should not matter if the buffer came from a lookaside, is a part of the device extension, etc.
Kernel memory is all the same, system-wide. The operating system has no concept of driver “ownership” of memory. When you create a lookaside list in a driver, it’s just a pointer to some memory. Your driver can hand that pointer to anyone you want to, and anyone who has the pointer can allocate lookaside buffers from that list and free them back again. It’s just memory. The PROBLEM comes when you need to shut things down, because YOU have to provide the syncronization. No one can delete the lookaside list until all the drivers are through using it. The system provides no mechanism for managing that.
Stepping back, it’s not at all clear to me that’s what you’re doing. ARE you actually sharing a lookaside list between two drivers? Your statements are contradictory. ARE you seeing this crash when you try to delete the while lookaside list? Why are you trying to delete it? Are you coordinating deleting the list with the other driver? Are you sure all of the buffers have been returned to the list before deleting it?
Or is your problem unrelated to “deleting the whole list”, and simply just some screwup with allocating and freeing buffers from the list?
When demand for allocations is high, the operating system increases the number of entries it holds in each lookaside list. When demand falls again, it frees surplus lookaside entries back to system pool.
And when demand for pool increases, it frees entries for your look-aside list arbitrarily. Thereby penalizing your driver, and canceling one good reason to use lookaside lists in the first place — which is to ensure you have memory preallocated.
But Doron knew I was going to say that, because it’s yet one more of my hot buttons.
When demand for allocations is high, the operating system increases the number of entries it holds in each lookaside list. When demand falls again, it frees surplus lookaside entries back to system pool.
And when demand for pool increases, it frees entries for your look-aside list arbitrarily. Thereby penalizing your driver, and canceling one good reason to use lookaside lists in the first place — which is to ensure you have memory preallocated.
But Doron knew I was going to say that, because it’s yet one more of my hot buttons.
Peter
Eh? How does the OS determine what entry is “surplus” and gets freed? I had always considered it would be there until I told the OS to make it go away … should I write something to each buffer to mark it, like a dog on a fencepost?
All entries freed and held in the lookaside are candidates for the kernel to reclaim and truly free. Remember that private list of lookasides in the kernel? Under memory pressure, the kernel can walk that list and for each lookaside, scavenge and free entries it holds.
Ah, that makes sense … so if I think that I might sometime need up to 64 blocks of 64K each then I would create a lookaside list [ExInitializeLookasideListEx] and then call [ExAllocateFromLookasideListEx] 64 times for 64K allocations, then once that’s done as long as I don’t “free” them but instead keep my own dirty/ clean state for those buffers then I’m OK … when the device/ context unloads then I can call [ExFreeToLookasideListEx] 64 times, then [ExDeleteLookasideListEx] to finish up …
You don’t need a lookaside for this pattern. The allocator in the lookaside is no different than allocating from pool in the driver. Just allocate your 64 buffers and then free them. The intent of the lookaside is to cooperate with the kernel when it is under memory pressure and return allocations, if you never want to return them don’t use a lookaside (aka Peters issue with the lookaside pattern).
I think I got confused totally here.
I am not sharing any lookaside list with any driver. The question was out of curiosity. Also I got confused with the system-wide set of active lookaside lists where every driver which initialize the lookasidelist adds an entry to its tail.
My dump analysis says that
When there is a request to delete the lookaside list, the entries in the system wide list are adjusted using its prev and next pointers.
The other driver tries to delete the lookaside list created by it using nt!ExDeleteNPagedLookasideList but when adjusting the existing entries in the system wide list, it found an invalid entry resulted in the crash.
the system wide lookaside list entered an invalid state with incorrect entries due to my driver not calling the appropriate Delete function (ExDeleteLookasideListEx).
I got confused with the system wide lookaside list with my lookaside list by itself created by the driver.
