StorPortGetLogicalUnit with dynamic allocations

For a virtual storport miniport, i need to store disk information that varies in size in response to certain scsi commands that occur at run time. The size of memory needed can range from 0 bytes (typical) to extremely large sizes (test environment). It is straight forward to perform a memory allocation and store the pointer in the StorPortGetLogicalUnit area to have an appropriate sized buffer for this on a per-disk basis. The question I have is at what point can one free the memory? Say in the case the adapter is removed.