Peculiar BSOD when running with secure boot

Alex_Grig · August 4, 2016, 11:13am

>The BSOD may be happening because of page-in error, which may happen if the file
that backs the driver section is deleted, but the handle is not closed.

It’s a bit different, though. The section object keeps a reference to the file object, but not a handle. When the last handle is closed, the file is deleted, even though the section keeps a reference. A page-in will read all zeros (if I remember it correctly).

Scott_Noone_OSR · August 4, 2016, 11:53am

You can’t delete a file with an active image mapping, the file system
prevents it. See the FASTFAT source:

//
// If the user wants write access access to the file make sure there
// is not a process mapping this file as an image. Any attempt to
// delete the file will be stopped in fileinfo.c
//
// If the user wants to delete on close, we must check at this
// point though.
//

if (FlagOn(*DesiredAccess, FILE_WRITE_DATA) || DeleteOnClose) {

Fcb->OpenCount += 1;
DecrementFcbOpenCount = TRUE;

if (!MmFlushImageSection( &Fcb->NonPaged->SectionObjectPointers,
MmFlushForWrite )) {

Iosb.Status = DeleteOnClose ? STATUS_CANNOT_DELETE :
STATUS_SHARING_VIOLATION;
try_return( Iosb );
}
}

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

The BSOD may be happening because of page-in error, which may happen if the
file
that backs the driver section is deleted, but the handle is not closed.

It’s a bit different, though. The section object keeps a reference to the
file object, but not a handle. When the last handle is closed, the file is
deleted, even though the section keeps a reference. A page-in will read all
zeros (if I remember it correctly).

Andrey_Bazhan · August 4, 2016, 2:10pm

You need to investigate how FILE_OBJECT FileName.Length field is getting damaged.
Seems like this is the reason why Device Guard is blocking your driver.

0: kd> r
Last set context:
rax=0000000000000000 rbx=0000000000000000 rcx=ffffe001765671f0
rdx=ffffe00196625000 rsi=ffffe001765671f0 rdi=0000000000000000
rip=fffff8005c0263ac rsp=ffffd00020e91968 rbp=ffffd00020e91a70
r8=0400000000020020 r9=fffff801bfcb3c40 r10=00000000000001c8
r11=ffffd00020e91960 r12=ffffffff8000139c r13=0000000000000200
r14=ffffe00196625000 r15=0000000020206f49
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010246
DataNow_Driver!GsDriverEntry:
fffff8005c0263ac 48895c2408 mov qword ptr [rsp+8],rbx ss:0018:ffffd00020e91970=0000000000000000

0: kd> dt nt!_DRIVER_OBJECT ffffe001765671f0 DriverSection
+0x028 DriverSection : 0xffffe001`76d2b8a0 Void

0: kd> dt nt!_KLDR_DATA_TABLE_ENTRY 0xffffe00176d2b8a0 +0x000 InLoadOrderLinks : _LIST_ENTRY [0xfffff801bfd5dcf0 - 0xffffe00176ebeba0] +0x010 ExceptionTable : 0xfffff8005c023000 Void
+0x018 ExceptionTableSize : 0x63c
+0x020 GpValue : (null)
+0x028 NonPagedDebugInfo : (null)
+0x030 DllBase : 0xfffff8005c010000 Void +0x038 EntryPoint : 0xfffff8005c0263ac Void
+0x040 SizeOfImage : 0x19000
+0x048 FullDllName : _UNICODE_STRING “\SystemRoot\system32\DRIVERS\DataNow_Driver.sys”
+0x058 BaseDllName : _UNICODE_STRING “DataNow_Driver.sys”
+0x068 Flags : 0x49104020
+0x06c LoadCount : 1
+0x06e u1 :
+0x070 SectionPointer : 0xffffc0018b662fc0 Void +0x078 CheckSum : 0x22ce9 +0x07c CoverageSectionSize : 0 +0x080 CoverageSection : (null) +0x088 LoadedImports : 0xffffc00182b1a190 Void
+0x090 Spare : (null)
+0x098 SizeOfImageNotRounded : 0x19000
+0x09c TimeDateStamp : 0x579a15e7

0: kd> dt nt!_SECTION 0xffffc0018b662fc0 u1.ControlArea +0x028 u1 : +0x000 ControlArea : 0xffffe0019666c010 _CONTROL_AREA

0: kd> dt nt!_CONTROL_AREA 0xffffe0019666c010 FilePointer.Object +0x040 FilePointer : +0x000 Object : 0xffffe00176cfe996 Void

0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 FileName. +0x058 FileName : "\Windows\System32\drivers\DataNow_Dri" +0x000 Length : 0x4a +0x002 MaximumLength : 0x78 +0x008 Buffer : 0xffffc001828d99a0 “\Windows\System32\drivers\DataNow_Dri”

0: kd> du /c 100 0xffffc001828d99a0 ffffc001828d99a0 “\Windows\System32\drivers\DataNow_Driver.sys”

0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 SectionObjectPointer +0x028 SectionObjectPointer : 0xffffe00176721688 _SECTION_OBJECT_POINTERS

0: kd> dt nt!_SECTION_OBJECT_POINTERS 0xffffe00176721688 +0x000 DataSectionObject : 0xffffe0017610f4c0 Void
+0x008 SharedCacheMap : (null)
+0x010 ImageSectionObject : 0xffffe0019666c010 Void 0: kd> dt nt!_CONTROL_AREA 0xffffe0017610f4c0 FilePointer.Object
+0x040 FilePointer :
+0x000 Object : 0xffffe001767217cf Void 0: kd> dt nt!_FILE_OBJECT 0xffffe001767217c0 FileName.
+0x058 FileName : “\Windows\System32\drivers\DataNow_Dri”
+0x000 Length : 0x4a
+0x002 MaximumLength : 0x78
+0x008 Buffer : 0xffffc001`8a49adc0 “\Windows\System32\drivers\DataNow_Dri”

–
@AndreyBazhan
http://www.andreybazhan.com

Peter_Viscarola_OSR · August 4, 2016, 2:26pm

Seriously: Try to repo this problem with a driver that does nothing and that’s built in the *standard* WDK build environment.

If it fails then, you’ve got a Device Guard bug… You’re done. Go file it, and provide them the two copies of the failing driver and your repro steps.

If it succeeds, you could *still* have a Device Guard bug… but it’s less likely.

If you enable debugging on the target machine, and attach a kernel debugger, that *should* let you bypass the signing requirement. The first thing I’d so is try this, and see if the problem with your driver still occurs (the question is… does having the debugger enabled also disable Device Guard doing its thing). If it does, you’ll be able to try new scenarios much more easily. Note that I don’t hold out much hope for this, but you never know.

Note that, for the purposes of expediency, also register another (non-EV) Class 3 Code Signing certificate with SYSDEV, and use THAT to sign your submissions. It won’t get MSFT to sign any faster, but it’ll get you out of having to deal with your company’s administrivia around the EV Cert.

Peter
OSR
@OSRDrivers

Alex_Grig · August 4, 2016, 4:44pm

@Scott:

NTFS may have slightly different behavior than FAT. I’ll write a test repro.

Tim_Roberts · August 4, 2016, 7:35pm

xxxxx@osr.com wrote:

Note that, for the purposes of expediency, also register another (non-EV) Class 3 Code Signing certificate with SYSDEV, and use THAT to sign your submissions. It won’t get MSFT to sign any faster, but it’ll get you out of having to deal with your company’s administrivia around the EV Cert.

Forgive the dumb question, but does that work? I thought an EV cert was
required in order to establish a Sysdev account, and I thought the
package you submitted had to be signed by the cert you used to sign up.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Maxim_S_Shatskih · August 5, 2016, 7:59am

>still occurs (the question is… does having the debugger enabled also disable Device Guard doing

Debugger is incompatible with Secure Boot. Will it be compatible with Device Guard?

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Scott_Noone_OSR · August 5, 2016, 8:38am

The FileName field of the File Object is only guaranteed valid during
PreCreate processing, once you’re out of the context the content is
undefined. Do you know for certain that Device Guard uses this field for
anything?

-scott
OSR
@OSRDrivers

“Andrey Bazhan” wrote in message
news:xxxxx@ntdev…
You need to investigate how FILE_OBJECT FileName.Length field is getting
damaged.
Seems like this is the reason why Device Guard is blocking your driver.

0: kd> r
Last set context:
rax=0000000000000000 rbx=0000000000000000 rcx=ffffe001765671f0
rdx=ffffe00196625000 rsi=ffffe001765671f0 rdi=0000000000000000
rip=fffff8005c0263ac rsp=ffffd00020e91968 rbp=ffffd00020e91a70
r8=0400000000020020 r9=fffff801bfcb3c40 r10=00000000000001c8
r11=ffffd00020e91960 r12=ffffffff8000139c r13=0000000000000200
r14=ffffe00196625000 r15=0000000020206f49
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b
efl=00010246
DataNow_Driver!GsDriverEntry:
fffff8005c0263ac 48895c2408 mov qword ptr [rsp+8],rbx ss:0018:ffffd00020e91970=0000000000000000

0: kd> dt nt!_DRIVER_OBJECT ffffe001765671f0 DriverSection
+0x028 DriverSection : 0xffffe00176d2b8a0 Void 0: kd> dt nt!_KLDR_DATA_TABLE_ENTRY 0xffffe00176d2b8a0
+0x000 InLoadOrderLinks : _LIST_ENTRY [0xfffff801bfd5dcf0 - 0xffffe00176ebeba0]
+0x010 ExceptionTable : 0xfffff8005c023000 Void +0x018 ExceptionTableSize : 0x63c +0x020 GpValue : (null) +0x028 NonPagedDebugInfo : (null) +0x030 DllBase : 0xfffff8005c010000 Void
+0x038 EntryPoint : 0xfffff8005c0263ac Void +0x040 SizeOfImage : 0x19000 +0x048 FullDllName : _UNICODE_STRING "\SystemRoot\system32\DRIVERS\DataNow_Driver.sys" +0x058 BaseDllName : _UNICODE_STRING "DataNow_Driver.sys" +0x068 Flags : 0x49104020 +0x06c LoadCount : 1 +0x06e u1 : <unnamed-tag> +0x070 SectionPointer : 0xffffc0018b662fc0 Void
+0x078 CheckSum : 0x22ce9
+0x07c CoverageSectionSize : 0
+0x080 CoverageSection : (null)
+0x088 LoadedImports : 0xffffc00182b1a190 Void +0x090 Spare : (null) +0x098 SizeOfImageNotRounded : 0x19000 +0x09c TimeDateStamp : 0x579a15e7 0: kd> dt nt!_SECTION 0xffffc0018b662fc0 u1.ControlArea
+0x028 u1 :
+0x000 ControlArea : 0xffffe0019666c010 _CONTROL_AREA 0: kd> dt nt!_CONTROL_AREA 0xffffe0019666c010 FilePointer.Object
+0x040 FilePointer :
+0x000 Object : 0xffffe00176cfe996 Void 0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 FileName.
+0x058 FileName : “\Windows\System32\drivers\DataNow_Dri”
+0x000 Length : 0x4a
+0x002 MaximumLength : 0x78
+0x008 Buffer : 0xffffc001828d99a0 "\Windows\System32\drivers\DataNow_Dri" 0: kd> du /c 100 0xffffc001828d99a0
ffffc001828d99a0 "\Windows\System32\drivers\DataNow_Driver.sys" 0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 SectionObjectPointer
+0x028 SectionObjectPointer : 0xffffe00176721688 _SECTION_OBJECT_POINTERS 0: kd> dt nt!_SECTION_OBJECT_POINTERS 0xffffe00176721688
+0x000 DataSectionObject : 0xffffe0017610f4c0 Void +0x008 SharedCacheMap : (null) +0x010 ImageSectionObject : 0xffffe0019666c010 Void

0: kd> dt nt!_CONTROL_AREA 0xffffe0017610f4c0 FilePointer.Object +0x040 FilePointer : +0x000 Object : 0xffffe001767217cf Void

0: kd> dt nt!_FILE_OBJECT 0xffffe001767217c0 FileName. +0x058 FileName : "\Windows\System32\drivers\DataNow_Dri" +0x000 Length : 0x4a +0x002 MaximumLength : 0x78 +0x008 Buffer : 0xffffc0018a49adc0
“\Windows\System32\drivers\DataNow_Dri”

–
@AndreyBazhan
http://www.andreybazhan.com

Peter_Viscarola_OSR · August 5, 2016, 8:43am

Yes, that does work.

No, the package you submit does not *currently* need to be signed with the EV cert you used to sign up. The package JUST needs to be signed with *a cert* that the submitter has registered with SYSDEV.

Please be aware that while signing the submission with any registered cert, and not the EV cert, CURRENTLY works, the plan of record is that the EV cert *will* eventually be required. I don’t know if the date for when the EV cert will be required has been announced publicly – We are hoping the decision to require the submission to be signed with the EV cert will be reversed.

In fact, I think I’l start a separate thread on this topic. Thank you, Mr. Roberts.

Peter
OSR
@OSRDrivers

Alex_Grig · August 5, 2016, 11:18am

Can you print full dt for the relevant FILE_OBJECTs?

OSR_Community_User · August 5, 2016, 11:38pm

ASLR concerns have eased or ASLR have disappeared with Windows 10 (for user-mode at least)). I made the test, loading and unloading a binary ten times in a row. The binary is always mapped at the same VA. That is not the case on Windows 7.

In Windows 10, the stack cookie is randomized by the loader and not by the stub present in the binary. Because this stub can not use functions that are protected by a stack cookie, this stub does not provide enough entropy. In Windows 10, the stub raises a software exception (int 29h, code 6) if the cookie was not randomized by the loader. The older stub uses the rdtsc instruction (read time stamp counter), whose result is xored with the address of the cookie to provide some entropy, the two upper bytes of the cookie are then zeroed.

So, the stub used in Windows 10 bugchecks the machine if the stack cookie was not randomized by the loader while the old stub just reclaims a new one based on the rdtsc instruction.

Andrey_Bazhan · August 6, 2016, 5:18am

Thanks Scott! I didn’t know that. Just learned that
NTFS!NtfsUpdateCcbsForLcbMove doesn’t bother to update FILE_OBJECT
FileName.Length.

No, I don’t.

By the way, I got .cat, .inf and .sys files from the dump, and they are not
damaged.
Very interesting case!

–
@AndreyBazhan
http://www.andreybazhan.com

“Scott Noone” wrote in message news:xxxxx@ntdev…

The FileName field of the File Object is only guaranteed valid during
PreCreate processing, once you’re out of the context the content is
undefined. Do you know for certain that Device Guard uses this field for
anything?

-scott
OSR
@OSRDrivers

“Andrey Bazhan” wrote in message
news:xxxxx@ntdev…
You need to investigate how FILE_OBJECT FileName.Length field is getting
damaged.
Seems like this is the reason why Device Guard is blocking your driver.

0: kd> r
Last set context:
rax=0000000000000000 rbx=0000000000000000 rcx=ffffe001765671f0
rdx=ffffe00196625000 rsi=ffffe001765671f0 rdi=0000000000000000
rip=fffff8005c0263ac rsp=ffffd00020e91968 rbp=ffffd00020e91a70
r8=0400000000020020 r9=fffff801bfcb3c40 r10=00000000000001c8
r11=ffffd00020e91960 r12=ffffffff8000139c r13=0000000000000200
r14=ffffe00196625000 r15=0000000020206f49
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b
efl=00010246
DataNow_Driver!GsDriverEntry:
fffff8005c0263ac 48895c2408 mov qword ptr [rsp+8],rbx ss:0018:ffffd00020e91970=0000000000000000

0: kd> dt nt!_DRIVER_OBJECT ffffe001765671f0 DriverSection
+0x028 DriverSection : 0xffffe00176d2b8a0 Void 0: kd> dt nt!_KLDR_DATA_TABLE_ENTRY 0xffffe00176d2b8a0
+0x000 InLoadOrderLinks : _LIST_ENTRY [0xfffff801bfd5dcf0 - 0xffffe00176ebeba0]
+0x010 ExceptionTable : 0xfffff8005c023000 Void +0x018 ExceptionTableSize : 0x63c +0x020 GpValue : (null) +0x028 NonPagedDebugInfo : (null) +0x030 DllBase : 0xfffff8005c010000 Void
+0x038 EntryPoint : 0xfffff8005c0263ac Void +0x040 SizeOfImage : 0x19000 +0x048 FullDllName : _UNICODE_STRING "\SystemRoot\system32\DRIVERS\DataNow_Driver.sys" +0x058 BaseDllName : _UNICODE_STRING "DataNow_Driver.sys" +0x068 Flags : 0x49104020 +0x06c LoadCount : 1 +0x06e u1 : <unnamed-tag> +0x070 SectionPointer : 0xffffc0018b662fc0 Void
+0x078 CheckSum : 0x22ce9
+0x07c CoverageSectionSize : 0
+0x080 CoverageSection : (null)
+0x088 LoadedImports : 0xffffc00182b1a190 Void +0x090 Spare : (null) +0x098 SizeOfImageNotRounded : 0x19000 +0x09c TimeDateStamp : 0x579a15e7 0: kd> dt nt!_SECTION 0xffffc0018b662fc0 u1.ControlArea
+0x028 u1 :
+0x000 ControlArea : 0xffffe0019666c010 _CONTROL_AREA 0: kd> dt nt!_CONTROL_AREA 0xffffe0019666c010 FilePointer.Object
+0x040 FilePointer :
+0x000 Object : 0xffffe00176cfe996 Void 0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 FileName.
+0x058 FileName : “\Windows\System32\drivers\DataNow_Dri”
+0x000 Length : 0x4a
+0x002 MaximumLength : 0x78
+0x008 Buffer : 0xffffc001828d99a0 "\Windows\System32\drivers\DataNow_Dri" 0: kd> du /c 100 0xffffc001828d99a0
ffffc001828d99a0 "\Windows\System32\drivers\DataNow_Driver.sys" 0: kd> dt nt!_FILE_OBJECT 0xffffe00176cfe990 SectionObjectPointer
+0x028 SectionObjectPointer : 0xffffe00176721688 _SECTION_OBJECT_POINTERS 0: kd> dt nt!_SECTION_OBJECT_POINTERS 0xffffe00176721688
+0x000 DataSectionObject : 0xffffe0017610f4c0 Void +0x008 SharedCacheMap : (null) +0x010 ImageSectionObject : 0xffffe0019666c010 Void

0: kd> dt nt!_CONTROL_AREA 0xffffe0017610f4c0 FilePointer.Object +0x040 FilePointer : +0x000 Object : 0xffffe001767217cf Void

0: kd> dt nt!_FILE_OBJECT 0xffffe001767217c0 FileName. +0x058 FileName : "\Windows\System32\drivers\DataNow_Dri" +0x000 Length : 0x4a +0x002 MaximumLength : 0x78 +0x008 Buffer : 0xffffc0018a49adc0
“\Windows\System32\drivers\DataNow_Dri”

–
@AndreyBazhan
http://www.andreybazhan.com

Alex_Grig · August 6, 2016, 4:17pm

@Scott:

I wrote a sample program and tested on Win2012R2.

If CreateFileMapping is called with SEC_IMAGE or SEC_IMAGE_NO_EXECUTE flag, and a handle (with or without FILE_SHARE_DELETE) is closed before calling DeleteFile, DeleteFile will succeed.

DeleteFile will not succeed if the source file handle is opened with regular mapping flags (not IMAGE).

MapViewOfFile fails on a SEC_IMAGE(_NO_EXECUTE) mapping handle, whether the file is deleted or not. Looks like it needs some secret recipe to obtain a mapping and actually see what will page-in bring.

Alex_Grig · August 6, 2016, 4:26pm

>and a handle (with or without FILE_SHARE_DELETE) is closed before calling DeleteFile,

This means file handle is closed.

Alex_Grig · August 6, 2016, 6:26pm

My suspicion about the OP bug is that even though the handle to the driver file is closed, something keeps a reference to its section (for example, the driver was unloaded but there is still a live device object). The file was then deleted. When reload was requested, the loader found the section with the target name, but which refers to a deleted file.

Ben_Staniford · August 8, 2016, 10:20am

Peter:

If you enable debugging on the target machine, and attach a kernel debugger,
that *should* let you bypass the signing requirement.

I’ve made an attempt at this, but I don’t believe it’s possible. You can’t enable device guard unless you’ve enabled secure boot, and you can’t attach a kernel debugger unless you’ve disabled secure boot. I found this from Mr Ionescu in a previous post which seems to agree with what I’ve found:

“Additionally, Device Guard is typically only active on systems with Secure
Boot. On systems with Secure Boot, debugging is not permitted. Therefore, you
are either booting with a strange policy that allows debugging on SecureBoot
devices, in violation of the standard, or you are booting with Device Guard and
UEFI disabled, a strange non-default and meaningless combination, or your issue
was actually not related to Device Guard”

I’ve made 4 drivers which I’m still waiting for Microsoft to sign:

A standard boilerplate filter driver, created using the Visual Studio wizard.
The same as above but with a large static array inside create a separate larger version.
Our driver but built in a brand new project with no external linkage and all default options (except for the specification of Windows7 as the target version)
The same as above but with some extra bloat.

I’m hoping I can recreate the problem with at least one of these. Thanks for your advice. I was going to follow your advice to register an extra cert with sysdev too but I’ve managed to get my company to let me grab the security token whenever I need it now so I probably don’t need to. My only issue now is the incredibly slow pace at which Microsoft sign things… I’ve been on step 5/8 in sysdev for the last 6 hours… I’ve no idea why each step such as “extracting contents from cab file” seem to take hours.

Regards,

Ben…

Peter_Viscarola_OSR · August 8, 2016, 10:56pm

Pure guesswork on my part, but I bet it has to do with load on the servers. The new OS update just shipped me after all… When did my test a month ago, it took about an hour, IIRC.

Peter
OSR
@OSRDrivers

Tom_McDermott · August 9, 2016, 11:14pm

Guys, I just saw this thread today, sorry for the delay in chiming in.

We have the same type of issue here, i.e. BSOD in GsDriverEntry, similar dump. Triggered by turning on Device Guard HVCI. Occurs in 10586(Th2), but not in 10240(Th1) or the new 14393(Redstone Anniversary).

Our issue is still under investigation by Microsoft, but right now I believe the behavior is due to HVCI (running in the Secure kernel) rejecting the driver, by means of hypervisor removing the execute privilege through the SLAT. GsDriverEntry just happens to be the entry point of the driver.

Although we believe our driver not to be fully HVCI compatible at this point, it is not clear why it is not rejected in 10240 or 14393. Perhaps they have been tweaking the enforcement levels.

Ben_Staniford · August 10, 2016, 5:37am

Hi Tom,

Thanks for the reply.

We have the same type of issue here, i.e. BSOD in GsDriverEntry, similar dump.
Triggered by turning on Device Guard HVCI. Occurs in 10586(Th2), but not in
10240(Th1) or the new 14393(Redstone Anniversary). Our issue is still under investigation by Microsoft.

Funny you should mention that. But Microsoft just got back to me 30 minutes before I saw your post and informed me that they were looking at another instance of our issue and that it only affected the builds you mention. My guess is that you guys are the other customer who they’re working with.

They’ve advised us to test the driver with Redstone 1, which I’ve handed off to our QA guys, so I’ll keep you posted with what we find. I got the impression from what he said that Microsoft are leaning towards believing this is a bug with Device Guard (in that build).

Although we believe our driver not to be fully HVCI compatible at this point

Out of interest, what do you believe you are currently lacking?

Ben_Staniford · August 10, 2016, 5:52am

> Pure guesswork on my part, but I bet it has to do with load on the servers. The

new OS update just shipped me after all… When did my test a month ago, it
took about an hour, IIRC.

I’m told by our Microsoft contact that the process is partly manual (depending on submission type). We’ve been waiting more than 48 hours now for our beta build to be signed and our beta is somewhat at risk so our Microsoft contact has suggested we contact the sysdev team directly with our ID numbers… It sounds like they can investigate why individual cases are taking so long.