TAEF extremely slow when uninstalling drivers during deployment

anszom · June 26, 2024, 2:12pm

I'm pretty new to Windows driver development, so I'm not sure what should be expected of the WDK. I've installed a recent version of VS2022 and the WDK extension.
When deploying a driver on a test machine, the "Remove previous driver versions before deployment" step takes several minutes. The console is stuck at the following:

Test Authoring and Execution Framework v10.66k for x64

StartGroup: DriverTestTasks::_DriverRemoval
Driver Certificate File: package.cer
Driver Inf File: Something.inf
Driver Hardware ID: root\Something
Driver package GUID: x64
Import Driver To Store: 1
Uninstall Driver: 1
WDTF_DRIVER_PACKAGE       :  - SetPackageInfFileName(C:\Windows\inf\oem46.inf)
WDTF_TARGET               :  - GetInterface("DriverSetup")
WDTF_TARGET               :          Target: RUNNER

Meanwhile, the TE.ProcessHost.exe executable is busy, it seems to be querying something about every driver in the system, as shown by the stacktrace:

	ntdll.dll!NtDeviceIoControlFile()	Unknown
 	KernelBase.dll!DeviceIoControl()	Unknown
 	kernel32.dll!00007ffbc0bf27f1()	Unknown
 	drvstore.dll!CmGetObjectProperty()	Unknown
 	drvstore.dll!DriverDatabaseGetObjectProperty()	Unknown
 	drvstore.dll!EnumDriverPackageCallback(struct _DRIVERSTORE_CONTEXT *,enum _DRIVERSTORE_OBJECT_TYPE,unsigned short const *,__int64)	Unknown
 	drvstore.dll!DriverDatabaseEnumObjects()	Unknown
 	drvstore.dll!DriverStoreEnumW()	Unknown
 	WDTFDevnodeInfData.dll!DriverStoreDll::_DriverStoreEnumW(struct HDRIVERSTORE__ *,unsigned long,int (*)(struct HDRIVERSTORE__ *,unsigned short const *,struct _DRIVERSTORE_DRIVERPACKAGE_INFOW *,__int64),__int64)	Unknown
 	WDTFDevnodeInfData.dll!DriverStore::Refresh(void)	Unknown
 	WDTFDevnodeInfData.dll!DriverStore::GetPackageCount(unsigned long)	Unknown
 	WDTFDevnodeInfData.dll!CTsDriverStore::EnumReset(void)	Unknown
 	WDTFDevnodeInfData.dll!CTsDriverStore::Open(void)	Unknown
 	WDTFDevnodeInfData.dll!CDevnodeInfInfoDetail_RawData::Refresh(class CGDGNode *)	Unknown
 	WDTFDevnodeInfData.dll!CGDGFieldBase::GetData(class CGDGNode *,class CGDGSetFieldData *)	Unknown
 	WDTF.DLL!CSubSource::ResolveField(class IDataBlock &,unsigned long &)	Unknown
 	WDTF.DLL!CCachedBlockField<unsigned long>::GetValue(void)	Unknown
 	WDTF.DLL!CDataBlock::GetValue(unsigned short const *)	Unknown
 	WDTF.DLL!CDataBlock::GetValue(unsigned short const *)	Unknown
 	WDTF.DLL!CField::Eval(class IDataBlock &)	Unknown
 	WDTF.DLL!CSubcontext::Eval(class IDataBlock &)	Unknown
 	WDTF.DLL!CTargetsCore<class std::list<struct IWDTFTarget2 *,class std::allocator<struct IWDTFTarget2 *> >,class ATL::ICollectionOnSTLImpl<struct IWDTFTargets2,class std::list<struct IWDTFTarget2 *,class std::allocator<struct IWDTFTarget2 *> >,struct IWDTFTarget2 *,class _CopyInterfaceFromListItem<struct IWDTFTarget2>,class ATL::CComEnumOnSTL<struct IEnumVARIANT,&struct __s_GUID const _GUID_00020404_0000_0000_c000_000000000046,struct tagVARIANT,struct _CopyVariantFromListItem<struct IWDTFTarget2>,class std::list<struct IWDTFTarget2 *,class std::allocator<struct IWDTFTarget2 *> >,class ATL::CComMultiThreadModel> >,class CWDTFTarget2,struct IWDTFTarget2,class CWDTFTargets2,struct IWDTFTargets2,struct IWDTF2>::InternalQuery(unsigned short *,struct IWDTFTargets2 * *)	Unknown
 	WDTF.DLL!CWDTFTargets2::InternalQuery(unsigned short *,struct IWDTFTargets2 * *)	Unknown
 	WDTF.DLL!CWDTFTargets2::Query(unsigned short *,struct IWDTFTargets2 * *)	Unknown
 	WDTF.DLL!CWDTFDeviceDepot2::Query(unsigned short *,struct IWDTFTargets2 * *)	Unknown
 	DriverTestTasks.dll!00007ffba942d501()	Unknown
 	DriverTestTasks.dll!00007ffba942cecd()	Unknown
 	DriverTestTasks.dll!00007ffba9419f93()	Unknown
 	DriverTestTasks.dll!00007ffba9419f4f()	Unknown
 	DriverTestTasks.dll!00007ffba94196af()	Unknown
 	DriverTestTasks.dll!00007ffba94128ef()	Unknown
 	TE.Loaders.dll!00007ffb8d5669a9()	Unknown
 	TE.Host.dll!`WEX::TestExecution::CommonTestHost::BeginInvokeImpl'::`2'::<lambda_1>::operator()()	Unknown
 	TE.Common.dll!TAEF::Common::ExceptionBoundary<`WEX::TestExecution::CommandThread::ExecuteCommandThread'::`7'::<lambda_1>>()	Unknown
 	TE.Common.dll!WEX::TestExecution::CommandThread::ExecuteCommandThread(class std::optional<class WEX::Logging::LogSession>)	Unknown
 	TE.Common.dll!TAEF::Common::Details::ThreadData<class std::_Mem_fn<long ( WEX::TestExecution::CommandThread::*)(class std::optional<class WEX::Logging::LogSession>) noexcept>,class WEX::TestExecution::CommandThread *,class std::optional<class WEX::Logging::LogSession> >::RunThread(class TAEF::Common::Details::ThreadData<class std::_Mem_fn<long ( WEX::TestExecution::CommandThread::*)(class std::optional<class WEX::Logging::LogSession>) noexcept>,class WEX::TestExecution::CommandThread *,class std::optional<class WEX::Logging::LogSession> > *)	Unknown
 	TE.Common.dll!`TAEF::Common::Thread::Thread<class std::_Mem_fn<long ( WEX::TestExecution::CommandThread::*)(class std::optional<class WEX::Logging::LogSession>) noexcept>,class WEX::TestExecution::CommandThread *,class std::optional<class WEX::Logging::LogSession> >(struct TAEF::Common::ThreadSettings &&,class std::_Mem_fn<long ( WEX::TestExecution::CommandThread::*)(class std::optional<class WEX::Logging::LogSession>) noexcept> &&,class WEX::TestExecution::CommandThread * &&,class std::optional<class WEX::Logging::LogSession> &&)'::`2'::<lambda_1>::operator()(void *)	Unknown
 	kernel32.dll!00007ffbc0c0257d()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

So far I've been coping by manually running a script which removes leftovers (in seconds, not minutes!), but it's a bit tedious and error-prone. Is this normal? Is there a better way?

Mark_Roddy · June 26, 2024, 4:26pm

Yes. First don’t use TAEF and don’t use VS to deploy drivers.

Pnputil.exe is on every supported system and can both install and remove your driver. It is trivial to wrap its usage in a script.

Use Windbg for debugging.

In development test/debug mode you generally just need to replace the driver binary after a bug fix. Windbg can be set up to load your latest built .sys file instead of the installed image when your driver is loaded by the system. No need to install/uninstall anything.

anszom · June 27, 2024, 7:47am

Thanks! This works MUCH better.

Peter_Viscarola_OSR · June 27, 2024, 2:42pm

We have to give this guidance to just-about everyone who comes here after reading trying to learn how to write/test drivers just the WDK docs.

It's the only way.

BTW, if you haven't tried it yet, WinDbg makes it very convenient to cause new versions of your driver to be loaded.

See Updating Drivers with KDFiles – OSR... but, in short, at the WinDbg command line, use a command like:

kd> .kdfiles -m mydriver.sys F:\MyDriverProject\x64\debug\mydriver.sys

where "mydriver.sys" is your executable, and the rest is the path (on your dev box) to your driver binary.

Now, every time to disable/enable your driver (via Device Manager or pnputil) WinDbg will cause the latest version of your driver binary to be pulled-across the debug connection and used on your test system.

This is what Mr. Roddy was alluding to, above, when he said "Windbg can be set up to load your latest built .sys file"

Once you've tried .kdfiles, you'll fall in love with it. Guaranteed.

Zac_Lockard · June 27, 2024, 6:35pm

That's really weird, it should be really fast to do this. I will look into what's happening. Is there anything else horrible in this WDK-based scenario? Happy to relay any feedback

Peter_Viscarola_OSR · June 27, 2024, 7:13pm

Hi Zac! Not sure what you're asking. We've been saying for years that:

Using the WDK and VS to deploy your driver to a target machine has been so error-prone that it's simply not even worth trying. Worse, it's basically impossible to debug when something goes wrong. Why automate something that's so simple to do (using .kdfiles or even by simply copying your driver binary to the target machine manually)? When somebody is new to learning Windows drivers, having to deal with a fussy process is the last thing they need, especially when the manual alternative is so easy and predictable.
Using the debugger within VS for driver debugging is not the best experience. The whole work cycle of driver debugging within VS just doesn't make a lot of sense: You can't edit your code while the debugger is running, and to make the debugger STOP running you need to DISCONNECT from the target machine. WinDbg is designed for kernel debugging. It's extremely rich. And, again, using dbgeng and debugging your driver from within VS provides no actual benefit.

Our views on this have been made very clear, on numerous occasions to anyone who'll listen, including the WDK team (who on multiple occasions have been very generous with their time in meeting with us on this topic). We've had the privilege of discussing this over the years with numerous generations of PMs, Leads, and devs of the individual features.

If somebody JUST wants to get their driver though the build, deploy, and run the driver under the debugger cycle... multiple times every day... the basic tools off the best and most expedient path. This seems to be the clear consensus of the community. Trying to "add value" to this process by providing VS integration just makes things more complex and more difficult.

anszom · June 27, 2024, 7:48pm

As a beginner I'll drop in my 0.03.

Configuring the test machine for deployment, and the deployment itself is a complex, opaque and brittle process. When it works correctly, it's quite impressive, albeit slow. On the other hand when it fails, it produces a next-to-useless error message.
I've been working with the WDK for several weeks so my experience is limited, but on several occasions I've seen random deployment failures which go away by just trying again.
And the cycle times - this is a major productivity killer. Why would the deployment process take several minutes (or at least tens of seconds, with "driver removal" disabled) if we have an alternative that works nearly instantly (kdfiles)?

VS integration is buggy too:

During my short experience, I've had devenv crash several times while debugging.
The debugger doesn't indicate the current line when stepping - at least not in the usual way. There is some kind of a "focused" marker around the line, but not like the regular debugger
Stepping is very slow compared to windbg
Breakpoint markers (red circle) only sometimes appear on the margin
You can't read the debugger output after detaching from the target, the "immediate window" disappears. Thankfully it can be brought back by reattaching, but this is not really convenient.
Breakpoints set before launching are not applied. I've needed to break into the debugger and issue ".reload" after the driver is loaded to convince the debugger to place the breakpoints.

That is true
Thank you @Mark_Roddy & @Peter_Viscarola_OSR

Zac_Lockard · June 28, 2024, 2:42am

You answered my question - that's what I was asking

I appreciate the feedback