Perplexing Build/Environment Issue

NTDEV
1

I am having a build/environment issue that I cannot seem to correct. I’d
appreciate any ideas/solutions

symptoms:
I can perform several scripted builds succesfully. At some point (after
5-7) builds the build script starts failing with errors like
‘can’t start tool’
‘Not enough quota is available to process this command.’
‘Error spawning ‘vcspawn.exe’. The build could not be performed.’
or vs98 can’t find include files, source files, etc. that exist

It takes a reboot to correct this issue. If I don’t reboot, at some time
in the future my profile gets corrupted. Logouts do not resolve the issue

Environment:
w2k sp1
vs98 w/sp4 applied
november platform sdk
w2k ‘sp1’ ddk
nt4ddk

The script is sh file using the MS windows unix utilities. The sh script
sets the environment and calls a batch file that consists of calls to msdev
with unique dsp files. There are about 12 dsps building usermode apps and,
through batch calls, 4 drivers.

I’ve rebuilt the build system with no change in behavior
I’ve modified the dddk\setenv and removed the vccheck calls, as this tends
to cause the problem to occur more rapidly.
The problem occurs if I just use the .bat file (removing MS Unix Utils from
the equation)

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

2

Do I understand you correctly, that you are calling Msdev.exe with dsp files
as arguments? I’ve never tried that, but I wouldn’t recommend doing that as
a way to do command line builds. Since you have the DDKs installed, it
appears that you are building device drivers? If so, why are you doing
things differently than the DDK provides? Use the build environments, and
use build.exe. BTW, you can also use build for any apps, as well.

All that said, from the information you are providing, it sounds like
something is making cumulative edits to your global environment, most likely
the path. Eventually, your path (or whatever environment variable is
causing the problem) will exceed the maximum allowable path length, and you
will have a problem. Granted, other failure mechanisms may cause the same
symptoms, but I am not aware of any. To test it, run your build several
times, then start a new command prompt, and look at your environment. If
you find an environment variable with a large number of the same entry,
you’ve confirmed my suspicion. Otherwise, you have a different problem, and
I don’t have any other suspects, without more information.

Phil
* Philip D. Barila | (503) 264-8386
* Intel Corp. | M/S JF2-53 Office JF2-2-G6
* Storage Architecture and Performance
* Internet Systems Lab

-----Original Message-----
From: xxxxx@emc.com [mailto:xxxxx@emc.com]
Sent: Tuesday, March 20, 2001 1:53 AM
To: NT Developers Interest List
Subject: [ntdev] Perplexing Build/Environment Issue

I am having a build/environment issue that I cannot seem to correct. I’d
appreciate any ideas/solutions

symptoms:
I can perform several scripted builds succesfully. At some point (after
5-7) builds the build script starts failing with errors like
‘can’t start tool’
‘Not enough quota is available to process this command.’
‘Error spawning ‘vcspawn.exe’. The build could not be performed.’
or vs98 can’t find include files, source files, etc. that exist

It takes a reboot to correct this issue. If I don’t reboot, at some time
in the future my profile gets corrupted. Logouts do not resolve the issue

Environment:
w2k sp1
vs98 w/sp4 applied
november platform sdk
w2k ‘sp1’ ddk
nt4ddk

The script is sh file using the MS windows unix utilities. The sh script
sets the environment and calls a batch file that consists of calls to msdev
with unique dsp files. There are about 12 dsps building usermode apps and,
through batch calls, 4 drivers.

I’ve rebuilt the build system with no change in behavior
I’ve modified the dddk\setenv and removed the vccheck calls, as this tends
to cause the problem to occur more rapidly.
The problem occurs if I just use the .bat file (removing MS Unix Utils from
the equation)

You are currently subscribed to ntdev as: xxxxx@intel.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

3

> ----------

From: Barila, Phil[SMTP:xxxxx@intel.com]
Reply To: NT Developers Interest List
Sent: Tuesday, March 20, 2001 5:54 PM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

Do I understand you correctly, that you are calling Msdev.exe with dsp
files
as arguments? I’ve never tried that, but I wouldn’t recommend doing that
as
a way to do command line builds.

Just a note: we also use this way because it is the easiest way how to
create automatic build environment for VS usermode projects. Of course,
people write and debug apps in VS but we needed an automatic build engine
for complete software suite including usermode apps, NT/w2k kernel drivers,
w9x VxDs etc. It is possible to use build.exe for this purpose but in
reality it is nightmare. It is too clumsy and one have to study makefile.def
to find all necessary undocumented variables. Instead, we use scripts which
start appropriate build engine for every piece of software – msdev for
apps, build for drivers, nmake for vxds etc.

Best regards,

Michal Vodicka
Veridicom
(RKK - Skytale)
[WWW: http://www.veridicom.com , http://www.skytale.com]

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

4

> w9x VxDs etc. It is possible to use build.exe for this purpose but in

reality it is nightmare.

Looks like using .DSP to build a driver sometimes introduces even more
problems then studying MAKEFILE.DEF
When one asks - “I’m building a driver via .DSP and having problems with
it” - then the “return back to SOURCES file” recommendation seems to be OK.

SOURCES file is not broken - no need to fix it.

Max

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

5

Phil,

[snip]

Do I understand you correctly, that you are calling Msdev.exe
with dsp files
as arguments? I’ve never tried that, but I wouldn’t
recommend doing that as
a way to do command line builds. Since you have the DDKs
installed, it
appears that you are building device drivers? If so, why are
you doing
things differently than the DDK provides? Use the build
environments, and
use build.exe. BTW, you can also use build for any apps, as well.

We use msdev as a automation mechanism. For the user apps in our
package we spawn with a true vs98 project. For the drivers, a
custom build action is trigger that calls ‘build’ in the correct
directory for the driver in question.

All that said, from the information you are providing, it sounds like
something is making cumulative edits to your global
environment, most likely
the path. Eventually, your path (or whatever environment variable is
causing the problem) will exceed the maximum allowable path
length, and you
will have a problem. Granted, other failure mechanisms may
cause the same
symptoms, but I am not aware of any. To test it, run your
[snip]

This is what I thought at first as well. I then modified our
top level script to use the setlocal/endlocal directives. This
restores the command window’s environment back to its original
state after performing a build.

It seems like msdev may have a resource leak when fired from the
command line. Unfortunately (or not:) ) I don’t have the time
to track this down right now. I was hoping someone had
encountered this issue and already solved it.

–
Dan

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

6

I do ‘nightly builds’ on a moderate sized development tree using a similar
approach, and I have not run into your problem. I don’t think that there is
anything intrinsic in visual studio or ddk build that is causing this
problem, so it is either your batch scripts that run build or it is the way
your are running visual studio that is killing your system. It of course
sounds like some application is not terminating, and the resources it is
using with each iteration ‘pile up’ until they exceed some limit that causes
the subsequent failure.

Mark Roddy
xxxxx@hollistech.com
www.hollistech.com
603 321 1032
WindowsNT Windows 2000 Consulting Services

-----Original Message-----
From: Vodicka, Michal [mailto:xxxxx@rkk.cz]
Sent: Wednesday, March 21, 2001 12:19 AM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

From: Barila, Phil[SMTP:xxxxx@intel.com]
Reply To: NT Developers Interest List
Sent: Tuesday, March 20, 2001 5:54 PM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

Do I understand you correctly, that you are calling Msdev.exe with dsp
files
as arguments? I’ve never tried that, but I wouldn’t recommend doing that
as
a way to do command line builds.

Just a note: we also use this way because it is the easiest way how to
create automatic build environment for VS usermode projects. Of course,
people write and debug apps in VS but we needed an automatic build engine
for complete software suite including usermode apps, NT/w2k kernel drivers,
w9x VxDs etc. It is possible to use build.exe for this purpose but in
reality it is nightmare. It is too clumsy and one have to study makefile.def
to find all necessary undocumented variables. Instead, we use scripts which
start appropriate build engine for every piece of software – msdev for
apps, build for drivers, nmake for vxds etc.

Best regards,

Michal Vodicka
Veridicom
(RKK - Skytale)
[WWW: http://www.veridicom.com , http://www.skytale.com]

You are currently subscribed to ntdev as: xxxxx@stratus.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

7

Max,

I’ve been using VS for driver development for 5 years. Since Walter Oney
published that wizard, I don’t even bother with BUILD and SOURCES any more.
The IDE provides a lot of features that just simply make life easier in
driver development. Symbol lookup, variable/function references, variable
function definitions, the browser, etc are just some of the things I use all
day every day. The ability to easily switch between checked and free is
another.

This is a religious issue though, and I’m liberal as opposed to a
traditionalist.

As some say here … just me $.02.

Gary

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Maxim S. Shatskih
Sent: Wednesday, March 21, 2001 4:01 AM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

w9x VxDs etc. It is possible to use build.exe for this purpose but in
reality it is nightmare.

Looks like using .DSP to build a driver sometimes introduces even more
problems then studying MAKEFILE.DEF
When one asks - “I’m building a driver via .DSP and having problems with
it” - then the “return back to SOURCES file” recommendation seems to be OK.

SOURCES file is not broken - no need to fix it.

Max

You are currently subscribed to ntdev as: xxxxx@delphieng.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

8

I use VS for driver development as well, but I use a modified version of
Mark’s excellent ddkbuild.bat. I get everything you get, and I am
absolutely sure that someone else can take my source tree unmodified and
build the driver in their own environment. The real point is that,
regardless of how you, me, or the cows out in Tillamook feel about it, the
DDK is assembled to produce binaries in a certain way. I have fought with
the VS settings to create the same thing, once, and won’t do it again.
Mark’s ddkbuild just works, and it works right, and gives me access to all
the features of the IDE. Sure, I want a completely transparent DD creation
and build in the VS IDE, but that’s not there in the wizard, either. I want
a lot, though! I want the IDE to front end for the 'bag, too, so I can do
it all in one spot. It’s not there yet, and may never be, but it’s
certainly closer than it used to be.

Phil

-----Original Message-----
From: Gary G. Little [mailto:xxxxx@inland.net]
Sent: Wednesday, March 21, 2001 7:15 AM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

Max,

I’ve been using VS for driver development for 5 years. Since Walter Oney
published that wizard, I don’t even bother with BUILD and SOURCES any more.
The IDE provides a lot of features that just simply make life easier in
driver development. Symbol lookup, variable/function references, variable
function definitions, the browser, etc are just some of the things I use all
day every day. The ability to easily switch between checked and free is
another.

This is a religious issue though, and I’m liberal as opposed to a
traditionalist.

As some say here … just me $.02.

Gary

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Maxim S. Shatskih
Sent: Wednesday, March 21, 2001 4:01 AM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

w9x VxDs etc. It is possible to use build.exe for this purpose but in
reality it is nightmare.

Looks like using .DSP to build a driver sometimes introduces even more
problems then studying MAKEFILE.DEF
When one asks - “I’m building a driver via .DSP and having problems with
it” - then the “return back to SOURCES file” recommendation seems to be OK.

SOURCES file is not broken - no need to fix it.

Max

You are currently subscribed to ntdev as: xxxxx@delphieng.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@intel.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

9

> ----------

From: green, daniel[SMTP:xxxxx@emc.com]
Reply To: NT Developers Interest List
Sent: Wednesday, March 21, 2001 2:03 PM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

It seems like msdev may have a resource leak when fired from the
command line. Unfortunately (or not:) ) I don’t have the time
to track this down right now. I was hoping someone had
encountered this issue and already solved it.

It seems we use something similar as you (sh scripts with cygwin utilities)
but never encountered this problem. We spawn msdev for tens project in one
build and even repeated builds don’t cause problems. From your original
description I doubt it is resource leak in msdev, it would be really huge
one. Try to check if there aren’t some msdev instances or other processes
which ‘forget’ to exit…

Best regards,

Michal Vodicka
Veridicom
(RKK - Skytale)
[WWW: http://www.veridicom.com , http://www.skytale.com]

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

10

Max,

you quoted my sentence and gave it different meaning :slight_smile: I assume you wanted
to express below opinion which I quite agree but anyway, I said we use msdev
for building apps and build utility for drivers. I wanted to say that using
build for building whole tree containing apps, drivers and other software is
possible but really complicated and there are more disadvantages than
benefits. Build is good for drivers, simple DLLs, command line apps and
combinations of mentioned. If I ever use msdev (why if there are much better
programmers editors?) for driver development, I would use the same approach
as Mark Roddy i.e. call build via batch from msdev project.

Best regards,

Michal Vodicka
Veridicom
(RKK - Skytale)
[WWW: http://www.veridicom.com , http://www.skytale.com]

From: Maxim S. Shatskih[SMTP:xxxxx@storagecraft.com]
Reply To: NT Developers Interest List
Sent: Wednesday, March 21, 2001 1:00 PM
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

> w9x VxDs etc. It is possible to use build.exe for this purpose but in
> reality it is nightmare.

Looks like using .DSP to build a driver sometimes introduces even more
problems then studying MAKEFILE.DEF
When one asks - “I’m building a driver via .DSP and having problems with
it” - then the “return back to SOURCES file” recommendation seems to be
OK.

SOURCES file is not broken - no need to fix it.

Max

You are currently subscribed to ntdev as: xxxxx@rkk.cz
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

11

A resource leak in msdev would not require a reboot to fix. Assuming
you looked for runaway processes, and you are not running out of swap space,
and you are not way undersized on RAM, this is a bug in OS code.

Any other little things, such as ClearCase, installed on the machine?
I’ve seen older versions of ClearCase leak memory when running with msdev, occasionally.

-DH

> ----------
> From: green, daniel[SMTP:xxxxx@emc.com]
> Reply To: NT Developers Interest List
> Sent: Wednesday, March 21, 2001 2:03 PM
> To: NT Developers Interest List
> Subject: [ntdev] RE: Perplexing Build/Environment Issue
>
> It seems like msdev may have a resource leak when fired from the
> command line. Unfortunately (or not:) ) I don’t have the time
> to track this down right now. I was hoping someone had
> encountered this issue and already solved it.
>
It seems we use something similar as you (sh scripts with cygwin utilities)
but never encountered this problem. We spawn msdev for tens project in one
build and even repeated builds don’t cause problems. From your original
description I doubt it is resource leak in msdev, it would be really huge
one. Try to check if there aren’t some msdev instances or other processes
which ‘forget’ to exit…

Best regards,

Michal Vodicka
Veridicom
(RKK - Skytale)
[WWW: http://www.veridicom.com , http://www.skytale.com]

You are currently subscribed to ntdev as: xxxxx@syssoftsol.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

12

For those of you following my build trials and tribulations…

My machine died again yesterday evening so I decided to be a little more
proactive in tracking this issue down. Using boot.ini I load a checked
kernel and hal. I then booted and started dbgview to catch kernel debug
messages. I also started a performance counter log tracking commit limit,
commit bytes, total system handles, total system processes and total system
threads. I then started the builds.

On reboot:
wuser32.exe crash do to unknown software exception (only occurs in
chk build)

Build 1.
lots of ‘RTL: Ignoring attempt to delete a pinned atom (c009)’
a couple of ‘EXWORKER: Created dynamic thread type 0, 0 total’
a ‘[580] A BuildEvent was created!’

Build 2.
same, but ‘[1152] A BuildEvent was created!’

Build 3.
same as 2

Build 4.
failed while environment being set up
cvs issued socket errors
lots of ‘EX: ExAllocatePool (0000C5EC, 0x1 ) returning NULL’
the size of the failure is always less that 65k.

Build 5.
same as 4

After the pool allocations started failing system becomes unusable and never
recovers (everything starts failing, new processi cant start). I can do a
shutdown/restart.

Viewing the perf log, almost all resources seem to be released (handles,
threads, processes return to steady state between builds).
However the ‘commit bytes’ continue to grow and bytes are never released.
The problem seems to be paged pool allocations fail once some internal state
has been reached.

I looked in Inside w2k for a description of ‘commit’ but didn’t find much.

Anybody got a good description of the ‘commit’ process and/or anymore ideas?

Thanks,
Dan

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

13

Hi Daniel,
Some Suggestions
1: Check al the cards in the PC are correctly seated in there sockets.
2: Have you tried removing and reinserting the memory chips on the
motherboard ?
3: Verify the power connectors are seated.

4: Can you connect another HD to your PC with a different Os / or another
copy of NT to verify it is not a failing HD problem.

Robert Fernando
Anite Telecoms Ltd
110 Fleet Road
Fleet
Hampshire GU51 4BL
United Kingdom
Tel: +44 (0) 1252 775200
Fax: +44 (0) 1252 775 321
Email: xxxxx@anitetelecoms.com

Anite Telecoms Limited Registered in England No. 1721900 Registered
Office: 100 Longwater Avenue, GreenPark, Reading, Berkshire RG2 6GP,
United Kingdom

-----Original Message-----
From: green, daniel [mailto:xxxxx@emc.com]
Sent: 23 March 2001 12:41
To: NT Developers Interest List
Subject: [ntdev] RE: Perplexing Build/Environment Issue

For those of you following my build trials and tribulations…

My machine died again yesterday evening so I decided to be a little more
proactive in tracking this issue down. Using boot.ini I load a checked
kernel and hal. I then booted and started dbgview to catch kernel debug
messages. I also started a performance counter log tracking commit limit,
commit bytes, total system handles, total system processes and total system
threads. I then started the builds.

On reboot:
wuser32.exe crash do to unknown software exception (only occurs in
chk build)

Build 1.
lots of ‘RTL: Ignoring attempt to delete a pinned atom (c009)’
a couple of ‘EXWORKER: Created dynamic thread type 0, 0 total’
a ‘[580] A BuildEvent was created!’

Build 2.
same, but ‘[1152] A BuildEvent was created!’

Build 3.
same as 2

Build 4.
failed while environment being set up
cvs issued socket errors
lots of ‘EX: ExAllocatePool (0000C5EC, 0x1 ) returning NULL’
the size of the failure is always less that 65k.

Build 5.
same as 4

After the pool allocations started failing system becomes unusable and never
recovers (everything starts failing, new processi cant start). I can do a
shutdown/restart.

Viewing the perf log, almost all resources seem to be released (handles,
threads, processes return to steady state between builds).
However the ‘commit bytes’ continue to grow and bytes are never released.
The problem seems to be paged pool allocations fail once some internal state
has been reached.

I looked in Inside w2k for a description of ‘commit’ but didn’t find much.

Anybody got a good description of the ‘commit’ process and/or anymore ideas?

Thanks,
Dan

You are currently subscribed to ntdev as: xxxxx@anitetelecoms.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com