Beware Of Deploying Debug Code In Production

By Milan Negovan

Print

Take a peek inside your web.config. Are you telling the framework to compile debug or release code? Does it have to stumble across and compile one page at a time, or compile them all in one pass? It's all in the <compilation> tag. Let's take a closer look at it.

The compilation element has many attributes. The one we are going to examine is debug. The attribute may take one of the two values:

• true to have the framework produce debug binaries
• false to produce release (retail) binaries

Let's dig in. ASP.NET maintains a directory where it stores compiled code, be it debug or release code. The debug switch determines the manner in which it compiles code. In the .NET 1.1 installation you can find this folder in \Windows\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files. ASP.NET creates a folder for each web application root in there.

For example, I created a sample project for this article and named it DebugTest. As promised ASP.NET created a folder called debugtest for me.

The sample project, TestDebug, has two pages named test1.aspx and test2.aspx. Note that every page is actually a class. The project itself compiles executable code into a dll, TestDebug.dll. Quite naturally, we expect to see both test1 and test2 classes inside this DLL.

Let's expand the tree of folders with cryptic names and locate our assembly. The last folder in this food chain has two files: TestDebug.dll and __AssemblyInfo__.ini. The ini file is a simple one:

[AssemblyInfo]
MVID=69266042ccf63445be7d9c996d9658aa
URL=file:///c:/inetpub/wwwroot/DebugTest/bin/DebugTest.DLL
DisplayName=DebugTest, Version=1.0.1584.40802, Culture=neutral, «
     PublicKeyToken=null

Now let's peek inside the DLL itself. If you own a copy of a disassembler, such as ILDASM or Reflector, fire it up and open the assembly.

Bingo! Test1 and test2 are there just like we anticipated! Also, it's interesting to see that globals.asax got compiled into its own class! Makes you wonder if you really need it and whether you have use for all those event hooks to the Session and Application objects.

When you create a brand new web project in Visual Studio.NET your web.config contains the following compilation tag by default:

<compilation defaultLanguage="c#" debug="true" />

Let's see now how the settings of the debug attribute affect output.

debug="false"

When the debug attribute is set to false ASP.NET conducts a batch compilation, i.e. all files in the current folder are compiled into one DLL.

Now, pay close attention to this: when an ASP.NET file is requested for the first time the framework conducts a batch compilation on that directory. It compiles files in that directory only into a DLL. It does not traverse subfolders. You need to hit a page in every folder to have each folder batch compiled. This is an important point.

For example, these are the files I end up with under my debugtest directory:

The xml files correspond to pages in the project. Each xml file indicates where a page was compiled and lists where the original page resides to track changes:

<preserve assem="qu_mr0sg" type="_ASP.test1_aspx" «
hash="ff19a32b35478349" batch="true">
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\test1.aspx" />
</preserve>

You will have as many xml files as there are pages in your project. Files with the .web extension are used by ASP.NET to manage file updates (this is to say I don't know how exactly they are used but ASP.NET keeps tabs on them for its own reasons).

You might've noticed there are two DLLs in this folder. Why so? Remember my statement about batch compilations? Well, both test1 and test2 were compiled into one assembly while globals.asax got its own assembly. I could have removed it from the project to keep this exercise cleaner. I decided to keep it and emphasize that globals.asax is an important part of every project and you should think twice if you really need it. So far it's been getting in the way a lot.

So now we know this much: all pages from the current folder are compiled into one DLL and each and every page has an xml file "assigned" to it. Let's make a simple change to, say, test1.aspx. Anything. Now run the page again and look what happens:

We have a new DLL! And... the xml file for test1.aspx now points to it:

<preserve assem="ugtscdas" type="ASP.test1_aspx" «
hash="e65cd41d34f3d4">
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\test1.aspx" />
</preserve>

Wait a second! The xml file for test2.aspx still points to the old DLL:

<preserve assem="qu_mr0sg" type="_ASP.test2_aspx" « 
hash="ff19a36e8f347176" batch="true">
   <filedep name="c:\inetpub\wwwroot\DebugTest\test2.aspx" />
   <filedep name="c:\inetpub\wwwroot\DebugTest\bin\DebugTest.DLL" />
</preserve>

We simply changed test1.aspx and now we have two assemblies? That's right! Fire up your favorite disassembler and load both the old DLL and the new one.

The DLL produced as the result of a minor change to test1.aspx led to compilation of that one page only into a separate DLL (it's the second one down, called ugtscdas.dll in this particular case)! The old DLL will be used to service current requests only.

Let's make another change to test1.aspx. Anything at all. Now run it. Yet another assembly! I think you get the point by now‚you keep changing a page and ASP.NET will keep recompiling it to have the latest code running. It won't bother recompiling everything. What does this tell you? If certain pages change a lot, move them out to a separate folder if possible to spare the rest of the code from constant recompilations!

If you keep changing test1.aspx and requesting it in your browser again and again you will see an interesting thing: in the temporary directories you'll have files with the .delete extension. These are the outdated DLLs, the ones replaced upon recompilations.

Basically, these DLLs are doomed to be purged. Go ahead and try deleting one. You can't because it's locked. The DLLs are loaded into an AppDomain. To evict a DLL from an AppDomain either change something in web.config or global.asax, or restart the application. Then start your application by requesting a page and take a look at your temporary directory. All .delete files should be gone.

Maximum Number Of Recompiles

We've talked about changing a page and causing the framework to recompile it. How much of this abuse can ASP.NET take before restarting the whole application and starting from scratch? The threshold is configured in machine.config. Go to your \Windows\Microsoft.NET\Framework\v1.1.4322\Config folder, load machine.config and find the <compilation> tag.

Do not change anything in machine.config. Changes to this file will affect all web applications on your computer. Do so only if you know what you are doing.

The attribute we're after is numRecompilesBeforeApprestart. MSDN states its purpose as follows:

Indicates the number of dynamic recompiles of resources that can occur before the application restarts. This attribute is supported at the global and application level but not at the directory level.

The default for this attribute is 15 recompilations. Once the limit is reached the AppDomain unloads (more on this in a minute), the application restarts upon a subsequent page request and old DLLs are finally deleted.

So there you have it. Excessive recompilations will cause your web application to restart often.

Maximum Number Of Pages Per Batch Compilation

Along the same lines, the maximum number of files from the same folder that can be compiled into a DLL during batch compilation is 1000. This default is configured right there, in the maxBatchFileSize attribute of the <compilation> element, too. According to MSDN maxBatchFileSize specifies the maximum number of pages per batched compilation.

Why set this limit? Batch compilation happens once you reach a web application for the first time. Since having a user wait is not a good idea there needs to be a sane limit to how long we can have the user's attention before they decide to bail. Also, the size of the DLL might become unacceptably large. Therefore the limit is imposed.

Unloading AppDomain To Release DLLs

In .NET you can load a DLL but can't unload. I don't know if it's a good thing or a bad thing. In this case it's a bad thing. Here's why: every time a page is recompiled a new DLL is loaded into the AppDomain. The smallest unit that you can unload in .NET is an AppDomain. This means we have to unload the entire AppDomian to release old DLLs and have them scavenged. I can think of a couple of options to make this happen:

1. Modify globals.asax or web.config. This causes a batch compilation. A fresh start, so to say.
2. Wait until numRecompilesBeforeApprestart compilations happen (15 by default).
3. Run iisreset or reboot.
4. ASP.NET worker process takes up too much memory and the framework restarts it. This is pretty much outside of your control.

debug="true"

All this time we were assuming the debug attribute of the <compilation> element was set to false, i.e. no debug information was being generated. Now, on with the fun part—set it to "true" and observe what happens.

The temporary folder contains a lot more files this time around. Batch compilation doesn't take place! To the contrary, every page is compiled into its own assembly! This is a very important point. More so, every page comes with a bunch of other files: debug symbol file, compiler command line file, compiler output file, etc. I'll reiterate: every page will have to be compiled individually into a separate assembly. Do I have to say this alone may kill performance of your web app when deployed live?

Note that you also get source code for free! This is what the compiler produces with the help of CodeDOM and compiles as DLLs. Except that in this case it leaves the source code and other byproducts of compilation on the disk.

These days you can't even watch on TV without being warned of yet another computer worm that is spreading fast. "Security" has become a very popular buzzword. In light of this I will just point out that exposing your source code in this manner is not a good idea. Nobody will pad you on the back for this. Just the opposite—you may get in trouble. Not that you exposed the source code in plain and clear as there is a lot of vagueness in the produced code, but it comes close.

Before we wrap up I wanted to share insight into a couple of problems you might encounter when dealing with page compilations, batch compiles, etc. I assume you've installed .NET Framework 1.1. A lot of bugs have been fixed since 1.0 and I won't address them here.

BUG: XCOPY Deployment Enlarges Temporary ASP.NET Folder Size (Q310450)

I've already mentioned that in the latest version of .NET, which is ver 1.1 as of now, you cannot unload individual DLLs from an AppDomain. As you modify pages and cause their recompilation new DLLs stay loaded until you do one of the following:

• Modify web.config or machine.cofig.
• Modify global.asax.
• Modify any file in the bin folder.

Any of these steps trigger unloading of the AppDomain and deletion of older DLLs. Otherwise old DLLs pile up and eat up your disk space. In an extreme case you'll run out of space on your hard drive.

PRB: CS0013 or CS0016 Compilation Errors in ASP.NET Web Applications (Q825791)

This one is easy. If you receive the error

CS0016: Could not write to output 
file 'c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary «
ASP.NET Files\xxxxx'. The directory name is invalid.

check your TEMP and TMP system variables. Chances are they point to a non-existing folder. Knowledge Base article Q825791 tells you how to fix it.

PRB: Access Denied Error When You Make Code Modifications with Index Services Running (Q329065)

This one is vicious! I see it come up in newsgroups all the time. Here's the scoop of the problem: the Indexing Service, if enabled, may engage in rescanning the Temporary ASP.NET Files folder for a couple of minutes. As the Knowledge Base article Q329065 states:

The length of time of the lock depends on the size of the directory that causes the Aspnet_wp.exeprocess (or W3wp.exe process for applications that run on Microsoft Internet Information Services [IIS] 6.0) to not load the particular DLL.

There are two ways to address this—either disable the Indexing Service altogether, or if you absolutely need it add the Temporary ASP.NET Files folder to the service for exclusion.

Insufficient User Rights To Run The ASP.NET Worker Process

You see different flavors of this predicament in the newsgroups all the time, too. Check out Q833444 and Q811320 on your own to see if they apply to you.

Keep The Code Warm

Microsoft published a great whitepaper titled Deploying .NET Framework-based Applications at Patterns & Practices. This prescriptive guide advocates "warming up" a web application by triggering a batch compilation. How?

• Manually. Navigate to each folder to have it batch-compiled. Next time the application restarts for whatever reason you have to do this again.
• Automatically. The said whitepaper suggests writing a script that simply places page requests. You can have a scheduler run this script every once in a while.

A Windows service might be even a more powerful option here. A Windows service may monitor the status of IIS as a whole or a particular web app and run the script automatically each time it restarts. WMI can lend a helping hand to monitor the state of processes.

The whitepaper also lists JIT compilation, but I won't go there. It's a questionable approach which is outside of the scope of this article. Please, refer to Chapter 4 of the whitepaper, "Deployment Issues for .NET Applications", to read more about NGEN and JIT compilation.

Conclusion

It's interesting how a little obscure switch somewhere in a configuration file can make so much difference. It helps to understand the underpinnings of ASP.NET and how it handles page compilations. Be careful when deploying code in production and make sure you set the <compilation> tag properly as outlined here.

Discuss

Liked it? Hated it? Discuss this article