There are a number of utilities you can use to debug IIS faults. David Wang's guide on how to understand and diagnose an application pool crash<\/a> gives some insight into how it should be done - attach a debugger and wait for the fault, then look at the stack trace. His suggestions include IISState<\/a>, an IIS resource kit utility, which will give you an output of the current state of an IIS process; and DebugDiag<\/a>, a much nicer and newer interactive debugger for IIS.<\/p>\n I spent a couple of days playing with DebugDiag, and unfortunately, due to the nature of my problem, I couldn't get anything useful. It seems that the application wasn't ever actually crashing or hanging - states that DebugDiag should be able to deal with easily. IISState, on the other hand, suggested I was regularly getting first chance exceptions<\/a> thrown. A first chance exception means an exception has been raised - if the application handles it, then nothing more is mentioned, else it becomes a second chance exception, and the debugger or system exception handler is called). So, more digging was required.<\/p>\n There are a number of blogs that mention running WinDbg, but none that actually put together all the pieces on how to debug this kind of problem. So here's my story for you.<\/p>\n First, attach the debugger to your w3wp.exe process. I found it's process ID in Task Manager - you need to add the PID column. At this point, I get a breakpoint hit - you might not, so use D<\/u>ebug\/B<\/u>reak or Ctrl-Break to stop the program.<\/p>\n Debugging symbols let you see not just the memory location of a failure, but the name of the function that it is in. They are not normally included with release software as they make it a lot larger, but can be downloaded separately in this case. DebugDiag is friendly and will automatically download appropriate debugging symbols for you from Microsoft's symbol server<\/a>, and put them in c:\\symcache\\. WinDbg is less friendly and requires you set this up yourself. I used the same path, as I already had some symbols downloaded - I told WinDbg to look at the local symbols first, and look at the online symbol server if they're not already cached there. Do this like so:<\/p>\n Now, we need to load the SOS debugger helper<\/a>, which lets you debug managed code in WinDbg (Nerd note: SOS stands for Son of Strike, which is a Microsoft in-joke<\/a>.) If you started the process inside the debugger, you can apparently do this:<\/p>\n I couldn't, so I'll do this:<\/p>\n If you're running .NET 1.1, you can .load clr10\\sos.dll<\/tt> for the version shipped with the debugger.<\/p>\n Now, we instruct it to break whenever there is an exception thrown:<\/p>\n 0:033> sxe CLR<\/tt><\/p>\n And hit D<\/u>ebug\/G<\/u>o, type 'g and hit Enter, or hit F5 to resume the program.<\/p>\n No sooner than I hit 'g', we get results!<\/p>\n I don't speak assembly and really have no desire to. Thankfully the SOS extension lets us look at the exception, and the call stack (the list of functions that we called to get where we are). Type !pe<\/tt> to print the exception, and !clrstack<\/tt> to see the call stack.<\/p>\n And there's the problem. A first-class exception is handled trying to connect to a database with incorrect login details, so it continues, probably never freeing the memory from the last connection. This particular exception doesn't get logged and doesn't get displayed on screen (probably because it's being loaded in the background) and causes an infinite timeout. Oops!<\/p>\n Armed with a little knowledge about the debugging tools, it's easy to see what's going on inside your code, even if it's running in IIS.<\/p>\n","protected":false},"excerpt":{"rendered":" Problem: after hitting \"Login\" on an ASP.NET web application on IIS6, w3wp.exe races to high CPU usage, and about 70MB of memory. CPU usage falls off, but memory usage increases a few KB every second. No indication is made that the application has crashed or hung, no output. The first utility I installed to check […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[66,68],"tags":[63,3,27,4],"_links":{"self":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/210"}],"collection":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=210"}],"version-history":[{"count":1,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/210\/revisions"}],"predecessor-version":[{"id":245,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/210\/revisions\/245"}],"wp:attachment":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=210"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=210"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=210"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"WinDbg,](\"http:\/\/craig.dubculture.co.nz\/blog\/wp-content\/uploads\/2007\/12\/windbg.thumbnail.PNG\")
<\/a>Enter the Windows debug tools<\/a>, and specifically WinDbg<\/a>. This is what the professionals use, and as such, it's ultimately powerful and incredibly scary. It's updated regularly, except for the interface, which is nicest described as \"circa-1990\"<\/a>.<\/p>\n.sympath SRV*c:\\symcache*http:\/\/msdl.microsoft.com\/download\/symbols<\/pre>\n
0:033> .loadby mscorlib sos\r\nUnable to find module 'sos'<\/pre>\n
0:033> .load C:\\WINDOWS\\Microsoft.NET\\Framework\\v2.0.50727\\SOS.dll<\/pre>\n
0:033> g\r\n(a78.8b4): CLR exception - code e0434f4d (first chance)\r\nFirst chance exceptions are reported before any exception handling.\r\nThis exception may be expected and handled.\r\neax=0230ec7c ebx=000d9c50 ecx=00000000 edx=00000024 esi=0230ed08 edi=e0434f4d\r\neip=77e4bee7 esp=0230ec78 ebp=0230eccc iopl=0 nv up ei pl nz na po nc\r\ncs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00200202\r\nkernel32!RaiseException+0x3c:\r\n77e4bee7 5e pop esi<\/pre>\n
0:012> !pe\r\nException object: 048f8a4c\r\nException type: System.Data.SqlClient.SqlException\r\nMessage: Cannot open database requested in login 'R622P1Sy1'. Login fails.\r\nLogin failed for user 'admin'.\r\nInnerException: <none>\r\nStackTrace (generated):\r\n<none>\r\nStackTraceString: <none>\r\nHResult: 80131904\r\n0:012> !clrstack\r\nOS Thread Id: 0x8b4 (12)\r\nESP EIP\r\n0230ed54 77e4bee7 [HelperMethodFrame: 0230ed54]\r\n0230edf8 6539bf14 System.Data.SqlClient.SqlInternalConnection.OnError(System.Data.SqlClient.SqlException, Boolean)\r\n0230ee0c 652f695d System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(System.Data.SqlClient.TdsParserStateObject)\r\n0230ee3c 652f79e5 System.Data.SqlClient.TdsParser.Run(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject)\r\n0230ee90 652e932a System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean)<\/pre>\n