Craig Box's journeys, stories and notes...

Posts Tagged ‘sysadmin’

More on Windows service permissions

Wednesday, March 12th, 2008

A while ago I wrote about the permission required to query the status of a Windows service.

Today, I started looking for a way to set this permission from the .NET framework, without using sc or subinacl. Registry and file system security is easy; services, not as much. After what seemed like an eternity of searching I came across the answer - an article on 'stackenbloggen'1 on setting ACLs on Windows services. There is magic in System.Security.AccessControl in .NET 2.0 which lets you control security on pretty much any Windows object, and here are some classes to expose it.

Words can be very specific in the English language, and even more so in code. For example, a method called "SetAccessControl" does not do what it would do if it was called "AddAccessControlEntry". So, if you create an ACE that contains just the item you want to add, and run SetAccessControl, you're screwed.

Never mind, I wrote about how to fix this:

C:\Craig\>sc sdset Service1 D:(A;;CCLS...)
[SC] OpenService FAILED 5:Access is denied.


Maybe I'll get a better message out of subinacl?

C:\Craig\>subinacl /service Service1 /sacl=D:(A;;CCLS...)
Service1 - OpenService Error : 5 Access is denied.

Double oops.

New error message though, worth searching for:

OpenService Error 5 Access is Denied - Google results

It's very irritating to only find your own post when searching for something! Lots of old-school #wlug'ers used to get this all the time with the WLUG wiki.

A bit more Googling, and the answer is found in Microsoft's knowledge base: how to reset a DACL on a Windows service. The crux of the matter is you need to run a command prompt as LocalSystem - type something like at 15:37 cmd.exe /interactive, where 15:37 is the time one minute from now. (Don't make it bigger. Waiting 30 seconds is tedious enough!)



C:\WINDOWS\system32>sc sdset Service1 D:(A;;CCLS...)
[SC] SetServiceObjectSecurity SUCCESS

Double booyah!

I'll be calling GetAccessControl and adding my own ACE to its result from now on, I think.

  1. I love the name 'stackenbloggen', but a disclaimer that says "[..] my own personal opinions and do not represent my employer's view in any way" is meaningless if you don't say who you are on your blog! 

Debugging w3wp.exe and IIS application pools

Saturday, December 29th, 2007

Problem: after hitting "Login" on an ASP.NET web application on IIS6, w3wp.exe races to high CPU usage, and about 70MB of memory. CPU usage falls off, but memory usage increases a few KB every second. No indication is made that the application has crashed or hung, no output.

The first utility I installed to check this out was IISTracer. Ignore the horrible web site - this is a nice little utility which lets you see what requests IIS is currently serving. In this instance the default.aspx page calls a dashboard.aspx, which was timing out indefinitely without returning a response to the client.

There are a number of utilities you can use to debug IIS faults. David Wang's guide on how to understand and diagnose an application pool crash gives some insight into how it should be done - attach a debugger and wait for the fault, then look at the stack trace. His suggestions include IISState, an IIS resource kit utility, which will give you an output of the current state of an IIS process; and DebugDiag, a much nicer and newer interactive debugger for IIS.

I spent a couple of days playing with DebugDiag, and unfortunately, due to the nature of my problem, I couldn't get anything useful. It seems that the application wasn't ever actually crashing or hanging - states that DebugDiag should be able to deal with easily. IISState, on the other hand, suggested I was regularly getting first chance exceptions thrown. A first chance exception means an exception has been raised - if the application handles it, then nothing more is mentioned, else it becomes a second chance exception, and the debugger or system exception handler is called). So, more digging was required.

WinDbg, the Windows DebuggerEnter the Windows debug tools, and specifically WinDbg. This is what the professionals use, and as such, it's ultimately powerful and incredibly scary. It's updated regularly, except for the interface, which is nicest described as "circa-1990".

There are a number of blogs that mention running WinDbg, but none that actually put together all the pieces on how to debug this kind of problem. So here's my story for you.

First, attach the debugger to your w3wp.exe process. I found it's process ID in Task Manager - you need to add the PID column. At this point, I get a breakpoint hit - you might not, so use Debug/Break or Ctrl-Break to stop the program.

Debugging symbols let you see not just the memory location of a failure, but the name of the function that it is in. They are not normally included with release software as they make it a lot larger, but can be downloaded separately in this case. DebugDiag is friendly and will automatically download appropriate debugging symbols for you from Microsoft's symbol server, and put them in c:\symcache\. WinDbg is less friendly and requires you set this up yourself. I used the same path, as I already had some symbols downloaded - I told WinDbg to look at the local symbols first, and look at the online symbol server if they're not already cached there. Do this like so:

.sympath SRV*c:\symcache*

Now, we need to load the SOS debugger helper, which lets you debug managed code in WinDbg (Nerd note: SOS stands for Son of Strike, which is a Microsoft in-joke.) If you started the process inside the debugger, you can apparently do this:

0:033> .loadby mscorlib sos
Unable to find module 'sos'

I couldn't, so I'll do this:

0:033> .load C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\SOS.dll

If you're running .NET 1.1, you can .load clr10\sos.dll for the version shipped with the debugger.

Now, we instruct it to break whenever there is an exception thrown:

0:033> sxe CLR

And hit Debug/Go, type 'g and hit Enter, or hit F5 to resume the program.

No sooner than I hit 'g', we get results!

0:033> g
(a78.8b4): CLR exception - code e0434f4d (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=0230ec7c ebx=000d9c50 ecx=00000000 edx=00000024 esi=0230ed08 edi=e0434f4d
eip=77e4bee7 esp=0230ec78 ebp=0230eccc iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00200202
77e4bee7 5e              pop     esi

I don't speak assembly and really have no desire to. Thankfully the SOS extension lets us look at the exception, and the call stack (the list of functions that we called to get where we are). Type !pe to print the exception, and !clrstack to see the call stack.

0:012> !pe
Exception object: 048f8a4c
Exception type: System.Data.SqlClient.SqlException
Message: Cannot open database requested in login 'R622P1Sy1'. Login fails.
Login failed for user 'admin'.
InnerException: <none>
StackTrace (generated):
StackTraceString: <none>
HResult: 80131904
0:012> !clrstack
OS Thread Id: 0x8b4 (12)
ESP       EIP
0230ed54 77e4bee7 [HelperMethodFrame: 0230ed54]
0230edf8 6539bf14 System.Data.SqlClient.SqlInternalConnection.OnError(System.Data.SqlClient.SqlException, Boolean)
0230ee0c 652f695d System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(System.Data.SqlClient.TdsParserStateObject)
0230ee3c 652f79e5 System.Data.SqlClient.TdsParser.Run(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject)
0230ee90 652e932a System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean)

And there's the problem. A first-class exception is handled trying to connect to a database with incorrect login details, so it continues, probably never freeing the memory from the last connection. This particular exception doesn't get logged and doesn't get displayed on screen (probably because it's being loaded in the background) and causes an infinite timeout. Oops!

Armed with a little knowledge about the debugging tools, it's easy to see what's going on inside your code, even if it's running in IIS.

Parsing quotes in cmd.exe

Monday, December 24th, 2007

The Microsoft world can be horribly inconsistent. Take escaping a quote character:

  • "" in VB and SQL Server
  • \" in C++/C#
  • Unless you use a @ symbol first, in which case you have to use "" as \ won't be interpreted as an escape character

It never seemed at all you could escape characters on a command line. Until, of course, you go to do something like this:

rem xcopy is old, let's not use it any more
rem the backslash on the end of bar tells xcopy
rem that it's a directory, not a file, it's copying to
rem xcopy /e "C:\foo\" "C:\bar\"
robocopy "C:\foo" "C:\bar\" /e /x "Extra Options"

Whereupon the much newer program parses the output as follows:

destination=C:\bar" /e /x "Extra Options"

This is a problem with anything that uses the .NET framework also. Watch your trailing slashes!

C#: How to tell if an IIS metabase property is inherited

Monday, December 24th, 2007

How can you tell, using managed code, if an IIS metabase property is inherited?

Scenario: I am using System.DirectoryServices to manage the IIS configuration (known as the "metabase", and stored in an XML file). I want to remove a property on a metabase node, but only if it is explicitly set here. This allows me to inherit the correct value from up the tree.

First attempt:

if (dirEntry.Properties["AnonymousUserName"].Count > 0) {
    Console.WriteLine("Log: Local value found for AnonymousUserName, removing");

My assumption was that this would remove the entry the first time, if it exists, and ignore it every time afterwards. However, every time it runs, Count is 1 and the property appears set.

It turns out that the Properties method on a DirectoryEntry object shows all the properties, including ones that are set on parent items in the tree.

How can we see only items that are defined on this particular node? This problem shows up all over the Internet with no solution. However, Microsoft's David Wang isn't one to leave a newsgroup question unanswered:

Never tried it in .Net, but with ADSI you can determine this.
Set objNode = GetObject( "IIS://foo/W3SVC" & "/" & "AppPools/AnAppPool" )
Set objNodeAttribute = objNode.GetPropertyAttribObj( "WamUserName" )
If ( objNodeAttribute.IsInherit = true ) Then
End If

I just needed a way to run GetPropertyAttribObj from C#, which was a much easier problem to solve. There were only a couple of relevant hits, the best one being some abandoned Pastebin code in Google's cache.

We now have an answer:

/// <summary>
/// Establish if a metabase property is set explicitly on this node.
/// </summary>
/// <param name="iisNode">Metabase node</param>
/// <param name="propertyName">Property name</param>
/// <returns>True if the property is defined on this node explicitly; false if the property is not defined at all, or is inherited.</returns>
private static bool PropertyDefinedExplicitly(DirectoryEntry iisNode, String propertyName)

    // check 1. this node has a valid key-type, and 2. it contains the required property
    if (iisNode.Properties.Contains("KeyType") &&
    iisNode.Properties["KeyType"].Value.ToString().Length > 0 &&

        // call ADSI GetPropertyAttribObject method directly
        Object propertyAttributes = iisNode.Invoke("GetPropertyAttribObj",
        new String[] { propertyName });
        IISOle.IISPropertyAttribute attributes =

        // Isinherit will be set if the property is inherited
        if (attributes.Isinherit)
            return false; // property was inherited
            return true; // property explicitly defined on this node

        return false; // not a valid IIsObject node, or property not found

We then rewrite the original function like so:

if (PropertyDefinedExplicitly(dirEntry, "AnonymousUserName")) {
    Console.WriteLine("Log: Local value found for AnonymousUserName, removing");

There is a lesson to learn here for Windows programmers: Microsoft support is in the newsgroups. Don't fear Usenet because it's older than you are!

With useful tips like this, I'll win the NZ .NET blog of the year for sure!

Querying the status of a Windows service

Friday, December 7th, 2007

An interesting puzzle presented itself to me at work yesterday. We have some code called from an ASP.NET application, which queries the status of a service. This works fine when you run the application pool as NetworkService, but if you run it as the IWAM_SERVERNAME user that is created for running IIS processes, it would fail.

I wrote a console application with exactly the same code that the ASP.NET application used, and it worked fine.

Interesting aside: you can find the password for the IWAM_SERVERNAME user by using the Metabase Explorer from the IIS 6 Resource Kit. It's under the Server/LM/W3SVC node as WAMUserPass, and you have to enable "View/Secure Data" to see it. Then, use runas /user:iwam_servername cmd.exe, and enter in the password from the metabase.

Why? Well, surely it must just be some kind of permissions issue, not the Winter Madness.

What can't be done with subinacl isn't worth doing, so I pulled it out and had a look at the service. Here's the important part:

/pace =interactive      ACCESS_ALLOWED_ACE_TYPE-0x0
        SERVICE_INTERROGATE-0x80           READ_CONTROL-0x20000               SERVICE_USER_DEFINED_CONTROL-0x0100

(People finding this useful post via search will want to read the full subinacl output.)

Turns out an interactive user can query a server status, but IIS doesn't run interactively. Assigning permission to the IWAM account is the only way to do this:

subinacl /service Service1 /grant=IIS_WPG=s

Problem solved for the web app.

Always one for complete proof, I tested removing interactive access (subinacl /service Service1 /deny=interactive=s). Result as requested - the console app now throws the same exception the web application does.

But wait...

C:\Program Files\Windows Resource Kits\Tools\>subinacl /service Service1 /revoke=interactiveService1
- OpenService Error : 5 Access is denied.

Oops. Big oops. Nothing I can do now seems to be able to let me fix this, because I've lost access to query the status of the server myself.

Some searching led me to an answer: use the sc.exe utility, which lets you control services from the command line. It also lets you set their security descriptors!

C:\Program Files\Windows Resource Kits\Tools\>sc sdshow Service1 

C:\Program Files\Windows Resource Kits\Tools\>sc sdshow Service2

[SC] SetServiceObjectSecurity SUCCESS

C:\Program Files\Windows Resource Kits\Tools\>subinacl /service Service1
+Service Service1

Whew. See how the SDs differ between the two services? Ignore the gobbledy-gook that is SDDL, and just copy an example from another similar service.

I suspect I could have granted access using with sc instead of subinacl, but the IWAM_SERVERNAME account and IIS_WPG group don't have well-known SIDs, so I would have had to find out the SID using another utility anyway.

(Hi Mum. This is what I do all day.)

Fixing the password problem on small business networks

Thursday, June 28th, 2007

I've been involved in commercially supporting Windows networks for almost 5 years now, having dealt with a hundreds of users across dozens of different companies.  Most of the clients we support are "small businesses", which makes sense, as it is quoted that as 97.3% of private enterprise in NZ is small/medium sized, accounting for 49.4% of private sector employment.

These companies, in the most part, don't have the infrastructure for a large, homogeneous IT environment.

And you know what?  They couldn't care less.

They make do with what they always have - buying PCs piecemeal, having Office 2000 on some PCs and Office 2003 on others, and - the kicker - knowing everyone else's passwords, instead of sharing data.  Even though products like Microsoft Exchange allow you to do things like delegate access to someone else's mailbox, they still claim they need to all have their passwords set to 'password', or documented in a book, in case they need to sit at someone else's PC.

People don't do what PCs suggest they should do. No one really wants the multi user functionality that PCs have now.  In small business, people want to be able to use the line-of-business application, a web browser, and access their own e-mail and files. They might like the idea of having some personalisation (some care for it, some don't), but overall, having to log out as you and log in as me takes longer than the effort required just to use the application as set up on your profile.

So, as a sysadmin, I want people to use strong passwords.  I have to wean them off the idea of needing someone else's password to get at their data.  And I want to work how they work, not how I think they want to.  They want the desktop you use when you're filling in for someone to look like it did when they were learning over the first person's shoulder.

The primary solution put forward by Microsoft is "roaming profiles", where you can log into any machine, and have your applications loaded.  Say you've got a shortcut to Word 2003 on your desktop, and you roam that profile to a machine with Word 2000 on it.  Doesn't work.  Good for volume licensed customers with the same software on all PCs, but not good for us.  Doubly bad when you look at how people actually work - the accounts clerk has MYOB and payroll software installed, some managers will have banking software for authorising transactions, sales people may have a line-of-business application that analysts don't need, etc.  It's not worth ensuring that the software is on everyone's machine, it means unnecessary licensing costs, and in the case of things like payroll software, people want to know it's not available to everyone.

So, roaming profiles are out.

When someone is away, their mail and phone are diverted, but their PC sits there unused, or someone has to sit at the desk - they try and find the icons you used to click, but their new profile doesn't have the shortcuts, or the per-user registry keys required for some random application.

Let's look at some other possible solutions:

Terminal services or Citrix MetaFrame

Put everyone on a thin client and make everyone use a central server.  Good plan, large investment required, takes a lot of time to change from an office of fat-clients to a thin-client environment, and not all SME apps are TS friendly.  Also, if you scale to needing more than one TS, then you're back at square 1 with needing the apps to be in synch across two machines.

Virtual machinery

Abstract the access away from the machine - have a bunch of passwords all able to unlock the same machine.  Wasteful.

Change someones password temporarily if you need to use their account

Tried this.  At present, there is no way for the Administrator to change someone's password, store the original hash, and set it back at a later date.  I think it's worth implementing though.

Cheat biometrics

Biometric sensors, like the fingerprint scanner on my T60 laptop, can be 'cheated': in an office of 10 people, with 10 fingerprints able to be stored, why not store everyone's fingerprint on everyone's computer?  Requires buying a scanner for everyone's PC.

Insecure machine accounts, delegated access to data

Why not have everyone have a 20 character password, but have a single password for logging into the machine in the morning?  You could have a "machine user" account on each machine, and delegate e-mail access for everyone necessary to the machine.  A bit more administrative overhead but a possible solution.

Craig's "Silver Bullet" answer

My favourite suggestion is delegating access to your profile, or your profile/PC combination.  This is what Exchange lets you do now with e-mail - why not extend this to user accounts also?  Presumably, the component (a "GINA") that that handles authentication for the fingerprint reader, could be made to start loading another account, separate from the one you entered the password for?

Therefore, we can have a 1:1 mapping of people to passwords, so no-one ever has to know anyone else's, and then we can have a 1:many between computers and users, without needing messy multiple profiles.

Anyone see any problems with this approach?  If not, why haven't you written it yet?  Look perhaps at pGina as a base. My (ex-)small businesses will pay.

USB devices and drive letters

Friday, June 22nd, 2007

A year ago, I ranted about the fact that Windows will map a newly added USB drive on the first available drive letter, even if there's a subst'd or mapped network drive on that letter.

Kyle pointed me to USBDLM, the USB Drive Letter Manager. This is a piece of software that, as well as working around this bug, will allow you to ensure that bad USB devices are always mapped to the same drive letter. It's free for personal or educational use, but costs for commercial use. Not that it's really a solution - Microsoft, this is a simple bug that you could fix today. It is an exercise for the reader to get Raymond Chen to tell me why it's not as simple as I think.

Mongrel upload progress problem - cause found

Friday, June 15th, 2007

Per my previous post on upload progress in Rails, I can now confirm:

The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA.

Which means, cat site.cer chain.cer Equifax_Secure_Global_eBusiness_CA-1.cer site.key > site.pem.

I've backported Pound to Ubuntu Dapper, from Debian Testing. Dapper only has 1.0, which might work, but the configuration has changed beween 1.x and 2.x, which makes the examples incorrect.

SpamAssassin 3.2.0 backport for Ubuntu Dapper

Wednesday, June 6th, 2007

I've built packages for SpamAssassin 3.2.0 for Ubuntu Dapper. They are available in my firewall repository with the dependencies (libnet-dns-perl, libnetaddr-ip-perl, libmail-spf-perl):

deb dapper firewall

If you use this repository, you'll get a new version of ClamAV, and some other packages also. Beware.

It was a bit of a mission to build, but made easier with the Prevu tool. This is like pbuilder for backports, and anyone doing anything with backports should use it. You can use the 0.4.1 release on Sourceforge on Dapper.

ACT presents "Object reference: Not set to an instance of an object." when connecting via Citrix

Wednesday, May 23rd, 2007

The ACT! contact management application is a pig. There, I've said it. It's one of those programs that keeps changing owners - that's how much of a pig it is. Seems no-one really wants it.

They re-wrote it in .NET a couple of versions ago, and it ought to be better, but it isn't. ACT! throws "Object reference: Not set to an instance of an object." (a reasonably common .NET error) whenever it feels like it.

In my particular case, I could open it fine when using a Citrix desktop session, but as a published application, it would die all the time.

I tracked the problem to the seamless window. When you run seamlessly, you get Citrix's WFSHELL.EXE running, and not EXPLORER.EXE. You can force a published application to run at a certain screen size, which runs as if you were in a desktop. Hopefully the ACT! forums will give me a better answer.