The one with gpedit.msc

It's amazing how lazy people can get these days with this whole Internet lying around. I just got reminded today.
I was trying to open Group Policy Editor on a Windows XP machine to import the ICA client adm file and configure some settings just for that machine ... when I got this:

Group Policy Editor Error - Access Denied. Wait .. what?

Pfff... Let me Google this I said to myself. It'll know the answer for me. As always, got some results one of them being :

http://windowsnurfingertips.blogspot.co.uk/2009/10/gpeditmsc-or-group-policy-editor-access.html

No success. Sad face.

Then I recalled my purpose and was snapped out of my laziness by Mark Russinovich's motto: "When in doubt, run Process Monitor". Of course...
I still can't believe how I dismissed Mark's tool just like that..running into Google's arms. Running procmon and filtering the results by ACCESS DENIED gave me this:

c:\windows\system32\grouppolicy\gpt.ini

Hmm...let's check the file.

That doesn't look right, does it? Nope.
After changing permissions I was able to open gpedit.

I guess the bottom line is...don't be afraid of a little troubleshooting..and when in doubt, run Process Monitor :).

The one with BSOD

Some few months ago I looked into a problem with a virtual machine not booting up. Well, it was booting up fine but right before the windows welcome screen was BSODing with this screen

 

A problem has been detected and windows has been shut down to prevent damage to your computer (your standard text) and ..

A process or thread crucial to system operation has unexpectedly exited or been terminated.

 

 

 

and STOP code 0x000000F4.

 

VM was a Windows 2008 server running on xenserver with shared storage. Right before it started to misbehave it's been powered down so that resources like RAM and HD space be added. Once this was done and the machine powered back on it started to BSOD. I'm usually happy when a computer BSODs rather than simply restarting because there's a dump file you can debug. Sudden restarts usually suggest problems at lower level like hardware. However, as always, there's a catch. If your system BSODs when running then that's cool. Take the dump, analyze it and take measures. You have access to the system...you can deploy a fix easily. This one was crashing before I could enter some type of interface. Normal, safe mode, command line, last known good configuration...crashing. Nevertheless, challenge accepted. As expected, there was a dump file on our server (booted with a live cd) and although this issue didn't look like a citrix one to me I had to get Citrix involved to debug it. Even VHD is a stable format and the technology is around for some time now, I never ruled out vhd corruption which could've led to problems. No luck...because it was a kernel dump it was not providing enough data to the engineers at Citrix. Sometimes you need a full memory dump to catch everything (user data). I know how to set a windows os to generate a full memory dump via the gui itself...but how do you do it without gui or registry access? I did it by booting the vm with a windows live cd and mounting the system hive from c:\windows\system32\config\. I then edited the CrashDumpEnabled flag as per

 

 http://support.microsoft.com/kb/969028  / http://support.microsoft.com/kb/254649 and powered the VM up.  

 

Needless to say...even though the vm continued to BSOD and the screen was saying writing data to disk...no dump file was being created. Sad face , dead end.

As I was implementing this reg change I could not ignore the fact that currentcontrolset was missing and all I had was controlset1 and 3. So like every normal human out there that doesn't know something but wants to know about it, I found this - http://support.microsoft.com/kb/100010

Aha. Now everything clicks. (for those that don't want to read the MS KB - CurrentControlSet exists only when windows is running. It is nothing more than the ControlSet1 key mapped under it. ControlSetx is your last known good configuration).
I went back to my registry and for some dumb reason I checked the Select key (which controls what control set the system should use for normal booting and last known good configuration booting). Hmm...something is fishy. The default flag has the same value as the lastknowngood flag, or vice versa. This means that each time I wanted to boot into last known boot configuration I was actually booting into the default one, current one. Made the changes and pointed lastknowngood flag to the backup controlset. Restarted vm and.....it booted just fine. Argh...there's something there in the registry that is causing this...but what? Went back to registry and devised a simple trial and error plan. Exported each sub key from the working controlset and had that imported into the non working one. Powered the vm. So on and so forth until I found the subkey that was making the vm stable. In this case the subkey Control. I went then one step further and exported every subkey from this key and repeated the test. Half a day later I ended up with the faulty key "hklm\controlset01\control\session manager\environment"

Right ...which flag is it then? Ran the same tests as above, excluding one by one and.....PATH was the one.

The one causing the problem was having some extra entries at the beginning. After removing those entries (in red in the picture) and leaving the default (what's with black in the picture) server booted just fine. I don't know if the length of the path was causing this or something else, but interesting enough.

Click it to enlarge.

 

The one with TOE

This is a good one. One day I was reported an issue at a customer's site. They were rolling into production a new web based portal and for some reason, the 30+ Citrix servers were not in the mood to display the page - Page not found. Same page, a windows xp machine - great success. As any normal person would say - you have there a citrix issue. I saw on many occasions this trend - if it happens on the citrix server...it's a citrix issue...I feel sorry for these people and their lack of judgement.

Nevertheless, this was an issue and needed fixing. Where to start from? Well...let's jump on some other citrix servers than the ones reported to have the problem and test the page. Hmm...I found a few where it works. That's strange...they have the same built. Where to start from? Luckily, after a while in IT you start to develop this extra sense...the troubleshooting sense :)). Hunch and a bit of luck - let's start with the network card. Working server - Intel, non working servers - Broadcom...every single server. Aha...that's your starting point.
It then followed - driver upgrade, downgrade, firmware upgrade, changing setting, undo'ing setting...nothing seemed to work. About 2 days later I come across an article about tcp offload engine...great idea. Let me check this then. I opened the page in the browser and hit netstat -t. Scrolled down all the way to the entries belonging to my connection and I saw the status Offloaded. Went after that on a working server, did the same steps and checked the results: Inhost. Hmm...so on the servers where the connections to the web server are offloaded the page doesn't display. Let me disable TOE then. Ran Netsh int ip set chimney DISABLED. Reloaded the page...it worked. Nice...

So..what was it? I then came across this which I think says it all. (check the two linked posts from this article as well)

http://msmvps.com/blogs/thenakedmvp/archive/2007/01/06/how-broadcom-and-dell-wasted-three-days-of-my-time.aspx

Result...a bit nervous when I see broadcom nics...:)

The one with lost library

Had to reinstall my OS the other day and not for one second have I thought of my beloved virtual machines running in vmware workstation. I woke up in the new operating system with my library gone. Then I asked myself - How does one migrate / copy / move the vmware workstation library from one machine to another or one operating system to the other ? Well everything is saved in this file: inventory.vmls located under the folder: C:\Users\<user>\AppData\Roaming\VMware

 

Just copy that across and start workstation.

I had a backup :). Pam pam.

Hello world :)

Hello world.

I guess everything nowadays starts with hello world...

This first post is merely a welcome to the blog...and a test for me to see how blogger works :)

The title says it all...I work in IT and I look (troubleshoot) after Citrix products. I thought it'll be fun to blog about my experience(s).  

Stay tuned.
Daniel.