The one with the 30 second delay

So you're troubleshooting a delay issue? Here's one that I came across today.

I'm at a customer setting up few things. One of the task is to make sure users can login from the thin client solution into Citrix published desktops. We're going to use ThinKiosk on top of the OS. The chosen OS is Windows Embedded 8.1 Industry (WES 8). The OS gets deployed part of an SCCM task sequence. Part of the same task sequence we install ThinKiosk and apply a write filter to prevent any changes to the C drive from sticking (the build in Windows write filter via the Embedded Lockdown Manager). Everything works fine except that there's a 30 second delay between pressing CTRL ALT DEL and getting the username and password prompt.

After much troubleshooting a colleague found the solution posted on the Microsoft support forums. Exclude the folder c:\windows\ccm\logs from the write filter. That'll fix the issue.

The one with the Network Profile

Hi there,

It's been a while since I posted something. I blame my employer for keeping me busy :)

So I'm on this project for few days where I have to build some physical servers, install the OS on them, patch them to the switches, etc.
Part of the build requires setting up a DC and joining the other servers to the domain. Done and dusted in no time. After joining the servers to the domain I moved from using the KVM in the datacenter to managing the servers remotely. Not that before it wasn't possible but I usually do things in one shot: set IP & join to the domain.

To my surprise I wasn't able to RDP to the boxes. The firewall in Windows was turned on so I said to check if the rule to allow RDP is on. It was on but only for Domain and Private networks. I checked the Network profile I was on and what do you know - I was on a Public Network profile. That is not right - it should've been Domain  Network profile.

After some more digging around to understand how the Network profile is set I fired up Process Monitor (When in doubt run process monitor :) ) to have a look at the network traffic. I used Wireshark at the same time but in this case it's easier to spot the problem in Process Monitor.

I noticed that after plugging the network cable in the server, Windows (2012 R2) was starting to identify the network type. It does this by by opening a connection to the domain controller on port 389 and performing some queries (I will not go into details).

As seen below - there are plenty of UDP send packets but no response:

I plugged the DC and a client into another switch (a 5 port - 10 year old one) and the network type was set to Domain right away. The only thing that could create such a problem is if for some reason the switch doesn't start forwarding packets right after the cable is plugged in. Aha - spanning tree protocol...:) What Windows does is that it tries to identify the network right away without actually knowing if the switch/switch port is in a state to process/forward packets.
After a chat with the networking admin and changing the ports' spanning tree state to "portfast" the problem went away. Now I can see replies to the outgoing packets and the network profile is set to the correct one. This means the firewall profile Windows is using is also the correct one.

The one with the slow first logon

Hi there,

I just came across this old entry which never got published...so here it is.

I want to share something with you, something that recently happened to me while deploying a XenApp 6.5 environment; might spare you some valuable time.

As said, I'm deploying at the moment a XenApp 6.5 farm and was presented with a frustrating issue: after a server restart, the first logon via ICA (RDP and console were always fine) was slower than normal. By slower than normal I mean that there was a delay of 30 to 40 seconds during logon. This was visible in both provisioned servers and normal non-provisioned servers. The farm is 6.5 HRP3.
After this first logon/logoff via ICA subsequent logins were fine - loading in the expected amount of time. This was happening to both published applications and published desktops.

Here's a print screen of where the hang was and what was visible on the screen during the hang:

 

To spare you the trouble of reading a long story I'm going to cut to the chase. Running a Process Monitor trace on the first login showed that some Edgesight components where trying to perform what I believe is a CRL check against some servers in Akamai. As the customer has a proxy with authentication the access was failing.

 

The fix came to me when I found a Citrix article explaining how to fix an issue with an Edgesight service. I told myself then that if the trick works for that particular service it should work for the other Edgesight executables - the ones that were giving me a headache.

So what I ended up doing was to create a file called SemsService.exe.config under the folder c:\Program Files (x86)\Citrix\Euem\Service.

A similar file called rscorsvc.exe.config was created under the folder c:\Program Files (x86)\Citrix\System Monitoring\Agent\Core

Both files contain the following text:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<runtime>
<generatePublisherEvidence enabled="false" />
</runtime>
</configuration>

 

...but you probably saw the above lines of config more times than needed.

 

Problem fixed.

 

 

 

 



The one where Discovery fails

Hi there,

Ever had one of those issues when you know it was not triggered by recent changes because there were none and you look around and you actually can't find any recent changes? Let me tell you - you need to look harder. There's always someone changing something, next to you, without you actually knowing it. I heard the answer "Nothing has changed" so many times so far that I think it's safe to say...I don't trust people anymore when it comes to this.
Back to our issue. We had a XenApp 6 farm (under Citrix PVS) that all of a sudden it decided not to perform discovery. There are nearly a dozen servers with one data collector. Basic error message:

Error occurred when using {server} in the discovery. On double clicking the message the following was being shown: An unexpected error occurred. Check that the server name is correct, that the server is on, that Citrix Presentation Server is installed on this server, and that the Citrix MFCOM Service is running.

The usual suspects: IMA and MFCOM. You got to love them.

After running all the sanity checks, everything seemed in place. IMA started, MFCOM started, LHC recreated. Farm was functioning properly, people could connect and there were no relevant errors in Event Logs. We even tried un and re / gistering MFCOM - no luck.

I started thinking of creepier scenarios - someone has made a joke and removed all Citrix admins from the farm (you get the same error message when you're not an admin). I used DSView to check there are still admins listed - there were. Time to pull out the big guns - CDFControl.

So I ran CDF Control with only Access Management Console category selected (AMC..nice memories huh?) and performed a discovery. Here's the output:

*the picture was cropped to show only the relevant data.

Aha - a server entry in the farm is causing problems. I bet dscheck will show more details.

C:\Users\user>dscheck

Data Store Validation Utility.    Version: 6.23

Server Consistency Check:  The Citrix XenApp record with HostName {server} at
DN 00001CAE may be invalid.  The Load Manager for Citrix XenApp entry was not found.

Finished data store validation.


I created a backup of the SQL Database holding the Citrix Datastore and then executed the same command as above with the /Clean switch, to fix the problems. For confirmation I ran dscheck again. There was still and error present so I ran dscheck /clean again. The second error was cleared the second time and I was able to run discovery.

To finish the story I started with - the server causing problems was a recently created test server booted from a new test vdisk (completely rebuilt). So the answer to the question: What recent changes were in the environment should have been: tested a new vdisk in the production farm.

Happy Citrixing.

The one with insufficient resources

A colleague of mine had the opportunity to work on a case recently that had a pretty interesting resolution so I thought I might as well just share it. The client was running a CAG 504 VPX serving a XenDesktop infrastructure. Every now and then, during the login the users were prompted with an error of: Intermittent HTTP 405: page cannot be displayed
Refreshing the page would've displayed the resources.

I will not go into details on this one but after months of troubleshooting Citrix was not able to understand why this happens. We narrowed down to where it happens by analyzing the network traces and other logs from the CAG but simply were unable to get to the WHY.

So we were one day in the office talking about VM resource reservations and how a colleague of ours saw some weird things with a NS VPX that was not given enough "juice" so we decided to go down this route and propose to our client to ensure his CAG VPX VM has at least 500 MHz and 500 MB RAM reserved on the Hypervisor side.
I'm happy to say that the issue never presented itself after that.

The one where everything is "Starting"

Got a call the other day from a customer that reported problems with a Citrix server. After a software installation and server restart Citrix IMA and MFCOM where hanging on Starting or IMA was throwing error 214783647

We believed that LHC got corrupted because of the restart so went on with recreating LHC. That did not help. We noticed then that a series of other services where stuck on Starting, not only Citrix Independent Management Architecture and Citrix MFCOM. DHCP Client, Windows Event Log and TCP/IP Netbios Helper were also stuck on Starting. We suspected at that moment a problem lower in the software stack, something at the OS level so we had to ask the client to attempt an OS repair.
To cut the long story short the problem was reproduced on freshly built servers so we gave troubleshooting another go. By luck I ended up on a forum post where a fellow IT guy had a similar problem and what got him going was to follow this procedure: http://pieter.wigleven.com/it/archives/54 (although not visibly related to his problem or ours)
To sum that post up, if you changed your System Locale / System Accounts locale (local system, local service, network service) to something than English United States this behaviour might be visible. Changing it back to English US and restarting the server resolves the problem.

The one with failing to update to HRP1

Another day, another case. This was some months ago. Windows 2008 with XenApp 5. The client was trying to deploy HRP1 onto the server but the installation was failing. With MSI logging in place it was discovered that the failing point/error was this one:

Error 1720. There is a problem with this Windows Installer package. A script required for this install to complete could not be run. Contact your support personnel or package vendor. Custom action SetHotFixName script error -2146827859, Microsoft VBScript runtime error: ActiveX component can't create object: 'Scripting.FileSystemObject' Line 7, Column 2

This was a long time ago so I don't recall all the steps I took to troubleshoot the issue, I only recall it took some time. Luckily, with some educated guesswork in place, I managed to get HRP 1 to install.

I blindly followed at that time all the steps described in this article, with no luck:
http://support.citrix.com/article/CTX764017

Then for some reason I realized I'm in a 64bit environment so the location is wrong. Changed the location and ran regsvr32 c:\windows\syswow64\scrrun.dll

Now that worked fine :)