This is a good one. One day I was reported an issue at a customer's site. They were rolling into production a new web based portal and for some reason, the 30+ Citrix servers were not in the mood to display the page - Page not found. Same page, a windows xp machine - great success. As any normal person would say - you have there a citrix issue. I saw on many occasions this trend - if it happens on the citrix server...it's a citrix issue...I feel sorry for these people and their lack of judgement.
Nevertheless, this was an issue and needed fixing. Where to start from? Well...let's jump on some other citrix servers than the ones reported to have the problem and test the page. Hmm...I found a few where it works. That's strange...they have the same built. Where to start from? Luckily, after a while in IT you start to develop this extra sense...the troubleshooting sense :)). Hunch and a bit of luck - let's start with the network card. Working server - Intel, non working servers - Broadcom...every single server. Aha...that's your starting point.
It then followed - driver upgrade, downgrade, firmware upgrade, changing setting, undo'ing setting...nothing seemed to work. About 2 days later I come across an article about tcp offload engine...great idea. Let me check this then. I opened the page in the browser and hit netstat -t. Scrolled down all the way to the entries belonging to my connection and I saw the status Offloaded. Went after that on a working server, did the same steps and checked the results: Inhost. Hmm...so on the servers where the connections to the web server are offloaded the page doesn't display. Let me disable TOE then. Ran Netsh int ip set chimney DISABLED. Reloaded the page...it worked. Nice...
So..what was it? I then came across this which I think says it all. (check the two linked posts from this article as well)
http://msmvps.com/blogs/thenakedmvp/archive/2007/01/06/how-broadcom-and-dell-wasted-three-days-of-my-time.aspx
Result...a bit nervous when I see broadcom nics...:)
Nevertheless, this was an issue and needed fixing. Where to start from? Well...let's jump on some other citrix servers than the ones reported to have the problem and test the page. Hmm...I found a few where it works. That's strange...they have the same built. Where to start from? Luckily, after a while in IT you start to develop this extra sense...the troubleshooting sense :)). Hunch and a bit of luck - let's start with the network card. Working server - Intel, non working servers - Broadcom...every single server. Aha...that's your starting point.
It then followed - driver upgrade, downgrade, firmware upgrade, changing setting, undo'ing setting...nothing seemed to work. About 2 days later I come across an article about tcp offload engine...great idea. Let me check this then. I opened the page in the browser and hit netstat -t. Scrolled down all the way to the entries belonging to my connection and I saw the status Offloaded. Went after that on a working server, did the same steps and checked the results: Inhost. Hmm...so on the servers where the connections to the web server are offloaded the page doesn't display. Let me disable TOE then. Ran Netsh int ip set chimney DISABLED. Reloaded the page...it worked. Nice...
So..what was it? I then came across this which I think says it all. (check the two linked posts from this article as well)
http://msmvps.com/blogs/thenakedmvp/archive/2007/01/06/how-broadcom-and-dell-wasted-three-days-of-my-time.aspx
Result...a bit nervous when I see broadcom nics...:)
This is funny :)))
ReplyDelete