VC No Comm Event

Score a temporary victory for the machines in Man vs. Machine. A couple of weeks ago, an external event triggered a rebellion of our HP Blade Servers.

Without getting too technical, a bug in the firmware meant that the Virtual Connect network switches were checking out an incorrect IP address when validating management communication. The Virtual Connect Manager apparently decided to stop talking to us when someone in China configured DNS for their new network using that address.

Once that happened, the Virtual Connect Manager started reporting NO_COMM and other disinformation. Just going into the manager and looking to see what was going on made things worse, and eventually rebooting the switch (to clean up the mess) abruptly took down every blade server (even though they are configured for complete redundancy).

(The HP Customer Advisory gives a long list of what can trigger disaster.)

Reminds me of the Uncle Remus Tar Baby story. Starts out with a lack of communication, but quickly becomes hopelessly sticky if you try to find out why and then keep slugging it out.

Daylight Saving Time 2007 (Y2K7)

Congress has decided that we should all get up an hour early tomorrow. I kind of wonder who is making money from this little experiment. The one thing that I have noticed so far is that the summer gasoline prices are coming about three weeks early. And it is just an experiment: the Energy Policy Act said that they would evaluate the "energy savings" and possibly change it back.

We have spent a few months planning and preparing for this computer software change. Every system and application seems to have a different way of computing wall clock time. Windows, Exchange, Java... Kronos, Exchange, PowerChart... All this work patching and upgrading systems and on Monday everyone else (hopefully) will say, "What's the big deal, nothing happened!"

But even yesterday, vendors were giving us notice that their software is not ready. The GE PACS software has their own copy of Java that needs to be patched on workstations. I think that GE was already trying to tell us about this one, but the critical applications that are closest to actual patient care are supported by Clinical Engineering, not IST.

I've recently learned a few things about time zones and Java... but who is Olson?

No N00S is Good N00S

I spent the evening responding to a new virus. This one runs a program called n00s.exe and attacks Symantec AntiVirus on TCP port 2967. Nearly all of the machines in our domain have Symantec AntiVirus, so there is a lot of potential for destruction. Some machines get a message when Symantec's rtvscan crashes, but others get infected and go to the dark side.

Crazy Wild Desktop Day

I knew it was going to be bad when the pager started going off at 7 AM, before I had crawled out of bed. They said that "a lot" of people were getting rebooted for security patches and after reboot, they didn't get their desktop. They also said that they didn't get the "I Accept".

Okay, I dialed in and checked the WSUS server: 1,800 desktops or so updated so far. Not a Good Thing if they were all broken.

"Not getting their desktop" was something that was reported from time to time. Also known as "lost the printer setup" or "lost email profile". When we do automated installs, it is generally by making the machine automatically log in to a special, captive desktop admin account. The account runs a login script that does the install. When a user reported that their desktop is blank, email profile is gone, printers are gone, I had written it off to something failing in the script and the user not knowing that they are logged in to a different account. TAC walked them them through getting it straightened out.

This was not like that. These updates were being done by Automatic Updates. Straight Microsoft code with no automatic login or anything.

The patches were released by Microsoft on Jan. 10 and we applied them to our 30 or so test desktops. No problems in the last couple of weeks. They were scheduled to be released to all desktops last Wednesday, but we had too many other problems going on that day. Finally released then Monday evening. They downloaded to the desktops during the day yesterday, then started installing at 4 AM this morning.

It looks like all of the ones that installed at 4 AM worked without problem. Machines that were powered off overnight were different. The user powered on the machine, Symantec AV kicked off a scheduled weekly scan (on Wednesday by coincidence), the patches started installing, and then the user logged on.

"Can't load profile... NTUSER.DAT file in use." (Was it being scanned?) Windows was kind enough to create a new profile by copying the default user profile... but none of the user's desktop shortcuts, printers, My Documents, etc. were to be seen. Some new profiles were worse than that: Default User\NTUSER.DAT could not be copied because it was also in use. Ugly profiles resulted. (They didn't get the "I Accept", which being interpreted, means the login script didn't run.)

1,880+ out of 3,000+ already installed and the others probably couldn't be stopped because most would already have been downloaded and waiting on the desktop. Vogue la galère. We'll just keep it going.

To shorten a long story... We developed a safe and effective fix to get the users back on their original profiles and "a lot" did not turn into "very many" or "every".