The Case of the missing 5 hours

Animated_Spiral_Clock_with_2_pointers_by_Robbert_van_der_SteegI had some Windows 2008 R2 servers in Amazon AWS EC2.  To save some money, they were turned off when they weren’t needed.   I noticed when I did boot them that they had some time issues apparently jumping from Eastern US time to UTC time for a while before switching back.

It seems when you search for time issues, specifically when you have a *nix Host Operating System set to UTC and a Windows guest OS set to a local timezone people will link you to the “RealTimeIsUniversal” registry key.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation\RealTimeIsUniversal = 1  REG_DWORD

The problem is, that registry key was already set.

Further searching brought me to Amazon’s article about setting the time for a Windows OS.

This had a couple of suggestions.  To make sure that KB2800213 and KB2922223 are installed.   After some searching it turned out that KB2800213 was superseded by KB2922223.   Also KB2922223 was already installed.

Checking the Windows Event Log found the time was changed by the Citrix Tools for Virtual Machines service.   “C:\Program Files (x86)\Citrix\XenTools\XenGuestAgent.exe”

I verified that this service was causing the issue by restarting just the service.  Sure enough, the time changed to UTC.  Then when I opened up time in Windows and had it check against the NTP server, it changed back to local time.

To resolve the problem, I upgraded the Amazon EC2 Paravirtual Driver.   This had a prerequisite to update EC2Config.

With a solution found and tested on one server, I turned over the resolution on the other servers to the System Administrators.

Incorrect time impacts security logs and any subsequent troubleshooting or investigation.  According to Amazon, issues like this can cause problem with DHCP leases.   There can any number of unknown application problems.   I expect Kerberos wouldn’t be very happy either.