This post originated from an RSS feed registered with .NET Buzz by Brendan Tompkins. Original Post: An Incorrect Server Time can Crash Your Applications! Feed Title: Brendan Tompkins Feed URL: /error.htm?aspxerrorpath=/blogs/brendan.tompkins/Rss.aspx Feed Description: Blog First. Ask Questions Later. Latest .NET Buzz Posts Latest .NET Buzz Posts by Brendan Tompkins Latest Posts From Brendan Tompkins

I was going to title this post â€œAn Incorrect Server Time can Crash Your ASP.NET 2.0 Applications!” but I thought that would be too sensational… But, it’s true, my ASP.NET 2.0 apps pooped out due to a server time being off. .  Here’s what happened:

The other day we started seeing one of our two web servers crashing.  We recently deployed our new public site onto ASP.NET 2.0, and everything had been working fine up until this point.  The strange thing about this server crashing was that both servers are identical hardware, had the same codebase deployed, and are load balanced with a Cisco CSS hardware.    Why one server would begin to fail and not the other was mysterious.

Well, I went looking in the event log and saw this error, at regular intervals, every 5 minutes. It turns out that after a few of these, the server would crash, and we’d get “Service Unavailable” when browsing to that server.  This is what the event log looked like:

Source: .NET Runtime 2.0 Error Reporting
Event Id: 5000
Description: EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 microsoft.web.services2, P5 2.0.3.0, P6 42177cce, P7 eb1, P8 47, P9 e3hnjzi4pzb2exb3atbffdtvqefg3pm4, P10 NIL.

What a great description, huh? Anyhow, I did some searching and found the following KB article that let me know what was causing the crash:

When an unhandled exception is thrown in a Microsoft ASP.NET-based application that is built on the Microsoft .NET Framework 2.0, the application unexpectedly quits. When this problem occurs, no exception information that you must have to understanding the issue is logged in the Application log.

Okay, nice feature.  Anyhow,  I knew that WSE was somehow involved (due to the web.services2 in the description) and that I had an unhandled exception in code somewhere (sloppy on my part).  I suspected a regular process since the errors were so periodic, and we do indeed have some processes that go off every 5 minutes and do some WSE stuff.  Well, finally with my trusty copy of SysInternal’s DebugView (which will show you Trace.Write, and Debug.Write output) I tracked it down to this error:  A WSE message failed to decrypt because it was “sent in the future.”  Sent in the future? Cool!  No wait, that’s impposible. I checked and sure enough, the server’s time was off by a minute. 

So the time was off, the message wasn’t decrypted, WSE threw an exception, my sloppy code failed to handle it, and bang!  Dead server.. I thought I’d pass this along,  this was the strangest error fix I’ve had in a while!

-Brendan