Roger's Information Security Blog

On Thursday we rolled out the Blue Coat web filter to the company. It was a bit more sudden than I had planned. I had planned to roll out slowly over a week and a half (still kind of quick), with the goal to be done by January 28th. Our Websense license expired on January 31st and I wanted to be done before then.
Unfortunately a company board meeting interfered in my plans as we were not allowed to roll out anything while they were in town. I was told that after license expiration, Websense would continue to filter, but not get any new updates. This was acceptable to the Director, so we pushed back the Blue Coat with a new goal of February 5th.
As it turned out at 11 pm on January 31st Websense stopped filtering. So on the morning of the 1st we rolled out Blue Coat to the entire company and disabled the Websense.
That afternoon, I received a report about slow FTP to our DMZ. I did some testing and the speed seemed reasonable. However, that wasn’t the end of it.
The next morning before I got to work, I had a voicemail about other people having trouble opening Flash and downloading large pdf files from the DMZ. It came to a head when another Director in our company emailed our Director claiming it was impossible to get any work done. The Director wanted to turn it off all together, but I felt that this would not provide a good troubleshooting environment. We had used Blue Coat within our department with no reported problems of this nature, so we needed to have the systems under close to a full load. A compromise was reached by removing the subnets of the complainers from filtering.
The network guys had already opened up cases with Cisco and Blue Coat. Everything appeared to be normal. The configuration was acceptable to the support people. The CPU and RAM seemed fine.
I checked the antivirus appliance to make sure it wasn’t running out of threads, but everything was well within spec there. Next, I checked the Blue Coat forums to see what other people had to say about this problem.
A quick check found that the most likely cause was mismatched speed or duplex issues on the switch. I called one of the network guys as 1:45 to ask him to check into that. I kept searching to get an idea of other things to try (and also establish some speed test baselines). A speed test reporting downloads of 800 kbps. Which is ludicrous when we have a 25 meg pipe.
We checked into the switch and found it wasn’t quite as intelligent as we had expected. We didn’t have the capability to hard code the connections to a specific speed and duplex value. We did however see the collision light was occurring on the connection to the core router. I should mention the switch is 10/100/1000 and the router interface is 100. We checked the router and saw the same errors there. The connection was already hardcoded to 100 Full so the network guys changed that to auto. That’s the opposite of what you normally do when you have this problem. The port negotiated 100 Full and the errors went away.
I performed a few speed tests and found that web requests were benchmarked 10-100 times faster. The speed test now reported something crazy like 80 meg down (due to the antivirus or caching I suspect). But it is at least and apples to apples comparison with the 800 kb test.
So all is solved. The problem was not with the Blue Coat, but I did take a few body blows and get a black eye.