October 2008

On Thursday, October 30, 2008, we will have an emergency downtime from 4:00am to 8:00am EDT.

During this time we will be shutting down our infrastructure so that we can apply updates to the file server. The vendor has requested that we apply these updates so that we can have the very latest configuration while we continue to diagnose the ongoing performance and connectivity issues.

Update: We applied the patches and have escalated the issue with the vendor. We are still seeing performance and connectivity problems with the file server. These problems impact project space, the cvs server, and the web site.

Update 11/1, 12:15pm: We continue to have issues. Our vendor has assigned personnel to work the problem over the weekend.

On Monday, October 27, 2008, we will have an emergency downtime from 4:00am to 8:00am EDT.

During this time we will make a change to the file server configuration that, we believe, will correct the recent performance/connectivity problems.

Update: The changes did not correct the issue; we are working with the vendor to identify and fix the problem.

On Tuesday, October 28, 2008, we will have a scheduled downtime from 4:00am to 8:00am EDT 8:00am to 8:30am EDT.

Scheduled work includes:

• Replace the penguins cycle servers: The three old 32-bit Intel-based machines (tux, opus, willy) will be replaced with two 64-bit AMD-based machines. The two new machines will take the names opus and tux. We will deprecate the name willy.
• OS updates for some of our infrastructure machines

SPECIAL NOTE: As we are replacing the hardware for the Linux cycle servers, all user crontabs on these machines will be deleted. You will need to backup your crontabs before the downtime, and restore them after the downtime.

Why is it happening:

• The penguins machines are being replaced because they are outdated and have become difficult to support.
• The infrastructure machines are getting OS patches as part of normal maintenance.

Update 10/27/08: Because we had an emergency downtime this morning, we were able to do most of the work scheduled above. We are now having a short downtime to swap the penguins cycle servers. Other parts of the department infrastructure will remain operational.