Author name: jrc

Who is affected:
All users of the CS Department Network

What is happening:
The Department\’s primary border firewall will be replaced with a new firewall device. During the replacement, connections from the CS Department to the main campus, the internet, or to portions of the CS Department network which are outside our firewall, will be interrupted. However, as OIT will also be performing router maintenance between 05:00 and 07:00 (see earlier announcement), we don\’t anticipate this work causing much additional impact.

Why is it happening:
This is the second step of our three-step process to provide 10Gbps networking from our network core out to the main campus, and, by extension, the internet. This upgrade will give our firewall the internal capacity to support 10Gbps connections.

Due to technical complications, the decision was made to postpone the work in order to minimize the risk of problems. Those issues having been worked out, we have rescheduled this maintenance for Monday, August 12, 2013.

Who is affected:
Potentially, all users of CS Department computing and storage infrastructure.

What is happening:
On Thursday, the storage cluster that provides file service for all CS Department public services will undergo upgrades to its network interfaces.

During this maintenance, connections to Samba/CIFS file shares will experience occasional, temporary interruptions. Users may need to re-establish connections to the file share in some cases.

We DO NOT anticipate actual downtime for other services, but since all maintenance involves the risk of broader problems, this announcement serves as a heads-up. In the event of problems with this maintenance, impacts could range up to outages of ALL CS Department public services that rely on the storage cluster (file service, public login hosts, e-mail, web service, etc). Note that wired and wireless networking connectivity will not be affected.

Why is it happening:
This upgrade will provide 10Gbit network connectivity to our storage cluster, getting us one step closer to end-to-end 10Gbit capability for our services.

We are working to resolve an issue with DropBox. The site is currently down. We will post status updates here.

Update 11:39 AM – DropBox is now working again. Please let us know if you have any issues.

The main CS website, virtual websites, wiki and CS Guide are currently down. We are working to correct the problem.

UPDATE 10:06am: we believe we have isolated the issue and continue to correct the problem.

UPDATE 11:03am: all systems are back up and working as expected.

From http://helpdesk.princeton.edu/outages/view.plx?ID=3906

\”OIT network administrators have informed the Help Desk that the OIT provided network is experiencing problems acccessing resources off of the Princeton campus. This includes all internet websites as well as mail resources. Connections to intranet resources (file shares, OIT mail servers, Princeton websites) will all still work from on campus connections. OIT administrators are working to resolve this issue and more information will be provided here as it becomes avaiable.\”

As of 4:39pm today both the CS and HPCRC external fiber connections are currently down. We are working with OIT to restore the connections and will post updates as we know more. All external traffic outside of the CS network will not work because the fiber connections are down.

UPDATE 8:02pm: we now have the external connection restored to the main CS router and have connectivity to the outside from the CS department. We still have no connections to HPCRC.

UPDATE 8:36pm: we now have the HPCRC fiber connection restored.

UPDATE 8:40pm: All network connections have been restored and connectivity to the outside should be working as expected.

EPILOG 9/26/2012 11:21am: Root cause may have been related to renovation at another site where our fiber is routed.

– CS Staff

We are still having trouble with the storage array that hosts our virtual infrastructure. We had a short outage on the CS websites but those sites are all back up again as of 12:35pm. Here is a list of things that are currently down:

1) The submission server that runs the check submit script on dropbox.cs
2) opus – the public server

We will post updates to this page as more information becomes available.

UPDATE 1:32pm: The submission server that runs the check submit script on dropbox.cs is now online again.

UPDATE 1:33pm: opus is now online again.

As of 12:12am, the main web pages are down. We are investigating.

UPDATE 9:35am: The filesystem for the webserver is corrupt and is in the process of being repaired.

UPDATE 10:00am: The main web pages are now online again.

UPDATE 10:10am: Tux is still down please use Opus until it can be restored.

2:20pm: Our e-mail service is currently down. We are working with the vendor to correct the problem.

2:30pm: E-mail is operating again.

2:45pm: The ProofPoint server is currently not blocking Spam e-mail we are working with the vendor to correct the issue.

3:30pm: The ProofPoint server is now blocking Spam e-mail again.

3:30pm: E-mail is operating normally.