Hello, everyone.

In order to accommodate several research and conference deadlines that were brought to our attention, we have adjusted the maintenance window to Wednesday, November 19 at 5:00 am.

Thank you,
CS Staff

—– Original Message —–
From: csstaff@cs.princeton.edu
To: downtime@lists.cs.princeton.edu
Sent: Monday, November 3, 2025 9:24:13 AM
Subject: [downtime] CS Storage Maintenance, Wednesday, November 12, 2025, 09:00-15:00

Date: Wednesday, November 12, 2025 (09:00-15:00)

Who is affected:
All users of the CS department storage or computing facilities.

What is happening:
We are upgrading our storage operating system, which requires the
components of the CS storage system to be rebooted. This upgrade is
expected to be mostly non-disruptive, and department services will continue
as usual, but sporadic moments of interruption may be noticeable at times.

All services that depend upon access to storage share the same minor risk
of performance hiccups for some periods during this window, including cycle
servers, ionic and neuronic clusters, web content, home directories, CIFS,
etc.

Why is it happening:
In July, we identified that a filesystem bug affecting critical shared
storage was causing intermittent failures of some of our web services. This
problem, which occasionally restricted required access and disrupted site
availability, and had no viable workaround, has been under review while we
awaited an official fix from Dell. During the upcoming upgrade, we will
apply the newly released OneFS patch to resolve this issue and enhance
overall service reliability.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Wednesday, November 12, 2025 (09:00-15:00)

Who is affected:
All users of the CS department storage or computing facilities.

What is happening:
We are upgrading our storage operating system, which requires the
components of the CS storage system to be rebooted. This upgrade is
expected to be mostly non-disruptive, and department services will continue
as usual, but sporadic moments of interruption may be noticeable at times.

All services that depend upon access to storage share the same minor risk
of performance hiccups for some periods during this window, including cycle
servers, ionic and neuronic clusters, web content, home directories, CIFS,
etc.

Why is it happening:
In July, we identified that a filesystem bug affecting critical shared
storage was causing intermittent failures of some of our web services. This
problem, which occasionally restricted required access and disrupted site
availability, and had no viable workaround, has been under review while we
awaited an official fix from Dell. During the upcoming upgrade, we will
apply the newly released OneFS patch to resolve this issue and enhance
overall service reliability.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Tuesday, August 5, 2025 (06:00-11:00)

Who is affected:
All users of the CS department storage or computing facilities.

What is happening:
We are upgrading our storage operating system, which requires the
components of the CS storage system to be rebooted. This upgrade is
expected to be mostly non-disruptive, and department services will continue
as usual, but sporadic moments of interruption may be noticeable at times.

All services that depend upon access to storage share the same minor risk
of performance hiccups for some periods during this window, including cycle
servers, ionic and neuronic clusters, web content, home directories, CIFS,
etc.

Why is it happening:
An upgrade is required to address a bug that is causing unreliable snapshot
expiration, generating unnecessary costs.

While we do not anticipate any extended service outages, you may find that
there are momentary interruptions, and some connections (especially
CIFS/SMB connections) may need to be reestablished.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Tuesday, July 15, 2025 (07:00-15:00)

Who is affected:
All users of the CS Department Beowulf high performance computing clusters,
known as ionic and neuronic.

All users of the CS Staff-managed public login systems, including the
cycles, courselab, and armlab systems.

What is happening:
CS Staff will upgrade the Ionic and Neuronic clusters, as well as the
Cycles, Courselab, and Armlab systems, to the latest Red Hat 9
distribution.

In addition, the cluster management and job scheduling system, Slurm, along
with its database, will also be upgraded. No data loss is anticipated.

Why is it happening:
This is part of the routine maintenance and will bring newer versions of
installed tools and software.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Tuesday, July 1, 2025 (06:00-16:00)

Who is affected:
All users of the CS department storage or computing facilities.

What is happening:
We are upgrading our storage operating system, which requires the
components of the CS storage system to be rebooted. This upgrade is
expected to be mostly non-disruptive, and department services will continue
as usual, but sporadic moments of interruption may be noticeable at times.

All services that depend upon access to storage share the same minor risk
of performance hiccups for some periods during this window, including cycle
servers, ionic and neuronic clusters, web content, home directories, CIFS,
etc.

Why is it happening:
Our current operating system has reached end-of-life status and needs an
upgrade.

While we do not anticipate any extended service outages, you may find that
there are momentary interruptions, and some connections (especially
CIFS/SMB connections) may need to be reestablished.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Monday, June 9, 2025 (07:30-08:00)

Who is affected:
All users of CS Department networked services, including email, web
service, and HPC computing.

What is happening:
Critical backbone switches will be upgraded to newer firmware and
rebooted.

We expect no lasting negative impacts from this maintenance.

Why is it happening:
Firmware upgrades are necessary on a few devices to address bugs that are
slowing progress on our Summer project list.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

Date: Thursday, March 27, 2025 (07:00-08:00)

Who is affected:
Users of CS Department IMAP/Pop/Webmail/SMTP services

What is happening:
Sometime during this window, CS email services will experience a brief
outage while the underlying LDAP service is reconfigured. The actual outage
time is expected to be less than 5 minutes, but the larger window is
scheduled to allow for the unexpected.

Why is it happening:
This outage will allow us to begin the longer process of updating our
Zimbra mail servers to the latest available software release. This is a
necessary first step that is intended to avert a much longer outage later
in the process, by enabling a live upgrade of the rest of the system.

No visible changes are anticipated until after the end of the semester, but
minor systems will be updated before that time to prepare for the major
upgrade in the Summer.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff