All

OIT Unplanned Outage

OIT Unplanned Outage

Thursday, March 13, 2025

The Office of Information Technology is investigating a service disruption affecting several applications and services. At this time, faculty, staff, researchers and students may have difficulty accessing WiFi and signing-in to various applications and services.

Updates are planned every 30 minutes or as information becomes available.

Due to the disruption to campus WiFi, please use mobile phones to dial 911 or to reach emergency services.

Updates will be provided as information becomes available.

Impacted services include, but not limited to: Eduroam WiFi, PeopleSoft (TigerHub, Self-Service), Canvas, ServiceNow, Listserv, GlobalProtect VPN, Prime, Virtual Machines.

Outage Status

Investigating

Updates

9:09 p.m.

The IT disruption affecting several Princeton University services has been resolved. The eduroam WiFi network and other services are now available.

If you continue to experience any issues, please contact the Service Desk via live chatemail or by opening a ticket.

We apologize for the inconvenience caused during this time and appreciate your patience.

8:10 p.m.

OIT has identified an issue related to network equipment and has implemented a fix. We are monitoring progress to confirm service has been restored.

Campus Update: March 13, 5:20 p.m.

The response to the ongoing OIT disruption continues. OIT is engaging with the respective vendor(s) and continues to investigate. 

Many Princeton University applications and services are unavailable. At this time, it may not be possible to use the single-sign-on service to sign-in to applications. Campus WiFi via the eduroam network is also affected.

Critical life safety systems such as the fire alarm system and 911 remain operational, but the OIT team has identified that some desk phones are no longer connected to the network. For that reason, the University recommends mobile phones to reach 911 or emergency services.

Additional personnel from Public Safety and Facilities will be on campus until the outages are resolved.

Due to the unavailability of some systems related to remotely monitoring building access, the following actions will be taken this evening:

  • Administrative buildings were manually locked at 5 pm. Card access is disabled.
  • Academic and research buildings will be manually locked at 8:30 pm.  In the meantime, local alarm systems such as fume hoods and refrigeration alarms may not work in these buildings. After 8:30 pm, these buildings should be vacated. Only emergency access will be permitted after this time.
  • Residence halls will remain on their normal locking schedule. Students should make sure to bring their TigerCards when leaving their buildings as systems enabling temporary cards are not functioning.  
  • Firestone Library circulation will remain open until 9 pm and the study areas will be open until 11 pm. Firestone Special Collections, the Milberg Gallery, and the Cotsen Children’s Library are closed for the remainder of the day.
  • Campus recreational facilities will remain open as scheduled.

5:10 p.m.

Our team has implemented a temporary workaround, enabling off-campus sign-in access to cloud-based solutions, such as Zoom, Canvas, GlobalProtect VPN, and ServiceNow.

3:30 p.m.

Investigation into the disruption continues. The issue may be related to hardware that controls internet traffic. OIT is engaging with the respective vendor(s) and continues to investigate.

2:10 p.m.

Our team has also identified that some desk phones are no longer connected to the network. For that reason, we recommend using mobile phones when dialing 911 or reaching emergency services. 

1:43 p.m.

The data center cooling has been restored. It may take 6-12 hours for Research Computing storage to come back online.

1:22 p.m.

OIT is also investigating overheating equipment in the data center. At this time, Research Computing equipment is being shut down. Additional updates to follow. 

1 p.m.

Investigation into the disruption continues. The issue may be related to hardware that controls internet traffic. OIT is engaging with the respective vendor(s) and continue to investigate.

12:30 p.m.

Technicians are currently restarting various systems and equipment as we work to identify the root cause. We are working to restore service as soon as possible. 

12 p.m.

OIT continues to investigate the root cause of the service disruption and is working to restore service as soon as possible. 

11:20 a.m.

OIT continues to investigate the root cause of the service disruption and is working to restore service as soon as possible. 

10:30 a.m.

The Office of Information Technology is investigating a service disruption affecting several applications and services. Additional communications are expected as impact and restoration information becomes available.

Many Princeton University applications and services are unavailable. At this time, it may not be possible to use the single-sign-on service to sign-in to applications.

Updates from OIT will be posted at https://oit.princeton.edu/oit-unplanned-outage 

OIT Unplanned Outage Read More »

CS Cycles/Ionic Downtime, Tuesday, March 11, 2025, 07:00-12:00

Date: Tuesday, March 11, 2025 (07:00-12:00)

Who is affected:
All users of the CS Department Beowulf high performance computing cluster
known as ionic.

All users of the CS Staff-managed public login systems, including the
cycles, courselab, and armlab systems.

What is happening:
CS Staff will upgrade the Ionic cluster as well as Cycles, Courselab, and
Armlab systems to the latest Redhat 9 distribution.

Additionally, MATLAB configurations will be updated. Please review the CS
Guide for new instructions:
csguide.cs.princeton.edu/software/matlab

Why is it happening:
This is part of the routine maintenance and will bring newer versions of
installed tools and software.

MATLAB changes will allow for us to have multiple versions of the software
available.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Cycles/Ionic Downtime, Tuesday, March 11, 2025, 07:00-12:00 Read More »

CS Cycles/Ionic/Neuronic System Downtime, Tuesday, January 7, 2025, 07:00-15:00

Date: Tuesday, January 7, 2025 (07:00-15:00)

Who is affected:
All users of the CS Department Beowulf high performance computing clusters,
known as ionic and neuronic.

All users of the CS Staff-managed public login systems, including the
cycles, courselab, and armlab systems.

What is happening:
Ionic and neuronic nodes will have Nvidia, CUDA, and kernel drivers updated
to fix GPU-related failures. In addition, cluster management and job
scheduling system slurm and its database will be upgraded. No data loss is
anticipated. After the upgrade, machines will be rebooted.

Cycles, courselab, and armlab machines will be rebooted during this window
to clear some defunct user processes interfering with research work.

Why is it happening:
Ionic nodes are experiencing various GPU-related failures. To address these
problems, we will be updating Nvidia, CUDA, and kernel modules.

Additionally, some user processes have entered a defunct state, hindering
research activities. To resolve this, a system reboot is necessary to clear
these processes.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Cycles/Ionic/Neuronic System Downtime, Tuesday, January 7, 2025, 07:00-15:00 Read More »

CS Cycles/Ionic/Neuronic System Downtime, Tuesday, July 2, 2024, 06:00-17:00

Date: Tuesday, July 2, 2024 (06:00-17:00)

Who is affected:
All users of the CS Department Beowulf high performance computing cluster,
known as ionic.

All users of the CS Staff-managed public login systems, including the
cycles, courselab, and armlab systems.

What is happening:
During this window, all CS managed systems (cycles, ionic, neuronic,
courselab and armlab) will be upgraded to the latest Red Hat Operating
System – 9.4. In addition, cluster management and job scheduling system
slurm and its database will be upgraded. No data loss is anticipated.

SPECIAL NOTE:
As we are reloading the Linux servers, all crontabs will be deleted. If you
have crontabs that you wish to persist, you will need to back up your
crontabs before the downtime, and restore them after.

In addition, all local disk storage will be wiped, thus resulting in a loss
of any data stored in the /scratch partition. If you have data in /scratch
that needs to survive the reload, please ensure it is copied somewhere safe
before the start of the maintenance.

Why is it happening:
This is part of regular maintenance to keep systems up-to-date.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Cycles/Ionic/Neuronic System Downtime, Tuesday, July 2, 2024, 06:00-17:00 Read More »

CS Mailman Upgrade, Monday, June 10, 2024, 07:00-10:00

Date: Monday, June 10, 2024 (07:00-10:00)

Who is affected:
All email recipients of the CS mailing lists.

What is happening:
CS Staff will upgrade the CS mailing list server as well as the Mailman
Suite to the latest version.

The web interface for the list server will undergo significant changes.

We do not expect any loss of data or mailing lists configurations.

Why is it happening:
Mailman will be upgraded from version 2.1.12 to 3.3.9.

This is part of maintenance to enhance software performance and security.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Mailman Upgrade, Monday, June 10, 2024, 07:00-10:00 Read More »

2024-01-29 – Unplanned Outage

Several services are suffering unplanned outage this morning, including DNS and Web services. At this time, staff are aware, en route, and looking into the issues. More updates will be published as we learn more.

07:50 Update – A problem was located and mitigated with the DNS servers. All services should be returning to normal at this time.
08:10 Update – We are still having issues with the CS DNS servers. We are still working on the issue.
08:59 Update – We are still working on the CS DNS issues. You can using the wireless EDUROAM network to connect to things outside CS.
09:38 Update – We have tracked down the issue for the CS DNS server and things should start returning to normal. The CS clusters are currently offline until we can track down an issue.

10:02 Update – The clusters are back online. All services should be returned to normal.

2024-01-29 – Unplanned Outage Read More »

CS Database Downtime, Monday, January 8, 2024, 07:00-10:00

Hello, everyone.

Reminder for the upcoming scheduled maintenance.

Thank you, and Happy New Year,
CS Staff

—– Forwarded Message —–
From: “CS Staff” <csstaff@cs.princeton.edu>
To: “downtime” <downtime@lists.cs.princeton.edu>
Sent: Wednesday, December 20, 2023 3:26:21 PM
Subject: [downtime] [rescheduled] CS Database Downtime, Monday, January 8, 2024, 07:00-10:00

Due to an unforeseen scheduling conflict, this downtime, previously
announced for Tuesday, is being rescheduled by one day to Monday,
January 8th, 2024.

Please contact CS Staff if it causes you undue hardship.

Thank you,
CS Staff

—– Original Message —–
From: “csstaff” <csstaff@cs.princeton.edu>
To: “downtime” <downtime@lists.cs.princeton.edu>
Sent: Wednesday, December 20, 2023 11:10:35 AM
Subject: [downtime] CS Database Downtime, Tuesday, January 9, 2024, 07:00-10:00

Date: Tuesday, January 9, 2024 (07:00-10:00)

Who is affected:
All users of the CS Department ”publicdb” database server, including
any dependent web properties and all CS Department Beowulf high-performance
computing cluster users, known as ionic.

All users of CS Department administrative web properties (Dropbox, CS
Guide, the Main website, etc.)

What is happening:
During this window, the ”publicdb” database server will be replaced
with a newer server. All existing MariaDB databases will be migrated to the
new server, so no data loss is anticipated. However, while Slurm jobs will
continue, new jobs cannot start during the migration.

In addition, the database server underlying the administrative systems
will be upgraded and replaced. During the upgrade, all database-dependent
administrative systems will be unavailable. This includes the CS Dropbox
service, the main website, the CS Guide, ADM, and any content feeds
provided by CS Staff.

Why is it happening:
The old servers running MariaDB 10.1.24 will be upgraded to newer ones
running MariaDB 10.5.22.

phpMyadmin web interface will be upgraded from version 4.4.14 to 5.2.1.

This is part of regular maintenance to enhance system performance and
security.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Database Downtime, Monday, January 8, 2024, 07:00-10:00 Read More »

[rescheduled] CS Database Downtime, Monday, January 8, 2024, 07:00-10:00

Due to an unforeseen scheduling conflict, this downtime, previously
announced for Tuesday, is being rescheduled by one day to Monday,
January 8th, 2024.

Please contact CS Staff if it causes you undue hardship.

Thank you,
CS Staff

—– Original Message —–
From: “csstaff” <csstaff@cs.princeton.edu>
To: “downtime” <downtime@lists.cs.princeton.edu>
Sent: Wednesday, December 20, 2023 11:10:35 AM
Subject: [downtime] CS Database Downtime, Tuesday, January 9, 2024, 07:00-10:00

Date: Tuesday, January 9, 2024 (07:00-10:00)

Who is affected:
All users of the CS Department ”publicdb” database server, including
any dependent web properties and all CS Department Beowulf high-performance
computing cluster users, known as ionic.

All users of CS Department administrative web properties (Dropbox, CS
Guide, the Main website, etc.)

What is happening:
During this window, the ”publicdb” database server will be replaced
with a newer server. All existing MariaDB databases will be migrated to the
new server, so no data loss is anticipated. However, while Slurm jobs will
continue, new jobs cannot start during the migration.

In addition, the database server underlying the administrative systems
will be upgraded and replaced. During the upgrade, all database-dependent
administrative systems will be unavailable. This includes the CS Dropbox
service, the main website, the CS Guide, ADM, and any content feeds
provided by CS Staff.

Why is it happening:
The old servers running MariaDB 10.1.24 will be upgraded to newer ones
running MariaDB 10.5.22.

phpMyadmin web interface will be upgraded from version 4.4.14 to 5.2.1.

This is part of regular maintenance to enhance system performance and
security.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

[rescheduled] CS Database Downtime, Monday, January 8, 2024, 07:00-10:00 Read More »

CS Database Downtime, Tuesday, January 9, 2024, 07:00-10:00

Date: Tuesday, January 9, 2024 (07:00-10:00)

Who is affected:
All users of the CS Department ”publicdb” database server, including
any dependent web properties and all CS Department Beowulf high-performance
computing cluster users, known as ionic.

All users of CS Department administrative web properties (Dropbox, CS
Guide, the Main website, etc.)

What is happening:
During this window, the ”publicdb” database server will be replaced
with a newer server. All existing MariaDB databases will be migrated to the
new server, so no data loss is anticipated. However, while Slurm jobs will
continue, new jobs cannot start during the migration.

In addition, the database server underlying the administrative systems
will be upgraded and replaced. During the upgrade, all database-dependent
administrative systems will be unavailable. This includes the CS Dropbox
service, the main website, the CS Guide, ADM, and any content feeds
provided by CS Staff.

Why is it happening:
The old servers running MariaDB 10.1.24 will be upgraded to newer ones
running MariaDB 10.5.22.

phpMyadmin web interface will be upgraded from version 4.4.14 to 5.2.1.

This is part of regular maintenance to enhance system performance and
security.

We will post updates to the status page: www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff

CS Database Downtime, Tuesday, January 9, 2024, 07:00-10:00 Read More »

Scroll to Top