SRI AIC Facilities @SRI_AIC_Action

Network at our ISP has problems since 11:30. We are waiting for them to fix it. Backup network up, but web access seems broken from our VPN.

Unable to talk to our ISP. On site looking at it.

ICS VPN should be stable.

About to reboot ICS VPN per Doug's note re upgrade. Expect it to be unusable until 6:45PM. MT

Lupe got us working again. ssh from off site is up.

ssh from off site is down. Appears file server Tarpit needs reboot, and it took down SSH server. Someone will have to go on site to reboot Tarpit

Corporate IT is having problems that are affecting our division. They are working on it

Power failed at 17:36. Report is power is back about 2 hours later. We will be bringing machines up tonight.

Power failed at 6:30AM Saturday & restored about 10:45AM. Staff onsite restoring services.

Power failure early Saturday resulted in corruption of several file systems. Restoration underway. No estimate of full service yet.

Power failed for an hour. Main fileserver still down. DG is on his way in.

Mabry wants to update router & switch software tonight (Sunday) or Monday, but backups are running. If done, there will be multiple short network interruptions as devices reboot. x5021

Outage delayed 1:45. ICS discovered a configuration issue that leaves us dependent on UPSes (basically batteries) for our switches in order for our minimal site to stay up during the outage. We may go down sometime during the day.

Resolved issue of unexpected power outage (our hosts didn't go down, but I suspect ISP's did). We are configured for Saturday as announced.

We're down because of a unexpected power failure. People on site trying to diagnose.

Mail system died early AM. Locked up other systems. Disk is being checked. May be up about 8:30AM. MT; DG

Lost main file server, due to a failed disk in a raid. It will take ~12 hrs to recover. Expected down to at least noon Sunday. Maybe later

Power has been restored and appears is stable. We're bringing systems back online. Email is up, so further updates will happen in email.

ICS and AIC Systems are down due to power problems. No info available on how long outage will last. Updates will follow when info available.

Rebooting a couple of network switches. Access to outside world will be down for 5 min. CSL will be down 5 min.