Open Computing Facility Blog

Web Server Downtime

2012-01-25T23:43:00.000-08:00

Tonight, the web server ran out of swap space and is currently experiencing downtime.

Since there are updates pending on the server, rebooting it runs the risk of it not booting up properly afterwards.

Work on the (much more powerful) virtual machine that will become the new web server began a week ago, and staff will meet this Saturday to finish it.

Update Jan 26 8pm: Fixed, but there may be downtime on Saturday as web hosting is migrated to a new server.

Intermittent downtime for maintenance

2011-12-17T15:08:00.000-08:00

On December 16, we rewired parts of our network between 5pm and 7pm, resulting in intermittent unavailability for OCF services. We believe that the end of the semester would be a less intrusive time to make changes, especially since the lab is closed until spring semester.

OCF, as with other student groups in Eshleman Hall, will be moved out of the building in August before it is demolished as part of Bears Initiative. We are aggressively consolidating our hardware (reducing the number of physical servers), since space for our hardware after the move will be limited, and working to prevent these changes from having an adverse effect on performance and reliability. We apologize for the inconvenience.

See also: Lab and printing services uncertain after Spring 2012

Lab and printing services uncertain after Spring 2012

2011-12-07T13:31:00.000-08:00

From: OCF announcements <announce@OCF.Berkeley.EDU>
To: All OCF members
Subject: OCF lab and printing ending after Spring 2012
Date: Wed, Dec 7, 2011 01:31p PST

Hello OCF members,

Last night OCF's volunteer staff unanimously decided to close lab services, including free printing, after Spring 2012.

OCF, as with other student groups in Eshleman Hall, will be moved out of the building in August before it is demolished as part of Bears Initiative. Until new space is obtained elsewhere on campus, we have been allocated temporary space in Hearst Gymnasium basement for the duration of Lower Sproul construction, which is projected to last for at least 3 years.

This area is limited both in space and technical resources, and we feel that it would be a disservice to all OCF members to provide an unreliable and inadequate level of service, in both quality and quantity. We would not be able to fit much of our computing and printing equipment into the space allotted to us, and time constraints limit what can be set up before the fall semester begins.

We understand and share your concern for the OCF. We too are students and members of the OCF. This is a drastic change for the organization, which has had a lab since 1989.

We want to restore the lab and its services as soon as possible, but we must also keep in mind the process of our last two moves (to Heller Lounge in the MLK Student Union, and to our present location in Eshleman). It would be dishonest if we were not to give notice of lab downtime, which has been unavoidable and
excessive during each move. Furthermore, the compressed space and needs of other student groups constrain us more than in the past.

We hope that by shifting resources, we will be able to compensate by expanding our other services, including currently web hosting, disk space, email, and shell accounts, and ensure that they would not be disrupted by a move as they have been before. We are looking at our options for negotiating server space and we will continue hosting websites, including those of student groups. We will better assist other students and student groups as a technical resource given our close proximity in the temporary space, and encourage you to make use of what we would be able to offer.

We will be considering all options, including other space if available, and appreciate any feedback or advice you can offer. We don't want to let you down...

Good luck on finals guys,
OCF staff members,
Eshleman basement

P.S. One of the printers is broken, so printing might be slower than we'd hope, but please, enjoy our lab while it lasts. 9am-9pm hours during dead week and finals, see our website for more info.

You have received this announcement to update you about important changes to the OCF. The Open Computing Facility is an all-volunteer student organization providing free shell accounts, disk space, web hosting, email, and printing from a lab, lounge, and server room in Eshelman Hall.

Update Dec 8: Daily Cal has published an article: Computing facility in Eshleman Hall to close after next semester

Update Dec 17: To clarify, OCF is not going away, but staff felt that, given our constraints, it would not be possible to guarantee a public lab next fall, and misleading if we did not inform members. The post's title has been updated to reflect that lab and printing services are uncertain, since no decision is made in stone, and the constraints which led to our decision can (and hopefully will) change. Sorry about the confusion.

Two years ago OCF completed its 15-month move from Heller Lounge in MLK to the larger space in Eshleman, with excessive lab and server downtime (caused in part by the discovery of asbestos and later a failed pump flooding the new lab, then a pipe burst dropping raw sewage, then a network and later power outage which was prolonged). The current move leaves us two weeks (Aug 1 to Aug 15) to move and bring the lab up in a much smaller space before the semester begins. That also includes getting rid of desktops, servers, and equipment that would not fit in the space, since Moore's Law makes the storage of aging computer hardware uneconomical.

We also hoped that by not committing ourselves to the lab immediately, we would leave ourselves time to ensure that non-lab services, like web hosting, would not be disrupted by the move. During the last move our web hosting, used by many student groups, was down for several months, and the complaints were not pretty. To this day we hear about OCF "unreliability" based on that downtime.

And lastly, we have every intent to restore our lab when it is possible and when we have the logistics of the space worked out. To quote an ex-staff member: "The group's choice to focus on back-end makes a lot of sense given the uncertainty with construction and space."

And yes, this makes us very unhappy.

Networking maintenance downtime

2011-11-18T09:19:00.000-08:00

Downtime is scheduled between 8pm and midnight tonight.

Update 4:00am

We replaced some wiring and networking equipment between 11:30pm and 01:15am. Most services were only briefly affected. Web, mail, SSH, and dependent services were temporarily moved to another switch during the rewiring.

We thank a generous donor for two new gigabit switches.

Networking maintenance

2011-10-31T21:45:00.001-07:00

There will be one day of downtime during the days of November 18-20 for networking maintenance. The anticipated downtime should be much less than one day. The exact date will be confirmed as the day approaches.

Mail service downtime

2011-10-10T03:42:00.001-07:00

Mail service, including SMTP, IMAP, and POP, was unavailable between 2:19am and 3:06am because of transient errors with LDAP (user database). Some mail sent or received during this time may have been incorrectly rejected.

Unexpected downtime, RAID failure

2011-09-21T01:06:00.000-07:00

Taking precautionary measures after RAID failure on our main disk array, we have taken down all services that depend on NFS, including web hosting, mail service, and home directory file access.

We are doing our best by working overnight so that service will be restored as soon as possible.

We apologize for the inconvenience.

Update Thurs Sep 22 05:45am

First, we want to apologize (again!) for the delay. We were hoping to restore service yesterday morning. Things didn't exactly work out...

While we don't want to blame the delay on excuses, we would also like to be straightforward about what is going on.

Last Sunday, our backup server began to fail, and by Tuesday, the largest volume, which contained backups of home and web directories, appeared to be unrecoverable. Data is stored in a RAID volume (meaning it is resilient to a certain number of hard drive failures), so the simultaneous hard drive failures/corruption that would have been required suggest hardware problems with the server itself and not (or not only) the hard drives. To be honest, hard drive or RAID controller failure is not completely unexpected for an aging machine with aging hard drives (the server easily predates all of current staff, so we don't know exactly how old). In 20/20 hindsight, we could have acquired and set up new hardware, but for a machine that hosts backup copies of data and is not directly accessible, the extra expense in time and money did not appear to be worthwhile.

On Tuesday at 10:30pm, the main disk array, which exports most data stored in NFS (home and web directories, and mail folders but not mail inboxes). Because RAID adds redundancy, no data was lost, but redundancy was lost, meaning future failures result in data loss. This is why we then took down all services that can access or edit this data and modified others (like printing) to not depend on it (this will however prevent you from being able to see your print quota, you will need to ask a staff member).

The disk array is in its third year of service, so hardware problems with the server itself are not really expected but not improbable either. However, even the most reliable hard drives, accessed constantly 24/7, can fail. We did not have any hard drives other than those in the disk array with capacities greater than or equal to 1 TB, meaning we could not immediately begin rebuilding the RAID volume (again, in hindsight, this was probably a mistake on our part), let alone any (more expensive) "enterprise-class" 1 TB hard drives as would be proper for (and which are currently used in) the disk array.

On Wednesday morning, we bought a temporary "desktop-class" hard drive. When we mounted the new hard drive in the disk array, it was detected but unrecognized and marked as "bad" on reboot. We tried unsuccessfully to work around the problem. Other hard drives (of smaller capacity; they cannot be used to rebuild the RAID volume) were recognized and usable for other purposes without errors. It seems highly unlikely that a brand new hard drive would be bad, and we could not find any sign of errors when testing and running diagnostics on the hard drive in other machines, so the disk array is suspect, but since it may work with other hard drives, is not clearly at fault either. (There appear to be firmware restrictions on intermixing "desktop class" and "enterprise class" drives.)

On Wednesday evening, as another precautionary measure, we planned out a procedure to replicate the data on other machines so that if another hard drive or the disk array were to fail, we would not have data loss or corruption. To prioritize, we are copying data in alphabetical order from enabled (i.e., not disabled) accounts to another hard drive on the disk array. We will remove this hard drive when done for safekeeping, and also copy the same data over our internal network to another server with a RAID 1 (mirror) setup.

We will try our best to restore service as soon as possible. We don't want to sound deceiving by suggesting a time earlier than what might end up happening, especially since we need to first make sure that existing data is safely backed up. Service downtime through the weekend is not acceptable but it is possible, and depending on any obstacles encountered, the length of downtime could be longer or shorter.

Our Board of Directors (comprised of interested OCF members, volunteer staff and "users" alike) currently meets weekly on Thursday at 6:45pm in the OCF lab. Our next meeting is today, and if you have any advice or comments, related or not to the downtime, we encourage you to attend.

Update Sat Sep 24 06:30pm

The ASUC Auxiliary is closed on weekends, and as a result we won't be able to obtain the package of hard drive replacements that we ordered until Monday, at which point we will be able to rebuild the array and bring services back online. We may be able to mount the disk array read-only before then if the local copy is complete.

Update Sun Sep 25 12:00am

The local copying of non-disabled accounts that was started on Thursday morning is about 90% finished. We're expecting it to be finished by the morning.

Update Sun Sep 25 10:30am

The login and mail servers are now mounting home directories read-only. SSH/SFTP will give you read-only access to your files, IMAP/POP/mutt/webmail will give you read-only access to your mail.

Unfortunately, the ASUC Auxiliary is closed on weekends, and as a result we won't be able to obtain the package of hard drive replacements that we ordered until Monday, at which point we will be able to repair the array and bring all services back up.

Update Sun Sep 25 11:30am

The web server is now serving web pages read-only. This may break some sites that require writing to the home or web directory. For the time being, you can optionally use our error message to give an HTTP 503 Service temporarily unavailable error with an explanation.

Update Mon Sep 26 02:30pm

We obtained and added the new hard drives to the array at 9am this morning. If there are no errors, we expect the resync to be complete by 5pm. We will then mount home directories with full read and write access in the state they were originally.

Update Mon Sep 26 05:15pm

Finally, all services are operational as before. This will hopefully be the last update...

Some incoming mail rejected

2011-09-19T02:43:00.000-07:00

Between September 1 and September 15, some incoming mail was incorrectly rejected as spam. A blacklist incorrectly included all mail servers, raising the score of incoming messages, and causing some high-scoring messages to be rejected at the SMTP stage. See Debian bug #641227.

Since these messages were rejected before their contents could be accepted, they cannot be recovered.

Email senders should have received a non-delivery report (bounce message) that stated:

Client host rejected: Mail appeared to be SPAM or forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settins or to get removed from DNSBLs;

Client host rejected: temporarily blocked because of previous errors - retrying too fast.

We apologize for the inconvenience.

Unanticipated Downtime

2011-09-18T00:45:00.000-07:00

Due to some unanticipated network issues on the nfs server, all servers that depend on nfs are down. We apologize for the inconvenience.

Update 1:40am: All services have been restored on all servers.
Update 11:20am: An unrelated issue was denying IMAP/POP/SMTP authentication, fixed.

Login server restarted

2011-09-10T19:47:00.000-07:00

The primary login server (tsunami, ssh.OCF) was restarted at 4:00am PDT to apply a kernel security update as part of scheduled maintenance.

MySQL downtime

2011-08-19T01:55:00.000-07:00

MySQL was unavailable between 9:08p and 1:52a for unanticipated maintenance.

Service downtime

2011-08-17T20:53:00.000-07:00

User information hosted in LDAP was unavailable from 4p to 9p PDT. Web hosting, email, and SSH/SFTP services were intermittently interrupted. Mail service was also taken down from 10p to 11p PDT for maintenance. We apologize for the inconvenience.

Although no user data was affected, some email messages sent to OCF accounts during the service interruption may have been rejected and bounced to the sender.

Login server restarted

2011-08-14T22:51:00.000-07:00

The login server (tsunami) was restarted at 10:14 PM for unanticipated maintenance. Zabbix, an uptime monitoring software, was not communicating with tsunami properly. The iptables firewall were flushed, causing network connectivity to be disrupted briefly. We apologize for the inconvenience.

Service downtime

2011-06-25T12:19:00.000-07:00

There was intermittent down time between 11a-12:30p PDT due to software updates.

Login server and wiki downtime

2011-06-13T15:32:00.000-07:00

Secure shell login and access to the wiki were unavailable Monday morning and part of the afternoon. These services are hosted on virtual machines whose hypervisor (once again) had problems accessing its SCSI drives. The machine was reset, and is running normally again.

Additionally, the switch connecting the alternate login machine (pileup) to the OCF had been switched off. This has also been remedied.

As was the case last time, no user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.

Internet outage

2011-06-05T16:03:00.000-07:00

The gateway which our uplink is connected to was offline during the following periods this morning:

06:41a - 06:43a
06:50a - 06:56a
07:06a - 07:43a

The OCF was not externally accessible during this time. Since the gateway is operated by the campus IST department, we cannot verify the cause of the outage.

We suspect this was related to a planned power outage at the IST data center, although the campus network which we are connected to was not planned to be affected.

Issues sending mail through webmail and webserver

2011-06-01T14:10:00.000-07:00

It was discovered that users could not send mail through webmail or the webserver for the past month.

If you sent an email from OCF webmail between March 25 and April 31, please confirm that it was actually received.

You will not have received emails from your website in the past 4 weeks.

These issues have now been fixed. We apologize for any inconvenience and for the delay in discovering them.

Off-peak mail maintenance

2011-05-12T06:01:00.000-07:00

The mail server was taken down from 04:39a to 05:55a today for maintenance and cleanup. SSH and SFTP logins may have failed during that time. All services should now be operational.

Login server, webmail, and wiki down

2011-05-03T13:49:00.000-07:00

The login server (tsunami aka ssh.OCF), webmail, and wiki are not accessible since 1:01pm today.

These services are provided by virtual machines. The physical hypervisor in which the virtual machines are run has issues accessing its data store, and we are investigating SCSI failure with the hard drives and controller.

No user data is stored on these machines, so your data is not affected. We apologize for the inconvenience.

If you need SFTP/SSH access, you may temporarily connect to one of the desktops, pileup.OCF.Berkeley.EDU, instead of connecting to ssh.OCF.Berkeley.EDU which is currently unavailable.

UPDATE 6:45p: The login server (SFTP/SSH) is back up.

UPDATE 6:52p: Webmail and wiki are back up.

OCF internet connection down

2011-04-24T05:46:00.000-07:00

The campus router which provides our Internet connection went down at 5:40am. All services are down.

UPDATE 5:45am: Router is online. Services should be up again.

Mail server rebuilt

2011-04-17T13:07:00.000-07:00

As the previous post notes, our primary mail server (war.OCF aka mail.OCF) was down on Friday and parts of Thursday and Saturday.

At around 3:30a Thursday the server began to slow down on processing mail, and by 3:15p Friday the server had crashed. We could not turn the server back on, suspecting file corruption on the SPARC hypervisor the Solaris LDOM runs on.

We began rebuilding the mail server with Debian 6 on physical x86 hardware Friday night, and after 9am Saturday services were coming up. IMAP, POP, SMTP, and SMTPAUTH should all be up as of 12:30p Saturday.

We're using a new SSL certificate on the new mail server.

Since we're still running parts of mail on a mix of other machines running Solaris and Debian 5, this is a temporary solution. Maintenance work should be expected soon. We apologize for the inconvenience.

As always, you can email any suggestions or comments to staff@OCF.Berkeley.EDU.

UPDATE Sat 5:15p: Our incoming mail server (sandstorm.OCF) has a backlog of emails to process. There may be delays in receiving emails.

UPDATE Sat 11:45p: special local (slocal) mail delivery mail processing is no longer supported, please update your .forward files accordingly.

UPDATE Tues 11:55p: We've had difficulties with the incoming mail server, so mail may have been temporarily (or possibly in some rare cases, permanently rejected) when sent to an OCF address. We've replaced the Solaris machine with the newly-rebuilt mail server. We apologize for the inconvenience, and hope that there will be no bugs.

Mail Server Down

2011-04-15T15:35:00.000-07:00

The mail server is experiencing some issues and is currently down. We apologize for any inconvenience this may have caused.

Web Server Crash

2011-04-05T15:45:00.000-07:00

The web server (death/www.OCF) had crashed again. We apologize for any inconvenience this may have caused.

UPDATE 12:03am: Fixed, web server is functioning again.

Websites down

2011-03-30T06:53:00.000-07:00

Our webserver ran into troubles with virtualization and crashed at 10:15pm last night. We are working to bring it back up and apologize for the inconvenience. User data should not been affected and is still accessible over SSH.

Update 11:07am: Should be back up.

flood.OCF rebuild

2011-03-24T15:42:00.000-07:00

flood.OCF has been taken offline as the machine is being rebuilt. Users should see no effect as all services running on flood have been transitioned to another server.

Print quotas

2011-03-02T00:42:00.000-08:00

Our print server crashed a week ago and users' print quotas were wiped. A Board of Directors resolution was passed to set each users' quota to 175 pages given that we are in the 7th week of the semester. Users can email staff and will be dealt with on a case-by-case basis.

Login servers

2011-02-10T04:50:00.000-08:00

We are taking apocalypse down for maintenance. Most users should not be affected, however SSHing directly into apocalypse.ocf.berkeley.edu will not work.

To SSH into a login server, use the hostname ocf.berkeley.edu, as specified by our documentation.

Tuesday outage

On an unrelated note, tsunami (our primary login server) was down between 7:00am and 2:40pm on Tuesday, Feb 8. We apologize for the inconvenience.

Update: Tuesday, February 22

apocalypse.OCF has been permanently discontinued as a SPARC login server. We will be discontinuing support for the SPARC architecture.

tsunami is currently our 32-bit x86 login server, and was upgraded on Sunday to Debian 6.0.

OCF Not Open Until Furthur Notice

2011-01-18T23:48:00.000-08:00

Due to unexpected problems with the Windows Server, the Windows lab machines do not work.
We will update you as progress happens.

Update 01/19 at 09:17 PM: We were able to implement a temporary solution so that some of the Windows computers in the lab work (as well as some of the Linux computers). The computers with blue post-it notes definitely do not work. The others worked when last checked. We apologize for the inconvenience.

Update 01/20: The machines previously with blue post-it notes are now working. All computers should be working as before.

Mail scheduled maintenance

2011-01-16T23:39:00.000-08:00

As part of our ongoing consolidation and maintenance of mail infrastructure, OCF mail will be scheduled for maintenance in short period(s) during the beginning of the spring semester.

We apologize for the inconvenience and hope for better performance, reliability, and redundancy in the long run.

Mail is currently scheduled for maintenance on Monday, January 17, while /var/mail is migrated to a permanent mail server. Incoming mail will not be accepted during this time, but read-only access will be available as much as possible through the login servers (but not on POP/IMAP and webmail).

As always, you can follow our progress here and on IRC.

UPDATE 05:25p: Maintenance will begin shortly. /var/mail will now be mounted read-only.

UPDATE 06:00p: /var/mail is currently being migrated to a permanent server. It will still be mounted read-only, and accessible through the login servers (but not on POP/IMAP and webmail).

UPDATE Tuesday 01:40p: Mail should now be back up, including the login servers, POP/IMAP, and webmail.

OCF Mail Down

2010-12-10T03:32:00.001-08:00

OCF mail is down; please stay tuned for updates.

UPDATE (10 December): The server which hosts /var/mail had a drive failure (one of the pins on the boot drive was bent, which short-circuited something and caused a system shutdown). We've restored the server using its other drive and are rebuilding RAID. All mail services should now be functional.

UPDATE (11 December): The mailspool server is down again, and its remote management system seems to be malfunctioning. Mail is down until an OCF staffer living in Berkeley can investigate; if the server is irreparably damaged, we'll have to wait until finals are over to rebuild it. Sorry for the inconvenience.

UPDATE (12 December): Some of you have expressed concern that your data may have been lost. Sorry for any alarm — rest assured, your data is safe, and we do have recent backups of /var/mail.

As part of our transition to a new mail infrastructure, we recently migrated /var/mail from our central disk array to a new machine with a better NFS server. We didn't realize until the server shut down two days ago that one of its drives was damaged, at which point we swapped out the drive and turned the machine on again. The server shut down again yesterday, and S.M.A.R.T. is predicting another drive failure, so we fear the motherboard or drive controller may have been damaged by the faulty drive. The server runs and your data is intact, but rather than just rebuilding RAID again and risking a major meltdown during finals week, we've decided to leave everything off after finals, when we'll have time to take care of this properly.

UPDATE (14 December): As of early morning today, mail is tentatively operational. Last night we transitioned /var/mail to an alternate hard disk in a temporary server. Maintenance work will be scheduled soon, especially over break. Special thanks to the ASUC for their cooperation.

Unexpected Downtime

2010-11-22T19:37:00.000-08:00

The OCF's DNS server just underwent spontaneous massive existence failure and we are trying to get the system to boot. Further bulletins as events warrant.

In other news, there was a hardware failure of unknown origin on the new mail server and /var/mail was down for a few minutes while we recovered the RAID array. Sorry for the inconvenience.

Update(10:30PM): DNS is back up. Sorry for the inconvenience again.

Update(11:00PM): The login servers had the blues during the DNS outage. They have been rebooted, and are fully functional again.

Scheduled downtime this weekend

2010-11-16T21:13:00.000-08:00

Hey OCFers,

In order to give OCF mail a much-needed reliability boost, we plan to incrementally replace our current mail infrastructure with a centralized, redundant new system. The current system is scattered and full of single points of failure — as an example, our SpamAssassin server didn't survive last week's power outage, and our mail servers were forced to reject all incoming mail until we were able to build a replacement spam filter on a different machine. Our hope is that this new system will be significantly less volatile and more maintainable.

To that end, we'll be taking OCF mail down this Saturday. We'll be migrating mail storage, outgoing SMTP, IMAP/POP, and webmail to the new mail server. Incoming mail will not be accepted while we're working on the servers, but we'll try to keep read-only access to your old mail online for most of the day. As always, you can follow our progress here and on IRC.

UPDATE (20 November, 12:15PM): It begins! We've taken incoming mail offline and made /var/mail read-only for the mailspool migration.

UPDATE (20 Nov, 2:30PM): We've finished copying data from the old mailstore to the new one and are in the process of switching NFS servers. IMAP/POP access to mail is now down.

UPDATE (20 Nov, 5:30PM): We're having a bit of trouble with the new NFS server; IMAP/POP is still down, and our login servers hung and had to be rebooted a short while ago. tsunami and conquest should be working now, though.

UPDATE (20 Nov, 6:45PM): The NFS server is now up and running, now that we've squashed a pesky NFSv4-related bug (thanks to sluo and dwc for their help!). /var/mail is no longer read-only. Once we finish migrating disk quota information, we'll bring the mail servers back online. Thanks for your patience.

UPDATE (21 Nov, 12:15AM): /var/mail has been fully migrated to the new NFS server, and we've re-enabled the old mail servers. Incoming mail and POP/IMAP still live on the old infrastructure, but now that the NFS server migration is complete we should be able to set up the new infrastructure with minimal downtime.

To be clear, we have not yet migrated SMTP, IMAP/POP, or webmail to the new mail server, but we plan to do so in the near future. Watch this space for updates.

ASUC Building Power Outage

2010-11-09T16:20:00.000-08:00

Hi,
Tonight from 2am-6am Esheleman Hall will have a power outage, stay tuned for updates.

Update 8:44: Everything except the OCF webserver/disk array/authentication servers has now been shutdown.

Update 9:00pm Shutting down mysql

Update: 9:15pm Webserver shutdown

Update: 9:20pm Disk array shutdown/authentication servers shut down. The only things that are still up are infrastructure related servers. These allow us to manage machines remotely. The UPS says 2:15 of runtime, the outage is 4hrs, lets hope 4hrs is a conservative estimate?

Update 6:36am Marginally restored our infrastructure, we are running tests to make sure everything that is up is working. Login Servers will be up shortly

Update 7:04am There are a few issues coming back up we are looking to get them resolved ASAP

Update 7:31am We had some disk array issues, they seemed to be resolved for the time being, all the windows machines work, printing works, and we will soon boot up the login servers after we are sure permissions and such are working properly

Update 7:56 Our DNS server is being stubborn, seems to be the root cause of recent issues. Mysql should be back up

Update 8:04am FSCK time, what a fun way to start the moring, fsck'ing broken filesystems

Update 8:12am The webserver should be working again

Update 8:22am DNS is plodding along, expect a delay between 9-10:30 since I have class at this time.

Update 8:41am FSCK on the DNS server, will likely be down for a while, login servers should be up and running, docs and webmail should work too.

Update 8:44am spoke too soon disregard the previous post

Update 10:56am still working on getting dns up.

Update 11:11am DNS should be up now

Update 6:40pm Reaching hour 30 of this adventure, most of our services have been restored. Mail is a work in progress, but your stored email should be fully accessible now. apocalypse.ocf.berkeley.edu doesn't seem to turn on, so we will keep that off for now (while we straighten out everything else).

Other failures

2010-10-20T19:59:00.000-07:00

In a rather unlucky streak of timing here are other known failures in the OCF (today was not the best day).

2 printers (1 critically)
3 infrastructure related servers

Will update you as we get things fixed.

Mysql Down

2010-10-20T11:00:00.001-07:00

Mysql went down for a short period of time yesterday, we hosted the service from backups in a read-only mode for the night.

UPDATE (15:41):
ETA of 8-10hrs before we get mysql up and running again.

UPDATE (19:41):
mysql, postgresql back in business. props to jaws.ocf.berkeley.edu, for performing admirably

Expected downtime

2010-10-10T10:51:00.000-07:00

Hi all,
Yesterday's reboot left behind some nasty problems in our mail volume, we are going to take some downtime to fix it. 7pm-11pm, I expect system-wide downtime, I apologize for whatever inconvenience this has caused. In hindsight we should have handled this better, but we learn from our mistakes.

Sorry for the trouble.

Update 4:56pm
We may start working a bit early at around 5:30-6:00 so don't be surprised if /var/mail disappears during that time.

Update 8:40pm
Stuff should be working again for the most part, thanks for your patience

Disk array reboot

2010-10-09T21:15:00.001-07:00

Hi all,
We rebooted our disk array, resulting about 30 mins of downtime system wide. Just notifying all in case there are any stray stale nfs handles hanging around, or if any scripts broke. Don't panic if you couldn't log into our machines during this time, it was nothing serious.

Webserver down for some maintenance

2010-09-01T17:16:00.001-07:00

We are moving the webserver onto some better hardware, it should be a bit faster after the transition...

Webmail Restored

2010-08-21T11:59:00.001-07:00

We've restored webmail functionality. Please let us know if there are any problems or hiccups.

Update

2010-08-14T13:16:00.000-07:00

Hi all,
Some of you may have noticed your cgi scripts have broken, this is probably due to the fact that our new webserver has a different directory structure, and your scripts rely on a hard coded directory path. Don't worry this is only temporary. I used different nfs mount points to make mounting user data easier while we were swapping ip's around. It should be resolved when the server is on its proper hardware.

Anticipate downtime this weekend

2010-08-13T15:28:00.000-07:00

Hi all,
We will try to rebuild one of our main servers this weekend, anticipate some downtime and/or loss of functionality.

Servicing our disk array

2010-07-31T15:40:00.000-07:00

The OCF will likely be down the entire weekend as we update, error check, and test our disk array. Please check back later for more updates

Mysql Upgrade

2010-07-18T20:33:00.000-07:00

Hi all,
We just upgraded our mysql server, let me know if anything doesn't work. Some of our client programs may be old, so if it says stuff like "Authentication protocol not supported" don't worry, its just that our clients are using old programs.

Users can now write to home directories

2010-07-13T01:35:00.000-07:00

The OCF has been reasonably stable for the past few days, I have decided to allow users to write to their home directories.

Further Updates

2010-07-02T17:46:00.001-07:00

Thank you for all your patience, its really appreciated. Our new disk array should be back up, and we will slowly restore services after the July 4th break. I don't want to rush anything since its just going to make it harder to fix if something goes wrong.

I want to make sure everyone knows we have multiple goals when addressing the current problems. While we try to maximize our uptime, we are also concurrently rebuilding our current system. At the beginning of summer, we did send a downtime announcement that spanned the whole summer. At some point we may decide the current system is not worth salvaging, and focus our efforts on the rebuild.

Furthermore, we will shortly attempt to rebuild our core servers, which will take down our entire system. We will try our best to build ad hoc login servers, but anticipate more downtime in July.

Read-only home directories, mailboxes, and html directories

2010-06-20T22:37:00.000-07:00

As you may know by now OCF services have been marginally restored and you can now access your files. However, you may have also noticed that you get strange errors, "Permission Denied", and are generally unable to edit your files. This is because we have mounted all your data Read-Only. This is a consequence of impending failure of our disk array, and we are in the process of migrating to a new one. Stay tuned for changes.

Unexpected Downtime

2010-06-13T18:28:00.001-07:00

As many of you have noticed, the OCF is currently down. One of our core servers experienced a hardware failure earlier today and we are working to restore it. We do not yet have an ETA for restored service, but will post updates as we have them. We sincerely apologize for the inconvenience.

UPDATE (16 June, 1 PM): The failed server has been partly restored. Web and SSH logins are intermittently working for usernames starting with the letters a-p, but mail is still down systemwide; we hope to have full functionality soon. As always, you can find us in the OCF's IRC channel if you have any questions. Thank you for your patience.

UPDATE (16 June, 11 PM): All OCF services, including mail, webmail, and SSH logins for accounts starting with the letters q-z, should be working now. Please let us know if anything appears to be broken, and thank you for your patience!

UPDATE (17 June, 1 PM): Webmail has been fixed.

UPDATE (17 June, 1:30 PM): The OCF's main disk array is having problems — we've taken all OCF services down while we investigate.

First Update on Rebuild

2010-05-26T20:21:00.000-07:00

Update 1 on OCF Rebuild

-Set up a new nat server and a new printhost
-Each one is a separate esxi guest
-Testing to see if I allocated enough memory, I think I may want to bump the printserver up to 512 MB of virtual ram (any thoughts?)
-New Lab IP ranges (formalized)
- So lab networking is being cleaned up, mostly wiring, I'm hoping to axe the windows vlan altogether since if you include the random shit we have acquired over the past year thats way more ports than what we traditionally allocated for the lab nat. (motivation for cleaning those IP's up)
-haven't set up any printers yet, one of them is possibly broken (or jammed)
-Hopefully next week we will start rolling out machine images...will start with linux
-8 pages of documentation so far, ask me if you want to read it. I'm doing them on word documents for now (not leaving any details out). traditionally we tend to leave stuff out like iptables and stuff out of staff wiki documentation, but staffers have messed stuff like that up before, so i'm not leaving even "trivial" stuff out...
-^perhaps we can make an ocf staff handbook or something basically command by command how to rebuild the ocf?

Summer Goings-On

2010-05-25T19:37:00.001-07:00

So, some of our users have been concerned about the planned downtime this summer. Here's the nitty-gritty:

We're rebuilding everything from the ground up. Partly this is because the current system has things scattered over multiple servers, a relic of the days when we didn't have powerful enough systems to consolidate mail, for example, onto one machine. Partly this is because Oracle (formerly Sun) can no longer be relied upon to provide free security updates for Solaris, which forms the majority of our back-end. Finally, this is partly because the current setup is simply untenable. It's the computer equivalent of an Land Rover held together with spit, baling wire, and chewing gum - it runs, more or less, but God help anyone who needs to poke around under the hood when things break.

So, we decided to move on. Rather than spending time and energy trying to keep fixing a 20 year old heap, we're starting fresh. We'll be changing some things on the backend - most notably a migration from Solaris to Debian and FreeBSD - and we'll be changing some things on the frontend - like bringing in Windows 7 and getting some new hardware for the Linux clients. For most of our users, the change will be mostly transparent. For some of our users, things will change a little bit. For a tiny minority, things will break. To those people, we apologize in advance, and we are, as always, happy to help you get things working again.

In theory, while we're making this transition, we'll be building replacements side-by-side with the current production servers, and swapping them out once we're fairly certain that everything's working correctly. So, for the majority of our services, there won't be much more than a blip in service. Moreover, we won't be swapping out more than one server at a time, so no more than one service should go offline at any given time. However, there are some services (notably web and MySQL) which will take longer to swap out. The fact of the matter is that we really only have one server powerful enough to be a web server, so we can't build it's replacement until we've shut it down. Even so, our daring team of sysadmins should have the server back up and running in no time flat (I believe the previous record for a ground-up rebuild of the webserver was less than a day).

Finally, I'd like to bring some attention to the "in theory" that started off that last paragraph. As anyone who's ever worked on any sort of project before knows, something will always go wrong. So we ask you to bear with us while we work out the kinks and the bugs. This is going to take at least a few weeks, and things will be a little hectic during that time. We may swap in a new system only to find some bug that escaped our testing, and we'll have to switch back until we get it sorted out. We'll do our best to post here when we're getting ready to swap something out, and we'll make sure that there are always avenues open to get in touch with us to let us know about problems.

Oh- and remember that we're all volunteers. We do our best, but sometimes other commitments (school, work, family, life) can get in the way for a little while. But like all true geeks, we can't stay away for long, so rest assured that things will get fixed, emails will get answered, and the agents of truth and light will win the day.

Thanks for reading.

Sorry...

2010-05-04T00:29:00.001-07:00

Our Incoming mail server was having some difficulties with disk-space, somewhere in the stress of finals and the stress of having to resurrect the server, I panicked and accidentally deleted some queued up mail. So mail might behave a bit weird for the next day or two. Sorry for any inconvenience I may have caused you.

Sanjay Krishnan
OCF Site Manager

Unexpected Downtime

2010-04-04T17:58:00.001-07:00

The OCF's central file server went down today around 4:30, and we're having a bit of trouble nursing it back to a working state. We'll post updates here as we have them.

Patch Day

2010-02-12T17:24:00.000-08:00

The OCF's patch day is set to begin in about half an hour, and is scheduled to run for 24 hours (6 PM Friday to 6 PM Saturday). Various services will drop in and out during that time, including mail, web, and SSH access. We'll post status updates here and will let you know about any unforseen events which would prolong the outage.

8 PM: The mail servers have been updated successfully, with no outage required thus far. We will need to shut down mail delivery, POP, and IMAP when we take the fileserver down later tonight, however. The login server and documentation server are taking longer than expected to patch; we'll hold of on the fileserver until those two are resolved.

10:45 PM: The final updates for the login and documentation servers have been applied, apparently with no adverse effects. We've just taken mail down in preparation for patching our main fileserver. We will be bringing down web next before commencing the patching.

1:45 AM: Looks like the bulk of the patching work is done. There might be a few more short outages throughout the day as we tweak settings and ferret out bugs, but there should be no more prolonged outages (cross your fingers).

Unexpected Downtime

2009-12-20T22:23:00.000-08:00

Sometime this morning, one of the OCF's mail servers stopped responding to user logins. We are working as fast as we can to resolve the issue, but do not have a time estimate. The server that hosts webmail also recently stopped responding; we are working on restoring it, but are focusing on bringing mail services back online.

The mail server in question — mail.OCF.Berkeley.EDU — provides IMAP, POP, and SMTP access to mail, but stores users' email on the central file server. Old mail is still accessible via command-line mail clients like mutt and pine on our login servers (apocalypse and tsunami), and incoming mail will be queued on a different server and delivered when mail.OCF is restored.

We will post updates when we have them. If you need assistance, someone in the OCF IRC channel may be able to assist you, but please be patient if nobody responds right away. :) We apologize for the downtime and inconvenience, and wish you a wonderful winter break.

UPDATE (23 December): mail.OCF is partially online. You can access your mail with an IMAP or POP client such as Mozilla Thunderbird, but we are still ironing out some issues with sending mail — please let us know if you have any problems. Webmail is still down, but we are hoping to have that service restored within the next few days.

UPDATE (27 December): We've restored the server that hosts webmail and documentation. Thank you for your patience — have a happy New Year!

UPDATE (30 December): Our SSL certificate issuing authority, ipsCA, kinda dropped the ball and forgot to renew their root certificates before they expired on December 29th. As a result, unless you're using IE8, you will get a warning about an untrusted SSL certificate whenever you try to access OCF services like POP, IMAP, SMTP, Webmail, or account tools. We are working on obtaining updated certificates from a slightly more on-the-ball certificate authority, but in the meantime, you may continue accessing your OCF services by overriding the certificate warnings. However, in general this is a bad practice to get in the habit of, and we suggest that you wait until we have installed the updated certificates before attempting to connect.

UPDATE (5 January): We have installed the new SSL certs for mail.ocf and webmail.ocf. Let us know if you have any issues connecting to these services, and happy new year!

Emergency Webserver Maintenance

2009-11-16T20:46:00.001-08:00

At around 8:30 PM this Monday, OCF staff noticed that the OCF website was slow to respond, and that our webserver either the victim of a DDOS attack or else swap space exhaustion. In any event, we've gone ahead and rebooted it. We will also use this opportunity to apply a few long-overdue patches to the machine.

We expect this downtime to last no longer than an hour. During this outage, most OCF services will be unavailable (the machine hosting our webserver also hosts a number of other critical services).

We apologize for the inconvenience and thank our users for their understanding.

UPDATE: We have since discovered a problem with the server hosting mail services. We are working on this issue now; we anticipate that this will add no more than an additional hour to our original schedule.

UPDATE 2: It looks like everything is back online as of 9:55 PM. Let us know if you discover anything out of the ordinary.

UPDATE 3: It looks like webmail is having some issues. We are investigating this issue, but have no timeframe for fixing it. UPDATE TO THE UPDATE: Looks like this was an easier fix than I had thought. Webmail should be working again.

Unexpected Downtime

2009-09-15T23:47:00.000-07:00

The OCF directory server went down today (Tuesday 9/15) at roughly 10:20 PM, taking pretty much everything else with it. We're still working on figuring out what went wrong and how to fix it.

We'll post updates here as we have them. You can also find us at on the #OCF channel on the CSUA irc server at irc.csua.berkeley.edu.

UPDATE 9/16/09 3:00 PM: It looks like everything is back up again. If you find anything missing or broken, please let us know.

And to everyone who was wondering, yes, this is why your passwords/email stopped working. This is also indirectly responsible for why you may have gotten funny looking prompts when logging in remotely.

General Meeting on 9/10

2009-09-05T21:07:00.000-07:00

Hello dear OCF members!
The new school year has begun, so it is time again for the OCF’s semi-annual General Meeting! It will be held at 7PM in 243 Dwinelle Hall. Come vote for this semester's new General and Site Managers, give us your suggestions, volunteer to help (we obviously need it), or just pelt us with your complaints! Pizza and various drinks shall be served to enhance the experience. See you all there!

Status of OCF Services

2009-08-27T07:12:00.001-07:00

We've been working hard to get all our services restored. As usual, we've run into some bumps along the way, but I'm happy to report that we've been able to bring up the following services:

Webmail: You can read your mail via our webmail interface. However, mail ~~sending and~~ receiving is still in the process of being restored.
Web hosting: The web server is mostly functional. The OS installation managed to corrupt itself during the move from our old server room in MLK to our new server room in Eshleman. We were able to hobble together a new installation from backup, but we're not sure if all our server components were restored. If something seems out of the ordinary, please comment and let us know.
MySQL databases: These should be completely functional.
Shell Access (SSH/SFTP/SCP): We know that many of you need access to your files. ~~Our goal is to bring up our login servers by Thursday night.~~ apocalypse is now online and accepting logins.
POP3/IMAP: You can now check mail using software on your computer (ex., Thunderbird, Postbox, Outlook, etc.)
Outgoing Mail: You can now send mail from the OCF.
Incoming Mail: Our mail infrastructure is split into many pieces that are hosted on multiple servers. Unfortunately, all the pieces need to be brought up before we can re-enable mail (otherwise, there would be a flood of spam, bounces, and incorrectly delivered mail). ~~We hope to have incoming mail restored by the end of the week.~~ Incoming mail is up as of Tuesday night.

Once again, we apologize for the extended downtime. We're exploring better ways of handling extended outages in the future, but we want to emphasize that this downtime was really an anomaly. A perfect storm of factors came together: a long power outage, a server room migration, a network disconnection, and an IP space change.

We'd also like to thank the wonderful people at Physical Plant, IST, Residential Computing, and other campus organizations for coming to our aid. Also, thanks to our users for their supportive and critical comments (hey, we deserve it!).

UPDATE (28-Aug-09, 9:30 AM): We had a bit of a hiccup in our services yesterday night. All services that were working yesterday morning should also be working now.

UPDATE (29-Aug-09, 7:00 PM): We're still restoring our mail services. We haven't forgotten about our other services (printing, web file access, etc.) We're just prioritizing our most important ones.

UPDATE (1-Sep-09, 9:00 AM): We're starting to ramp up incoming mail. Once we're sure that it's working perfectly, we'll begin restoring the rest of our services.

UPDATE (2-Sep-09, 8:00 AM): Incoming mail is completely operational. We're starting work on restoring the rest of our services.

Good News!! (About Time...)

2009-08-21T20:21:00.000-07:00

Today, the electrical work in our new server room was finally finished! Also, our network problems are over! Connectivity has been restored - all thanks to IST, who swooped in today to hook OCF up to their own network. Not only do we receive a steady fiber connection, but we also get IPv6 and our very own subnet.

So why aren't OCF services up and running, you ask? Well, all the IP addresses have changed because we are on a different subnet. This means that in addition to flashing all the switches, we must also renumber all the IPs on our servers, which we have many of. This is quite a task, requiring staff members to sit in the server room with a screen and keyboard and reconfigure every single server we have. Since a couple staff members are returning to Berkeley this weekend, I am planning to do this on Monday so that they can help me out. And since everything that can go wrong already has, I am going to be optimistic and hope that we won't run into any problems bringing servers online. If all goes well, OCF can be back up by Tuesday morning!

You Know What They Say About Academia...

2009-08-19T18:39:00.000-07:00

The cause of our networking outage has not been determined. We're working with campus technicians to debug the problem, but everything is going really s-l-o-w-l-y. We're just as upset as you are about the prolonged downtime, but, as a student organization, we just don't have as much sway as a campus department.

Continuing Downtime

2009-08-06T07:16:00.000-07:00

The downtime that began on 7/28 is expected to continue for a few more days as campus electricians procure parts to complete the electrical work. If there are no other unanticipated delays, we should be back up in the coming days.

We apologize for the downtime and inconvenience.

UPDATE: The electrical work will be completed by Friday, August 14, at 4 PM. Unfortunately, we have very limited staff presence in the Berkeley area due to the summer vacation, so it may take some time to get everything up and running.

SECOND UPDATE (14-Aug-2009, 3 PM): The electrical work is mostly complete, and power has been restored to our server room. Unfortunately, something happened to our network uplink during the downtime (maybe someone forgot to turn a switch back on), so we have no Internet access. We're looking into it.

THIRD UPDATE (18-Aug-2009, 4 PM): A campus network technician is working on restoring connectivity to the OCF. The cause of the problem is still undetermined, but we're pressuring them to get things fixed ASAP.

OCF Downtime Starting 7/28

2009-07-24T12:03:00.000-07:00

The OCF will be down next Tuesday due to electrical work being done in Eshleman Hall. Unfortunately, we have not been given a time frame, so servers may be down for up to a week. We apologize for the inconvenience, but will try hard to make sure services come back up as soon and as smoothly as possible.

The Pre joins OCF Mobile

2009-06-18T09:28:00.000-07:00

Last year, we launched our mobile site to load for the iPhone/iPod touch, and right after included other devices like the Blackberry Storm, Android-based G1, and Opera Mini (such as on the Blackberry Curve beyond the BB browser). All other devices could still enter the site manually through m.ocf.berkeley.edu.

The Palm Pre has now been added to that list. Data detection works just as it does with the iPhone OS, so your phone dialer, Google Maps, and email client will work with our pages there too. I'll be on the lookout for Windows Mobile devices to test upon.

Unexpected Downtime

2009-06-12T11:23:00.000-07:00

OCF servers and services are currently experiencing unexpected downtime, due to what may be a Denial of Service (DoS) attack on the subnet the OCF is on. Most staffers are currently out of town due to the summer break; while this is being looked into, it's a bit difficult without connectivity to the servers.

Further updates to come on this blog. While our Internet Relay Chat (IRC) server is down, you can connect to CSUA's IRC server at irc.csua.berkeley.edu and join us in #ocf for updates (mibbit IRC client).

2:18 PM: It seems like we're back online, but the causes of the outage is still unknown. No data loss is anticipated; please let us if otherwise.

apocalypse accepting user logins

2009-05-06T19:15:00.000-07:00

apocalypse.OCF.Berkeley.EDU (192.58.221.245) is now accepting user logins. You should be able to connect to the server via SSH/SFTP to access your home directory and web space. If you're comfortable using a command-line mail client (e.g. mutt or pine), you'll also be able to read and send mail.

There are a few lingering issues with the server. In particular:

~~Public key authentication does not work. We're working on fixing this problem.~~ Fixed.
Some pre-installed software and user commands may be missing or broken. We had to re-compile all of our software for the rebuild, and it's an ongoing process. We prioritized the most commonly used programs, but leave a comment if you think we've forgotten an important program. UPDATE: We know the software waitlist is building up, so we're getting more staff involved in this process.
~~Password changing does not work.~~ Use kpasswd to change your account password. Windows password changing is still broken.
~~Cron jobs are not yet available.~~ Cronjobs should now work as documented on our wiki.

The SSH host keys are:

DSA:
04:68:af:14:ab:a1:4e:79:c3:42:6f:51:62:10:a2:5a
xivit-hunuc-kogur-hizez-pazyr-bynat-pytop-zitos-dufuf-danec-sixax
RSA:
72:16:9c:7e:7b:cc:a0:47:56:6d:e3:59:aa:04:bc:7c
xipav-pizem-lysyd-kicyt-sococ-macuz-hyluc-vizoc-cotem-latig-muxyx

Thanks for your continued patience!

OCF downtime: frequently asked questions

2009-05-06T03:06:00.000-07:00

We've been getting a lot of questions over the past week, and (understandably) many of them are similar, so I thought I'd collect the answers to the most frequently asked ones here.

We thank you for your patience, and we'll continue to provide updates via our main website and this blog as things change and we get back on our feet.

Q: Can I access my email?
A: As of a few hours ago, there's a temporary webmail interface available at https://webmail.ocf.berkeley.edu/. Sending email from it doesn't work for the moment, and it lacks the spit and polish of the usual webmail interfaces, but you should at least be able to read your email. POP and IMAP access is still available, and you can send email using SMTP -- see our documentation (now also available again) for information on how to set this up.

Q: Can I access my files on the OCF?
Q: Can I modify my website hosted on the OCF?
A: The usual SSH and SFTP access is unavailable at the moment. For now, you can use the web FTP application to upload, download, and edit your files.

Q: What happened to my account application?
A: We can't process account applications at the moment. We'll get around to it once we have our services up and running.

Q: Can I use SSH or SFTP?
Q: I tried to log in via SSH/SFTP, but I'm getting a message that the host key changed.
A:The primary login servers that you get by logging into ocf.berkeley.edu are turned on again, but aren't accepting user logins at the moment. We'll let you know when you can log in to these machines again. The host keys have changed, and we'll post the new key fingerprints, along with instructions, when we're ready to allow logins again.

Q: So what's this about being hacked? Should I do anything to make sure I'm safe?
A: We recommend the following:

If you have other accounts which share the same password as your OCF account, those passwords should be changed immediately. Note that reusing passwords is a terrible idea for exactly this reason.
If you have logged into another machine from an OCF system using SSH, Telnet, or the r* commands using a password since April 15th, that password should be changed immediately.
If you entered a password on an OCF system, especially a Solaris or Linux system, at some point since April 15, consider changing that password.
If you kept an SSH private key on the OCF, immediately prevent it from being used to log in to any systems (remove the corresponding public key from authorized_keys everywhere it's listed).

We will be requiring that all users change their OCF account passwords soon, but you cannot do this now, nor do we recommend doing it now.

If you have other questions, feel free to email us (staff [AT] ocf.berkeley.edu).

2009-04-28T20:29:00.000-07:00

We apologize for the lack of notification and the lack of status updates, and are grateful for your patience. As you may have guessed, this is not a planned outage.

On April 19th, we discovered that one of our servers had been broken into. We initially tried to contain the problem, but further investigation revealed that the scope of the break-in was worse than we had originally thought. At our Board of Directors meeting on Thursday, April 23, it was decided that many, if not all, of the OCF servers would need to be rebuilt, to reduce the possibility that the hacker would be able to break back into our systems. Because of the nature of the incident, in the interests of security and damage control, the decision was made to begin reconstruction efforts immediately, rather than waiting the customary week to give notification.

Starting that night, various servers have been pulled from the network for forensic analysis and re-installation. However, we have been delayed in restarting some of our services due to an unforeseen difficulty in rebuilding one of our core servers. Until that issue is cleared up, we will be unable to resume anything approaching normal service.

For those of you worried about mail, the mail servers are continuing to accept incoming mail, and it will be available for pick-up once normal services resume.

I'm very sorry that I cannot give you a more solid timeframe than "soon." Please, rest assured that we are doing our very best to bring the OCF back online, and all of our senior technical staff are hard at work fixing this roadblock to normalcy. We ask your patience and understanding, noting that all our staff are volunteers and students who, like you, have class and work obligations, especially with finals looming ever closer.

As always, you are welcome to join us on the OCF IRC channel, irc.ocf.berkeley.edu, with questions, concerns, or anything else you think we should know.

OCF downtime

2009-04-24T18:08:00.000-07:00

The OCF login servers (conquest and apocalypse) and lab machines are currently down for maintenance, and many services in the lab itself are unavailable. We apologize for the inconvenience and the delay in communicating this, and hope to be able to provide more details soon.

New Printers

2009-04-06T18:02:00.000-07:00

We've purchased and installed two Xerox Phaser 4510DT laser printers to replace our old, broken, printer.

Let us know if you have any problems using the new printers!

Unexpected downtime

2009-03-30T09:57:00.000-07:00

Sometime this morning, our core fileserver started having some disk issues. We're working on figuring out what exactly went wrong and fixing it, but in the meantime, the OCF is effectively down (no lab, no SSH, no web, etc). We'll post updates as we have them.

UPDATE: The problem seems to have been fixed, though we're still not entirely sure what it was. We'll keep working on it throughout the day. In the meantime, though, the OCF is back up and ready for business.

RIP Logjam, You Were a Good Printer

2009-03-10T17:48:00.001-07:00

logjam, our HP LaserJet 4350 laser printer, died over the weekend. It's out of warranty, so we're currently exploring our options. In the mean time, we've brought one of our old printers back into service.

We apologize for the disruption in service and slower printing you may experience while we work on fixing/replacing our primary printer.

Yes We Can Upgrade Quotas

2009-02-25T10:42:00.000-08:00

Good news: we're providing storage stimulus by upgrading all user disk space quotas to 2 GB.

Bad news: to enact this change, the OCF will be unavailable between 12 AM, Friday, February 27 and 8 AM, Monday, March 2 (just over 72 hours).

We know three days is a long time, but the prolonged downtime is necessary to ensure the safety of user data while we perform our upgrades. Delivering data reliability you can believe in is important to us.

We're hoping this will be the first of many upgrades for our users. Who knows? With the new lab, this could signal a new beginning.

OCF Lab Reopening!

2009-02-06T15:57:00.000-08:00

The OCF lab will be reopening on Monday, February 9. We have a shiny new lab in the basement of Eshleman Hall, a cool lounge just for hanging out, and Airbears!

So come by to check it out next week, and thank you for being so patient with us for the past seven months!

Unexpected Downtime

2009-02-03T21:40:00.000-08:00

We're currently experiencing some hardware problems on one of our core servers. We are investigating the issue and will post updates here as we determine the scope of the problem.

UPDATE: Disk array issues again. We're running a check on the data, and we expect to be back online tomorrow morning. Sorry for the inconvenience. We understand that it's a school night, so please let us know (sm @ ocf.berkeley.edu) if you need emergency access to your files.

UPDATE (1:01 AM): The check completed successfully, and we're bringing all OCF services back right now. Let us know if you notice anything strange with your files.

General Meeting on February 5

2009-01-27T09:59:00.000-08:00

The OCF will be holding a general meeting next Thursday, February 5th, at 7 PM in 189 Dwinelle. Hear the latest updates on our lab relocation, meet the staff who volunteer daily to keep your services running, and elect this semester's OCF managers.

Everyone's invited, so come on by for food and socializing!

Login Server Upgrades

2008-11-01T09:11:00.001-07:00

We just upgraded the operating system of one of our login servers, conquest.OCF.Berkeley.EDU. The machine now is available for general use and will be re-added to our login server alias once we're sure that everything works. Feel free to give the server a test drive, but please note that we might have to reboot it a few times in the coming days.

This operating system upgrade allows us to deliver greater security and better application support to our users. In fact, I'm pleased to announce that we now support Mercurial, a popular distributed version control system.

We'll also be upgrading apocalypse.OCF.Berkeley.EDU soon. If you really need a place to work uninterrupted this weekend, you can try our Linux login server, tsunami.OCF.Berkeley.EDU.

Power Outage

2008-10-22T17:49:00.000-07:00

There will be a power outage in the building housing our core servers between 1-6 AM on Thursday, October 23. We anticipate shutting down our servers around midnight tonight and powering them back on as soon as we can.

I apologize for the short notice, but we were just notified today.

Maintenance Complete

2008-10-18T20:59:00.000-07:00

We've finished our maintenance ahead of schedule, and all OCF services should be working again.

Thanks for your patience and understanding!

Planned Maintenance

2008-10-18T00:29:00.000-07:00

The OCF is currently down for maintenance. Some of you may have noticed the unplanned downtime this past Tuesday, which was related to errors with our disk array, which houses mail as well as many of the critical files for the OCF. This downtime is meant to help fix some of those problems and stabilize our servers.

We don't expect it to last past 9 AM on Sunday, October 19.

Unplanned Downtime

2008-10-14T13:25:00.000-07:00

We began having problems communicating with our disk array this morning. Since most of our data is stored on the disk array, many of our services are currently offline.

We are working on fixing the disk array. The data on the disk array is backed up daily, so even if the disk array is broken, all data should be safe.

UPDATE: We've brought back all our services as of 3:30 PM. Let us know if anything is behaving strangely.

iPhone-tailored OCF website

2008-10-07T00:59:00.000-07:00

Been thinking about doing this for a while now, and as of today, owners of the iPhone and iPod Touch can point their devices to: ocf.berkeley.edu/iphone for a slimmed down essentials-only mobile version designed specifically for this viewport. There are some goodies in there, including the phone number linking to your dial pad, or the map location linking to your built-in Google Maps.

I've tested it against the iPhone SDK's simulator, and against Safari in general, and we'd love any feedback you have. For those of you without these devices, you can play around with it with TestiPhone.com, though it should work with most devices with similar viewports. Or you can simply view screenshots. And of course, you always have the option to view the full versions of these pages in these devices.

I have plans to develop a more general mobile version for other devices as time permits. Keep posted. Coming to a device near you.

OCF System Maintenance Fall 2008

2008-09-27T20:35:00.000-07:00

The OCF is down for maintenance. We don't anticipate that it will last more than 12 hours, but if it does, we will post updates here.

Buzz

2008-09-18T23:59:00.000-07:00

Check us out at The Daily Clog for an outsider perspective on our Heller-to-Eshleman relocation situation. Free computing is becoming a little hard to come by these days, but we will be back and better before the first snow falls. (Luckily, we're in California.)

FREE FOOD!

2008-09-10T23:32:00.000-07:00

The OCF is holding a General Meeting on Thursday, September 11 at 7PM in 263 Dwinelle.

There will be free food! Come hear the latest news about our lab relocation and meet the staff who keep your services running!

Unplanned Mail Downtime

2008-09-01T10:42:00.001-07:00

At 3AM on September 1, we began having issues with the hardware hosting our mail. We are currently working on identifying the problem and minimizing its impact upon the rest of our services.

We will update this post once we have more information.

UPDATE: It looks like our disk array had some hard drive issues. We were using mirrored hard drives, so we don't think any data was affected. Everything should be working again, but please let us know if you see anything out of the ordinary.

OCF Laboratory Move

2008-06-09T16:23:00.000-07:00

The OCF laboratory in Heller Lounge will be closed throughout the day as we relocate to our new location in Eshleman Hall.

We apologize for the short notice. Please contact us via email or IRC if you have any questions or concerns.

Harder Better Faster Stronger Webmail and Documentation (Redux)

2008-05-25T19:26:00.000-07:00

We just upgraded our webmail and documentation server to a machine with more than four times the processing power and RAM. Combined with the changes we made last semester, our webmail and documentation should be faster than ever.

We haven't forgotten about our other services, though, and we're working on moving them to faster and more reliable hardware this summer. We know that many of you have been suffering with the occasional slowness in our services, and we're working on fixing it! Please be patient as upgrades to our other services require lots of planning and testing.

Before I go, I'd like to extend a special thank you to Sun Microsystems, who provided us with the hardware that made this upgrade possible. They make awesome server hardware!

Network Restored

2008-05-23T18:12:00.001-07:00

The OCF's network access was just restored, and we're checking to see if everything works. I'll provide updates to this post if anything comes up.

Unplanned Network Outage

2008-05-23T15:25:00.000-07:00

Our upstream network provider is having some network problems, rendering the OCF inaccessible. We're investigating the problem and trying to restore network access to the OCF as quickly as possible.

Thanks for your patience!

PHP 4 Support Terminated

2008-05-23T01:13:00.001-07:00

As announced in March, we have ended our support for PHP 4. Any remaining PHP 4 users must migrate their code to PHP 5.

A Sight for Sore Eyes

2008-05-20T03:32:00.000-07:00

We've just rolled out a bunch of new widescreen monitors for our new workstations:

Besides an awesome 20" of viewable goodness and 1680x1050 resolution, you can also rotate the screen:

If you haven't visited our lab in awhile, come by and check out our new hardware, and if you don't already have an account, go get one!

Slow Web Services

2008-05-16T17:46:00.000-07:00

Our web server is currently being overwhelmed by a flood of spam. We're working on blocking the spam and restoring our web services.

UPDATE: I think we've got everything under control. Our web services should be working normally now. Send us an email if you notice anything strange.

LDAP and Kerberos Migration

2008-04-18T02:50:00.000-07:00

As of Wednesday, April 16, the OCF has begun using a new and improved account storage and authentication framework. Among other things, this transition brings faster and more reliable Windows printing and single-sign-on support for the Unix clients. It also means that you MUST sign on to at least one OCF service (Windows or Unix clients, webmail, POP, IMAP, online account tools, SSH (WITH YOUR PASSWORD), etc.) sometime in the next six months, or else you will become an obligatory beta tester for the password migration tool, coming in the fall.

OCF downtime Thursday 3/27 from 10 AM to 5 PM

2008-03-25T01:14:00.000-07:00

Eshleman Hall is scheduled to be dark this Thursday (3/27) from 10 AM to 5 PM for construction work. Since our connection to the campus network and the rest of the world runs through Eshleman, we'll be offline during this period. Incoming mail should be queued by senders and delivered normally after the outage, so it won't be lost.

Reminder: PHP 5 now the default

2008-03-21T13:39:00.000-07:00

As previously announced, the default PHP interpreter is now PHP 5 as of today. See the previous post or our documentation for more information on what this means and how to ensure your web applications continue to work.

OCF general meeting Thursday 3/20

2008-03-17T18:52:00.000-07:00

There will be a general meeting Thursday, March 20, at 7 PM in 109 Wheeler, to discuss amendments to the OCF bylaws and constitution. See http://www.ocf.berkeley.edu/~sluo/draft-bylaws.txt for details of the proposed changes. Everyone is welcome to attend, and food will be provided.

Print Queue Online and In Real Life

2008-03-15T00:24:00.000-07:00

We've talked about it, and now it's here! The print queue is now available on display in lab on a dedicated machine and monitor by the printer, as well as online at your convenience. It was formerly accessible only via command line (lpq -s) and printing status windows in Windows, neither of which were visible enough for the majority of our users to notice. Thank you Aaron Oaks, Genevieve Wong, and Mike Gasidlo for working on making this happen.

PHP 4 to PHP 5 transition

2008-03-11T17:26:00.000-07:00

Due to the PHP developers' decision to end support for PHP 4, the OCF will be requiring all users with PHP web applications, such as WordPress, phpBB, or Gallery, to use PHP 5. This transition, required by the campus Minimum Security Standards, will provide more features, security, and performance for PHP web applications on the OCF. As the current default PHP interpreter is PHP 4, almost all users of PHP will be affected, though few will need to make drastic changes to their applications. The transition will happen in two stages:

As of March 21st, 2008, the default PHP interpreter (the one used to run scripts with extension .php) will change from PHP 4.4 to PHP 5.2. Users who cannot migrate to PHP 5 by this date may choose to continue to use PHP 4 temporarily; see our transition documentation for details.
As of May 12, 2008, support for PHP 4 will be removed entirely. All PHP applications in use after this date will be run using PHP 5.

Users whose websites do not include web applications, or whose web applications are written in another language such as Perl, will not be affected.

All PHP users should review their web applications for compatibility with PHP 5 as soon as possible. Most widely-used web applications already support PHP 5, so in many cases, no action is needed; in most others, an application upgrade will be sufficient to ensure continued functionality. Users who use custom-written PHP applications need to review their code for PHP 5 compatibility; see the links provided in our documentation for details on what's changed.

We recommend that you test your web application with PHP 5 before the transition date; instructions on how to do this are in our documentation.

The latest information on the change, and details on how you can ensure your web applications continue to function, can be found at:

http://docs.ocf.berkeley.edu/wiki/Major_service_changes

If you have any questions or concerns, please don't hesitate to contact us (email staff@ocf.berkeley.edu).

The Long Promised Website Uploader

2008-02-29T04:31:00.000-08:00

The OCF website manager, our SFTP web-based interface, has now entered beta mode. What it does is allow you to upload files to your webspace and your home directory through a purely web-based interface, in our efforts to make this process as easy as possible. I would like to thank Stephen Le for setting up this installation into something functional. I've also re-skinned the default "Blue" theme with a custom OCF theme that is more or less complete. In time, this will become a standard tool for all OCF users, once we make sure everything works smoothly.

First General Meeting for Spring 2008

2008-01-31T01:58:00.000-08:00

Welcome back OCFers!

We hope everyone is excited for a new semester with OCF.

Our first general meeting will be held on Thursday, February 7th in 109 Wheeler at 7pm.

Come learn more about the OCF
Learn how to become a staff member
Help elect the OCF General Manager and Site Manager
Find out about our upcoming events

Also, stay afterwards for some food and socializing. We hope to see all of you there!

Open Computing Facility Blog

Web Server Downtime

Intermittent downtime for maintenance

Lab and printing services uncertain after Spring 2012

Networking maintenance downtime

Update 4:00am

Networking maintenance

Mail service downtime

Unexpected downtime, RAID failure

Update Thurs Sep 22 05:45am

Update Sat Sep 24 06:30pm

Update Sun Sep 25 12:00am

Update Sun Sep 25 10:30am

Update Sun Sep 25 11:30am

Update Mon Sep 26 02:30pm

Update Mon Sep 26 05:15pm

Some incoming mail rejected

Unanticipated Downtime

Login server restarted

MySQL downtime

Service downtime

Login server restarted

Service downtime

Login server and wiki downtime

Internet outage

Issues sending mail through webmail and webserver

Off-peak mail maintenance

Login server, webmail, and wiki down

OCF internet connection down

Mail server rebuilt

Mail Server Down

Web Server Crash

Websites down

flood.OCF rebuild

Print quotas

Login servers

Tuesday outage

Update: Tuesday, February 22

OCF Not Open Until Furthur Notice

Mail scheduled maintenance

OCF Mail Down

Unexpected Downtime

Scheduled downtime this weekend

ASUC Building Power Outage

Other failures

Mysql Down

Expected downtime

Disk array reboot

Webserver down for some maintenance

Webmail Restored

Update

Anticipate downtime this weekend

Servicing our disk array

Mysql Upgrade

Users can now write to home directories

Further Updates

More problems....

Read-only home directories, mailboxes, and html directories

Unexpected Downtime

First Update on Rebuild

Summer Goings-On

Sorry...

Unexpected Downtime

Patch Day

Unexpected Downtime

Emergency Webserver Maintenance

Unexpected Downtime

General Meeting on 9/10

Status of OCF Services

Good News!! (About Time...)

You Know What They Say About Academia...

Continuing Downtime

OCF Downtime Starting 7/28

The Pre joins OCF Mobile

Unexpected Downtime

apocalypse accepting user logins

OCF downtime: frequently asked questions

OCF downtime

New Printers

Unexpected downtime