Open Computing Facility Blog

The Pre joins OCF Mobile

2009-06-18T09:28:00.000-07:00

Last year, we launched our mobile site to load for the iPhone/iPod touch, and right after included other devices like the Blackberry Storm, Android-based G1, and Opera Mini (such as on the Blackberry Curve beyond the BB browser). All other devices could still enter the site manually through m.ocf.berkeley.edu.

The Palm Pre has now been added to that list. Data detection works just as it does with the iPhone OS, so your phone dialer, Google Maps, and email client will work with our pages there too. I'll be on the lookout for Windows Mobile devices to test upon.

Unexpected Downtime

2009-06-12T11:23:00.000-07:00

OCF servers and services are currently experiencing unexpected downtime, due to what may be a Denial of Service (DoS) attack on the subnet the OCF is on. Most staffers are currently out of town due to the summer break; while this is being looked into, it's a bit difficult without connectivity to the servers.

Further updates to come on this blog. While our Internet Relay Chat (IRC) server is down, you can connect to CSUA's IRC server at irc.csua.berkeley.edu and join us in #ocf for updates (mibbit IRC client).

2:18 PM: It seems like we're back online, but the causes of the outage is still unknown. No data loss is anticipated; please let us if otherwise.

apocalypse accepting user logins

2009-05-06T19:15:00.000-07:00

apocalypse.OCF.Berkeley.EDU (192.58.221.245) is now accepting user logins. You should be able to connect to the server via SSH/SFTP to access your home directory and web space. If you're comfortable using a command-line mail client (e.g. mutt or pine), you'll also be able to read and send mail.

There are a few lingering issues with the server. In particular:

~~Public key authentication does not work. We're working on fixing this problem.~~ Fixed.
Some pre-installed software and user commands may be missing or broken. We had to re-compile all of our software for the rebuild, and it's an ongoing process. We prioritized the most commonly used programs, but leave a comment if you think we've forgotten an important program. UPDATE: We know the software waitlist is building up, so we're getting more staff involved in this process.
~~Password changing does not work.~~ Use kpasswd to change your account password. Windows password changing is still broken.
~~Cron jobs are not yet available.~~ Cronjobs should now work as documented on our wiki.

The SSH host keys are:

DSA:
04:68:af:14:ab:a1:4e:79:c3:42:6f:51:62:10:a2:5a
xivit-hunuc-kogur-hizez-pazyr-bynat-pytop-zitos-dufuf-danec-sixax
RSA:
72:16:9c:7e:7b:cc:a0:47:56:6d:e3:59:aa:04:bc:7c
xipav-pizem-lysyd-kicyt-sococ-macuz-hyluc-vizoc-cotem-latig-muxyx

Thanks for your continued patience!

OCF downtime: frequently asked questions

2009-05-06T03:06:00.000-07:00

We've been getting a lot of questions over the past week, and (understandably) many of them are similar, so I thought I'd collect the answers to the most frequently asked ones here.

We thank you for your patience, and we'll continue to provide updates via our main website and this blog as things change and we get back on our feet.

Q: Can I access my email?
A: As of a few hours ago, there's a temporary webmail interface available at https://webmail.ocf.berkeley.edu/. Sending email from it doesn't work for the moment, and it lacks the spit and polish of the usual webmail interfaces, but you should at least be able to read your email. POP and IMAP access is still available, and you can send email using SMTP -- see our documentation (now also available again) for information on how to set this up.

Q: Can I access my files on the OCF?
Q: Can I modify my website hosted on the OCF?
A: The usual SSH and SFTP access is unavailable at the moment. For now, you can use the web FTP application to upload, download, and edit your files.

Q: What happened to my account application?
A: We can't process account applications at the moment. We'll get around to it once we have our services up and running.

Q: Can I use SSH or SFTP?
Q: I tried to log in via SSH/SFTP, but I'm getting a message that the host key changed.
A:The primary login servers that you get by logging into ocf.berkeley.edu are turned on again, but aren't accepting user logins at the moment. We'll let you know when you can log in to these machines again. The host keys have changed, and we'll post the new key fingerprints, along with instructions, when we're ready to allow logins again.

Q: So what's this about being hacked? Should I do anything to make sure I'm safe?
A: We recommend the following:

If you have other accounts which share the same password as your OCF account, those passwords should be changed immediately. Note that reusing passwords is a terrible idea for exactly this reason.
If you have logged into another machine from an OCF system using SSH, Telnet, or the r* commands using a password since April 15th, that password should be changed immediately.
If you entered a password on an OCF system, especially a Solaris or Linux system, at some point since April 15, consider changing that password.
If you kept an SSH private key on the OCF, immediately prevent it from being used to log in to any systems (remove the corresponding public key from authorized_keys everywhere it's listed).

We will be requiring that all users change their OCF account passwords soon, but you cannot do this now, nor do we recommend doing it now.

If you have other questions, feel free to email us (staff [AT] ocf.berkeley.edu).

2009-04-28T20:29:00.000-07:00

We apologize for the lack of notification and the lack of status updates, and are grateful for your patience. As you may have guessed, this is not a planned outage.

On April 19th, we discovered that one of our servers had been broken into. We initially tried to contain the problem, but further investigation revealed that the scope of the break-in was worse than we had originally thought. At our Board of Directors meeting on Thursday, April 23, it was decided that many, if not all, of the OCF servers would need to be rebuilt, to reduce the possibility that the hacker would be able to break back into our systems. Because of the nature of the incident, in the interests of security and damage control, the decision was made to begin reconstruction efforts immediately, rather than waiting the customary week to give notification.

Starting that night, various servers have been pulled from the network for forensic analysis and re-installation. However, we have been delayed in restarting some of our services due to an unforeseen difficulty in rebuilding one of our core servers. Until that issue is cleared up, we will be unable to resume anything approaching normal service.

For those of you worried about mail, the mail servers are continuing to accept incoming mail, and it will be available for pick-up once normal services resume.

I'm very sorry that I cannot give you a more solid timeframe than "soon." Please, rest assured that we are doing our very best to bring the OCF back online, and all of our senior technical staff are hard at work fixing this roadblock to normalcy. We ask your patience and understanding, noting that all our staff are volunteers and students who, like you, have class and work obligations, especially with finals looming ever closer.

As always, you are welcome to join us on the OCF IRC channel, irc.ocf.berkeley.edu, with questions, concerns, or anything else you think we should know.

OCF downtime

2009-04-24T18:08:00.000-07:00

The OCF login servers (conquest and apocalypse) and lab machines are currently down for maintenance, and many services in the lab itself are unavailable. We apologize for the inconvenience and the delay in communicating this, and hope to be able to provide more details soon.

New Printers

2009-04-06T18:02:00.000-07:00

We've purchased and installed two Xerox Phaser 4510DT laser printers to replace our old, broken, printer.

Let us know if you have any problems using the new printers!

Unexpected downtime

2009-03-30T09:57:00.000-07:00

Sometime this morning, our core fileserver started having some disk issues. We're working on figuring out what exactly went wrong and fixing it, but in the meantime, the OCF is effectively down (no lab, no SSH, no web, etc). We'll post updates as we have them.

UPDATE: The problem seems to have been fixed, though we're still not entirely sure what it was. We'll keep working on it throughout the day. In the meantime, though, the OCF is back up and ready for business.

RIP Logjam, You Were a Good Printer

2009-03-10T17:48:00.001-07:00

logjam, our HP LaserJet 4350 laser printer, died over the weekend. It's out of warranty, so we're currently exploring our options. In the mean time, we've brought one of our old printers back into service.

We apologize for the disruption in service and slower printing you may experience while we work on fixing/replacing our primary printer.

Yes We Can Upgrade Quotas

2009-02-25T10:42:00.000-08:00

Good news: we're providing storage stimulus by upgrading all user disk space quotas to 2 GB.

Bad news: to enact this change, the OCF will be unavailable between 12 AM, Friday, February 27 and 8 AM, Monday, March 2 (just over 72 hours).

We know three days is a long time, but the prolonged downtime is necessary to ensure the safety of user data while we perform our upgrades. Delivering data reliability you can believe in is important to us.

We're hoping this will be the first of many upgrades for our users. Who knows? With the new lab, this could signal a new beginning.

OCF Lab Reopening!

2009-02-06T15:57:00.000-08:00

The OCF lab will be reopening on Monday, February 9. We have a shiny new lab in the basement of Eshleman Hall, a cool lounge just for hanging out, and Airbears!

So come by to check it out next week, and thank you for being so patient with us for the past seven months!

Unexpected Downtime

2009-02-03T21:40:00.000-08:00

We're currently experiencing some hardware problems on one of our core servers. We are investigating the issue and will post updates here as we determine the scope of the problem.

UPDATE: Disk array issues again. We're running a check on the data, and we expect to be back online tomorrow morning. Sorry for the inconvenience. We understand that it's a school night, so please let us know (sm @ ocf.berkeley.edu) if you need emergency access to your files.

UPDATE (1:01 AM): The check completed successfully, and we're bringing all OCF services back right now. Let us know if you notice anything strange with your files.

General Meeting on February 5

2009-01-27T09:59:00.000-08:00

The OCF will be holding a general meeting next Thursday, February 5th, at 7 PM in 189 Dwinelle. Hear the latest updates on our lab relocation, meet the staff who volunteer daily to keep your services running, and elect this semester's OCF managers.

Everyone's invited, so come on by for food and socializing!

Login Server Upgrades

2008-11-01T09:11:00.001-07:00

We just upgraded the operating system of one of our login servers, conquest.OCF.Berkeley.EDU. The machine now is available for general use and will be re-added to our login server alias once we're sure that everything works. Feel free to give the server a test drive, but please note that we might have to reboot it a few times in the coming days.

This operating system upgrade allows us to deliver greater security and better application support to our users. In fact, I'm pleased to announce that we now support Mercurial, a popular distributed version control system.

We'll also be upgrading apocalypse.OCF.Berkeley.EDU soon. If you really need a place to work uninterrupted this weekend, you can try our Linux login server, tsunami.OCF.Berkeley.EDU.

Power Outage

2008-10-22T17:49:00.000-07:00

There will be a power outage in the building housing our core servers between 1-6 AM on Thursday, October 23. We anticipate shutting down our servers around midnight tonight and powering them back on as soon as we can.

I apologize for the short notice, but we were just notified today.

Maintenance Complete

2008-10-18T20:59:00.000-07:00

We've finished our maintenance ahead of schedule, and all OCF services should be working again.

Thanks for your patience and understanding!

Planned Maintenance

2008-10-18T00:29:00.000-07:00

The OCF is currently down for maintenance. Some of you may have noticed the unplanned downtime this past Tuesday, which was related to errors with our disk array, which houses mail as well as many of the critical files for the OCF. This downtime is meant to help fix some of those problems and stabilize our servers.

We don't expect it to last past 9 AM on Sunday, October 19.

Unplanned Downtime

2008-10-14T13:25:00.000-07:00

We began having problems communicating with our disk array this morning. Since most of our data is stored on the disk array, many of our services are currently offline.

We are working on fixing the disk array. The data on the disk array is backed up daily, so even if the disk array is broken, all data should be safe.

UPDATE: We've brought back all our services as of 3:30 PM. Let us know if anything is behaving strangely.

iPhone-tailored OCF website

2008-10-07T00:59:00.000-07:00

Been thinking about doing this for a while now, and as of today, owners of the iPhone and iPod Touch can point their devices to: ocf.berkeley.edu/iphone for a slimmed down essentials-only mobile version designed specifically for this viewport. There are some goodies in there, including the phone number linking to your dial pad, or the map location linking to your built-in Google Maps.

I've tested it against the iPhone SDK's emulator, and against Safari in general, and we'd love any feedback you have. For those of you without these devices, you can play around with it with TestiPhone.com. Or you can simply view screenshots. And of course, you always have the option to view the full versions of these pages in these devices.

I have plans to develop a more general mobile version for other devices as time permits. Keep posted. Coming to a device near you.

OCF System Maintenance Fall 2008

2008-09-27T20:35:00.000-07:00

The OCF is down for maintenance. We don't anticipate that it will last more than 12 hours, but if it does, we will post updates here.

Buzz

2008-09-18T23:59:00.000-07:00

Check us out at The Daily Clog for an outsider perspective on our Heller-to-Eshleman relocation situation. Free computing is becoming a little hard to come by these days, but we will be back and better before the first snow falls. (Luckily, we're in California.)

FREE FOOD!

2008-09-10T23:32:00.000-07:00

The OCF is holding a General Meeting on Thursday, September 11 at 7PM in 263 Dwinelle.

There will be free food! Come hear the latest news about our lab relocation and meet the staff who keep your services running!

Unplanned Mail Downtime

2008-09-01T10:42:00.001-07:00

At 3AM on September 1, we began having issues with the hardware hosting our mail. We are currently working on identifying the problem and minimizing its impact upon the rest of our services.

We will update this post once we have more information.

UPDATE: It looks like our disk array had some hard drive issues. We were using mirrored hard drives, so we don't think any data was affected. Everything should be working again, but please let us know if you see anything out of the ordinary.

OCF Laboratory Move

2008-06-09T16:23:00.000-07:00

The OCF laboratory in Heller Lounge will be closed throughout the day as we relocate to our new location in Eshleman Hall.

We apologize for the short notice. Please contact us via email or IRC if you have any questions or concerns.

Harder Better Faster Stronger Webmail and Documentation (Redux)

2008-05-25T19:26:00.000-07:00

We just upgraded our webmail and documentation server to a machine with more than four times the processing power and RAM. Combined with the changes we made last semester, our webmail and documentation should be faster than ever.

We haven't forgotten about our other services, though, and we're working on moving them to faster and more reliable hardware this summer. We know that many of you have been suffering with the occasional slowness in our services, and we're working on fixing it! Please be patient as upgrades to our other services require lots of planning and testing.

Before I go, I'd like to extend a special thank you to Sun Microsystems, who provided us with the hardware that made this upgrade possible. They make awesome server hardware!