The Pre joins OCF Mobile

Last year, we launched our mobile site to load for the iPhone/iPod touch, and right after included other devices like the Blackberry Storm, Android-based G1, and Opera Mini (such as on the Blackberry Curve beyond the BB browser). All other devices could still enter the site manually through m.ocf.berkeley.edu.

The Palm Pre has now been added to that list. Data detection works just as it does with the iPhone OS, so your phone dialer, Google Maps, and email client will work with our pages there too. I'll be on the lookout for Windows Mobile devices to test upon.

Unexpected Downtime

OCF servers and services are currently experiencing unexpected downtime, due to what may be a Denial of Service (DoS) attack on the subnet the OCF is on. Most staffers are currently out of town due to the summer break; while this is being looked into, it's a bit difficult without connectivity to the servers.

Further updates to come on this blog. While our Internet Relay Chat (IRC) server is down, you can connect to CSUA's IRC server at irc.csua.berkeley.edu and join us in #ocf for updates (mibbit IRC client).

2:18 PM: It seems like we're back online, but the causes of the outage is still unknown. No data loss is anticipated; please let us if otherwise.

apocalypse accepting user logins

apocalypse.OCF.Berkeley.EDU (192.58.221.245) is now accepting user logins. You should be able to connect to the server via SSH/SFTP to access your home directory and web space. If you're comfortable using a command-line mail client (e.g. mutt or pine), you'll also be able to read and send mail.

There are a few lingering issues with the server. In particular:

• Public key authentication does not work. We're working on fixing this problem. Fixed.
  
• Some pre-installed software and user commands may be missing or broken. We had to re-compile all of our software for the rebuild, and it's an ongoing process. We prioritized the most commonly used programs, but leave a comment if you think we've forgotten an important program. UPDATE: We know the software waitlist is building up, so we're getting more staff involved in this process.
  
• Password changing does not work. Use kpasswd to change your account password. Windows password changing is still broken.
• Cron jobs are not yet available. Cronjobs should now work as documented on our wiki.
  

The SSH host keys are:

• DSA:
  04:68:af:14:ab:a1:4e:79:c3:42:6f:51:62:10:a2:5a
  xivit-hunuc-kogur-hizez-pazyr-bynat-pytop-zitos-dufuf-danec-sixax
• RSA:
  72:16:9c:7e:7b:cc:a0:47:56:6d:e3:59:aa:04:bc:7c
  xipav-pizem-lysyd-kicyt-sococ-macuz-hyluc-vizoc-cotem-latig-muxyx

Thanks for your continued patience!

OCF downtime: frequently asked questions

We've been getting a lot of questions over the past week, and (understandably) many of them are similar, so I thought I'd collect the answers to the most frequently asked ones here.

We thank you for your patience, and we'll continue to provide updates via our main website and this blog as things change and we get back on our feet.

Q: Can I access my email?
A: As of a few hours ago, there's a temporary webmail interface available at https://webmail.ocf.berkeley.edu/. Sending email from it doesn't work for the moment, and it lacks the spit and polish of the usual webmail interfaces, but you should at least be able to read your email. POP and IMAP access is still available, and you can send email using SMTP -- see our documentation (now also available again) for information on how to set this up.

Q: Can I access my files on the OCF?
Q: Can I modify my website hosted on the OCF?
A: The usual SSH and SFTP access is unavailable at the moment. For now, you can use the web FTP application to upload, download, and edit your files.

Q: What happened to my account application?
A: We can't process account applications at the moment. We'll get around to it once we have our services up and running.

Q: Can I use SSH or SFTP?
Q: I tried to log in via SSH/SFTP, but I'm getting a message that the host key changed.
A:The primary login servers that you get by logging into ocf.berkeley.edu are turned on again, but aren't accepting user logins at the moment. We'll let you know when you can log in to these machines again. The host keys have changed, and we'll post the new key fingerprints, along with instructions, when we're ready to allow logins again.

Q: So what's this about being hacked? Should I do anything to make sure I'm safe?
A: We recommend the following:

• If you have other accounts which share the same password as your OCF account, those passwords should be changed immediately. Note that reusing passwords is a terrible idea for exactly this reason.
  
• If you have logged into another machine from an OCF system using SSH, Telnet, or the r* commands using a password since April 15th, that password should be changed immediately.
  
• If you entered a password on an OCF system, especially a Solaris or Linux system, at some point since April 15, consider changing that password.
  
• If you kept an SSH private key on the OCF, immediately prevent it from being used to log in to any systems (remove the corresponding public key from authorized_keys everywhere it's listed).
  

We will be requiring that all users change their OCF account passwords soon, but you cannot do this now, nor do we recommend doing it now.

If you have other questions, feel free to email us (staff [AT] ocf.berkeley.edu).

We apologize for the lack of notification and the lack of status updates, and are grateful for your patience. As you may have guessed, this is not a planned outage.

On April 19th, we discovered that one of our servers had been broken into. We initially tried to contain the problem, but further investigation revealed that the scope of the break-in was worse than we had originally thought. At our Board of Directors meeting on Thursday, April 23, it was decided that many, if not all, of the OCF servers would need to be rebuilt, to reduce the possibility that the hacker would be able to break back into our systems. Because of the nature of the incident, in the interests of security and damage control, the decision was made to begin reconstruction efforts immediately, rather than waiting the customary week to give notification.

Starting that night, various servers have been pulled from the network for forensic analysis and re-installation. However, we have been delayed in restarting some of our services due to an unforeseen difficulty in rebuilding one of our core servers. Until that issue is cleared up, we will be unable to resume anything approaching normal service.

For those of you worried about mail, the mail servers are continuing to accept incoming mail, and it will be available for pick-up once normal services resume.

I'm very sorry that I cannot give you a more solid timeframe than "soon." Please, rest assured that we are doing our very best to bring the OCF back online, and all of our senior technical staff are hard at work fixing this roadblock to normalcy. We ask your patience and understanding, noting that all our staff are volunteers and students who, like you, have class and work obligations, especially with finals looming ever closer.

As always, you are welcome to join us on the OCF IRC channel, irc.ocf.berkeley.edu, with questions, concerns, or anything else you think we should know.

OCF downtime

The OCF login servers (conquest and apocalypse) and lab machines are currently down for maintenance, and many services in the lab itself are unavailable. We apologize for the inconvenience and the delay in communicating this, and hope to be able to provide more details soon.

New Printers

We've purchased and installed two Xerox Phaser 4510DT laser printers to replace our old, broken, printer.

Let us know if you have any problems using the new printers!

Unexpected downtime

Sometime this morning, our core fileserver started having some disk issues. We're working on figuring out what exactly went wrong and fixing it, but in the meantime, the OCF is effectively down (no lab, no SSH, no web, etc). We'll post updates as we have them.

UPDATE: The problem seems to have been fixed, though we're still not entirely sure what it was. We'll keep working on it throughout the day. In the meantime, though, the OCF is back up and ready for business.