We apologize for the lack of notification and the lack of status updates, and are grateful for your patience. As you may have guessed, this is not a planned outage.
On April 19th, we discovered that one of our servers had been broken into. We initially tried to contain the problem, but further investigation revealed that the scope of the break-in was worse than we had originally thought. At our Board of Directors meeting on Thursday, April 23, it was decided that many, if not all, of the OCF servers would need to be rebuilt, to reduce the possibility that the hacker would be able to break back into our systems. Because of the nature of the incident, in the interests of security and damage control, the decision was made to begin reconstruction efforts immediately, rather than waiting the customary week to give notification.
Starting that night, various servers have been pulled from the network for forensic analysis and re-installation. However, we have been delayed in restarting some of our services due to an unforeseen difficulty in rebuilding one of our core servers. Until that issue is cleared up, we will be unable to resume anything approaching normal service.
For those of you worried about mail, the mail servers are continuing to accept incoming mail, and it will be available for pick-up once normal services resume.
I'm very sorry that I cannot give you a more solid timeframe than "soon." Please, rest assured that we are doing our very best to bring the OCF back online, and all of our senior technical staff are hard at work fixing this roadblock to normalcy. We ask your patience and understanding, noting that all our staff are volunteers and students who, like you, have class and work obligations, especially with finals looming ever closer.
As always, you are welcome to join us on the OCF IRC channel, irc.ocf.berkeley.edu, with questions, concerns, or anything else you think we should know.
Subscribe to:
Post Comments (Atom)
17 comments:
Thanks for the update, Michael. It's very much appreciated. The silence was getting a bit worrisome!
Good luck to you and staff with getting everything back up and running, and thanks to everyone for your efforts.
Thanks for working on this. This is a big task from a frustrating situation.
Is this break-in tied in with the accounts that were disabled as a result of supsected ssh key interception, and are there plans to re-enable these accounts?
Hi, OCF Staff:
Many thanks for the information. Just so you will know in case this will help in your work:
"I attempted to login via SFTP, got an error message about the change in the key footprint and finally an error that an FTP error occurred with a possible problem with password or login name."
I am assuming that is because of the break in and changes you are implementing.
Again, thanks and hopefully you will have it all resolved and up again soon.
Kindest Regards,
Charles
banway IST AS Release Management
For those who wish to upload files to their website location you can go to the OCF FTP web based application.
I just uploaded some htm files to my web site via this web URL:
https://secure.ocf.berkeley.edu/net2ftp/index.php
So it does work until they finish fixing the various servers.
Charles
I received a "WARNING" that host identification has changed and that it is not recommended that I login until I have contacted the system administrator. I chose not to proceed and did not login. Does anyone know what this message might mean? It said either the administrator of the remote host has changed the host identification, the SSH protocol has been upgraded from SSH1 to SSH2, or someone could be eavesdropping on me right now (man in the middle attack). The message gave me a lengthy fingerprint of the host public key.
p.s. - this happened just now when I tried to login to my email reader, which is in Pine/unix mode.
the rsa keys have changed on the login servers. This is expected given recent events---but it would be nice for ocf staff to confirm the new key/fingerprints for login servers. The main web site is down, the keys are normally published there.
Can I go ahead and try to log in, then, or shall I wait longer?
The login servers are NOT ready for general user login. We are still in the process of rebuilding them, and they're only online to allow staff to work on them.
We'll make an announcement when they are ready for use (along with information about the new SSH keys).
The main website actually is up, anonymous -- ocf.berkeley.edu goes to conquest/apocalypse (the login servers, which are down) and is supposed to redirect to www.ocf.berkeley.edu (the web server, which is up).
I know we cannot log in to our email accounts now, but is it possible to have incoming email to my ocf address automatically forwarded to another email address, or are all incoming emails being bounced back?
I am able to access my OCF email through Thunderbird at home and Outlook Express at work, but not able to send out mail from my Thunderbird client (something to do with a uclink certificate). My question, though, is there a way I can check my quota while OCF login is disabled, in order to get my quota under limit?
Stephen, can you post instructions on how to configure Outlook Express to access OCF email? (I normally use Pine, SSH, or webmail but none of those are working right now, and I am eager to read my email which I am sure has piled up over the past week.) Thanks!
"Stephen Denney said...
I am able to access my OCF email through Thunderbird at home and Outlook Express at work"
Kristine,
I've restored our documentation server. All the information you need should be available at http://docs.ocf.berkeley.edu/wiki/.
doc server is up, but has the ssh keys page been updated?
Is everything right, or not right? It looks lioke things are up but when I try to log in it says I've got my password wrong -- which I don't. So either my password got changed in some fascinating way, or something's still f*cked up, but I can't tell which.
Will we be notified when ssh is up and running again?
Post a Comment