Power Outage in Area where most openSUSE Servers are Located

Just a quick note: We have a power outage in the part of the city of Nürnberg where the Novell office and the main server room is.  This means that many of our servers are right down, especially the download redirector, the mailing lists, the openSUSE build service and users.opensuse.org.

I will post a message once the power has been restored and all machines are running again.  Current estimate (11am Nuernberg time) is that it will take another 4 hours (until 3pm Nuernberg time which is 13:00 UTC) at least to restore power.

Note: the power companies do not know yet exactly where the problem is.

This server and the wiki are located in another data center and are therefore available.

Updates:

13:15 CEST: New rumor: Current estimate for power restoring is six more hours, they need to dig up the street.

16:45 CEST: Bad news: It will take longer until power gets restored.  The local power company just stated “22:00 to 23:00”.  We will try to get then the first machines up but might not get everything running during the night.  Btw. currently it seems that it’s only our office complex that is without power, the rest of the area has power again.

17:15 CEST: I just chatted with our admins, and they currently hope to have everything up Saturday around 13:00 CEST (11:00 UTC) if - and only if - there are no major problems like hardware failures.

18:05 CEST: The admins will start early tomorrow morning - there’s no sense waiting for the power company this night.  The estimate stays at 13:00 CEST (11:00 UTC).  We’ve never experienced such a long outage before, this is exceptionally bad.

19:02 CEST: Beineri has uploaded some photos from the construction site (thanks!).

20:04 CEST: Marko has uploaded some photos as well (thanks!).  Some notes: I’ve heard (no official confirmation) that our office building has two power lines and currently both are getting repaired, they started with the first one and now dig out the second one as well.  Our building seems to be the last one in the area to get power back since it’s the only one with a 20kV line.

1:30 CEST: Power is back in the office - later than estimated.

9:20 CEST: Our admins have brought the basic net infrastructure up and will work on the rest now.

9:45 CEST: The first servers coming up, download.opensuse.org is available again.

10:20 CEST: lists.opensuse.org is up again, I’ve send an announcement out to the mailing lists.  I just don’t know when it will go through since some other systems are not running and I guess the mail queue is rather long.

10:33 CEST: After I approved my announcement, it went through directly and was sent out - this means, the infrastructure is indeed up and runing ;)

13:00 CEST: Most systems should be up, the only problems right now are login on users.opensuse.org and the build service.

15:00 CEST: Info from our admins:

It has turned out that the electric feeder cable outside the building was blown which had to be digged out and then repaired, so the first estimation  of the energy provider was a little bit optimistic. Connection was re-established Friday night at about 1AM (localtime) and reconstruction started this morning at 7AM and most important services were back at about 9AM.

15:08 CEST: We’re still working on users.o.o and the build service, everything else should be ok.

18:50 CEST: users.opensuse.org and build.opensuse.org are back online.  We should now be good enough for the weekend.  Currently still down are ideas.o.o, features.o.o and tracker.opensuse.org (for our torrents).  We will have these restored on monday.

20:18 CEST: tracker.opensuse.org (for torrents) is running again.