Working late...

Ugh. Just got home from a long 16 hour day.
A major electrical circuit that supplies power to the data center began failing. We only caught it because of the smell. YIKES!

Well we spent most of the day running extension cords and installing spot coolers so the ICU would not be down. No wrist shots, but here are a few iPhone photos.

We have to rebalance the loan in these three racks.


Another place where some cable management is needed.


Gotta love an impromptu UPS holder


Lectric stuffz


The bowels


Time to sleep. Another long one tomorrow.

By the way I wore this guy and wish I was there instead.

Sucks when everything blows up on you.
(In the entertainment business, nearly every day is a 16 hour day. )

porschefan wrote:

Sucks when everything blows up on you.
(In the entertainment business, nearly every day is a 16 hour day. )

No kidding - exactly why I had to leave it.

What is this thing called work?

jaw wrote:

porschefan wrote:

Sucks when everything blows up on you.
(In the entertainment business, nearly every day is a 16 hour day. )

No kidding - exactly why I had to leave it.

Damn...and I thought my 12 hours shifts were long.

Glad you caught it before you had a cascade of failures!

What do your servers manage Poli...that was a serious electrical room!

DAMN! The breaker is not working. What does that mean? No AC in the data center. Spot coolers are in place but we're holding on barely at 80F and climbing.

PoliMalaka wrote:

DAMN! The breaker is not working. What does that mean? No AC in the data center. Spot coolers are in place but we're holding on barely at 80F and climbing.

Isn't this a line that Scotty said to Kirk in one of the Star Trek movies?



Jaw, I haven't worked one of those entertainment days in a year and a half ... but my wife still does.

Sucks when that happens..................

It's been a fun week here, as well......I had this in my e-mail when I arrived at work Monday morning (I X'd out file references, since I'm too dumb to know if it's proprietary...but you still get the picture):

"This past Friday night we experienced a catastrophic loss of the data included on file server XXXX. This is our main file server....

Due to the the amount of data involved and the subsequent time needed to restore the files, the data will not be available for use until Monday afternoon at the earliest."

It was followed up by this one on Tuesday:

"Our data restore is progressing, although not as quickly as we would have hoped. While the good news is still that no data has been lost, returning it to a useable form is now estimated to take until Friday morning at the earliest."

Fun stuff.....

Holy COW !!! Look at ALL of that !!! AMAZING stuff Poli !
I hope things begin working a little smoother for you soon
with this. At least you still had a COOL watch there with
you to keep you company sir ! ..Some astounding
pictures here ! Thanks !

I feel your pain.....we have control rooms like that all over. I keep the juice flowing so the automations guys can keep the servers running. Good luck Poli!!

That electrical panel you show is a Square D model 5 Motor Control Center. The MCC is likely 480 volt 3 phase and feeds circuits in the server room through a transformer and sub panel more like you would see at home. A breaker can fail in several ways but a device failure on the load side is much more common than an issue internal to the feeder breaker itself or the line side of the breaker . Could be as simple as a bad contact in the breaker. Faulty breaker could be in the sub panel or an issue with the transformer itself. Should'nt be much of a problem for a qualified electrician. Unless you have a standby battery system or generator with auto transfer switch, an outage may be required to repair it. Good luck-I hope you guys get it straightened out before it ruins the weekend.

merichar wrote:

That electrical panel you show is a Square D model 5 Motor Control Center. The MCC is likely 480 volt 3 phase and feeds circuits in the server room through a transformer and sub panel more like you would see at home. A breaker can fail in several ways but a device failure on the load side is much more common than an issue internal to the feeder breaker itself or the line side of the breaker . Could be as simple as a bad contact in the breaker. Faulty breaker could be in the sub panel or an issue with the transformer itself. Should'nt be much of a problem for a qualified electrician. Unless you have a standby battery system or generator with auto transfer switch, an outage may be required to repair it. Good luck-I hope you guys get it straightened out before it ruins the weekend.

In surgery a 16 hour day would be welcomed. 24-36 is not abnormal... I had a colleague in transplant surgery who worked 3 days with only 4 hours of sleep total.

merichar.
That is not the panel that is jacked. The one you see there controls the air handlers and a few other things.
We could not run the generators as it would do no good. The "equipment branch" has (yes still has) the problem. Which means we could not run the generator so the panel could be de-energized.

The problem was NOT fixed. We have spot coolers in the data center and will have another downtime tomorrow night. We'll be running extention cables for the routers, PBX, and mission critical servers. The ICU needs to be up so they can chart on the kiddos.

Ahhh... Gotta love being a dork.

More pics of the main panel tomorrow night.

Ah yes another late night. Generators. Mmmmmm

Turbo Diesels... I thought you did computers.

Ah yes, more problems
The breaker keeps tripping.











Tim,
We can't use kumpyooterz without the lectrisitee.

Well I guess the problem spilled over onto the weekend. You can definitely see a problem with the phase C stabs. The plasma arcing indicates the breaker is doing its job-open the circuit to isolate the problem and prevent a fire or catastrophic failure. I hope you guys find whats causing the issue downstream soon.

Thats not minor arcing.Whats feeding the trunk line to that breaker?

Three different contractors an not a one has a clue. Here's the problem. The condensers and air handlers are on this circuit. So we're hot and sticky. Blech. Thankfully we can run a few things to minimize the impact on the overall building.

Motegi wrote:

In surgery a 16 hour day would be welcomed. 24-36 is not abnormal... I had a colleague in transplant surgery who worked 3 days with only 4 hours of sleep total.

If not for nothin I don't think I'd want to be the final patient your colleague performed surgery on!

Yowza, some mean machines right there ! GREAT photos too !
I hope things get better with all of this soon !