Data Center Managers?

[AllVols]

Are there any Data Center Managers out there? I would like to talk with people that might work in a Data Center, bounce ideas off each other, and just talk about daily issues that you deal with running a data center?

A little back ground. (I will not name the companies I have worked for and or currently work for), but I have managed DC's for two different companies.
My first DC job was a for a global company, and the main DC was in (Lets say Germany) we had DC's in several countries, but each DC had at least 2 guys managing the daily operations, Power, cooling, cabling, UPS's, asset management, etc...
We operated almost independently as far as standardization but ultimately our budget and over all objectives came from the mother ship in Germany. Great job., and learned a ton there.

I now work for a company that has no where near that foot print. We have 4 DCs through out the U.S. country. They had hired one guy in site 1 in another state and found out quickly he couldn't handle the job by himself. That is how I got hired. He was responsible for all four DC's and I backed him up in Site 2 and also handled all the generators, 8 in total, 4 at the DC's and 4 at other sites that need 100 percent up time. plus any kind of preventive maintenance (PM's), as well as standardizing MDF's and IDF's in large and smaller sites. Needless to say I stay busy.

Now, the DC manager has left the company at site 1, and I'm the guy managing all four DC's as well as all the other sites. The company I work for has hired a guy at site 1 to back fill my job, so that has helped a little I'm still very very busy trying to train him and deal with day to day operations.

Last year I went and finished my certification for International Data Center Authority IDCA. They handle the DC managers job up a little different that what I'm used to, but wouldn't mind bringing in more help at each site and using this model, but that isn't my call

Anyway I just wanted to see if there were any other people on this board that might be able to relate and as I said throw ides around.

 

[CopperHead_Vol]

I would be glad to discuss and bounce things around. Sounds like you have your hands full walking into this new role.

 

[Hurley Mooncalf]

I have a little knowledge on the subject so I'll continue to follow the thread.

 

[AllVols]

I would be glad to discuss and bounce things around. Sounds like you have your hands full walking into this new role.

I've been doing DC work for about 15 years. This new company I started in 2019. Anyway, I just wanted to hear how others run there DC? Do you have hot cold isle? isle containment systems? what kind of monitoring software do you use? What size (tier) DC do you run or work in?
We use Solarwinds, but that's very limited, unless you can get someone to package the apk packages, which I know very or less than very little about. At my first job we used SE Netbotz for monitoring the DC the software was called infrastruxre. They have re-packed it now and it's called struxureware (spelling is correct) anyways, it is much more robust compared to what I used to use and Solarwinds.

 

[CopperHead_Vol]

I oversee IT managed services operations for a global gov contractor. At this stage in my career, I have my hands in most all facets of high availability IT service delivery but not in the weeds on any one specific area. My earlier background is in global DC operations, NOC/SOC , system engineering, security and administration, etc....

Response to your questions....
Yes on hot/cold aisles
Cold Aisle containment system along with vented access floors for hot side...
monitoring -CA currently migrating to SolarWinds
Two main DCs are TIA 942 Tier 4 designated sites.
The StruxureWare tool set is nice and would provide a more holistic approach to DC monitoring/management. Looks like it checks all the boxes for sure.

What does your operational staffing coverage look like? Any clear set process and procedures already in place for you coming in the door? or yours to define?

 

[AllVols]

As for staffing, it's just me and one other guy, what I do is more management than having my hands busy. For "one off's" like cable runs from one rack to another I will do that myself. We are currently right in the middle of three huge projects. They tie into each other. (Project 1) We are closing 2 of our DC's and migrating everything from sites 3 and 4 to site 1 where I sit and splitting a few other things, AS400 and Vblock to site 2. To get that done, (Project 2) our DC UPS's system has aged out so we are going with a whole new UPS's system. We are building out a whole new room for those new UPS's, so I can use the space that the old UPS's system used for more rack space. (Project 3) We are bring in millions of dollars’ worth of VXrails, that run very hot, to visualize all the older physical hardware from site 3 and 4. So, back to your question I have to bring in a lot of contractors for help. I can't wait to get sites 3 and 4 shut down. I'm sick of flying to these states.


All of our sites are, if you stretch it a tier 2. We had looked into trying to get to a tier 3 solution, but they stopped short due to zero space to add extra gens at 3 of the 4 sites and getting a second leg of power from the power company at site 2 was running upwards of 2 million, and before they decided to close sites 3 and 4, three times that. Sites 3 and 4 are right in the middle of very large cities.

Oh, actully we are also changing our network solutions as well, going from the older Nexus/FEX Cisco type switches, that required a lot of home runs from rack to rack, to the newer ACI switches with Leafs (all in rack cabling) to splines. That's been a lot of late night re-cabling fun.

So, why are you going to Solarwinds? your call, or like me, way way above your head?

 

[VolInNorthCack]

I used to run the NIM (Network Infrastructure Monitoring) at a major bank. We used NetQoS from CA. I left there about 6 years ago and I think they have migrated to the new CA product. The new place I am at is much smaller and only 1 DC with offsite at SunGard in NJ. I haven't done any DC management in probably 10-15 years though. I am in IT/Network Administration now and working on creating and building out our MDM network. We use Solarwinds here for our monitoring. I will be glad to help out if I can; but it's been so long since I have done anything in a DC environment exclusively, that everything has changed so much and I doubt I would be much help.

Will follow along on the thread though as I am probably going to be changing jobs in the next 6-8 months (finally moving back to TN) and who knows, I may need to brush up on some things and this thread may help me do that.

 

[CopperHead_Vol]

As for staffing, it's just me and one other guy, what I do is more management than having my hands busy. For "one off's" like cable runs from one rack to another I will do that myself. We are currently right in the middle of three huge projects. They tie into each other. (Project 1) We are closing 2 of our DC's and migrating everything from sites 3 and 4 to site 1 where I sit and splitting a few other things, AS400 and Vblock to site 2. To get that done, (Project 2) our DC UPS's system has aged out so we are going with a whole new UPS's system. We are building out a whole new room for those new UPS's, so I can use the space that the old UPS's system used for more rack space. (Project 3) We are bring in millions of dollars’ worth of VXrails, that run very hot, to visualize all the older physical hardware from site 3 and 4. So, back to your question I have to bring in a lot of contractors for help. I can't wait to get sites 3 and 4 shut down. I'm sick of flying to these states.


All of our sites are, if you stretch it a tier 2. We had looked into trying to get to a tier 3 solution, but they stopped short due to zero space to add extra gens at 3 of the 4 sites and getting a second leg of power from the power company at site 2 was running upwards of 2 million, and before they decided to close sites 3 and 4, three times that. Sites 3 and 4 are right in the middle of very large cities.

Oh, actully we are also changing our network solutions as well, going from the older Nexus/FEX Cisco type switches, that required a lot of home runs from rack to rack, to the newer ACI switches with Leafs (all in rack cabling) to splines. That's been a lot of late night re-cabling fun.

So, why are you going to Solarwinds? your call, or like me, way way above your head?

Wow, you have your hands full for sure..... a lot of moving parts shutting down sites especially with legacy systems (AS400). I've always tried to do software swings where possible (VMs or new hardware migrating the data).

We are currently on mirrored CISCO/NetApp Flexpod converged infrastructure for two customer facing ITMS DCs. So very familiar with ACI deployments and management.

The migration of monitoring solutions is for our customer facing EOC toolset. CA has lost their mind in their annual maintenance pricing and how they support current customers. This cost is passed on to our end customers, (AAS) model for NOC/SOC service delivery. CA priced us out of continuing to do business with them as we need to keep our rates competitive. I was not part of the selection process just reviewed the selection based on a detailed AoA that was conducted on available options.

 

[AllVols]

Wow, you have your hands full for sure..... a lot of moving parts shutting down sites especially with legacy systems (AS400). I've always tried to do software swings where possible (VMs or new hardware migrating the data).

We are currently on mirrored CISCO/NetApp Flexpod converged infrastructure for two customer facing ITMS DCs. So very familiar with ACI deployments and management.

The migration of monitoring solutions is for our customer facing EOC toolset. CA has lost their mind in their annual maintenance pricing and how they support current customers. This cost is passed on to our end customers, (AAS) model for NOC/SOC service delivery. CA priced us out of continuing to do business with them as we need to keep our rates competitive. I was not part of the selection process just reviewed the selection based on a detailed AoA that was conducted on available options.

Do you have in house maintenance crews to handle PM's on your Gen's and other gear/stuff? We here a national contract with SE. We use Schneider Electric (SE). They are national and basically have contracts with local vendors all over the U.S. to handle everything from generator. CRAC, mini splits, Gen set maintenance... just about anything we need. I've got one number to call and can get someone sent to a site in Mesa AZ to deal with UPS's if needed.

 

[AllVols]

I used to run the NIM (Network Infrastructure Monitoring) at a major bank. We used NetQoS from CA. I left there about 6 years ago and I think they have migrated to the new CA product. The new place I am at is much smaller and only 1 DC with offsite at SunGard in NJ. I haven't done any DC management in probably 10-15 years though. I am in IT/Network Administration now and working on creating and building out our MDM network. We use Solarwinds here for our monitoring. I will be glad to help out if I can; but it's been so long since I have done anything in a DC environment exclusively, that everything has changed so much and I doubt I would be much help.

Will follow along on the thread though as I am probably going to be changing jobs in the next 6-8 months (finally moving back to TN) and who knows, I may need to brush up on some things and this thread may help me do that.

You're changing jobs within the same company? or doing the same kind of work but for a new company? Feel free to chime in anytime. I just like to chat with others about DC's and pick others brains on what they do or how they might have done things in the past. Just batting a ball around so to speak.

 

[VolInNorthCack]

You're changing jobs within the same company? or doing the same kind of work but for a new company? Feel free to chime in anytime. I just like to chat with others about DC's and pick others brains on what they do or how they might have done things in the past. Just batting a ball around so to speak.

No. I will be leaving this job and looking for something in TN. Honestly, I hate this place and I am just about burned out on IT. This may be my last IT job. I might look into doing NOC or even HelpDesk work. Something that is truly 9-5 and not stressful. Working every day and then every night when I get home is killing me lol. I turn 51 this year, so I am at the point in my life where I can just say F-U if I feel like it lol.

 

[CopperHead_Vol]

Do you have in house maintenance crews to handle PM's on your Gen's and other gear/stuff? We here a national contract with SE. We use Schneider Electric (SE). They are national and basically have contracts with local vendors all over the U.S. to handle everything from generator. CRAC, mini splits, Gen set maintenance... just about anything we need. I've got one number to call and can get someone sent to a site in Mesa AZ to deal with UPS's if needed.

That is a good setup to provide one stop support. We have internal facilities and site support groups that manage generator, CRAC, electric, etc...

 

[CopperHead_Vol]

Currently on a all night Zoom meeting with my network and Contact Center services teams working a site related Vlan issue.

 

[CopperHead_Vol]

No. I will be leaving this job and looking for something in TN. Honestly, I hate this place and I am just about burned out on IT. This may be my last IT job. I might look into doing NOC or even HelpDesk work. Something that is truly 9-5 and not stressful. Working every day and then every night when I get home is killing me lol. I turn 51 this year, so I am at the point in my life where I can just say F-U if I feel like it lol.

What part of TN are you looking to relocate to?

 

[AllVols]

Currently on a all night Zoom meeting with my network and Contact Center services teams working a site related Vlan issue.

Oh man you got to love those. I just spent at least 4 hours on a trouble shooting issue from an incident we had over this weekend at site one. I highly recommend getting one lines done of your electrical systems. Some of the re-work we are doing for the new UPS system involves moving air handles off of the electrical panel that is supported by the generator we have. The new subpanel will also be supported by the gen, but things needed to be move around. Anyway the previous guy and electrical company labeled some circuits wrong and when the pulled what they thought were circuits for air handlers for office AC, everything seemed to be working fine. Then about 30 minutes later we started getting calls the DC was over heating. Come to find out those circuits where my chillers on the roof. Needless to say the DC over heated before they could recover. We lost a few device including the F5/firewalls, so the DC was offline for about 30 minutes until I could get temps back down and get coms back up. UGH!!! nothing but fall out over that.

 

[VolInNorthCack]

What part of TN are you looking to relocate to?

Going to be building a house in Wears Valley.

 

[CopperHead_Vol]

Going to be building a house in Wears Valley.

My family is from that area.... was just up at the Little Greenbriar schoolhouse and Walker Sister's cabin a few weekends back.

Reason I asked was when you start this way let me know. We manage 30+ tier2 service desks, NOCs, etc for various gov customers. Sure I could get you first interviews with that or IT positions. Most are working remote at this point and will most likely stay in that ops model going forward.

 

[CopperHead_Vol]

Oh man you got to love those. I just spent at least 4 hours on a trouble shooting issue from an incident we had over this weekend at site one. I highly recommend getting one lines done of your electrical systems. Some of the re-work we are doing for the new UPS system involves moving air handles off of the electrical panel that is supported by the generator we have. The new subpanel will also be supported by the gen, but things needed to be move around. Anyway the previous guy and electrical company labeled some circuits wrong and when the pulled what they thought were circuits for air handlers for office AC, everything seemed to be working fine. Then about 30 minutes later we started getting calls the DC was over heating. Come to find out those circuits where my chillers on the roof. Needless to say the DC over heated before they could recover. We lost a few device including the F5/firewalls, so the DC was offline for about 30 minutes until I could get temps back down and get coms back up. UGH!!! nothing but fall out over that.

Ouch, yep been there done that... latest fiasco was a few years back. Some site manager insisted that the new Flexpod was install with all the pretty lights facing the glass wall. So when he gave tours he could show it off. 5 racks facing the wrong direction for hot/cold airflow in a small Server room. That went well....

 

[VolInNorthCack]

My family is from that area.... was just up at the Little Greenbriar schoolhouse and Walker Sister's cabin a few weekends back.

Reason I asked was when you start this way let me know. We manage 30+ tier2 service desks, NOCs, etc for various gov customers. Sure I could get you first interviews with that or IT positions. Most are working remote at this point and will most likely stay in that ops model going forward.

I appreciate that. Our timeline has been moved back a bit. We were hoping to be almost done by now, but with supply problems and lumber prices that went through the roof, we have held off for this year. Also, the first builder we were working with misrepresented some things to us after planning and preparing for 3 months with them.

I will most likely be looking for something that is almost 100% remote. I used to work from home for 2 or 3 years before I left the bank.

 

[AllVols]

I'm in Nashville this week, at my smallest DC of the four. This one is slated for closure soon. With everyone working from home these days it's hard to get things done or get people to get off their lazy bums and take ownership of their gear when needed.

 

[AllVols]

Alright, here's one for y'all. One of the hats I wear here is I take care of all IDF and MDF's. The standard is the room should be 10x10 if at all possible and have it's own cooling. Normally Mini splits. I have to depend on my site hands and feet at almost all of these sites. I just got a email from site #2 that is my main backup DC. It's a high rise and was in service well before I started here. I just got this picture of one of the mini splits condensers. I've never seen this before. BTW the building has 11 floors. This is on the 3rd floor.

 

[VolInNorthCack]

Alright, here's one for y'all. One of the hats I wear here is I take care of all IDF and MDF's. The standard is the room should be 10x10 if at all possible and have it's own cooling. Normally Mini splits. I have to depend on my site hands and feet at almost all of these sites. I just got a email from site #2 that is my main backup DC. It's a high rise and was in service well before I started here. I just got this picture of one of the mini splits condensers. I've never seen this before. BTW the building has 11 floors. This is on the 3rd floor.

View attachment 406196

Well, that's different.

 

[superdave1984]

No. I will be leaving this job and looking for something in TN. Honestly, I hate this place and I am just about burned out on IT. This may be my last IT job. I might look into doing NOC or even HelpDesk work. Something that is truly 9-5 and not stressful. Working every day and then every night when I get home is killing me lol. I turn 51 this year, so I am at the point in my life where I can just say F-U if I feel like it lol.

An error has occurred.

 

[VolInNorthCack]

An error has occurred.

lol thanks.

 

[VolInNorthCack]

Anybody ever build out an MDM network? I know it's not really DC related, but I am knee-deep in building out an MDM for the company I work for. Just curious to see if anyone has done it and what tool they used.