Archives

Invisible In America

Very early one morning the Chief Engineer received a mysterious phone call. “Would you be interested in doing a story on my building?” The answer was an astounding “Yes, we are always interested in a good story.”

“ But there are a few conditions,” the caller said. “You cannot use the name of the facility nor the location. Still interested?” As he explained the uniqueness of his facility and the reason for the secrecy, we understood.

So here we are in the middle of Nowhere, USA, reporting on a building that is one of only four buildings of its kind. What we can tell you is that the extraordinary measures taken to build this structure can be explained in only one word – redundancy.

Our technological world is controlled by a series of 1’s and 0’s. These numbers combined can tell anyone, anywhere, anything they want to know about you, provided they can get at and interpret the data. Thus, the reason for the amount of anonymity surrounding this building. It handles millions upon millions of data transmissions on a daily basis. The redundancy built into this building to ensure that there is no interruption in the flow of this data is spectacular, and was taken into consideration from the ground floor up.

Remarkable diligence was used in scouting out the precise location of this building. While most companies search for a building already capable of providing the necessities, the exact opposite was true in this circumstance. This 152,499-square-foot shell was chosen, and ultimately purchased, with the dirt floor as a prerequisite. With construction completed in less than seven months, the Tier 4 building now guarantees 99.99999% up time; In other words, the center guarantees they are uninterruptible by design.

We spoke that day with Dan Hickey, Vice President, Critical Environment, Rick Schuknecht, First Vice President, Critical Environment, and Tom Tabak, Technical Engineering Manager, Emcor Facilities Services, Inc. Dan, Rick and Tom have been with the building since its inception. Dan is a long-time member and Past President of The Chief Engineers Association. He is married to wife Patti, and they are kept very busy with three girls. Tom is also a member of the Chief Engineers Association. Attending as many meetings as possible, Tom tells us that the Association plays an important part in keeping him abreast of the vast changes taking place in today’s engineering world. Rick has retired from the Navy after 25 years of service. He has come here from California and is married with four grown children and four grandchildren.

Dan started by telling us that security is part of their guarantee of 99.9% accuracy. Every door on the premises has a badge reader that must be used before gaining entrance to the building. In addition, critical areas are protected by a hand reader. Cameras follow the movement of people in every hallway, and prospective employees undergo a rigorous background check, fingerprinting and drug testing procedure. One major project Rick took on was to develop a “working in critical environments” protocol which must be adhered to by each contractor hired to do work. The contractor must meet with Rick prior to beginning the job and agree to stipulations set forth by the company before being allowed to begin. The contractor must then sign off that they understand the standards of work rules for this facility.

To further ensure that the rules are being followed, a staff engineer is on the worksite at all times.

At the present time there are 15 engineers and two electricians who maintain all the equipment. Right now that equates to about one engineer for every 10,000 square feet of space. These numbers are reviewed annually; and while there is plenty of work to go around, the main concern is not to overtax the employees working in such critical circumstances. “This site doesn’t run if they don’t run it,” Rick told us. “And we will add to the staff when and where necessary.”

Beginning with the requisite dirt floor, crews dug down sixteen feet into the ground to lay the electrical conduit. In addition, the HVAC piping is located underneath a three-foot raised floor. Removable panels allow the engineers to maintain the equipment with ease. As we began our tour, we only needed to keep one thing in mind – redundant. What we were about to look at on the west side of the floor actually had a duplicate standing on the east side of the floor. If one entire side of the building were to be taken down for out-of-service maintenance, the other side would pick up the load until such time as the work is completed. This was a critical design criteria.

Further ensuring that this building will not experience any downtime, six air-cooled, Caterpillar diesel generators at 2,000 kW each provide standby power should it be called for. The generators are equipped with dual redundancy starters and battery chargers that have been added on. They are supplied by six, 400-gallon fuel tanks and backed up by two 15,000-gallon underground tanks. These engines are test run once each month to ensure that they are in perfect working order at all times. In the event of a catastrophe, the backup generators will allow the facility to run 24/7 for approximately one week.

But wait, that’s not all. Over 2,000 batteries in total stand ready for use in the UPS system. Each battery cell is constantly monitored. One side of the data floor will be fed by two UPS systems and the other side is fed by two separate UPS systems. Each and every piece of equipment is dual ported, meaning that one cord goes to each UPS system. This way if one side should fail, the other side picks up the load. Surprisingly so, (or maybe not), there is another identical bank of batteries located in another area of the building identical to this one. Ah, the epitome of redundancy.

The cooling tower system is another unique feature of this building. While most cooling towers and related pumps are located entirely on the roof, their four 1,200-ton cooling towers are on the roof while six 2880 GPM primary, eight 1440 GPM secondary and six, 3600 GPM condenser water pumps are located inside the building. Four, 1,200-ton York centrifugal chillers handle the chilled water needs of the facility. Water leaving the condenser is maintained at a minimum 50 degrees F. This allows the engineering staff to move in and out of free cooling while operating the chillers. In addition a backup water supply of 150,000 gallons stands ready and waiting. And, yes, there are two separate, identical chiller plants.

So what is all this equipment necessary for? Well, as we said, this particular facility is responsible for the transmission of millions of bytes of information on a daily basis. The most amazing part of the building is the data floor.

It was a stark white surrounding we entered. A strict 72 degree F temperature is maintained year round so it was a bit chilly as we walked through. A raised, open ceiling of 21 feet let the imagination run with the vast area of this room. As we looked around the reason for the redundancy suddenly became clearer. Here was where the critical part of this operation was located. Rows and rows of servers stood ready to move their information. Two fiber optic feeds facilitated the movement of this high traffic information. While it seemed to us that there was already an enormous amount of servers here, Dan told us that eventually they could see the number rising as high as 7,000 servers here.

Fire protection was another critical area of design for the building. The data room is equipped with flash fire indicators and smoke detection sensors have been placed above and below the raised floor. Sprinklers are installed in the event a fire should break out.

Because a virtual 100% uptime is a requirement for this building, training is an important issue. Assimilators provide a variety of power outage scenarios to train the staff how to handle select emergencies. Engineers are kept up-to-date on all the latest technology changes affecting their facility. Constant communication has been established with the critical environment readings through the use of laptop computers. But holding true to security issues, these laptops also have a high level of security installed on them. Fire walls prevent access to sites that have been deemed dangerous of carrying viruses. A list of programs are automatically prevented from being loaded and in the event that they would make it onto the laptop, signing on to the network will instantly remove that particular program from the computer.

We were amazed at the scrutiny paid to each and every aspect of getting this facility up and running. Security and redundancy were the key words for the day. There was no doubt that with the exception of a nuclear disaster right on this particular corner of the street, this facility will still be running even under the worst scenario. The high level of security achieved here actually made me think that the next time I used my ATM card, I could feel totally confident in the fact that at least if my information were being funneled through this facility, there was no danger in anyone finding out how poor I actually am.

Archives