Data Center and Telecommunication systems power and grounding

In a Data Center and Telecommunication POPS (Point of Presence), Wire Centers/Central Offices proper powering and grounding is critical to dependable operation of network and computer equipment. Bell Labs originally developed NEBS Network Equipment Building Systems in the early the 1970’s to address many issues important among them is power and grounding. Then with the divesture of AT&T into the RBOC’s (Regional Bell Operating Companies) they continued basic tenants of NEBS through to Telecommunications Act of 1996 were the local phone company was now known as the Incumbent Local Exchange Carriers (ILEC), the Long Distance Carriers now known as the Interstate Exchange Carriers (IXC) and then lastly the Competitive access providers now the Competitive Local Exchange Carrier all use a flavor of NEBS. Now as Cable TV companies start looking like a cross between CLEC’s and IXC’s and the Cable TV Headend is also becoming a Data Center/POP/Central office. It appears that some Cable Companies like Charter and Comcast are discovering the Value of the NEBS standard used for many years in the Telecommunications industry.. Especially now that Cable MSO’s (Multiple System Operator) are in the Voice (digital and VOIP) and Data business some are becoming aware of NEBS, especially as have started co-locate facilities with the traditional telecom providers.

Most of NEBS is involved in the development, manufacturing, testing and certification of Network Equipment
“NEBS Level 3” is a term derived from a Bellcore special report SR-3580 it details 3 distinct functional levels of NEBS compliance based on GR-63-CORE and GR-1089-CORE.

“NEBS Level 1” defines how people and equipment hazards are mitigated along minimizing network degradation.

NEBS Level 1 addresses the personnel and equipment safety requirements of GR-63-CORE and GR-1089-CORE. It provides guidelines for prototypes in lab trials. It’s requirements are typically imposed on Competitive Local Exchange Carriers (CLECs), ISP’s and IXC’s when they co-locate equipment into an RBOC’s network wire centers.

“NEBS Level 2” Assures limited equipment operability, but in a controlled environment or normal environment. Equipment used in data centers or in fault-tolerant services that support Applications. This is a very ambiguous level and is rarely used.

“NEBS Level 3” This is Carrier Class Equipment that has meet this stringent level. This equipment that operates under the environmental extremes found in a central office. Basically it is equipment that meets all of the requirements of GR-63-CORE and GR-1089-CORE.

GR-63-CORE (Central Office Relay Equipment) is mostly as manufacturing testing standard for network switching systems that need to certified before they can installed in a NEBS compliant site. The GR-63 parts included in the NEBS testing involve, space requirements, environment, temperature, humidity, altitude criteria, fire resistance, acoustic noise, earthquake, equipment handling criteria and transportation vibration.

GR-1089-CORE is a standard for Emissions Control (EMC) and Electrical Safety. It involves:
System-level Electrostatic Discharge
Electromagnetic Interference
Lightning and AC Power Fault
Electrical Safety Criteria
Bonding and Grounding

This chapter will focus components of GR-1089-CORE related to power and grounding and other elements that would be recommended best practices to follow that not yet defined under any specific standard.

Here is a link for more details on NEBS: http://www.nebs-faq.com/

To really understand how a grounding system for Telecommunications equipment like a PBX, router, a Windows 2003 or UNIX server is done properly we need to understand grounding is an integral part of the power distribution system.
In the Power grid the ground is an integral part of an un-balanced transmission system using earth ground as the return. This makes building Power lines much cheaper but more dangerous. In North America Power utilities use a 3 phase power distribution system (Figure 22), common voltages for three phase systems in North America include 4-wire Delta Systems with a phase to phase voltage of 208VAC/60Hz, or 480VAC/60Hz. It is also common to see 480VAC/60Hz 5-wire

WYE5

 

4 wire and 5 wire distribution
WYE systems with a phase-to-neutral voltage of 277 VAC, or 208/60Hz 5-wire WYE systems with a phase-to neutral voltage of 120 VAC. The power company generator outputs AC power simultaneously to the 3 phase conductors that are offset 120 degrees from each other. There are four wires coming out of every power plant: the three phases plus a neutral or ground common to all.

If AC power was a balanced transmission line it would be less dangerous but more expensive as you would need 6 wires instead of 4 as each phase would be a two wire pair with no need for a neutral.

Have you ever wondered why birds can sit on power lines and not be electrocuted?
The biggest danger for people working around power is touching anything connected to the earth that is conductive and then touching an energized phase of electricity; they become a path to ground. The birds sitting on the power line have not completed a transmission path, but consider a large bald eagle flying off of a line and his wing span touches an energized phase and the neutral, he is no longer in this life.

This also applies to Electrician or the Technician working around energized circuits that they should turn off breaker to circuit whether AC or DC before working on it.

If the power system was balanced it would not have a reference to earth ground and a person accidentally touching an energized conductor would be less likely to be electrocuted. Now if they touched one conductor with one hand and the other of a balanced pair with the other hand, then they would be electrocuted.
It is rare that you will ever see balanced power but they are made see this link http://www.equitech.com/articles/enigma.html a good article, that discusses balanced power.

In Utility power lower voltage (2300-7200 volts) distribution there are three phases at the top of a pole and below is the Neutral line on a transmission line which is typically sized a little smaller than the phase conductors and is bonded to vertical conductor going to earth ground at every pole or in the case of underground at every access point. On large transmission lines there are usually two overhead ground wires that are primarily used as a lighting screen and they bonded to vertical wires that run to bottom of pole or tower. I imagine that they could act as a neutral but the return is in the earth ground.
When ILEC’s , Cable TV, and CLEC,s attach there plant to a pole that the power company is also on they all attach a bond to the power company vertical wire. The vertical wire attaches the neutral to the bottom of the pole where a ground plate/rod is in place. In cable/phone/fiber optic outside plant the purpose for this is purely protection against shock of static electricity and other induced currents like 60 cycle hum from the power lines overhead or adjacent trench Radio frequency Interference or RFI. In underground applications each would use there own ground rod.
It is apparent at least in the Cable TV business that the old outside plant people get the promoted into the Cable TV Headend and bring with them what they know about grounding. They apply outside plant grounding concepts which are wrong for a data center (which is what a Cable TV Headend is now). All of the other Telecom providers have long since understood the difference.

3-phase-power

-phase power spaced 120 degrees a part, one of the three phases is always nearing a peak. This provides suitable power output to High-power 3-phase motors like the type used in large HVAC systems or manufacturing applications.
This was typical of electricity use for business and industry along with lighting. This type of equipment has little effect on the 60 Hertz (cycles per second) sine waveform of the electricity provided from utility power. These are linear loads, which are defined as current (amperage) that rises and falls in proportion to the voltage wave.

But in the last 25 years, the increased use of microprocessor based equipment has created non-linear loads on business power distribution systems. Examples are computer workstations and servers, AC/DC converters, lighting electronic ballasts, Hospital diagnostic equipment and the uninterruptible power supplies used in Data Centers.

A non-linear loads cause harmonic waveforms that are multiples of the fundamental 60 Hertz sine wave are additions to the base waveform. For example, the second harmonic is an octave (using the musical term) above 60 Hertz which is 2 times 60 or 120 Hertz, the third harmonic is a 180 Hertz waveform, and so on (The old Cable guy will recognize this same problem that happens when Cable TV frequencies are stacked on each other). The combination of the sine wave with all the harmonics creates a new, non-sinusoidal wave of entirely different shape. The change to the wave is called harmonic distortion. The unit measuring this is called Total Harmonic Distortion or THD. THD is more of a problem for motors in HVAC and other applications causing motor windings to overheat. It also can cause abnormal currents on the neutral wire causing overheating and thus a safety hazard.
For Computer and Telecom equipment the problem is the sporadic transient voltage that can spike quickly causing servers to suddenly reset, some times SCSI and IDE drives to fail or suddenly have data errors. Most typically it damages the computer power supply causing failure. Transients are usually the result common mode coupling into the neutral. Typically manufactures will have circuitry to reject common mode transients and many also depend on the Uninterruptible Power Supply (UPS) to arrest these transients. But the inverter portion of the UPS can also contribute electrical anomalies to a building power system. The use of UPS by itself is not a cure all for dirty power as I would like recite in the following case study.

Part 1 Case Study: Noise in the Predatory Finance Company PBX
This case study points out a common mistake when adding new power panels to a older power system.
A “Cash til payday” company which I will call the “Predatory Finance Company” had a small Call center with a 4 cabinet Nortel Option 11 PBX that hung on a wall and it was a mess. It was connected to an APC matrix UPS along with a SQL server, Web server, NT domain controllers, a Sun Solaris box that ran a predictive dialer. The phone system had a couple upgrades and then some re-configuration to install a local Primary Rate ISDN line for direct inward dial numbers. In parallel to this activity several more servers we installed and brought online, several for a web farm and to more SQL servers.
Not long after that complaints about buzzing sounds and blasts of noise while talking on the phone were coming in daily

The Predatory Finance company had a maintenance contract with the company the sold them the PBX. They were called in and the first suspicion was the new PBX configuration that was done by the in house company Technician..
The company Technician had worked for a Interstate Carrier and suspected that it might be a grounding and power may the issue. But he stood by while the Vendor started troubleshooting things like T-1’s (which were clean no bpv or crc errors). The in house Tech had suggested that this not source of problem, and showed them Nortel’s own installation manual about grounding. It was then that after removing the power panels and diagramming the wiring of the power that a ground loop discovered in the 4 wire power system.
Here are the details of the discovery (you will need to take time and study this closely):
Non-dedicated ground power the typical 4-wire, 480/277 V system

In a 4 wire power system the ground is carried by use of metallic conduit and metal enclosures to provide a what is called a safety ground.
Looking at the following drawing:

Power_ground_loop

Number 1 appears twice where to two ground rods in different locations, one at the Power Entrance and in at the Main panel.
Number 2 is where the Main Power entrance from the Power Company is, in a commercial building it is typical for the power to be feed off of the transformer in the building. This is a 4-wire, 208/120 V power distribution system common in older large buildings.
Note that the ground conductor is connected to the metallic building structure and a 6 foot ground rod. This type of ground was designed to suppress 60 cycle leakage and fault currents, but it does not do the job in suppressing higher frequency leakage currents created by non-Linear loads.
Number 3 is the ATS cabinet where the utility power connects to the un-energized side of an Automatic Transfer Switch while the backup Generator is connected to open contacts. When power fails the ATS energizes the switch relay and connects the output to of the Generator to now provide building power.
Note that in the conduit leaving the ATS are 4 conductors, 3 are the 208 phase conductors and the 4th being the Neutral.
The most important to note is the Ground and neutral wires bonded together in ATS panel and in the service entrance panel. Note: Since most businesses use Generator backup in this day and age this is a good example to use.

Number 4 is the Data panel Power distribution is where a 4 wire system now becomes a 3 phase five wire power system.

Dedicated ground power distribution

The 5 wire system consists of 3 phase conductors, a neutral conductor and a separate ground conductor. In fact about every panel sold in most markets now has isolated Neutral and Ground busses.

In a remodel to accommodate the Predatory Finance Company’s new computer room a new panel was added called the Data Room panel. Data room panel was connected by conduit to the Main panel. The Electrician ran 5 wires (the 3 phases, neutral and ground) from the Main panel to the Sub-panel. Note that in the main panel the neutral and ground are bonded to together on a wire terminal strip that is bonded to the panel. He drove a 6 foot ground rod and also connected it to the Neutral and ground going to the data sub panel.
The main panel has breakers that power circuits with over 200 workstation Computers, electronic lighting both being non-linear loads.

When the Electrician planted the second ground rod it created a second path for these spurious harmonics and noise to travel to.

Electrician was among the many confused about how to do grounding properly. Instead of fixing things he actually complicated things by creating a ground loop. This is a clear effect of the un-balanced power system were the neutral collects all of these transients from the different phases in a common mode, thus the term common mode noise. But disposal of this noise down one ground is futile as the other ground is the re-entry point for some portion of the noise and harmonics from the earth.
Removing the 2nd ground rod may help but it is not the entire solution to the problem. Before discussing the solutions I would like to establish what the difference is between Neutral and Ground

A simple rule could clarify the differences between Neutral and Ground.

It can be stated that Neutral can be grounded, but Ground is not neutral

A Neutral represents a reference point within an electrical distribution system. Conductors connected to this reference point (Neutral) should, normally, be non current carrying conductors, sized to handle momentary faults (short circuits) occurring in electrical equipment. However, with the introduction of non linear loads, such as computers, electronic lighting the role for the neutral conductor has increased.

The Ground represents an electrical path that is normally designed to carry fault current when an insulation breakdown occurs within electrical equipment. (Breakdown example would be dropping a metal tool across lug terminal with positive voltage potential to the steel structure within a facility.) Typically the ground is the metallic conduit in commercial installations this would be a safety ground. Sometimes it is an additional conductor. But Non-linear loads necessitate the use of a second ground which is known as a dedicated ground whose primary function is not safety but the reduction of electrical noise and harmonics.

Some current will always flow through the ground path, mainly from capacitive coupling and Inductive coupling with the other phase power conductors and the ground path (conductive conduit, conductive structure members, etc) are the typical sources of ground path current. Since a ground wire can couple to the other conductors it can be used to drain the harmonics another noise generated by the computers, servers, lighting etc to ground, which is a characteristic that we want to exploit.

SPG_power

 

New circuit from ATS with continuous ground conductor back to power entrance were neutral and ground are bonded together.

 

Single Point Grounding, Isolated circuits and panels

The best way to reduce electrical equipment from interfering from each other is use a single ground reference in a building. Figure 25 shows a solution for correcting the ground loop by installing a new 5 wire system from the building power entrance/ATS directly to the sub panel (by passing the main panel) that will feed the computer room. You only want one point where the neutral and the ground bond together and that is at the power service entrance. The ATS in this case study kind of muddles that concept but since it is connected by only 2 feet of conduit the ATS and Main power service panel would be considered the power service entrance for either utility power or generator power.
The new design all off the noise/harmonics from the lighting and computer work stations are isolated from the computer room circuits as they are now in separate conduits terminating to power service entrance and both are protected from interfering with other by the neutral to ground bond at the service entrance.

 

isolated_power

 

Solution 2 is least expensive way of eliminating ground loop by installing an AC isolation transformer

Electrical Isolation transformer eliminating Common mode noise

Installing the separate 150 foot runs cost about 5,000 with labor and 2 inch conduit. And in this case study we are trying eliminate the noise in the PBX so we used a cheaper way of fixing the problem by using an electrical isolation transformer. This is applying the same methodology to the power that runs the PBX that the other connections to the PBX use. This would be the analog POTS and T-1 line cards in the PBX. The twisted pairs that are connected to the T-1 cards and the POTS (Plain Old Telephone lines) are balanced and isolated by line matching transformers. Thus they have no common mode relationship to the dirty power and ground. This is also true of 10 base T/100/1000 copper Ethernet which also is balanced and transformer coupled. It is important to point out here that RS-232 serial cables are unbalanced and susceptible to the same problems as the AC power and can create ground loops.

In Figure 26 shows the electrical isolation transformer implementation to eliminate common mode noise. The isolation transformer creates a new ground reference were the ground conductor and the neutral are bonded with the second ground rod in the Main panel. The UPS was connected to the output of the transformer that was custom manufactured by TSI power in Wisconsin.
This fixed are noise problem. Of course buying a more expensive UPS that has a built in isolation transformer and has the orange dedicated ground outlets (will cover that in more detail in case Study 2).

After solving the voice problem The Predatory Finance Company kept expanding the data center adding several more Dell servers soon they started having problems with random reboots of the two critical SQl servers, Power supply failures in some of the Web servers and several SCSI drive failures
Much discussion went on the in house voice tech knowing that the rest o the electrical system was still wrong persuaded the company to spend 500 on a study by a consultant who used a fluke 430 power quality analyzer. A twenty four study logged several transient spikes, severe harmonics and ground to neutral noise of 1.5 volts indicating that the ground loop still existed.

They finally spent the money on Solution 1 They also used the orange colored outlets that are seen all over in Hospitals, as they are isolated ground outlets. After doing so many events like loosing SCSI drives and server power supplies subsided. The application of Isolated ground outlets I will explain in the next case study where everything is done right up front.

new_data_center

 

 

 

Case Study 2 NEBS compliant Power and Grounding

In Figure 26 the Predatory Finance Company moves to a new building.

First I would like to note that working with electrical contractors on doing dedicated ground wires and creating signal ground references can be a struggle. Hiring the right consultant to enforce these standards may be the easiest when building a new data center.

Referencing Number 1 find the building system ground which again consists of two components a large sized conductor to a ground rod and another to the building steel this bond is within feet of the main power entrance panel. Again an Automatic Transfer Switch is used to provide the Utility power by default and when Utility power is out the ATS is energized starting the generator and switching the outputs of ATS to generator power. Also notice that the Utility power is coming from a 5 wire WYE transformer.
The ATS is connected immediately to the main distribution panel that has these large 200 thermal magnetic breakers are three different 480 VAC output feeds One them goes to a panel that runs the HVAC for the building, another for 277 VAC for the lighting. And then outputs to separate 480 to 208 step down transformers one for all of the outlets and another for the Data Center. This uses both of the prior solutions of dedicated runs with isolated grounds and with the two step-down transformers can also perform the isolation transformer function.
In effect now the Computer workstations are isolated from the Data Center and both isolated from the HVAC and the lighting.

Number 2 Is the Telecom panel where the isolated ground is continues. It is Important that the Isolated ground is not bonded or connected to the panel itself, the grounding of the panel is performed by conduits and if required by code and additional ground. This grounds specific function is protection from electrical shock.

Here is where the coupling of the dedicated ground wire to the phase conductors and the neutral can be exploited. The isolated ground function is used to dispose of the unwanted harmonics and common mode noise as these components couple with dedicated ground wire. One could characterize that noise and non-linear components of the 60 HZ ac are drained to ground.

Number 3 is the single point ground is established with use of a NEBS compliant ground bar. A 6 AWG stranded wire is from the Isolated ground bar in the Telecom panel about 20 feet to a wall were to the NEBS compliant isolated ground bar is installed. All external grounds are connected to it. This would include all of the cabinets of the Nortel Option 11 (Number 6) cabinets running on AC UPS, the Nortel Option 61 running on DC power and then the grounding of 48 volt DC battery strings of the DC power plant. Number 4 Points out that the grounding of the batteries is done on the positive side of the battery strings. This is known as positive ground and why Telecom battery is referenced as -48 volts.
Number 5 shows a hard wire connection to 208 input UPS with as dedicated circuit and isolated ground wire.

Referencing Number 1 find the building system ground which again consists of two components a large sized conductor to a ground rod and another to the building steel this bond is within feet of the main power entrance panel. Again an Automatic Transfer Switch is used to provide the Utility power by default and when Utility power is out the ATS is energized starting the generator and switching the outputs of ATS to generator power. Also notice that the Utility power is coming from a 5 wire WYE transformer.
The ATS is connected immediately to the main distribution panel that has these large 200 thermal magnetic breakers are three different 480 VAC output feeds One them goes to a panel that runs the HVAC for the building, another for 277 VAC for the lighting. And then outputs to separate 480 to 208 step down transformers one for all of the outlets and another for the Data Center. This uses both of the prior solutions of dedicated runs with isolated grounds and with the two step-down transformers can also perform the isolation transformer function.
In effect now the Computer workstations are isolated from the Data Center and both isolated from the HVAC and the lighting.

Number 2 Is the Telecom panel where the isolated ground is continues. It is Important that the Isolated ground is not bonded or connected to the panel itself, the grounding of the panel is performed by conduits and if required by code and additional ground. This grounds specific function is protection from electrical shock.

Here is where the coupling of the dedicated ground wire to the phase conductors and the neutral can be exploited. The isolated ground function is used to dispose of the unwanted harmonics and common mode noise as these components couple with dedicated ground wire. One could characterize that noise and non-linear components of the 60 HZ ac are drained to ground.

Number 3 is the single point ground is established with use of a NEBS compliant ground bar. A 6 AWG stranded wire is from the Isolated ground bar in the Telecom panel about 20 feet to a wall were to the NEBS compliant isolated ground bar is installed. All external grounds are connected to it. This would include all of the cabinets of the Nortel Option 11 (Number 6) cabinets running on AC UPS, the Nortel Option 61 running on DC power and then the grounding of 48 volt DC battery strings of the DC power plant. Number 4 Points out that the grounding of the batteries is done on the positive side of the battery strings. This is known as positive ground and why Telecom battery is referenced as -48 volts.
Number 5 shows a hard wire connection to 208 input UPS with as dedicated circuit and isolated ground wire.

 

 

 

orange_isolatted_groung_outlet The orange Isolated dedicated ground receptacle is a critical piece with the use of isolated ground conductor . This isolates the dedicated ground from the safety ground that is used to ground the conduit and the outlet box.

 

SPG_ground_bar

 

Also note the use of the ground bar with two-hole lugs with holes that are 5/8 on center. The connections on the equipment must be the same to meet NEBS level 3 requirements.
Here is a link to an example of how Cisco meets Network Equipment Business Systems (NEBS) Level 3 and European Telecommunication Standards Institute (ETSI) standards.
http://www.cisco.com/univercd/cc/td/doc/product/access/multicon/3810nebs.htm

Measuring Ground to Neutral noise with a volt meter

Measuring_for_ground_loop

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s