Disaster Document
Goals and Values
Sample Mission Statement
Customer Service Plan
Employee Evaluation Form
Sample Collocation Agreement
Project Manager
B-PON
TDM TO IP TRANSITION
Internet Links
Network Operations Draft
Information System Operations
Project Cost Management
PMBOC Knowledge Maping
Home
e-mail me

   

 COMMUNICATION SITE

DISASTER PREVENTION,  MITIGATION PLAN

Jack H. Farley

PART I  DISASTER PREVENTION PRINCIPLES

1.0 Introduction  Document

For millions of users, including public safety organizations, communications is an essential part of everyday business, and a vital tool in maintaining and restoring public facilities in case of disaster.  It is essential, therefore, that we introduce plans and controls to fulfill the dual purpose of preventing damage to communications company facilities, and ensuring service continuity through the natural and man-made disasters that occur.

This document describes “The Company” plan for evaluating risks to service and preventing damage when abnormal conditions such as fires, floods, windstorms, and other such threats occur. 

The plan has the following objectives:

·        To ensure the safety of “The Company’s employees,  contractors, and the public that the company serves.

·        To identify the primary hazards that are common to all operations, and that are unique to some because of their

          locale.

·        To mitigate hazards by taking preventive measures that are balanced with the degree of risk.

·        To marshal the company's resources when damage does

          occur so the adverse effects on service and costs are

          minimized.

·        To enhance “The Company’s competitive position by assuring our customers of the survivability of service during abnormal conditions that may disrupt other forms of communication.

 

A disaster is defined as an occurrence that inflicts widespread destruction or distress.  A disaster can be as small as a blown fuse that takes the communications facilities out of service, or as large as a fire, earthquake, or tornado that damages an entire business complex. 

Disaster Prevention Planning is the process of developing strategies for mitigating the effects of abnormal conditions.  Our objective is to ensure that disasters do not occur despite conditions that inflict damage on other parts of the locale.

“The Company’s Disaster Prevention Plan examines each of the potential points of failure in a communications system and develops the most cost effective way to ensure continuity of service, and rapid restoration in case of service interruption.  The key term is cost effective.  Full duplication of all equipment and facilities could protect against most failures, but this would double the cost of doing business.  The cost of each solution must be weighed against the risks of a service outage.

This document deals with the transport, or call delivery portion of communications and describes how “The Company” can provide cost effective solutions to protect communications service.  This plan covers each of the four main elements of a communications system:

·        The Switch

·        The Network

·        The Remotes

·        The Software

1.1 Definitions

This section defines terms that are used throughout this plan.

Abnormal Condition: 

Any unusual event, such as fire, flood, earthquake, or civil disturbance that threatens to inflict damage or disrupt service.

Crisis: 

A time of acute danger or hardship resulting from an event that is not a normal occurrence.

Disaster: 

A sudden occurrence that results in extensive damage or failure of a resource.  Note that a disaster does not necessarily result in a crisis.

Disaster Prevention: 

The process of developing plans and taking precautions to prevent damage from abnormal situations.

Disaster Recovery: 

The process of restoring service or damaged property after a disaster occurs.

Emergency: 

An abnormal event that requires prompt action.

Hazard:

 A source of danger or damage.

Mitigation:  The process of taking actions that lessen the effects of an abnormal situation.

Risk: 

The possibility of suffering harm or loss from an abnormal event.

 

2.0 PURPOSES OF THE PLAN

This plan has multiple purposes, as discussed below.

 

2.1 Ensure Service Continuity to Key Customers

“The Company” Communications has a vital role in securing public safety.  In their disaster plans, most public safety agencies rely on communications to provide service if the public switched telephone network (PSTN) fails.  Therefore, it is of utmost importance that we provide service that is as invulnerable to failure as possible.

2.2 Take Countermeasures to Reduce Hazards

The Disaster Prevention Plan must take permanent countermeasures to mitigate the effects of hazards.  For example, by providing self-healing network services, the effects of hazards such as dig-up of T-1 carrier facilities can be avoided.  By designing equipment buildings to protect against collapse during earthquakes and tornado-force winds, and by building with fire resistant materials, the effects of common hazards can be minimized.  In earthquake-prone areas, secured furniture and equipment will prevent injury to employees.  These and other mitigation measures will help to save lives and protect property, and will reduce the level of activity required after an abnormal event occurs.

2.3 Minimize Financial Exposure from Risks

Another purpose of the plan is to reduce the amount of financial exposure the company has as a result of abnormal events.  The company's exposure generally falls into three categories:

·        Loss of plant and equipment

·        Loss of revenues

·        Loss of competitive advantage

The magnitude of the first two categories can be predicted with some accuracy.  The third, however, is almost impossible to predict.  The degree to which “The Company” is capable of providing service continuity through an abnormal situation has a significant effect on our ability to compete.

2.4 Specify Organizational and Individual Responsibilities

Another purpose of the Disaster Prevention Plan is to specify in each location who bears the major responsibilities. 

These include:

2.4.1 Who is Authorized to Declare an Emergency. 

During a disaster extraordinary measures are taken to protect and restore service.  These often require unusual expenditures of funds and extra efforts from personnel.  At the same time, the failure to declare an emergency may delay restoration or increase the magnitude of the disaster by the failure to take proper protective measures.  The decision to activate preventive or restorative measures must be made by someone who has the authority. 

2.4.2 Order of Succession. 

The appropriate levels of authority may not always be available in case of emergency.  Each manager must designate who is delegated authority to act on emergency matters in his or her absence.

2.5 Disaster Prevention. 

The key to disaster prevention planning is in taking steps to ensure that the damage from abnormal events is avoided or minimized.  Each location must take the following five steps to prevent or minimize the consequences of a disaster:

Risk Assessment: 

Review hazards to the business and assess the degree to which they involve a service risk.

Disaster Prevention Planning: 

Make plans to reduce the degree of service risk from particular hazards to an acceptable level.  When natural or man-made disasters occur, the effect on service will be avoided or controlled.

Disaster Recovery: 

Plan for how to minimize or restore damage from a disaster if it occurs.

Emergency Communications: 

Furnish information about the situation to all levels of management, to customers, and to the public.

Resume Business as Usual:  Dismantle the resources that were assembled for the emergency, and resuming business as usual.

2.6 Establish a Process for Coordinating Disaster Recovery Activities

The disaster recovery process must be thought out and documented in advance of an actual disaster.  When a disaster threatens or occurs, there is no time to establish plans about what is to be done and who is to do it.  These must be determined in advance so preventive or recovery activities can begin with a minimum of delay.

2.7 Assemble Information in an Accessible Location

Information is vital to Disaster Prevention Planning.  Information must be retained in a form that is common to all locations and kept where it is readily accessible to those involved in Disaster Prevention Planning and recovery.

2.8 Review and Test Preparatory Measures on a Scheduled Basis

A Disaster Prevention plan can never be tested completely in advance of an abnormal condition, but it is essential that plans be reviewed and tested periodically.  For example, emergency engines must be regularly run, emergency supplies and spares must be examined periodically to see that they are in place, and plans must be reviewed regularly to ensure that personnel and equipment lists are up to date.  2.9 Mobilize “The Company’s Resources to Support the Community in Case of Disaster

When a crisis such as a storm, earthquake, or other serious condition disrupts a community, our emergency equipment can be an important resource in supporting the community's restoration activities.  “The Company” may be able to provide communications telephones and other equipment such as emergency generators to aid the community in its efforts.

3.0 STRUCTURE OF THE PLAN

This plan contains three major parts.  Part I provides information on how to identify service risks and take countermeasures to mitigate the risks.  Part II is site-specific.  It includes forms and brief instructions on how to evaluate hazards and take protective measures for each site.  Forms are included for inventorying crucial resources such as personnel, test equipment, vehicles, portable power equipment, and other such resources that can be deployed in case of emergency.  Part III is reserved for regional plans to counter specific hazards such as hurricanes in the Gulf Coast and earthquakes in California.

4.0 RISK IDENTIFICATION

The first step in disaster prevention planning is to identify which hazards present the greatest risk to the business.  When hazards and service risks are identified, plans must be developed to mitigate their potential effects. 

Note:  Countermeasures require designing protective measures into new buildings and may require upgrading existing facilities.

There are a variety of resources you can call on to aid in risk evaluation: insurers, police and fire departments, state and federal disaster planners, professional consultants, and publications.  The bibliography at the end of this instruction lists publications that are available to assist in evaluating risk and planning countermeasures.  In addition, your employees can provide valuable knowledge and experience.  Involving them at the early stages of the Disaster Prevention Planning process increases their awareness of the company's efforts and increases their commitment to the Plan. 

4.1 Identifying Hazards

Each part of the country is more susceptible to certain types of hazards than others, which means that Disaster Prevention Plans must be directed to the nature of the hazards as they exist.  For example, tornadoes and hurricanes are virtually unknown on the west coast, but the earthquake hazards are the highest in the country.  The earthquake hazard is minimal in the Midwest, but the risk of storm damage is higher. 

Local management must examine facilities, assess the magnitude of the risk based on the probability of the hazard occurring, and take steps to minimize the potential of damage or service loss based on cost effectiveness. 

Throughout the process of evaluating hazards, you must be alert to common problems that can result from any of the hazards discussed.  The most common is commercial power outage, which can occur at any time; even during conditions that are otherwise not hazardous.  For example, unusually warm or cold weather can result in a power brownout that may disrupt commercial service for a lengthy period.  Also, common carrier telephone service may be fail, which has the secondary effect of overloading our services.  This section of the plan discusses how to identify external hazards and estimate their effects on your plant.

4.1.1 Fire

Fire hazards must be examined from three standpoints:

·        The risk of combustion from outside “The Company” facilities and consequent damage to our plant.

·        The risk of combustion within our facilities and consequent damage to plant.

·        The effectiveness of fire suppression systems in controlling combustion after it occurs.

4.1.1.1 Externally-Caused Combustion

The risk of loss from external combustion comes from two sources.  When “The Company” facilities are in a company-owned building, the hazards are from combustion of other buildings or grass or forest fires in the vicinity.  For example, the 1991 fire in Berkeley, CA destroyed a large area.  Protecting facilities in such an area is virtually impossible.  When our facilities are in leased buildings the risk comes from fires in quarters leased by others.  To evaluate the risks from external combustion, ask questions such as these:

·        Is there a clear zone free of combustibles around “The Company” facilities?

·        What kind of activities are carried on in adjoining facilities?  Do they represent an extraordinary risk of fire?

·        Is the entire area susceptible to fire (because of unusual dryness, high winds, etc.)?

·        If a fire occurs, how likely is it that fire departments will be able to extinguish it before severe damage results?

4.1.1.2  Internally-Caused Combustion

It is much easier to evaluate and minimize the risks of an internally-caused fire because the environment is under our control.  The first principle of fire prevention is good housekeeping.  Evaluate the risks by asking questions such as these:

·        Are housekeeping practices acceptable?  Is the area free of dust, and combustible materials stored in equipment areas?

·        Is smoking prohibited in all equipment areas?  If so, are no-smoking regulations enforced?

·        Is the building or room housing our equipment made of  combustible materials such as wood, or of non-combustibles such as concrete?

·        Are electrical circuits properly fused?

·        Does all cabling meet the flammability requirements of UL Standards 910 and 1666?

4.1.1.3  Fire Suppressants

All equipment areas in “The Company” should be equipped with fire suppressant systems.  The principles of fire suppression are discussed in Section 5 of this plan.  To evaluate the risks of fire damage, ask questions such as these:

·        Are equipment areas protected with some type of fire suppressant system?

·        Are all areas equipped with smoke detectors?

·        Are suppressant systems and smoke detectors alarmed to a “The Company” control center or protective agency?

·        Are equipment rooms and work areas equipped with portable fire extinguishers?  If so, are inspections current?

·        Are escape routes clearly marked?

4.1.2  Water

Water is the enemy of electronic equipment.  Whenever equipment is subjected to flooding or drenching, several undesirable effects occur.  Electric circuits are shorted by water flowing between the conductors, and permanent damage may result.  Also, water carries impurities that cause corrosion of circuit cards and connectors.  The risk from water damage generally comes from several sources, including these:

·        Flood waters that rise into equipment areas

·        Stopped-up or overflowing sewers, gutters, or other devices that are intended to carry away excess water.

·        Ruptured pipes or activated overhead sprinkler systems.

·        Leaking roofs or walls that are intended to be waterproof.

Assessing the hazards from water damage requires little more than common sense and close observation.  If a building is well above the flood plain there is little risk of damage from flooding, but there still may be risks of damage from ruptured pipes or leakage. 

To assess the damage, ask questions such as these:

·        Is the site near any streams that could rise high enough to reach the building?

·        Is there any evidence of leakage in the building such as discoloration on walls or floors?

·        Are there overhead sprinklers in the equipment room?  If so, are they required by code?

·        If gutters or sewers become stopped up, is there any way water can reach equipment areas?

4.1.3  Storms

Storms can be divided into three major categories; wind, lightning, and ice storms and blizzards.  Sometimes all three can combine to multiply the hazards.  Unlike other types of natural disasters, which tend to be localized, storms can wreak destruction over a wide area, and disrupt transportation and other forms of communication.  It is particularly important, therefore, that our facilities be constructed to withstand the effects of all types of storms. 

Storms present a service hazard from three primary sources:

·        Damage to antennas and towers from wind, icing, or lightning.

·        Damage to buildings that may result in secondary damage to equipment from falling debris or water that enters through normally protected areas.

·        Damage to equipment that results from lightning strikes

Hazards from the three categories of storms vary greatly with different areas of the United States.  The degree of risk can be assessed only with a knowledge of local conditions.  The areas that are most susceptible to hurricane damage, for example, are least vulnerable to snow and ice storms.  In assessing the degree of risk from storms, consider questions such as these:

·        What type of storm is most (and least) likely to occur in your part of the country? 

·        When a storm does occur, what is the most likely effect on the public at large?  Will this generate a high volume of calls?

·        Are buildings housing critical equipment constructed to withstand the most severe types of storms including snow loading?

·        Are antennas securely fastened to towers or supporting structures to prevent damage or disorientation during windstorms?

·        Are all towers constructed to EIA 222 specifications?

·        Are towers maintained (bolts and guys tightened, etc.) to enable them to withstand storm conditions?

·        Are antennas properly grounded to prevent damage from lightning strikes?

·        Are all circuits entering the building equipped with the necessary electrical protection (carbons, gas tube protectors, etc.)?

4.1.4  Earthquakes

The risk of earthquake damage is highly localized in the United States.  Figure 4.1 is a copy of the BellCore Earthquake Zoning Map [Note: permission to reproduce has been requested from Bellcore] that shows the degree of seismic shock hazard for the United States.  Some parts of the U.S., notably California, have building code requirements that must be met for buildings housing facilities such as hospitals, public telecommunications companies, and public safety agencies.  Building code agencies should be consulted on local requirements.

Identify all present sites that are in buildings that do not meet the code for new construction, and determine whether inadequate structures leave “The Company” open to the loss of critical sites.  The Federal Emergency Management Agency Disaster Mitigation Guide for Business and Industry lists the following types of buildings as seismically suspect:

·        Unreinforced masonry bearing wall buildings.

·        Nonductile reinforced concrete frame buildings.

·        Precast concrete tilt-up buildings.

·        "Soft story" buildings (those in which one story--usually the ground floor--has less stiffness than the others).

The hazards from earthquakes fall into several categories, any of which can be disastrous to service:

·        Equipment failure because of vibration and shock.

·        Direct damage to equipment from twisting or collapse of bays.

·        Secondary damage from falling debris.

·        Damage from fire or water from ruptured pipes.

·        Explosion hazard from ruptured gas mains.

·        Failure of fuel supply to emergency generators.

·        Liquefaction, which causes the earth to temporarily lose its strength and temporarily become viscous.

·        Landslides that are triggered by earth movement.

To assess the risk of earthquake damage, ask questions such as these:

·        Is the service in an area that is susceptible to earthquakes?  Refer to Figure 4.1.

·        Are buildings and towers constructed to be earthquake resistant?

·        Has earthquake bracing been installed on equipment where necessary?

·        Is the site vulnerable to secondary effects from an earthquake (e.g. protracted power outage, damage from other hazards, etc.)?

4.1.5  War, Sabotage, or Civil Disturbance

Fortunately, the United States appears to be less at risk of war today than any time in recent history, so it is not necessary to take steps to evaluate war hazards at this time.  The threat of sabotage or civil disturbance is always real, however, and the methods of assessment and risk mitigation are essentially the same.  The risks are somewhat greater in some parts of the country than others, but such events can occur anywhere, and at any time.  Therefore, management must evaluate the probability of such disturbances happening, and take necessary steps to protect service.

To evaluate the risk of such events, ask questions such as these:

·        Have appropriate security measures been taken to limit access to company facilities, both through physical entry and through dial-up circuits?

·        Are records closely guarded so the public cannot readily learn the location of “The Company” facilities?

·        Are unattended buildings protected by fences and warning signs?

·        Are facilities properly alarmed, and are management or security agencies alerted immediately when there is any attempt to penetrate our facilities?

4.1.5.1  Dial-up & Port Security

A major threat to “The Company” facilities is from electronic access through dial-up or IP access.  Unauthorized persons breaking into equipment through one of its access ports can either steal service or cause damage up to and including destroying the system software.  In most cases knowledge of system operations is required before an intruder can successfully do either of these, but there are many individuals with the knowledge of how communications systems work.  Both current and ex-employees and vendor personnel are risks that must be considered.

Management must determine the degree of risk each hazard poses, and devise adequate countermeasures to mitigate each one.  The hazards addressed in this section are of a general nature and pertain to any facility regardless of manufacturer.  Hazards specific to a certain type of switch or to a special configuration should be addressed at a local level.

While remote access to the switch is our first concern, we have broadened the definition to include any point of electronic access to the switch.  In their initial state, all points of access are equal, so it makes sense to treat them all as potential trouble areas.  

In this section, such points of access will be called "data ports."  Terminology specific to other switches may vary.

Possible points of failure are defined as any point at which a data port to the switch may be accessed.  These include, but are not necessarily limited to, switch terminals, dial-in ports, internal network terminals, hard wired local terminals, and remote access points.

Unauthorized access is the most serious problem.  The data ports have little potential for causing serious hardware faults, but they are the path by which someone could cause serious software faults, either inadvertently or maliciously.  A "hacker" who gets into the system could create a problem.  A disgruntled employee or former employee could do so easily, possibly interrupting service for an extended period. 

To evaluate the risk of loss from dial-up access, ask questions such as these:

·        Are passwords restricted to those with a need to know, secure from unauthorized access, and regularly changed?

·        How many levels of password must a person know to gain access?

·        Are passwords composed of random characters and enough digits that it is difficult for intruders to guess at the password?

·        Are passwords changed whenever the incumbent in a job changes or the person leaves the company?

·        Are lists of dial-up port telephone numbers closely guarded?

·        Does equipment provide evidence of attempts to break in such as listing incidents of multiple password failures?

·        Does equipment freeze the port after a specified number of ineffective access attempts?

 

There are two categories of intruders that are often described as "hackers."  One is a hobbyist whose primary interest is the challenge of breaking into systems.  Once in the system they explore to determine how the system works and what it is capable of doing.  Hobbyists often break into voice mail systems for the purpose of establishing a mail box that they and acquaintances can use free of charge.  Although their intent may not be malicious, hobbyists can create considerable damage, either inadvertently or through idle curiosity.  Moreover, they can keep the port tied up and deny access for legitimate maintenance activities.

The second category of intruder is the person with criminal intent.  His or her motivation may be stealing service or creating damage.  Whatever the motivation, hackers must be kept out of the system. 

The following questions should be asked in determining the degree of risk from intruders:

·        Is it possible for an intruder who gains access to the system to place long distance calls?

·        Is the equipment programmed to detect and report multiple ineffective access attempts?

4.2  Assessing Service Outage Hazards to Plant

This portion of the plan provides information on each of the major classes of plant and facilities and discusses the penalty of loss or impairment of service.  Methods of mitigating the hazards are discussed in Section 5.2.

4.2.1  Loss or Impairment of Communications Equipment

The CENTRAL OFFICE is the most vulnerable part of the communications network since all calls are switched and controlled through it. 

To evaluate the degree of risk to the CENTRAL OFFICE, management must evaluate the following:

·        The building in which the CENTRAL OFFICE is housed.

·        The circuit facilities that serve the CENTRAL OFFICE and its linkage to remotes, interexchange carriers (IECs) and local exchange carriers (LECs).

·        The common equipment such as power and fire suppression equipment that supports the CENTRAL OFFICE

·        The records and backup software that are used for restoring service if failures occur.

·        The ability of the CENTRAL OFFICE to withstand peak traffic loads.

This section discusses the principal considerations in assessing the vulnerability of the CENTRAL OFFICE to failure. 

Note:  all of the hazards enumerated in Section 4.1 must be evaluated to assess the environment within which the CENTRAL OFFICE operates.

4.2.1.1  Building

The building that houses the CENTRAL OFFICE is the first element that should be evaluated.  Bear in mind that company-owned buildings are the most desirable for protecting service because they can be designed and maintained to company specifications and the environment controlled to a high degree.  In leased buildings, particularly multi-tenant buildings, it is more difficult to control hazards to service. 

To evaluate the building, ask questions such as these:

·        Is the CENTRAL OFFICE in a building that is reasonably secure against damage from fire, earthquake, flooding, and other environmental hazards?

·        Is the CENTRAL OFFICE in a location that can be kept secure from access by unauthorized personnel?

·        Is the equipment room free of sprinklers and overhead water pipes that could operate or rupture and damage equipment?

·        If building facilities such as the elevator, security system, or power fail, will it prevent maintenance personnel from entering the CENTRAL OFFICE?

·        Are there other occupants of the building that present a hazard because they might produce a hazardous substance, create a labor disturbance, or generate other type of disorder?

·        Are power, water, heating, and air conditioning furnished by the building or by separate facilities dedicated to the equipment room?  If the facilities are common, are they vulnerable to a disaster?

·        Is the fuel supply to the emergency generator secure from disruption during the crisis?  For example, are fuel tanks and supply lines for a diesel generator or gas supply lines to a gas turbine likely to rupture during an earthquake?

4.2.1.2  Circuit Facilities

Without circuits to the remotes, LECs, the CENTRAL OFFICE is either inoperative or impaired.  In evaluating the effects of loss of circuits, it is important to know the degree to which backup facilities are available.  Also, “The Company” networks such as the internal data network and the SS7 network often ride over the same sets of facilities to the IECs.  Loss of these facilities also can disrupt service. 

It is usually less costly to use bulk facilities such as T-3 to link the CENTRAL OFFICE to IECs and LECs, but failure of a single fiber or the loss of a high-order multiplexer can disrupt a substantial portion of the CENTRAL OFFICE. 

Therefore, in assessing the risk of circuit loss, it is important to ask questions such as these:

·        To what degree has circuit diversity been provided to critical cell sites and to other carriers?

·        Have self-healing circuits been installed to back up critical facilities?

·        Are private microwave facilities equipped for hot standby redundancy?

·        Are “The Company” networks such as MAC-1 and SS7 backed up with redundant facilities that will prevent their loss if the primary route fails?

·        Are dial-up ports secured properly and policed regularly according to the principles in Section 4.1.5.1?

4.2.1.3  Common Equipment

Continued operation of the CENTRAL OFFICE depends on the availability of backup circuitry and equipment.  For example, power, heating, and air conditioning equipment are all vital to the health of the system.  Backup power is always required for the CENTRAL OFFICE.  Backup heating and air conditioning equipment may or may not be required depending on environmental conditions and the availability of auxiliary equipment such as fans and portable heaters.  Where the operation of the equipment through external emergencies is essential, it is just as important that the supporting equipment be protected from damage as that the CENTRAL OFFICE equipment itself be protected.  For example, if the switch requires cooling, it becomes critical if the air conditioning fails or is damaged in an earthquake or wind storm. 

To evaluate the common equipment, ask questions such as these:

·        Are redundant air conditioning or heating equipment required?  If so, have they been provided?

·        Is the fuel supply to auxiliary power equipment secure against failure during disaster conditions?

·        Is the battery supply properly maintained and secure against damage during disaster conditions?

·        Is redundant switching equipment kept operative?

4.2.1.4  Records and Software

If a disaster strikes and equipment is damaged, records are essential to restoring service.  Any software-controlled apparatus such as the CENTRAL OFFICE is vulnerable to loss of information stored in the data base.  To guard against data loss, software must be backed up regularly and backup copies stored in an off-site location. 

To assess the status of records and software, ask questions such as these:

·        Are vital records such as equipment wiring, circuit identification, and network diagrams protected from unauthorized access and kept available to facilitate restoration?

·        Are software backups made regularly and stored off-site?

·        Has a disaster prevention evaluation been made in the past year and documented in the Disaster Prevention Plan?  Have deficiencies been documented and plans made for correcting them?

·        Is the location of backup documentation and software posted so it can be located quickly during an emergency?

4.2.1.5  Central Office Overloads

During abnormal conditions, the communications network may be overloaded because of excessive demand.  This will be particularly true if the PSTN experiences failure or overloads.  The CENTRAL OFFICE can be protected against overload by the use of Line Load Control, which is a method of denying dial tone to non-essential users.  The network also can be protected by increasing numbers of circuits during overloads, or by reducing the number of circuits when excessive demand from a particular location is overloading the CENTRAL OFFICE. 

This factor can be evaluated by asking questions such as:

·        Is the CENTRAL OFFICE equipped for line load control?  If so, is there a written policy on who can decide to activate it and under what conditions?

·        Does Customer Service have a policy of determining which customers are defined as essential service?

·        Is the CENTRAL OFFICE data base marked with essential service customers?

·        Are all “The Company” communications telephones marked as essential class of service?

4.2.2  Loss or Impairment of a Remote Site

Each operating location must identify cases where the loss of a single remote site would result in a serious impact such as loss of service to an area. 

These instances should be identified, and appropriate protective measures taken.  Particular attention must be paid to critical remote sites, that provide access to the CENTRAL OFFICE for several other sites, regardless of the traffic that site processes.  If this site suffers a failure of a microwave link or common carrier facility to the CENTRAL OFFICE, the impact is immediate and costly.  Ideally, all hub sites should have at least two alternate routes to the CENTRAL OFFICE, so the sites that are served by each hub would continue to be fully functional, although somewhat slower in their processing.

The considerations discussed under the CENTRAL OFFICE should be evaluated for critical cell sites from a service protection standpoint, and for all cell sites to evaluate the risk of property loss.  In addition, this section discusses considerations for evaluating elements that are unique to cell sites.

4.2.2.1  Radio System

Radio system protection is offered by normal maintenance procedures together with retaining a suitable number of spares.

4.2.2.2  Remote Site Controller

The site controller is a duplicated device, the loss of which can render the site inoperative.  Because the controller is redundant, the probability of loss is low.

4.2.2.3  Central Office to Remote Site Facilities

The elements at risk between the site and CENTRAL OFFICE are CENTRAL OFFICE-to- site circuits, network backbone to CENTRAL OFFICE for switched services, and network connectivity to other switched services.  These circuits are carried over private microwave or over common carrier facilities, primarily DS-1 circuits.  The risks involved in common carrier facilities are discussed in Section 4.2.1.2.

Since microwave systems primarily connect the CENTRAL OFFICEs to the remote sites, and in some cases to other CENTRAL OFFICEs or to the interexchange and local access network, microwave becomes an important issue when planning for a disaster.  Microwave systems generally are hardier than traditional wired facilities such as metallic cable and fiber optics; however, they depend on atmospheric conditions and the continuity of power and antennas.

Microwave systems also are subject to failure because of fading.  If the path is properly engineered, however, the probability of path loss is less than 5 percent.  To ensure microwave path reliability, fade margin and space diversity studies should be done on every path.  Other factors such as rain attenuation should be evaluated and monitored over time.  Other local effects due to change of land use, structures being built such as buildings and  towers, and interference factors from other radio transmission systems such as earth stations at TV studios must also be evaluated. 

Microwave systems can be disabled by human error, but since microwave often requires little intervention by technicians, human-based risk is fairly low.  Because most microwave systems in the “The Company” networks are one-for-one protected with hot standby equipment, the risk of the microwave system loss is low. 

Evaluate the risk to microwave systems by reviewing issues such as these:

·        Is the radio equipped for hot standby redundancy?

·        If the area is vulnerable to path fading, is the microwave equipped for space diversity?

·        Is the waveguide pressurized to prevent the entry of water?

·        Have fade margin tests been made and recorded?

·        Is the antenna secured rigidly to a tower or supporting structure so it cannot become disoriented during high winds?

·        Is the antenna equipped with deicers or a radome to prevent path loss during icing conditions?

·        Is the radio connected to the battery and emergency power system?

5.0  Mitigating the Effects of Disaster Conditions

The key to “The Company’s Disaster Prevention Plan is the word Prevention.  Disaster conditions may prevail in the environment, but the disaster is conducted to the company only if we have failed to prepare adequately.  This section of the plan provides general guidelines on what measures should be taken commensurate with the risk involved.  This section follows the same general outline as the risk evaluation in Section 4.

5.1  Mitigating External Hazards

The most difficult hazards to control are those that are imposed from the outside world.  However, we generally have at least six ways of minimizing the probability that external hazards will cause serious damage or service outage:

·        Select site locations carefully with a view toward their inherent hazards.  Avoid leased facilities for critical sites wherever it is economically feasible to do so.

·        Design building space for maximum security against hazards.  Use fireproof materials wherever feasible, and design around inherent hazards such as overhead pipes and sprinklers.  Pay close attention to security.

·        Install equipment in accordance with recognized standards that protect it from damage from external hazards.

·        Maintain facilities with a view toward hazard prevention.  Store critical records off site where they would not be damaged along with equipment damage.  Exert good management controls over housekeeping and employee awareness of hazards.

·        Provide redundant equipment and facilities wherever it is economically feasible to do so.

·        Maintain disaster prevention and disaster recovery plans in a current condition so you are prepared to withstand hazards, and to minimize or recover from damage when it occurs.

Bellcore's Network Equipment-Building System (NEBS) Generic Equipment Requirements, which is listed in the bibliography, is an excellent source of information for selecting and configuring equipment buildings.  The NEBS recommendations should be followed wherever possible.

5.1.1 Fire

Fire prevention activities involve removing hazards, removing combustible materials, and removing critical elements such as oxygen that are needed to sustain combustion.  The elements that go into providing communications service are not, in themselves, hazardous.  The principal hazards are man made, and prevention revolves about removing the causes of hazards.

As a general principle, smoking is not permitted in any equipment areas or adjacent maintenance areas.  This principle is necessary not only for reduction of fire hazards, but also to reduce contaminants that are otherwise detrimental to equipment.  Smoking should also be discouraged in office areas adjacent to equipment rooms because of the danger of fire spreading into equipment areas.

Equipment and office areas must be constructed to ensure the safety of employees.  This requires that alarms be connected to alert employees to the presence of smoke or fire.  Fire escapes must be available and the access to them kept unblocked.  Extinguishers must be placed near exits and at points away from exits to give employees a chance to extinguish small fires.  All protective apparatus must be tested regularly to ensure that it is operative when needed.

The Fire Protection Handbook, which is listed in the bibliography, is an excellent reference for all aspects of mitigating fire hazards.

5.1.1.1 Externally-Caused Combustion

The principal method of preventing external fires from damaging “The Company” facilities is to remove combustible material around a suitably wide radius. 

The following are some methods of preventing external fires:

·        Locate unattended facilities such as cell sites in areas free from combustible materials or environments.

·        Remove combustible materials from outside areas so fires cannot spread from other sources to “The Company’s buildings.

·        In leased facilities, make regular inspections of building facilities to see that the owner maintains a fire- resistant environment.

·        Ensure that flammable or hazardous liquids are not stored in the building.

5.1.1.2  Internally-Caused Combustion

“The Company” personnel have maximum control over preventing internal fires.  There is relatively little fire risk in a properly installed and maintained equipment room; however, disastrous fires have occurred in telecommunications buildings.  Virtually all internal fires can be prevented.  If they do occur, proper design and management attention can limit the effects. 

The following are principles of preventing internal fires:

·        Remove all combustible materials from equipment areas.

·        Encourage fire department inspections of facilities and follow their recommendations.

·        Ensure that electrical circuits are properly fused and not overloaded.

·        Ensure that detection and suppression equipment is operative, alarmed, and regularly tested.

·        Act immediately on all alarms.  Do not allow alarms to remain disabled.

·        Use cabling that meets the flammability requirements of UL Standards 910 and 1666

5.1.1.3  Fire Suppressants

When fires do occur, suppression systems are designed to minimize the damage.  Suppression systems operate on the principle of removing the supply of oxygen that is essential to maintaining combustion.  The most common form of suppression is water, but as discussed in Section 4.1.2, water will damage electronic equipment, often beyond the point of economical repair.  Therefore, automatic sprinkler systems should be avoided in equipment rooms unless required by code.  If sprinkler systems are required, it is important that they have high temperature heads, and that they not be triggered by fires outside the equipment room.  In a multi-tenant building, it is possible that a minor fire in another tenant's quarters could trigger sprinklers in “The Company’s quarters and damage equipment that was never threatened by fire.

The most effective suppression systems for our environment are Halon.  However, because Halon contains hydrocarbons, which are environmentally harmful, the EPA is imposing a tax on Halon systems that may make them economically infeasible.

Carbon dioxide systems are generally effective, but at six to seven percent concentrations, the effects on humans will be detrimental.  CO2 systems should be confined to small portable extinguishers where the concentration is low.

Dry chemical systems are generally used for flammable liquid spills, and are not recommended for use in electronic equipment.  Foam extinguishers are likewise intended for liquid spills.  Since foam is conductive, it is not recommended for use in electrical fires.

Portable fire extinguishers, normally of the CO2 type, should be installed in all equipment rooms.  Extinguishers should be installed inside the main entrance door within easy reach from the outside for extinguishing fires when no one is in the room.  They also should be placed at the furthest point from the door so personnel inside the room can use them to work their way toward the exit.

5.1.2  Water

The main way of protecting against water hazards is to remove the source.  This is, of course, not possible for buildings located within a flood plain.  If floods occur, there is virtually no way to prevent damage.  The only feasible method is sandbagging, and this is often ineffective.  The first principle to follow, therefore, is never to locate sites of any kind where they are susceptible to flooding. 

The most common source of water damage is from broken pipes, leaking roofs, stopped up drains and sewers, and operated sprinkler systems.

The following are some principles for protecting against water damage from these sources:

·        Do not use sprinkler systems unless required by code.  If sprinklers are required, construct a metal shield over the equipment to keep it dry.

·        Inspect all water pipes for evidence of corrosion.  Keep such pipes out of equipment rooms wherever possible.

·        Keep sewers and gutters clean and operative.

·        If any part of the building sustains damage in a storm that could let water in, place temporary repairs immediately.  Cover equipment with tarps pending permanent repairs.

If fire occurs, there is also risk of damage from fire suppression efforts of the fire department.  Within the bounds of employee safety, make every effort to avoid letting the fire department pour water on the equipment.  If the fire has started elsewhere in the building, use tarps, if possible, to cover equipment so it does not sustain water damage.

5.1.3  Storms

The key to preventing storm damage is to construct the facilities initially to withstand the highest expected force, or the worst hazards. 

The following are examples of the protective measures that should be taken:

·        Equip antennas with deicers or radomes to prevent icing

·        Construct all towers to EIA 222 specifications.

·        Maintain towers to the manufacturer's specifications.  Keep guys tight, tighten bolts, eliminate rust, and keep them painted.

·        Keep roofs and siding in good repair so they will not be damaged in windstorms.

·        Select buildings that are capable of withstanding loads imposed by wind, icing, and snow.

·        Obtain underground facilities from the LECs wherever a choice is available.  If the location is served by aerial cable or wire, determine if it is feasible to bury it.

5.1.3.1 Electrical Protection

Lightning and crosses with power lines impose a particular hazard to electronic facilities.  Lightning seeks the path of least resistance to ground, so damage is prevented by providing low resistance paths to ground that lightning can follow without passing through delicate equipment.  Antenna structures are particularly vulnerable to lightning because of their height, and should be grounded adequately so any lightning strikes will be drawn to ground.  Many communications facilities share joint pole lines with power, and are vulnerable to damage from downed power lines.  Protective measures must be taken on all circuits entering “The Company” facilities where they are exposed to lightning.  Protection is discussed in Article 800-30 of the National Electric Code.

Communication facilities are considered as exposed when they are subjected to one or more of the following:

·        Disturbances from lightning

·        Contact with power circuits operating at more than 300 volts RMS to ground

·        Power faults that result in ground potential exceeding 300 volts RMS

·        60 Hz induction exceeding 300 volts RMS

For the purposes of this plan, all circuits entering “The Company” facilities should be considered as exposed unless the facility enters on fiber optic cable or copper facilities from the LEC or IEC are wholly contained within the same building.

Where cable is exposed, it must be protected with carbon block protectors that confine the voltage to 300 VRMS or less.  Note that carbon block protectors do not protect equipment from "sneak currents" that sometimes result from voltages that are too low to operate protectors.  To prevent damage to equipment, current-operated devices known as "heat coils," are usually provided.  In addition, some equipment that is particularly sensitive to voltage spikes requires fast-acting gas tube protectors to prevent damage.

The LECs normally provide carbon block protectors with all exposed circuits they provide.  All circuits, including private voice and data and switched telephone circuits should be equipped with carbon block protectors.  If they are not, discuss it with your LEC representative.  The only exception are circuits provided over fiber optics and T-1, which is protected by the LEC's interface block.  If heat coils and gas tube protectors are needed, they must be provided by “The Company”.

5.1.4  Earthquakes

After the nature of the hazards have been identified, it is usually evident what measures can and should be taken to prevent earthquake damage.  To protect personnel, it is important that objects on shelves be secured so they cannot be dislodged and injure someone or damage equipment.  Furniture and equipment must be secured to the floor, and equipment must be braced against the walls to keep bays from tipping during seismic action. 

The most effective guard against earthquake damage is selecting earthquake-resistant buildings to house critical sites.  In selecting new sites or evaluating present sites in leased buildings, review the building against seismic building codes.  If, after inspection, an CENTRAL OFFICE is found to be in a building with a high probability of earthquake loss, plans should be made to move it.  If there are questions about the structural soundness of a building, consult a structural engineer who has a proven record of providing seismic evaluations of buildings.  Note that there is no category or license for an "earthquake engineer."

5.1.5  War, Sabotage, or Civil Disturbance

The first line of defense against sabotage or civil disturbance is a viable security program.  If sites are secured against sabotage and civil disturbances, and meet the requirements for resistance against fire, water, storm, and earthquake damage, they will be as secure against war as it is feasible to make them, given the present world political situation.  Therefore, all preventive measures in this category that should be taken are to guard against sabotage and civil disturbances.

5.1.5.1 Controlling Access to “The Company” Facilities

In all “The Company” sites, access must be controlled.  Not only are we guarding against the loss of service and property, we also are guarding information about site locations, customer lists, facility routes, and other such information that could be valuable to competitors, saboteurs, or the merely curious.  To protect service, property, and information, access to company facilities shall be as secure as it is economically feasible to make it. 

This includes the following guidelines:

·        All attended equipment rooms shall be locked against intrusion, using code locks, the code to which is changed at least monthly or the company's Mosler access card system.

·        Access to unattended sites in leased buildings with shared space, shall be limited to the building owner and bonded security people.

·        All unattended sites in “The Company”-owned buildings shall be enclosed with a chain link fence at least six feet high with three strands of barbed wire on the top.  All gates shall be kept locked while the site is unattended.

·        Antennas shall be mounted on towers that are secured with the same degree of security as the equipment buildings.

·        Unescorted access to office areas shall not be permitted.  All visitors shall be screened by a receptionist, and if access is justified, escorted by a company employee or properly authorized contractor.

·        Proprietary information shall be restricted to employees and contractors with a need to know.

 

Information about equipment quantities, facility routes, customers, and the like is proprietary, and shall be guarded from unauthorized access.

5.1.5.2  Securing Remote Access Data Ports Lines

The first level of defense of any data system is the password.  Many systems provide multi-level passwords, meaning that a person needs to know more than one password to access the system.  Passwords are not capable of providing total assurance of security because given enough time, someone who is determined to access a system can usually defeat the password.  With two levels of passwords that are properly accounted for and managed, the risk is minimized.  The risk is reduced even further with external security devices that automatically change the password at intervals.  The caller carries a wallet-sized card that is synchronized with the host.  Only a person with the remote card knows at any instant what the current password is; a few minutes later it will be changed.  As long as the remote card is secured, this system is highly secure. 

The following principles shall be observed in securing passwords:

·        Passwords shall not be posted where they are visible.

·        Passwords shall be restricted to those with a need to know, and changed at least quarterly, or when personnel with knowledge of the password change to another job or leave the company.

·        Telephone numbers of dial-up ports shall be closely guarded.

·        Every access port shall be secured with at least two levels of password and/or dial-back modem access.

·        Prohibit the use of script languages in telecommunications programs that permit automatic logon to a system.

A second defense against unauthorized access is the security modem.  Security modems are of two general types.  One type requires a password before a caller is connected to the application.  Another type, dial-back modems, can be programmed with a list of authorized dial-back numbers.  A caller logs in and enters the dial-back telephone number.  The modem checks it against a list, and if authorized, dials the number back.  With dial-back security, the risk of intrusion is minimized.  If a security modem is programmed incorrectly, it is possible to deactivate the security features. 

The following principles shall be observed in managing security modems:

·        Only properly trained personnel shall change the passwords in the modems.

·        A written procedure shall be used for changing the passwords.

·        The validity of the dial-back number list contained within the modem shall be checked at least quarterly or whenever anyone who has the potential of misusing the facility leaves the employ of the company.

·        Data ports should not be available for unlimited attempts to logon, but should lock out after several ineffective attempts to access the port.

Hard wired terminals, (those that are directly wired into an CENTRAL OFFICE) provide a high degree of security risk because to use the terminal, a person must only gain physical access to the keyboard.  Unguarded terminals in open areas are dangerous.  Terminals must be located in physically secure areas, and access restricted to those with a need to be there.  Even with physical restrictions, such terminals should still require at least one level of password to access the system, and passwords must be guarded with the same provisions as dial-up passwords.

5.2  Mitigating Service Hazards to Plant

This section discusses specific steps that must be taken to minimize the risk of loss in the main elements of the communications network.  In some cases, a significant expenditure that is not currently budgeted will be needed to mitigate a hazard.

Management should identify all cases where the risk of service loss is unacceptable, and shall program corrective action using the normal procedures for justifying expenditures. 

In other cases mitigation requires administrative action or training, and deficiencies should be corrected immediately.

5.2.1  Loss or Impairment of CENTRAL OFFICE

Every reasonable contingency must be reviewed to prevent loss or serious impairment of the CENTRAL OFFICE.  All switching equipment is fully redundant, so loss of the switching system because of equipment failure should be rare.  Switching equipment is designed to have an average of no more than one hour's outage in 20 years because of equipment failure.  As discussed earlier, however, the CENTRAL OFFICE can be lost because of failure of supporting elements such as building structure or power, or it can be damaged by storms or earthquakes if the building is not capable of withstanding the stress.

5.2.1.1  Equipment Installation

The Bellcore book Network Equipment-Building System (NEBS) Generic Equipment Requirements establishes Bellcore's minimum requirements for all new telecommunications equipment used in central offices and other telephone buildings of their clients.  The standards include cable distribution systems, distributing and interconnecting frames, power equipment, operations support systems, and cable entrance facilities; specifications for many of which are included in other Bellcore publications. 

NEBS standards should be used for all new equipment installations if possible.

It is not uncommon for equipment engineers to design a communications room with all of the equipment of one type on the same side of the room.  Sometimes this is done for simplicity, and sometimes for aesthetics.  When more than one set of equipment, such as T-1 circuits and sub-rate multiplexes is installed, it is more effective from a preservation of service standpoint to separate the equipment physically.  With this arrangement, a small fire in one set of equipment that is quickly extinguished will not harm the other set.

The integrity of equipment rooms is vital to ensure the survivability of the CENTRAL OFFICE. 

Wherever possible, the following guidelines shall be observed in selecting and configuring equipment rooms:

·        NEBS standards shall be followed wherever possible.

·        Equipment shall be securely bolted to the floor, and equipped with earthquake bracing appropriate for the seismic zone.

·        No overhead plumbing shall be permitted in the equipment unless required by building codes.

·        If building codes require overhead sprinklers, they shall be equipped with high temperature heads and shall not be triggered by fires outside the equipment rooms.

·        Fire suppressant systems shall be provided in all CENTRAL OFFICEs

·        Spares shall be provided in accordance with the manufacturer's standards, and shall be administered by locally developed procedures.

5.2.1.2  Circuits and Circuit Equipment

This section discusses the principal methods that are available to mitigate hazards of loss of circuits and terminating equipment.

National Security Emergency Preparedness (NSEP).

The integrity of the communications system is heavily dependent on the circuits connecting the CENTRAL OFFICE to the cell sites, LECs, and IECs.  All carriers are required by the National Security Emergency Preparedness (NSEP) program to classify certain customers under a Telecommunications Service Priority (TSP) system.  Under TSP customers are assigned restoration priority according to levels of priority that are prescribed by the NSEP document.  NSEP services are classed as emergency and essential.  Essential services are assigned restoration priorities 1 to 5, with 5 lowest. 

“The Company” has a dual responsibility as a user and a provider of emergency and essential services.  To fulfill this role, it is important that all circuits “The Company” uses have received TSP treatment. 

The following general guidelines apply:

·        Ensure that all dial facilities are marked as essential service in LEC records

·        Ensure that the LEC has established highest restoration priority allowable for all critical circuits in their TSP process

·        Review customer records and provide plans for restoring service to “The Company’s essential customers in accordance with TSP principles.

Circuit Diversity

In most cases the best way of ensuring circuit integrity is having two diverse routes to the distant location.  Diversity is still a viable option when more than one circuit is needed, but many carriers offer self-healing networks that will automatically re-route interrupted service.  Usually, these self-healing networks are supplied by the LEC and provide two paths for T-1 services to the IEC.

A common misconception about diversity is that if you use two common carriers for your service, you have adequately protected against a single point of failure.  This is not always true.  If the LEC furnishes facilities for the IEC, the same route may be used for both services.  Anything that damages one cable may damage both.  Also, many carriers purchase service from their competitors when they do not have adequate facilities.  The most effective diversity method is to use different media: for example microwave backed up by cable or fiber optics.

It is impossible to provide absolute guidelines for providing diversity because of local differences. 

Cost and feasibility studies should be done to evaluate the cost of diversity and the feasibility of alternatives such as self-healing service.

Access to the IEC

There are three potential failure points in access to the IEC:

·        The access facility itself

·        The interface between the CENTRAL OFFICE and the access facility to the IEC switch

·        The IEC's switch

Of the three, the access facility is historically the source of most outages related to long distance failures.  These outages have a variety of sources such as construction activity, commercial power failures, and unintentional service disruption by technicians.

The second most likely failure point is the access facility interface.  The most common failure at this point is wiring.  The least likely failure is the interface equipment.  Normally, interface cards fail only as a result of some external activity, such as a power failure followed by poor surge protection or mishandling of the cards.

The least likely point of failure is the IEC switch.  “The Company’s T-1 facilities normally interface the IEC's switch via direct digital connection, which makes this an unlikely point of failure.  Because of diverse paths through the IEC's switch and network as well as advanced software and backup capabilities, such failures are rare. 

To protect against access circuit outages, consider either self-healing circuits or obtaining access from an alternate access carrier (AAC), or both.  AACs are private carriers that compete with the LEC in providing T-1 facilities, usually over fiber optic cable that takes a different route from that provided by the LEC.  Another alternative is to obtain a guaranteed diversity option from the IEC.  With this service, which is not always available, the LEC guarantees that the routes take different paths from the user's premises to the central office.

The next most likely source of failure is the interface between the access facility and the CENTRAL OFFICE switch.  This includes wiring between the switch and demarcation point, and any interface equipment such as channel service units (CSUs).  Wiring failures usually occur during circuit or maintenance activity on the same wiring block or backboard.  These are usually spotted quickly by an experienced technician and are generally easy to repair. 

To avoid problems of this sort, follow these guidelines:

·        Follow standard color codes in all wiring plans.

·        Maintain cross connect records on all circuits entering the site.

·        Follow workmanship standards in placing connections and maintaining the correct amount of slack.

·        Use care in placing new service to avoid disrupting existing connections.

If the circuit connection is electronic such as DACS, ensure that the DACS software is protected with the same methods as are used for backing up CENTRAL OFFICE software.  For CSUs ensure that there is a spare CSU for every four units, and that active CSUs receive power from two separate sources.  For channel banks, provide a spare set of common cards for every four banks. 

The least likely point of failure is the IEC's switch.  Where available, consider obtaining access to an alternate switch.  AT&T, for example, provides a service known as SAFER to provide access to a secondary 4ESS.

Microwave

Disaster planning and recovery for microwave systems largely depends on a good initial system design, thorough acceptance tests, and good maintenance practices.  The design engineer must provide an appropriate level of maintenance spares, and maintenance personnel must administer them professionally. 

The initial design includes factors such as these:

·        A theoretical design that includes an adequate fade margin.

·        Path loss testing that ensures that the path is within calculated limits.

·        Antennas with the appropriate amount of gain, properly oriented, and securely mounted to the tower.

·        A tower with sufficient rigidity to maintain antenna orientation, even during adverse weather.

·        Spare antennas, cables, wave guide, air compressors, power supplies, and radio equipment.

·        Equipment with a high mean time between failures, redundancy, and a low mean time to repair such that availability is no less than 99.97 percent.

Spare levels of 10 percent or less are usually adequate since the mean time between failures on most modern microwave systems is high.  It is expected that the actual volumes of maintenance spares should be low--at least one spare of each operating device or  circuit board should be retained in each metropolitan area.  The location of spare parts should be based on the driving time to working locations.

The following are the principal countermeasures against microwave system loss:

·        A properly engineered and tested initial design that provides adequate fade margin.

·        The provision of hot standby diversity for all microwave, plus space diversity if the region is subject to severe fading.

·        The provision of diverse routing; ie. an alternate microwave route or provision of a cable or fiber optic route between points.

5.2.1.3  Common Equipment

All sites, except those where a total site outage for prolonged periods can be tolerated, must have provisions for backup power.  If the site is not equipped with an emergency generator, the batteries should be sized to provide a minimum of eight hours service during the busiest hours of a normal business day.  The rectifiers should also be sized to serve the normal full load of the cell plus the capacity to recharge batteries that have been discharged.  If the site is not equipped with an emergency generator, provisions should be made to rent or lease backup power during emergencies. 

Where the survival of the network depends on leased emergency power, it is not enough to obtain a verbal commitment to supply a generator.  A written contract stating the terms and conditions of the agreement must be negotiated.

For those sites that are equipped with generators, the batteries should be sized to provide a minimum of two hours service in case the generator fails to start.  The reduced battery cost will help offset the cost of the generator.  All fixed generators should be equipped with an automatic transfer panel that will sense the loss of commercial power, start the generator, and transfer the load.  In addition, generators must be equipped for remote start and load transfer.

Emergency generators must be kept in operative condition by regular testing under load.  A log of such tests shall be maintained in each maintenance center.  Fuel tanks must be kept filled, and an available source of fuel should be kept under contract. 

The following guidelines apply to emergency power:

·        All CENTRAL OFFICEs and critical sites shall be equipped with emergency power.

·        All emergency generators shall be run weekly, and the site load transferred to the generator.

·        All critical emergency generator functions such as temperature, output voltage, output current, oil pressure, etc. shall be alarmed to the maintenance center.

·        The specific gravity and cell voltage of all open cell batteries shall be checked and logged weekly.

·        All critical sites that are not equipped with an emergency generator shall have a minimum of eight  hours battery reserve.  All such sites shall be equipped with standard couplers for connecting portable generators.

·        All portable generators that are to be leased in case of emergency shall be contracted for under an agreement that ensures they will be available when needed.

When more than one item of equipment of the same type is located in an equipment room, it is important to ensure that the two sets are powered by separate feeds through separate fuses.  If the function is critical enough, a backup power supply should be installed. 

5.2.1.4  Records and Software

As with any processor controlled device, the CENTRAL OFFICE is vulnerable to human-caused errors.  Proper training and supervision are the key to avoiding these types of outages. 

To ensure that the system can be restored rapidly in case of failure, an adequate set of records must be maintained.  These records must show how the system is equipped, and how circuits are interconnected.  Customer and billing records are essential and irreplaceable parts of the communications system.  These must be protected by maintaining backup records at an off-site location where they can be rapidly obtained in case of failure.

The following principles shall be followed to ensure the survivability of the CENTRAL OFFICE:

·        Dump two copies of the system load and billing records at least once per week.  Store one copy on site in a fire and water proof safe.  Store the other copy off-site in a local storage facility.  Maintain a local

·        record of the location, phone number, and contact person.  Provide the Corporate Network Service Center with a copy of the location record.

·        Maintain a log of all system changes other than to the subscriber data base. 

·        Dump additional system backup tapes, including billing data, prior to and during all major system changes.

·        Maintain records of the network configuration both on-site and off-site. 

Include at least the following:

·        Network diagram

·        Circuit list showing A and Z locations and carrier

·        Circuit cross connection diagram

·        Equipment wiring lists and wiring records

·        Manage system security according to the principles in Section 5.1.5

·        Arrange for potential backup billing vendors in case the primary billing vendor's facilities are damaged or destroyed.

5.2.1.5  Central Office Overloads

The nature of telephone traffic is such that a properly operating switch can be impaired by overloads that occur during disaster situations.  These are not always caused by natural disaster.  Important news events such as a presidential assassination attempt or an outbreak of civil disturbances or war can cause circuit overloads.  These overloads can affect any element of the communications network: radio channels at a cell site, circuits from the cell site to the CENTRAL OFFICE, and circuits from the CENTRAL OFFICE to the LEC, IEC, or other CENTRAL OFFICEs.  The CENTRAL OFFICE itself can be overloaded by an excessive number of call attempts that exceed the processor capacity of the system.  Overload conditions may be aggravated during a disaster by the loss of circuits to other sites.  Each market area must prepare a plan for coping with momentary overloads. 

Traffic administrators and maintenance personnel have four primary ways of protecting the CENTRAL OFFICE against overloads:

·        Revising trunking quantities to provide more trunks to destinations that are blocked because of heavy loads.

·        Revising traffic to restrict the number of trunks from sources that are originating excessive numbers of ineffective attempts.

·        Operating line load control to provide service to high priority users, while restricting access to non- essential services.

·        Temporarily providing alternate routes to relieve overload conditions.

The uniqueness of local conditions makes it impossible to specify actions that must be taken to relieve overloads. 

Each location in its disaster prevention plan shall take overload protection into consideration. 

Consider these elements to the plan:

·        Determine the most likely cause of overload.

·        Evaluate the points in the network that are the most vulnerable.

·        Develop information sources that can rapidly be tapped to determine the extent of the problem.  These can be data from intelligent devices such as switches, DACS, etc.

·        Develop emergency plans for managing the network until conditions return to normal.

5.2.2  Loss or Impairment of Remote Sites

The first level of security at a site is physical (locks, fences, etc.).  The probability of intrusion into a properly constructed site is low.  Furthermore, unless it is a critical site, the amount of damage that would be inflicted is costly, but not a severe hazard to service.

Tower

Towers are subject to failure due to high wind conditions, excessive icing, vandalism, and sabotage.  If a tower is lost, regulatory limitations may preclude restoring the tower immediately.  For example, regulatory authorities may dictate whether a damaged or destroyed tower may be repaired or replaced. 

There are several options available should a tower become unusable:

·        Operate the cell on antennas just above ground level (6 to 8 feet).  This is probably the most immediate solution, but it also provides the least coverage.

·        Install temporary antenna supports such as telephone poles.  This option may limit transmit height, and could also require approval by local authorities.

·        Relocate site to a new structure in the immediate vicinity.  This option is time consuming and potentially expensive.

 

The most effective means of controlling loss of a tower is in preventive steps during construction and maintenance.  New towers should be built in compliance with EIA RS-222 specification revision E.  This specification includes design requirements for wind and ice loading that are specific to the geographic region.  Any existing towers should also comply with these specifications. 

Many towers in “The Company’s network were built under an earlier revision of RS-222.  Assuming the tower was compliant when constructed, these earlier specifications were sufficient.  Any future analysis of existing towers should be performed using the latest EIA specification; however there is no need to perform a wholesale analysis of existing towers under the new specification. 

Any towers that are not engineered for wind and ice loading should be analyzed immediately and corrective action taken.  This requirement includes both “The Company”-owned and third party-owned towers. 

Proper maintenance of existing facilities can also aid in preventing tower failure.  Many tower manufacturers offer inspection services, and guidelines for inspection.  At a minimum, all towers should be inspected annually.  The inspection should include guy-wire tension, visual inspection of all welds and antenna mounts, checking bolt tension with a torque wrench, and a general inspection for rust, with repainting where necessary.  These inspections should be filed and compared with previous years' inspections.  Recurring problems can indicate a trend that may lead to tower failure.

Antenna

As with towers, the loss of a single antenna in a communications system is not likely to cause catastrophic loss of service, but the importance of any single antenna or array will depend on system design.  From a disaster recovery standpoint it is desirable to identify a limited number of generic antenna types to be kept on hand as spares.  These antenna types will vary by market depending on system design.

PART II 

SPECIFIC DISASTER PREVENTION AND RECOVERY PLANS

1.0 INTRODUCTION

Part I of this plan provides the general instructions and underlying philosophy of “The Company’s disaster prevention and recovery plans.  This part discusses the actions that every market area shall take in developing site-specific plans.  This part begins with a discussion of the process for developing the plan, including steps that each location must take to identify hazards and determine the most cost-effective methods of ensuring that when events occur, service is not disrupted.

Several forms are included in this section to assist in inventorying the facilities that must be protected.  Forms are included for maintaining lists of how to reach personnel with specific roles in coping with emergency situations.  Instructions are provided for reporting threats that may result in taking emergency action, and in reporting actions as they are taken. 

In developing this plan, consider four different categories of activity:

1.  Business as Usual.  No reporting requirements are set by this plan other than assurance that emergency plans have been developed, tested, and disseminated to those who have a role in carrying them out.

2.  Hazardous Situation.  When a disaster threatens; for example, a severe storm is approaching, or an CENTRAL OFFICE is without redundancy and restoral is delayed, certain protective measures are brought to play.  These are reported as covered in Section 3.1 of this plan.

3.  Emergency Situation.  When a significant loss of service or loss of company property is encountered, this plan requires activating the Disaster Prevention Plan.  Further reporting requirements are established by Section 3.1 of the plan.  Specific roles are identified as covered in Section 3.3 of the plan.

4.  Restoration.  When the abnormal condition no longer exists, the process of restoring plant and resuming business as usual begins.  Section 3.4 of the plan discusses reporting requirements during this process.

Note:  A significant loss of service is defined in Section 3.0

2.0 DEVELOPING THE DISASTER PREVENTION PLAN

The first and possibly most important task is to conduct a risk analysis for each site (CENTRAL OFFICE, cell, or microwave repeater). As in any analysis, the end product will be no better than the facts used in arriving at a conclusion. Factual, current, hands-on data is of prime importance.

A suggested sequence for conducting the analysis is:

1.  Conduct a site inventory to identify areas of risk

2.  Determine the probability of loss

3.  Identify the risks to service

4.  Inventory resources available for emergency situations

5.  Determine emergency operating methods

6.  Identify roles and order of succession during emergency conditions

2.1 Inventory Sites

The first step is to review each site and document the review so it can be used both in the risk evaluation process and as a base record if needed to recover from an actual disaster. 

The initial review should be made within six months of approval of this Plan, and annually thereafter.  Use the Disaster Prevention Review forms to conduct this review.  The forms must be approved by the responsible manager and at a minimum, copies retained at the site location, the site maintenance center, and a copy forwarded to the Corporate Network Service Center.

2.2  Determine Probability of Loss

Following the site inventory, a probability identification will be done using the Probability/Effect matrices that are included for each section of the Disaster Prevention Review form. 

The matrix is used as follows:

·        For each question on the form, evaluate the probability of an event occurring.

·        For the same question evaluate the effect if the event does occur.

·        Place the number of the question in the appropriate block on the form.

For example, assume that a site in a southern location is considering Question 1 on Page 1 of the form.  If heating fails, in the CENTRAL OFFICE, heat from the equipment will probably keep the site in an operative condition.  The probability of failure is low, but the effects of failure, which could be loss of redundancy in the CENTRAL OFFICE, are medium.  The block in the lower row of the middle column would contain a "1" for question one.

The probability assigned to an item does not depend on the importance of that item to the site, network, customers, or the corporation.  It depends on the likelihood of the event occurring.  The effect has nothing to do with whether an event is likely to occur.  It states the impact on the company if it does happen.  If management decides to accept a high or medium probability condition, the rationale should be discussed on the form together with a discussion of what recovery action would be appropriate.

Because the classification of probability and effects is subjective, risk identification and evaluation process must be done by an individual with broad overall system knowledge.  The overall evaluation must be approved by an appropriate level of management.

2.3  Identify Risks

Based on your knowledge of local conditions, identify the risks to service, and record the result in the blocks following the question. 

The evaluation should be based on your knowledge of operations, considering questions such as these:

·        Will the coverage provided by this location be assumed by

   another site with little or no degradation in service? 

·        Will the loss result in a reduction in traffic carrying capacity

   but not an isolation? 

·        Will the loss result in seriously degraded service for essential

   services?

·        Will the loss result in a total lack of coverage for an area?

·        Will significant lost revenues likely result?

·        Will the costs of restoral be higher than the costs of mitigating the hazard?

·        Will the loss result in adverse public perception of “TheCompany’s service?

·        Could access to a location be a problem? Is it a matter of travel time from the maintenance center to the site or one site to another?

·        How long does it take to supply replacement equipment from a central depot or the factory.

Check the appropriate block on the form indicating whether the situation is OK, Needs Attention, or Does Not Apply.  If the response is Needs Attention, briefly describe the unsatisfactory condition and indicate what is being done to correct it.  Attach additional sheets to the form if necessary.  If the response is Does Not Apply, briefly indicate why.

 

Even after taking all of the above into consideration, there will be situations where it will not make good business sense to provide the equipment spares, routing diversity, one-for-one backups, or other protective measures. In those cases, by going through the risk management process, and documenting your findings, the resulting recovery, the disaster will be much more manageable if it occurs.

2.4  Identify Resources

The resources available for coping with an expected emergency or recovering from a disaster include the following:

·        Personnel

·        Emergency spares and backup equipment

·        Test equipment

·        Communication equipment

·        Vehicles

This Plan includes forms that shall be used to record resources.  It is essential that these forms be kept current.

2.4.1  Market Area Directory

This form is an inventory of personnel resources that are available to assist in disaster prevention or recovery.  Space is provided to record the names, telephone numbers, vehicle numbers, and areas of responsibility of those with emergency assignments.

2.4.2  General Site Inventory

This form is an inventory of information common to all types of sites: CENTRAL OFFICE, cell site, and other type of site such as a radio repeater.  The purpose is to provide a uniform location in all sites for important information in case of emergency. 

2.4.3  Circuit Inventory by Site Location

This form records all circuits terminating in a site.  It is a working document that is kept current with circuit order activity.

The "A" location on the form is always the highest level site in the hierarchy of circuits.  For example, in a circuit from a cell site to the Central Office, the C.O. is the A location, and the cell site is the Z location.  In a circuit extending from a cell site to a hub cell site, the hub site would be the A location. 

2.4.5.  Determine Emergency Operating Methods

Insofar as possible, preparations should be made in advance on how operations will be conducted during various types of emergency situations. 

Remember that it is not enough to simply discuss and document emergency plans.  If you are depending on outside resources to supplement your normal resources, negotiate contracts and agreements to ensure the resources will be available when they are needed.

Since no emergencies are ever the same, it is impossible to foresee every contingency, but this section discusses considerations that all market areas should use in planning for the most likely type of emergency in each locality. 

·        Have emergency assignments been given to departments and

   key individuals?  Are all individuals aware of their emergency

   assignments?

·        How will vehicles be deployed during the emergency?  Have

    plans been made to lease vehicles, rent them from

    employees, or use other means of obtaining extraordinary

    resources?

·        Have arrangements been made for fuel sources for

   emergency vehicles? 

An excellent alternative may be to contract with a card-lock station for a supply of cards and an assured supply of fuel.

·        If emergency engines are in continual operation, have

   arrangements been made for replenishing fuel during

   emergency conditions?

·        If you are planning to lease generators to support critical

   sites during an emergency, are connectors available to

   connect them to the rectifiers? 

Have arrangements been made to refuel them?  Remember that in an emergency, engine-driven DC arc welders can be used to charge batteries.

·        Are there certain sites that are normally unmanned that should be manned during an emergency?

·        Have arrangements been made to house and feed personnel who are on emergency assignments?

·        Have arrangements been made for transportation of emergency equipment?

·        Are there agreements with the competing communications carrier to provide service to “The Company’s customers in case of major failure?

·        Where are emergency radios and communications telephones kept?  Are there enough of them to cope with an emergency?

·        Have provisions been made to obtain a supply of emergency cash or credit cards to purchase minor tools and supplies during a crisis?

·        Have arrangements been established with construction contractors who have cranes, bulldozers, trucks, and other such equipment available for use in a disaster?

·        Have arrangements been established with security firms to furnish personnel to guard company property during an emergency?

·        Are there sufficient quantities of emergency supplies such as tarps, sandbags, foul weather gear, lanterns, shovels, ropes, and snowshoes available to cope with the expected  emergency?  Are they stored where they will be accessible  when needed?

2.4.6.  Identify Roles and Order of Succession

The provisions of the Disaster Prevention Plan are generally not set into motion until someone declares that an emergency situation exists.  There is a natural tendency on the part of most people to attempt to handle an abnormal condition before they ask for assistance.  There is more risk from failing or waiting too long to declare an emergency than from declaring an emergency when none exists.  Each location must designate who is empowered to declare an emergency, and set the provisions of the Disaster Prevention Plan into effect.  An order of succession must also be established if the persons empowered to declare an emergency are not available.

Also necessary is a plan describing the roles of various parts of the organization during emergencies.  Section 3.3 provides guidelines for the roles of the various departments.

3.0 NOTIFICATION AND REPORTING OF EMERGENCY SITUATIONS

Local management is responsible for determining when an emergency situation exists and for notifying higher management.  The following are examples of situations that require notification using the procedures outlined in this section:

·        Total failure of an Central Office for one minute or more.

·        Failure of a major circuit facility that results in isolating a major community from communications service or causes loss of access to one of “The Company’s SS-7 or data networks.

·        Loss of a critical remote site

·        Loss of 50 percent or more of the circuits connecting the CENTRAL OFFICE to the LEC, IEC, or a critical cell-site.

·        Loss of property by damage or destruction from any of the hazards discussed in Section 4 of Part I where the monetary impact from lost revenues or restoral exceeds $10,000.

·        A high risk of service loss exists because of an impending storm, civil disturbance, fire, or other such disaster.

 

3.1  Reporting Abnormal Conditions

Reports shall be forwarded by the most expeditious means to higher management and to the Corporate Network Service Center.  Reports to the Corporate Network Service Center shall be sent by facsimile using the format in the Emergency Report form.  Supplemental reports are required every two hours until the hazard has abated.

Any communications system experiencing an emergency situation should alert its back up systems, other cities, and other Technical Operations groups that may have extra maintenance spares and extra technical support.  If necessary, vendors should be contacted if they are needed to supply spare parts and technical people.

3.2  Disaster Recovery Procedures

The first priority in recovering from a disaster situation is to see that the safety of personnel is not jeopardized.  All personnel must be cautioned not to take unnecessary risks to restore damaged facilities. 

Consistent with safety, the following are the primary activities to take in restoring damaged facilities:

·        Assess the extent of the damage.  Determine as rapidly as possible exactly what has been lost and what has caused the loss.

·        Determine what measures can be taken to restore service, temporarily if necessary.  Set priorities for restoring the most important services first.

·        Obtain resources for restoral.  Contact other “The Company”  locations, contractors, and vendors to furnish the personnel and equipment needed to restore service.

·        Deploy resources as available to restore service.

3.3  Invoking the Disaster Prevention Plan

Upon notification of an emergency, “The Company’s management will assess damage reports and determine which contingencies will be invoked.  Key personnel may be involved in the decision-making process in order to establish a unified effort to expedite restoration of service.  If needed, a "war room" will be established under the direction of the Operations Department.  Department heads shall obtain resources to perform the tasks outlined below.

3.3.1 Operations Department

The Operations Department has the following responsibilities under this plan, and shall determine which of these contingencies will be invoked:

·        Notify higher management and the Network Management Center of the emergency.

·        Inform all affected employees that the Disaster Prevention Plan has been invoked, and direct them to report to specified locations. 

·        Establish a "war room" to be used to direct system restoration. 

·        Provide communication facilities, telephones, desks, etc., as required. 

·        Provide, if required, temporary quarters for system equipment, war room, or any other need during the

·        Survey the damage and establish plans for restoring service.

 This information is to be determined as soon as possible and higher management notified.

·        Establish the restoration program.   Define timelines for each activity, and manage employees and vendors until restoration is complete.

·        Notify the FCC that communications system is not in service due to disaster condition.  If a tower is involved, the FAA must also be notified.

·        Notify key vendors that “The Company” is under its Disaster Prevention Plan and that they are to be prepared to supply personnel and equipment in accordance with agreements.

·        Obtain necessary documentation required for restoral. Establish equipment lists, specifications, materials, and costs required to accomplish restoration.

·        Provide the Administration Department with estimated cost to restore service and the target date for completion.  Obtain authorization to proceed with placing orders.

3.3.2 Administration Department

The Administration Department has the following responsibilities under this plan:

·        Notify employees outside Operations that the Disaster Prevention Plan has been invoked.

·        Notify the insurance carrier that the Disaster Prevention Plan has been activated.

·        Establish an emergency pool of funds to restore plant based on estimated costs from the Operations Department.

·        Establish a process for providing status reports of recovery activities to “The Company’s owners and higher management.

·        Provide personnel to staff the communications center and war room.

·        Assess human resource requirements for emergency duty.

·        Provide personnel to supplement the Operations Department staff

·        Establish an emergency accounting system to expedite transfer of funds for restoral activities. 

·        Maintain records for insurance company and regulatory bodies and prepare the appropriate documentation.

·        Contact competitors to notify them of the disaster condition and to negotiate emergency service for “The Company” subscribers. 

3.3.3 Customer Service

The Customer Service Department has the following responsibilities under this plan:

.  Invoke a pre-established text for informing customers that

   call Customer Service regarding service.

.  Develop a process for calling key subscribers to notify them

   of service interruption.

.  Assess resource requirements and advise Administration of

   help and/or material needed.

.  Notify the billing company that tapes will not be sent until

   further notice if the system is out of service.

.  Estimate lost revenue for insurance purposes based on

   historical data plus projections.

.  Maintain the customer data base for loading the switch, when

    required.

.  Develop and maintain plans in case the billing vendor's

   facilities are damaged or destroyed.  Customer Service shall

    review alternate vendors capabilities and negotiate a

    contract for billing services.

3.3.4  Accounting Department

The Accounting Department has the following responsibilities under this plan:

.  Establish and maintain functions required to track capital

   and expense costs for the disaster and its recovery functions.

.  Prepare periodic reports and continuous tracking of Purchase

   Orders, Expense Statements, vendor invoices, etc.

3.3.5 Marketing and Sales Department

The Marketing and Sales Department has the following responsibilities under this plan:

.  Notify all affected employees that the plan has been invoked

   and inform them where to report for disaster assignments.

.  Notify agents and sales persons of circumstances and

   activate pre-planned procedures.

.  Act as the source of all information, both internal and

   external, and issue approved statements to the media.

.  Prepare mailings and other means of notification to

   customers, advising them of alternative solutions.

.  Maintain contact with agents and customers to maintain the

   customer base during service interruption.

.  Prepare and submit plans for approval to recapture lost

   customers when service is restored.  Include budgetary

   requirements and forecasted results.

3.3.6  Credit and Collections

The Credit and Collections Department has no specifically assigned responsibilities under this plan.  However, they shall be prepared to furnish personnel to support other departments if necessary.

3.4  Deactivating the Disaster Prevention Plan

When an emergency occurs, the objective is to resume normal business operations as rapidly as possible.  Once activated, the Disaster Prevention Plan should remain in effect until one of the following occurs:

.  Service is restored to at least 95 percent of normal levels.  ie,

    no more than 5 percent of customers are experiencing lost

     or impaired service

.  Complete restoration is delayed pending replacement of a

    major system that cannot be replaced immediately.

.  Higher management directs that the emergency operations be

    discontinued, and the location resume normal business

    activities.

The plan is deactivated by discontinuing the war room and return   personnel to their normal assignments.  All the key organizations that were initially notified of the emergency, including, if applicable, the PUC and FCC, shall be notified through the appropriate channels.  A final report shall be prepared and forwarded to the Network Control Center to close the outstanding emergency report.

3.5  Plan Review and Testing

The Disaster Prevention Plan is intended to be a living document that changes with day-to-day activities.  For example, circuit records, which are essential for preventing disasters, change almost daily, so the process for keeping them up to date must be part of normal business activity.  Some parts of the Plan change slowly, if at all, but still must be reviewed occasionally. 

The Disaster Prevention Plan shall be reviewed annually in its entirety.  Only portions of the plan that have changed since the previous year must be changed.

It is generally not practical at this time to test the Disaster Prevention Plan in total.  Individual elements of the plan should be tested, however, to ensure that they are available when needed. 

This includes at least the following:

.  All emergency engines must be tested by running them under

   load at least weekly.

.  All backup battery supplies must be maintained and charged

   according to the manufacturer's specifications.

.  All alarms must be tested at least quarterly to ensure that

   they are operative on both the sending and receiving ends.

.  Batteries on emergency equipment such as radios, flashlights,

   and lanterns must be tested quarterly to ensure that they are

   available.

.  Stocks of emergency supplies must be reviewed annually at a

   time appropriate to their most likely need.  For example,

   supplies required for storm restoration should be checked at

   the beginning of the storm season.

PART III 

REGIONAL DISASTER PREVENTION AND RECOVERY PLANS

1.0      INTRODUCTION

This part of the Plan is reserved for adding plans that are designed to cope with particular regional hazards that are not covered by the main body of the plan.  Where such hazards exist, management shall develop and insert their locally developed plans. 

Examples are:

.  Plans for securing plant when a hurricane warning is received

.  Plans for providing community support when a serious

   earthquake occurs

.  Plans for maintaining service in the midst of a severe

   blizzard that makes it difficult or impossible to reach key

   sites

.  Plans for coping with flooding where key sites are located in

   a flood plain.

Disaster Mitigation and Recovery Plan

Treat Review

Companies are vulnerable to a disaster when management has not taken protective measures or prepared for recovery. A necessary first step in disaster prevention process is to identify which risk present the greatest threats to the business. This requires identification of service hazards. This section identifies the most common service hazards and provides space for management to identify the extent of the risk involved and planned action to mitigate unacceptable risk.

 

Instructions for Completing Review

This review shall be completed annually for each site. The review consists of the following parts:

Buildings

Physical condition

Site Location

Heating ventilation and air conditioning

Utilities

Fire prevention and suppression

Security

Environmental Alarms

Equipment

Emergency generator

Alarms

Routines

Spares

Data Security

Earthquake protection

Software

Network facilities

This review must be completed by personnel who are familiar with the site and how it fits into the companies total operations. It must be approved by an appropriate manager in the Operations Department for the market area. Each section of this review may be completed by a Different individual. To complete the review, follow these steps:

1.      Determine whether the site is critical. See Part I Section 1.1 for the definition of a critical site.

2.      Determine the effects and probability of failure for each question and enter the question number in the appropriate box in the evaluation matrix. Note that the effects and probability of failure must be evaluated independently. See part II section 2.2 and 2.3 for further information.

3.      Consider each question separately, and mark the appropriate check block on the form.

4.      If the item needs attention, explain in the space provided. This space can also be used to enter remarks to assist higher management auditors in understanding the rational for the ration. Note that items with high probability of failure and server effects from failure require the most explanation. Use the back of the form or attach additional sheets if necessary

5.      Have the form approved by the appropriate manager.

6.      File the form in the site binder in the Market’s operations center.

 

Buildings

 

Completed by__________________-

 

Date___________________________

Approved By___________________

Date___________________________

 

Physical Condition

With the physical condition of the building we are primarily concerned with protecting the building and equipment from natural forces or civil damage. The civil damage can be either intentional or unintentional. In evaluating physical condition, the manager must be alert to the different hazards presented by varying geographical and urban or rural locations.

 

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Buildings housing critical sites shall be structurally sound to survive the severest weather conditions expected.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      The building shall comply with current building code standards for earthquake resistance for emergency facilities.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Critical sites shall not be located in a liquefaction zone. (earthquake)

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4. All building structural items shall be kept in operative condition;

a. Doors (including special entrances)

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

b. Locks

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

c. Vents

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

d. Windows and hatches (secured)

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

e. Roofs

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

f. Drains (internal system)

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

g. Gutters and sewers (external drainage systems)

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      Building alarms shall be logged when they occur, and if not cleared by on-site personnel, maintenance personnel shall be dispatched.

     OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      Physical separation shall be maintained between equipment rooms and other areas. Separations shall be reviewed to ensure that they have not been compromised by construction activity.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      All doors, windows, and access hatches shall be alarmed.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      Tarps shall be available to protect equipment in case water enters the buildings from an overhead source.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

9.      Other areas of the building shall be inspected at least annually to see that the owner maintains a fire-resistant environment.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Site Location

Sometimes risk result from the location of the site itself. For example, in multi-tenant building owner or outsiders, and not by the Company. It is also essential that arrangements be made with public agencies to provide identification cards of other means of ensuring that our personnel have access to areas that are otherwise closed to the public because of disaster conditions.

 

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Immediate access must be available on a 24-hour basis and not subject to disruption by lack of facilities or activates of other building tenants.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      Arrangements must be made with public safety agencies to ensure that personnel are permitted in closed areas to maintain service.

 

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3. Critical sites shall be located close enough to fire department that fire fighters can arrive in 15 minutes or less.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4. Access to equipment areas should be restricted to the Companies employees, or to building owner and his authorized and bonded security service.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5. No overhead plumbing, including sprinkler systems should be located in the equipment room except when required by fire codes.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6. If overhead sprinkler systems are required by fire codes, high temperature heads are recommended and operation should not be triggered by sensors located in remote areas of the building. Sprinklers should be dry charged if possible.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      No hazardous material or work operations should be permitted in the equipment room or in adjacent areas.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      Critical sites must be located away from natural hazards such as high water, hazardous materials, explosives, etc.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

9.      Activities by other tenants in a multi-tenants in a multi-tenant building should not impose unacceptable hazard.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Heating, Ventilation and Air Conditioning (HVAC)

Date of last HVAC preventive maintenance

Most electronic equipment is designed to operate reliably with a given temperature and humidity range. If the temperature of humidity is allowed to exceed these tolerances, the result can range from service impairment to total failure of the site.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      If required, backup heating should be available during an HVAC failure.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      If required, backup cooling should be available during an HVAC failure.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      If required, backup dehumidifiers should be available during an HVAC failure.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4. Heating system alarms should be sent to the network operation center.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5. Cooling system alarms should be sent to the network operation center.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6. Dehumidifying system alarms should be sent to the NOC.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Utilities

Power is the principal utility that is evaluated in this section. The power company should be contacted to ensure that critical sites are high on their restoration priority list.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

1.      Power failures must be alarmed to the NOC.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      The power company shall be contacted to ensure that the site is on the power company’s priority restoration list or priority feeder route.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      All electrical circuits shall be protected with property size fuses or breakers.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      Emergency lighting should be in place and automatically activated during an AC power failure.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      Emergency lighting shall be tested and logged monthly.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      All communications circuits entering the building shall be equipped with electrical protection devices such as carbon blocks and gas tube protectors.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      The land line telephone installation must be adequately protected against failure. The entrance should preferably be underground. If aerial, the drop wire shall be securely attached and as secure against failure from icing, etc. as possible.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      The equipment ground shall be tested as least once per year with a ground megger and the results logged

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Fire Prevention and Suppression

Date of last fire suppression system inspection

_______________________________________

Date of last contact with fire department

_________________________________________

Date of last portable extinguisher inspection.

While most fires start from internal causes, attention must be paid to conditions outside our buildings to reduce the possibility of fire damage from other sources. In this area we must consider the protection of service and equipment in addition to the safety of employees.

                                                                                                                                                                                         Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Equipment rooms should be separated from office areas with doors that close automatically in case of fire.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      All equipment and battery rooms shall be protected with a fire suppression system.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Fire suppression system shall be of a type that limits damage to the equipment itself through release of the extinguishing agent.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.     Fire and/or smoke detectors must be installed in all equipment rooms.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      Fire and /or smoke detectors shall be tested at least annually, and the results of the test entered in the site log.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      Fire and / or smoke detectors shall alarm to the NOC.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      Fire and /or smoke detectors shall automatically release fire extinguishing agent.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      The fire suppression system must be professionally designed and reviewed whenever the building is expanded.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

9.      Release of fire extinguishing agent should automatically disconnect both AC and DC power to equipment.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

10.  The room shall be sealed sufficiently to ensure proper operation of the extinguishing agent.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

11.  All cabling shall meet the flammability requirements of UL standards 910

and 1066.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

12.  Power and communications cabling should be in separate cable trays, ducts, or raceways.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

13. Portable extinguishers shall located within 75’ of any inside building location.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

14.  A portable extinguisher shall be located at a point furthest from the exit door.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

15.  A portable extinguisher shall be located at a point jhust inside the entrance door.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

16.  A bank of portable extinguisher shall be provided for fire department use.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

17.  Escape routes shall be clearly marked. 

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

18.  No flammable liquids shall be stored within equipment buildings.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

19.  Flammable material (paper, rags, boxes) shall not be stored in equipment areas.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

20.  Grounds within fenced area shall be cleared of all flammable material and over-vegetation.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

21.  Fire alarms shall be logged when they occur, and if not cleared by on-site personnel, maintenance personnel shall be dispatched.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

22.  Smoking shall be prohibited in all equipment rooms, and the site shall show evidence that the policy is observed.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

23.  Good housekeeping practices shall be observed to keep equipment areas free of dust, debris, and other flammable materials.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

24.  The fire department shall be contracted to acquaint them with our concerns about water damage, to show them how to turn off power, and where portable extinguisher are located.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Security

Date of last lock code changed______________________

The company’s buildings are at risk from intruders who can cause malicious damage, steal property, disrupt service, or simply satisfy idle curiosity. Buildings and fenced enclos8ured must be kept as impervious to intrusion as possible.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Doors to all equipment areas shall be locked at all times unless they are accessible only through an area that is restricted by guard or receptionist to authorized persons.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      In areas with access card systems, the code shall be changed monthly.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Whenever possible, sites shall be fenced to include both building and generator structures. Fencing shall be a minimum of six feet high with three strands of barbed wire on top.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      All fenced areas shall be equipped with lockable gates, and gates shall be kept locked.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      All locked doors shall be alarmed to the NOC.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      Offices and remote sites shall both maintain logs of all personnel entering the site.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      All intrusion alarms shall be logged when they occur, and if not cleared by on-site personnel, maintenance personnel shall be dispatched.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      Property shall not be stored outside the building unless it can be secured.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

9.      Telephone numbers of the responsible police & fire departments shall be posted by the door and maintained at the office.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

10.  Sign shall be placed on all entrance gates or doors identifying the area as “Restricted” The site name and NOC 800 number for emergency access shall be posted on the door.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

11.  All windows accessible to the public shall be alarmed and have security screens in place.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

12. All sites shall have alternate means of communication such as order wire or land line telephones.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (   

Equipment 

 

Completed by_____________

 

Date_______________________

 

Approved By______________

 

Date______________________

The equipment installed at each site must be reviewed for both normal usage and abnormal conditions. Requirements to be consider are equipment deterioration, forces external to the system, and-man problem.

Emergency Generator and Batteries

All sites must either be equipped with a permanently mounted emergency generator, or wired for connection to a trailer-mounted generator. Connections for external generators must be tested under load at least monthly to ensure that the wiring and transfer circuitry are operational. Also, current requirements must be reviewed semi-annually to ensure that the generator designated for the site is adequate to handle the load.

Generator for this site is:____________________________________

 

Permanently mounted (  )

 

Located at

 

Date of last generator run

__________________________________________________________-

Date of last power requirements review

                                                                                                                                                                            Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      If the site is not equipped with emergency power, wiring shall be in place for emergency generator .

 

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      If the site is not generator-equipment, are there enough emergency generators available to ensure availability of one under expected disaster conditions?

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      If the site is set up for trailer-mounted generators, there must be adequate parking space within easy reach of the connector.

 

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      If the site is set up for portable generators, the generator and building wiring shall be tested monthly to ensure that they are correctly wired and operable.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      If the generator is permanently mounted, the fuel supply shall be secure from disruption during an emergency: i.e. fuel lines must remain intact, tank must not be susceptible to rupture, ETC.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      Batteries shall have enough capacity to run at full load for two hours if the site is equipped with an emergency generator, and eight hours if it is not.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

ALARM

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

1.      All critical equipment, emergency generator, and building functions shall be alarmed.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      Alarm sending capabilities shall be routinely verified and logged.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Equipment Routines

Continued operation of equipment is dependent on completing routines on schedule. Review personnel should check maintenance logs to verify that maintenance is not being deferred.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Equipment routine maintenance shall be scheduled and completed according to manufactures specifications and the results recorded.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      Permanent emergency generators shall be run weekley and the load transferred.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Trailer mounted emergency generators shall be tested under load at least monthly.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Spare Equipment

Spare equipment requirements must be determined with respect to: redundancy, traffic carrying capacity of remaining on-line equipment, distance and time to the central spare stock, and quantity of specific type of equipment installed.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

a.      Processor

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

b.      Microwave

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

c.      Radio

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

d.     Power

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

e.      DACS

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

f.       Multiplex Equipment

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

g.      Data Terminals

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Spare inventory located in technician’s vehicles shall be maintained at designated levels.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Data Security

The system software must be provided a level of security that is impervious to intentional or unintentional access. Unauthorized access to the software base can be extremely damaging from cost and service standpoints.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      All terminals that are not fully attended shall e located in secure areas that are not open to unauthorized access.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      All remote access ports shall have dual passwords and or security modem.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      All passwords shall be restricted to those who need to know.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      Passwords shall be changed on schedule, not to exceed one month, or when individual possessing the pass word leaves the company.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      Passwords shall consist of character strings that are long enough that intruders cannot easily guess the password. At least one character in the string shall be a number or a special character.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      The site must have a process for ensuring that keys and access cards are retrieved when a person leaves the employ of the company.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      Ports shall be designed to lock out after three unsuccessful attempts to enter the password, and the feature shall be activated.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Earthquake Protection

During an earthquake, equipment will be subjected to motions that depend on the magnitude of the seismic shock and the structural characteristics of the building. Earthquake protection takes into account the protection of equipment as well as the safety of employees. The equipment must be constructed and installed with sufficient bracing to withstand the effects of the seismic shock, and buildings, must be constructed not only to withstand the shock, but also to avoid damaging equipment through secondary effects such as collapsed ceilings, ruptured water mains, etc. Some parts of the country have building codes that prescribe the types of facilities that can house equipment that must remain operational.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      Has the degree of earthquake hazard been evaluated for the site.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      Earthquake bracing shall be installed if required by NEBS standards of building codes for the seismic zone.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      All bays, racks and cabinets shall be secured to the floor and or walls.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      Batteries shall be secured to racks with bracing appropriate for the earthquake zone.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Software

The availability of backup software is of major importance. The inability to regain system sanity or reconstruct lost customer and billing records demands that each site maintain fall-back software that is a duplicate of current data in the case of generic software, or is a reasonably recent copy of current customer records and billing information.

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

1.      Backup copies of system generic, customer records, and billing data shall be made at locally prescribed intervals.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      System generic backup tapes shall be verified prior to, and updated following, any system changes or new configurations.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Customer record and billing tapes shall be updated to tape or other storage daily.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      On site backup tapes shall be stored in a secure location remote from the on-site data and available for 24-hour per day access.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      On-site backup data shall be stored in a locked fire resistant container.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      A current copy of the generic backup date shall be maintained at a secure off site location. This must be the most current generation of the data.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

Network Facilities

Completed BY______________________________

Date_______________________________________

Approved By_______________________________

Date_______________________________________

To evaluate the strength of the network, facility structure consideration must be given to the strategic importance and anticipated reliability of each link or route. Redundancy, split routes, central office and facility diversity (dual facility, separate sheath, separate routes, different serving central offices, type of facility) are all different methods of maintaining network/facility reliability.

 

Effects of Failure

            

LOW

MEDIUM

HIGH

 

Probability of Failure

 

 

 

HIGH

 

 

 

MEDIUM

 

 

 

LOW

 

1.      All facilities serving customers who have Telecommunications Service Priority (TSP) shall be designated as Priority Services in the restoration sequence of all carriers furnishing the circuits.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

2.      Customer records shall be marked with TSP priorities wherever applicable.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

3.      Self healing features of facilities shall be tested in conjunction with the LEC at least semiannually.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

4.      Network configuration records shall be available for restoration if needed.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

5.      All facility cross connections (jumpers) shall be given special service identification at all cross connection points.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

6.      Critical sites shall have redundant or self-healing circuits furnished by different routes.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

7.      A plan for network overload protection shall be documented and tested.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

8.      All microwave radio serving critical sites shall be equipped with hot standby redundancy.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

9.      In areas that are subject to severe fading, microwave serving critical sites shall be equipped for space diversity.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

10. Fade margin test shall be made and recorded on all microwave radio.

OK (   )  NEED ATTENTION  (   )  DOES NOT APPLY (    )

 

General Site Inventory

 

Site Name___________________

Date________________________

Inventoried By_____________

Next Scheduled Review Date_________________

 

General

1.      Site Address

2.      Site landline telephone NO._________________________________________

3.      Type of Site______________________________________

4.      Special Access route Instructions

5.      Fire Department Tele#______________________________

6.      Police Department Telephone#____________________________

7.      Responsible Technician

Name_______________

Telephone Work____________________

Home_______________________

Mobile______________________

Pager_______________________________

8.      Responsible Supervisor:

Name__________________________________

Telephone Home_______________________________

Home___________________________________________

Mobil____________________________________________

Pager____________________________________

9.      Strategic Importance of Site to Network

 

Critical  (  ) Medium  (  )  Low  (  )  (explain)

__________________________________________________________________-

10. Strategic importance of site to major essential services concerning public safety and security

Critical  (  ) Medium  (  )  Low  (  )  (explain)

 

__________________________________________________________________-

Building

1.      Company Owned: yes (  )  NO (  )

2.      Other tenants:  Yes (  )  NO (  )

3.      Type of Construction :  Masonry ( ) Metal  (  )  Wood (  )

Other_____________________________________

4.      Supplementary heating provided  Yes  (  )  NO  (  )

5. Air conditioning provided   Yes  (  )  NO  (  )

6. Dehumidifiers provided  Yes  (  )  NO  (  )

7.      Fire

a.      Reporting: Automatic to control center

Yes  (  )  NO  (  )

Fire Dept Tele#____________________

Fire Dept Location________________________________________

b.      Detection/Suppression:

Type of Detectors_________________________________________________

Type of Extinguishers:_____________________________________

Automatic____________________________________

Manual_______________________________________

Test Sets

set_________________location: site (  )  Tech Veh (  )


set_________________location: site (  )  Tech Veh (  )

set_________________location: site (  )  Tech Veh (  )

set_________________location: site (  )  Tech Veh (  )

 

Power

1.      Commercial Power Supplied by_______________________________

2.      Batteries:

Manufacture_____________________________________________

Type/Model______________________________________________

Number of Hours Capacity________________________________

3. Emergency Generator on site Y  (  )   N  (  )

a.      If Yes:

Type of Fuel_________________________________________

Size of Fuel Storage Facility___________________________

Scheduled Re-Fill Date_________________________________

Minimum Hour of Fuel at Fuel at full AC load_____________

Fuel Vendor Name/Location____________________________

Tel# of above Vendor:________________________________________

Business Hours___________________________________________

Out of hours contact________________________________________

b.      If  NO

Size of Generator Required KW__________________________________

Hours required to re-charge batteries__________________________

Type of Emer. Gen. Interface at Site____________________________________

Special Instructions_____________________________________________________

Location of closest Company Owned Generator______________________________

Location of Closet Vendor-Supplied Generator__________________________________

Name of Above Vendor:___________________________________________________

Contract with Vendor yes  (  )    No  (  )

Contract Provisions_________________________________

Tel# of Above Vendor:____________________________

Business Hours__________________________________

Out of Hours contact_________________________________

 

Software

1.      System Processor:

Issue #_______________

Revision # ___________

2.      Software Backup

On site storage location

System________

Customer Records___________

Billing______________________

 

Market Area Disaster Prevention Directory

Date___________________________

Department

Name__________________________

Title____________________________

Home Phone_____________________

Work Phone______________________

Back up #_________________________

Pager #____________________________

Vehicle #__________________________

Area of Responsibility_________________________________

 

__________________Market Area

Site Listing

 

Site Name

Address

Telephone No.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Circuit Inventory By Site Location

 

NETWORK

 

Ckt ID

 

Primary System Chan

 

Primary Carrier

 

A Location

 

Z Location

 

Connects to CK ID

 

 

 

Self Healing

     Y   N

 

 

Backup system Chan

 

Backup Carrier

 

 

Bibliography

 


 

Disaster Prevention Plan

Bellcore Catalog of Technical Information

Includes a complete listing of Bellcore publications, many of which are applicable to the cellular industry. Obtain from Bellcore Customer Service Hotline (800)-521-2673

 

Disaster Planning Guide for Business and Industry Federal Emergency Management Agency (FEMA) 141 August, 1987

Provides information for disaster planning including vulnerability analysis, and planning for emergency opera­tions. Obtain from FEMA, PO Box 8181, Washington DC 20472

 

Disaster Mitigation Guide for Business and Industry Federal Emergency Management Agency (FEMA) 190 February, 1990

Provides information useful for preparing site-specific disaster plans. Obtain from FEMA, PO Box 8181, Wash­ington DC 20472 (no charge)

EIA standards and their publication Catalog of EIA and ]EDC Standards and Engineering Publications are available from Electronic Industries Association 1722 Eye St. N.W., Washington, DC 20006 (202) 457-4966

IEEE standards are available from IEEE Computer Society, Suite 608, 111 19th St. N. W. Washington, DC 20036 (202) 785-0017

 

National Electrical Code .

Provides technical regulations for electrical installations. Part 800 specifically discusses communications systems. Order from The National Fire Protection Association (NFP A) (800) 344-3555

Network Equipment-Building System (NEBS) Generic Equip­ment Requirements Bellcore Technical Reference TR-NWT­000063

Provides Bellcore's view of generic spatial and environ­mental requirements for new equipment systems deployed in central office and telephone buildings. Obtain from Bellcore Customer Service Hotline (800)-521-2673

 

Telecommunication Electrical Protection 1985 AT&T Select Code 350-060.

Discusses AT&T's requirements and specifications for protecting telecommunications circuits and equipment from damage by lighting, power cross, and other electrical disturbances. Order from AT&T Customer Information Center (800) 432-6600

 

Telecommunications Distribution Methods Manual Building Industry Consulting Service International (BICSI) Provides information on design of communications distri­bution facilities, both inside and outside buildings. Infor­mation is provided on fires topping, codes pertaining to cable installation, etc. Order from Testmark Laboratories Publications Department, 3050 Harrodsburg Rd. Lexing­ton, KY 40503

 

Telecommunications Service Priority (TSP) System for National Security Emergency Preparedness (NSEP) NCS Manual.

This manual provides procedures to implement the TSP system for NSEP as prescribed in National Communica­tions System Directive 3-1. Obtain from National Commu­nications System Washington IX: 20305-2010 or telephone the TSP Program Office at (703) 692-0040 or (703) 692-2115.

UL Standards. Available from Underwriters Laboratories, Inc. 1285 Walt Whitman Rd., Melville, NY 11747 (516) 271­6200

 

 





 

|Disaster Document| |Goals and Values| |Sample Mission Statement| |Customer Service Plan| |Employee Evaluation Form| |Sample Collocation Agreement| |Project Manager| |B-PON| |TDM TO IP TRANSITION| |Internet Links| |Network Operations Draft| |Information System Operations| |Project Cost Management| |PMBOC Knowledge Maping| |Home|