Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide November 2006 Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Text Part Number: OL-10853-01...
Page 2
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Inc.;...
Submitting a Service Request Definitions of Service Request Severity Obtaining Additional Publications and Information Product Overview C H A P T E R Introduction to Switches Switch Connectors Switch Components LEDs Power Supply Bay Power Supply Module System Features InfiniBand Connectivity...
Page 4
Safety Pre-installation Instructions Rack-Mounting the Switches Requirements Rack-Mounting the Switch Installing the Switch with One Person Installing the Switch with Two People Configuring Basic Connectivity Obtaining an IP Address Obtaining an IP Address from the DHCP Server for Ethernet Management...
Page 5
Power Supply Modules Locating the Power Supply Unit Failed Power Supply Modules Installing a Power Supply Module Removing Power Supply Modules Managing the Switch C H A P T E R LEDs Chassis Status LEDs InfiniBand Port LEDs Power Supply LEDs...
Page 6
Contents Specifications and Compliance Certifications A P P E N D I X Chassis and Management Interface Electrical Specifications Acronyms and Abbreviations A P P E N D I X Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide 78-10853-01...
Describes how to install the switches in a rack. Installing Field Describes how to install and remove field-replaceable Replaceable Units units (FRUs) on the switches. Managing the Switch Provides upgrade procedures for switch components. Hardware Diagnostic Describes how to monitor the performance of the Tests switches. Specifications and Provides specifications and compliance information.
Conventions Conventions This publication uses the following conventions: Convention boldface font italic font { x | y | z } [ x | y | z ] string font screen boldface screen italic screen font < > Notes use the following conventions: Means reader take note.
Page 9
Preface Warnings use the following conventions: IMPORTANT SAFETY INSTRUCTIONS Warning This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.
Page 10
Preface Conventions Avvertenza IMPORTANTI ISTRUZIONI SULLA SICUREZZA Questo simbolo di avvertenza indica un pericolo. La situazione potrebbe causare infortuni alle persone. Prima di intervenire su qualsiasi apparecchiatura, occorre essere al corrente dei pericoli relativi ai circuiti elettrici e conoscere le procedure standard per la prevenzione di incidenti. Utilizzare il numero di istruzione presente alla fine di ciascuna avvertenza per individuare le traduzioni delle avvertenze riportate in questo documento.
Page 11
Preface Conventions Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01...
Page 12
Preface Conventions Aviso INSTRUÇÕES IMPORTANTES DE SEGURANÇA Este símbolo de aviso significa perigo. Você se encontra em uma situação em que há risco de lesões corporais. Antes de trabalhar com qualquer equipamento, esteja ciente dos riscos que envolvem os circuitos elétricos e familiarize-se com as práticas padrão de prevenção de acidentes. Use o número da declaração fornecido ao final de cada aviso para localizar sua tradução nos avisos de segurança traduzidos que acompanham o dispositivo.
Page 13
Preface Conventions Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide xiii OL-10853-01...
Related Documentation Related Documentation For more information about the Cisco SFS 7000 switch product family, refer to these publications: Cisco SFS 7000D InfiniBand Server Switch Installation and Configuration Note • • Cisco SFS 7000P InfiniBand Server Switch Installation and Configuration Note •...
Preface You can access international Cisco websites at this URL: http://www.cisco.com/public/countries_languages.shtml Product Documentation DVD The Product Documentation DVD is a library of technical product documentation on a portable medium. The DVD enables you to access installation, configuration, and command guides for Cisco hardware and software products.
Product Alerts and Field Notices To see security advisories, security notices, and security responses as they are updated in real time, you can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS) feed. Information about how to subscribe to the PSIRT RSS feed is found at this URL: http://www.cisco.com/en/US/products/products_psirt_rss_feed.html Reporting Security Problems in Cisco Products Cisco is committed to delivering secure products.
Preface Obtaining Technical Assistance Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco Technical Support & Documentation website on Cisco.com features extensive online support resources. In addition, if you have a valid Cisco service contract, Cisco Technical Assistance Center (TAC) engineers provide telephone support.
Obtaining Additional Publications and Information Submitting a Service Request Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3 and S4 service requests are those in which your network is minimally impaired or for which you require product information.) After you describe your situation, the TAC Service Request Tool provides recommended solutions.
Page 19
You can access the Internet Protocol Journal at this URL: http://www.cisco.com/ipj • Networking products offered by Cisco Systems, as well as customer support services, can be obtained at this URL: http://www.cisco.com/en/US/products/index.html •...
Page 20
Preface Obtaining Additional Publications and Information Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01...
Cisco SFS 7000D Server Switch. Figure 1-1 Powe r Supp ly OL-10853-01 shows an illustration of the Cisco SFS 7000P Server Switch, and Cisco SFS 7000P Server Switch Powe r Suppl y Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide...
• One 10/100 Ethernet RJ-45 management-Ethernet port for out-of-band management • One RJ-45 console port used to configure and monitor the Cisco SFS 7000P Server Switch • The Cisco SFS 7000D Server Switch uses the following connectors: Twenty-four dual-speed 20-Gbps 4x copper InfiniBand ports that are capable of operating in either •...
World Wide Web tiers. The Cisco SFS 7000P Server Switch provides 10 Gbps connectivity to servers, while the Cisco SFS 7000D Server Switch provides 20 Gbps DDR or 10 Gbps SDR modes for connectivity to servers.
When an InfiniBand fabric includes more than one Cisco SFS 7000P Server Switch or Cisco SFS 7000D Server Switch, and the subnet manager fails on the Cisco SFS 7000P Server Switch or the Cisco SFS 7000D Server Switch that is acting as the master, another subnet manager takes over within seconds. (See...
A real-time clock maintains correct time regardless of power conditions or connectivity. Latency The Cisco SFS 7000P Server Switch has port to port latency of less than 200ns. The Cisco SFS 7000D Server Switch has port to port latency of less than 140ns...
Page 26
Chapter 1 Product Overview Administrative Features Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01...
Cisco SFS 7000P and SFS 7000D Server Switches. Environmental factors can adversely affect the performance and longevity of your system. Planning a proper location for the switch and layout of your equipment rack or wiring closet is essential for successful system operation. You should install the switch in an enclosed, secure area, ensuring that only qualified personnel have access to the switch and control of the environment.
Multiple switches can be rack-mounted with little or no clearance above and below the chassis. However, when mounting a switch in a rack with other equipment, or when placing it on the floor near other equipment, ensure that the exhaust from other equipment does not blow into the air intake vent of the switch chassis.
Chapter 2 Preparation for Installation Dust and Particles Exhaust fans cool power supplies and system fan trays cool systems by drawing in air and exhausting air out through various openings in the chassis. However, fans also ingest dust and other particles, causing contaminant buildup in the system and increased internal chassis temperature.
Site Requirements If you use twisted-pair cable in your plant wiring with sufficient distribution of grounding conductors, the plant wiring is unlikely to emit radio interference. If you exceed the recommended distances, use a high-quality twisted-pair cable with one ground conductor for each data signal when applicable. If the wires exceed the recommended distances or if wires pass between buildings, carefully consider the effect of a lightning strike in your vicinity.
2.5 inches (6.4 cm) of clearance between the chassis vents and the cabinet wall. • If placing the switch adjacent to a device that exhausts air towards the switch, the horizontal distance between the devices should be a minimum of 6 inches (15.2 cm).
Site Preparation Checklist Table 2-1 lists the site planning activities that you should perform prior to installing the Cisco SFS 7000P and SFS 7000D Server Switches. Completing each activity helps ensure a successful switch installation. Table 2-1 Task No. Planning Activity Space evaluation: •...
Page 33
Installing the Switches This chapter describes how to install and manage the Cisco SFS 7000P and SFS 7000D Server Switches hardware and contains these sections: Safety, page 3-2 • Pre-installation Instructions, page 3-2 • • Rack-Mounting the Switches, page 3-3 Configuring Basic Connectivity, page 3-7 •...
Switches, you need to do the following: • Check the contents of the switch package—For more information, see the parts list in the installation and configuration note that shipped with your switch. Always use an approved ESD-preventative ankle or wrist strap.
Rack-Mounting the Switch To mount the Cisco SFS 7000P and SFS 7000D Server Switches in a rack, follow these steps: Remove the switch, rack-mount brackets, contents of accessory kit, and documentation from the box. Step 1 Place the switch on a secure, clean surface.
Figure 3-1 Step 2 Attach the bracket that has the screw holes to the side of the switch, with the flange facing away from the switch, as shown in Figure 3-2 The standard method is to face the flange toward the front of the switch. However, you can also mount...
Page 37
Chapter 3 Installing the Switches Check the rack for clearance for the switch. The switch can be installed either directly on top of another Step 4 device or be suspended from the rack posts. Attach the remaining two rack brackets to your rack.
Rack-Mounting the Switches Carefully push the switch unit into the rack, as shown in Step 7 If the brackets do not slide easily, the alignment might be off. Pull the switch back toward you, and realign the brackets. Figure 3-5 Maintain at least six inches between the cooling vents and any obstructions.
Attach the rack brackets to their counterparts before inserting the switch into the rack. Step 3 Insert the switch into the rack with the rack bracket attached. You must tilt the switch unit to one side to Step 4 avoid hitting the sides of the rack bracket as they pass around the back rails.
A Cisco SFS 7000P Server Switch user must configure a static IP Address for Ethernet Management. A Cisco SFS 7000D Server Switch user can configure a static IP address and connect to a serial console port only if a DHCP server is not available.
Page 41
Chapter 3 Installing the Switches You can connect the serial console port to a PC serial port for local administrative access to the switch. A terminal emulation software product, such as HyperTerminal, makes communication between your switch and your PC possible during setup and configuration.
Step 3 Connect the power cords to the power connectors on the rear of the switch. The system automatically boots up. You can watch the operation status by using the serial console. For power connector locations for the Cisco SFS 7000P Server Switch, see...
Step 4 47 Hz and 63 Hz. The switch automatically starts and boots up. Use the correct external power source. Attach the switch only to approved power sources, as indicated by the electrical ratings label. If you are unsure of the correct power-source to use, contact your support personnel or your local power company.
Statement 1056 To use InfiniBand to connect to other workstations or switches, you need standard 4x InfiniBand cables. InfiniBand cables can be used to connect any two InfiniBand devices, whether a switch or a host. (Figure 3-11 Switches.) For more information about InfiniBand cables, see the InfiniBand SDR, DDR, and QDR Technology Overview.
Page 45
Chapter 3 Installing the Switches Connect InfiniBand cables from the host to the InfiniBand switch. Step 1 To connect an InfiniBand cable, push the connector into the interface until you hear or feel a click. Figure 3-13 cable with a pull connector.
Managing the System Figure 3-15 Press here To remove a cable with a pull connector, grasp the connector with one hand and push it toward the port, then pull the latch away from the port with your other hand and gently move the connector back and forth and away from the port.
Page 47
Chapter 3 Installing the Switches Command-Line Interface (CLI)—A text-based interface accessible through a direct serial • connection, Telnet over IP, or SSH over IP Chassis Manager (GUI)—A web-based graphical user interface • Element Manager (GUI)—A graphical interface installed on a workstation, accessible over IP •...
Page 48
Chapter 3 Installing the Switches Managing the System Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide 3-16 OL-10853-01...
Both power modules are located at the front of the switch. When facing the front of the switch, the power supply modules are located in the left and right slots of the switch.
Never place your hand inside an empty module bay or anywhere inside the Caution Cisco SFS 7000P and SFS 7000D Server Switches. Unused module bays always should have a switch filler panel over the bay to ensure proper safety, Caution ventilation, and cooling.
Page 51
For more information, see the Figure 4-3 Power Supply LEDs OL-10853-01 Removing the Power Supply from the Switch “Power Supply LEDs” section on page Power Supply LEDs Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide Installing a Power Supply Module...
Never operate the device without a filler panel or power supply module in place because the device might overheat. Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide Removing the Screws from the Power Supply of the Switch “Installing a Power Supply Module” section on Chapter 4...
Page 53
Managing the Switch This chapter describes how to manage the Cisco SFS 7000P and SFS 7000D Server Switches hardware. The following sections appear in this chapter: LEDs, page 5-1 • Managing the System with Element Manager, page 5-4 • •...
Green chassis status LED Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide Front View of Cisco SFS 7000P Server Switch Showing Chassis Status LED Front View of Cisco SFS 7000D Server Switch Showing Chassis Status LED...
• OL-10853-01 Figure 5-6 for the Cisco SFS 7000D Server Switch. InfiniBand Port LEDs Location for the Cisco SFS 7000P Server Switch Port LEDs InfiniBand Port LEDs Location for the Cisco SFS 7000D Server Switch Port LEDs InfiniBand Port LED Indication Logical link is not established.
Power Supply and Fan LEDs Location Power Supply Module LEDs Indication No DC input. AC connected, DC output OK. No failure on the power supply. Operator intervention required. Failure detected within the power supply. Health Status Window Chapter 5 Managing the Switch Figure 5-8.) OL-10853-01...
Chapter 5 Managing the Switch Using the Summary Tab Use the Summary tab to view the status of the power, fans, and temperature sensors simultaneously. (See Figure 5-8.) The Summary tab has the following fields: Power • A green check indicates that at least one power source is connected and functioning properly.
Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide Health Status Sensors Window Table A-1 on page Power-supply Information oper-status utilization voltage Power-supply Seeprom serial-number number 820000 820000 820000 820000 Chapter 5 Managing the Switch A-1. number OL-10853-01...
Chapter 5 Managing the Switch Displaying Fan Information To display information about the fans, enter the show fan command. An oper-status of up means that the fan is operating correctly within the power supply module. In the Speed field, the integer represents a percentage. The percentage changes based on the ambient temperature of the unit and increases as the temperature rises.
Page 60
Chapter 5 Managing the Switch Displaying System Information Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01...
Chassis Self-Test • • Fan Self-Test Power Supply Self-Tests • OL-10853-01 Chapter 5, “Managing the Switch” Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide C H A P T E R for more information on LEDs.
Self-test The example below shows how to run a card self-test. The sample text in the steps below shows command output for the Cisco SFS 7000P Server Switch. For Note the Cisco SFS 7000D Server Switch, you would see a similar display.
• The example below shows how to run a chassis standard test. The sample text in the steps below shows command output for the Cisco SFS 7000P Server Switch. For Note the Cisco SFS 7000D Server Switch, you would see a similar display.
Note The sample text in the steps below shows command output for the Cisco SFS 7000P Server Switch. For the Cisco SFS 7000D Server Switch, you would see a similar display. To perform a diagnostic self-test on a fan, follow these steps: Identify the number of the fan you want to diagnose, if you do not already know the number.
Page 65
Chapter 6 Hardware Diagnostic Tests Figure 6-1 P o w e r S u p p Enter the show diagnostic fan command. SFS-7000P# show diagnostic fan ====================================================================== ====================================================================== module-number : 2 percentage-completed : 100 result-string : Fan Self Test Completed, Final report : Passed=1, Failed=0, Total=1 Enter the diag fan id command.
• Self-Test • The sample text in this section shows command output for the Cisco SFS 7000P Server Switch. For the Note Cisco SFS 7000D Server Switch, you would see a similar display. Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide...
Chapter 6 Hardware Diagnostic Tests Running a LED Test on the Power Supply You can use the LED test to identify a specific power supply by entering these commands: SFS-7000P> enable SFS-7000P# config SFS-7000P(config)# diag power-supply 1 SFS-7000P(config-diag-power-supply-1)# test led SFS-7000P(config-diag-power-supply-1)# start SFS-7000P(config-diag-power-supply-1)# stop SFS-7000P(config-diag-power-supply-1)#...
Proceed with reload? [yes(default) | no] y Displaying Hardware Errors The sample text in the steps below shows command output for the Cisco SFS 7000P Server Switch. For Note the Cisco SFS 7000D Server Switch, you would see a similar display.
Page 69
POST that executes at power-up. SFS-7000P# show diagnostic post ====================================================================== ====================================================================== fru-slot -------------------------------------------------------------------------------- card(1) fan(1) fan(2) power-supply(1) passed Display FRU errors on your switch by entering the show diagnostic fru-error command. Step 2 SFS-7000P# show diagnostic fru-error ====================================================================== ====================================================================== fru-slot -------------------------------------------------------------------------------- card(1) fan(1) fan(2)
Length Weight OL-10853-01 Specification 0 to 45°C (32°F to 113°F)—Cisco SFS 7000P Server Switch 0 to 40°C (32°F to 104°F)—Cisco SFS 7000D Server Switch -40 to 70°C (-40 to 158°F) 20°C (68°F) max. per 60 min 0 to 3,048 m (0 to 10,000 ft)
Specification Auto-ranging 90 to 264 VAC, 47 to 63 Hz Power with external powered Power without external powered devices is 65 Watts—Cisco SFS 7000P Server Switch Power without external powered devices is 80 Watts—Cisco SFS 7000D Server Switch Appendix A...
R E V I E W D R A F T — C I S C O C O N F I D E N T I A L Acronyms and Abbreviations Table B-1 defines the acronyms and abbreviations that are used in this publication. Table B-1 List of Acronyms and Abbreviations Acronym...
Appendix B Acronyms and Abbreviations R E V I E W D R A F T — C I S C O C O N F I D E N T I A L Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01...
Page 75
adapter plug 3-10 administrative features air flow altitude audience, document bad wiring practice baud rate bi-color LED blackout Card Self-Test chassis interface power up 3-10 Chassis Self-Test chassis status LED bi-color single Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01 I N D E X configure connectivity...
Page 76
Index document audience conventions viii obtaining organization related dual-connected, HCAs dust electrical storms electrolytic capacitors electrostatic discharge, damage Element Manager Element Manager GUI empty, module bay emulation software environmental factors grounding strap preventative, ankle strap preventative,wrist strap Ethernet cable 3-14 exhaust fans Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide IN-6...
Page 77
ports hot-swappable, power supply module humidity 2-2, 2-3 HyperTerminal IEC connectors InfiniBand cables 3-12 connectivity devices, connect 3-12 port port LED install power supply module preparation with one person with two people interface chassis management interswitch connectivity latency latency, port to port Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01 list...
Page 78
Self-Test test unit without power supply module power up, chassis 3-10 pull connector 3-13, 3-14 rack-mount, switch radio transmitters real-time clock redundant, power supply remove power supply module safety warnings 2-1, 3-2 scalability 1-2, 1-3 OL-10853-01...
Page 79
Summary tab switch components system information, display temperature Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide OL-10853-01 conditions extreme fluctuations transients undervoltage...
Page 80
Index Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide IN-10 OL-10853-01...