Hp 3par service processor software rebuild instructions (qr483-96009, december 2012) (9 pages)
Summary of Contents for HP 3PAR StoreServ 7000
Page 1
HP 3PAR StoreServ 7000 and 7000c Storage Troubleshooting Guide Abstract This guide is intended for experienced users and system administrators who troubleshoot HP 3PAR StoreServ 7000 and 7000c Storage systems and have a firm understanding of RAID schemes. HP Part Number: QL226-98171...
Page 2
The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
Contents 1 Identifying Storage System Components............9 Understanding Component Numbering..................9 Drive Enclosures........................9 Nodes, Slots, and Ports......................10 Controller Nodes......................10 Node/Slot/Port Location Formats ...................11 Legend for Node:Slot:Port Location..................11 I/O Modules ........................13 Power Cooling Modules......................14 Power Distribution Units......................15 Service Processor.......................15 2 Understanding LED Indicator Status.............16 Enclosure LEDs........................16 Bezel LEDs........................16 Disk Drive LEDs........................17...
Page 4
cabling..........................37 Format of Possible cabling Exception Messages..............38 cabling Example 1......................38 cabling Suggested Action 1....................38 cabling Example 2......................38 cabling Suggested Action 2....................38 cage..........................39 Format of Possible cage Exception Messages..............39 cage Example 1......................39 cage Suggested Action 1....................39 cage Example 2......................40 cage Suggested Action 2....................40 cage Example 3......................41 cage Suggested Action 3....................41 cage Example 4......................41...
Page 5
ld Suggested Action 1....................48 ld Example 2........................49 ld Suggested Action 2....................49 ld Example 3........................49 ld Suggested Action 3....................49 ld Example 4........................50 ld Suggested Action 4....................50 license..........................50 Format of Possible license Exception Messages..............50 license Example......................50 license Suggested Action....................51 network..........................51 Format of Possible network Exception Messages..............51 network Example 1.......................51 network Suggested Action 1...................51 network Example 2.......................51...
Page 6
Suggested Action......................66 Troubleshooting Storage System Setup..................66 Storage System Setup Wizard Errors..................66 Collecting SmartStart Log Files.....................72 Collecting Service Processor Log Files...................72 Contacting HP Support about System Setup................73 6 Support and Other Resources..............74 Contacting HP........................74 HP 3PAR documentation......................74 Typographic conventions......................75 Documentation feedback......................75...
Page 7
HP 3PAR StoreServ 7400 Controller Nodes.................11 HP 3PAR StoreServ 7400 Node:Slot:Port Legend..............12 Location of Controller Node Ports..................13 I/O Module Numbering for HP M6710 (2U) and HP M6720 (4U) Drive Enclosures....14 PCM Numbering......................14 PCMs in a HP M6710 (2U) and HP M6720 (4U) Drive Enclosures...........15 HP 3PAR Service Processor DL 320e...................15...
The Storage system includes the following types of drive and node enclosures: The HP M6710 Drive Enclosure (2U24) holds up to 24, 2.5 inch small form factor (SFF) Serial Attached SCSI (SAS) disk drives arranged vertically in a single row on the front of the enclosure.
The controller nodes are located at the rear of the node enclosure. The HP 3PAR StoreServ 7200 Storage system contains two nodes numbered 0 and 1. The HP 3PAR StoreServ 7400 Storage system has either two nodes or four nodes. The four-node...
Figure 3 HP 3PAR StoreServ 7400 Controller Nodes Node/Slot/Port Location Formats The HP 3PAR CLI and the SSMC display the controller node FC, iSCSI, and Gigabit Ethernet port locations in the following format: <node>:<slot>:<port>. For example: 1:1:3 = Node 1: Slot 1: Port 3 2:3:1 = Node 2: Slot 3: Port 1 Node: Valid node numbers are 0-3 depending on the number of nodes installed in your system.
Figure 4 HP 3PAR StoreServ 7400 Node:Slot:Port Legend Table 1 describes the default port configurations for the HP 3PAR StoreServ 7000 and 7000c Storage systems. Table 1 Storage System Expansion Cards Expansion cards Nodes 0 and 1 Nodes 2 and 3...
Each node enclosure must have matching PCIe cards. Figure 5 (page 13) and the table that follows show the locations and descriptions of the controller node ports. NOTE: If you are upgrading from a two-node to a four-node configuration, you can have CNAs installed in node 0 and node 1, and FC HBAs installed in node 2 and node 3.
Figure 6 I/O Module Numbering for HP M6710 (2U) and HP M6720 (4U) Drive Enclosures NOTE: The I/O modules are located in slots 0 and 1 of the HP M6710 and M6720 drive enclosures. Power Cooling Modules The PCM is an integrated power supply, battery, and cooling fan. There are two types of PCMs: The 580 W PCM is used in drive enclosures and does not include a battery.
Two power distribution units (PDU) are mounted horizontally at the bottom of the rack. The PDUs are numbered 0 and 1 from bottom to top. The default configuration for the HP Intelligent Series Racks is two PDUs mounted vertically at the bottom of the rack so to provide a front-mounting unit space.
2 Understanding LED Indicator Status Storage system components have LEDs indicating status of the hardware. Use the LED indicators to help diagnose basic hardware problems. This chapter provides tables and illustrations of component LEDs. Enclosure LEDs Bezel LEDs The bezel LEDs are located at the front of the system on each side of the drive enclosure. The bezels have three LED indicators.
Disk Drive LEDs Disk Drive LEDs are located on the front of the disk drives. Disk drives have two LED indicators. Figure 1 1 Location of Disk Drive LEDs Table 4 Description of Disk Drive LEDs Callout LED Appearance Indicates Activity Green On –...
Figure 12 Location of Controller Node PCM LEDs Table 5 Description of Controller Node PCM LEDs Icon Description Appearance Indicates No AC power or PCM fault Flashing Firmware download AC input fail Amber AC present and PCM On / OK Flashing Standby mode PCM OK...
Table 5 Description of Controller Node PCM LEDs (continued) Icon Description Appearance Indicates Present and charged Flashing Charging or disarmed Battery Good Green Drive PCM LEDs The following figure shows the location of drive 580 W PCM LEDs. Table 6 (page 19) for details of PCM LEDs..
HBAs or other ports. Each port includes External Port Activity LEDs, labeled 0 to 3. The I/O module also includes a Power and Fault LED. Figure 14 Location of HP M6710/M6720 I/O Module LEDs Figure 15 I/O Module Power and Fault LEDs...
Table 7 Description of I/O module Power and Fault LEDs (continued) Icon Function Appearance State Indicates Fault Amber Fault Normal operation Flashing Locate command issued External Port Activity LEDs Figure 16 Location of External Port Activity LEDs Function Appearance State Indicates External Port Activity;...
Controller Node and Internal Component LEDs NOTE: Enter the locatenode command to flash the hotplug LED blue. Figure 17 Location of Controller Node LEDs Table 8 Description of Controller Node LEDs Callout Appearance Indicates Status Green Node status Good On – No cluster Quick Flashing –...
Table 9 Description of Ethernet LEDs (continued) Off – No link established or 10 Mb Link Activity Green On – No Link activity Off – No link established Flashing – Link activity FC Port LEDs The controller node has two FC ports. Each FC port has two LEDs. The arrow-head shaped LEDs point to the associated port.
Table 1 1 Description of SAS port LEDs Callout Appearance Indicates DP- 1 Green Off– SAS link is present or not, this LED does not remain lit Flashing–Activity on port DP-2 Green Off–SAS link is present or not, this LED does not remain lit Flashing–Activity on port Interconnect Port LEDs The controller node has two interconnect ports.
Figure 22 Location of Fibre Channel Adapter Port LEDs Table 13 Description of Fibre Channel Adapter Port LEDs Callout Appearance Indicates All ports No light Off – Wake up failure (dead device) or power is not applied Port speed Amber Off –...
16 G Fibre Channel (FC) Adapter LEDs Figure 24 16 G FC Adapter LEDs Table 15 16 G FC Adapter LEDs 1 – Green LED 2 – Yellow LED State Boot failure (dead board) POST failure (dead board) Slow blink Boot failure after POST Fast blink Not defined...
10 G Ethernet Adapter LEDs Figure 25 10 G Ethernet Adapter LEDs Table 16 10 G Ethernet Adapter LEDs State 1 – Amb=10G Link speed 10Gb/s Link speed 1Gb/s 2 – ACT/Port Link up Blinking Link activity 1 G Ethernet Adapter LEDs Figure 26 1 G Ethernet Adapter LEDs Table 17 1 G Ethernet Adapter LEDs State...
Service Processor LEDs The HP 3PAR SP (Proliant DL320e) LEDs are located at the front and rear of the SP. Figure 27 Front Panel LEDs Table 18 Front panel LEDs Item Appearance Description UID LED/button Blue Active Flashing Blue System is being managed remotely...
Power supply Green Normal NOTE: May not be applicable to Off = one or more of the following your system (for hot-plug HP CS conditions: power supplies ONLY) Power is unavailable Power supply has failed Power supply is in standby mode...
Before you power off, use either SPmaint or SPOCC to shut down the system (see Service Processor Onsite Customer Care in the HP 3PAR StoreServ 7000 and 7000c Storage Service Guide). The system must be shut down before powering off by using any of the following three methods: Using SPOCC Select InServ Product Maintenance.
Page 31
NOTE: To avoid any cabling errors, all drive enclosures must have at least one or more hard drive(s) installed before powering on the enclosure. Power on the node enclosure PCMs. Verify the status of the LEDs. See “Understanding LED Indicator Status” (page 16).
(cage:magazine:disk). The first and second numbers are sufficient to identify the exact disk in an HP 3PAR StoreServ 7000 and 7000c Storage system, since there is always only a single disk (disk 0) in a single magazine.
Page 33
On the suggested actions page, scroll through the list to find the message state listed in the alert message. The recommended action is listed next to the message state. Getting Recommended Actions...
5 Troubleshooting Using the checkhealth Command The HP 3PAR CLI checkhealth command checks and displays the status of storage system hardware and software components. For example, the checkhealth command can check for unresolved system alerts, display issues with hardware components, or display information about virtual volumes that are not optimal.
Component -----------Summary Description----------- Qty Alert New alerts Date Date is not the same on all nodes LDs not mapped to a volume License Golden License. vlun Hosts not connected to a port ------------------------------------------------------- 5 total The following information is included when you use the -detail option: Component ----Identifier---- -----------Detailed Description------- Alert sw_port:1:3:1...
-p -devtype NL -nodes 0,1, and showcage. In the example below, there are 72 NL PDs attached to nodes 2&3, but none attached to nodes 0&1. Contact HP support to request support for moving NL PDs (and possibly cages) from nodes 2&3 to nodes 0&1 for your system.
For cables that should be moved between different I/O modules and/or cages, contact HP support to request support for changing the cabling of your system. See “Contacting HP Support about System Setup”...
dar (Data Encryption at Rest) Checks for issues with Data Encryption. If the system is not licensed for Data Encryption. Format of Possible dar Exception Messages DAR -- "DAR Encryption status is unavailable" DAR -- "DAR Encryption is enabled but not licensed" DAR -- "DAR Encryption key needs backup"...
Checks for behavior altering files on the node disk drives Checks if an online upgrade is in progress Many issues reported by the file component will require you to contact HP support to request support your system. See “Contacting HP Support about System Setup” (page 73).
Format of Possible file Exception Messages File <nodeID> "Filesystem <filesystem> mounted on "<mounted on>" is over <count>% full" File <nodeID> "Behavior altering file "<filename>" exists, created on <date&time>" File <nodeID> "Dump or HBA core files found" File <nodeID> "sr_mnt is full" File -- "sr_mnt not mounted"...
CLI upgradesys -finish command to complete the upgrade after all nodes have been rebooted to the new version of software. Be very careful with aborting or reverting an offline upgrade. Contact HP support to request support for aborting or reverting the upgrade of your system. See “Contacting HP Support about System Setup”...
the FPG if no longer wanted. For other File Services issues, use the CLI showfs, showvfs, showvfs commands (and the associated set and stop commands) to investigate and solve issues. host Displays Adaptive Optimization issues Checks that all PD classes that exist on any node-pair are found on all node-pairs Checks that FC Fabric connected host ports are configured to support Persistent Ports Checks that the FC switch ports are configured for NPIV support Checks that FC Fabric connected host partner ports are found on same FC SAN...
multiple nodes, use the CLI showport command to determine if a port is offline or misconfigured. A missing or rebooting node will cause a port to be offline. cli% showvlun -v testvv Active VLUNs Lun VVName HostName -Host_WWN/iSCSI_Name- Port Type Status ID 2 testvv testhost 10000000C9E5E0B9 3:1:1 host active cli% showhost testhost...
LDs are normally mapped to (used by) VVs but they can be disassociated with a VV if a VV is deleted without the underlying LDs being deleted, or by an aborted tune operation. Normally, you would remove the unmapped LD to return its chunklets to the free pool. cli% showld vv.9.usr.3 Id Name RAID -Detailed_State- Own...
manually moved with movech or by specifying it during a tune operation or during failure conditions such as node, path, or cage failures. The HA levels from highest to lowest are port, cage, mag, and ch (disk). Examine the identified LDs using CLI commands such as showld, showld d, showldch, and showpd for any failed or missing drives.
license Suggested Action If desired, request a new or updated license from your Sales Engineer. network Displays Ethernet issues for the Administrative, File Services, and Remote Copy over IP (RCIP) networks that have been logged in the previous 24-hour sampling window. Reports if the storage system has fewer than two nodes with working admin Ethernet connections.
Node node:<nodeID> "VV <vvid> has <stuck_num> outstanding <command> with a maximum wait time of <slptime>" Node node:<nodeID> "<item> is not the same on all nodes" (where <item> is: BIOS version, NEMOE version, Control memory, Data memory, CPU Speed, CPU Bus Speed, HP 3PAR OS version, Package list)
Node node:1 Power supply 0 detailed state is AC Failed Node node:1 Power supply 0 AC state is Failed Node node:1 Power supply 0 DC state is Failed node Suggested Action 1 Examine the states of the power supplies with commands such as shownode, shownode -s, shownode -ps, and the like.
Displays Physical Drives (PDs) with states or conditions that are not optimal: Checks for failed and degraded PDs Checks for an imbalance of PD ports, for example, if Port-A is used on more drives than Port-B Checks for an Unknown Sparing Algorithm. For example, when it hasn't been set Checks for drives experiencing a high number of IOPS Reports if a servicemag operation is outstanding (servicemag status) Reports if there are PDs that do not have entries in the firmware DB file...
pd Example 4 Component --Identifier-- -------Detailed Description---------- disk:3 Detailed State: old_firmware pd Suggested Action 4 The identified drive does not have firmware that the storage system considers current. When a drive is replaced, the servicemag operation should upgrade the drive's firmware. When drives are installed or added to a system, the admithw command can perform the firmware upgrade.
Use the CLI showld, showpd, and shownode commands. If the ownership issue is not created due to a node missing or failed PD, Contact HP support to request support for moving the LDs to the desired location for your system. See “Contacting HP Support about System Setup”...
due to a service action, such as hardware or software upgrades, these exceptions can be ignored until that action has finished and the node is online. In this example, LD 35, with a name of R1.usr.3, is owned (Own) by nodes 3/2/0/1, respectively, and the primary/secondary physical paths to the drives (chunklets) in this LD are from nodes 3 and 2, respectively.
Check SFP statistics using CLI commands such as showport -sfp, showport -sfp -ddm. cli% showport -sfp 0:1:1 N:S:P -State-- -Manufacturer- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM 0:1:1 OK HP-F 8.5 No -------------------------------------------------------------------------- In the following example an RX power level of 522 microwatts (uW) for Port 0:1:1 DDM is a good reading;...
-sfp command. In this example, the SFP in 1:1:1 has been removed from the adapter: cli% showport -sfp N:S:P -State- -Manufacturer- MaxSpeed(Gbps) TXDisable TXFault RXLoss DDM 0:1:1 OK HP-F 8.5 No 0:1:2 OK HP-F 8.5 No...
port Suggested Action 4 This output indicates that the port's mode, such as an initiator or target, is not correct for the connection type, such as drive, host, ISCSI, FCoE, or RCFC. Useful CLI commands are showport, showport -c, showport -par, showport -rcfc, showcage, etc. cli% showport N:S:P Mode...
Remote Copy internal structure Too many Remote Copy targets configured as sync Format of Possible rc Exception Messages RC rc:<name> "All links for target <name> are down but target not yet marked failed." RC rc:<name> "Target <name> has failed." RC rc:<name> "Link <name>...
Format of Possible sp Exception Messages Network SP->InServ "SP ethernet Stat <stat> has increased too quickly check SP network settings" sp Example Component -Identifier- --------Detailed Description------------------------ ethernet "State rx_errs has increased too quickly check SP network settings" sp Suggested Action The <stat>...
Detailed status is as follows: 2010-10-22 10:35:36 PDT Created task. 2010-10-22 10:35:36 PDT Updated Executing "upgradecage -a -f" as 0:12109 2010-10-22 10:35:36 PDT Errored upgradecage: Invalid option: -f vlun Displays inactive Virtual LUNs (VLUNs) and those which have not been reported by the host agent Reports VLUNs that have been configured but are not currently being exported to hosts or host-ports...
72). Collect the SP log files. See “Collecting Service Processor Log Files” (page 72). Contact HP support and request support for your HP 3PAR StoreServ 7000 or 7000c Storage system. See “Contacting HP Support about System Setup” (page 73). Storage System Setup Wizard Errors This section describes possible error messages that may display while using the Storage System Setup Wizard.
Page 67
Could not communicate with the storage system. Make sure it is running and connected to the network. This message can display if the HP 3PAR OS loses network connectivity, either by becoming unplugged or by going down for some other reason.
Page 68
Major.Minor.Patch TPD package as the storage system’s HP 3PAR OS. If the package is not the same, then the SP cannot communicate with the HP 3PAR OS. {0} will be the version of the TPD package that the user must install so that the SP will work with the storage system.
Page 69
{0} will be the name of the cage with the problem. Although the firmware upgrade may have succeeded, this error might occur if the cage does not boot back up. Contact HP Support. For information about contacting HP Support, see “Contacting HP Support about System Setup”...
Page 70
The storage system found an error while checking cage health. There is a problem with a drive cage that has had a firmware upgrade. Cage {0} did not come back after the firmware upgrade. Contact HP support for help. This error message displays in a dialog box with Retry and Cancel buttons. This error might occur after the drive cages have had a firmware upgrade.
Page 71
This message displays in a dialog box. The error occurs if the storage system detects that the defined IPv4 gateway address could not be reached. Click Back and specify a valid IPv4 gateway address. If the error persists, contact HP Support. For information about contacting HP Support, see “Contacting HP Support about System Setup”...
Click Back and specify a valid time zone. Collecting SmartStart Log Files To collect the SmartStart log files for HP support, zip all the files in this folder: C:\Users\ <username>\SmartStart\log. NOTE: You can continue to access the SmartStart log files in the Users folder after you have removed SmartStart from your system.
Technical support registration number (if applicable) Product serial numbers Error messages Operating system type and revision level Detailed questions When contacting HP, specify that you are requesting support for your HP 3PAR StoreServ 7000 or 7000c Storage system. Troubleshooting Storage System Setup...
6 Support and Other Resources Contacting HP For worldwide technical support information, see the HP support website: http://www.hp.com/support Before contacting HP, collect the following information: Product model names and numbers Technical support registration number (if applicable) Product serial numbers Error messages...
HP. Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (docsfeedback@hp.com). Include the document title and part number, version number, or the URL when submitting your feedback.