Page 1
IBM Elastic Storage System 3000 Version 6.0.0 Hardware Planning and Installation Guide SC28-3107-01...
Page 2
IBM welcomes your comments; see the topic “How to submit your comments” on page xii. When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.
Site preparation and physical planning....................21 Planning for hardware installation......................21 Planning for racks..........................21 Reviewing Elastic Storage System 3000 location guidelines............. 22 ESS 3000 model........................... 22 Planning for power for server canister....................25 Physical installation planning........................27 Operating environment........................28 ESS 3000 requirements........................28...
Page 4
Installing the EMS server in a rack.......................40 Unpacking Elastic Storage System 3000 (IBM SSR task)..............40 Installing support rails for the ESS 3000 system (IBM SSR task)............41 Installing enclosures (IBM SSR task)....................44 Connecting the components (IBM SSR task)..................46 Connecting an Ethernet cable to SSR access port (IBM SSR task).............47...
2. Rear view of ESS 3000..........................15 3. ESS 3000 Networking..........................17 4. ESS 3000 Sharing Existing Networks......................17 5. ESS 3000 High-level architecture and topology..................18 6. High-level view of system management....................19 7. SSR access port............................20 8. Location of CPUs and DIMM slots within the server canister..............24 9.
Page 8
31. Canister SSR Tools menu options......................55 32. Connecting a laptop to the serial port in the EMS node................59 33. Front and rear view of IBM Elastic Storage System 3000............... 63 34. Rear view of ESS 3000..........................64 35. Management ports for each canister......................64 36.
Page 9
16. Summary of onboard Ethernet ports......................37 17. Summary of supported host interface adapters..................38 18. Selecting bracket pins for your rack......................43 19. ESS 3000 Network preference table......................66 20. Hardware locations of enclosures and other devices................68 21. Enclosure models and rack requirements....................69...
This information is intended as a guide for administering IBM Elastic Storage System (ESS) 3000. Who should read this information This information is intended for administrators of IBM Elastic Storage System (ESS) 3000 systems that include IBM Spectrum Scale RAID. Related information...
In the left margin of the document, vertical lines indicate technical changes to the information. How to submit your comments To contact the IBM Spectrum Scale development organization, send your comments to the following email address: scale@us.ibm.com xii IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead.
International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml.
DANGER: A danger notice indicates the presence of a hazard that has the potential of causing death or serious personal injury. (D002) 2. Locate the IBM Systems Safety Notices with the user publications that were provided with your system hardware.
Page 16
Electrical voltage and current from power, telephone, and communication cables are hazardous. To avoid a shock hazard: • If IBM supplied a power cord(s), connect power to this unit only with the IBM provided power cord. Do not use the IBM provided power cord for any other product.
Page 17
Use the reference numbers in parentheses at the end of each notice (for example, D005) to find the matching translated notice in IBM Systems Safety Notices. CAUTION: The battery contains lithium. To avoid possible explosion, do not burn or charge the battery.
Page 18
• Do not stand under overhanging load. • Do not use on uneven surface, incline or decline (major ramps). • Do not stack loads. (C048, part 1 of 2) 6 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 19
• Do not operate while under the influence of drugs or alcohol. • Do not support ladder against LIFT TOOL. • Tipping hazard. Do not push or lean against load with raised platform. • Do not use as a personnel lifting platform or step. No riders. •...
• Wear safety glasses when you are hammering, drilling, soldering, cutting wire, attaching springs, using solvents, or working in any other conditions that might be hazardous to your eyes. 8 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 21
Important: Good judgment must also be used to identify potential safety hazards due to the attachment of non-IBM features or options that are not covered by this inspection guide. If any unsafe conditions are present, you must determine how serious the apparent hazard might be and whether you can continue without first correcting the problem.
Attention: Depending on local conditions, the sound pressure can exceed 85 dB(A) during service operations. In such cases, wear appropriate hearing protection. Environmental notices This information contains all the required environmental notices for IBM Systems products in English and other languages. The IBM Systems Environmental Notices includes statements on limitations, product information, product recycling and disposal, battery information, flat panel display, refrigeration and water-cooling systems, external power supplies, and safety data sheets.
Electromagnetic compatibility notices The following Class A statements apply to IBM products and their features unless designated as electromagnetic compatibility (EMC) Class B in the feature information. When attaching a monitor to the equipment, you must use the designated monitor cable and any interference suppression devices that are supplied with the monitor.
This statement applies to products greater than 20 A, single phase. This statement applies to products greater than 20 A per phase, three-phase. Japan Voluntary Control Council for Interference (VCCI) Notice 12 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
People's Republic of China Notice Russia Notice Taiwan Notice IBM Taiwan Contact Information: United States Federal Communications Commission (FCC) Notice This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment.
Page 26
Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors, or by unauthorized changes or modifications to this equipment.
Elastic Storage System 3000 can contain up to 24 NVMe-attached SSD drives, 12 drives (half populated) or 24 drives (fully populated). The drives are accessible from the front of ESS 3000, as shown in Figure 1 on page 15.
Page 28
– EC64: PCIe4 x16 2-Port EDR 100GB InfiniBand Adapter – EC67: PCIe4 x16 2-Port 100GbE RoCE Adapter Networking details The networking details of ESS 3000 are shown in the following figure: 16 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 29
I/O Server – gssio1 HMC 1 I/O Server – gssio2 HMC 1 Figure 4. ESS 3000 Sharing Existing Networks NVMe transport protocol in Elastic Storage System 3000 The Elastic Storage System 3000 systems uses the Non-Volatile Memory express (NVMe) drive transport protocol.
Page 30
You need a minimum of one EMS node (5148-21L) as a part of your ESS cluster. The EMS node can be ordered as a part of ESS 3000. When ESS 3000 is added to an existing ESS cluster that has a 5148-21L EMS node, the same EMS node can also be used to manage ESS 3000.
The EMS node also serves as a third GPFS quorum node in a configuration with one building block. System management An EMS node in an ESS 3000 cluster provides system management functions. ESS 3000 GUI runs on the EMS nodes and provides management and health monitoring capabilities. The EMS node also runs a container with Ansible playbook that can provide orchestration of complex tasks, such as cluster configuration, file system creation, and code update.
Hardware specification sheets EMS node specifications The ESS 3000 system uses one EMS node in the basic configuration. The EMS node acts as the management server and provides infrastructure for hardware monitoring and hosts GUI. The EMS server has the following roles: •...
For detailed rack specification information on the 7965-S42 rack, see: Planning for the 7965-S42 rack. If you do not have a 7965-S42 rack and want to install the system into a non-IBM rack, see the ESS 3000 requirements topic. Ensure that the physical environment meet the specified requirements, such as rack space, power, and environmental conditions.
ESS 3000 contains two I/O server canister. Each ESS 3000 cluster consists of the following components: • One or more ESS 3000 systems – each system requires 2U (standard rack units) in a rack • One EMS node - requires 2U space in a rack •...
Page 35
Table 4. Available memory configurations for a server canister Total Memory per ESS 3000 768 GB 1,536 GB ACG1 Features ACG1 +ACGB Slots for CPU1 Content of each DIMM slot 32 GB 32 GB dummy 32 GB 32 GB 32 GB...
Page 36
I/O server canister. The following figure shows a proper orientation for the server canister that an IBM SSR would use to remove or replace a DIMM because it shows the labels in a way they are easily read.
Attention: The power cord is the main power disconnect. Ensure that the socket outlets are located near the equipment and are easily accessible. Figure 11 on page 26 shows the back of a Elastic Storage System 3000. Each PSU is located on the sides of ESS 3000. Chapter 3. Planning for hardware 25...
Page 38
(250V/10A) with C19/C20 connectors Attention: Ensure that sufficient power supply circuits are available to provide the total power requirements of the equipment that is connected to each power supply circuit. 26 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
• Do you have adequate rack space for your hardware? The system requires two Electronic Industries Alliance (EIA) units for each ESS 3000. • Do the power circuits you are planning to use have sufficient capacity and the correct sockets for your installation? Chapter 3.
Supported hosts IBM Spectrum Scale needs to be installed on the host server before it is connected to ESS 3000. IBM Spectrum Scale provides the high-performance scale-out clustering capabilities. For a list of supported host type and OS, see the IBM Spectrum Scale Frequently Asked Questions.
Page 41
If there is less airflow, additional mechanisms are required to cool the enclosure. An appropriate IBM rack configuration would be the 7014-T42 IBM Rack Model T42, with standard rear door and feature code 6069 Front Door For 2.0 Meter Rack (High Perforation).
Page 42
• The combined power and cooling module exhausts air from the rear of each canister. Ensure that your environment falls within the ranges that are listed in the Table 8 on page 31. 30 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 43
850 mm 44.85 kg 49.65 kg with 24 drive slots (3.46 in.) (19.0 in.) (33.5 in.) (98.87 lb) (109.46 lb) The following table shows the rack space requirements for ESS 3000 in tabular form: Chapter 3. Planning for hardware 31...
Page 44
See the Site preparation and physical planning section to help you prepare your physical site for the installation of Elastic Storage System 3000. Additional space requirements Ensure that these additional space requirements, as shown in Table 12 on page 33, are available around the enclosures. 32 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 45
Table 12. Clearances Location Additional space requirements Reason Left and right sides 50 mm (2 in.) Cooling air flow Back Minimum: 100 mm (4 in.) Cable exit Supported drives Table 13 on page 33 provides drive specifications for your Elastic Storage System 3000 system. Table 13.
Page 46
Shock and vibration specifications for Elastic Storage System 3000 enclosures Table 14 on page 35 and Table 15 on page 35 provide the shock and vibration testing results for your Elastic Storage System 3000 system. 34 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
As you plan your installation, you must consider IP address requirements and service access for the system. ESS 3000 uses 100GbE/100Gb EDR network for cluster communication and data transport. For IP port usage requirement, see IBM Spectrum Scale Knowledge Center.
Elastic Storage System 3000 contains two server canisters. A label on ESS 3000 identifies each server canister and power supply unit (PSU). As Figure 14 on page 36 shows, server canister 1 is on the top and server canister 2 is on the bottom.
Page 49
Make the following connections for Elastic Storage System 3000: • Each ESS 3000 system requires two Ethernet cables to connect it to an Ethernet switch. One cable connects to port 1 of the top server canister, and the other cable connects to port 1 of the bottom...
Note: The server canister in the upper slot (slot 1) of ESS 3000 is inverted, so the adapter slots and port numbers are numbered from right to left. The adapters and ports of the server canister in the lower slot (slot 2) are numbered left to right.
Chapter 4. Installing This section covers topics on how to rack, stack, unpack, and install an ESS 3000 system. It also guides the customer on how to fill out the installation worksheet. The SSR can refer to the “Elastic Storage System 3000 installation (SSR task)” on page 51 topic to perform a hardware checkout and set the management interface IP.
Detailed installation steps (SSR task) This section is intended for IBM authorized service personnel only. IBM service support representatives can access service installation information through the links in the following topics. Note: An IBM intranet connection is required. ESS system view Components required to set up EMS For information on the components required to set up EMS, see Before Installation (Sections 1 through 8).
5. Using the box knife, carefully cut the four corners of the carton from top to bottom. 6. Fold the sides and back of the carton down to uncover the rear of the ESS 3000 system. If necessary, carefully cut along the lower fold line of the sides and remove them.
Page 54
Each rail comes with four medium pins pre-installed (two in the front bracket and two in the rear bracket). Large pins are provided separately. Use the pins that are appropriate for the mounting holes in your rack (see Table 18 on page 43). 42 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 55
Table 18. Selecting bracket pins for your rack Mounting holes Bracket pins Round, Use the pre-installed medium pins. unthreaded Square Unscrew the medium pins and replace with the large pins that are supplied with the rails. 4. At each end of the rail, grasp the tab and pull firmly to open the hinge bracket.
Note: The rails are not designed to hold an enclosure that is partially inserted. The enclosure must always be in a fully inserted position. The ESS 3000 system must be installed only on the supplied ESS 3000 system rails. Do not install an ESS 3000 system on expansion enclosure rails.
Page 57
Figure 22. Inserting the enclosure 4. Secure each flange on the front of the enclosure with two black M5 screws. (See Figure 22 on page 45.) Some enclosures are secured by one silver captive screw and one black M5 screw on each flange. In this case, align the enclosure, fasten the two captive screws, and then fasten the two M5 screws.
Connecting the components (IBM SSR task) After installing rails and the ESS 3000 enclosure in the rack, the next step is to connect to the power and network. Connecting Ethernet cables to the server canisters To provide system management connectivity for your system, you must connect the Ethernet cables to Ethernet port 1 of each server canister in the ESS 3000 system.
Before you check the system, you must connect an Ethernet cable to the SSR access port on the AF8 enclosure. 1. Locate the SSR access ports on the ESS 3000 system, as shown in the following figure: Figure 27. SSR access ports...
Using these best practices the customer is aided to select IP addresses for the management interfaces and fill out the “Installation worksheet” on page 63. This information from the customer is required to 48 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 61
For more information on the essmgr.yml file, see the Quick Deployment guide in the IBM Elastic Storage System (ESS) 3000 Knowledge Center. The SSR take the inputs from the worksheet and implements the requested IP addresses Management server best practices (EMS) Ensure that each IP in the worksheet is on the same subnet.
• If the system did not come racked, the SSR would rack the necessary components before the installation starts. • ESS 3000 needs to be placed in the rack location provided by the user. For example, EIA 1-2 and 3-4. – Each additional ESS 3000 unit needs 2U space in the rack.
– Neither interface has a default IP. The EMS management interface (T1) is set up by the SSR. • Ethernet cables (2) run from the 1GbE or 10GbE management switch to the ESS 3000 units (1 per server canister to Ethernet port 1).
Page 64
8) Go to the SSR menu, run the 2. Check and validate various install parameters option, and then the 3. Quick storage configuration check option. Address all the hardware issues that you find. Note: Skip any option that says IO-Node Only. 52 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 65
Figure 28. SSR menu on the EMS node 9) Select the 5. Assign IP address to Interface option to set the management interface IP. Use the c key to change the IP address. A new panel will open which will allow the SSR to change the IP address.
If there are any issues, check Ethernet cable and switch, and verify the laptop IP address. If the issue persists, see the “ESS 3000 issues” on page 61 topic. e. Open putty and set the target address as 10.0.0.102/24 (port 22 for SSH), accept any security prompts.
Figure 30. essutils tool i. Run all steps in the SSR tools menu from option #2 through #7, as shown in the Figure 31 on page Figure 31. Canister SSR Tools menu options If a step fails, go through the command output. j.
Page 68
If there are any issues, check Ethernet cable and switch, and verify the laptop IP address. If the issue persists, see the “ESS 3000 issues” on page 61 topic. e. Open putty and set target address as 10.0.0.101/24 (port 22 for SSH), accept any security prompts.
Page 69
l. To test the ping connection with the EMS node, highlight the 8. Ping tests option. Use the c key to change the IP address, a new panel will open. Set the IP address that was added during the EMS setup phase and also defined on the customer worksheet.
Page 70
58 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 72
If you face issues logging in with username: essserv1 and password: <serial number of server>, perform the following steps: • Log in as a root user: – username: root – password: ibmesscluster 60 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
• Connect a terminal to the canister (monitor to VGA port and keyboard to USB port). • Put power back to the ESS 3000 unit. Note: You will be able to watch the boot process and take appropriate actions when problems arise.
Page 74
62 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Appendix A. Planning worksheets (customer task) Customers are responsible for completing the system planning worksheets. The customer then provides the worksheets to the IBM SSR when they install and configure the system. Installation worksheet This worksheet is intended for customers as part of the TDA process to outline the items required to be implemented by the SSR during setup.
Figure 36. Port mapping on the back of the EMS The SSR connects to C10-T4 (bottom port) on the 1Gb card to access the server and run the essutils tool. 64 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
SSR: General guidance • For each ESS 3000 canister in the flow (see Chapter 4, “Installing,” on page 39), connect your laptop point-to-point to the Service Port as shown in the Figure 34 on page 64. You need to use this port to access each canister, perform hardware checks, and set the management IP address.
Page 78
– Port with a gear symbol (see Figure 34 on page 64) – Service/Technician port. The SSR will connect a laptop to this port when performing the ESS 3000 tasks. You need to move the cable when working on a different canister (two per ESS 3000 unit)
Page 79
48. Customers must fill in the following values so that the SSR is able perform the required networking tasks. If there are more than one ESS 3000 node being installed, you will need to add the corresponding rows. IP address...
Cabling Scenario 1 IBM Elastic Storage System 3000 as a part of the entire ESS setup including ESS GL / GS / GH units with Protocol nodes and the EMS node. 72 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Cabling Scenario 2 IBM Elastic Storage System 3000 as a part of the entire ESS setup with Protocol nodes and the EMS node. Appendix B. Cabling Guide (SSR tasks) 73...
Cabling Scenario 3 High speed networking with IBM Elastic Storage System 3000, Protocol nodes, and the EMS node. 74 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Cabling Scenario 5 Management networking with IBM Elastic Storage System 3000, Protocol nodes, and the EMS node. 76 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.
Page 95
A node with a mounted GPFS file system that is used specifically to run a customer job. ESS 3000 disks are not directly visible from and are not managed by this type of node.
Page 96
IP addresses for interfaces and services. Elastic Storage System (ESS 3000) A high-performance, GPFS NSD solution made up of one or more building blocks that runs on IBM Power Systems servers. The ESS 3000 software runs on ESS 3000 nodes - management server nodes and I/O server nodes.
Page 97
failure group A collection of disks that share common access paths or adapter connection, and could all become unavailable through a single hardware failure. See file encryption key (FEK). file encryption key (FEK) A key used to encrypt sectors of an individual file. See also encryption key. file system The methods and data structures used to control how data is stored and retrieved.
Page 98
The primary communication protocol for relaying datagrams across network boundaries. Its routing function enables internetworking and essentially establishes the Internet. I/O server node An ESS node that is attached to the ESS 3000 storage enclosures. It is the NSD server for the GPFS cluster. See Internet Protocol (IP).
Page 99
(MS) An ESS 3000 node that hosts the ESS 3000 GUI and xCAT and is not connected to storage. It must be part of a GPFS cluster. From a system management perspective, it is the central coordinator of the cluster.
Page 100
(RG) A collection of disks that is set up by IBM Spectrum Scale RAID, in which each disk is connected physically to two servers: a primary server and a backup server.
Page 101
secure shell (SSH) A cryptographic (encrypted) network protocol for initiating text-based shell sessions securely on remote computers. Serial Attached SCSI (SAS) A point-to-point serial protocol that moves data to and from such computer storage devices as hard drives and tape drives. service network ®...
Page 102
90 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 103
Index accessibility features 79 license inquiries 81 air flow 46 location guidelines audience xi enclosure 22 comments xii notices Console environmental 10 user interfaces 28 control enclosure power cords 47 cooling 46 overview drive 19 of information xi documentation on web xi drive patent information 81 overview 19...
Page 104
SSR access port 47 SSR install 51 static-sensitive devices 10 submitting xii trademarks 2, 82 vibration specifications 34 documentation xi resources xi 92 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...
Page 105
Index accessibility features 79 license inquiries 81 air flow 46 location guidelines audience xi enclosure 22 comments xii notices Console environmental 10 user interfaces 28 control enclosure power cords 47 cooling 46 overview drive 19 of information xi documentation on web xi drive patent information 81 overview 19...
Page 106
SSR access port 47 SSR install 51 static-sensitive devices 10 submitting xii trademarks 2, 82 vibration specifications 34 documentation xi resources xi 94 IBM Elastic Storage System 3000: Hardware Planning and Installation Guide...