Hardware

• 1: Boards: Raspberry PI
• 2: Pi Clusters on the Internet
• 3: Cases: Pi Cluster Cases
• 4: Parts
• 4.1: Individual Parts
• 4.2: Parts for a 3 node cluster
• 4.3: Parts for a 8 node cluster

1 - Boards: Raspberry PI

Raspberry PI’s are a convenient cheap compute platform that allow us to explore create cloud clusters with various software that otherwise would not be accessible to most. The point is not to create a complex compute platform, but to create a testbed in which we can explore configuration aspects and prepare benchmarks that are run on larger and expensive cloud environments. In addition Raspberry Pis can be used as a simple Linux terminal to log into other machines.

We will give a small introduction to the platform next.

Raspberry PI 4B

Figure 1: Pi 4B board

The board has the following properties ¹:

• Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
• 2GB, 4GB or 8GB LPDDR4-3200 SDRAM (depending on model)
• 2.4 GHz and 5.0 GHz IEEE 802.11ac wireless, Bluetooth 5.0, BLE
• Gigabit Ethernet
• 2 USB 3.0 ports; 2 USB 2.0 ports.
• Raspberry Pi standard 40 pin GPIO header (fully backwards compatible with previous boards)
• 2 × micro-HDMI ports (up to 4kp60 supported)
• 2-lane MIPI DSI display port
• 2-lane MIPI CSI camera port
• 4-pole stereo audio and composite video port
• H.265 (4kp60 decode), H264 (1080p60 decode, 1080p30 encode)
• OpenGL ES 3.0 graphics
• Micro-SD card slot for loading operating system and data storage
• 5V DC via USB-C connector (minimum 3A*)
• 5V DC via GPIO header (minimum 3A*)
• Power over Ethernet (PoE) enabled (requires separate PoE HAT)
• Operating temperature: 0 – 50 degrees C ambient

Important for the cluster is the following comment:

• A good quality 2.5A power supply can be used if downstream USB peripherals consume less than 500mA in total.

Raspberry PI 3 B+

We plan to purchase a number of them so we can conduct performance experiments and leverage the faster hardware. The newest Raspberry PI 3 B+ is shown in Figure 2.

Figure 2: Raspberry PI 3B+ board

The board has the following properties:

• Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz
• 1GB LPDDR2 SDRAM
• 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac wireless LAN
• Bluetooth 4.2, BLE
• Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps)
• Extended 40-pin GPIO header
• Full-size HDMI
• 4 USB 2.0 ports
• CSI camera port for connecting a Raspberry Pi camera
• DSI display port for connecting a Raspberry Pi touchscreen display
• 4-pole stereo output and composite video port
• Micro SD port for loading your operating system and storing data
• 5V/2.5A DC power input
• Power-over-Ethernet (PoE) support (requires separate PoE HAT)

Raspberry PI 3 B

Till February 2018 the Raspberry PI 3 B was the newest model. Within this class we have access to about 100 of them. The Raspberry PI 3 B is shown in Figure 3.

Figure 3: Raspberry PI 3 B board

The board has the following properties:

• Quad Core 1.2GHz Broadcom BCM2837 64bit CPU
• 1GB RAM
• BCM43438 wireless LAN and Bluetooth Low Energy (BLE) on board
• 40-pin extended GPIO
• 4 USB 2 ports
• 4 Pole stereo output and composite video port
• Full size HDMI
• CSI camera port for connecting a Raspberry Pi camera
• DSI display port for connecting a Raspberry Pi touchscreen display
• Micro SD port for loading your operating system and storing data
• Switched Micro USB power source up to 2.5A

Raspberry PI Zero

In addition to the PI 3’s another interesting platform is the PI Zero, which is a very low-cost system that can serve as IoT board. However, it is also powerful enough to run more sophisticated applications on it. The newest Raspberry PI Zero is shown in Figure 4.

Figure 4: Raspbery Pi Zero ²

The board has the following properties:

• 1GHz single-core CPU
• 512MB RAM
• Mini HDMI port
• Micro USB OTG port
• Micro USB power
• HAT-compatible 40-pin header
• Composite video and reset headers
• CSI camera connector (v1.3 only)

Pin Layout

The PI 4B, 3B+, 3B, and Zero come with a number of pins that can be used to attach sensors. It is convenient to have the pinout available for your project. Hence, we provide a pinout layout in Figure 5. Other Pis will have a different pinout and you will have to locate them on the internet.

Figure 5. Pi pinout

Mounting diagram

In case you need to drill wholes in cases or plexiglas for mounting, the following mounting diagram is very useful:

• Mounting Diagram in PDF

Resources

Detailed information about it are available at

• https://www.raspberrypi.org/documentation/hardware/raspberrypi/README.md

2 - Pi Clusters on the Internet

There is a large number of projects related to creating Pi clusters on the internet. They vary in size and software installed on them. Naturally a Pi cluster is a useful training and development environment for research organizations and thus many bigger projects are located at universities as well as government labs. However, we also have many projects done by enthusiasts.

We distinguish two different efforts. First, we often find projects that target the creation of cases for such clusters and second, we find projects that develop software fort these clusters. This section will provide an overview of these activities and provide links to these activities.

Cluster Cases

Lego

When looking at the placement of the wholes on the Raspberry Pi, the width between the wholes on the small side seems to be exactly 7 Lego Technic beam wholes apart. This has the advantage that one could build a quick frame form Lego pieces such as a

• 2 * 11 ($0.22) piece [link]
• 2 * two piece * 4 * 4 ($0.192) [link]
• A number of connector pins [link] [link]

The cost is about $0.25 per piece = $2.74 per pi.

So if we are having 100 pis we end up with $274. However we also need still to get screw and Lego connectors which we at this time have not counted and included in this calculation.

Naturally Lego’s have been explored by others

• Images on google
• University of Southampton: [link] Instructions are no longer at the original link
• Lego Technic: [link], [link]
• Lego Technic [link]
• Old style Lego [link]

Other ideas using Lego’s include:

• Compact case: Single board No screws [link]
• Zero 3D print Thingverse [link]
• B+ 3D Pi case [link]

Other interesting but not cluster related links include

• Lego hat [link]
• Brick Pi [link]

Beast by resion.io

This company has provided some larger designs for Raspberry Pi clusters and tries to create a modular system to put a number of Pis on plates that than can be connected.

• Beast 3 [link]
• Beast 2 [link]
• Beast 2 Tile [link]

Clusters

Additional links which could be useful include:

• SDSC Raspberry Pi2 MPI and Tiled Wall Viz Cluster [link]
• Six Common Errors When Building a Raspberry Pi Cluster [link]
• 5 Most Popular Raspberry Pi Cluster Supercomputer Projects [link]
• 10 amazing Raspberry Pi clusters [link]
• Build an OctaPi [link]

An older document on how to create an MPI cluster is located at

• Pi 2 MPI cluster, Boise State [link]

Swarm

How to set up docker swarm is documented here

• Pi Docker Cluster [link]

3 - Cases: Pi Cluster Cases

NAS (1 Pi)

Although a NAS is not really a compute cluster the Pi has used many times to build a Network Attached Storage (NAS) server. In this configuration a HDD is attached to the Raspberry and the network features of the Raspberry is used to access the disk drive via software installed on the PI that make this easily possible. Many tutorials exists on the Web that help setting op such a device.

We like to hear from you if you have successfully developed such a NAS and provide us with such links. Links that may help include:

• https://hackmypi.com/NASpi.php

ClusterHat (4 Zero + 1 Pi)

The smallest cluster we came across is actually a hybrid cluster in which 4 Pi zeros attached to a Raspberry Pi 3. This sis achieved via an add on board to the Pi 3 allowing to plug in PI=i Zeros:

• https://clusterhat.com/

The Cluster HAT (Hardware Attached on Top) allows to attach 4 Raspberry Pi Zeros via to be attached to a regular Raspberry PI 3 to simulate a small cluster.

According to the Web Site it supports the following features:

• USB Gadget Mode: Ethernet and Serial Console.
• Onboard 4 port USB 2.0 hub.
• Raspberry Pi Zeros powered via Controller Pi GPIO (USB optional).
• Individual Raspberry Pi Zero power controlled via the Controller Pi GPIO (I2C).
• Connector for Controller Serial Console (FTDI Basic).
• Controller Pi can be rebooted without interrupting power to Pi Zeros (network recovers on boot).

Although this setup seems rather appealing, the issue is with obtaining Pi Zeros for the regional price of $5. Typically users can only by one for that price and must pay shipping. To by more one has to buy a kit for about $20. However, for that amount of money it may just be worth while to get Pi 3’s instead of zero’s. Nevertheless the form factor is rather appealing.

Additional information can be found at:

• https://www.raspberrypi.org/magpi/clusterhat-review-cluster-hat-kit/

Cluster Case With Cooling (5 Pi’s)

Many instructions on the Web exist describing how to build clusters with 3 or more Pi’s. One of the considerations that we have to think about is that we may run rather demanding applications on such clusters causing heat issues. To eliminate them we must provide proper cooling. In some cluster projects cooling is not adequately addressed. Hence we like to provide an example that discusses in detail how to add a fan and what the fan has for an impact on the temperature.

• http://climbers.net/sbc/add-fan-raspberry-pi/
• http://climbers.net/sbc/diy-raspberry-pi-3-cluster/

From the previous Web page we find the following information as shown in the following table. From the data in the table it is clear that we need to keep the Pi from throttling while being in a case by adding a fan as obvious from experiment No. 2.

Table: Temperature comparison of fan impact

No. Case Fan Direction RPM Idle 100% Load Performance

1 no no - - 41.0C 75.5C OK (barely) 2 yes no - - 45.0C 82.5C throttles 3 yes 5V in unknown 37.9C 74.5C OK (barely) 4 yes 7V in 800 35.6C 69.5C OK 5 yes 12V in 1400 32.5C 61.1C OK 6 yes 7V out 800 34.5C 66.4C OK

Interesting is also the design of the case that uses snaps instead of screws to affix the walls to each other. The case layout can be found at:

• http://climbers.net/sbc/diy-raspberry-pi-3-cluster-2017/

Octapi (8 Pi’s)

A set of instructions on building an is available at

• https://projects.raspberrypi.org/en/projects/build-an-octapi

Bitscope Case (40 Pi’s)

A company from Australia called BitScope Designs offers a number of cases that leverage their Pi Blade boards allowing up to four Pis to be put together and sharing the same power supply. The blades are shown in the next Figure. The rack to place 10 of them is shown in the Figure after that.

The cost of the blade rack is $ 795.45 + $60.00 shipping + import tax. This may originally sound expensive when compared to a single case, however as we can store 40 Pis in them and they can share the power-supply and reduce cabling we think this case is quite interesting overall due to its price-point of $20 per Pi.

BitScope Cluster (144 Pi’s)

• https://www.youtube.com/watch?v=78H-4KqVvrg

Together with LANL a new cluster module that holds 144 Pis is developed. This system is targeted to be placed into a rack to create a large Pi cluster. The cost for such a module is about $15K.

The next Figure shows the module.

The next figure shows how multiple modules can be placed into a single rack.

Additional information about this form factor can be found at the following links:

• https://cluster.bitscope.com/solutions
• https://www.pcper.com/news/General-Tech/BitScope-Unveils-Raspberry-Pi-Cluster-2880-CPU-Cores-LANL-HPC-RD
• http://my.bitscope.com/store/
• http://my.bitscope.com/store/?p=view\&i=item+7
• http://www.newark.com/bitscope/bb04b/quattro-pi-board-raspberry-pi/dp/95Y0643
• http://linuxgizmos.com/rpi-expansion-boards-support-up-to-40-pi-clusters/

Oracle Cluster (1060 Pi’s)

Oracle has displayed at Oracle World 2019 a 1060 node Raspberry Pi Cluster.

More images are available at this (link)[https://imgur.com/gallery/wx1hZ5D]. the supercomputer features scores of racks with 21 Raspberry Pi 3 B+ boards each. The system is used to demonstrate Oracle Autonomous Linux source.

Build Your Own 5 Node Pi Cluster

To experiment with building an elementary cluster one does not need to have a big budget. Such clusters are often dedicated to research tasks and are bound into security protocols that do not allow direct access. Instead it is possible to build such a cluster based on Raspberry Pi’s yourself if you are willing to spend the money or if you have access to Pi’s that you may loan from your department.

Table Parts lists one such possible parts list that will allow you to build a cluster for up to 5 nodes. However make sure to buy at least 3 Raspberry Pi’s with the appropriate memory. At minimum we recommend you get the 32GB SD card. We do not recommend any smaller as otherwise you will run out of memory. Additionally, you can add memory and disks on the USB ports. If you attach a HDD, make sure it has an external power supply and do not drive it from the USB power as otherwise the PI becomes unstable. A fan is at this time not yet included.

Naturally it is possible to modify the parts list and adapt. If you find better parts let us know. We have not included any case and you are welcome to share your suggestions with the class. For a case we are looking also for a good solution for a fan.

We suggest that when you build the cluster to do it on a table with a large white paper or board, or a tablecloth and take pictures of the various stages of the build so we can include it in this document.

Initially we just put Raspbian as Operating system on the SD cards and test out each PI. To do so you will naturally need an SD card writer that you can hook up to your computer if it does not have one. As you will have to potentially do this more than once it is not recommended to buy an SD card with the OS on it. Buy the SD card writer instead so you can redo the flashing of the card when needed. In addition to the SD card you need a USB mouse and keyboard and a monitor or TV with HDMI port.

Locate setup instructions and write a section in markdown that we will include here once it is finished. The section is to be managed on github.

¹ items were replaced with similar ² item was not available

Assembling the Pi Cluster

Figure	Description
	First, aluminum and copper heat syncs need to be attached to each Pi. The two aluminum heat syncs are attached to the Broadcom chip and the SMSC Ethernet controller located on the top of the Pi. The blades of the heat syncs are parallel to the longer side of the Pi as shown in black aluminum fanned heat syncs are attached to the top of the pi as shown.
	Flat copper heat sync is attached to the bottom of the pi as shown.
	After attaching the heat syncs, threaded hexagonal spacer supports are used to connect the Pis together. A fully-assembled 5-node Pi cluster is shown.
	Each node of the cluster is then attached to the switch using an Ethernet cables and to the power supply using a USB cables. The fully wired cluster is shown.

Virtual Raspberry Cluster

It is possible to create a virtual raspberry PI cluster while for example using virtual box. This requires two steps. First the deployment of a virtualized Raspberry PI. There are many resources on the internet that describe how to do that. Some of them are

• https://raspberrytips.com/run-raspberry-in-virtual-machine/
• http://dbakevlar.com/2015/08/emulating-a-raspberry-pi-on-virtualbox/

The next step includes the deployment of multiple VMs emulating Raspberry’s. Naturally each should have its own name so you can distinguish them. Instead of just using the GUI, it would be important to find out how to start them from a command line as a shell script as well as tear them down.

Next you will need to make sure you can communicate from the Pi’s to each other. This is naturally the same as on a real cluster

Note: We recently set up such a cluster and found it was extremely slow on an older MacBook Pro with 16GB memory. Thus make sure you have up to date hardware and lots of memory. For this reason we recommend doing a hradware version. Also remember that some software does require a lot of memory to be allocated to the VM. Make sure that you plan ahead before you start the virtualization route. It will be frustrating to work on a real slow virtual cluster.

4 - Parts

The parts are collected in a YAML file located at:

You can add parts you used by doing a pull request. Please also send us a picture of your cluster.

4.1 - Individual Parts

Parts

This section will have the list of tools and parts that we recommend you get to assemble a PI cluster

To add parts please visit the yaml file and add them to the parts list

Prices are in dollar as found at the time by an online retailers. We list only the cheapest prices found. Price updates may be listed in comments. We may round to the next largest dollar amount.

	vendor	description	included	price	comment	image
0	Cana Kit	Raspberry Pi 4 8GBSKU: PI4-8GB	1	75
1	Amazon	Heat Sinks	12	12.99
2	Amazon	SD Card 64GB EVO Select	1	9.99
3	Amazon	SanDisk 32GB Ultra microSDHC	1	10.99	Only buy those for PI3B+, for 4 use the cheaper afaster and bigger 64GB cards
4	Amazon	Duo SD Card Reader	1	7.99
5	Amazon	Compatible with Raspberry Pi 4 Power Supply with ON/Off Switch	1	9.99
6	Amazon	Compatible with Raspberry Pi 4 Power Supply with ON/Off Switch	1	9.99	cable is build in the utronics seems betetr as we can take it out
7	Amazon	USB Charger 10-Port 120W	1	39.99	Warning when using the power switsh listed in the table, you must always switch it on and off from the build in switch. Never put the power supply on a power strip that has its own switch and use that switch to switch on and off via the power supply and leave the switch on the supply just open. THis coudl destroy your supply.
8	Amazon	Anker 60W 6-Port USB Wall Charger	1	27.99	pay attention to get as much W as possible
9	Amazon	60 Ports USB Charging Station	1	119.99
10	Amazon	Cable with switch	2	8.89	has 2 cables included and as the 3 cable option is not available making tis a good option
11	Amazon	Power USB Switch Type-C Cable	1	5.89	expensive
12	Amazon	Cable with switch	1	9.99	3 pack, unavailable
13	Amazon	4 Pack Cable with switch	4	12	these are from USB-C to USB-C to go to a USB charger you stil need a cable or converter to USB-A
14	Amazon	Individual power switches	6	19.99	Please be casefull using them as they may lead to damage to PI power supplies in case of power spickes.
15	Amazon	50 Pcs M2 X 20mm + 5mm Hex Hexagonal Threaded Spacer	50	11.49	fit very good, a bit thin. This is the cheapest solution
16	Amazon	Spacers	50	11.85	slightoy too big screw portion, will work, but buy oothers next time
17	Amazon	HONJIE M2.5 x 11mm	50	7.56	2 of them stacked are needed between two Pis
18	Amazon	Standoffs	10	5.49
19	Amazon	CanaKit Raspberry Pi 4 Micro HDMI Cable - 6 Feet (Pack of 2)	2	9.99
20	Amazon	TP-Link 8 Port Gigabit Ethernet Network Switch	1	19.99
21	Amazon	Ethernet cable	6	6.64
22	Amazon	Case	1	59.39	The price of this case has significantly increase and is now about $139. Find alternatives. This case is used to custom fit multiple Pis. Dependent on How many PIs you fit in, buy 90 degree power cables. And get 2 feet versions or longer. To take the case shelf out you need to drill out the rivets ifrom the shelf. This case has a swing door for easy access. You wll need a USB to 12v converter to use the fans or a separate power supply for it. It may be easier to buy the $50 case without fans and buy 120mm 5V fans that you can directly attach to the USB. This case is big.
23	Amazon	Fan cable	1	10.99
24	Amazon	Case 8 layers	1	40	see commend in the 8 layer case
25	Amazon	Case 8 layers	1	64.99	not recommend due to to not able to use the faster SD Cards, works well for PI3B+. For PI4 and 64GB card the voltage of the SDCard needs to be increased, but that means its slower
26	Amazon	Case 8 layers	1	90	see comment in 8 layer case
27	Amazon	Case 8 layers	1	12	Heat Sinks and fans included
28	Amazon	Cloudlet Case, 7 Pi	1	70
29	Amazon	Utronics 1U Rack mount, 4 Pi’s	1	40
30	Amazon	Utronics 1U Rack mount with HDMI redirection, 4 Pi’s	1	60
31	Amazon	UCTRONICS 19 inch 3U Rack Mount	1	125	Comes only 8 trays that costs $100 and will have an open gap if you do not buy an extra 4 trays for $25. The frame can hold 12 trays
32	Amazon	MyElectronics 3U Rackmount, 12 Pi’s	1	139	see also UTRONICS

4.2 - Parts for a 3 node cluster

Parts

This section will have the list of tools and parts that we recommend you get to assemble a PI cluster

Note: If you at a later time like to update to more than 6 nodes, you need a different power supply or buy a second one of these. See our 8 node cluster list for an alternative.

To add parts please visit the yaml file and add them to the parts list

Prices are in dollar as found at the time on online retailers.

https://github.com/cloudmesh/pi/blob/main/parts/README-parts-list.yml

	vendor	description	included	price	count	total	comment	image
0	Cana Kit	Raspberry Pi 4 8GBSKU: PI4-8GB	1	75.0	3	225.0
1	Amazon	Heat Sinks	1	12.99	1	12.99
2	Amazon	SD Card 64GB EVO Select	1	9.99	3	29.97
3	Amazon	Duo SD Card Reader	1	7.99	1	7.99
4	Amazon	Compatible with Raspberry Pi 4 Power Supply with ON/Off Switch	1	9.99	3	29.97
5	Amazon	50 Pcs M2 X 20mm + 5mm Hex Hexagonal Threaded Spacer	50	11.49	1	11.49	fit very good, a bit thin
6	Amazon	CanaKit Raspberry Pi 4 Micro HDMI Cable - 6 Feet (Pack of 2)	2	9.99	1	9.99
7	Amazon	TP-Link 8 Port Gigabit Ethernet Network Switch	1	19.99	1	19.99
8	Amazon	Ethernet cable	6	6.64	1	6.64
9						========
10						354.03

4.3 - Parts for a 8 node cluster

Parts

This section will have the list of tools and parts that we recommend you get to assemble a PI cluster

Prices are in dollar as found at the time on online retailers.

To add parts please visit the yaml file and add them to the parts list

	vendor	description	included	price	count	total	comment	image
0	Cana Kit	Raspberry Pi 4 8GBSKU: PI4-8GB	1	75.0	8	600.0
1	Amazon	Heat Sinks	12	12.99	1	12.99
2	Amazon	SD Card 64GB EVO Select	1	9.99	8	79.92
3	Amazon	SanDisk 32GB Ultra microSDHC	1	10.99	8	87.92	Only buy those for PI3B+, for 4 use the cheaper afaster and bigger 64GB cards
4	Amazon	Duo SD Card Reader	1	7.99	1	7.99
5	Amazon	USB Charger 10-Port 120W	1	39.99	1	39.99	Warning when using the power switsh listed in the table, you must always switch it on and off from the build in switch. Never put the power supply on a power strip that has its own switch and use that switch to switch on and off via the power supply and leave the switch on the supply just open. THis coudl destroy your supply.
6	Amazon	Cable with switch	2	8.89	4	35.56	has 2 cables included and as the 3 cable option is not available making tis a good option
7	Amazon	CanaKit Raspberry Pi 4 Micro HDMI Cable - 6 Feet (Pack of 2)	2	9.99	1	9.99
8	Amazon	TP-Link 8 Port Gigabit Ethernet Network Switch	1	19.99	1	19.99
9	Amazon	Ethernet cable	6	6.64	2	13.28
10						========
11						907.63