This article is supplemental information for OpenMPI integration with PBS Professional Server and execution hosts.

The steps  in this article were tested using PBS Professional 2020.1.1 , however they should work for any versions 2020 or greater.  These steps were also tested on a CentOS 8 virtual machine, but should work on other versions and operating systems with minor modifications that someone with familiarity  of the operating system should be able to do. The specifications for the VM were a 4 core with 8GB RAM and 40GB drive space.  This is probably more than was needed.  The scope of the installation of the VM and OS are beyond the scope of this document.

To Start:

Install the needed PBS Professional RPMS.  In this case, PBS Professional 2021.1.1 was used:

rpm -ivvh pbspro-execution-2021.1.1.20210518090504-0.el8.x86_64.rpm pbspro-devel-2021.1.1.20210518090504-0.el8.x86_64.rpm

It doesn’t even actually matter if the PBS session works or not.  It’s just needed for the libraries to compile against.

Next, install some libraries that you will need for the compile and install:

yum install -y zlib gcc-c++ openssl-devel gcc

yum install gcc-gfortran # I think this is only needed if you want openmpi fortran support

yum group install "development tools"

Now, extract and set up the actual source RPM for OpenMPI.  They can be obtained from the OpenMPI homepage (https://www.open-mpi.org).  For this example, the package was downloaded into /root.

cd /root

rpm -ivvh openmpi-4.1.1-1.src.rpm

Locate the newly created source code directory.  For this install, the rpm created:

/root/rpmbuild/SOURCES

Extract the tar file found there:

cd /root/rpmbuild/SOURCES

tar -xzvf openmpi-4.1.1.tar.gz

At this point, you can choose to compile and build (which you can install into a shared filesystem), or create an RPM for distribution.  If you want to create an RPM, skip down to that section.  Otherwise, continue on to the next steps.

Build for shared filesystem:

cd openmpi-4.1.1

export LD_LIBRARY_PATH=”/opt/pbs/lib:$LD_LIBRARY_PATH”

export LDFLAGS="-L/opt/pbs/lib -lpbs -lpthread -lsec”

Depending on your version of PBS, you will also need to compile with the -lcrypto libraries.  If you do need that, make sure to install libcrypto into your compile environment.

Depending on where you want the libraries and programs to install, you should look at the configure help to determine what options to use. To stick with the defaults, just do this:

./configure --with-tm=/opt/pbs

make

make install

This process will take quite a long time.

Build for RPM creation:

First, install the needed rpm building tools:

yum install -y rpmdevtools rpmlint

Get ready to build the packaging environment:

cd openmpi-4.1.1/contrib/dist/linux/

rpmdev-setuptree

Now, initiate the build and packaging processes:

export QA_RPATHS=0x0003
rpmbuild --rebuild --define 'configure_options QA_RPATHS=0x0003 LD_LIBRARY_PATH=/opt/pbs/lib:$LD_LIBRARY_PATH LDFLAGS="-L/opt/pbs/lib -lpbs -lpthread -lsec" --with-tm=/opt/pbs' /root/openmpi-4.1.1-1.src.rpm

In the end of the compile (which will take quite a while), look for a line that says something like:

Wrote: /root/rpmbuild/RPMS/x86_64/openmpi-4.1.1-1.el8.x86_64.rpm

That's the rpm file you can use to install the rest of the cluster.