HPC Cluster Deployment Expert Guide

The following text describes how to deploy HEAppE Middleware and connect it to the HPC cluster. HEAppE should be deployed within HPC centre´s internal network. The deployment process is divided into several steps.

1. Server Environment Prerequisites

Server Environment (deploy on virtual machine)
For running instances of HEAppE Middleware is necessary to have these hardware/software requirements.

CPU

RAM

HDD

Operation system

2

4GB

50 GB

Any Linux distribution (validated on Centos/Ubuntu)
Package installation
Git (Highly recommended)

Network-utils (Highly recommended)

Epel-release (Highly recommended)

Wget (Highly recommended)

Nginx (required for ensure HTTPS)

Docker

Docker Compose
Required ports to open in firewall
22 - SSH access to virtual machine (open in restricted mode for specific IP subnet)

80/443 - HTTPS Nginx proxy (Ensure SSL connectivity)

5000 - HTTP HEAppE API (open in restricted mode for specific IP subnet)

6000 - MS-SQL Database (open in restricted mode for specific IP subnet)
HPC Robot accounts creation
Process of gaining robot accounts is described at the Robot accounts.

2. HEAppE Environment Setup

These steps should be performed as a sudo user (recommended). If you are unsure about the folder structure, please refer to the folder validation scheme at the end of this chapter.

Warning

This script will guide you through the entire HEAppE Middleware setup process. If you prefer to go through the process manually, follow the steps outlined below.

The script can be downloaded here: HEAppE-setup.sh

To run the script, execute the following commands in your console:

$ chmod +x HEAppE-setup.sh
$ ./HEAppE-setup.sh

The actions performed by the script are illustrated in this UML Activity diagram: HEAppE-env-setup-AD.svg

After executing the script, proceed with the next steps in 3. HEAppE configuration files preparation.

Create the HEAppE Middleware directory structure
Copy the following bash script to the server and execute it. This script sets up the necessary directory structure for the HEAppE Middleware.

Download the script here: HEAppE-dir-structure.sh

Grant execution permissions to the script.
$ chmod +x HEAppE-dir-structure.sh
Run the script with the project name argument as a sudo user.
$ ./HEAppE-dir-structure.sh HPC_PROJECT
Note

The script creates the following directory structure:
/opt/heappe ├── confs └── projects └── HPC_PROJECT ├── app │ ├── keys │ ├── logs │ └── confs ├── ssh_agent │ └── keys └── docker_configurations
Prepare .env file for docker containers
Copy the following .env file into the directory /opt/heappe/projects/HPC_PROJECT/docker_configuration/.

Download .env: .env
$ cp .env /opt/heappe/projects/HPC_PROJECT/docker_configuration/
Note

You can modify the file according to your needs - set container names, database master password, etc.
Prepare docker-compose.override.yml file for docker containers
Copy the following docker-compose.override.yml file into the directory /opt/heappe/projects/HPC_PROJECT/docker_configuration/.

Download docker-compose.override.yml: docker-compose.override.yml
$ cp docker-compose.override.yml /opt/heappe/projects/HPC_PROJECT/docker_configuration/
Note

You can modify the file according to your needs - set container names, database master password, etc. But specify the database password same as in the .env file.

Get the release of HEAppE Middleware official repository and copy docker-compose files

$ cd /opt/heappe/projects/HPC_PROJECT/
$ git clone https://github.com/It4innovations/HEAppE.git
$ cp /opt/heappe/projects/HPC_PROJECT/docker_configuration/docker-compose.override.yml ./HEAppE
$ cp /opt/heappe/projects/HPC_PROJECT/docker_configuration/.env ./HEAppE

Generation and storing of SSH keys for connection to the HPC cluster
There are several options for managing SSH keys based on your setup:
- Using Existing HPC Accounts
  If you have gained Robot HPC accounts (HEAppE Internal accounts) from the HPC centre, you need to put these keys to the directory /opt/heappe/projects/HPC_PROJECT/app/keys/.
- Using SSH Keys from SSH Agent
  If you will use SSH keys from the SSH Agent, you need to put these keys to the directory /opt/heappe/projects/HPC_PROJECT/ssh_agent/keys/.
  
  SSH keys need to be loaded into the SSH Agent (Docker container SshAgent).
- Using HEAppE Middleware for SSH Key Generation
  If you will use HEAppE Middleware to generate SSH keys, skip this step.
Note

More information about SSH keys generation is available at the Robot accounts.

3. HEAppE configuration files preparation

In order to configure the HEAppE environment, some additional setup files need to be prepared (appsettings.json, appsettings-data.json and seed.njon). The first mandatory configuration file is appsettings.json, which includes key values about user domain parameters, business logic, or HPC connection framework. For instance, the template of the configuration file is stored at HEAppE/RestApi/appsettings.example.json. The following section describes appsettings.json in detail. The configuration code is divided into separate blocks and explained individually.

Warning

You can go through the following steps or you can use interactive script for the HEAppE Middleware configuration. The script will guide you through the whole process of the HEAppE Middleware configuration.

The script is available at HEAppE-setup.sh

`appsettings.json` configuration

The HEAppE database connection string is an important setting. A database connection string is a string that specifies information about a data source and the means of connecting to it (see the below example for the Microsoft SQL Server connection string).

Following connection string can be used, but <mssql_password> needs to be replaced the with the password.
"ConnectionStrings": {
    "MiddlewareContext": "Server=mssql;Database=heappe;User=SA;Password=<mssql_password>;TrustServerCertificate=true;",
    "Logging": ""
}
Note

Store appsettings.json into the directory /opt/heappe/projects/HPC_PROJECT/app/confs.

More information about other sections of the configuration file is available at the REST API configuration.

Warning

Please check the password policy of the Microsoft SQL Server. The password must meet the requirements of the SQL Server password policy.

`appsettings-data.json` configuration

For the HEAppE Middleware Data Staging API configuration, you need to prepare the appsettings-data.json file. You need to specify same database connection string as in the appsettings.json file and also specify AuthenticationToken at ApplicationAPIConfiguration section of the file.

Following connection string can be used, but <mssql_password> needs to be replaced the with the password. The password must be same as in the appsettings.json connection string.
"ConnectionStrings": {
    "MiddlewareContext": "Server=mssql;Database=heappe;User=SA;Password=<mssql_password>;TrustServerCertificate=true;"
}
"ApplicationAPIConfiguration": {
    "AuthenticationToken": "<token>"
}
Note

Store appsettings-data.json into the directory /opt/heappe/projects/HPC_PROJECT/app/confs.

More information about other sections of the configuration file is available at the Data Staging API configuration.

`seed.njson` configuration

Another requirement for seamless functionality of the HEAppE Middleware is seed.njson file creation (database seed setup). When HEAppE is starting up database seed is applied.

An example template for HPC Cluster mode configuration seed is presented at the URL address HEAppE/RestApi/seed.example.njson

Note

Store seed.njson into the directory /opt/heappe/projects/HPC_PROJECT/app/confs.

Warning

If you want to use specific configuration, you can modify the seed.njson file according to your needs. You can find more information about seed.njson file at the Database seed.

4. Cluster script directory initialization

This part of the deployment will vary from case to case (or from HPC infrastructure to HPC infrastructure). The principle of the deployment process stays the same, but there are differences in project storage structures or shared locations from case to case. This step-by-step guide shows the HPC cluster mode HEAppE configuration at the IT4Innovations HPC infrastructure. The following figure 2 illustrates the comprehensive folder and shared locations structure that has to be created for successful deployment. Overall, you have to find a storage area for temporal folders/files (figure 2, part A), shared project folders/files (figure 2, part B), and account folders/files (figure 2, part C).

Preparation project shared folders for application scripts (figure, part B)

There are two ways of preparation application scripts:

Automatically by calling HEAppE REST API endpoint Management/InitializeClusterScriptDirectory after HEAppE is configured and started (more informations about REST API on page apicmdtemplate).

Manually with script below which have to be prepared under Master robot account identity.
$ mkdir -p {disk_location}/HEAppE
$ chmod 770 {disk_location}/HEAppE
$ cd HEAppE

$ mkdir -p Scripts
$ chmod 770 Scripts
$ cd Scripts

$ git clone https://github.com/It4innovations/HEAppE-scripts.git
$ mv HEAppE-scripts/HPC/.key_scripts .key_scripts

$ rm -Rf HEAppE-scripts

$ chmod -R 750 .key_scripts

$  ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts

Preparation symbolic links to scripts located in the shared project folder (figure, part C)

Have to be prepared under robot account identities.
$ ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts

Note

HEAppE .key_scripts that have to be modified by user

copy_data_to_temp.sh
Specify hpcprojectid variable (corresponding project id identifier). Eg. project id format DD-XX-XX
remote-cmd3.sh
Replace default value of baseprefixpath variable by the path of HEAppE/Executions folder.

5. HEAppE Middleware instance launching

Build Docker containers

For building containers (HEAppE, Database, etc..) use docker compose tool (multi-container Docker is specified in docker-compose.yml file).
$ cd /opt/heappe/heappe-core
$ docker compose build

Init and unseal Credentials Vault

Before the HEAppE Middleware deployment, the Vault needs to be initialized and unsealed. Initialization is a one-time process that sets up the Vault’s encryption keys and other configuration settings. During initialization, the Vault generates the initial root token and unseal keys, which are required to unseal the Vault.

The scripts can be also used for unsealing the Vault when it is sealed and already initialized, usage is the same as for initialization.

Note

Tis workflow requires ansible-vault and jq to be installed.

1. Navigate to the Vault scripts directory:
$ cd /opt/heappe/heappe-core/HashiCorpVaultScripts
2. Add execute permissions to the init and unseal scripts:
$ chmod +x *.sh
3. Run the initialization script:
$ ./initHVault.sh myVaultPassword -p /opt/heappe/projects/credentials -b ../../app/confs -i HPC_PROJECT
Note

If configuration files do not exist in the specified base path, default configuration files will be created.

Start Docker containers

Use the code below to start the HEAppE Middleware running containers. Running containers start listening on ports defined in .env file.
$ cd /opt/heappe/heappe-core
$ docker compose --profile sshagent --profile db --profile staging up -d
Check of running containers
$ docker ps
Adding private keys into the SSH Agent container (in case of using SSH Agent option)
$ docker exec -it SshAgent ./scripts/adding_keys.sh 2 "key_"

Shutdown Docker containers

Use the code below to shutdown the HEAppE Middleware running containers.
$ cd /opt/heappe/heappe-core
$ docker compose --profile sshagent --profile db --profile staging down
Check if containers are not running
$ docker ps -a