HPC Cluster Deployment Guide

The following text describes how to deploy HEAppE Middleware and connect it to the HPC cluster. HEAppE should be deployed within HPC centre´s internal network. The deployment process is divided into several steps.

1. Server Environment Prerequisites

Server Environment (deploy on virtual machine)
For running instances of HEAppE Middleware is necessary to have these hardware/software requirements.

CPU

RAM

HDD

Operation system

2

4GB

50 GB

Any Linux distribution (validated on Centos/Ubuntu)
Package installation
Git (Highly recommended)

Network-utils (Highly recommended)

Epel-release (Highly recommended)

Wget (Highly recommended)

Nginx (required for ensure HTTPS)

Docker

Docker Compose
Required ports to open in firewall
22 - SSH access to virtual machine (open in restricted mode for specific IP subnet)

80/443 - HTTPS Nginx proxy (Ensure SSL connectivity)

5000 - HTTP HEAppE API (open in restricted mode for specific IP subnet)

6000 - MS-SQL Database (open in restricted mode for specific IP subnet)
HPC Robot accounts creation
Process of gaining robot accounts is described at the Robot accounts.

2. HEAppE Environment & Configuration Setup

These steps should be performed as a sudo user (recommended). If you are unsure about the folder structure, please refer to the folder validation scheme at the end of this chapter.

HEAppE Middleware installation procedure

This script will guide you through the entire HEAppE Middleware setup process. If you prefer to go through the process manually, follow the steps at HPC Deployment Deployment Expert Guide.

The script can be downloaded here: HEAppE-setup.sh

To run the script, execute the following commands in your console as a sudo user:

$ chmod +x HEAppE-setup.sh -p PROJECT
$ ./HEAppE-setup.sh

Note

Please run the script with the -p / --project parameter to specify the project name. The project name is used to create the project directory structure.

If you want to run script without any user interaction, you can use the -d / --defaults parameter to use the default values in the configuration files.

Script actions:

Creation of the directory structure for the HEAppE Middleware.
Preparation of the .env file for docker containers.
Preparation of the docker-compose.override.yml file for docker containers.
Acquisition of the release of the HEAppE Middleware official repository and copy docker-compose files.
Creation of the appsettings.json file.
Creation of the appsettings-data.json file.
Creation of the seed.njson file.
Credentials Vault initialization and unsealing procedure.
Inicialization the docker containers.

Note

The script creates the following directory structure:

/opt/heappe
├── confs
└── projects
    └── PROJECT
        ├── app
        │   ├── keys
        │   ├── logs
        │   └── confs
        ├── heappe-core
        ├── ssh_agent
        │   └── keys
        └── docker_configurations
            ├── docker-compose.override.yml
            └── .env

Warning

The script will ask you to provide the SA password for the SQL Server - which is deyployed in the docker container and HEAppE Middleware will use it to connect to the database. Please check the password policy of the Microsoft SQL Server. The password must meet the requirements of the SQL Server password policy.

Generation and storing of SSH keys for connection to the HPC cluster

There are several options for managing SSH keys based on your setup:

Using HEAppE Middleware for SSH Key Generation (Preferred)
After the successful HEAppE deployment, you can generate SSH keys via the HEAppE REST API endpoint /heappe/Management/GenerateSecureShellKey. The keys will be stored in the directory /opt/heappe/projects/PROJECT/app/<ACCOUNTING_STRING>/keys/.

Using Existing HPC Accounts
If you have gained robot HPC accounts (HEAppE Internal accounts) from the HPC centre, you need to put these keys to the directory /opt/heappe/projects/PROJECT/app/keys/.

Using SSH Keys from SSH Agent
If you will use SSH keys from the SSH Agent, you need to put these keys to the directory /opt/heappe/projects/PROJECT/ssh_agent/keys/.

SSH keys need to be loaded into the SSH Agent (Docker container SshAgent).

Note

More information about SSH keys generation is available at the Robot accounts.

3. Cluster script directory initialization

The following figure illustrates the comprehensive directory and shared locations structure that has to be created for successful deployment.

A) HEAppE Job Execution directory structure

The directory structure for HEAppE Job Execution is created under the Master robot account identity automatically. HEAppE checks the existence of the directory structure when user creates Job by calling JobManagement/CreateJob endpoint and creates it if it does not exist. This directory structure is created on shared storage (usually scratch project directory) that is accessible from all nodes of the HPC cluster.

B) & C) HEAppE Scripts directory structure

The directory structure for HEAppE Scripts is created under the Master robot account identity. Description of the HEAppE cluster scripts is available at the Cluster scripts.

Automatic setup

This setup can be done automatically by calling HEAppE REST API endpoint Management/InitializeClusterScriptDirectory after HEAppE is configured and started. Each Robot account identity has to create symbolic links to this cluster script directory to be able to use it. Symbolic links are created at the Robot account identitity home directory.

Manual setup

Another way of preparation is manual setup. If you choose this way, you need to do the following steps:
Create the directory structure for the application scripts under the Master robot account identity.
$ mkdir -p {disk_location}/HEAppE
$ chmod 770 {disk_location}/HEAppE
$ cd HEAppE

$ mkdir -p Scripts
$ chmod 770 Scripts
Clone the repository with the application scripts.
$ cd Scripts

$ git clone https://github.com/It4innovations/HEAppE-scripts.git
Move the scripts to the .key_scripts directory.
$ mv HEAppE-scripts/HPC/.key_scripts .key_scripts
$ rm -Rf HEAppE-scripts
$ chmod -R 750 .key_scripts
$ ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts
Modify the scripts according to your needs.
Create symbolic links under the Robot account identities.
$ ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts
Warning

HEAppE .key_scripts that have to be modified at cluster by user when choosing manual way of preparation are listed below.

copy_data_to_temp.sh
Specify hpcprojectid variable (corresponding project id identifier). Eg. project id format DD-XX-XX

remote-cmd3.sh
Replace default value of baseprefixpath variable by the path of HEAppE/Executions folder.

Note

If everything is successfully deployed, HEAppE is available at http://localhost:5000 (Swagger at http://localhost:5000/swagger/index.html).

And you can authenticate with the default credentials: admin/admin via the UserAndLimitationManagement/AuthenticateUserPassword endpoint.