Deployment

The following text describes how to deploy HEAppE Middleware. HEAppE is usually deployed on a server or its instance can also be deployed locally on a user’s machine (for testing purposes).

HEAppE deployment modes

There are two possibilities how to deploy HEAppE:

  1. Local Cluster mode

    Local HPC is docker container that simulates the behavior of the workload manager on a real Super-Computing cluster. This mode is for testing purpose of the HEAppE Middleware capabilities (Figure 1, left).

  1. HPC Cluster mode

    This mode offers real connectivity with production HPC cluster. HEAppE supports the HPC clusters with “PBS Professional” or “Slurm” batch scheduler (Figure 1, right).

The HEAppE Middleware communicates via “SSH (service)” or “SCP/Rsync (data transfer)” protocols with the “High-Performance Computer” or “Local Cluster”.

Local and HPC cluster

Figure 1: HEAppE modes architecture

Step 1: HEAppE environment setup - Prerequisites

  1. Local Environment (deploy on user machine)

    For running a test instance of HEAppE Middleware is necessary to have an operating system with Docker support and the computer must have at least 2 GB of free RAM.

    Software installation
  2. Server Environment (deploy on virtual machine)

    For running instances of HEAppE Middleware is necessary to have these hardware/software requirements.

    CPU

    RAM

    HDD

    Operation system

    2

    4GB

    50 GB

    Any Linux distribution (validated on Centos/Ubuntu)
    Package installation

    • Git (Highly recommended)

    • Network-utils (Highly recommended)

    • Epel-release (Highly recommended)

    • Wget (Highly recommended)

    • Nginx (required for ensure HTTPS)

    • Docker

    • Docker-compose

    Required ports to open in firewall

    • 22 - SSH access to virtual machine (open in restricted mode for specific IP subnet)

    • 80/443 - HTTPS Nginx proxy (Ensure SSL connectivity)

    • 5000 - HTTP HEAppE API (open in restricted mode for specific IP subnet)

    • 6000 - MS-SQL Database (open in restricted mode for specific IP subnet)

  3. HPC functional accounts creation

    Another requirement for seamless HEAppE deployment is establishing at least two (not limited) functional HPC accounts (this step is mandatory). For proper deployment, the user needs HPC accounts from the HPC center where the HEAppE middleware will be deployed. This step is important due to the generation and storage of SSH keys for connection to the HPC cluster. The following figure presents an example of the complete procedure for HPC functional accounts establishment at IT4Innovations (via open access competition). But there are more methods how to obtain HPC computational resources such as LUMI supercomputer computational resources, EuroHPC JU grant competitions, directors` discretion, and rental of computational resources.

    ../_images/functionalaccountcreation.png

Step 2: HEAppE environment setup (for both modes)

These steps do as sudo user (recommended). In case of a misunderstanding of folder structure, please move to the end of this chapter and see folder validation scheme.

  1. Create the “heappe” directory and move to created directory
    mkdir -p /opt/heappe
cd /opt/heappe
  2. Create the “app” directory with the following subdirectories where configuration files, logs, and keys will be stored
    mkdir -p app

mkdir -p app/confs
mkdir -p app/keys
mkdir -p app/logs
  3. Create directory where keys uses by SSH agent will be stored
    mkdir -p ssh_agent/keys
  4. Create directory for storing docker backup files
    mkdir -p docker_configuration
  5. Create and prepare .env file for docker containers
    touch docker_configuration/.env
nano docker_configuration/.env

# .env file contains (see below)
COMPOSE_PROJECT_NAME=HEAppETesting

HOST_SSHAGENT_NAME=SshAgentTesting
HOST_SSHAGENT_HOST_MAP_FOLD_PATH=/opt/heappe/ssh_agent/keys
HOST_SSHAGENT_CONTAINER_MAP_FOLD_PATH=/opt/heappe/sshagent_keys:ro

HOST_MSSQL_NAME=MssqlDbTesting
HOST_MSSQL_PORT=6000
HOST_MSSQL_PASSWORD=Passw0rd

HOST_APP_NAME=HeappeTesting
HOST_APP_PORT=5000
HOST_APP_HOST_MAP_FOLD_NAME=/opt/heappe/app
HOST_APP_CONTAINER_MAP_FOLD_PATH=/opt/heappe

# Only for Local Cluster mode
HOST_LOCAL_NAME=LocalHPC
HOST_LOCAL_PORT=49005
  6. Prepare docker-compose.override.yml file for docker containers
    touch docker_configuration/docker-compose.override.yml
nano docker_configuration/docker-compose.override.yml

# docker-compose.override.yml file contains (see below)
version: '3.4'

services:
  sshagent:
    container_name: ${HOST_SSHAGENT_NAME}
    volumes:
      - ${HOST_SSHAGENT_HOST_MAP_FOLD_PATH}:${HOST_SSHAGENT_CONTAINER_MAP_FOLD_PATH}

  mssql:
    container_name: ${HOST_MSSQL_NAME}
    environment:
      SA_PASSWORD: ${HOST_MSSQL_PASSWORD}
    ports:
      - ${HOST_MSSQL_PORT}:1433

  heappe:
    container_name: ${HOST_APP_NAME}
    volumes:
      - ${HOST_APP_HOST_MAP_FOLD_NAME}:${HOST_APP_CONTAINER_MAP_FOLD_PATH}
    ports:
      - ${HOST_APP_PORT}:80
    extra_hosts:
      - "host.docker.internal:host-gateway"

    In case of usage of the “Local Cluster mode” , the docker-compose.override.yml file have to be extended with “localhpc” block.

    # Only for Local Cluster mode
localhpc:
  container_name: ${HOST_LOCAL_NAME}
  build:
    context: .
    dockerfile: LocalHPC/Dockerfile
  ports:
    - ${HOST_LOCAL_PORT}:80
  7. Get the release of HEAppE Middleware official repository and copy docker-compose files
    cd /opt/heappe
git clone https://github.com/It4innovations/HEAppE.git
cp docker_configuration/docker-compose.override.yml ./HEAppE
cp docker_configuration/.env ./HEAppE

# Only for Local Cluster mode
git clone https://github.com/It4innovations/HEAppE-scripts.git
mv  HEAppE-scripts/LocalHPC ./HEAppE/LocalHPC
rm -rf HEAppE-scripts
  8. Generation and storing of SSH keys for connection to the HPC cluster

    There are two options (both are possible):

    • HEAppE uses SSH keys from HEAppE Database (For Local and HPC cluster modes)

      This option is for using SSH keys from the file and loading the private key from the passphrase stored in HEAppE´s database.

      cd app/keys #keys generated by support will be stored here

# Super-Computing Service (SCS) generates  SSH keys (depends on number of functional accounts)
# Store generated keys

ssh-keygen -t ed25519 -f "key_01" -C "key_01" #example Local Cluster mode
ssh-keygen -t ed25519 -f "key_02" -C "key_02" #example Local Cluster mode
    • HEAppE uses SSH keys from SSH Agent (For HPC cluster mode)

      This option is for using SSH keys from SSH Agent. SSH keys are loaded into SSH Agent in startup by user.

    cd ssh_agent/keys #keys generated by support will be stored here

# Super-Computing Service (SCS) generates  SSH keys (depends on number of functional accounts)
# Store generated keys

    Directory and file tree should look like this (folder validation scheme).

    /heappe
    /app
        /confs
            /appsettings.json #file
            /seed.njson #file
        /keys
        /logs
    /docker_configuration
        /.env #file
        /docker-compose.override.yml #file
    /HEAppE
        /.env #file
        /docker-compose.override.yml #file
        /......
    /ssh_agent
        /keys

Step 3a: HEAppE configuration files preparation

In order to configure the HEAppE environment, some additional setup files need to be prepared (“appsettings.json” and “seed.njon”). The first mandatory configuration file is appsettings.json, which includes key values about user domain parameters, business logic, or HPC connection framework. For instance, the template of the configuration file is stored at HEAppE/RestApi/appsettings.example.json. The following section describes appsettings.json in detail. The configuration code is divided into separate blocks and explained individually.

Note

Store appsettings.json into the directory ‘app/confs’.

The HEAppE database connection string is an important setting. A database connection string is a string that specifies information about a data source and the means of connecting to it (see the below example for the Microsoft SQL Server connection string).

"ConnectionStrings": {
    "MiddlewareContext": "Server=mssql;Database=heappe;User=SA;Password=<mssql_password>;TrustServerCertificate=true;",
    "Logging": ""
}

Settings in the following block define a swagger configuration with user-domain parameters. Example of API settings:

"ApplicationAPISettings": {
    "AllowedHosts": [ "http://localhost:5000" ],
    "DeploymentSettings": {
        "Name": "Local instance",
        "Description": "Local testing instance",
        "Version": "v3.1.0",
        "DeployedIPAddress": "127.0.0.1",
        "Port": "5000",
        "Host": "http://localhost:5000",
        "HostPostfix": "",
        "DeploymentEnvironmentType": "Docker",
        "ResourceAllocationInfrastructureTypes": [ "HPC" ]
    },
    "SwaggerDocSettings": {
        "Title": "HEAppE Web API",
        "Description": "HEAppE middleware API v3.1.0",
        "PrefixDocPath": "swagger",
        "TermOfUsageUrl": "https://heappe.eu",
        "License": "GNU General Public License v3.0",
        "LicenseUrl": "https://www.gnu.org/licenses/gpl-3.0.html",
        "ContactName": "IT4Innovations",
        "ContactEmail": "support.heappe@it4i.cz",
        "ContactUrl": "https://twitter.com/it4innovations"
    }
}
This configuration block allows time-interval definition (in seconds), where each background thread is activated.

  • GetAllJobsInformationCheck - Update job status from batch schedulers

  • CloseConnectionToFinishedJobsCheck - Close tunnel connection for finished jobs

  • ClusterAccountRotationJobCheck - Check if Middleware is able to submit job (ClusterAccountRotation = true)

  • FileTransferKeyRemovalCheck - Removal FileTransfer temporary added keys

"BackGroundThreadSettings": {
    "GetAllJobsInformationCheck": 30,
    "CloseConnectionToFinishedJobsCheck": 30,
    "ClusterAccountRotationJobCheck": 30,
    "FileTransferKeyRemovalCheck": 10800
}
This setting block allows the HEAppE business logic configurations.

  • ClusterAccountRotation - Rotation of cluster service accounts

  • GeneratedFileTransferKeyLimitPerJob - Maximal active FileTransfer temporary keys per job

  • ValidityOfTemporaryTransferKeysInHours - Validity of FileTransfer temporary keys

  • SessionExpirationInSeconds - User session expiration

  • HTTPRequestConnectionTimeoutInSeconds - HTTP requests timeout

"BusinessLogicSettings": {
    "ClusterAccountRotation": false,
    "GeneratedFileTransferKeyLimitPerJob": 5,
    "ValidityOfTemporaryTransferKeysInHours": 24,
    "SessionExpirationInSeconds": 900,
    "HTTPRequestConnectionTimeoutInSeconds": 10
}
These configurations allow the setup of encryption ciphers (file transfer temporary key). You can choose from several options:

  • RSA 3072

  • RSA 4096

  • ECDSA 256

  • ECDSA 521

"CertificateGeneratorSettings": {
    "CipherSettings": {
        "TypeName": "RSA",
        "Size": 4096
    }
}

The following two blocks present the configuration of the HEAppE HPC connection framework (for HPC as well as Local Cluster).

Local Cluster:

"HPCConnectionFrameworkSettings": {
    "GenericCommandKeyParameter": "#HEAPPE_PARAM",
    "JobArrayDbDelimiter": "<JOB_ARRAY_ITERATION>",
    "TunnelSettings": {
        "LocalhostName": "127.0.0.1",
        "MinLocalPort": 4000,
        "MaxLocalPort": 4500
    },
    "CommandScriptsPathSettings": {
        "AddFiletransferKeyCmdPath": "~/.key_scripts/add_key.sh",
        "RemoveFiletransferKeyCmdPath": "~/.key_scripts/remove_key.sh",
        "CreateJobDirectoryCmdPath": "~/.key_scripts/create_job_directory.sh",
        "ExecuteCmdPath": "~/.key_scripts/run_command.sh",
        "ExecuteBackgroundCmdPath": "~/.key_scripts/run_background_command.sh",
        "CopyDataFromTempCmdPath": "~/.key_scripts/copy_data_from_temp.sh",
        "CopyDataToTempCmdPath": "~/.key_scripts/copy_data_to_temp.sh"
    },
    "ClustersConnectionPoolSettings":{
        "Local":{
            "ConnectionPoolMinSize": 0,
            "ConnectionPoolMaxSize": 2,
            "ConnectionPoolCleaningInterval": 60,
            "ConnectionPoolMaxUnusedInterval": 1800
        }
    },
    "LinuxLocalCommandScriptPathSettings": {
        "PrepareJobDirCmdPath": "~/.local_hpc_scripts/prepare_job_dir.sh",
        "RunLocalCmdPath": "~/.local_hpc_scripts/run_local.sh",
        "GetJobInfoCmdPath": "~/.local_hpc_scripts/get_job_info.sh",
        "CountJobsCmdPath": "~/.local_hpc_scripts/count_jobs.sh",
        "CancelJobCmdPath": "~/.local_hpc_scripts/cancel_job.sh"
    }
}

HPC Cluster:

"HPCConnectionFrameworkSettings": {
    "GenericCommandKeyParameter": "#HEAPPE_PARAM",
    "JobArrayDbDelimiter": "<JOB_ARRAY_ITERATION>",
    "TunnelSettings": {
        "LocalhostName": "127.0.0.1",
        "MinLocalPort": 4000,
        "MaxLocalPort": 4500
    },
    "CommandScriptsPathSettings": {
        "AddFiletransferKeyCmdPath": "~/.key_scripts/add_key.sh",
        "RemoveFiletransferKeyCmdPath": "~/.key_scripts/remove_key.sh",
        "CreateJobDirectoryCmdPath": "~/.key_scripts/create_job_directory.sh",
        "ExecuteCmdPath": "~/.key_scripts/run_command.sh",
        "CopyDataFromTempCmdPath": "~/.key_scripts/copy_data_from_temp.sh",
        "CopyDataToTempCmdPath": "~/.key_scripts/copy_data_to_temp.sh"
    },
    "ClustersConnectionPoolSettings":{
        "Cluster 1":{
            "ConnectionPoolMinSize": 0,
            "ConnectionPoolMaxSize": 2,
            "ConnectionPoolCleaningInterval": 60,
            "ConnectionPoolMaxUnusedInterval": 1800
        }
    },
    "LinuxLocalCommandScriptPathSettings": {
        "PrepareJobDirCmdPath": "",
        "RunLocalCmdPath": "",
        "GetJobInfoCmdPath": "",
        "CountJobsCmdPath": "",
        "CancelJobCmdPath": ""
    }
}

Step 3b: HEAppE configuration files preparation

Another requirement for seamless functionality of the HEAppE Middleware is seed.njson file creation (database seed setup). When HEAppE is starting up database seed is applied.

An example template for HPC Cluster mode configuration seed is presented at the URL address HEAppE/RestApi/seed.example.njson , and for Local Cluster mode is stored at HEAppE/RestApi/seed.example.localcomputing.njson.

Note

Store seed.njson into the directory ‘app/confs’.

The following text boxes provide seed.json with some details:

  • AdaptorUser

    Password is SHA512 hash with cryptographic salt, which consists of datetime value in yyyy-MM-dd HH:mm:ss format. For password creation you can use an online tool for generating SHA512 hashes with salt: https://www.convertstring.com/en/Hash/SHA512

    The same datetime which will be used for hashing as cryptographic salt must be specified in the attribute ‘CreatedAt’ of AdaptorUser.

    "AdaptorUsers": [
{
    "Id": 1,
    "Username": "heappe_user",
    "Password": "606756EAC81B78675E3CCC47708318484E556CE9B3261C8C3FB6F8A008C48A0ACCD3326C8CA54938F43DDB60E6126BC2CF4E50D575E5DED0A71E0FE50CF323E4",
    "LanguageId": 1,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null
}]

  • AdaptorUserGroup

    "AdaptorUserGroups": [
{
    "Id": 1,
    "Name": "Group_1",
    "Description": "This group is first",
    "AccountingString": "{AccountingString}",
    "ProjectId": 1
}]

    Note

    AccountingString must be the same project-ID that is used at remote cluster.

  • AdaptorUserRoles

    The HEAppE Middleware supports several user roles. The system is able to cascade assign permissions to a users according to their role. External user can be assigned to the following user roles (see below).

    For example, a user with the Administrator role has privileges of all defined roles.

    "AdaptorUserRoles": [
{
    "Id": 1,
    "Name": "Administrator",
    "Description": "HEAppE administrator role with access to the entire system."
},
{
    "Id": 2,
    "Name": "Maintainer",
    "Description": "HEAppE maintainer role for getting information about actual HEAppE status."
},
{
    "Id": 3,
    "Name": "Submitter",
    "Description": "Standard user, can submit and check his own jobs."
},
{
    "Id": 4,
    "Name": "Reporter",
    "Description": "Users with this role can watch other jobs in the same group."
}]
  • AdaptorUserUserGroupRoles

    Mapping AdaptorUsers to their roles and UserGroups. You can assign multiple AdaptorUsers to multiple AdaptorUserRoles [M:N]. You can also assign multiple AdaptorUsers to multiple AdaptorUserGroup [M:N].

    "AdaptorUserUserGroupRoles": [
{
    "AdaptorUserId": 1,
    "AdaptorUserGroupId": 1,
    "AdaptorUserRoleId": 1,
    "CreatedAt": "2015-01-01 00:00:00",
    "ModifiedAt": null,
   "IsDeleted": false
}]
  • ClusterProxyConnections

    Cluster proxy connection configuration. Proxy configuration is set up only if it is needed.

    "ClusterProxyConnections": [
{
    "Id": 1,
    "Host": "",
    "Port": 8000,
    "Type": 1,
    "Username": null,
    "Password": null
}]

    Note

    Attribute “type” can take following values:

    • Socks4 = 1

    • Socks5 = 2

    • Http = 3

  • Clusters

    Local Cluster:

    "Clusters": [
{
    "Id": 1,
    "Name": "Local",
    "Description": "Local computing machine",
    "MasterNodeName": "host.docker.internal",
    "SchedulerType": 1,
    "ConnectionProtocol": 2,
    "TimeZone": "CET",
    "Port": 49005,
    "UpdateJobStateByServiceAccount": false,
    "DomainName": null,
    "ProxyConnectionId": null
}]

    HPC Cluster:

    "Clusters": [
{
    "Id": 1,
    "Name": "Cluster 1",
    "Description": "Our cluster number one",
    "MasterNodeName": "cluster.domain.com",
    "SchedulerType": 2,
    "ConnectionProtocol": 2,
    "TimeZone": "CET",
    "Port": null,
    "UpdateJobStateByServiceAccount": true,
    "DomainName": "domain.com",
    "ProxyConnectionId": null
}]

    Note

    Attribute “SchedulerType” can take following values:

    • LinuxLocal = 1

    • PbsPro = 2

    • Slurm = 4

    Attribute “ConnectionProtocol” can take following values:

    • MicrosoftHpcApi = 1

    • Ssh = 2

    • SshInteractive = 4

  • ClusterAuthenticationCredentials

    Local Cluster:

    "ClusterAuthenticationCredentials":[
{
    "Id": 1,
    "Username": "heappeclient",
    "Password": "pass",
    "PrivateKeyFile": "/opt/heappe/keys/key_1",
    "PrivateKeyPassword": ""
},
{
    "Id": 2,
    "Username": "heappeclient",
    "Password": "pass",
    "PrivateKeyFile": "/opt/heappe/keys/key_2",
    "PrivateKeyPassword": ""
}]
    HPC Cluster:
    PrivateKeyFile authentication
    "ClusterAuthenticationCredentials":[
{
    "Id": 1,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile":" /opt/heappe/keys/key_1",
    "PrivateKeyPassword": ""
},
{
    "Id":2,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": "/opt/heappe/keys/key_2",
    "PrivateKeyPassword": ""
}]

    SshAgent authentication

    "ClusterAuthenticationCredentials":[
{
    "Id": 1,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": null,
    "PrivateKeyPassword": null
},
{
    "Id": 2,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": null,
    "PrivateKeyPassword": null
}]
  • Projects

    HEAppE integrates multi-HPC projects support. This block is needed for project configuration.

    "Projects":[
{
    "Id": 1,
    "Name": "Project1",
    "Description": "Project 1",
    "AccountingString": "DD-XX-XX",
    "StartDate": "2022-01-01 00:00:00",
    "EndDate": "2022-12-12 23:59:59",
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]
  • ClusterProjects

    Cluster project configuration block.

    Local Cluster:

    "ClusterProjects":[
{
    "Id": 1,
    "ClusterId": 1,
    "ProjectId": 1,
    "LocalBasepath": "/home/heappeclient/heappetests",
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]

    HPC Cluster:

    "ClusterProjects":[
{
    "Id": 1,
    "ClusterId": 1,
    "ProjectId": 1,
    "LocalBasepath": "/scratch/project/DD-XX-XX/heappe/heappetests",
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]
  • ClusterProjectCredentials

    Cluster project credentials allow assigned cluster authentication credentials to specific projects.

    "ClusterProjectCredentials":[
{
    "ClusterProjectId": 1,
    "ClusterAuthenticationCredentialsId": 1,
    "IsServiceAccount": true,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
},
{
    "ClusterProjectId": 1,
    "ClusterAuthenticationCredentialsId": 2,
    "IsServiceAccount": false,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]
  • ClusterNodeTypes
    Setting up cluster queues, using Job/Task Templates and etc… .
    Submission Job Template defines some of Cluster node type values.

    Local Cluster:

    "ClusterNodeTypes":[
{
    "Id": 1,
    "Name": "ExpressLocal",
    "Description": "Testing nodes Local",
    "NumberOfNodes": 1,
    "CoresPerNode": 1,
    "Queue": "qexp",
    "MaxWalltime": 3600,
    "ClusterId": 1,
    "FileTransferMethodId": 1,
    "ClusterAllocationName": null
 }]

    HPC Cluster:

    "ClusterNodeTypes":[
{
    "Id": 1,
    "Name": "Production",
    "Description": "Standard production nodes",
    "NumberOfNodes": 192,
    "CoresPerNode": 36,
    "Queue": "qprod",
    "MaxWalltime": 172800,
    "ClusterId": 1,
    "FileTransferMethodId": 1,
    "ClusterAllocationName": null
},
{
    "Id": 2,
    "Name": "Express",
    "Description": "Testing nodes",
    "NumberOfNodes": 16,
    "CoresPerNode": 36,
    "Queue": "qexp",
    "MaxWalltime": 3600,
    "ClusterId": 1,
    "FileTransferMethodId": 1,
    "ClusterAllocationName": null
}]
  • CommandTemplates

    Command Template defines what will be executed on the cluster. You can prepare these command templates in database seed. You can also create a “Generic Command Template” and later create other command templates from it via the HEAppE Rest API.

    "CommandTemplates":[
{
    "Id": 1,
    "Name": "TestTemplate",
    "Description": "TestTemplate",
    "Code": "TestTemplate",
    "ExecutableFile": "~/.key_scripts/test.sh",
    "CommandParameters": "%%{inputParam}",
    "PreparationScript": null,
    "ClusterNodeTypeId": 2,
    "IsGeneric": false,
    "IsEnabled": true,
    "ProjectId": null
},
{
    "Id": 2,
    "Name": "GenericCommandTemplate",
    "Description": "Command template for generic job.",
    "Code": "GenericCommandTemplate",
    "ExecutableFile": "~/.key_scripts/generic.sh",
    "CommandParameters": "%%{userScriptPath} %%{userScriptParametres}",
    "PreparationScript": null,
    "ClusterNodeTypeId": 2,
    "IsGeneric": true,
    "IsEnabled": true,
    "ProjectId": null
}]

  • CommandTemplateParameters

    "CommandTemplateParameters": [
{
    "Id": 1,
    "Identifier": "inputParam",
    "Query": "",
    "Description": "inputParam",
    "CommandTemplateId": 1,
    "IsVisible": true
},
{
    "Id": 2,
    "Identifier": "userScriptPath",
    "Query": "",
    "Description": "Path of the user script, to be executed via the generic job script",
    "CommandTemplateId": 2,
    "IsVisible": true
},
{
    "Id": 3,
    "Identifier": "userScriptParametres",
    "Query": "",
    "Description": "Generic parameters of the generic command template.",
    "CommandTemplateId": 2,
    "IsVisible": false
}]
  • FileTransferMethods

    Cluster server hostname with protocol specification.

    Local Cluster:

     "FileTransferMethods":[
{
     "Id": 1,
     "ServerHostname": "localhost:49005",
     "Protocol": 4,
     "ClusterId": 1
 }]

    HPC Cluster:

    "FileTransferMethods":[
{
    "Id": 1,
    "ServerHostname": "cluster.domain.com",
    "Protocol": 2,
    "ClusterId": 1
}]

    Note

    Attribute “Protocol” can take following values:

    • NetworkShare = 1

    • SftpScp = 2

    • LocalSftpScp = 4

Step 4 (HPC mode): HEAppE scripts/folders description

This chapter (Step 4) has a mainly descriptive purpose, and therefore you don’t need to fulfill all steps immediately. The step-by-step manual for folder creation and script preparation is explained in the next chapter (Step 5).

To complete the following part is expected to have access to the HPC functional accounts (since all steps will be done within the HPC account environment). Overall you need one account that will be operated as a “Master functional account” and the others as just a “functional account”. The number of functional accounts depends on the number of isolated jobs you plan to apply. For instance, if you want to run five jobs in parallel, you need to have one ‘’master functional account’’ and five ‘’functional accounts’’. The main purpose of the ‘’master’’ account is managing and reporting, whilst the others are job runners. Set up your HPC accounts with proper folder structure and mandatory script location (see below):

  1. Folders to create (must be created under “Master functional account privileges”)

    • “.key_scripts” folder containing the HPC cluster´s scripts used by the HEAppE Middleware, which are located in HPC cluster scripts repository. It is required for each HPC cluster system that will be using through HEAppE Middleware. (recommended location: home directory)

    • “HEAppE/Executions” folder for job data (recommended location: area for temporal directory)

    • “HEAppE/ExecutionsTemp” folder for allowing copy of job data from/to a temporary folder (recommended location: rea for temporal directory)

  2. Scripts preparation - The bash scripts have to be located in the “.key_scripts” folder with specific privileges (ACLs). “Master functional account” is the only account with “read, write, and execute” permissions. The other “functional accounts” have assigned only “read, and execute” permissions. The “HPC cluster scripts repository” contains the following scripts:

    • “create_job_directory.sh” - Create a job directory in “HEAppE/Executions” folder

    • “add_key.sh” - Adding temporary SSH key for job data transfer

    • “remove_key.sh” - Remove temporary SSH key for job data transfer

    • “remote-cmd3.sh” - Security check - allowing only access to data in “HEAppE/Executions” folder

    • “run_command.sh” - Decoding allocation from BASE64 and sending to the HPC scheduler

    • “generic.sh” - Allowing users to run customized generic scripts

    • “test.sh” - Testing execution script

    • “test_generic.sh” - Generic testing execution script

    • “copy_data_from_temp.sh” - Copy job data from “HEAppE/ExecutionsTemp” folder to “HEAppE/Executions” folder

    • “copy_data_to_temp.sh” - Copy job data from “HEAppE/Executions” folder to “HEAppE/ExecutionsTemp” folder

Step 4 (Local mode): HEAppE scripts/folders description

These steps lead to the configuration of a local cluster.

  • Add public keys into ./HEAppE/LocalHPC/.ssh/authorized_keys` file on the Local HPC cluster

  • HEAppE key scripts modifications
    • “copy_data_to_temp.sh”

      For instance, set “hpcprojectid” variable to “test” value.

    • “remote-cmd3.sh”

      Replace default value of “baseprefixpath” variable by ‘/home/heappeclient/heappetests/’.

“.key_scripts” folder will be automatically copied with the Local HPC Dockerfile configuration linked in HEAppE docker-compose.

Note

Please, make sure that you modify source files in the ./HEAppE/LocalHPC/ directory. Do not directly modify scripts on the Local cluster container. Directories and files will be always copied to the cluster when the container will be started through Dockerfile (or docker-compose). Just modify files in ./HEAppE/LocalHPC/ directory.

Step 5: HPC cluster HEAppE configuration (HPC and Local cluster modes)

This part of the deployment will vary from case to case (or from HPC infrastructure to HPC infrastructure). The principle of the deployment process stays the same, but there are differences in project storage structures or shared locations from case to case. This step-by-step guide shows the HPC cluster mode HEAppE configuration at the IT4Innovations HPC infrastructure. The following figure 2 illustrates the comprehensive folder and shared locations structure that has to be created for successful deployment. Overall, you have to find a storage area for temporal folders/files (figure 2, part A), shared project folders/files (figure 2, part B), and account folders/files (figure 2, part C).

HPC cluster folder structure

Figure 2: HPC cluster folder structure

Please follow these instructions carefully:

  1. Preparation temporary folders for the job executions (figure, part A)

    Have to be prepared under “Master functional account” identity.

    mkdir -p {disk_location}/HEAppE
chmod 770 {disk_location}/HEAppE
cd HEAppE

mkdir -p Executions
mkdir -p ExecutionsTemp

chmod 770 Executions
chmod 770 ExecutionsTemp
  2. Preparation project shared folders for application scripts (figure, part B)

    Have to be prepared under “Master functional account” identity.

    mkdir -p {disk_location}/HEAppE
chmod 770 {disk_location}/HEAppE
cd HEAppE

mkdir -p Scripts
chmod 770 Scripts
cd Scripts

git clone https://github.com/It4innovations/HEAppE-scripts.git
mv HEAppE-scripts/HPC/.key_scripts .key_scripts

rm -Rf HEAppE-scripts #Remove all unnecessary files

#Modification copy_data_to_temp.sh and remote-cmd3.sh scripts in the .key_scripts folder (see note in Step 4)
chmod -R 750 .key_scripts

ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts #Create symlink from home directory to project shared scripts folder
  3. Preparation symbolic links to scripts located in the shared project folder (figure, part C)

    Have to be prepared under “functional account” identities.

    ln -sf {disk_location}/HEAppE/Scripts/.key_scripts ~/.key_scripts #Create symlink from home directory to project shared scripts folder

Note

HEAppE “.key_scripts” that have to be modified by user

  • “copy_data_to_temp.sh”

    Specify “hpcprojectid” variable (corresponding project id identifier). Eg. project id format DD-XX-XX

  • “remote-cmd3.sh”

    Replace default value of “baseprefixpath” variable by the path of “HEAppE/Executions” folder.

Step 6: HEAppE Middleware instance launching (on user/virtual machine)

  1. Build Docker containers

    For building containers (HEAppE, Database, etc..) use docker compose tool (multi-container Docker is specified in docker-compose.yml file).

    cd /opt/heappe/HEAppE
docker compose build
  2. Start Docker containers

    Use the code below to start the HEAppE Middleware running containers. Running containers start listening on ports defined in “.env” file.

    cd /opt/heappe/HEAppE
docker compose up -d

# Check of running containers
docker ps

# Adding private keys into the SSH Agent container (in case of using SSH Agent option)
docker exec -it SshAgent ./scripts/adding_keys.sh 2 "key_"

    Note

    If local computing is setup. Local cluster will be available at “host.docker.internal” with port from “.env” file. Local cluster run at port “49005” for our setup.

  3. Shutdown Docker containers

    Use the code below to shutdown the HEAppE Middleware running containers.

    cd /opt/heappe/HEAppE
docker compose down

# Check if containers are not running
docker ps -a

Note

If everything is successfully deployed, HEAppE is available at “http://localhost:5000”.