Database seed

Requirement for seamless functionality of the HEAppE Middleware is seed.njson file creation (database seed setup). When HEAppE is starting up database seed is applied.

An example template for HPC Cluster mode configuration seed is presented at the URL address HEAppE/RestApi/seed.example.njson.

The file is in JSON format and contains the following configuration blocks:

AdaptorUser

Password is SHA512 hash with cryptographic salt, which consists of datetime value in yyyy-MM-dd HH:mm:ss format. For password creation you can use an online tool for generating SHA512 hashes with salt: https://www.convertstring.com/en/Hash/SHA512

Warning

The same datetime which will be used for hashing as cryptographic salt must be specified in the attribute CreatedAt of AdaptorUser.

"AdaptorUsers": [
{
    "Id": 1,
    "Username": "heappe_user",
    "Password": "606756EAC81B78675E3CCC47708318484E556CE9B3261C8C3FB6F8A008C48A0ACCD3326C8CA54938F43DDB60E6126BC2CF4E50D575E5DED0A71E0FE50CF323E4",
    "LanguageId": 1,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null
}]

AdaptorUserGroup

"AdaptorUserGroups": [
{
    "Id": 1,
    "Name": "Group_1",
    "Description": "This group is first",
    "AccountingString": "{AccountingString}",
    "ProjectId": 1
}]

Note

AccountingString must be the same project-ID that is used at remote cluster.

AdaptorUserRoles

The HEAppE Middleware supports several user roles. The system is able to cascade assign permissions to a users according to their role. External user can be assigned to the following user roles (see below).

For example, a user with the Administrator role has privileges of all defined roles.

"AdaptorUserRoles": [
{
    "Id": 1,
    "Name": "Administrator",
    "Description": "HEAppE administrator role with access to the entire system."
},
{
    "Id": 2,
    "Name": "Maintainer",
    "Description": "HEAppE maintainer role for getting information about actual HEAppE status."
},
{
    "Id": 3,
    "Name": "Submitter",
    "Description": "Standard user, can submit and check his own jobs."
},
{
    "Id": 4,
    "Name": "GroupReporter",
    "Description": "Users with this role car report in assigned groups."
},
{
    "Id": 5,
    "Name": "Reporter",
    "Description": "Users with this role can watch other jobs in the same group."
},
{
    "Id": 6,
    "Name": "ManagementAdmin",
    "Description": "User with this role can create new project and use management API."
}]

AdaptorUserUserGroupRoles

Mapping AdaptorUsers to their roles and UserGroups. You can assign multiple AdaptorUsers to multiple AdaptorUserRoles [M:N]. You can also assign multiple AdaptorUsers to multiple AdaptorUserGroup [M:N].

"AdaptorUserUserGroupRoles": [
{
    "AdaptorUserId": 1,
    "AdaptorUserGroupId": 1,
    "AdaptorUserRoleId": 1,
    "CreatedAt": "2015-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]

ClusterProxyConnections

Cluster proxy connection configuration. Proxy configuration is set up only if it is needed.

"ClusterProxyConnections": [
{
    "Id": 1,
    "Host": "",
    "Port": 8000,
    "Type": 1,
    "Username": null,
    "Password": null
}]

Note

Attribute type can take following values:

• Socks4 = 1

• Socks5 = 2

• Http = 3

Clusters

"Clusters": [
{
    "Id": 1,
    "Name": "Cluster 1",
    "Description": "Our cluster number one",
    "MasterNodeName": "cluster.domain.com",
    "SchedulerType": 2,
    "ConnectionProtocol": 2,
    "TimeZone": "CET",
    "Port": null,
    "UpdateJobStateByServiceAccount": true,
    "DomainName": "domain.com",
    "ProxyConnectionId": null
}]

Note

Attribute SchedulerType can take following values:

• LinuxLocal = 1

• PbsPro = 2

• Slurm = 4

• Slurm with HyperQueue = 8

Attribute ConnectionProtocol can take following values:

• MicrosoftHpcApi = 1

• Ssh = 2

• SshInteractive = 4

ClusterAuthenticationCredentials

PrivateKeyFile authentication

"ClusterAuthenticationCredentials":[
{
    "Id": 1,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile":" /opt/heappe/keys/key_01",
    "PrivateKeyPassword": ""
},
{
    "Id":2,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": "/opt/heappe/keys/key_02",
    "PrivateKeyPassword": ""
}]

SshAgent authentication

"ClusterAuthenticationCredentials":[
{
    "Id": 1,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": null,
    "PrivateKeyPassword": null
},
{
    "Id": 2,
    "Username": "cluster-user",
    "Password": null,
    "PrivateKeyFile": null,
    "PrivateKeyPassword": null
}]

Projects

HEAppE integrates multi-HPC projects support. This block is needed for project configuration.

"Projects":[
{
    "Id": 1,
    "Name": "Project1",
    "Description": "Project 1",
    "AccountingString": "DD-XX-XX",
    "StartDate": "2022-01-01 00:00:00",
    "EndDate": "2022-12-12 23:59:59",
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]

ClusterProjects

Cluster project configuration block.

"ClusterProjects":[
{
    "Id": 1,
    "ClusterId": 1,
    "ProjectId": 1,
    "LocalBasepath": "/scratch/project/DD-XX-XX/HEAppE/Executions",
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]

ClusterProjectCredentials

Cluster project credentials allow assigned cluster authentication credentials to specific projects.

"ClusterProjectCredentials":[
{
    "ClusterProjectId": 1,
    "ClusterAuthenticationCredentialsId": 1,
    "IsServiceAccount": true,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
},
{
    "ClusterProjectId": 1,
    "ClusterAuthenticationCredentialsId": 2,
    "IsServiceAccount": false,
    "CreatedAt": "2022-01-01 00:00:00",
    "ModifiedAt": null,
    "IsDeleted": false
}]

ClusterNodeTypes

Cluster queues/slurm partitions

"ClusterNodeTypes":[
{
    "Id": 1,
    "Name": "Production",
    "Description": "Standard production nodes",
    "NumberOfNodes": 192,
    "CoresPerNode": 36,
    "Queue": "qprod",
    "MaxWalltime": 172800,
    "ClusterId": 1,
    "FileTransferMethodId": 1,
    "ClusterAllocationName": null
},
{
    "Id": 2,
    "Name": "Express",
    "Description": "Testing nodes",
    "NumberOfNodes": 16,
    "CoresPerNode": 36,
    "Queue": "qexp",
    "MaxWalltime": 3600,
    "ClusterId": 1,
    "FileTransferMethodId": 1,
    "ClusterAllocationName": null
}]

CommandTemplates

Command Template defines what will be executed on the cluster. You can prepare these command templates in database seed. You can also create a “Generic Command Template” and later create other command templates from it via the HEAppE Rest API.

"CommandTemplates":[
{
    "Id": 1,
    "Name": "TestTemplate",
    "Description": "TestTemplate",
    "Code": "TestTemplate",
    "ExecutableFile": "~/.key_scripts/test.sh",
    "CommandParameters": "%%{inputParam}",
    "PreparationScript": null,
    "ClusterNodeTypeId": 2,
    "IsGeneric": false,
    "IsEnabled": true,
    "ProjectId": null
},
{
    "Id": 2,
    "Name": "GenericCommandTemplate",
    "Description": "Command template for generic job.",
    "Code": "GenericCommandTemplate",
    "ExecutableFile": "~/.key_scripts/generic.sh",
    "CommandParameters": "%%{userScriptPath} %%{userScriptParametres}",
    "PreparationScript": null,
    "ClusterNodeTypeId": 2,
    "IsGeneric": true,
    "IsEnabled": true,
    "ProjectId": null
}]

CommandTemplateParameters

"CommandTemplateParameters": [
{
    "Id": 1,
    "Identifier": "inputParam",
    "Query": "",
    "Description": "inputParam",
    "CommandTemplateId": 1,
    "IsVisible": true
},
{
    "Id": 2,
    "Identifier": "userScriptPath",
    "Query": "",
    "Description": "Path of the user script, to be executed via the generic job script",
    "CommandTemplateId": 2,
    "IsVisible": true
},
{
    "Id": 3,
    "Identifier": "userScriptParametres",
    "Query": "",
    "Description": "Generic parameters of the generic command template.",
    "CommandTemplateId": 2,
    "IsVisible": false
}]

FileTransferMethods

Cluster server hostname with protocol specification.

"FileTransferMethods":[
{
    "Id": 1,
    "ServerHostname": "cluster.domain.com",
    "Protocol": 2,
    "ClusterId": 1
}]

Note

Attribute Protocol can take following values:

• NetworkShare = 1

• SftpScp = 2

• LocalSftpScp = 4