File Transfer

HEAppE Middleware provides two ways to transfer files between the client and the server:

HTTP (only for downloading small files)
SCP/RSYNC

HEAppE Middleware also provides a way to list the files in the job execution directory. This is useful to know the files that are available to download.

ListChangedFilesForJob

List the files in the job execution directory. Method has the following parameters:

HEAppE ListChangedFilesForJob Parameters
Field	Type	Description	Example Values
SubmittedJobInfoId	integer	The job identifier returned by the `/heappe/JobManagement/SubmitJob` endpoint.	1
SessionCode	string	The session code returned by the `Authentication` endpoint.

Example Request:

curl -X 'GET' \
  'http://localhost:5000/heappe/FileTransfer/ListChangedFilesForJob?sessionCode=123e4567-e89b-12d3-a456-426614174000&submittedJobInfoId=1' \
  -H 'accept: application/json'

Example Response:

[
  {
    "FileName": "/1/stdout",
    "LastModifiedDate": "2024-05-03T06:53:52Z"
  },
  {
    "FileName": "/1/stderr",
    "LastModifiedDate": "2024-05-03T06:49:22Z"
  },
  {
    "FileName": "/1/resultFile.txt",
    "LastModifiedDate": "2024-05-03T06:54:22Z"
  }
]

Note

The response contains the list of files changed in the job execution directory. The FileName is the relative path of the file in the job execution directory and sould be used in the DownloadFileFromCluster method.

DownloadFileFromCluster (HTTP method)

This HTTP method is used to transfer small files from the client and the server using the HTTP protocol. Typically, this method is used to get the output files (stdout, stderr) from the HPC Job Execution directory to the client. Method has the following parameters:

HEAppE DownloadFileFromCluster Parameters
Field	Type	Description	Example Values
SubmittedJobInfoId	integer	The job identifier returned by the `/heappe/JobManagement/SubmitJob` endpoint.	1
SessionCode	string	The session code returned by the `Authentication` endpoint.
RelativeFilePath	string	The relative path of the file to download.	“stdout”

Example Request:

curl -X 'POST' \
  'http://localhost:5000/heappe/FileTransfer/DownloadFileFromCluster' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "SessionCode": "123e4567-e89b-12d3-a456-426614174000",
  "SubmittedJobInfoId": 1,
  "RelativeFilePath": "/1/stdout"
}'

Note

The RelativeFilePath parameter is the relative path of the file to download. The file must be in the job execution directory. To get the list of files in the job execution directory, use the /heappe/FileTransfer/ListChangedFilesForJob endpoint. Result of the method is the file content in the Base64 format.

DownloadPartsOfJobFilesFromCluster

This method is used to download parts of the files from the job execution directory. This method is useful when the file is larger and the client wants only to download the file in parts. Method has the following parameters:

HEAppE DownloadPartsOfJobFilesFromCluster Parameters
Field	Type	Description	Example Values
SubmittedJobInfoId	integer	The job identifier returned by the `/heappe/JobManagement/SubmitJob` endpoint.	1
SessionCode	string	The session code returned by the `Authentication` endpoint.
TaskFileOffsets	array	The array of the file offsets to download.	see example

The DownloadPartsOfJobFilesFromCluster method contains the FileType attribute at the TaskFileOffsets array. The FileType attribute is an enum that specifies the type of the file to download. The following table shows the FileType enum values:

FileType Enum
Value	Description
0	Log File
1	Progress File
2	Standard Error File
3	Standard Output File

Example Request:

Downloads the rest of the file from the 16th byte of the stdout file from the HPC job execution directory (HEAppE Task).

curl -X 'POST' \
  'http://localhost:5000/heappe/FileTransfer/DownloadPartsOfJobFilesFromCluster' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "SessionCode": "123e4567-e89b-12d3-a456-426614174000",
  "SubmittedJobInfoId": 1,
  "TaskFileOffsets": [
    {
      "SubmittedTaskInfoId": 1,
      "FileType": 3,
      "Offset": 16
    }
  ]
}'

RequestFileTransfer (SCP/RSYNC method)

This endpoint is used to transfer files between the client and the server using the SCP/RSYNC protocol. HEAppE Middleware will create temporary SSH key and propagates public key part to the HPC cluster (if the cluster supports to add key to the .ssh/authorized_keys). The client can use the temporary SSH key to transfer files between the client and the selected HPC Job Execution directory.

This temporary SSH key is generated by the HEAppE. The client can use these commands:

scp command to transfer files using the SCP protocol.

rsync command to transfer files using the RSYNC protocol.

ls command to list the files in the job execution directory.

mkdir command to create a directory in the job execution directory.

Note

These commands are available to be used only for the job execution directory. This directory has the following structure: SharedBasepath/SubmittedJobInfoId/SubmittedTaskId

SharedBasepath is the base path of the job execution directory and is returned by the RequestFileTransfer method.
SubmittedJobInfoId is the job identifier returned by the /heappe/JobManagement/SubmitJob endpoint.
SubmittedTaskId is the task identifier returned by the /heappe/JobManagement/SubmitTask endpoint.

The RequestFileTransfer method has the following parameters:

HEAppE RequestFileTransfer Parameters
Field	Type	Description	Example Values
SubmittedJobInfoId	integer	The job identifier returned by the `/heappe/JobManagement/SubmitJob` endpoint.	1
SessionCode	string	The session code returned by the `Authentication` endpoint.

Example Request:

curl -X 'POST' \
  'http://localhost:5000/heappe/FileTransfer/RequestFileTransfer' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "SessionCode": "123e4567-e89b-12d3-a456-426614174000",
  "SubmittedJobInfoId": 1
}'

Note

The response contains the temporary SSH key that the client can use to transfer files between the client and the server. The SSH key is valid for 24 hours by default and user can generate up to 5 keys for the job (can be changed in the configuration file in the BusinessLogicSettings section).

Example Response:

{
  "ServerHostname": "hpc.cluster.com",
  "SharedBasepath": "/project/PROJ/HEAppE/Executions",
  "Protocol": 1,
  "Port": 22,
  "ProxyConnection": {
    "Host": "",
    "Port": 0,
    "Type": 1,
    "Username": "",
    "Password": ""
  },
  "Credentials": {
    "Username": "string",
    "Password": "string",
    "CipherType": 0,
    "CredentialsAuthType": 0,
    "PrivateKey": "string",
    "PublicKey": "string",
    "Passphrase": "string"
  }
}

Protocol

The Protocol attribute is an enum that specifies the type of used protocol. The following table shows the Protocol enum values:

Protocol Enum
Value	Description
0	NetworkShare
2	SftpScp
4	LocalSftpScp (only for LocalHPC Simulator )

CipherType

The CipherType attribute is an enum that specifies the type of used cipher. The following table shows the CipherType enum values:

CipherType Enum
Value	Description
0	None
1	RSA3072
2	RSA4096
3	nistP256
4	nistP521
5	ED25519

CredentialsAuthType

The CredentialsAuthType attribute is an enum that specifies the type of used credentials. The following table shows the CredentialsAuthType enum values:

CredentialsAuthType Enum
Value	Description
0	None
1	Password
2	PasswordInteractive
3	PasswordAndPrivateKey
4	PrivateKey
5	PasswordViaProxy
6	PasswordInteractiveViaProxy
7	PasswordAndPrivateKeyViaProxy
8	PrivateKeyViaProxy
9	PrivateKeyInSshAgent

CloseFileTransfer

This method is used to close the file transfer session. The client can use this method to close the file transfer session and remove the temporary SSH key. The method has the following parameters:

Example Request:

curl -X 'POST' \
  'http://localhost:5000/heappe/FileTransfer/CloseFileTransfer' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "SessionCode": "123e4567-e89b-12d3-a456-426614174000",
  "SubmittedJobInfoId": 1,
  "PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ
}'

Warning

Temporary SSH key type can be changed in the configuration file in the CertificateGeneratorSettings section. The default value is RSA with the lengts of 4096 bits.