HyperQueue support

HyperQueue is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers. HEAppE Middleware directly supports HyperQueue as a task execution engine. This means that you can use HyperQueue to execute tasks on HPC clusters through HEAppE Middleware.

Warning

Prerequisity of using HyperQueue with HEAppE Middleware is to have Slurm workload manager and HyperQueue installed on HPC cluster as module.

To enable HyperQueue support in HEAppE Middleware, you need to set the SchedulerType attribute in the configuration file to 8 (HQ with Slurm) in the Clusters section. HEAppE Middleware uses the HyperQueue Auto Allocation feature to automatically allocate resources for task execution.

GPU nodes support

When using HyperQueue with HEAppE Middleware, you can also use GPU nodes for task execution. To enable GPU nodes support, you need to set the PlacementPolicy attribute in the HEAppE Task Specification to gpus/nvidia. Amount of the GPUs will be computed by the HEAppE Middleware based on the MaxCores attribute in the HEAppE Task Specification.

CPU nodes support

CPU nodes are used by default, you do not need to set any additional attributes in the HEAppE Task Specification.

Note

When using HyperQueue, also please make sure that you are using the corrrect HPC partition / queue for the task execution.