General Explanation of Custom Resources in PBS Professional
Altair EmployeeExplanation and Settings
Currently, there are six types of custom resources that can be created in PBS Professional. Those are: boolean, string, long, float, size, and string_array. Each resource is defined in its own section in this document, with explanations of their capabilities and use cases for each of them. All resource types can be used on the server (whether that be in a queue or just in the server definition), and on nodes and vnodes in the cluster.
A consumable resource can be incremented and/or decremented by some action on the server, queue, or nodes (depending on where the resource resides). Actions could be the allocation of a specific amount of that resource by a job scheduled on a node. Actions outside of job resource allocation such as hooks and scripts can also affect custom resources. . An example of a consumable resource is ncpus. The PBS server is told by a MoM node that it has a certain number of CPUs. This is represented by resources_available.ncpus = 72. When a job is assigned to a compute node (MoM), the number of CPUs available on that node is decreased by PBS. So, for example, a job requests 40 of those CPUs. Checking the node’s attributes after the resource is allocated, the value of resources_assigned.ncpus = 40. Thus, the available ncpus for the node would be 32. Not all resource types can be consumable. Those which can be consumable are: long, float, string_array, and size.
Non-consumable resources can be modified and re-assigned, but do not represent a variable amount of resources to be allocated. Any resource type can be defined as non-consumable.
NOTE: All custom resources are created using qmgr commands. However, in order for a resource to be allocated by the user (and tracked by PBS), it also needs to be in the sched_config file. To add resources to the sched_confg file, add the name of the resource into the line in the file which starts with "resources: ". If the resource does not need to be allocated by a user, it does not need to be added to sched_config.
Along with the resources themselves, there are flags which determine how they will act, as well as how, where, and by whom they can be accessed. The flag settings of (no flags), h and m define where the variable will exist in the cluster and what machines (server, scheduler, mom, etc.) can access it. Other flags explain how a resource can be accessed or modified. Here is the breakdown:
| Flag | defined on | consumable |
| No flags | server/queue | no |
| h | host/vnode | no |
| fh | first host/node | yes |
| nh | host/vnode | yes |
| q | server/queue | yes |
| m | server/queue (cached on nodes) | (depending on other flags) |
The “m” resource flag is a little different from the rest. In the case of this flag, it exists in two places. The actual resource is on the server/queue. But a copy of that value is also cached on the execution nodes.
Normally, a resource is visible to users and can be modified by users. There are two other flags which can be applied to the resource which can change those properties of the resource.
| Flag | Definition |
| i | The resource is invisible to users. |
| r | The resource is read only. Only able to be modified by higher authority users and hooks. |
The above charts are a basic description for the resource flags. For a better explanation, please refer to the Reference Guide. This chart was created to be a quick reference guide to see what flags would be needed while defining resources.
Definition and Examples
Type: boolean
Description:
A boolean resource only has two values, True or False.
Restrictions:
Can never be a consumable resource.
Possible values are:
TRUE, True, true, T, t, Y, y, 1
FALSE, False, false, F, f, N, n, 0
Use Cases:
Server level: Can be used as an on/off switch for hooks. For example, if the hooks were meant to be turned on and off all at one time or modify the behavior of the hooks, for a specific mode the user wants the cluster to be in. For example, there could be a diagnostic mode which would set a debug variable in the hooks. When set, the hooks would output more information about their execution. With the recent addition of the "m" flag, this could be used to globally turn on and off a given hook (or set of hooks). For instance, turning off a node health check, globally across the entire cluster.
Defined: qmgr -c 'create resource node_health_control type=boolean,flag=m'
Usage: qmgr -c 's s resources_available.node_health_control=true'
Access: <python> watcher_on = pbs.server().resources_available["node_health_control"]
Host level: Could be used to state whether or not that particular node is attached to a massive storage NAS device.
Defined: qmgr -c 'c r has_nas type=boolean,flag=h'
Usage: qmgr -c 's n worker425 resources_available.has_nas=true'
Access: echo 'hostname' | qsub -l select=1:ncpus=1:has_nas=true
Type: string
Description:
A string resource can contain any value supported by normal string in Python.
Restrictions:
Values are case sensitive.
String resources are never consumable.
Only one of the two quoting characters (" or ') can appear within the string at any given time.
Possible values are:
[_a-zA-Z0-9][[-_a-zA-Z0-9 ! " # $ % ´ ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ' { | } ~] ...]
- This is essentially all characters, including spaces.
Use Cases:
Server level: This could be used for a user readable comment, which could be changed by the admin at any time, which describes which processing environments are available on the cluster that day. As well as information about which environments are shutdown, and their expected length of that downtime. At submission time, a submit hook could read and interpret the string, and determine if the environment requested is available. And, if not, display the string to the user.
Defined: qmgr -c 'create resource environment_motd type=string'
Usage: qmgr -c 'set server resources_available.environment_motd="Available environments: RHEL6, RHEL7|Unavailable: Ubuntu18.4 until 2021/02/10"'
Access: <python> available_environments = [element.strip() for element in pbs.server().resources_available[“environment_motd”].split("|")[0].split(":")[1].split(",")]
Host level: Can be used to set a node into a logical grouping to coincide with a physical group. As an example, say this node resides within the 121st rack in the cluster.
Defined: qmgr -c 'c r cluster_place=string,flag=h'
Usage: qmgr -c 's n worker425 resources_available.custer_place="rack121"'
Access: echo 'sleep 60' | qsub -l select=1:ncpus=1:cluster_place=rack121
Type: long
Description:
A long is an integer value equivalent to the C type "long int". That being a 32-bit number where values are -2,147,483,647 to 2,147,483,647. This type can be consumable.
Restrictions:
None, beyond the bounds of the values stated above.
Use Cases:
Server level: You might use this as part of a hook to track the number of application licenses (app_lic) that are available for jobs in the system at a given time. For example, if another job was running and required the use of one or more of the licenses, then the user job could be held or rejected if there are insufficient resources. Since this will be managed by hooks, we do not want the users to be able to modify it. So, we will set the “r” flag to make it read only.
Defined: qmgr -c 'create resource app_lic type=string,flag=r'
Usage: qmgr -c 'set server resources_available.app_lic=10'
Access: <python> app_lic = pbs.server().resources_available["app_lic"]
Host level: Could be used as a maximum number of licenses that a specific node has for running jobs using a proprietary library. Perhaps the fictional library, "math_cruncher".
Defined: qmgr -c 'c r math_cruncher type=long,flag=h'
Usage: qmgr -c 's n worker425 resources_available.math_cruncher=9000
Access: echo 'hostname' | qsub -l select=1:ncpus=1:math_cruncher=42
Type: size
Description:
A size is a combination of a number and a size identifier. Examples of this would be 6gb for 6 gigabytes, or 24mb for 24 megabytes. This type can be consumable.
Restrictions:
Values must be in the format of #<size abbreviation>, for example 8 megabytes would be 8mb.
The abbreviation is not case sensitive.
Possible values are:
tb=terabytes
gb=gigabytes
mb=megabytes
kb=kilobytes
b=bytes
tw=terawords
gw=gigawords
mw=megawords
kw=kilowords
w=words
Use Cases:
Server level: This could be used to track the overall available drive space available in the entire cluster. Assuming that there is a NAS server connected to the cluster, it would be possible to track the available drive space on that NAS and make that available to the cluster via a server variable, for scheduling. The purpose being that, if a job was to be run that used the NAS for scratch space, then a job could be held or rejected based on their being enough scratch space for the job.
Defined: qmgr -c 'create resource cluster_drive_space type=size'
Usage: qmgr -c 'set server resources_available.cluster_drive_space=14tb'
Access: <python> space = pbs.server().resources_available["cluster_drive_space"]
Host level: Instead of tracking overall cluster drive space, it might be good for an individual node to report its drive space available for users' temporary storage. Then, a user could request nodes that have a least that amount of drive space available.
Defined: qmgr -c 'c r temp_space type=size,flag=h'
Usage: qmgr -c 's n worker425 resources_available.temp_space=5gb'
Access: echo 'hostname' | qsub -l select=1:ncpus=1:temp_space=500mb
Type: float
Description:
Any decimal number, positive or negative. This type can be consumable.
Restrictions:
None other than those imposed on bit width in the system.
Possible values are:
Any positive or negative floating-point number
Use Cases:
Server level: At the server level, the float could be used to modify how much of an effect certain attributes (such as eligible_time) can have on your job_sort_formula. Thus, allowing the admin to change the behavior of the cluster at any point in time, with minimal changes. For the example below, it would be easier to just reset the formula with a different static number. But, if you had a more extensive formula, this would reduce human error on resetting the entire formula.
Defined: qmgr -c 'create resource time_factor type=float,flag=r'
Usage: qmgr -c 's s resources_available.time_factor=3.27'
Access: set server job_sort_formula = ncpus+eligible_time*time_factor
Can be changed at any time by: qmgr -c 's s resources_available.time_factor=1.25'
Host level: At the host level, a similar hook could be created to store the load average for the last minute. That value could be queried and stored for load statistics on the cluster.
Defined: qmgr -c 'c r host_load type=float,flag=hr'
Usage: qmgr -c 's n worker425 resources_available.host_load=0.0'
Access: <python> pbs.event().vnode_list[pbs.get_local_nodename()].resources_available[“host_load”]=1.05
Type: string_array
Description:
A comma separated list (array) of strings. This acts as a normal array within PBS, where each individual value can be matched in the scheduler.
Restrictions:
Values are case sensitive.
String_array resources are never consumable.
Only one of the two quoting characters (" or ') can appear within the string at any given time.
Possible values are:
[_a-zA-Z0-9][[-_a-zA-Z0-9 ! " # $ % ´ ( ) * + - . / : ; < = > ? @ [ \ ] ^ _ ' { | } ~] ...]
- This is essentially all characters, including spaces. Excluding commas as they are a separator character.
Use Cases:
Server level: This could be used in a queuejob hook to determine if a user is in a group which will get a higher priority number added to their jobs. For instance, there are some tests results that are coming up which need to be analyzed. So, anyone in that user group would get a bump in their priority level. But, you still want to run normal jobs if there are enough resources. In this type of situation, a hook can be used to increase a priority for that group by an arbitrary amount.
Defined: qmgr -c 'create resource escalated_user_groups type=string_array'
Usage: qmgr -c 's s resources_available.escalated_user_groups=”mars_rover:5000,solar_studies:300,admins:20000”'
Access: <python> user_groups = str(pbs.server().resources_available["escalated_user_groups"]).split(',')
user_groups = [element.split(':') for element in user_groups]
# The resulting values would be:
user_groups = [['mars_rover','5000'],['solar_studies', '300'],['admins','20000']]
Host level: This can be used to store a list of attributes of the individual hosts. So, if a host has a large amount of memory, is tied to a NAS, is on switch one, and has SSD drives, the resulting string array could be: large_mem,has_nas,switch1,ssd. Then, a user could ask for nodes which have any of these features.
Defined: qmgr -c 'c r node_attr type=string_array,flag=hr'
Usage: qmgr -c 's n worker425 resources_available.node_attr="large_mem,gpus,ssd"'
Access: echo 'hostname' | qsub -l select=1:ncpus=1:node_attr=large_mem,ssd