Using Fairshare from a Role Account

Chuck Fannin
Chuck Fannin
Altair Employee
edited 2023 08 in Altair HPCWorks

Using Fairshare from a Role Account

 

NOTE: When Altair Accelerator™ commands are mentioned in text below, they will be formatted in bold italics (for example, nc why 12345).

Introduction

Fairshare is an algorithm for determining which job a scheduler will run next.  

A fairshare tree is created, and the algorithm uses the tree, combined with workloads currently executing, to determine which job will run next. This is called "ranking."

Each node at the same level of the fairshare tree has equal priority, or weight. Branches can be assigned different weights, but for the initial exercise, equal weighting is assumed.

The job of fairshare is to decide which job to run next. It is not supposed to keep available nodes empty when jobs are waiting. (There are other ways to do that, but they should be used with extreme caution.)

Fairshare and Role Accounts

By default, Accelerator assigns jobs to fairshare nodes by userid, giving all users equal weight unless adjusted otherwise.

It is common for continuous integration systems to run all jobs from a single role account userid. This turns Acellerator into a first come, first serve, scheduler, since all jobs will be assigned to the same node.  If multiple teams and/or projects are using the system, that is unlikely to be the desired outcome.

Fortunately, accelerator also allows for fairshare nodes to be assigned at the nc run command line. 

A solution is available:

Assuming that jobs will be executed for n different groups:

  • Create a fairshare node for each group.
  • Submit jobs to scheduler using -g option on nc run command to assign the job to a specific node.


 

Creating and Using Fairshare Groups

Setting Up Fairshare Groups

  • Only the Accelerator administrator can create/modify the fairshare configuration (the administrator is the account that was used to install/configure Accelerator). Regular users cannot create/modify the fairshare configuration because this would not end well.
  • It is a best practice not to create the new fairshare nodes at the top level of the tree, which leads to less flexibility in the future.
  • To view the existing fairshare tree:
    • nc cmd vovfsgroup show
    • If this command will not execute, you may not be using the administrator account. Do not go any further until this command executes properly.
  • To create a fairshare group:
    • nc cmd vovfsgroup create <group>

Executing Jobs

  • To assign a job to a specific fairshare node:
    • nc run -g <group> <nc options> −− <command to run>

Saving and Restoring Fairshare Configuration

  • To save fairshare configuration:
    • nc cmd vovfsgroup genconfig <swd directory>/fairshare/fsconfig.tcl
  • To restore fairshare configuration:
    • nc cmd vovfsgroup loadconfig <swd directory>/fairshare/fsconfig.tcl
  • While the fairshare configuration can be saved to any location, the Accelerator swd directory is a logical place to keep it.
  • To simplify saving and restoring FS configuration, the commands above can easily be wrapped into scripts. Every time the FS configuration is modified, the configuration can be automatically saved. Restoring the FS config after grid startup is only one command.

 


 

Examples

Examples: Setup and use new Fairshare Groups

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;"># Verify Access to FS Tree and Display Tree</span>

 

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">> nc cmd vovfsgroup show</span>

 <span style="font-family: Consolas; border: 1pt none windowtext; padding: 0in;"># Create new FS Groups</span>

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">> nc cmd vovfsgroup create</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">/groups/groupa</span>

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">> nc cmd vovfsgroup create</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">/groups/groubb</span>

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;"># etc</span>

 

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;"># Use the New FS Tree, by specifying FS group from command line</span>

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">> nc run -g</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">/groups/groupa</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">--</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">sleep</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">60</span>

 

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;"># (Optional) Remove unused/unneeded branches of the fstree:</span>

<span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">> nc cmd vovfsgroup delete</span> <span style="font-size: 10.5pt; font-family: Consolas; color: #333333; border: 1pt none windowtext; padding: 0in;">/groups/groupx</span>