How to get the system and user mode cputime of submitted jobs in PBSPro?

Luqman Hakim
Luqman Hakim Altair Community Member
edited February 26 in Community Q&A

When I use "qstat -a" or "qstat -f", I can get the cputime information, but is it possible to get in detail about kernel and user mode cputime? 

Maybe I have to modify the "cgroup hook" script?

Thank you.

Tagged:

Answers

  • Jake Goldingay
    Jake Goldingay
    Altair Employee
    edited February 19

    Hi Luqman,

    The cputime reported in qstat is user plus system cputime. There is no way to breakdown this reported cputime into user and system in qstat.

  • Luqman Hakim
    Luqman Hakim Altair Community Member
    edited February 20

    Hi Jake, 

    Thanks for the reply.

    As you said, seems like qstat cannot breakdown the cputime. 

    However, it was possible to modify the cgroup hook script (pbs_cgroups.PY) and make it gathers the information from cgroup's cpuacct_user and cpuacct_sys. The result will be displayed inside the log files, or you check it using **tracejob <job_id>** command. I don't care even if it is not from qstat.

    ```
    # tracejob 50
    [...]
    02/19/2024 17:29:36 M update_job_usage: CPU percent: 70
    02/19/2024 17:29:36 M update_job_usage: CPU usage: 29.934 secs
    02/19/2024 17:29:36 M update_job_usage: CPU usage_sys: 29.851 secs
    02/19/2024 17:29:36 M update_job_usage: CPU usage_user: 0.084 secs
    02/19/2024 17:29:36 M update_job_usage: Memory usage: mem=1892kb
    [...]
    ```

     

     

  • Luqman Hakim
    Luqman Hakim Altair Community Member
    edited February 20

    However, seems like the pbs_cgroups.PY only works with cgroupv1, and it will not work with cgroupv2. Would be appreciated if there is a cgroup hook script for cgroupv2.

  • Jake Goldingay
    Jake Goldingay
    Altair Employee
    edited February 20

    Hi Luqman,

    I am glad you were able to retrieve the information you needed.

    As you have noticed, there is only support for cgroupv1. However, thank you for your feedback here.

  • Luqman Hakim
    Luqman Hakim Altair Community Member
    edited February 26

    As of February 2024, for distro that has started using the cgroupv2 (RHEL9, Ubuntu 22.04, etc), you need to change it to cgroupv1. (Reference: https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_mounting-cgroups-v1_assembly_using-cgroupfs-to-manually-manage-cgroups). 

    The hook is executed using pbs_python command, and it you build it using python3.10 and above, the hook will not work. They are still working on how to make python3.10 and above supportable. So, right now, python3.9 is recommended.

  • Adarsh_20887
    Adarsh_20887
    Altair Employee
    edited February 26
    Luqman Hakim said:

    As of February 2024, for distro that has started using the cgroupv2 (RHEL9, Ubuntu 22.04, etc), you need to change it to cgroupv1. (Reference: https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_mounting-cgroups-v1_assembly_using-cgroupfs-to-manually-manage-cgroups). 

    The hook is executed using pbs_python command, and it you build it using python3.10 and above, the hook will not work. They are still working on how to make python3.10 and above supportable. So, right now, python3.9 is recommended.

    Hi Luqman,

    The upcoming release of  PBS Professional in 2024 would have support for  CgroupsV2.
    Once the Cgroupsv2 supported version of PBS Professional is released, we will update this channel.

     

     

Categories