How to get the system and user mode cputime of submitted jobs in PBSPro?

Hi Luqman,

The cputime reported in qstat is user plus system cputime. There is no way to breakdown this reported cputime into user and system in qstat.

Hi Jake, 

Thanks for the reply.

As you said, seems like qstat cannot breakdown the cputime. 

However, it was possible to modify the cgroup hook script (pbs_cgroups.PY) and make it gathers the information from cgroup's cpuacct_user and cpuacct_sys. The result will be displayed inside the log files, or you check it using **tracejob <job_id>** command. I don't care even if it is not from qstat.

```
# tracejob 50
[...]
02/19/2024 17:29:36 M update_job_usage: CPU percent: 70
02/19/2024 17:29:36 M update_job_usage: CPU usage: 29.934 secs
02/19/2024 17:29:36 M update_job_usage: CPU usage_sys: 29.851 secs
02/19/2024 17:29:36 M update_job_usage: CPU usage_user: 0.084 secs
02/19/2024 17:29:36 M update_job_usage: Memory usage: mem=1892kb
[...]
```

However, seems like the pbs_cgroups.PY only works with cgroupv1, and it will not work with cgroupv2. Would be appreciated if there is a cgroup hook script for cgroupv2.

Hi Luqman,

I am glad you were able to retrieve the information you needed.

As you have noticed, there is only support for cgroupv1. However, thank you for your feedback here.

As of February 2024, for distro that has started using the cgroupv2 (RHEL9, Ubuntu 22.04, etc), you need to change it to cgroupv1. (Reference: https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_mounting-cgroups-v1_assembly_using-cgroupfs-to-manually-manage-cgroups). 

The hook is executed using pbs_python command, and it you build it using python3.10 and above, the hook will not work. They are still working on how to make python3.10 and above supportable. So, right now, python3.9 is recommended.

As of February 2024, for distro that has started using the cgroupv2 (RHEL9, Ubuntu 22.04, etc), you need to change it to cgroupv1. (Reference: https://access.redhat.com/documentation/ja-jp/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/proc_mounting-cgroups-v1_assembly_using-cgroupfs-to-manually-manage-cgroups). 

The hook is executed using pbs_python command, and it you build it using python3.10 and above, the hook will not work. They are still working on how to make python3.10 and above supportable. So, right now, python3.9 is recommended.

Hi Luqman,

The upcoming release of  PBS Professional in 2024 would have support for  CgroupsV2.
Once the Cgroupsv2 supported version of PBS Professional is released, we will update this channel.