After the PCP software has been installed on each compute node it should be configured so that data are collected:
The archive data for each node are stored on a shared filesystem for subsequent processing by the job summarization software.
The PCP collection software should have been installed on the compute nodes.
The Job Summarization software includes template files that can be used to configure PCP collection on the compute nodes. The package itself should not be installed on the compute nodes, however you may wish to install it on a test node in order to obtain the PCP configuration file templates. Alternatively, the scripts may be extracted directly from the source tarball.
This script will run data collection at job start time and three additional samples at ten second intervals. This script should be merged into your existing prologue script. This script is designed for and tested with the Slurm resource manager.
This script will run data collection at job end. This script should be merged into your existing epilogue script. This script is designed for and tested with the Slurm resource manager.
This configuration file sets the parameters for process logging into the pcp archives. It prevents the logging of system processes, unless they consume significant resources. This will reduce disk usage by the pcp archives.
$ touch /var/lib/pcp/pmdas/slurm/.NeedInstall $ touch /var/lib/pcp/pmdas/nvidia/.NeedInstall $ touch /var/lib/pcp/pmdas/gpfs/.NeedInstall $ touch /var/lib/pcp/pmdas/nfsclient/.NeedInstall $ touch /var/lib/pcp/pmdas/perfevent/.NeedInstall $ touch /var/lib/pcp/pmdas/mic/.NeedInstall $ touch /var/lib/pcp/pmdas/libvirt/.NeedInstall
By default, PCP does not allow the capture of process information for all users. XDMoD
can display process information only if the pcp user is permitted to log this
information from each compute host. See the relevant documentation in
To enable this, you must add the
-A flag to the
/etc/pcp/pmcd/pmcd.conf like so:
proc 3 pipe binary /var/lib/pcp/pmdas/proc/pmdaproc -A
The daily housekeeping processes that run from cron for PCP
will attempt to do some cleanup that is not necessary when ingesting
the PCP archives into XDMoD. You should add the
-M flag to pmlogger_daily
line in the
/etc/cron.d/pcp-pmlogger file. This will disable the process
that runs daily to combine multiple archives into one file. XDMoD can
handle these files with no problem, and this process uses unnecessary resources.
You may also wish to adjust the retention period for old archives
-k parameter. See
man pmlogger_daily for more information. The
following line will disable the daily rollup and keep archives forever.
10 0 * * * pcp /usr/libexec/pcp/bin/pmlogger_daily -M -k forever
Adjust the retention policy to suit your needs of reprocessing old data.
After making configuration changes to the PMDAs, you will need to restart the pmcd service. On a systemd enabled system, this can be done with:
$ systemctl restart pmcd