NAME¶

pmlogextract - reduce, extract, concatenate and merge Performance Co-Pilot archives

SYNOPSIS¶

pmlogextract [-dfmwxz?] [-c configfile] [-D debug] [-S starttime] [-s samples] [-T endtime] [-V version] [-v volsize] [-Z timezone] input [...] output

DESCRIPTION¶

pmlogextractreads one or more Performance Co-Pilot (PCP) archivesidentified byinputand creates a merged and/or reduced PCP archive inoutput.Eachinputargument is either a name ora comma-separated list of names, and each nameis the name of one file from an archive orthe base name of an archive or the name of a directory containingone or more archives.The nature of merging is controlled by the number ofinputarchives, while the nature of data reduction is controlled bythe command line arguments.Theinputarguments must be archives created bypmlogger(1)with performance data collected from thesamehost, but usually over different time periods and possibly (althoughnot usually) with different performance metrics being logged.

If only oneinputis specified, then the default behavior simply copies theinputPCP archive (with possible conversion to a newerversion of the archive format, see-Vbelow), into theoutputPCP archive.When two or more PCP archives are specified asinput,the archives are merged (or concatenated) and written tooutput.

In theoutputarchive a<mark>record may be inserted at a timejust past the end of each of theinputarchive to indicatea possible temporal discontinuity between the end of oneinputarchive and the start of the nextinputarchive.See theMARK RECORDSsection below for more information.There is no<mark>record after the end of thelast(in temporal order) of the records from theinputarchive(s).

OPTIONS¶

The available command line options are:

-c config, --config=config

Extract only the metrics specified inconfigfrom theinputPCP archive(s).Theconfigsyntax accepted bypmlogextractis explained in more detail in theCONFIGURATION FILE SYNTAXsection.

-d, --desperate

Desperate mode.Normally if a fatal error occurs, all trace ofthe partially written PCP archiveoutputis removed.With the-doption, theoutputarchive is not removed.

-f, --first

For most common uses, all of theinputarchives will have been collected in the same timezone.But if this is not the case, thenpmlogextractmust choose one of the timezones from theinputarchives to beused as the timezone for theoutputarchive.The default is to use the timezone from thelastinputarchive.The-foption forces the timezone from thefirstinputarchive to be used.

-m, --mark

As described in theMARK RECORDSsection below, sometimes it is possible to safely omit<mark>records from theoutputarchive.If the-moption is specified, then theepilogueandprologuetest is skipped and a<mark>record will always be inserted at the end of eachinputarchive (except the last).This is the original behaviour forpmlogextract.

-S starttime, --start=starttime

Define the start of a time window to restrict the records processed;refer toPCPIntro(1).See also the-woption.

-s samples, --samples=samples

The argumentsamplesdefines the number of samples (or records) to be written tooutput.Ifsamplesis 0 or-sis not specified,pmlogextractwill continue until the end of all theinputarchivesor until the end of the time window as specified by-T,whichever comes first.The-soption will override the-Toption if it occurs sooner.

-T endtime, --finish=endtime

Define the end of a time window to restrict the records processed;refer toPCPIntro(1).See also the-woption.

-V version, --outputversion=version

Each PCP archive has a version for the physical record format,currently 2 or 3.By default, theoutputarchive is created with a version equal to themaximumof the version of theinputarchives.The-Voption may be used to explicitly force the version foroutput,providedversionis no smaller than the archive version that would have beenchosen by the default rule.

-v volsize

Theoutputarchive is potentially a multi-volume data set, and the-voption causespmlogextractto start a new volume afterreaching an archive volume size ofvolsize.Ifvolsizeis an integer then at mostvolsizerecords will be written to each volume.Ifvolsizeis an integer suffixed byborbytesthen at mostvolsizebytes will be written out to each volume.Other viable file size units include:K,Kb,KiB,Kbyte,Kilobytefor kilobytes andM,Mb,MiB,Mbyte,Megabytefor megabytes andG,Gb,GiB,Gbyte,Gigabytefor gigabytes.These units may be optionally suffixed by ansand may be of mixed case.Alternativelyvolsizemay be an integer or a floating point number suffixed using a time unitas described inPCPIntro(1)for theintervalargument (to the standard PCP-tcommand line option).

Some examples of different formats:


-s 100
-s 100bytes
-s 100K
-s 100Mb
-s 10Gbyte
-s 10mins
-s 1.5hours


    

-w

Where-Sand-Tspecify a time window within the same day, the-wflag will cause the data within the time window to be extracted,foreveryday in the archive.For example, the options-w -S @11:00 -T @15:00specify thatpmlogextractshould include archive records only for the periods from 11amto 3pm on each day.When-wis used, theoutputarchive will contain<mark>records to indicate the temporaldiscontinuity between the end of one time window and the start ofthe next.

-x

It is expected that the metadata(name, PMID, type, semantics and units)for each metricwill be consistent across all of theinputPCP archive(s) in which that metric appears.In rare cases, e.g. in development, in QA and when a PMDA is upgraded, this may not be the caseandpmlogextractwill report the issue and abort without creating theoutputarchive.This is done so the problem can be fixed withpmlogrewrite(1)before retrying the merge.In unattended or QA environments it may be preferable to force themerge and omit the metrics with the mismatched metadata.The-xoption does this.

-Z timezone, --timezone=timezone

Usetimezonewhen displaying the date and time in diagnostics.Timezoneis in the format of the environment variableTZas described inenviron(7).The default is to initially use the timezone of the local host.

-z, --hostzone

Use the local timezone of the host from theinputarchive(s) when displaying the date and time in diagnostics.The default is to initially use the timezone of the local host.

-?, --help

Display usage message and exit.

CONFIGURATIONFILESYNTAX¶

Theconfigfilecontains metrics of interest - only those metrics (or instances)mentioned explicitly or implicitly in the configuration file will beincluded in theoutputarchive.Each specification must begin on a new line, and may span multiple linesin the configuration file.Instancesmay also be specified, but they are optional.The format for each specification is

        metric
or


        metric [ instance ... ]

wheremetricmay be a leaf or a non-leaf name of a metric in the Performance MetricsName Space (PMNS, seePMNS(5)).If ametricrefers to a non-leaf node in the PMNS,pmlogextractwill recursively descend the PMNS and include all metricscorresponding to descendent leaf nodes.

Instances areoptional and are specified as a list space (or comma) separated ofinstanceidentifiers, with the list enclosed by square brackets.Eachinstanceidentifier may bea number or a string (enclosed in single or double quotes).instanceidentifiers that are numbers are assumed to be internalinstance identifiers, else the string values are assumed to beexternal instance identifiers; seepmGetInDom(3)for more information.If no instances are given, thenallinstances of the associated metric(s) will be extracted.

Any additional white space is ignored andcomments may be added with a `#' prefix.

CONFIGURATIONFILEEXAMPLE¶

This is an example of a validconfigfile:

#
# config file for pmlogextract
#

kernel.all.cpu
kernel.percpu.cpu.sys ["cpu0","cpu1"]
disk.dev ["dks0d1"]

MARKRECORDS¶

When more than oneinputarchive contributes performance data to theoutputarchive, then<mark>records may be inserted to indicate a possibletemporal discontinuity in the performance data.

A<mark>record contains a timestamp and no performance data andis used to indicate that there is a time periodin the PCP archive where we do not know the values ofanyperformance metrics, because there was nopmlogger(1)collecting performance data during this period.Since these periods areoften associated with the restart of a service orpmcd(1)or a system reboot, there may be considerable doubt as to the continuity ofperformance data across this time period.

Most current archives are created with aprologuerecord at the beginning and anepiloguerecord at the end.These records identify the state ofpmcd(1)at the time, and may be used bypmlogextractto determine that there is no discontinuity between the end ofone archive and the next output record, and as a consequence the<mark>record can safely be omitted from theoutputarchive.

The rationale behind<mark>records may be demonstrated with an example.Consider oneinputarchive that starts at 00:10 and ends at 09:15 on thesame day, and anotherinputarchive that starts at 09:20 on thesame day and ends at 00:10 the following morning.This would be a very common case for archives managed and rotated bypmlogger_check(1)andpmlogger_daily(1).

Theoutputarchive created bypmlogextractwould contain:
00:10.000    first record from first input archive
...
09:15.000    last record from first input archive
09:15.001    <mark> record
09:20.000    first record from second input archive
...
01:10.000    last record from second input archive

The time period where the performance data is missing starts just after09:15 and ends just before 09:20.When theoutputarchive is processed with any of the PCP reportingtools, the<mark>record is used to indicate a period of missing data.For example using theoutputarchive above, imagine one was reporting the averageI/O rate at 30 minute intervalsaligned on the hour and half-hour.The I/O count metric is a counter, so theaverage I/O rate requires two valid values fromconsecutive sample times.There would bevalues for all the intervals ending at 09:00,then no values at 09:30 because of the<mark>record, then no values at 10:00 because the ``prior'' value at 09:30 is notavailable, then the rate would be reported again at 10:30 and continueevery 30 minutes until the last reported value at 01:00.

The presence of<mark>records in a PCP archive can be establishedusingpmlogdump(1)where a timestamp and the annotation<mark>is used to indicate a<mark>record.

METADATACHECKS¶

When more than oneinputarchive is specified,pmlogextractperforms a number of checks to ensure the metadata is consistent formetrics appearing in more than one of theinputarchives.These checks include:

• metric data type is the same
• metric semantics are the same
• metric units are the same
• metric is either always singular or always has the same instance domain
• metrics with the same name have the same PMID
• metrics with the same PMID have the same name

If any of these checks fail,pmlogextractreports the details and aborts without creating theoutputarchive.

To address these semantic issues, usepmlogrewrite(1)to translate theinputarchives into equivalent archives with consistent metadata before usingpmlogextract.

Refer to the-xand-dcommand line options above for alternatives to thedefault handling of errors during metadata checks.

CAVEATS¶

Theprologuemetrics (pmcd.pmlogger.archive, pmcd.pmlogger.host,and pmcd.pmlogger.port), which are automatically recorded bypmloggerat the start of the archive, may not be present in the archive output bypmlogextract.These metrics are only relevant while the archive is being created,and have no significance once recording has finished.

DIAGNOSTICS¶

All error conditions detected bypmlogextractare reported onstderrwith textual (if sometimes terse) explanation.

If one of theinputarchives contains no archive records thenan ``empty archive''warning is issued and that archive is skipped.

Should one of theinputarchive(s) be corrupted (this can happenif thepmloggerinstance writing the archive suddenly dies), thenpmlogextractwill detect and report the position of the corruption in the file,and any subsequent information from that archive will not be processed.

If any error is detected,pmlogextractwill exit with a non-zero status.

FILES¶

For each of theinputandoutputarchive, several physical files are used.

archive.meta

metadata (metric descriptions, instance domains, etc.) for the archive

archive.0

initial volume of metrics values (subsequent volumes have suffixes1,2,...) - forinputthese files may have been previously compressed withbzip2(1)orgzip(1)and thus may have an additional.bz2or.gzsuffix.

archive.index

temporal index to support rapid random access to the other files in thearchive.

PCPENVIRONMENT¶

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

For environment variables affecting PCP tools, see pmGetOptions(3).

DEBUGGINGOPTIONS¶

The-Dor--debugoption enables the output of additional diagnostics onstderrto help triage problems, although the information is sometimes cryptic andprimarily intended to provide guidance for developers rather end-users.debugis a comma separated list of debugging options; usepmdbg(1)with the-loption to obtaina list of the available debugging options and their meaning.

Debugging options specific topmloggerare as follows:

SEE ALSO¶

PCPIntro(1), pmlc(1), pmlogdump(1), pmlogger(1), pmlogreduce(1), pmlogrewrite(1), pcp.conf(5), pcp.env(5) and PMNS(5).