table of contents
- bookworm 6.1.106-3
- bookworm-backports 6.10.11-1~bpo12+1
- testing 6.10.11-1
- unstable 6.11.2-1
- experimental 6.11-1~exp1
PERF-RECORD(1) | perf Manual | PERF-RECORD(1) |
NAME¶
perf-record - Run a command and record its profile into perf.data
SYNOPSIS¶
perf record [-e <EVENT> | --event=EVENT] [-a] <command> perf record [-e <EVENT> | --event=EVENT] [-a] -- <command> [<options>]
DESCRIPTION¶
This command runs a command and gathers a performance counter profile from it, into perf.data - without displaying anything.
This file can then be inspected later on, using perf report.
OPTIONS¶
<command>...
-e, --event=
where M, N, K are numbers (in decimal, hex, octal format). Acceptable values for each of 'config', 'config1' and 'config2' are defined by corresponding entries in /sys/bus/event_source/devices/<pmu>/format/* param1 and param2 are defined as formats for the PMU in: /sys/bus/event_source/devices/<pmu>/format/*
There are also some parameters which are not defined in .../<pmu>/format/*. These params can be used to overload default config values per event. Here are some common parameters: - 'period': Set event sampling period - 'freq': Set event sampling frequency - 'time': Disable/enable time stamping. Acceptable values are 1 for
enabling time stamping. 0 for disabling time stamping.
The default is 1. - 'call-graph': Disable/enable callgraph. Acceptable str are "fp" for
FP mode, "dwarf" for DWARF mode, "lbr" for LBR mode and
"no" for disable callgraph. - 'stack-size': user stack size for dwarf mode - 'name' : User defined event name. Single quotes (') may be used to
escape symbols in the name from parsing by shell and tool
like this: name=\'CPU_CLK_UNHALTED.THREAD:cmask=0x1\'. - 'aux-output': Generate AUX records instead of events. This requires
that an AUX area event is also provided. - 'aux-sample-size': Set sample size for AUX area sampling. If the '--aux-sample' option has been used, set aux-sample-size=0 to disable AUX area sampling for the event.
See the linkperf:perf-list[1] man page for more parameters.
Note: If user explicitly sets options which conflict with the params, the value set by the parameters will be overridden.
Also not defined in .../<pmu>/format/* are PMU driver specific configuration parameters. Any configuration parameter preceded by the letter '@' is not interpreted in user space and sent down directly to the PMU driver. For example:
perf record -e some_event/@cfg1,@cfg2=config/ ...
will see 'cfg1' and 'cfg2=config' pushed to the PMU driver associated with the event for further processing. There is no restriction on what the configuration parameters are, as long as their semantic is understood and supported by the PMU driver.
--filter=<filter>
In the case of tracepoints, multiple '--filter' options are combined using '&&'.
A hardware trace PMU advertises its ability to accept a number of address filters by specifying a non-zero value in /sys/bus/event_source/devices/<pmu>/nr_addr_filters.
Address filters have the format:
filter|start|stop|tracestop <start> [/ <size>] [@<file name>]
Where: - 'filter': defines a region that will be traced. - 'start': defines an address at which tracing will begin. - 'stop': defines an address at which tracing will stop. - 'tracestop': defines a region in which tracing will stop.
<file name> is the name of the object file, <start> is the offset to the code to trace in that file, and <size> is the size of the region to trace. 'start' and 'stop' filters need not specify a <size>.
If no object file is specified then the kernel is assumed, in which case the start address must be a current kernel memory address.
<start> can also be specified by providing the name of a symbol. If the symbol name is not unique, it can be disambiguated by inserting #n where 'n' selects the n'th symbol in address order. Alternately #0, #g or #G select only a global symbol. <size> can also be specified by providing the name of a symbol, in which case the size is calculated to the end of that symbol. For 'filter' and 'tracestop' filters, if <size> is omitted and <start> is a symbol, then the size is calculated to the end of that symbol.
If <size> is omitted and <start> is '*', then the start and size will be calculated from the first and last symbols, i.e. to trace the whole file.
If symbol names (or '*') are provided, they must be surrounded by white space.
The filter passed to the kernel is not necessarily the same as entered. To see the filter that is passed, use the -v option.
The kernel may not be able to configure a trace region if it is not within a single mapping. MMAP events (or /proc/<pid>/maps) can be examined to determine if that is a possibility.
Multiple filters can be separated with space or comma.
A BPF filter can access the sample data and make a decision based on the data. Users need to set an appropriate sample type to use the BPF filter. BPF filters need root privilege.
The sample data field can be specified in lower case letter. Multiple filters can be separated with comma. For example,
--filter 'period > 1000, cpu == 1' or
--filter 'mem_op == load || mem_op == store, mem_lvl > l1'
The former filter only accept samples with period greater than 1000 AND CPU number is 1. The latter one accepts either load and store memory operations but it should have memory level above the L1. Since the mem_op and mem_lvl fields come from the (memory) data_source, it'd only work with some events which set the data_source field.
Also user should request to collect that information (with -d option in the above case). Otherwise, the following message will be shown.
$ sudo perf record -e cycles --filter 'mem_op == load' Error: cycles event does not have PERF_SAMPLE_DATA_SRC
Hint: please add -d option to perf record. failed to set filter "BPF" on event cycles with 22 (Invalid argument)
Essentially the BPF filter expression is:
<term> <operator> <value> (("," | "||") <term> <operator> <value>)*
The <term> can be one of:
ip, id, tid, pid, cpu, time, addr, period, txn, weight, phys_addr,
code_pgsz, data_pgsz, weight1, weight2, weight3, ins_lat, retire_lat,
p_stage_cyc, mem_op, mem_lvl, mem_snoop, mem_remote, mem_lock,
mem_dtlb, mem_blk, mem_hops
The <operator> can be one of:
==, !=, >, >=, <, <=, &
The <value> can be one of:
<number> (for any term)
na, load, store, pfetch, exec (for mem_op)
l1, l2, l3, l4, cxl, io, any_cache, lfb, ram, pmem (for mem_lvl)
na, none, hit, miss, hitm, fwd, peer (for mem_snoop)
remote (for mem_remote)
na, locked (for mem_locked)
na, l1_hit, l1_miss, l2_hit, l2_miss, any_hit, any_miss, walk, fault (for mem_dtlb)
na, by_data, by_addr (for mem_blk)
hops0, hops1, hops2, hops3 (for mem_hops)
--exclude-perf
-a, --all-cpus
-p, --pid=
-t, --tid=
-u, --uid=
-r, --realtime=
--no-buffering
-c, --count=
-o, --output=
-i, --no-inherit
-F, --freq=
--strict-freq
-m, --mmap-pages=
-g
--call-graph
The unwinding method used for kernel space is dependent on the unwinder used by the active kernel configuration, i.e CONFIG_UNWINDER_FRAME_POINTER (fp) or CONFIG_UNWINDER_ORC (orc)
Any option specified here controls the method used for user space.
Valid options are "fp" (frame pointer), "dwarf" (DWARF's CFI - Call Frame Information) or "lbr" (Hardware Last Branch Record facility).
In some systems, where binaries are build with gcc --fomit-frame-pointer, using the "fp" method will produce bogus call graphs, using "dwarf", if available (perf tools linked to the libunwind or libdw library) should be used instead. Using the "lbr" method doesn't require any compiler options. It will produce call graphs from the hardware LBR registers. The main limitation is that it is only available on new Intel platforms, such as Haswell. It can only get user call chain. It doesn't work with branch stack sampling at the same time.
When "dwarf" recording is used, perf also records (user) stack dump when sampled. Default size of the stack dump is 8192 (bytes). User can change the size by passing the size after comma like "--call-graph dwarf,4096".
When "fp" recording is used, perf tries to save stack enties up to the number specified in sysctl.kernel.perf_event_max_stack by default. User can change the number by passing it after comma like "--call-graph fp,32".
-q, --quiet
-v, --verbose
-s, --stat
-d, --data
--phys-data
--data-page-size
--code-page-size
-T, --timestamp
-P, --period
--sample-cpu
--sample-identifier
-n, --no-samples
-R, --raw-samples
-C, --cpu
User space tasks can migrate between CPUs, so when tracing selected CPUs, a dummy event is created to track sideband for all CPUs.
-B, --no-buildid
-N, --no-buildid-cache
-G name,..., --cgroup name,...
If wanting to monitor, say, cycles for a cgroup and also for system wide, this command line can be used: perf stat -e cycles -G cgroup_name -a -e cycles.
-b, --branch-any
-j, --branch-filter
The option requires at least one branch type among any, any_call, any_ret, ind_call, cond. The privilege levels may be omitted, in which case, the privilege levels of the associated event are applied to the branch filter. Both kernel (k) and hypervisor (hv) privilege levels are subject to permissions. When sampling on multiple events, branch stack sampling is enabled for all the sampling events. The sampled branch type is the same for all events. The various filters must be specified as a comma separated list: --branch-filter any_ret,u,k Note that this feature may not be available on all processors.
-W, --weight
--namespaces
--all-cgroups
--transaction
--per-thread
-D, --delay=
-I, --intr-regs
--user-regs
--running-time
-k, --clockid
-S, --snapshot
In Snapshot Mode trace data is captured only when signal SIGUSR2 is received and on exit if the above e option is given.
--aux-sample[=OPTIONS]
--proc-map-timeout
--switch-events
--vmlinux=PATH
--buildid-all
--buildid-mmap
--aio[=n]
--affinity=mode
--mmap-flush=number
The maximal allowed value is a quarter of the size of mmaped data pages.
The default option value is 1 byte which means that every time that the output writing thread finds some new data in the mmaped buffer the data is extracted, possibly compressed (-z) and written to the output, perf.data or pipe.
Larger data chunks are compressed more effectively in comparison to smaller chunks so extraction of larger chunks from the mmap data pages is preferable from the perspective of output size reduction.
Also at some cases executing less output write syscalls with bigger data size can take less time than executing more output write syscalls with smaller data size thus lowering runtime profiling overhead.
-z, --compression-level[=n]
--all-kernel
--all-user
--kernel-callchains
--user-callchains
Don’t use both --kernel-callchains and --user-callchains at the same time or no callchains will be collected.
--timestamp-filename Append timestamp to output file name.
--timestamp-boundary
--switch-output[=mode]
Note: the precision of the size threshold hugely depends on your configuration - the number and size of your ring buffers (-m). It is generally more precise for higher sizes (like >5M), for lower values expect different sizes.
A possible use case is to, given an external event, slice the perf.data file that gets then processed, possibly via a perf script, to decide if that particular perf.data snapshot should be kept or not.
Implies --timestamp-filename, --no-buildid and --no-buildid-cache. The reason for the latter two is to reduce the data file switching overhead. You can still switch them on with:
--switch-output --no-no-buildid --no-no-buildid-cache
--switch-output-event
Uses the same syntax as --event, it will just not be recorded, serving only to switch the perf.data file as soon as the --switch-output event is processed by a separate sideband thread.
This sideband thread is also used to other purposes, like processing the PERF_RECORD_BPF_EVENT records as they happen, asking the kernel for extra BPF information, etc.
--switch-max-files=N
--dry-run
perf record --dry-run -e can act as a BPF script compiler if llvm.dump-obj in config file is set to true.
--synth=TYPE
Kernel (and some other) events are recorded regardless of the choice in this option. For example, --synth=no would have MMAP events for kernel and modules.
Available types are:
--tail-synthesize
--overwrite
When --overwrite and --switch-output are used perf records and drops events until it receives a signal, meaning that something unusual was detected that warrants taking a snapshot of the most current events, those fitting in the ring buffer at that moment.
overwrite attribute can also be set or canceled for an event using config terms. For example: cycles/overwrite/ and instructions/no-overwrite/.
Implies --tail-synthesize.
--kcore
--max-size=<size>
--num-thread-synthesize
--control=fifo:ctl-fifo[,ack-fifo], --control=fd:ctl-fd[,ack-fd]
Available commands:
-F Show just the sample frequency used for each event. -v Show all fields. -g Show event group information.
Measurements can be started with events disabled using --delay=-1 option. Optionally send control command completion (ack\n) to ack-fd descriptor to synchronize with the controlling process. Example of bash shell script to enable and disable events during measurements:
#!/bin/bash
ctl_dir=/tmp/
ctl_fifo=${ctl_dir}perf_ctl.fifo test -p ${ctl_fifo} && unlink ${ctl_fifo} mkfifo ${ctl_fifo} exec {ctl_fd}<>${ctl_fifo}
ctl_ack_fifo=${ctl_dir}perf_ctl_ack.fifo test -p ${ctl_ack_fifo} && unlink ${ctl_ack_fifo} mkfifo ${ctl_ack_fifo} exec {ctl_fd_ack}<>${ctl_ack_fifo}
perf record -D -1 -e cpu-cycles -a \
--control fd:${ctl_fd},${ctl_fd_ack} \
-- sleep 30 & perf_pid=$!
sleep 5 && echo 'enable' >&${ctl_fd} && read -u ${ctl_fd_ack} e1 && echo "enabled(${e1})" sleep 10 && echo 'disable' >&${ctl_fd} && read -u ${ctl_fd_ack} d1 && echo "disabled(${d1})"
exec {ctl_fd_ack}>&- unlink ${ctl_ack_fifo}
exec {ctl_fd}>&- unlink ${ctl_fifo}
wait -n ${perf_pid} exit $?
--threads=<spec>
<cpus mask 1>/<affinity mask 1>:<cpus mask 2>/<affinity mask 2>:...
CPUs or affinity masks must not overlap with other corresponding masks. Invalid CPUs are ignored, but masks containing only invalid CPUs are not allowed.
For example user specification like the following:
0,2-4/2-4:1,5-7/5-7
specifies parallel threads layout that consists of two threads, the first thread monitors CPUs 0 and 2-4 with the affinity mask 2-4, the second monitors CPUs 1 and 5-7 with the affinity mask 5-7.
<spec> value can also be a string meaning predefined parallel threads layout:
Predefined layouts can be used on systems with large number of CPUs in order not to spawn multiple per-cpu streaming threads but still avoid LOST events in data directory files. Option specified with no or empty value defaults to CPU layout. Masks defined or provided by the option value are filtered through the mask provided by -C option.
--debuginfod[=URLs]
If the URLs is not specified, the value of DEBUGINFOD_URLS system environment variable is used.
--off-cpu
Note that BPF can collect stack traces using frame pointer ("fp") only, as of now. So the applications built without the frame pointer might see bogus addresses.
INTEL HYBRID SUPPORT¶
Support for Intel hybrid events within perf tools.
For some Intel platforms, such as AlderLake, which is hybrid platform and it consists of atom cpu and core cpu. Each cpu has dedicated event list. Part of events are available on core cpu, part of events are available on atom cpu and even part of events are available on both.
Kernel exports two new cpu pmus via sysfs: /sys/devices/cpu_core /sys/devices/cpu_atom
The cpus files are created under the directories. For example,
cat /sys/devices/cpu_core/cpus 0-15
cat /sys/devices/cpu_atom/cpus 16-23
It indicates cpu0-cpu15 are core cpus and cpu16-cpu23 are atom cpus.
As before, use perf-list to list the symbolic event.
perf list
inst_retired.any [Fixed Counter: Counts the number of instructions retired. Unit: cpu_atom] inst_retired.any [Number of instructions retired. Fixed Counter - architectural event. Unit: cpu_core]
The Unit: xxx is added to brief description to indicate which pmu the event is belong to. Same event name but with different pmu can be supported.
Enable hybrid event with a specific pmu
To enable a core only event or atom only event, following syntax is supported:
cpu_core/<event name>/ or
cpu_atom/<event name>/
For example, count the cycles event on core cpus.
perf stat -e cpu_core/cycles/
Create two events for one hardware event automatically
When creating one event and the event is available on both atom and core, two events are created automatically. One is for atom, the other is for core. Most of hardware events and cache events are available on both cpu_core and cpu_atom.
For hardware events, they have pre-defined configs (e.g. 0 for cycles). But on hybrid platform, kernel needs to know where the event comes from (from atom or from core). The original perf event type PERF_TYPE_HARDWARE can’t carry pmu information. So now this type is extended to be PMU aware type. The PMU type ID is stored at attr.config[63:32].
PMU type ID is retrieved from sysfs. /sys/devices/cpu_atom/type /sys/devices/cpu_core/type
The new attr.config layout for PERF_TYPE_HARDWARE:
PERF_TYPE_HARDWARE: 0xEEEEEEEE000000AA AA: hardware event ID EEEEEEEE: PMU type ID
Cache event is similar. The type PERF_TYPE_HW_CACHE is extended to be PMU aware type. The PMU type ID is stored at attr.config[63:32].
The new attr.config layout for PERF_TYPE_HW_CACHE:
PERF_TYPE_HW_CACHE: 0xEEEEEEEE00DDCCBB BB: hardware cache ID CC: hardware cache op ID DD: hardware cache op result ID EEEEEEEE: PMU type ID
When enabling a hardware event without specified pmu, such as, perf stat -e cycles -a (use system-wide in this example), two events are created automatically.
------------------------------------------------------------ perf_event_attr:
size 120
config 0x400000000
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
exclude_guest 1 ------------------------------------------------------------
and
------------------------------------------------------------ perf_event_attr:
size 120
config 0x800000000
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
exclude_guest 1 ------------------------------------------------------------
type 0 is PERF_TYPE_HARDWARE. 0x4 in 0x400000000 indicates it’s cpu_core pmu. 0x8 in 0x800000000 indicates it’s cpu_atom pmu (atom pmu type id is random).
The kernel creates cycles (0x400000000) on cpu0-cpu15 (core cpus), and create cycles (0x800000000) on cpu16-cpu23 (atom cpus).
For perf-stat result, it displays two events:
Performance counter stats for 'system wide':
6,744,979 cpu_core/cycles/ 1,965,552 cpu_atom/cycles/
The first cycles is core event, the second cycles is atom event.
Thread mode example:
perf-stat reports the scaled counts for hybrid event and with a percentage displayed. The percentage is the event’s running time/enabling time.
One example, triad_loop runs on cpu16 (atom core), while we can see the scaled value for core cycles is 160,444,092 and the percentage is 0.47%.
perf stat -e cycles -- taskset -c 16 ./triad_loop
As previous, two events are created.
perf_event_attr:
size 120
config 0x400000000
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
exclude_guest 1
and
perf_event_attr:
size 120
config 0x800000000
sample_type IDENTIFIER
read_format TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
disabled 1
inherit 1
enable_on_exec 1
exclude_guest 1
Performance counter stats for 'taskset -c 16 ./triad_loop':
233,066,666 cpu_core/cycles/ (0.43%) 604,097,080 cpu_atom/cycles/ (99.57%)
perf-record:
If there is no -e specified in perf record, on hybrid platform, it creates two default cycles and adds them to event list. One is for core, the other is for atom.
perf-stat:
If there is no -e specified in perf stat, on hybrid platform, besides of software events, following events are created and added to event list in order.
cpu_core/cycles/, cpu_atom/cycles/, cpu_core/instructions/, cpu_atom/instructions/, cpu_core/branches/, cpu_atom/branches/, cpu_core/branch-misses/, cpu_atom/branch-misses/
Of course, both perf-stat and perf-record support to enable hybrid event with a specific pmu.
e.g. perf stat -e cpu_core/cycles/ perf stat -e cpu_atom/cycles/ perf stat -e cpu_core/r1a/ perf stat -e cpu_atom/L1-icache-loads/ perf stat -e cpu_core/cycles/,cpu_atom/instructions/ perf stat -e {cpu_core/cycles/,cpu_core/instructions/}
But {cpu_core/cycles/,cpu_atom/instructions/} will return warning and disable grouping, because the pmus in group are not matched (cpu_core vs. cpu_atom).
SEE ALSO¶
2024-10-03 | perf |