NAME¶

argdist - Trace a function and display a histogram or frequency count of its parameter values. Uses Linux eBPF/bcc.

SYNOPSIS¶

argdist [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-d DURATION] [-n COUNT] [-v] [-T TOP] [-H specifier] [-C specifier] [-I header] [-t TID]

DESCRIPTION¶

argdist attaches to function entry and exit points, collects specified parameter values, and stores them in a histogram or a frequency collection that counts the number of times a parameter value occurred. It can also filter parameter values and instrument multiple entry points at once.

Since this uses BPF, only the root user can use this tool.

REQUIREMENTS¶

CONFIG_BPF and bcc.

OPTIONS¶

-h

Print usage message.

-p PID

Trace only functions in the process PID.

-t TID

Trace only functions in the thread TID.

-z STRING_SIZE

When collecting string arguments (of type char*), collect up to STRING_SIZE characters. Longer strings will be truncated.

-i INTERVAL

Print the collected data every INTERVAL seconds. The default is 1 second.

-d DURATION

Total duration of trace in seconds.

-n NUMBER

Print the collected data COUNT times and then exit.

-v

Display the generated BPF program, for debugging purposes.

-T TOP

When collecting frequency counts, display only the top TOP entries.

-H specifiers, -C specifiers

One or more probe specifications that instruct argdist which functions to probe, which parameters to collect, how to aggregate them, and whether to perform any filtering. See SPECIFIER SYNTAX below.

-I header

One or more header files that should be included in the BPF program. This enables the use of structure definitions, enumerations, and constants that are available in these headers. You should provide the same path you would include in the BPF program, e.g. 'linux/blkdev.h' or 'linux/time.h'. Note: in many cases, argdist will deduce the necessary header files automatically.

SPECIFIER SYNTAX¶

The general specifier syntax is as follows:

{p,r,t,u}:{[library],category}:function(signature):type[,type...]:expr[,expr...][:filter]][#label]

{p,r,t,u}

Probe type - "p" for function entry, "r" for function return, "t" for kernel tracepoint, "u" for USDT probe; -H for histogram collection, -C for frequency count. Indicates where to place the probe and whether the probe should collect frequency count information, or aggregate the collected values into a histogram. Counting probes will collect the number of times every parameter value was observed, whereas histogram probes will collect the parameter values into a histogram. Only integral types can be used with histogram probes; there is no such limitation for counting probes.

[library]

Library containing the probe. Specify the full path to the .so or executable file where the function to probe resides. Alternatively, you can specify just the lib name: for example, "c" refers to libc. If no library name is specified, the kernel is assumed.

category

The category of the kernel tracepoint. For example: net, sched, block.

function(signature)

The function to probe, and its signature. The function name must match exactly for the probe to be placed. The signature, on the other hand, is only required if you plan to collect parameter values based on that signature. For example, if you only want to collect the first parameter, you don't have to specify the rest of the parameters in the signature. When capturing kernel tracepoints, this should be the name of the event, e.g. net_dev_start_xmit. The signature for kernel tracepoints should be empty. When capturing USDT probes, this should be the name of the probe, e.g. reloc_complete. The signature for USDT probes should be empty.

[type[,type...]]

The type(s) of the expression(s) to capture. This is the type of the keys in the histogram or raw event collection that are collected by the probes.

[expr[,expr...]]

The expression(s) to capture. These are the values that are assigned to the histogram or raw event collection. You may use the parameters directly, or valid C expressions that involve the parameters, such as "size % 10". Tracepoints may access a special structure called "args" that is formatted according to the tracepoint format (which you can obtain using tplist). For example, the block:block_rq_complete tracepoint can access args->nr_sector. USDT probes may access the arguments defined by the tracing program in the special arg1, arg2, ... variables. To obtain their types, use the tplist tool. Return probes can use the argument values received by the function when it was entered, through the $entry(paramname) special variable. Return probes can also access the function's return value in $retval, and the function's execution time in nanoseconds in $latency. Note that adding the $latency or $entry(paramname) variables to the expression will introduce an additional probe at the function's entry to collect this data, and therefore introduce additional overhead.

[filter]

The filter applied to the captured data. Only parameter values that pass the filter will be collected. This is any valid C expression that refers to the parameter values, such as "fd == 1 && length > 16". The $entry, $retval, and $latency variables can be used here as well, in return probes. The filter expression may also use the STRCMP pseudo-function to compare a predefined string to a string argument. For example: STRCMP("test.txt", file). The order of arguments is important: the first argument MUST be a quoted literal string, and the second argument can be a runtime string.

[label]

The label that will be displayed when printing the probed values. By default, this is the probe specifier.

EXAMPLES¶

Print a histogram of allocation sizes passed to kmalloc:

# argdist -H 'p::__kmalloc(u64 size):u64:size'

Print a count of how many times process 1005 called malloc with an allocation size of 16 bytes:

# argdist -p 1005 -C 'p:c:malloc(size_t size):size_t:size:size==16'

Snoop on all strings returned by gets():

# argdist -C 'r:c:gets():char*:$retval'

Print a histogram of read sizes that were longer than 1ms:

# argdist -H 'r::__vfs_read(void *file, void *buf, size_t count):size_t:$entry(count):$latency > 1000000'

Print frequency counts of how many times writes were issued to a particular file descriptor number, in process 1005:

# argdist -p 1005 -C 'p:c:write(int fd):int:fd'

Print a histogram of error codes returned by read() in process 1005:

# argdist -p 1005 -H 'r:c:read()'

Print a histogram of buffer sizes passed to write() across all processes, where the file descriptor was 1 (STDOUT):

# argdist -H 'p:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'

Count fork() calls in libc across all processes, grouped by pid:

# argdist -C 'p:c:fork():int:$PID;fork per process'

Print histogram of number of sectors in completing block I/O requests:

# argdist -H 't:block:block_rq_complete():u32:nr_sector'

Aggregate interrupts by interrupt request (IRQ):

# argdist -C 't:irq:irq_handler_entry():int:irq'

Print the functions used as thread entry points and how common they are:

# argdist -C 'u:pthread:pthread_start():u64:arg2' -p 1337

Print histograms of sleep() and nanosleep() parameter values:

# argdist -H 'p:c:sleep(u32 seconds):u32:seconds' -H 'p:c:nanosleep(struct timespec *req):long:req->tv_nsec'

Spy on writes to STDOUT performed by process 2780, up to a string size of 120 characters:

# argdist -p 2780 -z 120 -C 'p:c:write(int fd, char* buf, size_t len):char*:buf:fd==1'

Group files being read from and the read sizes from __vfs_read:

# argdist -C 'p::__vfs_read(struct file *file, void *buf, size_t count):char*,size_t:file->f_path.dentry->d_iname,count:file->f_path.dentry->d_iname[0]!=0'

SOURCE¶

This is from bcc.

https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

OS¶

Linux

STABILITY¶

Unstable - in development.

AUTHOR¶

Sasha Goldshtein