NAME¶

SYNOPSIS¶

DESCRIPTION¶

This data can then be inspected later on, using uftrace replay or uftrace report.

OPTIONS¶

-b SIZE, --buffer=SIZE

Size of internal buffer in which trace data will be saved. Default size is 128k.

-F FUNC, --filter=FUNC

Set filter to trace selected functions only. This option can be used more than once. See FILTERS.

-N FUNC, --notrace=FUNC

Set filter not to trace selected functions (or the functions called underneath them). This option can be used more than once. See FILTERS.

-T TRG, --trigger=TRG

Set trigger on selected functions. This option can be used more than once. See TRIGGERS.

-t TIME, --time-filter=TIME

Do not show functions which run under the time threshold. If some functions explicitly have the 'trace' trigger applied, those are always traced regardless of execution time.

--force

Allow running uftrace even if some problems occur. When uftrace record finds no mcount symbol (which is generated by compiler) in the executable, it quits with an error message since uftrace can not trace the program. However, it is possible that the user is only interested in functions within a dynamically-linked library, in which case this option can be used to cause uftrace to run the program regardless. Also, the -A/--argument and -R/--retval options work only for binaries built with -pg, so uftrace will normally exit when it tries to run binaries built without that option. This option ignores the warning and goes on tracing without the argument and/or return value.

-L PATH, --library-path=PATH

Load necessary internal libraries from this path. This is for testing purposes.

--no-libcall

Do not record library function invocations. Library calls are normally traced by hooking the dynamic linker's resolve function in the PLT. One can disable it with this option.

--no-pltbind

Do not bind dynamic symbol address. This option uses the LD_BIND_NOT environment variable to trace library function calls which might be missing due to concurrent (first) accesses. It is not meaningful to use this option with the --no-libcall option.

-D DEPTH, --depth=DEPTH

Set global trace limit in nesting level.

--max-stack=DEPTH

Set the max function stack depth for tracing. Default is 1024.

--nop

Do not record any functions. This is a no-op and only meaningful for performance comparisons.

--time

Print running time of children in time(1)-style.

-k, --kernel

Trace kernel functions as well as user functions. Only kernel entry/exit functions will be traced by default. Use the --kernel-depth option to override this.

-H HOST, --host=HOST

Send trace data to given host via the network, not writing to files. The uftrace recv command should be run on the destination host to receive the data.

--port=PORT

When sending data to the network (with -H), use the given port instead of the default (8090).

--disable

Start uftrace with tracing disabled. This is only meaningful when used with a trace_on trigger.

--demangle=TYPE

Use demangled C++ symbol names for filters, triggers, arguments and/or return values. Possible values are "full", "simple" and "no". Default is "simple" which ignores function arguments and template parameters.

-A SPEC, --argument=SPEC

Record function arguments. This option can be used more than once. See ARGUMENTS.

-R SPEC, --retval=SPEC

Record function return values. This option can be used more than once. See ARGUMENTS.

--num-thread=NUM

Use NUM threads to record trace data. Default is 1/4 of online CPUs (but when full kernel tracing is enabled, it will use the full number of CPUs).

--libmcount-single

Use single thread version of libmcount for faster recording. This is ignored if the target program calls pthread_create().

--rt-prio=PRIO

Boost priority of recording threads to real-time (FIFO) with priority of PRIO. This is particularly useful for high-volume data such as full kernel tracing.

-K DEPTH, --kernel-depth=DEPTH

Set kernel max function depth separately. Implies --kernel.

--kernel-buffer=SIZE

Set kernel tracing buffer size. The default value (in the kernel) is 1408k. Implies --kernel.

FILTERS¶

The first one is an opt-in filter. By default, it doesn't trace anything. But when one of the specified functions is executed, tracing is started. When the function returns, tracing is stopped again.

For example, consider a simple program which calls a(), b() and c() in turn.

$ cat abc.c
void c(void) {
    /* do nothing */
}
void b(void) {
    c();
}
void a(void) {
    b();
}
int main(void) {
    a();
    return 0;
}
$ gcc -pg -o abc abc.c

    

Normally uftrace will trace all the functions from main() to c().

$ uftrace ./abc
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main() {
            [ 1234] |   a() {
            [ 1234] |     b() {
   3.880 us [ 1234] |       c();
   5.475 us [ 1234] |     } /* b */
   6.448 us [ 1234] |   } /* a */
   8.631 us [ 1234] | } /* main */

    

But when the -F b filter option is used, it will not trace main() or a() but only b() and c().

$ uftrace record -F b ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
            [ 1234] | b() {
   3.880 us [ 1234] |   c();
   5.475 us [ 1234] | } /* b */

    

The second type of filter is opt-out. By default, everything is traced, but when one of the specified functions is executed, tracing stops. When the excluded function returns, tracing is started again.

In the above example, you can omit the function b() and all calls it makes with the -N option.

$ uftrace record -N b ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main() {
   6.448 us [ 1234] |   a();
   8.631 us [ 1234] | } /* main */

    

In addition, you can limit the print nesting level with the -D option.

$ uftrace record -D 3 ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main() {
            [ 1234] |   a() {
   5.475 us [ 1234] |     b();
   6.448 us [ 1234] |   } /* a */
   8.631 us [ 1234] | } /* main */

    

In the above example, uftrace only prints functions up to a depth of 3, so leaf function c() was omitted. Note that the -D option works with -F.

Sometimes it's useful to see long-running functions only. This is good because there are usually many tiny functions that are not interesting. The -t/--time-filter option implements the time-based filter that only records functions which run longer than the given threshold. In the above example, the user might want to see functions running more than 5 microseconds like below:

$ uftrace record -t 5us ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main() {
            [ 1234] |   a() {
   5.475 us [ 1234] |     b();
   6.448 us [ 1234] |   } /* a */
   8.631 us [ 1234] | } /* main */

    

The -t/--time-filter option works for user-level functions only. It does not work for recording kernel functions, but they can be hidden in replay, report, dump and graph commands with -t/--time-filter option.

You can also set triggers on filtered functions. See TRIGGERS section below for details.

When kernel function tracing is enabled, you can also set the filters on kernel functions by marking the symbol with the @kernel modifier. The following example will show all user functions and the (kernel) page fault handler.

$ sudo uftrace -k -F '*page_fault@kernel' ./abc
# DURATION    TID     FUNCTION
           [14721] | main() {
  7.713 us [14721] |   __do_page_fault();
  6.600 us [14721] |   __do_page_fault();
  6.544 us [14721] |   __do_page_fault();
           [14721] |   a() {
           [14721] |     b() {
           [14721] |       c() {
  0.860 us [14721] |         getpid();
  2.346 us [14721] |       } /* c */
  2.956 us [14721] |     } /* b */
  3.340 us [14721] |   } /* a */
 79.086 us [14721] | } /* main */

    

TRIGGERS¶

<trigger>  :=  <symbol> "@" <actions>
<actions>  :=  <action>  | <action> "," <actions>
<action>   :=  "depth="<num> | "trace" | "trace_on" | "trace_off" | "recover"

    

The depth trigger is to change filter depth during execution of the function. It can be used to apply different filter depths for different functions.

The following example shows how triggers work. The global filter maximum depth is 5, but when function b() is called, it is changed to 1, so functions below b() will not shown.

$ uftrace record -D 5 -T 'b@depth=1' ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main() {
            [ 1234] |   a() {
   5.475 us [ 1234] |     b();
   6.448 us [ 1234] |   } /* a */
   8.631 us [ 1234] | } /* main */

    

The backtrace trigger is only meaningful in the replay command.

The traceon and traceoff actions (the _ can be omitted from trace_on and trace_off) control whether uftrace records the specified functions or not.

The 'recover' trigger is for some corner cases in which the process accesses the callstack directly. During tracing of the v8 javascript engine, for example, it kept getting segfaults in the garbage collection stage. It was because v8 incorporates the return address into compiled code objects(?). The recover trigger restores the original return address at the function entry point and resets to the uftrace return hook address again at function exit. I was managed to work around the segfault by setting the recover trigger on the related function (specifically ExitFrame::Iterate).

Triggers only work for user-level functions for now.

ARGUMENTS¶

<argument>    :=  <symbol> "@" <specs>
<specs>       :=  <spec>  | <spec> "," <spec>
<spec>        :=  ( <int_spec> | <float_spec> | <ret_spec> )
<int_spec>    :=  "arg" N [ "/" <format> [ <size> ] ] [ "%" ( <reg> | <stack> ) ]
<float_spec>  :=  "fparg" N [ "/" ( <size> | "80" ) ] [ "%" ( <reg> | <stack> ) ]
<ret_spec>    :=  "retval" [ "/" <format> [ <size> ] ]
<format>      :=  "i" | "u" | "x" | "s" | "c" | "f"
<size>        :=  "8" | "16" | "32" | "64"
<reg>         :=  <arch-specific register name>  # "rdi", "xmm0", "r0", ...
<stack>       :=  "stack" [ "+" ] <offset>

    

The -A/--argument option takes argN where N is an index of the arguments. The index starts from 1 and corresponds to the argument passing order of the calling convention on the system. Note that the indexes of arguments are separately counted for integer (or pointer) and floating-point type, and they can interfere depending on the calling convention. The argN is for integer arguments and fpargN is for floating-point arguments.

Users can optionally specify a format and size for the arguments and/or return values. Without this, uftrace treats them as 'long int' type for integers and 'double' for floating-point numbers. The "i" format makes it signed integer type and "u" format is for unsigned type. Both are printed as decimal while "x" format makes it printed as hexadecimal. The "s" format is for string type and "c" format is for character type. Finally, the "f" format is for floating-point type and is meaningful only for return value (generally). Note that fpargN doesn't take the format field since it's always floating-point.

Please beware when using string type arguments since it can crash the program if the (pointer) value is invalid.

It is also possible to specify a certain register name or stack offset for arguments (but not for return value). The following register names can be used for argument:

•

x86: rdi, rsi, rdx, rcx, r8, r9 (for integer), xmm[0-7] (for floating-point)

•

arm: r[0-3] (for integer), s[0-15] or d[0-7] (for floating-point)

Examples are below:

$ uftrace record -A main@arg1/x -R main@retval/i32 ./abc
$ uftrace replay
# DURATION    TID     FUNCTION
 138.494 us [ 1234] | __cxa_atexit();
            [ 1234] | main(0x1) {
            [ 1234] |   a() {
            [ 1234] |     b() {
   3.880 us [ 1234] |       c();
   5.475 us [ 1234] |     } /* b */
   6.448 us [ 1234] |   } /* a */
   8.631 us [ 1234] | } = 0; /* main */
$ uftrace record -A puts@arg1/s -R puts@retval ./hello
Hello world
$ uftrace replay
# DURATION    TID     FUNCTION
   1.457 us [21534] | __monstartup();
   0.997 us [21534] | __cxa_atexit();
            [21534] | main() {
   7.226 us [21534] |   puts("Hello world") = 12;
   8.708 us [21534] | } /* main */

    

Note that these arguments and return value are recorded only if the executable was built with the -pg option. Executables built with -finstrument-functions will cause uftrace to exit with an error message. Recording of arguments and return values only works with user-level functions for now.

SEE ALSO¶

AUTHORS¶