NAME¶

Libpulp - Userspace Live Patching

INTRODUCTION¶

Libpulp is a framework that enables live patching of userspace processes. In other words, it allows that a running process be modified without restarting the whole application. Libpulp is composed of a library, which provides live patching capabilities to running processes, as well as of a collection of tools to perform live patching operations.

REQUIREMENTS AND RESTRICTIONS¶

The following requirements and restrictions must be met for live patching operations to work:

Process Startup

Live patches cannot be applied to every process in the system. In order for a process to be eligible for live patching, it must dynamically load libpulp.so, the runtime support for live patching operations. Typically, dynamically linked libraries needed by a program are automatically loaded into memory during process initialization. However, since live patching operations are not called by the program itself (but from an external tool), there is no real dependency, and libpulp.so might be missing from DT_NEEDED entries (see ld.so(8)). To bridge this gap, processes should be started with LD_PRELOAD=libpulp.so.

Target Libraries Rebuilding

Not every library in the system is eligible for live patching, only those that have been previously prepared to be so. These are referred to as target libraries. Making a target library does not require changes to its source code, however it requires rebuilding the library with patchable function entries (see -fpatchable-function-entry in gcc(1));

Patch Granularity

Live patches cannot be applied to arbitrarily tiny bits of a program. When a live patch is applied, it replaces entire functions at a time.

Function visibility

Live patches cannot be applied to every function in a process. First, only functions belonging to shared libraries can be live patched; functions in the application itself, or functions linked from static libraries are not eligible. Secondly, only externally visible (GLOBAL or WEAK) functions can be targets of a live patch.

Process Permissions

Libpulp uses ptrace to attach to target processes, thus only the owner of a process (or root) can apply a live patch to it. Moreover, on systems that have Linux Security Modules (LSM) enabled, some extra convincing might be required (see Ptrace access mode checking in ptrace(2)).

TOOLS¶

Creating and applying live patches is achieved with the following tools:

ulp_post(1)

Converts the sequences of one-byte nops at the entry of patchable functions into multi-byte nops.

ulp_packer(1)

Creates a live patch metadata file based on the live patch description file that it takes as argument. The metadata file can later be used by ulp_trigger(1) to actually apply the live patch to a running process. For more information about the description file format, see the METADATA section.

ulp_reverse(1)

Opens the metadata file passed as argument and creates a new metadata file that can be used to reverse a previously applied live patch.

ulp_dump(1)

Parses the metadata file passed as argument and prints its contents in human-readable format.

ulp_trigger(1)

Applies the live patch described by the metadata file received as argument to the process with the specified pid.

ulp_check(1)

Checks if the live patch referred to by the metadata file has already been applied to the process with the specified pid.

ANATOMY OF LIVE PATCHES¶

A live patch is very simple. It is composed of a dynamic shared object (DSO) containing replacement functions, and of a metadata file describing which target library and functions they are meant for. The DSO is no different than a regular shared library, in the sense that it is built from regular source code into a shared object. The metadata is described below.

METADATA¶

A metadata file describes a live patch. It contains three pieces of information:

Path to live patch DSO

Live patching operates on a function-by-function basis (see Patch Granularity). These functions are provided in a dynamic shared object file. The absolute path to this file is recorded in the metadata.

Path to the target library DSO

Each live patch contains replacement functions for a specific library. The absolute path to the in-disk location of the library is recorded in the metadata. Libpulp compares this path against the memory mappings of the target process, as provided by procfs(5), thus, even if the library is removed from disk between the starting of the process and the application of the live patch, the comparison works.

List of replacement functions

Libpulp needs a correlation between original functions in the target library and replacement functions provided by the live patch. This correlation is recorded in the metadata file as a list of pairs of functions.

Description file format

The metadata file is created based on a description file. The description file is rather simple. The first line contains the absolute path to the live patch DSO. The second line starts with the @ character, immediately followed by the absolute path to the target library DSO. Subsequent lines, when not preceded by the # character, provide the list of replacement functions, where each line starts with the name of the original function, followed by a colon, then by the name of the replacement function. Finally, subsequent lines that do start with # specify the offsets that local (not-exported) variables in the target library have from the beginning of the library load location, as well as the offsets of references to those variables in the live patch object. Each line is composed of four items separated by colons: the name of the variable in the target library; the name of a reference to it in the live patch object; the offset of the library variable within the library DSO; the offset of the reference variable within the live patch DSO. These offsets are used by Libpulp to enable access to local variables from the live patch.

For example, a live patch to the math library could have a description file that looked like the following snippet (paths may differ across distributions):

Notice, however, that even though the paths mentioned above refer to files in storage, the patches are not applied to the files themselves. Rather, they are applied to running processes that have loaded these files. See ulp_trigger(1).

EXAMPLES¶

The programs and commands below demonstrate how to use Libpulp.

First, a live patchable library must be created and properly compiled:

Library source

#include <stdlib.h>
#include <string.h>
char *
proverb(void)
{


  int selection;


  char *result;


  char *proverbs[] = {


    "A picture is worth a thousand words",


    "Actions speak louder than words",


    "An apple a day keeps the doctor away",


    "Birds of a feather flock together",


    "Do not judge a book by its cover",


    "Never look a gift horse in the mouth",


    "Practice makes perfect",


    "Slow and steady wins the race",


    "There is no place like home",


    "Too many cooks spoil the broth"


  };


  selection = rand() % (sizeof(proverbs) / sizeof(char *));


  result = strdup(proverbs[selection]);


  return result;
}
    

As explained in the Target Libraries Rebuilding section above, in order to be live patchable, a target library must be built with patchable function entries. Apart from that, it may be optionally post-processed with ulp_post(1):

$ gcc library.c -o library.so \


      -shared -fPIC \


      -fpatchable-function-entry=24,22
$ ulp_post library.so
    

Next, a program that uses the library:

Program source

#include <stdio.h>
#include <unistd.h>
char *proverb(void);
int
main(void)
{


  char buffer[128];


  printf("%d\n", getpid());


  while (fgets(buffer, sizeof(buffer), stdin))


    printf("%s\n", proverb());


  return 0;
}
    

Applications themselves do not require rebuilds, but for the sake of completeness, commands to build an application and link it to a library in a non-default location are shown below:

$ gcc program.c -L$PWD -lrary -Wl,-rpath=$PWD -o program
    

After startup, the program prints its own PID, which will be used further down in this example. Also, hitting ENTER causes the program to call into the library, which replies with a message.

$ LD_PRELOAD=libpulp.so ./program
libpulp loaded...
12345
<ENTER>
Birds of a feather flock together
<ENTER>
An apple a day keeps the doctor away
(and so on...)
    

Next, recall that a live patch can only replace entire functions (see Patch Granularity), thus the following live patch source provides a reimplementation of the proverbs function, giving it a different name to avoid clashes:

Live patch source

#include <string.h>
char *
proverb_v2(void)
{


  return strdup("All good things must come to an end");
}
    

Live patches must be built like shared libraries (notice the use of the -shared option):

$ gcc livepatch.c -shared -fPIC -o livepatch.so
    

Next, recall that a live patch is not only composed of the object created above; it also requires a metadata file, which lets Libpulp know which library the live patch refers to, as well as it provides the correlation between original and replaced functions. A metadata file is built out of a description file.

Description file

/absolute/path/to/livepatch.so
@/absolute/path/to/library.so
proverb:proverb_v2
    

Converting from description to metadata is accomplished with ulp_packer(1):

$ ulp_packer livepatch.dsc -o livepatch.ulp
    

Finally, ulp_trigger(1) can be used to connect to the target process and apply the live patch (note the PID specification, using the -p option):

$ ulp_trigger -p 12345 livepatch.ulp
    

Wrapping up, the target process is now live patched and should behave differently when ENTER is hit in its controlling terminal:

(...)
<ENTER>
All good things must come to an end
<ENTER>
All good things must come to an end
    

SEE ALSO¶

ptrace(2), ulp_packer(1), ulp_reverse(1), ulp_dump(1), ulp_post(1), ulp_trigger(1), ulp_check(1).