NAME¶

SYNOPSIS¶

#include <memkind.h>

Link with -lmemkind

DESCRIPTION¶

HEAP MANAGEMENT:
The functions described in this section define a heap manager with an interface modeled on the ISO C standard API's, except that the user must specify the kind of memory with the first argument to each function. See the KINDS section below for a full description of the implemented kinds. For file-backed kind of memory see memkind_create_pmem().

memkind_malloc() allocates size bytes of uninitialized memory of the specified kind. The allocated space is suitably aligned (after possible pointer coercion) for storage of any type of object. If size is 0, then memkind_malloc() returns NULL.

memkind_calloc() allocates space for num objects each size bytes in length in memory of the specified kind. The result is identical to calling memkind_malloc() with an argument of num*size, with the exception that the allocated memory is explicitly initialized to zero bytes. If num or size is 0, then memkind_calloc() returns NULL.

memkind_realloc() changes the size of the previously allocated memory referenced by ptr to size bytes of the specified kind. The contents of the memory are unchanged up to the lesser of the new and old sizes. If the new size is larger, the contents of the newly allocated portion of the memory are undefined. Upon success, the memory referenced by ptr is freed and a pointer to the newly allocated high bandwidth memory is returned.

Note: memkind_realloc() may move the memory allocation, resulting in a different return value than ptr.

If ptr is NULL, the memkind_realloc() function behaves identically to memkind_malloc() for the specified size. The address ptr, if not NULL, must have been returned by a previous call to memkind_malloc(), memkind_calloc(), memkind_realloc(), or memkind_posix_memalign() with the same kind as specified to the call to memkind_realloc(). Otherwise, if memkind_free(kind, ptr) was called before, undefined behavior occurs.

memkind_posix_memalign() allocates size bytes of memory of a specified kind such that the allocation's base address is an even multiple of alignment, and returns the allocation in the value pointed to by memptr. The requested alignment must be a power of 2 at least as large as sizeof(void *). If size is 0, then memkind_posix_memalign() returns NULL.

memkind_malloc_usable_size() function provides the same semantics as malloc_usable_size(3), but operates on specified kind.

Note: memkind_malloc_usable_size() is not supported by TBB heap manager described in ENVIRONMENT section.

memkind_free() causes the allocated memory referenced by ptr to be made available for future allocations. This pointer must have been returned by a previous call to memkind_malloc(), memkind_calloc(), memkind_realloc(), or memkind_posix_memalign(). Otherwise, if memkind_free(kind, ptr) was already called before, undefined behavior occurs. If ptr is NULL, no operation is performed. The value of MEMKIND_DEFAULT can be given as the kind for all buffers allocated by a kind that leverages the jemalloc allocator. In cases where the kind is unknown in the context of the call to memkind_free() 0 can be given as the kind specified to memkind_free() but this will require a look up that can be bypassed by specifying a non-zero value.

KIND MANAGEMENT:
There are built-in kinds that are always available and these are enumerated in the KINDS section. The user can also create their own kinds of memory. This section describes the API's that enable the tracking of the different kinds of memory and determining their properties.

memkind_create_pmem() is a convenient function used to create a file-backed kind of memory. It allocates a temporary file in the given directory dir. The file is created in a fashion similar to tmpfile(3), so that the file name does not appear when the directory is listed and the space is automatically freed when the program terminates. The file is truncated to a size of max_size bytes and the resulting space is memory-mapped.
Note that the actual file system space is not allocated immediately, but only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This allows to create a pmem memkind of a pretty large size without the need to reserve in advance the corresponding file system space for the entire heap. If the value of max_size equals 0, pmem memkind is only limited by the capacity of the file system mounted under dir argument. The minimum max_size value which allows to limit the size of kind by the library is defined as MEMKIND_PMEM_MIN_SIZE. Calling memkind_create_pmem() with a size smaller than that and different than 0 will return an error. The maximum allowed size is not limited by memkind, but by the file system specified by the dir argument. The max_size passed in is the raw size of the memory pool and jemalloc will use some of that space for its own metadata. Returns zero if the pmem memkind is created successfully or an error code from the ERRORS section if not.

memkind_create_kind() creates kind that allocates memory with specific memory type, memory binding policy and flags (see MEMORY FLAGS section). The memtype_flags (see MEMORY TYPES section) determine memory types to allocate, policy argument is policy for specifying page binding to memory types selected by memtype_flags. Returns zero if the specified kind is created successfully or an error code from the ERRORS section if not.

memkind_destroy_kind() destroys previously created kind object, which must have been returned by a previous call to memkind_create_pmem() or memkind_create_kind(). Otherwise, or if memkind_destroy_kind(kind) was already called before, undefined behavior occurs. Note that, when the kind was returned by memkind_create_kind() all allocated memory must be freed before kind is destroyed, otherwise this will cause memory leak. When the kind was returned by memkind_create_pmem() all allocated memory will be freed after kind will be destroyed.

memkind_check_available() returns zero if the specified kind is available or an error code from the ERRORS section if it is not.

MEMKIND_PMEM_MIN_SIZE The minimum size which allows to limit the file-backed memory partition.

DECORATORS:
The memkind library enables the user to define decorator functions that can be called before and after each memkind heap management API. The decorators that are called at the beginning of the function end are named after that function with _pre appended to the name and those that are called at the end of the function are named after that function with _post appended to the name. These are weak symbols and if they are not present at link time they are not called. The memkind library does not define these symbols which are reserved for user definition. These decorators can be used to track calls to the heap management interface or to modify parameters. The decorators that are called at the beginning of the allocator pass all inputs by reference and the decorators that are called at the end of the allocator pass the output by reference. This enables the modification of the input and output of each heap management function by the decorators.

LIBRARY VERSION:
The memkind library version scheme consist major, minor and patch numbers separated by dot. Combining those numbers, we got the following representation:
major.minor.patch, where:
-major number is incremented whenever API is changed (loss of backward compatibility),
-minor number is incremented whenever additional extensions are introduced or behavior has been changed,
-patch number is incremented whenever small bug fixes are added.

memkind library provide numeric representation of the version by exposing the following API:

memkind_get_version() return version number represented by a single integer number, obtained from the formula:
major * 1000000 + minor * 1000 + patch

Note: major < 1 means unstable API.

API standards:
-STANDARD API, API is considered as stable
-NON-STANDARD API, API is considered as stable, however this is not a standard way to use memkind
-EXPERIMENTAL API, API is considered as unstable and the subject to change

RETURN VALUE¶

KINDS¶

MEMKIND_DEFAULT

Default allocation using standard memory and default page size.

MEMKIND_HUGETLB

Allocate from standard memory using huge pages. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_GBTLB (DEPRECATED)

Allocate from standard memory using 1GB chunks backed by huge pages. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_INTERLEAVE

Allocate pages interleaved across all NUMA nodes with transparent huge pages disabled.

MEMKIND_HBW

Allocate from the closest high bandwidth memory NUMA node at time of allocation. If there is not enough high bandwidth memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL.

MEMKIND_HBW_ALL

Same as MEMKIND_HBW except decision regarding closest NUMA node is postponed until the time of first write.

MEMKIND_HBW_HUGETLB

Same as MEMKIND_HBW except the allocation is backed by huge pages. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_HBW_ALL_HUGETLB

Combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB properties. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_HBW_PREFERRED

Same as MEMKIND_HBW except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory.

MEMKIND_HBW_PREFERRED_HUGETLB

Same as MEMKIND_HBW_PREFERRED except the allocation is backed by huge pages. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_HBW_GBTLB (DEPRECATED)

Same as MEMKIND_HBW except the allocation is backed by 1GB chunks of huge pages. Note that size can take on any value, but full gigabyte pages will allocated for each request, so remainder of the last page will be wasted. This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED)

Same as MEMKIND_HBW_GBTLB except that if there is not enough high bandwidth memory to satisfy the request, the allocation will fall back on standard memory. Note: This kind requires huge pages configuration described in SYSTEM CONFIGURATION section.

MEMKIND_HBW_INTERLEAVE

Same as MEMKIND_HBW except that the pages that support the allocation are interleaved across all high bandwidth nodes and transparent huge pages are disabled.

MEMKIND_REGULAR

Allocate from regular memory using the default page size. Regular means general purpose memory from the NUMA nodes containing CPUs.

MEMORY TYPES¶

MEMKIND_MEMTYPE_DEFAULT

Standard memory, the same as process uses.

MEMKIND_MEMTYPE_HIGH_BANDWIDTH

High bandwidth memory (HBM). There must be at least two memory types with different bandwidth to determine which is the HBM.

MEMORY BINDING POLICY¶

MEMKIND_POLICY_BIND_LOCAL

Allocate local memory. If there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL.

MEMKIND_POLICY_BIND_ALL

Memory locality is ignored. If there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL.

MEMKIND_POLICY_PREFERRED_LOCAL

Allocate preferred memory that is local. If there is not enough preferred memory to satisfy the request or preferred memory is not available, the allocation will fall back on any other memory.

MEMKIND_POLICY_INTERLEAVE_LOCAL

Interleave allocation across local memory. For n memory types the allocation will be interleaved across all of them.

MEMKIND_POLICY_INTERLEAVE_ALL

Interleave allocation. Locality is ignored. For n memory types the allocation will be interleaved across all of them.

MEMKIND_POLICY_MAX_VALUE

Max policy value.

MEMORY FLAGS¶

MEMKIND_MASK_PAGE_SIZE_2MB

Allocation backed by 2MB page size.

ERRORS¶

memkind_posix_memalign()

returns the one of the POSIX standard error codes EINVAL or ENOMEM as defined in <errno.h> if an error occurs (these have positive values). If the alignment parameter is not a power of two or is not a multiple of sizeof(void *), then EINVAL is returned. If there is insufficient memory to satisfy the request then ENOMEM is returned.

All functions other than memkind_posix_memalign() which have an integer return type return one of the negative error codes as defined in <memkind.h> and described below.

MEMKIND_ERROR_UNAVAILABLE

Requested memory kind is not available

MEMKIND_ERROR_MBIND

Call to mbind(2) failed

MEMKIND_ERROR_MMAP

Call to mmap(2) failed

MEMKIND_ERROR_MALLOC

Call to jemalloc's malloc() failed

MEMKIND_ERROR_ENVIRON

Error parsing environment variable (MEMKIND_*)

MEMKIND_ERROR_INVALID

Invalid input arguments to memkind routine

MEMKIND_ERROR_TOOMANY

Error trying to initialize more than maximum MEMKIND_MAX_KIND number of kinds

MEMKIND_ERROR_BADOPS

Error memkind operation structure is missing or invalid

MEMKIND_ERROR_HUGETLB

Unable to allocate huge pages

MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE

Error requested memory type is not available

MEMKIND_ERROR_OPERATION_FAILED

Error memkind operation failed

MEMKIND_ERROR_ARENAS_CREATE

Call to jemalloc's arenas.create() failed

MEMKIND_ERROR_RUNTIME

Unspecified run-time error

FILES¶

/usr/bin/memkind-hbw-nodes

Prints a comma separated list of high bandwidth nodes.

MEMORY TYPES¶

MEMKIND_MEMTYPE_DEFAULT

Standard memory, the same as process uses.

MEMKIND_MEMTYPE_HIGH_BANDWIDTH

High bandwidth memory (HBM). There must be at least two memory types with different bandwidth to determine which is the HBM.

MEMORY BINDING POLICY¶

MEMKIND_POLICY_BIND_LOCAL

Allocate local memory. If there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL.

MEMKIND_POLICY_BIND_ALL

Memory locality is ignored. If there is not enough memory to satisfy the request errno is set to ENOMEM and the allocated pointer is set to NULL.

MEMKIND_POLICY_PREFERRED_LOCAL

Allocate preferred memory that is local. If there is not enough preferred memory to satisfy the request or preferred memory is not available, the allocation will fall back on any other memory.

MEMKIND_POLICY_INTERLEAVE_LOCAL

Interleave allocation across local memory. For n memory types the allocation will be interleaved across all of them.

MEMKIND_POLICY_INTERLEAVE_ALL

Interleave allocation. Locality is ignored. For n memory types the allocation will be interleaved across all of them.

MEMKIND_POLICY_MAX_VALUE

Max policy value.

MEMORY FLAGS¶

MEMKIND_MASK_PAGE_SIZE_2MB

Allocation backed by 2MB page size.

ENVIRONMENT¶

MEMKIND_HBW_NODES

This environment variable is a comma separated list of NUMA nodes that are treated as high bandwidth. Uses the libnuma routine numa_parse_nodestring() for parsing, so the syntax described in the numa(3) man page for this routine applies: e.g. 1-3,5 is a valid setting.

MEMKIND_ARENA_NUM_PER_KIND

This environment variable allows leveraging internal mechanism of the library for setting number of arenas per kind. Value should be a positive integer (not greater than INT_MAX defined in limits.h). The user should set the value based on the characteristics of application that is using the library. Higher value can provide better performance in extremely multithreaded applications at the cost of memory overhead. See section "IMPLEMENTATION NOTES" of jemalloc(3) for more details about arenas.

MEMKIND_HOG_MEMORY

Controls behavior of memkind with regards to returning memory to underlaying OS. Setting MEMKIND_HOG_MEMORY to "1" causes memkind to not release memory to OS in anticipation of memory reuse soon. This will improve latency of 'free' operations but increase memory usage.

MEMKIND_DEBUG

Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to "1" enables printing messages like errors and general information about environment to stderr.

MEMKIND_HEAP_MANAGER

Controls heap management behavior in memkind library by switching to one of the available heap managers.
Values: JEMALLOC – sets the jemalloc heap manager TBB – sets the Intel Threading Building Blocks heap manager. This option requires installed Intel Threading Building Blocks library. If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager will be used by default.

SYSTEM CONFIGURATION¶

HUGETLB (huge pages)

Current number of "persistent" huge pages can be read from /proc/sys/vm/nr_hugepages file. Proposed way of setting hugepages is: "sudo sysctl vm.nr_hugepages=<number_of_hugepages>". More information can be found here: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

STATIC LINKING¶

When linking statically against memkind, libmemkind.a should be used together with its dependencies libnuma and pthread. Pthread can be linked by adding /usr/lib64/libpthread.a as a dependency (exact path may vary). Typically libnuma will need to be compiled from sources to use it as a static dependency. libnuma can be reached on github: https://github.com/numactl/numactl

KNOWN ISSUES¶

HUGETLB (huge pages)

There might be some overhead in huge pages consumption caused by heap management. If your allocation fails because of OOM, please try to allocate extra huge pages (e.g. 8 huge pages).

COPYRIGHT¶

SEE ALSO¶