NAME¶

ibv_reg_mr, ibv_reg_mr_iova, ibv_reg_dmabuf_mr, ibv_dereg_mr - register or deregister a memory region (MR)

SYNOPSIS¶

#include <infiniband/verbs.h>

struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr,
                          size_t length, int access);

struct ibv_mr *ibv_reg_mr_iova(struct ibv_pd *pd, void *addr,
                               size_t length, uint64_t hca_va,
                               int access);

struct ibv_mr *ibv_reg_dmabuf_mr(struct ibv_pd *pd, uint64_t offset,
                                 size_t length, uint64_t iova,
                                 int fd, int access);

int ibv_dereg_mr(struct ibv_mr *mr);

DESCRIPTION¶

ibv_reg_mr() registers a memory region (MR) associated with the protection domain pd. The MR's starting address is addr and its size is length. The argument access describes the desired memory protection attributes; it is either 0 or the bitwise OR of one or more of the following flags:

IBV_ACCESS_LOCAL_WRITE

Enable Local Write Access

IBV_ACCESS_REMOTE_WRITE

Enable Remote Write Access

IBV_ACCESS_REMOTE_READ

Enable Remote Read Access

IBV_ACCESS_REMOTE_ATOMIC

Enable Remote Atomic Operation Access (if supported)

IBV_ACCESS_FLUSH_GLOBAL

Enable Remote Flush Operation with global visibility placement type (if supported)

IBV_ACCESS_FLUSH_PERSISTENT

Enable Remote Flush Operation with persistence placement type (if supported)

IBV_ACCESS_MW_BIND

Enable Memory Window Binding

IBV_ACCESS_ZERO_BASED

Use byte offset from beginning of MR to access this MR, instead of a pointer address

IBV_ACCESS_ON_DEMAND

Create an on-demand paging MR

IBV_ACCESS_HUGETLB

Huge pages are guaranteed to be used for this MR, applicable with IBV_ACCESS_ON_DEMAND in explicit mode only

IBV_ACCESS_RELAXED_ORDERING

This setting allows the NIC to relax the order that data is transferred between the network and the target memory region. Relaxed ordering allows network initiated writes (such as incoming message send or RDMA write operations) to reach memory in an arbitrary order. This can improve the performance of some applications. However, relaxed ordering has the following impact: RDMA write-after-write message order is no longer guaranteed. (Send messages will still match posted receive buffers in order.) Back-to-back network writes that target the same memory region leave the region in an unknown state. Relaxed ordering does not change completion semantics, such as data visibility. That is, a completion still ensures that all data is visible, including data from prior transfers. Relaxed ordered operations will also not bypass atomic operations.

If IBV_ACCESS_REMOTE_WRITE or IBV_ACCESS_REMOTE_ATOMIC is set, then IBV_ACCESS_LOCAL_WRITE must be set too.

Local read access is always enabled for the MR.

To create an implicit ODP MR, IBV_ACCESS_ON_DEMAND should be set, addr should be 0 and length should be SIZE_MAX.

If IBV_ACCESS_HUGETLB is set, then application awares that for this MR all pages are huge and must promise it will never do anything to break huge pages.

ibv_reg_mr_iova() ibv_reg_mr_iova is the same as the normal reg_mr, except that the user is allowed to specify the virtual base address of the MR when accessed through a lkey or rkey. The offset in the memory region is computed as 'addr + (iova - hca_va)'. Specifying 0 for hca_va has the same effect as IBV_ACCESS_ZERO_BASED.

ibv_reg_dmabuf_mr() registers a dma-buf based memory region (MR) associated with the protection domain pd. The MR starts at offset of the dma-buf and its size is length. The dma-buf is identified by the file descriptor fd. The argument iova specifies the virtual base address of the MR when accessed through a lkey or rkey. It must have the same page offset as offset. The argument access describes the desired memory protection attributes; it is similar to the ibv_reg_mr case except that only the following flags are supported: IBV_ACCESS_LOCAL_WRITE, IBV_ACCESS_REMOTE_WRITE, IBV_ACCESS_REMOTE_READ, IBV_ACCESS_REMOTE_ATOMIC, IBV_ACCESS_RELAXED_ORDERING.

ibv_dereg_mr() deregisters the MR mr.

RETURN VALUE¶

ibv_reg_mr() / ibv_reg_mr_iova() / ibv_reg_dmabuf_mr() returns a pointer to the registered MR, or NULL if the request fails. The local key (L_Key) field lkey is used as the lkey field of struct ibv_sge when posting buffers with ibv_post_* verbs, and the the remote key (R_Key) field rkey is used by remote processes to perform Atomic and RDMA operations. The remote process places this rkey as the rkey field of struct ibv_send_wr passed to the ibv_post_send function.

ibv_dereg_mr() returns 0 on success, or the value of errno on failure (which indicates the failure reason).

NOTES¶

ibv_dereg_mr() fails if any memory window is still bound to this MR.

SEE ALSO¶

ibv_alloc_pd(3), ibv_post_send(3), ibv_post_recv(3), ibv_post_srq_recv(3)

AUTHORS¶

Dotan Barak <dotanba@gmail.com>