table of contents
- NAME
- CONFIGURING TLS
- COMMON TASKS
- ADVANCED CONNECTIONS
- CONNECTION POOLING
- CURSORS
- BULK WRITE OPERATIONS
- AGGREGATION FRAMEWORK EXAMPLES
- DISTINCT AND MAPREDUCE
- USING LIBMONGOC IN A MICROSOFT VISUAL STUDIO PROJECT
- CREATING INDEXES
- AIDS FOR DEBUGGING
- USING CLIENT-SIDE FIELD LEVEL ENCRYPTION
- AUTHOR
- COPYRIGHT
- bookworm 1.23.1-1
MONGOC_GUIDES(3) | libmongoc | MONGOC_GUIDES(3) |
NAME¶
mongoc_guides - Guides
CONFIGURING TLS¶
Configuration with URI options¶
Enable TLS by including tls=true in the URI.
mongoc_uri_t *uri = mongoc_uri_new ("mongodb://localhost:27017/"); mongoc_uri_set_option_as_bool (uri, MONGOC_URI_TLS, true); mongoc_client_t *client = mongoc_client_new_from_uri (uri);
The following URI options may be used to further configure TLS:
Constant | Key | Description |
MONGOC_URI_TLS | tls | {true|false}, indicating if TLS must be used. |
MONGOC_URI_TLSCERTIFICATEKEYFILE | tlscertificatekeyfile | Path to PEM formatted Private Key, with its Public Certificate concatenated at the end. |
MONGOC_URI_TLSCERTIFICATEKEYFILEPASSWORD | tlscertificatekeypassword | The password, if any, to use to unlock encrypted Private Key. |
MONGOC_URI_TLSCAFILE | tlscafile | One, or a bundle of, Certificate Authorities whom should be considered to be trusted. |
MONGOC_URI_TLSALLOWINVALIDCERTIFICATES | tlsallowinvalidcertificates | Accept and ignore certificate verification errors (e.g. untrusted issuer, expired, etc.) |
MONGOC_URI_TLSALLOWINVALIDHOSTNAMES | tlsallowinvalidhostnames | Ignore hostname verification of the certificate (e.g. Man In The Middle, using valid certificate, but issued for another hostname) |
MONGOC_URI_TLSINSECURE | tlsinsecure | {true|false}, indicating if insecure TLS options should be used. Currently this implies MONGOC_URI_TLSALLOWINVALIDCERTIFICATES and MONGOC_URI_TLSALLOWINVALIDHOSTNAMES. |
MONGOC_URI_TLSDISABLECERTIFICATEREVOCATIONCHECK | tlsdisablecertificaterevocationcheck | {true|false}, indicates if revocation checking (CRL / OCSP) should be disabled. |
MONGOC_URI_TLSDISABLEOCSPENDPOINTCHECK | tlsdisableocspendpointcheck | {true|false}, indicates if OCSP responder endpoints should not be requested when an OCSP response is not stapled. |
Configuration with mongoc_ssl_opt_t¶
Alternatively, the mongoc_ssl_opt_t struct may be used to configure TLS with mongoc_client_set_ssl_opts() or mongoc_client_pool_set_ssl_opts(). Most of the configurable options can be set using the Connection String URI.
mongoc_ssl_opt_t key | URI key |
pem_file | tlsClientCertificateKeyFile |
pem_pwd | tlsClientCertificateKeyPassword |
ca_file | tlsCAFile |
weak_cert_validation | tlsAllowInvalidCertificates |
allow_invalid_hostname | tlsAllowInvalidHostnames |
The only exclusions are crl_file and ca_dir. Those may only be set with mongoc_ssl_opt_t.
Client Authentication¶
When MongoDB is started with TLS enabled, it will by default require the client to provide a client certificate issued by a certificate authority specified by --tlsCAFile, or an authority trusted by the native certificate store in use on the server.
To provide the client certificate, set the tlsCertificateKeyFile in the URI to a PEM armored certificate file.
mongoc_uri_t *uri = mongoc_uri_new ("mongodb://localhost:27017/"); mongoc_uri_set_option_as_bool (uri, MONGOC_URI_TLS, true); mongoc_uri_set_option_as_utf8 (uri, MONGOC_URI_TLSCERTIFICATEKEYFILE, "/path/to/client-certificate.pem"); mongoc_client_t *client = mongoc_client_new_from_uri (uri);
Server Certificate Verification¶
The MongoDB C Driver will automatically verify the validity of the server certificate, such as issued by configured Certificate Authority, hostname validation, and expiration.
To overwrite this behavior, it is possible to disable hostname validation, OCSP endpoint revocation checking, revocation checking entirely, and allow invalid certificates.
This behavior is controlled using the tlsAllowInvalidHostnames, tlsDisableOCSPEndpointCheck, tlsDisableCertificateRevocationCheck, and tlsAllowInvalidCertificates options respectively. By default, all are set to false.
It is not recommended to change these defaults as it exposes the client to Man In The Middle attacks (when tlsAllowInvalidHostnames is set), invalid certificates (when tlsAllowInvalidCertificates is set), or potentially revoked certificates (when tlsDisableOCSPEndpointCheck or tlsDisableCertificateRevocationCheck are set).
Supported Libraries¶
By default, libmongoc will attempt to find a supported TLS library and enable TLS support. This is controlled by the cmake flag ENABLE_SSL, which is set to AUTO by default. Valid values are:
- AUTO the default behavior. Link to the system's native TLS library, or attempt to find OpenSSL.
- DARWIN link to Secure Transport, the native TLS library on macOS.
- WINDOWS link to Secure Channel, the native TLS library on Windows.
- OPENSSL link to OpenSSL (libssl). An optional install path may be specified with OPENSSL_ROOT.
- LIBRESSL link to LibreSSL's libtls. (LibreSSL's compatible libssl may be linked to by setting OPENSSL).
- OFF disable TLS support.
OpenSSL¶
The MongoDB C Driver uses OpenSSL, if available, on Linux and Unix platforms (besides macOS). Industry best practices and some regulations require the use of TLS 1.1 or newer, which requires at least OpenSSL 1.0.1. Check your OpenSSL version like so:
$ openssl version
Ensure your system's OpenSSL is a recent version (at least 1.0.1), or install a recent version in a non-system path and build against it with:
cmake -DOPENSSL_ROOT_DIR=/absolute/path/to/openssl
When compiled against OpenSSL, the driver will attempt to load the system default certificate store, as configured by the distribution. That can be overridden by setting the tlsCAFile URI option or with the fields ca_file and ca_dir in the mongoc_ssl_opt_t.
The Online Certificate Status Protocol (OCSP) (see RFC 6960) is fully supported when using OpenSSL 1.0.1+ with the following notes:
- •
- When a crl_file is set with mongoc_ssl_opt_t, and the crl_file revokes the server's certificate, the certificate is considered revoked (even if the certificate has a valid stapled OCSP response)
LibreSSL / libtls¶
The MongoDB C Driver supports LibreSSL through the use of OpenSSL compatibility checks when configured to compile against openssl. It also supports the new libtls library when configured to build against libressl.
When compiled against the Windows native libraries, the crl_file option of a mongoc_ssl_opt_t is not supported, and will issue an error if used.
Setting tlsDisableOCSPEndpointCheck and tlsDisableCertificateRevocationCheck has no effect.
The Online Certificate Status Protocol (OCSP) (see RFC 6960) is partially supported with the following notes:
- The Must-Staple extension (see RFC 7633) is ignored. Connection may continue if a Must-Staple certificate is presented with no stapled response (unless the client receives a revoked response from an OCSP responder).
- Connection will continue if a Must-Staple certificate is presented without a stapled response and the OCSP responder is down.
Native TLS Support on Windows (Secure Channel)¶
The MongoDB C Driver supports the Windows native TLS library (Secure Channel, or SChannel), and its native crypto library (Cryptography API: Next Generation, or CNG).
When compiled against the Windows native libraries, the ca_dir option of a mongoc_ssl_opt_t is not supported, and will issue an error if used.
Encrypted PEM files (e.g., setting tlsCertificateKeyPassword) are also not supported, and will result in error when attempting to load them.
When tlsCAFile is set, the driver will only allow server certificates issued by the authority (or authorities) provided. When no tlsCAFile is set, the driver will look up the Certificate Authority using the System Local Machine Root certificate store to confirm the provided certificate.
When crl_file is set with mongoc_ssl_opt_t, the driver will import the revocation list to the System Local Machine Root certificate store.
Setting tlsDisableOCSPEndpointCheck has no effect.
The Online Certificate Status Protocol (OCSP) (see RFC 6960) is partially supported with the following notes:
- The Must-Staple extension (see RFC 7633) is ignored. Connection may continue if a Must-Staple certificate is presented with no stapled response (unless the client receives a revoked response from an OCSP responder).
- When a crl_file is set with mongoc_ssl_opt_t, and the crl_file revokes the server's certificate, the OCSP response takes precedence. E.g. if the server presents a certificate with a valid stapled OCSP response, the certificate is considered valid even if the crl_file marks it as revoked.
- Connection will continue if a Must-Staple certificate is presented without a stapled response and the OCSP responder is down.
Native TLS Support on macOS / Darwin (Secure Transport)¶
The MongoDB C Driver supports the Darwin (OS X, macOS, iOS, etc.) native TLS library (Secure Transport), and its native crypto library (Common Crypto, or CC).
When compiled against Secure Transport, the ca_dir and crl_file options of a mongoc_ssl_opt_t are not supported. An error is issued if either are used.
When tlsCAFile is set, the driver will only allow server certificates issued by the authority (or authorities) provided. When no tlsCAFile is set, the driver will use the Certificate Authorities in the currently unlocked keychains.
Setting tlsDisableOCSPEndpointCheck and tlsDisableCertificateRevocationCheck has no effect.
The Online Certificate Status Protocol (OCSP) (see RFC 6960) is partially supported with the following notes.
- The Must-Staple extension (see RFC 7633) is ignored. Connection may continue if a Must-Staple certificate is presented with no stapled response (unless the client receives a revoked response from an OCSP responder).
- Connection will continue if a Must-Staple certificate is presented without a stapled response and the OCSP responder is down.
COMMON TASKS¶
Drivers for some other languages provide helper functions to perform certain common tasks. In the C Driver we must explicitly build commands to send to the server.
Setup¶
First we'll write some code to insert sample data:
doc-common-insert.c
/* Don't try to compile this file on its own. It's meant to be #included
by example code */ /* Insert some sample data */ bool insert_data (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
enum N { ndocs = 4 };
bson_t *docs[ndocs];
bson_error_t error;
int i = 0;
bool ret;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
docs[0] = BCON_NEW ("x", BCON_DOUBLE (1.0), "tags", "[", "dog", "cat", "]");
docs[1] = BCON_NEW ("x", BCON_DOUBLE (2.0), "tags", "[", "cat", "]");
docs[2] = BCON_NEW (
"x", BCON_DOUBLE (2.0), "tags", "[", "mouse", "cat", "dog", "]");
docs[3] = BCON_NEW ("x", BCON_DOUBLE (3.0), "tags", "[", "]");
for (i = 0; i < ndocs; i++) {
mongoc_bulk_operation_insert (bulk, docs[i]);
bson_destroy (docs[i]);
docs[i] = NULL;
}
ret = mongoc_bulk_operation_execute (bulk, NULL, &error);
if (!ret) {
fprintf (stderr, "Error inserting data: %s\n", error.message);
}
mongoc_bulk_operation_destroy (bulk);
return ret; } /* A helper which we'll use a lot later on */ void print_res (const bson_t *reply) {
char *str;
BSON_ASSERT (reply);
str = bson_as_canonical_extended_json (reply, NULL);
printf ("%s\n", str);
bson_free (str); }
explain Command¶
This is how to use the explain command in MongoDB 3.2+:
explain.c
bool explain (mongoc_collection_t *collection) {
bson_t *command;
bson_t reply;
bson_error_t error;
bool res;
command = BCON_NEW ("explain",
"{",
"find",
BCON_UTF8 (COLLECTION_NAME),
"filter",
"{",
"x",
BCON_INT32 (1),
"}",
"}");
res = mongoc_collection_command_simple (
collection, command, NULL, &reply, &error);
if (!res) {
fprintf (stderr, "Error with explain: %s\n", error.message);
goto cleanup;
}
/* Do something with the reply */
print_res (&reply); cleanup:
bson_destroy (&reply);
bson_destroy (command);
return res; }
Running the Examples¶
common-operations.c
/*
* Copyright 2016 MongoDB, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/ #include <mongoc/mongoc.h> #include <stdio.h> const char *COLLECTION_NAME = "things"; #include "../doc-common-insert.c" #include "explain.c" int main (int argc, char *argv[]) {
mongoc_database_t *database = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *collection = NULL;
mongoc_uri_t *uri = NULL;
bson_error_t error;
char *host_and_port;
int res = 0;
if (argc < 2 || argc > 3) {
fprintf (stderr,
"usage: %s MONGOD-1-CONNECTION-STRING "
"[MONGOD-2-HOST-NAME:MONGOD-2-PORT]\n",
argv[0]);
fprintf (stderr,
"MONGOD-1-CONNECTION-STRING can be "
"of the following forms:\n");
fprintf (stderr, "localhost\t\t\t\tlocal machine\n");
fprintf (stderr, "localhost:27018\t\t\t\tlocal machine on port 27018\n");
fprintf (stderr,
"mongodb://user:pass@localhost:27017\t"
"local machine on port 27017, and authenticate with username "
"user and password pass\n");
return EXIT_FAILURE;
}
mongoc_init ();
if (strncmp (argv[1], "mongodb://", 10) == 0) {
host_and_port = bson_strdup (argv[1]);
} else {
host_and_port = bson_strdup_printf ("mongodb://%s", argv[1]);
}
uri = mongoc_uri_new_with_error (host_and_port, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
host_and_port,
error.message);
res = EXIT_FAILURE;
goto cleanup;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
res = EXIT_FAILURE;
goto cleanup;
}
mongoc_client_set_error_api (client, 2);
database = mongoc_client_get_database (client, "test");
collection = mongoc_database_get_collection (database, COLLECTION_NAME);
printf ("Inserting data\n");
if (!insert_data (collection)) {
res = EXIT_FAILURE;
goto cleanup;
}
printf ("explain\n");
if (!explain (collection)) {
res = EXIT_FAILURE;
goto cleanup;
} cleanup:
if (collection) {
mongoc_collection_destroy (collection);
}
if (database) {
mongoc_database_destroy (database);
}
if (client) {
mongoc_client_destroy (client);
}
if (uri) {
mongoc_uri_destroy (uri);
}
bson_free (host_and_port);
mongoc_cleanup ();
return res; }
First launch two separate instances of mongod (must be done from separate shells):
$ mongod
$ mkdir /tmp/db2 $ mongod --dbpath /tmp/db2 --port 27018 # second instance
Now compile and run the example program:
$ cd examples/common_operations/$ gcc -Wall -o example common-operations.c $(pkg-config --cflags --libs libmongoc-1.0)$ ./example localhost:27017 localhost:27018 Inserting data explain {
"executionStats" : {
"allPlansExecution" : [],
"executionStages" : {
"advanced" : 19,
"direction" : "forward" ,
"docsExamined" : 76,
"executionTimeMillisEstimate" : 0,
"filter" : {
"x" : {
"$eq" : 1
}
},
"invalidates" : 0,
"isEOF" : 1,
"nReturned" : 19,
"needTime" : 58,
"needYield" : 0,
"restoreState" : 0,
"saveState" : 0,
"stage" : "COLLSCAN" ,
"works" : 78
},
"executionSuccess" : true,
"executionTimeMillis" : 0,
"nReturned" : 19,
"totalDocsExamined" : 76,
"totalKeysExamined" : 0
},
"ok" : 1,
"queryPlanner" : {
"indexFilterSet" : false,
"namespace" : "test.things",
"parsedQuery" : {
"x" : {
"$eq" : 1
}
},
"plannerVersion" : 1,
"rejectedPlans" : [],
"winningPlan" : {
"direction" : "forward" ,
"filter" : {
"x" : {
"$eq" : 1
}
},
"stage" : "COLLSCAN"
}
},
"serverInfo" : {
"gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25" ,
"host" : "MacBook-Pro-57.local",
"port" : 27017,
"version" : "3.2.6"
} }
ADVANCED CONNECTIONS¶
The following guide contains information specific to certain types of MongoDB configurations.
For an example of connecting to a simple standalone server, see the Tutorial. To establish a connection with authentication options enabled, see the Authentication page.
Connecting to a Replica Set¶
Connecting to a replica set is much like connecting to a standalone MongoDB server. Simply specify the replica set name using the ?replicaSet=myreplset URI option.
#include <bson/bson.h> #include <mongoc/mongoc.h> int main (int argc, char *argv[]) {
mongoc_client_t *client;
mongoc_init ();
/* Create our MongoDB Client */
client = mongoc_client_new (
"mongodb://host01:27017,host02:27017,host03:27017/?replicaSet=myreplset");
/* Do some work */
/* TODO */
/* Clean up */
mongoc_client_destroy (client);
mongoc_cleanup ();
return 0; }
TIP:
It is recommended to use a seed list of members of the replica set to allow the driver to connect to any node.
Connecting to a Sharded Cluster¶
To connect to a sharded cluster, specify the mongos nodes the client should connect to. The C Driver will automatically detect that it has connected to a mongos sharding server.
If more than one hostname is specified, a seed list will be created to attempt failover between the mongos instances.
WARNING:
#include <bson/bson.h> #include <mongoc/mongoc.h> int main (int argc, char *argv[]) {
mongoc_client_t *client;
mongoc_init ();
/* Create our MongoDB Client */
client = mongoc_client_new ("mongodb://myshard01:27017/");
/* Do something with client ... */
/* Free the client */
mongoc_client_destroy (client);
mongoc_cleanup ();
return 0; }
Connecting to an IPv6 Address¶
The MongoDB C Driver will automatically resolve IPv6 addresses from host names. However, to specify an IPv6 address directly, wrap the address in [].
mongoc_uri_t *uri = mongoc_uri_new ("mongodb://[::1]:27017");
Connecting with IPv4 and IPv6¶
If connecting to a hostname that has both IPv4 and IPv6 DNS records, the behavior follows RFC-6555. A connection to the IPv6 address is attempted first. If IPv6 fails, then a connection is attempted to the IPv4 address. If the connection attempt to IPv6 does not complete within 250ms, then IPv4 is tried in parallel. Whichever succeeds connection first cancels the other. The successful DNS result is cached for 10 minutes.
As a consequence, attempts to connect to a mongod only listening on IPv4 may be delayed if there are both A (IPv4) and AAAA (IPv6) DNS records associated with the host.
To avoid a delay, configure hostnames to match the MongoDB configuration. That is, only create an A record if the mongod is only listening on IPv4.
Connecting to a UNIX Domain Socket¶
On UNIX-like systems, the C Driver can connect directly to a MongoDB server using a UNIX domain socket. Pass the URL-encoded path to the socket, which must be suffixed with .sock. For example, to connect to a domain socket at /tmp/mongodb-27017.sock:
mongoc_uri_t *uri = mongoc_uri_new ("mongodb://%2Ftmp%2Fmongodb-27017.sock");
Include username and password like so:
mongoc_uri_t *uri = mongoc_uri_new ("mongodb://user:pass@%2Ftmp%2Fmongodb-27017.sock");
Connecting to a server over TLS¶
These are instructions for configuring TLS/SSL connections.
To run a server locally (on port 27017, for example):
$ mongod --port 27017 --tlsMode requireTLS --tlsCertificateKeyFile server.pem --tlsCAFile ca.pem
Add /?tls=true to the end of a client URI.
mongoc_client_t *client = NULL; client = mongoc_client_new ("mongodb://localhost:27017/?tls=true");
MongoDB requires client certificates by default, unless the --tlsAllowConnectionsWithoutCertificates is provided. The C Driver can be configured to present a client certificate using the URI option tlsCertificateKeyFile, which may be referenced through the constant MONGOC_URI_TLSCERTIFICATEKEYFILE.
mongoc_client_t *client = NULL; mongoc_uri_t *uri = mongoc_uri_new ("mongodb://localhost:27017/?tls=true"); mongoc_uri_set_option_as_utf8 (uri, MONGOC_URI_TLSCERTIFICATEKEYFILE, "client.pem"); client = mongoc_client_new_from_uri (uri);
The client certificate provided by tlsCertificateKeyFile must be issued by one of the server trusted Certificate Authorities listed in --tlsCAFile, or issued by a CA in the native certificate store on the server when omitted.
See Configuring TLS for more information on the various TLS related options.
Compressing data to and from MongoDB¶
MongoDB 3.4 added Snappy compression support, zlib compression in 3.6, and zstd compression in 4.2. To enable compression support the client must be configured with which compressors to use:
mongoc_client_t *client = NULL; client = mongoc_client_new ("mongodb://localhost:27017/?compressors=snappy,zlib,zstd");
The compressors option specifies the priority order of compressors the client wants to use. Messages are compressed if the client and server share any compressors in common.
Note that the compressor used by the server might not be the same compressor as the client used. For example, if the client uses the connection string compressors=zlib,snappy the client will use zlib compression to send data (if possible), but the server might still reply using snappy, depending on how the server was configured.
The driver must be built with zlib and/or snappy and/or zstd support to enable compression support, any unknown (or not compiled in) compressor value will be ignored. Note: to build with zstd requires cmake 3.12 or higher.
Additional Connection Options¶
The full list of connection options can be found in the mongoc_uri_t docs.
Certain socket/connection related options are not configurable:
Option | Description | Value |
SO_KEEPALIVE | TCP Keep Alive | Enabled |
TCP_KEEPIDLE | How long a connection needs to remain idle before TCP starts sending keepalive probes | 120 seconds |
TCP_KEEPINTVL | The time in seconds between TCP probes | 10 seconds |
TCP_KEEPCNT | How many probes to send, without acknowledgement, before dropping the connection | 9 probes |
TCP_NODELAY | Send packets as soon as possible or buffer small packets (Nagle algorithm) | Enabled (no buffering) |
CONNECTION POOLING¶
The MongoDB C driver has two connection modes: single-threaded and pooled. Single-threaded mode is optimized for embedding the driver within languages like PHP. Multi-threaded programs should use pooled mode: this mode minimizes the total connection count, and in pooled mode background threads monitor the MongoDB server topology, so the program need not block to scan it.
Single Mode¶
In single mode, your program creates a mongoc_client_t directly:
mongoc_client_t *client = mongoc_client_new (
"mongodb://hostA,hostB/?replicaSet=my_rs");
The client connects on demand when your program first uses it for a MongoDB operation. Using a non-blocking socket per server, it begins a check on each server concurrently, and uses the asynchronous poll or select function to receive events from the sockets, until all have responded or timed out. Put another way, in single-threaded mode the C Driver fans out to begin all checks concurrently, then fans in once all checks have completed or timed out. Once the scan completes, the client executes your program's operation and returns.
In single mode, the client re-scans the server topology roughly once per minute. If more than a minute has elapsed since the previous scan, the next operation on the client will block while the client completes its scan. This interval is configurable with heartbeatFrequencyMS in the connection string. (See mongoc_uri_t.)
A single client opens one connection per server in your topology: these connections are used both for scanning the topology and performing normal operations.
Pooled Mode¶
To activate pooled mode, create a mongoc_client_pool_t:
mongoc_uri_t *uri = mongoc_uri_new (
"mongodb://hostA,hostB/?replicaSet=my_rs"); mongoc_client_pool_t *pool = mongoc_client_pool_new (uri);
When your program first calls mongoc_client_pool_pop(), the pool launches monitoring threads in the background. Monitoring threads independently connect to all servers in the connection string. As monitoring threads receive hello responses from the servers, they update the shared view of the server topology. Additional monitoring threads and connections are created as new servers are discovered. Monitoring threads are terminated when servers are removed from the shared view of the server topology.
Each thread that executes MongoDB operations must check out a client from the pool:
mongoc_client_t *client = mongoc_client_pool_pop (pool); /* use the client for operations ... */ mongoc_client_pool_push (pool, client);
The mongoc_client_t object is not thread-safe, only the mongoc_client_pool_t is.
When the driver is in pooled mode, your program's operations are unblocked as soon as monitoring discovers a usable server. For example, if a thread in your program is waiting to execute an "insert" on the primary, it is unblocked as soon as the primary is discovered, rather than waiting for all secondaries to be checked as well.
The pool opens one connection per server for monitoring, and each client opens its own connection to each server it uses for application operations. Background monitoring threads re-scan servers independently roughly every 10 seconds. This interval is configurable with heartbeatFrequencyMS in the connection string. (See mongoc_uri_t.)
The connection string can also specify waitQueueTimeoutMS to limit the time that mongoc_client_pool_pop() will wait for a client from the pool. (See mongoc_uri_t.) If waitQueueTimeoutMS is specified, then it is necessary to confirm that a client was actually returned:
mongoc_uri_t *uri = mongoc_uri_new (
"mongodb://hostA,hostB/?replicaSet=my_rs&waitQueueTimeoutMS=1000"); mongoc_client_pool_t *pool = mongoc_client_pool_new (uri); mongoc_client_t *client = mongoc_client_pool_pop (pool); if (client) {
/* use the client for operations ... */
mongoc_client_pool_push (pool, client); } else {
/* take appropriate action for a timeout */ }
See Connection Pool Options to configure pool size and behavior, and see mongoc_client_pool_t for an extended example of a multi-threaded program that uses the driver in pooled mode.
CURSORS¶
Handling Cursor Failures¶
Cursors exist on a MongoDB server. However, the mongoc_cursor_t structure gives the local process a handle to the cursor. It is possible for errors to occur on the server while iterating a cursor on the client. Even a network partition may occur. This means that applications should be robust in handling cursor failures.
While iterating cursors, you should check to see if an error has occurred. See the following example for how to robustly check for errors.
static void print_all_documents (mongoc_collection_t *collection) {
mongoc_cursor_t *cursor;
const bson_t *doc;
bson_error_t error;
bson_t query = BSON_INITIALIZER;
char *str;
cursor = mongoc_collection_find_with_opts (collection, query, NULL, NULL);
while (mongoc_cursor_next (cursor, &doc)) {
str = bson_as_canonical_extended_json (doc, NULL);
printf ("%s\n", str);
bson_free (str);
}
if (mongoc_cursor_error (cursor, &error)) {
fprintf (stderr, "Failed to iterate all documents: %s\n", error.message);
}
mongoc_cursor_destroy (cursor); }
Destroying Server-Side Cursors¶
The MongoDB C driver will automatically destroy a server-side cursor when mongoc_cursor_destroy() is called. Failure to call this function when done with a cursor will leak memory client side as well as consume extra memory server side. If the cursor was configured to never timeout, it will become a memory leak on the server.
Tailable Cursors¶
Tailable cursors are cursors that remain open even after they've returned a final result. This way, if more documents are added to a collection (i.e., to the cursor's result set), then you can continue to call mongoc_cursor_next() to retrieve those additional results.
Here's a complete test case that demonstrates the use of tailable cursors.
NOTE:
An example to tail the oplog from a replica set.
mongoc-tail.c
#include <bson/bson.h> #include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> #ifdef _WIN32 #define sleep(_n) Sleep ((_n) *1000) #endif static void print_bson (const bson_t *b) {
char *str;
str = bson_as_canonical_extended_json (b, NULL);
fprintf (stdout, "%s\n", str);
bson_free (str); } static mongoc_cursor_t * query_collection (mongoc_collection_t *collection, uint32_t last_time) {
mongoc_cursor_t *cursor;
bson_t query;
bson_t gt;
bson_t opts;
BSON_ASSERT (collection);
bson_init (&query);
BSON_APPEND_DOCUMENT_BEGIN (&query, "ts", >);
BSON_APPEND_TIMESTAMP (>, "$gt", last_time, 0);
bson_append_document_end (&query, >);
bson_init (&opts);
BSON_APPEND_BOOL (&opts, "tailable", true);
BSON_APPEND_BOOL (&opts, "awaitData", true);
cursor = mongoc_collection_find_with_opts (collection, &query, &opts, NULL);
bson_destroy (&query);
bson_destroy (&opts);
return cursor; } static void tail_collection (mongoc_collection_t *collection) {
mongoc_cursor_t *cursor;
uint32_t last_time;
const bson_t *doc;
bson_error_t error;
bson_iter_t iter;
BSON_ASSERT (collection);
last_time = (uint32_t) time (NULL);
while (true) {
cursor = query_collection (collection, last_time);
while (!mongoc_cursor_error (cursor, &error) &&
mongoc_cursor_more (cursor)) {
if (mongoc_cursor_next (cursor, &doc)) {
if (bson_iter_init_find (&iter, doc, "ts") &&
BSON_ITER_HOLDS_TIMESTAMP (&iter)) {
bson_iter_timestamp (&iter, &last_time, NULL);
}
print_bson (doc);
}
}
if (mongoc_cursor_error (cursor, &error)) {
if (error.domain == MONGOC_ERROR_SERVER) {
fprintf (stderr, "%s\n", error.message);
exit (1);
}
}
mongoc_cursor_destroy (cursor);
sleep (1);
} } int main (int argc, char *argv[]) {
mongoc_collection_t *collection;
mongoc_client_t *client;
mongoc_uri_t *uri;
bson_error_t error;
if (argc != 2) {
fprintf (stderr, "usage: %s MONGO_URI\n", argv[0]);
return EXIT_FAILURE;
}
mongoc_init ();
uri = mongoc_uri_new_with_error (argv[1], &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
argv[1],
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "local", "oplog.rs");
tail_collection (collection);
mongoc_collection_destroy (collection);
mongoc_uri_destroy (uri);
mongoc_client_destroy (client);
return EXIT_SUCCESS; }
Let's compile and run this example against a replica set to see updates as they are made.
$ gcc -Wall -o mongoc-tail mongoc-tail.c $(pkg-config --cflags --libs libmongoc-1.0) $ ./mongoc-tail mongodb://example.com/?replicaSet=myReplSet {
"h" : -8458503739429355503,
"ns" : "test.test",
"o" : {
"_id" : {
"$oid" : "5372ab0a25164be923d10d50"
}
},
"op" : "i",
"ts" : {
"$timestamp" : {
"i" : 1,
"t" : 1400023818
}
},
"v" : 2 }
The line of output is a sample from performing db.test.insert({}) from the mongo shell on the replica set.
SEE ALSO:
mongoc_cursor_set_max_await_time_ms().
BULK WRITE OPERATIONS¶
This tutorial explains how to take advantage of MongoDB C driver bulk write operation features. Executing write operations in batches reduces the number of network round trips, increasing write throughput.
Bulk Insert¶
First we need to fetch a bulk operation handle from the mongoc_collection_t.
mongoc_bulk_operation_t *bulk =
mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
We can now start inserting documents to the bulk operation. These will be buffered until we execute the operation.
The bulk operation will coalesce insertions as a single batch for each consecutive call to mongoc_bulk_operation_insert(). This creates a pipelined effect when possible.
To execute the bulk operation and receive the result we call mongoc_bulk_operation_execute().
bulk1.c
#include <assert.h> #include <mongoc/mongoc.h> #include <stdio.h> static void bulk1 (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *doc;
bson_t reply;
char *str;
bool ret;
int i;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
for (i = 0; i < 10000; i++) {
doc = BCON_NEW ("i", BCON_INT32 (i));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
}
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
fprintf (stderr, "Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk1-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "test");
bulk1 (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Example reply document:
{"nInserted" : 10000,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"nUpserted" : 0,
"writeErrors" : []
"writeConcernErrors" : [] }
Mixed Bulk Write Operations¶
MongoDB C driver also supports executing mixed bulk write operations. A batch of insert, update, and remove operations can be executed together using the bulk write operations API.
Ordered Bulk Write Operations¶
Ordered bulk write operations are batched and sent to the server in the order provided for serial execution. The reply document describes the type and count of operations performed.
bulk2.c
#include <assert.h> #include <mongoc/mongoc.h> #include <stdio.h> static void bulk2 (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *query;
bson_t *doc;
bson_t *opts;
bson_t reply;
char *str;
bool ret;
int i;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
/* Remove everything */
query = bson_new ();
mongoc_bulk_operation_remove (bulk, query);
bson_destroy (query);
/* Add a few documents */
for (i = 1; i < 4; i++) {
doc = BCON_NEW ("_id", BCON_INT32 (i));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
}
/* {_id: 1} => {$set: {foo: "bar"}} */
query = BCON_NEW ("_id", BCON_INT32 (1));
doc = BCON_NEW ("$set", "{", "foo", BCON_UTF8 ("bar"), "}");
mongoc_bulk_operation_update_many_with_opts (bulk, query, doc, NULL, &error);
bson_destroy (query);
bson_destroy (doc);
/* {_id: 4} => {'$inc': {'j': 1}} (upsert) */
opts = BCON_NEW ("upsert", BCON_BOOL (true));
query = BCON_NEW ("_id", BCON_INT32 (4));
doc = BCON_NEW ("$inc", "{", "j", BCON_INT32 (1), "}");
mongoc_bulk_operation_update_many_with_opts (bulk, query, doc, opts, &error);
bson_destroy (query);
bson_destroy (doc);
bson_destroy (opts);
/* replace {j:1} with {j:2} */
query = BCON_NEW ("j", BCON_INT32 (1));
doc = BCON_NEW ("j", BCON_INT32 (2));
mongoc_bulk_operation_replace_one_with_opts (bulk, query, doc, NULL, &error);
bson_destroy (query);
bson_destroy (doc);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk2-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "test");
bulk2 (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Example reply document:
{ "nInserted" : 3,
"nMatched" : 2,
"nModified" : 2,
"nRemoved" : 10000,
"nUpserted" : 1,
"upserted" : [{"index" : 5, "_id" : 4}],
"writeErrors" : []
"writeConcernErrors" : [] }
The index field in the upserted array is the 0-based index of the upsert operation; in this example, the sixth operation of the overall bulk operation was an upsert, so its index is 5.
Unordered Bulk Write Operations¶
Unordered bulk write operations are batched and sent to the server in arbitrary order where they may be executed in parallel. Any errors that occur are reported after all operations are attempted.
In the next example the first and third operations fail due to the unique constraint on _id. Since we are doing unordered execution the second and fourth operations succeed.
bulk3.c
#include <assert.h> #include <mongoc/mongoc.h> #include <stdio.h> static void bulk3 (mongoc_collection_t *collection) {
bson_t opts = BSON_INITIALIZER;
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *query;
bson_t *doc;
bson_t reply;
char *str;
bool ret;
/* false indicates unordered */
BSON_APPEND_BOOL (&opts, "ordered", false);
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, &opts);
bson_destroy (&opts);
/* Add a document */
doc = BCON_NEW ("_id", BCON_INT32 (1));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
/* remove {_id: 2} */
query = BCON_NEW ("_id", BCON_INT32 (2));
mongoc_bulk_operation_remove_one (bulk, query);
bson_destroy (query);
/* insert {_id: 3} */
doc = BCON_NEW ("_id", BCON_INT32 (3));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
/* replace {_id:4} {'i': 1} */
query = BCON_NEW ("_id", BCON_INT32 (4));
doc = BCON_NEW ("i", BCON_INT32 (1));
mongoc_bulk_operation_replace_one (bulk, query, doc, false);
bson_destroy (query);
bson_destroy (doc);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk);
bson_destroy (&opts); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk3-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "test");
bulk3 (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Example reply document:
{ "nInserted" : 0,
"nMatched" : 1,
"nModified" : 1,
"nRemoved" : 1,
"nUpserted" : 0,
"writeErrors" : [
{ "index" : 0,
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: test.test.$_id_ dup key: { : 1 }" },
{ "index" : 2,
"code" : 11000,
"errmsg" : "E11000 duplicate key error index: test.test.$_id_ dup key: { : 3 }" } ],
"writeConcernErrors" : [] } Error: E11000 duplicate key error index: test.test.$_id_ dup key: { : 1 }
The bson_error_t domain is MONGOC_ERROR_COMMAND and its code is 11000.
Bulk Operation Bypassing Document Validation¶
This feature is only available when using MongoDB 3.2 and later.
By default bulk operations are validated against the schema, if any is defined. In certain cases however it may be necessary to bypass the document validation.
bulk5.c
#include <assert.h> #include <mongoc/mongoc.h> #include <stdio.h> static void bulk5_fail (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *doc;
bson_t reply;
char *str;
bool ret;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
/* Two inserts */
doc = BCON_NEW ("_id", BCON_INT32 (31));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
doc = BCON_NEW ("_id", BCON_INT32 (32));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
/* The above documents do not comply to the schema validation rules
* we created previously, so this will result in an error */
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk); } static void bulk5_success (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *doc;
bson_t reply;
char *str;
bool ret;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
/* Allow this document to bypass document validation.
* NOTE: When authentication is enabled, the authenticated user must have
* either the "dbadmin" or "restore" roles to bypass document validation */
mongoc_bulk_operation_set_bypass_document_validation (bulk, true);
/* Two inserts */
doc = BCON_NEW ("_id", BCON_INT32 (31));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
doc = BCON_NEW ("_id", BCON_INT32 (32));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk); } int main (void) {
bson_t *options;
bson_error_t error;
mongoc_client_t *client;
mongoc_collection_t *collection;
mongoc_database_t *database;
const char *uri_string = "mongodb://localhost/?appname=bulk5-example";
mongoc_uri_t *uri;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
database = mongoc_client_get_database (client, "testasdf");
/* Create schema validator */
options = BCON_NEW (
"validator", "{", "number", "{", "$gte", BCON_INT32 (5), "}", "}");
collection =
mongoc_database_create_collection (database, "collname", options, &error);
if (collection) {
bulk5_fail (collection);
bulk5_success (collection);
mongoc_collection_destroy (collection);
} else {
fprintf (stderr, "Couldn't create collection: '%s'\n", error.message);
}
bson_free (options);
mongoc_uri_destroy (uri);
mongoc_database_destroy (database);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Running the above example will result in:
{ "nInserted" : 0,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"nUpserted" : 0,
"writeErrors" : [
{ "index" : 0,
"code" : 121,
"errmsg" : "Document failed validation" } ] } Error: Document failed validation { "nInserted" : 2,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"nUpserted" : 0,
"writeErrors" : [] }
The bson_error_t domain is MONGOC_ERROR_COMMAND.
Bulk Operation Write Concerns¶
By default bulk operations are executed with the write_concern of the collection they are executed against. A custom write concern can be passed to the mongoc_collection_create_bulk_operation_with_opts() method. Write concern errors (e.g. wtimeout) will be reported after all operations are attempted, regardless of execution order.
bulk4.c
#include <assert.h> #include <mongoc/mongoc.h> #include <stdio.h> static void bulk4 (mongoc_collection_t *collection) {
bson_t opts = BSON_INITIALIZER;
mongoc_write_concern_t *wc;
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *doc;
bson_t reply;
char *str;
bool ret;
wc = mongoc_write_concern_new ();
mongoc_write_concern_set_w (wc, 4);
mongoc_write_concern_set_wtimeout_int64 (wc, 100); /* milliseconds */
mongoc_write_concern_append (wc, &opts);
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, &opts);
/* Two inserts */
doc = BCON_NEW ("_id", BCON_INT32 (10));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
doc = BCON_NEW ("_id", BCON_INT32 (11));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk);
mongoc_write_concern_destroy (wc);
bson_destroy (&opts); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk4-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "test");
bulk4 (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Example reply document and error message:
{ "nInserted" : 2,
"nMatched" : 0,
"nModified" : 0,
"nRemoved" : 0,
"nUpserted" : 0,
"writeErrors" : [],
"writeConcernErrors" : [
{ "code" : 64,
"errmsg" : "waiting for replication timed out" } ] } Error: waiting for replication timed out
The bson_error_t domain is MONGOC_ERROR_WRITE_CONCERN if there are write concern errors and no write errors. Write errors indicate failed operations, so they take precedence over write concern errors, which mean merely that the write concern is not satisfied yet.
Setting Collation Order¶
This feature is only available when using MongoDB 3.4 and later.
bulk-collation.c
#include <mongoc/mongoc.h> #include <stdio.h> static void bulk_collation (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
bson_t *opts;
bson_t *doc;
bson_t *selector;
bson_t *update;
bson_error_t error;
bson_t reply;
char *str;
uint32_t ret;
/* insert {_id: "one"} and {_id: "One"} */
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
doc = BCON_NEW ("_id", BCON_UTF8 ("one"));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
doc = BCON_NEW ("_id", BCON_UTF8 ("One"));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
/* "One" normally sorts before "one"; make "one" come first */
opts = BCON_NEW ("collation",
"{",
"locale",
BCON_UTF8 ("en_US"),
"caseFirst",
BCON_UTF8 ("lower"),
"}");
/* set x=1 on the document with _id "One", which now sorts after "one" */
update = BCON_NEW ("$set", "{", "x", BCON_INT64 (1), "}");
selector = BCON_NEW ("_id", "{", "$gt", BCON_UTF8 ("one"), "}");
mongoc_bulk_operation_update_one_with_opts (
bulk, selector, update, opts, &error);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
bson_destroy (update);
bson_destroy (selector);
bson_destroy (opts);
mongoc_bulk_operation_destroy (bulk); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk-collation";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "db", "collection");
bulk_collation (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
Running the above example will result in:
{ "nInserted" : 2,
"nMatched" : 1,
"nModified" : 1,
"nRemoved" : 0,
"nUpserted" : 0,
"writeErrors" : [ ] }
Unacknowledged Bulk Writes¶
Set "w" to zero for an unacknowledged write. The driver sends unacknowledged writes using the legacy opcodes OP_INSERT, OP_UPDATE, and OP_DELETE.
bulk6.c
#include <mongoc/mongoc.h> #include <stdio.h> static void bulk6 (mongoc_collection_t *collection) {
bson_t opts = BSON_INITIALIZER;
mongoc_write_concern_t *wc;
mongoc_bulk_operation_t *bulk;
bson_error_t error;
bson_t *doc;
bson_t *selector;
bson_t reply;
char *str;
bool ret;
wc = mongoc_write_concern_new ();
mongoc_write_concern_set_w (wc, 0);
mongoc_write_concern_append (wc, &opts);
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, &opts);
doc = BCON_NEW ("_id", BCON_INT32 (10));
mongoc_bulk_operation_insert (bulk, doc);
bson_destroy (doc);
selector = BCON_NEW ("_id", BCON_INT32 (11));
mongoc_bulk_operation_remove_one (bulk, selector);
bson_destroy (selector);
ret = mongoc_bulk_operation_execute (bulk, &reply, &error);
str = bson_as_canonical_extended_json (&reply, NULL);
printf ("%s\n", str);
bson_free (str);
if (!ret) {
printf ("Error: %s\n", error.message);
}
bson_destroy (&reply);
mongoc_bulk_operation_destroy (bulk);
mongoc_write_concern_destroy (wc);
bson_destroy (&opts); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string = "mongodb://localhost/?appname=bulk6-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "test");
bulk6 (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
The reply document is empty:
{ }
Further Reading¶
See the Driver Bulk API Spec, which describes bulk write operations for all MongoDB drivers.
AGGREGATION FRAMEWORK EXAMPLES¶
This document provides a number of practical examples that display the capabilities of the aggregation framework.
The Aggregations using the Zip Codes Data Set examples uses a publicly available data set of all zipcodes and populations in the United States. These data are available at: zips.json.
Requirements¶
Let's check if everything is installed.
Use the following command to load zips.json data set into mongod instance:
$ mongoimport --drop -d test -c zipcodes zips.json
Let's use the MongoDB shell to verify that everything was imported successfully.
$ mongo test connecting to: test > db.zipcodes.count() 29467 > db.zipcodes.findOne() {
"_id" : "35004",
"city" : "ACMAR",
"loc" : [
-86.51557,
33.584132
],
"pop" : 6055,
"state" : "AL" }
Aggregations using the Zip Codes Data Set¶
Each document in this collection has the following form:
{
"_id" : "35004",
"city" : "Acmar",
"state" : "AL",
"pop" : 6055,
"loc" : [-86.51557, 33.584132] }
In these documents:
- The _id field holds the zipcode as a string.
- The city field holds the city name.
- The state field holds the two letter state abbreviation.
- The pop field holds the population.
- The loc field holds the location as a [latitude, longitude] array.
States with Populations Over 10 Million¶
To get all states with a population greater than 10 million, use the following aggregation pipeline:
aggregation1.c
#include <mongoc/mongoc.h> #include <stdio.h> static void print_pipeline (mongoc_collection_t *collection) {
mongoc_cursor_t *cursor;
bson_error_t error;
const bson_t *doc;
bson_t *pipeline;
char *str;
pipeline = BCON_NEW ("pipeline",
"[",
"{",
"$group",
"{",
"_id",
"$state",
"total_pop",
"{",
"$sum",
"$pop",
"}",
"}",
"}",
"{",
"$match",
"{",
"total_pop",
"{",
"$gte",
BCON_INT32 (10000000),
"}",
"}",
"}",
"]");
cursor = mongoc_collection_aggregate (
collection, MONGOC_QUERY_NONE, pipeline, NULL, NULL);
while (mongoc_cursor_next (cursor, &doc)) {
str = bson_as_canonical_extended_json (doc, NULL);
printf ("%s\n", str);
bson_free (str);
}
if (mongoc_cursor_error (cursor, &error)) {
fprintf (stderr, "Cursor Failure: %s\n", error.message);
}
mongoc_cursor_destroy (cursor);
bson_destroy (pipeline); } int main (void) {
mongoc_client_t *client;
mongoc_collection_t *collection;
const char *uri_string =
"mongodb://localhost:27017/?appname=aggregation-example";
mongoc_uri_t *uri;
bson_error_t error;
mongoc_init ();
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
collection = mongoc_client_get_collection (client, "test", "zipcodes");
print_pipeline (collection);
mongoc_uri_destroy (uri);
mongoc_collection_destroy (collection);
mongoc_client_destroy (client);
mongoc_cleanup ();
return EXIT_SUCCESS; }
You should see a result like the following:
{ "_id" : "PA", "total_pop" : 11881643 } { "_id" : "OH", "total_pop" : 10847115 } { "_id" : "NY", "total_pop" : 17990455 } { "_id" : "FL", "total_pop" : 12937284 } { "_id" : "TX", "total_pop" : 16986510 } { "_id" : "IL", "total_pop" : 11430472 } { "_id" : "CA", "total_pop" : 29760021 }
The above aggregation pipeline is build from two pipeline operators: $group and $match.
The $group pipeline operator requires _id field where we specify grouping; remaining fields specify how to generate composite value and must use one of the group aggregation functions: $addToSet, $first, $last, $max, $min, $avg, $push, $sum. The $match pipeline operator syntax is the same as the read operation query syntax.
The $group process reads all documents and for each state it creates a separate document, for example:
{ "_id" : "WA", "total_pop" : 4866692 }
The total_pop field uses the $sum aggregation function to sum the values of all pop fields in the source documents.
Documents created by $group are piped to the $match pipeline operator. It returns the documents with the value of total_pop field greater than or equal to 10 million.
Average City Population by State¶
To get the first three states with the greatest average population per city, use the following aggregation:
pipeline = BCON_NEW ("pipeline", "[",
"{", "$group", "{", "_id", "{", "state", "$state", "city", "$city", "}", "pop", "{", "$sum", "$pop", "}", "}", "}",
"{", "$group", "{", "_id", "$_id.state", "avg_city_pop", "{", "$avg", "$pop", "}", "}", "}",
"{", "$sort", "{", "avg_city_pop", BCON_INT32 (-1), "}", "}",
"{", "$limit", BCON_INT32 (3) "}", "]");
This aggregate pipeline produces:
{ "_id" : "DC", "avg_city_pop" : 303450.0 } { "_id" : "FL", "avg_city_pop" : 27942.29805615551 } { "_id" : "CA", "avg_city_pop" : 27735.341099720412 }
The above aggregation pipeline is build from three pipeline operators: $group, $sort and $limit.
The first $group operator creates the following documents:
{ "_id" : { "state" : "WY", "city" : "Smoot" }, "pop" : 414 }
Note, that the $group operator can't use nested documents except the _id field.
The second $group uses these documents to create the following documents:
{ "_id" : "FL", "avg_city_pop" : 27942.29805615551 }
These documents are sorted by the avg_city_pop field in descending order. Finally, the $limit pipeline operator returns the first 3 documents from the sorted set.
DISTINCT AND MAPREDUCE¶
This document provides some practical, simple, examples to demonstrate the distinct and mapReduce commands.
Setup¶
First we'll write some code to insert sample data:
doc-common-insert.c
/* Don't try to compile this file on its own. It's meant to be #included
by example code */ /* Insert some sample data */ bool insert_data (mongoc_collection_t *collection) {
mongoc_bulk_operation_t *bulk;
enum N { ndocs = 4 };
bson_t *docs[ndocs];
bson_error_t error;
int i = 0;
bool ret;
bulk = mongoc_collection_create_bulk_operation_with_opts (collection, NULL);
docs[0] = BCON_NEW ("x", BCON_DOUBLE (1.0), "tags", "[", "dog", "cat", "]");
docs[1] = BCON_NEW ("x", BCON_DOUBLE (2.0), "tags", "[", "cat", "]");
docs[2] = BCON_NEW (
"x", BCON_DOUBLE (2.0), "tags", "[", "mouse", "cat", "dog", "]");
docs[3] = BCON_NEW ("x", BCON_DOUBLE (3.0), "tags", "[", "]");
for (i = 0; i < ndocs; i++) {
mongoc_bulk_operation_insert (bulk, docs[i]);
bson_destroy (docs[i]);
docs[i] = NULL;
}
ret = mongoc_bulk_operation_execute (bulk, NULL, &error);
if (!ret) {
fprintf (stderr, "Error inserting data: %s\n", error.message);
}
mongoc_bulk_operation_destroy (bulk);
return ret; } /* A helper which we'll use a lot later on */ void print_res (const bson_t *reply) {
char *str;
BSON_ASSERT (reply);
str = bson_as_canonical_extended_json (reply, NULL);
printf ("%s\n", str);
bson_free (str); }
distinct command¶
This is how to use the distinct command to get the distinct values of x which are greater than 1:
distinct.c
bool distinct (mongoc_database_t *database) {
bson_t *command;
bson_t reply;
bson_error_t error;
bool res;
bson_iter_t iter;
bson_iter_t array_iter;
double val;
command = BCON_NEW ("distinct",
BCON_UTF8 (COLLECTION_NAME),
"key",
BCON_UTF8 ("x"),
"query",
"{",
"x",
"{",
"$gt",
BCON_DOUBLE (1.0),
"}",
"}");
res =
mongoc_database_command_simple (database, command, NULL, &reply, &error);
if (!res) {
fprintf (stderr, "Error with distinct: %s\n", error.message);
goto cleanup;
}
/* Do something with reply (in this case iterate through the values) */
if (!(bson_iter_init_find (&iter, &reply, "values") &&
BSON_ITER_HOLDS_ARRAY (&iter) &&
bson_iter_recurse (&iter, &array_iter))) {
fprintf (stderr, "Couldn't extract \"values\" field from response\n");
goto cleanup;
}
while (bson_iter_next (&array_iter)) {
if (BSON_ITER_HOLDS_DOUBLE (&array_iter)) {
val = bson_iter_double (&array_iter);
printf ("Next double: %f\n", val);
}
} cleanup:
/* cleanup */
bson_destroy (command);
bson_destroy (&reply);
return res; }
mapReduce - basic example¶
A simple example using the map reduce framework. It simply adds up the number of occurrences of each "tag".
First define the map and reduce functions:
constants.c
const char *const COLLECTION_NAME = "things"; /* Our map function just emits a single (key, 1) pair for each tag
in the array: */ const char *const MAPPER = "function () {"
"this.tags.forEach(function(z) {"
"emit(z, 1);"
"});"
"}"; /* The reduce function sums over all of the emitted values for a
given key: */ const char *const REDUCER = "function (key, values) {"
"var total = 0;"
"for (var i = 0; i < values.length; i++) {"
"total += values[i];"
"}"
"return total;"
"}"; /* Note We can't just return values.length as the reduce function
might be called iteratively on the results of other reduce
steps. */
Run the mapReduce command. Use the generic command helpers (e.g. mongoc_database_command_simple()). Do not the read command helpers (e.g. mongoc_database_read_command_with_opts()) because they are considered retryable read operations. If retryable reads are enabled, those operations will retry once on a retryable error, giving undesirable behavior for mapReduce.
map-reduce-basic.c
bool map_reduce_basic (mongoc_database_t *database) {
bson_t reply;
bson_t *command;
bool res;
bson_error_t error;
mongoc_cursor_t *cursor;
const bson_t *doc;
bool query_done = false;
const char *out_collection_name = "outCollection";
mongoc_collection_t *out_collection;
/* Empty find query */
bson_t find_query = BSON_INITIALIZER;
/* Construct the mapReduce command */
/* Other arguments can also be specified here, like "query" or
"limit" and so on */
command = BCON_NEW ("mapReduce",
BCON_UTF8 (COLLECTION_NAME),
"map",
BCON_CODE (MAPPER),
"reduce",
BCON_CODE (REDUCER),
"out",
BCON_UTF8 (out_collection_name));
res =
mongoc_database_command_simple (database, command, NULL, &reply, &error);
if (!res) {
fprintf (stderr, "MapReduce failed: %s\n", error.message);
goto cleanup;
}
/* Do something with the reply (it doesn't contain the mapReduce results) */
print_res (&reply);
/* Now we'll query outCollection to see what the results are */
out_collection =
mongoc_database_get_collection (database, out_collection_name);
cursor = mongoc_collection_find_with_opts (
out_collection, &find_query, NULL, NULL);
query_done = true;
/* Do something with the results */
while (mongoc_cursor_next (cursor, &doc)) {
print_res (doc);
}
if (mongoc_cursor_error (cursor, &error)) {
fprintf (stderr, "ERROR: %s\n", error.message);
res = false;
goto cleanup;
} cleanup:
/* cleanup */
if (query_done) {
mongoc_cursor_destroy (cursor);
mongoc_collection_destroy (out_collection);
}
bson_destroy (&reply);
bson_destroy (command);
return res; }
mapReduce - more complicated example¶
You must have replica set running for this.
In this example we contact a secondary in the replica set and do an "inline" map reduce, so the results are returned immediately:
map-reduce-advanced.c
bool map_reduce_advanced (mongoc_database_t *database) {
bson_t *command;
bson_error_t error;
bool res = true;
mongoc_cursor_t *cursor;
mongoc_read_prefs_t *read_pref;
const bson_t *doc;
/* Construct the mapReduce command */
/* Other arguments can also be specified here, like "query" or "limit"
and so on */
/* Read the results inline from a secondary replica */
command = BCON_NEW ("mapReduce",
BCON_UTF8 (COLLECTION_NAME),
"map",
BCON_CODE (MAPPER),
"reduce",
BCON_CODE (REDUCER),
"out",
"{",
"inline",
"1",
"}");
read_pref = mongoc_read_prefs_new (MONGOC_READ_SECONDARY);
cursor = mongoc_database_command (
database, MONGOC_QUERY_NONE, 0, 0, 0, command, NULL, read_pref);
/* Do something with the results */
while (mongoc_cursor_next (cursor, &doc)) {
print_res (doc);
}
if (mongoc_cursor_error (cursor, &error)) {
fprintf (stderr, "ERROR: %s\n", error.message);
res = false;
}
mongoc_cursor_destroy (cursor);
mongoc_read_prefs_destroy (read_pref);
bson_destroy (command);
return res; }
Running the Examples¶
Here's how to run the example code
basic-aggregation.c
/*
* Copyright 2016 MongoDB, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/ #include <mongoc/mongoc.h> #include <stdio.h> #include "constants.c" #include "../doc-common-insert.c" #include "distinct.c" #include "map-reduce-basic.c" #include "map-reduce-advanced.c" int main (int argc, char *argv[]) {
mongoc_database_t *database = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *collection = NULL;
mongoc_uri_t *uri = NULL;
bson_error_t error;
char *host_and_port = NULL;
int exit_code = EXIT_FAILURE;
if (argc != 2) {
fprintf (stderr, "usage: %s CONNECTION-STRING\n", argv[0]);
fprintf (stderr,
"the connection string can be of the following forms:\n");
fprintf (stderr, "localhost\t\t\t\tlocal machine\n");
fprintf (stderr, "localhost:27018\t\t\t\tlocal machine on port 27018\n");
fprintf (stderr,
"mongodb://user:pass@localhost:27017\t"
"local machine on port 27017, and authenticate with username "
"user and password pass\n");
return exit_code;
}
mongoc_init ();
if (strncmp (argv[1], "mongodb://", 10) == 0) {
host_and_port = bson_strdup (argv[1]);
} else {
host_and_port = bson_strdup_printf ("mongodb://%s", argv[1]);
}
uri = mongoc_uri_new_with_error (host_and_port, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
host_and_port,
error.message);
goto cleanup;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
goto cleanup;
}
mongoc_client_set_error_api (client, 2);
database = mongoc_client_get_database (client, "test");
collection = mongoc_database_get_collection (database, COLLECTION_NAME);
printf ("Inserting data\n");
if (!insert_data (collection)) {
goto cleanup;
}
printf ("distinct\n");
if (!distinct (database)) {
goto cleanup;
}
printf ("map reduce\n");
if (!map_reduce_basic (database)) {
goto cleanup;
}
printf ("more complicated map reduce\n");
if (!map_reduce_advanced (database)) {
goto cleanup;
}
exit_code = EXIT_SUCCESS; cleanup:
if (collection) {
mongoc_collection_destroy (collection);
}
if (database) {
mongoc_database_destroy (database);
}
if (client) {
mongoc_client_destroy (client);
}
if (uri) {
mongoc_uri_destroy (uri);
}
if (host_and_port) {
bson_free (host_and_port);
}
mongoc_cleanup ();
return exit_code; }
If you want to try the advanced map reduce example with a secondary, start a replica set (instructions for how to do this can be found here).
Otherwise, just start an instance of MongoDB:
$ mongod
Now compile and run the example program:
$ cd examples/basic_aggregation/ $ gcc -Wall -o agg-example basic-aggregation.c $(pkg-config --cflags --libs libmongoc-1.0) $ ./agg-example localhost Inserting data distinct Next double: 2.000000 Next double: 3.000000 map reduce { "result" : "outCollection", "timeMillis" : 155, "counts" : { "input" : 84, "emit" : 126, "reduce" : 3, "output" : 3 }, "ok" : 1 } { "_id" : "cat", "value" : 63 } { "_id" : "dog", "value" : 42 } { "_id" : "mouse", "value" : 21 } more complicated map reduce { "results" : [ { "_id" : "cat", "value" : 63 }, { "_id" : "dog", "value" : 42 }, { "_id" : "mouse", "value" : 21 } ], "timeMillis" : 14, "counts" : { "input" : 84, "emit" : 126, "reduce" : 3, "output" : 3 }, "ok" : 1 }
USING LIBMONGOC IN A MICROSOFT VISUAL STUDIO PROJECT¶
Download and install libmongoc on your system, then open Visual Studio, select "File→New→Project...", and create a new Win32 Console Application. [image]
Remember to switch the platform from 32-bit to 64-bit: [image]
Right-click on your console application in the Solution Explorer and select "Properties". Choose to edit properties for "All Configurations", expand the "C/C++" options and choose "General". Add to the "Additional Include Directories" these paths:
C:\mongo-c-driver\include\libbson-1.0 C:\mongo-c-driver\include\libmongoc-1.0
[image]
(If you chose a different CMAKE_INSTALL_PREFIX when you ran CMake, your include paths will be different.)
Also in the Properties dialog, expand the "Linker" options and choose "Input", and add to the "Additional Dependencies" these libraries:
C:\mongo-c-driver\lib\bson-1.0.lib C:\mongo-c-driver\lib\mongoc-1.0.lib
[image]
Adding these libraries as dependencies provides linker symbols to build your application, but to actually run it, libbson's and libmongoc's DLLs must be in your executable path. Select "Debugging" in the Properties dialog, and set the "Environment" option to:
PATH=c:/mongo-c-driver/bin
[image]
Finally, include "mongoc/mongoc.h" in your project's "stdafx.h":
#include <mongoc/mongoc.h>
Static linking¶
Following the instructions above, you have dynamically linked your application to the libbson and libmongoc DLLs. This is usually the right choice. If you want to link statically instead, update your "Additional Dependencies" list by removing bson-1.0.lib and mongoc-1.0.lib and replacing them with these libraries:
C:\mongo-c-driver\lib\bson-static-1.0.lib C:\mongo-c-driver\lib\mongoc-static-1.0.lib ws2_32.lib Secur32.lib Crypt32.lib BCrypt.lib
[image]
(To explain the purpose of each library: bson-static-1.0.lib and mongoc-static-1.0.lib are static archives of the driver code. The socket library ws2_32 is required by libbson, which uses the socket routine gethostname to help guarantee ObjectId uniqueness. The BCrypt library is used by libmongoc for TLS connections to MongoDB, and Secur32 and Crypt32 are required for enterprise authentication methods like Kerberos.)
Finally, define two preprocessor symbols before including mongoc/mongoc.h in your stdafx.h:
#define BSON_STATIC #define MONGOC_STATIC #include <mongoc/mongoc.h>
Making these changes to your project is only required for static linking; for most people, the dynamic-linking instructions above are preferred.
Next Steps¶
Now you can build and debug applications in Visual Studio that use libbson and libmongoc. Proceed to Making a Connection in the tutorial to learn how connect to MongoDB and perform operations.
CREATING INDEXES¶
To create indexes on a MongoDB collection, execute the createIndexes command with a command function like mongoc_database_write_command_with_opts() or mongoc_collection_write_command_with_opts(). See the MongoDB Manual entry for the createIndexes command for details.
WARNING:
Example¶
example-create-indexes.c
/* gcc example-create-indexes.c -o example-create-indexes $(pkg-config --cflags
* --libs libmongoc-1.0) */ /* ./example-create-indexes [CONNECTION_STRING [COLLECTION_NAME]] */ #include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> int main (int argc, char *argv[]) {
mongoc_client_t *client;
const char *uri_string =
"mongodb://127.0.0.1/?appname=create-indexes-example";
mongoc_uri_t *uri;
mongoc_database_t *db;
const char *collection_name = "test";
bson_t keys;
char *index_name;
bson_t *create_indexes;
bson_t reply;
char *reply_str;
bson_error_t error;
bool r;
mongoc_init ();
if (argc > 1) {
uri_string = argv[1];
}
if (argc > 2) {
collection_name = argv[2];
}
uri = mongoc_uri_new_with_error (uri_string, &error);
if (!uri) {
fprintf (stderr,
"failed to parse URI: %s\n"
"error message: %s\n",
uri_string,
error.message);
return EXIT_FAILURE;
}
client = mongoc_client_new_from_uri (uri);
if (!client) {
return EXIT_FAILURE;
}
mongoc_client_set_error_api (client, 2);
db = mongoc_client_get_database (client, "test");
/* ascending index on field "x" */
bson_init (&keys);
BSON_APPEND_INT32 (&keys, "x", 1);
index_name = mongoc_collection_keys_to_index_string (&keys);
create_indexes = BCON_NEW ("createIndexes",
BCON_UTF8 (collection_name),
"indexes",
"[",
"{",
"key",
BCON_DOCUMENT (&keys),
"name",
BCON_UTF8 (index_name),
"}",
"]");
r = mongoc_database_write_command_with_opts (
db, create_indexes, NULL /* opts */, &reply, &error);
reply_str = bson_as_json (&reply, NULL);
printf ("%s\n", reply_str);
if (!r) {
fprintf (stderr, "Error in createIndexes: %s\n", error.message);
}
bson_free (index_name);
bson_free (reply_str);
bson_destroy (&reply);
bson_destroy (create_indexes);
mongoc_database_destroy (db);
mongoc_uri_destroy (uri);
mongoc_client_destroy (client);
mongoc_cleanup ();
return r ? EXIT_SUCCESS : EXIT_FAILURE; }
AIDS FOR DEBUGGING¶
GDB¶
This repository contains a .gdbinit file that contains helper functions to aid debugging of data structures. GDB will load this file automatically if you have added the directory which contains the .gdbinit file to GDB's auto-load safe-path, and you start GDB from the directory which holds the .gdbinit file.
You can see the safe-path with show auto-load safe-path on a GDB prompt. You can configure it by setting it in ~/.gdbinit with:
add-auto-load-safe-path /path/to/mongo-c-driver
If you haven't added the path to your auto-load safe-path, or start GDB in another directory, load the file with:
source path/to/mongo-c-driver/.gdbinit
The .gdbinit file defines the printbson function, which shows the contents of a bson_t * variable. If you have a local bson_t, then you must prefix the variable with a &.
An example GDB session looks like:
(gdb) printbson bson ALLOC [0x555556cd7310 + 0] (len=475) {
'bool' : true,
'int32' : NumberInt("42"),
'int64' : NumberLong("3000000042"),
'string' : "Stŕìñg",
'objectId' : ObjectID("5A1442F3122D331C3C6757E1"),
'utcDateTime' : UTCDateTime(1511277299031),
'arrayOfInts' : [
'0' : NumberInt("1"),
'1' : NumberInt("2")
],
'embeddedDocument' : {
'arrayOfStrings' : [
'0' : "one",
'1' : "two"
],
'double' : 2.718280,
'notherDoc' : {
'true' : NumberInt("1"),
'false' : false
}
},
'binary' : Binary("02", "3031343532333637"),
'regex' : Regex("@[a-z]+@", "im"),
'null' : null,
'js' : JavaScript("print foo"),
'jsws' : JavaScript("print foo") with scope: {
'f' : NumberInt("42"),
'a' : [
'0' : 3.141593,
'1' : 2.718282
]
},
'timestamp' : Timestamp(4294967295, 4294967295),
'double' : 3.141593 }
LLDB¶
This repository also includes a script that customizes LLDB's standard print command to print a bson_t or bson_t * as JSON:
(lldb) print b (bson_t) $0 = {"x": 1, "y": 2}
The custom bson command provides more options:
(lldb) bson --verbose b len=19 flags=INLINE|STATIC {
"x": 1,
"y": 2 } (lldb) bson --raw b '\x13\x00\x00\x00\x10x\x00\x01\x00\x00\x00\x10y\x00\x02\x00\x00\x00\x00'
Type help bson for a list of options.
The script requires a build of libbson with debug symbols, and an installation of PyMongo. Install PyMongo with:
python -m pip install pymongo
If you see "No module named pip" then you must install pip, then run the previous command again.
Create a file ~/.lldbinit containing:
command script import /path/to/mongo-c-driver/lldb_bson.py
If you see "bson command installed by lldb_bson" at the beginning of your LLDB session, you've installed the script correctly.
Debug assertions¶
To enable runtime debug assertions, configure with -DENABLE_DEBUG_ASSERTIONS=ON.
USING CLIENT-SIDE FIELD LEVEL ENCRYPTION¶
New in MongoDB 4.2, Client-Side Field Level Encryption (also referred to as Client-Side Encryption) allows administrators and developers to encrypt specific data fields in addition to other MongoDB encryption features.
With Client-Side Encryption, developers can encrypt fields client side without any server-side configuration or directives. Client-Side Encryption supports workloads where applications must guarantee that unauthorized parties, including server administrators, cannot read the encrypted data.
Automatic encryption, where sensitive fields in commands are encrypted automatically, requires an Enterprise-only process to do query analysis.
Installation¶
libmongocrypt¶
There is a separate library, libmongocrypt, that must be installed prior to configuring libmongoc to enable Client-Side Encryption.
libmongocrypt depends on libbson. To build libmongoc with Client-Side Encryption support you must:
- 1.
- Install libbson
- 2.
- Build and install libmongocrypt
- 3.
- Build libmongoc
To install libbson, follow the instructions to install with a package manager: Install libbson with a Package Manager or build from source with cmake (disable building libmongoc with -DENABLE_MONGOC=OFF):
$ cd mongo-c-driver $ mkdir cmake-build && cd cmake-build $ cmake -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF -DENABLE_MONGOC=OFF .. $ cmake --build . --target install
To build and install libmongocrypt, clone the repository and configure as follows:
$ cd libmongocrypt $ mkdir cmake-build && cd cmake-build $ cmake -DENABLE_SHARED_BSON=ON .. $ cmake --build . --target install
Then, you should be able to build libmongoc with Client-Side Encryption.
$ cd mongo-c-driver $ mkdir cmake-build && cd cmake-build $ cmake -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF -DENABLE_MONGOC=ON -DENABLE_CLIENT_SIDE_ENCRYPTION=ON .. $ cmake --build . --target install
mongocryptd¶
The mongocryptd binary is required for automatic Client-Side Encryption and is included as a component in the MongoDB Enterprise Server package. For detailed installation instructions see the MongoDB documentation on mongocryptd.
mongocryptd performs the following:
- Parses the automatic encryption rules specified to the database connection. If the JSON schema contains invalid automatic encryption syntax or any document validation syntax, mongocryptd returns an error.
- Uses the specified automatic encryption rules to mark fields in read and write operations for encryption.
- Rejects read/write operations that may return unexpected or incorrect results when applied to an encrypted field. For supported and unsupported operations, see Read/Write Support with Automatic Field Level Encryption.
A mongoc_client_t configured with auto encryption will automatically spawn the mongocryptd process from the application's PATH. Applications can control the spawning behavior as part of the automatic encryption options. For example, to set a custom path to the mongocryptd process, set the mongocryptdSpawnPath with mongoc_auto_encryption_opts_set_extra().
bson_t *extra = BCON_NEW ("mongocryptdSpawnPath", "/path/to/mongocryptd"); mongoc_auto_encryption_opts_set_extra (opts, extra);
To control the logging output of mongocryptd pass mongocryptdSpawnArgs to mongoc_auto_encryption_opts_set_extra():
bson_t *extra = BCON_NEW ("mongocryptdSpawnArgs",
"[", "--logpath=/path/to/mongocryptd.log", "--logappend", "]"); mongoc_auto_encryption_opts_set_extra (opts, extra);
If your application wishes to manage the mongocryptd process manually, it is possible to disable spawning mongocryptd:
bson_t *extra = BCON_NEW ("mongocryptdBypassSpawn",
BCON_BOOL(true), "mongocryptdURI", "mongodb://localhost:27020"); mongoc_auto_encryption_opts_set_extra (opts, extra);
mongocryptd is only responsible for supporting automatic Client-Side Encryption in the driver and does not itself perform any encryption or decryption.
Automatic Client-Side Field Level Encryption¶
Automatic Client-Side Encryption is enabled by calling mongoc_client_enable_auto_encryption() on a mongoc_client_t. The following examples show how to set up automatic client-side field level encryption using mongoc_client_encryption_t to create a new encryption data key.
NOTE:
Providing Local Automatic Encryption Rules¶
The following example shows how to specify automatic encryption rules using a schema map set with mongoc_auto_encryption_opts_set_schema_map(). The automatic encryption rules are expressed using a strict subset of the JSON Schema syntax.
Supplying a schema map provides more security than relying on JSON Schemas obtained from the server. It protects against a malicious server advertising a false JSON Schema, which could trick the client into sending unencrypted data that should be encrypted.
JSON Schemas supplied in the schema map only apply to configuring automatic client-side field level encryption. Other validation rules in the JSON schema will not be enforced by the driver and will result in an error:
client-side-encryption-schema-map.c
#include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> #include "client-side-encryption-helpers.h" /* Helper method to create a new data key in the key vault, a schema to use that
* key, and writes the schema to a file for later use. */ static bool create_schema_file (bson_t *kms_providers,
const char *keyvault_db,
const char *keyvault_coll,
mongoc_client_t *keyvault_client,
bson_error_t *error) {
mongoc_client_encryption_t *client_encryption = NULL;
mongoc_client_encryption_opts_t *client_encryption_opts = NULL;
mongoc_client_encryption_datakey_opts_t *datakey_opts = NULL;
bson_value_t datakey_id = {0};
char *keyaltnames[] = {"mongoc_encryption_example_1"};
bson_t *schema = NULL;
char *schema_string = NULL;
size_t schema_string_len;
FILE *outfile = NULL;
bool ret = false;
client_encryption_opts = mongoc_client_encryption_opts_new ();
mongoc_client_encryption_opts_set_kms_providers (client_encryption_opts,
kms_providers);
mongoc_client_encryption_opts_set_keyvault_namespace (
client_encryption_opts, keyvault_db, keyvault_coll);
mongoc_client_encryption_opts_set_keyvault_client (client_encryption_opts,
keyvault_client);
client_encryption =
mongoc_client_encryption_new (client_encryption_opts, error);
if (!client_encryption) {
goto fail;
}
/* Create a new data key and json schema for the encryptedField.
* https://dochub.mongodb.org/core/client-side-field-level-encryption-automatic-encryption-rules
*/
datakey_opts = mongoc_client_encryption_datakey_opts_new ();
mongoc_client_encryption_datakey_opts_set_keyaltnames (
datakey_opts, keyaltnames, 1);
if (!mongoc_client_encryption_create_datakey (
client_encryption, "local", datakey_opts, &datakey_id, error)) {
goto fail;
}
/* Create a schema describing that "encryptedField" is a string encrypted
* with the newly created data key using deterministic encryption. */
schema = BCON_NEW ("properties",
"{",
"encryptedField",
"{",
"encrypt",
"{",
"keyId",
"[",
BCON_BIN (datakey_id.value.v_binary.subtype,
datakey_id.value.v_binary.data,
datakey_id.value.v_binary.data_len),
"]",
"bsonType",
"string",
"algorithm",
MONGOC_AEAD_AES_256_CBC_HMAC_SHA_512_DETERMINISTIC,
"}",
"}",
"}",
"bsonType",
"object");
/* Use canonical JSON so that other drivers and tools will be
* able to parse the MongoDB extended JSON file. */
schema_string = bson_as_canonical_extended_json (schema, &schema_string_len);
outfile = fopen ("jsonSchema.json", "w");
if (0 == fwrite (schema_string, sizeof (char), schema_string_len, outfile)) {
fprintf (stderr, "failed to write to file\n");
goto fail;
}
ret = true; fail:
mongoc_client_encryption_destroy (client_encryption);
mongoc_client_encryption_datakey_opts_destroy (datakey_opts);
mongoc_client_encryption_opts_destroy (client_encryption_opts);
bson_free (schema_string);
bson_destroy (schema);
bson_value_destroy (&datakey_id);
if (outfile) {
fclose (outfile);
}
return ret; } /* This example demonstrates how to use automatic encryption with a client-side
* schema map using the enterprise version of MongoDB */ int main (void) { /* The collection used to store the encryption data keys. */ #define KEYVAULT_DB "encryption" #define KEYVAULT_COLL "__libmongocTestKeyVault" /* The collection used to store the encrypted documents in this example. */ #define ENCRYPTED_DB "test" #define ENCRYPTED_COLL "coll"
int exit_status = EXIT_FAILURE;
bool ret;
uint8_t *local_masterkey = NULL;
uint32_t local_masterkey_len;
bson_t *kms_providers = NULL;
bson_error_t error = {0};
bson_t *index_keys = NULL;
char *index_name = NULL;
bson_t *create_index_cmd = NULL;
bson_json_reader_t *reader = NULL;
bson_t schema = BSON_INITIALIZER;
bson_t *schema_map = NULL;
/* The MongoClient used to access the key vault (keyvault_namespace). */
mongoc_client_t *keyvault_client = NULL;
mongoc_collection_t *keyvault_coll = NULL;
mongoc_auto_encryption_opts_t *auto_encryption_opts = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *coll = NULL;
bson_t *to_insert = NULL;
mongoc_client_t *unencrypted_client = NULL;
mongoc_collection_t *unencrypted_coll = NULL;
mongoc_init ();
/* Configure the master key. This must be the same master key that was used
* to create the encryption key. */
local_masterkey =
hex_to_bin (getenv ("LOCAL_MASTERKEY"), &local_masterkey_len);
if (!local_masterkey || local_masterkey_len != 96) {
fprintf (stderr,
"Specify LOCAL_MASTERKEY environment variable as a "
"secure random 96 byte hex value.\n");
goto fail;
}
kms_providers = BCON_NEW ("local",
"{",
"key",
BCON_BIN (0, local_masterkey, local_masterkey_len),
"}");
/* Set up the key vault for this example. */
keyvault_client = mongoc_client_new (
"mongodb://localhost/?appname=client-side-encryption-keyvault");
keyvault_coll = mongoc_client_get_collection (
keyvault_client, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_collection_drop (keyvault_coll, NULL);
/* Create a unique index to ensure that two data keys cannot share the same
* keyAltName. This is recommended practice for the key vault. */
index_keys = BCON_NEW ("keyAltNames", BCON_INT32 (1));
index_name = mongoc_collection_keys_to_index_string (index_keys);
create_index_cmd = BCON_NEW ("createIndexes",
KEYVAULT_COLL,
"indexes",
"[",
"{",
"key",
BCON_DOCUMENT (index_keys),
"name",
index_name,
"unique",
BCON_BOOL (true),
"partialFilterExpression",
"{",
"keyAltNames",
"{",
"$exists",
BCON_BOOL (true),
"}",
"}",
"}",
"]");
ret = mongoc_client_command_simple (keyvault_client,
KEYVAULT_DB,
create_index_cmd,
NULL /* read prefs */,
NULL /* reply */,
&error);
if (!ret) {
goto fail;
}
/* Create a new data key and a schema using it for encryption. Save the
* schema to the file jsonSchema.json */
ret = create_schema_file (
kms_providers, KEYVAULT_DB, KEYVAULT_COLL, keyvault_client, &error);
if (!ret) {
goto fail;
}
/* Load the JSON Schema and construct the local schema_map option. */
reader = bson_json_reader_new_from_file ("jsonSchema.json", &error);
if (!reader) {
goto fail;
}
bson_json_reader_read (reader, &schema, &error);
/* Construct the schema map, mapping the namespace of the collection to the
* schema describing encryption. */
schema_map =
BCON_NEW (ENCRYPTED_DB "." ENCRYPTED_COLL, BCON_DOCUMENT (&schema));
auto_encryption_opts = mongoc_auto_encryption_opts_new ();
mongoc_auto_encryption_opts_set_keyvault_client (auto_encryption_opts,
keyvault_client);
mongoc_auto_encryption_opts_set_keyvault_namespace (
auto_encryption_opts, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_auto_encryption_opts_set_kms_providers (auto_encryption_opts,
kms_providers);
mongoc_auto_encryption_opts_set_schema_map (auto_encryption_opts,
schema_map);
client =
mongoc_client_new ("mongodb://localhost/?appname=client-side-encryption");
/* Enable automatic encryption. It will determine that encryption is
* necessary from the schema map instead of relying on the server to provide
* a schema. */
ret = mongoc_client_enable_auto_encryption (
client, auto_encryption_opts, &error);
if (!ret) {
goto fail;
}
coll = mongoc_client_get_collection (client, ENCRYPTED_DB, ENCRYPTED_COLL);
/* Clear old data */
mongoc_collection_drop (coll, NULL);
to_insert = BCON_NEW ("encryptedField", "123456789");
ret = mongoc_collection_insert_one (
coll, to_insert, NULL /* opts */, NULL /* reply */, &error);
if (!ret) {
goto fail;
}
printf ("decrypted document: ");
if (!print_one_document (coll, &error)) {
goto fail;
}
printf ("\n");
unencrypted_client = mongoc_client_new (
"mongodb://localhost/?appname=client-side-encryption-unencrypted");
unencrypted_coll = mongoc_client_get_collection (
unencrypted_client, ENCRYPTED_DB, ENCRYPTED_COLL);
printf ("encrypted document: ");
if (!print_one_document (unencrypted_coll, &error)) {
goto fail;
}
printf ("\n");
exit_status = EXIT_SUCCESS; fail:
if (error.code) {
fprintf (stderr, "error: %s\n", error.message);
}
bson_free (local_masterkey);
bson_destroy (kms_providers);
mongoc_collection_destroy (keyvault_coll);
bson_destroy (index_keys);
bson_free (index_name);
bson_destroy (create_index_cmd);
bson_json_reader_destroy (reader);
mongoc_auto_encryption_opts_destroy (auto_encryption_opts);
mongoc_collection_destroy (coll);
mongoc_client_destroy (client);
bson_destroy (to_insert);
mongoc_collection_destroy (unencrypted_coll);
mongoc_client_destroy (unencrypted_client);
mongoc_client_destroy (keyvault_client);
bson_destroy (&schema);
bson_destroy (schema_map);
mongoc_cleanup ();
return exit_status; }
Server-Side Field Level Encryption Enforcement¶
The MongoDB 4.2 server supports using schema validation to enforce encryption of specific fields in a collection. This schema validation will prevent an application from inserting unencrypted values for any fields marked with the "encrypt" JSON schema keyword.
The following example shows how to set up automatic client-side field level encryption using mongoc_client_encryption_t to create a new encryption data key and create a collection with the Automatic Encryption JSON Schema Syntax:
client-side-encryption-server-schema.c
#include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> #include "client-side-encryption-helpers.h" /* Helper method to create and return a JSON schema to use for encryption. The caller will use the returned schema for server-side encryption validation. */ static bson_t * create_schema (bson_t *kms_providers,
const char *keyvault_db,
const char *keyvault_coll,
mongoc_client_t *keyvault_client,
bson_error_t *error) {
mongoc_client_encryption_t *client_encryption = NULL;
mongoc_client_encryption_opts_t *client_encryption_opts = NULL;
mongoc_client_encryption_datakey_opts_t *datakey_opts = NULL;
bson_value_t datakey_id = {0};
char *keyaltnames[] = {"mongoc_encryption_example_2"};
bson_t *schema = NULL;
client_encryption_opts = mongoc_client_encryption_opts_new ();
mongoc_client_encryption_opts_set_kms_providers (client_encryption_opts,
kms_providers);
mongoc_client_encryption_opts_set_keyvault_namespace (
client_encryption_opts, keyvault_db, keyvault_coll);
mongoc_client_encryption_opts_set_keyvault_client (client_encryption_opts,
keyvault_client);
client_encryption =
mongoc_client_encryption_new (client_encryption_opts, error);
if (!client_encryption) {
goto fail;
}
/* Create a new data key and json schema for the encryptedField.
* https://dochub.mongodb.org/core/client-side-field-level-encryption-automatic-encryption-rules
*/
datakey_opts = mongoc_client_encryption_datakey_opts_new ();
mongoc_client_encryption_datakey_opts_set_keyaltnames (
datakey_opts, keyaltnames, 1);
if (!mongoc_client_encryption_create_datakey (
client_encryption, "local", datakey_opts, &datakey_id, error)) {
goto fail;
}
/* Create a schema describing that "encryptedField" is a string encrypted
* with the newly created data key using deterministic encryption. */
schema = BCON_NEW ("properties",
"{",
"encryptedField",
"{",
"encrypt",
"{",
"keyId",
"[",
BCON_BIN (datakey_id.value.v_binary.subtype,
datakey_id.value.v_binary.data,
datakey_id.value.v_binary.data_len),
"]",
"bsonType",
"string",
"algorithm",
MONGOC_AEAD_AES_256_CBC_HMAC_SHA_512_DETERMINISTIC,
"}",
"}",
"}",
"bsonType",
"object"); fail:
mongoc_client_encryption_destroy (client_encryption);
mongoc_client_encryption_datakey_opts_destroy (datakey_opts);
mongoc_client_encryption_opts_destroy (client_encryption_opts);
bson_value_destroy (&datakey_id);
return schema; } /* This example demonstrates how to use automatic encryption with a server-side
* schema using the enterprise version of MongoDB */ int main (void) { /* The collection used to store the encryption data keys. */ #define KEYVAULT_DB "encryption" #define KEYVAULT_COLL "__libmongocTestKeyVault" /* The collection used to store the encrypted documents in this example. */ #define ENCRYPTED_DB "test" #define ENCRYPTED_COLL "coll"
int exit_status = EXIT_FAILURE;
bool ret;
uint8_t *local_masterkey = NULL;
uint32_t local_masterkey_len;
bson_t *kms_providers = NULL;
bson_error_t error = {0};
bson_t *index_keys = NULL;
char *index_name = NULL;
bson_t *create_index_cmd = NULL;
bson_json_reader_t *reader = NULL;
bson_t *schema = NULL;
/* The MongoClient used to access the key vault (keyvault_namespace). */
mongoc_client_t *keyvault_client = NULL;
mongoc_collection_t *keyvault_coll = NULL;
mongoc_auto_encryption_opts_t *auto_encryption_opts = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *coll = NULL;
bson_t *to_insert = NULL;
mongoc_client_t *unencrypted_client = NULL;
mongoc_collection_t *unencrypted_coll = NULL;
bson_t *create_cmd = NULL;
bson_t *create_cmd_opts = NULL;
mongoc_write_concern_t *wc = NULL;
mongoc_init ();
/* Configure the master key. This must be the same master key that was used
* to create
* the encryption key. */
local_masterkey =
hex_to_bin (getenv ("LOCAL_MASTERKEY"), &local_masterkey_len);
if (!local_masterkey || local_masterkey_len != 96) {
fprintf (stderr,
"Specify LOCAL_MASTERKEY environment variable as a "
"secure random 96 byte hex value.\n");
goto fail;
}
kms_providers = BCON_NEW ("local",
"{",
"key",
BCON_BIN (0, local_masterkey, local_masterkey_len),
"}");
/* Set up the key vault for this example. */
keyvault_client = mongoc_client_new (
"mongodb://localhost/?appname=client-side-encryption-keyvault");
keyvault_coll = mongoc_client_get_collection (
keyvault_client, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_collection_drop (keyvault_coll, NULL);
/* Create a unique index to ensure that two data keys cannot share the same
* keyAltName. This is recommended practice for the key vault. */
index_keys = BCON_NEW ("keyAltNames", BCON_INT32 (1));
index_name = mongoc_collection_keys_to_index_string (index_keys);
create_index_cmd = BCON_NEW ("createIndexes",
KEYVAULT_COLL,
"indexes",
"[",
"{",
"key",
BCON_DOCUMENT (index_keys),
"name",
index_name,
"unique",
BCON_BOOL (true),
"partialFilterExpression",
"{",
"keyAltNames",
"{",
"$exists",
BCON_BOOL (true),
"}",
"}",
"}",
"]");
ret = mongoc_client_command_simple (keyvault_client,
KEYVAULT_DB,
create_index_cmd,
NULL /* read prefs */,
NULL /* reply */,
&error);
if (!ret) {
goto fail;
}
auto_encryption_opts = mongoc_auto_encryption_opts_new ();
mongoc_auto_encryption_opts_set_keyvault_client (auto_encryption_opts,
keyvault_client);
mongoc_auto_encryption_opts_set_keyvault_namespace (
auto_encryption_opts, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_auto_encryption_opts_set_kms_providers (auto_encryption_opts,
kms_providers);
schema = create_schema (
kms_providers, KEYVAULT_DB, KEYVAULT_COLL, keyvault_client, &error);
if (!schema) {
goto fail;
}
client =
mongoc_client_new ("mongodb://localhost/?appname=client-side-encryption");
ret = mongoc_client_enable_auto_encryption (
client, auto_encryption_opts, &error);
if (!ret) {
goto fail;
}
coll = mongoc_client_get_collection (client, ENCRYPTED_DB, ENCRYPTED_COLL);
/* Clear old data */
mongoc_collection_drop (coll, NULL);
/* Create the collection with the encryption JSON Schema. */
create_cmd = BCON_NEW ("create",
ENCRYPTED_COLL,
"validator",
"{",
"$jsonSchema",
BCON_DOCUMENT (schema),
"}");
wc = mongoc_write_concern_new ();
mongoc_write_concern_set_wmajority (wc, 0);
create_cmd_opts = bson_new ();
mongoc_write_concern_append (wc, create_cmd_opts);
ret = mongoc_client_command_with_opts (client,
ENCRYPTED_DB,
create_cmd,
NULL /* read prefs */,
create_cmd_opts,
NULL /* reply */,
&error);
if (!ret) {
goto fail;
}
to_insert = BCON_NEW ("encryptedField", "123456789");
ret = mongoc_collection_insert_one (
coll, to_insert, NULL /* opts */, NULL /* reply */, &error);
if (!ret) {
goto fail;
}
printf ("decrypted document: ");
if (!print_one_document (coll, &error)) {
goto fail;
}
printf ("\n");
unencrypted_client = mongoc_client_new (
"mongodb://localhost/?appname=client-side-encryption-unencrypted");
unencrypted_coll = mongoc_client_get_collection (
unencrypted_client, ENCRYPTED_DB, ENCRYPTED_COLL);
printf ("encrypted document: ");
if (!print_one_document (unencrypted_coll, &error)) {
goto fail;
}
printf ("\n");
/* Expect a server-side error if inserting with the unencrypted collection.
*/
ret = mongoc_collection_insert_one (
unencrypted_coll, to_insert, NULL /* opts */, NULL /* reply */, &error);
if (!ret) {
printf ("insert with unencrypted collection failed: %s\n", error.message);
memset (&error, 0, sizeof (error));
}
exit_status = EXIT_SUCCESS; fail:
if (error.code) {
fprintf (stderr, "error: %s\n", error.message);
}
bson_free (local_masterkey);
bson_destroy (kms_providers);
mongoc_collection_destroy (keyvault_coll);
bson_destroy (index_keys);
bson_free (index_name);
bson_destroy (create_index_cmd);
bson_json_reader_destroy (reader);
mongoc_auto_encryption_opts_destroy (auto_encryption_opts);
mongoc_collection_destroy (coll);
mongoc_client_destroy (client);
bson_destroy (to_insert);
mongoc_collection_destroy (unencrypted_coll);
mongoc_client_destroy (unencrypted_client);
mongoc_client_destroy (keyvault_client);
bson_destroy (schema);
bson_destroy (create_cmd);
bson_destroy (create_cmd_opts);
mongoc_write_concern_destroy (wc);
mongoc_cleanup ();
return exit_status; }
Explicit Encryption¶
Explicit encryption is a MongoDB community feature and does not use the mongocryptd process. Explicit encryption is provided by the mongoc_client_encryption_t class, for example:
client-side-encryption-explicit.c
#include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> #include "client-side-encryption-helpers.h" /* This example demonstrates how to use explicit encryption and decryption using
* the community version of MongoDB */ int main (void) { /* The collection used to store the encryption data keys. */ #define KEYVAULT_DB "encryption" #define KEYVAULT_COLL "__libmongocTestKeyVault" /* The collection used to store the encrypted documents in this example. */ #define ENCRYPTED_DB "test" #define ENCRYPTED_COLL "coll"
int exit_status = EXIT_FAILURE;
bool ret;
uint8_t *local_masterkey = NULL;
uint32_t local_masterkey_len;
bson_t *kms_providers = NULL;
bson_error_t error = {0};
bson_t *index_keys = NULL;
char *index_name = NULL;
bson_t *create_index_cmd = NULL;
bson_t *schema = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *coll = NULL;
mongoc_collection_t *keyvault_coll = NULL;
bson_t *to_insert = NULL;
bson_t *create_cmd = NULL;
bson_t *create_cmd_opts = NULL;
mongoc_write_concern_t *wc = NULL;
mongoc_client_encryption_t *client_encryption = NULL;
mongoc_client_encryption_opts_t *client_encryption_opts = NULL;
mongoc_client_encryption_datakey_opts_t *datakey_opts = NULL;
char *keyaltnames[] = {"mongoc_encryption_example_3"};
bson_value_t datakey_id = {0};
bson_value_t encrypted_field = {0};
bson_value_t to_encrypt = {0};
mongoc_client_encryption_encrypt_opts_t *encrypt_opts = NULL;
bson_value_t decrypted = {0};
mongoc_init ();
/* Configure the master key. This must be the same master key that was used
* to create the encryption key. */
local_masterkey =
hex_to_bin (getenv ("LOCAL_MASTERKEY"), &local_masterkey_len);
if (!local_masterkey || local_masterkey_len != 96) {
fprintf (stderr,
"Specify LOCAL_MASTERKEY environment variable as a "
"secure random 96 byte hex value.\n");
goto fail;
}
kms_providers = BCON_NEW ("local",
"{",
"key",
BCON_BIN (0, local_masterkey, local_masterkey_len),
"}");
/* The mongoc_client_t used to read/write application data. */
client =
mongoc_client_new ("mongodb://localhost/?appname=client-side-encryption");
coll = mongoc_client_get_collection (client, ENCRYPTED_DB, ENCRYPTED_COLL);
/* Clear old data */
mongoc_collection_drop (coll, NULL);
/* Set up the key vault for this example. */
keyvault_coll =
mongoc_client_get_collection (client, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_collection_drop (keyvault_coll, NULL);
/* Create a unique index to ensure that two data keys cannot share the same
* keyAltName. This is recommended practice for the key vault. */
index_keys = BCON_NEW ("keyAltNames", BCON_INT32 (1));
index_name = mongoc_collection_keys_to_index_string (index_keys);
create_index_cmd = BCON_NEW ("createIndexes",
KEYVAULT_COLL,
"indexes",
"[",
"{",
"key",
BCON_DOCUMENT (index_keys),
"name",
index_name,
"unique",
BCON_BOOL (true),
"partialFilterExpression",
"{",
"keyAltNames",
"{",
"$exists",
BCON_BOOL (true),
"}",
"}",
"}",
"]");
ret = mongoc_client_command_simple (client,
KEYVAULT_DB,
create_index_cmd,
NULL /* read prefs */,
NULL /* reply */,
&error);
if (!ret) {
goto fail;
}
client_encryption_opts = mongoc_client_encryption_opts_new ();
mongoc_client_encryption_opts_set_kms_providers (client_encryption_opts,
kms_providers);
mongoc_client_encryption_opts_set_keyvault_namespace (
client_encryption_opts, KEYVAULT_DB, KEYVAULT_COLL);
/* Set a mongoc_client_t to use for reading/writing to the key vault. This
* can be the same mongoc_client_t used by the main application. */
mongoc_client_encryption_opts_set_keyvault_client (client_encryption_opts,
client);
client_encryption =
mongoc_client_encryption_new (client_encryption_opts, &error);
if (!client_encryption) {
goto fail;
}
/* Create a new data key for the encryptedField.
* https://dochub.mongodb.org/core/client-side-field-level-encryption-automatic-encryption-rules
*/
datakey_opts = mongoc_client_encryption_datakey_opts_new ();
mongoc_client_encryption_datakey_opts_set_keyaltnames (
datakey_opts, keyaltnames, 1);
if (!mongoc_client_encryption_create_datakey (
client_encryption, "local", datakey_opts, &datakey_id, &error)) {
goto fail;
}
/* Explicitly encrypt a field */
encrypt_opts = mongoc_client_encryption_encrypt_opts_new ();
mongoc_client_encryption_encrypt_opts_set_algorithm (
encrypt_opts, MONGOC_AEAD_AES_256_CBC_HMAC_SHA_512_DETERMINISTIC);
mongoc_client_encryption_encrypt_opts_set_keyid (encrypt_opts, &datakey_id);
to_encrypt.value_type = BSON_TYPE_UTF8;
to_encrypt.value.v_utf8.str = "123456789";
to_encrypt.value.v_utf8.len = strlen (to_encrypt.value.v_utf8.str);
ret = mongoc_client_encryption_encrypt (
client_encryption, &to_encrypt, encrypt_opts, &encrypted_field, &error);
if (!ret) {
goto fail;
}
to_insert = bson_new ();
BSON_APPEND_VALUE (to_insert, "encryptedField", &encrypted_field);
ret = mongoc_collection_insert_one (
coll, to_insert, NULL /* opts */, NULL /* reply */, &error);
if (!ret) {
goto fail;
}
printf ("encrypted document: ");
if (!print_one_document (coll, &error)) {
goto fail;
}
printf ("\n");
/* Explicitly decrypt a field */
ret = mongoc_client_encryption_decrypt (
client_encryption, &encrypted_field, &decrypted, &error);
if (!ret) {
goto fail;
}
printf ("decrypted value: %s\n", decrypted.value.v_utf8.str);
exit_status = EXIT_SUCCESS; fail:
if (error.code) {
fprintf (stderr, "error: %s\n", error.message);
}
bson_free (local_masterkey);
bson_destroy (kms_providers);
mongoc_collection_destroy (keyvault_coll);
bson_destroy (index_keys);
bson_free (index_name);
bson_destroy (create_index_cmd);
mongoc_collection_destroy (coll);
mongoc_client_destroy (client);
bson_destroy (to_insert);
bson_destroy (schema);
bson_destroy (create_cmd);
bson_destroy (create_cmd_opts);
mongoc_write_concern_destroy (wc);
mongoc_client_encryption_destroy (client_encryption);
mongoc_client_encryption_datakey_opts_destroy (datakey_opts);
mongoc_client_encryption_opts_destroy (client_encryption_opts);
bson_value_destroy (&encrypted_field);
mongoc_client_encryption_encrypt_opts_destroy (encrypt_opts);
bson_value_destroy (&decrypted);
bson_value_destroy (&datakey_id);
mongoc_cleanup ();
return exit_status; }
Explicit Encryption with Automatic Decryption¶
Although automatic encryption requires MongoDB 4.2 enterprise or a MongoDB 4.2 Atlas cluster, automatic decryption is supported for all users. To configure automatic decryption without automatic encryption set bypass_auto_encryption=True in mongoc_auto_encryption_opts_t:
client-side-encryption-auto-decryption.c
#include <mongoc/mongoc.h> #include <stdio.h> #include <stdlib.h> #include "client-side-encryption-helpers.h" /* This example demonstrates how to set up automatic decryption without
* automatic encryption using the community version of MongoDB */ int main (void) { /* The collection used to store the encryption data keys. */ #define KEYVAULT_DB "encryption" #define KEYVAULT_COLL "__libmongocTestKeyVault" /* The collection used to store the encrypted documents in this example. */ #define ENCRYPTED_DB "test" #define ENCRYPTED_COLL "coll"
int exit_status = EXIT_FAILURE;
bool ret;
uint8_t *local_masterkey = NULL;
uint32_t local_masterkey_len;
bson_t *kms_providers = NULL;
bson_error_t error = {0};
bson_t *index_keys = NULL;
char *index_name = NULL;
bson_t *create_index_cmd = NULL;
bson_t *schema = NULL;
mongoc_client_t *client = NULL;
mongoc_collection_t *coll = NULL;
mongoc_collection_t *keyvault_coll = NULL;
bson_t *to_insert = NULL;
bson_t *create_cmd = NULL;
bson_t *create_cmd_opts = NULL;
mongoc_write_concern_t *wc = NULL;
mongoc_client_encryption_t *client_encryption = NULL;
mongoc_client_encryption_opts_t *client_encryption_opts = NULL;
mongoc_client_encryption_datakey_opts_t *datakey_opts = NULL;
char *keyaltnames[] = {"mongoc_encryption_example_4"};
bson_value_t datakey_id = {0};
bson_value_t encrypted_field = {0};
bson_value_t to_encrypt = {0};
mongoc_client_encryption_encrypt_opts_t *encrypt_opts = NULL;
bson_value_t decrypted = {0};
mongoc_auto_encryption_opts_t *auto_encryption_opts = NULL;
mongoc_client_t *unencrypted_client = NULL;
mongoc_collection_t *unencrypted_coll = NULL;
mongoc_init ();
/* Configure the master key. This must be the same master key that was used
* to create the encryption key. */
local_masterkey =
hex_to_bin (getenv ("LOCAL_MASTERKEY"), &local_masterkey_len);
if (!local_masterkey || local_masterkey_len != 96) {
fprintf (stderr,
"Specify LOCAL_MASTERKEY environment variable as a "
"secure random 96 byte hex value.\n");
goto fail;
}
kms_providers = BCON_NEW ("local",
"{",
"key",
BCON_BIN (0, local_masterkey, local_masterkey_len),
"}");
client =
mongoc_client_new ("mongodb://localhost/?appname=client-side-encryption");
auto_encryption_opts = mongoc_auto_encryption_opts_new ();
mongoc_auto_encryption_opts_set_keyvault_namespace (
auto_encryption_opts, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_auto_encryption_opts_set_kms_providers (auto_encryption_opts,
kms_providers);
/* Setting bypass_auto_encryption to true disables automatic encryption but
* keeps the automatic decryption behavior. bypass_auto_encryption will also
* disable spawning mongocryptd */
mongoc_auto_encryption_opts_set_bypass_auto_encryption (auto_encryption_opts,
true);
/* Once bypass_auto_encryption is set, community users can enable auto
* encryption on the client. This will, in fact, only perform automatic
* decryption. */
ret = mongoc_client_enable_auto_encryption (
client, auto_encryption_opts, &error);
if (!ret) {
goto fail;
}
/* Now that automatic decryption is on, we can test it by inserting a
* document with an explicitly encrypted value into the collection. When we
* look up the document later, it should be automatically decrypted for us.
*/
coll = mongoc_client_get_collection (client, ENCRYPTED_DB, ENCRYPTED_COLL);
/* Clear old data */
mongoc_collection_drop (coll, NULL);
/* Set up the key vault for this example. */
keyvault_coll =
mongoc_client_get_collection (client, KEYVAULT_DB, KEYVAULT_COLL);
mongoc_collection_drop (keyvault_coll, NULL);
/* Create a unique index to ensure that two data keys cannot share the same
* keyAltName. This is recommended practice for the key vault. */
index_keys = BCON_NEW ("keyAltNames", BCON_INT32 (1));
index_name = mongoc_collection_keys_to_index_string (index_keys);
create_index_cmd = BCON_NEW ("createIndexes",
KEYVAULT_COLL,
"indexes",
"[",
"{",
"key",
BCON_DOCUMENT (index_keys),
"name",
index_name,
"unique",
BCON_BOOL (true),
"partialFilterExpression",
"{",
"keyAltNames",
"{",
"$exists",
BCON_BOOL (true),
"}",
"}",
"}",
"]");
ret = mongoc_client_command_simple (client,
KEYVAULT_DB,
create_index_cmd,
NULL /* read prefs */,
NULL /* reply */,
&error);
if (!ret) {
goto fail;
}
client_encryption_opts = mongoc_client_encryption_opts_new ();
mongoc_client_encryption_opts_set_kms_providers (client_encryption_opts,
kms_providers);
mongoc_client_encryption_opts_set_keyvault_namespace (
client_encryption_opts, KEYVAULT_DB, KEYVAULT_COLL);
/* The key vault client is used for reading to/from the key vault. This can
* be the same mongoc_client_t used by the application. */
mongoc_client_encryption_opts_set_keyvault_client (client_encryption_opts,
client);
client_encryption =
mongoc_client_encryption_new (client_encryption_opts, &error);
if (!client_encryption) {
goto fail;
}
/* Create a new data key for the encryptedField.
* https://dochub.mongodb.org/core/client-side-field-level-encryption-automatic-encryption-rules
*/
datakey_opts = mongoc_client_encryption_datakey_opts_new ();
mongoc_client_encryption_datakey_opts_set_keyaltnames (
datakey_opts, keyaltnames, 1);
ret = mongoc_client_encryption_create_datakey (
client_encryption, "local", datakey_opts, &datakey_id, &error);
if (!ret) {
goto fail;
}
/* Explicitly encrypt a field. */
encrypt_opts = mongoc_client_encryption_encrypt_opts_new ();
mongoc_client_encryption_encrypt_opts_set_algorithm (
encrypt_opts, MONGOC_AEAD_AES_256_CBC_HMAC_SHA_512_DETERMINISTIC);
mongoc_client_encryption_encrypt_opts_set_keyaltname (
encrypt_opts, "mongoc_encryption_example_4");
to_encrypt.value_type = BSON_TYPE_UTF8;
to_encrypt.value.v_utf8.str = "123456789";
to_encrypt.value.v_utf8.len = strlen (to_encrypt.value.v_utf8.str);
ret = mongoc_client_encryption_encrypt (
client_encryption, &to_encrypt, encrypt_opts, &encrypted_field, &error);
if (!ret) {
goto fail;
}
to_insert = bson_new ();
BSON_APPEND_VALUE (to_insert, "encryptedField", &encrypted_field);
ret = mongoc_collection_insert_one (
coll, to_insert, NULL /* opts */, NULL /* reply */, &error);
if (!ret) {
goto fail;
}
/* When we retrieve the document, any encrypted fields will get automatically
* decrypted by the driver. */
printf ("decrypted document: ");
if (!print_one_document (coll, &error)) {
goto fail;
}
printf ("\n");
unencrypted_client =
mongoc_client_new ("mongodb://localhost/?appname=client-side-encryption");
unencrypted_coll = mongoc_client_get_collection (
unencrypted_client, ENCRYPTED_DB, ENCRYPTED_COLL);
printf ("encrypted document: ");
if (!print_one_document (unencrypted_coll, &error)) {
goto fail;
}
printf ("\n");
exit_status = EXIT_SUCCESS; fail:
if (error.code) {
fprintf (stderr, "error: %s\n", error.message);
}
bson_free (local_masterkey);
bson_destroy (kms_providers);
mongoc_collection_destroy (keyvault_coll);
bson_destroy (index_keys);
bson_free (index_name);
bson_destroy (create_index_cmd);
mongoc_collection_destroy (coll);
mongoc_client_destroy (client);
bson_destroy (to_insert);
bson_destroy (schema);
bson_destroy (create_cmd);
bson_destroy (create_cmd_opts);
mongoc_write_concern_destroy (wc);
mongoc_client_encryption_destroy (client_encryption);
mongoc_client_encryption_datakey_opts_destroy (datakey_opts);
mongoc_client_encryption_opts_destroy (client_encryption_opts);
bson_value_destroy (&encrypted_field);
mongoc_client_encryption_encrypt_opts_destroy (encrypt_opts);
bson_value_destroy (&decrypted);
bson_value_destroy (&datakey_id);
mongoc_collection_destroy (unencrypted_coll);
mongoc_client_destroy (unencrypted_client);
mongoc_auto_encryption_opts_destroy (auto_encryption_opts);
mongoc_cleanup ();
return exit_status; }
AUTHOR¶
MongoDB, Inc
COPYRIGHT¶
2017-present, MongoDB, Inc
October 20, 2022 | 1.23.1 |