NAME¶

PCRE2 - Perl-compatible regular expressions (revised API)

PCRE2 REGULAR EXPRESSION DETAILS¶

The syntax and semantics of the regular expressions that are supported by PCRE2 are described in detail below. There is a quick-reference syntax summary in the pcre2syntax page. PCRE2 tries to match Perl syntax and semantics as closely as it can. PCRE2 also supports some alternative regular expression syntax (which does not conflict with the Perl syntax) in order to provide some compatibility with regular expressions in Python, .NET, and Oniguruma.

Perl's regular expressions are described in its own documentation, and regular expressions in general are covered in a number of books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by O'Reilly, covers regular expressions in great detail. This description of PCRE2's regular expressions is intended as reference material.

This document discusses the regular expression patterns that are supported by PCRE2 when its main matching function, pcre2_match(), is used. PCRE2 also has an alternative matching function, pcre2_dfa_match(), which matches using a different algorithm that is not Perl-compatible. Some of the features discussed below are not available when DFA matching is used. The advantages and disadvantages of the alternative function, and how it differs from the normal function, are discussed in the pcre2matching page.

SPECIAL START-OF-PATTERN ITEMS¶

A number of options that can be passed to pcre2_compile() can also be set by special items at the start of a pattern. These are not Perl-compatible, but are provided to make these options accessible to pattern writers who are not able to change the program that processes the pattern. Any number of these items may appear, but they must all be together right at the start of the pattern string, and the letters must be in upper case.

EBCDIC CHARACTER CODES¶

PCRE2 can be compiled to run in an environment that uses EBCDIC as its character code instead of ASCII or Unicode (typically a mainframe system). In the sections below, character code values are ASCII or Unicode; in an EBCDIC environment these characters may have different code values, and there are no code points greater than 255.

CHARACTERS AND METACHARACTERS¶

A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern

The quick brown fox

matches a portion of a subject string that is identical to itself. When caseless matching is specified (the PCRE2_CASELESS option or (?i) within the pattern), letters are matched independently of case. Note that there are two ASCII characters, K and S, that, in addition to their lower case ASCII equivalents, are case-equivalent with Unicode U+212A (Kelvin sign) and U+017F (long S) respectively when either PCRE2_UTF or PCRE2_UCP is set.

The power of regular expressions comes from the ability to include wild cards, character classes, alternatives, and repetitions in the pattern. These are encoded in the pattern by the use of metacharacters, which do not stand for themselves but instead are interpreted in some special way.

There are two different sets of metacharacters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized within square brackets. Outside square brackets, the metacharacters are as follows:

\ general escape character with several uses
^ assert start of string (or line, in multiline mode)
$ assert end of string (or line, in multiline mode)
. match any character except newline (by default)
[ start character class definition
| start of alternative branch
( start group or control verb
) end group or control verb
* 0 or more quantifier
+ 1 or more quantifier; also "possessive quantifier"
? 0 or 1 quantifier; also quantifier minimizer
{ start min/max quantifier

Part of a pattern that is in square brackets is called a "character class". In a character class the only metacharacters are:

\ general escape character
^ negate the class, but only if the first character
- indicates character range
[ POSIX character class (if followed by POSIX syntax)
] terminates the character class

If a pattern is compiled with the PCRE2_EXTENDED option, most white space in the pattern, other than in a character class, and characters between a # outside a character class and the next newline, inclusive, are ignored. An escaping backslash can be used to include a white space or a # character as part of the pattern. If the PCRE2_EXTENDED_MORE option is set, the same applies, but in addition unescaped space and horizontal tab characters are ignored inside a character class. Note: only these two characters are ignored, not the full set of pattern white space characters that are ignored outside a character class. Option settings can be changed within a pattern; see the section entitled "Internal Option Setting" below.

The following sections describe the use of each of the metacharacters.

BACKSLASH¶

The backslash character has several uses. Firstly, if it is followed by a character that is not a digit or a letter, it takes away any special meaning that character may have. This use of backslash as an escape character applies both inside and outside character classes.

For example, if you want to match a * character, you must write \* in the pattern. This escaping action applies whether or not the following character would otherwise be interpreted as a metacharacter, so it is always safe to precede a non-alphanumeric with backslash to specify that it stands for itself. In particular, if you want to match a backslash, you write \\.

Only ASCII digits and letters have any special meaning after a backslash. All other characters (in particular, those whose code points are greater than 127) are treated as literals.

If you want to treat all characters in a sequence as literals, you can do so by putting them between \Q and \E. This is different from Perl in that $ and @ are handled as literals in \Q...\E sequences in PCRE2, whereas in Perl, $ and @ cause variable interpolation. Also, Perl does "double-quotish backslash interpolation" on any backslashes between \Q and \E which, its documentation says, "may lead to confusing results". PCRE2 treats a backslash between \Q and \E just like any other character. Note the following examples:

Pattern PCRE2 matches Perl matches

\Qabc$xyz\E abc$xyz abc followed by the
contents of $xyz
\Qabc\$xyz\E abc\$xyz abc\$xyz
\Qabc\E\$\Qxyz\E abc$xyz abc$xyz
\QA\B\E A\B A\B
\Q\\E \ \\E

The \Q...\E sequence is recognized both inside and outside character classes. An isolated \E that is not preceded by \Q is ignored. If \Q is not followed by \E later in the pattern, the literal interpretation continues to the end of the pattern (that is, \E is assumed at the end). If the isolated \Q is inside a character class, this causes an error, because the character class is not terminated by a closing square bracket.

CIRCUMFLEX AND DOLLAR¶

The circumflex and dollar metacharacters are zero-width assertions. That is, they test for a particular condition being true without consuming any characters from the subject string. These two metacharacters are concerned with matching the starts and ends of lines. If the newline convention is set so that only the two-character sequence CRLF is recognized as a newline, isolated CR and LF characters are treated as ordinary data characters, and are not recognized as newlines.

Outside a character class, in the default matching mode, the circumflex character is an assertion that is true only if the current matching point is at the start of the subject string. If the startoffset argument of pcre2_match() is non-zero, or if PCRE2_NOTBOL is set, circumflex can never match if the PCRE2_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below).

Circumflex need not be the first character of the pattern if a number of alternatives are involved, but it should be the first thing in each alternative in which it appears if the pattern is ever to match that branch. If all possible alternatives start with a circumflex, that is, if the pattern is constrained to match only at the start of the subject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.)

The dollar character is an assertion that is true only if the current matching point is at the end of the subject string, or immediately before a newline at the end of the string (by default), unless PCRE2_NOTEOL is set. Note, however, that it does not actually match the newline. Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

The meaning of dollar can be changed so that it matches only at the very end of the string, by setting the PCRE2_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion.

The meanings of the circumflex and dollar metacharacters are changed if the PCRE2_MULTILINE option is set. When this is the case, a dollar character matches before any newlines in the string, as well as at the very end, and a circumflex matches immediately after internal newlines as well as at the start of the subject string. It does not match after a newline that ends the string, for compatibility with Perl. However, this can be changed by setting the PCRE2_ALT_CIRCUMFLEX option.

For example, the pattern /^abc$/ matches the subject string "def\nabc" (where \n represents a newline) in multiline mode, but not otherwise. Consequently, patterns that are anchored in single line mode because all branches start with ^ are not anchored in multiline mode, and a match for circumflex is possible when the startoffset argument of pcre2_match() is non-zero. The PCRE2_DOLLAR_ENDONLY option is ignored if PCRE2_MULTILINE is set.

When the newline convention (see "Newline conventions" below) recognizes the two-character sequence CRLF as a newline, this is preferred, even if the single characters CR and LF are also recognized as newlines. For example, if the newline convention is "any", a multiline mode circumflex matches before "xyz" in the string "abc\r\nxyz" rather than after CR, even though CR on its own is a valid newline. (It also matches at the very start of the string, of course.)

Note that the sequences \A, \Z, and \z can be used to match the start and end of the subject in both modes, and if all branches of a pattern start with \A it is always anchored, whether or not PCRE2_MULTILINE is set.

FULL STOP (PERIOD, DOT) AND \N¶

Outside a character class, a dot in the pattern matches any one character in the subject string except (by default) a character that signifies the end of a line.

When a line ending is defined as a single character, dot never matches that character; when the two-character sequence CRLF is used, dot does not match CR if it is immediately followed by LF, but otherwise it matches all characters (including isolated CRs and LFs). When any Unicode line endings are being recognized, dot does not match CR or LF or any of the other line ending characters.

The behaviour of dot with regard to newlines can be changed. If the PCRE2_DOTALL option is set, a dot matches any one character, without exception. If the two-character sequence CRLF is present in the subject string, it takes two dots to match it.

The handling of dot is entirely independent of the handling of circumflex and dollar, the only relationship being that they both involve newlines. Dot has no special meaning in a character class.

The escape sequence \N when not followed by an opening brace behaves like a dot, except that it is not affected by the PCRE2_DOTALL option. In other words, it matches any character except one that signifies the end of a line.

When \N is followed by an opening brace it has a different meaning. See the section entitled "Non-printing characters" above for details. Perl also uses \N{name} to specify characters by Unicode name; PCRE2 does not support this.

MATCHING A SINGLE CODE UNIT¶

Outside a character class, the escape sequence \C matches any one code unit, whether or not a UTF mode is set. In the 8-bit library, one code unit is one byte; in the 16-bit library it is a 16-bit unit; in the 32-bit library it is a 32-bit unit. Unlike a dot, \C always matches line-ending characters. The feature is provided in Perl in order to match individual bytes in UTF-8 mode, but it is unclear how it can usefully be used.

Because \C breaks up characters into individual code units, matching one unit with \C in UTF-8 or UTF-16 mode means that the rest of the string may start with a malformed UTF character. This has undefined results, because PCRE2 assumes that it is matching character by character in a valid UTF string (by default it checks the subject string's validity at the start of processing unless the PCRE2_NO_UTF_CHECK or PCRE2_MATCH_INVALID_UTF option is used).

An application can lock out the use of \C by setting the PCRE2_NEVER_BACKSLASH_C option when compiling a pattern. It is also possible to build PCRE2 with the use of \C permanently disabled.

PCRE2 does not allow \C to appear in lookbehind assertions (described below) in UTF-8 or UTF-16 modes, because this would make it impossible to calculate the length of the lookbehind. Neither the alternative matching function pcre2_dfa_match() nor the JIT optimizer support \C in these UTF modes. The former gives a match-time error; the latter fails to optimize and so the match is always run using the interpreter.

In the 32-bit library, however, \C is always supported (when not explicitly locked out) because it always matches a single code unit, whether or not UTF-32 is specified.

In general, the \C escape sequence is best avoided. However, one way of using it that avoids the problem of malformed UTF-8 or UTF-16 characters is to use a lookahead to check the length of the next character, as in this pattern, which could be used with a UTF-8 string (ignore white space and line breaks):

(?| (?=[\x00-\x7f])(\C) |
(?=[\x80-\x{7ff}])(\C)(\C) |
(?=[\x{800}-\x{ffff}])(\C)(\C)(\C) |
(?=[\x{10000}-\x{1fffff}])(\C)(\C)(\C)(\C))

In this example, a group that starts with (?| resets the capturing parentheses numbers in each alternative (see "Duplicate Group Numbers" below). The assertions at the start of each branch check the next UTF-8 character for values whose encoding uses 1, 2, 3, or 4 bytes, respectively. The character's individual bytes are then captured by the appropriate number of \C groups.

SQUARE BRACKETS AND CHARACTER CLASSES¶

An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not special by default. If a closing square bracket is required as a member of the class, it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash. This means that, by default, an empty class cannot be defined. However, if the PCRE2_ALLOW_EMPTY_CLASS option is set, a closing square bracket at the start does end the (empty) class.

A character class matches a single character in the subject. A matched character must be in the set of characters defined by the class, unless the first character in the class definition is a circumflex, in which case the subject character must not be in the set defined by the class. If a circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash.

For example, the character class [aeiou] matches any lower case vowel, while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the characters that are in the class by enumerating those that are not. A class that starts with a circumflex is not an assertion; it still consumes a character from the subject string, and therefore it fails if the current pointer is at the end of the string.

Characters in a class may be specified by their code points using \o, \x, or \N{U+hh..} in the usual way. When caseless matching is set, any letters in a class represent both their upper case and lower case versions, so for example, a caseless [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a caseful version would. Note that there are two ASCII characters, K and S, that, in addition to their lower case ASCII equivalents, are case-equivalent with Unicode U+212A (Kelvin sign) and U+017F (long S) respectively when either PCRE2_UTF or PCRE2_UCP is set.

Characters that might indicate line breaks are never treated in any special way when matching character classes, whatever line-ending sequence is in use, and whatever setting of the PCRE2_DOTALL and PCRE2_MULTILINE options is used. A class such as [^a] always matches one of these characters.

The generic character type escape sequences \d, \D, \h, \H, \p, \P, \s, \S, \v, \V, \w, and \W may appear in a character class, and add the characters that they match to the class. For example, [\dABCDEF] matches any hexadecimal digit. In UTF modes, the PCRE2_UCP option affects the meanings of \d, \s, \w and their upper case partners, just as it does when they appear outside a character class, as described in the section entitled "Generic character types" above. The escape sequence \b has a different meaning inside a character class; it matches the backspace character. The sequences \B, \R, and \X are not special inside a character class. Like any other unrecognized escape sequences, they cause an error. The same is true for \N when not followed by an opening brace.

The minus (hyphen) character can be used to specify a range of characters in a character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as indicating a range, typically as the first or last character in the class, or immediately after a range. For example, [b-d-z] matches letters in the range b to d, a hyphen character, or z.

Perl treats a hyphen as a literal if it appears before or after a POSIX class (see below) or before or after a character type escape such as as \d or \H. However, unless the hyphen is the last character in the class, Perl outputs a warning in its warning mode, as this is most likely a user error. As PCRE2 has no facility for warning, an error is given in these cases.

It is not possible to have the literal character "]" as the end character of a range. A pattern such as [W-]46] is interpreted as a class of two characters ("W" and "-") followed by a literal string "46]", so it would match "W46]" or "-46]". However, if the "]" is escaped with a backslash it is interpreted as the end of range, so [W-\]46] is interpreted as a class containing a range followed by two other characters. The octal or hexadecimal representation of "]" can also be used to end a range.

Ranges normally include all code points between the start and end characters, inclusive. They can also be used for code points specified numerically, for example [\000-\037]. Ranges can include any characters that are valid for the current mode. In any UTF mode, the so-called "surrogate" characters (those whose code points lie between 0xd800 and 0xdfff inclusive) may not be specified explicitly by default (the PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES option disables this check). However, ranges such as [\x{d7ff}-\x{e000}], which include the surrogates, are always permitted.

There is a special case in EBCDIC environments for ranges whose end points are both specified as literal letters in the same case. For compatibility with Perl, EBCDIC code points within the range that are not letters are omitted. For example, [h-k] matches only four characters, even though the codes for h and k are 0x88 and 0x92, a range of 11 code points. However, if the range is specified numerically, for example, [\x88-\x92] or [h-\x92], all code points are included.

If a range that includes letters is used when caseless matching is set, it matches the letters in either case. For example, [W-c] is equivalent to [][\\^_`wxyzabc], matched caselessly, and in a non-UTF mode, if character tables for a French locale are in use, [\xc8-\xcb] matches accented E characters in both cases.

A circumflex can conveniently be used with the upper case character types to specify a more restricted set of characters than the matching lower case type. For example, the class [^\W_] matches any letter or digit, but not underscore, whereas [\w] includes underscore. A positive character class should be read as "something OR something OR ..." and a negative class as "NOT something AND NOT something AND NOT ...".

The only metacharacters that are recognized in character classes are backslash, hyphen (only where it can be interpreted as specifying a range), circumflex (only at the start), opening square bracket (only when it can be interpreted as introducing a POSIX class name, or for a special compatibility feature - see the next two sections), and the terminating closing square bracket. However, escaping other non-alphanumeric characters does no harm.

POSIX CHARACTER CLASSES¶

Perl supports the POSIX notation for character classes. This uses names enclosed by [: and :] within the enclosing square brackets. PCRE2 also supports this notation. For example,

[01[:alpha:]%]

matches "0", "1", any alphabetic character, or "%". The supported class names are:

alnum letters and digits
alpha letters
ascii character codes 0 - 127
blank space or tab only
cntrl control characters
digit decimal digits (same as \d)
graph printing characters, excluding space
lower lower case letters
print printing characters, including space
punct printing characters, excluding letters and digits and space
space white space (the same as \s from PCRE2 8.34)
upper upper case letters
word "word" characters (same as \w)
xdigit hexadecimal digits

The default "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), and space (32). If locale-specific matching is taking place, the list of space characters may be different; there may be fewer or more of them. "Space" and \s match the same set of characters.

The name "word" is a Perl extension, and "blank" is a GNU extension from Perl 5.8. Another Perl extension is negation, which is indicated by a ^ character after the colon. For example,

[12[:^digit:]]

matches "1", "2", or any non-digit. PCRE2 (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered.

By default, characters with values greater than 127 do not match any of the POSIX character classes, although this may be different for characters in the range 128-255 when locale-specific matching is happening. However, if the PCRE2_UCP option is passed to pcre2_compile(), some of the classes are changed so that Unicode character properties are used. This is achieved by replacing certain POSIX classes with other sequences, as follows:

[:alnum:] becomes \p{Xan}
[:alpha:] becomes \p{L}
[:blank:] becomes \h
[:cntrl:] becomes \p{Cc}
[:digit:] becomes \p{Nd}
[:lower:] becomes \p{Ll}
[:space:] becomes \p{Xps}
[:upper:] becomes \p{Lu}
[:word:] becomes \p{Xwd}

Negated versions, such as [:^alpha:] use \P instead of \p. Three other POSIX classes are handled specially in UCP mode:

[:graph:]

This matches characters that have glyphs that mark the page when printed. In Unicode property terms, it matches all characters with the L, M, N, P, S, or Cf properties, except for:

U+061C Arabic Letter Mark
U+180E Mongolian Vowel Separator
U+2066 - U+2069 Various "isolate"s

[:print:]

This matches the same characters as [:graph:] plus space characters that are not controls, that is, characters with the Zs property.

[:punct:]

This matches all characters that have the Unicode P (punctuation) property, plus those characters with code points less than 256 that have the S (Symbol) property.

The other POSIX classes are unchanged, and match only characters with code points less than 256.

COMPATIBILITY FEATURE FOR WORD BOUNDARIES¶

In the POSIX.2 compliant library that was included in 4.4BSD Unix, the ugly syntax [[:<:]] and [[:>:]] is used for matching "start of word" and "end of word". PCRE2 treats these items as follows:

[[:<:]] is converted to \b(?=\w)
[[:>:]] is converted to \b(?<=\w)

Only these exact character sequences are recognized. A sequence such as [a[:<:]b] provokes error for an unrecognized POSIX class name. This support is not compatible with Perl. It is provided to help migrations from other environments, and is best not used in any new patterns. Note that \b matches at the start and the end of a word (see "Simple assertions" above), and in a Perl-style pattern the preceding or following character normally shows which is wanted, without the need for the assertions that are used above in order to give exactly the POSIX behaviour.

VERTICAL BAR¶

Vertical bar characters are used to separate alternative patterns. For example, the pattern

gilbert|sullivan

matches either "gilbert" or "sullivan". Any number of alternatives may appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left to right, and the first one that succeeds is used. If the alternatives are within a group (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the group.

INTERNAL OPTION SETTING¶

The settings of the PCRE2_CASELESS, PCRE2_MULTILINE, PCRE2_DOTALL, PCRE2_EXTENDED, PCRE2_EXTENDED_MORE, and PCRE2_NO_AUTO_CAPTURE options can be changed from within the pattern by a sequence of letters enclosed between "(?" and ")". These options are Perl-compatible, and are described in detail in the pcre2api documentation. The option letters are:

i for PCRE2_CASELESS
m for PCRE2_MULTILINE
n for PCRE2_NO_AUTO_CAPTURE
s for PCRE2_DOTALL
x for PCRE2_EXTENDED
xx for PCRE2_EXTENDED_MORE

For example, (?im) sets caseless, multiline matching. It is also possible to unset these options by preceding the relevant letters with a hyphen, for example (?-im). The two "extended" options are not independent; unsetting either one cancels the effects of both of them.

A combined setting and unsetting such as (?im-sx), which sets PCRE2_CASELESS and PCRE2_MULTILINE while unsetting PCRE2_DOTALL and PCRE2_EXTENDED, is also permitted. Only one hyphen may appear in the options string. If a letter appears both before and after the hyphen, the option is unset. An empty options setting "(?)" is allowed. Needless to say, it has no effect.

If the first character following (? is a circumflex, it causes all of the above options to be unset. Thus, (?^) is equivalent to (?-imnsx). Letters may follow the circumflex to cause some options to be re-instated, but a hyphen may not appear.

The PCRE2-specific options PCRE2_DUPNAMES and PCRE2_UNGREEDY can be changed in the same way as the Perl-compatible options by using the characters J and U respectively. However, these are not unset by (?^).

When one of these option changes occurs at top level (that is, not inside group parentheses), the change applies to the remainder of the pattern that follows. An option change within a group (see below for a description of groups) affects only that part of the group that follows it, so

(a(?i)b)c

matches abc and aBc and no other strings (assuming PCRE2_CASELESS is not used). By this means, options can be made to have different settings in different parts of the pattern. Any changes made in one alternative do carry on into subsequent branches within the same group. For example,

(a(?i)b|c)

matches "ab", "aB", "c", and "C", even though when matching "C" the first branch is abandoned before the option setting. This is because the effects of option settings happen at compile time. There would be some very weird behaviour otherwise.

As a convenient shorthand, if any option settings are required at the start of a non-capturing group (see the next section), the option letters may appear between the "?" and the ":". Thus the two patterns

(?i:saturday|sunday)
(?:(?i)saturday|sunday)

match exactly the same set of strings.

Note: There are other PCRE2-specific options, applying to the whole pattern, which can be set by the application when the compiling function is called. In addition, the pattern can contain special leading sequences such as (*CRLF) to override what the application has set or what has been defaulted. Details are given in the section entitled "Newline sequences" above. There are also the (*UTF) and (*UCP) leading sequences that can be used to set UTF and Unicode property modes; they are equivalent to setting the PCRE2_UTF and PCRE2_UCP options, respectively. However, the application can set the PCRE2_NEVER_UTF and PCRE2_NEVER_UCP options, which lock out the use of the (*UTF) and (*UCP) sequences.

GROUPS¶

Groups are delimited by parentheses (round brackets), which can be nested. Turning part of a pattern into a group does two things:

1. It localizes a set of alternatives. For example, the pattern

cat(aract|erpillar|)

matches "cataract", "caterpillar", or "cat". Without the parentheses, it would match "cataract", "erpillar" or an empty string.

2. It creates a "capture group". This means that, when the whole pattern matches, the portion of the subject string that matched the group is passed back to the caller, separately from the portion that matched the whole pattern. (This applies only to the traditional matching function; the DFA matching function does not support capturing.)

Opening parentheses are counted from left to right (starting from 1) to obtain numbers for capture groups. For example, if the string "the red king" is matched against the pattern

the ((red|white) (king|queen))

the captured substrings are "red king", "red", and "king", and are numbered 1, 2, and 3, respectively.

The fact that plain parentheses fulfil two functions is not always helpful. There are often times when grouping is required without capturing. If an opening parenthesis is followed by a question mark and a colon, the group does not do any capturing, and is not counted when computing the number of any subsequent capture groups. For example, if the string "the white queen" is matched against the pattern

the ((?:red|white) (king|queen))

the captured substrings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of capture groups is 65535.

As a convenient shorthand, if any option settings are required at the start of a non-capturing group, the option letters may appear between the "?" and the ":". Thus the two patterns

(?i:saturday|sunday)
(?:(?i)saturday|sunday)

match exactly the same set of strings. Because alternative branches are tried from left to right, and options are not reset until the end of the group is reached, an option setting in one branch does affect subsequent branches, so the above patterns match "SUNDAY" as well as "Saturday".

DUPLICATE GROUP NUMBERS¶

Perl 5.10 introduced a feature whereby each alternative in a group uses the same numbers for its capturing parentheses. Such a group starts with (?| and is itself a non-capturing group. For example, consider this pattern:

(?|(Sat)ur|(Sun))day

Because the two alternatives are inside a (?| group, both sets of capturing parentheses are numbered one. Thus, when the pattern matches, you can look at captured substring number one, whichever alternative matched. This construct is useful when you want to capture part, but not all, of one of a number of alternatives. Inside a (?| group, parentheses are numbered as usual, but the number is reset at the start of each branch. The numbers of any capturing parentheses that follow the whole group start after the highest number used in any branch. The following example is taken from the Perl documentation. The numbers underneath show in which buffer the captured content will be stored.

# before ---------------branch-reset----------- after
/ ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
# 1 2 2 3 2 3 4

A backreference to a capture group uses the most recent value that is set for the group. The following pattern matches "abcabc" or "defdef":

/(?|(abc)|(def))\1/

In contrast, a subroutine call to a capture group always refers to the first one in the pattern with the given number. The following pattern matches "abcabc" or "defabc":

/(?|(abc)|(def))(?1)/

A relative reference such as (?-1) is no different: it is just a convenient way of computing an absolute group number.

If a condition test for a group's having matched refers to a non-unique number, the test is true if any group with that number has matched.

An alternative approach to using this "branch reset" feature is to use duplicate named groups, as described in the next section.

NAMED CAPTURE GROUPS¶

Identifying capture groups by number is simple, but it can be very hard to keep track of the numbers in complicated patterns. Furthermore, if an expression is modified, the numbers may change. To help with this difficulty, PCRE2 supports the naming of capture groups. This feature was not added to Perl until release 5.10. Python had the feature earlier, and PCRE1 introduced it at release 4.0, using the Python syntax. PCRE2 supports both the Perl and the Python syntax.

In PCRE2, a capture group can be named in one of three ways: (?<name>...) or (?'name'...) as in Perl, or (?P<name>...) as in Python. Names may be up to 32 code units long. When PCRE2_UTF is not set, they may contain only ASCII alphanumeric characters and underscores, but must start with a non-digit. When PCRE2_UTF is set, the syntax of group names is extended to allow any Unicode letter or Unicode decimal digit. In other words, group names must match one of these patterns:

^[_A-Za-z][_A-Za-z0-9]*\z when PCRE2_UTF is not set
^[_\p{L}][_\p{L}\p{Nd}]*\z when PCRE2_UTF is set

References to capture groups from other parts of the pattern, such as backreferences, recursion, and conditions, can all be made by name as well as by number.

Named capture groups are allocated numbers as well as names, exactly as if the names were not present. In both PCRE2 and Perl, capture groups are primarily identified by numbers; any names are just aliases for these numbers. The PCRE2 API provides function calls for extracting the complete name-to-number translation table from a compiled pattern, as well as convenience functions for extracting captured substrings by name.

Warning: When more than one capture group has the same number, as described in the previous section, a name given to one of them applies to all of them. Perl allows identically numbered groups to have different names. Consider this pattern, where there are two capture groups, both numbered 1:

(?|(?<AA>aa)|(?<BB>bb))

Perl allows this, with both names AA and BB as aliases of group 1. Thus, after a successful match, both names yield the same value (either "aa" or "bb").

In an attempt to reduce confusion, PCRE2 does not allow the same group number to be associated with more than one name. The example above provokes a compile-time error. However, there is still scope for confusion. Consider this pattern:

(?|(?<AA>aa)|(bb))

Although the second group number 1 is not explicitly named, the name AA is still an alias for any group 1. Whether the pattern matches "aa" or "bb", a reference by name to group AA yields the matched string.

By default, a name must be unique within a pattern, except that duplicate names are permitted for groups with the same number, for example:

(?|(?<AA>aa)|(?<AA>bb))

The duplicate name constraint can be disabled by setting the PCRE2_DUPNAMES option at compile time, or by the use of (?J) within the pattern, as described in the section entitled "Internal Option Setting" above.

Duplicate names can be useful for patterns where only one instance of the named capture group can match. Suppose you want to match the name of a weekday, either as a 3-letter abbreviation or as the full name, and in both cases you want to extract the abbreviation. This pattern (ignoring the line breaks) does the job:

(?J)
(?<DN>Mon|Fri|Sun)(?:day)?|
(?<DN>Tue)(?:sday)?|
(?<DN>Wed)(?:nesday)?|
(?<DN>Thu)(?:rsday)?|
(?<DN>Sat)(?:urday)?

There are five capture groups, but only one is ever set after a match. The convenience functions for extracting the data by name returns the substring for the first (and in this example, the only) group of that name that matched. This saves searching to find which numbered group it was. (An alternative way of solving this problem is to use a "branch reset" group, as described in the previous section.)

If you make a backreference to a non-unique named group from elsewhere in the pattern, the groups to which the name refers are checked in the order in which they appear in the overall pattern. The first one that is set is used for the reference. For example, this pattern matches both "foofoo" and "barbar" but not "foobar" or "barfoo":

(?J)(?:(?<n>foo)|(?<n>bar))\k<n>

If you make a subroutine call to a non-unique named group, the one that corresponds to the first occurrence of the name is used. In the absence of duplicate numbers this is the one with the lowest number.

If you use a named reference in a condition test (see the section about conditions below), either to check whether a capture group has matched, or to check for recursion, all groups with the same name are tested. If the condition is true for any one of them, the overall condition is true. This is the same behaviour as testing by number. For further details of the interfaces for handling named capture groups, see the pcre2api documentation.

REPETITION¶

Repetition is specified by quantifiers, which can follow any of the following items:

a literal data character
the dot metacharacter
the \C escape sequence
the \R escape sequence
the \X escape sequence
an escape such as \d or \pL that matches a single character
a character class
a backreference
a parenthesized group (including lookaround assertions)
a subroutine call (recursive or otherwise)

The general repetition quantifier specifies a minimum and maximum number of permitted matches, by giving the two numbers in curly brackets (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example,

z{2,4}

matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special character. If the second number is omitted, but the comma is present, there is no upper limit; if the second number and the comma are both omitted, the quantifier specifies an exact number of required matches. Thus

[aeiou]{3,}

matches at least 3 successive vowels, but may match many more, whereas

\d{8}

matches exactly 8 digits. An opening curly bracket that appears in a position where a quantifier is not allowed, or one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantifier, but a literal string of four characters.

In UTF modes, quantifiers apply to characters rather than to individual code units. Thus, for example, \x{100}{2} matches two characters, each of which is represented by a two-byte sequence in a UTF-8 string. Similarly, \X{3} matches three Unicode extended grapheme clusters, each of which may be several code units long (and they may be of different lengths).

The quantifier {0} is permitted, causing the expression to behave as if the previous item and the quantifier were not present. This may be useful for capture groups that are referenced as subroutines from elsewhere in the pattern (but see also the section entitled "Defining capture groups for use by reference only" below). Except for parenthesized groups, items that have a {0} quantifier are omitted from the compiled pattern.

For convenience, the three most common quantifiers have single-character abbreviations:

* is equivalent to {0,}
+ is equivalent to {1,}
? is equivalent to {0,1}

It is possible to construct infinite loops by following a group that can match no characters with a quantifier that has no upper limit, for example:

(a?)*

Earlier versions of Perl and PCRE1 used to give an error at compile time for such patterns. However, because there are cases where this can be useful, such patterns are now accepted, but whenever an iteration of such a group matches no characters, matching moves on to the next item in the pattern instead of repeatedly matching an empty string. This does not prevent backtracking into any of the iterations if a subsequent item fails to match.

By default, quantifiers are "greedy", that is, they match as much as possible (up to the maximum number of permitted times), without causing the rest of the pattern to fail. The classic example of where this gives problems is in trying to match comments in C programs. These appear between /* and */ and within the comment, individual * and / characters may appear. An attempt to match C comments by applying the pattern

/\*.*\*/

to the string

/* first comment */ not comment /* second comment */

fails, because it matches the entire string owing to the greediness of the .* item. However, if a quantifier is followed by a question mark, it ceases to be greedy, and instead matches the minimum number of times possible, so the pattern

/\*.*?\*/

does the right thing with the C comments. The meaning of the various quantifiers is not otherwise changed, just the preferred number of matches. Do not confuse this use of question mark with its use as a quantifier in its own right. Because it has two uses, it can sometimes appear doubled, as in

\d??\d

which matches one digit by preference, but can match two if that is the only way the rest of the pattern matches.

If the PCRE2_UNGREEDY option is set (an option that is not available in Perl), the quantifiers are not greedy by default, but individual ones can be made greedy by following them with a question mark. In other words, it inverts the default behaviour.

When a parenthesized group is quantified with a minimum repeat count that is greater than 1 or with a limited maximum, more memory is required for the compiled pattern, in proportion to the size of the minimum or maximum.

If a pattern starts with .* or .{0,} and the PCRE2_DOTALL option (equivalent to Perl's /s) is set, thus allowing the dot to match newlines, the pattern is implicitly anchored, because whatever follows will be tried against every character position in the subject string, so there is no point in retrying the overall match at any position after the first. PCRE2 normally treats such a pattern as though it were preceded by \A.

In cases where it is known that the subject string contains no newlines, it is worth setting PCRE2_DOTALL in order to obtain this optimization, or alternatively, using ^ to indicate anchoring explicitly.

However, there are some cases where the optimization cannot be used. When .* is inside capturing parentheses that are the subject of a backreference elsewhere in the pattern, a match at the start may fail where a later one succeeds. Consider, for example:

(.*)abc\1

If the subject is "xyz123abc123" the match point is the fourth character. For this reason, such a pattern is not implicitly anchored.

Another case where implicit anchoring is not applied is when the leading .* is inside an atomic group. Once again, a match at the start may fail where a later one succeeds. Consider this pattern:

(?>.*?a)b

It matches "ab" in the subject "aab". The use of the backtracking control verbs (*PRUNE) and (*SKIP) also disable this optimization, and there is an option, PCRE2_NO_DOTSTAR_ANCHOR, to do so explicitly.

When a capture group is repeated, the value captured is the substring that matched the final iteration. For example, after

(tweedle[dume]{3}\s*)+

has matched "tweedledum tweedledee" the value of the captured substring is "tweedledee". However, if there are nested capture groups, the corresponding captured values may have been set in previous iterations. For example, after

(a|(b))+

matches "aba" the value of the second captured substring is "b".

ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS¶

With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy") repetition, failure of what follows normally causes the repeated item to be re-evaluated to see if a different number of repeats allows the rest of the pattern to match. Sometimes it is useful to prevent this, either to change the nature of the match, or to cause it fail earlier than it otherwise might, when the author of the pattern knows there is no point in carrying on.

Consider, for example, the pattern \d+foo when applied to the subject line

123456bar

After matching all 6 digits and then failing to match "foo", the normal action of the matcher is to try again with only 5 digits matching the \d+ item, and then with 4, and so on, before ultimately failing. "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides the means for specifying that once a group has matched, it is not to be re-evaluated in this way.

If we use atomic grouping for the previous example, the matcher gives up immediately on failing to match "foo" the first time. The notation is a kind of special parenthesis, starting with (?> as in this example:

(?>\d+)foo

Perl 5.28 introduced an experimental alphabetic form starting with (* which may be easier to remember:

(*atomic:\d+)foo

This kind of parenthesized group "locks up" the part of the pattern it contains once it has matched, and a failure further into the pattern is prevented from backtracking into it. Backtracking past it to previous items, however, works as normal.

An alternative description is that a group of this type matches exactly the string of characters that an identical standalone pattern would match, if anchored at the current point in the subject string.

Atomic groups are not capture groups. Simple cases such as the above example can be thought of as a maximizing repeat that must swallow everything it can. So, while both \d+ and \d+? are prepared to adjust the number of digits they match in order to make the rest of the pattern match, (?>\d+) can only match an entire sequence of digits.

Atomic groups in general can of course contain arbitrarily complicated expressions, and can be nested. However, when the contents of an atomic group is just a single repeated item, as in the example above, a simpler notation, called a "possessive quantifier" can be used. This consists of an additional + character following a quantifier. Using this notation, the previous example can be rewritten as

\d++foo

Note that a possessive quantifier can be used with an entire group, for example:

(abc|xyz){2,3}+

Possessive quantifiers are always greedy; the setting of the PCRE2_UNGREEDY option is ignored. They are a convenient notation for the simpler forms of atomic group. However, there is no difference in the meaning of a possessive quantifier and the equivalent atomic group, though there may be a performance difference; possessive quantifiers should be slightly faster.

The possessive quantifier syntax is an extension to the Perl 5.8 syntax. Jeffrey Friedl originated the idea (and the name) in the first edition of his book. Mike McCloskey liked it, so implemented it when he built Sun's Java package, and PCRE1 copied it from there. It found its way into Perl at release 5.10.

PCRE2 has an optimization that automatically "possessifies" certain simple pattern constructs. For example, the sequence A+B is treated as A++B because there is no point in backtracking into a sequence of A's when B must follow. This feature can be disabled by the PCRE2_NO_AUTOPOSSESS option, or starting the pattern with (*NO_AUTO_POSSESS).

When a pattern contains an unlimited repeat inside a group that can itself be repeated an unlimited number of times, the use of an atomic group is the only way to avoid some failing matches taking a very long time indeed. The pattern

(\D+|<\d+>)*[!?]

matches an unlimited number of substrings that either consist of non-digits, or digits enclosed in <>, followed by either ! or ?. When it matches, it runs quickly. However, if it is applied to

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

it takes a long time before reporting failure. This is because the string can be divided between the internal \D+ repeat and the external * repeat in a large number of ways, and all have to be tried. (The example uses [!?] rather than a single character at the end, because both PCRE2 and Perl have an optimization that allows for fast failure when a single character is used. They remember the last single character that is required for a match, and fail early if it is not present in the string.) If the pattern is changed so that it uses an atomic group, like this:

((?>\D+)|<\d+>)*[!?]

sequences of non-digits cannot be broken, and failure happens quickly.

BACKREFERENCES¶

Outside a character class, a backslash followed by a digit greater than 0 (and possibly further digits) is a backreference to a capture group earlier (that is, to its left) in the pattern, provided there have been that many previous capture groups.

However, if the decimal number following the backslash is less than 8, it is always taken as a backreference, and causes an error only if there are not that many capture groups in the entire pattern. In other words, the group that is referenced need not be to the left of the reference for numbers less than 8. A "forward backreference" of this type can make sense when a repetition is involved and the group to the right has participated in an earlier iteration.

It is not possible to have a numerical "forward backreference" to a group whose number is 8 or more using this syntax because a sequence such as \50 is interpreted as a character defined in octal. See the subsection entitled "Non-printing characters" above for further details of the handling of digits following a backslash. Other forms of backreferencing do not suffer from this restriction. In particular, there is no problem when named capture groups are used (see below).

Another way of avoiding the ambiguity inherent in the use of digits following a backslash is to use the \g escape sequence. This escape must be followed by a signed or unsigned number, optionally enclosed in braces. These examples are all identical:

(ring), \1
(ring), \g1
(ring), \g{1}

An unsigned number specifies an absolute reference without the ambiguity that is present in the older syntax. It is also useful when literal digits follow the reference. A signed number is a relative reference. Consider this example:

(abc(def)ghi)\g{-1}

The sequence \g{-1} is a reference to the most recently started capture group before \g, that is, is it equivalent to \2 in this example. Similarly, \g{-2} would be equivalent to \1. The use of relative references can be helpful in long patterns, and also in patterns that are created by joining together fragments that contain references within themselves.

The sequence \g{+1} is a reference to the next capture group. This kind of forward reference can be useful in patterns that repeat. Perl does not support the use of + in this way.

A backreference matches whatever actually most recently matched the capture group in the current subject string, rather than anything at all that matches the group (see "Groups as subroutines" below for a way of doing that). So the pattern

(sens|respons)e and \1ibility

matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If caseful matching is in force at the time of the backreference, the case of letters is relevant. For example,

((?i)rah)\s+\1

matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original capture group is matched caselessly.

There are several different ways of writing backreferences to named capture groups. The .NET syntax \k{name} and the Perl syntax \k<name> or \k'name' are supported, as is the Python syntax (?P=name). Perl 5.10's unified backreference syntax, in which \g can be used for both numeric and named references, is also supported. We could rewrite the above example in any of the following ways:

(?<p1>(?i)rah)\s+\k<p1>
(?'p1'(?i)rah)\s+\k{p1}
(?P<p1>(?i)rah)\s+(?P=p1)
(?<p1>(?i)rah)\s+\g{p1}

A capture group that is referenced by name may appear in the pattern before or after the reference.

There may be more than one backreference to the same group. If a group has not actually been used in a particular match, backreferences to it always fail by default. For example, the pattern

(a|(bc))\2

always fails if it starts to match "a" rather than "bc". However, if the PCRE2_MATCH_UNSET_BACKREF option is set at compile time, a backreference to an unset value matches an empty string.

Because there may be many capture groups in a pattern, all digits following a backslash are taken as part of a potential backreference number. If the pattern continues with a digit character, some delimiter must be used to terminate the backreference. If the PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, this can be white space. Otherwise, the \g{} syntax or an empty comment (see "Comments" below) can be used.

ASSERTIONS¶

An assertion is a test on the characters following or preceding the current matching point that does not consume any characters. The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are described above.

More complicated assertions are coded as parenthesized groups. There are two kinds: those that look ahead of the current position in the subject string, and those that look behind it, and in each case an assertion may be positive (must match for the assertion to be true) or negative (must not match for the assertion to be true). An assertion group is matched in the normal way, and if it is true, matching continues after it, but with the matching position in the subject string reset to what it was before the assertion was processed.

The Perl-compatible lookaround assertions are atomic. If an assertion is true, but there is a subsequent matching failure, there is no backtracking into the assertion. However, there are some cases where non-atomic assertions can be useful. PCRE2 has some support for these, described in the section entitled "Non-atomic assertions" below, but they are not Perl-compatible.

A lookaround assertion may appear as the condition in a conditional group (see below). In this case, the result of matching the assertion determines which branch of the condition is followed.

Assertion groups are not capture groups. If an assertion contains capture groups within it, these are counted for the purposes of numbering the capture groups in the whole pattern. Within each branch of an assertion, locally captured substrings may be referenced in the usual way. For example, a sequence such as (.)\g{-1} can be used to check that two adjacent characters are the same.

When a branch within an assertion fails to match, any substrings that were captured are discarded (as happens with any pattern branch that fails to match). A negative assertion is true only when all its branches fail to match; this means that no captured substrings are ever retained after a successful negative assertion. When an assertion contains a matching branch, what happens depends on the type of assertion.

For a positive assertion, internally captured substrings in the successful branch are retained, and matching continues with the next pattern item after the assertion. For a negative assertion, a matching branch means that the assertion is not true. If such an assertion is being used as a condition in a conditional group (see below), captured substrings are retained, because matching continues with the "no" branch of the condition. For other failing negative assertions, control passes to the previous backtracking point, thus discarding any captured strings within the assertion.

Most assertion groups may be repeated; though it makes no sense to assert the same thing several times, the side effect of capturing in positive assertions may occasionally be useful. However, an assertion that forms the condition for a conditional group may not be quantified. PCRE2 used to restrict the repetition of assertions, but from release 10.35 the only restriction is that an unlimited maximum repetition is changed to be one more than the minimum. For example, {3,} is treated as {3,4}.

NON-ATOMIC ASSERTIONS¶

The traditional Perl-compatible lookaround assertions are atomic. That is, if an assertion is true, but there is a subsequent matching failure, there is no backtracking into the assertion. However, there are some cases where non-atomic positive assertions can be useful. PCRE2 provides these using the following syntax:

(*non_atomic_positive_lookahead: or (*napla: or (?*
(*non_atomic_positive_lookbehind: or (*naplb: or (?<*

Consider the problem of finding the right-most word in a string that also appears earlier in the string, that is, it must appear at least twice in total. This pattern returns the required result as captured substring 1:

^(?x)(*napla: .* \b(\w++)) (?> .*? \b\1\b ){2}

For a subject such as "word1 word2 word3 word2 word3 word4" the result is "word3". How does it work? At the start, ^(?x) anchors the pattern and sets the "x" option, which causes white space (introduced for readability) to be ignored. Inside the assertion, the greedy .* at first consumes the entire string, but then has to backtrack until the rest of the assertion can match a word, which is captured by group 1. In other words, when the assertion first succeeds, it captures the right-most word in the string.

The current matching point is then reset to the start of the subject, and the rest of the pattern match checks for two occurrences of the captured word, using an ungreedy .*? to scan from the left. If this succeeds, we are done, but if the last word in the string does not occur twice, this part of the pattern fails. If a traditional atomic lookhead (?= or (*pla: had been used, the assertion could not be re-entered, and the whole match would fail. The pattern would succeed only if the very last word in the subject was found twice.

Using a non-atomic lookahead, however, means that when the last word does not occur twice in the string, the lookahead can backtrack and find the second-last word, and so on, until either the match succeeds, or all words have been tested.

Two conditions must be met for a non-atomic assertion to be useful: the contents of one or more capturing groups must change after a backtrack into the assertion, and there must be a backreference to a changed group later in the pattern. If this is not the case, the rest of the pattern match fails exactly as before because nothing has changed, so using a non-atomic assertion just wastes resources.

There is one exception to backtracking into a non-atomic assertion. If an (*ACCEPT) control verb is triggered, the assertion succeeds atomically. That is, a subsequent match failure cannot backtrack into the assertion.

Non-atomic assertions are not supported by the alternative matching function pcre2_dfa_match(). They are supported by JIT, but only if they do not contain any control verbs such as (*ACCEPT). (This may change in future). Note that assertions that appear as conditions for conditional groups (see below) must be atomic.

SCRIPT RUNS¶

In concept, a script run is a sequence of characters that are all from the same Unicode script such as Latin or Greek. However, because some scripts are commonly used together, and because some diacritical and other marks are used with multiple scripts, it is not that simple. There is a full description of the rules that PCRE2 uses in the section entitled "Script Runs" in the pcre2unicode documentation.

If part of a pattern is enclosed between (*script_run: or (*sr: and a closing parenthesis, it fails if the sequence of characters that it matches are not a script run. After a failure, normal backtracking occurs. Script runs can be used to detect spoofing attacks using characters that look the same, but are from different scripts. The string "paypal.com" is an infamous example, where the letters could be a mixture of Latin and Cyrillic. This pattern ensures that the matched characters in a sequence of non-spaces that follow white space are a script run:

\s+(*sr:\S+)

To be sure that they are all from the Latin script (for example), a lookahead can be used:

\s+(?=\p{Latin})(*sr:\S+)

This works as long as the first character is expected to be a character in that script, and not (for example) punctuation, which is allowed with any script. If this is not the case, a more creative lookahead is needed. For example, if digits, underscore, and dots are permitted at the start:

\s+(?=[0-9_.]*\p{Latin})(*sr:\S+)

In many cases, backtracking into a script run pattern fragment is not desirable. The script run can employ an atomic group to prevent this. Because this is a common requirement, a shorthand notation is provided by (*atomic_script_run: or (*asr:

(*asr:...) is the same as (*sr:(?>...))

Note that the atomic group is inside the script run. Putting it outside would not prevent backtracking into the script run pattern.

Support for script runs is not available if PCRE2 is compiled without Unicode support. A compile-time error is given if any of the above constructs is encountered. Script runs are not supported by the alternate matching function, pcre2_dfa_match() because they use the same mechanism as capturing parentheses.

Warning: The (*ACCEPT) control verb (see below) should not be used within a script run group, because it causes an immediate exit from the group, bypassing the script run checking.

CONDITIONAL GROUPS¶

It is possible to cause the matching process to obey a pattern fragment conditionally or to choose between two alternative fragments, depending on the result of an assertion, or whether a specific capture group has already been matched. The two possible forms of conditional group are:

(?(condition)yes-pattern)
(?(condition)yes-pattern|no-pattern)

If the condition is satisfied, the yes-pattern is used; otherwise the no-pattern (if present) is used. An absent no-pattern is equivalent to an empty string (it always matches). If there are more than two alternatives in the group, a compile-time error occurs. Each of the two alternatives may itself contain nested groups of any form, including conditional groups; the restriction to two alternatives applies only at the level of the condition itself. This pattern fragment is an example where the alternatives are complex:

(?(1) (A|B|C) | (D | (?(2)E|F) | E) )

There are five kinds of condition: references to capture groups, references to recursion, two pseudo-conditions called DEFINE and VERSION, and assertions.

COMMENTS¶

There are two ways of including comments in patterns that are processed by PCRE2. In both cases, the start of the comment must not be in a character class, nor in the middle of any other sequence of related characters such as (?: or a group name or number. The characters that make up a comment play no part in the pattern matching.

The sequence (?# marks the start of a comment that continues up to the next closing parenthesis. Nested parentheses are not permitted. If the PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is set, an unescaped # character also introduces a comment, which in this case continues to immediately after the next newline character or character sequence in the pattern. Which characters are interpreted as newlines is controlled by an option passed to the compiling function or by a special sequence at the start of the pattern, as described in the section entitled "Newline conventions" above. Note that the end of this type of comment is a literal newline sequence in the pattern; escape sequences that happen to represent a newline do not count. For example, consider this pattern when PCRE2_EXTENDED is set, and the default newline convention (a single linefeed character) is in force:

abc #comment \n still comment

On encountering the # character, pcre2_compile() skips along, looking for a newline in the pattern. The sequence \n is still literal at this stage, so it does not terminate the comment. Only an actual character with the code value 0x0a (the default newline) does so.

RECURSIVE PATTERNS¶

Consider the problem of matching a string in parentheses, allowing for unlimited nested parentheses. Without the use of recursion, the best that can be done is to use a pattern that matches up to some fixed depth of nesting. It is not possible to handle an arbitrary nesting depth.

For some time, Perl has provided a facility that allows regular expressions to recurse (amongst other things). It does this by interpolating Perl code in the expression at run time, and the code can refer to the expression itself. A Perl pattern using code interpolation to solve the parentheses problem can be created like this:

$re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x;

The (?p{...}) item interpolates Perl code at run time, and in this case refers recursively to the pattern in which it appears.

Obviously, PCRE2 cannot support the interpolation of Perl code. Instead, it supports special syntax for recursion of the entire pattern, and also for individual capture group recursion. After its introduction in PCRE1 and Python, this kind of recursion was subsequently introduced into Perl at release 5.10.

A special item that consists of (? followed by a number greater than zero and a closing parenthesis is a recursive subroutine call of the capture group of the given number, provided that it occurs inside that group. (If not, it is a non-recursive subroutine call, which is described in the next section.) The special item (?R) or (?0) is a recursive call of the entire regular expression.

This PCRE2 pattern solves the nested parentheses problem (assume the PCRE2_EXTENDED option is set so that white space is ignored):

\( ( [^()]++ | (?R) )* \)

First it matches an opening parenthesis. Then it matches any number of substrings which can either be a sequence of non-parentheses, or a recursive match of the pattern itself (that is, a correctly parenthesized substring). Finally there is a closing parenthesis. Note the use of a possessive quantifier to avoid backtracking into sequences of non-parentheses.

If this were part of a larger pattern, you would not want to recurse the entire pattern, so instead you could use this:

( \( ( [^()]++ | (?1) )* \) )

We have put the pattern into parentheses, and caused the recursion to refer to them instead of the whole pattern.

In a larger pattern, keeping track of parenthesis numbers can be tricky. This is made easier by the use of relative references. Instead of (?1) in the pattern above you can write (?-2) to refer to the second most recently opened parentheses preceding the recursion. In other words, a negative number counts capturing parentheses leftwards from the point at which it is encountered.

Be aware however, that if duplicate capture group numbers are in use, relative references refer to the earliest group with the appropriate number. Consider, for example:

(?|(a)|(b)) (c) (?-2)

The first two capture groups (a) and (b) are both numbered 1, and group (c) is number 2. When the reference (?-2) is encountered, the second most recently opened parentheses has the number 1, but it is the first such group (the (a) group) to which the recursion refers. This would be the same if an absolute reference (?1) was used. In other words, relative references are just a shorthand for computing a group number.

It is also possible to refer to subsequent capture groups, by writing references such as (?+2). However, these cannot be recursive because the reference is not inside the parentheses that are referenced. They are always non-recursive subroutine calls, as described in the next section.

An alternative approach is to use named parentheses. The Perl syntax for this is (?&name); PCRE1's earlier syntax (?P>name) is also supported. We could rewrite the above example as follows:

(?<pn> \( ( [^()]++ | (?&pn) )* \) )

If there is more than one group with the same name, the earliest one is used.

The example pattern that we have been looking at contains nested unlimited repeats, and so the use of a possessive quantifier for matching strings of non-parentheses is important when applying the pattern to strings that do not match. For example, when this pattern is applied to

(aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()

it yields "no match" quickly. However, if a possessive quantifier is not used, the match runs for a very long time indeed because there are so many different ways the + and * repeats can carve up the subject, and all have to be tested before failure can be reported.

At the end of a match, the values of capturing parentheses are those from the outermost level. If you want to obtain intermediate values, a callout function can be used (see below and the pcre2callout documentation). If the pattern above is matched against

(ab(cd)ef)

the value for the inner capturing parentheses (numbered 2) is "ef", which is the last value taken on at the top level. If a capture group is not matched at the top level, its final captured value is unset, even if it was (temporarily) set at a deeper level during the matching process.

Do not confuse the (?R) item with the condition (R), which tests for recursion. Consider this pattern, which matches text in angle brackets, allowing for arbitrary nesting. Only digits are allowed in nested brackets (that is, when recursing), whereas any characters are permitted at the outer level.

< (?: (?(R) \d++ | [^<>]*+) | (?R)) * >

In this pattern, (?(R) is the start of a conditional group, with two different alternatives for the recursive and non-recursive cases. The (?R) item is the actual recursive call.

GROUPS AS SUBROUTINES¶

If the syntax for a recursive group call (either by number or by name) is used outside the parentheses to which it refers, it operates a bit like a subroutine in a programming language. More accurately, PCRE2 treats the referenced group as an independent subpattern which it tries to match at the current matching position. The called group may be defined before or after the reference. A numbered reference can be absolute or relative, as in these examples:

(...(absolute)...)...(?2)...
(...(relative)...)...(?-1)...
(...(?+1)...(relative)...

An earlier example pointed out that the pattern

(sens|respons)e and \1ibility

matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If instead the pattern

(sens|respons)e and (?1)ibility

is used, it does match "sense and responsibility" as well as the other two strings. Another example is given in the discussion of DEFINE above.

Like recursions, subroutine calls used to be treated as atomic, but this changed at PCRE2 release 10.30, so backtracking into subroutine calls can now occur. However, any capturing parentheses that are set during the subroutine call revert to their previous values afterwards.

Processing options such as case-independence are fixed when a group is defined, so if it is used as a subroutine, such options cannot be changed for different calls. For example, consider this pattern:

(abc)(?i:(?-1))

It matches "abcabc". It does not match "abcABC" because the change of processing option does not affect the called group.

The behaviour of backtracking control verbs in groups when called as subroutines is described in the section entitled "Backtracking verbs in subroutines" below.

ONIGURUMA SUBROUTINE SYNTAX¶

For compatibility with Oniguruma, the non-Perl syntax \g followed by a name or a number enclosed either in angle brackets or single quotes, is an alternative syntax for calling a group as a subroutine, possibly recursively. Here are two of the examples used above, rewritten using this syntax:

(?<pn> \( ( (?>[^()]+) | \g<pn> )* \) )
(sens|respons)e and \g'1'ibility

PCRE2 supports an extension to Oniguruma: if a number is preceded by a plus or a minus sign it is taken as a relative reference. For example:

(abc)(?i:\g<-1>)

Note that \g{...} (Perl syntax) and \g<...> (Oniguruma syntax) are not synonymous. The former is a backreference; the latter is a subroutine call.

CALLOUTS¶

Perl has a feature whereby using the sequence (?{...}) causes arbitrary Perl code to be obeyed in the middle of matching a regular expression. This makes it possible, amongst other things, to extract different substrings that match the same pair of parentheses when there is a repetition.

PCRE2 provides a similar feature, but of course it cannot obey arbitrary Perl code. The feature is called "callout". The caller of PCRE2 provides an external function by putting its entry point in a match context using the function pcre2_set_callout(), and then passing that context to pcre2_match() or pcre2_dfa_match(). If no match context is passed, or if the callout entry point is set to NULL, callouts are disabled.

Within a regular expression, (?C<arg>) indicates a point at which the external function is to be called. There are two kinds of callout: those with a numerical argument and those with a string argument. (?C) on its own with no argument is treated as (?C0). A numerical argument allows the application to distinguish between different callouts. String arguments were added for release 10.20 to make it possible for script languages that use PCRE2 to embed short scripts within patterns in a similar way to Perl.

During matching, when PCRE2 reaches a callout point, the external function is called. It is provided with the number or string argument of the callout, the position in the pattern, and one item of data that is also set in the match block. The callout function may cause matching to proceed, to backtrack, or to fail.

By default, PCRE2 implements a number of optimizations at matching time, and one side-effect is that sometimes callouts are skipped. If you need all possible callouts to happen, you need to set options that disable the relevant optimizations. More details, including a complete description of the programming interface to the callout function, are given in the pcre2callout documentation.

BACKTRACKING CONTROL¶

There are a number of special "Backtracking Control Verbs" (to use Perl's terminology) that modify the behaviour of backtracking during matching. They are generally of the form (*VERB) or (*VERB:NAME). Some verbs take either form, and may behave differently depending on whether or not a name argument is present. The names are not required to be unique within the pattern.

By default, for compatibility with Perl, a name is any sequence of characters that does not include a closing parenthesis. The name is not processed in any way, and it is not possible to include a closing parenthesis in the name. This can be changed by setting the PCRE2_ALT_VERBNAMES option, but the result is no longer Perl-compatible.

When PCRE2_ALT_VERBNAMES is set, backslash processing is applied to verb names and only an unescaped closing parenthesis terminates the name. However, the only backslash items that are permitted are \Q, \E, and sequences such as \x{100} that define character code points. Character type escapes such as \d are faulted.

A closing parenthesis can be included in a name either as \) or between \Q and \E. In addition to backslash processing, if the PCRE2_EXTENDED or PCRE2_EXTENDED_MORE option is also set, unescaped whitespace in verb names is skipped, and #-comments are recognized, exactly as in the rest of the pattern. PCRE2_EXTENDED and PCRE2_EXTENDED_MORE do not affect verb names unless PCRE2_ALT_VERBNAMES is also set.

The maximum length of a name is 255 in the 8-bit library and 65535 in the 16-bit and 32-bit libraries. If the name is empty, that is, if the closing parenthesis immediately follows the colon, the effect is as if the colon were not there. Any number of these verbs may occur in a pattern. Except for (*ACCEPT), they may not be quantified.

Since these verbs are specifically related to backtracking, most of them can be used only when the pattern is to be matched using the traditional matching function, because that uses a backtracking algorithm. With the exception of (*FAIL), which behaves like a failing negative assertion, the backtracking control verbs cause an error if encountered by the DFA matching function.

The behaviour of these verbs in repeated groups, assertions, and in capture groups called as subroutines (whether or not recursively) is documented below.

SEE ALSO¶

pcre2api(3), pcre2callout(3), pcre2matching(3), pcre2syntax(3), pcre2(3).

AUTHOR¶

Philip Hazel
University Computing Service
Cambridge, England.

REVISION¶

Last updated: 06 October 2020
Copyright (c) 1997-2020 University of Cambridge.