regex_syntax/
lib.rs

1/*!
2This crate provides a robust regular expression parser.
3
4This crate defines two primary types:
5
6* [`Ast`](ast/enum.Ast.html) is the abstract syntax of a regular expression.
7  An abstract syntax corresponds to a *structured representation* of the
8  concrete syntax of a regular expression, where the concrete syntax is the
9  pattern string itself (e.g., `foo(bar)+`). Given some abstract syntax, it
10  can be converted back to the original concrete syntax (modulo some details,
11  like whitespace). To a first approximation, the abstract syntax is complex
12  and difficult to analyze.
13* [`Hir`](hir/struct.Hir.html) is the high-level intermediate representation
14  ("HIR" or "high-level IR" for short) of regular expression. It corresponds to
15  an intermediate state of a regular expression that sits between the abstract
16  syntax and the low level compiled opcodes that are eventually responsible for
17  executing a regular expression search. Given some high-level IR, it is not
18  possible to produce the original concrete syntax (although it is possible to
19  produce an equivalent concrete syntax, but it will likely scarcely resemble
20  the original pattern). To a first approximation, the high-level IR is simple
21  and easy to analyze.
22
23These two types come with conversion routines:
24
25* An [`ast::parse::Parser`](ast/parse/struct.Parser.html) converts concrete
26  syntax (a `&str`) to an [`Ast`](ast/enum.Ast.html).
27* A [`hir::translate::Translator`](hir/translate/struct.Translator.html)
28  converts an [`Ast`](ast/enum.Ast.html) to a [`Hir`](hir/struct.Hir.html).
29
30As a convenience, the above two conversion routines are combined into one via
31the top-level [`Parser`](struct.Parser.html) type. This `Parser` will first
32convert your pattern to an `Ast` and then convert the `Ast` to an `Hir`.
33
34
35# Example
36
37This example shows how to parse a pattern string into its HIR:
38
39```
40use regex_syntax::Parser;
41use regex_syntax::hir::{self, Hir};
42
43let hir = Parser::new().parse("a|b").unwrap();
44assert_eq!(hir, Hir::alternation(vec![
45    Hir::literal(hir::Literal::Unicode('a')),
46    Hir::literal(hir::Literal::Unicode('b')),
47]));
48```
49
50
51# Concrete syntax supported
52
53The concrete syntax is documented as part of the public API of the
54[`regex` crate](https://docs.rs/regex/%2A/regex/#syntax).
55
56
57# Input safety
58
59A key feature of this library is that it is safe to use with end user facing
60input. This plays a significant role in the internal implementation. In
61particular:
62
631. Parsers provide a `nest_limit` option that permits callers to control how
64   deeply nested a regular expression is allowed to be. This makes it possible
65   to do case analysis over an `Ast` or an `Hir` using recursion without
66   worrying about stack overflow.
672. Since relying on a particular stack size is brittle, this crate goes to
68   great lengths to ensure that all interactions with both the `Ast` and the
69   `Hir` do not use recursion. Namely, they use constant stack space and heap
70   space proportional to the size of the original pattern string (in bytes).
71   This includes the type's corresponding destructors. (One exception to this
72   is literal extraction, but this will eventually get fixed.)
73
74
75# Error reporting
76
77The `Display` implementations on all `Error` types exposed in this library
78provide nice human readable errors that are suitable for showing to end users
79in a monospace font.
80
81
82# Literal extraction
83
84This crate provides limited support for
85[literal extraction from `Hir` values](hir/literal/struct.Literals.html).
86Be warned that literal extraction currently uses recursion, and therefore,
87stack size proportional to the size of the `Hir`.
88
89The purpose of literal extraction is to speed up searches. That is, if you
90know a regular expression must match a prefix or suffix literal, then it is
91often quicker to search for instances of that literal, and then confirm or deny
92the match using the full regular expression engine. These optimizations are
93done automatically in the `regex` crate.
94
95
96# Crate features
97
98An important feature provided by this crate is its Unicode support. This
99includes things like case folding, boolean properties, general categories,
100scripts and Unicode-aware support for the Perl classes `\w`, `\s` and `\d`.
101However, a downside of this support is that it requires bundling several
102Unicode data tables that are substantial in size.
103
104A fair number of use cases do not require full Unicode support. For this
105reason, this crate exposes a number of features to control which Unicode
106data is available.
107
108If a regular expression attempts to use a Unicode feature that is not available
109because the corresponding crate feature was disabled, then translating that
110regular expression to an `Hir` will return an error. (It is still possible
111construct an `Ast` for such a regular expression, since Unicode data is not
112used until translation to an `Hir`.) Stated differently, enabling or disabling
113any of the features below can only add or subtract from the total set of valid
114regular expressions. Enabling or disabling a feature will never modify the
115match semantics of a regular expression.
116
117The following features are available:
118
119* **unicode** -
120  Enables all Unicode features. This feature is enabled by default, and will
121  always cover all Unicode features, even if more are added in the future.
122* **unicode-age** -
123  Provide the data for the
124  [Unicode `Age` property](https://www.unicode.org/reports/tr44/tr44-24.html#Character_Age).
125  This makes it possible to use classes like `\p{Age:6.0}` to refer to all
126  codepoints first introduced in Unicode 6.0
127* **unicode-bool** -
128  Provide the data for numerous Unicode boolean properties. The full list
129  is not included here, but contains properties like `Alphabetic`, `Emoji`,
130  `Lowercase`, `Math`, `Uppercase` and `White_Space`.
131* **unicode-case** -
132  Provide the data for case insensitive matching using
133  [Unicode's "simple loose matches" specification](https://www.unicode.org/reports/tr18/#Simple_Loose_Matches).
134* **unicode-gencat** -
135  Provide the data for
136  [Uncode general categories](https://www.unicode.org/reports/tr44/tr44-24.html#General_Category_Values).
137  This includes, but is not limited to, `Decimal_Number`, `Letter`,
138  `Math_Symbol`, `Number` and `Punctuation`.
139* **unicode-perl** -
140  Provide the data for supporting the Unicode-aware Perl character classes,
141  corresponding to `\w`, `\s` and `\d`. This is also necessary for using
142  Unicode-aware word boundary assertions. Note that if this feature is
143  disabled, the `\s` and `\d` character classes are still available if the
144  `unicode-bool` and `unicode-gencat` features are enabled, respectively.
145* **unicode-script** -
146  Provide the data for
147  [Unicode scripts and script extensions](https://www.unicode.org/reports/tr24/).
148  This includes, but is not limited to, `Arabic`, `Cyrillic`, `Hebrew`,
149  `Latin` and `Thai`.
150* **unicode-segment** -
151  Provide the data necessary to provide the properties used to implement the
152  [Unicode text segmentation algorithms](https://www.unicode.org/reports/tr29/).
153  This enables using classes like `\p{gcb=Extend}`, `\p{wb=Katakana}` and
154  `\p{sb=ATerm}`.
155*/
156
157#![deny(missing_docs)]
158#![forbid(unsafe_code)]
159
160pub use error::{Error, Result};
161pub use parser::{Parser, ParserBuilder};
162pub use unicode::UnicodeWordError;
163
164pub mod ast;
165mod either;
166mod error;
167pub mod hir;
168mod parser;
169mod unicode;
170mod unicode_tables;
171pub mod utf8;
172
173/// Escapes all regular expression meta characters in `text`.
174///
175/// The string returned may be safely used as a literal in a regular
176/// expression.
177pub fn escape(text: &str) -> String {
178    let mut quoted = String::with_capacity(text.len());
179    escape_into(text, &mut quoted);
180    quoted
181}
182
183/// Escapes all meta characters in `text` and writes the result into `buf`.
184///
185/// This will append escape characters into the given buffer. The characters
186/// that are appended are safe to use as a literal in a regular expression.
187pub fn escape_into(text: &str, buf: &mut String) {
188    for c in text.chars() {
189        if is_meta_character(c) {
190            buf.push('\\');
191        }
192        buf.push(c);
193    }
194}
195
196/// Returns true if the give character has significance in a regex.
197///
198/// These are the only characters that are allowed to be escaped, with one
199/// exception: an ASCII space character may be escaped when extended mode (with
200/// the `x` flag) is enabld. In particular, `is_meta_character(' ')` returns
201/// `false`.
202///
203/// Note that the set of characters for which this function returns `true` or
204/// `false` is fixed and won't change in a semver compatible release.
205pub fn is_meta_character(c: char) -> bool {
206    match c {
207        '\\' | '.' | '+' | '*' | '?' | '(' | ')' | '|' | '[' | ']' | '{'
208        | '}' | '^' | '$' | '#' | '&' | '-' | '~' => true,
209        _ => false,
210    }
211}
212
213/// Returns true if and only if the given character is a Unicode word
214/// character.
215///
216/// A Unicode word character is defined by
217/// [UTS#18 Annex C](http://unicode.org/reports/tr18/#Compatibility_Properties).
218/// In particular, a character
219/// is considered a word character if it is in either of the `Alphabetic` or
220/// `Join_Control` properties, or is in one of the `Decimal_Number`, `Mark`
221/// or `Connector_Punctuation` general categories.
222///
223/// # Panics
224///
225/// If the `unicode-perl` feature is not enabled, then this function panics.
226/// For this reason, it is recommended that callers use
227/// [`try_is_word_character`](fn.try_is_word_character.html)
228/// instead.
229pub fn is_word_character(c: char) -> bool {
230    try_is_word_character(c).expect("unicode-perl feature must be enabled")
231}
232
233/// Returns true if and only if the given character is a Unicode word
234/// character.
235///
236/// A Unicode word character is defined by
237/// [UTS#18 Annex C](http://unicode.org/reports/tr18/#Compatibility_Properties).
238/// In particular, a character
239/// is considered a word character if it is in either of the `Alphabetic` or
240/// `Join_Control` properties, or is in one of the `Decimal_Number`, `Mark`
241/// or `Connector_Punctuation` general categories.
242///
243/// # Errors
244///
245/// If the `unicode-perl` feature is not enabled, then this function always
246/// returns an error.
247pub fn try_is_word_character(
248    c: char,
249) -> std::result::Result<bool, UnicodeWordError> {
250    unicode::is_word_character(c)
251}
252
253/// Returns true if and only if the given character is an ASCII word character.
254///
255/// An ASCII word character is defined by the following character class:
256/// `[_0-9a-zA-Z]'.
257pub fn is_word_byte(c: u8) -> bool {
258    match c {
259        b'_' | b'0'..=b'9' | b'a'..=b'z' | b'A'..=b'Z' => true,
260        _ => false,
261    }
262}
263
264#[cfg(test)]
265mod tests {
266    use super::*;
267
268    #[test]
269    fn escape_meta() {
270        assert_eq!(
271            escape(r"\.+*?()|[]{}^$#&-~"),
272            r"\\\.\+\*\?\(\)\|\[\]\{\}\^\$\#\&\-\~".to_string()
273        );
274    }
275
276    #[test]
277    fn word_byte() {
278        assert!(is_word_byte(b'a'));
279        assert!(!is_word_byte(b'-'));
280    }
281
282    #[test]
283    #[cfg(feature = "unicode-perl")]
284    fn word_char() {
285        assert!(is_word_character('a'), "ASCII");
286        assert!(is_word_character('à'), "Latin-1");
287        assert!(is_word_character('β'), "Greek");
288        assert!(is_word_character('\u{11011}'), "Brahmi (Unicode 6.0)");
289        assert!(is_word_character('\u{11611}'), "Modi (Unicode 7.0)");
290        assert!(is_word_character('\u{11711}'), "Ahom (Unicode 8.0)");
291        assert!(is_word_character('\u{17828}'), "Tangut (Unicode 9.0)");
292        assert!(is_word_character('\u{1B1B1}'), "Nushu (Unicode 10.0)");
293        assert!(is_word_character('\u{16E40}'), "Medefaidrin (Unicode 11.0)");
294        assert!(!is_word_character('-'));
295        assert!(!is_word_character('☃'));
296    }
297
298    #[test]
299    #[should_panic]
300    #[cfg(not(feature = "unicode-perl"))]
301    fn word_char_disabled_panic() {
302        assert!(is_word_character('a'));
303    }
304
305    #[test]
306    #[cfg(not(feature = "unicode-perl"))]
307    fn word_char_disabled_error() {
308        assert!(try_is_word_character('a').is_err());
309    }
310}