Parser combinator

In any programming language that has first-class functions, parser combinators can be used to combine basic parsers to construct parsers for more complex rules. For example, a production rule of a context-free grammar (CFG) may have one or more alternatives and each alternative may consist of a sequence of non-terminal(s) and/or terminal(s), or the alternative may consist of a single non-terminal or terminal or the empty string. If a simple parser is available for each of these alternatives, a parser combinator can be used to combine each of these parsers, returning a new parser which can recognise any or all of the alternatives.

In languages that support operator overloading, a parser combinator can take the form of an infix operator, used to glue different parsers to form a complete rule. Parser combinators thereby enable parsers to be defined in an embedded style, in code which is similar in structure to the rules of the formal grammar. As such, implementations can be thought of as executable specifications with all the associated advantages. (Notably: readability)

To keep the discussion relatively straightforward, we discuss parser combinators in terms of recognizers only. If the input string is of length #input and its members are accessed through an index j, a recognizer is a parser which returns, as output, a set of indices representing positions at which the parser successfully finished recognizing a sequence of tokens that began at position j. An empty result set indicates that the recognizer failed to recognize any sequence beginning at index j. A non-empty result set indicates the recognizer ends at different positions successfully.

• The empty recognizer recognizes the empty string. This parser always succeeds, returning a singleton set containing the current position:

${\displaystyle empty(j)=\{j\}}$

• A recognizer term x recognizes the terminal x. If the token at position j in the input string is x, this parser returns a singleton set containing j + 1; otherwise, it returns the empty set.

${\displaystyle term(x,j)={\begin{cases}\left\{\right\},&j\geq \#input\\\left\{j+1\right\},&j^{th}{\mbox{ element of }}input=x\\\left\{\right\},&{\mbox{otherwise}}\end{cases}}}$

Note that there may be multiple distinct ways to parse a string while finishing at the same index: this indicates an ambiguous grammar. Simple recognizers do not acknowledge these ambiguities; each possible finishing index is listed only once in the result set. For a more complete set of results, a more complicated object such as a parse tree must be returned.

Following the definitions of two basic recognizers p and q, we can define two major parser combinators for alternative and sequencing:

• The ‘alternative’ parser combinator, ⊕, applies both of the recognizers on the same input position j and sums up the results returned by both of the recognizers, which is eventually returned as the final result. It is used as an infix operator between p and q as follows:

${\displaystyle (p\oplus q)(j)=p(j)\cup q(j)}$

• The sequencing of recognizers is done with the ⊛ parser combinator. Like ⊕, it is used as an infix operator between p and q. But it applies the first recognizer p to the input position j, and if there is any successful result of this application, then the second recognizer q is applied to every element of the result set returned by the first recognizer. ⊛ ultimately returns the union of these applications of q.

${\displaystyle (p\circledast q)(j)=\bigcup \{q(k):k\in p(j)\}}$

Consider a highly ambiguous context-free grammar, s ::= ‘x’ s s | ε. Using the combinators defined earlier, we can modularly define executable notations of this grammar in a modern functional language (e.g. Haskell) as s = term ‘x’ <*> s <*> s <+> empty. When the recognizer s is applied on an input sequence xxxxx at position 1, according to the above definitions it would return a result set {5,4,3,2}.

Parser combinators, like all recursive descent parsers, are not limited to the context-free grammars and thus do no global search for ambiguities in the LL(k) parsing First_k and Follow_k sets. Thus, ambiguities are not known until run-time if and until the input triggers them. In such cases, the recursive descent parser may default (perhaps unknown to the grammar designer) to one of the possible ambiguous paths, resulting in semantic confusion (aliasing) in the use of the language. This leads to bugs by users of ambiguous programming languages, which are not reported at compile-time, and which are introduced not by human error, but by ambiguous grammar. The only solution that eliminates these bugs is to remove the ambiguities and use a context-free grammar.

The simple implementations of parser combinators have some shortcomings, which are common in top-down parsing. Naïve combinatory parsing requires exponential time and space when parsing an ambiguous context-free grammar. In 1996, Frost and Szydlowski demonstrated how memoization can be used with parser combinators to reduce the time complexity to polynomial.[5] Later Frost used monads to construct the combinators for systematic and correct threading of memo-table throughout the computation.[6]

Like any top-down recursive descent parsing, the conventional parser combinators (like the combinators described above) will not terminate while processing a left-recursive grammar (e.g. s ::= s <*> term ‘x’|empty). A recognition algorithm that accommodates ambiguous grammars with direct left-recursive rules is described by Frost and Hafiz in 2006.[7] The algorithm curtails the otherwise ever-growing left-recursive parse by imposing depth restrictions. That algorithm was extended to a complete parsing algorithm to accommodate indirect as well as direct left-recursion in polynomial time, and to generate compact polynomial-size representations of the potentially exponential number of parse trees for highly ambiguous grammars by Frost, Hafiz and Callaghan in 2007.[8] This extended algorithm accommodates indirect left recursion by comparing its ‘computed context’ with ‘current context’. The same authors also described their implementation of a set of parser combinators written in the Haskell programming language based on the same algorithm.[4][9]

• parser-combinator: Common Lisp parser combinator implementation
• Parsec: Industrial strength, monadic parser combinator library for Haskell
• parsec: Go version of Parsec
• FParsec: F# adaptation of Parsec
• csharp-monad: C# port of Parsec
• Jparsec: Java port of Parsec
• Arcsecond: Javascript fantasy-land compliant monadic parser combinator library
• Ramble: R parser combinator implementation
• nom: Rust parser combinator implementation using zero copy.
• pyparsing: Python parser combinator implementation, though it does not call itself that in its documentation.
• ts-parsec: TypeScript parser combinator library
• Diesel: Swift parser combinator library