Parsing is everywhere in software development — from configuration files to query languages, from log analysis to domain-specific languages. Yet many developers shy away from building custom parsers, thinking they’re complex and difficult to maintain. Enter NimbleParsec, an Elixir library that makes parser construction not just approachable, but genuinely enjoyable.
What Makes NimbleParsec Special?
NimbleParsec is a parser combinator library that compiles your parser definitions into efficient Erlang bytecode at compile time. Unlike runtime parsing solutions, this approach gives you both excellent performance and compile-time guarantees about your parser’s correctness.
The library follows the combinator pattern, where you build complex parsers by combining simpler ones. Think of it like LEGO blocks for parsing — each piece is simple and well-defined, but together they can create sophisticated structures.
Real-World Applications
Before diving into code, let’s understand where NimbleParsec shines:
Configuration Languages: Instead of wrestling with complex regex patterns or hand-written state machines, you can elegantly parse custom configuration syntax.
Query Languages: Build domain-specific query languages for your applications that feel natural to your users.
Log Processing: Parse structured log formats with complex nested data efficiently.
Data Transformation: Convert between different data formats with parsers that are both readable and maintainable.
Template Languages: Create custom templating systems with sophisticated syntax.
The Power of DSLs
Domain-Specific Languages (DSLs) are where parser combinators truly shine. Let’s see the difference between traditional parsing and the NimbleParsec approach:
Traditional Approach (Hand-written parser):
def parse_assignment(input) do
case Regex.run(~r/^(\w+)\s*=\s*(.+)$/, input) do
[_, var, value] ->
case parse_value(value) do
{:ok, parsed_value} -> {:ok, {var, parsed_value}}
error -> error
end
nil -> {:error, "Invalid assignment"}
end
endCode language: Elixir (elixir)def parse_value(input) do
cond do
Regex.match?(~r/^\d+$/, input) ->
{:ok, String.to_integer(input)}
Regex.match?(~r/^\d+\.\d+$/, input) ->
{:ok, String.to_float(input)}
# ... more cases
true -> {:error, "Invalid value"}
end
endCode language: Elixir (elixir)NimbleParsec Approach:
defmodule MiniLang do
import NimbleParsec
number =
integer(min: 1)
|> unwrap_and_tag(:number)
float_number =
integer(min: 1)
|> string(".")
|> integer(min: 1)
|> reduce({Enum, :join, [""]})
|> map({String, :to_float, []})
|> unwrap_and_tag(:float)
identifier =
ascii_char([?a..?z, ?A..?Z])
|> repeat(ascii_char([?a..?z, ?A..?Z, ?0..?9, ?_]))
|> reduce({Enum, :join, [""]})
|> unwrap_and_tag(:identifier)
assignment =
identifier
|> ignore(string("="))
|> choice([float_number, number])
|> tag(:assignment)
defparsec(:parse_assignment, assignment)
endCode language: Elixir (elixir)The NimbleParsec version is not only more readable but also more composable and testable. Each parser component has a single responsibility and can be reused in different contexts.
Building a Mini Calculator DSL
Let’s build something practical that showcases the real power of parser combinators. We’ll create a calculator that can handle variables, basic arithmetic, and multiple statements.
Our target syntax:
x = 10
y = x * 2 + 5
result = y / 3
area = 3.14 * radius * radiusCode language: Elixir (elixir)Here’s the complete implementation:
defmodule Calculator do
import NimbleParsec
# Basic building blocks
whitespace = ascii_char([?\s, ?\t]) |> times(min: 0)
number =
optional(string("-"))
|> integer(min: 1)
|> optional(string(".") |> integer(min: 1))
|> reduce({IO, :iodata_to_binary, []})
|> map({String, :to_float, []})
|> unwrap_and_tag(:number)
identifier =
ascii_char([?a..?z, ?A..?Z, ?_])
|> repeat(ascii_char([?a..?z, ?A..?Z, ?0..?9, ?_]))
|> reduce({Enum, :join, [""]})
|> unwrap_and_tag(:variable)
# Arithmetic operators with precedence
defcombinatorp(:expression,
parsec(:term)
|> repeat(
choice([string("+"), string("-")])
|> concat(parsec(:term))
)
|> reduce(:build_expression)
)
defcombinatorp(:term,
parsec(:factor)
|> repeat(
choice([string("*"), string("/")])
|> concat(parsec(:factor))
)
|> reduce(:build_expression)
)
defcombinatorp(:factor,
choice([
number,
identifier,
ignore(string("("))
|> parsec(:expression)
|> ignore(string(")"))
])
|> ignore(whitespace)
)
# Assignment statement
assignment =
identifier
|> ignore(whitespace)
|> ignore(string("="))
|> ignore(whitespace)
|> parsec(:expression)
|> reduce(:build_assignment)
# Program is multiple statements
statement =
choice([assignment])
|> ignore(whitespace)
|> ignore(optional(string(";")))
|> ignore(whitespace)
program =
ignore(whitespace)
|> repeat(statement)
|> eos()
defparsec(:parse, program)
# Helper functions for building the AST
def build_expression([left | rest]) do
Enum.chunk_every(rest, 2)
|> Enum.reduce(left, fn [op, right], acc ->
{:binary_op, op, acc, right}
end)
end
def build_assignment([{:variable, var}, expr]) do
{:assignment, var, expr}
end
# Evaluation engine
def evaluate(program_text) do
case parse(program_text) do
{:ok, ast, "", _, _, _} ->
execute_program(ast, %{})
{:ok, _, remaining, _, _, _} ->
{:error, "Unexpected input: #{remaining}"}
{:error, reason, _, _, _, _} ->
{:error, reason}
end
end
defp execute_program(statements, context) do
Enum.reduce_while(statements, {:ok, context}, fn statement, {:ok, ctx} ->
case execute_statement(statement, ctx) do
{:ok, new_ctx} -> {:cont, {:ok, new_ctx}}
error -> {:halt, error}
end
end)
end
defp execute_statement({:assignment, var, expr}, context) do
case evaluate_expression(expr, context) do
{:ok, value} -> {:ok, Map.put(context, var, value)}
error -> error
end
end
defp evaluate_expression({:number, value}, _context), do: {:ok, value}
defp evaluate_expression({:variable, name}, context) do
case Map.get(context, name) do
nil -> {:error, "Undefined variable: #{name}"}
value -> {:ok, value}
end
end
defp evaluate_expression({:binary_op, op, left, right}, context) do
with {:ok, left_val} <- evaluate_expression(left, context),
{:ok, right_val} <- evaluate_expression(right, context) do
case op do
"+" -> {:ok, left_val + right_val}
"-" -> {:ok, left_val - right_val}
"*" -> {:ok, left_val * right_val}
"/" when right_val != 0 -> {:ok, left_val / right_val}
"/" -> {:error, "Division by zero"}
end
end
end
endCode language: Elixir (elixir)Let’s test our calculator:
program = """
x = 10
y = x * 2 + 5
result = y / 3
radius = 5
area = 3.14159 * radius * radius
"""Code language: PHP (php)case Calculator.evaluate(program) do
{:ok, variables} ->
IO.inspect(variables)
# Output: %{"area" => 78.53975, "radius" => 5.0, "result" => 8.333333333333334, "x" => 10.0, "y" => 25.0}
{:error, reason} ->
IO.puts("Error: #{reason}")
endCode language: PHP (php)What This Demonstrates
This calculator showcases several powerful aspects of NimbleParsec:
Composability: We built complex parsing behavior by combining simple parsers. The expression parser reuses term, which reuses factor.
Operator Precedence: Mathematical operator precedence is handled naturally through the parser structure, not complex conditional logic.
Error Handling: The parser provides meaningful error messages and handles edge cases gracefully.
Separation of Concerns: Parsing and evaluation are cleanly separated, making the code easier to test and maintain.
Extensibility: Adding new operators or language features is straightforward — you modify the relevant parser combinator without touching unrelated code.
Advanced Features Worth Exploring
Custom Error Messages: NimbleParsec allows you to provide custom error messages that help users understand exactly what went wrong and where.
Streaming Support: For large inputs, NimbleParsec can work with streams, processing data incrementally rather than loading everything into memory.
Compile-time Optimizations: The library performs optimizations at compile time, eliminating backtracking where possible and generating efficient code.
Integration with LiveView: Parser combinators work beautifully with Phoenix LiveView for building interactive code editors with real-time syntax validation.
Performance Considerations
One of NimbleParsec’s key strengths is performance. Because parsers compile to efficient bytecode, they often outperform runtime alternatives significantly. The library also provides tools for profiling and optimizing your parsers.
For our calculator example, parsing and evaluating a 1000-line program takes just a few milliseconds on typical hardware — fast enough for interactive applications.
Getting Started
Adding NimbleParsec to your project is straightforward:
# In mix.exs
defp deps do
[
{:nimble_parsec, "~> 1.0"}
]
endCode language: Elixir (elixir)Start small with simple parsers and gradually build complexity. The combinator approach makes it easy to develop incrementally, testing each piece as you go.
Conclusion
NimbleParsec transforms parsing from a daunting task into an elegant composition exercise. Whether you’re building configuration languages, query processors, or domain-specific tools, parser combinators provide a path to maintainable, efficient, and genuinely readable parsing code.
The calculator we built demonstrates that even complex language features like operator precedence and variable scoping become natural when expressed through combinators. Most importantly, the resulting code tells a story about what it’s parsing — future developers (including yourself) will thank you for the clarity.
Start experimenting with NimbleParsec in your next project that involves parsing. You might be surprised how many parsing problems you’ve been solving the hard way, and how much more enjoyable they become with the right tools.
If you want the deep dive in NimbleParsec & see what amazing project I built, let me know I will publish it for you.
You can support me by https://buymeacoffee.com/y316nitka if you want more in-depth and practical blogs, Thank You!
