M5 Text Processing Language User’s Guide

M4 has full access to its host environment (similar to most programming and scripting languages, but unlike many macro preprocessors). Malware can easily do harm. Third- party M5 code should be carefully vetted before use, or M5 should be run within a contained environment. M5 provides a simple mechanism for library inclusion by URL (or it will). This enables easy execution of public third-party code, so use it with extreme caution.

M4 does not provide any library, namespace, and version management facilities. Though M5 does not currently address these needs, plans have been sketched in code comments.

While macro processing is all about string processing, safely manipulating arbitrary strings is not possible in M4 or it is beyond awkward at best. M4 provides m4_regexp, m4_patsubst, and m4_substr. These return unquoted strings that will necessarily be elaborated, potentially altering the string. While M5 is able to jump through hoops to provide m5_regex and m5_substr (for strings of limited length) that return quoted (literal) text, m4_patsubst cannot be fixed (though m5_for_each_regex is similar). The result of m4_patsubst can be quoted only by quoting the input string, which can complicate the match expression, or by ensuring that all text is matched, which can be awkward, and quoting substitutions.

In addition to these issues, care must be taken to ensure that resulting text does not contain mismatching quotes or parentheses or combine with surrounding text to result in the same. Such resulting mismatches are difficult to debug. M5 provides a notion of "unquoted strings" that can be safely manipulated using m5_regex, and m5_substr.

Additionally the regex configuration used by M4 is quite dated. For example, it does not support lookahead, lazy matches, and character codes.

Introspection is essentially impossible. The only way to see what is defined is to dump definitions to a file and parse this file.

Recursion has a fixed (command-line) depth limit, and this limit is not applied reliably.

M4 is an old tool and was built for ASCII text. UTF-8 is now the most common text format. It is a superset of ASCII that encodes additional characters as two or more bytes using byte codes (0x10-0xFF) that do not conflict by those defined by ASCII (0x00-0x7F). All such bytes (0x10-0xFF) are treated as characters by M4 with no special meaning, so these characters pass through, unaffected, in macro processing like most others. There are two implications to be aware of. First, m5_length provides a length in bytes, not characters. Second, m5_substr and regular expressions manipulate bytes, not characters. This can result in text being split in the mid-character, resulting in invalid character encodings.

M4’s facilities for associating output with input only map output lines to line numbers of top-level calls. M4 does not maintain a call stack with line numbers.

M4 and M5 have no debugger to step through code. Printing (see m5_DEBUG is the debugging mechanism of choice.

M5 is intended for text processing, not for compute-intensive algorithms. Use a programming language for that.

M5 is for text processing only.

Major next steps include:

See issues file in the M5 repository for more details.

M5 comments are one form of syntactic sugar. They look like:

Block comments begin with /** and end with **/. Line comments begin with /// and end with a newline. Both are stripped prior to any other processing. As such:

Whitespace preceding a line comment is also stripped. Newlines from block comments are preserved.

In case /// or /** are needed in the resulting file, quotes can be used, e.g.: ['//']['/'], to disrupt the syntax.

Comments in the target language are not recognized as comments by M5. To disable M5 code, it is important to use M5 comments, not target-language comments. (Thus it can be especially problematic when one’s editor mode highlights target-language comments in a manner that suggests the code has no impact.)

These are specific to Code Blocks, introduced later.

A "body" is a parameter or macro value that is to be be evaluated in the context of a caller. Macros, like m5_if and m5_loop have "immediate" body parameters. These bodies are to be evaluated by calls to these macros themselves. The final argument to a function or macro declaration is an "indirect" body argument. This body is to be evaluated, not by the declaration macro itself, but by the caller of the macro it declares.

Code Blocks are convenient syntactic sugar constructs for multi-line body arguments formatted like code.

[Text blocks] are syntactic sugar for specifying multi-line blocks of arbitrary text, indented with the code.

A body argument can be provided as a quoted string of text:

Note that the quoting of ['Yes, '] prevents misinterpretation of the , as an argument separator as the body is evaluated.

This syntax is fine for simple text substitutions, but it is essentially restricted to a single line which is unreadable for larger bodies that might define local variables, perform calculations, evaluate code conditionally, iterate in loops, call other functions, recurse, etc.

M5 supports special multi-line syntactic sugar convenient for body arguments, called "code blocks". These look more like blocks of code in a traditional programming language. Aside from comments and whitespace, they contain only macro calls and variable elaborations ("statements"). The resulting text of the code block is constructed from the results of these macro calls.

The code below is equivalent to the example above, expressed using a code body (and assuming it is itself called from within a code body).

The block begins with [, followed immediately by a newline. It ends with a line that begins with ], indented consistently with the beginning line. The above code block is "unscoped". A "scoped" code block uses, instead, { and }. Scopes are detailed in Scoped Code Blocks.

The first non-blank line of the block determines the indentation of the block. Indentation uses spaces; tabs are discouraged, but must be used consistently if they are used. All non-blank lines at this level of indentation (after stripping M5 comments) begin a "statement". Lines with deeper indentation would continue a statement. A continuation line either begins a macro argument or is part of its own (nested) code block argument.

Essentially, the body, when evaluated, results in the text produced by its statements, which are macros or variables, listed without their m5_ prefix, or inline text.

Specifically, statements can be:

Statements that produce output (as all statements in the above example’s code block do) must be preceded by ~ (and others may be). This simply helps to identify the source of code block ouput. The ~(…​) syntax produces the given text. A m5_ prefix is implicit on statements. In the rare (and discouraged) event that a macro without this prefix is to be called, such as use of an m4_ macro, using ~out(m4_…​) will do the trick.

The earlier example behaves the same as:

The (internal) m5__block macro evaluates its argument and results in any text captured by m5_out.

Scoped Code Blocks are delimited by { / } quotes. Within a code block, variable declarations (e.g. made by m5_var) are scoped. Their definitions are pushed by the declaration, and popped at the end of their scope. (See Macro Stacks regarding pushing and popping.)

It is recommended that all indirect body arguments (see Multi-line Constructs: Blocks and Bodies), such as those of m5_fn be scoped. Immediate body arguments (see Multi-line Constructs: Blocks and Bodies), such as those of m5_if, are most often unscoped, but scope may be used to isolate the side effects of the block to explicit m5_out_eval calls. Scoped and unscoped blocks are illustrated in the following example:

Declarations from outer scopes are visible in inner scopes. Similarly, declarations from calling scopes are visible in callee scopes, though functions should generally be written without any assumptions about the calling scope. Exceptions should be clearly documented/commented.

By convention, scoped variables and macros use Pascal case, e.g. MyVar. (See [Macro Naming Conventions].)

"Text blocks" provide a syntax for multi-line quoted text that is indented with its surroundings. They are delimited similarly to code blocks, but use standard ([' / '] ) quotes. The openning quote must be followed by a newline and the closing quote must begin a new line that is indented consistently with the line beginning the block. Their indentation is defined by the first non-blank line in the block. All lines must contain at least this indentation (except the last). This fixed level of indentation and the beginning and ending newline are removed. For example:

This is equivalent to:

The text of the block is in source context, thus syntactic sugar is interpretted under the assumption that the text is to be evaluated. Text blocks that contain literal (quoted) text that is not evaluated should avoid entering argument list context with m5_, using quotes or $ (if within a macro body), and it should be understood that vanishing comments would be removed.

It can be convenient to form non-body arguments by evaluating code. Syntactic sugar is provided for this in the form of a * preceding the block open quote.

For example, here a scoped evaluate code block is used to form an error message by searching for negative arguments:

Proper use of quotes can get a bit tedious, especially when it is necessary to escape out of several levels of nested quotes. It can improve maintainability, code clarity, and performance to make judicious use of block labels. Note, however, that the need for block labels is rare and is mostly replaced by mechanisms provided by Functions.

Blocks can be labeled using syntax such as:

Labels can be used in two ways.

Both use cases are illustrated in the following example that attempts to declare a macro for parsing text. This macro declares a helper macro ParseError for reporting parse errors that can be used many times by my_parser.

This code contains, potentially, two mistakes in the error message. First, m5_What will be substituted at the time of the call to ParseError. As long as my_parser does not modify the value of What, this is fine, but it might be preferred to expand m5_What in the definition itself to avoid this potential Variable Masking issue in case What is reused.

Secondly, $1 will be substituted upon calling my_parser, not upon calling ParseError, and it will be substituted with a null string.

The corrected example would use:

This code corrects both issues:

Source context generally passes text through to the output. It is the default context and is also the context of text blocks.

Features supported in source context are supported in all contexts. For text that is intended to be literal, caution must be taken to avoid inadvertent use of these syntaxes. (See Ensure No Impact.)

The following are recognized in source context:

Text context is the default context entered by (block or non-block) [' quotes.

In addition to the features of source context, the following are recognized in text context:

Argument list context is entered from source and text contexts by, for example, m5_foo(. This context is exited by the corresponding ). In addition to text context features, the following are recognized in argument list context:

Code context is for Code Blocks, supporting syntactic sugar for formatting macro code more like programs.

Code context is entered by [/{ that end a line (after stripping vanishing comments) and is exited by the corresponding ]/} beginning a line at matching indentation (also after stripping vanishing comments).

In addition to argument list context features, the following are recognized in code context:

M5 checks that indentation is consistent for code and text blocks.

Parenthesis and quote matching is performed on the code after stripping comments. Quotes (including [ / ] and { / } quotes for code blocks) must be balanced.

Within each level of quotes, parentheses must be balanced. Parentheses in source and text context are excluded from this check, thus requiring parentheses for macros and parentheses that appear unquoted within macro arguments to be balanced.

Within a line, '] / [' quotes may be used (including nesting) to escape from and return to the same quoted context. This applies to contexts of all quote types, including code blocks, even though they are bound using different quote syntax. The context that is escaped from and returned to is the same context, thus parenthesis matching happens across the escaping. Thus, the parentheses on this code statement line are matching:

Here are some other examples:

See also, m5_open_quote, m5_close_quote, and m5_pragma_[enable/disable]_paren_checks in Pragmas.

In certain cases quote and parenthesis checking gets in the way. It is possible to disable checking and control debug behavior using pragmas. Pragmas processing happens after M5 comments are stripped. The following strings are recognized as pragmas:

Since the pragmas would pass through to the target file, pragmas are generally expressed using the following macro calls which elaborate to nothing:

For nested declarations, the use of numbered parameters ($1, $2, …​) and special parameters ($@, $*, $#, and $0) can be extremely awkward. Nested declarations are declarations within the bodies of other declarations. Since nested bodies are part of outer bodies, numbered and special parameters within them would actually substitute based on the outer bodies. This can be prevented by generating the body with macros that produce the numbered parameter references, but this requires an unnatural and bug prone use of quotes. Therefore the use of functions with named parameters is preferred for inner macro declarations. Use of m5_fn_args and m5_fn_arg is also simpler than using special parameters. If parameters are named, these are helpful primarily to access …​ arguments or to pass argument lists to other functions.

Additionally, and in summary:

Function calls must have arguments for all non-optional, non-inherited (^) parameters. Arguments are positional, so misaligning arguments is a common source of errors. There is checking, however, that required arguments are provided and that no extra arguments are given. m5_foo() is permitted for a function foo declared with no parameters, though it is passed one emtpy parameter. (m5_call(foo) might be preferred.)

In argument list context, function foo is declared below to display its parameters.

And it can be called (again, in argument list context):

And this expands to:

It is possible for a function to make assignments (and, actually do anything) in the calling scope. This can be done using m5_on_return or m5_return_status.

This is important for:

Each of these is discussed in its own section, next.

Functions can pass variables by reference and make assignments to the referenced variables upon returning from the function. For example:

A similar function could be defined to declare a referenced variable by using var instead of set.

The use of m5_on_return avoids the potential masking issue that would result from:

A function’s m5_status should be returned via the function’s aftermath, using m5_return_status, e.g.

Functions automatically restore m5_status after body evaluation to its value prior to body evaluation, so the evaluation of the body has no impact on m5_status. Aftermath is evaluated after this. It is fine to call m5_return_status multiple times. Only the last call will have a visible effect.

The example below illustrates a function if_neg that takes an argument that is a body to evaluate. The body is defined in a calling function, e.g. my_fn on lines 15-16. Such a body is expected to evaluate in the context of the calling function, my_fn. Its assignment of Neg, on line 15, should be an assignment of its own local Neg, declared on line 12. Its side effects from m5_return_status on line 15 should be side effects of my_fn.

If the body is evaluated inside the function body, its side effects would be side effects of if_neg, not my_fn. The body should instead be evaluated as aftermath, using m5_on_return, as on line 6.

Note that m5_return_status is called after evaluating m5_Body. Both m5_on_return and m5_return_status add to the Aftermath of the function, and m5_status must be set after evaluating the body (which could affect m5_status).

Example of a body argument.

Since m5_macro does not support Aftermath, it is not recommended to use m5_macro with a body argument.

Recursive calls tend to grow the stack significantly, and this can result in an error (see m5_recursion_limit) as well inefficiency. When recursion is the last act of the function ("tail recursion"), the recursion can be performed in aftermath to avoid growing the stack. For example:

Variable "masking" is an issue that can arise when a macro has side effects determined by its arguments. For example, an argument might specify the name of a variable to assign, or an argument might provide a body to evaluate that could declare or assign arbitrary variables. If the macro declares a local variable, and the side effect updates a variable by the same name, the local variable may inadvertently be the one that is updated by the side effect. This issue is addressed differently depending how the macro is defined. Note that using function Aftermath is the preferred method, but all options are listed here for completeness:

m5_var(Name, Value, …​)

m5_set(Name, Value)

m5_push_var(Name, Value)

m5_pop(Name)

m5_null_vars(…​)

m5_fn(…​)
m5_lazy_fn(…​)

m5_macro(Name, Body)
m5_null_macro(Name, Body)

m5_set_macro(Name, Body)

m5_push_macro(Name, Body)

m5_get(Name)

m5_must_exist(Name)
m5_var_must_exist(Name)

m5_status (Universal variable)

m5_sticky_status (Universal variable)

m5_sticky_status()

m5_reset_sticky_status()

m5_if(Cond, TrueBody, …​)
m5_unless(Cond, TrueBody, FalseBody)
m5_else_if(Cond, TrueBody, …​)

m5_if_eq(String1, String2, TrueBody, …​)
m5_if_neq(String1, String2, TrueBody, …​)

m5_if_null(Var, Body, ElseBody)
m5_if_var_def(Var, Body, ElseBody)
m5_if_var_ndef(Var, Body, ElseBody)
m5_if_defined_as(Var, Value, Body, ElseBody)

m5_else(Body)
m5_if_so(Body)

m5_else_if_def(Name, Body)

m5_case(Name, Value, TrueBody, …​)

m5_loop(InitList, DoBody, WhileCond, WhileBody)

m5_repeat(Cnt, Body)

m5_for(Var, List, Body)

m5_recurse(max_depth, macro, …​)

m5_nl()

m5_open_quote()
m5_close_quote()

m5_arg_comma (Universal variable)

m5_orig_open_quote()
m5_orig_close_quote()

m5_printable_open_quote()
m5_printable_close_quote()

m5_UNDEFINED()

m5_append_var(Name, String)
m5_prepend_var(Name, String)
m5_append_macro(Name, String)
m5_prepend_macro(Name, String)

m5_substr(String, From, Length)
m5_substr_eval(String, From, Length)

m5_join(Delimiter, …​)

m5_translit(String, InChars, OutChars)
m5_translit_eval(String, InChars, OutChars)

m5_uppercase(String)
m5_lowercase(String)

m5_replicate(Cnt, String)

m5_strip_trailing_whitespace_from(Var)

m5_format_eval(string, …​)

m5_length(String)

m5_index_of(String, Substring)

m5_num_lines(String)

m5_for_each_line(Text, Body)

m5_dequote(String)
m5_requote(String)

m5_output_with_restored_quotes(String)

m5_no_quotes(String)

Regular expressions in M5 use the same regular expression syntax as GNU Emacs. (See GNU Emacs Regular Expressions.) This syntax is similar to BRE, Basic Regular Expressions in POSIX and is regrettably rather limited. Extended Regular Expressions are not supported.

m5_regex(String, Regex, Replacement)
m5_regex_eval(String, Regex, Replacement)

m5_var_regex(String, Regex, VarList)

m5_if_regex(String, Regex, VarList, Body, …​)
m5_else_if_regex(String, Regex, VarList, Body, …​)

m5_for_each_regex(String, Regex, VarList, Body)

m5_defn(Name)

m5_call(Name, …​)

m5_quote(…​)

m5_nquote(…​)

m5_eval(Expr)

m5_comment(…​)
m5_nullify(…​)

See Macro Stacks.

m5_get_ago(Name, Ago)

m5_depth_of(Name)

m5_shift(…​)
m5_comma_shift(…​)

m5_nargs(…​)

m5_argn(ArgNum, …​)

m5_comma_args(…​)

m5_echo_args(…​)

m5_calc(Expr, Radix, Width)

m5_equate(Name, Expr)
m5_operate_on(Name, Expr)

m5_increment(Name, Amount)
m5_decrement(Name, Amount)

These have boolean (0 / 1) results. Note that some m5_calc expressions result in boolean values as well.

m5_is_null(Name)
m5_isnt_null(Name)

m5_eq(String1, String2, …​)
m5_neq(String1, String2, …​)

m5_fn_args()
m5_comma_fn_args()

m5_fn_arg(Num)

m5_fn_arg_cnt()

m5_out(String)
m5_out_eval(String)

m5_return_status(Value)

m5_on_return(…​, D, MacroName, …​)

These macros output text to the standard error output stream (STDERR) (with [' / '] quotes represented by single characters). (Note that STDOUT is the destination for the evaluated output.)

m5_errprint(text)
m5_errprint_nl(text)

m5_warning(message)
m5_error(message)
m5_fatal_error(message)
m5_DEBUG(message)

m5_warning_if(condition, message)
m5_error_if(condition, message)
m5_fatal_error_if(condition, message)
m5_DEBUG_if(condition, message)

m5_assert(message)
m5_fatal_assert(message)

m5_verify_min_args(Name, Min, Actual)
m5_verify_num_args(Name, Min, Actual)
m5_verify_min_max_args(Name, Min, Max, Actual)

m5_recursion_limit (Universal variable)

m5_abbreviate_args(max_args, max_arg_length, …​)