yaql/doc/source/language_reference.rst

Language reference

YAQL is a single expression language and as such does not have any block constructs, line formatting, end of statement marks or comments. The expression can be of any length. All whitespace characters (including newline) that are not enclosed in quote marks are stripped. Thus, the expressions may span multiple lines.

Expressions consist of:

• Literals
• Keywords
• Variable access
• Function calls
• Binary and unary operators
• List expressions
• Dictionary expressions
• Index expressions
• Delegate expressions

Terminology

• YAQL - the name of the language - acronym for Yet Another Query Language
• yaql - Python implementation of the YAQL language (this package)
• expression - a YAQL query that takes context as an input and produces result value
• context - an object that (directly or indirectly) holds all the data available to expression and all the function implementations accessible to expression
• host - the application that hosts the yaql interpreter. The host uses yaql to evaluate expressions, provides initial data, and decides which functions are going to be available to the expression. The host has ultimate power to customize yaql - provide additional functions, operators, decide not to use standard library or use only parts of it, override function and operator behavior
• variable - any data item that is available through the context
• function - a Python callable that is exposed to the YAQL expression and can be called either explicitly or implicitly
• delegate - a Python callable that is available as a context variable (in expression data rather than registered in context)
• operator - a form of implicit function on one (unary operator) or two (binary operator) operands
• alphanumeric - consists of latin letters and digits (A-Z, a-z, 0-9)

Literals

Literals refer to fixed values in expressions. YAQL has the following literals:

• Integer literals: 123
• Floating point literals: 1.23, 1.0
• Boolean and null literals represented by keywords (see below)
• String literals enclosed in either single (') or double (") quotes: "abc", 'def'. The backslash () character is used to escape characters that otherwise have a special meaning, such as newline, backslash itself, or the quote character
• Verbatim strings enclosed in back quote characters, for example `abc, are used to suppress escape sequences. This is equivalent to r'strings'` in Python and is especially useful for regular expressions

Keywords

Keyword is a sequence of characters that conforms to the following criteria:

• Consists of non-zero alphanumeric characters and an underscore (_)
• Doesn't start with a digit
• Doesn't start with two underscore characters (__)
• Is not enclosed in quote marks of any type

YAQL has only three predefined keywords: true, false, and null that have the value of similar JSON keywords.

There are also four keyword operators: and, or, not, in. However, this list is not fixed. The yaql host may decide to have additional keyword operators or not to have any of the four aforementioned keywords.

All other keywords have the value of their string representation. Thus, except for the predefined keywords and operators they can be considered as string literals and can be used anywhere where string is expected. However the opposite is not true. That is, keywords can be used as string literals but string literals cannot be used where a token is expected.

Examples:

• John + Snow - the same as "John" + "Snow"
• true + love - syntactically valid, but cannot be evaluated because there is no plus operator that accepts boolean and string (unless you define one)
• not true - evaluates to false, not is an operator
• "foo"() - invalid expression because the function name must be a token
• John Snow - invalid expression - two tokens with no operator between them

Variable access

Each YAQL expression is a function that takes inputs (arguments) and produces the result value (usually by doing some computations on those inputs). Expressions get the input through a context - an object that holds all the data and a list of functions, available for expression.

Besides the argument values, expressions may populate additional data items to the context. All these data are collectively known as a variables and available to all parts of an expression (unless overwritten with another value).

The syntax for accessing variable values is $variableName where variableName is the name of the variable. Variable names may consist of alphanumeric and underscore characters only. Unlike tokens, variable names may start with digit, any number of underscores and even be an empty string. By convention, the first (usually the single) function parameter is accessible through $ expression (i.e. empty string variable name) which is an alias for $1. The usual case is to pass the main expression data in a single structure (document) and access it through the $ variable.

If the variable with given name is not provided, it is assumed to be null. There is no built-in syntax to check if a variable exists to distinguish cases where it does not and when it is just set to null. However in the future such a function might be added to yaql standard library.

When the yaql parser encounters the $variable expression, it automatically translates it to the #get_context_data("$variable") function call. By default, the #get_context_data function returns a variable value from the current context. However the yaql host may decide to override it and provide another behavior. For example, the host may try to look up the value in an external data source (database) or throw an exception due to a missing variable.

Function calls

The power of YAQL comes from the fact that almost everything in YAQL is a function call (explicit or implicit) and any function may be overridden by the host. In YAQL there are two types of functions:

• explicit function - those that can be called from expressions
• implicit (system) functions - functions with predefined names that get called upon some operations. For example, 2 + 3 is translated to #operator_+(2, 3). In this case, #operator_+ is the name of the implicit function. However, because #operator_+(2, 3) is not a valid YAQL expression (because of #), implicit functions cannot be called explicitly but still can be redefined by the host.

The syntax for explicit function is:

call: funcName "(" [parameters] ")" funcName: token parameters: positionalParameters | : keywordParameters | : positionalParameters "," keywordParameters positionalParameters: parameter ("," parameter)* parameter: expression | empty-string keywordParameters: keywordParameter ("," keywordParameter) keywordParameter: parameterName "=>" expression parameterName: token

In simple words:

• The function name must be a token.
• Parameters may be positional, keyword or both. But keyword parameters may not come before positional.
• Positional parameters can be skipped if they have a default value, for example, foo(1,,3).
• Keyword arguments must have a token name that must match the parameter name in the function declaration. Therefore, you must know the function signature for the right name.

Examples:

• foo(2 + 3)
• bar(hello, world)
• baz(a,b, kwparam1 => c, kwparam2 => d)

Functions have ultimate control over how they can be called. In particular:

• Each parameter may (and usually does) have an associated type check. That is, the function may specify that the expected parameter type and if it can be null.
• Usually, any parameters can be passed either by positional or keyword syntax. However, function declaration may force one particular way and make it positional-only or keyword-only.
• A function may have a variable number of positional (aka *args) and/or keyword (aka **kwarg) arguments.
• In most languages, function arguments are evaluated prior to function invocation. This is not always true in YAQL. In YAQL, a function may declare a lazy argument. In this case, it is not evaluated and the function implementation receives a passed value as a callable or even as an AST, depending on how the parameter was declared. Thus in YAQL there is no special syntax for lambdas. foo($ + 1) may mean either "call foo with value of $ + 1" or "call foo with expression $ + 1 as a parameter". In the latter case it corresponds to foo(lambda *args, **kwargs: args[0] + 1) in Python. Actual argument interpretation depends on the parameter declaration.
• Function may decide to disable keyword argument syntax altogether. For such functions, the name => expr expression will be interpreted as a positional parameter yaql.language.utils.MappingRule(name, expr) and the left side of => can be any expression and not just a keyword. This allows for functions like switch($ > 0 => 1, $ < 0 => -1, $ = 0 => 0).

Additionally, there are three subtypes of explicit functions. Suppose that there is a declared function foo(string, int). By default, the syntax to call it will be foo(something, 123). But it can be declared as a method. In this case, the syntax is going to be something.foo(123). Because of the type checking, something.foo(123) will work since something is a string, but not the 123.foo(456). Thus foo becomes a method of a string type.

A function may also be declared as being an extension method. If foo were to be declared as an extension method it could be called both as a function (foo(string, int)) and as a method (something.foo(123)).

YAQL makes use of a full function signature to determine which function implementation needs to be executed. This allows several overloads of the same function as long as they differ by parameter count or parameter type, or anything else that allows unambiguous identification of the right overload from the function call expression. For example, something.foo(123) may be resolved to a completely different implementation of foo from that in foo(something, 123) if there are two functions with the name foo present in the context, but one of them was declared as a function while the other as a method. If several overloads are equally suitable for the call expression, an AmbiguousFunctionException or AmbiguousMethodException exception gets raised.

Operators

YAQL has both binary and unary operators, like most other languages do. Parentheses and => sequence are not considered as operators and handled internally by the yaql parser. However, it is possible to configure yaql to use sequence other than => for that purpose.

The list of available operators is not fixed and can be modified by the host. The following operators are available by default:

Binary operators:

Group	Operators
math operators	+, -, *, /, mod
comparision operators	>, <, >=, <=, =, !=
logical operators	and, or
method/member access	., ?.
regex operators	=~, !~
membership operator	in
context passing operator	->

Unary operators:

Group	Operators
math operators	+, -
logical operators	not

YAQL supports for both prefix and suffix unary operators. However, only the prefix operators are provided by default.

In YAQL there are no built-in operators. The parser is given a list of all possible operator names (symbols), their associativity, precedence, and type, but it knows nothing about what operators are applicable for what operands. Each time a parser recognizes the X OP Y construct and OP is a known binary operator name, it translates the expression to #operator_OP(X, Y). Thus. 2 + 3 becomes #operator_+(2, 3) where #operator_+ is an implicit function with several implementations including the one for number addition and defined in standard library. The host may override it and even completely disable it. For unary operators, OP X (or X OP for suffix unary operators) becomes #unary_operator_OP(X).

Upon yaql parser initialization, an operator might be given an alias name. In such cases, X OP Y is translated to *ALIAS(X, Y) and OP X to *ALIAS(X). This decouples the operator implementation from the operator symbol. For example, the = operator has the equal alias. The host may configure yaql to have the == operator instead of = keeping the same alias so that operator implementation and all its consumers work equally well for the new operator symbol. In default configuration only = and != operators have alias names.

For information on default operators, see the YAQL standard library reference.

List expressions

List expressions have the following form:

listExpression: "[" [expressions] "]" expressions: expression ("," expression)*

When a yaql parser encounters an expression of the form [A, B, C], it translates it into #list(A, B, C) (for arbitrary number of arguments).

Default #list function implementation in standard library produces a list (tuple) comprised of given elements. However, the host might decide to give it a different implementation.

Map expressions

Map expressions have the following form:

mapExpression: "{" [mappings] "}" mappings: mapping ("," mapping)* mapping: expression "=>" expression

When a yaql parser encounters an expression of the form {A => X, B => Y}, it translates it into #map(A => X, B => Y).

The default #map implementation disables the keyword arguments syntax and thus receives a variable length list of mappings, which allows dictionary keys to be expressions rather than a keyword. It returns a (frozen) dictionary that itself can be used as a key in another map expression. For example, {{a => b} => {[2 + 2, 2 * 2] => 4}} is a valid YAQL expression though yaql REPL utility will fail to display its output due to the fact that it is not JSON-compatible.

Index expressions

Index expressions have the following form:

indexExpression: expression listExpression

Examples:

• [1, 2, 3][0]
• $arr[$index + 1]
• {foo => 1, bar => 2}[foo]

When a yaql parser encounters such an expression, it translates it into #indexer(expression, index).

The standard library provides a number of #indexer implementations for different types.

The right side of the index expression is a list expression. Therefore, an expression like $foo[1, x, null] is also a valid YAQL expression and will be translated to #indexer($foo, 1, x, null). However, any attempt to evaluate such expression will result in NoMatchingFunctionException exception because there is no #indexer implementation that accepts such arguments (unless the host defines one).

Delegate expressions

Delegate expressions is an optional language feature that is disabled by default. It makes possible to pass delegates (callables) as part of the context data and invoke them from the expression. It has the same syntax as explicit function calls with the only difference being that instead of function name (keyword) there is a non-keyword expression that must produce the delegate.

Examples:

• $foo(1, arg => 2) - call delegate returned by $foo with parameters (1, arg => 2)
• [$foo, $bar][0](x) - the same as $foo(x)
• foo()() - can be written as (foo())() - foo() must return a delegate

Delegate expressions are translated into #call(callable, arguments). Thus $foo(1, 2) becomes #call($foo, 1, 2).

The default implementation of #call invokes the result of the evaluation of its first arguments with the given arguments.