Metaprogramming¶

Symbols¶

The : character has two syntactic purposes in Julia. The first form creates a Symbol, an interned string used as one building-block of expressions:

julia>:foo:foojulia>typeof(ans)Symbol

Symbols can also be created using symbol(), which takes any number of arguments and creates a new symbol by concatenating their string representations together:

julia>:foo==symbol("foo")truejulia>symbol("func",10):func10julia>symbol(:var,'_',"sym"):var_sym

In the context of an expression, symbols are used to indicate access to variables; when an expression is evaluated, a symbol is replaced with the value bound to that symbol in the appropriate scope.

Sometimes extra parentheses around the argument to : are needed to avoid ambiguity in parsing.:

julia>:(:):(:)julia>:(::):(::)

Quoting¶

The second syntactic purpose of the : character is to create expression objects without using the explicit Expr constructor. This is referred to as quoting. The : character, followed by paired parentheses around a single statement of Julia code, produces an Expr object based on the enclosed code. Here is example of the short form used to quote an arithmetic expression:

julia>ex=:(a+b*c+1):(a+b*c+1)julia>typeof(ex)Expr

(to view the structure of this expression, try ex.head and ex.args, or use dump() as above)

Note that equivalent expressions may be constructed using parse() or the direct Expr form:

julia>:(a+b*c+1)==parse("a + b*c + 1")==Expr(:call,:+,:a,Expr(:call,:*,:b,:c),1)true

Expressions provided by the parser generally only have symbols, other expressions, and literal values as their args, whereas expressions constructed by Julia code can have arbitrary run-time values without literal forms as args. In this specific example, + and a are symbols, *(b,c) is a subexpression, and 1 is a literal 64-bit signed integer.

There is a second syntactic form of quoting for multiple expressions: blocks of code enclosed in quote...end. Note that this form introduces QuoteNode elements to the expression tree, which must be considered when directly manipulating an expression tree generated from quote blocks. For other purposes, :(...) and quote..end blocks are treated identically.

julia>ex=quotex=1y=2x+yendquote# none, line 2:x=1# none, line 3:y=2# none, line 4:x+yendjulia>typeof(ex)Expr

Interpolation¶

Direct construction of Expr objects with value arguments is powerful, but Expr constructors can be tedious compared to “normal” Julia syntax. As an alternative, Julia allows “splicing” or interpolation of literals or expressions into quoted expressions. Interpolation is indicated by the $ prefix.

In this example, the literal value of a is interpolated:

julia>a=1;julia>ex=:($a+b):(1+b)

Interpolating into an unquoted expression is not supported and will cause a compile-time error:

julia>$a+bERROR:unsupportedormisplacedexpression$

In this example, the tuple (1,2,3) is interpolated as an expression into a conditional test:

julia>ex=:(ain$:((1,2,3))):($(Expr(:in,:a,:((1,2,3)))))

Interpolating symbols into a nested expression requires enclosing each symbol in an enclosing quote block:

julia>:(:ain$(:(:a+:b)))^^^^^^^^^^quotedinnerexpression

The use of $ for expression interpolation is intentionally reminiscent of string interpolation and command interpolation. Expression interpolation allows convenient, readable programmatic construction of complex Julia expressions.

eval() and effects¶

Given an expression object, one can cause Julia to evaluate (execute) it at global scope using eval():

julia>:(1+2):(1+2)julia>eval(ans)3julia>ex=:(a+b):(a+b)julia>eval(ex)ERROR:UndefVarError:bnotdefinedjulia>a=1;b=2;julia>eval(ex)3

Every module has its own eval() function that evaluates expressions in its global scope. Expressions passed to eval() are not limited to returning values — they can also have side-effects that alter the state of the enclosing module’s environment:

julia>ex=:(x=1):(x=1)julia>xERROR:UndefVarError:xnotdefinedjulia>eval(ex)1julia>x1

Here, the evaluation of an expression object causes a value to be assigned to the global variable x.

Since expressions are just Expr objects which can be constructed programmatically and then evaluated, it is possible to dynamically generate arbitrary code which can then be run using eval(). Here is a simple example:

julia>a=1;julia>ex=Expr(:call,:+,a,:b):(1+b)julia>a=0;b=2;julia>eval(ex)3

The value of a is used to construct the expression ex which applies the + function to the value 1 and the variable b. Note the important distinction between the way a and b are used:

• The value of the variablea at expression construction time is used as an immediate value in the expression. Thus, the value of a when the expression is evaluated no longer matters: the value in the expression is already 1, independent of whatever the value of a might be.
• On the other hand, the symbol:b is used in the expression construction, so the value of the variable b at that time is irrelevant — :b is just a symbol and the variable b need not even be defined. At expression evaluation time, however, the value of the symbol :b is resolved by looking up the value of the variable b.

Functions on Expressions¶

As hinted above, one extremely useful feature of Julia is the capability to generate and manipulate Julia code within Julia itself. We have already seen one example of a function returning Expr objects: the parse() function, which takes a string of Julia code and returns the corresponding Expr. A function can also take one or more Expr objects as arguments, and return another Expr. Here is a simple, motivating example:

julia>function math_expr(op,op1,op2)expr=Expr(:call,op,op1,op2)returnexprendjulia>ex=math_expr(:+,1,Expr(:call,:*,4,5)):(1+4*5)julia>eval(ex)21

As another example, here is a function that doubles any numeric argument, but leaves expressions alone:

julia>function make_expr2(op,opr1,opr2)opr1f,opr2f=map(x->isa(x,Number)?2*x:x,(opr1,opr2))retexpr=Expr(:call,op,opr1f,opr2f)returnretexprendmake_expr2(genericfunction with1method)julia>make_expr2(:+,1,2):(2+4)julia>ex=make_expr2(:+,1,Expr(:call,:*,5,8)):(2+5*8)julia>eval(ex)42

Basics¶

Here is an extraordinarily simple macro:

julia>macrosayhello()return:(println("Hello, world!"))end

Macros have a dedicated character in Julia’s syntax: the @ (at-sign), followed by the unique name declared in a macroNAME...end block. In this example, the compiler will replace all instances of @sayhello with:

:(println("Hello, world!"))

When @sayhello is given at the REPL, the expression executes immediately, thus we only see the evaluation result:

julia>@sayhello()"Hello, world!"

Now, consider a slightly more complex macro:

julia>macrosayhello(name)return:(println("Hello, ",$name))end

This macro takes one argument: name. When @sayhello is encountered, the quoted expression is expanded to interpolate the value of the argument into the final expression:

julia>@sayhello("human")Hello,human

We can view the quoted return expression using the function macroexpand() (important note: this is an extremely useful tool for debugging macros):

julia>ex=macroexpand(:(@sayhello("human"))):(println("Hello, ","human"))^^^^^^^interpolated:nowaliteralstringjulia>typeof(ex)Expr

Hold up: why macros?¶

We have already seen a function f(::Expr...)->Expr in a previous section. In fact, macroexpand() is also such a function. So, why do macros exist?

Macros are necessary because they execute when code is parsed, therefore, macros allow the programmer to generate and include fragments of customized code before the full program is run. To illustrate the difference, consider the following example:

julia>macrotwostep(arg)println("I execute at parse time. The argument is: ",arg)return:(println("I execute at runtime. The argument is: ",$arg))endjulia>ex=macroexpand(:(@twostep:(1,2,3)));Iexecuteatparsetime.Theargumentis::((1,2,3))

The first call to println() is executed when macroexpand() is called. The resulting expression contains only the second println:

julia>typeof(ex)Exprjulia>ex:(println("I execute at runtime. The argument is: ",$(Expr(:copyast,:(:((1,2,3)))))))julia>eval(ex)Iexecuteatruntime.Theargumentis:(1,2,3)

Macro invocation¶

Macros are invoked with the following general syntax:

@nameexpr1expr2...@name(expr1,expr2,...)

Note the distinguishing @ before the macro name and the lack of commas between the argument expressions in the first form, and the lack of whitespace after @name in the second form. The two styles should not be mixed. For example, the following syntax is different from the examples above; it passes the tuple (expr1,expr2,...) as one argument to the macro:

@name(expr1,expr2,...)

It is important to emphasize that macros receive their arguments as expressions, literals, or symbols. One way to explore macro arguments is to call the show() function within the macro body:

julia>macroshowarg(x)show(x)# ... remainder of macro, returning an expressionendjulia>@showarg(a)(:a,)julia>@showarg(1+1):(1+1)julia>@showarg(println("Yo!")):(println("Yo!"))

Building an advanced macro¶

Here is a simplified definition of Julia’s @assert macro:

macroassert(ex)return:($ex?nothing:throw(AssertionError($(string(ex)))))end

This macro can be used like this:

julia>@assert1==1.0julia>@assert1==0ERROR:AssertionError:1==0

In place of the written syntax, the macro call is expanded at parse time to its returned result. This is equivalent to writing:

1==1.0?nothing:throw(AssertionError("1==1.0"))1==0?nothing:throw(AssertionError("1==0"))

That is, in the first call, the expression :(1==1.0) is spliced into the test condition slot, while the value of string(:(1==1.0)) is spliced into the assertion message slot. The entire expression, thus constructed, is placed into the syntax tree where the @assert macro call occurs. Then at execution time, if the test expression evaluates to true, then nothing is returned, whereas if the test is false, an error is raised indicating the asserted expression that was false. Notice that it would not be possible to write this as a function, since only the value of the condition is available and it would be impossible to display the expression that computed it in the error message.

The actual definition of @assert in the standard library is more complicated. It allows the user to optionally specify their own error message, instead of just printing the failed expression. Just like in functions with a variable number of arguments, this is specified with an ellipses following the last argument:

macroassert(ex,msgs...)msg_body=isempty(msgs)?ex:msgs[1]msg=string(msg_body)return:($ex?nothing:throw(AssertionError($msg)))end

Now @assert has two modes of operation, depending upon the number of arguments it receives! If there’s only one argument, the tuple of expressions captured by msgs will be empty and it will behave the same as the simpler definition above. But now if the user specifies a second argument, it is printed in the message body instead of the failing expression. You can inspect the result of a macro expansion with the aptly named macroexpand() function:

julia>macroexpand(:(@asserta==b)):(ifa==bnothingelseBase.throw(Base.Main.Base.AssertionError("a == b"))end)julia>macroexpand(:(@asserta==b"a should equal b!")):(ifa==bnothingelseBase.throw(Base.Main.Base.AssertionError("a should equal b!"))end)

There is yet another case that the actual @assert macro handles: what if, in addition to printing “a should equal b,” we wanted to print their values? One might naively try to use string interpolation in the custom message, e.g., @asserta==b"a($a)shouldequalb($b)!", but this won’t work as expected with the above macro. Can you see why? Recall from string interpolation that an interpolated string is rewritten to a call to string(). Compare:

julia>typeof(:("a should equal b"))ASCIIStringjulia>typeof(:("a ($a) should equal b ($b)!"))Exprjulia>dump(:("a ($a) should equal b ($b)!"))Exprhead:Symbolstringargs:Array(Any,(5,))1:ASCIIString"a ("2:Symbola3:ASCIIString") should equal b ("4:Symbolb5:ASCIIString")!"typ:Any

So now instead of getting a plain string in msg_body, the macro is receiving a full expression that will need to be evaluated in order to display as expected. This can be spliced directly into the returned expression as an argument to the string() call; see error.jl for the complete implementation.

The @assert macro makes great use of splicing into quoted expressions to simplify the manipulation of expressions inside the macro body.

Hygiene¶

An issue that arises in more complex macros is that of hygiene. In short, macros must ensure that the variables they introduce in their returned expressions do not accidentally clash with existing variables in the surrounding code they expand into. Conversely, the expressions that are passed into a macro as arguments are often expected to evaluate in the context of the surrounding code, interacting with and modifying the existing variables. Another concern arises from the fact that a macro may be called in a different module from where it was defined. In this case we need to ensure that all global variables are resolved to the correct module. Julia already has a major advantage over languages with textual macro expansion (like C) in that it only needs to consider the returned expression. All the other variables (such as msg in @assert above) follow the normal scoping block behavior.

To demonstrate these issues, let us consider writing a @time macro that takes an expression as its argument, records the time, evaluates the expression, records the time again, prints the difference between the before and after times, and then has the value of the expression as its final value. The macro might look like this:

macrotime(ex)returnquotelocalt0=time()localval=$exlocalt1=time()println("elapsed time: ",t1-t0," seconds")valendend

Here, we want t0, t1, and val to be private temporary variables, and we want time to refer to the time() function in the standard library, not to any time variable the user might have (the same applies to println). Imagine the problems that could occur if the user expression ex also contained assignments to a variable called t0, or defined its own time variable. We might get errors, or mysteriously incorrect behavior.

Julia’s macro expander solves these problems in the following way. First, variables within a macro result are classified as either local or global. A variable is considered local if it is assigned to (and not declared global), declared local, or used as a function argument name. Otherwise, it is considered global. Local variables are then renamed to be unique (using the gensym() function, which generates new symbols), and global variables are resolved within the macro definition environment. Therefore both of the above concerns are handled; the macro’s locals will not conflict with any user variables, and time and println will refer to the standard library definitions.

One problem remains however. Consider the following use of this macro:

moduleMyModuleimportBase.@timetime()=...# compute something@timetime()end

Here the user expression ex is a call to time, but not the same time function that the macro uses. It clearly refers to MyModule.time. Therefore we must arrange for the code in ex to be resolved in the macro call environment. This is done by “escaping” the expression with esc():

macrotime(ex)...localval=$(esc(ex))...end

An expression wrapped in this manner is left alone by the macro expander and simply pasted into the output verbatim. Therefore it will be resolved in the macro call environment.

This escaping mechanism can be used to “violate” hygiene when necessary, in order to introduce or manipulate user variables. For example, the following macro sets x to zero in the call environment:

macrozerox()returnesc(:(x=0))endfunction foo()x=1@zeroxx# is zeroend

This kind of manipulation of variables should be used judiciously, but is occasionally quite handy.

An advanced example¶

Julia’s base library has a sub2ind() function to calculate a linear index into an n-dimensional array, based on a set of n multilinear indices - in other words, to calculate the index i that can be used to index into an array A using A[i], instead of A[x,y,z,...]. One possible implementation is the following:

function sub2ind_loop{N}(dims::NTuple{N},I::Integer...)ind=I[N]-1fori=N-1:-1:1ind=I[i]-1+dims[i]*indendreturnind+1end

The same thing can be done using recursion:

sub2ind_rec(dims::Tuple{})=1sub2ind_rec(dims::Tuple{},i1::Integer,I::Integer...)=i1==1?sub2ind_rec(dims,I...):throw(BoundsError())sub2ind_rec(dims::Tuple{Integer,Vararg{Integer}},i1::Integer)=i1sub2ind_rec(dims::Tuple{Integer,Vararg{Integer}},i1::Integer,I::Integer...)=i1+dims[1]*(sub2ind_rec(tail(dims),I...)-1)

Both these implementations, although different, do essentially the same thing: a runtime loop over the dimensions of the array, collecting the offset in each dimension into the final index.

However, all the information we need for the loop is embedded in the type information of the arguments. Thus, we can utilize generated functions to move the iteration to compile-time; in compiler parlance, we use generated functions to manually unroll the loop. The body becomes almost identical, but instead of calculating the linear index, we build up an expression that calculates the index:

@generatedfunction sub2ind_gen{N}(dims::NTuple{N},I::Integer...)ex=:(I[$N]-1)fori=N-1:-1:1ex=:(I[$i]-1+dims[$i]*$ex)endreturn:($ex+1)end

What code will this generate?

An easy way to find out, is to extract the body into another (regular) function:

julia>@generatedfunction sub2ind_gen{N}(dims::NTuple{N},I::Integer...)sub2ind_gen_impl(dims,I...)endsub2ind_gen(genericfunction with1method)julia>function sub2ind_gen_impl{N}(dims::Type{NTuple{N}},I...)length(I)==N||return:(error("partial indexing is unsupported"))ex=:(I[$N]-1)fori=N-1:-1:1ex=:(I[$i]-1+dims[$i]*$ex)endreturn:($ex+1)endsub2ind_gen_impl(genericfunction with1method)

We can now execute sub2ind_gen_impl and examine the expression it returns:

julia>sub2ind_gen_impl(Tuple{Int,Int},Int,Int):(((I[1]-1)+dims[1]*(I[2]-1))+1)

So, the method body that will be used here doesn’t include a loop at all - just indexing into the two tuples, multiplication and addition/subtraction. All the looping is performed compile-time, and we avoid looping during execution entirely. Thus, we only loop once per type, in this case once per N (except in edge cases where the function is generated more than once - see disclaimer above).