I am participating in a learning circle with the goal of gaining a better understanding of the Julia language. To better retain what we learn, I will be turning my notes into small blog posts. The posts should be simple, quick, but hopefully enjoyable reads.
The code snippets in this post are run on Julia 1.6.0-DEV.1440.
Just-in-Time Compilation
The first time a method is run, it will just-in-time (JIT) be compiled.
The compilation time can be measured with @time.
julia>A=randn(Float64,3,3);julia>@timeinv(A);0.244590seconds(559.50kallocations:31.983MiB,2.82%gctime,99.94%compilationtime)julia>@timeinv(A);0.000015seconds(4allocations:1.953KiB)The method inv(::Vector{Float64}) is now compiled and fast to call.
However, for example inv(::Vector{Float32}) is not yet compiled, and will consequently incur compilation time.
julia>A=randn(Float32,3,3);julia>@timeinv(A);0.188690seconds(449.85kallocations:25.852MiB,96.79%compilationtime)julia>@timeinv(A);0.000017seconds(4allocations:1.125KiB)The Julia JIT is simple: it compiles a method once the method is required. This, however, comes at the cost of start-up time and delays during runtime. Other approaches, like PyPy, first run the code on an interpreter, profile the code, and then compile bits of the code based on the profiling results; this is called profile-guided optimisation (POGO).
Method Invalidation
Once a method is compiled, it can happen that it needs to be recompiled. Namely, a method is compiled under certain assumptions, and these assumptions may not hold anymore as more code is loaded.
For example, suppose that a compiled method m uses the instance my_add(x::Float64, y::Float64) obtained from the implementation for my_add(x::Real, y::Real).
If a direct implementation of my_add(x::Float64, y::Float64) is then added, the compiled method m needs to be recompiled to make use of this direct implementation: m gets invalidated.
Here’s that example:
julia>my_add(x::Real,y::Real)=x+ymy_add(genericfunction with1method)julia>my_sum(x::Vector{T})whereT<:Real=reduce(my_add,x;init=one(T))my_sum(genericfunction with1method)julia>my_sum(randn(10))0.65443378603631We then add a direct implementation for my_add(x::Float64, y::Float64).
To detect the method invalidation, we use SnoopCompile.jl.
julia>trees=invalidation_trees(@snooprbeginmy_add(x::Float64,y::Float64)=x+yend)1-elementVector{SnoopCompile.MethodInvalidations}:insertingmy_add(x::Float64,y::Float64)inMainatREPL[12]:2invalidated:backedges:1:supersedingmy_add(x::Real,y::Real)inMainatREPL[8]:1withMethodInstanceformy_add(::Float64,::Float64)(10children)1mt_cachejulia>trees[1].backedges[end]MethodInstanceformy_add(::Float64,::Float64)atdepth0with10childrenjulia>show(trees[1].backedges[end];minchildren=0,maxdepth=100)MethodInstanceformy_add(::Float64,::Float64)(10children)MethodInstancefor(::Base.BottomRF{typeof(my_add)})(::Float64,::Float64)(9children)MethodInstancefor_foldl_impl(::Base.BottomRF{typeof(my_add)},::Float64,::Vector{Float64})(8children)MethodInstanceforfoldl_impl(::Base.BottomRF{typeof(my_add)},::Float64,::Vector{Float64})(7children)MethodInstanceformapfoldl_impl(::typeof(identity),::typeof(my_add),::Float64,::Vector{Float64})(6children)MethodInstancefor_mapreduce_dim(::typeof(identity),::typeof(my_add),::Float64,::Vector{Float64},::Colon)(5children)MethodInstanceforvar"#mapreduce#665"(::Colon,::Float64,::typeof(mapreduce),::typeof(identity),::typeof(my_add),::Vector{Float64})(4children)MethodInstancefor(::Base.var"#mapreduce##kw")(::NamedTuple{(:init,),Tuple{Float64}},::typeof(mapreduce),::typeof(identity),::typeof(my_add),::Vector{Float64})(3children)MethodInstanceforvar"#reduce#667"(::Base.Iterators.Pairs{Symbol,Float64,Tuple{Symbol},NamedTuple{(:init,),Tuple{Float64}}},::typeof(reduce),::typeof(my_add),::Vector{Float64})(2children)MethodInstancefor(::Base.var"#reduce##kw")(::NamedTuple{(:init,),Tuple{Float64}},::typeof(reduce),::typeof(my_add),::Vector{Float64})(1children)MethodInstanceformy_sum(::Vector{Float64})(0children)This shows the whole call stack.
You can interactively navigate the stack with ascend(trees[1].backedges[end]), which uses Cthulhu.jl.
Let’s perform some timings to see whether we can detect delays due to method invalidations. Start up a fresh Julia REPL.
julia>usingSnoopCompilejulia>x=randn(10);julia>my_add(x::Real,y::Real)=x+y;julia>my_sum(x::Vector{T})whereT<:Real=reduce(my_add,x;init=one(T));julia>@timemy_sum(x);0.023856seconds(79.31kallocations:4.761MiB,99.88%compilationtime)julia>my_add(x::Float64,y::Float64)=x+y;julia>@timemy_sum(x);0.016896seconds(53.17kallocations:2.952MiB,99.94%compilationtime)julia>usingSnoopCompilejulia>x=randn(10);julia>my_add(x::Real,y::Real)=x+y;julia>my_sum(x::Vector{T})whereT<:Real=reduce(my_add,x;init=one(T));julia>@timemy_sum(x);0.023979seconds(79.31kallocations:4.761MiB,99.89%compilationtime)julia>my_add(x::Float32,y::Float32)=x+y;julia>@timemy_sum(x);0.000004seconds(1allocation:16bytes)In the first case, where my_add(::Float64, ::Float64) gets invalidated, the second call of my_sum(x) again incurs compilation time.
This does not happen in the second case.
Lastly, we discuss one more common scenario in which method invalidations happen. Consider
julia>f(x::Int)=1;julia>g(x)=f(x);julia>g("1")ERROR:MethodError:nomethodmatchingf(::String)Closestcandidatesare:f(::Int64)atREPL[8]:1Stacktrace:[1]g(x::String)@Main./REPL[9]:1[2]top-levelscope@REPL[10]:1The compiled method instance g(::String) gives back a MethodError.
In particular, it assumes that there is no implementation for f(::String).
If we add that implementation, then g(::String) needs to be recompiled to make use of the then-available f(::String).
Invalidations of this kind link back to the method table.
They show up in the property mt_backedges of MethodInvalidations:
julia>invalidation_trees(@snooprbeginf(x::String)=1end)1-elementVector{SnoopCompile.MethodInvalidations}:insertingf(x::String)inMainatREPL[11]:1invalidated:mt_backedges:1:signatureTuple{typeof(f),String}triggeredMethodInstanceforg(::String)(0children)