Makie

Appearance

Conversion, Transformation and Projection Pipeline ​

This section describes all the processing stages that are applied to data given to a plot on its way to being displayed.

Overview ​

The pipeline can be broadly be summarized in 3 parts each with a few steps:

  1. Conversions which mainly normalize types
    1. `expand_dimensions()` adds defaulted/generated data (e.g. x, y in `image()`)
    2. `dim_convert` processes special types like Units
    3. `convert_arguments()` normalizes numeric types & data formats
  2. Transformations which transform data on a per-plot basis
    1. `transform_func` is a function applied to data
    2. `model` matrix applies linear transformations
  3. Projections which project data from one coordinate system to another
    1. `view` matrix moves data from "world" space to a camera "view/eye" space
    2. `projection` matrix moves from the camera space to "clip" space
    3. `viewport` moves "clip" space to "pixel/screen" space

As a user you have direct control over the model matrix (1.2) with the scale!(), translate!() and rotate!() functions. You have indirect control over projections (3) with the space attribute. It sets what coordinate system is used as the initial space and adjusts the projections as a result. You also have indirect control over transform_func, which can be set by passing a Transformation() directly to a plot. However it usually inherited and controlled by the Axis.

As a developer, i.e. someone who wants to extend Makie, you can interact with most these steps. Most likely you will extend with convert_arguments() to allow special types to be plotted. But you can also implement more dim_converts, add methods for expand_dimensions(), implement more transform functions or add a camera which produces its ownviewandprojectionmatrix. Onlymodelandviewporthandling as well as the interpretation ofspace are set.

Argument Conversions ​

When calling a plot function, e.g. scatter!(axis_or_scene, args...) a new plot object is constructed. It keeps track of the original input arguments as an Observables in plot.args. Those input arguments are then converted by the conversion pipeline and stored in plot.converted. The pipeline consists of 3 steps as mentioned above:

Data Generation ​

The first step is to generate "missing" data. For example, you can create an image plot with just image(rand(10, 10)). The most general form however also includes a ClosedInterval for the x and y dimensions, declaring the size of the image. This data is generated by expand_dimensions(::Trait, args...) where the Trait = conversion_trait(::PlotType, args...).

Special Type Processing ​

The second step handles special types like Unitful types, Dates types or categorical values which need to be synchronized within the scene. For example, if one plot uses "hours" as unit for its x values other plots need to also use time units for x. If the scale of the unit differs between plots, i.e. one uses hours, the other minutes, then a common unit must be found and the values need to be scaled appropriately. This is what dim_converts handles. You can find more documentation on them in the Dimension conversions docs.

Convert Arguments ​

The last step and main work-horse in the conversion pipeline is the convert_arguments() function. It's purpose is to convert different data types and layouts into one or a select few formats. For example, any data passed to scatter() is converted to a Vector{Point{D, T}} where D = 2 or 3 and T = Float32 or Float64. These conversions can happen based on the plot type or its conversion trait. For scatter() the conversion trait PointBased is used.

convert_arguments() can also accept keyword arguments sourced from plot attributes. For this the attribute needs to be marked with used_attribute(::PlotType) = (names...). Any name in that list will be removed from the attributes of the final plot and be passed to convert_arguments() instead.

If you want to plot your own custom types you may want to extend convert_arguments(). Let's say you have some custom type MySimulation and some function positions(::MySimulation) which returns positions you want to plot when calling scatter(::MySimulation). In this case you can define

julia
function Makie.convert_arguments(PT::PointBased, sim::MySimulation)
    return Makie.convert_arguments(PT, positions(sim))
end

to make that possible. You can use the plot type (i.e. Scatter) as the first argument as well.

Transformations ​

After conversions have normalized the type and layout of plot data, transformations can now adjust it. They are handled by the Transformation object, which exists both on the plot and scene level in the transformation field. It contains a transform_func which is a function-like object applied to the data, and a model matrix which handles linear transformations of the data.

The plot Transformation object may inherit from it's parent scene or plot if it acts in the same space. For transform_func this means using the same function as the parent. For model it means merging the parents model matrix with the local one as plot.model = parent.model * local_model.

Transformation Function ​

The transformation function or transform_func is part of the Transformation object. It handles non-linear transformations like log transform of a logarithmic axis.

Each transform_func implements at least

julia
Makie.apply_transform(transform_func, arg::VecTypes{N, T}) where {N, T}

where the transform function can be represented by any type, not just a Function. That way it can carry auxiliary information that may be important to the transformation. Additionally methods with other arg types such as numbers of Vectors thereof may also be implemented to more efficiently apply the transform_func.

Typically a transform_func also implements

julia
Makie.inverse_transform(transform_func)
Makie.apply_transform(transform_func, arg::Rect3)

The inverse allows transforming data back, which is used for example in Axis limits. If a reasonable inverse exists (even if it is incomplete or ambiguous) it should be given by inverse_transform. The other apply_transform method is used in boundingbox() and defaults to transforming the corners of the bounding box. It should be implemented if the default returns wrong results, e.g. with a Polar transform.

Model Transformations ​

The model matrix takes care of linear transformations of plot data. This includes scaling with scale!(), translations with translate!() and rotations with rotate!(). See the Transformation reference docs for more details.

Projections ​

Projections are matrix transformations that move data from one coordinate system or "space" to another. The matrices are part of the camera(scene) and managed either by the cameracontrols(scene) or by the parent Block. The plot cannot change these matrices but can control which will be used via the space attribute.

Camera Controller ​

The camera controller, be that the object in the Scene or the parent Block, generates the view and projection matrices. These matrices are stored in camera(scene) which also combines them into projectionview = projection * view. The view matrix moves plot data from a "world" coordinate system to centered and oriented based on the viewer. This coordinate system is typically called camera, eye or view space. From there the projection matrix may apply perspective projection and scales data to move to "clip" space. This space is a normalized space where everything outside a -1 .. 1 box is clipped. The final projection to pixel or screen space is handled implicitly by the Graphics API or explicitly in CairoMakie based on viewport(scene). The camera(scene) also holds onto a pixel_space matrix, which transforms pixel space to clip space, the scenes resolution as well as some auxiliary information about the cameras orientation. Note that the camera controller is often referred to as the "camera" because the camera(scene) is just inactive storage.

Space Attribute ​

The space attribute controls which projection matrices the plot uses. The options refer to the input space, which generally transforms to clip space. The options include:

  • space = :data: Apply the cameras view and projection matrices. (This is usually called world space in Graphics APIs.)

  • space = :pixel: Apply the camera pixel_space matrix.

  • space = :clip: Apply an identity matrix.

  • space = :relative: Apply a constant translation-scale matrix.

Note that all of these act after transformations, i.e. after the model matrix is applied.

Marker Space Attribute ​

A few plots include a markerspace attribute. For these, the projections above are split into two steps, going from space to markerspace to clip space. The same options as above apply. If needed, some of these matrices may also be inverted (e.g. going from :pixel -> :data -> :clip space).

Float32Convert ​

The Float32Convert is an optional step that doesn't fit as cleanly into the pipeline. It's job is to make sure that the data, projection matrices and model matrix are save to convert to Float32 types and thus can be passed to the graphics API without precision issues.

Currently only Axis actually defines this transformation. When calling plot!(axis, ...) the data_limits() of the plot are recorded by the Axis and combined with existing limits. The limits eventually trigger a camera update, where the transform_func gets applied. The result gets passed to the Float32Convert, which updates its linear transformation to make the transformed limits Float32-safe. The projection matrices are then derived from the safe limits. At this point the linear transformation of the Float32Convert exists just before view. If possible, it is permuted with model so that the model matrix can processed by the graphics API, i.e. on the GPU.