Chapter 9: Native Rendering

In this chapter we will learn:

  • How to use RenderArea
  • How to draw any shape
  • How to efficiently show an image as a texture
  • How to change the currently used blend mode
  • How to apply a 3D transform to a shape GPU-side
  • How to compile a GLSL Shader and set its uniforms
Compat

In v0.2.0 and earlier, features from this chapter were not available on MacOS. Since v0.3.0, 100% of Mousetrap is portable, meaning MacOS is now fully supported.


Native Rendering on Linux Wayland

Linux users using the Wayland windowing system may encounter the following error message when starting Mousetrap:

In gdk_window_create_gl_context: Faled to create EGL display

This means we would be unable to use the RenderArea widget, which this chapter centers around.

To address this, we need to locate the directory egl_vendor.d, which has to be non-empty and contain a json file

locate egl_vendor.d
/etc/glvnd/egl_vendor.d
/usr/share/glvnd/egl_vendor.d
/usr/share/glvnd/egl_vendor.d/50_mesa.json

Here we choose /usr/share/glvnd/egl_vendor.d instead of /etc/glvnd/egl_vendor.d, because only the former contains a json file, 50_mesa.json in this case.

Armed with this path, we then execute, in a terminal, before Julia is started:

export __EGL_VENDOR_LIBRARY_DIRS=/usr/share/glvnd/egl_vendor.d

After which any OpenGL-related features from this chapter will become available.

To make this change permanent, we can paste the above line into the ~/.bashrc text file, which will be loaded automatically anytime a terminal starts.

Manually Disabling the OpenGL Component

We can disable all features from this chapter by setting the environment variable MOUSETRAP_DISABLE_OPENGL_COMPONENT to TRUE. This may be necessary for machines that do not have a OpenGL 3.3-compatible graphics card driver. See here for more information.


In the chapter on widgets, we learned that we can create new widgets by combining already predefined widgets as a compound widget. We can create a new widget that has a Scale, but we cannot render our own scale with, for example, a square knob. In this chapter, this will change.

By using the native rendering facilities Mousetrap provides, we are free to create any shape we want, assembling new widgets pixel-by-pixel, line-by-line, then adding interactivity using the event controller system.

RenderArea

The central widget of this chapter is RenderArea, which is a canvas used to display native graphics. At first, it may seem very simple:

render_area = RenderAre()

This will render as a transparent area, because RenderArea has no graphic properties of its own. Instead, we need to create separate shapes, then bind them for rendering, after which RenderArea will display the shapes for us. Still, RenderArea will follow its properties just like any other widget, for example, it will have an allocated size that follows size-hinting and the expansion property.

Shapes

In general, shapes are defined by a number of vertices. A vertex has a position in 2D space, a color, and a texture coordinate. In this chapter, we will learn what each of these properties does and how they coalesce to form a shape.

Vertex Coordinate System

A shape's vertices define where inside the RenderArea it will be drawn. The coordinate system shapes use is different from the one we use for widgets. OpenGL, on which the native rendering component of Mousetrap is based, uses the right-hand coordinate system, which is familiar from traditional math:

(source: learnopengl.com)

We assume the z-coordinate for any vertex is set to 0, reducing the coordinate system to a 2D plane.

We will refer to this coordinate system as GL coordinates, while the widget coordinate system used for ClickEventController and the like as widget space coordinates.

To further illustrate the difference between GL and widget space coordinates, consider this table, where w is the widgets' width, h is the widgets' height, in pixels:

Conceptual PositionGL CoordinatesWidget Space Coordinates
top left(-1, +1)(0, 0)
top( 0, +1)(w / 2, 0)
top right(+1, +1)(w, 0)
left(-1, 0)(0, y / 2)
center( 0, 0)(w / 2, y / 2)
right(+1, 0)(w, y / 2)
bottom left(-1, -1)(0, y)
bottom( 0, -1)(w / 2, y)
bottom right(+1, -1)(w, y)

We see that the OpenGL coordinate system is normalized, meaning the values of each coordinate are inside [-1, 1], while the widget-space coordinate system is absolute, meaning the values of each coordinate take the allocated size of the widget into account, being inside [0, w] and [0, h] for the x- and y-coordinates, respectively, in pixels.

The gl coordinate system is anchored at the center of the render areas allocated area, while widget space is anchored at the top left.

At any point, we can convert between the coordinate systems using from_gl_coordinates and to_gl_coordinates, which take the RenderArea as their first argument. These functions convert gl-to-widget-space and widget-space-to-gl coordinates, respectively. Of course, the widget space coordinates depend on the current size of the RenderArea. When it is resized, the old coordinates may be out of date, which is why using the normalized GL system is preferred in an application where the canvas can change size frequently.

Rendering Shapes

We'll now create our first shape, which is a point. A point is always exactly one pixel in size.

shape = Shape()
as_point!(shape, Vector2f(0, 0))

Where we use Vector2f(0, 0) as the points position, meaning it will appear at the origin of the RenderArea, its center.

The above is directly equivalent to the following:

shape = Point(Vector2f(0, 0))

We see that

typeof(shape)
Mousetrap.Shape

The variable shape is still of type Shape. Point is simply a convenience function for initializing a shape, then calling as_point! on that instance.

For this shape to show up on screen, we need to bind it for rendering. To do this, we create a RenderTask which wraps the shape, then use add_render_task! to add it to the scheduled tasks of our RenderArea. From this point onwards, anytime the RenderArea goes through a render cycle, it will also draw all its tasks, including our point:

shape = Point(Vector2f(0, 0))
add_render_task!(render_area, RenderTask(shape))

How to generate this Image
using Mousetrap
main() do app::Application

    set_current_theme!(app, THEME_DEFAULT_DARK)

    window = Window(app)
    set_title!(window, "Mousetrap.jl")
    render_area = RenderArea()

    shape = Point(Vector2f(0, 0))
    add_render_task!(render_area, RenderTask(shape))

    frame = AspectFrame(1.0, render_area)
    set_size_request!(frame, Vector2f(150, 150))
    set_margin!(frame, 10)
    set_child!(window, frame)
    present!(window)
end

If we want to remove a task from our render area, we need to call clear_render_tasks!, then add all other render tasks again.

Shape Types

Mousetrap offers a wide variety of pre-defined shape types. Thanks to this, we don't have to manually adjust each vertex position.

Point

As we've seen, Point is always exactly one pixel in size. Its constructor takes a single Vector2f:

point = Point(Vector2f(0, 0))

Points

Points is a number of points. Instead of taking a single Vector2f, its constructor takes Vector{Vector2f}:

points = Points([
    Vector2f(-0.5, 0.5), 
    Vector2f(0.5, 0.5), 
    Vector2f(0.0, -0.5)
])

Rendering a number of points using Points is much more performant, a four-vertex Points is much faster than rendering four Point with one vertex each.

Line

A Line is defined by two points, between which a 1-pixel thick line will be drawn:

line = Line(
    Vector2f(-0.5, +0.5), 
    Vector2f(+0.5, -0.5)
)

Lines

Lines will draw a number of unconnected lines. It takes a vector of point pairs. For each of these, a 1-pixel thick line will be drawn between them.

lines = Lines([
    Vector2f(-0.5, 0.5) => Vector2f(0.5, -0.5),
    Vector2f(-0.5, -0.5) => Vector2f(0.5, 0.5)
])

LineStrip

LineStrip, not to be confused with Lines, is a connected series of lines. Thus, it takes a vector of points, as opposed to a vector of point pairs.

A line will be drawn between each successive pair of coordinates, meaning the last vertex of the previous line will be used as the first vertex of the current line. If the supplied vector of points is {a1, a2, a3, ..., a(n)} then LineStrip will render as a series of lines with coordinate pairs {a1, a2}, {a2, a3}, ..., {a(n-1), a(n)}

line_strip = LineStrip([
    Vector2f(-0.5, +0.5),
    Vector2f(+0.5, +0.5),
    Vector2f(+0.5, -0.5),
    Vector2f(-0.5, -0.5)
])

Wireframe

Wireframe is similar to a LineStrip, except that it also connects the last and first vertex. For a supplied vector of points {a1, a2, a3, ..., an}, the series of lines will be {a1, a2}, {a2, a3}, ..., {a(n-1), a(n)}, {a(n), a1}, the last vertex-coordinate pair is what distinguishes it from a LineStrip. As such, Wireframe is sometimes also called a line loop.

wireframe = Wireframe([
    Vector2f(-0.5, +0.5),
    Vector2f(+0.5, +0.5),
    Vector2f(+0.5, -0.5),
    Vector2f(-0.5, -0.5)
])

Note how this shape takes the exact same coordinates as LineStrip, but draws one more line, connecting the last to the first vertex.

Triangle

A Triangle is constructed as one would expect, using three Vector2f, one for each of its vertices:

triangle = Triangle(
    Vector2f(-0.5, 0.5),
    Vector2f(+0.5, 0.5),
    Vector2f(0.0, -0.5)
)

Rectangle

A Rectangle has four vertices. It is defined by its top-left point and its width and height. As such, it is always axis-aligned.

rectangle = Rectangle(
    Vector2f(-0.5, 0.5), # top left
    Vector2f(1, 1)       # width, height
)

Circle

A Circle is constructed from a center point and radius. We also need to specify the number of outer vertices used for the circle. This number will determine how "smooth" the outline is. For example, a circle with 3 outer vertices is an equilateral triangle; a circle with 4 outer vertices is a square; a circle with 5 is a pentagon, etc.

As the number of outer vertices increases, the shape approaches a mathematical circle, but will also require more processing power.

circle = Circle(
    Vector2f(0, 0), # center
    0.5,            # radius
    32              # n outer vertices
)

Ellipse

An Ellipse is a more generalized form of a Circle. It has two radii, the x- and y-radius:

ellipse = Ellipse(
    Vector2f(0, 0), # center
    0.6,            # x-radius
    0.4,            # y-radius
    32              # n outer vertices
)

Polygon

The most general form of convex shapes, Polygon is constructed using a vector of vertices, which will be sorted clockwise, then their outer hull will be calculated, which results in the final convex polygon:

polygon = Polygon([
    Vector2f(0.0, 0.75),
    Vector2f(0.75, 0.25),
    Vector2f(0.5, -0.75),
    Vector2f(-0.5, -0.5),
    Vector2f(-0.75, 0.0)
])

We note that a 4-vertex polygon is a rectangle. Therefore, if we want to render a non-axis-aligned rectangle, we should instead use Polygon with four vertices.

Rectangular Frame

A RectangularFrame takes a top-left vertex, a width, a height, and the x- and y-width, the latter of which is the thickness of the frame along the x- and y-axes:

rectangular_frame = RectangularFrame(
    Vector2f(-0.5, 0.5),  # top-left
    Vector2f(1, 1),       # width, height
    0.15,                 # x-thickness
    0.15,                 # y-thickness
)

Note how the top left and size govern the position and size of the outer perimeter of the rectangular frame.

Circular Ring

For the round equivalent of a rectangular frame, we have CircularRing, which takes a center, the radius of the outer perimeter, as well as the rings thickness. Like Circle and Ellipse, we have to specify the number of outer vertices, which decides the smoothness of the ring:

circular_ring = CircularRing(
    Vector2f(0, 0),  # center
    0.5,             # radius of outer circle
    0.15,            # thickness
    32               # n outer vertices
)

As before, the center and radius determine the position and size of the outer perimeter.

Elliptical Ring

A generalization of CircularRing, EllipticalRing has an ellipse as its outer shape. Its thickness along the horizontal and vertical dimension are supplied separately, making it more flexible than CircularRing.

elliptcal_ring = EllipticalRing(
    Vector2f(0, 0),  # center
    0.6,             # x-radius
    0.4,             # y-radius
    0.15,            # x-thickness
    0.15,            # y-thickness
    32               # n outer vertices
)

Outline

Lastly, we have a special shape. Outline does not take any vertex positions for its constructor. Instead, we construct an Outline shape from another shape. It will then generate a wireframe for the outer perimeters of the original shape.

As the name suggests, this is useful for generating outlines of another shape. By rendering the Outline on top of the original shape, we can achieve make it so it better stands out from the background.

outline = Outline(triangle)

outline = Outline(rectangle)

outline = Outline(circle)

outline = Outline(ellipse)

outline = Outline(polygon)

outline = Outline(rectangular_frame)

outline = Outline(circular_ring)

outline = Outline(elliptical_ring)

While possible, it doesn't make much sense to create an Outline from a shape that does not have a volume, such as Point or Wireframe.

Rendering white outlines on top of a white shape would make little sense, of course. To achieve the desired effect, we need to make the outline another color, which brings us to the additional properties of Shape.

Shape Properties

Vertex Properties

Shapes are made up of vertices, whose properties we can manually edit. To set the property of a single vertex of a shape, we use set_vertex_color!, set_vertex_position!, and set_vertex_texture_coordinate!. Each of these take an index, which is the index of the vertex in clockwise order, 1-based. To know how many vertices a shape actually has, we use get_n_vertices.

We will rarely need to modify individual vertices, as working on the Shape as a whole is much more convenient.

Centroid

The centroid of a shape is the intuitive "center of mass". In mathematical terms, it is the component-wise mean of all vertex coordinates. In practice, for many symmetrical shapes such as rectangles, triangles, circles, and ellipses, the centroid will be the "center" of the shape, as it is defined in common language.

We can access the centroid using get_centroid. To move a shape a certain distance, we move its centroid by that distance by calling set_centroid!, which will automatically move all other vertices of the shape such that its new centroid is as specified.

Rotation

We can rotate all of a Shape's vertices around a point in GL coordinates by calling rotate!, which takes an Angle as its first argument:

# rotate shape around its center
rotate!(shape, degrees(90), get_centroid(shape))

Color

To change the color of a shape as a whole, we use set_color!. This simply calls set_vertex_color! on all of a shape's vertices. By default, a shape's color will be RBGA(1, 1, 1, 1), white.

Visibility

We can prevent a shape from being rendered by setting set_is_visible! to false. This is different from making all vertices of a shape have an opacity of 0. is_visible directly hooks into the shape's render function and prevents it from being called, as opposed to it completing rendering and not being visible on screen. This means making a shape invisible completely removes any performance penalty that would have been incurred during the render step, which is also called culling.

Bounding Box

We can access the axis-aligned bounding box of a shape with get_bounding_box, which returns an AxisAlignedRectangle. This is the smallest axis-aligned rectangle that still contains all of a shape's vertices.

Using this, we can query the top-left coordinate and size of the bounding box.


Lastly, each shape has an optional texture, which is what the texture coordinate properties of each vertex are used for. If a shape does not have a texture, it will be rendered as a solid color. If it does, the color of each pixel in the texture will be multiplied with the shape's color.

Textures

In the chapter on widgets, we learned that we can use the ImageDisplay widget to display static images. This works, but has a number of disadvantages:

  • Image data is costly to update
  • Downloading image data is impossible
  • Scaling the image will always trigger linear interpolation
  • The image is always shown in full, as a rectangle

If we need the additional flexibility, we should instead use a Shape along with a Texture, which represents an image living on the graphics card.

We create a texture from an Image like so:

image = Image()
load_from_file!(image, "path/to/image.png")

texture = Texture()
create_from_image!(texture, image)

Once create_from_image! is called, the image data is uploaded to the graphics cards' RAM, so we can safely discard the Image instance.

To display the texture on screen, we need to bind it to a shape, then render that shape:

texture_shape::Shape = Rectangle(Vector2f(-1, 1), Vector2f(2, 2))
set_texture!(texture_shape, texture)

add_render_tasK!(render_area, RenderTask(texture_shape))

How and where the texture is displayed depends on the shape's vertices texture coordinate. These coordinates are in ([0, 1], [0, 1]), meaning the x- and y- component are in [0, 1] each. We will call this coordinate system texture space.

Conceptual PositionTexture Space Coordinates
top left(0, 0)
top(0.5, 0)
top right(1, 0)
left(0, 0.5)
center(0.5, 0.5)
right(1, 0.5)
bottom left(0, 1)
bottom(0.5, 1)
bottom right(1, 1)

We see that, due to the normalized nature of this coordinate system, a texture coordinate is unable to reference a specific pixel. Instead, we use a floating-point coordinate, for which the graphics card will return an interpolated color. This is the color any specific pixel on the monitor should assume when the shape is displayed.

Scale Mode

Similar to Images as_scaled, we have options as to how we want the texture to behave when scaled to a size other than its native resolution. Mousetrap offers the following texture scale modes, which are represented by the enum TextureScaleMode:

TextureScaleModeMeaningEquivalent InterpolationType
TEXTURE_SCALE_MODE_NEARESTNearest Neighbor ScalingINTERPOLATION_TYPE_NEAREST
TEXTURE_SCALE_MODE_LINEARLinear InterpolationINTERPOLATION_TYPE_BILINEAR

While the resulting image behaves similarly to how InterpolationType will result in the final image, operating on a texture is much, much more performant. Rescaling a texture is essentially free when done by the graphics card, which is in stark contrast to the capabilities of a CPU, as would be needed for Images as_scaled.

Wrap Mode

Wrap mode governs how the texture behaves when a vertices' texture coordinate components are outside [0, 1]. Mousetrap offers the following wrap modes, which are all part of the enum TextureWrapMode:

TextureWrapModePixel will be filled with
TEXTURE_WRAP_MODE_ZERORGBA(0, 0, 0, 0)
TEXTURE_WRAP_MODE_ONERGBA(1, 1, 1, 1)
TEXTURE_WRAP_MODE_STRETCHNearest outer Edge
TEXTURE_WRAP_MODE_REPEATEquivalent pixel in ([0, 1], [0, 1])
TEXTURE_WRAP_MODE_MIRROREquivalent pixel in (1 - [0, 1], 1 - [0, 1])

How to generate this Image
using Mousetrap

# compound widget that displays a texture with a label
struct TexturePage <: Widget
    center_box::CenterBox
    label::Label
    render_area::RenderArea
    texture::Texture
    shape::Shape

    function TexturePage(label::String, image::Image, wrap_mode::TextureWrapMode)
        out = new(
            CenterBox(ORIENTATION_VERTICAL),
            Label("<tt>" * label * "</tt>"),
            RenderArea(),
            Texture(),
            Rectangle(Vector2f(-1, 1), Vector2f(2, 2))
        )

        set_expand!(out.render_area, true)
        set_size_request!(out.render_area, Vector2f(150, 150))    

        set_start_child!(out.center_box, AspectFrame(1.0, Frame(out.render_area)))
        set_end_child!(out.center_box, out.label)
        set_margin!(out.label, 10)

        create_from_image!(out.texture, image)
        set_wrap_mode!(out.texture, wrap_mode)
        
        set_texture!(out.shape, out.texture)

        # zoom out texture coordinates by 1 unit
        set_vertex_texture_coordinate!(out.shape, 1, Vector2f(-1, -1))
        set_vertex_texture_coordinate!(out.shape, 2, Vector2f(2, -1))
        set_vertex_texture_coordinate!(out.shape, 3, Vector2f(2, 2))
        set_vertex_texture_coordinate!(out.shape, 4, Vector2f(-1, 2))

        add_render_task!(out.render_area, RenderTask(out.shape))
        return out
    end
end
Mousetrap.get_top_level_widget(x::TexturePage) = x.center_box

main() do app::Application

    window = Window(app)
    set_title!(window, "Mousetrap.jl")

    render_area = RenderArea()

    image = Image()
    create_from_file!(image, "docs/src/assets/logo.png")
        # this assumes the script is run in `Mousetrap.jl` root

    # replace RGBA(0, 0, 0, 0) pixels with rainbow color
    size = get_size(image)
    hue_step = 1 / size.x
    for i in 1:size.y
        for j in 1:size.x
            if get_pixel(image, i, j).a == 0
                set_pixel!(image, i, j, HSVA(j * hue_step, 1, 1, 1))
            end
        end
    end

    box = Box(ORIENTATION_HORIZONTAL)
    set_spacing!(box, 10)
    set_margin!(box, 10)

    push_back!(box, TexturePage("ZERO", image, TEXTURE_WRAP_MODE_ZERO))
    push_back!(box, TexturePage("ONE", image, TEXTURE_WRAP_MODE_ONE))
    push_back!(box, TexturePage("STRETCH", image, TEXTURE_WRAP_MODE_STRETCH))
    push_back!(box, TexturePage("REPEAT", image, TEXTURE_WRAP_MODE_REPEAT))
    push_back!(box, TexturePage("MIRROR", image, TEXTURE_WRAP_MODE_MIRROR))

    set_child!(window, box)
    present!(window)
end

Where the default wrap mode is TEXTURE_WRAP_MODE_REPEAT.

By being able to modify the vertex coordinates for any of a shape's vertices, we have much more control over how image data is displayed on screen. Only the part of the texture that conceptually overlaps a shape will be displayed, which is governed by that shape's texture coordinates.


RenderArea Size

Because shapes do not take into account the size and aspect ratio of their RenderArea, we, as developers, should take care that shapes are displayed correctly when this size changes.

Consider the following example:

render_area = RenderArea()

shape = Ellipse(Vector2f(0, 0), 0.5, 0.5, 32)
add_render_task!(render_area, RenderTask(shape))

set_child!(window, render_area)

Where an ellipse with identical x- and y-radii is a circle.

Despite defining the shape as a circle, on screen, it appears stretched. This is because the shapes vertices use the GL coordinate system, which is normalized. Thus, how long the x- and y-radii of a circle are depends on the width and height of the RenderArea canvas. By widening the window, the render area expands, increasing its width and thus stretching our circle.

We have two ways to correct this. The easiest of which is putting the render area inside an AspectFrame, which forces it to always maintain the correct aspect ratio, square in this case:

render_area = RenderArea()

shape = Circle(Vector2f(0, 0), 0.5, 32)
add_render_task!(render_area, RenderTask(shape))

set_child!(window, AspectFrame(1.0, render_area)) # force 1:1 aspect ratio

While this corrects our circle, the entire RenderArea is now restrained in size, making this solution unviable for applications where we need a RenderArea to fill its entire area regardless of its aspect ratio.

The other way to correct the issue is to modify our circle when RenderArea changes shape. This is made possible by the resize signal of RenderArea, which is emitted whenever its allocated area changes:

resize requires a signal handler with the following signature:

(::RenderArea, width::Integer, height::Integer, [::Data_t]) -> void

Where width and height are the new sizes of the RenderArea widget, in pixels.

Using this information and some simple geometry, we can change the x- and y-radius dynamically whenever the RenderArea changes aspect ratio:

# define resize callback
function on_resize(::RenderArea, width::Integer, height::Integer, shape::Shape)

    # calculate y-to-x-ratio
    new_ratio = height / width

    # resize the shape by adjusting x-radius
    as_ellipse!(shape, 
        Vector2f(0, 0),     # old center
        0.5 * new_ratio,    # new x-radius
        0.5,                # old y-radius
        32                  # n vertices
    )
end

main() do app::Application
    window = Window(app)
    render_area = RenderArea()

    shape = Ellipse(Vector2f(0, 0), 0.5, 0.5, 32)
    add_render_task!(render_area, RenderTask(shape))

    # connect callback, providing our shape as `Data_t` argument
    connect_signal_resize!(render_area, on_resize, shape)

    set_child!(window, render_area)
    present!(window)
end

Here, the RenderArea has a non-square aspect ratio, yet the shape is still displayed as a proper circle. Using signal resize like this, we can protect ourselves against side effects from the normalized nature of GL coordinates.


Anti Aliasing

When shapes are drawn to the screen, they are rasterized, which is when the graphics card takes the mathematical shape in memory and transforms it such that it can be displayed using a limited number of pixels. This process is imperfect, as no number of pixels will be able to draw a perfect curve. One artifact that can appear during this process is aliasing, which, in non-technical terms, is when lines appear "jagged":

(Source: learnopengl.com)

To address the unsightly nature of this issue, a number of remedies are available, the most appropriate of which is called multi-sampled anti-aliasing (MSAA). User of Mousetrap are not required to understand the algorithm behind it, only that it causes jagged edges to appear smoother.

To enable MSAA, we provide an enum value of type AntiAliasingQuality to RenderAreas constructor:

msaa_on = RenderArea(ANTI_ALIASING_QUALITY_BETTER)
msaa_off = RenderArea(ANTI_ALIASING_QUALITY_OFF)
AntiAliasingQuality Value# MSAA Samples
ANTI_ALIASING_QUALITY_OFF0 (no MSAA)
ANTI_ALIASING_QUALITY_MINIMAL2
ANTI_ALIASING_QUALITY_GOOD4
ANTI_ALIASING_QUALITY_BETTER8
ANTI_ALIASING_QUALITY_BEST16

Where ANTI_ALIASING_QUALITY_OFF will be used when calling the RenderArea constructor with no arguments, as we have so far.

The higher the number of samples, the better the smoothing will be. MSAA comes at a cost: any quality other than OFF will induce the RenderArea to take about twice as much space in the graphic card's memory. Furthermore, the higher the number of samples, the more time each render step will take.

It's difficult to convey the result of MSAA using just pictures on a web page due to compression. Instead, readers are encouraged to run the following main.jl, which will show off the anti-aliasing in high resolution on the screen:

How to generate this Image
main() do app::Application

    window = Window(app)
    set_title!(window, "Mousetrap.jl")

    # create render areas with different MSAA modes
    left_area = RenderArea(ANTI_ALIASING_QUALITY_OFF)
    right_area = RenderArea(ANTI_ALIASING_QUALITY_BEST)

    # paned that will hold both areas
    paned = Paned(ORIENTATION_HORIZONTAL)

    # create singular shape, which will be shared between areas
    shape = Rectangle(Vector2f(-0.5, 0.5), Vector2f(1, 1))
    add_render_task!(left_area, RenderTask(shape))
    add_render_task!(right_area, RenderTask(shape))

    # rotate shape 1° per frame
    set_tick_callback!(paned) do clock::FrameClock

        # rotate shape 
        rotate!(shape, degrees(1), get_centroid(shape))

        # force redraw for both areas
        queue_render(left_area) 
        queue_render(right_area)

        # continue callback indefinitely
        return TICK_CALLBACK_RESULT_CONTINUE
    end

    # setup window layout for viewing
    for area in [left_area, right_area]
        set_size_request!(area, Vector2f(150, 150))
    end

    # caption labels
    left_label = Label("<tt>OFF</tt>")
    right_label = Label("<tt>BEST</tt>")

    for label in [left_label, right_label]
        set_margin!(label, 10)
    end

    # format paned
    set_start_child_shrinkable!(paned, false)
    set_end_child_shrinkable!(paned, false)
    set_start_child!(paned, vbox(AspectFrame(1.0, left_area), left_label))
    set_end_child!(paned, vbox(AspectFrame(1.0, right_area), right_label))

    # present
    set_child!(window, paned)
    present!(window)
end

Render Task

Tip

The rest of this chapter will assume that readers are familiar with the basics of OpenGL, how to write GLSL shaders, what a shader uniform is, how blending works, and how a linear transform allows us to move a point in 3D space.

With what we have learned so far in this chapter, we are already well-equipped to be able to accomplish most tasks that require the native rendering component, such as displaying static images or rendering shapes. Any information after this point should be considered optional to learn.

So far, we have registered render tasks using add_render_task!(render_area, RenderTask(shape)). Sometimes, we will have to deal with RenderTask on its own. Its constructor actually has the following signature:

RenderTask(::Shape ; [shader::Shader, transform::GLTransform, blend_mode::BlendMode])

Where any name after the ; are keyword arguments, which are optional.

We see that a RenderTask actually bundles the following objects:

  • a Shape, which is the shape being rendered
  • a Shader, which is a shader program containing a vertex- and fragment- shader
  • a GLTransform, which is a spatial transform that will be applied to the shape using the vertex shader
  • a BlendMode, which governs which type of blending will take place during the blit step

Using these four components, RenderTask gathers all objects necessary to render a shape to the screen. All components except for the Shape are optional. If not specified, a default value will be used instead. This is what allows less experienced users to fully ignore shaders, transforms and blend modes, simply calling RenderTask(shape) will take care of everything for us.


Transforms

GLTransform is an object representing a spatial transform. It is called GLTransform, because it uses the GL coordinate system. Applying a GLTransform to a vector in widget- or texture-space will produce incorrect results. They should only be applied to the position attribute of a Shape's vertices.

Internally, a GLTransform is a 4x4 matrix of 32-bit floats. It is of size 4x4 because it's intended to be applied to OpenGL positions, which are vectors in 3D space. In Mousetrap, the last coordinate of a spatial position is assumed to be 0, but it is still part of each vectors' data.

At any time, we can directly access the underlying matrix of a GLtransform using getindex or setindex!:

transform = GLTransform()
for i in 1:4
    for j in 1:4
        print(transform[i, j], " ")
    end
    print("\n")
end
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1

We see that after construction, GLTransform is initialized as the identity transform. No matter the current state of the transform, we can reset it back to this identity matrix by calling reset!.

GLTransform has many common spatial transforms already available as convenient functions, which means we rarely have to modify its values manually.

It provides the following transformations, which behave identically to those familiar from linear algebra:

  • translate!, translates in 3D space
  • scale!, scales along the x- and y-axis
  • rotate!, rotates around a point in 3D space

We can combine two transforms using combine_with. If we wish to apply the transform CPU-side to a Vector2f or Vector3f, we can use apply_to.

While we could apply the transform to each vertex of a Shape manually, then render the shape, it is much more performant to do this kind of math GPU-side. By registering the transform with a RenderTask, the transform will be forwarded to the vertex shaders, which, for the default vertex shader, is then applied to the shape's vertices automatically:

shape = Shape() 

transform = Transform()
translate!(transform, Vector2f(-0.5, 0.1))
rotate!(transform, degrees(180))

task = RenderTask(shape; transform = transform)

Where we used the transform keyword argument to specify the transform, while leaving the other render task component unspecified. This means the transform is applied automatically during rendering, allowing us to take advantage of the increased performance gained from the GPU architecture.


Blend Mode

As the third component of a render task, we have the blend mode. This governs how two colors behave when rendered on top of each other.

Let the color currently in the frame buffer be destination, while the newly added color will be origin. Each BlendMode, then, behaves as follows:

BlendModeResulting Color
BLEND_MODE_NONEorigin.rgb + 0 * destination.rgb
BLEND_MODE_NORMALtraditional alpha-blending
BLEND_MODE_ADDorigin.rgba + destination.rgba
BLEND_MODE_SUBTRACTorigin.rgba - destination.rgba
BLEND_MODE_REVERSE_SUBTRACTdestination.rgba - origin.rgba
BLEND_MODE_MULTIPLYorigin.rgba * destination.rgba
BLEND_MODE_MINmin(origin.rgba, destination.rgba)
BLEND_MODE_MAXmax(origin.rgba, destination.rgba)

Where +, *, and - are component-wise operations.

Users may be familiar with the names of the blend modes from traditional image editors such as GIMP or Photoshop.

If left unspecified, RenderTask will use BLEND_MODE_NORMAL.


Shaders

As the last component of a RenderTask, we have Shader, which represents an OpenGL shader program that contains already compiled fragment and vertex shaders. These shaders are written in GLSL, which will not be taught in this manual.

Compiling Shaders

To create a shader, we first instantiate a Shader, then use create_from_file! or create_from_string! to compile the shader from GLSL code. After which the component is automatically bound it to the shader program, overriding whatever shader component was there before.

We use SHADER_TYPE_FRAGMENT or SHADER_TYPE_VERTEX to specify which of the two shader types we are targeting.

shader = Shader()

# compile fragment shader
create_from_string!(shader, SHADER_TYPE_FRAGMENT, """
    #version 330
        
    in vec4 _vertex_color;
    in vec2 _texture_coordinates;
    in vec3 _vertex_position;

    out vec4 _fragment_color;

    void main()
    {
        vec2 pos = _vertex_position.xy;
        _fragment_color = vec4(pos.y, dot(pos.x, pos.y), pos.x, 1);
    }
""")

# bind with render task, which will automatically apply it to each fragment of `shape`
task = RenderTask(shape; shader = shader)
add_render_task!(render_area, task)

How to generate this Image
using Mousetrap
main() do app::Application

    window = Window(app)
    set_title!(window, "Mousetrap.jl")
    render_area = RenderArea()
    shape = Rectangle(Vector2f(-1, 1), Vector2f(2, 2))

    shader = Shader()
    create_from_string!(shader, SHADER_TYPE_FRAGMENT, """
        #version 330
            
        in vec4 _vertex_color;
        in vec2 _texture_coordinates;
        in vec3 _vertex_position;
    
        out vec4 _fragment_color;
    
        void main()
        {
            vec2 pos = _vertex_position.xy;
            _fragment_color = vec4(pos.y, dot(pos.x, pos.y), pos.x, 1);
        }
    """)
    
    task = RenderTask(shape; shader = shader)
    add_render_task!(render_area, task)

    frame = AspectFrame(1.0, Frame(render_area))
    set_size_request!(frame, Vector2f(150, 150))
    set_margin!(frame, 10)
    set_child!(window, frame)
    present!(window)
end

If we do not initialize the vertex- or fragment shader, the default shader component will be used. It may be instructive to see how the default shaders are defined, as any user-defined shader should build upon them.

Default Vertex Shader

This is the default vertex shader, used whenever we do not supply a custom vertex shader for a Shader instance:

#version 330

layout (location = 0) in vec3 _vertex_position_in;
layout (location = 1) in vec4 _vertex_color_in;
layout (location = 2) in vec2 _vertex_texture_coordinates_in;

uniform mat4 _transform;

out vec4 _vertex_color;
out vec3 _vertex_position;
out vec2 _texture_coordinates;

void main()
{
    gl_Position = _transform * vec4(_vertex_position_in, 1.0);
    _vertex_color = _vertex_color_in;
    _vertex_position = _vertex_position_in;
    _texture_coordinates = _vertex_texture_coordinates_in;
}

Where any variable name prefixed with _ signals that it was defined outside of main.

We see that it requires OpenGL 3.3 due to the location syntax. In terms of behavior, this shader simply forwards the interpolated vertex attributes to the fragment shader.

The current vertices' position is supplied via _vertex_position_in, the vertices' texture coordinates as _vertex_color_in, and the vertices' texture coordinates are _vertex_texture_coordinates. These values will contain the data from the Julia-side Shape. We should take care that the location attribute exactly matches this order, 0 for vertex position, 1 for vertex color, 2 for texture coordinate.

The output variables of the vertex shader are _vertex_color, _texture_coordinates and _vertex_position, which need to be assigned with results gained from within the vertex shader. The shader has furthermore access to the uniform _transform, which holds the GLTransform the current RenderTask associates with the current Shape.

Default Fragment Shader

#version 330

in vec4 _vertex_color;
in vec2 _texture_coordinates;
in vec3 _vertex_position;

out vec4 _fragment_color;

uniform int _texture_set;
uniform sampler2D _texture;

void main()
{
    if (_texture_set != 1)
        _fragment_color = _vertex_color;
    else
        _fragment_color = texture2D(_texture, _texture_coordinates) * _vertex_color;
}

The fragment shader is handed _vertex_color, _texture_coordinate and _vertex_position, which we recognize as the output of the vertex shader. The output of the fragment shader is _fragment_color.

The default fragment shader respects two uniforms, _texture, which is the texture of the shape we are currently rendering, and _texture_set, which is 1 if we called set_texture! on the current Shape instance, 0 otherwise.

Users aiming to experiment with shaders should take care to not modify the name or location of any of the in or out variables of either shader. These names, along with the required shader version, should not be altered.

Binding Uniforms

Both the vertex and fragment shaders make use of uniforms. We've seen that _transform, _texture, and _texture_set are assigned automatically. Mousetrap users can furthermore freely add new uniforms, assigning them in a convenient manner using RenderTask.

To bind a uniform, we first need a CPU-side value that should be uploaded to the graphics card. Let's say we want to use a certain color in our fragment shader, replacing the shape's fragment color with that color. We would write the fragment shader as follows:

#version 330

in vec4 _vertex_color;
in vec2 _texture_coordinates;
in vec3 _vertex_position;

out vec4 _fragment_color;

uniform vec4 _color_rgba; // new uniform

void main()
{
    _fragment_color = _color_rgba; // set fragment color to uniform
}

To set the value of _color_rgba, we use set_uniform_rgba!, which is called on the render task, not the shader itself. This will make the render task store the CPU-side value as long as it is needed, automatically forwarding it to the shader during rendering.

set_uniform_rgba! is one of many set_uniform_*! functions which allow us to bind Julia-side values to OpenGL shader uniforms:

The following types can be assigned this way:

Julia TypeRenderTask functionGLSL Uniform Type
Cfloatset_uniform_floatfloat
Cintset_uniform_intint
Cuintset_uniform_uintuint
Vector2fset_uniform_vec2vec2
Vector3fset_uniform_vec3vec3
Vector4fset_uniform_vec4vec4
GLTransformset_uniform_transformmat4x4
RGBAset_uniform_rgbavec4
HSVAset_uniform_hsvavec4

We would therefore set the _color_rgba uniform value like so:

# create shader
shader = Shader()
create_from_string!(shader, SHADER_TYPE_FRAGMENT, """
    #version 330

    in vec4 _vertex_color;
    in vec2 _texture_coordinates;
    in vec3 _vertex_position;

    out vec4 _fragment_color;

    uniform vec4 _color_rgba; // uniform we want to assign

    void main()
    {
        _fragment_color = _color_rgba;
    }
""")

# create shape and task
shape = Shape()
task = RenderTask(shape; shader = shader) # shader bound to `shader` keyword argument

# set uniform
set_uniform_rgba!(task, "_color_rgba", RGBA(1, 0, 1, 1))

How to generate this Image
using Mousetrap
main() do app::Application

    window = Window(app)
    set_title!(window, "Mousetrap.jl")
    render_area = RenderArea()
    shape = Rectangle(Vector2f(-1, 1), Vector2f(2, 2))

    shader = Shader()
    create_from_string!(shader, SHADER_TYPE_FRAGMENT, """
        #version 330
            
        in vec4 _vertex_color;
        in vec2 _texture_coordinates;
        in vec3 _vertex_position;
    
        out vec4 _fragment_color;

        uniform vec4 _color_rgba;
    
        void main()
        {
            _fragment_color = _color_rgba;
        }
    """)
    
    task = RenderTask(shape; shader = shader)
    set_uniform_rgba!(task, "_color_rgba", RGBA(1, 0, 1, 1))

    add_render_task!(render_area, task)

    frame = AspectFrame(1.0, Frame(render_area))
    set_size_request!(frame, Vector2f(150, 150))
    set_margin!(frame, 10)
    set_child!(window, frame)
    present!(window)
end

Where the name used in set_uniform_*! has to exactly match the variable name in GLSL.

With this, we have a convenient way to specify shader uniforms without having to manually update the shader each time it is bound for rendering. RenderTask does this for us.


Rendering to a Texture

Compat

This feature is not yet implemented, this section is incomplete.