HCI Framework for Java

Getting Started with sqUIrt

User documentation
08/17/2009

Using sqUIrt¶

Using the sqUIrt framework is rather straightforward. The process can be summarized as follows:

We create a new Java application
This Java application obtains an instance of an NUIController
We add Components to the NUIController
We add listeners to Components that execute our application code
We start() the NUIController

Let us examine this process via an example.

Hello World¶

Let's begin with a bare bones minimum example of a sqUIrt based application.

import nui.squirt.NUIController;
import nui.squirt.component.Label;

public class SquirtHelloWorld {

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();
        theController.add(new Label(0, 0, "hello world"));
        theController.start();
    }

}

As you can see, executing this application creates the text "hello world" in the middle of the screen. This text can be dragged around using either the mouse pointer or touch input, and likewise, can be scaled and rotated using pinch and stretch gestures.

But what is really going on in this code? This simple application begins by obtaining a NUIController instance by calling the static method getInstance() on the NUIController class. We then add a Component to the NUIController instance with the add() method. In this case we are creating a new Label Component. The parameters to the Label constructor are as follows:

The x position of the Component (this is relative to the center of the parent Component, more on this...)
The y position of the Component (this is also relative to the center of the parent)
And the String text to display

Children, Parents, Components, Containers, and the Scene Graph¶

Now what is this talk of parent Components? In our applications, the sqUIrt framework is used to construct a scene graph.

In a scene graph we build a tree structure that describes the graphical output of the program. All nodes of this tree are instances of Component objects. Components are thus the building blocks of all sqUIrt applications. We include Components in our sqUIrt application by adding them to something called Containers.

Now, Containers are merely a special kind of Component, and act as internal nodes of our scene graph tree. Containers support two very important methods, add(Component c) and remove(Component c).

In our Hello World example above we are actually using a Container. The NUIController implements the Container interface, and coincidentally acts as the root node of all sqUIrt-based scene graphs. As you can see above, we add a Label to the NUIController. Label implements the Component interface, but not the Container interface, so Label objects do not support add() and remove() methods.

An interesting note is that the Container interface extends the Component interface. This means that we can recursively build complex, multi-level trees by adding Containers to other Containers. For instance, let's modify our Hello World example from above to the following:

import nui.squirt.NUIController;
import nui.squirt.component.Button;
import nui.squirt.component.Frame;
import nui.squirt.component.Label;

public class SquirtHelloWorld {

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();
        Frame aContainer = new Frame(0, 0, 300, 200);
        aContainer.add(new Label(0, -25, "hello world"));
        aContainer.add(new Button(0, 50, "button"));
        theController.add(aContainer);
        theController.start();
    }

}

What we now have is a scene graph that is three levels! As you can see, we now create an instance of a Frame. A Frame implements the Container interface. Its constructor takes four arguments:

The x position
The y position
The width
The height

After creating the new Frame we add two different kinds of Components to it. The first is a Label, very similar to the one we had before. Although now, we change the y position of the Label from 0 to -50. This means that the center of the Label will be placed 50 units above the center of the Frame, regardless of where the Frame is positioned.

We then add another Component that we haven't seen before, a Button. The Button is a simple Component that the user can press and release using either the mouse pointer or touch input. When this happens, you can see that the color of the Button becomes slightly darker to provide feedback to the user that they Button is currently being held down.

The constructor of the Button takes three arguments, very similar to the constructor of the Label:

The x position
The y position
The text to display on the Button

Again, the x and y position are relative to the parent Container.

You can observe the qualities of the scene graph by grabbing the Frame and dragging it around or rotating and scaling it. Whenever the Frame undergoes these changes, all of its children Components also undergo the same changes. This is because child Components are always drawn relative to their parent Containers. This is the power behind scene graph based APIs. (Also note that a Frame can be resized by dragging at the corners. This changes the width and height of the Frame rather than scaling it.)

Now that you have a basic understanding of Containers and Components, take some time to experiment on your own. Try adding a Button directly to the NUIController. Try adding a Frame to a Frame. Try adding a two different Frames to the NUIController, each with their own set of child Components. See how the different sets of children respond to operations on either of the parent Frames.

So what kind of Components can I use in my sqUIrt applications?¶

There are a variety of Components available that come as part of the sqUIrt library. We have already examined three of these: the Label, the Frame, and the Button. But let us take a look at some other basic Components.

The TextField¶

Quite often there are times where we want our applications to obtain textual input from the user. For example, we may ask a user to enter a user-name and password to login to the system. Or perhaps, we'd like the user to enter data to be inputted into a database.

The sqUIrt framework supports this kind of input via the TextField Component. The TextField Component creates a white box on the screen where the user can enter text. If the user clicks on the TextField, or touches it, a caret begins flashing indicating where text will go if they start typing. The user can select text by highlighting using either the mouse pointer or a finger. All in all, it operates just about how you would expect.

To include a TextField in our applications, we need to instantiate one with a constructor.

TextField(float x, float y, float width)
TextField(float x, float y, float width, String initialText)

Like always, the first two arguments specify the position of the Component relative to the parent Container. The third argument describes how wide you would like the TextField to be. The fourth, optional argument is a String that will be preloaded into the TextField.

The TextField also supports two other important methods: getText() and setText(String text). The method getText() will return whatever String currently populates the TextField. This allows you to read input from the user. The method setText(String text) will change whatever text populates the TextField to the String you pass in as an argument.

The Image¶

Sometimes we want to display some kind of image in our applications. The sqUIrt framework supports this functionality, rendering an image on the screen that reacts to dragging, as well as rotation and scaling. To include an Image in our application we must use a constructor.

Image(float x, float y, String imagePath)
Image(float x, float y, float width, float height, String imagePath)

The first two arguments again specify a position. The imagePath argument describes a file path to the image. The image must reside inside a directory called data within your projects directory. Yet, imagePath may also be a URL to an image file online. The width and height arguments are optional and will force the image to render with those constraints. If not provided, the image will just be rendered to its native width and height.

The Knob¶

The sqUIrt framework also includes several Components to describe a continuous value between some bounds. For instance, a sqUIrt application to control a media player might want the user to be able to adjust the audio volume. The Knob offers such functionality.

The Knob is a circular Component that the user can rotate to adjust a continuous value. There are several parameters to specify when creating a Knob.

Knob(float x, float y, float radius)
Knob(float x, float y, float radius, float minimumValue, float maximumValue)
Knob(float x, float y, float radius, float minimumValue, float maximumValue, float initialValue)
Knob(float x, float y, float radius, float minimumValue, float maximumValue, float minimumAngle, float maximumAngle)
Knob(float x, float y, float radius, float minimumValue, float maximumValue, float minimumAngle, float maximumAngle, float initialValue)

x and y again describe position. radius describes how large to draw the Knob circle. The value that the Knob can represent is bounded by minimumValue and maximumValue, if these are not specified 0 and 1 are used by default. When the Knob is created, its starting value is described by initialValue, if left unspecified the initial value will be halfway between the lower and upper bounds. The possible rotation of the Knob can also be bounded by the minimumAngle and maximumAngle arguments, if these are not specified, the Knob can rotate between 0 and 2*pi radians.

The application can obtain the current value of the Knob by calling the getValue() method.

The Slider¶

Another Component that can be used to control a continuous value is the Slider. Instead of providing a circle to be rotated like with the Knob, the Slider provides a handle that can move along a line. For example, the Slider works well for instances where you might want to provide the user a way to adjust balance levels between a right and left speaker.

Let's take a look at the Slider constructors:

Slider(float x, float y, float length)
Slider(float x, float y, float length, float minimumValue, float maximumValue)
Slider(float x, float y, float length, float minimumValue, float maximumValue, float initialValue)

x and y again describe position. length describes how long to draw the Slider. The value that the Slider can represent is bounded by minimumValue and maximumValue, if these are not specified 0 and 1 are used by default. When the Slider is created, its starting value is described by initialValue, if left unspecified the initial value will be halfway between the lower and upper bounds.

The application can obtain the current value of the Slider by calling the getValue() method.

Other Components¶

sqUIrt comes packaged with additional Components not discussed here. These other Components are deemed outside the scope of this "Getting Started with sqUIrt" tutorial documentation. Feel free to reference all Components packaged with sqUIrt via the source code available at http://nuicode.com/projects/gsoc-hci-java/repository

Actual Javadoc coming soon!

Making a sqUIrt Interface Interact with My Application Code¶

Let's say that you have application code, for which you'd like to build a multi-touch front-end. You're in luck, because sqUIrt is designed for this. But how exactly do we tie application code and sqUIrt code together?

The Bad Way...¶

We have seen that we can access information within different Components via methods like getValue() for Knobs and Sliders, and getText() for TextFields. Additionally, you can call isPressed() on a Button to see if the user is currently holding it down. These are all very useful, but a naive way to tie this information into your application code would be something like the following:

import nui.squirt.NUIController;
import nui.squirt.component.Button;
import nui.squirt.component.Frame;
import nui.squirt.component.Label;

public class SquirtHelloWorld {

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();
        Frame aContainer = new Frame(0, 0, 400, 200);
        Label label = new Label(0, -25, "hello world");
        aContainer.add(label);
        Button button = new Button(0, 50, "button");
        aContainer.add(button);
        theController.add(aContainer);
        theController.start();

        boolean waiting = false;
        int count = 0;

        while(true) {
            // Check to see if the button has been pressed down
            if (button.isPressed())
                waiting = true; // We are waiting to perform an action, once that button is released

            // Check to see if the button has been released
            if (!button.isPressed() && waiting) {
                count++; // Increase the count of button presses
                waiting = false; // We are no longer waiting
            }

            // Update the label text
            if (count > 0)
                label.setText("button pressed " + count + " times");
        }
    }

}

In the above example, we maintain a infinite while loop that repeated polls the information within our sqUIrt Components. By executing this code, you can see that this method works, but lacks elegance and the while loop will quickly grow unwieldy when we start adding lots of Components to our user interface.

The Good Way... (aka Using Events)¶

Luckily for us, the sqUIrt framework supports a full Event-based structure for tying interface code and application code together.

This is all well and good, but what is an "Event-based structure" and how do we use it? An Event structure is a design pattern that allows us to specify what should happen whenever certain "Events" occur. An Event might be a Button being released, the text of a TextField changing, or the value stored by a Knob or Slider changing. What we do is we register Event-listeners with instances of different Components. Whenever a Component senses that an Event has occurred it tells the Event-listener to do whatever it is supposed to do.

Let's take a chance to examine how our "Bad" example from above would change as a result of using Events:

package nui.squirt.demo;

import nui.squirt.NUIController;
import nui.squirt.component.Button;
import nui.squirt.component.Frame;
import nui.squirt.component.Label;
import nui.squirt.event.ActionEvent;
import nui.squirt.listener.ActionListener;

public class SquirtHelloWorld {

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();
        Frame aContainer = new Frame(0, 0, 400, 200);
        final Label label = new Label(0, -25, "hello world");
        aContainer.add(label);

        Button button = new Button(0, 50, "button");
        button.addActionListener(new ActionListener() {
            private int count = 0;
            public void actionPerformed(ActionEvent e) {
                label.setText("button pressed " + (++count) + " times");
            }
        });
        aContainer.add(button);

        theController.add(aContainer);
        theController.start();
    }

}

As you can see above we are adding a special kind of Event listener to our Button. Now Button implements the Actionable interface, which means that you can register ActionListeners with it. The Button fires an ActionEvent every time the user releases the Button. When the ActionEvent is fired, the actionPerformed() method is called on every ActionListener added to the Button. You can see above the use of an anonymous class to define our ActionListener, but just as well, we can define our own regular class that implements a Listener interface.

Below we create our own class to respond to ValueEvents coming from Sliders and Knobs.

import nui.squirt.NUIController;
import nui.squirt.component.Frame;
import nui.squirt.component.Knob;
import nui.squirt.component.Label;
import nui.squirt.component.Slider;
import nui.squirt.event.ValueEvent;
import nui.squirt.listener.ValueListener;

public class SquirtHelloWorld {

    private static class ValueLabel implements ValueListener {
        private Label label;
        public ValueLabel(Label l) {
            this.label = l;
        }
        public void valueChanged(ValueEvent e) {
            label.setText(Float.toString(e.getNewValue()));
        }    
    }

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();

        Frame sliderFrame = new Frame(-400, 0, 400, 400);
        Label sliderLabel = new Label(-75, 0, "slider label");
        Slider slider = new Slider(150, 0, 275);
        slider.addValueListener(new ValueLabel(sliderLabel));
        sliderFrame.add(sliderLabel);
        sliderFrame.add(slider);

        Frame knobFrame = new Frame(400, 0, 400, 400);
        Label knobLabel = new Label(0, -25, "knob label");
        Knob knob = new Knob(0, 100, 75);
        knob.addValueListener(new ValueLabel(knobLabel));
        knobFrame.add(knobLabel);
        knobFrame.add(knob);

        theController.add(sliderFrame);
        theController.add(knobFrame);
        theController.start();
    }
}

By creating a class definition for our ValueLabel, we can create multiple instances without rewriting the code every time. As you can see above, we are implementing the ValueListener interface. Both Sliders and Knobs fire ValueEvents whenever the value the represent is modified. When this occurs, they call the valueChanged() method on all listeners registered with the Component.

The valueChanged() method is passed a ValueEvent object. The ValueEvent object supports several methods. In the above example we call the getNewValue() method. This returns whatever new value the Slider or Knob has assumed after the change. There is also the getOldValue() method for ValueEvent objects; this returns the value stored by the Slider or Knob before the change. ValueEvent, like all Event objects (including ActionEvent), supports the getSource() method. getSource() returns whatever Component generated the event. For example, let's say our interface has SliderA and KnobB. If the user moves the handle for SliderA, SliderA will call valueChanged() on all listeners it has registered, and will pass a ValueEvent object. Calling getSource() on this ValueEvent object will return a reference to SliderA.

Extensibility of sqUIrt, or Creating our own Components¶

The sqUIrt framework is built with flexibility in mind. Although sqUIrt ships with a handful of useful Components, we want to be able to define our own Components as well. To implement new Components with new functionality we need to create a new class that either implements the Component interface, or extends an existing Component type.

Let's take a look at a modification of a previous example, but this time we achieve the same kind of functionality by extending the Label Component with new behavior.

import nui.squirt.NUIController;
import nui.squirt.component.Frame;
import nui.squirt.component.Knob;
import nui.squirt.component.Label;
import nui.squirt.component.Slider;
import nui.squirt.event.ValueEvent;
import nui.squirt.listener.ValueListener;

public class SquirtHelloWorld {

    public static class ValueLabel extends Label implements ValueListener {    
        public ValueLabel(float x, float y, String t) {
            super(x, y, t);
        }
        public void valueChanged(ValueEvent e) {
            super.setText(Float.toString(e.getNewValue()));
        }
        @Override
        public void setText(String t) {
            // Empty. We don't want anything external to change the text of this Label.
        }
    }

    public static void main(String[] args) {
        NUIController theController = NUIController.getInstance();

        Frame sliderFrame = new Frame(-400, 0, 400, 400);
        ValueLabel sliderLabel = new ValueLabel(-75, 0, "slider label");
        Slider slider = new Slider(150, 0, 275);
        slider.addValueListener(sliderLabel);
        sliderFrame.add(sliderLabel);
        sliderFrame.add(slider);

        Frame knobFrame = new Frame(400, 0, 400, 400);
        ValueLabel knobLabel = new ValueLabel(0, -25, "knob label");
        Knob knob = new Knob(0, 100, 75);
        knob.addValueListener(knobLabel);
        knobFrame.add(knobLabel);
        knobFrame.add(knob);

        theController.add(sliderFrame);
        theController.add(knobFrame);
        theController.start();
    }

}

Now our ValueLabel isn't just an implementation of ValueListener, it is also an actual Label! By extending existing Component implementations we can modify behavior by simply overwriting the methods that we want to change.

This is the easiest way to create new Components. Although, sometimes we desire a new Component type that is so novel, it cannot be created by modifying an existing Component type. In these cases it is best extend the abstract class AbstractComponent. At a minimum the developer will be forced to implement the following methods in their new Component:

public void render(PApplet p)
public void update()
public boolean canAcceptMoreControlPoints()
public void controlPointCreated(ControlPoint cp)
public void controlPointDied(ControlPoint cp)
public void controlPointUpdated(ControlPoint cp)
public boolean isUnderPoint(ControlPoint cp)
public boolean offer(ControlPoint cp)

This kind of extension is beyond the scope of this "Getting Started with sqUIrt" tutorial, but expect further documentation in the future covering this material!