Recognizing Mouse Gestures

Материал из Wiki.crossplatform.ru

Qt Quarterly | Выпуск 18 | Документация

For fast typists, there are keyboard shortcuts. These are fullysupported by Qt and easy to set up. The mouse-oriented equivalent ofkeyboard shortcuts is mouse gestures. In this article, I will presenta few classes that you can use to support mouse gestures in your Qtapplication.

Содержание 1 The Recognition Algorithm 1.1 Step 1: Filtering the mouse movement 1.2 Step 2: Limiting the directions 1.3 Step 3: Simplifying the direction list 1.4 Step 4: Matching and reducing 2 The Qt Classes 3 Bridging the Gap 4 Bringing Gestures to the Application 5 Running the Example 6 Conclusions and Possible Improvements

Mouse gesture support is becoming increasingly common, not only onPDAs but also in desktop applications such as Opera and MozillaFirefox, not to mention games (e.g., Black & White). The conceptis simple: The user gestures a shape while holding a mouse button,and if the shape is recognized the associated action is executed.

We will start by looking at a basic mouse gesture framework I havedeveloped using Qt 4. We will then examine how to use theframework from a Qt application.

[Download source code]

The Recognition Algorithm

Recognizing a mouse gesture isn't as difficult as it may sound,because it can be reduced to four fairly straightforwardsteps: filtering, limiting, simplifying, and matching.

Step 1: Filtering the mouse movement

The filtering is applied while a mouse movement (a segment in thegesture) is recorded. It prevents very short mouse movements frombeing recorded. This step is necessary because different hardwarereports mouse positions at different rates, and because the rest ofthe algorithm might fail if the movement is too small.

Step 2: Limiting the directions

The limiting step is about determining what the user actually meant.This is done by limiting each segment of the gesture to one of thefour directions up, down, left, or right. More precisely, we need tocompare the x and y components of each segment and zero outthe smaller of the two.

Step 3: Simplifying the direction list

The third step, simplifying, consists of finding consecutivemovements in the same direction and joining these. This gives us alist with the directions that make up the gesture (e.g., "right, up,right, up"). This list can then be matched against predefinedgestures.

Step 4: Matching and reducing

The matching part is the most difficult part of the algorithm, sinceit is common for users to change the direction slightly as theyrelease the mouse button. In addition, we must deal with smallglitches along long movements.

The algorithm starts by trying to match the gesture against apredefined list. If that fail, we remove the shortest segment fromthe gesture and try again. This is repeated until a match is found ora too large proportion of the original gesture has been removed.

The Qt Classes

The mouse gesturing framework described in this article makes it easyto add gesture support to existing Qt applications. The Qt-specificpart of the framework consists of two classes:

• QjtMouseGestureFilter is an event filter that catchesrelevant mouse activity on a specified Qt widget.
• QjtMouseGesture represents a gesture.

Let's start with the QjtMouseGestureFilter classdefinition:

class QjtMouseGestureFilter : public QObject
{
public:
    QjtMouseGestureFilter(Qt::MouseButton button = Qt::RightButton,
                          QObject *parent = 0);
    ~QjtMouseGestureFilter();
 
    void addGesture(QjtMouseGesture *gesture);
    void clearGestures(bool deleteGestures = false);
 
protected:
    bool eventFilter(QObject *obj, QEvent *event);
    ...
};

To use it, we simply need to create an instance, callinstallEventFilter() on the windows or widgets that shouldsupport gesture input, and populate it with a list of mouse gestures.Here's the definition of the gesture class:

class QjtMouseGesture : public QObject
{
    Q_OBJECT
 
public:
    QjtMouseGesture(const DirectionList &directions,
                    QObject *parent = 0);
    ~QjtMouseGesture();
 
    const DirectionList directions() const;
 
signals:
    void gestured();
 
private:
    ...
};

A gesture is essentially a list of directions and a signal. Theavailable directions are Up, Down, Left, Right,AnyHorizontal, AnyVertical, and NoMatch.AnyHorizontal means "left or right", whereas AnyVerticalmeans "up or down". The NoMatch direction is used to create agesture that is matched if no other gesture is matched.

The Direction enum that defines the available directions is partof the Gesture namespace, described in the next section. To savea few keystrokes, we use the following typedefs and usingdirectives:

typedef Gesture::Direction Direction;
 
using Gesture::Up;
using Gesture::Down;
using Gesture::Left;
using Gesture::Right;
using Gesture::AnyHorizontal;
using Gesture::AnyVertical;
using Gesture::NoMatch;
 
typedef QList<Direction> DirectionList;

Bridging the Gap

The actual mouse gesture recognition classes are available as a setof framework-neutral classes in the Gesture namespace. The Qtspecific classes map Qt's event filtering mechanism to a set offramework-independent mouse-following functions, and a callback classto a Qt signal. The neutral classes corresponding toQjtMouseGesture are shown below:

typedef enum { Up, Down, Left, Right, AnyHorizontal,
               AnyVertical, NoMatch } Direction;
typedef std::list<Direction> DirectionList;
 
class MouseGestureCallback
{
public:
    virtual void callback() = 0;
};
 
struct GestureDefinition
{
    GestureDefinition(const DirectionList &amp;d, MouseGestureCallback *c)
        : directions(d), callbackClass(c) {}
 
    DirectionList directions;
    MouseGestureCallback *callbackClass;
};

To bridge the gap with Qt, we must subclass MouseGestureCallbackand reimplement callback() as follows:

class GestureCallbackToSignal
        : public Gesture::MouseGestureCallback
{
public:
    GestureCallbackToSignal(QjtMouseGesture *object) {
        m_object = object;
    }
 
    void callback() {
        m_object->emitGestured();
    }
 
private:
    QjtMouseGesture *m_object;
};

To allow the bridging class to emit the gestured() signal onbehalf of the QjtMouseGesture object, we use theemitGestured() private function. This requiresGestureCallbackToSignal to be declared as a friend ofQjtMouseGesture. The private section of QjtMouseGesture lookslike this:

class GestureCallbackToSignal;
 
class QjtMouseGesture : public QObject
{
    ...
 
private:
    friend class GestureCallbackToSignal;
    void emitGestured();
 
    DirectionList m_directions;
};

The framework-neutral class corresponding to QjtMouseGestureFilteris called MouseGestureRecognizer:

class MouseGestureRecognizer
{
public:
    MouseGestureRecognizer(int minimumMovement = 5, double minimumMatch = 0.9);
    ~MouseGestureRecognizer();
 
    void addGestureDefinition(
            const GestureDefinition &gesture);
    void clearGestureDefinitions();
 
    void startGesture(int x, int y);
    void addPoint(int x, int y);
    void endGesture(int x, int y);
    void abortGesture();
 
private:
    ...
    class Private;
    Private *d;
};

The startGesture(), addPoint(), and endGesture()functions are invoked from the event filtering part of theQjtMouseGestureFilter class. The actual bridging takes place inaddGesture():

void QjtMouseGestureFilter::addGesture(
        QjtMouseGesture *gesture)
{
    Gesture::DirectionList dl = gesture->directions().toStdList();
 
    d->bridges.append(GestureCallbackToSignal(gesture));
    d->gestures.append(gesture);
 
    d->mgr.addGestureDefinition(
          Gesture::GestureDefinition(dl, &d->bridges.last()));
}

We copy all the directions from a QList to an STL list,then we wrap the gesture into a GestureCallbackToSignal instance;finally we add a GestureDefinition to theMouseGestureRecognizer.

The bridges and gestures added to the QjtMouseGestureFilter areheld in the private member variable d along with theMouseGestureRecognizer instance.

Bringing Gestures to the Application

From the perspective of the Qt application programmer, what is reallyinteresting is how we can bring gestures to an application. To illustrate this,we will use the simple MainWindow class shown below as an example.

Here's the class definition:

class MainWindow : public QMainWindow
{
    Q_OBJECT
 
public:
    MainWindow();
 
public slots:
    void clearAll();
    void setAll();
 
    void noMatch();
 
private:
    ...
};

The original main() function, with no gesture support, looks likethis:

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    MainWindow mainWin;
    mainWin.show();
    return app.exec();
}

To add gestures to the application, we must create an instance ofQjtMouseGestureFilter and install it on the MainWindow. Wemust also define gestures and hook them up to the application. Forexample:

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    MainWindow mainWin;
 
    QjtMouseGestureFilter filter;
    mainWin.installEventFilter(&filter);
 
    // Clear all by making three sideways moves
    QjtMouseGesture g1(DirectionList() << AnyHorizontal
                      << AnyHorizontal << AnyHorizontal);
    filter.addGesture(&amp;g1);
    connect(&amp;g1, SIGNAL(gestured()), &amp;mainWin, SLOT(clearAll()));
 
    // Set all by moving up, then left
    QjtMouseGesture g2(DirectionList() << Up << Left);
    filter.addGesture(&amp;g2);
    connect(&amp;g2, SIGNAL(gestured()), &amp;mainWin, SLOT(setAll()));
 
    mainWin.show();
    return app.exec();
}

To make the application slightly more user friendly, we can add anoMatch() slot to MainWindow and associate it with a special"no match" gesture:

int main(int argc, char **argv)
{
    ...
    // When nothing else matches
    QjtMouseGesture g3(DirectionList() << NoMatch);
    filter.addGesture(&amp;g3);
    connect(&amp;g3, SIGNAL(gestured()),
            &amp;mainWin, SLOT(noMatch()));
    ...
}

Running the Example

So far, we have talked about how the framework recognizes gestures, andtaken a brief tour of the code for a simple example. Building this exampleshould just be a simple case of running qmake and make.

Now, let's run the example. The window (shown in the opposite column) displaysfive checkboxes that will be set or unset when an appropriate gesture is used.If you hold down the right mouse button – the default gesturebutton in the QjtMouseGestureFilter constructor – the mousemovements will be recorded. Releasing the mouse button causes recording tostop, and the example will either update the checkboxes if it recognized thegesture, or it will display a dialog reminding you which gestures it knowsabout.

One recognized gesture involves moving the mouse up then to theleft; another requires three horizontal movements. There's also asecret gesture that you'll have to discover for yourself!

Conclusions and Possible Improvements

With the mouse gesture framework presented in this article, mousegesture support can easily be added to any Qt application. Theframework-independent gesture-recognizing code has been wrapped in a Qtinterface that makes the integration with other Qt classes seamless.Qt-specific enhancements could include support for QSettings-basedstorage of gestures, provision of visual feedback to help users learnnew gestures, and even gesture editing facilities.

The gesture recognizing algorithm outlined here is simple to getstarted with but has some limitations. In particular, it doesn'tsupport diagonal directions or relative lengths. Pen-based userinterfaces such as PDAs and tablet computers would benefit most fromsuch improvements.