Gesture Analysis Made Easy with INTERACT
Detailed gesture analysis consists of multiple nested levels that indicate not only the movement of the active hand itself, but also more specific details like the structure, focus and function of a movement.
Coding of simple gestures and automatic calculation of complex gesture combinations from this simple initial data.
At first glance, logging this kind of nested information seems to be a simple matter of lexical coding in INTERACT. But due to the nature of the nesting, this approach involves a huge amount of double entries while coding – making it far more time consuming than necessary!
This article describes how you can simplify data collection by logging all levels seperately (using the coding mode ‘Standard’ and mutually exclusive codes whenever possible).
Goal
To tell the movements apart, the required gesture information is first collected in a separate DataSet for each hand and person, but once the data is collected, you can take advantage of the various restructuring routines available in INTERACT to transform your events into coherent segments that reflect the exact structure of each part of the movement. Finally, the data can be joined into a single DataSet per person and used for complex analysis.
The biggest challenge we face here
Collecting multi-level Codes for which the duration of the subordered levels may differ from the initial activation, without the need to re-enter the upper-level information again and again for every tiny change in any of those levels.
That’s How it's Done
Example Study Design
Dyadic communications are recorded with two cameras, each one focusing on a participant.
Each hand is observed separately and each gesture level is coded in a single pass, but if you feel you can handle two levels at a time, you are free to do so.
For easier handling, we recommend logging the gesture data in a separate data file and only merging the final, restructured data with the other behavioral Events at the end of the process.
Defining a Coding Scheme
We create code definitions for each level like this:
In the first pass, you can log the activation of both hands simultaneously if you want. Those activation Events cover the total duration of a movement, independent of any changes in structure, focus or function during the movement!
Additional levelscan of course be implemented, but for this example the following 3-level setup enables us to show and tell the logic behind this sophisticated method of collecting complicated information the easy way:
Prepare Gesture Data Collection
The approach explained here takes advantage of the very clever command Create Coding Segments to get the information structure we need, after all data has been collected.
To make this work, it is important that you can properly tell apart the information collected per hand. Therefore, we start by collecting or splitting the basic hand movements into a separate DataSet per hand:
If you have already coded the activation per hand, you better extract and split up those Events into separate Sets before you continue.
TIP: If you collected the activation for both hands in one pass and thus within the same DataSet, you can use the command Group Events with identical Codes into DataSets to automate the split-up!
Collect Gesture Observations
Once you have collected/split the initial hand movements in separate Sets per hand, it is time to get into analyzing each movement in detail.
To specify the structure of each move, open the Code definition file that holds the ‘Structure’ Codes. Verify that the coding mode ‘Standard’ is active and start the observation mode.
- Double click on an ‘Activation’ Event to watch the movement or that particular hand, so you know what structure is required for the first part of the movement.
- Double click on the onset of this Event to position the video just there, so you can use this time information for the start of your first ‘Structure’ Event.
- Click the ‘Structure’ Code that applies to the first part of the movement and doubleclick the ‘Activation’ Event again to replay that Event.
TIP: Reduce the play speed for more accuracy. - The moment the structure of the movement changes, click the structure Code that matches the second part of the movement.
Due to the exclusion lists specified in the Code definitions, the previous structure Event is automatically closed the moment the new structure Event is started.
As long as you do not manually pause the video, playback will end the moment the offset of the Activation Event is reached. The moment you pause the video, this no longer works until you double-click that Event again.
- To close the last ‘Structure’ Event of the first movement, click the currently open Structure Code.
TIP: If you are sure the current Structure Event lasts until the end of the movement, you do not need to wait until the end of the selected Event is reached: Double-click on the offset of the ‘Activation’ Event and click the currently open ‘Structure’ Code to close it.
- Repeat those steps for each Activation Event for both hands.
- When finished, stop the observation session BEFORE you open the ‘Focus’ Codes within the Code Definition window.
For coding this Focus level, as well as any of the next levels, you can decide whether to replay the initial ‘Activation’ Event, like described, or if you need to focus on the Events created in the previous pass – in this case the ‘Structure’ Events.
Time Line Chart
After you finished coding, your data file will contain data like this for each hand like seen here in the Time Line Chart window of INTERACT.
Call the Descriptive Statistics function in INTERACT and this it will provide you with information about the frequency of movements, the total duration of those movement etc. per hand.
Similar for all Structure, Focus and Function Codes, independent of their combination.
Visualize Co-occurrences
Visualizing the collected hand movement per hand using the State-Space-Grid can provide you an interesting insight as well, depending on the selected Classes.
Each hand has a different color and the cells show you the co-occurring situations for two Classes at the time:
Extracting Combined Gesture Patterns
To get details on the combinations of more than two levels however, we need to re-structure the data by making use of a few clever routines INTERACT has to offer.
So far, we have independently collected data for the different levels of detail for each movement. Now we want to extract Events that represent every gesture pattern (any combination of Activation, Structue, Focus and Function) that occurred.
This means, that whenever there was a change in Structure, Focus or Function during a movement, this should be considerend a different type of movement and thus be represented by an Event that holds all of those co-occurring Codes:
Extracting this information, based on the data structured in one DataSet per hand, takes just a single mous click:
Select the command Create Coding Segments to create a new data file that now holds completely new Events, each of which representing the exact sequence that holds the current combination of Structure, Focus and Function of each movement:
This data now looks like it has been coded lexically.
For analysis, especially the new Class ‘AllCodes’ is of interest, because this provides us with the information about all the co-occurring combinations:
Join Events for both Hands
Now each Event can be identified effortlessly, so it is time to join the Events of both hands, at least if you want to find specific co-occurrences between hands: Select the Restructuring functionality Combine all DataSets in a new file to join the Events of all DataSets per DataGroup.
Analysis
To find all instances where both hands were active at the same time, you can now simply run the co-occurrence filter on the two Activation Codes ‘Left Hand’ and ‘Right Hand’.
For finding co-occurrences on specific types of movement, you might need to create new combination Classes, similar to the ‘Allcodes’ Class. If, for example, you are interested in the situations where both hands were moved with the same focus, independent of the structure or function, you need to run the Move & Combine Codes command to create a combined code for just the Classes ‘Activation’ and ‘Focus’.
The complete segments, the Co-occurrence filter should best be run on the ‘AllCodes’ Class.
Your advantage with INTERACT
One of the biggest advantage of INTERACT is its ability to re-organize, rename and move collected data, should you notice that your initial structure is not perfect to find the answers you are looking for.