Monday, September 27, 2010

Posting from the lab today, running my 2nd (out of 3) experiment. Feedback from initial subjects and pilot testing over last weekend and yesterday resulted in the following changes for today:
  • Increased the annotation interval from 10 to 20 seconds - We will get fewer data points (120 vs 240), but 10 seconds was found to lengthen the experiment to the point that subjects became bored and agitated.
  • Removed the need of spontaneous annotations from the first part of the experiment - It was found that it distracted subjects from the actual game if they were instructed to give spontaneous feedback.
  • Instruct the subject to minimise movement to increase recording accuracy
As far as data analysis goes, I plan to use weka ( as a tool to analyse my data.

Still technically running on schedule, but I would like to get most of the experiments out of the way and start analysis this week, as I am relatively unfamiliar with the machine learning aspect of things and may need some time to read into this subject. I'll start to analyse these 3 experiments this week and see if i need to find more participants next week.

Monday, September 20, 2010

Running the pilot study

The LATTE group has software that I need to run the second part of my experiment (annotation of the video after physiological recording). Thus I will adapt my original prototype to be used in viewing the media during physiological recording to record spontaneous emotional response.

I'll be running a pilot study this week on myself first. The total experiment time should take no longer than 2 hours.

I've found 3 participants so far, and those tests will be run next week (mid-semester break).

Monday, August 16, 2010

Progress on prototype

Development of the prototype has been making progress. Some code from the Interactive Tutoring video annotation program has been adapted into my prototype. I plan to have a basic interface done by the end of this weekend. This puts me on track to finalise the prototype (week 5, which will be on schedule).

I have also found 3 or 4 subjects suitable for using NRL game footage as stimulus. At this point I will have to start looking into how I will acquire footage for an entire match.

Currently the plan is to use a 40 minute video segment (half a game) followed by 40 minutes of annotation, with 10 minutes changeover and setup. 90 minutes seems reasonable, but there may not be enough data for all different classifier combinations.

If there aren't enough data points, I might see if I lengthen each experiment session or do physiological recording and self-annotation stages in separate sessions. There are issues with either decision:
  • By lengthening the recording sessions, I will gain more data points, which will probably increase the experiment's accuracy. On the other hand, if the session is too long (as I experienced in the siento experiments I participated in earlier) the subject can lose interest/concentration which could yield varying results.
  • By splitting two stages up, I gain more data points and the subject will be 'fresh' for each session. However, the subject's emotional state may change from each session, especially if they cannot remember their emotional response during the session when annotating footage.
I will finalise a decision after testing the video annotation software against half a match of an NRL game and seeing how many data points are recorded.

Sunday, August 8, 2010

Creating a prototype

Over the next few weeks a prototype will be developed for the experiment. Also, the experimental outcome and procedure has been finalised:
  1. Pre-annotate a 40 minute video with 20 second intervals between annotations of valence and arousal (3 levels each, for a total of 6 classes. We need at least 20 samples per class, so 120 samples all up) *To ensure we have enough samples, an interval of 10 or 15 seconds may have to be used instead*
  2. Record (physiological signals) a subject viewing the video.
  3. Have the subject self-annotate the video after first viewing.
Total experiment time for the subject should be around 100 minutes in total (10 minutes setup, 40 minutes initial viewing, 50 minutes self-annotation).

The user interface for the self-annotation phase will have to be very intuitive (1 click to record valence and arousal in under 3 seconds) to shorten the time of the experiment and prevent the subject from losing interest.

The project will be coded up in MATLAB, with the code following the protocols used by other projects in the LATTE group. Schedule for the next few weeks:
  • This week (week 3) - Review code for video annotation and start on own prototype
  • Next week (week 4) - Work on prototype
  • Following week (week 5) - Finish basic functioning prototype
Hopefully I can have a fully functioning prototype by week 7, run the experiments from weeks 8 through 10, analyse data from weeks 9 through 11 and write up the final report in weeks 12 and 13.

Sunday, August 1, 2010

Finalising experimental procedure

A basic outline of the experimental procedure has been drawn up:

  1. Annotate sports footage with expected levels of valency and arousal
  2. Expose test subject to the same footage whilst gathering their feedback on valency and arousal
  3. Measure test subject's physiological response whilst watching footage
  4. By comparing these results to the pre-annotated results, we can see how well we can predict levels of valency and arousal from this media.
  5. If there is enough time, we can use steps 1-4 on other forms of sports footage to see any correspondence in valency and arousal levels between different types of sports (e.g. if a test subject's team scores in any sport, we would at least expect positive valency)
The stimulus to be used will either be 1 half (45 minutes) of a UEFA Europa Cup game (football/soccer), or 1 half of a locally broadcast NRL game (rugby league). This will be finalised by Tuesday (2 days from now), as I'm contacting potential test subjects for their availability with this experiment.

Tuesday, May 25, 2010

A possible change in stimulus?

I've asked around for possible test subjects, and with the 2010 FIFA world cup happening soon, I might consider either using that or one of the European leagues currently playing as stimulus. Advantages with this stimulus is that it harder to watch entire games live due to the time difference (hence, it is more likely that test participants will not know the outcome of given stimulus). However, as previously mentioned, there aren't too many definite moments in a football match (e.g. scoring a goal) where affect can be annotated.

Another stimulus i could use are UFC (Ultimate Fighting Championship) or Boxing matches. The format of UFC matches (3 or 5 rounds of 5 minutes each with 1 minute of rest between each round) aligns closely with previous studies carried out using video stimulus. Also, the sport is not widely broadcast locally, which makes it easier to find subjects who won't know the result of a match (so their recordings will be more 'authentic'). The downside is that fights are very unpredictable and short; Most fights last for less than 2 or 3 rounds.

Analysing psychophysiological signals

After a week off last week due to a few assignments, I've been making an effort to catch up on work, especially with the progress report due next week. Reading this week was on a text book recommended by Rafa called "Psychophysiological Recording" (Stern, 2001). Currently I am familiarising myself with the equipment used in physiological recording and the analysis of EEG and EMG signals.

Wednesday, April 28, 2010

Familiarisation with equipment

Last Friday I participated in a trial experiment measuring physiological signals using images from IAPS as stimulus using protocol/software developed by Sazzad, Payam and Omar. There was also an additional experiment where they measured my physiological response from using an interactive tutoring program. I familiarised myself with the equipmen test setup and typical protocol used in recording physiological signals.

I observed some difficulties that my project will encounter:
  • The setup is a bit intrusive. The participant has to wear electrodes on part of their face and arms and movment is partially restricted. Too much movement will result in 'noisy' signals.
  • There are several types of sensors measuring arousal levels (ECG, skin conductivity, respiration rates), but only one type measuring valence (EMG).
  • Because of the above points, the stimulus must be of relatively short duration to limit movement and 'noise'.

Monday, April 19, 2010

Project proposal submitted

Project proposal was submitted on Friday. The title was: "Hedonics of Sports Fans"

Basically, I plan to use either live or annotated video footage of an NRL game as stimulus while recording a subject's physiological response. Events in a sports match might be easier to match to predicted emotions (e.g. you would expect a supporter to be joyful if their team scores a try, or be nervous if the scores are close).

Some difficulties I predict will be finding the right subjects and scheduling. The subjects need to have some sort of emotional attachment to the stimulus, so I need to find pretty big supporters of NRL teams, then schedule a session where I can record their reactions. A live observation of the match would be preferable, but this will be much harder to organize.

Also, meeting with Sazzad and Omar on Friday to familiarize myself with the test equipment (I am volunteering to be a test subject in one of their experiments).

Monday, April 12, 2010

Closing in on a topic proposal

Meeting with Rafa today resulted in a thesis topic: Hedonics of sports fans. Generally (in the field of affective computing) this involves measuring the emotional response of a subject when exposed to certain stimuli.

In this case, the stimulus will be a sports game and the user will be a fan of one of the teams playing. A sports game was chosen as emotional peaks should be easy to predict. For instance, if the user is a fan of a team, and that team scores a goal, try or point, then we should expect they would be have a positive valence of affect. If their favourite player gets injured or if their team loses at the last moment, then we might think that they would experience strong negative emotions. User affect should be much more objective and discrete in this stimulus when compared to other types of multimedia.

Which sport to use?
I was thinking of maybe using NRL (National Rugby League) coverage as stimulus, for the following reasons:
  • Games are frequently televised and it is in-season for the duration of my project. 26 rounds are played from March until Septempber (plus additional rounds for the finals), with up to 8 matches per round, all of which are televised (albeit, only 3 on free to air TV during the weekends).
  • I know of many friends who are pretty big fans, which should make signals easier to extract (I'm guessing they would exhibit stronger emotions).
  • Most of the televised broadcasts are live, which should also yield stronger physiological signals. If a game is a replay, then it would take some of the 'excitement' away.
I also considered using the NBA (Basketball) or A-League (Soccer) games for stimulus, but with the NBA playoffs starting in April, I felt there wasn't enough time or opportunity to have subjects watch a live game. Also, I didn't think there would be enough goals scored in a soccer match to collect signals from. Which brings me to issues:

Live vs Replay?
  • Viewing of a live game would almost certainly result in a stronger physiological response. However, it would be harder to organise and there would be issues in annotation (you would have to annotate on the fly, maybe keeping a record of who scores, or what you consider to be important moments in the match).
  • With a replay, you can annotate a video beforehand (thus collecting more accurate and numerous points of interest), but collected signals might be weaker.

Foreseeable difficulties:
  • Timetabling will be a tricky issue if a live viewing is chosen. I'll have to find strong supporters for the games and have them come in when the teams play.
  • Recording data for the entire game will have difficulties. It might be uncomfortable for a subject to be hooked up to the equipment for the length of a whole game (an NRL game lasts more than 80 minutes, Football matches go for over 90).

Thursday, April 8, 2010

Deciding on a Project Proposal

After meeting with Omar today, I've narrowed my project proposal down to either:
  • An investigation into individual users' affect profiles when presented with stimuli.
  • Compare affect response using different types of stimuli (e.g. images from IAPS vs video clips).
The first choice would involve quite a bit of technical knowledge in either data mining or signal analysis, and would require a more specialized experimental protocol. The second choice would be easier to complete but I'm not sure if it is unique enough compared to previous studies.

Tuesday, March 30, 2010

Literature review plan and 'Writing a Thesis'

After meeting with Rafa and other project students yesterday, I had a look through the handout titled "Making a Strong Start" (from "How to Write a Better Thesis", Second Edition, David Evans, Paul Gruba. Melbourne University Press, 2002), and lecture slides on how to write academic introductions.

At the end of the chapter, the article summarizes the following about thesis structure:

The Standard Structure is composed of 4 parts:
  1. Introduction - Outline the problem and how you'll be tackling it, scope and how the thesis will be structured.
  2. Background chapters - Material required (essentially a literature review) prior to your work.
  3. An account of your own work - Begin with a formal statement of your hypotheses or research questions, propose a methodology to test your hypotheses (including reasoning for selection) and report the results.
  4. Synthesis - Discuss the implications of results, draw conclusions and modify existing theory (or create a new one).
I also had a look through another article ("Towards a Framework of Literature Review Process in Support of Information Systems Research", Yair Levy, Timothy J. Ellis, 2006 Informing Science and IT Education Joint Conference, 2006) for my literature review (and eventually for my actual Treatise).

Sunday, March 14, 2010

First meetings

Last week I met briefly with Rafa who he gave me an overview of the nature of work that the LATTE group did. After this, I also met with Omar Alzoubi (one of the LATTE group members) who showed me around the lab, the equipment they used to collect physiological signals (EEG, ECG, EMG, SC, etc.), and possible areas I could focus my project on.

Edit: Omar suggested I could run a full study (with 25-30 candidates) measuring physiological responses using their gear and then analysing the data, or I could work on improving some of the DSP algorithms they use to analyse data (e.g. ECG waves).

Currently, Rafa has suggested that I focus on Affective computing modelling. The problem involves taking the physiological response from subjects when exposed to stimuli and create a unique/personalized 'profile' for a subject's emotional response.

This could require the development of software that analyzes the results and qualitatively or quantitatively measures the response. E.g. For a particular subject, an increase in defensive blinking and an increase in rate of respiration could indicate a that this subject is experiencing strong emotions related to panic or sadness.

Difficulties in this project I can foresee:
  • Creating an individual profile of affect. Omar explained briefly that a certain emotion might be more prevalent given different pre-conditions (e.g. perhaps subjects won't have the same physiological response on different days), so repeated test data might be necessary.
  • How to relate or categorize emotions given different physiological signals and present them as qualitative/quantitative data. A framework must be established that would map certain responses to certain types of emotions (e.g. How do we measure 'happiness'? What sort of units would we use?).

Things to do for this week:
  • Read into articles about diagnostic assessment of emotional recognition (previously I had only read into predictive assessment).
  • Research previous suitable frameworks available that fits the processes LATTE uses.

Thursday, March 4, 2010

Affective Computing Thesis/Project

This online journal will be a record of my progress in my ELEC4712/4713 project. The topic is in the area of Affective Computing: "Electrophysiological signals for the recognition of emotion"

My primary supervisor is Rafael Calvo. Another point of reference from last year (just before the break) was Judy Kay, but at this point she hasn't been listed as a co-supervisor yet (some admin stuff I'll have to sort out soon!).

Actual research related material:

During the break I have been reading up on material given to me last year by Judy and Rafael about affective computing, user modelling and Dynamic Bayesian networks. Although I think my project will ultimately focus on using a diagnostic approach to model dynamic decision networks (DDNs), it is interesting to look at the alternative methods. Judy pointed me towards a paper that looks into modelling DDNs by predictive assessment ("Empirically building and evaluating a probabilistic model of user affect", Conti and Maclaren, 2009).

The article above also made reference into OCC theory, so I looked further into this in an article by Adam, Herzig and Longin ("A logical formalization of the OCC theory of emotions", 2009).