Interactive movies are films where audiences either as a group or individuals make choices at certain points to change the plot. The ability to do this has been around for many years, in fact since the 1960’s. However, the way these films are put together has not changed, for example ‘Bandersnatch’ uses the same method of interaction used in the 1960’s!
The AIM project is investigating how to make interactive movies that do not explicitly ask the audience to make a decision to change content, but rather adapts to the viewer as they watch. Our goal is to build and embed an algorithm into a Unity platform, to create an App through which people can watch interactive movies. Underpinning research conducted by Richard Ramchurn used a brain scanner electroencephalogram (EEG) sensor which edited a film in real time depending on the viewers’ levels and frequencies of attention. Moving this concept forward and in recognition of the fact that audiences do not have access to EEGs, we wanted to explore how to create an adaptive cinema experience, using technology and equipment people have readily to hand.
Our work commenced with the development of a computer vision machine learning algorithm that can detect emotion as arousal and valence and this will act as a driver for a specific interactive film. The machine learning process considers the personality of the audience – which is novel. A key aspect of our work has been how to develop this in a safe and secure way to ensure that the audience has control over their personal data.
The making of our specific interactive film took place at the Cobot Maker Space at the University of Nottingham.
‘Before We Disappear’ co-stars Spot, a Boston Dynamics robot dog. The production of the film was part funded by the Arts Council England with support from Nottingham City Council and Nottingham Trent University. Before We Disappear is set 20 years from now and is about climate change. The film has a backbone of 5 scenes, each of which can detour to an additional scene, providing different routes to continue the journey of the film. During the making of ‘Before We Disappear’, we employed 18 filmmaking professionals and provided training to 13 film students – most of which were halfway through their degree.
In addition, we conducted a personality study with 700 participants to support grouping and producing a pool of people and their preferred genre of film. The pool provided data about peoples’ media watchlists which we used validate what people said they liked. From the pool we then identified and invited a representative group to watch a section of film which enabled us to monitor facial expressions and gain insights into what people look like when they are engaged in narrative moments.
We are now working on building a mark-up tool to identify strength of positive and negative emotions. Once the tool is developed, we will be introducing it to expert filmmakers and film academics and working with them to mark-up sections of ‘Before We Disappear’ with emotions that the directors are trying to elicit in the audience.
Further updates on our work to follow.