From a technical standpoint, the aim of this project was to create a location aware binaural audio game running natively on Android. The main design considerations were to create a system which gave clear and consistent user feedback, responded consistently to the user’s input, and was easy to use and learn. The application was to feature dynamic audio and visuals which responded to the direct input of the player and their physical location. Key to the success of the code was for this information to be portrayed as simply as possible in order to avoid confusion and aid learning.
Below I have given a descriptive account of the application, giving a overall structure of the code, followed by a more detailed breakdown of the processes involved in each stage. I also give comments regarding technical problems that arose during the test session, the current state of the code, and future solutions.
For a more technical understanding of the application, the fully commented code should be downloaded from: adamcampbell.org/dmsp/TheLostSoulsOfGeorgeSquare.zip
The source Java file is located at
The Pure Data patch is contained in
‘Finding’ state refers to the player searching for the ghost, before they have crossed the distance threshold, ‘battle’ state is when user has crossed the threshold and is interacting with the ghost.
GPS sensor – Used to measure the player’s current location and calculate their bearing to and distance from pre-set dummy locations.
Magnetometer – Used to calculate player’s orientation to dummy ghost locations.
Accelerometer – Used to calculate shake behaviour.
Pure Data & LibPD – Pure Data is used for all audio components and some game logic. Embedded into application with LibPD.
Processing – Used for all visual components
Java and Eclipse – Used for collecting sensor data and the majority of game logic.
External libraries / code – ProcessingTouchEvents and UiImageButton (twobigears.com/), tbe_3DCeption~ external for Pure Data (twobigears.com/)
Overall Programme Structure
– read sensor data
– calculate user’s location
– calculate distance of user to next game location
– calculate user’s bearing to game location
– if user is within 10 meters of ghost GPS location, trigger Battle state and allow shake behaviour
– calculate overall magnitude of three poles of accelerometer
– if magnitude is over threshold trigger victory sound and display victory screen
– if user has completed less than 4 stages direct player to next GPS location
– if user has completed 4 stages trigger final victory sound and end screen.
Detailed breakdown of each section
– Static visual screen.
– Pre-recorded introduction sequence is played.
– Hold image on screen until sound file has finished, then trigger next game state.
– Compass which rotates and points towards next ghost location relative to user’s current orientation (low pass filter applied to smooth out data).
– Counter of the number of ghosts saved and the current game level.
– Reset button to restart the game.
– Distance meter which fills as the player gets closer. This moves in stages rather than in a smooth movement to counteract the slow and unpredictable rate at which GPS data updates.
– Ghost voice, which is triggered randomly (frequency and phrase) and is panned binaurally towards ghost location, using azimuth.
– Music is a loop with added chorus effect which increases as the user gets closer to the ghost location, adding a warbling / out of tune effect.
– User’s GPS location measured by phone.
– Distance from user’s location and pre-set ghost GPS location is calculated
– User’s bearing to ghost GPS location is measured.
– If user is within distance threshold then trigger thunder sound and next state.
– ‘Shake’ text instructing the user to shake the phone
– ‘Magic symbol’ image to indicate user can interact with ghost
– Ghost image
– Reset button
– Background colour is linked to the magnitude of the accelerometer and changes as the user shakes the phone harder.
– When state is first triggered, play thunder sound.
– Ghost voice, triggered more frequently in this stage. Binaural panning is random and moves around the players location to give the impression of the ghost moving around the players head.
– A short music loop which is different for each stage.
– ‘Shake’ audio, linked to the phone magnitude. Increases in intensity and modulation as the player shakes the phone harder.
– Calculate overall magnitude of accelerometer (low pass filter applied to smooth out sensor data).
– If magnitude passes threshold, trigger Victory state
– Static screen, displays an image of whichever animal has been saved (i.e. level 1 owl, level 2 chipmunk etc.)
– Reset button
– Pre-composed sound file is played
– Hold image on screen until sound file has finished, then trigger next state.
– Static screen image
– Reset button
– Pre-composed victory sound
– Hold image on screen until user presses ‘reset’ button to start game again
Comments and possible improvements
During the test session a bug was discovered which allowed the user to advance the game while not in ‘battle’ mode. This lead to user’s becoming confused as the cues did not clearly communicate the game logic. This bug has since been fixed and all planned interactions and behaviours work consistently.
The main issue with the current build is that game logic is shared between Pure Data and Java. This came about due to the manner in which the project was built up, with early prototypes being constructed in Pure Data, but caused difficulties with debugging and editing as the size of the project increased. As a result the code became more difficult to follow, as messages are passed back and forth between the two environments. If continuing with the project I would re-write the code to be as fully implemented in Java as possible, using Pure Data to generate and modulate sounds, but not handle game logic.
This and other style issues in the code (mainly present in the pd patches) are due to a lack of thorough planning before beginning building the project, although don’t affect the running of the application.