Case Study

Client

Transport for NSW

Industry

Government, transport

Primary Brief

The 2015 Sydney storms and the Lindt Cafe siege were critical incidents that both cased major city-wide transport disruptions, with commuters stranded up to several hours or more in the Sydney CBD. Transport for NSW, as part of their major incident alert review, saw the need to standardise the visual language of their alert systems across their websites and apps.

The objective was to understand how commuters interpret various alerts, and use this information to deliver the appropriate detail and severity level of public transport disruptions. Ultimately, this would to allow commuters to take precaution, arrange alternative travel or avoid specific areas during such incidents.

What we did

Audit and prototype design

After an initial audit of all alert types across key Transport NSW’s apps and websites, we then designed three versions of an optimised alert framework for home pages, travel planning pages and timetable pages.

Usability and A/B split testing

A select group of commuters were recruited and tested on specific channels. The 0rder of testing device (channel) use randomised to control for interface learning and channel bias.

Independent variables such as icon, colour, timestamp Vs. no timestamp, and alert position were A/B split-tested across the respondent sample group.

Eye tracking

As participants completed tasks, we tracked eye gaze fixation to determine visual priority as well as correlate low or high performance on certain tasks with eye gaze patterns.

Results

Perceived Relevance and Criticality (PRC) scores

A measurable constant, Perceived Relevance and Criticality (PRC) score was recorded for each task. That is, for each scenario, the level of relevance to the individual as well as the criticality of the alert message was collected.

Interestingly, the poorest results were observed when the original timestamp was displayed without being updated in the second part of each participant session. In other words, there was an immediate drop in the level of how relevant or critical the alert was if the timestamp was seen to not have changed from the original alert.

This was most evident in scenarios where the incident displayed yesterday’s date and the participant was presented with a scenario of the alert for the next morning. When presented with a hypothetical choice of whether to travel to work the next day or not, there was a higher inclination to travel to work the day after an incident if the PRC based on timestamp was low.

The positive affect of alert colour was slightly higher on desktop than on mobile, despite there being a higher tendency to read the alert on mobile than on desktop. Speculatively, this may indicate a reduction in “banner blindness” for alerts on mobile.

Importantly, the affects of timestamp variations were found to be independent of banner colour.

Additional findings – The rise of social media in critical incidents

As with any digital research engagement, we explored the various methods that each participant used during the critical incidents as well as any other recent incident. It was interesting to see a reduction in trust of various mainstream news outlets as well as the providers of the service, and an increased trend and reliance on social media such as Twitter for quicker and “more accurate” updates on situations.