Queensland University of Technology Digital Observatory: Studying social media, music, and streaming behavior
About QUT Digital Observatory
Based at QUT, the Digital Observatory tracks public communication and content consumption on a range of platforms and services.
Tell us your challenge. We're here to help.
Contact usUsing Google BigQuery, Queensland University of Technology (QUT) Digital Observatory has captured large and fast-growing datasets for analysis and visualization.
Google Cloud Results
- Enabled QUT Digital Observatory to identify 500,000 meaningfully active Twitter accounts from 3.7 million accounts in Australia
- Supported projects with industry partners, such as analyses of social media activity around natural crises
- Helped deliver insights about the sharing and consumption of news sources on Twitter
Capturing a dataset of 2.4 billion tweets daily
Digital platforms and services are transforming the way individuals interact and consume creative works such as music, movies, and television shows. At Queensland University of Technology (QUT), a group of researchers and academics are undertaking an innovative project to track public communication and content consumption on these platforms and services.
Initiated by the QUT Digital Media Research Centre and operated by the QUT Institute for Future Environments, the QUT Digital Observatory project brings together a number of projects that emerged independently at the institution to create a single dataset accessible to QUT academics, researchers, and students. “We are interested in going beyond short-term analyses of the response to and interaction with particular events on digital platforms and towards a much longer-term tracking of public communication and consumption of content,” Professor Axel Bruns, Creative Industries Faculty, QUT Digital Media Research Centre, explains.
Analytics data warehouse essential
To develop the Digital Observatory, the QUT Digital Media Research Centre needed an analytics data warehouse to capture fast-growing datasets from platforms and services that could then be analyzed, visualized, and reported on. “We looked at the market in 2015 and quickly decided on Google BigQuery,” Professor Bruns says. “The product met our cost, scalability, and performance requirements and integrated seamlessly with the tool we elected to use for reporting.”
Google BigQuery now captures data for all modules and projects of the Digital Observatory. The largest and most advanced of these projects is known as Tracking Infrastructure for Social Media Analysis (TrISMA).
TrISMA aims to gather data from the social media platforms used most widely in Australia for public communication. “We’ve conducted a great deal of work to identify as many Australian Twitter accounts as possible and track the public activities of these accounts on an ongoing basis,” Professor Bruns says. “While a lot of existing analysis looks at how Twitter hashtags or small populations of Twitter accounts react to events such as an election or a celebrity death, we look at what is going on day-to-day, including how Australian user activity relates to big events.
“For example, we try to understand whether the reaction to a specific event comprises 1%, 10%, or nearly 100% of the overall conversation taking place at that time. This enables us to assess how important to the public conversation such events really are.”
“For the Twitter part of TrISMA alone, we needed a product that could store a live dataset with 2.4 billion tweets—growing at about 1.3 million tweets per day—and about 3.7 million Australian Twitter accounts, as well as a dataset of the global Twitter user base. Google BigQuery does this extremely successfully.”
—Professor Axel Bruns, Creative Industries Faculty, QUT Digital Media Research CentreOther components of the Digital Observatory include the Australian Music Observatory that pulls together comprehensive live data from industry sources about radio airplay, Spotify charts, and recorded music in live venues to provide a view of how digital music is consumed in Australia. “We analyse how these different forms of consumption influence each other,” Professor Bruns says. “So, do you hear things on the radio and start streaming them on Spotify, or does the radio play songs that are already popular in Spotify? Do venues lag behind these types of services?
“How does that work around launch and promotion campaigns for particular artists, and does artists touring influence consumption patterns?”
Data about songs and artists from thousands of radio stations is gathered once a week or daily and pushed into Google BigQuery, from which researchers can compare and derive conclusions from patterns across different datasets.
Another Digital Observatory project—the Digital Media Observatory—involves gathering data from services such as Netflix to determine what programming is available in Australia relative to the United States and other territories. This gives researchers an understanding of the choices streaming services such as Netflix and content providers make.
To deliver the Twitter component of the TrISMA project, the QUT Digital Media Research Centre team deployed a tool that gathers large datasets through the Twitter API. The tool then pushes the data into Google BigQuery for processing, reporting, and analysis using a third-party product.
“Ultimately, with Google BigQuery and reporting and visualisation tools, we are trying to get to a strong and detailed approximation of how Australians use these three major platforms for public communication.”
—Professor Axel Bruns, Creative Industries Faculty, QUT Digital Media Research CentreCapturing billions of tweets
“For the Twitter part of TrISMA alone, we needed a product that could store a live dataset with 2.4 billion tweets—growing at about 1.3 million tweets per day—and about 3.7 million Australian Twitter accounts, as well as a dataset of the global Twitter user base,” Professor Bruns says. “Google BigQuery does this extremely successfully.”
The organization arrived at the number of accounts in Australia by locating accounts that had set an Australian timezone, or mentioned Australia or one of its 45 largest towns and cities in their profile settings. Of the 3.7 million Australian accounts, the QUT Digital Media Research Centre team identified about 500,000 as meaningfully active in tweeting.
The seamless integration between Google BigQuery and the analysis tool enabled the QUT Digital Media Research Centre team to make the dataset available to a broad set of internal users for research. “We just need to train users on the one reporting tool, which makes the process easy,” Professor Bruns explains.
“We are working on a similar project with Facebook and Instagram,” he adds. “On Facebook we are looking to track as many public pages that relate to Australia as possible, so we can see what is going on day-to-day, whereas on Instagram we are trying to identify as many locations as we can around Australia that people are making posts to.”
Using social media for public communication
“Ultimately, with Google BigQuery and reporting and visualization tools, we are trying to get to a strong and detailed approximation of how Australians use these three major platforms for public communication,” Professor Bruns adds.
The project is enabling QUT researchers to work with businesses and government agencies to understand social media activities in their sector. “For example, we completed one project with Queensland Fire and Emergency Services to help them understand how people were using social media in relation to natural disasters involving cyclones, bushfires, floods, and the like,” Professor Bruns says. “We’ve also collaborated with various companies to help them understand how people are talking about their products and services.”
“Thanks to Google BigQuery and our reporting tools, we’ve found on Twitter that quality news publications do a lot better than they should, relative to market share data accessible from other sources. We can draw conclusions from this about Twitter users wanting to appear knowledgeable to their followers.”
—Professor Axel Bruns, Creative Industries Faculty, QUT Digital Media Research CentrePowering a Twitter index
In addition, Professor Bruns is using the project architecture to understand what news articles are being shared—including fake news items.
“I’ve been running the Australian Twitter News Index for several years, tracking the volume of link-sharing on Twitter for about 30-odd Australian news publications,” he explains. “This allows us to assess everything from the market for each publication down to the performance of individual stories on the social media network. As a result, we can make some fine-grained assessments involving demographics and stories that are compelling enough to be shared by Twitter users who don’t normally take an interest in the issue.”
The qualitative and interpretive work undertaken by the QUT Digital Media Research Centre team has delivered a range of valuable insights. “We’ve found on Twitter that quality news publications do a lot better than they should, relative to market share data accessible from other sources,” Professor Bruns says. “We can draw conclusions from this about Twitter users wanting to appear knowledgeable to their followers.
“We also find that Twitter users are more likely to share stories they think their followers will not have seen before, rather than articles covering widely known events.”
Through the data stored in Google BigQuery, the organization has also been able to build a picture of who follows whom and to identify networks of bots used to tweet, retweet, follow other accounts, or perform other actions within Twitter in Australia.
Evaluating Looker Studio
With Google BigQuery successfully powering TrISMA and other Digital Observatory projects, the QUT Digital Media Research Centre is now examining the potential of Looker Studio to provide dashboards. “We would like to have a dashboard that picks out from the daily throughput some of the most active hashtags and accounts, or the most popular keywords and URLs,” Professor Bruns explains. “And for more specific projects such as the Australian Twitter News Index, we’d like to have a dashboard that shows the most widely shared news sources, firstly on the domain level and potentially down to the level of individual stories.” This console would increase awareness of the project by providing a public-facing view of its activities and potentially secure commissions to undertake more bespoke work.
Tell us your challenge. We're here to help.
Contact us
About QUT Digital Observatory
Based at QUT, the Digital Observatory tracks public communication and content consumption on a range of platforms and services.