Economic Forecasting Using Social Media Data Is Now a Reality
Berkshire Media, a leading social analytics firm in Malaysia recently discovered a breakthrough approach in forecasting the state of an economy through social media data. This approach provides an alternative measurement to complement traditional economic indicators such as Gross Domestic Product (GDP), Gross National Income (GNI), Consumer Confidence Index (CCI), employment rates, and others.
We managed to secure an exclusive interview with Founder and Managing Director, Shahid Shayaa, to share more about the new frontier of social analytics and the impact to businesses and organizations.
You have been working on this revolutionary approach in forecasting the economy of Malaysia, tell us more how did you come up with this idea?
Economists around the world have been known to rely on economic indicators, or perhaps providing anecdotal perspectives which may cause cognitive bias when representing economic realities on the ground. Moreover, there have been debates around the methodology and limited sample size used to arrive at the economic indicators such as the Consumer Confidence Index (CCI).
We discovered a new metric known as Consumer Sentiment Index (CSI), a measurement of citizen's sentiment on the current state of the economy. CSI uses machine learning approach based on large scale datasets extracted from more than 10 digital media & social media channels. Social media data from netizens' conversations and comments, as well as news and blog posts were extracted into CSI categories such as Consumer Purchase, Employment, Personal Finance and Price of Controlled Goods.
For example, under the CSI category of Consumers Purchase, we extracted and cleaned social media data from netizens who expressed an interest to purchase or an action of past purchase decisions - from going to the cinema, eating out, purchasing flight tickets and to booking hotels.
Similar to CCI which are published every quarter by every country, our goal is to roll out CSI as a widely accepted economic indicator that can be refreshed every week or even days. We have been working on a National Consumer Predictor (NACOP) project using CSI as the underpinning to publish a more robust economic indicator for Malaysia.
Has there been any similar work or are you the first to come up with this idea?
We constantly push the boundaries on what we can do with social media data and our work was first inspired by the European Central Bank research study; 3 billion worth of social data was used to predict the state of the economy in Netherlands. However the data was entirely in Dutch.
In terms of methodology and the ability to process social media data in Bahasa Melayu & English language and dialects, we are the first company to have taken a deep-dive to arrive at the CSI numbers based on millions of social data with a robust set of accuracy across all CSI categories.
We have been extremely happy with the results due to our specialization in Bahasa Melayu (i.e Malay language which is used by 290 million people in Malaysia, Indonesia, Singapore, Brunei and Thailand). With the help of our proprietary text algorithm engine, it allows us to understand the complexity of the language vis-a-vis local dialects, nuances and non-dictionary based keywords.
What major discoveries have you found and how accurate was the results?
Interestingly, our CSI results mimics the trends and patterns from the quarterly CCI numbers published in Malaysia since 2015. Our data analysis was based on three years of social media data from 2015 to 2017 with millions of social data included in the study.
Through our proprietary text sentiment engine SentiRobo, we discovered that Consumer Purchase sentiment was highly positive, indicating strong positive sentiment on consumer spending in 2015 and 2016, with robust spending activities across multiple industries. Only 7.24% were tagged as negative, with the main grouses on why they are not spending being expensive goods, products, low quality of service and others.
The economic trend in Malaysia between 2015 and 2016 showed marginal improvement, with increased consumer spending activities from 91% in 2015 to 94% in 2016. This corresponds with recent reports stating that empty malls may not necessarily indicate that people are not spending, as more Malaysians are now doing their shopping online, with a 161% increase in online spending between the 2015 – 2016 period, at RM3.8 billion in transacted value.
One of the interesting findings from CSI was the large volume of negative sentiment on unemployment; showing posts about retrenchments or layoffs, salary cuts and others in 2016. This is in line with public data in 2016 where approximately 40,000 Malaysian employees were laid off from their jobs, while Labour Department statistics showed that more than 17,798 workers have been retrenched as at June 2016.
How will your work impact organizations and businesses?
We have been approached by many organizations; namely financial institutions, research organizations and economists to use our data as a means to predict the future. This is an exciting time for social analytics firms around the world as organizations are moving from using automated sentiment tools to service based social analytics firms.
Large organizations are now turning to social media data for fresh and new insights as the digital economy continue to flourish. Various research has shown that the CSI holds significant predictive power in explaining consumer behavior and to a certain extent, predicting the stock market volatility. For B2C marketers, this provide an opportunity to plan for product launches based on consumer's sentiment.
We believe that the key to unlock the value of social media data is to transform relevant data sets into meaningful insights that can contribute towards problem-solving and understanding human behavior, or in this case, consumer behavior and perception. On that note, for unstructured data, no tools or AI can automate that function to achieve the desired results.
To read more about this, visit here: