We live in a world where the combination of Moore’s Law and Metcalfe’s Law heralds a data revolution. The billions of smartphone and broadband users today already generate massive quantities of data. And cheap sensors in Internet of Things (IoT) devices and new 5G networks optimized for machine-to-machine (M2M) mean that this first wave of users will soon be joined by tens of billions of machines.
The IoT is here and 5G is coming
The IoT is already happening—with 13 billion devices in use today. And 5G is just around the corner, with the first networks to be commercially available around 2020. 5G has been designed for capacity but also specifically with M2M communications in mind—scale, low power, energy efficiency, and support for a massive number of devices.
Volume and velocity of big data
The volume and velocity of data today is almost unlimited, while the constraints on collecting and moving it around the world are disappearing. For the most part, this is happening in the enterprise space. The question, then, for enterprises—and for their CIOs, specifically—is whether all of this data can be gathered, often in real time, stored, and then analyzed?
These kinds of questions highlight the IT challenges that enterprises will face from this data explosion, with perhaps the most important being the question of ingestion.
The question of ingestion
The ingestion of data has to be done, in real time, using a distributed messaging layer that decouples the data capture from the storage, processing, and analysis of the data. Apache Kafka and MapR Streams have been built to easily capture any type of data and move the data in a publish-subscribe model between various server and software components.