1. Introduction to Data

Big Data


In recent decades, the amount of data in various fields has increased massively. Given the explosive growth of global data, the term “Big Data” is primarily used to describe huge datasets.

To give an idea of how large data must be for “Big Data,” the 3V model from Gartner (the same company known for the Hype Cycle) is often used. In the 3V model, the “Big” refers to three dimensions.

Big Data


Volume

Volume refers to the continuously increasing amount of data.

A lot of data are generated within just a few seconds on social networks: every video upload or tweet, but also every click and purchase produces data. The scale of the resulting data can be seen in this graphic. For example, over 500,000 gigabytes of data are generated in just 30 seconds.

Variety

The increasing variety refers to the many different types of data (images, emails, audio recordings, bank transactions, tracking data) that are generated, stored, and processed.

                                         


This variety is one of the major challenges of “Big Data.” In addition to so-called structured data, which we have already used before, more and more unstructured or semi-structured data are being used.

 

  • If data can also be stored in tables in a structured way, we call them structured data. Examples of structured data include John Snow’s data that we used in the last chapter or customer master data of an online shop. Such data have the advantage that management and access are efficient (We want all customers’ first names? No problem!).
  • The term unstructured data refers to digitized information that is not organized in such a formal structure. This can be images, digital texts, audio recordings, but also continuous data from a sensor.
  • Semi-structured data represent a mix of structured and unstructured data. For example, in an email some structured data can be found (sender, recipient, time, subject). The body of the message consists of arbitrary text, images, or other attachments and is therefore unstructured.

 

Velocity

The increasing speed has two dimensions. On the one hand, the speed at which data are generated continuously increases (see also Volume). On the other hand, this also increases the required processing speed.

Often, this processing must even occur in real-time (i.e., almost simultaneously with real events or at least with minimal delay), which is no easy task given the sheer volume of generated data. For example, when you pay with your credit card during online shopping, not only is it checked whether the entered card number is registered with your bank, but also whether you are within your credit limit, whether the payment looks suspicious, etc. Only when all checks are successfully completed is the payment initiated. All of this must, of course, take only a few seconds. Similarly, an earthquake early warning system should be able to issue warnings at the first signals and not wait several hours to analyze the data.