Data are flooding in at rates never seen before—doubling every 18 months—as a result of greater access to customer data from public, proprietary, and purchased sources, as well as new information gathered from Web communities and newly deployed smart assets. These trends are broadly known as “big data.” Technology for capturing and analyzing information is widely available at ever-lower price points. But many companies are taking data use to new levels, using IT to support rigorous, constant business experimentation that guides decisions and to test new products, business models, and innovations in customer experience. In some cases, the new approaches help companies make decisions in real time. Big data is generating an intense amount of attention among businesses, media and even consumers, along with analytics, cloud-based technologies, digital channels and data visualization.
What is big data?
Big data involves the data produced by different devices and applications, like data from Black Box, Social Media, Stock Exchange, Power Grid, Transport, Search Engine etc. Thus Big Data includes huge volume, high velocity, and extensible variety of data and thus could be of three broad types –
• Structured data – Relational Data
• Semi Structured Data – XML Data
• Unstructured Data – Word, PDF, Text, Media Logs. It also could be typically characterized by the four “V’s”:
• Volume: the amount of data being created is vast compared to traditional data sources
• Variety: data comes from different sources and is being created by machines as well as people
• Velocity: data is being generated extremely fast — a process that never stops, even while we sleep
• Veracity: big data is sourced from many different places, as a result you need to test the veracity/quality of the data
The impact of big data
To understand how data has transformed our daily lives, look at how the movie rental process has been changed. When movies were rented from nearby stores, the store would recommend on which movies the customer said they liked and also by sharing their own opinions.
Today, movie rental companies and content delivery services can utilize an array of data points to generate recommendations. By analyzing what was viewed, when, on what device, as well as user activities such as internet searches, and browsing and scrolling within a webpage, recommendations can be tailored for customers in real time and approximately 75 percent of views at a leading provider are now driven by these recommendations.
Big data life cycle
There are various types of data that we were generating for long but rarely been using in a meaningful manner:
1. Location of a Person
2. Restaurant they visit, food they consume
3. Credit Card expense pattern etc.
New technology such as advanced sensors and customized software can now record this information for analysis.
Too much large volumes of data were not traditionally captured and processed for many reasons, mostly because the cost was far larger than the value - companies could derive from its analysis. But due to many of the factors and new technologies the lowered cost is now leveraging the technology barrier for effective data processing, allowing mid size companies also, to be able to unlock the value contained in different
For instance, it is difficult for conventional relational databases to handle unstructured data, so software frameworks like Hadoop, for distributed storage and parallel processing of large datasets have been introduced to process non-structured data at high speed; to perform a more detailed analysis of big data.
Though these days it is easier and cheaper to capture, store and process data, still it will not be useful unless the information is relevant; it should also be available to
the right people who require the proper and accurate input in order to make logical decisions leading to successful results.
There are three key enablers:
Mobile — mobile networks have allowed easier distribution of information in real-time
Visual/interactive — technologies have brought the ability to review large and complex data sets into the average business users’ reality
Human resource — there is a new breed of employees with the knowledge to handle the complexities of big data and with the ability to simplify the output for daily use
Resources and processes
An important factor to achieve big data success is to have resources that have relevant knowledge and competency. This extends beyond the so-called data scientists who have deep knowledge and experience in handling, analyzing and reporting on big data sets. While these skill-sets are high in demand, success requires more than having a handful of specialists on the workforce.
Big data has big potential
A key success factor for companies is the availability of right information at the right time. Businesses need to know what decisions should be made, when to take action and how these decisions will impact on financial results and achieve operational goals. Demand for this type of insight, ignites the growth of big data to enable them to make better, smarter, real-time, data-driven decisions that will change the way they handle their operations and compete in the marketplace.