The UK's Leading Data & Analytics Jobs Board

Overcoming the challenges of Big Data

Big data is an innovation that businesses can use to drive competitive advantage. But the value of big data lies in the insights that your business can draw from it, rather than in the information itself.

Big data refers to the huge and increasing volume of the data available, and the ways it can be processed. Data comes in many forms, structured or unstructured, and it may be generated by organizations themselves or obtained from third parties.

Analytics is the means for extracting value from this data by generating useful insights. Without analytics, businesses have no way of using their big data to establish competitive advantage.

Issues and risks

Your business needs to overcome practical problems like the shortage of people with the technical skills to generate insights, and strategic challenges such as implementing big data adoption across the organization.

There are also serious legal and regulatory risks around issues as data privacy that can result in breach of trust. Most consumers do not want companies to share personal information, and many say that they are less willing to share personal information in the future.

For real value from big data, you need a holistic and strategic plan for identifying the opportunities, overcoming the hurdles and managing risks.


“There is certainly a lot of hype in the market today about big data and analytics. I think people should keep in mind that we tend to overestimate things in the near term … BUT underestimate them in the long term. There is no doubt in my mind that analytics is going to transform how businesses make decisions and shift where and how value gets created in most sectors.”

Christopher J. Mazzei

Businesses know they should extract value from their data. But many organizations find it difficult to overcome barriers and sector-specific issues. There is a clear reward to businesses who are able to extract maximum value from their data. Our research shows that companies that successfully use data are already outperforming their peers by as much as 20%.

Nevertheless, many organizations are frustrated with the limited progress they have made so far. They are confronted by these barriers that threaten to prevent them reaching the destinations they know are possible.

Analytics is the means for extracting value from this data by generating useful insights. Without analytics, businesses have no way of using their big data to establish competitive advantage.

The unknown destinations. Many companies understand that they possess valuable and useful data, but do not have a clear idea of what questions to ask of their data.
The underlying technology challenges. Many organizations lack the means to cope with the sheer scale of data flowing into the business. They can be intimidated by ideas such as real-time analytics, and do not know which technological tools can help them with their concerns.
The lack of a holistic approach. Too often, businesses approach big data on a project-by-project basis, and with distinct silos within the business. Some businesses are now creating specialist roles, but these are not recognized enough at the C-suite level.
The shortage of talent. Businesses need data scientists, visualization experts, business intelligence analysts, data warehousing professionals and data privacy experts. Developing these skills inhouse is difficult in the short term, but buying them in externally is expensive and simply not possible in many markets.
The fear of cyber attack. As your business’ dependence on data and analytics increases, so does your vulnerability to cyber attack, and the level of impact and damage that breaches will cause. This includes regulatory risk as well as outright business and reputation loss.
The difficulty of building the business case. While executives may accept the argument for big data initiatives in general, they want to understand the potential of specific projects before signing off on any related investments. Yet with relatively few such projects completed, it is difficult to provide such information.
The need for legal and regulatory compliance. Many organizations understand that data privacy and security will have an enormous impact on their work in big data and analytics. But if they do not address the regulatory and legal risks, it can cost them a lot of money, and their reputation.
The need for customer data. Customers appear to be less willing to share data with companies, for the fear that their data is being misused and their privacy violated. Companies must address how to use the data they have to better serve customers.

Is it possible to overcome these difficulties in order to unlock the value of your business’s data? Here are seven steps to follow as you seek to drive value.

Decide what you want to achieve with the data
Rather than starting with the data, you should come up with different hypotheses based on the decisions you need to make. Then use data and analytics to prove or disprove these hypotheses.
Determine what is relevant
Your company should focus on a relatively small amount of data to identify current and anticipated problems. With a better idea of which issues they wish to tackle, you can start to work out how to capture the data you need.
Continue driving business benefits
The success of big data and analytics endeavors will ultimately be judged on the basis of their impact on profitability: through lower costs or through higher revenues. That is why your organization needs to develop ways to measure the specific impact of your data efforts.
Prepare for the transformation across the organization
In order to extract the full value from big data, all strategic functions — CEO, CFO, CIO, CMO and CSO — need to work more closely. C-suite executives need to embrace this change, and empower senior executives and the rest of the organization to adopt it.
Generate insight from data 
This is about practical considerations like data storage, analytics tools, and the availability of talent. Create specialist roles, such as chief data officer and chief analytics officer. Consider building a center of excellence to provide a single point of focus and a platform for information exchange and the sharing of insights. This may also resolve some of the skills shortages.
Generate value from insight 
Your business needs processes and systems that will disseminate the generated insights throughout the whole organization. A data strategy driven from the center will not be successful if it does not apply to the rest of the business.
Manage risk 
The need to minimize and mitigate risk will be an ongoing challenge. Risk-planning, scenario mapping and fire-drill-type exercises will build awareness throughout the business of key risks, and an emphasis should be placed on flexibility, adaptability and responsibility.

The demanding and ever-changing legal environment represents the most important risk of all.

In addition to the strategic challenges, there are several legal challenges when working with big data, which can vary by country or region. You should address the legal issues around big data and analytics at the same time as the strategic issues. Any data strategy that fails to consider the legal issues will leave your business vulnerable to regulatory and reputational damage.

The most important legal area is data protection — particularly of personal data. This will be the source of most legal and ethical challenges.

The increasing significance of personal data. The protection of personal data is a central concern for consumers. So compliance with regulations on personal data protection is not only a way to adhere to the law, but also an effective way to convey your ethical and social commitment.
The need to define and manage personal data carefully. Big data and cloud computing can increase the risks raised by various key questions on how data is managed and defined. For example, what is the nature of the data the organization holds, where is personal data stored, how is personal data secured, do individuals still have control over their data, how can they prevent the processing of their data, and how can individuals recover their data?
The volume of data. Businesses that do not keep track of what data they hold, or keep checks on the accuracy, cannot guarantee they are complying with the law. The high volume of data held also makes it increasingly difficult for organizations to anonymize personal data to meet regulatory requirements.
The changing legislative environment. Legislators and regulators in many marketplaces are scrambling to keep up with organizations’ efforts to exploit the value of their data.
The European Commission is moving toward implementation of a data protection regulation that will ensure common standards across all member states of the EU, and apply to any organization that operates with personal data inside the bloc.
The need to protect the company’s own data. As the threat of cyber attack grows, companies’ own data may be vulnerable, leaving them open to legal and reputational risk. And while data and analytics sit at the heart of businesses’ digital innovation, the legal instruments available to protect and enhance data currently seem too limited given its rapidly increasing value.
The possibility of a big data backlash. Consumers are becoming more and more selective and careful about who they share their data with. Many are never happy for companies to share their personal information.
This might indicate that companies will have to offer an incentive for data sharing, and to consider seriously how they are gathering, and using, big data.

The US and Europe: two different approaches to data protection

The US and Europe have radically different definitions of concepts such as “protection of privacy” and “personal data.” The EU operates under a single regulatory regime. In the US, federal laws sit alongside the laws of each of the 50 states, with numerous laws to protect data.

There is also a philosophical difference. While privacy law in the US aims to achieve a balance between privacy and effective business, the EU sees respect of privacy as a citizen’s fundamental right.

One crucial difference is the emphasis placed by US legislation on the protection of data security — especially the obligation to declare breaches of security. In Europe, no such obligation currently exists, though the commission proposes to introduce something similar in its forthcoming overhaul of EU data privacy legislation.

How do organizations begin to grapple with these legal risks? From our work with businesses to address legal and strategic issues simultaneously, we have developed five key principles to address the demands that big data and analytics are now making on organizations.

Confront the legal issues on an enterprise-wide basis
Legal issues must be addressed by the business as a whole, rather than through silos. This also offers an opportunity to bring together other policies and solutions, so your company can create a single, coherent strategy for the business.
Identify legally questionable practices
The lack of good information management practices can cause your organization to unwittingly hold legally questionable data. This is a good example of where the legal and strategic responses to the challenges of big data go hand-in-hand: excessive, low-quality data represents an unnecessary legal risk, as well as a pointless cost.
Build a new structure for handling the legal issues 
A new discipline requires new types of leaders to confront the challenges it presents. New roles such as chief privacy officer and chief information strategy officer will need the same credibility and authority as chief risk officers currently have.
Create a virtue of data privacy 
Commercial organizations increasingly recognize that there is value to be gained from having a reputation for good practice with data. Confronting the legal and regulatory challenges presents an internal opportunity for your organization to embed a culture of data awareness at every level of the business.
Consider the customer in all legal decisions 
Your company should always think about what benefits you can generate for your customers and for the public in general. If companies focus their big data efforts on things that solely benefit themselves, they risk damaging the social contract that allows them access to such data.

Protection of personal data generates trust and business opportunities, and any ignorance of legal issues leads to financial, legal and reputational risks.

Copyright © 2019 Big Data Jobs Board. SEO by SEMtelligence