Build a Data Integration Strategy

Integrate your data or disintegrate your business.

ANALYST PERSPECTIVE

Integrate your data or disintegrate your business.

"Point-to-point integration is an evil that builds up overtime due to ongoing business changes and a lack of integration strategy. At the same time most businesses are demanding consistent, timely, and high-quality data to fuel business processes and decision making.

A good recipe for successful data integration is to discover the common data elements to share across the business by establishing an integration platform and a canonical data model.

Place yourself in one of our use cases and see how you fit into a common framework to simplify your problem and build a data-centric integration environment to eliminate your data silos."

Rajesh Parab, Director, Research & Advisory Services

Info-Tech Research Group

Our understanding of the problem

This Research Is Designed For:

• Data engineers feeling the pains of poor integration from inaccuracies and inefficiencies during the data integration lifecycle.
• Business analysts communicating the need for improved integration of data.
• Data architects looking to design and facilitate improvements in the holistic data environment.
• Data architects putting high-level architectural design changes into action.

This Research Will Also Assist:

• CIOs concerned with the costs, benefits, and the overall structure of their organization’s data flow.
• Enterprise architects trying to understand how improved integration will affect overall organizational architecture.

This Research Will Help You:

• Understand what integration is, and how it fits into your organization.
• Identify opportunities for leveraging improved integration for data-driven insights.
• Design a loosely coupled integration architecture that is flexible to changing needs.
• Determine the needs of the business for integration and design solutions for the gaps that fit the requirements.

This Research Will Help Them:

• Get a handle on the current data situation and how data interacts within the organization.
• Understand how data architecture affects operations within the enterprise.

Executive summary

Situation

• As organizations process more information at faster rates, there is increased pressure for faster and more efficient data integration.
• Data integration is becoming more and more critical for downstream functions of data management and for business operations to be successful. Poor integration holds back these critical functions.

Complication

• Investments in integration can be a tough sell for the business, and it is difficult to get support for integration as a standalone project.
• Evolving business models and uses of data are growing rapidly at rates that often exceed the investment in data management and integration tools. As a result, there is often a gap between data availability and the business’ latency demands.

Resolution

• Create a data-centric integration solution that supports the flow of data through the organization and meets the organization’s requirements for data accuracy, relevance, availability, and timeliness.
• Build your data-centric integration practice with a firm foundation in governance and reference architecture; use best-fit reference architecture patterns and the related technology and resources to ensure that your process is scalable and sustainable.
• The business’ uses of data are constantly changing and evolving, and as a result the integration processes that ensure data availability must be frequently reviewed and repositioned to continue to grow with the business.

Info-Tech Insight

1. Every IT project requires data integration.Any change in the application and database ecosystem requires you to solve a data integration problem.
2. Integration problem solving needs to start with business activity. After understanding the business activity, move to application and system integration to drive optimal data integration activities.
3. Integration initiatives need to be backed by requirements that depend on use cases. Info-Tech’s use cases will help identify organizational requirements and the ideal data-centric integration solution.

Your data is the foundation of your organization’s knowledge and ability to make decisions

Integrate the Data, Not the Applications

Data is one of the most important assets in a modern organization. Contained within an organization’s data are the customers, the products, and the operational details that make an organization function. Every organization has data, and this data might serve the needs of the business today.

However, the only constant in the world is change. Changes in addresses, amounts, product details, partners, and more occur at a rapid rate. If your data is isolated, it will quickly become stale. Getting up-to-date data to the right place at the right time is where data-centric integration comes in.

"Data is the new oil." – Clive Humby, Chief Data Scientist Source: Medium, 2016

Organizations are having trouble keeping up with the rapid increases in data growth and complexity

To keep up with increasing business demands and profitability targets and decreasing cost targets, organizations are processing and exchanging more data than ever before.

To get more value from their information, organizations are relying on more and more complex data sources. These diverse data sources have to be properly integrated to unlock the full potential of your data:

The most difficult integration problems are caused by semantic heterogeneity (Database Research Technology Group, n.d.).

80% of business decisions are made using unstructured data (Concept Searching, 2015).

85% of businesses are struggling to implement the correct integration solution to accurately interpret their data (KPMG, 2014).

Break Down Your Silos

Integrating large volumes of data from the many varied sources in an organization has incredible potential to yield insights, but many organizations struggle with creating the right structure for that blending to take place, and data silos form.

Data-centric integration capabilities can break down organizational silos. Once data silos are removed and all the information that is relevant to a given problem is available, problems with operational and transactional efficiencies can be solved, and value from business intelligence (BI) and analytics can be fully realized.

Data-centric integration is the solution you need to bring data together to break down data silos

On one hand…

Data has massive potential to bring insight to an organization when combined and analyzed in creative ways.

On the other hand…

It is difficult to bring data together from different sources to generate insights and prevent stale data.

How can these two ideas be reconciled?

Answer: Info-Tech’s Data Integration Onion Framework summarizes an organization’s data environment at a conceptual level, and is used to design a common data-centric integration environment.

Info-Tech’s Data Integration Onion Framework

Poor integration will lead to problems felt by the business and IT

The following are pains reported by the business due to poor integration:

59% Of managers said they experience missing data every day due to poor distribution results in data sets that are valuable to their central work functions. (Experian, 2016)

42% Reported accidentally using the wrong information, at least once a week. (Computerworld, 2017)

37% Of the 85% of companies trying to be more data driven, only 37% achieved their goal. (Information Age, 2019)

"I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts." – Sir Arthur Conan Doyle, Sherlock Holmes

Poor integration can make IT less efficient as well:

90% Of all company generated data is “dark.” Getting value out of dark data is not difficult or costly. (Deloitte Insights, 2017)

5% As data sits in a database, up to 5% of customer data changes per month. (Data.com, 2016)

"Most traditional machine learning techniques are not inherently efficient or scalable enough to handle the data. Machine learning needs to reinvent itself for big data processing primarily in pre-processing of data." – J. Qiu et al., ‎2016

Understand the common challenges of integration to avoid the pains

There are three types of challenges that organizations face when integrating data:

1. Disconnect from the business

Poor understanding of the integration problem and requirements lead to integrations being built that are not effective for quality data.

50% of project rework is attributable to problems with requirements. (Info-Tech Research Group)

45% of IT professionals admit to being “fuzzy” about the details of a project’s business objectives. (Blueprint Software Systems Inc., 2012)

2. Lack of strategy

90% Of organizations will lack an integration strategy through to 2018. (Virtual Logistics, 2017)

Integrating data without a long-term plan is a recipe for point-to-point integration spaghettification:

3. Data complexity

Data architects and other data professionals are increasingly expected to be able to connect data using whatever interface is provided, at any volume, and in any format – all without affecting the quality of the data.

36% Of developers report problems integrating data due to different standards interpretations. (DZone, 2015)

These challenges lead to organizations building a data architecture and integration environment that is tightly coupled.

A loose coupling integration strategy helps mitigate the challenges and realize the benefits of well-connected data

Loose Coupling

Most organizations don’t have the foresight to design their architecture correctly the first time. In a perfect world, organizations would design their application and data architecture to be scalable, modular, and format-neutral – like building blocks.

Benefits of a loosely coupled architecture:

• Increased ability to support business needs by adapting easily to changes.
• Added ability to incorporate new vendors and new technology due to increased flexibility.
• Potential for automated, real-time integration.
• Elimination of re-keying/manual entry of data.
• Federation of data.

Vs. Tight Coupling

However, this is rarely the case. Most architectures are more like a brick wall – permanent, hard to add to and subtract from, and susceptible to weathering.

Problems with a tightly coupled architecture:

• Delays in combining data for analysis.
• Manual/Suboptimal DI in the face of changing business needs.
• Lack of federation.
• Lack of flexibility.
• Fragility of integrated platforms.
• Limited ability to explore new functionalities.