The Biggest Challenges IT Teams Face On Data Integration Initiatives

IT organizations don’t want to have to invest in and learn a bunch of different integration technologies to support different use cases

IT developers are rarely starting from scratch when it comes to integration. Instead, they’re typically building on, replacing, or complementing integrations that are already in place, writes Mike Kiersey, Principal Technologist at Boomi.

Too often, these integrations are quick-and-dirty coding projects that were written to meet an immediate need without long-term goals such as reliability or scalability. In many organizations, most of the IT budget ends up going to maintaining old code and old systems, and most integration work is spent maintaining these brittle, hand-coded integrations. Old systems and technical debt are costly. In an IDG survey, 64% of organizations said legacy systems and processes were among their top obstacles to digital transformation.

Whenever the occasion arises, IT teams should replace these old, brittle integrations with new integrations built on a modern iPaaS (integration platform as a service) platform, ensuring stability, scalability, and flexibility. If the iPaaS platform features a low-code development interface, the new integration can probably be built in mere days or even hours, so that old, error-prone code can be replaced with new code quickly and easily.

Accelerate connectivity: Build integrations faster and easier with low-code development.

In any organization, there’s going to be some integration work that requires hand-coding and custom work. But the vast majority of integration work doesn’t. Most data integrations and data transformations can be built with a low-code development environment with a drag-and-drop graphical interface and ready-to-use connectors for popular business applications and data technologies. Using low-code development accelerates integration work by 5-10X. Business users quickly get the data connections they need, and the IT department frees itself to tackle the really hard, custom, and organization specific work that requires developers’ time.

Stop re-inventing the wheel: Use an Integration Center of Excellence to establish standards and build tested, proven reusable components

You have two development teams hard at work building integrations. When they’re done, you discover that each of them has recreated over half of the functionality of the other, but using custom code. Without central coordination, it’s too easy for a well-intentioned development team to spend long hours creating code that another team has already written, tested, and debugged.

The best solution? Establish an Integration Center of Excellence (ICoE) to coordinate and manage integration work across the organization. The ICoE is a cross-departmental team that ensures that all stakeholders’ needs are met when an integration is built. For example, the ICoE ensures that integrations comply with the organization’s security standards and guidelines. The ICoE can also run development projects to build a common foundation of code for future integrations. We find that over time these re-usable components can cover 80% of the work in future integrations, substantially saving time and money while improving code quality and reliability.

Don’t overlook data quality: Ensure your data integrations deliver complete and accurate data

Too often integrations are built without examining the quality of the data being delivered. Users are left wondering, “Where did this data come from? How do I know it’s accurate?” They might even ask, “Why is this different that the data my colleagues are getting from another application?” When moving data from one application to another, it’s critical that the data is accurate and complete. Otherwise, the new integration is simply perpetuating an old error.

To solve this problem, IT organizations should build data governance into data integrations. This is now easier than ever before, since data governance is now available with some iPaaS (integration platform as a service) platforms. The same platform can manage data connections and ensure that key data types and data records are accurate and consistent across the organization.

Start small: tackle the integrations being worked on currently, rather than trying to institute data governance across the organization all at once. Once you’ve built data governance into your new integrations, review your data standards every few months. Why? Because in most organizations, data is going to change, and you’ll need to keep your data governance and data integration work in sync.

Stay flexible: Build and run integrations wherever they’re needed

Most enterprises today keep some data on-premise and some in the cloud. For security reasons, they might keep some data on-premise, never allowing it to pass through the corporate firewall. For efficiency reasons, they might want their cloud applications to communicate directly without making a detour to any processes running on-premise. They might want some integrations to work in a multi-cloud environment, straddling cloud platforms and vendors. And they might want other integrations to run in a very small footprint at the network edge, supporting Internet of Things (IoT) devices.

IT organizations don’t want to have to invest in and learn a bunch of different integration technologies to support all these different use cases. They want a single integration platform that will let them build whatever integrations they need and then run them wherever they belong: in the cloud, on-premise, in a multi-cloud environment, or at the network edge.

IT organizations should make sure they have a data integration platform that provides this flexibility. IT is becoming only more diverse and distributed. It makes sense to have a global integration technology that can handle every use case, enabling integration processes to run quickly and efficiently, supporting any use case business units request.

Security and control: setting up parent-child relationships for integrations

Integrations handle something that’s vital to any organization: its data. In this age of stealthy data breaches and increased regulatory scrutiny, it’s critical that IT organization keep tight control over the integrations they build, ensuring that only authorized users can run them, access them, and configure them.

Look for an integration platform that supports role-based access controls and parent-child privileges, so that permissions for integrations can be precisely assigned to divisions, departments and individual users. One bonus of this precision and control: your IT organization will be able to track integration usage and report charge-back data to the Finance team. It turns out that assigning parent-child access privileges improve both data security and the accuracy of internal accounting.