Why data catalogs are the standard for data intelligence

Why data catalogs are the standard for data intelligence

How data catalogs can help companies find the right foundation for data intelligence while increasing employee productivity and bottom line.

Data culture. Data Literacy. Data intelligence. All of these things are deemed essential by the market. It’s easy to fall into the trap of feeling intimidated, falling behind, or not knowing where to start. If this happens to you, don’t worry. The truth is, many feel exactly the same and are trying to figure out how to improve with data.

In my role as director of product and cloud marketing, I speak with large companies that are making a real effort to foster a culture of data and use their data in a smarter way. Many admit that they are just getting started or are at the beginning of their journey. Of course, every organization is at a different stage and developing a data culture doesn’t happen overnight. The question therefore becomes: “How do we establish the right foundations for data intelligence?”

Data catalogs are the answer.

Gartner positions a data catalog as the foundation “for accessing all types of metadata and representing it in a connected knowledge graph”. To illustrate, I’ll share a personal experience on why I think a data catalog is crucial for data intelligence. A few years ago, when I was working at a large global technology company, my manager said to me, “I want you to find out what metrics we should be measuring and tell us if our product is making our customers successful. We don’t have the data or the analytics today. I was surprised. How could that be? How can a successful company not have the data model in place to measure a leading product on the market? Did they base their decisions on instinct?

As part of my work, I had to create hypotheses, collect data, analyze it and create a recommendation. To get started, I needed to find an expert who had a significant amount of tribal knowledge and could explain what data existed, where it was, what it meant, how I should use it, and what pitfalls I might encounter using it. Then I had to fetch the data from the data warehouse and write lots of SQL queries, while finding the data scientists for help.

The worst part? Whenever questions about the validity, accuracy, and reliability of data arose, I had to trust what people were telling me or rely on my own judgement. After months of working on the project, it turned out to be a worthwhile exercise that yielded insights.

Fast forward to today. After experimenting with the capabilities of a proper data management solution, this whole experience would have been much better with a data catalog. Why? Suppose the organization had a rich catalog of data. I could have researched and discovered the data myself, understood the context of the data, identified subject matter experts and asked them questions in the data catalog, written SQL on multiple data sources with active guidance on data to use and not to use, repurposed existing SQL queries so you don’t have to write them from scratch, and collaborated with data scientists and others to analyze data in one place. I would have saved a lot of time and effort. With this time saved, I could have worked on something new and even more valuable for the company.

How does this explain why a data catalog is the foundation of data intelligence? It’s simple. If you extrapolate my unique experience to anyone who needs or wants to use data for their work, a data catalog makes data smarter and more useful for an array of solutions, from enabling self-service to analytics data governance or migrating data to the cloud. Think of my time spent on the aforementioned project multiplied by thousands of users. This represents significant productivity gains.

Data producers can connect data sources and automatically scan data into the catalog. Indexed data can then be organized and governed to comply with data policies. DataOps can develop high-quality data products that enable data consumers to find, understand, and use data to make faster data-driven decisions that can lead to increased revenue or competitive advantage.

Research from the latest Forrester report shows that data catalogs provide a foundation for organizations to create a data culture. Technology is driving the use of data in organizations and improving data literacy. Organizations that have successfully implemented data catalogs have seen positive changes in the speed and quality of data analysis and in the engagement of the teams that need to perform the data analysis.

Managing data in the era of big data, data lakes, and self-service is a challenge, and data catalogs are the solution. Data catalogs have gone from a “nice to have” tool to a “must have” that organizations expect in their arsenal. Without a powerful data catalog, the status quo is impossible to maintain.