Today, only one in three AI projects are succeeding, and, perhaps more importantly, it is taking businesses more than six months to go from concept to production, reveals new research from unified analytics firm Databricks.
The primary reasons behind these challenges are that 96 percent of organizations face data-related problems like silos and inconsistent datasets, and 80 percent cite significant organizational friction like lack of collaboration between data scientists and data engineers. IT executives point to unified analytics as a solution for these challenges with 90 percent of respondents saying the approach of unifying data science and data engineering across the machine learning lifecycle will conquer the AI dilemma.
“Data is the fuel that powers AI. Large amounts of reliable data that data scientists can iterate on is the key to AI success. But organizational silos between data science and engineering cripples the iterative model development process. And to make matters worse the divide between today’s data and AI technologies increases complexity throughout the lifecycle further slowing down AI,” said Bharath Gowda, vice president of product marketing at Databricks, in a news release. “Unified analytics addresses this AI dilemma by providing an end-to-end analytics platform that unifies big data and AI, while fostering better collaboration between data science and engineering teams.”
The survey, commissioned by Databricks through IDG’s CIO Research Services, surveyed 200 IT executives at larger companies (1, 000+ employees) across the U.S. and Europe. The results speak to the complexity and organizational confusion being creating as companies pursue AI initiatives:
- 98 percent of those surveyed believe preparation and aggregation of large datasets in a timely fashion is a major challenge;
- 96 percent of respondents found data exploration and iterative model training challenging;
- 90 percent cited the deployment of models to production quickly and reliably as a significant challenge
- 87 percent of organizations invest in an average of seven different machine learning tools, adding to the organizational complexity
So, what will help these organizations conquer the AI dilemma? The surveyed executives said they need end-to-end solutions that combine data processing with machine learning capabilities. These streamlined solutions would simplify workflows, improve efficiency and ultimately accelerate business value.
In fact, nearly 80 percent of executives surveyed said they highly valued the notion of a unified analytics platform. Unified analytics makes AI more achievable for enterprise organizations by unifying data processing and AI technologies. Unified analytics solutions provide collaboration capabilities for data scientists and data engineers to work effectively across the entire AI development-to-production lifecycle.
With more than 90 percent of large companies facing data-related challenges and increasing complexity driven by an explosion of machine learning tools, the need for platforms and processes that can remove technology and organizational silos is more pronounced than ever.
