SANTA CLARA, Calif., Sept. 11, 2018 (GLOBE NEWSWIRE) -- Qubole, the data activation company, today announced the results of its 2018 Big Data Trends and Challenges report — an annual, global survey of IT and data professionals on the trends, challenges, and solutions of big data and machine learning initiatives across the enterprise. The survey found that the value achieved from big data in on-premises environments has significantly lagged behind expectations, and as a result, companies are rapidly shifting to the cloud for applications centered around machine learning (ML), and analytics.
According to the survey, 73 percent of businesses are now performing their big data processing in the cloud, up from 58 percent in 2017. The shift towards cloud is necessitated in part due to the ever-growing volume and diversity of data that companies are dealing with, as 44 percent of organizations now report working with massive data lakes over 100 terabytes in size.
Facilitated by this shift to the cloud, machine learning programs are expected to expand across a wide range of use cases in the next year. A majority of respondents cited improving data security and threat protection as the top priority for their machine learning initiatives while optimizing customer experience (49 percent) and predictive maintenance (43 percent) were also high on the list of ML priorities.
Apache Spark and Presto have also shown impressive gains among big data frameworks in the last year. 31 percent of respondents now report using Spark as their framework, with a 29 percent growth from 2017. Presto, while in use by a smaller 13 percent of companies, saw its user base grow 63 percent in the last year. The survey results also indicated a shift by organizations away from homegrown approaches in favor of open source technologies.
“The size, diversity, and applications of big data are accelerating at a near-exponential rate, and businesses are quickly discovering that traditional data management systems and strategies are no longer capable of supporting their demands,” said Ashish Thusoo, co-founder and CEO, Qubole. “Instead, a new generation of cloud-native, self-service platforms have become essential to the success of data programs, especially as companies look to expand their operations with new AI, machine learning and analytics initiatives.”
Additional findings from the report include:
Data teams continue to face a number of challenges when implementing both big data and machine learning projects:
- Respondents cited a lack of experience slowing project progress (44 percent), struggling to keep up with new data sources (42 percent) and issues with constantly evolving use cases (41 percent) as their top challenges.
- On the machine learning side, analyzing extremely large data sets (40 percent), ensuring adequate staffing and resources (38 percent) and integrating new data into existing pipelines (38 percent) were cited as the primary obstacles to machine learning projects.
Talent shortages are also a major issue for businesses:
- 79 percent of companies are looking to increase their data team headcount in the next year, but 83 percent also say it is difficult to find data professionals with the right skills and experience.
- 75 percent of respondents also reported that a sizeable gap exists between the potential value of the big data available to them, and dedicated tools and talent dedicated to delivering it.
A majority of businesses expressed plans for moving to a self-service analytics model, but few currently have it in place:
- 9 percent of respondents reported that their business already supports self-service analytics.
- 61 percent reported that they have plans to move to a self-service analytics model.
To view the complete survey findings, the 2018 Big Data Trends and Challenges report is available for download on the Qubole website here.
Research Methodology
The research report, commissioned by Qubole and conducted by independent research agency Dimensional Research, polled 401 IT and data professionals with big data responsibilities globally. The survey was administered electronically and participants were offered a token compensation for their participation.
About Qubole
Qubole is revolutionizing the way companies activate their data--the process of putting data into active use across their organizations. With Qubole's cloud-native Data Platform for analytics and machine learning, companies exponentially activate petabytes of data faster, for everyone and any use case, while continuously lowering costs. Qubole overcomes the challenges of expanding users, use cases, and variety and volume of data while constrained by limited budgets and a global shortage of big data skills. Qubole's intelligent automation and self-service supercharge productivity, while workload-aware auto-scaling and real-time spot buying drive down compute costs dramatically. Qubole offers the only platform that delivers freedom of choice, eliminating legacy lock in--use any engine, any tool, and any cloud to match your company's needs. Qubole investors include CRV, Harmony Partners, IVP, Lightspeed Venture Partners, Norwest Venture Partners, and Singtel Innov8. For more information visit us online.
About Dimensional Research
Dimensional Research provides practical market research to help technology companies make their customers more successful. Our researchers are experts in the people, processes, and technology of corporate IT and understand how technology organizations operate. We partner with our clients to deliver actionable information that reduces risks, increases customer satisfaction, and grows the business. For more information visit www.dimensionalresearch.com.
Qubole Media Contact:
Qubole
Ed Cruz
415-432-2400
qubole@teamlewis.com