Data has long been the currency on which the enterprise operates – and it goes right to the very top. Analysts and thought leaders almost universally urge the importance of the CEO being actively involved in data initiatives. But what gets buried in the small print is the acknowledgement that many data projects never make it to production. In 2016, Gartner assessed it at only 15%.

The operationalisation of data projects has been a key factor in helping organisations turn a data deluge into a workable digital transformation strategy, and DataOps carries on from where DevOps started. But there is a further Gartner warning: organisations who lack a sustainable data and analytics operationalisation framework by 2024 will see their initiatives set back by up to two years.

Operationalisation needs good orchestration to make it work, as Basil Faruqui, director of solutions marketing at BMC, explains. “If you think about building a data pipeline, whether you’re doing a simple BI project or a complex AI or machine learning project, you’ve got data ingestion, data storage and processing, and data insight – and underneath all of those four stages, there’s a variety of different technologies being used,” explains Faruqui. “And everybody agrees that in production, this should be automated.”

This is where Control-M from BMC, and in particular BMC Helix Control-M comes in. Control-M has been an integral part of many organisations for upwards of three decades, enabling businesses to run hundreds of thousands of batch jobs daily and help optimise complex operations such as supply chain management. But an increasingly complex technological landscape, across on-premises to cloud, as well as a greater usage of SaaS-based orchestration alongside consumption, made it a no-brainer to launch BMC Helix Control-M in 2020.

“CRMs and ERPs had been going the SaaS route for a while, but we started seeing more demands from the operations world for SaaS consumption models,” explains Faruqui.

The upshot of being a mature company – BMC was founded in 1980 – is that many customers have simply extended Control-M into more modern use cases. One example of a large organisation – and long-standing BMC customer – running an extremely complex supply chain is food manufacturer Hershey’s.

Apart from the time-sensitive necessity of running a business with perishable, delicate goods, the company has significantly adopted Azure, moving some existing ETL applications to the cloud, while Hershey’s operations are built on a complex SAP environment. Amid this infrastructure Control-M, in the words of Hershey’s analyst Todd Lightner, ‘literally runs our business.’

Faruqui returns to the stages of data ingestion, storage, processing, and insight to explain how Hershey’s would tackle a significant holiday campaign, or decide where to ship product. “It’s all data driven,” Faruqui explains. “They’re ingesting data from lots of systems of record, that are ingesting data from outside of the company; they’re pulling all that into massive data lakes where they’re running AI and ML algorithms to figure out a lot of these outcomes, and feeding into the analytics layer where business executives can look at dashboards and reports to make important decisions.

“They’re a really good example of somebody who has used orchestration and automation with Control-M as a strategic option for them,” adds Faruqui.

Yet this leads into another important point. DataOps is an important part of BMC’s strategy, but it is not the only part. “Data pipelines are dependent on a layer of applications both above and below them,” says Faruqui. “If you think about Hershey’s, trying to figure out what kind of promotion they should run, a lot of that data may be coming from SAP. And SAP is not a static system; it’s a system that is constantly being updated with workflows.

“So how does the data pipeline know that SAP is actually done and the data is ready for the data pipeline to start? And when they figure out the strategy, all that information needs to go back to SAP because the ordering of raw materials and everything is not going to happen in the data pipeline, it’s going to happen in ERPs,” adds Faruqui.

“So Control-M is able to connect across this layer, which is different from many of the tools that exist in the DataOps space.”

Faruqui is speaking at the AI & Big Data Expo Europe in Amsterdam in September around how orchestration and operationalisation is the next step in organisations’ DataOps journeys. So expect not only stories and best practices on what a successful journey looks like, and how to create data pipeline orchestration across hybrid environments combining multiple clouds with on-prem, but also a look at the future – and according to Faruqui, the complexity is only going one way.

“I think one of the things that will continue to be challenging is there’s just lots of different tools and capabilities that are coming up in the AI and ML space,” he explains. “If you look at AWS, Azure, Google, and you go to their website, and you click on their AI/ML offerings, it is quite extensive, and every event they do, they announce new capabilities and services. So that’s on the vendor side.

“On the customer side, what we’re seeing is they want to rapidly test and figure out which [tools] are going to be of use to them,” Faruqui adds. “So as an orchestration vendor, and orchestration in general within DataOps, this is both the challenge and the opportunity.

“The challenge is you’re going to have to keep up with this because orchestration doesn’t work if you can’t integrate into something new – but the opportunity here is that our customers are asking for this.

“They don’t want to have to reinvent the orchestration wheel every time they go and adopt new technology.”