This initiative will help meet increasing demands for data and analytics support

South Australia’s Department for Education has been actively using a wealth of operational and education-related data for more than a decade. Its System Performance division – which houses the department’s data and analytics capability – was established in 2012 with the aim of better leveraging the department’s considerable data assets to improve learning and development outcomes for young South Australians.

“At the beginning, it was just three people, and we now have over 90 full-time employees working in our System Performance division,” says the department’s Chief Operating Officer, Ben Temperly. “And we now provide a mature set of data products through Microsoft Power BI to our educators working in schools.

“A consequence of prioritising children and student data and the provision of information to schools is our provision and use of data across other areas of the department’s operations is much less mature. To be able to operate at a truly enterprise level, changes were needed in our operating model and technology.”

In addition, a lot of the department’s data had become siloed, making it difficult for its 30,000 employees to access and generate insights from. So, the department partnered with local data analytics and consulting company Exposé to establish and refine its technology, change management and data governance requirements.

“That was our initial framing of the business need – to have that single repository of data,” says Temperly. “But it was through engagement with experts like Exposé that we broadened our vision to start thinking about the full value chain of the data, as opposed to focusing on the storage of data.”

A multi-phase data transformation

The department and Exposé began a multi-phase data transformation project in March 2022 to build the edAnalytics Hub – a world-class data platform that aims to improve educational and development outcomes for young South Australians using Microsoft’s advanced data analytics, machine learning and cloud technologies.

This initiative will help the department meet the increasing demand for data and analytics support by enabling faster and more direct access to accurate information from multiple data sets and data sources.

Temperly says the first phase of the program is focused on deploying technology to replace its legacy systems, progressively migrating massive volumes of data into the new cloud-based environment, and then building data products in Power BI for employees to use. This phase is expected to be completed in mid-2023.

“We’re focused on three main domains of data in the first phase – student data, finance data, and people and culture data – but the department’s data holdings extend well beyond that,” he explains.

Aside from using Power BI for data visualisation and analysis, the edAnalytics Hub also leverages several Microsoft Azure services for data management and machine learning. These include Azure Databricks, Azure Data Factory, Azure Data Lake, Azure Data Share, Azure DevOps, Azure Synapse Analytics and Microsoft Purview.

The Azure services are configured in a ‘lakehouse’ architecture, which combines the cost-efficiency, scale and flexibility features of a data lake with the data and transaction management capabilities of a data warehouse.

Exposé co-founder and Chief Technology Officer Etienne Oosthuysen says the department and his team have adopted an agile approach for the entire program of work, which has enabled continuous improvement, delivery, and integration.

“Throughout that whole process and through the ceremonies in the agile space, users are continuously seeing the iterative status of what is coming their way,” he says. “Feedback is refitted into the sprints so that we can adjust things accordingly.”

Breaking new data governance ground in the cloud

Oosthuysen says the edAnalytics Hub also required a technology architecture to access data managed by South Australia’s Department of the Premier and Cabinet (DPC).

“There’s quite a unique aspect here in South Australia where DPC is actually the custodian of some of the data sources – you can’t just reach into the DPC domain and grab data,” he says. “So, we had to come up with an architecture aligned to the Conditions of Connection, which was approved by DPC and is now used by other agencies as well. And that makes use of Azure Data Share to bring data across the boundary.”

According to Oosthuysen, the edAnalytics Hub is breaking a lot of other new data governance ground using Purview.

“It’s not just the Power BI and machine learning aspects that are key [to this project],” he explains. “It’s the governance aspect as well and changing the processes that incorporate all of the edAnalytics Hub, which involves data workers from a very wide set of areas.

“Currently, it’s focused on the more central corporate teams [in the department], but that may eventually be expanded. So, something like Purview is going to be very valuable when that matures.”

Accelerating the data maturity journey

The department is on the cusp of releasing its first new data product – a people and culture dashboard – via the edAnalytics Hub, before launching new student and finance data products in 2023.

While the data transformation project is in the early stages, Temperly says it’s already prompted important and constructive conversations about how the department uses and manages data.

“It’s been quite a pleasant surprise, the extent to which other senior executive colleagues have looked to engage in those conversations around data governance and questions around data ownership,” he says.

“We’re seeing high expectations for what the project’s going to deliver across the department. And it’s emphasised what we already knew, which was the demand for usable data in a format that helps people with their day-to-day work.

“This program will work, and our partnership with Exposé is helping us to accelerate our journey across the data maturity curve. We’re going to get a level of progress over the next two or three years that we wouldn’t have otherwise reached.”

All about the people

Training is a crucial element of the project’s change management pillar. The department has focused on helping its System Performance division and its broader information and communications technology team to become familiar with the edAnalytics Hub.

The department is also looking to introduce self-service capabilities for authorised data users. Temperly hopes this will allow its System Performance team to focus on more sophisticated data analysis and value-adding tasks, rather than mundane data activities such as responding to ad hoc queries for basic information.

“In time, we’ll have to run a training program across all parts of the department on how to use new technology, how to access the data they need and, really importantly, how they can start to build some of their own data products through the use of Power BI and so on,” he says.

Oosthuysen adds: “Although this is a data-disrupting program of work, make no mistake, it really was all about people and less so about technology, whether that is the existing teams being brought along the journey or those potential future data workers.

“There is an opportunity to empower many kinds of personas in the organisation to really do better with the data that we will be exposing through the edAnalytics Hub.”

Tags: ExpośeMicrosoftSouth Australia’s Department for Education