Data analytics

We pride ourselves on our history of delivering state of the art analytics across a range of domains, from social policy across to national security issues. We harness the power of SQL (including Hive SQL ), Python / R (including Pyspark for big data) and cloud based technologies to deliver enterprise grade analytics. Some of the problems where we have delivered immense value using analytics include:

• Data wrangling / engineering
  • Leading the development of rich datasets by combining together disparate and messy data from multiple big data systems from social security, tax and health systems, to answer the most pressing policy questions.
  • Development of a backend data engine to a data app to crunch through millions of data points and automate batch processing of analytics to serve hundreds of concurrent users.
• Predictive analytics
  • Developing predictive models to detect various threats and risks to a large administrative program, automating large parts of manual assessments.
• Natural language processing
  • Using state of the art embedding based deep learning models for entity classification, and production grade named entity recognition systems.
• Network analysis
  • Using graph databases and network modelling to uncover new relationships and contagion impacts.
• Data visualisation
  • Developing award winning geospatial visualisations using Business Intelligence tools (Qlik, PowerBI and R Shiny).

Data management

While it doesn’t take the limelight in data projects like data analytics, successful data projects implement good data management practices. Successful data governance need not come at the cost of lost productivity. Our bottom approach to strategy means we use the right tools and frameworks to ensure good data management practices become second nature to any analytics project, some the ways we achieve this include:

• Tools we have experience in for better data management:
  • Azure Devops / Atlassian Confluence and Bitbucket – to record specialist knowledge and know-how through wiki pages, reproducible code through git code repositories, and task boards definitions.
  • Azure Data Catalog/Purview – to simplify data cataloguing, glossary, schema management and data lineage.
  • Qlik / PowerBI – to automatically sync shared excel workbooks / knowledge repositories/Abbreviation Data Warehouses and present data asset registers in a discoverable way.
• Frameworks
  • DAMA-DMBOK2 – defines key knowledge areas that allow us to assess organisational capabilities, uplifts required, and responsible entities. These include data security, data quality management and data storage.
  • Five Safes – essential in large data linkage projects especially with linkage units such as ABS, to balance privacy disclosure risk with security. We have assisted in steps to meet organisational privacy obligations and requirements.
  • Agile/Scrum – We use agile methodology to prioritise backlog and user stories, so that the most important features are developed first

Data architecture

In an environment where there are an abundance of tools and implementation options, its easy to get lost and confused. We make it easy to support you choosing the right infrastructure for your needs. Some areas where particularly have experience include:

• Analytics platforms market analysis
  • Perform market analysis of similar platforms by comparing the desired analytical capabilities, skills sets, processes and administrative burden with vendor offerings.
  • Develop rapid prototypes and proof of concepts solutions comparing implementation of big data solutions across various product offerings.
• Cloud analytics platform implementation
  • Lead business design documents to implement analytics platform to migrate on-premise transactional data to highly available data lake storage (using ADLS Gen2).
  • Process, analyse and curate datasets using Python, R or Spark notebooks using Azure Databricks and data copy or incremental load activities using Azure Data Factory.
  • Implement a medallion architecture to process data from unstructured (bronze), to business ready data stores (silver), and consumable project data (gold) for consumption from apps and APIs.

We have the most experience with Azure technology stack, which is highly prevalent in the public sector due to seamless integrations with Office 365, and the ability for it to provide end-to-end analytical capabilities. We also are highly experienced in RStudio offerings, which also offer a compelling technology stack for clients who prefer keeping deployment skills in-house and powered by their data scientists or analysts.