Data Warehouse Modernization Initiative

The Data Warehouse was initially created and is still known to this day as the Decision Support Database (DSD).

The Decision Support Database was created to generate and maintain the critical data sets required to support Institutional Research and reporting.  

• Production of consistent, timely data becomes limited from out-of-date and non-upgradable technology. 
• Loss of institutional knowledge over time regarding data definitions or changes in how data was being entered or how data is used.
• Hand-coded warehouse management methods became unsustainable and complex.

Executive Leadership committed to long-term support for the creation of a distinct Data Warehouse team, managed by a shared services governance model.

• Permanent staffing was increased from 1 to 2 FTE via broad system office reorganization.
• The Institutional Research Shared Services Committee (IRSSC) was formed, in which the 4 IR Directors equally participate in guiding and supporting the Data Warehouse functions.  
• High-level goals were drafted:
  • Continue to maintain the frozen data sets required by the four Institutional Research offices to generate reports and insights used in decision-making.
  • Optimize resources by increasing common data sets, increasing automations, and centralizing the warehousing tasks common to all four Institutional Research offices.
  • Promote consistency by upholding common, official, custom data definitions and applying data quality standards.

IRSSC and the new Data Warehousing team began gaps analysis, examining our assets and needs compared to industry best practices.

• Industry trends in data warehouse methods and technology have seen dramatic improvements, offering significantly more advanced capabilities than the current system can support.
• Demand for data in standardized, refreshed, and ad-hoc formats exceeds current system capacity creating a strain on UA’s reporting resources both in technology and personnel.
• Authority and stewardship for data unclear, lack of policy and procedures.
• Increase in the number of data systems in use across UA equally increases the complexity of our data ecosystem and introduces significant challenges to orchestrating vast amounts of data in a unified and meaningful way.

The ETL, Data Warehouse and Data Catalog Project was initiated. 

Research identified the following needs:

• A modern data Extraction, Transformation, and Loading (ETL) tool to help bring data from various sources into one location.
• A robust Data Warehouse management tool to ensure robust security, access, process automations, and to enhance data quality.
• A comprehensive Data Catalog to capture both shared and unique data definitions and the various critical reporting metrics across UA.

Issued an RFP seeking a technology platform to better manage our data assets across disparate data systems and silos, and to accommodate the University’s common and distinct business needs. 

With support from OIT, each MAU IR office, and Procurement the Data Warehouse RFP was launched in September 2023.

• We received “a historically unprecedented number of inquiries” and “an unexpectedly large number of proposals” in response.
• The project scoring team is committed to thorough examination and comparison of the diverse options proposed.
  • In first quarter 2024 the scoring committee is working to identify a smaller, top-ranked group of 2-5 proposals.
  • Scheduling a first round of demonstrations that will be technical in nature, focused on the ETL and data warehousing capabilities.
  • Soliciting feedback from the broader UA community, inviting them to attend a second round of demonstrations focused on business-oriented elements of the data catalog and related reporting features.

Following an extensive RFP process, Plante Moran has been selected to help the University implement a Data Warehouse & Data Catalog  solution leveraging Informatica and Snowflake.  Implementation will likely involve;

• Redesigning the new Data Warehouse to include MAU-specific tables that reflect the local business practices of each University. University-level data can then be used to inform aggregated UA system-level tables.
• Focus on value-added data utility while addressing current challenges, such as increasing clarity and context of our data definitions, setting demarcation of historical/new warehouse data, and providing a common framework to support future system-level initiatives.
• The implementation is reliant upon; the capabilities/limits of the vendor product selected, the level of support and resources invested by UA, and our ability to balance current workload demands with the work efforts required to stand up a new system.

Significant pre-assessment and discovery activities to solidify scope, guide implementation, and identify potential expansion phases.