Five-Year Departmental Evaluation Plan 2021-2022 to 2025-2026

2.1 Departmental transition

In August 2017, the Prime Minister announced the dissolution of Indigenous and Northern Affairs Canada (INAC), and the creation of two new departments, ISC and Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC), designed to better meet the needs of the people they serve, to accelerate self-determination and the closing of socio‑economic gaps, and to advance reconciliation. ISC was created to ensure a consistent, distinctions-based approach to the delivery of services to Indigenous peoples. ISC believes that transferring control to Indigenous partners is the best way to improve services and support their inherent right to self-determination with flexible approaches that are inclusive and responsive to the diversity of Indigenous peoples.

2.2 Departmental mandate and vision

ISC's mandate is to work collaboratively with partners to improve access to high quality services for First Nations, Inuit and Métis. Its vision is to support and empower Indigenous peoples to independently deliver services and address the socio-economic conditions in their communities.

2.3 Strategic Policy and Partnerships Sector (SPP)

SPP sector supports the Department's legislative mandate by advancing, in an integrated, co-developed, sustainable and evidence-based way, the improvement and transfer of Indigenous services to Indigenous peoples, for Indigenous peoples, by Indigenous peoples. SPP Sector will serve as an internal driver of change by supporting sectors in improving services and enabling Indigenous control.

2.4 Evaluation and Policy Re-Design Branch

The role of the Branch is to lead horizontal policy integration to advance departmental priorities, identify innovative policy interventions and partnerships to improve and transfer services, evaluate the success of programs in a neutral manner and support the advancement of Indigenous evaluation capacities and functions. 

The Branch further articulates its role to identify forward-looking, innovative, cross-cutting, horizontal policy interventions to support ISC sectors, other government departments and Indigenous partners in accelerating the transfer of Indigenous services, and leveraging partnerships to advance and evaluate progress on Indigenous-centric service design and delivery.

2.5 Evaluation Directorate

Aligned with the departmental and sector visions, the role of ISC Evaluation is to offer robust, timely and meaningful evidence to help shape and support the re-design of the Department's policies and programs, and to inform and influence improved service delivery and the transfer of services to Indigenous partners.

In this context, the evaluation function is anchored in three main areas: 

1. Delivering and managing core work on program evaluations and activities in accordance with the TBS Policy on Results (2016) and the Financial Administration Act (FAA);
2. Discovering and implementing innovative evaluation methods and approaches with Indigenous partners that specifically support service transfer; and
3. Exploring and implementing ways to support Indigenous evaluation capacity and advance Indigenous-led evaluation functions. This work is crosscutting to (1) and (2) above and underpins all of the evaluation work in this 5-year evaluation plan. 

ISC Evaluation has undertaken a great deal of transformational work with Indigenous partners in 2020-2021 on evaluation innovation projects and on other foundational pieces that are priorities in fulfilling its role.

2.6 ISC Evaluation innovation projects and other priorities

ISC Evaluation is increasingly emphasizing models of co-development and co-creation with Indigenous partners in all of its evaluation projects. In the short- to medium-term, this includes ways to integrate Indigenous evaluation expertise, knowledge, world views and/or Indigenous capacity development at key points in evaluations (planning, methodology, data collections, and development of findings and recommendations). In the long-term, the ultimate goal is to support the establishment of Indigenous evaluation functions outside of government.

In general, the evaluation innovation projects have the following objectives:

• To support Indigenous evaluators, research institutes, think tanks, and/or organizations to advance the implementation of ISC program evaluations with a focus on service transfer.  
• To support Indigenous organizations to strengthen their capacity to implement program evaluations, enhance and coordinate Indigenous policy and evaluation networks, and organize the Indigenous policy and evaluation field (e.g., conferences, a center of excellence, etc.)    
• To partner with Indigenous evaluators, researchers, think tanks, and/or organizations to advance evaluations tools, guides, policies, strategies and other evaluative practices that integrate Indigenous knowledge and worldviews.

In this context, ISC Evaluation began to support two multi-year evaluation innovation projects with Indigenous partners in 2020-2021: 

3.1 Status of evaluation completion in 2020-2021

Looking back on the previous year, four (4) evaluations were completed:

1. Evaluation of Elementary and Secondary Education Program
2. Evaluation of Post-Secondary Education Program
3. Evaluation of the Water and Wastewater On-Reserve Program
4. ESDC-led Horizontal Summative Evaluation of the Youth Employment Strategy

Also, as of the end of the 2020-21 fiscal year, ISC Evaluation published the following:

• Evaluation of the First Nations and Inuit Home and Community Care Program (March 2020)
• Evaluation of the Assisted Living Program (March 2020)
• A Review of Jordan's Principle (August 2020)
• Evaluation of the Aboriginal Health Human Resources Initiative 2013-14 to 2017-18 (August 2020)
• Five-Year Departmental Evaluation Plan 2020-2021 to 2024-2025 (January 2021)

The following evaluation projects were launched in 2020-2021:

• Evaluation of Education Facilities
• Evaluation of Healthy Child Development
• Cluster Evaluation of Health Infrastructure Support for First Nations
  1. Evaluation of Health Planning, Quality Management
  2. Health Services Integration Fund
  3. Quality Improvement and Accreditation Program
  4. Evaluation of Health Facilities
  5. Evaluation of BC Tripartite Governance
• Evaluation of Land, Natural Resources and Environmental Management

4.1  Scope of Work

One of the consequences of the transition was a broader scope of work and increased workload because of the additional programs for which the Department is now responsible. In 2017, the First Nations and Inuit Health Branch (FNIHB) was transferred from Health Canada to ISC. With this transfer, fourteen programs were moved to ISC. Further to the ISC and CIRNAC enabling legislation in July 2019, five programs were transferred from CIRNAC to ISC: Individual Affairs (includes Treaty Annuities); Indigenous Entrepreneurship and Business Development; Economic Development Capacity and Readiness; Land, Natural Resources and Environmental Management; Statutory, Legislative and Policy Support to First Nations Governance. In total 19 programs were added to ISC - the resulting impact to the ISC Evaluation workload was significant, approximately doubling the number of programs to be evaluated with no additional evaluation resources to carry out the evaluation work. Additional challenges include the inherited backlog of several evaluations from the previous shared evaluation function between ISC and CIRNAC and the many recent budget announcements and investments that have expanded the scope of existing programs or introduced new funding streams. For these reasons, there have been ongoing and persistent challenges in completing the planned evaluations. This also limits the flexibility of the Directorate to take on new evaluations that may be of interest to Senior Management as the Directorate is challenged to meet the due dates of the mandatory evaluations it must conduct to be compliant with the FAA. It will likely take a couple of years to stabilize the evaluation planning cycle.

4.2 Program structures

Most departmental programs, representing a majority of spending, are delivered through partnerships with Indigenous communities and in very few cases federal-provincial or federal-territorial agreements are in place. However, other funding structures have been established that are expected to become increasingly common as the Department moves to transfer services to Indigenous entities. For example, the British Columbia Tripartite Health Governance program involves a joint agreement between the Province of British Columbia, the First Nations Health Authority and ISC on health governance. Under this agreement, the parties agree to a tripartite evaluation, led by the First Nations Health Authority, with the final report being shared between the parties. In the future, more non-standard funding arrangements can be expected, with ISC Evaluation examining the 'readiness' of Indigenous governments or institutions to assume program delivery.

Currently, ISC has 30 authorities (voted grants and contributions) to support the 34 programs in its program inventory. However, a one-to-one relationship between the authorities listed in the Main Estimates and the Program Inventory of ongoing programs of grants and contributions deemed a 'Program' by the Department exists in very few instances. A number of programs draw on multiple grants and contributions; and conversely, there are grants and contributions that contribute to multiple programs. This Plan has been developed using the Treasury Board's Interim Guide on Results, which was updated most recently in August 2018, where there is the recognition that a 'program' of grants and contributions can cut across two or more programs in the Program Inventory and the relationship is not always one-to-one. This structure has presented the Department with some challenges in ensuring evaluation coverage for all authorities because of the segregation of a single authority over multiple programs with different evaluation timelines.

The program structure has also resulted in the number of evaluations exceeding the number of programs in the Program Inventory because of the distinct sub-programs that are included in the larger Programs. There are some sub-programs that have their own funding authority which alone supports the rationale for a separate evaluation that can identify improvements that might otherwise be lost in a larger evaluation. There are some sub-programs that are generally aligned with their Program, but when evaluators begin scoping the evaluation, the sub-program is quite unique and a single evaluation is not practical. One planned evaluation becomes two evaluation projects. This results in challenges in meeting TB commitments and FAA requirements as evaluations must sometimes be deferred.  

4.3 Evaluation planning methodology

The planning methodology is a multi-step process that begins with a review of the previous year's planning approach to update the steps where necessary and to review all relevant guidance and planning documents, such as the current Departmental Plan and Treasury Board commitments.

For 2021-2022, ISC refreshed its Departmental Results Framework (DRF) and its Program Inventory. Evaluation planning was conducted in alignment with the 2021-2022 framework. While there were no new programs added to the Program Inventory, there were two new Departmental Results added to Governance and Community Development Services. The 2021-2022 Core Responsibilities for ISC are:

• Services and Benefits to Individuals – with 4 supporting programs
• Health and Social Services – with 14 supporting programs
• Governance and Community Development Services – with 13 supporting programs
• Indigenous Self-Determined Services – with 3 supporting programs

Appendix A presents the entire Program Inventory for ISC by core responsibility.

The Plan is also expected to align with the Department's Risk-Based Audit Plan, which is prepared by the Audit and Assurance Services Branch (AASB). The AASB is part of the Audit and Evaluation Sector of CIRNAC and it serves both CIRNAC and ISC. As has been past practice, consideration is given to timing audits and evaluations in a way that does not overburden programs.

Each evaluation included in the Plan has been further assessed as to the type of evaluation required based on the review of Financial Administration Act or Treasury Board requirements and stakeholder consultations, as well as noting whether it is a horizontal or non‑horizontal evaluation. It should be noted that in January 2018, the Treasury Board Secretariat released a new Guide to Departments on the Management and Reporting of Horizontal Initiatives. This updated guide contains new requirements for horizontal initiatives, including a horizontal framework, specific reporting requirements and the creation of a governance structure comprising senior-level officials. TBS also went through an exercise of identifying mature horizontal initiatives that could submit close-out reporting rather than comply with the new requirements. Under the new guidance, ISC has no official Horizontal Initiatives, although it does participate in several led by other departments.

According to Section 42.1 of the Financial Administration Act, government departments and agencies must review, at least once every five years, the relevance and effectiveness of each ongoing grants and contributions program. Credit for evaluation coverage is granted upon deputy head approval of an evaluation report, under the premise that approval allows for the sharing of information in support of management action and the sharing of results with the public. Evaluation planning is conducted with that requirement in mind, reviewing the last fiscal year in which a program was evaluated, and scheduling completion and approval of the program evaluation five years later. Note that programs that are not funded through voted grants and contributions are not subject to the Financial Administration Act requirements.

However, as per the Policy on Results (Section 2.5), programs with five-year average actual expenditures of less than $5 million per year can be exempted from Section 42.1 and do not need to be evaluated every five years. Guidance from Treasury Board Secretariat states the five-year average is to be calculated on three years of actuals (Public Accounts) and two years of planned spending (Main Estimates). These calculations were conducted as part of the planning process and, based on these parameters, the following program is not scheduled for evaluation:

The Evaluation and Policy Redesign Branch conducted individual consultations in May 2021 with ISC senior management on the draft five-year schedule of evaluations to obtain their input on priorities for their respective sectors and on the timing of evaluations. Their input was synthesized and considered as part of the finalization process for the five-year schedule. The following senior management offices were consulted:

• Senior ADM Regional Operations
• Associate ADM Regional Operations
• Senior ADM First Nations and Inuit Health Branch
• ADM First Nations and Inuit Health Branch Regional Operations
• Chief Finances, Results and Delivery Officer
• ADM Child and Family Services Reform
• ADM Lands and Economic Development
• ADM Education and Social Development Program and Partnership
• ADM Strategic Policy and Partnerships Sector
• DG Communications
• DG Human Resources and Workplace Services

The planning methodology is largely driven by the requirements of the TBS Policy on Results and the Financial Administration Act. In future planning exercises, ISC Evaluation expects to engage national Indigenous organizations to ensure greater transparency of the planning process and to take into consideration the priorities of Indigenous partners.

4.4 Planned spending

Based on the Main Estimates, total planned spending for 2021-22 is outlined in the table below according to the Department's core responsibilities.

4.5 COVID-19 emergency funds

As of December 18, 2020, over $4.2 billion has been committed in specific COVID-19 support to Indigenous and northern communities and organizations, the bulk of which will be delivered by ISC. At this time, there is not a specific evaluation project identified to evaluate these emergency funding initiatives. Instead, as part of ongoing and future evaluations, ISC will take into consideration and/or address what impact COVID-19 emergency funds have had on the activities and results of core programming. ISC Evaluation will adjust the Plan to reflect any changes to that approach as the pandemic evolves and as priorities and needs change.

4.6  Coverage

Both of the following tables list 14 evaluations for the fiscal year 2022-23, as that year includes the Cluster Evaluation of Health Infrastructure Support for First Nations and Inuit. This cluster evaluation covers seven of the Department's First Nations and Inuit Health Branch programs that were grouped together to take a holistic view of how effectively programs are working in concert to improve, or put in place, health infrastructure for Indigenous communities. Consultations with senior management indicated that the cluster evaluation should take a broad strategic approach in order to examine whether the mix of programs meet the current needs of Indigenous communities and the alignment of goals of both FNIHB and ISC, that it covers all health programs related to health infrastructure, and that is in sync with the Department's mandate of reconciliation with Indigenous peoples in Canada. A cluster evaluation is also expected to alleviate the burden on program areas by evaluating their program only once in this five-year evaluation period and to alleviate the backlog of evaluations the Department currently faces.

ISC covers 100% of its voted grants and contributions amounts. The contribution amount for the Statutory, Legislative and Policy Support to First Nations Governance Program ($180,207 for 2021-22) is negligible.

The Department will also be participating in two horizontal evaluations that do not address coverage but will involve ISC Programs. These are evaluations of:

• It's Time: Canada's Strategy to Prevent and Address Gender-Based Violence, a horizontal initiative led by Women and Gender Equality Canada.
• The National Action Plan to Combat Human Trafficking, a horizontal initiative led by Public Safety Canada

Although there is no departmental funding attached to these horizontal initiatives, both initiatives involve the Department's Family Violence Prevention Program in reporting and evaluation, given its prevention programming and operational funding for shelters.

4.7  Resources

The Evaluation Directorate has the following planned expenditures for 2021-22 to deliver on the commitments made in the Plan. 

4.8 Challenges

4.9 Implementing the plan

Evaluations conducted by ISC Evaluation are used to inform decision-making and reporting (such as the Departmental Plan, the Departmental Results Report) to assess progress made by programs in achieving expected results and to incorporate lessons learned in order to improve program design and delivery. As noted, ISC Evaluation posts approved evaluation reports on its website in a timely manner, after sharing reports with Treasury Board. 

ISC Evaluation will work collaboratively with internal and external partners to tailor evaluation projects to meet the needs of decision makers, with a growing emphasis on innovating and implementing with Indigenous partners novel methodologies and approaches. This work will set the foundation for meaningful and substantive improvements to service delivery and service transfer while at the same time advance culturally appropriate evaluation capacities and functions. This work remains within the flexibilities of the Treasury Board Secretariat Policy on Results.  

ISC Evaluation is moving towards developing and implementing protocols for engagement with Indigenous representative organizations so that First Nations, Inuit and Métis peoples can contribute their knowledge and experience throughout the evaluation process. The Department is also working to build internal culturally-responsive evaluation capacity, as well as to support external capacity in collaboration with Indigenous partners with a focus on service delivery.

The Directorate is committed to delivering the evaluation projects that have been identified for 2021-22. As noted, it is addressing its capacity challenges though the ongoing recruitment, development, and retention of talented staff, with an emphasis on Indigenous recruitment, and is developing a strategy to address the backlog of evaluation projects using a risk-based and strategic approach. 

The consequences of the COVID-19 pandemic continue to be felt by the Department as it prioritizes the health and safety of Indigenous communities and peoples. It is expected that the pandemic will continue to have an impact on the implementation of evaluation work in 2021-22.  However, the extent of this is unknown. Should factors arise to compromise the delivery of evaluation projects, it will be brought to the attention of the SMC-PMEC in order that the senior management is fully informed.