Evaluation of the Harvest Sample Program - Final Report

Methodology

Evaluation approach

The evaluation relied on multiple lines of evidence including a document review, review of program data, interviews with key informants, surveys of producers and a literature review. By using multiple lines of evidence and triangulating the findings, the research methodology supported a comprehensive evaluation of the Harvest Sample Program.

Document review

The document review addressed evaluation issues related to the relevance and performance of the Harvest Sample Program. A total of 47 documents were reviewed as part of the evaluation (Appendix 5), including:

  • legislation including the Canada Grain Act
  • the Program Logic Model
  • annual harvest quality reports
  • speeches from the Throne
  • Departmental Performance Reports
  • Reports on Plans and Priorities
  • program expenditure reports
  • past reviews and evaluations of the Harvest Sample Program

Review of program data

Program data was reviewed to obtain information related to the relevance and performance of the Harvest Sample Program. Data analyzed as part of the evaluation included the number of registered producers in the producer database, previous customer satisfaction survey data, annual program summary reports which include producer participation rates, and performance data on visits to the program webpage and Canadian Grain Commission annual harvest quality report webpages.

Key informant interviews

A total of 83 interviews were conducted with key informants between October 2015 and February 2016 to address evaluation issues dealing with the relevance and performance of the program. Respondents were identified in consultation with the Project Steering Committee and sent an email explaining the purpose and timing of the interviews, a letter of introduction from the Canadian Grain Commission, and a copy of the interview questionnaire. Interviews were conducted by telephone and the responses were compiled and analyzed using Excel. Due to constraints scheduling telephone interviews with time differences and language barriers, some international buyers were provided the opportunity to submit their feedback via a written questionnaire. The number of respondents interviewed and the response rate for each stakeholder group is provided in Table 3.1.

Table 3.1: key informant interview metrics by cohort
Group Initial sample Valid sample Target Actual % of Target Response rate Est.
sample errorFootnote1
Canadian Grain Commission representatives 28 28 20 26 130% 92.8% 5.2%
Associations 49 47 15 19 127% 40.4% 17.5%
Domestic grain companies and buyers 44 40 15 21 140% 52.5% 14.9%
International buyers and processors 69 63 20 17 85% 26.9% 20.5%
Total 190 178 70 83 119%  46.6% 7.9%

Surveys of producers

Between November and December 2015, surveys were conducted with 1,207 producers who participated in the Harvest Sample Program, and 125 producers who registered for but have not participated in the program. The survey of participating producers was designed to obtain information from producers who have submitted samples regarding the use and benefit of the unofficial grade and quality assessment, their motivation for participating, and their degree of satisfaction with the services received. The survey of producers who registered but did not participate was designed to obtain information regarding their motivation for registering and the reasons why they have never submitted samples.

Survey respondents were sourced from the Harvest Sample Program producer database of 6,727 active producers. The database included 2,449 producers without an email address and 4,278 with an email address, 193 of whom had multiple registries and were excluded. The remaining 4,085 producers with emails were sent invitations to participate in the survey. In addition, 200 randomly selected producers with no email address were mailed a hard copy of the survey. Participants were sent pre-survey communication which included a letter of introduction and a copy of the questionnaire. A total of 3 email reminders were sent to producers with an email address and all participants received at least one telephone reminder. Updated email addresses were obtained for producers whose initial invitation had bounced and new invitations were reissued. Survey responses were coded and cleaned using Excel and summarized according to evaluation issue.

Table 3.2 outlines the number of survey completions and response rate for each cohort.

Table 3.2: key survey metrics by cohort
Group Initial sample Invalid or
bounced
Valid sample Target Completed % of target Response
rate
Margin
of errorFootnote1
Participating
producers
4,285 413 3,872 1,000 1,207 121% 31.2% 2.3%
Non participating
producers
N/A N/A N/A 150 125 83% N/A N/A

Literature review and comparative analysis of similar programs

The literature review and comparative analysis were conducted to obtain information related to the need for the Harvest Sample Program and to compare the effectiveness and efficiency of the program with similar programs in Canada and in other jurisdictions. Other programs were identified in consultation with the Evaluation Steering Committee, and included:

  • the Canadian International Grains Institute
  • Grain Farmers of Ontario
  • the US Wheat Association
  • the Australian Grain Growers Association / Australian Export Grains Innovation Centre
  • FranceAgrimer/Arvalis
  • the United Kingdom’s Agriculture and Horticulture Development Board’s Cereals and Oilseeds Division

A total of 50 websites, journal articles and grey literature sources were reviewed (Appendix 6).

Methodological considerations

Some of the limitations of the evaluation and the strategies employed to mitigate these limitations are as follows.

There is a potential response bias in the findings of the key informant interviews. This limitation was mitigated by:

  • clearly communicating the purpose of the evaluation, its design and methodology, and strict confidentiality of responses with respondents
  • cross-checking the responses with those of other stakeholder groups
  • using multiple lines of evidence
  • triangulating the evaluation findings

A relatively small number of completed interviews with international buyers and processors of Canadian grains makes it more difficult to ensure the results are significant, and to generalize the findings to the population of international stakeholders who use the annual harvest quality reports and other information.