Elsevier, Inc., ScienceDirect High School Access Program

CRA was contracted by the international publisher, Elsevier, Inc., to conduct an evaluation of its ScienceDirect High School Access Program. The purpose of the evaluation was to identify the strengths, challenges, and opportunities of this program, which provides high school students across the U.S. with access to ScienceDirect, an online database of science research literature and resources. CRA collected evidence from documents and from program participants (librarians, teachers, and students) during site visits to two participant schools, through interviews, focus groups, and questionnaires. CRA also collected evidence from teachers and librarians at participating schools through online questionnaires and telephone interviews. CRA analyzed, synthesized, and interpreted the evidence and prepared a report, a PowerPoint presentation, and a white paper, incorporating recommendations for program adjustments and opportunities. The project white paper, ScienceDirect High School Access Program: Strengths, Challenges, and Opportunities, is available online at http://libraryconnect.elsevier.com/whitepapers/02/lcwp02.pdf.

University of Colorado—Colorado Springs, Beth-El College of Nursing and Health Sciences, Doctor of Nursing Practice Program

CRA was contracted by the University of Colorado—Colorado Springs, Beth-El College of Nursing and Health Sciences to collaborate in the final development of a doctor of nursing practice (DNP) program. CRA also helped prepare and submit an application for federal grant funding for the DNP program through the Health Resources and Services Administration, Advanced Nursing Education Grant program. The University was awarded a three-year grant of approximately $800,000 to implement this fully online DNP program. CRA developed and is implementing the program monitoring and evaluation plan incorporated as part of the grant proposal to HRSA.

Goals of the evaluation are to (a) monitor and assess program development and implementation processes; (b) develop evaluation instruments and methods to measure the quality of multiple program elements across the three-year grant period; and (c) develop documentation to support HRSA grant reporting and national accreditation requirements.

As program evaluator, CRA has been a participant-observer in the full range of program planning and implementation processes since the program’s inception. CRA’s program evaluation strategies and instruments have been developed in collaboration with DNP faculty, staff members, program consultants, and grant program officers. CRA has developed and implemented program-specific assessment tools to measure the effectiveness of graduate level online nursing courses and student electronic portfolios. Additional data collected through interviews, focus groups, and online surveys is being analyzed using content analysis and appropriate statistical analysis techniques. CRA is providing documentation of program development, implementation, and progress toward program quality benchmarks for use in HRSA grant reporting and for the process currently being conducted for national accreditation of the UCCS DNP program.

CRWRC Nationwide AmeriCorps Program

The Christian Reformed World Relief Committee (CRWRC) contracted with CRA in 2007 to conduct a program-wide evaluation of its AmeriCorps program across 18 sites in 9 states, as well as the program functions of the CRWRC Home Office in Michigan. The evaluation (a) asked exploratory questions in order to reveal areas and issues for further evaluation; (b) established baselines for setting program goals; and (c) assessed the organization’s internal evaluation capacity. In addition, CRA developed recommendations for internal program monitoring and data collection to prepare for an evaluation of the program’s effectiveness, which CRA will conduct in 2010.

The 2007 CRA evaluation implemented a range of data collection methods including document review, onsite visits, interviews, focus groups, and online and paper-based questionnaires. The primary informants for the study were the Program Director, 8 Program Consultants, 18 Onsite Supervisors, and 21 AmeriCorps Members. In addition, approximately 10 community members from 3 target communities participated in interviews or focus groups. CRA developed questionnaires and interview guides intended to allow informants to express their ideas and feelings about their experiences in their own words. In addition, some questions asked informants to respond using rating scales so that some evaluation data could be readily aggregated over all program sites and informant groups. Data collection was conducted in English and Spanish.

CRA analyzed data collected as part of the evaluation, as well as data provided by CRWRC, using content analysis and descriptive statistical techniques. Due to small sample sizes in all informant groups, other statistical analysis techniques were not appropriate. CRWRC submitted CRA’s complete evaluation report to the Corporation for National and Community Service with its application for a second three-year AmeriCorps award and received a grant of nearly $1 million, a 27 percent increase over the previous award. The CRWRC-AmeriCorps program director believes that CRA’s evaluation report “played an important role in CRWRC’s successful application for refunding.”

The Keystone Center (Colorado)

The Keystone Center, an international nonprofit organization focused on addressing environmental issues, contracted with CRA to evaluate one of its teacher professional development programs, The Key Issues Institute—Bringing Environmental Issues to Middle- and High School Classrooms. The purpose of the evaluation was to determine program strengths, challenges, and opportunities. CRA worked collaboratively with program leaders and staff to define evaluation questions and plan for data collection. CRA collected data onsite during Key Issues Institute sessions through observations, focus groups, interviews, and questionnaires. Additional data was collected from Institute participants and program sponsors through online surveys and telephone interviews. CRA analyzed, synthesized, and interpreted evaluation data collected over a 14-month period. CRA then prepared a written report of results from the sponsor survey as well as a comprehensive written report of overall evaluation findings. The Keystone Center distributed CRA’s evaluation report widely to the organization’s governing board, program sponsors, program participants, and other program constituents.