STRENGTHENING ORGANIZATIONAL CAPACITY AND PRACTICES FOR HIGH-PERFORMING NONPROFIT ORGANIZATIONS: EVIDENCE FROM THE NATIONAL ASSESSMENT OF THE SOCIAL INNOVATION FUND--A PUBLIC-PRIVATE PARTNERSHIP.

Zhang, Xiaodong ; Griffith, Janet ; Pershing, Janet 等

INTRODUCTION

In the past decade, interest has intensified in the practice of developing organizational capacity to achieve programmatic goals. According to Harsh (2012), capacity building increases the ability of an organization to respond effectively to change by providing its staff with the skills and tools needed to identify and resolve problems over time. Although the nonprofit sector is widely viewed as a catalyst for change and a mechanism for serving societal needs, it faces substantial capacity challenges. De Vita and her colleagues (2001) observed that many small, community-based organizations are structurally fragile and many larger nonprofits are stretched to their limits. As demands for community-based services grow, with the identification of new needs and emergence of new service paradigms, nonprofits are continually challenged to strengthen their organizational capacity. On the other hand, there is a dearth of evidence-based information about what works and what does not work in building the organizational capacity of nonprofits. This is primarily because the philanthropic sector historically has paid little attention to capacity building, which was rarely supported by funders and of secondary importance to nonprofit managers whose priority is to deliver programs and services to people in need (Venture Philanthropy Partners, 2001). Nonprofits were further stretched during the 2008-12 recession. According to IRS data, charities with $50,000 or more fared somewhat worse during the recession-driven years of 2008-12 than during the previous four years (2004-08), although the difference was more modest than some had expected (Brown et.al., 2013). This situation is beginning to evolve, however, as more funders dedicate attention and financial support to developing organizational capacity.

This article contributes to the literature in two important ways. First, it provides a case study to test and advance theories about developing organization capacity that can be applied to a broader context, and presents empirical evidence concerning the ways that a high-profile federal program has addressed the twin goals of promoting evidence-based innovation while strengthening the organizational capacity of high-performing nonprofits. Second, the study features several methodological elements that enhance the evaluation's rigor. The evaluation employs two comparison groups, measures organizational capacity in terms of organizational behaviors, and supplements self-report with documentary evidence. In the rest of this article, we first present an overview of theoretical perspectives and empirical evidence that informed the national assessment. Next, we describe the SIF program and the national assessment. Finally, we present the findings and conclusions.

THEORETICAL PERSPECTIVES AND EMPIRICAL EVIDENCE

Organizational capacity refers to the wide range of capabilities, knowledge, and resources that an organization can employ to solve problems and achieve goals (Grantmakers for Effective Organizations, n.d.). Harsh (2012) presents a "multiple-dimension" approach to capacity building. Her four essential dimensions include: type of capacity (human, structural, organizational, or material); stage of capacity (exploration, emerging, full, or sustainability); level of capacity (information, skills, structures, or processes); and desired outcome of capacity. She argues that identifying the organization's starting point and capacity goals within these four dimensions is essential to designing a successful change initiative. She also speaks to the importance of monitoring implementation continuously to make mid-course corrections.

Wandersman and colleagues (2012) provide a logic model and roadmap for addressing gaps and improving organizational capacity. Much like Harsh's framework, the first step involves identifying existing capacity and the desired outcomes of capacity building initiatives. This existing capacity is then built upon using an iterative process that applies four support components--capacity tools, training, technical assistance (TA), and quality assurance/quality improvement--until goals are achieved. They argue for the integration of all four support components in order to overcome the limitations and challenges of each individual component. These four components also rely upon relationships, both within organizations and between organizations and their external capacity building providers. This iterative process offers a flexible structure and approach that ultimately guides organizations to accomplish capacity building outcomes and goals.

Nonprofit organizations face substantial challenges in developing organizational capacity. Traditionally, foundations make grants based on their assessment of a program's potential efficacy. Although that approach creates an incentive for nonprofits to devise innovative programs, it does not necessarily encourage their attention to assessing the strengths, goals, and needs of the organization itself. Thus, they may lack the organizational resources to carry out the programs that they have designed and tested (Letts et al., 1997). According to a report from the Center for Effective Philanthropy (Buteau & Buchanan, 2013), providing TA to nurture organizational capacity can often mean the difference between making a grant and making an impact. Grantees that receive comprehensive assistance from their foundation funders report a substantially greater impact on their organizations than grantees that receive no assistance. Providing this kind of TA requires a substantial commitment of time, money, and resources, however. Grantees funded by the Social Innovation Fund (SIF) are confronted with additional challenges in what is, for many, a new role as the recipient of a federal grant. By asking nonprofits to execute a range of essential tasks quickly--including fundraising, program implementation, and evaluation--the SIF has highlighted core strengths as well as areas in which nonprofits would need to improve to meet grant requirements (Grantmakers for Effective Organizations, 2013).

Empirical research on the effectiveness of capacity building efforts varies in scope and methodological rigor. For example, using surveys of 125 grassroots organizations that participated in capacity building events over five years, Sobeck (2008) found a positive relationship between investment in capacity building and organizational practices in planning, using evaluation strategies, and having grant writing knowledge and awareness of opportunities. Many evaluations of capacity building tend to rely on self-reports, with limited examples of more rigorous designs involving counterfactuals. One such example is the impact evaluation of the Compassion Capital Fund (CCF) demonstration program. CCF was a federal grant that enabled intermediaries to provide support to nonprofits designed to improve their organizational capacity, strengthen long-term sustainability, and enhance their ability to work effectively on target issues. (The level of investment was much smaller than the SIF.) The evaluation employed a randomized controlled trial (RCT) design and compared 217 treatment nonprofits with 168 control nonprofits served by 10 intermediary grantees in the 2006 grant cycle. The survey methodology featured baseline and 15-month follow-up web-based surveys. The evaluation found that the nonprofits that received capacity building support experienced significantly higher growth in several areas, including organizational development, leadership development, and community engagement (Minzner et. al., 2014).

Evidence on organizational impact began to emerge from some of the early cohorts of the SIF grantees and subgrantees. Stakeholder interviews by the Mile High United Way suggested that benefits on subgrantee organizations ranged from tangible increases in staffing capacity and related structural improvements, to improvement in ability to implement programming with fidelity across sites, and the new capacity to track and monitor programmatic efforts and related child outcomes (University of Denver, nd). Similarly, case studies of two United Way SIF grantees described how the grants have changed the ways organizations approach their work, in particular enabling them to adopt or enhance evidence-based practices for finding, selecting, and supporting high-performing nonprofit organizations (Stiefvater & Education Northwest, 2015). On the other hand, participation in the SIF has not been without challenges. The financial requirements of the program including match requirements, background checks and related fiscal documentation burden, as well as the need to keep programming consistent for research studies can be overwhelming at times (University of Denver, nd).

The literature suggests that organizational capacity is key to an organization's ability to solve problems and achieve goals. Nonprofit organizations, which have traditionally focused on delivering programs, face challenges in developing such capacity. However, recent trends suggest that more funders are starting to pay attention to developing organizational capacity and some of these efforts have begun to show promise in increasing organizational practices and abilities to design and implement programs. This article adds to the body of literature by providing empirical evidence from a nationwide grant program.

SOCIAL INNOVATION FUND

The SIF is a federal program designed to identify and implement innovative and effective evidence-based solutions to improve the lives of people in low-income communities throughout the United States. Authorized by the 2009 Edward M. Kennedy Serve America Act, it is administered by the Corporation for National and Community Service (CNCS). The SIF is one of the federal government's "tiered-evidence initiatives," established to promote social innovation based on evidence.

Since its authorization, the SIF has received an annual Congressional appropriation of approximately $50-70 million. The program leverages federal funds through public-private collaborations by awarding grants to larger nonprofit grantmaking organizations, which serve as intermediaries for the program. These SIF grantees match the federal funds and in turn fund local nonprofits (subgrantees), which also provide matching funds to implement community-based programs in one of three priority areas: youth development, economic opportunity, and healthy futures. The SIF's design integrates six key elements:

* Reliance for mobilization of the network on intermediary grantmaking institutions, which find, select, monitor, support, evaluate, and report on the nonprofit organizations that actually implement the community-based interventions.

* Requirement of a minimum of preliminary evidence of effectiveness for all funded programs/interventions, as tiered-evidence initiatives.

* Rigorous evaluation that will build upon the program/intervention's level of evidence.

* Scaling evidence-based programs to enable intermediaries to increase their impact within communities, locally or across the country, bringing them face-to-face with the widely shared nonprofit sector challenge of achieving successful and efficient scale up.

* A unique funding model that leverages public-private partnerships to support nonprofits and effect large-scale community impact in ways that neither a traditional federal grant nor a philanthropic investment could achieve on its own.

* A commitment to improving the effectiveness of nonprofits, funders, and federal agencies by capturing and learning about best practices and promoting approaches that generate the greatest impact for individuals and communities.

By late 2014, CNCS had completed four rounds of SIF grantmaking (in 2010, 2011, 2012, and 2014) and awarded 27 grants to 26 grantees (one organization received two grants). (No grantees were funded in 2013.) These grantees include both large nonprofits and foundations such as the Edna McConnell Clark Foundation, Jobs for the Future, and Venture Philanthropy Partners, and regional intermediaries like United Way Southeast Michigan and Mile High United Way. These grantees have in turn selected as subgrantees more than 200 nonprofit organizations through open, competitive processes.

Another fundamental purpose of the SIF program is to transform the way that nonprofit organizations conduct their work by requiring both grantees and subgrantees to build organizational capacity in targeted areas of practice. The SIF selects its grantees through a competitive process that requires them to demonstrate both preliminary evidence of their programmatic effectiveness and extensive capacity to support their subgrantees. To increase support for subgrantees and their programs, CNCS has provided extensive TA and other support to SIF grantees to foster capacity around evaluation, federal grants management and compliance, and communication and outreach. CNCS deploys capacity building through its SIF program officers, the CNCS Office of Research and Evaluation and its evaluation TA contractor, and peer learning opportunities for grantees such as convenings, conference calls, and on-line resource sharing.

Lastly, the SIF demands accountability and enforces high expectations among grantees by emphasizing the creation of an evidence base and the replication and expansion of what works. The application of evidence permeates all phases of the work from subgrantee selection and program improvement to demonstration of results and scale-up.

THE SIF NATIONAL ASSESSMENT

In 2013, CNCS funded an independent national assessment of the SIF program. Its main objective was to examine the impact of the SIF on the organizational capacity and practices of its grantees--their entire organizations, not only those sections that manage the SIF grant. CNCS's objectives were to increase effective program evaluation and management among grantees and further the national goal of learning from the federal government's tiered-evidence initiatives. Between 2014 and 2015, the national assessment team measured 14 key aspects of organizational capacity and practice, comparing the SIF grantees with other grantmaking organizations.

This section describes the evaluation questions that guided this research as well as the design, sample, instruments, data collection strategy, and analytic approach.

Evaluation Questions

To inform CNCS about the SIF program's implementation and impact on the organizational capacity and practice of grantees, the SIF national assessment addressed four key research questions:

1. Does participation in the SIF increase the capacity of organizations (for example, skills, attitudes, and behaviors) in the use of evidence-based grantmaking strategies?

2. Does participation in the SIF increase the willingness and ability of organizations to build the evidence base for high-impact models?

3. Does participation in the SIF increase organizational knowledge of how to scale effective program models?

4. Does participation in the SIF facilitate collaborative approaches to addressing local community needs?

Evaluation Design and Sample

The national assessment employed a quasi-experimental design (QED) to evaluate the SIF's impact on organizational change by comparing the grantees with themselves over time as well as creating two counterfactual groups that enabled evaluators to compare the experiences of SIF grantees with the experiences to be expected in the SIF's absence. The analysis compared SIF grantees with:

* Themselves over time. The evaluation assessed capacity building from a pre-SIF baseline (2009) to the time of data collection in 2014. This comparison allowed SIF grantees to serve as their own controls. This comparison did not rule out the possibility that any observed change resulted primarily from a general trend or high motivation among grantees.

* "Non-selected SIF applicants." The evaluation compared SIF grantees with applicants that had submitted satisfactory SIF applications but did not receive a SIF grant, providing an important counterfactual for the SIF grantee experience. Comparing the performance of non-selected SIF applicants with that of SIF grantees controlled for motivation and reduced the potential effects of selection bias.

* A nationally representative sample of grantmaking nonprofits. The evaluation compared SIF grantees with a sample of U.S. grantmaking nonprofits similar to the SIF grantees in terms of revenue and grantmaking size. Including this comparison group controlled for changes in the broader philanthropic/ nonprofit world, such as increasing engagement in rigorous evaluation, funding of evidence-based programs, and scaling up of programs with demonstrated effectiveness.

Although the SIF had funded four cohorts at the time of data collection, this article examines findings from the first three cohorts of 20 grantees (2010, 2011, and 2012) and the comparison groups. Because the six 2014 grantees were in their first year of SIF funding at the time of data collection, it was not realistic to anticipate changes in capacity and practice attributable to the SIF.

Differences in sample sizes for the SIF grantees and the two counterfactual groups resulted in variations in statistical power for the findings of these comparisons. Although the SIF grantee group included only 20 organizations (excluding those funded in the 2014 cohort), it represented the entire population and a response rate of 100%. With 19 cases in the analytic sample for the non-selected SIF applicants, tests for statistically significant differences between these two groups at [alpha] = 0.05 achieve 0.8 power for effect sizes of [alpha] = 0.68, considered medium-large effects (Cohen, 1988). With 262 cases in the analytic sample for the grantmaking nonprofits, tests for statistically significant differences between this group and SIF grantees at [alpha] = 0.05 achieve 0.8 power for effect sizes of [alpha] = 0.17, relatively small effects.

Instruments, Data Collection, and Analysis

To portray the organizational changes experienced by the SIF grantees, the evaluation drew on multiple sources of data, including (a) surveys of SIF grantees and the two comparison groups, (b) interviews with SIF grantees and other stakeholders, and (c) administrative data, including IRS form 990 (for financial data).

Instrument development. The national assessment examined five domains targeted by the SIF program: use of evidence-based grantmaking strategies, implementation and use of rigorous evaluations, scaling up of programs based on evidence of effectiveness, knowledge sharing, and collaboration to address community needs. To measure capacity, the study drew on respondent reports of changes in organizational behavior, because behavior is ultimately the measure that matters in considering organizational change. Finally, to allow for comparison over a common time frame, the survey asked SIF grantees and the comparison organizations about organizational behaviors in 2009 (before SIF implementation) and in 2014 (at or toward the end of SIF implementation).

The evaluators applied the retrospective pretest approach because the CNCS funded the national assessment several years after the grantees began work. Research has shown that retrospective pretest is not only simple and convenient but also effective in addressing the response shift bias commonly present in traditional pretest-posttest methodology, as respondents tend to overestimate pretest outcomes (Pratt et al., 2000; Hill & Betz, 2005; Cantrell, 2010). Cognitive interviews and pretests determined that respondents understood and could report on changes in organizational behavior, distinguish degrees of organizational behavior using the survey's 7-point scale, and report on factors that contributed to the changes. The survey instruments featured both closed-ended items (most using Likert-type scales) and open-ended items. Separate survey instruments for each group included all core questions, plus additional questions targeted to that group's distinctive experience.

Data collection. The evaluators sent survey participants a respondent-specific email link to an online survey tool. Respondents had the option of completing the entire questionnaire in one sitting or over several sessions. They were invited to involve others in their organization in answering any questions. Strategies to increase response rates included: tapping multiple sources to compile accurate and up-to-date contact information; establishing a user-friendly online survey platform; ensuring respondent confidentiality; allowing ample time for response; and sending multiple reminders and follow-ups. Once a SIF grantee submitted its survey, the evaluators sent a request for a 15-30 minute follow-up interview that gave SIF grantees the opportunity to discuss evidence and supply documentation that would support their observations in the survey about organizational change.

The evaluators computed response rates for all three target populations (SIF grantees, non-selected applicants, and the national sample of grantmaking nonprofits), abiding by the American Association for Public Opinion Research (n.d.) standard definitions for web surveys of named persons. The surveys achieved 100% response from SIF grantees, but lower response rates from non-selected SIF applicants (35%) and the national sample of grantmaking nonprofits (12%). However, a non-response analysis suggested little non-response bias. Weighting for the responses from these two groups improved the precision of the estimates.

Data analysis. Data analysis first documented and analyzed change in grantmaking strategies reported by SIF grantees. Because the estimates from the SIF grantees were population values, no statistical significance test was required. Comparisons between SIF grantees and the two counterfactual groups provided information about whether to attribute observed change to SIF participation. The evaluators used different analytic approaches to account for differences in sample size between the two comparison groups. To account for small sample sizes, we used the non-parametric Mann-Whitney test to assess differences between the SIF grantees and non-selected SIF applicants. To compare the differences between SIF grantees and the national sample of grantmaking nonprofits, we calculated the confidence interval at 95% around the values for the nonprofit sample.

In addition to the statistical significance test, we employed effect size estimates to gauge the practical significance of differences in magnitude. Statistical significance, which measures the likelihood that a result could occur by chance, is highly dependent on sample sizes. On the other hand, effect size, which measures the size of difference between groups, is less dependent on sample sizes. A small effect can be statistically significant if the sample size is large, while a large effect may not achieve statistical significance if the sample size is small. Cohen (1988) defines effect sizes as "small, d = 0.2," "medium, d = 0.5," and "large, d = 0.8."

Finally, analyses of open-ended questions from the surveys, as well as interview data and documentary evidence, addressed "how" and "why" questions about the factors that contributed to the observed change, and provided insights that supplemented the survey data. Analysis of qualitative data highlighted themes and emergent patterns and used illustrative quotes, examples, or vignettes.

Limitations and Solutions

The appraisal of organizational change conducted for the SIF national assessment had both strengths and limitations. One strength was the inclusion of two comparison groups, which enabled the evaluation to address questions about SIF grantees relative to both similar, highly committed organizations (non-selected SIF applicants) and to broader trends in the field of nonprofit grantmaking (the national sample). In addition, combining multiple data collection strategies--survey, interview, and document review methodologies--allowed the collection of specific, objective information about change as well as insights into factors that contributed to reported change.

The limitations of this study include the small number of SIF grantees and non-selected SIF applicants in the sample, which renders survey data analyses vulnerable to such factors as the presence of outliers. Several strategies were used to mitigate this issue, including effect size calculations and non-parametric assumptions in statistical significance testing. Other limitations are the reliance on self-report and retrospective reporting. The evaluation addressed these limitations by obtaining evidence from SIF grantees to document changes reported in the survey. Finally, our discussions with CNCS staff and grantees indicated that the experience of the first SIF cohort differed from that of subsequent cohorts. They perceived, for example, that the first cohort grantees experienced more programmatic "growing pains" than subsequent cohorts. These observations may suggest that the potential to generalize from this study to the broader population of future SIF grantees is limited. This study reports evidence about the experience of a specific group of organizations, and the future mix of organizations and therefore the experiences of later SIF cohorts may differ.

FINDINGS

This section first describes the organizational characteristics of the respondents. Next, it reviews findings related to impact on organizational capacity and practices through comparison of SIF grantees with the comparison groups.

Organizational Characteristics

Data analysis began with a comparison between SIF grantees and the two comparison groups on three financial indicators (obtained from 2013 federal tax submissions): gross revenue, total value of government grants received, and total value of grants made to U.S. organizations. The results suggest that SIF grantees were similar to both the comparison groups in terms of gross revenue and government grants received. Although similar to non-selected SIF applicants in terms of the value of grants made to U.S. organizations, SIF grantees gave significantly more to U.S. organizations than did the national sample of grantmaking nonprofits (Table 1).

The survey asked about prior experience with federal funding in part because some SIF grantees, especially those that had not previously received federal funds, found it challenging to comply with federal funding requirements. These data provide context for analysis of the SIF experience and capacity development, and suggest considerations for subsequent federal funding. The data show that two-thirds of SIF grantees receiving federal funding in 2009 and 2014 had no prior federal funding. Among the national sample of grantmaking nonprofits, about two-thirds had neither current nor previous federal funding (Table 2).

Impact on Organizational Capacity

This section examines five aspects of organizational capacity targeted by the SIF program: selection of subgrantees, support for subgrantees, evaluation, scaling up of evidence-based programs, and collaboration. The discussion of each aspect is organized into three sections:

1. SIF grantee organizational changes (comparison between the organizational capacity of SIF grantees in 2009 and 2014)

2. Changes for SIF grantees vs. comparison groups (comparisons between the organizational capacity of SIF grantees and two comparison groups in 2009 and 2014)

3. Observations about organizational change (responses concerning major factors affecting change and summary of open-ended question responses)

Selection of subgrantees. The survey asked SIF grantees about subgrantee selection in both 2009 and 2014. If the responses for the two years differed, the survey asked about changes that the SIF grantees experienced, as intermediary organizations, during this period.

SIF grantee organizational changes. About one-third of SIF grantees reported changes in subgrantee selection practices. Between 2009 and 2014, SIF grantees revised their funding eligibility guidelines to require applicant organizations to provide evidence of intervention effectiveness (Item B: 5.0 to 5.3) and submit a plan for rigorous evaluation of the intervention (Item C: 4.8 to 5.1). However, the practice of operating an open and competitive process to solicit, review, and approve or reject applications changed in a negative direction between 2009 and 2014 (Item A: 4.6 to 4.4). (The decrease primarily reflects the account of one SIF grantee of the six that responded to this question, of using open competition in 2009 but never in 2014 (Table 3).

Changes for SIF grantees vs. comparison groups. Compared to the national sample of grantmaking nonprofits, SIF grantees experienced significantly less change between 2009 and 2014 in the extent to which they used an open and competitive process to solicit, review, and approve or reject applications, again largely due to negative change reported by one grantee. The results also show medium to small positive differences in favor of the national sample of grantmaking nonprofits in all three areas. These findings were somewhat inconsistent with the comparison to non-selected SIF applicants. The same pattern held true for using an open and competitive process, but small differences were found in favor of SIF grantees, in effect sizes, with regard to requiring applicants to provide evidence of effectiveness and submit a rigorous evaluation plan to be eligible for funding (Table 3).

Observations about change. A majority of SIF grantees attributed changes in their subgrantee selection process to strict selection requirements imposed by their participation in the SIF, although they also mentioned other factors that contributed, such as trends in the field and direction from the organization's leadership. The other two groups were not asked about SIF participation effects, but were asked about other factors. A majority of the national sample of grantmaking nonprofits attributed changes to guidance from their board or other leadership. Both comparison groups also referenced trends in the broader grantmaking world as important factors that contributed to change. (The tables do not include statistics for these findings.) One respondent noted, "Donor tolerance for funding that does not have measurable results has changed." Some added that their organizations had always required evidence of effectiveness in grant applications, but in some cases, it is challenging to identify earlier evidence. For example, one nonprofit respondent observed, "We fund arts experiences; [it is] difficult to quantify the impact of arts education experiences in one year."

In open-ended comments about the impact of their participation in the SIF on their approach to selecting subgrantees, SIF grantees reiterated the themes of implementing application-based competitions, emphasizing evidence and evaluation, and increasing use of tools. Below are comments that offer context. The SIF RFP process was the first ever for this organization. We found this process to be helpful and plan to use it in the future. For another program, we now use a competitive selection process for all of our new, non-SIF sites. We found the selection process to be a valuable method of pre-planning and gauging interest and dedication to the model.

Support for subgrantees. Grantmaking organizations vary in the extent to which they support subgrantees to develop their capacity to carry out programs. The survey asked respondents about both the financial and non-financial assistance they provided to help subgrantees implement and evaluate their programs.

SIF grantee organizational changes. About one-third of SIF grantees reported changes in the ways that they support their subgrantees. Typically, SIF grantees expanded their support to subgrantees by more frequently funding an evaluation or hiring an external evaluator (Item A: 3.0 to 3.9), providing training or TA to conduct rigorous evaluation (Item B: 3.6 to 4.3), or supporting program implementation (Item C: 4.4 to 5.0) (Table 4).

Changes for SIF grantees vs. comparison groups. Compared to the national sample of grantmaking nonprofits, SIF grantees experienced significantly greater changes between 2009 and 2014 in the extent to which the organization: (a) provided funding to carry out an evaluation or hire an external evaluator and (b) provided training or TA to support rigorous evaluation. The effect sizes in the latter two areas were large, but negligible for TA to support program implementation. Although the change patterns for SIF grantees were not statistically significant, we found small differences in favor of SIF grantees, compared to non-selected SIF applicants, in the magnitude of TA provided to support evaluation and program implementation (Table 4).

Observations about change. Among organizations that reported changes in their TA support to grantees for program implementation, the majority of SIF grantees attributed the changes to their participation in the SIF, but the majority of the national sample of grantmaking nonprofits attributed such changes to direction from their board or other leadership and to trends in the broader grantmaking world. Another topic that SIF grantees identified in open-ended responses was a shift in their organizational business model toward a greater emphasis on grantee support, either for evaluation or TA or both. Some SIF grantees specifically credited the SIF with changing how they provide TA to subgrantees--not only within their SIF programs, but across their broader portfolios. SIF offered both the opportunity and the mandate to increase the level of TA for our subgrantees around evaluation and program administration. In 2009 we offered small TA grants around program implementation, and in 2014 we were able to offer larger one-on-one TA awards to our SIF subgrantees, as well as group TA and training to support data collection and other evaluation activities. SIF's requirements around TA and shared learning have impacted our impact work more generally, not just the body of work that is funded by SIF. More importantly, SIF has been a lever that we've used. By demonstrating the government's commitment to this kind of support, we can make a stronger case for providing it.

Use of data, evidence, and evaluation. The survey asked about ways that SIF grantees used evaluation in 2009 and 2014, their evaluation resources, and infrastructure. In addition, we asked respondents whether their organizations had changed their use of data and evidence, and what factors contributed to those changes.

SIF grantee organization changes. SIF grantees identified the greatest changes in their use of evaluation, with about two-thirds reporting increases in their capacity to conduct rigorous program evaluation (Item A: 3.1 to 4.5) and to interpret evaluation findings to improve programs (Item B: 3.9 to 4.9) and demonstrate and communicate program effectiveness (Item C: 3.7 to 4.9) (Table 5). This outcome is consistent with the SIF program's substantial focus on evaluation.

Changes for SIF grantees vs. comparison groups. Compared to the national sample of grantmaking nonprofits, SIF grantees experienced significantly more growth between 2009 and 2014 in all three evaluation areas, including (a) conducting rigorous program evaluations; (b) using evaluation findings to improve programs; and (c) using evaluation findings to demonstrate and communicate the effectiveness of programs funded by the organization. The effect size differences in all three areas were large.

Compared to non-selected SIF applicants, SIF grantees experienced significantly greater changes between 2009 and 2014 in the extent to which they conducted rigorous evaluations of programs funded by the organization, and the effect size was large. Although the differences in the other two evaluation areas were not statistically significant, the effect sizes were medium and small, respectively (Table 5).

Given the SIF's intense emphasis on evaluation, the survey also asked about changes in evaluation staffing and budget. SIF grantees, non-selected SIF applicants, and the national sample of grantmaking nonprofits all reported increases between 2009 and 2014 in several indicators of evaluation capacity: (a) staff position(s) or groups within the organization dedicated to evaluation; (b) external evaluation partners that provide the organization with evaluation services; and (c) line items in the organization's budget dedicated to evaluation. SIF grantees had a significantly higher baseline for evaluation capacity in 2009 than did the national sample of grantmaking nonprofits: SIF grantees were more likely to have (a) staff position dedicated to evaluation; (b) an external evaluation partner; and (c) dedicated line items. SIF grantees also experienced significantly more growth in these areas than did the national sample of grantmaking nonprofits(Table 6).

The perception data in Table 6 are further substantiated by organizations' financial data (Table 7). The majority of SIF grantees and non-selected SIF applicants reported significantly higher evaluation budgets in 2014 than in 2009, both in total dollar amount and percentage of the organization's annual budget. In contrast, the national sample of grantmaking nonprofits was more likely to report that their evaluation budgets were about the same in 2014 as in 2009.

SIF grantees significantly exceeded the national sample of grantmaking nonprofits in terms of annual evaluation budget as a percent of the organization's total budget (4.1% higher) in 2009, and in terms of increase in that budget (1.7%) in 2014 (Table 8). One caveat is that, in some cases, the evaluation budget may be part of the program budget rather than a separate line item. In addition, the mean values for the national sample of grantmaking nonprofits were skewed by one outlier with a much higher evaluation budget than any others. Across the three groups, we found no differences over time in the number of full-time equivalent (FTE) staff with primary responsibility for evaluation. One explanation is that SIF grantees tend to hire external evaluators, because internal staff are not equipped to conduct the rigorous evaluations required. Nevertheless, additional staff may be hired to oversee these evaluations.

Observations about change. A majority of those SIF grantees reporting changes in their use of evaluation attributed the change to their participation in the SIF. In contrast, a majority of the national sample of grantmaking nonprofits credited direction from the organization's board or other leadership for the change. In the open-ended responses, SIF grantees noted other factors such as increased capacity in terms of staff, software, evidence-based programs, and framework, with an emphasis on collective impact evaluation. When asked how participation in the SIF had contributed to their organization's approach to evaluation, SIF grantees noted an increase in their communication of findings, appreciation of evaluation, and use of resources and tools. SIF's emphasis on rigorous evaluation is giving us an opportunity to demonstrate to our organization as a whole how evaluation can impact results. It's allowing us to show what it means to fund for impact. The SIF helped us look at regional evaluation capacity--both from a grantee and evaluator standpoint. We are working with other local funders to build capacity. We are much more focused on evaluating at both a program level and our own community change efforts.

Scaling up programs with evidence of effectiveness. Increasingly, federal and nonprofit grantmakers are seeking to scale up programs that have demonstrated evidence of effectiveness. Scale-up may be broadly defined as increasing the impact of a program within a community or expanding it to other communities or populations.

SIF grantee organizational changes. About half the SIF grantees reported changes in their efforts to scale up existing programs (Item A: 4.2 to 5.1) and select programs for scale-up based on rigorous evaluation that demonstrates their effectiveness (Item B: 3.5 to 4.6) (Table 9).

Changes for SIF grantees vs. comparison groups. Compared to the national sample of grantmaking nonprofits, SIF grantees experienced significantly greater changes between 2009 and 2014 in the extent to which they selected programs for scale-up based on rigorous evaluation that demonstrates effectiveness; the effect size was medium. Although we found no statistically significant differences in scale up among the three groups, we found small distinctions in favor of SIF grantees in the magnitude of those differences. Although the change patterns for SIF grantees were not statistically significant, compared to non-selected SIF applicants, we detected small differences in favor of SIF grantees in the magnitude of differences in both capacities.

Observations about change. For those SIF grantees reporting changes in scaling up programs, a majority attributed them primarily to the organization's participation in the SIF, whereas a majority of the national sample of grantmaking nonprofits attributed changes to direction from the organization's board or other leadership. (SIF grantees also cited direction from their board/leadership as a major factor.) In response to open-ended questions, SIF grantees noted increased capacity in terms of staffing, shifts in organizational strategies such as adoption of "results-based accountability," and more funds raised for scaling up. Asked how participation in SIF contributed to their approach to scaling up, SIF grantees reported a greater understanding of scaling, more available funding, newly created staff positions, and evaluation readiness: SIF is providing us with the evidence needed to scale certain programs. We also have a deeper understanding of what it takes to scale programs and how to support organizations to grow. We would not have had that "forced learning" without SIF. We are taking what we've learned from SIF that has had positive impact in terms of process (such as competitive bids, an emphasis on accountability), and we are applying those standards to other bodies of work we fund.

Collaboration to address community needs. By awarding SIF grants, CNCS also anticipates prompting wider and deeper collaboration by SIF grantees--with subgrantees, other SIF grantees, and grantmakers--to address community needs and improve collective impact. The requirement for raising and sustaining match funding requires collaboration with other funders, but CNCS is interested in fostering broader community among funders.

SIF grantee organizational changes. About half the SIF grantees indicated changes in collaboration between 2009 and 2014 and reported considerable increases in their participation in funding alliance(s) with other nonprofit sector organizations (Item A: 3.6 to 4.3), knowledge sharing (Item B: 4.2 to 5.2), and collaboration for purposes of advocacy (Item C: 3.5 to 4.1) (Table 10).

Changes for SIF grantees vs. comparison groups. No statistically significant differences emerged between SIF grantees and the comparison groups. However, compared to the national sample of grantmaking nonprofits, we found small differences in favor of SIF grantees in the magnitude of their participation in funding alliances with other nonprofits and collaborations with other nonprofits to share knowledge. Compared to non-selected SIF applicants, we found small differences in favor of SIF grantees in the magnitude of collaborations to share knowledge (Table 10).

Observations about change. A majority of SIF grantees that reported changes in collaboration through funding alliances and knowledge sharing attributed these changes to their participation in SIF and to trends in the broader grantmaking world. A majority of the national sample of grantmaking nonprofits, on the other hand, attributed such changes to direction from their board or other leadership.

SIF grantees and the national sample of grantmaking nonprofits both cited direction from the organization's board or other leadership as a major factor behind increased collaboration for advocacy. (SIF participation was not a major factor, because grantees cannot legally use federal funds for lobbying.) This is one illustration of the impact on SIF grantees of trends in the broader grantmaking world. In response to open-ended questions, a few SIF grantees noted other factors that contributed to increased collaboration, such as the recession and budget cuts, consolidation of services to augment collective impact, and the need for intensified advocacy.

Regarding the ways that participation in the SIF influenced their approaches to collaboration, SIF grantees emphasized various forms of collaboration among grantees, funders, and industries. For example, To bolster support of our subgrantees, the organization entered into a strategic collaboration with the evaluator in our application to become a SIF intermediary. This collaboration extended beyond our SIF work to support most of our grantees. Our previous grantmaking did not necessarily have a match requirement. However, SIF funding does have this requirement, and so we are participating in these efforts with our SIF subgrantees. We've always participated in industry-specific exchanges; however, now we participate in cross-industry exchanges via the SIF Economic Opportunity Group.

CONCLUSIONS

The SIF program has sought to increase the capacity of social entrepreneurs to tackle social issues, seed experimental initiatives, and provide resources to replicate and expand effective programs. The national assessment described in this article sought to identify the SIF's influence on grantee capacity to meet these goals. The evaluation found evidence of positive changes in almost all target capacities. Table 11 shows that between 2009 and 2014, SIF grantees reported improvement in 13 of 14 measures of organizational capacity. However, not all areas of capacity evolved in the same way.

Compared to non-selected SIF applicants and the national sample of grantmaking nonprofits, as measured by effect size estimates, SIF had a large impact on evaluation capacity. Followup interviews suggested that the SIF focus on evaluation helped grantees plan and implement more--and more formal--evaluation; grantees increased their evaluation capacity with evaluation staff hires and the use of external evaluation partners; the SIF focus on evaluation helped grantees use evidence to improve results; and the assistance helped position SIF subgrantees to attract new funders, because they had the skills to conduct rigorous evaluation and communicate program effectiveness.

Data suggested a medium impact of the SIF on grantees' support for subgrantees Followup interviews further indicated that SIF grantees had increased their investment in growing the capacity of their subgrantees in terms of compliance capacity, evaluation capacity, and communication capacity; the SIF helped grantees become more strategic in how they support their subgrantees; and grantee support gave subgrantees a solid foundation upon which to grow.

A medium effect was also observed in the area of scaling up by SIF grantees. Themes related to scaling noted that the SIF helped grantees develop methods for choosing which interventions to scale up; and grantees packaged what they learned from the SIF to support scaling among the subgrantees.

While the SIF appears to have had a small impact on subgrantee collaboration, the followup interviews still showed that the SIF gave grantees a variety of collaborative learning opportunities; the SIF influenced how grantees support peer learning among their subgrantees; and the SIF augmented the extent to which grantees engaged in external partnerships.

Although a small effect was observed for grantees' practices regarding subgrantee selection, followup interviews indicated some promising trends including that the SIF helped grantees build a more systematic process for subgrantee selection; the selection process helped grantees identify subgrantees with a stronger evidence base; and the SIF's competitive solicitation process allowed grantees to reach subgrantees in new markets and geographic areas.

The SIF grantees regarded rigorous evaluation and engaging in peer-learning experience as major strengths of the SIF model. They also valued the transparency and accessibility of the CNCS staff and their flexibility. Even with these changes, many organizations continue to have room to improve their capacity. When asked how often they undertake activities indicative of strong organizational capacity in the survey, most SIF grantees scored themselves at or below 5 on a scale of 1-7, with 1 meaning "not at all" and 7 "meaning always."

While the evidence of progress is encouraging, the grantees offered recommendations for improving the SIF program. Many grantees reported that meeting the match requirement was a major challenge. Even among those who had early success, by years 3 to 5, several grantees mentioned undertaking an exercise to move existing funds around to meet the match, but not truly leveraging new dollars. In addition, although many of the federal requirements were not SIF-specific, grantees considered the federal compliance in financial regulations and criminal background checks as particularly burdensome. SIF grantees and applicants offered additional recommendations to improve the SIF application process, such as adjusting the timeframe, including a pre-qualification phase, more transparency about implementation requirements, and further streamlining and clarifying of application requirements. Finally, SIF grantees recommended continuing collaboration among SIF grantees and funders, and reiterated the importance of securing engagement and feedback from private foundations and philanthropies.

Building organizational capacity requires a multi-faceted approach and lengthy commitment through a recurring cycle of intervention, support, and growth (Harsh, 2012). The SIF national assessment has provided promising evidence that the SIF moved grantee organizations in the direction of increased capacity to make and support investments in program improvement and expansion. It also reminds funders and nonprofits that organizational changes require patience. As several respondents pointed out, almost everything about developing capacity takes more time and presents more complexity than one might anticipate. Nevertheless, building capacity in nonprofits has the potential to strengthen not only their individual organizational competency, but that of the community of nonprofit organizations as well.

REFERENCES

American Association for Public Opinion Research. (n.d.). Response rates: An overview. Retrieved from https://www.aapor.org/Education-Resources/For-Researchers/Poll-Survey-FAQ/Response-Rates-An-Overview.aspx.

Brown, M.S., McKeever, B., Dietz, N., Koulish, J., and Pollak, T. (2013). The impact of the great recession on the number of charities. National Center for Charitable Statistics. Washington, DC: Urban Institute.

Buteau, E., & Buchanan, P. (2013). Working well with grantees: A guide for foundation program staff. Cambridge, MA: The Center for Effective Philanthropy.

Cantrell, P., (2010). Traditional vs. retrospective pretests for measuring science teaching efficacy beliefs in preservice teachers. School Science and Mathematics, 103(4), 177-185.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.

Corporation for National and Community Service. (n.d.). Social Innovation Fund: Evaluation plan guidance. A step-by-step guide to designing a rigorous evaluation. Washington, DC: Corporation for National and Community Service.

De Vita, C. J., Fleming, C., & Twombly E. C. (2001). Building nonprofit capacity: A framework for addressing the problem. In C. J. De Vita & C. Fleming (Eds.). Building capacity in nonprofit organizations. Washington DC: The Urban Institute.

Grantmakers for Effective Organizations. (2013). Investing for impact: Building the capacity and infrastructure of high-performing nonprofits. Washington, DC: Grantmakers for Effective Organizations.

Grantmakers for Effective Organizations. (n.d.). What is nonprofit capacity and why does it matter? Washington, DC: Grantmakers for Effective Organizations.

Harsh, S. (2012). Taking successful change initiatives beyond capacity: A multiple-dimension approach to capacity building. Fairfax, VA: ICF International.

Hill, L.G., & Betz, D.L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-517.

Letts, C.W., Ryan, W., & Grossman, A. (1997). Virtuous capital: What foundations can learn from venture capitalists. Harvard Business Review (March-April), 1-8.

Minzner, A., Klerman, J. A., Markovitz, C. E., & Fink, B. (2014). The impact of capacity-building programs on nonprofits: A random assignment evaluation. Nonprofit & Voluntary Sector Quarterly, 43(3), 547-569. doi: 10.1177/0899764013491013.

Pratt, C. C, McGuigan, W. M., Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341-349.

Sobeck, J. L. (2008). How cost-effective is capacity building in grassroots organizations? Administration in Social Work, 32(2), 49-68. doi:10.1300/J147v32n0204.

Stiefvater, E., & Education Northwest (2015). Lessons and Stories from the Social Innovation Fund: From Transactional and Transformational--United ways Partner for Impact. Washington, DC: Corporation for National and Community Service, Social Innovation Fund.

University of Denver (n.d.) Mile High United Way: Impact of the Social Innovation Fund on Participating Organizations. Denver, CO: University of Denver, Graduate School of Social Work Butler Institute for Families.

Venture Philanthropy Partners. (2001). Effective capacity building in nonprofit organizations. Washington, D.C.: Venture Philanthropy Partners.

Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50.

XIAODONG ZHANG

JANET GRIFFITH

JANET PERSHING

JING SUN

LARA MALAKOFF

WHITNEY MARSLAND

KURT PETERS

ELLY FIELD

ICF Table 1 Financial indicators by respondent group National Financial Non-selected sample of indicators SIF grantmaking (2013) SIF applicants nonprofits Gross revenue N 15 9 147 Mean $76,573,121 $74,927,426 $98,431,599 Median $44,479,384 $22,234,541 $ 6,833,348 Total value of N 14 6 121 government Mean $5,307,259 $980,788 $3,741,735 grants received Median $1,411,197 $618,451 $ 660,614 Total value of N 15 9 218 grants made to Mean $21,573,525 $33,890,743 $12,483,065 U.S. (b+) organizations Median $11,462,022 $ 8,278,806 $ 4,111,065 Statistical significance (p<.05) is noted as: a: significant difference between SIF and non-selected SIF applicants b: significant difference between SIF and the national sample of grantmaking nonprofits +: favoring SIF -: favoring the comparison group Sample sizes for Table 1 are smaller than those reported in other exhibits due to missing data. Table 2 Federal funding experience of SIF grantees and the comparison groups Group % n (yes) n % (yes) Change SIF 19 36.8% 20 45.0% 5.3% Non-selected SIF 8 62.5% 8 62.5% 0.0% applicants National sample of 181 38.4% 180 41.6% 3.5% grantmaking nonprofits Table 3 Changes in how SIF grantees and comparison groups approach subgrantee selection To what extent did your organization do this...? Group 2009 2014 A. Used an open, SIF 4.6 4.4 competitive process to (20) (19) solicit and review Non-selected 5.0 (9) 5.3 (9) applications and to SIF applicants make selection National 4.8 5.1 decisions sample of (207) (207) grantmaking nonprofits B. Required applicant SIF 3.5 4.0 organizations to (20) (19) provide evidence of Non-selected 4.8 (9) 5.1 (9) intervention SIF applicants effectiveness to be National 3.7 4.5 eligible for funding sample of (202) (206) (includes pre- and grantmaking posttest outcome data nonprofits or other evidence based on evaluation studies) C. Required applicants SIF 2.0 2.3 to submit a plan for (20) (19) rigorous evaluation of Non-selected 2.9 (9) 3.1 (9) intervention to be SIF applicants eligible for funding National 2.4 2.9 (that is, quasi-experimental sample of (201) (200) designs grantmaking with a comparison nonprofits group, experimental designs, or other similarly rigorous designs) To what extent did your organization do this...? Effect Respondents Change size with change A. Used an open, -0.1 (b-) -0.1 6 competitive process to solicit and review 0.3 -0.3 4 applications and to make selection 0.3 -0.7 53 decisions B. Required applicant 0.6 0.2 8 organizations to provide evidence of 0.3 0.4 3 intervention effectiveness to be 0.8 -0.7 84 eligible for funding (includes pre- and posttest outcome data or other evidence based on evaluation studies) C. Required applicants 0.4 0.2 5 to submit a plan for rigorous evaluation of 0.2 0.3 2 intervention to be eligible for funding 0.5 -0.2 46 (that is, quasi-experimental designs with a comparison group, experimental designs, or other similarly rigorous designs) Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Sample sizes for Table 3 are smaller than those reported in other exhibits due to missing data. Table 4 Changes in support for subgrantees from SIF grantees and comparison groups To what extent did your organization do this ...? Group 2009 2014 Change A. Provided funding to SIF 3.0 3.9 0.9 (b+) carry out an evaluation (20) (b+) (20) or hire an external Non-selected 3.2 (9) 4.3 1.1 evaluator, as part of SIF (9) the grant or through applicants other means National 2.2 2.6 0.4 sample of (189) (189) grantmaking nonprofits B. Provided training or SIF 3.6 4.3 0.8 (b+) TA by your staff (20) (b+) (20) consultants or other Non-selected 3.7 (9) 3.9 0.2 means to conduct SIF (9) rigorous evaluation applicants National 2.5 2.9 0.4 sample of (189) (192) grantmaking nonprofits C. Provided training or SIF 4.4 5.0 0.6 TA by your staff (20) (b+) (20) consultants or other Non-selected 3.6 (9) 3.9 0.3 means to support SIF (9) implementation of the applicants program National 2.9 3.5 0.6 sample of (193) (193) grantmaking nonprofits To what extent did your organization do this ...? Effect Respondents size with change A. Provided funding to 0.5 8 carry out an evaluation or hire an external -0.1 4 evaluator, as part of the grant or through other means 1.3 41 B. Provided training or 0.4 7 TA by your staff consultants or other 0.5 4 means to conduct rigorous evaluation 0.9 48 C. Provided training or 0.3 7 TA by your staff consultants or other 0.3 3 means to support implementation of the program 0.1 64 1 Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 5 Changes in how SIF grantees and comparison groups approach evaluation To what extent did your organization do this...? Group 2009 2014 Conducted rigorous SIF 3.1 4.5 evaluations of programs (20) (20) funded by your Non-selected SIF 3.3 3.6 organization applicants (8) (8) National sample 3.4 4.1 of grantmaking (184) (185) nonprofits B. Used evaluation SIF 3.9 4.9 findings to improve (20) (20) programs funded by Non-selected SIF 5.1 5.4 your organization applicants (9) (9) National sample 3.6 4.3 of grantmaking (183) (184) nonprofits C. Used evaluation SIF 3.7 4.9 findings to demonstrate (20) (20) and communicate Non-selected SIF 4.9 5.7 effectiveness of applicants (9) (9) programs funded by National sample 3.8 4.6 your organization of grantmaking (181) (182) nonprofits To what extent did your organization do this...? Effect Respondents Change size with change Conducted rigorous 1.4 (a+ b+) 0.7 14 evaluations of programs funded by your 0.4 0.9 2 organization 0.7 1.9 62 B. Used evaluation 1.0 (b+) 0.5 12 findings to improve programs funded by 0.3 0.7 2 your organization 0.7 0.8 57 C. Used evaluation 1.2 (b+) 0.7 12 findings to demonstrate and communicate 0.8 0.3 3 effectiveness of programs funded by 0.8 0.8 63 your organization 1 Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 6 Changes in the evaluation infrastructure of SIF grantees and comparison groups Did/does your 2009 organization have Group n % (yes) the following? Staff position(s) or a SIF 20 50.0% (b+) group within your Non-selected SIF 8 50.0% organization that is applicants dedicated to National sample evaluation of grantmaking 187 30.0% nonprofits External evaluation SIF 20 70.0% (b+) partner(s)--consultant(s) Non-selected SIF 8 62.5% or applicants organization(s)--that National sample provide your of grantmaking 186 34.9% organization with nonprofits evaluation services Part of the SIF 20 75.0% (b+) organization's budget Non-selected SIF 8 62.5% is dedicated to applicants evaluation National sample of grantmaking 187 35.5% nonprofits Did/does your 2014 organization have n % (yes) Change the following? Staff position(s) or a 20 70.0% 20.0% (b+) group within your 8 75.0% 25.0% organization that is dedicated to evaluation 189 38.9% 7.7% External evaluation 20 95.0% 25.0% (b+) partner(s)--consultant(s) 8 87.5% 25.0% or organization(s)--that provide your 189 45.4% 9.6% organization with evaluation services Part of the 20 95.0% 20.0% (b+) organization's budget 8 87.5% 25.0% is dedicated to evaluation 189 47.2% 11.7% Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 7 Reported Changes in the evaluation budgets of SIF grantees and comparison groups Was your 2014 evaluation budget... than the evaluation About budget in S ubstantially Somewha the 2009? Group n higher higher same SIF (b+) 20 50.0% 20.0% 30.0% Non- 6 33.3% 50.0% 16.7% Total selected SIF evaluation applicants budget in National 160 16.7% 24.6% 57.5% dollars sample of grantmaking nonprofits SIF (b+) 20 30.0% 25.0% 35.0% Evaluation Non- 6 16.7% 33.3% 50.0% budget as a selected SIF percentage o applicants organization's National 156 12.9% 16.2% 62.5% total annual sample of budget grantmaking nonprofits Was your 2014 evaluation budget... than the evaluation budget in Somewhat Substantially 2009? lower lower 0.0% 0.0% 0.0% 0.0% Total evaluation budget in 0.4% 0.8% dollars 5.0% 5.0% Evaluation 0.0% 0.0% budget as a percentage o organization's 6.8% 1.6% total annual budget Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 8 Changes in the evaluation budgets of SIF grantees and comparison groups Did/does your 2009 organization have the Group n Mean Median following? Total annual SIF 10 $860,590 $170,450 evaluation Non-selected budget (in $) -- SIF applicants 1 $150,000 $150,000 for in-house National evaluators or sample of external grantmaking 66 $ 568,817 $0 partners nonprofits Annual SIF 13 4.1% (b+) 1.5% evaluation Non-selected budget as a SIF applicants 0 -- -- percentage of National the sample of 76 1.2% 0.0% organization's grantmaking total budget nonprofits Number of SIF 17 1.0 0.5 FTE staff with Non-selected primary SIF applicants 3 0.5 0.5 responsibility National for evaluation sample of grantmaking 101 1.8 0 nonprofits Did/does your 2014 Change organization have the n Mean Median following? Total annual 10 $1,789,800 $425,000 $939,344 evaluation budget (in $) -- 1 $200,000 $200,000 $50,000 for in-house evaluators or external 58 $1,115,810 $0 $43,195 partners Annual 13 5.8% 4.0% 1.7% (b+) evaluation budget as a 0 -- -- -- percentage of the 80 2.9% 1% 0.8% organization's total budget Number of 17 1.7 1 0.7 FTE staff with primary 3 1.3 1 0.8 responsibility for evaluation 104 3.1 .40 0.9 Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 9 Changes in approach to scaling up programs for SIF grantees and comparison groups To what extent did your organization do this ...? Group 2009 2014 A. Undertook efforts SIF 4.2 5.1 to scale up existing (19) (b+) (19) program(s)-- that is., Non-selected SIF 4.5 (8) 5.0 (8) to expand the applicants program(s) within National sample 3.1 3.9 the community or to of grantmaking (173) (174) other communities or nonprofits populations B. Selected programs SIF 3.5 (19) (b+) 4.6 for scale-up based on (19) rigorous evaluation Non-selected SIF 3.4 (8) 4.0 (8) that showed them to applicants be effective National sample 2.5 3.2 of grantmaking (170) (172) nonprofits To what extent did your organization do this ...? Effect Respondents Change size with change A. Undertook efforts 0.9 0.5 9 to scale up existing program(s)--that is., 0.5 0.4 3 to expand the program(s) within 0.8 0.3 79 the community or to other communities or populations B. Selected programs 1.1 (b+) 0.5 9 for scale-up based on rigorous evaluation 0.6 0.3 2 that showed them to be effective 0.8 0.6 53 (1) Statistical significance (p<.05) is noted as: a: significant difference between SIF and non-selected SIF applicants b: significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 10 Changes in approach to collaboration by SIF grantees and comparison groups To what extent did your organization do this...? Group 2009 2014 A. Participated in SIF 3.6 (20) (b+) 4.3 (20) funding alliance(s) with other nonprofit Non-selected 3.9 (8) 4.5 (8) sector organizations. SIF applicants (For example, co- National 3.1 3.7 funding programs sample of (172) (172) through joint funding; grantmaking providing or receiving nonprofits matching funds; or other collaboration) B. Participated in SIF 4.2 (20) 5.2 (20) collaborations with Non-selected 5.3 (8) 5.9 (8) other nonprofit SIF applicants organizations to share National 4.1 4.9 knowledge sample of (172) (172) grantmaking nonprofits C. Collaborated with SIF 3.5 (20) 4.1 (20) other organizations for Non-selected 4.0 (8) 4.8 (8) purposes of SIF applicants advocacy--to National 3.3 3.9 advocate for or sample of (169) (170) develop public support grantmaking for programs or nonprofits approaches to addressing social problems To what extent did your organization do this ...? Effect Respondents Change size with change A. Participated in 0.8 0.4 9 funding alliance(s) with other nonprofit 0.6 0.1 2 sector organizations. (For example, co- 0.7 0.2 66 funding programs through joint funding; providing or receiving matching funds; or other collaboration) B. Participated in 1.0 0.7 12 collaborations with 0.6 0.3 2 other nonprofit organizations to share 0.8 0.4 68 knowledge C. Collaborated with 0.6 0.3 8 other organizations for 0.8 -0.2 3 purposes of advocacy--to 0.7 -0.2 56 advocate for or develop public support for programs or approaches to addressing social problems (1) Statistical significance (p<.05) is noted as: (a:) significant difference between SIF and non-selected SIF applicants (b:) significant difference between SIF and the national sample of grantmaking nonprofits (+:) favoring SIF (-:) favoring the comparison group Table 11 Summary table for impact on SIF grantee organizational capacity Change in SIF grantees from 2009 to Organizational capacity 2014 Evaluation A. Conducted rigorous evaluations of programs [up arrow] funded by your organization B. Used evaluation findings to improve [up arrow] programs funded by your organization C. Used evaluation findings to demonstrate and [up arrow] communicate effectiveness of programs funded by your organization Support for subgrantees A. Provided funding to carry out an evaluation [up arrow] or hire an external evaluator, as part of the grant or through other means B. Provided training or TA by your staff [up arrow] consultants or other means to conduct rigorous evaluation C. Provided training or TA by your staff [up arrow] consultants or other means to support implementation of the program Scale up A. Undertook efforts to scale up existing [up arrow] program(s)--that is, to expand the program(s) within the community or to other communities or populations B. Selected programs for scale-up based on [up arrow] rigorous evaluation that shows them to be effective Collaboration A. Participated in funding alliance(s) with other [up arrow] nonprofit sector organizations (for example, co- funding programs through joint funding; providing or receiving matching funds; or other collaboration) B. Participated in collaborations with other [up arrow] nonprofit organizations to share knowledge C. Collaborated with other organizations for [up arrow] purposes of advocacy--to advocate for or develop public support for programs or approaches to addressing social problems Selection of subgrantees to fund A. Used an open, competitive process to solicit and review applications and to make selection [down arrow] decisions B. Required applicant organizations to provide evidence of intervention effectiveness to be [up arrow] eligible for funding (includes pre- and posttest outcome data or other evidence based on evaluation studies) C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible [up arrow] for funding (that is, quasi-experimental designs with a comparison group, experimental designs, or other similarly rigorous designs) Difference in changes: SIF 2010-2012 grantees vs. national sample of grantmaking Organizational capacity nonprofits Evaluation A. Conducted rigorous evaluations of programs +Large (*) funded by your organization B. Used evaluation findings to improve +Large (*) programs funded by your organization C. Used evaluation findings to demonstrate and +Large (*) communicate effectiveness of programs funded by your organization Support for subgrantees A. Provided funding to carry out an evaluation +Large (*) or hire an external evaluator, as part of the grant or through other means B. Provided training or TA by your staff +Large (*) consultants or other means to conduct rigorous evaluation C. Provided training or TA by your staff No consultants or other means to support implementation of the program Scale up A. Undertook efforts to scale up existing +Small program(s)--that is, to expand the program(s) within the community or to other communities or populations B. Selected programs for scale-up based on +Medium (*) rigorous evaluation that shows them to be effective Collaboration A. Participated in funding alliance(s) with other +Small nonprofit sector organizations (for example, co- funding programs through joint funding; providing or receiving matching funds; or other collaboration) B. Participated in collaborations with other +Small nonprofit organizations to share knowledge C. Collaborated with other organizations for No purposes of advocacy--to advocate for or develop public support for programs or approaches to addressing social problems Selection of subgrantees to fund A. Used an open, competitive process to solicit and review applications and to make selection -Medium (*) decisions B. Required applicant organizations to provide evidence of intervention effectiveness to be -Medium eligible for funding (includes pre- and posttest outcome data or other evidence based on evaluation studies) C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible -Small for funding (that is, quasi-experimental designs with a comparison group, experimental designs, or other similarly rigorous designs) Difference in changes: SIF 2010-2012 grantees vs. non-selected SIF Organizational capacity applicants Evaluation A. Conducted rigorous evaluations of programs +Large (*) funded by your organization B. Used evaluation findings to improve +Medium programs funded by your organization C. Used evaluation findings to demonstrate and +Small communicate effectiveness of programs funded by your organization Support for subgrantees A. Provided funding to carry out an evaluation No or hire an external evaluator, as part of the grant or through other means B. Provided training or TA by your staff +Small consultants or other means to conduct rigorous evaluation C. Provided training or TA by your staff +Small consultants or other means to support implementation of the program Scale up A. Undertook efforts to scale up existing +Small program(s)--that is, to expand the program(s) within the community or to other communities or populations B. Selected programs for scale-up based on +Small rigorous evaluation that shows them to be effective Collaboration A. Participated in funding alliance(s) with other No nonprofit sector organizations (for example, co- funding programs through joint funding; providing or receiving matching funds; or other collaboration) B. Participated in collaborations with other +Small nonprofit organizations to share knowledge C. Collaborated with other organizations for No purposes of advocacy--to advocate for or develop public support for programs or approaches to addressing social problems Selection of subgrantees to fund A. Used an open, competitive process to solicit and review applications and to make selection -Small decisions B. Required applicant organizations to provide evidence of intervention effectiveness to be +Small eligible for funding (includes pre- and posttest outcome data or other evidence based on evaluation studies) C. Required applicants to submit a plan for rigorous evaluation of intervention to be eligible +Small for funding (that is, quasi-experimental designs with a comparison group, experimental designs, or other similarly rigorous designs) ([up arrow]) SIF grantees experienced positive change ([down arrow]) SIF grantees experienced negative change (*) statistically significant difference at 0.05 level. A lack of significant difference between SIF grantees and non-selected SIF applicants is partly attributed to the small sample sizes. According to Cohen, large effect size at 0.8 or more; medium effect size at 0.5-0.79; small effect size at 0.2-0.49; no effect size smaller than 0.2. (+) difference favoring SIF grantee (-) difference favoring comparison group