Accountability and monitoring government in the digital era: Promise, realism and research for digital-era governance.

Lindquist, Evert A. ; Huse, Irene

Introduction

Furthering the accountability of elected governments and the public administration apparatus which serves them is a fundamental principle of democratic societies. It is understood that in the Canadian system of responsible government relying on Westminster principles, in exchange for assigning power to duly elected governments that elected representatives, various public and other watchdogs, and citizens have a right to scrutinize the allocation of public funds, whether they are consistent with statutory and other requirements, how well they are managed, and whether initiatives achieve intended results. Over the last fifty years, there have been significant debates about how to operationalize and balance the principles of accountability, particularly as governments and programs grew in size and as successive reforms added and emphasized new principles for governments to balance. These include--but are not confined to--open data and governance movements, such as the Open Government Partnership.' It is often said that digital technology, along with the culture and possibilities they produce, will transform societies and their governments; it is widely presumed that better services, improved governance, increased transparency, and more accountability will result. The emergence and proliferation of Web 2.0 capabilities (for example, cloud services, social media, wikis, open data, big data, data analytics, open-by-default initiatives, digital platforms and portals, collaboration and co-production, etc.) and advocates for their use in government has led to new rounds of experimentation, initiatives and reform under the banner of Government 2.0 in many jurisdictions inside and outside Canada.

The purpose of this article is to survey the Canadian and international literature on accountability in the digital era, to identify whether Canada is lagging or leading international contributions in this area, and to suggest lines of research. Canadian scholars have made important contributions to the literature on accountability (for example, Thomas 1998; Aucoin and Heintzman 2000; Aucoin and Jarvis 2005; Aucoin 2012), and a new generation of scholars are asking new questions (Clarke and Francoli 2014; Jarvis 2014a,2014b; Small 2012). However, the claims and changes wrought by the digital movement traverse governance domains and levels of analysis; a concerted effort is required to monitor and assess whether digital tools and approaches are reshaping governance and public administration. Many recent contributions to the literature on accountability in the digital era come from scholars in disciplines and professional schools with interests in information technology, transparency and digital culture, without deep backgrounds in governance and public administration, providing new perspectives, welcome granularity, and new possibilities for research initiatives. Digital-era initiatives arrived in the slipstream of the waning of the New Public Management (NPM) movement as the dominant challenger to traditional public administration and governance reform.

This article has five parts. Section 1 surveys the "public-administration-centered" literature which dealt with the topic of accountability until the digital era firmly took root. Section 2 reviews the "digital-centered" literature which focuses squarely on information and communications technology (ICT) and digital tools and related approaches, including research on transparency, open data and government, engagement, and participatory budgeting, with most authors presuming these improve accountability. Section 3 surveys ICT)-centered and public-administration-centered literature on whether digital tools have been improving accountability. Scholars seem to agree that, although much has changed with the advent of digital tools, many intervening factors are at play and democratic governance has not been transformed. Section 4 steps back to consider whether digital tools have fulfilled the expectations of many advocates or if findings and issues identified in the literature reflect enduring themes of public administration. Given that these streams of literature do not closely track and assess how digital tools work in accountability processes, Section 5 sets out a detailed research agenda to investigate the "New Accountability" in the digital era, inspired by Schillemans, Van Twist, and Vanhommerig (2013) identification of interactive, dynamic, and citizen-initiated accountability.

Public administration and accountability

Accountability is a well-worn subject, and justly so, because it is a crucial principle, expectation, and foundation of democratic governance systems. In such systems, governments and executive branches must account for the use of public funds and performance to elected representatives in legislatures and ultimately to citizens through the media, public debate, elections, the courts, and public inquiries (Gregory 2003; Aucoin and Jarvis 2005). This section provides a sense of key enduring themes where the literature stood as Canadian governments and civil society began to feel the initial bites (and bytes) of the digital era, and how digital tools and approaches initially promised to improve monitoring and accountability in our democratic system.

Several critical developments have affected the exercise of accountability in Canada. First, the policy movement of the 1960s and 1970s spawned a myriad of policy and evaluation units in public service institutions, and expanded mandates for central agencies and external auditors (Jordan and Sutherland 1979; Thomas 1979; French 1980; Prince and Chenier 1980). Second, the NPM reforms of the late 1980s and early 1990s focused on providing improving service to citizens and clients by increasing managerial flexibility in exchange for a focus on results and performance reporting (Aucoin 1995; Lindquist 1998). Third, the HRDC grants and contributions scandal (Good 2003) and the sponsorship affair (Gomery 2006) led to more rules and oversight of departments and agencies, including the adoption of the Government Accountability Act (2006), the introduction of accounting officer concept (Jarvis 2009), departmental audit committees (Shepherd 2011), and the creation of the Management Accountability Framework (Lindquist 2009). Finally, the number of internal and external watchdogs monitoring government activities has proliferated (Good 2014). By the mid-2000s, even without factoring in the effects of the Global Financial Crisis and the proliferation of new digital tools, it was a considerably more difficult environment for governments and public servants to navigate (Clark and Swain 2005).

Many years ago, accountability was comprised of a relatively simple principle of answerability, focused on ministers holding executive authority in hierarchical Westminster government contexts. Key issues emerged around the personal responsibility of ministers versus answering, and even resigning, for the actions and inactions of their departments and officials. Reflecting more complex governance environments and successive reform movements, the literature has identified various sites, instruments, and functions of accountability, with a proliferation of analytic frameworks and typologies. Romzek and Ingraham (2000) identified different modalities and key principles for accountability (hierarchical, legal, professional, political, and internal and external expectations), while Mulgan (2003) reminded us that personal accountability of office-holders remains important. Aucoin and Heintzman (2000) pointed to three functions of accountability (control, assurance, and continuous improvement or learning), later complemented by Bovens, Schillemans, and Goodin's (2008) distillation of three perspectives on public accountability based on a literature survey (democratic, constitutional, learning) and concerns about accountability deficits, costs and overloads (see also Bovens 2005, 2007). Peters (2014) identified distinctive accountability instruments: hierarchy, mutuality, competition, and contrived randomness. In reflecting on the evolving balance between individual and collective responsibility in Canada's Westminster system, Brown (2013) identified four "rooms" where accountability is exercised: policy (middle room), service delivery (front room), corporate services (back room), and advertising and communications.

Well-known reform public administration movements have had implications for accountability. Some observers see NPM reforms (2) as creating new avenues for accountability between service providers to clients and service recipients, which is different from traditional and post-NPM modalities for accountability (Laegreid 2014; Christensen and Laegreid 2015). But this did not eliminate the exposure on ministers who, without as much control, were still held accountable for issues and poor performance for the actions of staff and other service providers. Governments responded with more control of inputs and processes, one of the key attributes of Aucoin's (2012) "New Political Governance" (see Harris 2015; Page 2015). Growing interest in more collaborative, citizen-oriented, tailored, distributed and network governance and service delivery models--labeled as the New Public Governance (Osborne 2010)--has led to more interest in "horizontal accountability" for initiatives working across boundaries within and across levels of government, and with different sectors (Howard and Phillips 2012; Conteh 2016). Margetts and Dunleavy (2010) suggested that digital-era governance was replacing the NPM approaches but arguably the digital capabilities they pointed to could be seen as fueling all of these approaches, including traditional and NPM approaches.

Since none of these reform initiatives disappear, layers of accountability requirements accrete, resulting in competing values and cross-pressures on governments (Considine 2002; Mulgan 2000). Concerns have arisen about the costs of reporting to overseers and watchdogs (Good 2014), the opaqueness of reporting, contending accountability values or "multiple-accountability disorder" (Koppell 2005; Romzek and Ingraham 2000), accountability overloads (Halachmi 2014), the aggressiveness of media exacerbated by social media and quickly spreading news, and blurry accountabilities among ministers and officials (Aucoin and Jarvis 2005; Jarvis 2009). Moreover, increased reporting has not necessarily led to more accountability or better organizational performance; rather, it might crowd-out other forms of assessment such as evaluation and reduce the ability of legislatures to hold the executive to account (Dubnick 2005; Van de Walle and Cornelissen 2014). Despite much discourse looking beyond conventional vertical accountability, Jarvis (2014a, 2014b) usefully reminds us that "internal" accountability in government hierarchies between executives and front-line delivery staff and contractors (not between ministers and top executives) is an important but very under-studied domain. Howard and Phillips (2012) suggest that Westminster's vertical, hierarchical lines of accountability are not necessarily incongruent with diagonal and horizontal accountability.

It is into this mix of increasing demand for accountability and proliferation of instruments that modern governance systems are encountering digital innovations and expectations. Many governments have committed to becoming more "open" by sharing banks of data, increasing transparency, releasing information sooner, and being "open-by-default," often buttressed by joining the Open Government Partnership. Persistent calls emanate from inside and outside government for more data and lines of evidence to inform policy design and accountability, along with better real-time monitoring of policy implementation. Many observers are excited about more sophisticated ways of showing how government really works, and the many actors and capabilities involved in delivering policy and services (Lindquist 2015). Digital sensors, social media, and video streaming hold promise for better monitoring and oversight inside core government and across service delivery chains, and relying on citizens and outside experts. However, few public administration scholars have systematically explored how digital tools affect accountability and implications for Westminster systems.

Transparency, open data and government, engagement, and participatory budgeting

The international literature on ICTs and digital tools for government does not map easily onto how the public administration literature approaches accountability, despite overlaps and promising new approaches. Contributors with digital as a point of departure tend to focus on open data and social media, the extent of government transparency, and the quality and relevance of data for accountability. Other contributors explore how transparency can further public engagement to improve budgeting and reduce corruption, the use of social media, and whether open data and transparency have led to more citizen participation and creating more public value.

Transparency as foundation

One stream of literature focuses on the transparency of governments, with empirical research focusing on the content, accessibility and usability of their web sites. It presumes transparency is foundational for accountability. Piotrowski (2007) suggested transparency allows the public to better assess how well governments and their many parts perform, and to hold them to account.

Empirical work tends to operationalize "transparency" as the amount of data and information on government web sites. Baseline studies set out a method for measuring the openness of web sites (Demchak, Friis, and La Porte 2000; La Porte, de Jong, and Demchak 2002). Subsequent studies explored the links with other features of countries and governance systems. Chadwick and May (2003) reviewed US, UK, and EU government web sites, suggesting that most are non-interactive and non-deliberative. Wong and Welch (2004: 291) compared the openness of government web sites across fourteen countries (3) and agency types, concluding that transparency and public accountability are generally increasing, with country and agency differences. Jang, Cho, and Drori (2014) look at UN Statistical Yearbooks to explore the amount of national accounts data supplied by member countries, driven initially by economic factors, and later political ones. The findings suggest that one driver for transparency can be "functional imperatives" (furthering trust, prosperity, stability) and others are "transnational norms" from others governments, international organizations, and professions, stemming "from a cultural model of proper governance" (pp. 97-98). Some studies examine how web sites and social media might discourage corruption with more transparency (Bertot, Jaeger, and Grimes 2010; Andersen 2009; Bhatnagar 2003). Cucciniello, Nasi, and Valotti (2012) collected data on citizen preferences about the information provided on Italian local government web sites about day care services, revealing that governments provided institutional and political information, not the financial and service-delivery information sought by citizens.

Many authors explore what goals increased transparency serves in the digital era. Chadwick and May (2003) suggest that the ICT policy statements of governments might have three possible functions: improving service delivery and policy; better informing government policies; and improving deliberative democracy. They saw the first two as traditional government modalities and deliberation as more complex and multi-directional, not receiving as much attention from national and state governments, and more from state and local governments (p. 294). Koppell (2005) coined the concept of "multiple accountabilities disorder" (MAD) deriving from five competing dimensions of accountability--transparency, liability (or culpability), controllability, responsibility, and responsiveness--public organizations must meet simultaneously. Several contributors identify long lists of values associated with transparency, open government, and accountability (Halachmi and Greiling 2013; Bannister and Connolly 2014; Heald 2003, 2012; Abu-Shanab 2015), agreeing that it is difficult to simultaneously achieve what Heald (2012) called "symmetric transparency." Heald (2003, 2012) astutely observes that with so many values, goals and actors involved in governance, developing transparency regimes in the digital era requires balancing competing values, with the significant tensions among efficiency, equity, and democratic accountability. Many of these values are variously aligned with the traditional, NPM and post-NPM approaches such as the New Public Governance (Osborne 2010) and Public Value (Moore 1995) formulations with differing emphasis on whether transparency is to serve: ministers, legislatures or international organizations; clients and customers receiving public services; or citizens and democracy.

While transparency is seen as essential for accountability, little work explores its effectiveness in fostering accountability, but arrives at conclusions based on the information on government web sites. Early on, Chadwick and May (2003) concluded that government portals did not seem to further interaction or deliberation. Welch and Wong (2004) suggested that one could not hold governments to account with such information; later Jang, Cho, and Drori (2014) argued that functionally-based reporting may not be connected to quality of governance practice, similar to Heald's (2012) distinction between nominal and effective transparency. Mclvor, McHugh, and Cadden (2002) identified many barriers to furthering transparency, which requires cultivating the right skills and broader culture change in governing systems (see also Aman, Al-Shbail, and Mohammed 2013). Chadwick and May (2003) also noted that increasing transparency for service delivery implies citizens developing new and more direct connections with government, which would blur the lines between the accountability relationships between the executive and legislatures, and raise its own accountability and other issues (p. 293), which Laegreid (2014) suggests holds more generally for NPM reforms. Looking more broadly, Wong and Welch (2004) concluded that, despite ICTs increasing levels of transparency, the prevailing patterns of power persisted between the government and citizens, as between political leaders and officials. Halachmi and Greiling (2013) suggest that operationalizing transparency and accountability as concepts and variables is complicated because they are situated in broader institutional contexts, and other values and capabilities are required to further transparency and accountability.

More generally, many observers worry that dysfunction can arise (Bannister and Connolly 2011). Some argue that too much transparency and sharing of data and information can lead to overload, increased costs, and goal displacement for the government agencies supplying it (e.g. Halachmi 2011). This echoes Wildavsky's (1972) seminal thought experiment about the perils of the "self-evaluating" organization (see also Koppell 2005). Sunstein (2016), while calling for "output transparency" worries that the increased focus on "input transparency" by open government advocates will lead to a focus on whether advisors are aligned and increase the risks of providing advice to policy-makers, displacing attention on the merits of arguments and evidence and eventual decisions. Heald (2012) suggested that accountability is no longer limited to the release of reports, pointing to retrospective and real-time transparency, which creates a stream of accountability demands rather than episodic moments. Finally, Halachmi and Greiling (2013) note that increased transparency will expose governments to more public criticism, which will not necessarily increase trust or confidence in them, and argue that digital-era accountability will markedly change as citizens move from more passive to active stances.

Looking forward, researchers suggest that multi-faceted approaches are required for transparency to increase accountability and its effectiveness in the digital era. Fung, Graham, and Weil (2007) called for "targeted transparency" and mandating disclosure. Heald (2003, 2012) calls instead for an optimal mix of information for budget oversight to address the vertical, horizontal, and external needs for oversight and accountability. Bertot, Jaeger, and Grimes (2010: 267) suggest that cultivating "a sustained culture of transparency" involves not only better designing government web sites, ensuring the right mix of information, and increasing government and bureaucratic acceptance, but also increasing accessibility, empowering citizens and groups, building trust and social capital, and increasing literacy. This broader orientation is more consistent with the values of furthering deliberative democracy as well as Public Value and New Public Governance perspectives.

Open government data and open governance initiatives

Yu and Robinson (2012) urge caution in equating open data with open government, noting that each can exist without the other, and that the term "open data" is vague and ambiguous. For many, improving government transparency invokes broader discussions of freedom of the press and freedom-of-information legislation (Sunstein 2016). Here the focus is narrower, on the supply of data sets more generally and information on specific administrative or program areas, which could inform the monitoring and accountability of government (Piotrowski 2007; Dawes and Helbig 2010). Open data initiatives are less tied to particular programs and geared more to encouraging governments share more data for many different purposes. Open data initiatives involve releasing myriad data sets while open government initiatives focus more on open-data portals and information-sharing frameworks but can involve more (Clarke and Francoli 2014).

Several studies have sought to ascertain what goals open data and portal initiatives further. Pina, Torres, and Acerete (2007) and Pina, Torres, and Roya (2010) reviewed, respectively, the open-data initiatives of 19 OECD nations and the open-data web sites of 75 local governments in Europe to ascertain which accountability goals the governments appeared to further: financial accountability, transparency, or citizen engagement and dialogue. They concluded that typically governments sought to satisfy the minimum legal or statutory requirements for posting data and tended to communicate "out" - this information and data, though increasing transparency and availability of information for citizens, had little impact on public participation. Orelli, Padovani, and Scorsone (2010) reviewed the action plans of ten European countries about how ICTs could achieve inclusion, efficiency and effectiveness, impact, enabling and connections among IT systems, and e-participation. (4) Similar to Pina, Torres, and Acerete (2007) and Pina, Torres, and Roya (2010) they concluded that most governments focused on increasing efficiency and effectiveness, and relied more on traditional political accountability, rather than enhancing engagement and inclusion with better information platforms and portals and deliberative tools. In contrast to the broader themes associated with accountability, Lassinantti (2014) and Bedini et al. (2014) saw great possibility in the potential of open data for "innovation and value creation" by firms, non-profit organizations, and others outside government. Few studies rely on surveys or interviews of users and citizens, or case-studies of how citizens acquire information or entrepreneurs access data. (5)

Some work considers the quality of open data portals and their data sets. Clarke and Margetts (2014) point out that open data portals can contain both good and bad data (the latter not being reviewed for quality or relevance), and much relevant data may have not been released. Lnenicka (2015) created an assessment framework on open data and portals based on a literature review, suggesting that releasing data does not equate to more transparency or increased accountability. Lourenco (2015: 326) identified the characteristics of open data portals (6) in six countries (Australia, Canada, France, Singapore, UK, US) and concluded many portals were not designed to increase transparency or accountability, but rather, geared more to creating public value, commercialization, and innovation by external users (p. 331). This empirical work highlights the Yu and Robinson's (2012) distinction between using data to further service delivery or public accountability, and their call not to confuse open data initiatives with ensuring that government is indeed more politically accountable. Castro and Korte (2015) reviewed the amount and quality of information on G8 government portals according to Open Data Charter principles, and although they did not survey user communities or the potential of the data for improving accountability, they were highly critical of the meta-data supplied. Peled and Karine (2015), after comparing UK and US open-data initiatives, complained that governments were offering only high-level informational meta-data, insufficient for improving accountability, with potential users "swimming helplessly and without aim in a vast ocean of meaningless data" (p. 10).

This literature considers how open government data initiatives could be improved to increase accountability, among other objectives. Intriguingly, Peled and Karine (2015) observe that increasing effectiveness may require releasing less data and information, thinking more carefully about what citizens and others seek to hold governments accountable for, and identifying what other information need to be linked in order to "effectively link disparate OGD assets" (pp.10-11). This squares with Perrin (2015) who points to many reasons for why performance, evaluation and accountability indicators are insufficient or counter-productive for increasing accountability, and that supplying data and indicators may well increase accountability but not increase performance. In an OECD working paper, Ubaldi (2013) suggested the supply of open data is not enough for increasing transparency and economic and public value, which will also require empowering and increasing the capacity of ecosystems of external users and intermediaries, which includes data scientists, data journalists, scholars, and firms and non-profits working in this niche. (7) Likewise, Roy (2014) suggests creating a "national architecture" for open data and related initiatives and fostering more collaboration and participation in the governance ecosystem. It would facilitate and lever social media, allow for sharing information in new ways, foster social and public activism, and potentially lead to integrated perspectives on fragmented government service delivery and policy development systems (p. 422).

Engaging citizens: social media, deliberation, and open budgeting

There has been no shortage of instruments available for engaging citizens, but governments have typically relied on Question Period, legislative hearings, public enquiries, the work of auditors general and other watchdogs, court challenges, and media pressure. Governments are less likely to engage citizens for accountability. That said, digital tools can bring issues and information to the attention of governments and the public far more quickly, richly, and at lower cost. What follows reviews literature on how digital advances such as social media, new deliberative tools, and enhanced open budgeting might increase accountability in government.

Deliberation

The idea of increasing citizen engagement is certainly not new. Most recently, Nabatchi (2010) made an appeal to governments and public administrators to increase citizen engagement and use more deliberative approaches. Many new digital tools (for example, social media, engagement platforms, crowd-sourcing ideas, etc.) could be added to the arsenal of existing deliberative approaches, or older approaches could be digitally enhanced when information-gathering, adding online forums, relying on streaming video, or issuing reports for comment (Lindner and Riehm 2009; Aitamurto 2012; Swift 2013; Dutil 2015). Here we are interested in whether new digital and digitally-enhanced engagement models are taken up for accountability or monitoring government. As with the term "open government" (Clarke and Francoli 2014), the terminology about digitally enhanced citizen engagement can be quite elastic, with open budgeting, citizen e-petitions, and open dialogues all counting as crowd-sourcing (Aitamurto 2012; Roy 2012a, 2012b, 2012c), while others would not necessarily see these as "deliberative" (for example, Dutil 2015).

Digital tools have the potential to more richly or widely engage citizens. These include obtaining input on initiatives and policies with citizen advisors, panels or assemblies, or on the costs and benefits of programs. For example, Lindner and Riehm (2009) point to the increasing use of e-petitions (for example, Scotland, Queensland, Germany, and many others) to express citizen frustrations or ideas on new policy initiatives, whether initiated by government or not. Swift (2013) shows how the People's Lobby process uses an online platform and structured process to identify issues, winnow them, select a monitoring group, develop suggestions, and monitor government take-up of those ideas (like a citizen's assembly filtering which issues to pursue). Papaloi and Gouscos (2013) review the opportunities for information visualization in legislatures to increase openness and accountability. Longo (2017, this collection) surveys several approaches, but there is endless variety, since digital tools can be used in combination with a variety of long-standing engagement and deliberative approaches. Many advocates and observers cast digital and digitally-enhanced engagement as increasing transparency and accountability. Most examples focus on informing the upstream of the policy cycle--such as identifying issues, moving issues up the public and government agenda, indicating preferences on policies or delivery models, and then making decisions about new programs, policies or service delivery models. This is different from monitoring performance and holding governments to account about program, policy or service delivery choices (Halachmi and Holzer 2010). Papaloi and Gouscos (2013) suggest that downstream information visualization and information disclosure can enhance accountability, without supplying examples.

However, early reviews of digital tools for engagement are not positive. Johnston (2010) reviewed the surge of Obama administration and UK Open Government Initiative engagement exercises but argues they were "one-off and superficial" (p. 163). More generally, Hartz-Karp and Sullivan (2014) suggest there has been little take-up of new digital engagement technologies due to the challenges of "scaling up" and "scaling out," wrestling with complexity, and ensuring genuine participation and dialogue in online modalities. They note:

The inherent nature of online deliberation, with its typically asynchronous environment, is not conducive to intensive, empathetic, collaborative discourse. Rather, it is conducive to direct democracy that merely aggregates the unreflective opinions of self-selected voters, conveniently weighing in on every issue. Unfortunately we can't achieve nuanced solutions to contentious issues by aggregating opinions (p. 2).

They suggest digital tools should be reserved for collecting views, which could inform more genuine deliberation. Some policy domains have little, if any, meaningful oversight in the digital era (Roy 2016, suggests that "participative bodies" review Canada's security and intelligence activities). Finally, despite the marketing of open government initiatives by higher levels of government, municipal governments have experimented more with open data, social media and other digital tools for deliberation, building budgets, and encouraging development of apps (for example, Moon 2002; Norris and Moon 2005; Gruzd and Roy 2016; Roy 2016).

Social media

Social media rapidly joined the portfolio of communications tools of governments, political parties, media organizations, and many other non-governmental entities for announcing events, initiatives, and other developments. Conversely, most governments assiduously monitor social media, identifying when citizens, politicians and groups identify issues or failures of government departments and agencies, contracted providers, or regulated non-governmental organizations. If they do not monitor social media, governments will soon learn about issues from individuals and organizations that do. In this sense, social media is used for accountability, and undoubtedly accelerates the speed at which failures and issues move higher on the public agenda, increasing the rate and volume of "naming and shamin" in the public domain, and sharing notice of when key accountability sessions will occur. Less clear is whether social media facilitates more thorough monitoring and accountability.

Some see social media as potentially increasing the openness and accountability of governments through increased engagement (Stamati, Papadopoulos, and Anagnostopoulos 2015). Bertot, Jaeger, and Grimes (2012) review the potential of social media (blogs, wikis, social networking, media sharing, and micro-blogging), but focus more on anti-corruption, fostering direct interactions between service recipients and service providers, and collaborative and crowd-sourcing approaches for solutions to problems. Gruzd and Roy (2016) note local government politicians and public administrators use social media for diverse purposes like communication, citizen engagement, community-building and mobilization, different service models and apps, participatory budgeting, etc. (8) However, the literature suggests that, other than to share information on issues and events, deliberative space has not been expanded. Small (2012) reports that Canadian departments and agencies use Twitter more as a communications tool (with the feeds not "followed" by many citizens) and really not for improving service delivery or fostering citizen participation. In their study of social media use by Canadian, Australian and US governments Deschamps, McNutt, and Zhu (2012) concluded that "Despite some experimentation by the public sector, the use of Web technologies to enhance collaborative interaction between government, stakeholders and citizens remains limited" (p. 1). Roy (2014) reviews examples of engagement initiatives (mainly municipal), identifying clashes between traditional, topdown, control-oriented approach with Gov 2.0 themes of more social-media-drive collaboration, social participation, and public activism.

Open and participatory budgeting and ICTs

The open budgeting movement embraces not only the disclosure of budgetary data and information by governments in developing and developed jurisdictions to monitor expenditures and outcomes (Justice, Melitski, and Smith 2006; Carlitz 2013; Hudson 2014), but also opportunities for citizens and communities to review and provide input on budget priorities. As exemplified by the Open Government Partnership, this movement tends to focus on budget transparency (Harrison and Sayogo 2013, 2014) and seeks to, directly or indirectly, encourage citizen participation and government accountability (Khagram, Fung, and de Renzio 2013). An enormous literature on participatory budgeting has emerged over twenty-five years, reflecting over 1500 initiatives around the world, mainly by local governments (Baiocchi and Ganuza 2014; Dias 2014). Studies have reviewed specific and groups of participatory budget exercises, including Canada (Secondo and Jennings 2014), which along with the US is seen as a late adopter. Guelph and Edmonton stand out as early Canadian adopters (Pinnington, Lerner, and Schugurensky 2009; Mao and Adria 2013). (9) Influence typically obtains when focusing on more specific budget priorities and conventional power-wielders hold sway (Masser 2013). The sharing of data and information on the allocation of budgets, particularly if it is more granular, allows for better questions and more accountability.

Recently researchers have explored whether digital tools further or undermine participatory budgeting. Some advocates see the possibility of reaching out to reach more citizens, while others believe it does not encourage face-to-face interaction and the building of trust crucial to deliberation. The latter was confirmed by Barros and Sampaio's (2016) case study. (10) However, Allegretti (2012) and Sampaio and Peixoto (2014) argue that digital tools can complement the more intensive, face-to-face deliberations associated with participatory budgeting: recruiting participants, sharing data and information with participants, providing more tactile opportunities to participants to work through budget scenarios with simulations, and sharing information with the public as they deliberate.

Many governments are investing in more attractive and substantial portals which use data and different visualization software to allow the public to delve into and play with budgets, often at different levels of analysis (Kayser-Bril 2016). Examples include the portal of Edmonton, New York City, and Oakland. (11) Firms and associations have been springing up to provide the software to facilitate initiatives (e.g. Open North in Canada and OpenBudgets.eu in Europe). (12) Edmonton's Open Government portal contains dashboard and information analytics on almost all aspects of amenity, services, and performance. It provides a foundation for accountability on the use of allocated budgets in many areas, ranging from crime to economic indicators, using census, geo-spatial, and other data. (13)

Have digital tools improved accountability and democratic governance?

Relatively little work in the digital-centred and public-administration-centred literature has empirically explored whether ICTs and digital tools have improved accountability. However, promising work from both streams point to a similar empirical research agenda.

Several observers predicted that ICTs would not transform democratic governance, but rather, reinforce existing patterns in governance and provide governments with more tools to monitor and control citizens (Kraemer and Dedrick 1997; Welch and Wong 1998; Margolis and Resnick 2000; Weare 2002). After reviewing government web sites, Wong and Welch (2004) concluded that even though ICTs and Web 2.0 digital tools can improve service delivery, availability of information, citizen engagement, open governance, and accountability, there remains divergence in take-up across jurisdictions reflecting existing patterns in governance. They suggest that there will be greater use and impact of ICTs and digital tools in more open societies and governance systems. Likewise, after analyzing high-level indexes of e-government, democracy and integrity for forty diverse developed and developing countries, Islam and Gronlund (2012) conclude these variables move together and together increase accountability. Lacharite (2011) argues that the promise of digital democracy in Canada has not been realized; existing patterns in power and authority in governance have been reinforced. In his survey of the state of Canada's Westminster system, Brown (2013) argues that ICTs will reinforce the long ensconced tendency towards collective cabinet responsibility over ministerial responsibility, in part because of the challenging media environment, the need to create integrative narratives across the waterfront of government, and the power of the Prime Minister. Roy (2016) provides a thorough review and assessment of the extent of transparency and oversight of Canadian security and intelligence agencies, leading to insufficient accountability. These authors concede that digital tools have led to nontrivial change, but agree most countries have not moved to new thresholds of democratic governance.

Many authors in the digital-centred and public-administration-centred literature are certainly interested in how accountability is evolving in the digital era--pointing to new constraints, pressures, and flows of information--but they do not systematically explore how digital tools complicate, amplify or mitigate the broader trends they identify. Nor do they specifically identify how ICTs have permeated internal and external work environments, generating new demands on elected leaders and public servants, or new ways of sharing of information or complaints (for example, Bovens, Schillemans, and Goodin 2008; Jarvis 2014a, 2014b; Halachmi 2014). While Jarvis (2013) calls for more investigation of how the "black box" of internal accountability has evolved, there is no specification of how digital monitoring systems might be used by executive teams and central agencies; likewise, Halachmi (2014) invokes the useful concept of "accountability overloads" but does not systematically produce data or examples of different sources and efforts to address overload. Even Koppell's well-known (2005) study on how ICANN (the domain-name organization) dealt simultaneously with five competing accountability values did not explore the effect of ICTs. Aman, Al-Shbail, and Mohammed (2013) looked squarely at how e-government tools can enhance accountability, outlining a framework comprised of positive pathways, but without empirical analysis or identification of barriers. No studies have systematically reviewed possibilities improving accountability with the expanded menu of digitally-enhanced visual tools beyond normal charts, tables, and pictures (Davison 2014), such as dashboards, real-time data, data analytics, interactive data and gaming sets, etc. (see Lindquist 2015, 2018 forthcoming).

On the other hand, Bovens and Zouridis (2002) explored two case studies and argued that ICTs further system-level monitoring of service delivery, which reduces the administrative discretion of street-level bureaucracies and increases central control. Schillemans, Van Twist, and Vanhommerig (2013: 408), while noting that there can be costs and benefits to accountability, observed that the "critical undercurrent of the debate on public accountability, spurred on by changes in society and by technological innovations, has resulted in the creation of an abundance of new, alternative, mechanisms of accountability." Like Behn (2001, 2014a, 2014b), they point to the emergence of real-time reporting, learning and accountability. They see the internet, social media, mobile platforms, and other streams of data and information complementing traditional ways of exercising accountability. Schillemans, Van Twist, and Vanhommerig (2013) see three forms of real-time accountability:

* interactive accountability (among central officials and delivery partners using repertoires like deliverology, Compstat, and PerformanceStat that rely on monitoring and continual adjustment);

* dynamic accountability (essentially co-design and co-implementation with external stakeholders such as FixMyStreet and Recovery.gov); and

* citizen-initiated accountability (relying on their own data for accountability and engaging when it suits them).

Vanhommerig and Karre (2014) further argue that citizens will engage only as issues affect their interests, rather than continuously monitor government (pp. 206-07), leading to random "flare ups" along with regular government reports for the purposes of accountability. Moreover, since citizen-initiated accountability relies on other sources of data regardless of quality, they suggest governments will lose control of the data used for accountability (that is, not based on reports from government or auditors general) (p. 213). They take Bovens and Zouridis (2002) a step further, suggesting that sharing detailed data on local providers of services by higher order governments in the name of transparency can be seen as a "blame-shifting" strategy (p. 215).

Despite exploring how ICTs and digital tools might further government accountability in the digital era, some authors similarly conclude that the new tools on their own--or even in the hands of certain actors--will not make a significant difference. In their extensive literature review, Sandoval-Almazan and Gil-Garcia (2015) first suggest looking at government use of ICTs and web sites, and exploring information availability, transparency, open data, legal frameworks, and accountability, (14) but they also call for increased attention on participation, collaboration, and coproduction (reflecting post-NPM and New Public Governance themes), along with a more dynamic model of how all this might work for citizens (p. 177).In budget-related transparency initiatives, many of which rely on digital tools, Carlitz (2013) pointed to broader conditions such as fostering vertical and horizontal alliances, providing good data and information, benefitting from international support, and legal pressure.

Others more explicitly point to a distributed governance approach. Lips (2012) considers whether the initial service delivery focus of e-government initiatives have been superseded by later themes about working across internal government boundaries, with other governments and sectors, and with citizens to design policy and deliver services. She concluded that both models are in play, constituting a new model (Public Administration 2.0) which she sees as "dynamic, unpredictable, complex, and non-linear." Lips argues improving government will be about applying new technologies and changing the relationships among institutions and governance regimes. When Brown (2013) looked at different "rooms" for government business, he was essentially suggesting that, prime-ministerial power aside, the take-up of digital tools and approaches by governments was a distributed endeavour. Francoli (2014) chronicled how national governments like Canada's have signed onto the Open Government Partnership, an international movement requiring signatories to meet certain standards for reporting and share information about progress and best practices. Smith, Noorman, and Martin (2010), having explored the competing values and multiple directions for accountability, called for information architectures which allow for diverse data and information to be shared across levels and organizations inside government. Similarly, Roy (2016) proposes that information architecture should be designed at the national level to coordinate work and flows across governments and sectors. Together, these observations suggest that improving government accountability in the digital era requires contributions from diverse institutions as well as evolution in practice at different levels of analysis, echoing Bertot, Jaeger, and Grimes (2010) notion of building a "sustained culture of transparency."

Accountability in the digital era: transformation or enduring tensions?

Digital technology has led to a considerably different operating environment for governments and their elected and public service leaders. Vastly more data and information is collected, and the insights and issues they reveal can be circulated inside government and civil society ever more quickly, creating continuous pressure. Digitally-fueled developments in transparency, open data and portals, engagement, and information-sharing have been taking place, but governments have selectively embraced these tools, and not yet widely embraced them to move governance and accountability to new thresholds. Governments have tended to meet minimum thresholds for legal requirements and external optics, using new technologies to monitor the external environment, communicate messages, increase control over service delivery systems inside and outside government, and electronically elicit ideas and issues from citizens, which is not the same thing as opening up deliberative space. Observers have concluded that new supports will have to be put in place in order to better engage an expanding diverse set of actors inside and outside government. New forms of digital monitoring, information-sharing, and information display still hold considerable promise for ministers, legislatures, and citizens, but it will require political leadership and exemplars to show the way.

Although the accountability environment has changed dramatically, with more accountability-holders having greater access to more information, there is little evidence that digital tools have improved the amount and quality of oversight. Despite the open data and open government movement, governments remain remarkably opaque. Indeed, the proliferation of reporting requirements (internal and external) may be creating overloads on governments to supply data and information and, conversely, insufficient capacity on the part of non-governmental actors to use and analyze it. Public service institutions are arguably whiplashed among the expectations of ministers, citizens, and firms with unrelenting demands: ministers want the public service to become more innovative and efficient; the media and observers want the government to become more transparent but the "naming and shaming" impulse rules and digital records proliferate; and citizens, firms and other actors continue to have increasing expectations about the channels and quality of the service they receive. In short, despite the abiding promise of digital technology, and without imagining a governance nirvana, we are still in the early days of taking up digital tools and spreading around the benefits to all actors inside and outside governments.

This review has surfaced several enduring themes and tensions. First, there is the gap between the expectations for new digital technologies and governance realities. Recall that the promise of the social sciences and the policy analysis movement during the 1960s and 1970s was bolstered by access to data and new statistical and computing techniques: they changed the face of government with a profusion of policy capabilities, but did not "solve" problems and challenges (Radin 2000; French 1980). Likewise, the reforms associated with the NPM movement--intended to improve reporting to legislatures, citizens, and other actors served by government--fell short of expectations, with many observers arguing that accountability has not improved and government has become more opaque. Second, governments have long sought to look au courant and transparent--now by using social media and adopting open data and open government initiatives but do so in limited ways, attempting to preserve power and control the agenda. Third, it has long been possible for governments to empower legislatures and the public with more accessible and pertinent data, and more engagement and deliberative opportunities; the reluctance to share better information and expand deliberative space in the digital era is not new. However, the ease with which data and information can be assembled and move inside and outside organizations is new and potentially disruptive, leading to the phenomena of interactive, dynamic, and citizen-driven accountability identified by Schillemans, Van Twist, and Vanhommerig (2013), which deserves further, deeper investigation inside and outside government.

A research agenda for the "new accountability" in the digital era

The literature on accountability in the digital era seems to be converging: the digital-centred stream has generally concluded that institutions, national traditions, and broader networks of actors are critical for constraining and furthering the possibilities of digital tools for improving accountability; conversely, the public-administration-centred stream, though well aware of the pressures generated by various digital tools and pointing to their general effects on governments and stakeholders, is only starting to develop concepts and undertake empirical work about how those effects obtain. Canadian researchers have taken up key themes and stand among leading contributors to the international literature. With a few exceptions (Small 2012; Mao and Adria 2013; Gruzd and Roy 2016), Canadian researchers have not undertaken systematic quantitative research but have conceptualized and reflected on accountability, and chronicled and appraised key initiatives and different stresses on governments (for example, Brown 2013; Francoli 2014; Clarke and Francoli 2014; Jarvis 2014a, 2014b; Clarke and Margetts 2014 and Roy 2014-16). This suggests we have the base for a promising program of research over the next few years, which might include these themes and initiatives:

1. Using accountability information. The literature has examined open data portals, but not undertaken longitudinal case studies of external monitoring of the policy and program performance of governments, including the supply of reports and data by government and watchdogs, the "chatter" in social media and more systematic monitoring by the ecology of individuals and groups outside government, and how issues spring onto the agenda. This should shed light on the balance between episodic reporting and continuous real-time monitoring, both elements of dynamic and citizen accountability. Parallel research could focus on the needs of citizens, legislators, journalists, and other stakeholders as users.

2. Supplying accountability information. Research here could focus on the three levels: (1) monitoring the efforts of Canadian governments to access and link administrative and external data for monitoring the performance of policies and program; (2) monitoring how much of this is shared with watch-dogs, Parliament, and the public through open data portals and if such information allows for government to meaningfully be heldto account; and (3) monitoring the roll-out of the Government of Canada's revised Policy on Results and reporting regime and whether its version of "deliverology" (Lindquist 2016) furthers external accountability with legislators and citizens, or is used for internal purposes.

3. Studying participatory budgeting and performance portals. Many Canadian municipalities which have experimented with participatory budgeting initiatives with citizens to explain the budget, identify priorities, and inform decisions. These initiatives should be monitored to ascertain the depth of such engagement and whether it is sustained over time. Parallel studies could review the extent and patterns in citizen participation and use of budgeting and performance portals, and whether they affect accountability dynamics.

4. Enabling deliberative space in Parliament and beyond. Another strand of comparative research could focus on the quality of data and information available to legislatures, from the government and external entities, as well as the procedures, traditions and culture fostering different kinds of scrutiny and accountability. A complementary study could identify whether governments or other entities have created deliberative spaces such as "accountability" days to more broadly review government progress and performance on key societal challenges. More generally, case-study research could explore whether such enhanced engagement by traditional institutions competes with or complements ICT-enabled sharing of data and other forms of disclosure directly with the public as part open-government initiatives. (16)

5. Studying how digital tools are changing internal oversight. Using Bovens and Zouridis (2002) and Jarvis (2014a, 2014b) as points of departure, we need granular and systematic description of how government's internal accountability works in the digital era, which would include the oversight from ministers, department executive teams, and central agencies. A study to emulate is Good's (1980) Politics of Anticipation on tax policy-making, supplemented by survey data from staff. Another area to examine would be whether reporting for the Canadian government's Management Accountability Framework and government watchdogs has some purchase on staff decision-making, risk assessment, and innovation, or if in the digital era, such reporting has become easier cordon off corporate services specialists from regular policy work and operations (see Clark and Swain 2005).

6. New ways to convey data and information. Public servants and contractors rely increasingly on dashboards, project-management software, and visual analytics and other visual tools to display and analyze data, not only for monitoring performance but also to engage legislators, citizens and outside organizations. To what extent governments are making use of these digital tools to further transparency and accountability? Does making open data and "open-by-default" commitments along with open data and performance portals suffice? What kind of contextual information is needed to capture the requisite complexity to enable meaningful accountability? Comparative studies would review the range, quality and granularity of such information, as well as its take-up and use by various actors, and ascertain if Canada was meaningfully living up to undertakings required by the Open Government Partnership.

7. Furthering the "new" accountability. Considine (2002), Schillemans, Van Twist, and Vanhommerig (2013), Halachmi (2014), Laegreid (2014), and Mulgan (2014) essentially argue that governments in the digital era are dealing with new modes of horizontal, interactive, dynamic, and citizen-initiated accountability, which challenge traditional hierarchical or vertical approaches. We need systematic studies of emerging multi-dimensional or hybrid approaches, perhaps considering mutuality and reflexivity, and whether they need new ways to display and share information.

This is an extensive, inter-related research agenda. It requires drawing together diverse disciplinary and methodological expertise, and fostering research collaborations among scholars and practitioners. If Canadian universities, governments, and other experts and organizations collaborate on moving forward and realizing a good portion of this research agenda, it promises to make important contributions to Canadian governance worthy of international attention.

Notes

(1) The Open Government Partnership was spearheaded by the Obama administration but now involves scores of countries around the world. See http://www.opengovpartnership.org.

(2) NPM reforms have been characterized as emphasizing disaggregation of services, competition, and incentives, as well as managerial discretion and different oversight regime, with reporting on outputs and outcomes to political leaders and the public (see Dunleavy et al. 2006). They also include rankings, league tables, web sites, and other reporting (Van de Walle and Cornelissen 2014).

(3) Australia, Canada, China, Egypt, France, Germany, Indonesia, Japan, Korea, the Netherlands, New Zealand, Singapore, the United Kingdom, and the United States.

(4) These were considered as forms of "accountability" but really were different governance values.

(5) But see Meng (2014) which uses the Open Data Barometer data to compare its potential social impact in different countries by gauging different degrees of social capital.

(6) These were: quality, completeness, access and visibility, usability and comprehensibility, timeliness, value and usefulness, granularity, and comparability.

(7) I am indebted to the reviewer who suggested specifying this "ecology." The reviewer also drew attention to the Open Data 150 Survey undertaken by GovLab and the Open Data Exchange (see http://canada.opendata500.com/) for Canadian companies using open data, noting that there is not an equivalent survey of how governments, nonprofits, and scholars are using open data for various purposes.

(8) They argue that parallel and overlapping use of social media by political leaders and administrators can cross important political and administrative boundaries, and should be carefully managed in the future. Similar concerns have emerged about higher orders of government (for example, Brown 2013; Marland, Lewis, and Flanagan 2016).

(9) See the City of Edmonton's current web sites: http://yegcitybudget.ca/budget/ and http://budget.edmonton.ca/.

(10) This parallels a conclusion of the engagement literature: while digital tools assist with expressing and aggregating preferences, they are less helpful in furthering genuine deliberation.

(11) Examples of visualization in support of budgeting include: Edmonton: http://budget, edmonton.ca/; New York City: http://checkbooknyc.com/spending_landing/yeartype/ B/year/118; and Oakland: http://openbudgetoakland.org/2016-17-adjusted-budget-flow. html.

(12) See http://www.opennorth.ca/ and http://openbudgets.eu/.

(13) See https://data.edmonton.ca/ and http://ace.edmonton.ca/.

(14) Sandoval-Almazan and Gil-Garcia (2015) do not discuss "accountability" in the sense of "holding to account"--it is implied by reference to information availability, transparency, open data, and legal requirements.

(15) Much of the promise of open government and digital tools has been less about improving accountability and more about furthering innovation, public value, and market returns; non-government sector can use public data to create apps and other inventions (Cordelia and Bonina 2012; Gil-Garcia, Helbig, and Ojo 2014). By encouraging for-profit and non-profit innovation, society benefits from things government might not be able to or get around to doing. Likewise, some argue that by relying on digital solutions to improve service delivery through greater channel choice, adaptation, and citizen satisfaction through feedback, citizens might develop greater trust and confidence in government (West 2004, 2005; Welch, Hinnant, and Moon 2005; Reddick and Turner 2012; Kernaghan 2013; Marson and Heintzman 2009).

(16) We thank one of the anonymous reviewers for this latter point.

References

Abu-Shanab, E.A. 2015. "Reengineering the open government concept: An empirical support for a proposed model." Government Information Quarterly 32: 453-63.

Aitamurto, T. 2012. Crowdsourcing for Democracy: A New Era in Policy-Making. Committee of the Future, Standing Committee, Parliament of Finland. Available at https://cddrl.fsi. Stanford .edu / sites / default / files / Crowdsourcing_for_DemocracyF_www.pel f.

Allegretti, G. 2012. "From skepticism to mutual support: Towards a structural change in the relations between participatory budgeting and the information and communication technologies?" In Legitimacy 2.0: E-Democracy and Public Opinion in the Digital Age, edited by P. Mindus, A. Greppit, and M. Cuono. Frankfurt am Main: Goethe-University Press, pp. 145-81.

Aman, A., T. A. Al-Shbail, and Z. Mohammed. 2013. "Enhancing public organizations accountability through e-government systems." International Journal of Conceptions on Management and Social Sciences 1 (1): 2357-787.

Andersen, T. B. 2009. "e-Government as an anti-corruption strategy." Information Economics and Policy 21: 201-10.

Aucoin, P. 1995. The New Public Management: Canada in Comparative Perspective. Montreal: Institute for Research on Public Policy.

--. 2012. "New political governance in Westminster systems: Impartial public administration and management performance at risk." Governance: An International Journal of Policy, Administration, and Institutions 25 (2): 177-99.

Aucoin, P., and R. Heintzman. 2000. "The dialectics of accountability for performance in public management reform." In Governance in the Twenty-First Century: Revitalizing the

Public Service, edited by B. G. Peters and D. Savoie. Ottawa: Canadian Centre for Management Development, pp. 244-80.

Aucoin, P., and M. D. Jarvis. 2005. Modernizing Government Accountability: A Framework for Reform. Ottawa: Canada School of Public Service.

Baiocchi, G., and E. Ganuza. 2014. "Participatory budgeting as if emancipation mattered." Politics & Society 42 (1): 29-50.

Bannister, F., and R. Connolly. 2011. "The trouble with transparency: A critical review of openness in e-government." Policy & Internet 3 (1): 158-87.

--. 2014. "ICT, public values and transformative government: A framework and programme for research." Government Information Quarterly 31: 119-28.

Barros, S. A. R., and R. C. Sampaio. 2016. "Do citizens trust electronic participatory budgeting? Public expression in online forums as an evaluation method in Belo Horizonte." Policy & Internet 8 (3): 292-312.

Bedini, I., F. Farazi, D. Leoni, J. Pane, I. Tankoyeu, and S. Jeucci. 2014. "Open government data: Fostering innovation." Journal of e-Democracy 6 (1): 69-79.

Behn, R. D. 2001. Rethinking Democratic Accountability. Washington, DC: The Brookings Institution.

--. 2014a. The Performance Stat Potential: A Leadership Strategy for Producing Results. Washington, DC: Brookings Institution.

--. 2014b. "Performance stat." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford: Oxford University Press.

Bertot, J. C., P. T. Jaeger, and J. M. Grimes. 2010. "Using ICTs to create a culture of transparency: E-government and social media as openness and anti-corruption tools for societies." Government Information Quarterly 27: 264-71.

--. 2012. "Promoting transparency and accountability through ICTs, social media, and collaborative e-government." Transforming Government: People, Process and Policy 6 (1): 78-91.

Bhatnagar, S. 2003. E-Government and Access to Information. Global Corruption Report 2003. Washington, DC: Transparency International.

Bovens, M. 2005. "Public accountability." In The Oxford Handbook of Public Management, edited by E. Ferlie, L. Lynn, and C. Pollitt. New York: Oxford University Press, pp. 182-208.

--2007. "New forms of accountability and EU-governance." Comparative European Politics 5: 104-20.

Bovens, M., Robert E. Goodin, and Thomas Schillemans. eds. 2014. The Oxford Handbook of Public Accountability. Oxford: Oxford University Press.

Bovens, M., T. Schillemans, and R. E. Goodin. 2008. "Does public accountability work? An assessment tool." Public Administration 86 (1): 225-42.

--. 2014. "Public accountability." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford University Press.

Bovens, M., and S. Zouridis. 2002. "From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control." Public Administration Review 62 (2): 174-84.

Brown, D. C. G. 2013. "Accountability in a collectivized environment: From Glassco to digital public administration." Canadian Public Administration 56 (1): 47-69.

Carlitz, R. 2013. "Improving transparency and accountability in the budget process: An assessment of recent initiatives." Development Policy Review 31 (1): 49-67.

Castro, D., and T. Korte. 2015. Open Data in the G8: A Review of Progress on the Open Data Charter. Washington, DC: Centre for Data Innovation.

Chadwick, A., and C. May. 2003. "Interaction between states and citizens in the age of the internet: 'e-Government' in the United States, Britain, and the European Union." Governance 16 (2): 271-300.

Christensen, T., and P. Laegreid. 2015. "Performance and accountability--A theoretical discussion and an empirical assessment." Public Organizational Review 15: 207-25.

Clark, I. D., and H. Swain. 2005. "Distinguishing the real from the surreal in management reform: Suggestions for beleaguered administrators in the government of Canada." Canadian Public Administration 48 (4): 453-76.

Clarke, A., and M. Francoli. 2014. "What's in a name? A comparison of 'open government' definitions across seven Open Government Partnership members." Journal of e-Democracy 6 (1): 248-66.

Clarke, A., and H. Margetts. 2014. "Governments and citizens getting to know each other? Open, closed, and big data in public management reform." Policy & Internet 6 (4): 393-417.

Considine, M. 2002. "The end of the line? Accountable governance in the age of networks, partnerships, and joined-up services." Governance 15 (1): 21-40.

Conteh, C. 2016. "Rethinking accountability in complex and horizontal network delivery systems." Canadian Public Administration 59 (2): 224-44.

Cordelia, A., and C. M. Bonina. 2012. "A public value perspective for ICT enabled public sector reforms: A theoretical reflection." Government Information Quarterly 29: 512-20.

Cucciniello, M., G. Nasi, and G. Valotti. 2012. Assessing transparency in government: Rhetoric, reality and desire. In 45th Hawaii International Conference on System Sciences (HICSS45), Maui, HI, USA, pp. 2451-61.

Davison, J. 2014. "Visual accountability." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford University Press.

Dawes, S., and N. Helbig. 2010. "Information strategies for open government: Challenges and prospects for deriving public value from government transparency." In EGOV 2010, edited by Maria A. Wimmer, J.-L. Chappelet, M. Janssen, and H. ]. Scholl. New York: Springer, pp. 50-60.

Demchak, C. C., C. Friis, and T. M. La Porte. 2000. "Webbing governance: National differences in constructing the face of public organizations." In Handbook of Public Information Systems, edited by G. David Garson. New York: Marcel Dekker, pp. 179-95.

Deschamps, R., K. McNutt, and W. Zhu. eds. 2012. Environmental Scan on Social Media Use by the Public Sector Administration, Public Engagement and Citizen Centered Services. Regina: Johnson-Shoyama School of Public Policy.

Dias, Nelson, ed. 2014. Hope for Democracy - 25 Years of Participatory Budgeting Worldwide. Lisbon: In Local Association.

Dubnick, M. J. 2005. "Accountability and the promise of performance." Public Performance and Management Review 28(3): 376-417.

Dunleavy, P., H. Margetts, S. Bastow, and J. Tinkler. 2006. "New public management is dead. Long live digital-era governance." Journal of Public Administration Research and Theory 16 (3): 467-94.

Dutil, P. 2015. "Crowdsourcing as a new instrument in the state's arsenal: Explorations and considerations." Canadian Public Administration 58(3): 363-83.

Francoli, M. 2014. "Developing a strategy for effective e-Government: Findings from Canada." In Government e-Strategic Planning and Management, Vol. 3, edited by L. G. Anthopoulos and C. G. Reddick. Public Administration and Information Technology, pp. 143-55.

French, R. 1980. How Ottawa Decides: Planning and Industrial Policy-Making 1968-1980. Tornto: Lorimer and Canadian Institute for Economic Policy.

Fung, A., M. Graham, and D. Weil. 2007. Full Disclosure: The Perils and Promise of Targeted Transparency. Cambridge: Cambridge University Press.

Gil-Garcia, J. R., N. Helbig, and A. Ojo. 2014. "Being smart: Emerging technologies and innovation in the public sector." Government Information Quarterly 31: 11-18.

Gomery, J. H. 2006. "Restoring accountability." Report of the Commission of Inquiry into the Sponsorship Program & Advertising Activities. Ottawa: Public Works and Government Services Canada.

Good, D. A. 1980. The Politics of Anticipation: Making Canadian Federal Tax Policy. Ottawa: School of Public Administration, Carleton University.

--. 2003. The Politics of Public Management: The HRDC Audit of Grants and Contributions. Toronto: University of Toronto Press.

--. 2014. The Politics of Public Money, 2nd ed. Toronto: University of Toronto Press.

Gregory, R. 2003. "Accountability in modern government." In Handbook of Public Administration, edited by B. G. Peters and J. Pierre. London: Sage, pp. 557-68.

Gruzd, A., and J. Roy. 2016. "Social media and local government in Canada: An examination of presence and purpose." In Social Media and Local Governments, edited by M. Z. Sobaci. Public Administration and Information Technology, Vol. 25. New York: Springer, pp. 79-94.

Halachmi, A. 2011. "Accountability and governance." In Accounting, Accountability and Governance in the Public Sector, edited by Greiling D, Halachmi, A. and Schauer, R. Johannes Kepler University, Linz, Austria: Trauner Verlag Press, pp. 226-243. http:// www.gbv.de/dms/ilmenau/toc/647499304.PDF

--2014. "Accountability overloads." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford: Oxford University Press.

Halachmi, A., and G. Greiling. 2013. "Transparency, e-government, and accountability." Public Performance b Management Review 36(4): 562-84.

Halachmi, A., and M. Holzer. 2010. "Citizen participation and performance measurement: Operationalizing democracy through better accountability." Public Administration Quarterly 34 (3): 338-76. Available at http://www.egov4dev.org/transparency/.

Harris, M. 2015. Party of One: Stephen Harper and Canada's Radical Makeover. Toronto: Viking.

Harrison, T. M., and D. S. Sayogo. 2013. "Open budgets and open government: Beyond disclosure in pursuit of transparency, participation and accountability." In Proceedings of the 14th Annual International Conference on Digital Government Research, ACM, June 17-20, 2013, Quebec City, QC, Canada, New York: ACM, pp. 235-44.

--. 2014. "Transparency, participation, and accountability practices in open government: A comparative study." Government Information Quarterly 31: 513-25.

Hartz-Karp, J., and B. Sullivan. 2014. "The unfulfilled promise of online deliberation." Journal of Public Deliberation 10 (1): Article 16.

Heald, D. 2003. "Fiscal transparency: Concepts, measurement and UK practice." Public Administration 81 (4): 723-59.

--2012. "Why is transparency about public expenditure so elusive?" International Review of Administrative Sciences 78 (1): 30-49.

Howard, C., and S. Phillips. 2012. "Moving away from hierarchy: Do horizontality, partnerships and distributed governance really signify the end of accountability?" In From New Public Management to Neil' Public Governance, edited by H. Bakvis and M. D. Jarvis. Montreal: McGill-Queens University Press, pp. 314-41.

Hudson, Alan. 2014. "Open budgeting brings better development results." Available at http://alanhudson.info/wp-content/ uploads/2014/01 /ONE-Open-Budgeting-Case-Studies-FINAL-11-Jan-13.pdf.

Islam, M. S., and A. Gronlund. 2012. "Investigating the relationships between accountability and governments' transformation to e-government." In ICEGOV '12, October 22-25 2012, Albany, New York: ACM pp. 297-300.

Jang, Y. S., M. Cho, and G. S. Drori. 2014. "National transparency: Global trends and national variations." International journal of Comparative Sociology 55 (2): 95-118.

Jarvis, M. D. 2009. "The adoption of the accounting officer system in Canada: Changing relationships?" Canadian Public Administration 52 (4): 525-47.

--. 2014a. "The black box of bureaucracy: Interrogating accountability in the public service." Australian Journal of Public Administration 73 (4): 450-66.

--. 2014b. "Hierarchy." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford University Press, pp. 405-420.

Johnston, P. 2010. "Transforming government's policy-making processes: Why encouraging more and easier citizen input into policy-making is not enough." Journal of e-Democracy 2 (2): 162-69.

Jordan, J. M., and S. L. Sutherland. 1979. "Assessing the results of public expenditure: Program evaluation in the Canadian federal government." Canadian Public Administration 22 (4): 581-609.

Justice, J. B., J. Melitski, and D. L. Smith. 2006. "E-government as an instrument of fiscal accountability and responsiveness do the best practitioners employ the best practices?" American Review of Public Administration 36 (3): 301-22.

Kayser-Bril, N. 2016. "Turns out people love to investigate public spending (if you do it right)." Blog, March 27. Available at http://openbudgets.eu/post/2016/03/07/making-investigating-budgets-exciting.

Kernaghan, K. 2013. "Changing channels: Managing channel integration and migration in public organizations." Canadian Public Administration 56 (1): 121-41.

Khagram, S., A. Fung, and P. de Renzio. eds. 2013. Open Budgets: The Political Economy of Transparency, Participation, and Accountability. Washington, DC: The Brookings Institution.

Koppell, J. G. S. 2005. "Pathologies of accountability: ICANN and the challenge of 'multiple accountabilities disorder'." Public Administration Review 65 (1): 94-108.

Kraemer, Kenneth, and Jason Dedrick. 1997. "Computing and public organizations." Journal of Public Administration Research and Theory 7: 89-112.

Lacharite, J. R. 2011. "The Internet and electronic democracy in Canada: Reaching the limits of e-government and the false promise of digital democracy?" Canadian Political Science Review 5 (1): 1-19.

Laegreid, P. 2014. "Accountability and new public management." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford: Oxford University Press.

La Porte, T. M., M. de Jong, and C. Demchak. 2002. "Democracy and bureaucracy in the age of the web." Administration and Society 34: 411-26.

Lassinantti, J. 2014. Public Sector Open Data-Slmping an Arena for Innovation and Value Creation. Lulea University of Technology, Dept. of Computer Science, Electrical and Space Engineering, Division of Information Systems.

Lindner, R., and U. Riehm. 2009. "Electronic petitions and institutional modernization: International parliamentary e-petition systems in comparative perspective." Journal of e-Democracy 1 (1): 1-11.

Lindquist, E. 1998. "Getting results right: Reforming Ottawa's estimates." In How Ottawa Spends 1998-99: Balancing Act: The Post-Deficit Mandate, edited by L. A. Pal. Toronto: Oxford University Press, pp. 153-90.

--. 2009. "How Ottawa assesses department/agency performance: Treasury Board's management accountability framework." In Holy Ottawa Spends 2009-2010: Economic Upheaval and Political Dysfunction, edited by A. M. Maslove. Montreal: McGill-Queen's University Press, pp. 47-88.

--. 2015. "Visualization meets policy making: Visual traditions, policy complexity, strategic investments." In Governance in the Information Era: Theory and Practice of Policy Informatics, edited by E. W. Johnston. New York: Routledge, pp. 62-82.

--. 2016. "The lessons and prospects of deliverology." Canadian Government Executive 22(3): 6-9.

--. 2018 forthcoming. "Visualization practice and government: Strategic investments for more democratic governance." In Policy Analytics, Modelling, and Informatics: Innovative Tools for Solving Complex Social Problems, edited by J. R. Gil-Garcia, T. A. Pardo, and L. F. Luna-Reyes. New York: Springer.

Lips, M. 2012. "E-government is dead: Long live public administration 2.0." Information Polity 17: 239-50.

Lnenicka, M. 2015. "An in-depth analysis of open data portals as an emerging public e-service." International Journal of Social, Behavioral, Educational, Economic, Business and Industrial Engineering 9 (2): 589-99.

Lourenco, R. P. 2015. "An analysis of open government portals: A perspective of transparency for accountability." Government Information Quarterly 32: 323-32.

Mao, Y., and M. Adria. 2013. "Deciding who will decide: Assessing random selection for participants in Edmonton's Citizen Panel on budget priorities." Canadian Public Administration 56 (4): 610-37.

Margetts, H., and P. Dunleavy. 2010. "The second wave of digital-era governance: a quasiparadigm for government on the Web." Philosophical Transactions of the Royal Society A 371: 20120382. http://dx.doi.org/10.1098/rsta.2012.0382.

Margolis, M., and D. Resnick. 2000. Politics as Usual: The Cyberspace "Revolution". Thousand Oaks, CA: Sage.

Marland, A., J. P. Lewis, and T. Flanagan. 2016. "Governance in the age of digital media and branding." Governance 30 (1): 125-11.

Marson, B., and R. Heintzman. 2009. From Research to Results: A Decade of Results-Based Service Improvements in Canada. Toronto: Institute of Public Administration of Canada.

Masser, K. 2013. "Participatory budgeting as its critics see it." Blog, April 30. Available at http://www.buergerhaushalt.org/en/article/participatory-budgeting-its-critics-see-it.

Mclvor, R., M. McHugh, and C. Cadden. 2002. "Internet technologies: Supporting transparency in the public sector." International Journal of Public Sector Management 15 (3): 170-87.

Meng, A. 2014. "Investigating the roots of open data's social impact." Journal of e-Democracy 6 (1): 1-13.

Moon, M. J. 2002 "The evolution of e-government among municipalities: Rhetoric or reality?" Public Administration Review 62 (4): 424-33.

Moore, M. H. 1995. Creating Public Value: Strategic Management in Government. Cambridge, MA: Harvard University Press.

Mulgan, R. G. 2000. "'Accountability': An ever-expanding concept?" Public Administration 78: 555-73.

--. 2003. Holding Power to Account: Accountability in Modern Democracies. Basingstoke: Pal grave Macmillan.

--. 2014. "Accountability deficits." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford University Press.

Nabatchi, T. 2010. "Addressing the citizenship and democratic deficits: The potential of deliberative democracy for public administration." The American Review of Public Administration 40 (4): 376-99.

Norris, P., and M. J. Moon. 2005. "Advancing e-government at the grassroots: Tortoise or hare?" Public Administration Review 65 (1): 64-75.

Orelli, R. L., E. Padovani, and E. Scorsone. 2010. "E-government, accountability, and performance: Best-in-class governments in European Union countries." In Comparative E-Government, Integrated Series in Information Systems 25, edited by C. G. Reddick. New York: Springer, pp. 561-85.

Osborne, S. ed. 2010. The New Public Governance? Emerging Perspectives on Theory and Practice of Public Governance. Oxford: Routledge.

Page, K. 2015. Unaccountable: Truth and Lies on Parliament Hill. Toronto: Viking.

Papaloi, A., and D. Gouscos. 2013. "Parliamentary information visualization as a means for legislative transparency and citizen empowerment? Investigating the rationale, aspirations, feasibility and effectiveness of PIV efforts." journal of e-Democracy 5 (2): 174-86.

Peled, A., and N. Karine. 2015. "Towards open data for public accountability: Examining the US and the UK models." In Paper presented to iConference 2015. Available at https:// www.ideals.illinois.edu/bitstream/handle/2142/73632/101_ready.pdf?sequence=2.

Perrin, B. 2015. "Bringing accountability up to date with the realities of public sector management in the 21st century." Canadian Public Administration 58 (1): 183-203.

Peters, B. G. 2014. "Accountability in public administration." In The Oxford Handbook of Public Accountability, edited by M. Bovens, R. E. Goodin, and T. Schillemans. Oxford University Press.

Pina, V., L. Torres, and B. Acerete. 2007. "Are ICTs promoting government accountability? A comparative analysis of e-governance developments in 19 OECD countries." Critical Perspectives on Accounting 18: 583-602.

Pina, V., L. Torres, and S. Roya. 2010. "Is e-government leading to more accountable and transparent local governments? An overall view." Financial Accountability & Management 26 (1): 3-20.

Pinnington, E., J. Lerner, and D. Schugurensky. 2009. "Participatory budgeting in North America: The case of Guelph, Canada." Journal of Public Budgeting, Accounting & Financial Management 21 (3): 455-84.

Piotrowski, S. J. 2007. Governmental Transparency in the Path of Administrative Reform. Albany, NY: State University of New York Press.

Prince, M. J., and J. Chenier. 1980. "The rise and fall of policy planning and research units: An organizational perspective." Canadian Public Administration 23 (4): 519-41.

Radin, B. 2000. Beyond Machiavelli: Policy Analysis Comes of Age. Washington, DC: Georgetown University Press.

Reddick, C. G., and M. Turner. 2012. "Channel choice and public service delivery in Canada: Comparing e-government to traditional service delivery." Government Information Quarterly 29: 1-11.

Romzek, B. S., and P. W. Ingraham. 2000. "Cross pressures of accountability: Initiative, command, and failure in the Ron Brown plane crash." Public Administration Review 60 (3): 240-53.

Roy, J. 2012a. "Social media's democratic paradox: Lessons from Canada." European journal of ePractice 16: 5-15.

--. 2012b. "Electronic democracy's deliberative potential: Dissecting the Canadian polity and the challenges ahead." International journal of Electronic Governance 5 (1): 63-81.

--. 2012c. "Secrecy versus openness: Democratic adaptation in a Web 2.0 era." Public Administration and Information Technology 1 (3): 249-57.

--. 2014. "Open data and open governance in Canada: A critical examination of new opportunities and old tensions." Future Internet 6: 414-32.

--. 2016. "Secrecy, security and digital literacy in an era of meta-data: Why the Canadian Westminster model falls short." Intelligence and National Security 31 (1): 95-117.

Sampaio, R. C., and T. Peixoto. 2014. "Electronic participatory budgeting: False dilemmas and true complexities." In Hope for Democracy--25 Years of Participatory Budgeting Worldwide, edited by Nelson Dias. Lisbon: In Local Association, pp. 413-25.

Sandoval-Almazan, R., and J. R. Gil-Garcia. 2015. "Toward an integrative assessment of open government: Proposing conceptual lenses and practical components." Journal of Organizational Computing and Electronic Commerce 26 (1-2): 170-92.

Schillemans, T., M. Van Twist, and I. Vanhommerig. 2013. "Innovations in accountability." Public Performance & Management Review 36 (3): 407-35.

Secondo, D., and P. Jennings. 2014. "Building sustainable empowerment: Participatory budgeting in North America." In Hope for Democracy--25 Years of Participatory Budgeting Worldwide, edited by Nelson Dias. Lisbon: In Local Association, pp. 241-52.

Shepherd, R. 2011. "Departmental audit committees and governance: Making management and accountability the priority from the top down." Canadian Public Administration 54 (2): 277-304.

Small, T. A. 2012. "e-Government in the age of social media: An analysis of the Canadian government's use of Twitter." Policy & Internet 4 (3-4): 1944-2866.

Smith, M. L., M. E. Noorman, and A. K. Martin. 2010. "Automating the public sector and organizing accountabilities." Communications of the Assoc. for Information Systems 26: Article 1.

Stamati, T., T. Papadopoulos, and D. Anagnostopoulos. 2015. "Social media for openness and accountability in the public sector: Cases in the Greek context." Government Information Quarterly 32: 12-29.

Sunstein, C. R. 2016. "Output transparency vs. input transparency." Draft Paper, August 16. Available at http://ssrn.com/abstract=2826009.

Swift, J. C. 2013. "The people's lobby: A model for online activist deliberation." Journal of Public Deliberation 9 (2): Article 13.

Thomas, P. G. 1979. "The Lambert report: Parliament and accountability." Canadian Public Administration 22 (4): 557-71.

--. 1998. "The changing nature of accountability." In Taking Stock: Assessing Public Sector Reforms, edited by B. Guy Peters and Donald J. Savoie. Kingston and Montreal: Canadian Centre for Management Development and McGill-Queen's University Press, pp. 348-53.

Ubaldi, B. 2013. "Open government data: Towards empirical analysis of open government data initiatives." OECD Working Papers on Public Governance, No. 22. Paris: OECD Publishing.

Van de Walle, S., and Floor Cornelissen. 2014. "Performance reporting." In The Oxford Handbook of Public Accountability, edited by Mark Bovens, Robert E. Goodin, and Thomas Schillemans. Oxford University Press.

Vanhommerig, I., and P. M. Karre. 2014. "Public accountability in the internet age: Changing roles for governments and citizens." International Review of Public Administration. 19 (2): 206-217.

Weare, C. 2002. "The Internet and democracy: The causal links between technology and politics." International Journal of Public Administration 25 (5): 659-91.

Welch, E., C. Hinnant, and M. J. Moon. 2005. "Linking citizen satisfaction with e-government and trust in government." Journal of Public Administration Research and Theory 15 (3): 371-91.

Welch, E., and W. Wong. 1998. "Public administration in a global context: Bridging the gaps of theory and practice between Western and non-Western nations." Public Administration Review 58: 40-49.

West, D. M. 2004. "E-government and the transformation of service delivery and citizen attitudes." Public Administration Review 64 (1): 15-27.

--. 2005. Digital Government: Technology and Public Sector Performance. Princeton: Princeton University Press.

Wildavsky, A. 1972. "Self-Evaluating Organization." Public Administration Review 32 (5): 509-520.

Wong, W., and E. Welch. 2004. "Does e-government promote accountability? A comparative analysis of website openness and government accountability." Governance 17 (2): 275-97.

Yu, H., and D. G. Robinson. 2012. "The new ambiguity of 'open government'" UCLA Law Review Discourse 59: 178-208.

Evert Lindquist is Professor and Irene Huse is a doctoral student in the School of Public Administration, University of Victoria, British Columbia. The authors thank the anonymous reviewers and colleagues Amanda Clarke and Jeffrey Roy for useful comments on an earlier version of this manuscript.