Useful Data for Everyone: The City of Boston and Open Data Strategy

Customer: City of Boston | Site: Data Boston

Boston’s Principal Data Scientist, Curt Savoie, is all about data. “I like poking around in data, running analytics, thinking about business intelligence. And, back when our data wasn’t automated I helped Performance Management get the data they needed for their reports, to run their meetings.”

Performance Management, headed by Chris Dwelley, is responsible for tracking the City’s performance metrics. His office relied on data procured by Savoie to measure results. For example, the police department would report their data upon request -- the number of 911 calls, crime statistics, and other metrics relevant to police activity. Performance Management would compare reported data against the City’s goals to evaluate the police department’s success. The City used the same approach for other initiatives, like tracking the number of potholes filled or streetlights fixed.

Because of his work with Dwelley and other data-driven analysis projects, Savoie recognized there had to be a better way. Local departments were fielding requests from the City, the public, researchers, and the press, and the process was inconsistent. Savoie approached Boston’s CIO, Bill Oates, about the improvement of Boston’s open data program. Oates, now CIO for the State of Massachusetts, had been a participant in a G7[1] style cooperative with CIOs from other large cities – they would regularly discuss the challenges of open data and open government. During that time, Socrata was running open data pilot programs with San Francisco and New York City; Boston’s open data initiative lagged behind. With an endorsement from both Oates and Boston Mayor Marty Walsh, Savoie set out to transform Boston’s underwhelming open data program into a more comprehensive and user-friendly system.

During the early phases of the data project Boston received a D- financial transparency rating from U.S. PIRG. U.S. PIRG (United States Public Interest Research Group) evaluates financial transparency by assessing how much financial information a city provides and how easy it is for citizens to access the information[2]. Boston's low score gave the City incentive to speed up their efforts to not just bring their financial data online, but to do so in way that would be genuinely beneficial for everyone who uses the City’s data.

Why Socrata?

Savoie began his research in spring, 2012. He met with Rufus Pollack, President of the Open Knowledge Foundation. Savoie set up a test instance of CKAN, an open source data portal platform. He also researched Socrata – there was a lot of positive feedback about Socrata from the cities involved in the CIO G7.

“Philosophically, I like open source. But you have to have server resources and there’s a lot of development, so it becomes a resource question. Socrata has end to end service. Our web team gave Socrata our graphics and our style sheet, and I loaded the data. Socrata is a good solution for smaller cities that are technically advanced, or who don’t have software developers on staff. In Boston, we had the resources, but this was a choice we made.”

The City selected Socrata as their platform in summer of 2012; the first version of the portal went live in October that same year. In their earliest implementation of open data, the City published about ten datasets, but with the new Socrata driven portal, that number increased to 70 datasets. Later, the City implemented Socrata’s Open Checkbook and Open Budget to publish both the annual capital budget and the City’s operation expenses.

Open Data Is a Long Game

City stakeholders all the way up to Boston Mayor Walsh’s office supported the expansion of the open data program, but this enthusiasm wasn’t reflected throughout the entire organization. “In the past, public data often meant the press had been foraging for information and the results were not always flattering.” But, Savoie says, “[…] fears have faded over time as we’ve opened our data.”

Naysayers have seen the apps, analysis, maps, and visualizations that result from making data publicly available. “Once we put stuff out there, we can engage the public to help us solve problems and inform our citizens. And the press can stop calling -- they can go directly to the portal. Sure, there are some bad stories, but there are a million good ones, too.”

Positive examples have paid off. Divisions who were resistant at first are now proactive. “A few years ago, to get assessment data, you had to go to the office and pay 35 dollars for a CD. This year the Deputy Commissioner for Assessments emailed me right after their data was compiled to ask when it would go live on the portal. Before we had the portal, they had no interest in releasing their data. It’s a complete turnaround.”

Boston’s police department has changed their stance, too. In the beginning, they didn’t object to having their data publicized, but they weren’t engaged. Now, they track the health of their data feeds and Savoie hears from them right away if something doesn’t look right. “They rely on having a stable and consistent data feed -- a big difference from the apathy of the department before the open data initiative.”

Security Matters

Expanding the amount of data available to the public was a primary goal, but the City also wanted to ensure the data was released in a responsible way. As the Citywide Performance Manager, Dwelley is deeply involved with the City’s financial data. Based in the CFO’s office, Dwelley was Savoie’s contact when it came to releasing budget and expenditure data where security was a top concern. "We worked very closely with the stakeholders and the department heads who owned the data to make sure we weren't pushing out anything private or protected," said Dwelley.

In order to make sure released data did not create a security compromise, stakeholders worked with department heads and data owners to exclude sensitive items, like social security and account numbers. The team also consulted with the City's legal resources to ensure the protection of individual and private information. And Boston consulted with other cities about what they did not release. “There’s no standard rulebook about what to release,” said Dwelley, “we had to learn as we went through the process.”

Dwelley’s team repeatedly refined their process. They studied each dataset to identify which fields should not be publicized. Through repeated analysis, Dwelley’s team made sure they did not compromise on security. Even now, the City runs spot checks to ensure no sensitive information -- like account and social security numbers -- is included in published datasets.

“I think we've shown we can release useful, beneficial data and protect privileged information at the same time," says Dwelley.

Savoie confirms that the security process has held water. “We’ve had no breaches,” he says, “and with each new department that comes online, we can build on our success.”

Data for Everyone

The City of Boston's internal staff found the open data helpful -- city analysts now have fast access to information and no longer have to rely on the IT department for data. The portal is loaded with commonly used queries so those accessing the data don't have to be SQL experts or database analysts. And the results of those reports are consistent because everyone their data from the same source.

Opening up municipal data means that residents, advocates, students, web developers, journalists, businesses, government watchdogs and government employees alike can be civic innovators for our city.-- WGBH News

Academics, the press, and developers have also benefited. Reporters are now able to pull the data they’re looking for directly from the open data portal. The Boston Area Research Initiative has authored several papers using the City’s data. But a lot of the buzz around open data focuses on its accessibility by the public, and citizens are an important audience for Boston’s data transparency initiatives.

In January 2015, the Knight Foundation awarded a grant to Open Data to Open Knowledge. The organization plans to make the City's open data more readily accessible through Boston's libraries[3]. The City encourages this kind of collaboration because it creates a more informed public who, in turn, engage more effectively with the City’s government.

Boston's Open Future

The City is focused not on simply releasing more data, but making sure it's worthwhile to do so. They will continue to expand their offerings, but they're also investing in improving how the data is presented. They want to create visualizations to help citizens better understand each dataset. And they’re creating data categories -- public works, public safety, emergency medical services – for data analysis by sector or subject area.

The City plans to provide a contact for each dataset, connecting researchers with data owners. They’re building a "data dictionary" to define each type of published data. And Boston is taking a broader view; being open about their data has led the City to question if they could create a better website for all of the City’s data, not just their financial data.

The open data philosophy goes both ways – the people of Boston are encouraged to gather and share data with the City, too. This open data culture is behind initiatives like Street Bump, an app for pothole reporting. This citizen generated data is used to help the city plan how – and where – to spend road improvement funds. Boston’s inspection data is available through Food Police, an app for restaurant health and safety scores. Ungentry, a Code for Boston project, mashes up census data with Boston Foundation data to see how government policy is affecting gentrification in Boston neighborhoods.

These creative efforts illustrate the value making data more than open – the City’s data is now a building block for larger projects. "Throwing data up to say you do open data is a disservice to everybody," says Dwelley. "We want it to be easier for every day citizens to understand this data, relate to it and actually WANT to use it."

Savoie shares the vision of making data accessible for an audience beyond the press, researchers, academics, and developers. “Visualizations make our data accessible to the public, but there’s more that can be done. How can people who aren’t data experts mash up different datasets? Or ask relevant questions about their concerns? How do we move beyond silos of data into something custom, interesting, relevant, and timely to constituents at large? That’s the future of open data.”


[1] G7: CIOs From Seven Big-Cities Work Together to Develop Open-Source IT Solutions

[2] Report: Transparent & Accountable Budgets

[3] Boston library projects land $700,000 from Knight Foundation to teach public about privacy and city data