The EveryBlock Blog

We're excited to announce our latest feature: the ability to comment on news items throughout our site.

Now, if you're viewing a piece of neighborhood news on our site and have more information about it, you can add your observations directly to the page.

Comment threads are pretty common across the Web these days, but we've put a different spin on our comment system by tying it into our geographic engine. If you post a comment to a piece of news, obviously the comment will show up on that page — but it will also be published in the local news feeds for that area. For example, this comment was posted to a crime reported on the 5700 block of W. Roosevelt Road in Chicago, which means that the comment was then published in the news feeds for the block, the neighborhood, the ZIP and any custom locations that contain that point. (This is our attempt to tackle an interesting "long tail" problem — our site is so granular that the chances are slim that a large group of people will all view the same piece of information.)

We know that people on the street have more information about news near them than we or our news sources could ever have. We're hoping that people will use this feature to improve the information on our site, contributing to the collective neighborhood knowledge of a given incident, permit or other such news report.

Over the past week or so, we've been quietly rolling out this feature across the EveryBlock cities, and folks have posted some interesting comments already, including a Seattle restaurant owner explaining inspection results, a hospital worker explaining a crime, and various opinions on the news. We're looking forward to seeing how this feature unfolds.

Today we've added a new data type in Boston — calls for service.

If you live in Boston, you'll be notified any time the city publishes information about a street-light that fails or is knocked down, a new street sign, or other service requests.

This data comes from Boston's GIS Data Hub, a project of their Enterprise GIS department. As they say on their homepage, the city's MIS (Management & Information Services) Department plans to use this application as a foundation for additional maps and data sets over time.

The movement for cities to publish more data online is gaining steam, and the city of Boston is one of the major players. We look forward to working with Boston and other cities as they move to more open government. Contact us if you are a government official interested in sharing data.

Today we've launched a new way of diving into EveryBlock's local news: by individual street address.

Previously, the most granular way to view information on our site was by city block, with a one-block radius. Enter an address, then click the "1 block" link near the top of the page, and you get a page devoted to that small area of the city (example). But even this intense level of granularity didn't cut it for some people, particularly researcher-types who were wanting to get historical information on a particular building, or people who live in very large buildings where lots of things happen.

Starting today, if you search EveryBlock for an address, you'll still get the block page as before, but, if we have news available for that specific address, you'll see a new link to "Show news only for" that address:

Click that link, and you'll get a page devoted to the exact address you searched for, with whatever news we've found there.

For example, here's the page for 900 N. Michigan Ave. in Chicago -- a skyscraper that contains condos, an upscale shopping mall and a medical clinic, among other things. You can see that we find news at that address almost every day, from a building permit for a new leather glove retail kiosk in the ground floor atrium, to frequent references in the newspapers, to restaurant inspections at the many restaurants in the building. It's like a city within a city, and we're happy it finally has its own page on EveryBlock.

Just as with neighborhoods, ZIP codes or blocks, you can sign up for daily/weekly e-mail alerts or a customizable RSS feed for addresses.

Note that we've only activated these address-specific pages for addresses at which at least one thing has happened in our database of news. In other words, if you search for an address and you don't see the "Show news only for..." link, that means we haven't found anything specifically at that address since we launched EveryBlock in your city. But as soon as something happens there, a page will be created for that address automatically.

Give the new feature a spin and let us know what you think at feedback@everyblock.com.

Today we've added a new data type in San Francisco — calls for service.

If you live in San Francisco, you'll be notified any time the city gets notified about graffiti on private or public property, or someone requests a street or sidewalk cleaning, or asks that a pothole be filled.

This data comes from DataSF, a "central clearinghouse for datasets published by the City & County of San Francisco". This is one of the most ambitious of the new projects being launched by municipalities to unleash public data for the common good. San Francisco Mayor Gavin Newsom, now running for Governor of the State of California, deserves a lot of credit for launching this project.

One thing I've learned during my work at EveryBlock is that it takes dozens of hard-working policy, operations, and technical people, toiling in city government for months and years, to achieve something like this. And we want to encourage more and more comprehensive programs like this all over the country. We're just getting started in our goal to deliver news near you down to the block level. If you are a government official with access to data, we need help in the months and years to come. The open government data movement is picking up steam. We need more people like the leaders of DataSF all over the country.

Chief among these people is Edward Reiskin, who started as San Francisco 311 Director on the day I started at EveryBlock. He was an early believer in our work because he was a main force behind the creation of Washington, DC Citywide Data Warehouse, which has come to be known as the gold standard in the delivery of civic data in the U.S. This kind of cross-pollination among cities is a key to progress. Later, as Director of the San Francisco Department of Public Works, Ed had his staff set up feeds for excavation permits, street space permits, and street use permits more than a year and a half ago. And now nearly all of the data in the calls for service feed are items serviced by his department.

Nancy Alfaro, the Director of the 311 system in San Francisco, and Andy Maimoni, Deputy Director, also deserve a lot of credit for getting this data into the hands of the public. And Jay Nath, Innovations Manager at the Department of Technology and a leader of the DataSF project, has been a tireless worker on this stuff. He also worked with closely our Managing Editor, Paul Wilson, as he defined every service type in as much detail as possible. Often we get civic data that uses obscure terminology, so we try to do the work of defining things for our users; Jay has been a huge help here.

Behind these people are thousands more. Technology workers who built all the legacy data systems, GIS people who maintain street centerlines and district maps, public works laborers who put asphalt in holes, and even the citizens of San Francisco, who care enough about their block to make a call to 311.

We've created our first screencast, a video that shows you how to create and edit a custom location.

Since we launched our custom locations feature a few weeks ago, lots of you have added public custom locations to track news in a custom area. Thanks to EveryBlock users, you can now see all the news in lower nw dc, San Francisco's Tendernob, more or less, and the Unfancy Downtown area of Dallas. Get started now on your own custom location by checking out the new screencast, which walks you through the process.

Where does your neighborhood begin and end?

Does it stretch all the way to Rockwell St., or does it end at Western Ave.? Does it include the bordering cemetery? Does it stop at the railroad tracks, or continue north for a few blocks?

As a neighborhood news site, we try to maintain accurate lists of neighborhoods and their boundaries, but we're inevitably incomplete. Neighborhoods change, areas get renamed and redeveloped, and even the most well-established districts can have ambiguous boundaries. (In fact, some argue that neighborhoods have no true boundaries, only centers, but a computer needs to be able to draw the line somewhere.)

More importantly, the boundaries of a neighborhood don't necessarily correspond to the boundaries of the area you're interested in. For example, your apartment might border multiple neighborhoods — or you might never set foot in a specific area, in which case you might not care much about what happens there. Neighborhood boundaries don't always match up to emotional geographies.

After getting steady streams of feedback from users who say our boundaries are off or that we overlooked their particular neighborhood, we've decided to address this problem once and for all, and we're incredibly excited about our new approach. Today, we're launching a feature that puts the control of geographic boundaries in your own hands.

With our new custom locations tool, you can draw your own geographic area, however you'd like. You can draw simple rectangles, or you can be completely arbitrary, gerrymandering it to include or exclude specific streets, drawing oddly shaped polygons to your heart's content. Once you're done, we'll give you a news feed for that specific area, no matter how oblong or oddly shaped it is.

When you create a custom location, you have the option of making it public or keeping it private. If you make it public, it'll show up on the public list (example) for the benefit of your neighbors and anybody else who might find it useful.

The most interesting way to use it is to create a super-detailed map of your own area that includes only the specific streets you're interested in. For example, I find myself walking down the same side streets and going to the same restaurants, so I made myself a highly customized, private custom location representing that specific part of Chicago:

This is similar to (but much more powerful than) the old "crimes along a route" feature of our predecessor site, chicagocrime.org.

As with any other area on EveryBlock (like blocks, neighborhoods or ZIP codes), you can sign up to get regular e-mail alerts or an RSS feed for news that happens in your custom locations. And our fine-grained customization applies, too, so, for example, you can elect to get notified only for photos, crime and local blog coverage, but not for building permits.

We think this is an interesting approach because we've resigned ourselves to the fact that no list of neighborhoods will ever be 100% complete and accurate — so why push the boulder up that hill any longer? Of course, we still feature our legacy lists of neighborhoods (example), but over time, we'll likely phase that out in favor of a hybrid approach mixing user-contributed custom locations and the neighborhoods we've obtained from various "official" sources.

Take a look at the new tool and let us know what you think by e-mailing feedback [at] everyblock.com.

A new version of the EveryBlock iPhone app is now available.

Although we've made a number of small tweaks to the app since originally releasing it in April, this is the first major upgrade. Here's what's new in this version:

Ability to bookmark locations

Click the plus icon at the bottom of a location's page to bookmark the location for later. You can give it a nickname like "Home" or "Work."

To access your bookmarks, choose your city and click the blue icon that looks like an open book. (This is the same icon used for bookmarks in the iPhone's standard map application.)

Quick access to previously viewed locations

The app now keeps track of the locations you view, so you can easily return to them later. Choose your city and click the blue icon that looks like an open book, then click "Recent" at the bottom.

Ability to change search radius for blocks

Block searches now work as they do on our Web site, in that you can change the search radius to one block, three blocks or eight blocks. (Previously, the iPhone app only allowed an eight-block radius.)

Easier navigation on neighborhood list

The neighborhood list now includes an index for browsing by first letter (as in the iPhone's Contacts app).

Support for landscape mode in the Web browser

Just rotate your phone to view Web pages in landscape mode.

Faster performance

We've added a bunch of caching and sped up other bits of the application.

For screenshots of these new features, check out our updated iPhone app page. If you're already using the app, visit the App Store on your phone or computer to get the free upgrade.

We've got some huge news today: EveryBlock has been acquired by MSNBC.com.

If you've been following along with us, you'll know that we were funded for our first two years by a grant from the Knight Foundation. That grant ended on June 30, and we've been spending the last few months figuring out how to keep this site alive and make it sustainable. After considering a number of options (some wildly different from others), we decided that working with MSNBC.com was the best fit for our site and our team.

What does this mean for EveryBlock users? A couple of things.

First, obviously, it means that we're going to keep the site going. We'll continue to run the first and best microlocal news Web site on the planet, with the same six people, with the same logo and design, with the same everyblock.com domain. MSNBC.com has hired our whole team, and they've made it clear to us that we'll be driving the site's strategy and implementation, and that our site will remain an independent destination as a community service. Plus, while MSNBC.com is based in the Seattle area, our headquarters will continue to be in Chicago.

(Of course, we're always looking out for more local information. If you're a government official with local data to share, contact us.)

Second, it means that we'll have resources to expand EveryBlock profoundly. MSNBC.com is the most-visited news Web site in the U.S. and is in solid financial shape in a time when news organizations around the world are struggling. We're excited about the possibilities of pointing a massive audience at EveryBlock and having the resources to beef up our technological infrastructure and staff.

Our site is very young -- it's only been live for about a year and a half -- and we have a lot of ideas and expansion plans. I often tell friends and industry colleagues that EveryBlock in is current incarnation is only about 5 percent of what we want to do with it. We're now in a position to make this happen.

Finally, we'd like to thank the Knight Foundation, the many people who have helped advise us over the last two years, and, of course, the people who use our site. Here's to a bright future.

We've made some updates to our Los Angeles crime data, and the story behind these updates is a good example of the sort of behind-the-scenes work we do at EveryBlock to improve our data sets and advocate open government data in general.

Last week, the Los Angeles Times reported that the Los Angeles Police Department's online mapping system was missing nearly 40 percent of crimes, compared to data the LAPD released separately to the Times. It also reported that the LAPD's public-facing crime site is separate from its internal system (CompStat) — which implies at least two separate databases are being maintained for Los Angeles crime data.

Given that this online mapping system is our source for EveryBlock's Los Angeles crime section, we were pretty interested in finding out whether the LAPD would address this discrepancy and maybe even combine the two systems to avoid the problem entirely.

One part of the problem was addressed quickly. Shortly after the Times story was published, the LAPD added thousands of crime reports to its mapping system — reports from earlier in 2009 that had previously been omitted. After speaking with the Times and the police, we at EveryBlock reimported the entirety of crime data available on the LAPD's crime site, which contains reports going back to the beginning of this year. (EveryBlock users in Los Angeles might have noticed this change in daily alerts today, which might have contained crime reports that were several months old.)

We thank the Times for its reporting on this, as it improved the amount of information available on the city's public crime-mapping Web site as well as on EveryBlock. The crime-mapping system is our only source for the data, as we note in our LA crime section.

A couple of items are worth noting in this episode. In a follow-up story, LAPD officials indicated that they were reviewing their data policies, including their agreement with their crime-mapping vendor as well as their decision to share a separate set of data with the Times.

We have contacted the department and asked that EveryBlock be considered in any new data-sharing solution. We also shared some insights with the LAPD on the provision of crime data, based on the years of work we've done with police departments and other agencies across the country.

At EveryBlock, we seek to have good working relationships with all of the dozens of agencies we deal with in the 15 cities we cover. In the course of our work, we've seen many approaches to the publication of data, especially crime. We suggested that the LAPD take a look at the example of the San Francisco Police Department.

As we note in our San Francisco police calls section, the SFPD publishes a raw data feed, exported from their system. Because it's a single source of data directly from the department's system, it easier to maintain and eliminates the errors that arise from multiple databases.

The Times article noted that Los Angeles Police Chief William J. Bratton "was worried, he said, that he'd have to give it to anyone 'in his underwear in a basement somewhere.'" By adopting a feed like this — and directing all third-party services to use it — we could all get the most complete information, at the same time and from the same source. And that's good for everyone, regardless of where they spend their time or what they happen to be wearing.

Gale Brewer, chair of the Committee on Technology in Government of the New York City Council, has introduced a draft law that would adopt open data sharing standards for the city's government.

At EveryBlock, we've been working with Council Member Brewer's office for more than two years. As we worked to launch our New York City site, we saw that she was instrumental in the passage Local Law 47, requiring the publication of monthly reports regarding data collected on calls made to the 311 system. We saw it as a step in the right direction, and Council Member Brewer as someone we wanted to work with.

Recently, we were invited to submit testimony in favor of this Local Law, with a focus on concrete examples of our experience with the publication of data in New York and other EveryBlock cities. We show that publishing frequently updated, well-structured, complete data sets can be a simple matter.

This is an active piece of legislation. If you live in New York, please contact your council member and express your support for the strongest possible Open Data law.

Here's the testimony we submitted in support of open data standards in New York City:

EveryBlock strongly supports the introduction of a Local Law to create open data standards in New York City

EveryBlock is a neighborhood news site serving 15 cities, including New York. You can see our work at http://nyc.everyblock.com/, where we combine public records from New York City's government with news articles, business reviews, images, and other items collected from across the Web.

In a typical month, we add thousands of crime reports, building permits, restaurant inspections, street closings, business licenses, news articles, and other news for every block in the city. Wherever possible, items are published at the block level, so that people can see what's going on near them. We also offer the public the ability to subscribe to daily updates through e-mail and RSS feeds. We continually search for new ways to add information in more useful ways.

In the course of our work in the past two years, we've worked with city leaders — department heads, council members, technology developers, policy makers, and so on in each of the cities we cover. We share some of what we've learned about data sharing in New York and other places below, and we look forward to working with DOITT, the New York City Council Technology Committee, and other stakeholders on fashioning an effective local law in ways to benefit all New Yorkers. We really hope that this law — and the data published as a result of it — serve as an example for other municipalities.

Some thoughts on the draft language of the Local Law

We reviewed the draft amendment to title 23 of the administrative code of the city of New York as seen here on the Council Web site. Setting aside any word-parsing, we find this to be a strong move forward in open data law for municipalities. There are three areas we see as especially promising:

Raw formats

The provision that "all public records shall also be made available in their raw or unprocessed form" is especially welcome. Very often, municipalities seek overly expensive, complicated technology projects to present data. Leveraging the efforts of citizen developers working with powerful tools is the way to go. As part of the EveryBlock project, for instance, we recently open-sourced the site's backend code.

Structured formats

"All public records shall be presented and structured in a format that permits automated processing." This is a much less complicated requirement than it seems. See below for many examples of structured formats that are available in existing tools and technologies that are widely available in New York and other municipalities.

Frequent updates

The draft local law states that, "All public records shall be updated as often as necessary to preserve the integrity and usefulness of the records." Often, we see municipalities publish information once and fail to update it on a regular basis.

Examples of data published by the City of New York and displayed on EveryBlock

The City of New York already publishes a significant amount of data to its Web site. Here's a quick review of a number of these data types.

Building permit actions

This data is published in Excel spreadsheet format as Job Weekly Statistical Reports by the Department of Buildings. The department updates its data weekly, and we at EveryBlock publish it shortly thereafter.

Sign permit actions

This data comes from the Sign Monthly Statistical Reports published by the New York City Department of Buildings in Excel spreadsheet format. The data is updated regularly and we at EveryBlock publish it shortly thereafter.

Property sales

This data is published in Excel spreadsheet format on the Rolling Sales Update section of the city Web site. The New York City Department of Finance maintains the data, and updates it once a month. In general, the Finance Department deserves a lot of credit for the amount of data it publishes in this format. The department also provides RSS feeds to alert users when new data is published.

Examples of data we'd like to see or existing data that fails to meet standards set forth in this local law

Crime data

This data comes from the precinct reports published by the police department. These reports are not comprehensive (they only include seven crime types), unspecific (they are only collected to precinct level), and infrequent (published weekly). This data lags far behind many other cities in each of these criteria.

311

On EveryBlock, we publish information that we collect from the NYC*scout page, run by the Mayor's Office of Operations. We obtain this data by scraping the maps in the NYC*scout application. This is a tiny sliver of the service requests completed by the city. It would be much better to have formal feed of all 311 data, along with details on the final disposition of service requests.

Graffiti cleaned and Graffiti cleanup requests

This data comes from this database of completed graffiti cleanup locations and this database of pending graffiti cleanup locations, maintained by the Mayor's Community Affairs Unit. It would be better if this data was published in formal feeds with structured formats. This allows for more sustainable methods than scraping Web databases.

Restaurant inspections

This data comes from the online restaurant inspection database published by the Department of Health and Mental Hygiene. It would be better if this data was published in a formal feed with a structured format. This allows for more sustainable methods than scraping Web databases.

Landmark building permits

This data is no longer updated on EveryBlock because the source database, which used to be maintained by the Center for New York City Law CityAdmin search tool, is no longer available. This is a great example of why it's important to have reliable, centralized, well-structured datasets available to the public.

Other technology methods from other cities

Here are some examples of extremely lightweight data sharing at the civic level — next to zero effort with huge utility to developers like EveryBlock.

XML: San Francisco police calls

We worked with the San Francisco Police Department to help them create an XML file that they update for the public daily. The police do this as an export straight out of their CAD system — no development costs, no maintenance.

Database dumps: San Francisco restaurant inspections

The City and County of San Francisco does a .mdb file database dump in their native format and publishes it to an FTP server with access to EveryBlock and other approved entities.

Text files: San Jose building permits

The City of San Jose publishes raw text files like this directly to its Web site on a weekly basis.

Conclusion

We see this ordinance as a welcome next step in New York City government. If affirms the importance of data sharing, provides clear instruction on what should be published, and begins to set technology standards for format and structure.

Today's a big day for us at EveryBlock. We're making our source code available.

Over the past two years, EveryBlock has been funded by a grant from the Knight Foundation. The purpose of the grant was twofold: to launch this experiment in "microlocal" news, and to release the source code. Today, as our grant period comes to an end, we're fulfilling that second purpose.

You can read more about the open-sourcing and download the code at our source code page. (Keep in mind it'll probably make sense only if you're a Web developer/programmer.) We hope this extensive code base helps spark lots of great work.

But what about EveryBlock.com proper, now that the grant period is over? We've put a lot of love into this project over the past two years, and we're going to continue operating the site as a private company. Beyond continuing our steady expansion of new cities and more data types in existing cities, we have some exciting ideas planned around revolutionizing the whole EveryBlock experience itself. We're only getting started. :-)

We expect this maintenance to take a few hours. We’ll update this post and Twitter when the site is back up. Thanks for your patience.

Update: The maintenance period is over, and the site is back to normal.

Today we're launching beta versions of EveryBlock in four cities: Atlanta, Dallas, Detroit and Houston.

Why the "beta" designation? We got so much demand for these cities in our city expansion poll that we wanted to get something out there for people to start using — but we know we have more work to do in finding new data sets, news sources and neighborhood boundaries. Each city has a smaller amount of news available than in our other cities, but we plan to expand over time.

Take a look, let us know what you think (there's a feedback form at the bottom of each page), and tell your friends/family in these cities!

Today we're officially launching EveryBlock accounts — our first step at letting you customize the information you see on our site. You can create a free account by clicking the "Register" link in the upper right of any EveryBlock page.

For launch, we offer three simple features to registered users: e-mail alert management, saved places and timeline display preferences. Here's a quick rundown of each feature:

E-mail alert management

We've offered e-mail alerts for a long time, but they've been one-offs — i.e., you had to sign up for each one individually, and there was no way to manage them centrally. Now, when you sign up for an e-mail alert, it'll be associated with your account, so you can view and manage all your alerts in one place.

(We've created an EveryBlock account for each existing e-mail alert subscriber. If this is the case for you, you don't have a password yet; use the "Forgot your password?" link to get a fresh password e-mailed to you. Then you can change your password to one of your choosing.)

Saved places

This is a small but useful feature: if you're viewing a block, neighborhood or ZIP code page while you're logged in, you can click "Save this place" for easy access in the future. Once you save a place, it'll always be available in the "Saved places" section of the navigation (which is only visible if you're logged in).

Timeline display preferences

Another small but useful feature: you can now hide specific types of information from EveryBlock timeline pages. Say, for example, that you never want to see business reviews; just click the "Hide" link next to that section in the timeline, and all business reviews will be hidden from then on. You can turn them back on by going to your dashboard page (the first thing you see after you log in).

Next steps

These are just a few small features people have requested over the past few months. We're thinking about much more substantial ways for people to customize and contribute to EveryBlock. Stay tuned, and drop us an e-mail at feedback at everyblock.com if you've got any suggestions.

We've made a change to our San Francisco site to cut down on information overload.

On our San Francisco site, we publish calls made to the San Francisco Police Department. It's interesting data — but there's a lot of it, and it can get overwhelming, particularly in very large neighborhoods. With that in mind, we've changed our system to collapse them into daily totals rather than displaying each individual police call.

Here's what it used to look like (taken from the page for San Francisco's Mission neighborhood).

Note that there were 256 police calls on this given day, so this screenshot doesn't really do justice to the sheer length of the data. Because each police call was displayed on its own line, this resulted in some huge lists.

Here's what the same data looks like now:

It's much more compact and gives you a higher-level look at the data. Of course, you can click the appropriate category (e.g., "3 gun-related calls") to view all of the individual records.

We'll be applying similar treatment to a couple of other data sets across our 11 cities, and we're going to continue to experiment with different approaches to collapsing the data intelligently. If you've got any ideas, let us know by e-mailing feedback at everyblock.com.