The EveryBlock Blog

At EveryBlock, where my main role is to work with municipal governments to uncover new data sets, we're experimenting with a new form of journalism where we treat freshly updated public records as block-level news. It's a big job to acquire ongoing feeds of government data, and we have a broader goal of spreading the gospel of open data.

The two objectives:

• Get more datasets for EveryBlock so it can be a better Web site
• Convince governments to share that data with everyone, not just us

can lead to some cognitive dissonance in the minds of government leaders. They have two dominant templates of interaction for a project like ours, and neither of them fit very well.

First and foremost, we're a journalism project. The template for government dealings with media usually revolves around a particular document or dataset that is frozen in time. The data will most often lead to a single exclusive article or series of articles for a single publication or media company. When I tell them that we want all of the data, everyday, forevermore -- preferably published somewhere over the Web so that others can use the data as well -- this message just doesn't compute.

The other template that governments bring to EveryBlock is that of a technology vendor. They see our site and immediately grasp its utility. They naturally begin to think in terms of EveryBlock serving whatever technology or public communications goals they're currently working. When they start applying this template to us, I have to steer them back to our fundamental nature -- that we're journalists executing on a long-term project, not software vendors looking to lock them into a long-term contract.

This is a lot of heavy lifting, so it's a good thing EveryBlock is just one of many projects that seek to make government data more available, useful and usable. Here's a few of the efforts we've made recently with others:

• In December of last year, we met with a couple dozen other organizations to brainstorm the 8 Principles of Open Government Data. The principles attempt to define "open" in the context of the principles and lay down some fundamental definitions for primary terms like "public" and "data". I've been able to use this set of principles in my work with municipalities as we seek to get more data for EveryBlock. It's a very powerful message to tell them that I am not there just as an EveryBlock person. I tell them that it's not about us, it's about the data
• Last month, I participated in a panel for the Open the Government and Sunshine Week called, "Citizen Self-Help: Finding the Information You Need". We shared a number of tips & tricks on gathering information and talked about the challenges in presenting large data sets in ways that make sense. Sheila Krumholz (Center for Responsive Politics/OpenSecrets.org), David Moore (OpenCongress.org), and Sean Moulton (FedSpending.org) were the other panelists. One of the best questions asked by moderator Greg Elin of Sunlight Labs was, "what if all government data was suddenly made open? would your work be done?" See the entire panel discussion here
• EveryBlock is the local host for the Independent Government Observer's Task Force, a "non-conference" structured around 3 sets of working group activities: Case Law, Municipal Governments, and Government and Copyright Issues. We're looking forward to having a wide mix of people inside and outside government who want to make civic data more useful. I am helping organize the Municipal Government Working Group, so if you have some ideas and want to attend, let me know at at danx at everyblock.com or 773-321-8146.

More on the IGOTF as the summer wears on.

We've added Chicago building permits to EveryBlock.

The data comes from the Chicago Department of Buildings and is updated on the fifth of every month. We worked closely with the DOB over the last few months to get access to this data, which has not been available for years and has never been presented in this format. We thank them for their cooperation and efforts—it is a great example for others.

We've added the ability to customize per-block and per-neighborhood RSS feeds.

Since January, we've offered RSS feeds for every block and neighborhood in each EveryBlock city. (Confused? Visit What is RSS? for an explanation.) Until now, those feeds have included every news item—crimes, news articles, restaurant inspections, and everything else—and there was no way to opt out of particular types. Now there is.

On our new feed customization pages (example), you can specify exactly which types of news you'd like to appear in your feed. So, for example, if you live in a restaurant-heavy part of town and don't want to get overwhelmed with business reviews from Yelp, just uncheck that box to get a feed that excludes business reviews. You can exclude as many news types as you like; we'll give you a single feed that includes exactly what you want and excludes exactly what you don't want.

As you might be able to tell by reading this blog, we regularly add new types of information to EveryBlock. By default, our block and neighborhood RSS feeds automatically include new data types as soon as they're added. But if you're looking for a stable, specific feed—like, say, all restaurant inspections in North Beach, for use on your North Beach restaurant blog—you can get a feed that only includes the information you requested, with a guarantee that new data types won't be added to that feed. To do this, uncheck the box that says "When new data types are added, include them in the feed automatically."

To access the new feeds, search for a block or neighborhood, then click "Custom RSS feeds" in the upper right. If you have any comments or suggestions about this new feature, please get in touch with us either through the contact form at the bottom of each EveryBlock page or by e-mail at feedback at everyblock.com.

We've added Chicago property transfers to EveryBlock.

The data comes from the Cook County Recorder of Deeds and is updated every two weeks.

We’ve fixed a problem with pages not loading correctly in certain situations under Internet Explorer.

If you use IE—particularly version 6 or 7—you may have noticed that sometimes certain pages would not load, or would load slowly. This was the result of a configuration problem with our web server, which we’ve now corrected. We apologize for any inconvenience this caused you. And please don’t hesitate to let us know if you‘re having a less-than-optimal EveryBlock browsing experience.

We've added real estate listings to all three EveryBlock city sites -- Chicago, New York and San Francisco.

The data comes from Trulia and is updated twice a day. In San Francisco, where we'd already been publishing real estate listings; we've simply added Trulia's data to the mix.

We've added New York sign permit data to EveryBlock.

In short, this means that New Yorkers can now get notified whenever a new sign goes up in their neighborhood.

This data comes from the Department of Buildings, which issues permits for new outdoor signs. Following our "newspaper for your block" philosophy, we're publishing each "sign permit action" as a separate news item -- i.e., each step of the approval process, such as a DOB approval, is news to you, if you're near it.

We publish a number of interesting bits of data about each sign permit, including the text of the sign ("EAT AT JOE'S"), whether/how the sign is illuminated and the sign's size.

For those of you EveryBlock fans who are technically inclined, you may be interested in the latest issue of A List Apart (a popular Web development magazine) which features two articles written by EveryBlockers.

Both articles explain some of the behind the scenes work that went into building some features of EveryBlock.

In Take Control of Your Maps, Paul Smith expands on his earlier post about our maps with an in-depth description of how to build high-quality custom maps with freely available tools.

In Accessible Data Visualization with Web Standards , Wilson Miner talks about how we built some of the charts on EveryBlock (like the ones on this overview of crime in Chicago) with standards-based HTML and CSS.

In addition to the new search features we launched yesterday, we've also added maps to the custom filter pages.

Now you can use the custom filter to drill down to just the information you're looking for, and you'll see your search results on a map as you browse.

We've also added some controls to customize the size of the map on the page. It starts out small, but you can make it wider or taller to make room for more information on the map.

We've added two much-requested features to our custom filter pages: text search and address/block range search.

Custom filter pages (example), the data-lover's dream, let you mix and match search criteria for any data type in any of our cities. For example, check out the custom filter for San Francisco building permits, which lets you ask questions such as which San Francisco theaters have received demolition permits?

We've had custom filter pages since we launched two months ago, but we just added two key improvements:

• Each custom filter page now has an "Address" filter, which lets you limit the results to those near a particular address.
• Some types of information, such as Chicago business licenses, now offer a text search. In the case of business licenses, you can search for a business by name.

We've got more custom-filter improvements coming, too. Stay tuned, and contact us if you have any suggestions.

Thanks to data from our friends at Redfin, we've added real estate listings to the mix in our San Francisco site. This means we'll notify you of any new Redfin real estate listings in your San Francisco neighborhood.

San Francisco is the only EveryBlock city that's currently served by Redfin, but we look forward to working with them as both of our services expand to other cities.

Here's an example of what this data might look like on your block page:

Now that EveryBlock has been live for about two months, we've accumulated a number of frequently asked questions and created a FAQ page. It's linked-to from the bottom of each page.

We've slightly redesigned our news index pages to make the various browsing methods easier to find.

Previously, we listed the browsing methods — "By date," "By neighborhood," etc. — in small blue links at the top of news index pages, like this:

Now, we've made those links bigger and given example searches under each one, like this:

Some folks might recognize this as an homage to our previous project chicagocrime.org, which is no longer running but had a very similar homepage look:

Thanks to the EveryBlock users who requested this feature, both directly (by asking for it) and indirectly (by reporting they were having trouble finding stuff). Please keep the feedback coming.

We recently rolled-out a change to our maps that improves the way we display multiple news items.

Previously, every news item would get its own marker, indicating the location of the news item. You could click on the marker and a small window would pop-up, giving you the headline, link, date, and a short description of that news item. The problem is that there are often several news items with the exact same or nearly the same location: when you click on a marker to get more information, you can’t differentiate between them if there are more than one on the same spot. In some cases, you could zoom in to tell apart news items that were merely near to each other. But that wouldn’t help in cases where they were at the exact same location.

Our solution was to cluster nearby news items into a single, larger marker. The size of the marker indicates the number of news items it is clustering: the larger the marker, the greater the number of news items. (We also put the exact number on the marker for precision.) The position of the cluster marker on the map is the average of the locations of the news items it contains. And the pop-up window that shows you more information has buttons for paging through the news items.

We will continue to revisit our maps for areas of improvement, especially when it makes the information we’re presenting easier to read and understand. We’d love to hear what you think about them.

We've added our very first batch of real estate data to EveryBlock: New York property sales.

Using data from the city's Department of Finance, we've given it the full EveryBlock treatment, making it browsable by sale date, by neighborhood, by borough, by ZIP code, by building class, by building class category and even by year built. As with all EveryBlock's public records, we also offer a custom filter that lets you mix and match search criteria, and, of course, property sales now show up on your block and neighborhood news pages, alongside everything else we publish.

The data is updated on a monthly basis. This is significantly less frequent than some of the other information we publish, but we're limited by the data-release schedule of the Department of Finance. If you have access to other sources of real estate data in New York, Chicago or San Francisco, please get in touch by e-mailing feedback at everyblock.com. We'd love to add it to the mix.