The EveryBlock Blog

Today's a big day for us at EveryBlock. We're making our source code available.

Over the past two years, EveryBlock has been funded by a grant from the Knight Foundation. The purpose of the grant was twofold: to launch this experiment in "microlocal" news, and to release the source code. Today, as our grant period comes to an end, we're fulfilling that second purpose.

You can read more about the open-sourcing and download the code at our source code page. (Keep in mind it'll probably make sense only if you're a Web developer/programmer.) We hope this extensive code base helps spark lots of great work.

But what about EveryBlock.com proper, now that the grant period is over? We've put a lot of love into this project over the past two years, and we're going to continue operating the site as a private company. Beyond continuing our steady expansion of new cities and more data types in existing cities, we have some exciting ideas planned around revolutionizing the whole EveryBlock experience itself. We're only getting started. :-)

We expect this maintenance to take a few hours. We’ll update this post and Twitter when the site is back up. Thanks for your patience.

Update: The maintenance period is over, and the site is back to normal.

Today we're launching beta versions of EveryBlock in four cities: Atlanta, Dallas, Detroit and Houston.

Why the "beta" designation? We got so much demand for these cities in our city expansion poll that we wanted to get something out there for people to start using — but we know we have more work to do in finding new data sets, news sources and neighborhood boundaries. Each city has a smaller amount of news available than in our other cities, but we plan to expand over time.

Take a look, let us know what you think (there's a feedback form at the bottom of each page), and tell your friends/family in these cities!

Today we're officially launching EveryBlock accounts — our first step at letting you customize the information you see on our site. You can create a free account by clicking the "Register" link in the upper right of any EveryBlock page.

For launch, we offer three simple features to registered users: e-mail alert management, saved places and timeline display preferences. Here's a quick rundown of each feature:

E-mail alert management

We've offered e-mail alerts for a long time, but they've been one-offs — i.e., you had to sign up for each one individually, and there was no way to manage them centrally. Now, when you sign up for an e-mail alert, it'll be associated with your account, so you can view and manage all your alerts in one place.

(We've created an EveryBlock account for each existing e-mail alert subscriber. If this is the case for you, you don't have a password yet; use the "Forgot your password?" link to get a fresh password e-mailed to you. Then you can change your password to one of your choosing.)

Saved places

This is a small but useful feature: if you're viewing a block, neighborhood or ZIP code page while you're logged in, you can click "Save this place" for easy access in the future. Once you save a place, it'll always be available in the "Saved places" section of the navigation (which is only visible if you're logged in).

Timeline display preferences

Another small but useful feature: you can now hide specific types of information from EveryBlock timeline pages. Say, for example, that you never want to see business reviews; just click the "Hide" link next to that section in the timeline, and all business reviews will be hidden from then on. You can turn them back on by going to your dashboard page (the first thing you see after you log in).

Next steps

These are just a few small features people have requested over the past few months. We're thinking about much more substantial ways for people to customize and contribute to EveryBlock. Stay tuned, and drop us an e-mail at feedback at everyblock.com if you've got any suggestions.

We've made a change to our San Francisco site to cut down on information overload.

On our San Francisco site, we publish calls made to the San Francisco Police Department. It's interesting data — but there's a lot of it, and it can get overwhelming, particularly in very large neighborhoods. With that in mind, we've changed our system to collapse them into daily totals rather than displaying each individual police call.

Here's what it used to look like (taken from the page for San Francisco's Mission neighborhood).

Note that there were 256 police calls on this given day, so this screenshot doesn't really do justice to the sheer length of the data. Because each police call was displayed on its own line, this resulted in some huge lists.

Here's what the same data looks like now:

It's much more compact and gives you a higher-level look at the data. Of course, you can click the appropriate category (e.g., "3 gun-related calls") to view all of the individual records.

We'll be applying similar treatment to a couple of other data sets across our 11 cities, and we're going to continue to experiment with different approaches to collapsing the data intelligently. If you've got any ideas, let us know by e-mailing feedback at everyblock.com.

We've released an EveryBlock iPhone app, which combines the iPhone location-awareness with EveryBlock's rich assortment of local news. It's available for free now at the App Store.

We've written much more about it, including screenshots, over here. Check it out!

We've launched a number of improvements today that help solve two problems: ungeocoded data and inaccurately geocoded data. Some of this is unavoidably technical, but I'll try my best to explain it without getting too geeky.

A quick geocoding primer

Geocoding is the process of converting an address into a longitude/latitude pair. For example, the address 53 W. Jackson, Chicago, IL, corresponds to the longitude/latitude point (-87.6296034, 41.87813100). You can geocode any U.S. address for free using tools such as geocoder.us — enter an address there, and the site will tell you its longitude/latitude.

Geocoding is essential to what we do at EveryBlock, because it makes geographic querying possible. Once you have a set of longitude/latitude points, a computer can place them on a map, calculate which points are within a given area and determine which points are near a given block.

But geocoding isn't perfect. To get the longitude/latitude for an address, geocoding software relies on large databases that map streets to points — and these databases inevitably omit certain addresses or streets. As new buildings and streets are built, these databases can be slow to catch up. (If you've ever searched MapQuest or Google Maps for a valid address that it couldn't find, you've come across this issue.)

Another problem is that of misspellings and ambiguity. If you make a typo in a street name, an overly strict geocoder may not be able to correct the misspelling. An address like "800 Halsted" in Chicago can't be geocoded with certainty because both 800 N. Halsted and 800 S. Halsted exist as distinct places.

Finally, a fundamental problem with geocoding is that it's usually interpolative. A standard geocoding database, like the one we use at EveryBlock, doesn't map every address to a longitude/latitude — it only maps address ranges to longitude/latitude lines. A geocoder then works by searching the database for the appropriate address range, then doing a bit of arithmetic to figure out where on the line the exact address is. (The logic goes something like this: "If the addresses in this particular range go from 128 to 156 Main St., then the address 142 Main St. is exactly between those numbers, which implies the longitude/latitude point is exactly in the middle of that line.") The problem here is that the technique assumes addresses/houses are arranged uniformly on a given block, but different houses and city lots have different sizes — and, hence, the resulting longitude/latitude may be a couple of meters off. However, this interpolative approach is an acceptable compromise given the lack of availability of more granular geographic data. If you look closely, you'll notice this problem on even the largest mapping sites like Google Maps.

On EveryBlock, if we cannot geocode a given piece of news or data, we're not able to map it, and it won't show up on block, neighborhood or ZIP pages. It'll only be accessible if you browse by type of information. For example, this restaurant inspection in New York City has an address of "000 TERMINAL 5, QUEENS," which our system couldn't geocode. (Looks like it's a food establishment in JFK Airport.) If you live near this establishment, you won't see it on your EveryBlock neighborhood page, because it hasn't been geocoded and, hence, the system doesn't know that it's geographically near you. However, it's still accessible if you search for restaurant inspections by name or drill down by date.

With this background information in mind, here are the two improvements we've made today:

Improvement #1: Handling inaccurately geocoded data

In some cases, our data sources provide us the raw longitude/latitude point for each piece of data. For example, the Los Angeles Police Department includes longitude/latitude points in its crime data, which we publish on our L.A. site. We like it when this is the case, because it saves us from having to geocode the addresses on our end, and because we've found from experience that government agencies' geocoding systems are generally higher quality than our own, in-house geocoder. But we've recently learned that relying on our data sources' geocoding results isn't always accurate.

The Los Angeles Times got in touch with us on Friday afternoon to point out that some of our L.A. crime data was incorrectly geocoded to a point near City Hall. We confirmed that our source for the data, the LAPD's crime Web site, was also incorrectly geocoding the same crimes to the same incorrect point — so although we weren't originating the error, we were perpetuating it. On Sunday, the Times wrote an article about the LAPD's map glitch and called us out on it.

Thanks to this heads-up from the Times, we've now improved our system to fix the problem. From now on, rather than relying blindly on our data sources' longitude/latitude points, we cross-check those points with our own geocoding of the address provided. If the LAPD's geocoding for a particular crime is significantly off from our own geocoder's results, then we won't geocode that crime at all, and we publish a note on the crime page that explains why a map isn't available. (If you're curious, we're using 375 meters as our threshold. That is, if our own geocoder comes up with a point more than 375 meters away from the point that LAPD provides, then we won't place the crime on a map, or on block/neighborhood pages.)

For example, this crime from Jan. 1, 2009, was reported at Western Ave. and Pico Blvd., but the LAPD's geocoder placed it near City Hall — more than three miles away from that intersection. Because of this large discrepancy, we're not mapping the crime, and we're displaying a message on the crime's page that explains this.

We were curious about the LAPD system's geocoding accuracy and did a bit of number crunching. Here's a map of LA crimes where the LAPD system's longitude/latitude was more than one kilometer away from the result of our own geocoder. (This does not include any crimes whose addresses could not be geocoded by our own geocoder.) The numbers on the map represent the number of crimes at that given cluster, and we used the longitude/latitude points from the LAPD's system.

Starting today, none of these crimes will be mapped on EveryBlock, because the data is too inconsistent to be trustworthy.

This change goes a long way in preventing further inaccuracies of this sort. Now, for an L.A. crime to be mapped, it must not only have an LAPD longitude/latitude — it must also be geocoded by our own internal geocoder database such that the two geocoder results are significantly similar.

Improvement #2: Surfacing ungeocoded data

The second change we've made has to do with records on our site whose addresses could not be geocoded (and, hence, are not mapped). We want to be clearer about how many records can't be geocoded, so that EveryBlock users examining our aggregate statistics can keep that caveat in mind.

Starting today, wherever we have aggregate charts by neighborhood, ZIP or other boundary, we include the number, and percentage, of records that couldn't be geocoded. Each location chart has a new "Unknown" row that provides these figures.

Note that technically this figure includes more than nongeocodable records — it also includes any records that were successfully geocoded but don't lie in any neighborhood. For example, in our Philadelphia crime section, you can see that 1% percent of crime reports in the last 30 days are in an "Unknown" neighborhood; this means those 35 records either couldn't be geocoded or lie outside any of the Philadelphia neighborhood boundaries that we've compiled.

We hope this article, along with these new site changes, helps you get a better understanding of data caveats throughout our site. Let us know if you have any other suggestions by e-mailing feedback at everyblock.com.

EveryBlock users in Charlotte might have noticed something odd last month.

In the eight days beginning March 11, the Charlotte-Mecklenburg Police Department reported nearly 6,800 Charlotte "zone checks" — the department's term for checking problem spots for crimes or patrolling areas at residents' requests.

The number seemed strange, because the department only reported about 600 zone checks in Charlotte in the previous nine-plus months. And from March 19 to the end of the month, no zone checks were reported at all.

The numbers come from EveryBlock Charlotte's police calls for service section, which collects and maps call data from the department's "calls for service" page. With sites in 11 cities, we at EveryBlock often see data oddities, and when we do, we contact the agency in charge of the information for clarification. We called and e-mailed the police department, and asked what — a data problem, a new classification system, clerical error, etc. — might be going on. It took about a week, but we got some answers, from police spokesman Brian Cunningham.

During some maintenance, the department changed the way data was filtered for its Web site, such that all zone checks were being published. "It's now been reversed," Cunningham said.

The department's "calls for service" page states that it shows a "summary of calls for service made to a 911 operator and dispatched to a CMPD officer in the Charlotte area." Despite the reversal, that page could still display a small number of zone checks in the future, just as it did before March 11, Cunningham said. He wasn't sure why some "trickle through," but displaying all zone checks, as was happening during the eight-day spike, would be "overwhelming," he said.

There's something to that description. The 1,277 zone checks in Charlotte on March 17 actually is a higher number than the total number of police calls on March 19, the first day after the spike.

The spike also made zone checks the most common event in our police-calls section — even though it didn't crack the top 10 three weeks ago.

So, you can see why we're glad the department provided some answers, especially given Police Chief Rodney Monroe's aggressive strategy since being appointed last year, and recent questions surrounding police statistics in Charlotte. Early last month, Deputy Chief Kerr Putney chafed when asked whether a drop in crime rates in January and February stemmed from the department classifying results differently.

"We want to be in people's faces who are committing the crime," Putney told the Observer. "We're not sitting back and waiting for the calls.

"It annoys me — the implication that some of our reporting techniques have changed."

In the face of a major data oddity, we think the department's responsiveness could prevent others from drawing the wrong conclusions in this case. And we hope the department's actions — combined with our practice of checking out of the facts in these cases — will help EveryBlock users in Charlotte.

Today I testified before the Chicago City Council Joint Committee on Finance and Economic, Capital and Technology Development to urge them to pass the TIF Sunshine Ordinance co-sponsored by Alderman Scott Waguespack and Alderman Manny Flores.

"TIF" stands for "Tax Increment Financing", which is a financing method used by the City of Chicago to encourage development in certain areas. The wide use of TIFs in Chicago has led some to question where the money is spent and why, so it's a pretty hot topic in the EveryBlock hometown. The ordinance proposed today calls for some modest changes in the way TIF-related documents are published and maintained on the city Web site.

This kind of work is right on target for EveryBlock. Part of our mission is to make as much information as possible available in the most useful formats. We're also members of the Illinois Data Exchange Affiliates, a voluntary group of government agencies and not for profit organizations working to improve data sharing in Illinois. This group has worked for years on the very subject covered by this ordinance. As part of the testimony, we also filed a copy of the 8 Principles of Open Government Data with the City Council, and urged them to take these first steps in the right direction. EveryBlock founder Adrian Holovaty and I participated in a meeting of 30 open government advocates to develop these principles, and they are highly instructive in any effort to publish government data today.

There was an abundance of other testimony arguing that adherence to the ordinance would require very little city resources of time and money that would provide outsized benefits to the people of Chicago. University of Illinois at Chicago Associate Professor Rachel Weber and consultant Valerie Leonard spoke eloquently of the value of the data in academia and community development. And a live demonstration of the city's existing Web site by the Deputy Commissioner of the department responsible for administering TIFs went rather poorly.

Despite all of this the ordinance was tabled. We think this was a terrible outcome, but we have hopes for eventual passage next month. And we will continue to work to bring sunshine to all of the cities we work in.

Starting last week, we began hearing from users in San Francisco noting that their EveryBlock pages and e-mail alerts no longer included police calls. We investigated and found out our data source had stopped updating around March 3. We got in touch with Lieutenant Greg Yee of the San Francisco Police Department, and he looked into it.

They were able to fix the feed in short order, and we again have an uninterrupted record of San Francisco police calls on EveryBlock. We appreciate the quick work of Lt. Yee and the SFPD.

One thing casual EveryBlock users might not know is that we work to develop and maintain good relationships with government agencies in all of the cities we cover. In this case, we had worked with the department for more than a year before they created a public data feed, and we're glad we were able to notify them about the issue as soon as we found out.

Thanks to everyone who contacted us. You may have noticed a plethora of police calls in your e-mails this afternoon, but fret not — it's just catch-up!

We've added a new type of information to each EveryBlock city: local deals and coupons.

Now, you'll get notified whenever there's a new coupon for a local business near you. Check out this Los Angeles block page for an example, and check out this overview of coupons in Washington, DC, on Feb. 10.

The information is updated every day and comes from ValPak, a company that provides neighborhood coupons across the country.

Additionally, we have a special treat for our users in Chicago. Our friends at the Daily Groupon are also contributing their daily Chicago deals to the mix, so you'll get notified whenever there's a Groupon deal near you.

Two stories about EveryBlock were in the Chicago media today:

• ChicagoWorksForYou.com Never Hits the Web: Chicago Public Radio interviews our people person Dan O'Neil. (Audio is available.)
• Big ideas, small profits: Crain's Chicago Business covers us in a video about Chicago's thriving local-news startup scene. Gotta love the headline.

Hands down, the most common feedback we get is "Please bring EveryBlock to my city!" Now, we've finally added a formal way to cast your vote for EveryBlock in your part of the world.

The main EveryBlock homepage now has a "Don't see your city?" link that'll take you to a simple form to cast a vote for your city. We'll use these votes to help guide our future plans.

We're excited to announce a collaboration with The New York Times and a new section on EveryBlock New York City: political news items.

One of our goals at EveryBlock is to notify you when local news sites or blogs mention something in your neighborhood. This far, we've focused on specific addresses and places, which you can see in our "locations in the media" section. There, we catalog local news coverage to figure out which articles mention which particular places.

Now, we're taking this philosophy further by applying it to local elected representatives. This new section, "political news items," notifies you whenever your local elected representatives — such as your city councilman or state assemblyman — are mentioned in The New York Times.

Here's an example of what this might look like, from the EveryBlock page for 170 5th Ave. in Manhattan:

This is made possible by a collaboration between EveryBlock and The New York Times, whose excellent Represent site catalogs Times news coverage by elected representative. The Times makes its data available to us so that it can get added to the local news mix on EveryBlock.

Enjoy the new feature, and, as always, let us know if you have any ideas for more local information to add to our site by e-mailing feedback at everyblock.com.

One year ago today, we flipped the switch and launched EveryBlock to the world.

There are lots of ways to measure how far we’ve come since that first day. We launched in three cities, and today we cover eleven cities across the U.S. We started with 37 types of data, and today we have more than 130 unique data types, with 602 different sources of news and blogs alone.

But probably the most obvious sign that we’ve made some progress over the last year is also the most visible. We’ve got a kick out of digging up screenshots from launch day and comparing them to what the site looks like now. We thought it would be fun to share, and take a quick look back to celebrate our first birthday.

City homepages

The original city homepage design had one thing going for it: it was simple:

From the beginning, we wanted the homepage for each city to get people to the place they were looking for as soon as possible. We must have done something right, because the page was almost entirely navigation. We’ve added a lot of content to the site since then, and we’ve learned a lot about how people use the site.

In September, we redesigned the city homepages to include a lot more content up front and give a better idea of what the site is all about. Here’s what it looks like now:

Section pages

The section pages on EveryBlock let you drill down into one particular type of content (like crimes, articles or restaurant inspections) and get an overview of trends across the entire city.

When we launched, the goal of the design was to be as flexible as possible, since we didn’t always know what it would look like for new types of data.

We redesigned the section pages in October to make the layout more consistent and to make the navigation more prominent and easy to use. In November, we revamped the entire system for searching and browsing data on EveryBlock and integrated that “custom filter” with the section page navigation. Here’s what it looks like now:

Place pages

Probably the most important pages on EveryBlock are what we call “place pages.” When you search for an address or browse to a neighborhood, this is what you get. The core of it is a timeline of the most recent content near that place that we’ve found on EveryBlock.

In the beginning, it was about as bare-bones as you could get.

Once you get past the enormous icons, it’s amazing just how basic the information is. The map is on a separate page, and we didn’t even have thumbnails for the photos! Fairly soon after launch we created a system so that we could customize how each different type of content appeared in these lists. That allowed us to show thumbnails for photos and color codes for restaurant inspections, and generally show more relevant information in context. We also eventually came to our senses and made smaller versions of the icons (thanks to the multi-talented Nathan Borror, who designed all the icons for EveryBlock) and put the map back next to the list where it belonged.

More recently (just a couple of weeks ago, in fact) we added a new “overview” feature to the place pages in addition to the main timeline view that lets you see trends and breakdowns for different types of information in the area. To put it into perspective, all the citywide information we had on the section pages when we launched is now available for every block and neighborhood in the city.

We’re really proud of how far EveryBlock has come since we launched a year ago, but we can’t take all the credit ourselves. From day one, we’ve received great feedback and ideas from our users. Whether it’s something you liked, or something we needed to fix or improve, it’s all been incredibly helpful as we continue to develop the site. Keep it coming, and thanks to everyone for spreading the word about EveryBlock.