Skip to content

Instantly share code, notes, and snippets.

@auremoser
Last active January 5, 2018 22:20
Show Gist options
  • Save auremoser/da1b8fe8de23fe6365923b1a4d8f7b7d to your computer and use it in GitHub Desktop.
Save auremoser/da1b8fe8de23fe6365923b1a4d8f7b7d to your computer and use it in GitHub Desktop.
DSI - Maps Class

Intro to Mapping Data

###SVA-DSI ###Metrics and Data Visualiation I October 11th, 2016

Satellite ProPublica, Losing Ground.

Find this document here:

Outline

  1. Mapping Basics
    • You are Here.
    • Types + Topics
    • Anatomy of a Webmap
  2. Mapping Data
    • Data + Date Formats
    • Collecting + Sync Tables
    • Storing
  3. Mapping Tools
    • Examples
    • Tour of the interface
    • APIs / JS Libs
    • Data Import + Sync Tables
    • Customizing UI + CartoCSS
  4. Creating Custom Maps
    • Mapping in the Editor
    • Mapping in Code
      • Python
      • Node
      • R
  5. Building A Narrative
    • Case Study 1: NYC Building Choropleth Map
    • Case Study 2: NYC Bikelane Category Map
    • Cast Study 3: Multivariate Data Map App
    • Tell Time/Stories: Odyssey + Torque
    • Partner Graphics: Graphs + Charts
    • Bl.ocks for Mapping display
  6. Resources

YOU ARE HERE.

SVA

Where are you? If I gave you directions here, you could say you were at

132 W 21st St, New York, NY 10011

But if I wanted to describe the location as a point on a map, I might describe the same location using latitude and longitude:

40° 74' 20.70", -73° 99' 53.37"

It reads as "40 degrees, 73 minutes and 53.04 seconds, negative 73 degrees, 99 minutes and 41.35 seconds"

We describe locations with a spatiotemporal notation, it's also somewhat obscure. So let's describe the same place using longitude and longitude but using decimal degrees instead of minutes and seconds. There are a number of conversion tools available online to do this:

40.742070, -73.995337

###Types + Topics

Gallery of maps

There are many different types of map you can make, browse this gallery to view some options. Most maps illustrate some kind of geospatial change.

DYNAMIC MAPSs Time Travel Map

Source: Time Travel Between Counties, Carto.JS

WEATHER MAPS Realtime Precipitation

Source: Realtime Precipitation Map | NOAA Weather data

Realtime Climate Tracking

Source: Vizonomy + NOAA collaboration on Asterra's Climate Dashboard

TRAFFIC MAPS Realtime Traffic Map

Source: Traffic Change Data for SF | Municipal Traffic in SF App | Carto + Firebase

CRISIS MONITORING MAPS

Realtime Earthquake Map

Source: Andy's Earthquake Workshop

DATA SEARCH TOOLS GDELT Geographic News Search Tool

Source: GDELT Geographic News Search Tool

Journalists have used the GDELT data to track wildlife crime, and the spread of the Islamic State in the Middle East among other things.

You can fork the GDELT hourly synced data set from the Carto DataLibrary and add it as a layer on your map or use the Geographic Search Tool linked above to search for tags of interest.

CHART GRAPHICS

County Chart

Source: Geogia County Car Crash Counts, C3.JS

COMPARISON NARRATIVE

Michael's Syrian Refugee visualization Map population by relative density

  • Maps give you more context than most visualizations.
  • They allow you to apply data to a recognizable topography.

Sources

  • Carto: light open source library and graphical user interface application for hosting and visualizing geospatial data
  • ChartJS: light library for creating charts and graphs
  • GDELT: the global database of events languages and tones

Anatomy of a WebMap

Webmap Gif

Maps are composed of:

####1. Tiles (often)

Stamen Tiles

Tiles are 256x256 pixels squares that plot one next to the other to stitch together a pseudo-seamless image; the tiles draw dynamically as you focus and zoom in on your map so that the map draws on a continuous canvas

zoom pyramid

####2. Feature Layers

Map layers that populate on top of your basemap

####3. Javascript/HTML/CSS for rendering on the web

With these languages you can publish your map with the basetiles loaded and your data layers appropriately geocoded; with javascript you can also add to the interactivity of your map, revealing metadata in the tooltips

Mapping Data

Formats + Free Sources

You can get geospatial data for your feature layers in lots of places, here's a nice resource with a list of common data formats that you might encounter. Here are some examples.

Collecting + Sync Tables

We'll be collecting data from multiple sources. Most of which are available in the repository where our data will be stored. In this case, point (library), lines (bikelanes), and polygons (building footprints).

BIKELANES

LIBRARIES

Overpass

You can read more about Overpass on the Open Street Map Wiki.

Steps:

  • Pan manually to an area in OverPass Turbo
  • Go to the "Wizard"
  • Look up the appropriate OSM tag in the OSM wiki
  • Search for "amenity=library" or whatever
  • Export your data as GeoJson or KML
  • Upload into Carto or another interface

Notes:

  • only nodes, only ways, only relations (-> limit so it doesn't map data you're not interested in)
  • search for something that is too large it freezes (-> define your bounding box)

Turbo Interface

BUILDINGS

  • Carto Data Library

Data Library

NYC Pluto Downloader

There are lots of geospatial data formats that may or may not be easy to parse. Here's a short list of ones that are commonly imported into Carto.

Case Study 1: Libraries!

So let's take a look at this library data. You want to lead people to all of the libraries in NYC. You can use the kml file here in the data folder.

You can get the libraries from OSM or NYC Open Data but let's go with the KML from OSM that we extracted with OverPass.

.kml which is a notation for XML that first made popular by Google Earth before becoming standardized. Google Mapping Products happily use kml files. Check to see how the locations are described:

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2"><Document>
<name>overpass-turbo.eu export</name>
<description>Filtered OSM data converted to KML by overpass turbo.
Copyright: The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Timestamp: 2015-08-19T17:02:03Z</description>
<Placemark><name>SC Williams Library</name><Polygon>
	<outerBoundaryIs>
		<LinearRing><coordinates>-74.0254348,40.7450718 -74.0255769,40.7445846 -74.0251885,40.7445156 -74.0251471,40.7446466 -74.0251306,40.7446862 -74.0250324,40.7450032 -74.0254348,40.7450718</coordinates>
		</LinearRing>
	</outerBoundaryIs></Polygon><ExtendedData><Data name="@id"><value>way/26974141</value></Data>
	<Data name="amenity"><value>library</value></Data>
	...

Notice that the point coordinates are in the format longitude, latitude. Some geoformats are long, lat and others are long, lat. This makes everyone sad.

When coordinates cannot easily be parsed they go to "null island," a fictional place at coordinates (0,0).

Null-island

In a bit, we'll show how you can convert, process and use a file like this to plot data on your map!

Storing

Carto is a PostGres database in the cloud, which means it handles a lot of your backend data needs and allows you to query for data and pull those data and basemap tiles into your front-end code.

You can also store your data in Github, or in another service that makes it web accessible. Read more about that here and here.

Mapping Tools

Logos

There are loads of ways to approach a map here are a few approaches to mapping the same dataset.

Map Making Exercise

We can make a map with the libraries data, or other data in the data folder for this workshop.

Using Google Maps

  • https://www.google.com/maps/d/
  • create account if you don't already have a Gmail account
  • click on Import Map in top left hand menu (or My Maps -> Create map in some Google Maps UIs)
  • upload libraries.kml
  • explore changing the map features if you would like

Using Mapbox

  • https://mapbox.com/
  • create account if you don't already have a Mapbox account
  • click on the Data tab at the top right hand corner of the screen; * click on import
  • upload libraries.kml
  • select map features if you would like then click on Import Features explore changing the map features if you would like

Using Carto

  • https://carto.com/
  • create account if you don't already have a CartoDB account, use this URL to get boosted features
  • click on Create Map; select Map View at the top of the screen
  • click on the '+' or Add Layer option at the top of the right side menu
  • upload libraries.kml
  • explore changing the map features if you would like

Using Leaflet

You can also make a map from scratch using Leaflet.js (http://leafletjs.com/) to attach a set of points to a map made of tiles provided by OpenStreetMap.

You will first need to convert your kml file into GeoJSON (although I have both in the data folder) for this workshop.

GeoJSON is a file format that is easily digestable by JavaScript. If you have a data format (shp, kml) that is not geojson you can convert it to the right format for your code with GeoJSON.io/

{
  "type": "FeatureCollection",
  "generator": "overpass-turbo",
  "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.",
  "timestamp": "2015-08-19T17:02:03Z",
  "features": [
    {
      "type": "Feature",
      "id": "way/26974141",
      "properties": {
        "@id": "way/26974141",
        "amenity": "library",
        "designation": "Library",
        "name": "SC Williams Library"
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              -74.0254348,
              40.7450718
            ],

Examples

urban-interface

urban-reviewer

Tour of the interface

We'll use Carto for the subsequent exercises because of it's flexibility, support for SQL and JS, and backend handling for data management and geocoding.

int-data int-upload int-vis int-share

APIs / JS Libs

You can read more about the Carto APIs and JS Library here

  • CartoJS - JS library for interacting with CartoDB
  • Maps API - generate public/private maps with data hosted on your CDB account
  • SQL API - run sql in your code to dynamically filter/request/query your mapped data stored in CartoDB via http calls
  • Import API - push data to your CartoDB Account

Data Import + Sync Tables

Geospatial data is info that ids a geolocation and its characteristic features/frontiers, typically represented by points, lines, polygons, and/or complex geographic features. You can find a recommended list of geospatial data here

Issues:

Geocoding + SQL/PostGIS The most basic SQL statement is:

SELECT * FROM table_name
  1. SELECT is what you're requesting (required)
  2. FROM is where the data is located (required)
  3. WHERE is the filter on the data you're requesting (optional)
  4. GROUP BY and ORDER BY are optional additions, you can read more about aggregate/other functions below.

There are two special columns in Carto:

  1. the_geom
  2. the_geom_webmercator

The first of these is in the units of standard latitude/longitude, while the second is a projection based on the original Mercator projection but optimized for the web.

If you want to run SQL commands and see your map update, make sure to SELECT the the_geom_webmercator because this is the column that's used for mapping--the other is more of a convenience column since most datasets use lat/long.

Sync Tables

Sync Tables

The Editor is especially setup to process realtime data updates via sync tables.

You can import data that lives online via a URL, and set it to pull and update your map at regular intervals.

Check the file types supported in sync tables; keep in mind that it also works with dropbox + google drive.

Notes:

  • to auto-geocode a sync table, verify that it contains the following:
    • country column, a latitude column and a separate longitude column
    • a column of IP addresses
  • to edit sync tables you need to be disconnected from the datasource, so during syncing, you can use SQL to manipulate the dataset on the fly
  • you can create sync tables in both the Editor and via the Import API

Customizing UI + CartoCSS

Basemaps

You have myriad customization options in the in-browser editor:

  • sql - run sql and postgis functions across your data
  • builder - adjust the type, colors and fills in your map
  • infowindow - create hovers, tooltips with information from your datatables
  • css - customize the css and style of your map outside the builder
  • legends - create keys for your map
  • filters - filter the data without sql

Likewise, many types of visualizations:

  • Simple -- most basic visualization
  • Cluster -- counts number of points within a certain binned region
  • Choropleth -- makes a histogram of your data and gives bins different colors depending on the color ramp chosen
  • Category -- color data based on unique category (works best for a handful of unique types)
  • Bubble -- size markers based on column values
  • Intensity -- colors by density
  • Density -- data aggregated by number of points within a hexagon
  • Torque -- temporal visualization of data
  • Heat -- more fluid map of concentration; emphasis on far over near-view

Creating Custom Maps

Pluto Builder

Mapping in the Editor

The Editor/Builder GUI allows you to select your visualization "type" and customize color palettes, design details, and otherwise set the tone for your map. It also gives you ways to run sql and set your cartocss in the browser before you input those things as templates in your code.

Styles
  • Marker Fill: change the size, color, and opacity of all markers
  • Marker Stroke: change the width, color, and opacity of every marker's border
  • Composite Operation: change the color of markers when they overlap
  • Label Text: Text appearing by a marker (can be from columns)
Infowindows/hovers

Old Interface

Infowindow options

New Interface

Pluto Builder

  • Select which column data appear in infowindow by toggling column on
  • Customize further by selecting HTML-view
Basemaps

Basemap options

Check out the base Carto Basemaps from Stamen.

Mapping in Code

Python

Use the CartoDB Python module to create and update tables; check out Andy's IPythonNB demos.

Node

Use the CartoDB Uploader package available in NPM for working in Node.js

R

Use Kyle Walker's R2CartoDB project to create and update tables in R.

Building a Narrative

The data folder includes some point (libraries), line (bikelanes), and polygons (building footprints) for NYC. You can make a map with any one of these, or a dataset of your choosing.

Let's try the Building Footprint Data, making something like this.

  • Navigate to your Data Library in CartoDB
  • Search for 'PLUTO' and select the NYC data; or Use the NYC PLUTO data Downloader linked above
  • "Connect Dataset"
  • "Create Map"
  • Adjust aspects in the GUI

Pluto Map

EXPLORE THE DATA

Pluto Data

CHANGE YOUR BASEMAP

Pluto Basemap

MAKE A CHOROPLETH IN THE BUILDER

Pluto Builder

STYLE IT WITH CARTOCSS

Pluto CSS

CREATE TOOLTIPS AND LEGENDS

Pluto Legend

Case Study 2: NYC Bikelane Category Map

Let's try the Bikelane Data, making something like this.

  • Go to the Class Repo
  • Clone the repository to access the biklane data
  • Upload it to CartoDB

Try to do the above modifications with these data!

For code samples, you can check our class repo, where you'll find a template for making a map as a site online, in this case using blackout country-level data to illustrate which countries receive more "blackouts":

  • index.html: setting up your map div and canvas
  • main.js: pulling in your basemap, cartodb feature layers and other details

If you run this locally, you'll get a map that looks like this:

Blackout Map

Here's another example of BikeData from Madrid modified with SQL and hosted on Github. And here's a demo of how to create selectors that reload data in CartoDB: http://bl.ocks.org/ohasselblad/e5e438b4e83a4c1d12e9

Bikestorming is a mobile app for urban cycling built with open data on CartoDB with MapBox Baselayers. They extract from public and social data sources to build both the iOS app and their open data portal for bike info.

Bkx Data Store

Bkx Mobile screens

BiciDB

Tell Time/Stories: Odyssey + Torque

Outside of the CartoJS library, we have others to help you build dynamic narrative with your data.

Maps that tell Time - Torque

Realtime Traffic Map

Source: Traffic Change Data for SF Demo: Municipal Traffic in SF App Blog: CartoDB + Firebase

  1. Demonstrations in Brazil
  2. Animal migration patterns
  3. Diwali Celebrated
  4. Ramadan Tweets w/OdysseyJS
  5. Alcatraz Escapees
  6. Lynching and the Press

Twitter Import

Maps that tell Stories - Odyssey JS

  1. Al Jazeera: Israeli-Palestinian Conflict by Tweets
  2. The Sounds of 11M
  3. Berlin Wall Historic Tour

Partner Graphics: Graphs + Charts

You can use CartoDB's SQL API to query your data and pull it into any charting library of your choosing.

You can easily wire up a chart of pollution data for example, set it as a sync table to update continuously, check the code here.

chart

Learn more about it here!

Here are some examples:

Type Title Link/Demo BlogPost
Chart.js Bar Graph Traffic Data Aurelia's Block
Highcharts Sensor Data Github / Demo MOW Post
Highcharts Weather Data Aurelia's Block Tutorial
Chart.js Line Graph Tornado Data Andrew's Block
Plot.ly Earthquake Data Plotly Tutorial CartoDB Blog

Resources

XKCD

GeoSpatial Data

  1. Local Data Resources
  2. National Data Resources

Mapping + CartoDB

  1. Map Academy
  2. CartoDB Tutorials
  3. CartoDB Editor Documentation
  4. CartoDB APIs
  5. Community help on StackExchange
  6. CartoDB Map Gallery
  7. CartoDB Bootstrap Template by Chris Wong
  8. Listing of Map Data Formats

Visualization

  1. Charting Tools Repository
  2. Workshops @ CartoDB
  3. Recommended tools for Visualizations
  4. Perception Concerns
  5. Gestalt Theory
  6. Color Brewer or Geocolor
<!DOCTYPE html>
<html>
<head>
<title>Blackout Map | Carto</title>
<meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
<meta http-equiv="content-type" content="text/html; charset=UTF-8"/>
<link rel="shortcut icon" href="http://cartodb.com/assets/favicon.ico" />
<style>
html, body, #map {
height: 100%;
padding: 0;
margin: 0;
}
</style>
<!-- include the carto.css stylesheet -->
<link rel="stylesheet" href="http://libs.cartocdn.com/cartodb.js/v3/3.15/themes/css/cartodb.css" />
</head>
<body>
<div id="map"></div>
<!-- include carto.js library -->
<script src="http://libs.cartocdn.com/cartodb.js/v3/3.15/cartodb.js"></script>
<!-- place your code betwee the script tags below -->
<script src="main.js"></script>
</body>
</html>
function main() {
var map = new L.Map('map', {
zoomControl: false,
center: [43, 0],
zoom: 3
});
L.tileLayer('http://tile.stamen.com/toner/{z}/{x}/{y}.png', {
attribution: 'Stamen'
}).addTo(map);
cartodb.createLayer(map, 'https://team.cartodb.com/u/aureliamoser/api/v2/viz/67f88c2a-b77d-11e4-aae0-0e853d047bba/viz.json')
.addTo(map)
.on('done', function(layer) {
layer.setInteraction(true);
layer.on('featureOver', function(e, latlng, pos, data) {
cartodb.log.log(e, latlng, pos, data);
});
layer.on('error', function(err) {
cartodb.log.log('error: ' + err);
});
}).on('error', function() {
cartodb.log.log("some error occurred");
});
}
// you could use $(window).load(main);
window.onload = main;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment