Summary and Setup

One of the main challenges in using digital data in the social sciences is knowing how to collect it: where to find existing data, how to judge whether it can be trusted, and how to gather original data when nothing suitable exists. This module covers both sides, from data repositories, the U.S. Census Bureau and geocoding to satellite imagery and field data collection with a phone.

Episode What you will learn Tools
Introduction to Data Collection When to use existing data or collect your own; vector and raster data; judging data quality; FAIR data Concepts
Finding Data in Repositories Where to find free vector data, how to read metadata, and how to load data from the web Web browser, QGIS
Data from the U.S. Census Bureau Census geography, the Decennial Census and the ACS, margins of error, and the Census API Web browser, Python notebooks
Geocoding and Adding Spatial Context Turning addresses into points, checking match rates, and downloading features from OpenStreetMap Python notebooks
Acquiring Imagery Scanned historic maps, declassified CORONA imagery and Sentinel-2 satellite data Web browser, QGIS
Collecting Data in the Field Methods and apps, designing a schema, GPS accuracy, ethics and quality checks Concepts

Several episodes use one example: access to grocery stores in Lafayette and West Lafayette, Indiana. The grocery access case study in the Case Studies module completes it, with a QField field mapping demonstration and a network analysis in QGIS.

Prerequisite

Prerequisites

  • A Google account, to open the notebooks in Google Colab
  • A free Census API key (see Setup below)
  • QGIS installed on your computer (see Setup below)
  • For the notebooks: basic Python, such as running cells and working with a pandas DataFrame
  • No prior GIS experience is needed

This module is part of Enabling the Geospatial Turn in the Social Sciences, a set of training modules developed at Purdue University with support from the National Science Foundation.

This module uses free tools only:

  • A web browser for the repositories, the Census API and the imagery websites
  • Google Colab for the Python notebooks in the Census and Geocoding episodes. It runs in your browser, so there is nothing to install.
  • QGIS for the repositories and imagery episodes. It is a desktop program you install once.

What you need

  • A Google account (for Colab)
  • A computer where you can install software (for QGIS)
  • A free Census API key for Census notebooks 1b and 1c and the API examples (see below)
  • Optional: free accounts on USGS EarthExplorer and the Copernicus Browser to download satellite imagery

Part 1: Get a Census API key

Every learner needs their own key. It is free and takes a few minutes.

  1. Go to https://api.census.gov/data/key_signup.html.
  2. Enter your organization (for example, Purdue University) and your email address, and agree to the terms.
  3. Check your email. The key arrives within a few minutes.
  4. Click the activation link in the email. The key does not work until you do.
  5. Keep the key somewhere safe. Treat it like a password: do not post it online or leave it in a notebook you share.

Part 2: Google Colab

  1. Click a notebook link in the table in the Census or Geocoding episode. It opens in Colab.
  2. Click File > Save a copy in Drive so your changes are kept.
  3. Paste your Census API key where the notebook asks for it.
  4. Run cells from top to bottom: click a cell and press Shift + Enter.
  5. If Colab warns that the notebook was not written by Google, click Run anyway.

The notebooks read all their data straight from the web (the Census Bureau’s website, the Census Geocoder, OpenStreetMap and this lesson’s repository), so you never need to download anything. The geocoding notebooks do not need a Census API key.

Part 3: Install QGIS

Install QGIS (the free Long Term Release) by following the central Setup page of the Foundations module. If you are new to QGIS, its Getting Started with QGIS episode introduces the interface.

Loading data from a web address in QGIS

The repositories episode loads data straight from the web:

  1. Layer > Add Layer > Add Vector Layer
  2. Source Type: Protocol: HTTP(S), cloud, etc.
  3. Paste the address given in the episode into URI, then click Add

If you want to try field data collection with QField, see the setup page of the Case Studies module, where the QField demonstration lives.

Troubleshooting

Problem Fix
The Census API returns a “Missing Key” or “Invalid Key” page Check that you clicked the activation link in the email, and that the key was pasted completely
A notebook cell says NameError A cell above it was skipped. Use Runtime > Run before
QGIS will not open on macOS Right-click the app, choose Open, then confirm
A layer loaded from a web address shows nothing Right-click the layer and choose Zoom to Layer; check that the address was pasted completely
The Census Geocoder returns no match Check the address has a street number; campus and landmark addresses are often missing, so try Nominatim (notebook 2c)
An OpenStreetMap or Nominatim request times out The free server is busy or rate limited; wait a minute and rerun the cell