Summary and Setup

Data rarely arrives ready to analyze. This module teaches the everyday work between collecting data and analyzing it: cleaning raw tables and layers, geoprocessing (buffers, overlays, distances and areal weighting), raster processing (map algebra, reclassification and zonal statistics) and automating the steps you repeat. It is written for social scientists and other beginners who have finished the Foundations module.

Each topic is taught twice, once in QGIS (point and click) and once in Python (Google Colab), with the same data and the same results. Follow either one, or both. All examples use Tippecanoe County, Indiana (Lafayette and West Lafayette) and Indiana as a whole.

Episode Question it answers Tools
Introduction to Data Processing What happens between download and analysis, and how do I keep it repeatable? Concepts
Cleaning and Preparing Data (Python / QGIS) How do I fix identifiers, types, special codes, duplicates and geometry problems? pandas, geopandas / Refactor Fields, Check Validity
Geoprocessing (Python / QGIS) What lies within a distance, inside a boundary, or nearest? geopandas / QGIS Vector menu
Raster Processing (Python / QGIS) How do rasters combine, and what do they say about each tract? rasterio, rasterstats / Raster Calculator, Zonal statistics
Automating Repeated Steps (Python / QGIS) How do I run the same workflow for many inputs without mistakes? Functions and loops / Model Designer, batch processing

Start with the Introduction, then take the episodes in order: each one uses results or ideas from the one before. The advanced methods (network analysis, spatial autocorrelation, clustering and NDVI) follow in the Data Analysis module.

Prerequisite

Prerequisites

  • For the Python episodes: a Google account, and basic Python, pandas and GeoPandas (Foundations notebooks 2a to 4c)
  • For the QGIS episodes: QGIS installed, and the Foundations QGIS episodes (Getting Started with QGIS; Tables, Joins and Selections)
  • No Census API key is needed: the Census tables are provided as downloaded

New to Python, Colab or QGIS? Start with the Foundations module, then read the Setup page.

This module is part of Enabling the Geospatial Turn in the Social Sciences, a set of training modules developed at Purdue University with support from the National Science Foundation.

The Python episodes run in Google Colab, a free online notebook service, so you do not need to install Python. The QGIS episodes need QGIS on your computer; follow the Foundations setup to install it. No Census API key is needed in this module.

What you need

  • A Google account (Python episodes)
  • QGIS 3.34 or later; the episodes were checked in QGIS 4.0 (QGIS episodes)
  • A web browser (Chrome, Firefox, Edge or Safari)

Opening a notebook

  1. Click a notebook link in the Notebooks table at the end of any Python episode. It opens in Colab.
  2. Click File > Save a copy in Drive. Your own copy keeps your changes; the original stays read only.
  3. Run cells from top to bottom. Click a cell and press Shift + Enter, or click the play button to its left.
  4. The first time you run a cell, Colab may warn that the notebook was not written by Google. Click Run anyway.
Callout

Colab tips

  • Run cells in order. Most errors come from skipping a cell, for example the one that reads the data.
  • Start over cleanly with Runtime > Restart session and run all.
  • Data lives in memory and files you write (such as the county maps in notebook 4b) live in the session; download them from the Files panel on the left before the session ends.
  • Cells that start with !pip install add libraries Colab does not include by default (rasterstats in the Raster Processing notebooks). They take a few seconds and only need to run once per session.

Loading data in QGIS

The QGIS episodes load every layer from a web address: Layer > Add Layer > Add Vector Layer (or Add Raster Layer), Source Type: Protocol: HTTP(S), cloud, etc., Type: HTTP/HTTPS/FTP (in QGIS 4 the Type list starts with cloud services, so check it), and paste the address. Layers loaded this way are read only: tools write their results to new temporary layers, which disappear when QGIS closes unless you save them (right-click > Make Permanent).

Data

You never need to download data by hand. Every notebook and QGIS episode reads its data from the web.

Episode Data Source
Cleaning Raw ACS 2024 table for Tippecanoe County tracts; raw IPEDS directory of Indiana colleges; raw OpenStreetMap parks U.S. Census Bureau API; NCES; OpenStreetMap
Geoprocessing Lafayette food stores and city limits; Tippecanoe County tracts with 2024 ACS estimates OpenStreetMap; U.S. Census Bureau (pre-fetched)
Raster Processing Tippecanoe County elevation (USGS 3DEP, 30 m) and land cover (NLCD 2021), with tracts and 2024 ACS estimates USGS; Multi-Resolution Land Characteristics Consortium; U.S. Census Bureau
Automating The Geoprocessing layers and parks; raw ACS 2024 table and tract boundaries for all of Indiana As above; U.S. Census Bureau

Working on your own computer (optional)

To run the notebooks locally, install Anaconda and create an environment:

BASH 

conda create -n spatialturn -c conda-forge python=3.11 geopandas rasterio rasterstats mapclassify matplotlib jupyterlab
conda activate spatialturn
jupyter lab

Then download a notebook from GitHub (File > Download > Download .ipynb in Colab) and open it in JupyterLab. The data is still read from the web, so you need an internet connection.