Key Points
Introduction to Data Processing
- Processing turns collected data into analysis-ready data: inspect, clean, transform, combine, derive
- Keep raw data unchanged and write results somewhere else
- Record every step: a notebook or script, the QGIS Processing History or a model, and a short log
- Check after every step, with counts, totals and a quick map
- Use QGIS for quick, visual work and Python (or a QGIS model) for work you will repeat
Cleaning and Preparing Data in Python
- Inspect before changing anything: shape, types, minimum and maximum reveal most problems
- Read identifiers as text, convert numbers stored as text, and turn special codes into missing values
- ACS estimates have margins of error; the coefficient of variation shows which ones are too uncertain to rank
- Decide what makes two records the same before removing duplicates
- Define a CRS to say what coordinates are; reproject to change them
- Check every step with a count, a join or a map, and write your decisions down
Cleaning and Preparing Data in QGIS
- Inspect before changing anything: sorting a column quickly shows text that should be numbers and special codes
- Refactor Fields renames fields, sets their types and fills them with expressions in one step, without changing the original
-
to_int(),if(... < 0, NULL, ...)andnullif(x, 0)handle text numbers, codes and division by zero - ACS estimates have margins of error; the coefficient of variation shows which are too uncertain to rank
- Create points from coordinates with the CRS they are in; decide what makes two records the same before deleting
- Check validity, remove unwanted features and dissolve split ones, and confirm each step with a count
Geoprocessing in Python
- Project every layer to a CRS in meters (EPSG:26916 for Indiana) before buffering, measuring or overlaying
- Buffer draws zones, dissolve merges them, clip and intersect cut shapes, difference keeps what lies outside, and a nearest join finds the closest feature and its distance
- The spatial predicate (intersects, within, centroid within) changes which features are selected; report it
- Keep the original area before cutting, so areal weighting can split counts by area share
- Sum counts, never rates or medians; areal weighting assumes an even spread and straight lines are not travel
Geoprocessing in QGIS
- Reproject every layer to a CRS in meters (EPSG:26916 for Indiana) before buffering, measuring or overlaying; on-the-fly display does not change the data
- Buffer draws zones, Dissolve merges them, Clip and Intersection cut shapes, Difference keeps what lies outside, and Join attributes by nearest finds the closest feature and its distance
- The predicate in Select by Location (intersect, are within, centroid within) changes the answer; report it
- Store the original area before cutting, then use
area($geometry) / "tract_m2" * countfor areal weighting - Sum counts, never rates or medians; processing results are temporary until you make them permanent
Raster Processing in Python
- Map algebra works cell by cell on rasters that share CRS, cell size and extent; resample categories with nearest neighbor
- Comparisons make Boolean rasters, lookup tables reclassify codes,
and
&combines conditions - Zonal statistics summarize cells inside polygons: means for measurements, class counts for categories
- Cell counts times cell area give areas, in a CRS measured in meters
- Dasymetric estimation uses land cover to place people; compare it with areal weighting
Raster Processing in QGIS
- Rasters combined cell by cell must share CRS, cell size and extent; resample categories with nearest neighbor
- The Raster Calculator makes comparisons and combines conditions
(
AND,OR); results are 1 and 0 - Reclassify by table groups codes; set the range boundaries to
min <= value <= maxfor whole-number codes - Zonal statistics summarize measurements per polygon; Zonal histogram counts categories per polygon
- Check results with the unique values report: cells times 900 m² gives areas
Automating Repeated Steps in Python
- Automate work you will repeat; record work you will not
- A function packages steps; parameters are the parts that change, defaults the usual values
- Test on a known answer, check inputs, and report counts so every run leaves evidence
- Loops (or
groupby) run the same workflow for many inputs and collect the results in one table - Combine units by adding counts first, then computing rates
- Catch errors in long loops, but record them and check the totals at the end
Automating Repeated Steps in QGIS
- Automate work you will repeat; record work you will not (Processing > History)
- A model packages several tools; its inputs are the parts that change
- Test a model on a known answer before using it on new questions
- Run as Batch Process repeats a tool or model with different settings, one row per run
- Iterate over this layer runs a tool once per feature
- Make common mistakes impossible inside the model, for example by reprojecting every input