Key Points

Introduction to Data Management

  • Data management is the day-to-day organization, storage, protection and maintenance of data during a project
  • The research data lifecycle runs from planning through collection, processing, analysis, preservation, sharing and reuse
  • Geospatial data needs extra care: multi-file formats, coordinate systems, large sizes, geometry validity and sensitive locations
  • Many funders, including NSF and NIH, require a data management and sharing plan, and a plan is worth writing even when not required

Geospatial Data Models

  • Vector data represents discrete features as points, lines or polygons, each with a row of attributes
  • Raster data represents surfaces as a grid of cells with a resolution, bands, a data type and a NoData value
  • Topology describes how features connect and share boundaries; network and boundary analysis depend on it
  • The right data model and geometry depend on the question and the scale of the analysis
  • Finer raster resolution grows data size with the square of the change in cell size

Geospatial File Formats

  • Formats differ in openness, number of files, size limits and what attribute information they preserve
  • Shapefiles are universal but limited: several files, 10-character field names, 2 GB, weak null support
  • GeoPackage is a good default for your own vector work; GeoTIFF for rasters; CSV for tables
  • Prefer open, well-documented formats for sharing and preservation
  • Check the CRS, field names, data types, nulls and feature counts after every conversion

Coordinate Reference Systems

  • Coordinates are meaningless without their coordinate reference system: datum, coordinate system, units and projection
  • Store and exchange data in geographic systems; measure and analyze in projected ones
  • Use EPSG codes to name CRSs exactly; 4326, 4269, 3857, 5070 and 26916 are common in U.S. work
  • Defining a CRS labels existing coordinates; reprojecting calculates new ones. Confusing the two moves data to the wrong place
  • Before analysis, put every layer, and the project, in the same projected CRS

Designing Your Data

  • Design data from the research question: the things you record become layers, and the questions become attributes
  • A data dictionary defines each field’s name, meaning, type, length, allowed values and null rule
  • Store codes such as ZIP codes and GEOIDs as text; use domains with an Unknown option for categories
  • Every feature needs a permanent unique ID; joins match primary keys to foreign keys
  • Keep repeated information in separate related tables rather than copying it

Organizing, Versioning and Backing Up

  • Use a consistent folder structure that separates raw, processed and output data, and keep raw data read-only
  • Name files descriptively and consistently, without spaces, with ISO 8601 dates and zero-padded numbers
  • Keep data inside the project folder and use relative paths so QGIS projects stay portable
  • Track versions with numbered files and a change log, or with Git for code and small text data
  • Follow the 3-2-1 backup rule, remember that syncing is not backup, and test your backups

Data Quality

  • Data quality means fitness for a particular use, and every dataset has limitations
  • The elements of quality are completeness, logical consistency, positional accuracy, thematic accuracy and temporal quality, with lineage recorded alongside
  • Invalid geometries, gaps, overlaps, dangles and duplicates are common vector errors; QGIS has tools to find each
  • Prevent errors with good design (QA) and find them with systematic checks (QC)
  • Record what you checked and report limitations honestly