Dataset Lifecycle
This page gives you a tour through the Peekbank stack by describing the stages a newly added dataset goes through. You can click on an element in the diagram below to jump to the related section.
Raw data in peekbank_files
Section titled “Raw data in peekbank_files”A lab sends their LWL data (or a link to a data repository) to someone on the Peekbank team, who uploads it to peekbank_files on Redivis. Each dataset has its own folder there, named after its identifier (for example weaver_zettersten_2024), holding the raw files.
Import
Section titled “Import”The import process turns the raw data into preprocessed data in the Peekbank format. For each dataset, a collaborator writes a separate, scaffolded import.R script in peekbank-data-import, which downloads the raw data from peekbank_files, reshapes it (outputting one .csv per table of the schema) and validates the result. A dataset specific README.md documents any import decisions.
The import scripts use some internal helper functions from PeekbankR.
Once the data passes validation, we perform an internal review step where another collaborator looks through the import script and the paper the data came from in tandem. Until then, a marker file in the dataset’s folder keeps it out of the pipeline and the releases. If no further issues are found, the reviewer removes the marker from the folder, which makes the dataset part of the next release.
With a flag in the import script, the processed data and README.md can also be uploaded to peekbank_files, into a subfolder of _processed_data folder. Releases do not use these uploads, but the cached workflow for testing does, to allow for faster end-to-end testing of import changes.
Ingestion
Section titled “Ingestion”The ingestion builds a draft of the next release of Peekbank on Redivis from the raw data and the import code. It is triggered via a command in peekbank-data-import (./pbi ingest <release-name>), which runs in Docker and pins both the raw data and the state of the import repository to the release. It downloads the raw data, reruns every import script and then combines the processed .csv files of all datasets into one set of tables, offsetting the row IDs to their position in the joined dataset. The process stops if any dataset fails to import or validate. Peekbank uses release names in the format <YEAR>.<Incrementing Number>.
Every dataset’s README.md and raw data are bundled with the release in the dataset_files table. These files enable us to reproduce the ingestion if needed and allow PeekbankR to fetch versioned stimulus images and READMEs. Raw data that may not be shared publicly is not uploaded here, but the release notes what version of the private data was used.
Other than the schemas main tables, the ingest also creates an aoi_timepoints_rle table, a run-length-encoded version of aoi_timepoints, which is used by PeekbankR to cut transmission size. Similarly, the ingestion creates a release_info table with metadata, which has a single row with the release name of the dataset, the commit of peekbank-data-import it was built from, and the version of the private raw data it used.
Testing with the cached workflow
Rerunning every import takes a while, so for quickly testing import changes there is a quicker way, depicted in the “Cached workflow for quick testing” tab above. ./pbi ingest <release-name> --cache builds a draft from the cached .csv files uploaded to _processed_data. Such a draft records no import repository commit and is labelled as a test build on Redivis. Individuals with edit access on the Peekbank dataset on Redivis can read it with peekbank("next") in PeekbankR for testing purposes.
Release
Section titled “Release”Releasing the draft with ./pbi release <release-name> (or the Redivis interface) publishes it as a new release of the Peekbank dataset. Redivis numbers its versions on its own (e.g., v2.1), so our release_info table is used to keep track of Peekbank release names (e.g., 2026.2). Previous versions stay available to keep old analyses reproducible. The list of supported releases can be seen on the Releases page.
Access
Section titled “Access”End users have three ways to access the data of a release:
- PeekbankR, the R package most people use. It allows you to specify a release and read the tables from Redivis into your R environment.
- Redivis itself, where you can browse and query the tables in the browser or read them with one of its clients.
- This website, as it features interactive visualisations that let you explore some analyses on the data directly in the browser.
Furthermore, certain elements of this website (e.g., the dataset list on the Citation page) are populated with data from the releases. These, as well as the analyses, don’t query Redivis live. When the website is deployed, a Python script pulls the releases and turns them into compressed per-dataset JSON files that the browser loads.
Corrections
Section titled “Corrections”When someone finds a problem in the data, it goes into the issue tracker of peekbank-data-import. That dataset’s import script is then fixed, and the corrected data reaches everyone with the next release.