datagov-harvester

This repository holds the source code the Data.gov Harvester 2.0, which includes three applications applications:

datagov-harvest-runner: This is a python application, chiefly composed of files in the harvester directory.
datagov-harvest-admin: This is a Flask app which manages the configuration of harvest sources, organizations, and the creation of harvest jobs.
datagov-harvest-proxy: This is an nginx app which owns the public route and proxies traffic to the internal Flask app route.
datagov-harvest-validator: A minimal API (validator_api/) serving /api/v1/validate, which the admin app’s /validate/ page and external callers use. It shares the dcatus_validation/ package with the harvest runner and needs no database or secrets.

There is further documentation in the developer quickstart.

Documentation