Datasaur
Search…
How It Works
For a quickstart guide, refer to the this page.

Process / Flow

Project Creation

  • The script will start iterating from the root folder that is specified on the configuration, specifically attribute create.files.
  • Each folder will be treated as a project and the project name is derived from the folder name.
  • All files inside each folder will be the documents for that particular project.
  • All successful project creation is saved on the state. So, the next time you run the same command, there will be no project duplication. It will only process the new project(s) or the failed ones.

Project Export

  • Just like creation, it still relies on the configuration, specifically the export attribute.
  • It will export all the projects that fulfilled the filter and not yet recorded by the state. It will only export each project once.

Configuration File

Robosaur works by providing a JSON payload to determine the correct behavior. The current supported commands (create and export) always need this configuration file, which consist of the following attributes.
  • datasaur: specifies the host, clientId, and clientSecret which are already covered on the quickstart section.
  • projectState: specifies where to store the state files, see the object storage section below.
  • export: specifies the behavior for exporting projects.
  • create: specifies the project settings to be create.
An in-depth breakdown is available as a TypeScript file in src/config/interfaces.ts .

Object Storage

Each source attribute could be configured to local, AWS S3, or Google Cloud Storage. The detailed guide could be found here.