User Tools

Site Tools


sift:batch_processing

Batch Processing Overview

Batch processing enables rapid processing of large datasets and ensures consistent results across projects, labs, and time.

Sift provides two methods of batch processing:

  1. the GUI, and manually adding the library and pipelines
  2. the command line
    • As a one off
    • Time-based automation
    • Directory scanning for new data automation

V3D Engine

Sift runs on Visual3D's engine, and can batch run V3D pipelines across multiple CMZs concurrently.

When to use: Use the V3D Engine for processing smaller datasets, single-instance runs, or when first getting started with Sift.

Sift Command Line

The Sift command line enables large scale batch processing right from the command line.

When to use: Use the Sift Command Line to process large quantities of data, to run continuous automated updates over time, or, for those with experience in development.

Time-Based Triggers

Set up automatic processes to run at specific times.

When to use: Use this method for continuous processing on a daily, weekly, or monthly basis.

Directory Watchers

When to use: Use this method for automated processing when new data enters a specific directory.

sift/batch_processing.txt · Last modified: by wikisysop