User Tools

Site Tools


sift:batch_processing

This is an old revision of the document!


Batch Processing Overview

Batch processing enables rapid processing of large datasets and ensures consistent results across projects, labs, and time.

Sift provides two methods to batch processing:

  1. the GUI, and manually adding the library and pipelines
  2. the command line
    • As a one off
    • Time-based automation
    • Directory scanning for new data automation

V3D Engine

Sift runs on Visual3D's engine, and can batch run V3D pipelines across multiple CMZs concurrently.

When to use: Use the V3D Engine for processing smaller datasets, single-instance runs, or when first getting started with Sift.

Sift Command Line

The Sift command line enables large scale batch processing right from the command line.

When to use: Use the Sift Command Line to process large quantities of data, to run continuous automated updates over time, or, for those with experience in development.

Time-Based Triggers

Set up automatic processes to run at specific times.

When to use: Use this method for continuous processing on a daily, weekly, or monthly basis.

Directory Watchers

Set up automatic processes to run when new data enters a specific directory.

When to use: Use this method for automated processing when new data is added.

sift/batch_processing.1774367573.txt.gz · Last modified: by wikisysop