User Tools

Site Tools


sift:batch_processing

This is an old revision of the document!


Batch Processing Overview

Batch processing is great for quickly processing large datasets, and ensuring consistent results across projects, labs, and time.

There are two methods to batch processing in Sift. Using:

  1. the GUI, and manually adding the library and pipelines
  2. the command line

V3D Engine

Sift runs on Visual3D's engine, and can batch run V3D pipelines across multiple CMZ's concurrently.

When to use: Either of these methods may be appropriate for the same applications. Use this tool when needing to batch update in smaller doses, in a single instance, or when just getting started in Sift.

Sift Command Line

The Sift command line enables large scale batch processing right from the command line.

When to use: Use this option to batch update large quantities of data, or batch updating continuously over time, or for those with experience in development.

Time-Based Triggers

Set up automatic processes to run at specific times.

When to use: Use this method for continuous processing on a daily, weekly, or monthly bases.

Directory Watchers

Set up automatic processes to run when new data enters a specific directory.

When to use: Use this method for automated processing when new data is added.

sift/batch_processing.1774367187.txt.gz · Last modified: by wikisysop