This is an old revision of the document!

Batch Command Line Processing Tutorial

In this tutorial, we will guide you through running Sift on the command line. You'll learn how to load a library, execute queries, run PCA (Principal Component Analysis), and use it for outlier detection. After walking through each command step by step, we’ll create a batch script to automate these tasks. We'll also include some logic to further streamline the process. To get started, download the sample data folder. automation_tutorial.zip

Running on Command Line

First, we need to run Sift via the command line. On your keyboard, press Windows Key + R, type “cmd” into the run window, and press Enter to open a command prompt.
Next, navigate to Sift's installation directory.
To launch Sift, call the .exe file and pass the –NoGUI argument
This will open new console where we can interact with Sift
In this console, load a library by entering the command: –LoadLib directory “Path to downloaded library”
Next load and run a predefined query by entering: –LoadQuery file “Path to query definition file”
Select all signals by running –SelectSignals (Ensure all signal groups and work-spaces have been selected)
Run a PCA named “Outlier_PCA” –RunPCA name “Outlier_PCA”
Now, let's perform the first round of outlier detection. Run a Local Outlier Factor (LOF) test with the command: –RunLOF grouping “group” autoExclude This will find any group level outliers and exclude them automatically
To export the results of this LOF test, run: –ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThrehsold
Next, run another LOF test, this time on workspaces: –RunLOF grouping “workspace” autoExclude
Export the results of this LOF test as well: –ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThreshold
Finally, save the project to allow for future review in the GUI, if needed: –SaveProject file “Path to saved project”

Now our processing is complete, we have two exports detailing group and workspace level outliers and a saved project file that has excluded all of the found outliers. We can use this as a baseline for future processing.

Batch Processing

Now that we have an understanding of what we want as a baseline for processing, we can take the steps to automate it!

The Scenario

In order to fully understand how we should automate a process we need to understand all the details, so lets create a scenario.
Lets say we are working with a baseball team, and every day at practice we get some data that needs to be processed, that data will be stored in dated folders in a root directory, within each of these folders are three trials which contain relevant C3Ds

The Plan

Now that we know what we are working with we can think of how to build our batch script, first we will need to check each folder within the root directory, and if the folder does not already contain our exports (.txt files and .i3d) then we know this folder needs to be processed. Below is a batch script that will do just that, you will need to change the path in targetDir as well as the path pointing to the Sift.exe to match your own computer, then save the file as a .bat, to do so you can create a .txt file and then rename it to change the file extension to .bat

@echo off
setlocal

set "targetDir=C:\Users\shane\OneDrive\Documents\practice"

for /D %%d in ("%targetDir%\*") do (
    set "i3dFileFound="
    for %%f in ("%%d\*.i3d") do (
        if exist "%%f" (
            set "i3dFileFound=true"
        )
    )
    
    if not defined i3dFileFound (
        set "folderWithoutI3d=%%d"
        "C:\Program Files (x86)\Sift\Sift.exe" --NoGui --LoadLib directory "%%d" --LoadQuery file "%%d\query_definition.q3d" --selectSignals --RunPCA name "Outlier_PCA" --RunLOF grouping "group" autoExclude --ExportPCA file "%%d\Group Outliers.txt" exportFormat "transposed" lof lofThreshold --RunLOF grouping "workspace" autoExclude --ExportPCA file "%%d\Workspace Outliers.txt" exportFormat "transposed" lof lofThreshold --SaveProject file "%%d\Outliers.i3d" --Exit
    )
)

endlocal

The Execution

Now that we have a script ready to go, executing our processing will be as easy as double clicking the .bat file, but we can take the automation even further! Leveraging windows Task Scheduler we can set the batch script to execute at a certain time each day

Open the start menu and type Task Scheduler into the search bar
Open Task Scheduler
In the Task Scheduler select create basic task in the top right of the screen
In the popup dialog enter a meaningful name in this case we went with “Baseline Processing” press next
In the next step ensure daily is selected and press next
Next select what time of day you would like the task to run, we picked 12:00AM
In the next step ensure start a program is selected and press next
Now we will browse and select the batch script we created
On the final screen review and press finish
Finally check the existing task to ensure it has been created successfully