This is an old revision of the document!

Batch Command Line Processing Tutorial

In this tutorial we will learn how to run sift on the command line, loading a library, running queries then running PCA and using that for outlier detection. once we run through the commands step by step we will use them to create a batch script that can be used to execute them automatically, we will include so logic to automate the process even further. To get started download the sample data folder automation_tutorial.zip

Running on Command Line

First we need to run Sift on the command line, on your keyboard press the Windows Key + R in the run window type in cmd and press enter, this will open a command line terminal.
Then change the directory to Sift's install location
Next to launch sift we need to call the .exe and pass the argument –NoGUI
This will open another console that we can use to interact with Sift
In the new console we will load a library, enter the command –LoadLib directory “Path to downloaded library”
Next load and run a predefined query by entering the command –LoadQuery file “Path to query definition file”
Select all signals by running –SelectSignals (Ensure all signal groups and work-spaces have been selected)
Run a PCA with the name “Outlier_PCA” –RunPCA name “Outlier_PCA”
Now we will run our first round of outlier detection run an Local Outlier Factor test with the command –RunLOF grouping “group” autoExclude This will find any group level outliers and exclude them automatically
Now lets export the results of this LOF test run the command: –ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThrehsold
Next we will run another LOF test, this time on workspaces: –RunLOF grouping “workspace” autoExclude
Now export the results of this LOF test as well: –ExportPCA file “Path to exported txt file” exportFormat “transposed” LOF LOFThreshold
Finally, lets save the project so we can view it in the GUI if needed: –SaveProject file “Path to saved project”

Now our processing is complete, we have two exports detailing group and workspace level outliers and a saved project file that has excluded all of the found outliers. We can use this as a baseline for future processing.

Batch Processing

Now that we have an understanding of what we want as a baseline for processing, we can take the steps to automate it!

The Scenario

In order to fully understand how we should automate a process we need to understand all the details, so lets create a scenario.
Lets say we are working with a baseball team, and every day at practice we get some data that needs to be processed, that data will be stored in dated folders in a root directory, within each of these folders are three trials which contain relevant C3Ds

The Plan

Now that we know what we are working with we can think of how to build our batch script, first we will need to check each folder within the root directory, and if the folder does not already contain our exports (.txt files and .i3d) then we know this folder needs to be processed. Below is a batch script that will do just that, you will need to change the path in targetDir as well as the path pointing to the Sift.exe to match your own computer, then save the file as a .bat, to do so you can create a .txt file and then rename it to change the file extension to .bat

@echo off
setlocal

set "targetDir=C:\Users\shane\OneDrive\Documents\practice"

for /D %%d in ("%targetDir%\*") do (
    set "i3dFileFound="
    for %%f in ("%%d\*.i3d") do (
        if exist "%%f" (
            set "i3dFileFound=true"
        )
    )
    
    if not defined i3dFileFound (
        set "folderWithoutI3d=%%d"
        "C:\Program Files (x86)\Sift\Sift.exe" --NoGui --LoadLib directory "%%d" --LoadQuery file "%%d\query_definition.q3d" --selectSignals --RunPCA name "Outlier_PCA" --RunLOF grouping "group" autoExclude --ExportPCA file "%%d\Group Outliers.txt" exportFormat "transposed" lof lofThreshold --RunLOF grouping "workspace" autoExclude --ExportPCA file "%%d\Workspace Outliers.txt" exportFormat "transposed" lof lofThreshold --SaveProject file "%%d\Outliers.i3d" --Exit
    )
)

endlocal

The Execution

Now that we have a script ready to go, executing our processing will be as easy as double clicking the .bat file, but we can take the automation even further! Leveraging windows Task Scheduler we can set the batch script to execute at a certain time each day

Open the start menu and type Task Scheduler into the search bar
Open Task Scheduler
In the Task Scheduler select create basic task in the top right of the screen
In the popup dialog enter a meaningful name in this case we went with “Baseline Processing” press next
In the next step ensure daily is selected and press next
Next select what time of day you would like the task to run, we picked 12:00AM
In the next step ensure start a program is selected and press next
Now we will browse and select the batch script we created
On the final screen review and press finish
Finally check the existing task to ensure it has been created successfully