HAS-Motion Software Documentation

User Tools

Site Tools

You are here: Main Page » sift » tutorials » Clean your Data
sift:tutorials:clean_your_data

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
sift:tutorials:clean_your_data [2024/07/17 15:22] – sgrangersift:tutorials:clean_your_data [2024/11/28 19:13] (current) – [Clean your Data] wikisysop
Line 1: Line 1:
 ====== Clean your Data ====== ====== Clean your Data ======
  
-This tutorial will show you how to use Sift as a data cleaning, or quality assurance, tool. You will learn how to check for and correct faulty force assignments from all files at once, thereby removing any artifacts from the data. This is particularly useful for lab managers (or supervisors) who may not be familiar with the raw data, or collaborators that were not involved in the collection process.+This tutorial will show you how to use Sift as a data cleaning, or quality assurance, tool. You will learn how to check for and correct faulty force assignments from all files at once. This is particularly useful for lab managers (or supervisors) who may not be familiar with the raw data, or collaborators that were not involved in the collection process.
  
 +If you prefer, a video tutorial is available outlining the same process. It is available at this link: [[https://youtu.be/zgjnGTxhjEc?feature=shared|Sift Tutorial Video 2: Cleaning Data]]
 ==== Data ==== ==== Data ====
  
Line 11: Line 12:
 ==== Loading the library ==== ==== Loading the library ====
  
-{{:CleanTut1.png}}+{{ :CleanTut1.png?800}}
  
-1. Click {{:sift_library_load.png}} **Load Library** in the [[Sift:Application:Toolbar|toolbar]] to open the [[Sift:Application:Load_Library_Dialog|Load Library dialog]].+1. Click {{:sift_library_load.png?20}} **Load Library** in the [[Sift:Application:Toolbar|toolbar]] to open the [[Sift:Application:Load_Library_Dialog|Load Library dialog]].
  
-2. Click {{:sift_browser.png}} **Browse** and select the folder where the CMZ files are stored.+2. Click {{:sift_browser.png?20}} **Browse** and select the folder where the CMZ files are stored.
  
-3. Click {{:sift_reload_lib.png}} **Load** button to import the data.+3. Click {{:sift_reload_lib.png?20}} **Load** button to import the data.
  
 This step selects the path to the data you are using. If you intend to modify the data, e.g., by correcting invalid assignments, then your data must be saved in a directory where you have read and write access. This tutorial uses a folder located on the desktop. The data folder must contain the .cmx file associated with every [[Visual3D:Documentation:Definitions:File_Formats:CMZ_Format_|.cmz]] file, since without both file types Sift will be unable to load the files. Each .cmz file has multiple .c3d files with the different walking speeds identified using Tags. This step selects the path to the data you are using. If you intend to modify the data, e.g., by correcting invalid assignments, then your data must be saved in a directory where you have read and write access. This tutorial uses a folder located on the desktop. The data folder must contain the .cmx file associated with every [[Visual3D:Documentation:Definitions:File_Formats:CMZ_Format_|.cmz]] file, since without both file types Sift will be unable to load the files. Each .cmz file has multiple .c3d files with the different walking speeds identified using Tags.
Line 26: Line 27:
 ==== Defining queries and calculating groups ==== ==== Defining queries and calculating groups ====
  
-1. Navigate to the [[Sift:Application:Explore_Page|Explore Page]] and click on the {{:sift_query_builder.png}} **Query Builder** icon to open the [[Sift:Application:Query_Builder_Dialog|Query Builder dialog]].+1. Navigate to the [[Sift:Application:Explore_Page|Explore Page]] and click on the {{:sift_query_builder.png?20}} **Query Builder** icon to open the [[Sift:Application:Query_Builder_Dialog|Query Builder dialog]].
  
-2. To create a new query definition, click the {{:sift_action_add.png}} addition button in the empty **Queries** list.+2. To create a new query definition, click the {{:sift_action_add.png?20}} addition button in the empty **Queries** list.
  
 2.1. Type GRF in the **Query Name** text box in top-right and click **Save**. 2.1. Type GRF in the **Query Name** text box in top-right and click **Save**.
  
-{{:QueryBuilderCleanTut.png}}+{{:QueryBuilderCleanTut.png?800}}
  
-3. While the GRF query is selected, click the {{:sift_action_add.png}} green addition button in the empty **Conditions** list.+3. While the GRF query is selected, click the {{:sift_action_add.png?20}} green addition button in the empty **Conditions** list.
  
 3.1. Type R_GRF in the **Condition Name** text box in the top-right. 3.1. Type R_GRF in the **Condition Name** text box in the top-right.
Line 42: Line 43:
 3.2.1. **Signals**: This tab allows the user to select specific signals according to their names in the [[Visual3D:Documentation:Visual3D_Signal_Types:Data_Tree_Summary_|Visual3D Data Tree]]. For this tutorial, select the signal: TYPE - LINK_MODEL_BASED, FOLDER - ORIGINAL, NAME - R_GRF, and COMPONENT - Z. 3.2.1. **Signals**: This tab allows the user to select specific signals according to their names in the [[Visual3D:Documentation:Visual3D_Signal_Types:Data_Tree_Summary_|Visual3D Data Tree]]. For this tutorial, select the signal: TYPE - LINK_MODEL_BASED, FOLDER - ORIGINAL, NAME - R_GRF, and COMPONENT - Z.
  
-{{:ConditionCleanTut.png}}+{{:ConditionCleanTut.png?400}}
  
 3.2.2. **Events**: This tab allows the user to specify the desired event sequence to extract data from. For instance, the right gait cycle could be extracted using the event sequences RON, ROFF or RHS, RHS. For this tutorial select the RON and ROFF events and leave the default normalization values of 101 for the number of points and cubic for the spline type. 3.2.2. **Events**: This tab allows the user to specify the desired event sequence to extract data from. For instance, the right gait cycle could be extracted using the event sequences RON, ROFF or RHS, RHS. For this tutorial select the RON and ROFF events and leave the default normalization values of 101 for the number of points and cubic for the spline type.
  
-{{:ConditionsEventsCleanTut.png}}+{{:ConditionsEventsCleanTut.png?400}}
  
 3.2.3. **Refinements**: This tab allows the user to refine their selected signal using two separate and distinct methods: according to the signals contained within the [[Visual3D:Documentation:Definitions:File_Formats:CMO_Format|CMO files]], or according to Tags. For this tutorial, Refine using tags by leaving the **Use AND Logic** checkbox unchecked and selecting the SLOW, NORMAL, and FAST tags. 3.2.3. **Refinements**: This tab allows the user to refine their selected signal using two separate and distinct methods: according to the signals contained within the [[Visual3D:Documentation:Definitions:File_Formats:CMO_Format|CMO files]], or according to Tags. For this tutorial, Refine using tags by leaving the **Use AND Logic** checkbox unchecked and selecting the SLOW, NORMAL, and FAST tags.
  
-{{:RefinementsCleanTut.png}}+{{:RefinementsCleanTut.png?400}}
  
-3.3. Click {{:sift_apply.png}} **Save** to create the condition.+3.3. Click {{:sift_apply.png?20}} **Save** to create the condition.
  
 4. For this tutorial, define a second condition within the same group to account for the left side. This can be done by modifying the existing sub-group: 4. For this tutorial, define a second condition within the same group to account for the left side. This can be done by modifying the existing sub-group:
  
-4.1. While the GRF group is selected, click the {{:sift_action_add.png}} green addition button in the **Sub-Groups** list and type L_GRF in the **Sub-Group Name** text box.+4.1. While the GRF group is selected, click the {{:sift_action_add.png?20}} green addition button in the **Sub-Groups** list and type L_GRF in the **Sub-Group Name** text box.
  
 4.1.1. In the Signals tab, change the NAME to L_GRF and leave the other parameters as they are. 4.1.1. In the Signals tab, change the NAME to L_GRF and leave the other parameters as they are.
Line 64: Line 65:
 4.1.3. Leave the Refinements tab as it was. 4.1.3. Leave the Refinements tab as it was.
  
-4.2. Click {{:sift_apply.png}} **Save** to create this second condition.+4.2. Click {{:sift_apply.png?20}} **Save** to create this second condition.
  
-{{:QueriesMadeCleanTut.png}}+{{:QueriesMadeCleanTut.png?800}}
  
 You should now see one query (GRF) in the **Queries** list and two conditions (R_GRF and L_GRF) in the **Conditions** list. At this point in the tutorial these definitions have been created but they have not been applied to the signals in the loaded library. To do this, click on **Calculate All Queries** (or **Calculate Selected Queries**, since there is only one group in this tutorial). You should now see one query (GRF) in the **Queries** list and two conditions (R_GRF and L_GRF) in the **Conditions** list. At this point in the tutorial these definitions have been created but they have not been applied to the signals in the loaded library. To do this, click on **Calculate All Queries** (or **Calculate Selected Queries**, since there is only one group in this tutorial).
Line 87: Line 88:
 Your graph should now resemble the image below. Your graph should now resemble the image below.
  
-{{:DataCleanOriginalPlot.png}}+{{:DataCleanOriginalPlot.png?800}}
  
  
 2. Use your cursor to select only lines on the graph that you wish to inspect. Click on single traces in order to examine individual curves without the other curves 'polluting' the graph. When a trace is clicked, a tooltip will appear to describe the source of the trace in terms of both Sift's groups and workspaces and the underlying .c3d files. 2. Use your cursor to select only lines on the graph that you wish to inspect. Click on single traces in order to examine individual curves without the other curves 'polluting' the graph. When a trace is clicked, a tooltip will appear to describe the source of the trace in terms of both Sift's groups and workspaces and the underlying .c3d files.
  
-{{:SingleSelectedTrace.png}}+{{:SingleSelectedTrace.png?800}}
  
  
Line 108: Line 109:
 3.1 When traces are excluded, two notable differences should appear in the Queried Data subwindow. First, the previously selected, but now excluded data should not be visible on the graph. Second, the **Workspaces** widget will indicate when traces have been excluded. Specifically, an X should replace the original check mark beside each workspace where signal traces have been excluded, two horizontal bars should replace the check mark beside each .c3d file with excluded traces, and a red X should replace the checkmark beside each excluded trace. Additionally, the fraction of traces excluded for each workspace is displayed on the right-hand side of the **Workspaces** widget: in this case all of the workspaces contained poor quality events. If unintentional exclusion occurred for a workspace, you can right click on the workspace and select 'Re-Include Data'. 3.1 When traces are excluded, two notable differences should appear in the Queried Data subwindow. First, the previously selected, but now excluded data should not be visible on the graph. Second, the **Workspaces** widget will indicate when traces have been excluded. Specifically, an X should replace the original check mark beside each workspace where signal traces have been excluded, two horizontal bars should replace the check mark beside each .c3d file with excluded traces, and a red X should replace the checkmark beside each excluded trace. Additionally, the fraction of traces excluded for each workspace is displayed on the right-hand side of the **Workspaces** widget: in this case all of the workspaces contained poor quality events. If unintentional exclusion occurred for a workspace, you can right click on the workspace and select 'Re-Include Data'.
  
-{{:ExcludedTraces.png}}+{{:ExcludedTraces.png?800}}
  
 4. If you want to visualize the data you have excluded alongside the remaining data, go to the [[Sift:Application:General_Options_Dialog|General Options]] dialog and select the "Show excluded data" and "Use Specific Styles for Excluded Data" options. Then, navigate to the [[Sift:Application:Data_Options_Dialog|Data Styles]] dialog where you can choose to use a specific line style for excluded data. 4. If you want to visualize the data you have excluded alongside the remaining data, go to the [[Sift:Application:General_Options_Dialog|General Options]] dialog and select the "Show excluded data" and "Use Specific Styles for Excluded Data" options. Then, navigate to the [[Sift:Application:Data_Options_Dialog|Data Styles]] dialog where you can choose to use a specific line style for excluded data.
  
-{{:ShowExcludedCleanTut.png}}+{{:ShowExcludedCleanTut.png?800}}
  
  
Line 119: Line 120:
 5.1 Select the traces of interest by clicking and dragging on the plot. 5.1 Select the traces of interest by clicking and dragging on the plot.
  
-5.2 Click the {{:sift_show_animation.png}} **Show Animation** button.+5.2 Click the {{:sift_show_animation.png?20}} **Show Animation** button.
  
 5.3 The Show [[Sift:Application:Animation_Dialog|Animation dialog]] can display animations for those selected traces with enough data in the .cmz file to permit a recreation. Each of the selected traces can be selected from the drop-down box near the top of the dialog. This view can be used to determine what (if any) issue exists in the data and whether this merits exclusions. Excluded traces are removed from the drop-down box within the dialog as well as the Queried Data subwindow. 5.3 The Show [[Sift:Application:Animation_Dialog|Animation dialog]] can display animations for those selected traces with enough data in the .cmz file to permit a recreation. Each of the selected traces can be selected from the drop-down box near the top of the dialog. This view can be used to determine what (if any) issue exists in the data and whether this merits exclusions. Excluded traces are removed from the drop-down box within the dialog as well as the Queried Data subwindow.
Line 125: Line 126:
 6. Once all of the desired exclusions have been made, the original data is ready to be updated. 6. Once all of the desired exclusions have been made, the original data is ready to be updated.
  
-{{:ShowAnimationCleanTut.png}}+{{:ShowAnimationCleanTut.png?800}}
  
  
 ==== Updating the original data ==== ==== Updating the original data ====
  
-1. +{{:UpdateCMZActive.png?400}} 
-{{:UpdateCMZActive.png}}+ 
 +1. Now that the incorrect force assignments have been identified in our data set, it is possible to update the original [[Visual3D:Documentation:Definitions:CMO_Library_|CMZ]] data by clicking the {{:sift_update_cmz.png?20}} **Update CMZ Files** item in the application's tool bar to open the [[Sift:Application:Update_CMZs_Dialog|Update CMZ Dialog]]. This opens a dialog that provides different options for processing excluded data.
  
-Now that the incorrect force assignments have been identified in our data set, it is possible to update the original [[Visual3D:Documentation:Definitions:CMO_Library_|CMZ]] data by clicking the {{:sift_update_cmz.png}} **Update CMZ Files** item in the application's tool bar to open the [[Sift:Application:Update_CMZs_Dialog|Update CMZ Dialog]]. This opens a dialog that provides different options for processing excluded data. 
 2. The **Excluded Traces** options allow the user to add a BAD event to the excluded traces. This option is helpful if the intention is to [[Visual3D:Documentation:Definitions:Tag|tag]] the bad data, re-open the original data, and make manual corrections one trace at a time. 2. The **Excluded Traces** options allow the user to add a BAD event to the excluded traces. This option is helpful if the intention is to [[Visual3D:Documentation:Definitions:Tag|tag]] the bad data, re-open the original data, and make manual corrections one trace at a time.
  
sift/tutorials/clean_your_data.1721229739.txt.gz · Last modified: 2024/07/17 15:22 by sgranger