# An Experiment on the Effects of using Color to Visualize Requirements Analysis Tasks: Open Science Supplement

Authors: Yesugen Baatartogtokh, Irene Foster and Alicia M. Grubb.

This repository contains the supplemental information for the above named paper. It consists of the study instrument and data. We also include the script required to complete our statistical analysis. Researchers may use this supplement for the purpose of study replication. Educators may also use the models for educational purposes.

With these materials, the user may:

* Review and reuse the study materials.
* Run scripts for statistical tests and data analysis.

## Description of Artifact

*Artifact-Repo*  
|\_\_**README.md**  —  This file.  
|\_\_**LICENSE.md**  —  The license for this artifact.  
|\_\_**Supplement.pdf**  —  Open Science Supplement.   
|\_\_*pdfs/*  —  Folder containing the .pdf files used in this study.  
|  |\_\_*BK-Model-CTL*  —   Bike Evolution Control  
|  |\_\_*BK-Model-EVO*  —   Bike Evolution EVO  
|  |\_\_*BK-Q11-CTL*  —   Bike Evolutions Comparison Control  
|  |\_\_*BK-Q11-EVO*  —   Bike Evolutions Comparison EVO  
|  |\_\_*SM-Model-CTL*  —   Summer Evolution Control  
|  |\_\_*SM-Model-EVO*  —   Summer Evolution EVO  
|  |\_\_*SM-Q11-CTL*  —   Summer Evolutions Comparison Control  
|  |\_\_*SM-Q11-EVO*  —   Summer Evolutions Comparison EVO  
|  |\_\_*TNE-Handout*  —   Evolving EVO training  
|  |\_\_*TNG-Handout*  —   Initial Goal Modeling Training  
|  |\_\_*TNS-Handout*  —   Function Types Training  
|  |\_\_*TNS-Sim-Path*  —   Evolution of Training Model  
|  |\_\_*Video-Slides*  —   Training Video Slides   
|\_\_*statistics/*  —  Folder containing the raw anonymized data from the study, as well as the Python and RStudio scripts used in analyzing the data.  
|  |\_\_*data_anova.csv*  —   Within-subjects analysis data.   
|  |\_\_*data_scored.csv*  —   Scored data for analysis.   
|  |\_\_*data.csv*  —   Raw study data after anonymization used for quantitative analysis.   
|  |\_\_*final.csv*  —   Final dataset for between-subjects analysis.   
|  |\_\_*qualitative-data-scored.csv*  —   Scores after qualitative analysis was performed.   
|  |\_\_*qualitative-data.csv*  —   Raw study data after anonymization used in qualitative analysis.   
|  |\_\_*script-analysis.R*  —   Statistical analysis for Results section.   
|  |\_\_*script-anonymize-data.R*  —   Anonymizes survey data for further analysis.   
|  |\_\_*script-anova.R*  —   Calculates the analysis of variance for the discussion section.   
|  |\_\_*script-pairing4anova.py*  —   Creates dataset for within-subjects analysis.   
|  |\_\_*script-scoring-data.py*  —   Scores question data for statistical analysis.   

## Steps to review study materials.

One of the goals of this supplement is to allow researchers to review and reuse our study materials. `Supplement.pdf` provides the exact wording of all instruments used in the study. It also references the files in the `pdfs/` folder. 

[BloomingLeaf (Release 2.5)](https://github.com/amgrubb/BloomingLeaf/releases/tag/v2.5) was used to create the model shown in Supplement.pdf. BloomingLeaf was not used for any other part of this study and analysis.


## Steps to reproduce analysis.

**Preconditions:** The R script used for the paper were were created using RStudio for MAC (Version: 2022.12.0+353 | Released: 2022-12-15), which required R 3.3.0 (but we used the R 4.2.2 binary). 

**Required Packages:** Additionally, the R scripts require the instaltion of the follow packages directly in RStudio: `readr`, `tidyverse`, `ggplot2`, `reshape2`, `"dplyr"`.

1. Run `script-anonymize-data.R`. This script produces `data.csv` and `qualitative-data.csv`. _Note: The file `script-anonymize-data.R` will not run because the required source file is not included in this repository, as it contains personally identifying information. We include this script for reference and review._  
The resulting `qualitative-data.csv` file contains the raw data after anonymization. We then scored the anonymous results manually, in the `qualitative-data-scored.csv` file, using expert judgement and the question answers. For each question, responses were given a 0 or 1 score depending on whether they demonstrated reasoning that would follow from the model. During scoring, we did not have access to the subjects' grouping, to limit any researcher bias. After scoring the scored qualitative data responses and those scores were manually added to `data.csv`, for the next step.
2. Run `script-scoring-data.py`, which takes as input `data.csv` and outputs `data_scored.csv`. 
3. Run `script-analysis.R`. This removes extra columns not required for analysis and then completes the analysis required for Section IV. Results of the paper. It also outputs `final.csv`, which is used in the next step.
4. Run `script-pairing4anova.py`. This script creates a new dataset to enable within-subjects analysis and outputs `data_anova.csv`. 
5. Run `script-anova.R`, which applies the statistics discussed in Section V. Discussion of the paper. 

_Note: Apologies for the many steps, we were using a distributed development model and can restructure these elements._
	
## License

This work is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https://creativecommons.org/licenses/by-nc-nd/4.0/). See LICENSE.md for more information.

## Citation Information
Recommended Citation:  
Baatartogtokh, Yesugen; Foster, Irene; and Grubb, Alicia M., "Supplemental Material for IRB Protocol #20-026: Representing and Merging Diverse Goal Models in Requirements Engineering" (2023). Computer Science: Faculty Publications, Smith College, Northampton, MA. https://scholarworks.smith.edu/csc_facpubs/343

DOI: doi.org/10.35482/csc.002.2023