Project Checkpoint 3 - Design of Experiments/Case Studies
| Possible Points | Due Date |
|---|---|
| 50 pts | March 24th - Before Class |
Overview
For the third project checkpoint, you will be writing the first draft of the design of experiments or case study for your project. This means you will be describing the experimental methodology that you will be using to evaluate how well your approach works, or if you are conducting a study, describing your research questions and how you will answer those research questions. It is OK if you do not know all of the details of your evaluation yet, and we hope that writing this section will help you reason about and finalize your design for the project.
Checkpoint Instructions
Design of Experiments/Case Studies
Expected Length: (~2 pages)
Your overall goal in the approach description section of the paper is threefold: (i) you should provide the reader with a (i) the research questions that you are trying to answer, these should be the key aspects of your approach that you wish to evaluate, or the key aspects that you want to learn from your study (ii) you should provide the context of your study, that is what are the materials that you are using to evaluate it, and (iii) what are the metrics and methodology that you are using to measure how well your approach performs, or how complete the answers to your RQs are. You should use figures, tables, and examples where necessary to more clearly explain concepts.
In order to receive full points for this project checkpoint, your Design of Experiments section should include the following:
- Research Questions: In order for your audience to understand how well your proposed approach is working, or the findings of a study that you are conducting, you must explicitly state what it is you are measuring, or what phenomenon you hope to observe. This is best done through carefully worded research questions (RQs). These questions essentially state what you are investigate in an interrogative manner. For approach papers, each research question is typically formed around one quality attribute that you aim to measure. For example, this could be precision, recall, or a user opinion from a user study. For study papers, this typically takes the form of different things that you want to lean about the data. For instance, observing a trend, or looking for certain patterns.
- Study Context: After introducing your research questions, you then need to explain the context within which they will be measured. In this section you should describe your data, how it was collected, as well as any salient configurations of the approach that you intend to evaluate. For example, for a paper evaluating an Android testing tool, I might describe any app-related data used to train an ML approach, and the apps that I will then evaluate the test generation approach on.
- Methodology for Each RQ: Finally, after introducing the context, you then need to explain precisely how you will answer each research question. Thus, you should create a sub-section for each RQ that explains how you will answer that RQ. This should include a discussion of metrics, how your approach will be used to generate results, and other experimental factors.
For an example of a recent paper that follows this format, please see my recent paper on software traceability.
Submission Instructions
Project Checkpoints should be submitted as a PDF via Blackboard to the appropriate assignment.