QALMRI Short answer questions

QALMRI-C Instructions (20 points)
Each student will write four papers this semester. For each paper, students can select an article
(from a set of two choices) to summarize. The summaries should follow a modified QALMRI
method. Below is a list of the components of the modified QALMRI summary, along with the
points for each component. Further descriptions of what should be covered in each section
appear on the following pages. An example summary is posted on Cougar Courses.
Title page (1 point)
Questions (2 points)
Alternatives (2 points)
Logic (2 points)
Methods (4 points)
Results (4 points)
Inferences (2 points)
Connections (2 points)
References (1 point)
Questions:
What was the broad question being asked by this research project?
What was the specific question being asked by this research project?
Alternatives:
What were the authors’ hypotheses?
What were the alternative hypotheses?
Logic:
What was the logic of the hypotheses?
If the authors’ hypothesis is correct, then what should happen?
Method:
Briefly describe the study design. What were the independent variables? What were the levels
of the independent variables? Where the independent variables manipulated within- or
between-subjects? What were the dependent variables? How were these measured?
Briefly describe the study procedure in everyday terms. Describe what participants did.
Results:
What were the important results?
Which of the authors’ hypotheses were supported?
Inferences:
What did the authors conclude from their study?
How did the authors use the results to make inferences and conclusions about the hypotheses
and questions?
Connections:
How does this study relate to course material (textbook and lectures)?
How does this study relate to the real world?
Sections of the Paper
Title page
The first page of your paper should be an APA-formatted (7th edition student format) title page.
Q: Question
What was the broad question(s) being asked by this research project?
What was the specific question(s) being asked by this research project?
All research begins with a question and trying to answer the question is the point of conducting
research. The first step to understanding any empirical article is to identify the question or
questions that were asked by the authors and understand why the question was important
enough that we should care about the answer. In general, there are often two categories of
questions being asked: broad and specific. Broad questions are typically too general to be
answered by any single experiment and provide an overview of the general topic of interest
(e.g., “What is the influence of playing video games on our daily behavior?”). Specific questions,
on the other hand, can be addressed by a single experiment or set of experiments (e.g., “Does
playing violent video games cause children to engage in more violent behavior?”). Answering
one or more specific questions should be considered steps made toward addressing a broad
question.
Generally, the first few paragraphs of the introduction of an empirical article should include the
questions the article is addressing. The broad question can often be found in the very first
paragraph of the introduction, where the authors introduces the broad topic of interest that is
being examined. It should be noted that sometimes the broad question is not made explicit and
may require some work on the part of the reader to draw a connection from the specific
question to the broad topic. Additionally, specific questions might tap multiple broad issues
making it difficult to identify a single broad question.
Hint: If you are having difficulty identifying the broad question, first identify the specific
question, then try to connect that question to the broader topic. For example, the article might
quickly introduce the specific question, “Does playing violent video games cause children to
engage in more violent behavior?” and from this you might conclude that the broad question is
about how video games impact our behavior. However, the topic could have been introduced
by describing factors that can cause children to engage in violent behavior, and not by
describing other ways video games have been shown to influence behavior. In this case, the
larger issue might not be video games per se but might center on aggression and childhood
behavior. The specific question in this example is related to both of these broad questions, but
it is important to separate the broad question as understood by the authors, and other
potential issues the question might relate to.
Depending on the writing style, the specific question might be found early in the introduction,
shortly after the broad topic is introduced, or near the end of the introduction after the authors
has provided a review of previous work on the topic. The review of the previous work should
also provide some context, explaining why the questions being addressed are interesting,
important, and worth spending time and resources addressing.
A: Alternatives
What was the authors’ hypothesis?
What were the alternative hypotheses?
A good empirical article will consider at least two possible answers to the specific question(s)
being asked. Each possible answer proposed in the article is called a hypothesis. The authors
should explain why each possible answer is plausible, usually referencing previous articles and
theories. However, the authors’ preferred or favorite answer is called “the hypothesis,” while
other proposed answers are called “alternative hypotheses.” It is important to note that some
studies have multiple questions. Each of these questions will require its own hypothesis and
alternative hypotheses. The hypotheses can usually be found in the general introduction. After
describing the questions, the authors should provide the hypotheses and explain why those
hypotheses are plausible.
Again, it is worth reminding students that the hypotheses are the possible answers to the
question. However, sometimes the alternative hypotheses are not explicitly stated, or even
considered by the authors. If the alternatives are not stated clearly, the student should try to
figure out what they could be on your own. For example, if the study is attempting to confirm
the predictions of a single theory, what might other theories predict? Could a different
interpretation of the theory proposed by the authors make a different prediction?
L: Logic
What was the logic of the hypotheses?
If the authors’ hypothesis is true, then what should happen?
The goal of any research project is to carry out an experiment or set of experiments to
discriminate between alternative hypotheses. The logic, therefore, is the general idea
underlying how the alternatives might be distinguished, using empirical data as evidence for or
against each hypothesis. Ideally, you should be able to state the logic in the form of an if–then
statement. That is, if the authors’ hypothesis is correct, we would predict that manipulating a
particular variable should change the participant’s behavior in a specific way. If the alternative
hypothesis is correct, then we would predict that manipulating a specific variable would change
the participant’s behavior in a different way. The logic of the study generally appears near the
end of the general introduction. Often, the authors will provide an overview of the research
methods being applied in the study, and the predictions made in accordance with relevant
hypotheses. From these predictions, you should be able to derive if–then logical statements.
M: Methods
The methods are the details of what the researchers did in the study. The amount of
methodological information included in an article can be overwhelming. As a reader, you
should first determine what your goals are, and what level of detail you wish to learn about
research methodology. We will make a distinction between having a general understanding of
research design and understanding all the methodological details. Depending on your goals as a
reader, a general overview may be sufficient.
A general overview of the research design. Before diving into the gritty details, you should first
familiarize yourself with the general design of the study. It is important to note that there are a
variety of different methods for testing a hypothesis and you should first try to identify the
general method being applied. Some of the most common designs used in psychological
research are experimental, quasi-experimental, and correlational. Experimental designs
manipulate an independent variable to see changes in a dependent variable. For example, you
could manipulate the time of day students take a test (evening vs. morning) and measure the
change in test scores. Critically, in an experimental design, participants are randomly assigned
to one of the groups (evening or morning). On the other hand, quasi-experimental designs do
not randomly assign participants to groups and instead rely on existing group memberships
(e.g., musician vs. non-musician, married vs. single). For example, you could measure the
difference in test scores (dependent variable) for male and female students. In this case, gender
is treated as an independent variable even though we do not randomly assign participants to
each condition. Finally, in correlational designs we measure variables and look for relationships
between them. For example, you could measure the amount of money people make and how
happy they are to examine whether there is a relationship between salary and happiness. Once
you have determined the general type of research design, you should identify all the key
components of the design. If the study uses an experimental or quasi-experimental design you
should be able to identify the dependent variable(s) (what is being measured) and the
independent variable(s). Are the independent variables within- or between-subjects? Is the
independent variable manipulated (i.e., experimental) or using existing group memberships
(i.e., quasi-experimental)? If it is a factorial design, you should identify the levels for each
factor. It might be worthwhile to draw a table, renaming and mapping the factorial design onto
the appropriate cells of the table.
A general overview of the research design is usually described at the end of the general
introduction, or if there are multiple studies, in the introduction to each study. If there are any
details that are unclear from the general overview, you can search them out in the Methods
section.
Methodological details. The methodological details found in the Methods section, provide indepth descriptions of all the materials and procedures used throughout the study. For example,
the methods section will describe who participated in the study and how they were selected. It
will also describe what materials they used, how they were constructed, etc. There should be
enough detail included that any researchers could replicate the study without tracking down
the original authors to ask questions. The Methods section is usually broken down into the
following subsections:
Participants. While reading this section, you should try to understand who participated in the
study. How were they selected? Are multiple groups being compared? To what population are
the researchers intending to generalize their results? Is the sample representative of that
population? If a study was conducted to examine a particular population (e.g., men in their
early 20’s), then the participants should be as similar to that group as possible. If no particular
group is specified, then the sample should be representative of the population in general. If the
study uses more than one group (i.e., a between-subjects design), they should be equivalent on
important demographic variables such as age, education, or gender. You should try to think of
any demographic variables that are not described, or controlled for by the experimenter, that
could influence the results.
Materials. The apparatus is any equipment used during data collection, whether to deliver
stimuli or measure responses. The materials can include scripts, surveys, and software used for
data collection, as well as any stimuli presented to participants throughout the study. The
authors should describe exactly how stimuli were presented, how those stimuli were
constructed or chosen, and how responses were recorded. For example, if the researchers used
a computer apparatus, they should describe the software used, how long stimuli were
presented on the screen, the size of the stimuli, and the types of responses recorded. You
should think about how the apparatus and stimuli would have looked to the participant. Is
there any aspect that would have been distracting or confusing to the participant? Is the
apparatus, stimuli, response collection, etc., appropriate for the specified task?
Procedure. The procedure is the step-by-step listing in chronological order of what a participant
did in the study, and if appropriate, a step-by-step listing of what a participant did for any given
individual trial. You should try to picture yourself as a participant in the study. Does the task
seem easy or difficult? What were the instructions given to participants? Were the instructions
clear enough that participants would have understood them? Is it possible the researchers
treated participants in different groups differently?
R: Results
What were the outcomes? The outcome of the study will be detailed in the Results section.
First, you should try to gain an understanding of how participants generally performed in the
task. The results of the study are often summarized using descriptive measures of central
tendency (means, medians, or modes) and variability (e.g., standard deviation). These
descriptive measures are usually displayed in a table or figure that provides you with an easyto-understand summary of the results.
Second, how do you know the differences you see in the descriptive measures are reliable and
should be taken seriously? We rely on inferential statistics to help us make judgments about
our data. In the results section, the authors will report the statistical tests (e.g., t-test, ANOVA)
that they used to analyze their data and the resulting p-values. The p-value of a statistical test
represents the probability you would have observed the reported difference in the sample, by
chance alone assuming there are no true differences in the population (for more information,
see Gigerenzer, 2004). The p-value is always a number between 0 and 1, and the commonly
accepted standard is 0.05. If the p-value is less than 0.05, (say, 0.049) the result is said to be
“significant,” and we can be reasonably certain that the difference found in the sample,
represents an actual difference in the population. Other common p-values include 0.01 and
0.001, each of which increases the probability that your results were not due to chance and
represent an actual difference in the population.
Sometimes, the results section can be difficult to navigate. There can be numerous statistical
tests with many different results. You should try to identify the important results. Which
statistical tests directly relate to the questions asked? In other words, the hypotheses made
predictions about the changes we would expect to observe in our dependent variable for each
of the groups or conditions. For example, there might be predicted differences between specific
groups or, in the case of correlational designs, predicted associations. Try to find the statistical
tests that test those specific predictions.
I: Inferences
The results section details the results of the authors’ measurements, and statistical inferences
about whether differences between those measurements should be considered reliable. But
what does it all mean? The real payoffs of conducting an empirical study are the inferences one
can draw from the results that bear on the questions asked and help identify which of the
possible answers (i.e., hypotheses) are most likely to be true. Given the results, what did the
authors conclude? The Discussion section will contain the inferences (note that the use of the
word inference is separate from inferential statistics) the authors made about their results.
Ideally, if the logic and methodology are sound, the results should be more consistent with only
one of the hypotheses, allowing the authors to eliminate one or more alternatives. At this
point, you should be able to work backwards through the first half of our QALMRI answering all
the questions the authors originally set up. For example:
(Logic) How do the results line up with the logical if–then statements?
(Alternatives) Given the results, which hypothesis does the logic implicate?
(Questions) What does that hypothesis say about the specific and broad questions?
As a goal, you should try to summarize the authors’ conclusions in a short paragraph, as they
relate to the logic, alternatives, and questions.
Once you gain an understanding of the inferences the authors have made regarding the results,
you should try to think critically about their conclusions and about broader implications. Do the
statistical tests support the authors’ conclusions? Were any of the alternatives not convincingly
ruled out? Why? Were there any limitations or confounding variables that could alter their
interpretation of the results? For example, if the study uses existing group memberships (e.g.,
musician vs. non-musician), are there any other differences between the groups that could also
explain the results? What specific and broad questions did this particular study fail to address?
Do the results of this study create new specific questions that might help us understand the
broader question?
C: Connections
Describe how the material in the article related to course material (textbook and lecture). Were
there similar studies described in the course? Were the findings similar or were they different?
Which concepts for the course were mentioned in the article? These connections should be
described in one paragraph. Cite the textbook and lecture in APA formatted in-text citations.
References
The last page of your paper should be an APA-formatted reference list.
The above instructions were adapted from:
Brosowsky, N., Parshina, O. (2017). Using the QALMRI method to scaffold reading of primary
sources. In R. Obeid, A., Schartz, C. Shane-Simpson, & P. J. Brooks (Eds.) How we teach
now: The GSTA guide to student-centered teaching.
http://teachpsych.org/resources/Documents/ebooks/gstaebook.pdf
Brosowsky, N., Parshina, O., Locicero, A., & Crump, M. J. C. (2020). Teaching undergraduate
students to read empirical articles: An evaluation and revision of the QALMRI method.
PsyArXiv. https://doi.org/10.31234/osf.io/p39sc