APPENDIX A
I •
Pilot Study
The design of the questionnaires calls for the use of the paired comparisons technique. The purpose of the pilot study was to construct the stories for the questionnaires so as to make them equivalent in ali important respects except the one which is of interest to the theories being tested, so that by eliminating artifactual biases in the instruments, any differences that may be found in the stories can be attributed to the theories that are being tested, and not to the intruments themselves.
The dimensions on the basis of which the stories would be made equivalent were obtained mostly from Nevvcomb's articles (1959, p.
384). The questions and the respective dimensions of each are pre-sented below:
a) "How favorable or unfavorable does P feel about (issue)3?"
b) "Hovv favorable or unfavorable does O feel about (issue)?"
c) "How important do you think (P's attitude to issue) is to P?"
d) "How important do you think (O's attitude to issue) is to O ?"
e) "How Committed would you say P is in his attitude tovvard (issue)?"
f) "Hovv Committed would you say O is in his attitude tovvard (issue)?"
g) "To vvhat extent do you think that agreement or disagree-ment about this issue influences hovv much P and O like each other?"
h) "To vvhat extent does O's behavior or opinion on this topic help or hinder P's behavior or opinion about it?"
8 The parentheses are replaced by a statement of the issue or by a stateraent of either P's or O's attitude to the issue appropriate to the particular story.
97
CONCEPT LABEL used in Tables on pages 139-142 a) Favorability of P to the issue ia question P-Favorability b) Favorability of O to the issue in question O-Favorability c) Importance of the issue for P P-lmportance d) Importance of the issue for O O-Importance e) Committedness of P to the issue P-Committedness f) Committedness of O to the issue O-Committedness g) Relevance of the issue to the relationship
of P and O Relevance
h) instrumentality of the issue for p instrumentality
Four balanced (in Heiderian sense) stories and four unbalanced ones were needed to compose the four pairs of stories vvith ali the configurations of interest (please refer back to page 45); the stories are presented in Appendix F. As noted in Chapter 3 on methods and procedures, the instructions that the subjects read along with the stories varied from tereatment to treatment, but the stories themselves did not vary in wording, except in the all-sentiment-relations manipulation (i.e., all-sentiment/consistency;
all-sentiment / pleasantness; maximum/pleasantness conditions ali containing the same affective content). Hence in the pilot study, I examined subjects5 reactions to the stories in only tvvo condi-tions— the Standard condition and the all-sentiment condition.
Judgment Phase
A set of 6 graduate students (from Sociology and Psychology) served as judges rating the drafts of the 8 stories of the Standard condition on the 8 dimensions. Follovving their ratings, the experi-menter conducted an informal intervievv vvith each judge, asking him hovv he interpreted each story, discussing the specific vvording and probing for ambiguities. With the help of the feedback received from their responses and from the intervievvs vvith the judges, the stories vvere altered by the experimenter. A second set of 6 gradu-ate students further helped to revise and refine the stories vvith respect to the dimensions in question. A third set of 5 judges (2 of vvhom vvere upperclass undergraduates) also served in a similar vvay to modify the content of the stories in the desired direction.
98
The judgment phase of the pilot study for the all-sentiment-relations questionnaire was run in similar fashion except that, having pretty much exhausted the graduate judges pool, a third set of judges could not be obtained.
At the beginning the judgments were made on 11 point scales.
Since the judges tended to ignore the fine gradations provided by the 11-point scale but instead responded as though the scale had fewer gradations than it actually did, it was later reduced to a 6-point scale. Nevertheless, the judges could stili not utilize the whole range, but would tend to make their ratings in terms of a narrovver (a 4 point) range. The interviews revealed that the subject would give a rating of 5 (e.g., committedness) to one story and could give a rating of 4 to the other story in the pair, for no other reason than a search for "variety". Since none of the pre-cautions I took could solve this tendency, and because ali the stories presented characters taking rather "extreme" starıds, they did not present too much variety from story to story. Therefore, I decided that a difference of one unit between the paired stories would have to be tolerable. A vocabulary of such fine gradations on meaning proved to be unavailable to the judges that I inter-viewed. Besides, the efforts to eliminate such "minute" differences would result in the addition of more and more factual information that would svvell the size of the individual stories which were already cognitively oriented and maximally structured.
A similar problem with language was encountered with the wording of the questions of the last two dimensions- -i.e., relevance and instrumentality. The questions that were meant to test the two concepts were repeatedly called by the subjects "hard to rate", "a rather diffi-cult question to answer", "depends on the situation". I decided (after 3 rounds of trial with graduate students) that these questions fell short of tapping the respective dimensions, and that they might have to be dropped. Later, I decided to keep the questions but without predicting story equivalance based on these. Even though the two questions might not be too useful for testing hypotheses of equivalence, they could serve for some post hoc explanations.
Further, the experimental design included one condition with ali unit relations in the triad. This called for the (-) P /O relation of the stories also to be modified by changing them from a negative senti-ment relation into a negative unit relation. There was only one state-99
ment to be transformed: "P dislikes O" The best three candidate statements in terms of unit relations to replace this sentiment relation were: "P ignores O"; "P avoids O"; "P is dissimilar to O". Because the spring semester had already ended, very few "naive" potential judges were left. For this reason, only four persons (2 undergraduates, 1 graduate student and 1 faculty member) volunteered their comparative judgments on these statements. The modal judgment favored the use
of "P avoids O".
The intervievvs with the judges lasted from the 2nd vveek of Feb-ruary until the beginning of April, 1976.
Test Phase
Following the judgment phase of the pilot study, the experimenter decided upon a final version of the wording of each story. She submit-ted this version to a test to determine whether or not the pairs of stories which differed in verbal content and vvhich differed in their Heiderian balance configurations were nevertheless sufficiently com-parable in ali other relevant dimensions, such that it would be legiti-mate to use these pairs in the remainder of this study.
With the help of a S 100 fund supplied by the Sociology depart-ment, altogether 40 subjects (20 subjects for standard, 20 subjects for all-sentiment questionnaire) were hired (for $ 2 for 1 /2 hour) to rate each of the stories on the given 8 dimensions. The questionnaires were administered individually to each subject. The researcher kept working at a desk in the same room during the 30 odd minutes it took answering ali the questions. By way of instructions, the subjects received the following message:
"This is not an experiment, but an important phase of the pre-parations for an experiment. The task is called a judgment task.
You will be asked to rate several passages on certain dimensions.
You will find ali the information you need in the questionnaire. You may begin now." This message was designed to alleviate any suspi-cion and defensiveness on the part of the subjects, and so prevent their giving unauthentic ansvvers to the questions.
While the stories were presented in pairs to the judges, the sub-jects rated the 8 stories (plus 2 filler stories) one by one, the order having been randomized by means of a table of random numbers.
100
The subjects were obtained by way of sign-up sheets hung in the halis of North Campus and Noyes Center Dorms (mostly freshmen dorms) at Cornell University. They were contacted by phone and vvere scheduled to take the questionnaire one person at a time. The running of subjects (without interviews) started in the first week of April and ended the 2nd week of May 1976.
Results
The results from the test-phase of the pilot study are presented in Tables 3JL and 3-2, separately for each pair of stories. Likewise, the results are presented separately for the questionnaire used in the standard condition (stories having only one sentiment relation), and for the questionnaire used in the all-sentiment condition (stories having three sentiment relations).
The "t-method for putting confidence bands on means" (Darling-ton, p. 96, method EE2) was used to interpret the results. Since I am interested in seeing whether the difference betvveen pairs of ratings (for example, the difference between the rating vvhich the seascape story gets and vvhich the other story in the pair, namely African mask, gets) is bigger than the tolerance interval of one unit, I would like to put confidence intervals around this difference. More specif-ically, I vvould like to see vvhere the mean difference from my sample would fail in a distribution of mean differences.
Looking at the 95 % confidence bands for the mean differences, we see that most of the differences are confined betvveen the limits of
+ 1 and -1 range. In other vvords, the probability of two stories in a a given pair differing by more than 1 interval is about 5 %. In fact, on the average, only one out of 6 such differences slightly exceeded the 1-unit of tolerance interval.
Since the research design calls for positive favorability in some stories but negative favorability in others, such stories had to attain opposite ratings on the dimensions in question. For example, P is rated very high on favorability tovvard the jukebox (has a positive attitude tovvard the jukebox) vvhile P is rated very low on favorability tovvard soap operas (has a negative attitude tovvard soap operas). For those particular dimensions vvhere a very large difference was intended by the experimenter, the confidence limits shovv that the probability that the difference is less than 4 units is 2.5 %. The results shovv that these dif-101
TABLE A.l
Means and Confidence Limits of Pilot Ratings on Eight Dimensions, for the Four Pairs of Stories of the Questionnaire in the Standard Condition (n=20).
Means
African 95 % confidence limits Dimensions Seascape Mask Difference for the Mean Difference P-Favorability 4.70 4.65 0.05 -0.19 0.29 P-Importance 4.80 4.45 0.35 0.12 0.58 P-Commitment 4.80 4.20 0.60 0.13 1 .07 O-Favorability 0.40 4.10 -3.70 -4.18 -3 .22*
O-Importance 4.35 3.70 0.65 0.16 1 .14 O-Commitment 4.05 3 .55 0.50 0.01 0.99 Relevance 4.40 3 .40 1.00 0.34 1 .66 instrumentality 2.42 3 .58 -1.16 -2.03 -0.29
Dimensions
Means
95 % confidence limitş for the Mean difference Dimensions Jukebox
Soap
Opera Difference 95 % confidence limitş for the Mean difference P-Favorability 4.80 0.35 4.45 4.04 4.87*
P-Importance 4.25 4.00 0.25 -0.18 0.68 P-Commitment 4.20 4.20 0.00 -0.50 0.50
O-Favorability 4.30 4.85 -0.55 -0.87 -0.23 O-Importance 3 .80 4.60 -0.80 -1 .13 -0.47 O-Commitment 4.20 4.40 -0.20 -0.56 0.16 Relevance 3 .45 3 .50 -0.05 -0.91 0.82 instrumentality 3.95 2.75 1 .20 0.53 1 .87
Dimensions
Means
95 % confidence limits for the Mean difference Dimensions Wagner Dali Difference 95 % confidence limits for the Mean difference P-Favorability 0.45 0.55 -0.10 -0.50 0.30 P-Importance 4.20 4.30 -0.10 -0.50 0.30 P-Commitment 4.30 4.15 0.15 -0.20 0.50 O-Favorability 4.80 0.65 4.15 3 .84 4.46*
O-Importance 4.50 4.30 0.20 -0.16 0.56 O-Commitment 4.60 4.35 0.25 -0.12 0.62 Relevance 3.75 3 .40 0.35 -0.14 0.84 instrumentality 2.10 3 .70 -1.60 -2.36 -0.84
Dimensions
Means
95 % confidence limits for the Mean difference Dimensions Svvamp River Difference
95 % confidence limits for the Mean difference P-Favorabilityv 0.20 4.85 -4.65 -4.92 4.38*
P-Importance 4.25 4.65 -0.40 -0.81 0.01 P-Commitment 4.40 4.85 -0.45 -0.96 0.06 O-Favorability 0.20 0.53 -0.35 -0.66 -0.04 O-Importance 4.40 3.95 0.45 -0.29 1 .19 O-Commitment 4.50 3.90 0.60 -0.24 1 .44 Relevance 4.10 4.45 -0.35 -0.62 -0.08 instrumentality 4.10 3.65 0.45 -0.41 1 .31
* This large difference vvas intended by the experimener.
TABLE .
Means and Confidence Limits of Pilot Ratings on Eiglıt Dimensions, for the Four Pairs of Stories of the Ouestionnaire in the AU-sentiment Condition (n=20).
Dimensions Seascape Means African
Mask Difference
95 % Confidence limits for the Mean difference P-Favorability 4.85 4.70 0.15 -0.08 -.38 P-Importance 3.75 3.90 -0.15 -0.61 0.31 P-Commitment 4.10 4.30 -0.20 -0.67 -0.27 O-Favorability 0.20 4.60 -4.40 -4.89 -3 .91*
O-Importance 4.10 3.75 0.35 -0.41 1 .11 O-Commitment 3.75 3.65 0.10 -0.74 0.94 Relevance 2.85 2.85 0.00 -0.86 0.86 Instrumentality 2.20 3.15 -0.95 -1 .53 -0.37
Dimensions Jukebox
Means Soap
Opera Difference
95 % Confidence limits for the Mean Difference P-Favorability 4.70 0.25 4.45 4.10 4.80*
P-Importance 4.40 4.65 -0.25 -0.70 0.20 P-Commitment 4.20 4.75 -0.55 -0.94 -0.16 O-Favorability 4.35 4.90 -0.55 -0.83 -0.27 O-Importance 3 .70 4.65 -0.95 -1 .34 -0.56 O-Commitment 4.15 4.60 -0.45 -0.94 0.04 Relevance 3.58 3.84 -0.26 -1.06 0.53 Instrumen'ality 4.20 3.25 0.95 0.02 1 .88
Dimensions Wagner
Means
Dali Difference
95 % Confidence limits for Mean difference P-Favorability 0.25 0.30 -0.05 -0.23 0.13 P-Importance 4.70 4.60 0.10 -0.16 0.36 P-Commitment 4.65 4.60 0.05 -0.30 0.41 O-Favorability 4.90 0.80 4.10 3.70 4.50*
O-Importance 4.60 4.55 0.05 -0.34 0.44 O-Commitment 4.70 4.50 0.20 -0.16 0.56 Relevance 3.80 2.85 0.95 0.05 1 .85 Instrumentality 2.90 4.10 -1 .20 -2.15 -0.25
Dimensions
Means
95 % Corfidence limits for Mean Difference Dimensions Swamp River Differencp 95 % Corfidence limits
for Mean Difference P-Favorability 0.50 4.60 4.10 -4.83 -3 .37*
P-Importance 4.35 4.90 -0.55 -0.99 -0.11
P-Commitment 4.40 4.95 -0.55 -1 .09 -0.01 O-Favorability 0.50 0.60 -0.10 -0.62 0.42
O-Importance 4.40 4.70 -0.30 -0.93 0.33 O-Commitment 4.35 4.40 -0.05 -0.90 0.80 Relevance 3 .60 4.65 -1.05 -1 .68 -0.42 Instrumentality 4.10 3.60 0.50 -0.30 1 .31
•This large difference was intended by the experimenter.
i ferences are as large as intended, putting the stories at opposite extremes.
Conclusions:
On the basis of the repeated trials with the interviewees for imp-roving on the content of the stories, and on the basis of the results obtained from the subject-judges, I concluded that the stories inteııd-ed for pairing could be regardinteııd-ed as practically equivalent and could be placed in the questionnaires.
APPENDIX B