|
DEVELOPMENT OF THE
DISRUPTIVE STUDENT BEHAVIOR SCALE
BY
WILLIAM L. MOSES
A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN
PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
MAY 1986
Copyright 1986
by
William L. Moses
To Billy
Who would have been proud
ACKNOWLEDGEMENTS
I wish to express appreciation to my parents for their
love, understanding, and help; my committee chairperson,
Dr. McDavis, for his counsel and encouragement; my
committee members, Dr. Ziller for his confidence in my
ability to work independently and Dr. Loesch for stepping
into the breach and contributing so much so quickly while
continuing his friendship and support; and my employers at
Pasco-Hernando Community College for their financial
support.
Special thanks go to my friend and colleague, Dr. Tom
Floyd, who listened for hours and encouraged for years,
and to my friends and lovers who were usually supportive,
sometimes distracting, and always worth it.
TABLE OF CONTENTS
ACKNOWLEDGMENTS . . . . . . . . .
LIST OF TABLES. . . . . . . . . .
ABSTRACT. . . . . . . . . . .
CHAPTER ONE INTRODUCTION . . .
Statement of the Problem . .
Purpose of the Study . . .
Need for the Study . . . .
Significance of the Study . .
Definition of Terms. . . .
Organization of the Study . .
* .. . .
*. . . .
.
CHAPTER TWO REVIEW OF LITERATURE . . . . .
Definition of Disruptive School Behavior (DSB)
Identification, Assessment, and Placement. .
Rating Scale Development . . . . . .
Psychometric Properties of Rating Scales . .
Uses of Behavior Rating Scales . . . .
Summary . . . . . . . . .
CHAPTER THREE METHODOLOGY. . . . . . . .
Research Questions . . . . . . . .
Construction of the DSBS . . . . . .
Validation of the DSBS . . . . . . .
Reliability of the DSBS. . . . . . .
Field Study . . . . . . . . .
Data Analyses . . . . . . . .
Validity . . . . . . . . .
Reliability . . . . . . . .
Limitations . . . . . . . . .
CHAPTER FOUR RESULTS AND DISCUSSION . . . .
Results . . . . . . . . . .
The Severity Factor . . . . . .
The Samples . . . . . . . .
Research Question One . . . . ..
Research Question Two . . . . . .
Research Question Three . . . . ..
Research Question Four. . . . . .
Summary . . . . . . . .
Discussion . . . . . . . . . .
S. vii
. .viii
S 21
S 38
S 57
S 66
S 68
S 70
S 70
S 71
. 77
. 80
. 80
. 85
. 85
S 87
S 87
S 89
. 89
S 89
. 92
S 93
S 98
S100
. 110
. 117
. 113
* *
* .
. .
TABLE OF CONTENTS CONTINUED
CHAPTER FIVE CONCLUSIONS, IMPLICATIONS, SUMMARY, AND
RECOMMENDATIONS. . . . . . .. . . 118
Conclusions. . . . . . . . . . 118
Implications . . ... . . . . . 119
Summary. . . . . . . . . ... . 121
Recommendations. . . . . . . . . 123
APPENDICES
A CONSTRUCT DEVELOPMENT STUDY . . . . .. 124
B BEHAVIORS COLLECTED FROM DISCIPLINARY RECORDS 130
C ORAL INSTRUCTIONS FOR THE EDITING STUDY . . 133
D ITEMS DEVELOPED FROM CONTENT VALIDATION STUDY 134
E INSTRUCTIONS FOR CONTENT VALIDATION STUDY . 137
F THE DISRUPTIVE STUDENT BEHAVIOR SCALE (DSBS). 138
G INSTRUCTIONS FOR SEVERITY FACTOR STUDY. .. 144
H SCORING TEMPLATE FOR THE DSBS . . . .. 145
I SUMMARY OF TEACHER RATINGS ON THE DSBS. . . 151
J PRESCRIPTIVE PROFILE WORKSHEET FOR THE DSBS . 152
K INSTRUCTIONS FOR THE PILOT STUDY. . . .. .158
L RATERS EVALUATION OF THE DSBS . . . .. 160
M ASSIGNMENT OF CONSTRUCTS BY ITEM NUMBER . . 161
REFERENCES. . . . . . . . . ... . . 162
BIOGRAPHICAL SKETCH . . . . . . . . 193
LIST OF TABLES
Table rage
1. Domains of Student Life Influenced by
the School Experience . . . . 76
2. Potential Adverse Consequences of DSBS
Behaviors . . . . . .... ... 90
3. Rating Form Distribution by Demographic
Categories--Norming Group . . . .. 94
4. Rating Form Distribution by Demographic
Categories--Disruptive Group. . . ... 95
5. Frequency of Observed DSBS Behaviors by
Constructs . . . .... . . .. 97
6. DSBS Constructs by Number. . . . . ... 99
7. Assignment of Proposed Scale Items to
Constructs for Content Validation 101
8. Follow-up Study for Assignment of Proposed
Scale Items to Constructs. . . . ... 103
9. Comparison of Disruptiveness Ratings by
Teachers and Non-teaching Personnel. . 106
10. DSBS Ratings and z-scores for the Disruptive
Group . . . . . . . . . 107
11. DSBS Ratings and z-scores for the
Norming Group. . . . . . . . 108
12. Test-Retest Correlations . . .. . 112
vii
Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial
Fulfillment of the Requirements for the
Degree of Doctor of Philosophy
DEVELOPMENT OF THE
DISRUPTIVE STUDENT BEHAVIOR SCALE
By
William L. Moses
May 1986
Chairperson: Roderick McDavis, Ph.D.
Major Department: Counselor Education
Disruptive behavior is currently seen by both educa-
tors and the public as a major problem in American
education. A procedure for quantitatively assessing
disruptive behavior in schools is required to show a need
for intervention programs and to select students for
placement in either special education or alternative
education programs. The purpose of this study was to
develop and validate an instrument, the Disruptive Student
Behavior Scale (DSBS). The DSBS is intended for use in
assessing quantitatively the disruptive school behaviors
viii
of middle and junior high students referred for placement
in special education and alternative education programs.
This study investigated the position that disruptive
school behavior (DSB) can best be described in terms of
its type, frequency, and severity. The use of teachers as
observers and raters of disruptive school behavior is
discussed. Using teacher-generated behavioral statements
from disciplinary referrals to better describe DSB is
suggested. A review of various rating scale development
procedures attempted by business, industry, and government
is summarized.
A set of 10 constructs was selected to define DSB.
Scale items were developed from referral statements on
disciplinary records in a junior high school. A severity
factor was incorporated into the scoring system so that
behaviors rated as more detrimental to the student were
given a higher DSBS rating.
The DSBS was field tested in a public middle school.
Students in a norming group and a criterion, or disrup-
tive, group were rated by their classroom teachers using
the DSBS. A norm for disruptive behavior for the target
school was calculated and a criterion for classifying a
student as disruptive was established.
Results indicated the DSBS could identify the crite-
rion group of disruptive students, classify individual
students as disruptive, and exclude non-disruptive
students from the disruptive group. A follow-up study
suggested the results were consistent over time for all
DSBS ratings except those at the lowest end of the scale.
CHAPTER ONE
INTRODUCTION
The public school system in the United States has
been assigned a major role in socializing and enculturating
American youth (Filipczak, 1978). The U.S. Supreme Court
in its 1954 landmark civil rights decision (Brown v. Board
of Education of Topeka, 74 S.Ct. 686, 691) described
education as "a principal instrument in awakening the
child to cultural values, in preparing him for later
professional training, and in helping him to adjust
normally to his environment." The materialistic emphasis
of American society and culture ordains that the educa-
tional institution at all levels be driven by the broadly
defined goal of career success for its graduates (Bell,
1984; DiPrete, 1981, p. 199; National Education Associa-
tion (NEA), 1975, p. 108).
Unfortunately, a significant number of students are
detoured from this goal when educators describe them as
displaying behaviors inappropriate to the school environ-
ment and not attributable to legally-defined mental or
emotional handicaps. Suspensions, expulsions, and assign-
ments to alternative programs are evidence of failure by
the educational system to effect students' adherence to
current social norms and culturally-specified behaviors.
The consequences to the schools for this failure include
loss of both funds and credibility, neither of which the
educational system has in sufficient quantity to squander.
Attempts to correct this failure to convey effec-
tively norms and behaviors have included both exceptional
child education and alternative schooling programs. The
Education for All Handicapped Children Act of 1975 (P.L.
94-142)(Department of Health, Education, and Welfare,
1977) effectively administered the coup de grace to
exceptional child education approaches in Florida by
failing to include a category appropriate to disruptive
behavior (Florida Department of Education, 1975, 1985).
Alternative programs frequently fail to provide for
selection and discharge criteria, rendering evaluation
virtually impossible (Pinellas County School District,
1982). A primary reason for failure to specify behavioral
criteria for alternative schooling programs is the lack of
appropriate instruments for quantifying disruptive
behavior (Salvia & Ysseldyke, 1981, pp. 8, 9).
Inadequacies of existing behavioral assessment
instruments include failure to provide for local norming,
inclusion of inappropriate items, omission of the severity
factor, and inadequacy of prescriptive information
(Mesinger, 1982). An instrument providing both a
theoretical and a pragmatic rationale for identifying
disruptive students is a requirement for reconsidering the
inclusion of this category in special education legisla-
tion and enhancing the credibility of alternative
education programs (Reeves, Perkins, & Hollon, 1978).
Statement of the Problem
Disruptive behavior in the public school system is
not a new phenomenon (Garibaldi, 1979). That it remains a
problem is emphasized by Robert J. Rubel in introducing a
collection of papers on crime and violence in public
schools:
The issue in the 1980's no longer centers on
whether or not violence in American schools is
serious; the issue no longer centers on whether
violence is increasing or decreasing; the issue
no longer centers on technical anomalies concern-
ing under- or over-reporting of incidents. In
the debate of the 1980's, the primary issue
before large proportions of our urban schools
(and sizeable numbers of our suburban and even
rural schools) revolves around the continued
viability of American education as it existed a
generation ago. (1980, p. 5)
The U. S. government has acknowledged the existence
of disruptive behavior by awarding federal grants for
alternative education pilot programs (Law Enforcement
Assistance Administration, 1979; Moses, 1976).
Included in definitions of disruptive school
behavior (DSB) are such varied activities as talking,
hitting, yelling (Mayer & Butterworth, 1979); defy-
ing rules and procedures (Walker, 1979); aggressive
behavior which interrupts the instructional program
(Foley, 1982); and conduct disorders (American Psychiatric
Association, 1980 pp. 45-50). Forness and Cantwell (1982)
and Forness, Sinclair, and Russell (1984) have identified
these categories as likely to be ineligible for special
education services under P.L. 94-142.
The U.S. government (Department of Health, Education,
and Welfare, 1977), in implementing P.L. 94-142, specif-
ically denied services to the "socially maladjusted."
Florida law provides essentially the same restrictions
(State Board of Education Rule 6A-6.3016), although Bower
(1982), whose research (Bower, 1958) formed the basis for
the P.L. 94-142 definition of emotionally disturbed,
called this exclusion "contradictory in intent and content
with . the research from which it came" (1982, p.
60).
The need for alternative education services for
disruptive students seems supported by reports of the
widespread existence of DSB. Individuals and institutions
reporting on the continuing crisis in school discipline
include the California Department of Education (1973),
the National Education Association (1975), the U.S.
Congress (Bayh, 1975; Tygart, 1980), the Michigan
Department of Education (Vergon & Williams, 1978), the
National Institute of Education (Feldhusen, 1978), Cross
and Kohl (1978), Duke (1978), the New York State United
Teachers (1979), and the National Education Association
(1980). The Safe School Study Report to Congress (National
Institute of Education, 1978) indicated 5,000 teacher
assaults per month occurred across the nation. The Gallup
Poll on Education (Gallup, 1984) continues to report lack
of student discipline as the number one concern of
Americans about the public school system.
In Florida, the Governor's Task Force on Disrupted
Youth (GTFDY) found 17,983 student-days lost to suspen-
sions over a 2-year period in the 10 school districts
studied (GTFDY, 1973, p. 11). An analysis of conduct code
violations in Duval County, Florida, schools for 1980-1981
revealed more than 33,000 violations resulting in 13,679
days lost from school (Moses, 1981).
The aversive consequences of chronic DSB for students
include lowered self-esteem and functioning level (Caliste,
1979); dropping out and underemployment (Grise, 1980; NEA,
1975; Safer, Heaton, & Parker, 1981); alienation
(Garbarino, 1980; Moyer & Motta, 1982); and criminal
activity (Edwards, Roundtree, Kent, & Parker, 1981;
Mitchell & Rosa, 1981). Likewise, from the perspective of
the school system DSB is undesirable, involving excessive
teacher attention (Rubel, 1977, Chap. 1), litigation
(Lufler, 1982), vandalism costs (Goldstein, Apter, &
Harootunian, 1984), teacher stress (Pettegrew & Wolf,
1982), and weakened public support (Amos, 1980). Conse-
quences for the community include criminal actions and
psychiatric referrals (Faretra, 1981). Levin (1972) esti-
mated the expense of inadequate education to be about 6
billion dollars a year (1972 dollars) for costs associated
with welfare and crime.
Researchers have identified the middle and junior
high school age student as particularly prone to behavior
disorder (Geiger & Turiel, 1983; Loeber, 1982; Nielsen &
Gerber, 1979; Quay, 1978). These studies suggest the
middle and junior high schools as a focus for identifying
and remediating disruptive school behavior. Unfortu-
nately, no adequate instruments are available specifically
for this population (Mesinger, 1982). Instruments
developed from clinical populations contain some items
irrelevant to the non-clinical population in the public
schools (Quay & Peterson, 1967). Instruments offered with
norms developed from research samples and no procedure for
developing local norms for disruptive behavior do not
consider the placement needs of local school districts
(Messick, 1980). Levels of disruptive behavior that can
be managed within the regular school environment vary
across settings because of differences in such factors as
facilities, experience of teachers and administrators, and
school board policies.
Current instruments fail to consider the widely
differing consequences of specific disruptive acts (Kane &
Bernardin, 1982). Some possible effects of this omission
may be to group together students whose behaviors differ
widely in their severity, to encourage conceptualizing all
disruptive behavior as equally deleterious, and to base
placement decisions on personal judgments about the seri-
ousness of a particular type of behavior. Neither does
any available instrument provide procedures for creating a
prescriptive profile of a student based on the authors'
conceptual model of disruptive school behavior (Salvia &
Ysseldyke, 1981). This failure may seriously limit the
interpretation and application of rating scale results.
Purpose of the Study
The purpose of this study was to develop and validate
an instrument, the Disruptive Student Behavior Scale
(DSBS). The DSBS would be used to assess quantitatively
the disruptive school behaviors of students referred for
placement in either special education or alternative
education programs.
Need for the Study
Salvia and Ysseldyke (1981, pp. 443, 444, 450) have
called for norm-referenced instruments to support
placement decisions, evaluate student progress, evaluate
programs, provide intervention suggestions, and help
parents understand their children's abilities in relation
to other students. Reeves et al. (1978) called for
reliable instruments to use in placing handicapped
children. Also, Camp (1981) notes that
there is very little current, objective,
research-based information in existence to help
identify specific student behavior problems
occurring in the schools. A need exists for
research of this nature to quantitatively
establish the actual, current situation with
regard to student discipline problems in the
public secondary schools. (p. 48)
Presumably, these calls for reliable and valid instruments
apply both to special education and alternative schooling
programs, as both to some degree remove the student from
mainstream classroom activities. However, the Florida law
(State Board of Education Rule 6A-6.3017) providing for
special education programs for the socially maladjusted
was repealed July 24, 1981.
"Educational alternative programs" were created in
Florida in 1978 (Florida Statute 230.2315) specifically to
reduce disruptive behavior and truancy. Florida Statute
229.565 provides for the evaluation of "procedures for
identification and placement of students in educational
alternative programs." As an example of practice, in 1982
the alternative education program in the Pinellas County
School District did not require quantitative behavioral
assessment prior to placement.
Studies, however, have identified problems in using
subjective criteria for alternative education placement.
Disagreements in ranking behaviors (Pisarra & Giblette,
1981), value systems (Messick, 1980), labels applied to
students (Leyser & Abrams, 1982), teaching experience
(Rubel, 1977, p.51), level of frustration (Walker &
Holland, 1979), race (Arnove & Strout, 1978; Bennett &
Harris, 1982; Florida DOE, 1983; Goldsmith, 1982;
Mesinger, 1982), sex (Bennett & Harris, 1982), and
socioeconomic status (Arnove & Strout, 1978; NEA, 1975)
are variables that may confound perceptions of disruptive
behavior.
One way to help neutralize these confounding vari-
ables is to use quantitative measures. A review of
current literature indicates that appropriate instruments
may not exist. After a major study of alternative educa-
tion programs, Mesinger (1982) was unable to recommend
even one instrument for use in selecting students.
Messick (1964, 1965, 1980) argued against applying to
local environments behavioral norms developed elsewhere.
Stott, Marston, and Neill (1975, p. 8), Wodarski and Pedi
(1978, p. 480), and Quay and Peterson (1975, 1979) advised
the setting of local norms. However, no instrument
located in this review provides a specific procedure for
determining local norms.
Another advantage of locally-developed norms is the
opportunity to compute the mean DSB level for individual
schools. Intervention program entry and exit criteria may
be defined by the deviation of an individual student's
mean DSB score from the school mean. This may provide the
type of quantitative assessment required by state (SBE
Rule 6A-6.3016) and federal (P.L. 94-142) law for special
education placement and may meet the need noted by
Mesinger (1982) for quantitative instruments to assist in
selecting students for alternative education programs.
A major need in intervention programs is prescriptive
information (Lovitt, 1967 p. 238; Spivack & Swift, 1977).
However, many instruments do not provide operationally-
defined items which are useful in the classroom. For
example, the Behavior Problem Checklist (Quay & Peterson,
1979) items used to identify conduct problem students
include "restlessness," "disruptiveness," and "irresponsi-
bility." These items originally were taken from the files
of a child guidance clinic (Quay, 1977).
Defining disruptive behavior on the dimensions of
type, frequency, and severity has received support from
numerous sources (American Psychiatric Association, 1980,
p. 45; Bernardin, LaShells, Smith, & Alvares, 1976; Camp,
1980, 1981; Grosek, 1979; Taylor, Warren, & Slocumb,
1979). Criticisms of assessment procedures not incorporat-
ing a severity factor have been made by Kane and Bernardin
(1982) and Pisarra and Giblette (1981). Nevertheless, no
instrument was located which specifically recommended
using a severity factor in assessing disruptive school
behavior.
An instrument which provides for quantifying DSB may
help to protect students from placement in school programs
according to inappropriate criteria. To be most effec-
tive, the instrument should include provisions for
establishing locally-determined placement norms, for
comparing with those norms the scores of individual
students, for providing prescriptive information, and for
systematically considering the type, frequency, and
severity of the disruptive behaviors.
Significance of the Study
This study investigated the theoretical position that
disruptive school behavior (DSB) can best be described
in terms of its type, frequency, and severity. Theoretical
considerations in the use of teachers as observers and
raters of disruptive school behavior were discussed.
The feasibility of using teacher-generated behavioral
statements from disciplinary referrals to better specify
the parameters of DSB was suggested. A review of various
rating scale development procedures attempted by business,
industry, and government were summarized.
The instrument developed by this study will initially
be most appropriate as a research tool for conducting
studies of DSB. The availability of a process for
establishing local norms for DSB may facilitate local
research studies in evaluating the effectiveness of
disciplinary measures, in-service training, and alterna-
tive education programs. This study will likely suggest
additional areas for other investigations.
The identification of disruptive students for inter-
ventions is not standardized. This instrument may assist
in establishing quantitative criteria for selection,
placement, and treatment of disruptive students. This, in
turn, may lead to recognition of DSB as a category for
exceptional student education funding.
A major premise in much of the literature concerning
DSB is the role of school personnel in exacerbating
disruptive behavior. It may be that an instrument which
provides a behavioral profile of the disruptive student
will suggest goals for in-service training programs.
Definition of Terms
For the purposes of this study, the following
definitions apply:
Alternative education program. An educational
procedure which provides intervention outside the regular
classroom for students exhibiting some predetermined level
of disruptive or disinterested school behavior.
Disruptive school behavior (DSB). Behavior that
disrupts the learning of self and/or others and is not
attributable to severe emotional disturbance or other
exceptional education categories.
Delinquent behavior. Behavior by persons under 18
years of age which violates laws and regulations pertain-
ing to them.
Exceptional child (student) education programs.
Programs which receive additional funding in order to
better serve the needs of students meeting governmental
guidelines for special assistance.
Experienced teachers. Full-time, regular classroom
teachers who have held that position at least two academic
years.
Expulsions. Removal from school for at least the
remainder of the school year.
Locally developed norms. Criteria for comparing an
individual student's DSB with the expected DSB of a
specific reference population in the local school or
community.
Maladaptive social behavior. Behavior not of organic
origin which would be judged by impartial observers to be
inappropriate for the social situation and which ulti-
mately results in aversive consequences for the person
exhibiting the behavior.
Method bias. The influence on ratings of the type of
rating method used.
Non-quantitative assessment. See Qualitative assessment.
Qualitative assessment. Evaluation based on individ-
ual opinion and lacking a systematic basis.
Quantitative assessment. The use of numbers in
describing behavior so that a higher number indicates a
higher level of the behavior.
Severity. A prediction, stated quantitatively, of
the potentially detrimental consequences a disruptive
behavior would likely have for a student.
Special education programs. See Exceptional child
education programs.
Suspensions. Temporary removal from the regular
educational program of a school, usually involving
exclusion from school facilities for a specified number of
days.
Organization of the Study
There are four remaining chapters in this
dissertation. Chapter Two will present a review of the
literature related to the development of an instrument to
assess disruptive school behavior (DSB). Specifically,
consideration will be given to disruptive behavior in the
schools, existing assessment methods, rating scale develop-
ment, the psychometric properties of rating scales, and
the possible uses of results from a disruptive behavior
'rating scale.
Chapter Three will present the methodology employed
in the development, validation, and field testing of the
Disruptive Student Behavior Scale (DSBS). Included are
the research questions, information on the population,
procedures used in developing the scale, pilot testing,
data analyses, and possible limitations of the study.
Chapter Four will present the results of this study,
including the data and the information inferred from the
data. An explanation of the results will be given and
they will be related to past research.
Chapter Five will include conclusions from this
study, along with implications for theory, research,
practice, and training. A summary of the entire study
will be presented, followed by recommendations for addi-
tional research.
CHAPTER TWO
REVIEW OF LITERATURE
This study requires an investigation of the history
and current status of attempts to define disruptive
behavior in public schools; identification, assessment,
and placement efforts directed toward disruptive students;
rating scale development procedures; research into the
psychometric properties of rating scales; and the use by
schools of results obtained from rating scales. Accord-
ingly, this chapter will review research and opinion
covering both theoretical and applied considerations
relating to these topics.
Definition of Disruptive School Behavior (DSB)
According to Camp (1981), the major issue in student
discipline in the secondary schools is how to describe
quantitatively the kinds of disruptive behavior currently
occurring. Summarizing a 1978 survey of state directors
of special education, Hirshoren and Heller (1979) reported
that while individual states define emotional disturbance
consistently, there is considerable variation in the kinds
of children so identified. That is, children meeting
program criteria in one state appeared to be excluded in
another. Much has been written in an attempt to resolve
this situation. A review of the literature suggests the
emergence of five discrete perspectives: (a) empirical,
(b) clinical, (c) conceptual, (d) educational, and (e)
school.
The empirical approach of applying factor analysis
(Cattell, 1978; Gorsuch, 1974) to a variety of items has
resulted in the identification of some common behaviors
associated with disruptive school behavior and has contri-
buted to defining DSB (Achenbach, 1978; Achenbach &
Edelbrock, 1978; Edelbrock, 1979; Peterson, 1961; Quay,
1964, 1978; Quay & Peterson, 1967). However, researchers
utilizing the empirical approach have included a broad
range of behaviors, including many which identify delin-
quency and personality disorders (Freemont & Wallbrown,
1979), and so the scales developed from these studies have
limited application for school personnel in defining the
specific category of DSB.
The classification of disorders contained in the
Diagnostic and Statistical Manual of Mental Disorders,
3/e. (DSM-III) (American Psychiatric Association, 1980)
and research studies incorporating these classifications
and descriptions exemplify clinical efforts to define
disruptive school behavior. Hewett and Forness (1982)
pointed to the necessity of finding a common frame of
reference between educational and psychiatric diagnoses in
order for school personnel to accurately interpret
clinical reports. Forness and Cantwell (1982) concluded
that the respective diagnostic systems of psychiatry and
special education remain dissimilar. Likewise, other
studies (Loeber, 1982; Werry, Methuen, Fitzpatrick, &
Dixon, 1983) failed to find support for the use of
psychiatric diagnoses to assign students to special
education programs.
The conceptual approach utilizes experience,
research, and opinion in formulating descriptions of what
is usually referred to in this perspective as "problem
behavior" (Jessor & Jessor, 1977, p. 4). Cullinan (1975),
Howell (1978), and Richard Jessor (1982) are among those
applying a psychosocial conceptualization of problem
behavior to the study of adolescent behavior. Neverthe-
less, while the conceptual perspective gives support to
the notion of comparing the behavior of an individual
student with the behavior of peers before declaring the
student to be deviant, this perspective fails to provide
specific criteria for making such a comparison.
The educational perspective includes the definitions
contained in federal and state statutes, guidelines
proposed by governmental agencies, and district codes of
student conduct. In 1977, the U.S. government, without
defining the term, specifically excluded the socially
maladjusted student from receiving exceptional child
education services under P.L. 94-142. The term "socially
maladjusted" is not defined in the latest Florida guide-
lines for providing special education for exceptional
students (Florida DOE, 1985). The U.S. Bureau of Educa-
tion for the Handicapped has sponsored the compilation of
a manual on behavior disorders (Yard, 1977). However,
these items are too general for use in a quantitative
instrument.
Codes of student conduct contain lists of behaviors
for which punishment may be administered. Offenses listed
in the codes may be violations of either school rules
(e.g., inappropriate display of affection) (Duval County
Public Schools, 1980) or of law (e.g., vandalism) (Pinellas
County Schools, 1983). While these offenses must be
considered in defining disruptive school behavior, they
exclude many of the disruptive behaviors frequently
occurring within the classroom. Federal, state, and local
guidelines seem insufficient for operationally defining
DSB specifically enough to be useful in a selection
instrument.
The school perspective focuses on the interactions of
students, teachers, and administrators within schools.
Disruptive school behavior is seen as a product of these
interactions. H. M. Walker, author of The Walker Problem
Behavior Identification Checklist (1970), described the
acting-out child as one who usually defies rules and
ignores classroom procedures, is difficult to manage,
avoids failure by attempting little academic work, and
alienates teachers and other students by behaving
aversively.
Specific behaviors often include hitting, yelling,
leaving seat, arguing, having temper tantrums, and provok-
ing others and often lead to confrontations. These
confrontations may be verbal, physical, or both. Acting-
out behavior may occur in the classroom, in nonclassroom
areas, or both. Walker (1970) proposed that acting-out
children are differentiated from other students by the
frequency, or quantity, of these behaviors, not by the
type of behaviors. Thus, a measuring instrument must
provide for a frequency component.
Camp (1981) explored the types of behavior considered
to be disciplinary problems, the perceived degree of
severity of these behaviors, and the frequency with which
these behaviors were observed. Camp found that the types
of behaviors rated most serious were rarely observed and
concluded that the most serious problem may be the
frequent, though mild, behaviors that undermine student
and teacher morale. A study of 21 secondary school
administrators' attitudes toward aggressive behavior
suggested that suspensions were awarded according to the
administrators' attitudes toward the referred behavior,
rather than according to a consistent standard for the
school district (Pisarra & Giblette, 1981).
An evaluation of literature of the school perspective
suggests that DSB can be defined in terms which students,
teachers, and administrators understand; that the three
factors of type, severity, and frequency need to be
considered; and that measures of DSB need to be standard-
ized. In this section five perspectives for defining
disruptive school behavior were presented. Each perspec-
tive offers some assistance in differentiating this
category from other behavioral categories. There appears
to be support for an instrument which operationally
defines types of behaviors occurring throughout the school
environment, assigns a quantity to each descriptive item
based on the perceived frequency of occurrence and sever-
ity, and provides for comparing the score of an individual
student to a predetermined norm for that environment.
Identification, Assessment, and Placement
"Measurement is the construction of a model of some
property of the world" (Fraser, 1980, p. 27) and in
education this property is often the behavior of a
student. One role of the model provided by a measure is
to give accurate prescriptive information for planning
interventions with students (Forness, 1983). Several
studies have suggested this is being performed
inadequately (Greenwood, Walker, & Hops, 1977; Schenck,
1980; Sinclair, 1980; Sinclair & Kheifets, 1982; Spivack &
Swift, 1973; Strain, Cooke, & Apolloni, 1976).
Fraser (1980) acknowledged that psychological mea-
surement has been regarded as being quantitatively and
qualitatively of a lower order than physical measurement.
To achieve improvement, Ysseldyke and Marston (1982) have
argued for the use of direct observations of target behav-
iors by either teachers or trained observers. However,
Jones, Reid, and Patterson (1975) found observer reli-
ability varied inversely with the complexity of the
behaviors being observed.
Attempts to improve the validity of observations have
included such sophisticated approaches as Multidimensional
Scaling (MDS) (Torgerson, 1958). Sanson-Fisher and
Mulligan (1977), using adolescent student models, found
only marginal improvement for this technique over ratings
by classroom teachers. A comparison of a computer-driven
program for selecting behavioral/emotional disorders with
two expert psychologists' selections indicated no mean-
ingful differences existed (McDermott & Hale, 1982).
Weinrott (1979) summarized studies that indicated global
ratings could be significantly influenced by expectations,
while post hoc ratings of the same children by the same
raters when recorded on an instrument accurately reflected
discrete behavioral events. Gaynor and Gaynor (1976)
argued for instruments written to define behaviors so they
may be described quantitatively by teachers.
Beltramini (1982) suggested that scale-item content
is more important than other variables in obtaining reli-
able and valid results. A review by Albaum, Best, and
Hawkins (1981) of measurement literature found evidence to
support the use of from five to seven categories on Likert-
type scales, with no significant losses in reliability,
validity, or discrimination when compared with instruments
using more intervals. Fewer intervals sometimes resulted
in a loss of discriminative power and validity. It
appears that teachers using instruments which operation-
ally describe disruptive behaviors can be effective post
hoc raters and are able to provide reliable and valid
identification of disruptive school behavior (Edelbrock,
1979; Gresham, 1982; O'Leary & Johnson, 1979).
A review of current assessment techniques suggests
the emergence of a quantitative/qualitative dichotomy,
which will now be explored. In two reviews (Spivack &
Swift, 1973, 1977) of instruments for measuring secondary
school classroom behaviors no instrument was located which
limits its focus to disruptive school behavior, uses only
behaviorally-stated items, and provides for calculating
local norms. Descriptions follow of representative
instruments currently in use.
The Behavior Problem Checklist (BPC) (Quay & Peterson
1967, 1975, 1979) is a 55-item scale of behavioral traits
developed from a review of clinical records of kinder-
garten through eighth grade students referred for
psychiatric treatment (Quay, 1977). The items were
assigned by factor analysis to four scales plus a grouping
suggestive of psychosis. Epstein, Cullinan, and Rosemier
(1983, p. 172) and Gresham (1982, p. 137) reported that
the BPC is one of the behavior rating scales most widely
used in school studies.
The BPC has been used extensively both as a research
device (Eaves, 1975; Jacob, Grounds, & Haley, 1982;
Kelley, 1981; Touliatos & Lindholm, 1981) and in selecting
students for interventions (Algozzine, 1977; Balow, 1979;
R. Bower, 1969; Gerard, 1970; Ingram, Gerard, Quay, &
Levinson, 1970; McCarthy & Paraskevapoulas, 1969). Jacob
et al. (1982) reported that reviews of studies utilizing
the BPC suggested reliability and validity issues in need
of further study. The inability of the BPC to provide
other than broadband classifications has been noted
(Achenbach & Edelbrock, 1978).
Comprehensive normative data are not available for
the BPC for adolescents (Kelley, 1981). In an investiga-
tion of the effects of race on BPC ratings, Eaves (1975)
found that white teachers consistently rated black
students higher than white students on three of the
subscales. Black teachers showed no such bias. Eaves
(1975) concluded this bias could have a major effect on
the reported norms for the BPC. Touliatos and Lindholm
(1981) found that grade level, sex, and social class had a
significant effect on BPC ratings. However, differences
between schools and teachers contributed more variance in
the BPC ratings than grade, sex, and social class.
Touliatos and Lindholm (1981) suggested that Quay and
Peterson's (1967) recommendations be followed and
individual assessment be based on norms calculated for
particular schools and individual teachers.
Spivack and Swift (1973) concluded that the BPC was a
reasonably reliable measurement tool. Potential users
were cautioned, however, that most items are not specifi-
cally observable, but more like labels which imply
behaviors and designate traits. Likewise, Stott (1971,
p. 232) cited certain BPC items as requiring a teacher to
make inferences about students' feelings (e.g., "feelings
of inferiority"), being vague or ambiguous (e.g., "oddness,
bizarre behavior"), and relating to behaviors unobservable
by a teacher (e.g., "stays out late at night," "bed
wetting"). This review has identified several areas of
the BPC for which additional research has been suggested.
The Behavior Rating Profile (BRP) (Brown & Hammill,
1978) is composed of five rating scales and a sociogram.
Three of the scales (60 items) are completed by the target
student, one (30 items) by the teacher, and one (30 items)
by parents. The sociogram is a peer nominating techni-
que. The student scales provide self-ratings of behaviors
at home, at school, and with peers.
The BRP is based on an ecological approach which,
according to the authors, recognizes that students'
behaviors are dependent on the settings in which they
occur. Its purposes are the identification of students
with behavior problems and the differentiations among
learning disabled, emotionally disturbed, and behaviorally
disordered students in grades 1-12. Each of the six
measures is described as independent and individually
normed, allowing any scale to be used alone or in conjunc-
tion with any of the others.
The BRP manual (Brown & Hammill, 1978) reports
internal consistency reliability coefficients exceeding
.80. Concurrent validity was investigated by correlating
the BRP with measures obtained from other rating scales.
Adequate construct and content validity also are reported
by the authors. Norms are provided using scale scores
with means of 10 and standard deviations of 3, with scores
from 7 to 13 considered to be in the normal range.
One study (Reisberg, Fudell, & Hudson, 1982) of
behavior disordered students indicated that regular
classroom teachers gave higher ratings than special
educators (X=8.85 vs. X=6.87). Thus, norms may vary
according to the type of respondent (e.g., regular teacher
or special education teacher). Also, students' self-
ratings were inflated relative to other respondents'
ratings. Other investigators have noted problems
associated with attempts at multiple and self-ratings.
Lessing and her associates (Lessing & Clarke, 1982;
Lessing, Williams, & Gil, 1982; Lessing, Williams, &
Revelle, 1981) have reported on their unsuccessful
attempts to develop parallel checklists for use by
parents, teachers, and clinicians in psychiatric diag-
noses. Lobitz and Johnson (1975) found low correlations
between parent ratings and observed behaviors. Variables
confounding self-ratings include halo effect (Holzbach,
1978), social desirability (Dunnett, Koun, & Barber, 1981;
Seidman,, Rappaport, Kramer, Linney, Herzberger, & Alden,
1979), and lack of self-knowledge (Beitchman & Raman,
1979).
Ledingham, Younger, Schwartzman, and Bergeron (1982)
investigated teacher, peer, and self-ratings of 801
elementary school students. Self-ratings yielded the
lowest ratings for deviant behavior, aggression, and
withdrawal and the highest ratings for likability. Accu-
racy of self-evaluation has been found to be positively
correlated with high intelligence, high achievement
status, and internal locus of control, characteristics not
usually associated with DSB (Dunnett et al., 1981).
Reported research using the Behavior Rating Profile is
sparse. Additional verification of the assumptions of
equivalency of norms within respondent categories and the
validity of the self-report scales seems indicated.
The Bristol Social Adjustment Guides, 5/e.(BSAG)
(Stott, 1972) consist of 110 behaviorally-stated items
from which teachers select those descriptive of a
student's behavior in the month prior to the rating. The
items were originally developed in 1955 from clinical
observations of children aged 6 to 14 and modified by
classroom teachers (Stott & Sykes, 1956). A primary goal
was to incorporate context into the behavioral descrip-
tions (Stott, 1971).
The BSAG has been used extensively in clinical and
research studies (Davis, Butler, & Goldstein, 1972;
McDermott, 1980; Stott, 1978; Stott & Wilson, 1977).
Reliability and validity data were obtained through
extensive research (Stott et al. 1975) but are not
reported in a manner that is easily abstracted. Normative
data are available only for elementary school populations
(Stott, 1972). More recent research (McDermott, 1980,
1981; McDermott & Hale, 1982) has questioned the
specificity of the core syndromes of the BSAG and called
for further investigation of construct and predictive
validities (Hale & Zuckerman, 1981). At present, it
appears that not all of the core syndromes of the BSAG
have the specificity required in an instrument to be used
in educational placement.
The Hahneman High School Behavior Rating Scale (HHSB)
is a 13-factor, 45-item scale published in 1971 (Spivack &
Swift, 1971). The HHSB items were developed from observa-
tions of actual classroom behaviors, operationally stated
in educational terms. The items cover both academic and
interpersonal issues and can be rated by teachers in the
classroom. The intent is to provide prescriptive informa-
tion (Spivack & Swift, 1977). The factor scores for each
student are found by adding the raw scores for the three
or four items comprising each factor. These scores are
then combined into a profile, which is used to classify
students on the basis of their ability to adapt to total
classroom demands.
According to the authors (Spivack & Swift, 1973),
validity studies suggest consistent and significant
relationships between factor scores and academic grades.
No data are available on test-retest or interrater reli-
ability (Spivack & Swift, 1973). Norms are available
separately for suburban and urban samples. The HHSB is
limited as a selection device for special education pro-
grams by lack of reliability data, use of only three or
four items per factor, and overlapping among profile
descriptions.
The Behavior Evaluation Scale (BES) (McCarney, Leigh,
& Cornbleet, 1983) is a 52-item rating scale for use by
school personnel. Each item is assigned to a subscale
associated with one of the five characteristics of the
Bower (1958) definition of behavior disorders used in
Public Law 94-142. The BES was developed to aid in
diagnosis, placement, and program planning under federal
guidelines. Since federal criteria specifically exclude
the "socially maladjusted" student, the BES is inappropri-
ate for assessing DSB.
The Portland Problem Behavior Checklist (PPBC)
(Waksman & Loveland, 1980) was developed to aid in
assessment, evaluation, and intervention planning for
school children. The 29 items cover teacher-rated
behaviors for grade levels K-12. Norms are not avail-
able. Items are very generally stated (e.g., aggressive-
physical, destructive) and are rated on a scale of 0 (no
problem) to 5 (severe). It is not clear if this is a
rating of frequency of behavior or severity of the
consequences of the behavior. These features of the PPBC
would seem to limit the preciseness and reduce the confi-
dence level of quantitative scores intended to support
evaluation and placement for professional services.
The Pupil Classroom Behavior Scale (PCBS) (Dayton,
1967) is a 24-item, teacher-administered rating scale
intended to measure the effectiveness of special education
services for students displaying inappropriate classroom
behaviors. Most items are behaviorally stated and yield a
profile of three factors, achievement orientation, socio-
academic creativity, and socio-cooperativeness. Dayton
(1967) suggested using the scales for research on groups
rather than to describe individual students. Norms are
not available. Spivack and Swift (1973) concluded that
the PCBS is flawed by having overlapping items in the
factors and lacking data to support a relationship between
scale scores and emotional adjustment.
The 36-item Conners Teachers' Rating Scale (CTRS)
(Conners, 1969) has been used primarily in clinical diagno-
sis of children, particularly in the area of hyperactivity
(Goyette, Conners, & Ulrich, 1978). It does, however,
cover a wide range of school problem behaviors (Roberts,
Milich, Loney, & Caputo, 1981). There appears to be a
high intercorrelation between the Conduct Problem and
Hyperactivity subscales, limiting the usefulness of the
CTRS in identifying DSB.
The Brief Behavior Rating Scale (BBRS) (Kahn &
Ribner, 1982) was developed from the Devereux series of
rating scales (Spivack, Haimes, & Spotts, 1967). A cross-
validation study (Kahn & Ribner, 1982) reported that 61%
of a socially maladjusted group and 27% of an emotionally
handicapped group were correctly identified. These
results suggest that additional development is needed
to obtain support for the discriminant validity of the
BBRS.
Some of the most complete research in instrument
development has been conducted in attempts to improve the
diagnosis of clinical populations in the school environ-
ment. Although these efforts are not directly comparable
to the intent of the present study, six instruments having
potential interest to researchers working in the school
setting will be summarized.
The Child Behavior Check List (CBCL) (Achenbach,
1978) contains 118 behavior problem items and 20 social
competence items. Parallel forms exist for parents and
teachers. A review by Achenbach and Edelbrock (1978) of
empirical attempts to derive syndromes of child behavior
problems concluded with the recommendation that these
efforts be linked to the existing mental health system.
Recent efforts by these researchers and their associates
(Edelbrock & Achenbach, 1980; Reed & Edelbrock, 1983)
continue to pursue this objective. At present the
applicability of this instrument for educational measure-
ment is limited.
The role of parent observations in describing chil-
dren's behavior is formalized in the Louisville Behavior
Check Lists (Miller, 1967, 1980). A study (Tarte, Vernon,
Luke, & Clark, 1982) confirmed the validity of parent
observations of clinical symptoms in their children.
The items require inferences and judgments by raters.
Eight subscales were created through factor analysis and
although several appear to relate to school activities
(e.g., hyperactivity, antisocial), the content of
individual items comprising the subscales renders them
only marginally useful for school assessments.
The Children's Behaviour Questionnaire (Rutter, 1967)
was developed for teachers' use in screening for psychi-
atric assessment large numbers of school children. Many
of the 26 items are vaguely stated and some appear to
require inferences by the rater. The two subscales are
labeled neurotic and antisocial, terms which lack direct
application to the school setting.
The Devereux Adolescent Behavior Rating Scale
(Spivack et al., 1967) was developed to measure behavior
requiring professional intervention. The subscales are
oriented to clinical diagnosis and offer little specific
information for use in placement decisions.
The Pupil Behavior Inventory: 7-12 Grades (Vinter,
Sarri, Vorwaller, & Schafer, 1966) is a 34-item, teacher-
administered rating scale intended to furnish information
on students referred for agency treatment. Behavioral
items were collected from teachers, screened and factor-
analyzed, and grouped into five factors. Lack of data on
reliability, validity, and norms suggests caution in
using this instrument to select students for special
services (Spivack & Swift, 1973).
The Mooney Problem Check List (MPCL) (Mooney, 1942),
has been widely used by counselors to identify problems of
individuals seeking counseling or to explore the problem
profile of a group of students (Sundberg, 1961). However,
two studies (Joshi, 1964; Stewart & Deiker, 1976) of the
underlying factors of the MPCL scales have identified only
a single general factor. The MPCL may be further limited
by utilizing items generated from problems mentioned by
high school students in 1942.
Several instruments designed for other populations
include behaviors often used in descriptions of disruptive
school behavior. The Adolescent Behavioral Classification
Project instrument (Dreger, 1980) was developed for
assessing problems of institutionalized adolescents. An
analysis of the first-order factors indicates some common-
alities with both the Hahnemann High School Behavior
Rating Scale (Spivack & Swift, 1977) and Achenbach and
Edelbrock's (1978) syndromes, but many are couched in
clinical terms that have little or no relevance to the
classroom setting.
Ostrov and associates (Ostrov, Marohn, Offer, Curtiss,
& Feczko, 1980) developed and validated the Adolescent
Antisocial Behavior Check List (AABCL) for delinquents
housed in an institutional treatment setting. The authors
called for modification of the instrument for use in other
settings; however, extensive rewriting of items would seem
to be required.
The Jesness Inventory (Jesness, 1972) was created to
measure attitude change in youthful offenders undergoing
treatment. One study (Graham, 1981) found the Jesness
Inventory did not have the power to discriminate between
non-adjudicated and normal populations and thus would not
be useful in a school setting. The Jesness Inventory
appears best suited for research (Buros, 1978, pp.
876-878).
The Jesness Behavior Checklist (JBC)(Jesness, 1970)
is also a measure of delinquent behavior. The reliability
and validity of this instrument have been questioned and
the JBC is recommended only for research purposes (Buros,
1978, pp. 873-876).
Non-quantitative assessment often uses nonsystematic
observations to provide the information from which judg-
ments will be made. Judgments about individuals are
required in all assessment. Inaccurate, biased, or sub-
jective judgments can be misleading and harmful (Salvia &
Ysseldyke, 1981). The Russell Sage Foundation Conference
Guidelines (Goslin, 1969) and the 1974 Family Educational
Rights and Privacy Act (P.L. 93-380--the Buckley amend-
ment) established guidelines for the proper collection,
maintenance, and dissemination of data concerning students.
For data to be used in making judgments, it must be
verified. For standardized tests, this verification is
implicit in the psychometric qualities of the instrument.
For observational data, verification requires confirmation
by persons other than the original observers (Salvia &
Ysseldyke, 1981). When the observation is nonsystematic,
verification may be difficult to establish and support and
the assessment and resulting evaluation may be open to
challenge.
After a classroom teacher nominates a child for
evaluation for exceptional child education services, that
teacher's observation is verified by required legal proce-
dures (P.L. 94-142). There may be no such procedures for
other interventions. The Duval County, Florida, School
District has used teacher and principal nominations as the
criteria for admittance and dismissal from a program to
intervene with students displaying inappropriate social
behaviors (Duval County Public Schools, 1980). Short-term
suspensions in many school districts do not require hear-
ings and are based solely on a judgment by the school
principal (Lines, 1972; Pisarra & Giblette, 1981).
Subjective assessment practices such as these may
allow extraneous variables to influence judgments
(Poulton, 1976). Four such variables are bias, the influ-
ence of observer expectations, inaccurate perceptions,
and vagueness of the criteria for intervention.
Pupil characteristics were found by Ysseldyke and
Marston (1982) to influence rater bias. Variables
contributing to bias include perceived physical attrac-
tiveness (Ross & Salvia, 1975); sex, socioeconomic status,
and reason for referral (Matusek & Oakland, 1979;
Ysseldyke & Algozzine, 1982; Ysseldyke, Algozzine, Regan,
& McGue, 1979, 1981); race (Florida Department of
Education Report on Public Schools, 1983; Sikes, 1975);
type of behavior displayed by the student (Algozzine,
1980); and the theoretical orientation of the observer
(Messick, 1980; Salvia & Ysseldyke, 1981).
Erickson (1974) and Shuller and McNamara (1976) found
naive observers' reports coincided with experimenter-
induced expectancies about problem behavior. After
observing decisions made by educators, Weinrott (1979);
Ysseldyke, Algozzine, and Richey (1982); and Algozzine and
Ysseldyke (1981) speculated that these judgments were
influenced by an expectancy factor created by the
situation itself. A more direct measure of expectation
was reported on by Green and Brydon (1975). They found
teachers' attitudes were much more favorable toward
middle-income children than low-income children and that
43% of teachers' comments about black children were
negative as opposed to 17% of comments about white
children.
I
Dunlap and Dillard (1980) investigated 164 school
principals' perceptions of the factors indicative of
emotional disturbance in children. The factor least
chosen by the principals was the one considered by the
researchers most predictive of emotional disturbance.
The vagueness of criteria for suspension in one
school district was investigated by Pisarra and Giblette
(1981). They found the criterion to be improper conduct,
which was not further defined. The researchers concluded
that a student reported for fighting would be suspended,
possibly suspended, or not suspended depending on the
individual administrator who had jurisdiction.
A few of the possible sources of error in nonsystem-
atic observation leading to inaccurate, biased, or
subjective judgments have been presented to suggest their
ubiquitous nature and the necessity of providing for
systematic observations in judgments leading to educa-
tional placement decisions.
Rating Scale Development
Designing a rating scale requires addressing four
major issues: (a) what to measure (parameters), (b) how
to measure (item content and format), (c) how to record
(response format), and (d) how to interpret the results
(statistical analysis). Literature pertaining to these
issues will be reviewed in this section.
In a frequently cited longitudinal study of deviant
behavior, Robins (1966) found the variables of type of
behavior, frequency of occurrences, and severity of
consequences to be indicators of future behavior pat-
terns. More recent studies supporting these criteria
include those of Kohn, Koretzky, and Haft (1979); Camp
(1980); Forness and Cantwell (1982); Gresham (1982);
Loeber (1982); and a United States Department of Justice
report (1982, p. 1).
The types of behavior to be measured by a rating
scale are determined by its authorss, who must consider
content, sources, format, number, and order of presenta-
tion of the items to be included. Halo effects, or the
tendency to rate individuals holistically (Thorndike,
1920, p. 25; Willingham & Jones, 1958), were found by
Cooper (1981; 1983) to be reduced by having more specific
item content. Kreitler and Kreitler (1981) demonstrated
that items deemed irrelevant by raters tended to be scored
neutrally, thus limiting the derived information. Never-
theless, scales for rating disruptive behavior sometimes
include prosocial behavior content (Miller, 1980).
However, Deno (1979) suggested that to observe non-
disruptive behavior ignores the purpose of these ratings,
i.e., to determine whether inappropriate behaviors are
actually excessive. Schriesheim and Hill (1981) mixed
positive and negative statements on a questionnaire and
concluded that the effect was to reduce response validity.
Many scales do limit their items to behaviors that focus
on problem behavior (DiPrete, 1981; Duke, 1978; Governor's
Task Force on Disrupted Youth, 1974; Spivack & Swift, 1966;
Walker, 1979, p. 55), although not necessarily school
problems. Camp (1980) suggested that only school problems
directly observable by teachers and/or administrators be
included in scales for rating disruptive school behavior.
Logically, items taken from the setting in which the
ratings will be made best meet the criteria for relevant
content. Smith and Kendall (1963) used this premise in
devising Behavioral Expectation Scales (BES). Numerous
examples exist of the application of this premise in
education (Brown & Hammill, 1978; Camp, 1980; Duval County
School Board, 1979; Ross, Lacey, & Parton, 1965; Sherry,
1979; Spivack & Swift, 1977; Stott et al., 1975), mental
health (Kaufman, Swan, & Wood, 1979; Kohn et al., 1979;
Lachar & Gdowski, 1979; Miller, 1980) and industry (Vance,
Kuhnert, & Farr, 1978).
Item format refers to the various forms used in
presenting the information to which the rater is asked to
respond. It is often related to response format, which
refers to the methods of collecting information from the
raters. Response format literature will be presented in
the section covering the frequency characteristic.
Four types of item formats are currently in use in
behavioral rating scales. Behavioral Observation Scales
(BOS) describe the target behavior in specific terms that
require direct observation at the time the rating is made
(Latham & Wexley, 1977). Behaviorally Anchored Rating
Scales (BARS) provide a specific description of a behavior
for each successive rating point (anchor) of an item and
assess cumulative behavior over some time period (Smith &
Kendall, 1963). The Mixed Standard Scale (MSS) uses sev-
eral scales, with three levels of behavioral description
for each trait to be measured, and randomizes the order of
presentation (Blanz & Ghiselli, 1972).
Summated rating scales (Edwards, 1957), referred to
as Likert scales (LT) (1932) or graphic rating scales
(Waters, Reardon, & Edwards, 1982), present for each item
one statement that may be specific or general. Likert
scales have been used with both direct and deferred obser-
vation. BOS scales are developed using summated rating
procedures (Likert, 1932), while BARS and MSS use the
Thurstone (Thurstone & Chave, 1929) scale development
process (Bruvold, 1969).
Conflicting conclusions have resulted from numerous
investigations into the advantages and disadvantages of
these scale formats. Fay and Latham (1982) found BOS to
be superior to BARS in rating video-taped behavior during
job interviews. However, Murphy, Martin, and Garcia
(1982) questioned the theoretical basis for BOS and found
evidence to suggest that BOS tapped recall for behavior
traits as well as immediate observation. Several studies
(Hom, DeNisi, Kinicki, & Bannister, 1982; Ivancevich,
1980; Keaveny & McGann, 1975; Lee, Malone, & Greco, 1981)
failed to find significant advantages for the BARS format
over summated rating scales or other alternative methods
(Jacobs, Kafry, & Zedeck, 1980; Kingstrom & Bass, 1981;
Schwab, Heneman, & DeCotiis, 1975).
In opposition to MSS theory, Finley, Osburn, Dubin,
and Jeanneret (1977) found evidence to suggest that an
obvious scale format may be superior to a hidden contin-
uum. Dickinson and Zellinger (1980) compared MSS, BARS,
and LT formats and found MSS produced less method bias,
BARS produced as much discriminant validity as MSS and
provided the best feedback to rates, and LT scales were
easiest to understand and use. When Bruvold (1969) tested
the application of summated scales (Likert, 1932) and
successive interval scales (Edwards & Thurstone, 1952) to
the same data set, no significant differences were found
between the two scaling methods. According to Bernardin
and Smith (1981), one explanation may be that scale
constructors have deviated from the original procedures
(Smith & Kendall, 1963) in developing BARS instruments.
In addition to the Thurstone and Likert scaling
procedures, a third method is available. According to
Edwards (1957, p. 172), a Guttman (1944, 1945, 1947a,
1947b), or cumulative scale, requires that the construct
to be measured be unidimensional. Since disruptive school
behavior consists of many discrete behaviors, a Guttman
scale is not suitable for the instrument developed in this
study. At present, it appears that no item format is
superior enough to warrant relinquishing the clarity of
understanding and ease of use (Dickinson & Zellinger, 1980)
of the Likert scale, which presents one descriptive item
at a time to which the rater assigns a quantitative value
from a given range of values.
In determining the number of items to include in a
rating scale, some researchers (Quay & Peterson, 1967,
1979; Spivack & Swift, 1971; Stott, 1972) have relied on
factor analysis, using an arbitrarily chosen factor score
as the cut-off score. Edwards (1957) suggested an intui-
tive approach, utilizing 20-25 items that discriminate
between the groups at the extremes of the scale. A
comprehensive study (Achenbach & Edelbrock, 1978) of 18
rating scales found the range of items to be from 36 to
287 (median = 68 items; mean = 90.4 items). Of the 6
scales intended for use by teachers, 4 contained fewer
than 50 items and 2 between 50 and 100 items.
In a study of preferred scale length, Meredith (1981)
found half of the respondents preferred from 20 to 40
items, with 25 the median preferred length. In another
study, Meredith (1975) found a 52-item scale was judged
too long. Seidman and his associates (Seidman et al.,
1979) concluded their 46-item Teacher Behavior Description
Form was too cumbersome and reduced it to 23 items. While
item complexity is probably a factor (Meredith, 1981),
this review suggests a scale using no more than 40 items
would probably be acceptable to most teachers.
The ordering of items within a scale has been
suggested as a possible source of leniency error, halo
effect, and impaired discriminant validity (Blanz &
Ghiselli, 1972). Schriesheim and DeNisi (1980) and
Schriesheim (1981b) found that grouping according to
constructs rather than randomizing questionnaire items
resulted in impaired discriminant validity. Increased
leniency response bias was also found when items were
grouped (Schriesheim, 1981a).
Dickinson and Zellinger (1980) concluded that a
randomized scale contributed as much discriminant validity
as an ordered scale while displaying less method bias. In
a comparison of randomized and grouped scales, the
randomized scale engendered as much convergent and
discriminant validity (Waters et al., 1982). Thus, a
randomized order of presentation seems indicated.
Obtaining a meaningful measure of the frequency of
target behaviors requires attention to the variables of
response format, length of the observation period, and
type and number of raters. According to Tzeng (1983),
four response formats are most frequently cited in the
literature. They can be differentiated in terms of two
psychometric criteria. First, the existence of a neutral
response option defines the free choice format. Absence
of a neutral rating option defines the forced choice
format. Second, categorical (qualitative) ratings answer
the question "Does the ratee fit this category?" while
discriminatory (quantitative) ratings answer the question
"To what degree does the ratee fit?"
Tzeng (1983) criticized forced choice measures for
their omission of a valid response category, i.e., uncer-
tainty or neutrality of the raters' perceptions. King,
Hunter, and Schmidt (1980) concluded that a forced choice
format was ineffective in reducing rater halo. Dunnette
(1963, p. 96) reported that rater resistance to forced
choice formats led to their abandonment.
Categorical, or qualitative, formats used in
checklists cannot detect relative differences in degree
between two behaviors performed by the same ratee or
between the same behaviors among rates (Tzeng, 1983).
Johnson, Smith, and Tucker (1982) found less response
skewness on a 5-point Likert discriminatory scale compared
to a yes/?/no categorical format. A zero-based discrimina-
tory, free choice response format seems most appropriate
(Likert, 1932). The absence of a behavior can be indicated
by the 0 position or, if present, the perceived frequency
can be indicated by choosing a value from the remainder of
the scale (Edwards, 1957).
The number of value choices permitted to the rater is
a critical issue. If few points are used some information
may be lost, but the scales are less ambiguous for the
rater. If there are too many points the discrimination
may be too fine for the rater to make. Albaum et al.
(1981) attempted to show superiority for a continuous
scale format, but concluded that equivalent aggregate
measurements were obtained from a 5-category, discrete
rating scale.
Likewise, Bernardin et al. (1976) and Bardo and
Yaeger (1982) failed to find continuous scales superior to
discrete scales. The superiority of a 5-point, discrete
rating scale has been suggested by Cowen, Dorr, Clarfield,
Kreling, McWilliams, Pokracki, Pratt, Terrell, and Wilson
(1973); Lissitz and Green (1975); McKelvie (1978); Neumann
and Neumann (1981); and Broadbent, Cooper, Fitzgerald, and
Parkes (1982).
Conversely, Bardo and his associates (Bardo & Yeager,
1982; Bardo, Yeager. & Klingsporn, 1982) found obtained
means and variances closer to the expected values for
4-point scales over 5- and 7-point scales. These results
appear contrary to most other studies. Edwards (1957, pp.
150-151) gives Likert's original statistical rationale for
the use of a 5-point scale, anchored with the integers 0
through 4, and the summation of scores for individual
items as a total score for each ratee. Current research
provides no compelling evidence for departing from this
original format.
An anchor, e.g., "always," "sometimes," "never," is
usually associated with each scale point of a Likert-type
summated rating scale (Pohl, 1981). While a variety of
anchors has been used, the basis for the selection is
often not stated (Beatty, Schneier, & Beatty, 1977;
Broadbent et al., 1982; Camp, 1980; Cowen et al., 1973;
Hunter, Hunter, & Lopis, 1979; Kassin & Wrightsman, 1983;
Moses, 1974; Siegel, Dragovich, & Marholin, 1976; Solomon
& Kendall, 1977; White, 1977).
Several studies have investigated the assumptions
involved in the selection of one popular set of anchors:
always, often, occasionally, seldom, and never. Parducci
(1968), Chase (1969), and Pepper and Prytulak (1974) con-
cluded that the meanings of anchor words were influenced
by context. The effects of individual differences among
raters on their interpretations of anchor words were
demonstrated by Helson (1969) and Goocher (1965). These
studies suggested that the above anchors may not define
perceptually equal intervals along the rating continuum.
Four studies (Bass, Cascio, & O'Conner, 1974;
Schriesheim & Shriesheim, 1974, 1978; Spector, 1976) have
sought to select five anchor words that would be perceived
by raters as defining equally spaced rating intervals.
However, the most definitive study appears to be Pohl's
(1981) partial replication of the Bass et al. (1974) and
Shriesheim and Shriesheim (1974, 1978) studies. Using
responses from 164 college students, Pohl (1981) calcu-
lated the means and standard deviations for 39 expressions
of frequency.
Comparing these with the theoretical mean responses
for a 5-point equal interval scale, Pohl (1981) derived
the response set of always, quite often, sometimes, very
infrequently, and none of the time. The calculated mean
(26.71) for the mid-point term "sometimes" differed signi-
ficantly (p < .001) from the theoretical mean (29.05), but
nevertheless was the value closest to the optimal for a
5-point scale. The other calculated values were not
significantly different from the theoretical profile.
Thus, with the exception of the mid-point term, it appears
that the anchors produced by the Pohl (1981) study
adequately defined equal-appearing intervals on a 5-point
rating scale.
The length of the period for which behaviors are to
be rated has been little studied. For instance, the
manual for the Behavior Problem Checklist (Quay &
Peterson, 1967, 1975, 1979) does not specify for the rater
the inclusive time period to be considered in rating the
listed behaviors. The authors of the Devereux Elementary
School Behavior Rating Scales (Spivack & Swift, 1966)
instructed their raters to "consider recent and current
behavior" (p. 75). The same authors (Spivack & Swift,
1977), in developing the Hahnemann High School Behavior
Rating Scale, instructed teachers to base ratings on
behavior observed "over the past month" (p. 300).
A study (Hinton, Webster, & O'Neill, 1978) of hospi-
talized clinical patients used a 6-week time period. An
investigation (Beatty et al., 1977) of performance rating
in a data processing firm utilized three assessment
periods of two months each for a total of six months. In
a study of several response formats, Broadbent et al.
(1982) used a 6-month inclusive time period. However, in
none of these studies was a rationale given for selection
of the time period.
Two attempts at aggregating measures over specific
time periods have provided more precise instructions to
the rater. Cowen et al. (1973) defined each of five
rating points in terms of the inclusive time periods to be
considered when aggregating occurrences of behavior. For
example, the fourth anchor point, often, was defined as
"you have seen this behavior more often than once a week
but less often than daily" (p. 16). Camp (1980) used the
following scale:
Frequency of occurrence
0 Never observed
1 Once or more in semester
2 Once or more monthly
3 Once or more weekly
4 Once or more daily (p. 11)
The work of Seymour Epstein (1980), in support of the
stability over time of personality traits, bears directly
on the issue of aggregating behavior ratings over some
time period. Epstein (1980) stated that "stability can be
demonstrated . as long as the behavior in question is
averaged over a sufficient number of occurrences" (p.
791). In testing this hypothesis, Epstein conducted four
studies in which he used, among other types, ratings
performed in classrooms by teachers. Epstein suggested
aggregating behavior over subjects, stimulus situations,
time, and modes of measurement in order to establish
predictive reliability and validity (p. 797).
Ratings of middle and junior high school students by
their teachers in different courses would meet the
conditions of subjects and situations. Epstein (1980)
suggested that ratings at a single time following multiple
or extended observations represent an intuitive averaging
that has the "potential for producing highly replicable
and valid results" (p. 802). Harrop (1979) also
challenged the common assumptions (Fay & Latham, 1982;
Latham, Fay, & Saari, 1979) that coding of directly
observed behaviors produced superior results to aggregat-
ing behaviors over time.
A related concern in the assessment of school-related
behavior is selection of the time of year in which the
ratings will be made. Several studies (Cowen et al.,
1973; Epstein, et al., 1983; Larrivee & Bourque, 1980)
recommend allowing student behavior and teacher percep-
tions to stabilize. Supporting these decisions are data
from the Texas Junior High School Study (Evertson,
Anderson, & Brophy, 1979).
Evertson and Veldman (1981) found a moderate but
steady increase in serious misbehavior over the course of
the school year and an increase in general misbehavior in
April. Evertson and Veldman (1981) concluded that short-
term studies should avoid ratings made either early or
late in the school year. The available literature seems
to suggest the feasibility of aggregating behaviors over
time periods specified in the rating scale instructions
and after teachers have had at least two months to observe
student behavior.
Deciding on the most appropriate type of rater to use
in assessing children's behavior has long been a problem.
In 1965, Ross et al. recognized the potential usefulness
of teacher ratings. Teacher's ratings have been found to
be more accurate than peer ratings of classroom behaviors
(Bailey, Bender, & Montgomery, 1983); other school profes-
sionals' ratings (Bower & Lambert, 1971, p. 143; Freemont
& Wallbrown, 1979), and institutional child care workers'
ratings (Kohn et al., 1979) and to be equivalent to the
ratings obtained by a multidimensional scaling technique
(MDS) applied to classroom behavior (Sanson-Fisher &
Mulligan, 1977).
A number of researchers have found support for
teacher ratings as appropriate measures of general class-
room behaviors (Solomon & Kendall, 1977), social behavior
(Loranger, Lacroix, & Kaley, 1982), assertive vs. aggres-
sive behavior (Roberts & Jenkins, 1982), acting out
behavior (Walker, 1970), and behavior that would likely
result in referrals for exceptional child education (Dean,
1980; Epstein et al., 1983; Home & Larrivee, 1979; Lahey,
Green, & Forehand, 1980; McKinney & Forman, 1982; Roberts
et al., 1981).
Not all studies have yielded positive results. Morris
and Arrant (1978) found that regular classroom teachers
tended to see more behavior problems in students referred
for evaluation than did school psychologists. A study
(Kazdin, Esveldt-Dawson, & Loar, 1983) of psychiatric
inpatient children found extra-class raters' evaluations
of overt classroom behaviors to correspond more closely to
direct observational data than did teachers' ratings.
However, teachers were more accurate than the extra-class
raters in identifying hyperactive children using a behav-
ior checklist. Overall, the evidence suggests strong
support for the use of teachers as raters of classroom
behavior.
An associated issue is the use of multiple raters to
increase reliability and reduce halo effect (Epstein,
1980). Ratings of students commonly are obtained from all
teachers having direct classroom contact (Linton & Chavez,
1979; Wixson, 1980). This procedure could result in as
few as one or perhaps as many as seven ratings, depending
on the grade level and local practice.
More recent research efforts have focused on
empirically determining the most effective number of
raters. Prinz and Kent (1978) increased from 1 to 4 the
number of raters of parent-adolescent interactions in a
clinical setting and reported increased reliabilities.
Both reliability and concurrent validity of clinical
judgments were shown to increase when the number of judges
was increased from one to ten (Horowitz, Inouye, &
Siegelman, 1979). Strahan (1980) extended the Horowitz et
al. (1979) study and concluded that after using four
raters, adding additional ones contributed little to
measurement effectiveness. Another study (Green, Bigelow,
O'Brien, Stahl, & Wyatt, 1977) of inpatient clinical
behaviors found little improvement when using more than
four raters.
Although in general agreement with the above studies,
a cautionary note was added by Kenny and Berman (1980),
who pointed out that if raters are completely unreliable,
increasing their numbers will not increase reliability.
The number of teachers usually available in a middle or
junior high school to serve as raters would appear to be
adequate to contribute to both improved reliability and
concurrent validity.
Various classifications of severity have been adopted
in school settings. Student conduct codes typically use
some method of indicating seriousness of offenses, such as
"serious misconduct" (Pinellas County Schools, 1983, p. 7)
and "minor, intermediate, and major" (Duval County Public
Schools, 1980, p. 16). Researchers (Pisarra & Giblette,
1981) have used categories emphasizing the targets of the
behavior (e.g., offenses against persons, offenses against
state laws). Teachers often focus on specific behaviors
(e.g., use of drugs, striking teacher) (Camp, 1981) and
administrators have used a combination of both (National
School Public Relations Association, 1973).
There is little consensus on the number of levels to
be used in assigning degrees of severity. Taylor et al.
(1979) used levels ranging from 1 (not very severe) to 4
(extremely severe). Camp (1980) used 0 for "not con-
cerned" through 4 for "extremely concerned." In an
earlier study, Moses (1974) used three levels, 1 (mild), 2
(moderate), and 3 (severe) in asking mental health and
criminal justice professionals to rate a list of problem
behaviors. To use too many levels may imply a degree of
confidence in discrimination not supported by the subjec-
tive nature of such ratings.
Not all rating scale authors and researchers accept
the necessity for including a severity rating (Searls,
Isett, & Bowders, 1981; Spivack & Swift, 1977). Even
when, as in the Behavior Problem Checklist (Quay &
Peterson, 1967, 1975. 1979), a severity factor is provided
for, the author does not always recommend its use. How-
ever, at the practitioners' level the degree of severity
of behaviors is a major concern.
Algozzine (1979), using items characteristic of
several behavior rating scales, developed the Disturbing
Behavior Checklist which asks teachers to rate the degree
of disturbance they experience as a result of different
student behaviors. This suggests a consequence to the
teacher based not on the frequency of the behavior, but on
the type and severity. After noting irregularities and
lower reliabilities, Taylor et al. (1979) had teachers
rate for severity 26 items of Part Two of the Adaptive
Behavior Scale (ABS) (Nihira, Foster, Shellhaas, & Leland,
1969). Teachers were able both to categorize behaviors
and rate them in terms of severity, leading Taylor et al.
(1979) to conclude that this additional information would
be useful in refining the scale and adding to its clinical
efficacy.
Inasmuch as the instrument developed in this study is
intended to have locally developed norms, the statistical
techniques used in the norming procedure and the comparing
of individual scores to the derived local norms are not
complex. While some more recent studies have focused on
problems associated with such common procedures as the
calculation of measures of central tendency (Mosteller &
Tukey, 1977; Stavig, 1978, 1982), many researchers
continue to rely on descriptive statistics utilizing raw
scores, arithmetic means, standard deviations, and stan-
dard scores.
White (1977) compared individual student's scores on
classroom behavior to the computed mean score for five
classes of "Follow Through" program students in order to
identify immature students. In a business setting, Fay
and Latham (1982) used means and standard deviations in
comparing scores obtained using two different rating
methods. A study (Lyness & Cornelius, 1982) comparing
judgment strategies and ratings of college instructors
supported the use of a rating scale composed of discrete
items, with an overall rating calculated by weighting the
items and summing the weighted scores. To obtain mean
sub-scores for subjects, Algozzine (1980) summed scores
across the items defining each of four factors of
disturbing behaviors and used means and standard devia-
tions in analyzing the results.
The cited studies seem to support the use of descrip-
tive statistics in both obtaining individual scores (i.e.,
sum of weighted ratings)-and deriving a local norm (i.e.,
mean) from ratings of a representative sample of a total
population. Salvia and Ysseldyke (1981, chap. 4) offer
definitions of common terms for descriptive statistics
applied to assessment.
Psychometric Properties of Rating Scales
Historically, rating techniques have aroused contro-
versy over estimations of validity and reliability (Ryan,
1958). Validity is the relevance of the scale to the
variables being measured. Most sources recognize three
types of validity, i.e., content, criterion-related or
concurrent, and construct (American Psychological
Association, 1966; Cronbach, 1970; Kerlinger, 1972).
Reliability is the accuracy or precision of a measuring
instrument and has been usually classified as either
temporal, inter-rater, or internal (Cronbach, 1970).
However, investigations (Epstein, 1980) into the
effects of situations on behavior have recently introduced
a fourth consideration, situational reliability, or the
consistency of behavior across settings. The development
of norms against which to compare results obtained from
individual administrations of rating scales is another
area of active investigation (Mendelsohn & Erdwins, 1978;
Messick, 1980). Research on these issues is reviewed in
this section.
Content validity refers to the relevance and repre-
sentativeness of the items used in construction of a scale
(Epstein, 1980). Often, this is determined by obtaining
judgments from experts not otherwise involved in the scale
construction (DiStefano, Pryer, & Erffmeyer, 1983; Jones
et al., 1975, p. 83; Lawshe, 1975; Thorne, 1978).
Kreitler and Kreitler (1981) found that item content
determined the rater's perception of the central theme of
an instrument. Items not perceived as relevant to the
central theme tended to be given neutral responses, thus
limiting the information contributed by the rater.
Criterion-related validity is studied by comparing
scores obtained from an instrument with one or more
external criteria of the variable being measured
(Kerlinger, 1972, p. 459). Criterion-related validity
encompasses both concurrent and predictive qualities
(Epstein, 1980). The comparison of scale results with an
independent judgment or diagnosis of a subject is an
example of an attempt at estimating criterion-related
validity. If the judgment or diagnosis confirms the scale
indications, the inference may be drawn that the scale is
in agreement with the concurrent diagnosis and is
predictive that others given a similar rating would also
be diagnosed similarly (Kohn et al., 1979; Mendelsohn &
Erdwins, 1978).
In one validation study, Harris, Kreil, and Orpet
(1977) used the school principal, guidance counselor, and
two teachers as judges in selecting both disruptive and
prosocial students for rating by the Behavior Coding
System (Patterson, Ray, Shaw, & Cobb, 1969). In develop-
ing the Pittsburgh Adjustment Survey Scales (Ross et al.,
1965), school principals were used to nominate adjusted,
withdrawn, and aggressive students for rating by their
teachers and scale results were compared with these
nominations.
According to Kerlinger (1972, p. 461) and Cronbach
(1970), the significance of construct validity is its
concern with the theory behind the variable being
measured. Guion (1977) argues that construct validity
integrates both content and criterion considerations.
Likewise, the usefulness of content and concurrent
validity is questioned by Sanson-Fisher and Mulligan
(1977) and construct validity is supported.
A definition of construct validity as the process of
ascribing meaning to scores is offered by Stenner and
Smith (1982). Messick (1980) broadens the concept of
validity to include both test interpretation and test
use. Messick (1980) describes construct validity as
"interpretive meaningfulness" (p. 1015) and suggests that
it rests on four bases: convergent and discriminant
validity, ethical interpretation, relevance and utility
for the specific application, and the consequences follow-
ing use of the instrument.
To be interpretable, a rating scale must be reli-
able. That is, a scale must produce similar results when
applied to the same person over several administrations,
the instrument must be relatively free of errors of mea-
surement, and the results must closely approximate the
"true" value of the variable for the person being rated
(Cronbach, 1970; Kerlinger, 1972).
Typically, test-retest data are compiled for varying
time periods between administrations. The correlation
between the two obtained scores is used to justify esti-
mations of temporal stability and, in the case of rating
scales, intra-rater reliability. Examples of reported
test-retest intervals include one week (Duval County
School Board, 1979; Quay, 1977), two weeks (Mendelsohn &
Erdwins, 1978; Russell, Lankford, & Grinnell, 1981) and
two years (Quay, 1977). However, Masterson (1968) pointed
out that low test-retest correlation coefficients may
reflect the transitory nature of the measured variable and
suggested high coefficients of internal consistency may be
more indicative of reliability for some instruments.
Internal consistency has often been estimated by
inter-item and item-total analysis (Edwards, 1957;
Kerlinger, 1972). In these procedures, an individual's
rating on one item is compared with the rating on all
other items or with the total score from the scale or
subscale to estimate the degree to which each item is
similar to the other items. Item analysis may be impor-
tant in reducing errors of measurement attributable to the
composition of the instrument (Benson & Clark, 1982).
However, internal consistency may not provide good reliabi-
lity estimation for a rating scale assessing constructs
comprised of many discrete behaviors (Kerlinger, 1972).
Some research (Rosenthal & Jacobson, 1968; Sulzbacher,
1973) into observer bias has suggested that beliefs about
rates may affect rater perceptions and, consequently, the
reliability of the ratings. In three studies (O'Leary &
Kent, 1973; Shuller & McNamara, 1976; Siegel et al., 1976)
of disruptive classroom behavior, while biasing informa-
tion experimentally introduced was found to influence
global ratings, it had no significant effect upon results
obtained from behaviorally stated scales. Siegel et al.
(1976) suggested that behaviorally specific items reduce
bias and improve inter-rater and intra-rater reliability.
The degrees of agreement among different raters on
measures of the same subjects at the same time in the same
setting have been used to indicate the inter-rater
reliability of an instrument (Cronbach, 1970). Also, the
agreement among different raters of subjects in the same
settings at different times has been used for the same
purpose (Cronbach, 1970). In middle and junior high
schools, these conditions do not usually occur naturally.
Fortunately, investigations of trait consistency in
subjects (Abikoff, Gittelman, & Klein, 1980; Epstein,
1980; Mischel, 1969) have encouraged the comparisons of
ratings by different raters over the same elapsed time
periods, but for different settings and situations,
conditions which do occur naturally in the secondary
school setting.
Epstein (1980) concluded that subjects do manifest
trait consistency, if aggregation techniques are applied
in assessing behaviors. Epstein (1980) suggested aggre-
gation over raters (e.g., teachers), situations (e.g.,
classrooms), occasions (e.g., class periods), and measures
(e.g., disciplinary records). Epstein further suggested
that when single ratings are made after extended periods
of observation, these ratings are similar to aggregated
ratings in that they represent an intuitive averaging of
ratings over many observations. Thus, reliability may be
improved by combining different teachers' ratings of the
same student over the same portion of the school year.
According to Cooper (1981), perhaps the most ubiqui-
tous challenge to inter-rater reliability is halo error
(Thorndike, 1920) or the tendency of a rater to allow
overall impressions of an individual to influence judgment
of specific areas of behavior (Holzbach, 1978). Attempts
(Landy, Vance, Barnes-Farrell, & Steele, 1980; Landy,
Vance, & Barnes-Farrell,' 1982) to statistically control
for halo effects have apparently not succeeded (Harvey,
1982; Hulin, 1982; Mossholder & Giles, 1983; Murphy, 1982).
One exploration of ways to reduce halo error resulted in a
restatement of classic advice: do not use rating cate-
gories that are imprecise and overlapping (Cooper, 1983).
In an extensive review of the literature, Cooper (1981)
concluded that of nine methods currently employed to reduce
halo effect, all leave residual illusory halo.
Studies of variables affecting reliability have iden-
tified several other challenges to the accuracy of school
behavior ratings. The sex of the teacher was found in two
studies (Levine, 1977; Silvern, 1978) to be correlated
with ratings of classroom behavior, with male teachers
consistently reporting lower levels of disruptive behav-
ior. Teachers' ratings seemed to be influenced by special
education labels in one study (Fogel & Nelson, 1983). In
two studies (Marwit, 1982; Marwit, Marwit, & Walker, 1978),
perceived unattractiveness of students has been shown to
correlate with higher ratings of disruptive behavior.
While challenges to reliability from a variety of
sources have been observed, several studies (Bernardin &
Pence, 1980; Fay & Latham, 1982; Latham, Wexley, & Pursell,
1975; Madle, Neisworth, & Kurtz, 1980; Pursell, Dossett, &
Latham, 1980) have suggested that training in the use of
rating scales may be effective in reducing errors of
measurement. This review of studies of validity and
reliability has identified some sources of and counter-
measures for errors of measurement. Next, studies of the
variables affecting the norming of rating scales will be
reviewed.
Several writers have shown concern for the relation-
ship between behavior and the context in which it occurs.
The social value of a test, according to Messick (1980),
is determined by its instrumental value for a particular
setting. Willems (1975) stated that few phenomena have
meaning independent of the context in which they occur.
Likewise, researchers were cautioned by Dickinson (1978)
to evaluate behavior only in an environmental context.
Epstein (1980) referred to the "extreme situational
specificity of behavior" (p. 794) and warned that experi-
ments conducted in a single situation cannot be relied on
to generalize across even minor variations in stimulus
conditions. Others supporting this psychosocial approach
include Sherif (1954); Erickson (1963), quoted in Tinto,
Paclilio, and Cullen (1978); Salvia and Ysseldyke (1981,
p. 378); and Zammuto, London, and Rowland (1982).
Schools were described by Garbarino (1980) as "con-
texts for behavior and development" (p. 19). Some of the
characteristics of schools which may influence levels and
interpretations of disruptive behavior are size of enroll-
ment (DiPrete, 1981, p. 86; Garbarino, 1980; Kowalski,
Adams, & Gundlach, 1983); public or private administration
(DiPrete, 1981, p. 81); control orientation (e.g., human-
istic vs. custodial) (Deibert & Hoy, 1977; Gaynor &
Gaynor, 1976); degree of person-environment fit (Kulka,
Klingel, & Mann, 1980); traditional vs. open classrooms
(Solomon & Kendall, 1975); length of faculty tenure
(DiPrete, 1981, p. 107); socioeconomic level of the host
community (Kowalski et al., 1983); and region of the
country (DiPrete, 1981, p. xx; Kowalski et al., 1983).
Researchers advocating the use of local norms for behav-
ioral measurements include Fremont and Wallbrown (1979);
Mendelsohn and Erdwins (1978); Quay and Peterson (1967);
Smith (1976); Walker and Hops (1976); and Wallbrown,
Wallbrown, and Blaha (1976).
The effects of sex, age, race, and socioeconomic
status on ratings of disruptive behavior have been
frequently studied. The types of disruptive behavior
displayed in both educational and clinical settings have
not been found to be significantly different for the
variables of sex (Behar & Stewart, 1984; Epstein et al.,
1983; Morris & Arrant, 1978; Stott et al., 1975, p. 166),
age (Behar & Stewart, 1984; Ghodsian, Fogelman, Lambert, &
Tibbenham, 1980; Stott et al., 1975, p. 83), race (Gajar &
Hale, 1982), or socioeconomic status (Behar & Stewart,
1984; Stott et al., 1975, p. 97). Thus, providing for
separate norms for these variables seems unnecessary in
any scale rating only disruptive behaviors.
Uses of Behavior Rating Scales
Bailey et al. (1983) supported the use of rating
scales in program planning and evaluation. Likewise, the
lack of effective measurement devices was seen by Hirshoren
and Heller (1979) as limiting the evaluation of program
effectiveness. Mesinger (1982) called for the use of
appropriate measurement devices in providing services for
deviant youth within the public school setting. Cooper
(1983), Peed and Pinsker (1978), and Beatty et al. (1977)
have suggested providing rating scale results to rates to
influence behavior changes. Using rating scales to pro-
vide a standardized description of behavioral problems has
been suggested (Edelbrock & Achenbach 1978).
In a study comparing resource room delivery models,
Wixson (1980) used a behavior rating scale in developing
and evaluating intervention programs for various cate-
gories of handicapped children. Morton Bortner (Buros,
1978, p. 493), reviewing the AAMD Adaptive Behavior Scale,
pointed out its usefulness for evaluating the progress of
individuals and evaluating program goals. The Duval
County School Board (1979) used a locally constructed
behavior checklist to evaluate their grant-funded program
for disruptive students.
Several programs which retained students in their
regular classrooms have used behavior scales for evalua-
tion purposes. Walker and Holland (1979) and Linton and
Chavez (1979) developed and used rating scales for this
purpose in elementary and junior high schools, respec-
tively. The Hahnemann High School Behavior Rating Scale
(Spivack & Swift, 1977) was intended to provide teachers
with a practical means of describing disruptive classroom
behavior to parents and other school personnel. In a
study of junior high school truants, Nielsen and Gerber
(1979) used a behavior rating scale to match school inter-
ventions with student needs.
A quantitative measure of disruptive behavior was
developed by Mendelsohn and Erdwins (1978) to assist
community agencies in devising programs for expelled
students. Haskell (1979) developed a method of quantify-
ing clinical behavior in institutional settings to provide
a basis for planning individual programs and evaluating
results. McSweeney and Trout (1979) used the Jesness
Behavior Checklist (Jesness, 1970) to evaluate the social
progress of deviant children in a wilderness camp pro-
gram. Five reasons for obtaining measures of students
are offered by Salvia and Ysseldyke (1981): "Screening,
placement, program planning, program evaluation, and
assessment of individual programs" (p. 14). Behavior
rating scales have been used to obtain measures for each
of these needs.
Summary
This review has identified five approaches in the
literature to define disruptive school behavior (DSB). A
conceptualization of DSB based on the interactions of
students, teachers, and administrators within the school
setting was suggested as most relevant for the development
of an instrument to quantify DSB.
Psychometric challenges to the use of rating scales
for identifying behavioral characteristics were consid-
ered. Research was cited to suggest that teachers using
reliable and valid scales could accurately identify DSB.
Nineteen instruments available for assessing problem
behaviors were reviewed. None appeared to meet the
psychometric criteria required for educational placement
decisions. Possible sources of error in nonsystematic
observations were presented with the suggestion that
inaccurate, biased, or subjective judgments may result.
Type, frequency, and severity of behaviors were
related to item content, item format, and response
format. Support was found for the inclusion of these
measurement parameters in assessing DSB. The use of
descriptive statistics in current research for obtaining
individual behavior ratings and deriving local norms was
demonstrated.
A review of the sources of error in measurement was
conducted and counter-measures for improving validity and
reliability estimations were suggested. A number of
variables affecting the norming of rating scales were
investigated. Research evidence rejected separate norms
based on gender, race, or socioeconomic status. The
effective use of behavioral instruments in a variety of
settings was documented, suggesting the suitability of
such a device for describing students who display DSB.
CHAPTER THREE
METHODOLOGY
The purpose of this study was to develop and validate
an instrument, the Disruptive Student Behavior Scale
(DSBS). The DSBS is intended to be used to assess
quantitatively the disruptive school behaviors of students
referred for placement in either special education or
alternative education programs. This chapter presents the
research questions, defines the target population,
presents a plan for constructing the scale, describes
procedures for a pilot study, details statistical tests
and procedures for the data analyses, and discusses
possible limitations of the study.
Research Questions
1. Does the content of the DSBS represent behaviors
recognized and accepted by educators as occurring
in and disruptive to the school environment?
2. In the judgment of experts, does the DSBS contain
an equitable distribution of items descriptive of
the underlying theoretical constructs that
identify disruptive students and discriminate them
from non-disruptive students?
3. To what degree does the DSBS demonstrate
criterion, convergent, and discriminant validity?
4. To what degree does the DSBS provide ratings which
are stable over time?
Construction of the DSBS
The following plan is a modification of a suggested
procedure (Benson & Clark, 1982) for rating scale construc-
tion. A review of disruptive school behavior (DSB)
literature provided a research base for defining the
constructs comprising DSB. A total of 303 descriptive
items and 22 categories were found in 36 studies. After
eliminating duplications and items not pertaining directly
to DSB, 56 items remained. Combining similar categories
resulted in a total of 13 potential categories of behav-
iors associated with DSB.
In a project conducted by the Research Committee of
the Psychological Services Department of the Duval County,
Florida School District, the 56 items and 13 categories
were presented to 16 teachers of middle school students
enrolled in a behavior management program for disruptive
students. The rating group was composed of 10 females and
6 males, and all had at least two years full-time teaching
experience. Group members were instructed to assign each
item to one or more of the 13 categories. Instructions
and results are reproduced in Appendix A.
The judges' ratings and comments resulted in the
retention of 10 categories, which were considered to be
one set of constructs which could be used in identifying
DSB. A tentative definition of each construct was
formulated using the descriptive items assigned by the
teachers. A verification was attempted of the inclu-
siveness of these derived constructs. A frequency
distribution was prepared for all of the conduct code
violations reported for chronic violators in a sample of
Duval County, Florida, elementary, middle/junior high,
high, and alternative schools (Moses, 1981). Of 7717
behavior violations, 7686, or 99.6%, were included within
the definitions of the proposed constructs.
The items, as taken from the studies and used in
developing the constructs, were not considered specific
enough for use in a quantitative rating scale. However, a
readily available pool of potential items was located in
the disciplinary referral records of an inner-city junior
high school in a metropolitan Florida school district.
Verbatim transcriptions were made of the reasons recorded
on the referral forms by teachers when sending students to
the deans. All active folders for the 1980-1981 school
year were reviewed. A total of 395 items, including dupli-
cations, were recorded without regard for gender, age,
race, or grade level. Combining obvious duplications and
similarities resulted in 66 items (Appendix B) to be
considered for inclusion in a scale for rating DSB.
All of the 66 items were then presented individually
to six male and five female volunteers, experienced
secondary school regular classroom teachers from suburban
Florida middle and junior high schools. Instructions are
reproduced in Appendix C. These teachers were asked to
verify the specificity of the items and edit those consid-
ered ambiguous. This review yielded 40 items for possible
use on an instrument. These items were then stated in the
past tense to reflect the intention to measure students'
past behavior (Appendix D). This preliminary study indi-
cated the feasibility of using research-based constructs
and teacher-generated items as the basis for a rating scale
for disruptive school behavior.
In order to reduce halo and leniency errors, it has
been suggested (Blanz & Ghiselli, 1972) that a scale be
arranged so that items from the same construct will not be
contiguous. Accordingly, items were initially randomly
ordered, then inspected and rearranged to meet this crite-
rion. (Appendix M). Research studies previously cited
suggested that in addition to specifying the type, a
quantitative measure of disruptive behavior must provide
for rating both frequency and severity. Frequency rating
was provided for by the choice of response format selected
for the instrument. The literature review suggested the
suitabilty of a 5-point, equal interval, summated rating
scale (Likert, 1932) using the following anchors:
0 None of the time
1 Very infrequently
2 Sometimes
3 Quite often
4 Always (Pohl, 1981, p. 239)
The rating scale (Appendix F) utilized this response
format.
The severity rating for each scale item was estab-
lished with assistance from the faculty, staff, and
administrators of two alternative schools located in two
metropolitan Florida school districts. From their experi-
ence with disruptive students, these educators were
particularly aware of the consequences for students who
display DSB. Respondents were selected from volunteers,
including the principal and assistant principal, school
psychologist, social worker, educational evaluator, and
faculty members. This group contained both males and
females in approximately equal numbers. All had more than
two years' experience working with disruptive students.
The school experience may be conceptualized as influ-
encing the social, personal, and academic domains of a
student's life. Each of these domains may be subdivided
to facilitate closer study of the consequences of the
school experience (See Table 1). One way for educators to
assign a severity factor to a disruptive activity is to
have them estimate which domains of student life would
likely be affected adversely by that particular behavior.
Instructions for this procedure are reproduced in Appendix
G. The number of adverse consequences assigned by at
least 50% of the raters, divided by a constant of three to
keep the numbers small and with fractions rounded up to
the next whole number, gave a severity rating of 1, 2, or
3 to each of the items on the rating scale. Results are
reported in Chapter Four.
A scoring template incorporating the severity factor
was prepared for the DSBS (Appendix H). This template has
five holes, one corresponding to each possible frequency
rating (i.e., 0, 1, 2, 3, 4) for each rating scale item.
Through the holes are read the rater's mark (X) indicating
the frequency rating assigned. Above each hole is printed
a number which is the product of that frequency rating and
the previously determined severity factor for that item.
Thus, the weighted score for that item may be read by the
scorer directly from the scoring template and recorded on
the DSBS rating form beside each item.
These item scores were then added to give the page
score and form score (see Appendix F) and recorded onto a
summary sheet (Appendix I). The Summary of Teacher
Ratings form (Appendix I) contains for'each student the
DSBS rating; the deviation, in z-scores, from the local
0
o I
r. a 0
4) rz
HI a) V) > a
Q) (0 > bO I.
Q. 0 bO O 0
X r C bO.r-,
0 (*M r- .
0
OQC
0 a) CM ii
o
4O
.0
C C 4.)
c oo
1) ., o .-1
0 --- > >
C C O
0 0 0
4 0Q -
4- bOi 4-4
-4 .0
CC Oi U
0
C 0u 4
* LO 0
r= ., CO.
.0 4-)0
sC 0 -
S( Oa O
VC 0 *0
4- ) Q
(0 0 *P
Iu 0 >
T- t/3 Q .
r-1 r- o0
(U CO CO
- NCJ
norm; a comparison of ratings by each teacher; and the
basis for constructing a DSBS profile for prescriptive use
(Appendix J). These data are intended to provide local
school authorities with criteria for estimating the devia-
tion of any student's rating from the local DSB norm and
are intended to assist in determining a student's need for
an intervention program. The DSBS is normed locally
within each school district. Norms from this study are
reported in Chapter Four for information, but are not to
be used as criteria for judgments about students in other
settings.
Validation of the DSBS
To assure content validity, the 40 items and the 10
constructs developed from this preliminary study were
presented to a group of 24 teachers with instructions to
assign each item to a construct category or to no category.
The instructions are reproduced in Appendix E. Each judge
had at least two years of regular classroom teaching
experience in a middle or junior high school. Thirteen
male and 11 female teachers participated. The judges were
also asked to verify the specificity of the retained items
and reword those considered ambiguous. Revisions were
made as suggested and confirmed by a follow-up study using
another group of eight similarly-qualified teachers.
As described in the field study section, at a Florida
middle school a criterion group of disruptive students was
selected by nomination by seven non-teaching school person-
nel, including two deans, three guidance counselors, and
two administrators. Students in the disruptive group were
ranked numerically on a continuum from non- to severely
disruptive, based on subjective ratings from all the
nominating personnel. DSBS ratings from teachers were
compared to these subjective ratings to determine how well
high DSBS teacher ratings correlated with high levels of
disruptiveness as perceived by non-teaching school person-
nel.
To estimate how well the DSBS identified the disrup-
tive group, the mean of DSBS ratings for the disruptive
group students were compared with the mean of DSBS ratings
for a norming group representing a sample, stratified by
grade, of the school population. If the DSBS demonstrated
agreement with the concurrent judgments of disruptiveness
made by non-teaching school officials, there would be made
a prima facie case for predicting that students in other
settings identified by the DSBS as disruptive would also
be judged disruptive by non-teaching school officials.
Messick (1980) described construct validity as based
on convergent and discriminant validity, ethical
interpretation, relevance and utility for the specific
application, and the consequences following use of the
instrument. Convergent validity requires that the DSBS be
able to identify all students who are considered exces-
sively disruptive. To demonstrate satisfactory convergent
validity, the DSBS ratings of 100% of the students in the
disruptive group would have to be significantly above the
local DSB norm. The disruptive group ratings are reported
in Chapter Four.
Discriminant validity requires that the DSBS be able
to reject those students who are not considered exces-
sively disruptive. To demonstrate satisfactory discrimi-
nant validity, the DSBS ratings of only those students in
the disruptive group, or eligible for inclusion, could be
significantly above the local DSB norm. Ratings of the
norming group are reported in Chapter Four.
Ethical interpretation of DSBS ratings requires an
understanding of both the theoretical and practical
concepts underlying development of this instrument. There-
fore, a manual will be prepared before the DSBS is offered
for research use. Relevance was supported by the theoreti-
cal basis on which the 10 constructs were chosen to define
DSB for this study. Utility was provided by the proce-
dures used to select appropriate items, score the forms,
interpret the ratings, and present the results. The conse-
quences of using the DSBS cannot be predicted until it is
thoroughly researched. The intent is to improve the
validity of the selection process for programs assisting
disruptive students.
Reliability of the DSBS
The DSBS rating for each student is an aggregate of
scores from at least four teachers. A test-retest measure
compared two DSB ratings obtained from individual teachers.
Fourteen days after the receipt of teacher ratings, a
follow-up rating by the same teachers of approximately 10%
of both the norming and disruptive groups was made. These
results are reported in Chapter Four.
The internal consistency of the DSBS was protected by
choosing only items previously used by teachers to
describe DSB. Item analysis is not an effective technique
for establishing reliability of individual administrations
of the DSBS. Patterns of disruptive behavior are often
narrow and stereotypical, while the DSBS contains items
descriptive of a broad range of possible behaviors. Thus,
item scores were not likely to correlate with each other.
No attempt was made to assess interrater reliability.
Classroom settings are conceptualized as discrete environ-
ments, whose norms for behavior are determined by the
personality of the teacher. The behavior of interest is
the interaction of students with their teachers totally,
not individually.
Field Study
The purpose of the field study was to identify and
correct any problems, actual or potential, with item
content, response format, or administration and scoring
procedures of the DSBS. Following a successful field
study, the instrument may be offered to the profession for
further research and development (Benson & Clark, 1980).
Accordingly, the operational goal of this present effort
was to conduct a field study to determine the readiness of
the DSBS for use as a research instrument.
The target population consisted of students enrolled
in grades six through nine (i.e., middle and junior high
school grades) in public schools anywhere in the United
States. No restrictions were placed on age, gender, race
or socioeconomic status. The selection criteria for the
host school were a heterogeneous ethnic population, an
urban or suburban location, public middle (grades 6, 7, 8)
or junior high (grades 7, 8, 9) school status, random
assignment of students to basic courses, and an average
daily attendance figure of at least 500 students. Special
schools, such as alternative schools and special education
centers, were not considered.
A public middle school meeting these criteria was
located in a predominately urban school district on the
west coast of Florida. The student enrollment was
approximately 76% white, 22% black, and 2% Asian- and
Hispanic-American, with an average daily attendance of
733. Socioeconomic status was said by the principal to be
primarily upper-lower class and lower-middle class.
For the norming group, a sample consisting of 90 stu-
dents was selected using one English and one mathematics
class, with randomly-assigned enrollments, at each grade
level. A total of six classes containing 203 students and
ranging from 32 through 35 students each were sampled.
The numbers 1 through 35 were written on individual slips
of paper and 15 numbers drawn randomly using the replace-
ment procedure. For each class, the students whose class
roll numbers matched the 15 randomly selected numbers were
included in the norming groups.
The disruptive group was selected by nomination by
non-teaching school personnel, who were asked to list the
names of all of the excessively disruptive students
encountered during the current school year. It was thus
possible for a student's name to be included in both the
norming and the disruptive groups. The nominating process
initially produced a group of 64 students. After a confer-
ence among the raters, this group was reduced to 36
students.
All students finally nominated into the disruptive
group were assigned to one of four levels of disruptive-
ness (none, mild, moderate, or severe) by each nominating
person working independently. Nominated students were
assigned a numerical rating according to the following
scale:
Level of Disruptiveness Rating
None 0
Mild 1
Moderate 2
Severe 3
Students were ranked according to the average of these rat-
ings. This ranking permitted the correlation, reported in
Chapter Four, of levels of disruptiveness between the DSBS
results and the qualitative assessments by school person-
nel for each disruptive group student.
Schedules for the sample students were obtained from
school records. No contact was made with any student.
Training of all participating teachers took place in a
meeting at which a DSBS form for each period of a sample
student's current schedule was distributed. Appendix K
contains these instructions. The purpose of the study was
explained and a date and procedure for returning the forms
agreed upon. Emphasis was placed on the need to respond
to only the behaviors actually mentioned on the instrument
and to perform the ratings independently of other teachers.
Provision was made for a faculty member to either answer
or refer questions that might arise during the rating
period.
Teachers not submitting all their DSBS forms by the
agreed upon date were contacted and reminded of the
importance of their participation. Upon receipt of at
least four completed DSBS forms for each student, the DSBS
rating for that student was calculated. At least four
scorable forms, totaling 622, were received for 108 stu-
dents, 76 in the norming group and 32 in the disruptive
group. The scoring template (Appendix H) provided for
calculating item scores weighted for severity.
The item scores were totaled to produce a form score,
which was entered on the Summary of Teacher Ratings form
(Appendix I). This summary form contains spaces for the
student's name, grade, age, and sex; school name; evalua-
tor's name and title; individual form scores; each rater's
name, subject, and class period; and calculation of the
student's DSBS rating and z-score. Each sample student's
form scores were summed to give a total score. The total
score was divided by the number of raters to yield the
average score, which is the student's DSBS rating.
After the DSBS ratings for all the norming group stu-
dents were calculated, the mean DSBS rating and standard
deviation for the group were obtained. This mean of the
means is the local DSBS rating, or norm, for the target
school. The local DSBS norm was subtracted from the
students' DSBS rating, giving their deviation from the
local norm.
Dividing this deviation by the local standard devia-
tion gave the number of standard deviation units, or
z-scores, the student's DSBS rating differed from the
local DSBS norm. The criterion of two standard deviation
units above the local DSBS norm translates to a disrup-
tiveness score higher than approximately 98% of the
predicted scores from the school population. The distri-
bution of scores obtained from the norming group was
inspected to assure the existence of sufficient variance
to make the z-scores meaningful.
A reliability check was performed. Fourteen days
after all the rating forms were collected, approximately
10% of the students from both the norming and disruptive
groups were selected to be representative of the range of
scores. New forms were submitted to the original raters
for rerating the same students and the results compared.
These results are reported in Chapter Four. After comple-
tion of the data analyses, all participants were invited
to a meeting to discuss the results, offer comments, and
receive appreciation for their participation.
Data Analyses
Validity
To establish content validity, 24 expert judges
assigned proposed DSBS items to construct categories.
Results of the judges assignments were totaled for each
item. An item was dropped if not assigned to at least one
category by each judge. If this content validation pro-
cedure had resulted either in fewer than 30 items being
assigned to at least one construct or in having a con-
struct with fewer than three items assigned by 80% of the
respondents, enough items would have been constructed and
validated to meet these criteria. The judges' item assign-
ments are reported in Chapter Four.
To ascertain how well the DSBS identified the
disruptive group, the t-test was used to estimate the
significance of the difference between the means of the
norming group and disruptive group. An obtained prob-
ability level of .05 or less was considered evidence of
statistical significance. The magnitude of the difference
between the means was used to evaluate the practical
significance of the instrument and its potential for
identifying disruptive students. These results are be
reported and discussed in Chapter Four.
To estimate convergent validity for the DSBS, the DSBS
rating for each disruptive group member was compared with
the mean DSBS rating of the norming group. For the
purposes of this study, a DSBS rating of at least two
z-scores above the norming group mean was accepted as
evidence that the DSBS had correctly identified a disrup-
tive group member. The standard error of the mean was
used to include students when evaluating borderline cases.
The criterion for satisfactory convergent validity was the
correct identification of 100% of the disruptive group.
Discriminant validity also was estimated by using
ratings, means, and z-scores. The DSBS rating for each
norming group member was compared with the mean of that
group. Any norming group member whose DSBS rating
exceeded the mean by at least two z-scores was considered
identified by the DSBS as excessively disruptive. Identi-
fied cases, not members of or eligible for the disruptive
group, were considered challenges to the discriminant
validity of the DSBS. All cases not meeting the construct
validity criteria were investigated. Construct validity
results are reported and discussed in Chapter Four.
Reliability
The Pearson product-moment correlation statistic was
used to compare the original ratings on approximately 10%
of the completed forms with follow-up ratings made after
14 days. Individual coefficients of at least .80 were set
arbitrarily to establish an acceptable level of test-
retest reliability.
Limitations
1. The school for the field study was selected based on
the willingness to cooperate by both the school and
the faculty. This may have mitigated problems that
would occur in a less favorable environment.
2. Teacher resistance and/or concerns about this type of
research may have biased or limited their partici-
pation.
3. The study was limited to exploration and the results
are not intended to generalize beyond the administra-
tion and scoring procedures. Specifically, the
calculated DSB norm is valid only for this school.
4. No provision was made to assess the possible effects
of grade and sex on DSB norms. Studies have indicated
the influences are not significant, but at some point
this should be investigated.
5. The disruptive sample group was likely composed of
students who had been referred to the dean. The same
teachers who referred these students to the dean may
have rated their behaviors, with bias a possibility.
6. The use of expert judges in the validation procedures
may have introduced personal bias into the items used
on the instrument.
CHAPTER FOUR
RESULTS AND DISCUSSION
The purpose of this study was to develop and validate
an instrument, the Disruptive Student Behavior Scale
(DSBS). The study focused on identifying components of
disruptive school behavior as perceived by middle and
junior high school teachers and constructing an instrument
to quantify these behaviors. To accomplish this, an
instrument was constructed using behaviors taken from
disciplinary referrals and field tested on a representa-
tive sample of students from a Florida middle school.
Teacher ratings for a norm group and a disruptive group
were collected and analyzed as outlined in Chapter Three.
These results are reported in this chapter.
Results
The Severity Factor
Results of the assignment of potential adverse conse-
quences resulting from DSBS behaviors are reported in
Table 2. Twenty packets containing 40 DSBS items and an
instruction sheet were distributed and 16 were returned.
At least 50%, or 8, of the raters had to assign a DSBS item
to a particular domain before that domain was
Tmm 4--- n -j- N MN~ I
I- mj Crt-~m~r0 0I1
=r'-immemminmmewo0 rc'jm
OV E
V) 4-)
> a
0) w
CO
0
H
E
Q)
V0
C.
o
u-I
r_
0
E
a
0
Qi
QI
CMr0o 0e r oo Co
o0 % n M Ch wNOOOON N O OCN ON N 0
S- r r
OO0000-OOOOOONOO-0000
cM -Moo~~b0oom mMO~MM
m01~o- 0 0J '00 '- oC~c 0
y- OJO '-011 ~-O
OOm mO I. 0 0- M C1 V-m M--:o 3 nO iMm -coom
SeMm-c m~00o0 mom ~m
0C m^0m0 ccOci rmo oorO
r-i-r0T- T-t- --CM M 0 0C
nmoom -rwoON
M
cc
*r
E
0
1m
0C
CM
-ID
V -
0
en
E
Vo
|
RSS
