School of Education
University of Wisconsin Milwaukee
The Urban Teacher Selection Interview was developed by identifying functions which discriminate between the behaviors of star teachers and quitter/failure teachers. The number of star teachers (app. 8%) and the number of quitter/failure teachers (app. 40%) in the 120 major urban school districts was both sufficiently large and accessible. Stars were willing to be interviewed because they had been identified on seven criteria as being extraordinary teachers. Quitter/failure teachers were equally willing to be interviewed because they believed that the problems they encountered with students were the fault of the students, their parents or the schools systems. Indeed, they were generally eager to conduct exit interviews and explain in detail the reasons why “good” teachers such as themselves could not continue to teach in these systems.
After identifying the functions which discriminate between the two groups questions were developed which required subjects to respond to whether they would perform these functions and the degree to which they would perform them. In effect, subjects’ answers are assessed in terms of the proximity or distance of their responses from the responses made by stars or quitter/failures. The best answers are closest to how star teachers respond. The poorest answers are closest to how quitter/failure teachers respond.
The interview predicts who will remain in teaching in highly bureaucratic school systems, relate positively to diverse students in poverty and be effective versus those who will quit or fail. This interview is now used in over 220 major urban school districts any of which can provide additional evidence of its predictive validity.
The development of the Star Urban Principal Selection Interview could not be developed following the same procedures as the star teacher interview. Quitter/failure principals are reluctant to be interviewed for a variety of reasons. Some have been told they will get a good reference if they leave the district quietly. Others are promised other jobs in the district if they keep quiet about the causes of their stepping down. Some have been given lucrative buyouts to leave quietly. Some have relatives who still work in the district and fear retribution, while others save face by claiming they were simply taking early retirement or leaving for health reasons. While classroom teachers fail more quietly in their classrooms, principals fail publicly. Because it is more difficult for them to pretend they have been successful they are reluctant to be interviewed. This limits the number of those who would cooperate in the development of an instrument to those who would not be representative of the total group of quitter/failure principals.
The method used to develop the Star Urban Principal Selection therefore drew upon both the written knowledge base and the best practice cited by star principals to explain their effectiveness. In this sense, the star principal methodology was more complete than the process used to develop the star teacher interview.
Research and Development
The development of this questionnaire involved merging the knowledge and research base with the most effective practices of star urban principals. The research and theory base was summarized in the 24 domains of the Knowledge and Skill Base and laid out in Principals for Our Changing Schools published by The National Policy Board for Educational Administration. Star urban principals in three great city school districts were identified: 27 in Houston, 18 in Milwaukee and 84 in Chicago. “Star” principals were invited to participate using the following criteria: achievement scores had risen in their schools for a three year period; they were rated by their faculties as effective instructional leaders; central office personnel identified them as accountable fiscal managers; and parents described them as effective in developing community support for their schools. These stars then engaged in a process of explaining their effective leadership behaviors. They participated in consensus building activities which involved grouping and ranking the performance functions which they believed constituted best practice and which they believed explained their success. The domains of the written knowledge base and the functions performed by the urban principals were then synthesized into eleven functions. This synthesis represents the functions that star urban teachers identified as their effective behaviors which can also supported in the research literature.
Questions designed to assess the eleven functions of star urban principals were then developed to assess this synthesis of research and practice. In order to validate that the content of the questions dealt with the content they purported to be assessing, all the principals of the Milwaukee Public Schools in 2001 (167) were personally interviewed by Prof. Haberman over a period of 53 days. This process established content validity. Respondents, regardless of their level of administrative effectiveness, agreed that the questions dealt with the stated functions. The results of this study indicated that the effective functions cited by star principals which were also supported in the literature were indeed communicating common meanings to respondents. In addition, all question wordings that were ambiguous were clarified or discarded. In an ancillary study, 51 assistant principals were also interviewed. In spite of the fact that assistant principals were typically relegated to disciplinary duties, they identified ten of the eleven functions on the questionnaire as explanations of star principals’ effectiveness.
In addition to establishing content validity, this lengthy, in-depth process also provided a pool of responses to the same questions from principals deemed to be less than satisfactory as well as responses from star principals. (Unsatisfactory or “failure” principals were those with attributes opposite to stars: their schools had declining achievement; they were not regarded as instructional leaders by their faculties; they were identified by central office administrators as “in trouble”; and they were not supported by their parents and communities. These were individuals in the process of retiring, being assigned principal coaches or being moved out of schools and reassigned.)
As a result of these procedures, eleven functions representing sound theory and practice were developed into valid interview questions. Since our studies had included both stars and failure principals’ responses it was also possible to score responses. The scores reflect the degree to which respondents’ answers are closer to those made by star urban principals or to those made by failure principals to the same questions.
These procedures required one year to accomplish. At the conclusion of the year the questionnaire was taken back to the original three groups of star principals in Houston, Chicago and Milwaukee. The numbers of these groups had declined slightly(2 less in Houston, 1 less in Milwaukee and 8 less in Chicago). The star principals were asked to repeat the very same process they had engaged in initially; that is, they engaged in a process of consensus building in which they identified and ranked the behaviors they believed explained their effectiveness. The results of these activities indicated that the behaviors star urban principals had identified the previous year were the same ones they identified a year later. The second finding was that the answers of all the initial respondents’ identified as stars were, in every case, closer to the star respondents identified in the Milwaukee sample than to the responses of the failing principals. The third finding was that the questionnaire could be administered with inter-rater reliability; different interviewers scored respondents answers in the same ways.
In sum, the developmental approach followed here has yielded a questionnaire which synthesizes what the knowledge base indicates makes principals effective and what star urban principals themselves identify as explanations for their success. When this synthesis was replicated one year later it yielded the same explanations of success. The interview questions developed from this synthesis have content validity for both star principals and failure principals. The scoring of respondents is reliable when used by various questioners who have been trained to use the interview..
Individuals have been trained to use the interview in Washington, D.C.; Rochester, N.Y.; Buffalo, N.Y.; San Francisco and numerous smaller cities in Kansas, Missouri, Michigan and Texas. Dallas is in the process of receiving training. All cities report that the quality of the principals that they have hired using the interview has markedly improved. Each city collects its own achievement data and may be contacted for further information.
The Knowledge and Skill Base
The Danforth and the Geraldine Rockefeller Dodge foundations supported a year-long project to summarize the best practice of school principals.The Eli Lilly Foundation also contributed several forms of support and expertise. With the cooperation of all the major professional associations that represent school administrators, the National Policy Board for Educational Administration compiled Principals for Changing Schools: Knowledge and Skill Base. Fairfax,VA: 1992.
The process the group followed was to define the scope of the principal’s role as covering 21 domains of knowledge and skill. Teams of 6-12 academic and practitioner scholars then summarized the knowledge base in each of these domains including critical behaviors to be performed, ways of measuring effectiveness and the supporting bibliography. The domains of the principal’s knowledge and skill base covered were: leadership, information collection, problem analysis, judgment, organizational oversight, implementation, delegation, instruction, curriculum design, guidance, staff development, measurement and evaluation, resources allocation, motivation, interpersonal sensitivity, oral and non-verbal expression, written expression, contextual domains, legal and regulatory applications, policy and political influences, and public relations.
Selecting the Outstanding Principals
The practitioners selected to serve as resources and jurors were drawn from the Milwaukee Public Schools, the Chicago Public Schools and the Houston Independent School District. The method used to identify star principals was as follows. Principals with a minimum of five years of service were evaluated by their superiors on the basis of their demonstrated performance of the following functions: creating a common vision, team building, securing teacher buy in, staff development, instructional leadership, parental and community relations, performing within the contractual obligations of the various unions representing the school staff, implementing innovative programs, budgeting, and garnering resources for their schools. Principals were also asked to evaluate themselves against the same criteria used by their superiors. An anonymous survey was made of assistant principals asking them to assess these same principals on the same criteria. Traditional criteria such as student learning, absenteeism, suspensions, expulsions, and graduation rates were also assessed. All principals identified were leading schools making annual improvement and growth for the preceding five years on these criteria. In addition, some less traditional means were used to select star principals. Surveys of principals identifying colleagues they regarded as stars were used to rank those most frequently named. These surveys were also used to rank principals most frequently cited as being the most desired principal coaches. Finally, supplemental data were collected on those considered as possible stars: these data included, the number of the transfer requests made by teachers in their buildings who had been rated as effective, teacher absenteeism and the degree to which their schools were used after school, weekends and summers. The result of this extensive process yielded 27 star principals in Milwaukee, 36 in Chicago and 18 in Houston.
Developing the Practitioners’ Knowledge Base
This phase of the work, including the identification of star principals described above, covered a period of three years from 1993 through 1996. Daylong workshops of the 81 star principals were held in each of the three cities. The principals engaged in a process of team deliberations which yielded their perceptions of 14 functions they performed that they believed accounted for their success. They also produced specific behaviors which were behavioral manifestations of each of these functions.
Working with a team of doctoral students in the 1993-94 school year, the 21 domains of the research knowledge base in Principles for Changing Schools were merged with the 14 functions identified by outstanding principals as the explanation for their success. The 21 Milwaukee principals serving as a jury agreed that the merged document preserved the intent and the functions of both the literature and the practitioners’ knowledge bases.
The 1994-95 year was spent developing questions that could assess the functions identified. Through an iterative process trial questions were tested for their ability to orally communicate the functions correctly to a variety of constituencies; principals, acting principals and principals in training. Respondents were not asked to answer the questions but were asked to explain what they believed the question was asking them. These trials also produced the prompts interviewers might use to ensure that respondents understood and answered the questions being asked. At the end of these year-long trials the questions and prompts were worded so that 100 per cent of English speaking respondents agreed upon what was being asked in the questions.
During 1996 in-person day-long meetings were again held with the three star principal groups in Milwaukee, Chicago and Houston. Of the 81 original participants 63 were available and participated. Acting independently, the three panels agreed that the functions they had developed earlier were reflected in the questions developed to assess them. The juries also agreed that the behavioral manifestations of each of the functions used on the draft of the questionnaire were accurate reflections of the functions they were intended to assess.
Essentially, the interview seeks to assess what respondents regard as good practice and why they would engage in those practices. In the course of this development it became apparent that the questions were not mutually exclusive. Some of the content used in answering some of the questions are likely to be repeated in answering others. As a result, during this next phase of development three of the questions were identified as redundant and dropped. At this point the interview was developed in its present form: i.e. questions related to eleven functions and requiring approximately one hour to administer.
For the next five years the Star Principals Selection Interview was used in ways that could both assist school districts and assure that the interview would be tested as a valid predictor of principals’ effectiveness. In these trials the interview was used by districts as one criteria of principal selection but not as the determining one. This practice provided an opportunity for the scores of respondents who were hired to be compared with their subsequent performance as practicing principals. This process was used to assess the interview’s power to predict respondents’ behavior from their total scores, as well as from their scores on each of the eleven specific functions assessed by the interview. In other words, if Principal Z was evaluated more highly than Principal B then Z’s total score should be higher. Even more importantly, if Principal B performed more poorly than Z with parents and community, or on creating a common vision among his faculty, his score on those specific questions would have to be lower. These validation trials demonstrated the power of the interview to predict subsequent behavior. At that point school districts began using the interview as a critical or even deciding factor in principal selection.
There is a two day workshop offered for training those who would conduct the interview. It is recommended that at least two and not more than three conduct the interview and no more than two others be present as silent observers. In most cases, the two interviewers are the only ones present. It is also common for interviews to be audio recorded or video taped. Confidentiality and Bias
It is necessary for all those trained to use the interview sign a confidentiality agreement. There is no “train the trainer” model: those trained cannot train others. Once trained, an individual and school district may use the interview as frequently as desired with no further cost or retraining. As a non-profit foundation, the Haberman Educational Foundation uses funds district pay for training to train districts that are bankrupt or under fiscal takeovers which prevent them from paying for training.
There is an on-going problem of individuals who have been trained breaking confidentiality to help colleagues secure positions in districts which use the interview. This practice is engaged in by both school administrators and college faculty. There is also the problem of bias. Frequently, central office administrators know and favor particular individuals and do not score according to the rubrics as trained. The Haberman Educational Foundation provides free consultation to prevent and cut down on breaches of confidentiality.
A Final Test
In 2001 I interviewed all 167 individuals functioning as principals in the Milwaukee Public Schools. This is the only documented instance of an entire principal corps in a major urban school district being interviewed. As a result of this effort the district began a system of principal coaching which is still operating. The particular weaknesses of the principals identified to receive coaching matched their low total scores and their low specific scores on those same functions in the initial interview.
The list of school districts using the Star Urban Principals Selection Interview are available from the Haberman Educational Foundation. Each district can also provide its own data on the predictive value of the instrument.