The Connoisseur, The Critic, The Ethnographer and the Evaluator

Qualitative Research and Subjectivity:

Second Discussion of Qualitative Approaches (April 10)

Reading for the Week:

Eisner, E. W. (1994). The forms and functions of educational connoisseurship and educational criticism. In The educational imagination: On the design and evaluation of school programs (3rd ed.) (pp. 212-249). New York: Macmillan.

Eisner, E. W. (1991). Taking a second look: Educational connoisseurship revisited.

In M. W. McLaughlin & D. C. Phillips (Eds.), Ninetieth yearbook of the National Society for the Study of Education, Part II. Evaluation and education: At quarter century (pp. 169-187). Chicago, IL: The University of Chicago Press.

 Hemwall, M. K. (1991). Ethnography as evaluation: Hearing-impaired students in the mainstream. In D. M. Fetterman (Ed.), Ethnography in educational evaluation (pp. 133-152). Beverly Hills, CA: Sage Publications.

Pitman, M. A., & Dobbert, M. L. (1986). The use of explicit anthropological theory in educational evaluation: A case study. In D. M. Fetterman & M. A. Pitman (Eds.), Educational evaluation: Ethnography in theory, practice, and politics (pp. 78-100). Beverly Hills, CA: Sage Publications.

Through the use of the scientific methods, modern natural science has produced thousands of findings that have reshaped the path of modern man, yet man itself remains an enigma. We are yet to fully understand some of the questions posed by the complexity of the human mind and the endless variability of individuals’ perceptions and understandings of phenomena. Since no two people are alike and can be fully comprehended it is misleading and reductionist to reduce us to symbol or a number. If to Eisner the beauty of a simple rock can be studied for a lifetime, can anyone truly understand a human being? A qualitative study of education provides a glimpse into the complexity of the person.

General conclusions are sometimes hard to draw from a qualitative study because they focus on large quantities of information which are particular to a situation, a school, a program, or a person. As more and more information is produced by human being, there is an eagerness, a desire to organize data through algorithms and equations, to find trends and patterns. But when we do so, when we reduce reality, what do we lose in the process?

When reading the papers for this week, these are some of the questions that intrigued me. I hope you also find them interesting. Looking at accepted ethnographic methods, is there a best way in which to conduct educational ethnographies? What is the role of theories in conducting a high quality educational evaluation? Can we generalize from an individual’s auto-ethnography and, from it, gain a better understanding of other people? What is the role of variability? Who is a connoisseur and how important is it to be able to tell a “good” story and explain knowledge in both discursive and non-discursive ways? Finally, given the subjectivity of a qualitative study, can we trust this information, and to what extent is it valid? These are some of the questions that are of importance to a qualitative researcher. This week’s readings discussed some of these concerns, as well as the value of qualitative research to the field of educational evaluations.

The most significant articles discussed this week were Eisner’s (1994) The Forms And Functions Of Educational Connoisseurship And Educational Criticism, where he discusses the concepts of connoisseurship and critics, and Eisner’s (1991) Taking a second look: Educational connoisseurship revisited. Eisner, who both studied art and social sciences, is well versed in the subject of aesthetics and art and, from his work at Stanford, he shares with the reader the importance of understanding a subject beyond its description. He emphasizes the importance of having a good perception of an object so that we can better understand the qualities of a project that would otherwise be lost through variable-level analysis. Eisner (1994) explained how it often takes many years for a person to become a connoisseur, a term referring to an individual with a high level of expertise and an accurate perception of a subject.

From his work as an artist, he believes that connoisseurship is a skill that comes to some more naturally than to others, but that through time and the analysis of subjects to minute detail, individuals can obtain a better appreciation of a subject (be it schools, music, poetry, swimming, etc.). By doing so they are more capable of evaluating a subject and later using that knowledge (transferring) to better understand similar projects. He explains the process by dividing a critic’s task into four components: description, interpretation, evaluation, and thematic.

Due to the difficulty of developing a good perception Eisner argues that time is not the only element that should be considered in becoming a connoisseur. Most individuals working within a field develop a better understanding of the field over time. To some extent teachers, students, administrators, and parents all have a good understanding of a schools, but they likely do not have the fine-tuned perceptions skills of a  connoisseur and the storytelling skills of a critic. As stated by Dewey (1934), recognition is perception aborted (Pg. 324). To Eisner (1994) “connoisseurship is the art of appreciation”, while “criticism is the art of disclosure” (Pg. 215).

Implicitly, Eisner (1994, 1991) argues that to some extent being a critic is akin to being a good storyteller. A good storyteller understand the elements of a good story, and he or she will help you to feel those unspeakable emotions, understand that which is poetic and beyond the scientific. In this sense each critic has a unique voice and it is therefore very important for a critic to use his or her powers of persuasion in honesty and truthfulness, according to his or her perception. “Criticism is the art of disclosing the qualities of events or objects that connoisseurship perceives” (Pg. 219).

In illustrating how critics address validity questions, Eisner (1994) mentions how critics are like lawyers putting the pieces of a case together in an attempt to convince their jurors. The attorneys, through a process of placing all of the pieces in order, also known as “structural corroboration”, are able to convince the court of their version of events. However, because of the co-constructed nature of knowledge, and the current psychological understanding that we recreate events when we remember events (changing elements of our stories), Eisner (1994) also acknowledges that there may be multiple versions of events and, depending on how divided different possible “structural corroborations” are, this method may not be sufficient to ascertain the validity of the critique. When thinking of connoisseurs he mentions how critics mainly agree on most elements (something somewhat true if we look at Rotten Tomatoes). Yet, if we consider Supreme Court justices as connoisseurs then there are clear examples where experts in a field (and it would be hard to argue that the Justices are not connoisseurs and critics of their field), after deliberating and deconstructing cases may disagree and vote 5 to 4. With this example, can we then argue that structural corroboration may not be enough to increase validity?

The other element Eisner (1994) discusses for increasing validity is referential adequacy. This refers to the process of “testing the criticism against the phenomena it seeks to describe, interpret and evaluate” (Pg 241). This process is sometimes done by going back to the site of study or the data set and testing the findings with a portion of the data that is part of the larger data set but a part that was not previously analyzed. If the framework or the analysis previously developed also applies, then the critique in question would increase in validity.

Another topic discussed by Eisner (1994) is the generalizability of critiques. While some qualitative researchers emphasize transferability over generalizability, Eisner (1991, 1994) argues that some form of generalizability is possible. Unlike a mathematical understanding based on laws, theorems and axioms, according to Eisner (1994) qualitative studies can be generalized as they allow the critic to apply the skills learned from becoming a connoisseur to other related subjects as one is able to appreciate them with a better or improved perception. Eisner (1994) discusses how generalization is also possible by the acquisition of “new forms of anticipation” (Pg. 242). By understanding the complexities of a particular situation, we can then better understand other situations. According to Eisner’s definition, generalizability of critiques appears to be internal (within the critic) rather than external (within his work), as it is the critic alone who can critique according to his own subjectivity.

Another critic may have a similar opinion about a subject but it will be somewhat different subjectively and in its critique. While Eisner argues that critics generally agree about broad conclusions, would they agree with all of the words in a descriptive paragraph that discusses Paul Brach pictures? This is unlikely, as it is improbable that two individuals, two connoisseurs, will criticize a subject in the same manner. While each analysis may have its own strengths, complications when generalizing the data raise questions about the claims of Pittman and Dobbert (1986) regarding the design of a “true” ethnographic educational evaluation.

Their use of the word “true”, is, in my opinion, somewhat controversial even taking into account the use of quotations, as it implies that there is possibly a best or an ideal form of applying anthropological theory in educational evaluation. It seems as if in this way we get closer to fully deciphering the true figure of the shadow emanating from the cave. It can be argued that Pitman and Dobbert (1986) have a more objective view of reality than other ethnographers. However, despite this concern, their inclusion and valuation of theory as an essential element in educational evaluation raises some important questions. In including theory as a primary element and illustrating its benefits effectively in a teacher training program in a day care center, the authors strengthened their claims. Yet, they also illustrated some major limitations.

The main critique to their design is discussed by Pitman and Dobbert (1986) as they state that “a traditional (i.e. Mallnowskian) approach was not possible” (Pg. 79) and explain that they were only able to study the school during a month long study. Yet because of the rigor of their evaluation, and each evaluator having spent 120 hours per person for around a month (April 24 to May 22) conducting the study, the authors have a strong claim that the study did include the level of quality required of an ethnographic evaluation.

The authors emphasize the importance of the explicit use of theory in conducting education evaluations. They illustrate their case by citing the importance of the literature review in noticing patterns and comparing and contrasting the study findings to various theories including symbolic interactionism, cognitive anthropological approaches, rites of passage theory, small group theory, gearing notion of old hands and new hands in transaction, role theory, among others. The value of these theories in increasing the validity of the study was illustrated in various examples including how “relying on Homans’s and Herzberg’s theories, [they] recommended that the center take an additional step to create positive student perceptions by distributing a handout that would provide solid information about its policies and philosophies.” (Pg. 94) The article contains various instances in which theory informed the findings, increasing their external validity.

Having a team of 6 members also increased the rigor of their evaluation. The steps taken through planning included testing a procedural outline, identifying and refining a manageable team evaluation process, and lobbying for the increased use of qualitative methodology. (They do not share their value premises for these and other decisions in their study). By conducting secondary, and sometimes triple interviews, as well as dividing into subgroups to analyze the data and conduct a pattern analysis, the study increases the validity of their findings, and potential impact (transferability and generalizability) of their conclusions. The study also had a high level of utility as the recommendations of the study were mostly implemented soon afterwards, likely improving the program’s outlook for future years.

The findings of the study have been reinforced by other studies, with limited time available for instructors being one of the most common findings. Despite the best attempts of the lead teachers to serve as mentors and guides to the practicum students, they had limited time to communicate effectively with the students and address their concerns. One of the biggest concerns expressed by practicum students was the lack of awareness of how they were being evaluated and whether or not they were being seen as successful in completing their tasks. The teachers, on the other hand, argued that while not all students were willing to listen to their advice, some of them were very attentive to comments and criticisms. However, despite some of the differences of opinion, the students seemed to agree that “the lead teachers were very responsible and related well with children and their needs” (pg 88). While the authors argued that all students agreed with this statement, the individual’s political position is not discussed as it in Hemwall’s (1991) study.

Pitman and Dobbert’s (1986) article was very helpful in illustrating how a team of researchers can conduct an in-depth study within a short time frame. Yet is it possible for most evaluations to assemble a team and develop an extensive system for member checking and triangulation? In addition, in the end of the paper, the authors argue that “in this chapter we have sketched the distinguishing characteristics of “true” ethnography versus other kinds of fieldwork” (Pg. 97). Despite discussing their methodology as a “true” ethnography, they do not include many of the essential elements of a feminist ethnography or a critical ethnography. This ethnography is one out of many possible types of ethnographic research, being “true” only to the extent that it is perceived as “true” by the author and its reader.

Hemwall (1991) was the fourth reading for the week and it discussed the mainstreaming or integration of a group of hearing impaired students, a program which is formatively evaluated by the use of classroom observations, short structured interviews, and distributing questionnaires. Similar to Pitman and Dobbert (1986), this study also took only a few days, which leads the researcher to question the richness of the study. Hemwall (1991) provides extensive examples of how, despite the interest in increasing the hearing impaired student’s mainstreaming, many of them had yet to be fully accepted by their peers, break their dependency on each other and on the interpreters, and to develop a relationship that was similar to that of other teacher-instructor relationships. In her discussion of events, Hemwall saw the attachment of interpreters to students as a negative change which broke protocol. In addition, students who were not hearing impaired felt their hearing impaired classmates were being treated in a beneficial way by the instructor. “Aw, those guys just get away with murder!” (Pg. 141).

Hemwall (1991) cites examples of when the teacher spoke directly to the interpreter instead of trying to learn from the student in question. The student expressed frustration and decided not to ask a question to the instructor. An element in this paper that was missing to me was a discussion of the author’s opinions about mainstreaming, or her value premises. Hemwall (1991) also argued for the breaking up of the group of hearing impaired students, so that they could better relate to their non-hearing impaired classmates. The researcher feared that “the students in these groups had become so dependent on each other that the groups isolated themselves, thus preventing further integration.” (Pg 145). Yet her argument does not discuss how social integration of all students in high school can be difficult for many students, not only students that are hearing impaired. Her finding also seems to illustrate that many students felt marginalized by their peers.

The researcher concluded by providing a list of recommendations including: skill tutoring, establishment of better support systems, which would be ideally “a deaf adult who was previously mainstreamed” (pg 154), more informal academic and social experiences, separating the groups of hearing-impaired student. She highlighted the need for intensive in-servicing and training available for everyone, and lastly special training for interpreters, which despite its importance had been difficult to implement. To finalize, the article highlights the strengths of ethnographic research as well as its weaknesses. The three main weakness of the study were (1) obtaining access to the classrooms to observe; (2) the careful formulation of the results of the ethnographic evaluation to avoid any unintended consequences; and (3) a discussion as to how to adequately protect the informants.

To conclude, this article provides a close and detailed look about the experiences of mainstreaming students in this school while also raising questions about generalizability and transferability, as well as the subjectivity of the critic. One of my concerns when reading this study is its lack of a discussion of what is considered “normal”? Or is there such thing as a “normal” child? Is “normal” an objective term, or rather another social construct that is culturally dependent? The article is effective in explaining, as mentioned by the PE teacher, that hearing impaired students are as diverse a group as other students; “handicapped kids run the gamut like hearing kids… some brats, some good kids, some smart, some slow”.  This is important to highlight, but also that “normal” itself may be a misconception. Students have different learning modalities and living experiences, many of which are unique. While debating what is “normal” is beyond the extent of this paper, it is important to discuss it as a social construct and subjective concept.

Questions for Class Discussion:

1 – Would you consider Supreme Court justices to be connoisseurs and critics? Why or why not?

2 – Can a qualitative study be reproduced?

3 – Are aggregators of critics’ opinions better than a critic’s opinion?

4 – Is there an objective truth? Are there infinite ways of seeing things?

5 – Do you consider yourself to be normal? Why or why not?

6 – In Outliers, Gladwell argues that it takes 10,000 hours for someone to master a subject (with regular critical feedback), how is this related to Eisner’s concept of the connoisseur?

7 – What comes to mind when you think of an artistic teacher?

8 – Is a test fair (is testing fair)? What is fairer; a critic’s opinion, or a test?

9 – Is there a “true” ethnography? What are the components of a “true” ethnography?

10 – What is “Contemporary Time”? (Pittman and Dobbert)

11 – Three Cups of Tea’s author was recently involved in a controversy. His story, however, had been very captivating and felt “real”, leading to millions of dollars in donations. With this in mind, what is the responsibility, if any, of the storyteller to explain his experiences honestly?

12 – Are the 10 weeks mentioned in Pitman Dobbert sufficient time for an ethnographic study?

13 – Do using different qualitative methodologies lead to the same or different results? Would different experts yield different results with the same methodologies?