In evaluating published research and opinion, two separate processes must take place. First, you must decide what you expect a particular work to accomplish for you. Then you must evaluate the work in hand and decide how well it accomplishes the desired task. To define your expectations, consider the following questions.
- What do you want the literature to do for you?
a. Familiarize you with a subject about which you know almost nothing?
b. Serve as evidence and “building blocks” for your argument?
c. Provide you with fresh approaches to a topic already familiar to you?
d. Tell you the latest developments in a field of scholarship you are already familiar with?
2. How much do you already know about this subject?
a. What general idea do you have of the subject?
b. Are you familiar with the names of prominent scholars and schools of thought?
c. Do you have definite opinions as to right and wrong in reference to this subject?
This should result in a statement of what you want the published research or opinion to do for you or tell you.
Now that you have an idea of what you want to accomplish, this guide in intended to provide you with a detailed list of questions for critically evaluating a published work, or published works of a particular discipline. Many of the points raised here may seem superfluous and too obvious to deserve mentioning, while other points may seem too detailed and probing to be done for every published work you may look at. Use your own judgment as to how much of this guide you wish to use at any one time — think of this guide as an idealized model for the type of evaluation you should be doing. By providing you with this framework, we hope to get you into the habit of considering all of these points when looking at an author’s work, and also when evaluating your own research.
The following questions will provide you with the basic information you need to identify your published work and make a preliminary decision as to its value to your research project. By skimming through a work and answering these questions, you should be able to judge whether or not you should read it in depth.
A. Becoming familiar with a published work.
- What is the full title and year of publication?
a. Is the information current and timely? Or has it been superseded by more current works, and is now only useful as historical background material?
b. Is current information necessary for your research? Or would a somewhat older work containing a more basic explanation of the material be more appropriate?
c. Do you need “current events”-type information from a particular historical period? Does this material contain that?
2. What type of publication is this?
a. Is this an encyclopedia article, intended to present a background summary of the topic?
b. Is this a book, covering a single subject in great depth?
c. Is this a chapter in a collection of essays or reports, all giving different perspectives and opinions on the same general topic?
d. Is this a journal article, giving in-depth research on a narrowly focused topic?
e. Is this a review article, bringing a range of new ideas together and integrating them into the mainstream of thought within the discipline?
f. Is this an editorial or opinion piece, giving one person’s or group’s opinion on a topic of which they are knowledgeable?
g. Is this a news report, attempting to convey only factual information without adding editorial interpretation?
3. If this is a journal article, what journal is it published in?
a. Is this a scholarly journal, publishing refereed original research?
b. Is this a trade journal, containing news for professionals in the field?
c. Is this a popular magazine, with stories suitable for the general public?
4. Who are the author(s) and publisher?
a. Who is the author?
b. What is his/her occupation, position, education, experience, etc.?
c. Based on the above, is the author qualified (or not) to write on this topic?
d. Who publishes the work?
e. What type(s) of material does the publisher normally produce?
f. From what you know about the publisher, what can you infer about the quality of this work?
5. What is included in the table of contents or headings of this work?
a. Is there a listing of key words that the author feels are important in describing the work?
b. Does this information seem to promise that this piece is indeed about your subject?
6. Are there any features, such as an index, bibliography, glossary, illustrations, etc., included?
a. Are there any significant attachments or appendices, such as charts, maps, photos, documents, tests or questionnaires also included?
The results from your work in this section should be:
1. A detailed citation for the work, with all the information needed for your bibliography, and
2. A biographical summary on the author.
The section above was primarily concerned with the physical identification of the published work and its author. The following questions will deal more with identifying the piece and its intellectual context.
B. Becoming familiar with the purpose of a published work.
- What is the intention and scope of the work, as indicated by the title, introduction, or abstract?
a. What is the purpose for writing the article or doing the research?
2. Is the language and approach suitable for my level of research?
a. Who is the intended audience that the author is writing for?
b. Is it intended for the general public, for scholars, policy makers, teachers, professionals, practitioners, etc.?
c. How is this reflected in the author’s style of writing or presentation?
d. Do I think I understand the author’s language and style?
e. Has the piece been reviewed, via a peer-review process?
3. Does this work appear to address my questions?
4. Is the conclusion relevant to my interest?
The results from your work in this section should be:
- An overview (outline or list) of areas covered by this work, and
2. An evaluation of how the author approaches these areas.
The following questions are designed to examine the intellectual structure and methodology of the work in some depth. They seek to answer the question: “What is the author trying to accomplish?”
C. Understanding a published work.
- What is the author’s thesis?
a. Is the thesis clearly stated in the introduction or expressed clearly in the title or abstract?
2. What are the key concepts covered?
3. What is the research methodology used?
a. Is the article (or book) based on personal opinion or experience, critical rational analysis, common sense argument, interviews, library research, questionnaires, laboratory experiments, standardized personality tests, etc.?
b. Is this a standard method of obtaining data/information in the field? If not, does the author clearly state the rationale behind the method?
4. What are the findings and conclusions of the author?
a. Does the author satisfactorily justify the conclusion from the research or experience? Why or why not?
The results from your work in this section should be:
1. A list of key terms, concepts, data, and conclusions about the work, and
2. A summary or abstract of the work.
This section will assist in analyzing how well the author actually accomplishes what has been attempted, particularly in reference to the structure and language used in the work.
D. Analyzing the structure of a published work.
- What are the author’s frame of reference and intentions?
a. Is the author associated with a particular group of researchers?
2. What viewpoints or biases are evident?
a. Does the author have a bias or make assumptions upon which the rationale of the article or the research rests?
b. What are they?
3. Is the tone of the work appropriate?
a. Is the language overly scholarly or overly casual for the intended audience?
4. Are the structure, parts and transitions clear?
5. Is the writing of the piece coherent?
6. What patterns of thought emerge?
a. Does the author lead the reader from accepted fact to new theory in a logical manner?
7. Are the sources current and reliable?
a. Has the author cited the sources accurately? Or are the quotes taken out of context incorrectly?
8. Do the conclusions follow from the data?
9. Is there consistency within the piece and with other materials in the field?
The results from your work in this section should be responses to:
- the author’s viewpoint as presented in the work;
2. the approach he used in presenting this;
3. the style of research and writing in the piece; and
4. the results presented therein.
This section will help you bring together your reactions to the work you have been examining, and draw some conclusions about its place in the field and utility for your research. No one can come to any published work in their field without some preconceived ideas as to what is “right”. These questions will help you understand your expectations about the work, and let you judge how well the piece fulfills those expectations.
E. Assessing the usefulness of the content of a published work.
- What are your own biases on this subject and your expectations for this work?
2. Did the author fulfill the stated goals?
3. Do you feel confident in challenging points in this work?
4. Does the piece make a contribution to the literature?
a. How does this study compare with similar studies?
b. Is it in tune with or in opposition to conventional wisdom, established scholarship, professional practice, government policy, etc.?
c. Are there specific studies, writings, schools of thought, philosophies, etc… with which this one agrees or disagrees and that one should be aware of?
5. Does the piece make a contribution to your research?
The results from your work in this section should be:
- An evaluative statement about the work and its usefulness to your research (i.e., an annotation).
If you are interested in reading more about the critical evaluation of sources, you may want to look at these sites: