ASM events
This conference is managed by the American Society for Microbiology

Assignment 4: Annotated Bibliography

Table of contents
No headers


Pre-Institute Assignment 4 - Annotated Bibliography – Due July 7, 2009


Building on what others have done – searching for previous research in the SoTL field
1.   Learn where to find published resources relevant to your SoTL project
2.   Identify appropriate key words to use in your searches
3.   Begin to build an annotated bibliography for use in our program
Your assignment:
1.   Find 5 references relevant for your project – either as work you will build on and/or providing technique(s) you are interested in using.  Be sure that you have chosen a range of references from journal articles through digital portfolios or posters.  Be sure that at least two sources were found using an ERIC search.
2.   Annotate these 5 references and tell us why they are useful for your project (see below for more information how to write an annotated bibliography)
We provide at the end of this text a full example from Susan Godfrey (a Scholar from the 2006-2007 cohort).
These annotated bibliographies are due by Tuesday July 7th.  We will compile them into a single document and distribute for all to use in our work. 
Why are you engaging in this detailed literature search?
A significant part of the process of scholarly work is to build on what others have done.  However, the literature of a field can often seem overwhelming and impenetrable when you are new to it, and it seems easier to “do your own thing”.  And it is all too common in teaching to teach by instinct and not make use of the rapidly expanding literature of good research into learning.  That is why we are taking the relative luxury of time available in the summer to let you begin to immerse yourself in the literature.
Where will you find previous research?
For scientists moving to educational research it is not always clear where to look for this information because it is usually not in the databases with which we are familiar, such as PubMed.  Most confusingly, much of the research being done in the SoTL field is represented most richly in a diversity of on-line sources.  Some of the key sites for finding educational research and SoTL research are:
  •                  The articles and books and electronic resources we read in class.  Don’t forget to use research that you have already encountered to help you find new research.  Almost everything that you read was an excerpt from a larger source.  Read more of that text, look in the bibliography, … Use the ‘six degrees of separation’ principle to move in a productive trajectory through the literature.
  •                  ERIC, Educational Resources Information Center.  (  This is a searchable bibliographic database where you will find many of the references important in educational research.  The database is run by the government and is free. There is a special talent to searching using ERIC, and it is described at the end of this document in more detail. 
  •                  PubMed.  Despite what was said above, there are several education-oriented journals published by scientific societies and science education articles within mainstream science journals.  These can be found in PubMed.  You can also go directly to some of the top ones, such as: Life Science Education ( and Journal of Microbiology & Biology Education (
  •                  Various psychology databases (such as PsycINFO).  There is a rich field of educational psychology – the origin of much of the work represented in the How People Learn series of books. 
  •                  Journal of Cognitive and Affective Learning.  A slightly more established journal that also features SoTL research.  ( Past issues still exist but no longer printing new issues.
  •                  Lesson Study.  This site focuses on the use of Lesson Study principles in teaching at the college level, but is has an emerging array of case studies available, too.  (
  •                  Carnegie Foundation.  There is a great richness of SoTL case studies on the Carnegie site.  Be sure to check out:
o                          the Higher Ed work in the CASTL program (
o                          the K-12 work in both the Quest and CASTL programs (  
o                          the galleries (
o                          and the publications section of their website (
  •                  The Visible Knowledge Project.  (  This site contains electronic portfolios from one of the first national SoTL projects involving over 70 professors from institutions across the country.
What is an annotated bibliography and how do you write one?
An annotated bibliography adds to the traditional list of citations a paragraph that provides both a description of the research, a critical evaluation of the quality of the content, and the relevance of the citation to your work.   These annotations are typically ~150 words.  There are many on-line sites that describe annotated bibliographies and how to create them.  I found one of the most helpful to be at the University of Toronto ( 
There is a good example of an annotated bibliography up at the Georgetown Writing Center’s website ( 
You should do your references in APA style.  You can find information about how to cite within the text and list in the bibliography a wide variety of sources in APA style at the Duke library site (
ERIC searches:
As with any database search, knowing the key words that get you what you want is critical.  I will give you a search to do in ERIC that will get you started.  Use the following search terms in the "Advanced Search" option and perform a search with these terms:
  1. Biotechnology
  2. High School
Look at one of the results you got back – should be something like 74 results.
o                          The top line "ERIC #" will start with either ED or EJ.  ED=ERIC document, EJ = ERIC journal.  I personally am more interested in the EJ because these are articles in peer reviewed journals.  ED listings are documents people have placed in ERIC and do not necessarily undergo any review.  Do not ignore ED documents, they may be dissertations, etc. that you think are credible and provide useful information.
o                          "Descriptors" can provide you with ideas of other terms to search with.
If at this point you go back and add “literacy” into your search terms, you can narrow the search down to 5 references.  So keep in mind that this search works well with an array of search term categories (level of education, scientific topic, pedagogical term, etc).
Example from Susan Godfrey (2006-2007 Cohort)
In assembling this beginning annotated bibliography for my project I am focusing on articles that illustrate investigation of the effectiveness of a specific test question style, as that helps me think about issues like hypothesis formulation, relation between test item format and learning objective, and methodology of this specific type of SoTL. The papers I’ve found most useful so far, listed below, were mostly found by finding their citations in other papers I turned up searching in ERIC. I think this is because I don’t yet know many of the relevant “buzzwords” and am just beginning to learn those from my reading, and also because some of the obvious keywords have too many unrelated applications.
My project will have to do with testing whether some atypical (at Pitt) formats I am using for MCQ (multiple choice question) actually test different cognitive skills as defined in Bloom’s Taxonomy (there are many web sources on this taxonomy, I like &, among ones I’ve looked at lately)
1. Williams JB, “Assertion-reason multiple-choice testing as a tool for deep learning: a qualitative analysis”, Assessment & Evaluation in Higher Education 31(3): 287-301, 2006.
This article concerns a type of MCQ in which the student identifies the reason for the correctness or incorrectness of some statement, and gives examples illustrating the format. The author’s context is graduate school of business, and in the study students were using a MCQ survey and also open-ended questions to give their opinions of the ARQ (assertion-reason) style of question. Among the interesting student perceptions is that these questions are difficult because they require having better language skills than do rote memorization type MCQ, a comment I also get from students about conceptually similar MCQ. Having discovered the article in March, I did a limited try of the ARQ format in one of my classes: my students didn’t like it because the format was unfamiliar, and because they didn’t like having only one choice for the “because” aspect of the question (in my alternative format the answers are all possible “becauses”).
2. Wilson RB and Case SM, “Extended matching questions: an alternative to multiple-choice or free-response questions”, J. Veterinary Medical Education 20(3): 7 pages, 1993 (I got this online at
This paper is mainly a description of the “extended matching multiple-choice question” (EMQ) format, showing up more now in medical education references: the format includes a statement of a theme, a list of options (an “extended” list, that is, more than 4 or 5),  a lead-in statement (general enough to set up 2 or more individual question stems), and 2 or more item (question) stems. The items can vary in complexity: in one example set given in the paper the items contain laboratory test results. This method is said to obviate difficulties in choice of distractors, as well as providing different correct answers for different items in the set. It seems to me this method might be quite successful in medical microbiology courses. The authors provide their view of comparative success of this format compared to essay, short answer, and conventional multiple choice questions, which is that the EMQ format may evaluate application of knowledge less well than essay type questions but allows much better coverage of course materials. Another advantage of the format is that the theme and options list could be reused simply by providing different stems.
3. Beullens J, Struyf E, & Van Damme B, “Do extended matching multiple-choice questions measure clinical reasoning?”, Medical Education 39(4): 410-417, 2005.
This article again concerns the EMQ format. The format has been thought effective in testing factual knowledge and the authors wished to find out whether it tests clinical reasoning. As they stated, there is no “good assessment instrument of clinical reasoning” that they consider a “gold standard”. They evaluated the format using oral responses given by students and observed and rated by professionals, comparing a group of medical students to a group of residents, who are presumed to have better developed clinical reasoning skills. Examples of the questions are provided. I can see adapting the format for use in medical microbiology, especially.
4. Deutch CE, “Using data analysis problems in a large general Microbiology course”, American Biology Teacher 59(7): 396-402, 1997.
This paper describes an investigation using a “setup” that describes some simple/simplified results from a technical article, gives one or more data tables from the article, and then a set of 2 to 5 MCQ that relate to that information. The questions use the setup as a context in which to ask for recall of terms or concepts learned in the class generally (type A),  or ask students to show comprehension of data presentation in the table(s) (or figure) (question type B), or ask students to draw conclusions from the data presented (type C). Deutch used such question groups for about 20% of his exam points. I like this format and may adopt it along with the other MCQ formats I favor. Examples of 3 sets of questions are included as an appendix to the paper.
Although I have used individual (not sets of) questions like this, at the moment my interest is in somewhat similar questions except that following the setup the student is asked to choose, from among 5 small data tables, the one that is predicted to result from the experiment described in the setup. The way we’ve used such questions to date makes it likely that students who have been attentive in recitation will be able to answer correctly, so that the ability to form suitable hypotheses predicting expected results may not be tested directly, and I’d like to devise a way to investigate this.
I haven’t so far figured out any way to search ERIC to turn up examples of investigation of MCQ of this type.
5. Gronlund NE, Assessment of Student Achievement, Allyn and Bacon. It’s available in 8th edition 2006 but I’m currently reading the 6th ed. 1998 because the other is checked out of my library.
There is a huge amount of instructional material “out there” on how to write MCQ: I’m appreciating this source at the moment because it addresses the issues of question design, exam planning, grading, and so on in terms that relate directly to my concerns and seem to be more helpful to me than other sources I’ve stumbled across so far.
Tag page
You must login to post a comment.