EdNews.org - The Internets #1 source for Education News and Information  - http://theednews.org
CD-ROM Dictionaries, High Stakes Testing, and the Need for a Vocabulary Question Familiarity-Difficulty Measurement Scale
http://theednews.org/articles/5684/1/CD-ROM-Dictionaries-High-Stakes-Testing-and-the--Need-for-a-Vocabulary-Question-Familiarity-Difficulty-Measurement-Scale/Page1.html
Robert Oliphant Columnist EdNews.org

Robert Oliphant’s best known book is “A Piano for Mrs. Cimino” (Prentice Hall), which was made into an award-winning EMI film (Monte Carlo, US Directors) starring Bette Davis.  His best known work for musical theater (music, lyrics, and libretto) is “Oscar Wilde’s Earnest: A Chamber Opera for Eight Voices and Chorus.”  He has a PhD from Stanford, where he studied medieval lexicography under Herbert Dean Meritt, and  taught there as a visiting professor of English and Linguistics.  He currently serves as executive director of The Alliance for High Speed Recreational Reading, and formerly served as executive director of Californians for Community College Equity.  A resident of Thousand Oaks, CA, and an overseas Air Force veteran, he is an emeritus professor of English at Cal State Northridge.
 
By Robert Oliphant Columnist EdNews.org
Published on 12/18/2006
 

By Robert Oliphant Columnist EdNews.org
Blow as political winds may, the future of English as an internationalist language, especially in science and industry, is linked to high stakes vocabulary testing.It is for professionals, not tourists, that test-administration services like Pearson and Thomson Prometric dot the planet, including China.Nor is it by chance that our own major pre-professional exams (GRE, GMAT, LSAT, and MCAT) as a group devote over half of their examination time to language skills, principally vocabulary power — a field whose standards of achievement are frustratingly whimsical and lacking in authority.


CD-ROM Dictionaries, High Stakes Testing, and the Need for a Vocabulary Question Familiarity-Diffic

By Robert Oliphant Columnist EdNews.org

Blow as political winds may, the future of English as an internationalist language, especially in science and industry, is linked to high stakes vocabulary testing.It is for professionals, not tourists, that test-administration services like Pearson and Thomson Prometric dot the planet, including China.Nor is it by chance that our own major pre-professional exams (GRE, GMAT, LSAT, and MCAT) as a group devote over half of their examination time to language skills, principally vocabulary power — a field whose standards of achievement are frustratingly whimsical and lacking in authority.

Right now our most reliable sources of vocabulary authority are American dictionaries, especially our five major CD-ROM dictionaries: the American Heritage College Dictionary, Webster's New World College Dictionary, Merriam-Webster's Collegiate Dictionary, the Random House Webster's College Dictionary, and the Random House Webster's Unabridged Dictionary. Despite surface differences their word entries all follow the same headword, pronunciation, definition(s) pattern, enough so that any one of these dictionaries could be used to construct a standard by which to measure the relative ease or difficulty of individual vocabulary-test questions.

Headwords and lexical relationships. . . .The practicality of such a measure can be summed up in one word: CONNECTEDNESS. . . . Though presented in alphabetical sequence, the boldfaced headwords in any dictionary are linked together in a complex network that parallelsthe complexity of the English vocabulary as a whole (600,000 words).To learn a new word is to master a myriad of linking relationships in our living English vocabulary (lexicon, as linguists call it) and the dictionary entry which gives that lexicon a physically accessible address.Appendix One therefore draws upon the five most important of these relationships to present a scale for rating the comparative difficulty of vocabulary-based test questions.

Educationally considered, we can now use the five-element formula presented in Appendix One to determine a familiarity/ difficulty rating for ANY of the roughly 200,000 one-word-answer questions (crossword-style or spelling bee) a test designer might construct using the resources of an American "college" dictionary.More ambitiously, we can also reasonably expect our test designers to use such a scaling formula (word by word or sample thereof) to determine the overall familiarity/ difficulty level of any group of crossword-style questions they design for various groups, ranging from third grade up to pre-professional admissions examinations.

Difficulty scales and educational accountability. . . . At a time when the moral authority of high stakes testing has been seriously challenged, the use of a dictionary-based scale like this will be immensely helpful to the testing industry and to educational advocates who see it as a healthy step toward improving the level of modern day educational accountability — worldwide.

Appendix Two transfers our focus from the test designer to the learner.It does this by converting our test-question scale into an all-purpose word familiarity scale via eliminating its one highly specific element, namely, the relative unfamiliarity of the definition offered.

Since it has a dictionary's authority behind it, this four-element scale can be used to produce study lists that are far more understandable to students and parents than those currently in use.Since it has a dictionary's resources behind it (phonetics, definitions, etymologies, etc.) the scale can be used to produce study "addresses,"as opposed to the high cost transfer of dictionary information to other pieces of paper.

Best of all, these two tools open the door to personal-best learning and test-taking preparation at every level of American society, not just formal education.We've already got the dictionaries, the tests, and the study lists.Now, with our CD ROM dictionaries, every American can achieve a high level of vocabulary power on his or her own — measurably so for both the learner and for society as a whole.

TO CONCLUDE. . . . Our two scales should remind educators that a dictionary is fundamentally an inventory of word-relationships, not just an alphabetized list.As every parent knows, the words in a three-year old's vocabulary reflect the frequency with which those words occur at home, in nursery school, and in society as a whole.And as every first grader knows, it's socially essential to "get" a multiple-meaning joke-riddle like "Why did Mickey Mouse take off in a space ship? . . . Answer: He was looking for PLUTO."

Even more important, as every test designer knows, or should know, it's essential to have high stakes vocabulary-test questions whose answers are backed up by a reputable American dictionary, not by an anonymous psychometrist's gut instinct that FAMISHED: HUNGRYis a closer analogical parallel to POUR: DRIP than it is to BREEZE: GALE.If you can't explain it, don't retain it — the old marketing maxim is still worth taking seriously, especially by those whose test designs create escalating levels of anxiety in elementary schoolchildren and their parents.

It's important to emphasize here, I feel, that high stakes testing, including vocabulary testing, can never take the place of credit-diploma education American style, which is still more productive, especially in our graduate schools, than the kind of examination bureaucracies that still bedevil the Europeans.But quite apart from education, the growing internationalist role of English, especially Latinate English, calls for more attention to the unifying role of vocabulary tests, external standards, and dictionaries as an authoritative source of those standards.

When it comes to phonetics and grammar, it's true that the great anthropologist Franz Boas had good reason to say that "there are no primitive languages."But civilization, on the other hand, is clearly a Big Vocabulary (ours is the largest on the planet, measurably so).So our psychometrists and leximetrists are fully justified in testing our command of it.And so is the time we individually spend on learning it — and keeping control of it as we grow older.

A familiarity-difficulty scale for vocabulary test questions and a word familiarity scale for producing word lists that make sense to those who are going to use them — there's nothing overly ambitious about either of these.But therein lies their practicality, I feel. . . . along with their potential usefulness.

APPENDIX ONE. . . . AVocabulary-Question Familiarity/ Difficulty Rating Scale

As far as vocabulary testing goes, it's fair to say that both traditional spelling bees and crossword-style questions are going online.From our own direct experience online, we each know what it's like to be asked a question calling for a single fill-the-blank answer with as many as 20 components: numerals, letters, or an alphanumeric combination.And we also know how quickly our answer can be corrected on a right-or-wrong basis.Usually single words, these answers require more time and thought from us than guesser-friendly multiple choice questions.But for test designers, including non-professionals, the questions themselves are very easy to construct — in large numbers!

By way of illustration:Spelling bee questions and crossword-style questions each ask for a one-word answer on the basis of a single definition-clue.Practically considered, this means that an average-size dictionary with 70,000 head words and 200,000 definitionsrepresents a test-question pool with 200,000 possible questions whose level of test-taking difficulty can be determined on the basis of the following factors.

A headword's frequency of use and its number of definitions. . . . From Rudolph Flesch and Dr. Seuss on, we've known that frequently used words are more familiar to us and therefore easier to spell and to remember than rare-tropical-bird words that catch us by surprise.Even without a special search procedure, it takes only a few minutes to convert an alphabetized spelling list into a frequency-ranked list by checking the number of definitions in the entry for each word in the Random House Unabridged.The fourteen numbered definitions listed for reference, indicates it's more familiar to us than, say, abacus, even though the latter has fewer letters. .

A headword's etymological cross-references and its clarity of structure. . . . We've all learned in school that most words are built up from word elements: prefixes, suffixes, roots, separate words, etc.Practically considered, our clearest indication of these building blocks lies in an entry's capitalized cross references, e.g.,"REFER +-ENCE" for reference.If we add these in, along the RE- and BEAR (an etymological relative of fer) for REFER, we can now state the familiarity rating of reference as 18, not just 14.

Multiple meanings and the ordinal position of a specific definition. . . . Words appear in sentences, not just in dictionaries, and our ability to recognize which meaning of a word applies in the context of a specific sentence or test question is crucially important in understanding what the sentence itself means. The designers of crossword puzzles exploit this multiple-meaning feature, and so do designers of standardized vocabulary and reading-comprehension tests, as in, What does SLOW in the sign "Slow children at play" mean— (a) mentally dull, or (b) slacken in speed?

Since dictionaries customarily position frequently used definitions of a word before less frequently used definitions, the ordinal position (1st, 2nd, 3rd, etc.) of the definition used is a good indication of a vocabulary question's level of difficulty.For the target reference, then, the clue "a statement as to a person's character" (D12) would be less likely to produce a correct answer than the clue "a mention, allusion" (D2) for the target.

As a difficulty factor, especially with crossword-style questions, we can include the ordinal position of the specific definition that appears as a clue in a specific question, e.g., 12 for "a statement as to a person's character" (as opposed to 2 for the definition-clue "a mention, allusion."By including ordinal position as a difficulty factor, we now have a familiarity/ difficulty formula with three elements we can express thus: (overall number of definitions + number of etymological references)/ ordinal position of target definition or 14 +4/ 12 = 1.5. — a lower "tricky crossword" rating for reference than we get with the "easier" 2-value for "a mention, allusion," that is (14 + 4) 2 = 9.

Multi-letter words and question-item difficulty. . . . Where crossword-style questions exploit the multiple meaning feature of our vocabulary, the Scripps National Spelling Bee and the Merriam Webster "Word of the Day" (a web site feature) both specialize in one-definition words like bacciferous ("producing berries") whose unfamiliarity matches up with their number of letters,By way of accommodating technical-term questions like these, then, we can add a word's number of letters (11 for bacciferous) as a difficulty element to our familiarity/ difficulty formula.

Aural memory and spelling inconsistency. . . . Our giant technical-term vocabulary is composed almost exclusively of Latinate English words whose Anglicized pronunciations often clash with their Renaissance Latin spellings.We can assign a numerical value to this difficulty feature by recognizing how many spelling-letters in a word do not reappear in its phonetic transcription.For bacciferous/bak sif"euhr euhs/, the letters are C, C, S, and O, adding up to a numerical value of 4 — a fifth element we can add to our formula.

Using the five-element formula. . . . A full representation of the question-word bacciferous, as we've seen, must recognize the following five elements: (a) number of Definitions (1); (b) number of Etymological cross references (3 — BACC-, -FER, and –OUS); (c) Ordinal position of the definition used as a clue (1); (c) number of letters (11); number of Non-phonetic spelling letters transcription (4). As a familiarity/ difficulty formula we would represent these five elements algebraically as (d+e)/ (o+l+n) or "D-plus-A divided by O-plus-L-plus-M." . . . . For bacciferous specifically, we can represent its familiarity/ difficulty rating numerically as (1+3)/ (1+11+4) = 4/16 = . 25.For test designers, this .25 rating indicates that our bacciferous question will probably be less familiar and more difficult for most test takers than both of our reference-based questions — measurably so.

It also means that ANYONE can use a CD ROM college-level dictionary as a 200,000 item question pool for constructing randomly chosen crossword-style and spelling bee questions whose specific level of difficulty can be determined — even by the test taker — by using this five-element familiarity-difficulty scale.

APPENDIX TWO. . . . Converting the five-element scale into a four-element vocabulary familiarity scale.

The primary obstacle that vocabulary learners face can be summed up in one phrase:ALPHABETICAL TYRANNY. . . . The internet, our book stores, and classrooms are amply stocked with vocabulary lists to help aspiring test takers — all of them beginning with letter A and ending with letter Z.But none of them present them in a familiar-to-difficult learner-friendly sequence that's backed up by a publicly accessible CD ROM dictionary.

By way of illustration, the first three target words in the Webesee web site GRE 1,000-word list are abacus, abate, and abdication.If we eliminate our question-specific ordinal position element, the resulting four-element scale, via the Random House Unabridged CD ROM dictionary will produce the following ranked sequence: abate, 1.0; abdication, 3.6; and abacus, .25.The relatively high position of abdication, incidentally, is produced by its four etymological cross references in RHU, namely, AB-, DICT-, -ATE, and –ION — each of which contributes to the word's familiarity and memorability.

CD ROM dictionaries and lexical variables. . . . From RUN down to SUPERCALIFRAGILISTICEXPIALIDOCIOUS (yes, it's in RHU) and from THE down to MACADEMIA, any CD ROM dictionary represents a total-coverage word familiarity list that is fundamentally superior to word-shape frequency lists like the British National Corpus.Such a list reminds all of us, not just first graders, that a "word" is not a shape, but a relationship between two rather slippery variables: (a) a physical-shape variable (abbreviations, variant pronunciations and spellings, etc.) and (b) a range-of-meaning variable (dictionary definitions, special senses in context like irony, etc.) — both of which are included in linguistic terms like lexicon, lexicology, lexeme, leximetrics (vocabulary tests, etc.), and leximetrist.

Practically considered, there's nothing new about word familiarity lists.The Los Angeles newspaper Opinión uses an informal list for its news reporting, while allowing columnists far more lexicological leeway.Educators as well have used lists like the Dolch K-13 as a basis for textbook style decisions and as vocabulary growth indicators.What is new about this particular list is its authoritative source (the dictionary), its verifiable numerical elements (definitions, etymologies, spelling, non-phonetic letters), and its ranking capability.

Internationalist English dictionaries. . . . A vocabulary question familiarity-difficulty scale and a word familiarity scale — I believe these two tools will be immensely useful as what's now thought of as "American English" becomes more and more an internationalist language, especially its Latinate professional and pre-professional vocabularies.

Twenty years from now, will more CD ROM dictionaries of internationalist English be sold in China than in the United States?Though conjectural and unanswerable, the question is one that makes sense right now — encouragingly so, I feel.