There are numerous existing methods for measuring grammatical and textual competence (syntax, vocabulary, cohesion, etc.), but no generally accepted measures of the other components of cross-cultural communicative ability — components such as illocutionary and sociolinguistic competence. The purpose of this project is to validate a framework for instruments designed to evaluate such cross-cultural pragmatic ability as reflected in the speech acts of requests, refusals and apologies. One of the primary issues in developing instruments that assess cross-cultural pragmatic ability is associated with the variability of speaker behavior in discourse. Consequently, the study of pragmatic ability inherently involves addressing two contributors to variability in performance: (1) variability associated with the social properties of the speech event and the speaker’s choices for how to achieve communicative goals; and (2) variability due to the particular types of assessment instruments. This project attempts to address both of these issues.
At the present time, prototypes are being developed and evaluated for various test formats in English, Korean and Japanese. These formats include both cued response and free response types for paper and pencil indirect measures, oral semi-direct measures, and self-assessment. Of primary concern is the consistency and validity of all of these measures. Various statistical methods for estimation of test reliability and decision dependability are applied. The focus is on the validation of the test instruments through a multi-method strategy examining the test methods with degree of control in response. The current validation study involves piloting, scoring, revising, and analysing the results of the English and Japanese versions. Each of the six assessment instruments will be administered to subjects in Hawai‘i and Japan.
Technical Report #2 outlining the development process was published in 1992, and in 1995, Technical Report #7 including all instruments in English was published. A Korean version of the test instruments is available as Research Note #13.