Contexts and constructs: Implications for the testing of listening in pilots’ communication with air traffic controllers




ESP Assessment, Aeronautical English, Listening Comprehension, Testing Listening, ICAO


The International Civil Aviation Organization (ICAO) published the Language Proficiency Requirements for pilots and air traffic controllers (ATCOs) in 2003. Research has shown that there is still a lack of clarity regarding what needs to be assessed in terms of the English used by pilots and ATCOs in radiotelephony (DOUGLAS, 2014; EMERY, 2014; KIM; BILLINGTON, 2016; KIM; ELDER, 2015; MONTEIRO, 2019; READ; KNOCH, 2009). The purpose of the present study was to investigate the nature of the listening tasks performed by pilots as an essential step in test development and validation. This explanatory sequential mixed-methods study (CRESWELL, 2015), elicited questionnaire responses from 156 pilots (Phase 1) and subsequently interviewed six aeronautical English experts (Phase 2) to better understand the characteristics of the listening tasks that pilots undertake during radiotelephony communications. Quantitative and qualitative data were analysed, and findings were merged. They provide information that may usefully inform the development of the listening test construct and the test specifications.


Metrics Loading ...

Author Biographies

Angela Carolina de Moraes Garcia, Carleton University

Angela Carolina de Moraes Garcia is in her first year of the PhD program in Applied Linguistics and Discourse Studies at Carleton University. She has been working as a language test developer and raters’ trainer at the Brazilian Civil Aviation National Agency (ANAC) since January 2008. Angela holds a Master’s degree in Language Testing from Lancaster University. Her research interests include English for Specific Purposes and Language Testing, especially aeronautical English testing, validity, reliability, construct definition and rating scale development. E-mail:

Janna Fox, Carleton University

Janna Fox, PhD, is Emeritus Professor of Applied Linguistics, in the School of Linguistics and Language Studies at Carleton University, Ottawa, Canada. Her current research interests include validation theory, language testing, classroom based assessment, and pre-service/in-service teacher development. Email:


ADANK, P.; B. G. EVANS; J. STUART-SMITH; S. K. SCOTT. 2009. Comprehension of familiar and unfamiliar native accents under adverse listening conditions. Journal of Experimental Psychology: Human Perception and Performance, 35.2: 520-529. doi: 10.1037/a0013552

ALDERSON, J. C.; C. CLAPHAM.; D. WALL. 1995. Language Test Construction and Evaluation. Cambridge: Cambridge University Press.

BACHMAN, L.F. 2007. What is the construct? The dialectic of abilities and contexts in defining constructs in language assessment. In FOX, J. et al. (Eds.), Language Testing Reconsidered, pp. 41-71. Ottawa, ON: University of Ottawa Press.

BACHMAN, L.F.; A. S. PALMER. 1996. Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.

________________________________. 2010. Language assessment in practice. Oxford: Oxford University Press.

BARSHI, I. 1997. Effects of linguistic properties and message length on misunderstandings in aviation communication. Doctoral thesis. University of Colorado, Boulder, CO, USA.

BRINDLEY, G. 1998. Assessing listening abilities. Annual Review of Applied Linguistics, 18.1: 171–191. doi: 10.1017/S0267190500003536

BUCK, G. 2001. Assessing Listening. Cambridge: Cambridge University Press.

CRESWELL, J. W. 2015. A Concise Introduction to Mixed Methods Research. Los Angeles, CA: Sage.

_______________. 2014. Research design: Qualitative, quantitative and mixed method approaches. Thousand Oaks, CA. Sage Publications.

CUSHING, S. 1994. Fatal words: Communication clashes and aircraft crashes. University of Chicago Press.

DOUGLAS, D. 2001. Language for specific purposes assessment criteria: Where do they come from? Language Testing, 18.2: 171-185. doi:10.1177/026553220101800204

____________. 2004. Assessing the language of international civil aviation: Issues of validity and impact. Proceedings from the International Professional Communication Conference, IEEE Professional Communication Society, p. 248-252. Minneapolis: IEEE.

____________. 2010. Understanding language testing. London: Hodder Education.

____________. 2014. Nobody seems to speak English here today: Enhancing assessment and training in aviation English. Iranian Journal of Language Teaching Research, 2.2: 1- 12.

EMERY, H. J. 2014. Developments in LSP testing 30 years on? The case of aviation English. Language Assessment Quarterly, 11.2: 198-215. doi: 10.1080/15434303.2014.894516

ESTIVAL, D.; B. R. C. MOLESWORTH. 2009. A Study of EL2 Pilots Radio Communication in the General Aviation Environment. Australian Review of Applied Linguistics, 32: 24.1-24.16.

_____________________________________. 2016. Native English speakers and EL2 pilots: an experimental study. In ESTIVAL, D.; C. FARRIS.; B. MOLESWORTH. Aviation English: A lingua franca for pilots and air traffic controllers. New York: Routledge.

GARCIA, A. C. M. 2015. What do ICAO language proficiency test developers and raters have to say about the ICAO language proficiency requirements 12 years after their publication? A qualitative study exploring experienced professionals’ opinions. Master`s dissertation. Lancaster University, Lancaster, UK.

FIELD, J. 2019. Rethinking the Second Language Listening Test – From Theory to Practice. Equinox.

FOX, J.; ARTEMEVA, N. 2017. From diagnosis toward academic support: Developing a disciplinary, esp-based writing task and rubric to identify the needs of entering undergraduate engineering students. ESP Today, 5.2: 148–171.

HUGHES, A. 2003. Testing for language teachers (2nd ed.). Cambridge: Cambridge University Press.

INTERNATIONAL CIVIL AVIATION ORGANIZATION. 2010. Manual on the Implementation of ICAO Language Proficiency Requirements, 2nd Edition (Doc. 9835). Montreal, Canada: Author.

JACOBY, S.; T. MCNAMARA. 1999. Locating competence. English for Specific Purposes, 18.3: 213–241.

KIM, H.; R. BILLINGTON. 2016. Pronunciation and comprehension in English as a Lingua Franca Communication: Effect of L1 influence in international aviation communication. Applied Linguistics, first published online January 14, 2016. doi:10.1093/applin/amv075

KIM, H.; C. ELDER. 2009. Understanding Aviation English as a lingua franca: Perceptions of Korean aviation personnel. Australian Review of Applied Linguistics, 32.3: 23.1-23.17.

_________________. 2015. Interrogating the construct of aviation English: Feedback from test takers in Korea. Language Testing, 32.2: 129−149.

KNOCH, U. 2009. Collaborating with ESP stakeholders in rating scale validation: the case of the ICAO rating scale. Spaan Fellow Working Papers in Second or Foreign Language Assessment 7: 21-46.

LYNCH, T.; D. MENDELSHON. 2010. Listening. In N. Schmitt (Ed.), An introduction to applied linguistics (2nd ed.), p. 180-196. London: Hodder Arnold.

MESSICK, S. 1996. Validity and washback in language testing. ETS Research Report Series, i–18. doi:10.1002/j.2333-8504.1996.tb01695.x

MONTEIRO, A. L. T. 2019. Reconsidering the measurement of proficiency in pilot and air traffic controller radiotelephony communication: from construct definition to task design. Doctoral thesis. Carleton University, Ottawa, Canada.

O’MALLEY, J. M.; A. CHAMOT; L. KUPPER. 1989. Listening comprehension strategies in second language acquisition. Applied Linguistics, 10: 418–37.

PFEIFFER, A. 2009. Inter-rater reliability in an aviation speaking test. Master’s dissertation. Lancaster University, Lancaster, UK.

PRINZO, O.V. 2009. The ICAO English language proficiency rating scale applied to enroute voice communications of U.S. and foreign pilots. (DOT/FAA/AM‐09/10). Washington D.C., Federal Aviation Administration.

PRINZO, O.V.; T. W. BRITTON. 1993. ATC/Pilot Voice Communications: A Survey of the Literature. (DOT/FAA/AM-93/20). Washington, DC: Federal Aviation Administration.

PRINZO, O.V.; A. M. HENDRIX; R. HENDRIX. 2009. The outcome of ATC message length and complexity on En Route pilot readback performance. DOT/FAA/AM-09/2. Washington, DC: Federal Administration.

READ, J.; U. KNOCH. 2009. Clearing the air: Applied linguistic perspectives on aviation communication. Australian Review of Applied Linguistics, 32.3: 21.1–21.11. doi: 10.2104/aral0921.

RÉVÉSZ, A.; T. BRUNFAUT. 2013. Text Characteristics of Task Input and Difficulty in Second Language Listening Comprehension. Studies in Second Language Acquisition, 35: 31-65. doi:10.1017/S0272263112000678.

SALDAÑA, J. 2013. The coding manual for qualitative researchers qualitative data analysis: a methods sourcebook (2nd ed.). London: Sage Publications.

SCHMIDT-RINEHART, B. 1994. The effects of topic familiarity on second language listening comprehension. Modern Language Journal, 18: 179–89.

SHIMITZU, T.; K. MAKISHIMA; M. YOSHIDA; H. YAMAGISHI. 2002. Effect of background noise on perception of English speech for Japanese listeners. Auris Nasus Larynx, 29: 121–125.

SPOLSKY, B.; SIGURD, B.; SATO, M.; WALKER, E.; ARTERBURN, C. 1968. Preliminary studies in the development of techniques for testing overall second language proficiency. Language Learning, 18: 79–101.

TAYLOR, L.; A. GERANPAYEH. 2011. Assessing listening for academic purposes: Defining and operationalising the test construct. Journal of English for Academic Purposes, 10.2: 89–101.

TIEWTRAKUL, T.; S. R. FLETCHER. 2010. The challenge of regional accents for aviation English language proficiency standards: A study of difficulties in understanding in air traffic control–pilot communications, Ergonomics, 53.2: 229-239. DOI: 10.1080/00140130903470033?

WAGNER, E. 2014. Assessing listening. In A. KUNNAN (Ed.), Companion to language assessment, 1: 47-63. Oxford, UK: Wiley-Blackwell.

WU, Q.; B. MOLESWORTH; D, ESTIVAL. 2019. An investigation into the factors that affect miscommunication between pilots and air traffic controllers in commercial aviation. The International Journal of Aerospace Psychology, p. 53-63.

WU, Y. 1998. What do tests of listening comprehension test? – A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing, 15: 21-44.



2020-10-26 — Updated on 2023-06-12


How to Cite

Garcia, A. C. de M., & Fox, J. (2023). Contexts and constructs: Implications for the testing of listening in pilots’ communication with air traffic controllers. The ESPecialist, 41(4). (Original work published October 26, 2020)