Researching Visual Semiotics Online

Authors

DOI:

https://doi.org/10.23925/1984-3585.2020i21p116-145

Keywords:

Mixed-methods research, Google Image search, Visual content analysis, Semiotic theory, Semiotic methods, Peircean semiotics

Abstract

Analyzing visual meaning online and curating digitized images are topics of increasing relevance, but many potential methodologies for doing so remain merely implicit, underthematized, or unexplored. The potential for testing and developing semiotic theory through the exploration of visual data online also requires far more careful attention. In response, this paper provides an integrated, reflexive, Peircean account of two case studies featuring research projects focused on visual data drawn primarily from sources online, relying heavily on Google Image Search as a data collection tool. The first study illustrates the comparative analysis of brand mark logos to test and refine a theory of embodied semiotics involving oppositional relations. The second study illustrates the comparative analysis of images depicting the Tibetan Wheel of Life and Yama the monster of death, in order to test the embodied grounding hypothesis for the semiotic square. Issues of hypothesis formation, research parameters, data collection, database construction, operationalization, coding parameters, open data archiving and related issues are addressed in order to further develop and encourage practices of researching visual semiotics online in the context of digital humanities scholarship.

Author Biography

Jamin Pelkey, Ryerson University, Ontario, Toronto, Canada

Dr. Jamin Pelkey is Associate Professor in the Department of Languages, Literatures and Cultures and a core member of the Ryerson-York graduate program in Communication & Culture. He joined the department in 2013 following five years of university lecturing in British Columbia and more than a decade of field experience in the eastern hemisphere--including China, Thailand and Australia. Dr. Pelkey holds a PhD in Linguistics from La Trobe University, Melbourne, funded in full and awarded with merit in 2009. His areas of specialization include philosophy of language, embodied cognition, Tibeto-Burman linguistics, historical linguistics, C. S. Peirce, and semiotics. He is an award winning teacher and researcher with two federal grants, including “Steps to a Grammar of Embodied Symmetry” (SSHRC-IDG #430-2015-01226), two Dean's Awards—for teaching (2015) and research (2018)—the Mouton d’Or Award for best article in Semiotica (2017) and the Ryerson University Early Research Career Excellence Award (2018). He is currently editing Bloomsbury Semiotics, a major reference work in four volumes.

References

ATSUMI, Keisuke; KOIZUMI, Itsuro. Web image search revealed large-scale variations in breeding season and nuptial coloration in a mutually ornamented fish, Tribolodon hakonensis. Ecological Research, v. 32, n. 4, p. 567–578, 2017. DOI: 10.1007/s11284-017-1466-z.

BANDYOPADHYAY, Shreya; GHOSH, Kapil; DE, Sunil Kumar. A proposed method of bank erosion vulnerability zonation and its application on the River Haora, Tripura, India. Geomorphology, v. 224, p. 111–121, 2014. DOI: 10.1016/j.geomorph.2014.07.018.

BARTHES, Roland. Elements of semiology. New York: Hill and Wang, 1964.

BARTHES, Roland. Mythologies. New York: Hill and Wang, 1972.

BERNSTEIN, Richard J. Beyond objectivism and relativism: Science, hermeneutics and praxis. Philadelphia: University of Pennsylvania Press, 1983.

BELL, Philip. Content analysis of visual images. In: VAN LEEUWEN, Theo; JEWITT, Carey (eds.). The handbook of visual analysis. London: SAGE, 2004. p. 10–34. DOI: 10.4135/9780857020062.n2.

BOJE, D.; SMITH, R. Re-storying and visualizing the changing entrepreneurial identities of Bill Gates and Richard Branson. Culture and organization, v. 16, 2010, p. 307–331.

BOWDEN, Calida; SHEEHAN, Athena; FOUREUR, Maralyn. Birth room images: What they tell us about childbirth. A discourse analysis of birth rooms in developed countries. Midwifery, v. 35, p. 71–77, 2016. DOI: 10.1016/j.midw.2016.02.003.

CARA, Mariane. The Semiotic layers of Instagram: Visual tropes and brand meaning. American Journal of Semiotics, v. 34, n. 3–4, p. 331–352, 2018. DOI: doi.org/10.5840/ajs201931146.

CHATTERJEE, Karishma. What can we learn about the female condom online? An analysis of visual representations of the female condom on the Internet. Atlantic Journal of Communication, v. 26, n. 3, p. 149–163, 2018. DOI: 10.1080/15456870.2018.1474217.

COBLEY, Paul. Cultural implications of biosemiotics. Cham: Springer, 2016.

CRUNDWELL, G.; HARMER, J.; MALTBY, M.; MILLS, T.; NEUMANN, C.; WALSH, L.; BAGULEY, D. Images of otoscopy: rate and extent of non-compliance with good practice standards. Journal of Laryngology & Otology, v. 129, n. 1, p. 27–31, 2015. DOI: 10.1017/S0022215114003004.

DOLATABADI, Hadi; TARI, Zeinab Ghasemi. A visual framing analysis of French and US political cartoons on Trump’s withdrawal from the Iran nuclear deal. Journal of World Sociopolitical Studies, Teheran, v. 3, n. 3, p. 605–647, 2019. DOI: http://dx.doi.org.ezproxy.lib.ryerson.ca/10.22059/WSPS.2020.289255.1119.

GLASER, Barney G.; STRAUSS, Anselm L. Discovery of grounded theory: strategies for qualitative research. New Brunswick, N.J.: Aldine, 1967.

GROENEWALD, Thomas. A phenomenological research design illustrated. International Journal of Qualitative Methods, v. 3, n. 1, p. 42–55, 2004. DOI: 10.1177/160940690400300104.

GRBICH, Carol. Qualitative data analysis: an introduction, 2nd ed. London : Thousand Oaks, 2012.

GREIMAS, A. J. On meaning: selected writings in semiotic theory. Minneapolis, MN: University of Minnesota Press, 1987.

HÉBERT, Louis. An introduction to applied semiotics: tools for text and image analysis. New York: Routledge, 2020.

HUNTER, William Cannon. The visual representation of border tourism: Demilitarized zone (DMZ) and Dokdo in South Korea. International Journal of Tourism Research, v. 17, n. 2, p. 151–160, 2015. DOI: 10.1002/jtr.1973.

HUSSAIN, Aabid; GUL, Sumeer; SHAH, Tariq Ahmad; SHUEB, Sheikh. Retrieval effectiveness of image search engines. The Electronic Library, v. 37, n. 1, p. 173–184, 2019. DOI: 10.1108/EL-07-2018-0142.

KIM, Kwanghyun; HONG, Sungjun; CHOI, Baehoon; KIM, Euntai. Probabilistic ship detection and classification using deep learning. Applied Sciences, Basel, v. 8, n. 6, 2018. DOI: http://dx.doi.org.ezproxy.lib.ryerson.ca/10.3390/app8060936.

KRESS, Gunther R. Multimodality: a social semiotic approach to contemporary communication. London: Taylor & Francis, 2010.

KRESS, Gunther R.; VAN LEEUWEN, Theo. Reading images: the grammar of visual design. 2nd ed. London: Routledge, 2006.

KRIPPENDORFF, Klaus. Content analysis: an introduction to its methodology. London: Sage, 2004.

KUCUK, S. Umit. A semiotic analysis of consumer-generated antibranding. Marketing Theory, v. 15, n. 2, p. 243–264, 2015. DOI: 10.1177/1470593114540677.

LEIGHTON, Gabriella R. M.; HUGO, Pierre S.; ROULIN, Alexandre; AMAR, Arjun. Just Google it: assessing the use of Google Images to describe geographical variation in visible traits of organisms. Methods in Ecology and Evolution, v. 7, n. 9, p. 1060–1070, 2016. DOI: 10.1111/2041-210X.12562.

LIU, Xi; SHI, Zhi-Ping; SHI, Zhong-Zhi. A co-boost framework for learning object categories from Google Images with 1st and 2nd order features. The Visual Computer, v. 30, n. 1, p. 5–17, 2014. DOI: 10.1007/s00371-012-0772-2.

LIVINGSTONE, Devon; CHAU, Justin. Otoscopic diagnosis using computer vision: An automated machine learning approach. The Laryngoscope, v. 130, n. 6, p. 1408–1413, 2020. DOI: 10.1002/lary.28292.

LOBINGER, Katharina. Visual research methods. In: The International Encyclopedia of Communication Research Methods. American Cancer Society, p. 1–10, 2017. DOI: 10.1002/9781118901731.iecrm0265.

LORENCIN, Ivan; ANĐELIĆ, Nikola; MRZLJAK, Vedran; CAR, Zlatan. Marine objects recognition using convolutional neural networks. Nase More, Dubrovnik, v. 66, n. 3, p. 112–119, 2019. DOI: http://dx.doi.org.ezproxy.lib.ryerson.ca/10.17818/NM/2019/3.3.

MANERI, Marcello. Breaking the race taboo in a besieged Europe: How photographs of the “refugee crisis” reproduce racialized hierarchy. Ethnic and Racial Studies, p. 1–17, 2020. DOI: 10.1080/01419870.2020.1723672.

MARGOLIS, Eric; PAUWELS, Luc (eds.). The SAGE handbook of visual research methods. London: SAGE, 2011.

MARSDEN, Jamie; THOMAS, Briony. Brand values: Exploring the associations of symmetry within financial brand marks. Design Management Journal, v. 8, n. 1, p. 62–71, 2013. DOI: http://dx.doi.org/10.1111/dmj.12004.

MARSHALL, Zack; BRUNGER, Fern; WELCH, Vivian; ASGHARI, Shabnam; KAPOSY, Chris. Open availability of patient medical photographs in google images search results: Cross-sectional study of transgender research. Journal of Medical Internet Research, v. 20, n. 2, p. e70, 2018. DOI: 10.2196/jmir.8787.

MIKULA, Peter; HADRAVA, Jiří; ALBRECHT, Tomáš; TRYJANOWSKI, Piotr. Large-scale assessment of commensalistic-mutualistic associations between African birds and herbivorous mammals using internet photos. PeerJ, 2018. DOI: http://dx.doi.org.ezproxy.lib.ryerson.ca/10.7717/peerj.4520.

NET MARKET SHARE (NMS), 2020. Search engine market share. Net MarketShare. https://netmarketshare.com/search-engine-market-share.

NÖTH, Winfried. Semiotic foundations of natural linguistics and diagrammatic iconicity. In: WILLEMS, Klaas; DE CUYPERE, Ludovic (orgs.). Naturalness and iconicity in language. Amsterdam: Benjamins, 2008. p. 73–100.

O’HALLORAN, Kay. Multimodal digital humanities, In: TRIFONAS, Peter Pericles (ed.), International handbook of semiotics, v. 1. Cham: Springer, 2015. p. 389–415.

O’HALLORAN, Kay; CHUA, Alvin; PODLASOV, Alexey. The role of images in social media analytics: A multimodal digital humanities approach. In: Visual communication. Berlin: de Gruyter, 2014. p. 565–588.

O’HALLORAN, Kay L.; E, Marissa K. L.; PODLASOV, Alexey; TAN, Sabine. Multimodal digital semiotics: the interaction of language with other resources. Text & Talk, v. 33, n. 4–5, p. 665–690, 2013. DOI: 10.1515/text-2013-0030.

PEIRCE, Charles S. The collected papers of Charles Sanders Peirce, v. 1–6, HARTSHORNE, C.; WEISS P. (eds.) (1931–1935); v. 7–8, BURKS, A. (ed.) (1958). Cambridge, MA: Harvard University Press. (Cited as CP followed by volume number and section number.)

PEIRCE, Charles S. The essential Peirce, v. 2, PEIRCE EDITION PROJECT (ed.). Bloomington, IN: Indiana University Press, 1998. (Cited as EP 2 followed by page number.)

PEIRCE, Charles S. Historical perspectives on Peirce’s logic of science: a history of science. EISELE, Carolyn (ed.), 2 vols. Berlin: Mouton, 1985. (Cited as HP followed by volume and page number.)

PELKEY, Jamin. Greimas embodied: How kinesthetic opposition grounds the semiotic square. Semiotica, n. 214, p. 277–305, 2017a.

PELKEY, Jamin. The semiotics of X: Chiasmus, cognition, and extreme body memory. London: Bloomsbury Academic, 2017b.

PRIDGEN, B. C.; JOPLING, J. K.; SHECKTER, C. C.; DESPO, O.; YEUNG, S.; KARANAS, Y.; LI, F.; MILSTEIN, A. S. 33 Automated burn assessment using deep learning and computer vision. Journal of Burn Care & Research, v. 40, n. Suppl. 1, p. S25–S26, 2019. DOI: 10.1093/jbcr/irz013.037.

PRITCHARD, Katrina. Examining web images: a combined visual analysis (CVA) Approach. European Management Review, v. 17, n. 1, p. 297–310, 2020. DOI: 10.1111/emre.12376.

RODRIGUEZ, Lulu; ASORO, Ruby Lynn. Visual representations of genetic engineering and genetically modified organisms in the online media. Visual Communication Quarterly, v. 19, n. 4, p. 232–245, 2012. DOI: 10.1080/15551393.2012.735585.

RAVELLI, Louise J.; VAN LEEUWEN, Theo. Modality in the digital age. Visual Communication, v. 17, n. 3, p. 277–297, 2018. DOI: 10.1177/1470357218764436.

SEBEOK, Thomas A. Global semiotics. Bloomington, IN: Indiana University Press, 2001.

SEBEOK, Thomas A.; UMIKER-SEBEOK, Donna Jean. “You know my method” : a juxtaposition of Charles S. Peirce and Sherlock Holmes. In: ECO Umberto, SEBEOK, Thomas A. (orgs.). The sign of three. Bloomington, IN: Indiana University Press, 1983. p. 11–54.

SONESSON, Göran. The phenomenological semiotics of iconicity and pictoriality – Including some replies to my critics. Language and Semiotic Studies, v. 2, n. 2, p. 1–73, 2016.

SPENCE, Des. Imagine all the people. BMJ: British Medical Journal, v. 346, n. 7894, p. 39–39, 2013.

STAMPOULIDIS, Georgios. Polysemiotic communication vs. multimodality. SAUC – Street Art & Urban Creativity Scientific Journal, v. 5, n. 2, p. 26–31, 2019. DOI: 10.25765/sauc.v5i2.156.

STEPHENS, Neil; RUIVENKAMP, Martin. Promise and ontological ambiguity in the in vitro meat imagescape: From laboratory myotubes to the cultured burger. Science as Culture, v. 25, n. 3, p. 327–355, 2016. DOI: 10.1080/09505431.2016.1171836.

TEISER, Stephen F. Reinventing the wheel: paintings of rebirth in medieval Buddhist temples. Seattle, WA: University of Washington Press, 2007.

VELASQUEZ, Scott E. Images of gender arrangements of three American social organizations: A content analysis of top ranked Google images of individuals preforming organizational roles. 2016. Ph.D. – Kansas State University, 2016.

WALSH, Wendy E. Investigating public perception of occupational therapy: an environmental scan of three media outlets. American Journal of Occupational Therapy, Bethesda, v. 72, n. 3, p. 1–10, 2018. DOI: doi.org/10.5014/ajot.2018.024513.

WASHBURN, Dorothy K.; CROWE, Donald W. (eds.). Symmetries of culture: Theory and practice of plane pattern analysis. Seattle, WA: University of Washington Press, 1988.

WASHBURN, Dorothy K.; CROWE, Donald W. (eds.). Symmetry comes of age: the role of pattern in culture. Seattle: University of Washington Press, 2004.

WEBB, Eugene J.; CAMPBELL, Donald T.; SCHWARTZ, Richard D.; SECHREST, Lee. Unobtrusive measures, rev. ed. Thousand Oaks, CA: SAGE, 2000.

WILDFEUER, Janina; PFLAEGING, Jana; BATEMAN, John; SEIZOV, Ognyan; TSENG, Chiao-I. Multimodality, disciplinary thoughts and the challenge of diversity. Berlin: de Gruyter, 2019. DOI: 10.1515/9783110608694.

WILKINSON, Mark D. et al. The FAIR guiding principles for scientific data management and stewardship. Scientific Data, v. 3, p. 1–9, 2016. DOI: 10.1038/sdata.2016.18.

WISEMAN, Richard; OWEN, Adrian M. Turning the other lobe: directional biases in brain diagrams. i-Perception, v. 8, n. 3, p. 1–4, 2017. DOI: 10.1177/2041669517707769.

WOLCOTT, Harry F. Ethnography: A way of seeing. 2nd ed. Lanham, MD: AltaMira, 2008.

WORRING, Marcel; SNOEK, Cees. Visual content analysis. In: LIU, Ling; ÖZSU, M. Tamer (orgs.). Encyclopedia of database systems. Boston, MA: Springer, 2009. p. 3360–3365. DOI: 10.1007/978-0-387-39940-9_1019.

WSPANIALY, Patrick; BROOKS, Justin; MOUSSA, Medhat. An image labeling tool and agricultural dataset for deep learning. arXiv preprint arXiv:2004.03351, 2020, 2020.

WU, Amy. Hillary’s Heels: Examining gender and power through semiotics. In: PELKEY, Jamin; WALSH-MATTHEWS, Stephanie; SBROCCHI, Leonard (eds.) Semiotics 2014: The Semiotics of Paradox. Yearbook of the Semiotic Society of America. Ottawa: Legas, 2015. p. 473–490. DOI: 10.5840/cpsem201436.

YU, Ning; WANG, Tianfang; HE, Yingliang. Spatial subsystem of moral metaphors: a cognitive semantic study. Metaphor and Symbol, v. 31, n. 4, p. 195–211, 2016. DOI: 10.1080/10926488.2016.1223470.

ZLATEV, Jordan. Mimesis Theory, Learning, and polysemiotic communication. In: PETERS, Michael A. (org.). Encyclopedia of Educational Philosophy and Theory. Singapore: Springer, 2019. p. 1–6. DOI: 10.1007/978-981-287-532-7_672-1.

Downloads

Published

2020-12-14

Issue

No. 21 (2020): Humanidades Digitais

Section

Artigos

License

Esta revista oferece acesso livre imediato ao seu conteúdo de acordo com a licença CC BY 4.0, em conformidade com a definição de acesso público do Directory of Open Access Journals (DOAJ).

Ao submeter um texto à TECCOGS, os autores asseguram que o material submetido à avaliação e eventual publicação não infringe de modo algum qualquer direito proprietário ou copyright de outros. Com a submissão, o autor transfere em efetivo os direitos de publicação do artigo para a TECCOGS. A transferência de copyright cobre os direitos exclusivos de publicação e distribuição do artigo, incluindo reimpressões ou quaisquer outras reproduções de natureza similar, além de traduções. Os autores mantém o direito de usar todo ou partes deste texto em trabalhos futuros de sua autoria e de conceder ou recusar a permissão a terceiros para republicar todo ou partes do texto ou de suas traduções. Para republicar números da revista na íntegra, qualquer interessado precisa obter permissão por escrito tanto dos autores como também dos editores da TECCOGS. A TECCOGS por si só pode conceder direitos relativos a emissões de periódicos como um todo.

Imagens com direitos autorais pertencentes a terceiros, que não foram concedidos ao autor do texto, devem ser utilizadas somente quando necessárias à análise e ao argumento da pesquisa, sempre indicando as respectivas fontes e autoria. A TECCOGS dispensa o uso de imagens meramente ilustrativas. Se desejar ilustrar um conceito, o autor deve indicar, em forma de URL ou referência bibliográfica, uma referência em que a ilustração esteja disponível.

---------------------------------------------------------------------------------

This journal offers free immediate access to its content under CC BY 4.0, in accordance with Directory of Open Access Journals' (DOAJ) definition of Open Acess.

When submitting a text to TECCOGS, authors ensure that the material submitted for evaluation and eventual publication does not infringe any proprietary right or copyright. Upon submission, authors effectively transfer the publication rights of the article to TECCOGS. The copyright transfer covers the exclusive rights of publication and distribution of the article, including reprints or any other reproduction of similar nature, in addition to translations. Authors retain the right to use all or parts of the text in future works of their own, as well as to grant or refuse permission to third parties to republish all or parts of the text or its translations. In order to fully republish issues of the magazine, anyone interested must obtain written permission from both the authors and the editors of TECCOGS. TECCOGS alone can grant rights relating to issues of journals as a whole.

Images whose copyright belongs to third parties that have not been granted to the author of the text should be used only when essential for the analysis and argument, always indicating theirs respective sources and authorship. TECCOGS dismisses any use of merely illustrative images. To illustrate a concept, the author must indicate, in the form of a URL or bibliographic reference, a source in which the illustration is available.