Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security Advice

Ortloff, Anna-Marie and Fassl, Matthias and Ponticello, Alexander and Martius, Florin and Mertens, Anna and Krombholz, Katharina and Smith, Matthew
(2023) Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security Advice.
In: CHI Conference on Human Factors in Computing Systems (CHI '23), Hamburg, Germany.
Conference: CHI International Conference on Human Factors in Computing Systems

Full text not available from this repository.

Abstract

When conducting qualitative research it is necessary to decide how many researchers should be involved in coding the data: Is one enough or are more coders beneficial? To offer empirical evidence for this question, we designed a series of studies investigating qualitative coding. We replicated and extended a usable security and privacy study by Ion et al. to gather both simple survey data and complex interview data. We had a total of 65 students and seven researchers analyze different parts of this data. We analyzed the codebook creation process, similarity of outcomes, inter-rater reliability, and compared the student to the researcher outcomes. We also surveyed five years of SOUPS-PC members about their views on coding. The reviewers view on coding practices for complex and simple data are almost identical. However, our results suggest that the coding process can be different for the two types of data, with complex data benefiting more from interaction between coders.

Actions

Actions (login required)

View Item View Item