Publications

2025

1. The Effects of Demographic Instructions on LLM Personas

   Angel Felipe Paula, J. Shane Culpepper, Alistair Moffat, and 3 more authors

   In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, Padua, Italy, 2025

   Abs DOI PDF Poster

   Social media platforms must filter sexist content in compliance with governmental regulations. Current machine learning approaches can reliably detect sexism based on standardized definitions, but often neglect the subjective nature of sexist language and fail to consider individual users’ perspectives. To address this gap, we adopt a perspectivist approach, retaining diverse annotations rather than enforcing gold-standard labels or their aggregations, allowing models to account for personal or group-specific views of sexism. Using demographic data from Twitter, we employ large language models (LLMs) to personalize the identification of sexism. Our empirical results show that OpenAI’s LLMs (GPT-3.5, GPT-4, and GPT-4o) and two open-source LLMs (Mistral and Qwen) exhibit higher Krippendorff’s alpha label agreement with female annotators than with male annotators. As well, each LLM presents higher Krippendorff’s alpha agreement with a specific annotator age group. We then sought to counter these trends by providing ”persona” instructions as part of the LLM prompt, with somewhat surprising outcomes, highlighting the potential of user-centered perspectivist methods to improve content moderation systems.

2024

1. Towards Investigating Biases in Spoken Conversational Search

   Sachin Pathiyan Cherumanal, Falk Scholer, Johanne R Trippas, and 1 more author

   In Companion Proceedings of the 26th International Conference on Multimodal Interaction, San Jose, Costa Rica, 2024

   Abs DOI PDF Poster

   Voice-based systems like Amazon Alexa, Google Assistant, and Apple Siri, along with the growing popularity of OpenAI’s ChatGPT and Microsoft’s Copilot, serve diverse populations, including visually impaired and low-literacy communities. This reflects a shift in user expectations from traditional search to more interactive question-answering models. However, presenting information effectively in voice-only channels remains challenging due to their linear nature. This limitation can impact the presentation of complex queries involving controversial topics with multiple perspectives. Failing to present diverse viewpoints may perpetuate or introduce biases and affect user attitudes. Balancing information load and addressing biases is crucial in designing a fair and effective voice-based system. To address this, we (i) review how biases and user attitude changes have been studied in screen-based web search, (ii) address challenges in studying these changes in voice-based settings like SCS, (iii) outline research questions, and (iv) propose an experimental setup with variables, data, and instruments to explore biases in a voice-based setting like Spoken Conversational Search.

2. Everything We Hear: Towards Tackling Misinformation in Podcasts

   Sachin Pathiyan Cherumanal, Ujwal Gadiraju, and Damiano Spina

   In Proceedings of the 26th International Conference on Multimodal Interaction, San Jose, Costa Rica, 2024

   Abs DOI PDF Video Slides

   Advances in generative AI, the proliferation of large multimodal models (LMMs), and democratized open access to these technologies have direct implications for the production and diffusion of misinformation. In this prequel, we address tackling misinformation in the unique and increasingly popular context of podcasts. The rise of podcasts as a popular medium for disseminating information across diverse topics necessitates a proactive strategy to combat the spread of misinformation. Inspired by the proven effectiveness of auditory alerts in contexts like collision alerts for drivers and error pings in mobile phones, our work envisions the application of auditory alerts as an effective tool to tackle misinformation in podcasts. We propose the integration of suitable auditory alerts to notify listeners of potential misinformation within the podcasts they are listening to, in real-time and without hampering listening experiences. We identify several opportunities and challenges in this path and aim to provoke novel conversations around instruments, methods, and measures to tackle misinformation in podcasts.

3. Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search

   Kaixin Ji, Sachin Pathiyan Cherumanal, Johanne R. Trippas, and 4 more authors

   In Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction, Melbourne, VIC, Australia, 2024

   Abs DOI PDF Video Poster

   Spoken Conversational Search (SCS) poses unique challenges in understanding user-system interactions due to the absence of visual cues, and the complexity of less structured dialogue. Tackling the impacts of cognitive bias in today’s information-rich online environment, especially when SCS becomes more prevalent, this paper integrates insights from information science, psychology, cognitive science, and wearable sensor technology to explore potential opportunities and challenges in studying cognitive biases in SCS. It then outlines a framework for experimental designs with various experiment setups to multimodal instruments. It also analyzes data from an existing dataset as a preliminary example to demonstrate the potential of this framework and discuss its implications for future research. In the end, it discusses the challenges and ethical considerations associated with implementing this approach. This work aims to provoke new directions and discussion in the community and enhance understanding of cognitive biases in Spoken Conversational Search.

4. Walert: Putting Conversational Information Seeking Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot

   Sachin Pathiyan Cherumanal, Lin Tian, Futoon M. Abushaqra, and 7 more authors

   In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, Sheffield, United Kingdom, 2024

   Abs DOI PDF Video Poster

   Creating and deploying customized applications is crucial for operational success and enriching user experiences in the rapidly evolving modern business world. A prominent facet of modern user experiences is the integration of chatbots or voice assistants. The rapid evolution of Large Language Models (LLMs) has provided a powerful tool to build conversational applications. We present Walert, a customized LLM-based conversational agent able to answer frequently asked questions about computer science degrees and programs at RMIT University. Our demo aims to showcase how conversational information-seeking researchers can effectively communicate the benefits of using best practices to stakeholders interested in developing and deploying LLM-based chatbots. These practices are well-known in our community but often overlooked by practitioners who may not have access to this knowledge. The methodology and resources used in this demo serve as a bridge to facilitate knowledge transfer from experts, address industry professionals’ practical needs, and foster a collaborative environment. The data and code of the demo are available at https://github.com/rmit-ir/walert.

5. Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024

   Alexander Frummet, Andrea Papenmeier, Maik Fröbe, and 15 more authors

   In , Aug 2024

   Abs DOI PDF

   Conversational Agents are increasingly integrated into our daily routines, assisting us with various tasks, from simple commands such as scheduling events to more complex conversational search interactions. Such conversational search systems are traditionally evaluated with word-overlap metrics such as F1 score and accuracy. The full-day workshop on Search-Oriented Conversational Artificial Intelligence (SCAI) at CHIIR 2024 explored the evaluation of conversational search systems from the user’s perspective. This interactive workshop included multiple panel discussions and working groups focused on developing and discussing innovative, user-centered evaluation methods for these systems. This paper, co-authored by both organizers and participants of the workshop, presents a summary of the insights gathered from the panel discussions and working groups.Date: 14 March 2024.Website: https://scai.info/scai-2024/.

2023

1. RMIT_IR at the NTCIR-17 FairWeb-1 Task

   Sachin Pathiyan Cherumanal, Kaixin Ji, Danula Hettiachchi, and 3 more authors

   In The Seventeenth NTCIR Conference on Evaluation of Information Access Technologies Proceedings, Aug 2023

   Abs PDF Video Code Poster

   This report describes the participation of the RMIT IR group at the NTCIR-17 FairWeb-1 task. We submitted five runs with the aim of exploring the role of explicit search result diversification (SRD) and ranking fusion to generate fair rankings considering multiple fairness attributes. We also explored the use of a linear combination-based technique (LC) to take into consideration the relevance while re-ranking. In this report, we compared results from all our submitted runs against each other and the retrieval baselines along each topic type separately (i.e., Researcher, Movie, YouTube). Overall, our results show that neither the SRD-based runs nor the linear combination-based runs show any statistically significant improvement over the retrieval baselines.

2022

1. RMIT CIDDA IR at the TREC 2022 Fair Ranking Track

   Sachin Pathiyan Cherumanal, Marwah Alaofi, Reham Abdullah Altalhi, and 3 more authors

   In The Thirty-First Text REtrieval Conference (TREC 2022) Proceedings, Aug 2022

   Abs PDF

   This report describes the participation of the RMIT CIDDA IR group at the TREC 2022 Fair Ranking Track (Task 1). We submitted 8 runs with the aim to explore the role of explicit search result diversification, ranking fusion, and the use of a multi-criteria decision-making method to generate fair rankings considering multiple protected attributes.

2. Fairness-Aware Question Answering for Intelligent Assistants

   Sachin Pathiyan Cherumanal

   In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, Aug 2022

   Abs DOI PDF

   Conversational intelligent assistants, such as Amazon Alexa, Google Assistant, and Apple Siri, are a form of voice-only Question Answering (QA) system and have the potential to address complex information needs. However, at the moment they are mostly limited to answering with facts expressed in a few words. For example, when a user asks Google Assistant if coffee is good for their health, it responds by justifying why it is good for their health without shedding any light on the side effects coffee consumption might have citegao2020toward. Such limited exposure to multiple perspectives can lead to change in perceptions, preferences, and attitude of users, as well as to the creation and reinforcement of undesired cognitive biases. Getting such QA systems to provide a fair exposure to complex answers – including those with opposing perspectives – is an open research problem. In this research, I aim to address the problem of fairly exposing multiple perspectives and relevant answers to users in a multi-turn conversation without negatively impacting user satisfaction.

3. RMIT at TREC 2021 Fair Ranking Track

   Sachin Pathiyan Cherumanal, Damiano Spina, Falk Scholer, and 1 more author

   In The Thirtieth Text REtrieval Conference (TREC 2021) Proceedings, Aug 2022

   Abs PDF Poster

   This report describes the data, the assumptions, methodology, and our results involved in the participation at the TREC 2021 Fair Ranking Track. While most of the fairness-aware re-ranking techniques require explicitly defining protected attributes, we tried to leverage the implicit features of the Wikimedia articles by using an implicit diversification technique to study the impact of diversification on a fair ranking problem.

2021

1. Evaluating Fairness in Argument Retrieval

   Sachin Pathiyan Cherumanal, Damiano Spina, Falk Scholer, and 1 more author

   In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, Virtual Event, Queensland, Australia, Aug 2021

   Abs DOI PDF Code Poster

   Existing commercial search engines often struggle to represent different perspectives of a search query. Argument retrieval systems address this limitation of search engines and provide both positive (PRO) and negative (CON) perspectives about a user’s information need on a controversial topic (e.g., climate change). The effectiveness of such argument retrieval systems is typically evaluated based on topical relevance and argument quality, without taking into account the often differing number of documents shown for the argument stances (PRO or CON). Therefore, systems may retrieve relevant passages, but with a biased exposure of arguments. In this work, we analyze a range of non-stochastic fairness-aware ranking and diversity metrics to evaluate the extent to which argument stances are fairly exposed in argument retrieval systems.Using the official runs of the argument retrieval task Ttouché at CLEF 2020, as well as synthetic data to control the amount and order of argument stances in the rankings, we show that systems with the best effectiveness in terms of topical relevance are not necessarily the most fair or the most diverse in terms of argument stance. The relationships we found between (un)fairness and diversity metrics shed light on how to evaluate group fairness – in addition to topical relevance – in argument retrieval settings.

2. Report on the future conversations workshop at CHIIR 2021

   Damiano Spina, Johanne R. Trippas, Paul Thomas, and 17 more authors

   SIGIR Forum, Jul 2021

   Abs DOI PDF

   The Future Conversations workshop at CHIIR’21 looked to the future of search, recommendation, and information interaction to ask: where are the opportunities for conversational interactions? What do we need to do to get there? Furthermore, who stands to benefit?The workshop was hands-on and interactive. Rather than a series of technical talks, we solicited position statements on opportunities, problems, and solutions in conversational search in all modalities (written, spoken, or multimodal). This paper -co-authored by the organisers and participants of the workshop- summarises the submitted statements and the discussions we had during the two sessions of the workshop. Statements discussed during the workshop are available at https://bit.ly/FutureConversations2021Statements.

2017

1. Reducing the Cold-User and Cold-Item Problem in Recommender System by Reducing the Sparsity of the Sparse Matrix and Addressing the Diversity-Accuracy Problem

   K. R. Bindu, Rhama Lalgudi Visweswaran, Sachin Pathiyan Cherumanal, and 2 more authors

   In Proceedings of International Conference on Communication and Networks, Jul 2017

   Abs

   Recommender Systems are a subclass of information filtering systems that seek to predict the preferences of a user or the preference that a user would give to an item. The most common problem faced by these systems is the lack of data. Such a situation leads to a matrix that is extremely sparse thus reducing the accuracy of prediction. Cold-start problem is one such problem that is faced by the recommender systems when a new user or a new item enters the system. We are hoping to reduce the cold-user and the cold-item problem by reducing the sparsity of the sparse matrix with the help of Iterative Local Least Squares algorithm and a hybrid of Heat Spreading algorithm and Probability Spreading algorithm.