Previous research has shown that voters often struggle to understand political statements in Voting Advice Applications (VAAs) and tend to make minimal efforts to resolve these comprehension difficulties. Conversational Agent Voting Advice Applications (CAVAAs) present a promising solution by incorporating chatbots that provide users with additional information at low cognitive cost. This study investigates an experiment conducted in the lead-up to the Dutch National Elections on November 22, 2023, comparing a standard VAA without supplementary information to two CAVAA versions: one with a proactive chatbot that initiates interactions and one with a reactive chatbot that responds to user queries.
A total of 148 university students were randomly assigned to one of the three conditions. Results revealed that information options in the two CAVAA conditions were used extensively, with information being requested in about 45% of the cases. Participants primarily sought opinion-based information, followed by details on the status quo, and lastly, semantic clarification of terms. This suggests that voters are not just searching for objective facts but also for argumentative insights.
Comparing the two CAVAA conditions to the regular VAA, both CAVAA versions showed a reduced proportion of non-directional responses, indicating that chatbots effectively help users resolve comprehension issues. Furthermore, while all tools were rated similarly in terms of ease of use and playfulness, CAVAA users found the tool more useful and they also felt better informed. These improvements were largely consistent across both proactive and reactive chatbot versions, suggesting that CAVAAs can enhance the quality of voter decision-making, regardless of the chatbot’s engagement style.
Please ask permission from the first author before accessing the data.
Data files
SPSS
Qualtrics_surveydata_Dataverse.sav
Cleaned dataset of the relevant survey data.
Chatbot_gedrag.sav
The registration of the clicks in the CAVAA
MLWIN
Model_nietdirectioneleantwoorden.wsz en Model_semantisch_Infomratie.wsz
File with the model used to analyzed the % of non-directional responses in MLWIN and the % of semantic information requested (please keep this file in the Dataverse as it contains examples of the exact statistical models reported; the information in these files is also available in chatbot_gedrag.sav so contains no new information)
Supplemental material
Materials
A Word-version of the experimental materials used
Clip
Three video clips demonstrating the chatbot can be found at this SURF Research Drive folder.
Structure of the data package
In the package we have included the experimental materials (pdf file), a video in which the tools are demonstrated (included in the metadata), the survey data and the data related to the behavior in the tool itself (.sav files).
We also like to include wsz-files including the statistical models we constructed in MLWIN. These files offer the same information as the .sav-files but researchers with an MLWIN account can open the statistical models used for analyzing the response data.
Method: Data were collected through an experimental study
Universe: The population consisted of theoretically educated students from Tilburg University
Data sources: Qualtrics and the data of the conversations have been collected through the chatbot developer (genius voice)
Country / Nation: the Netherlands
