How to use Chatbots to fasten Qualitative data analysis

UX research, Human-computer interaction, Future of Technology

The objective of this substack is to help you fasten the qualitative data analysis process with the collaboration of a chatbot, in this case, we will use Copilot, a generative conversational AI chatbot.

The method

• In what situations will this be useful?
  During any research project, it’s important to collect and synthesise information that already exists about the product/service you are working on. This phase is extremely important, it will help you frame the problem you are trying to solve, fill in the knowledge gaps and avoid repeating unnecessary studies. So for instance, if you found a big chunk of qualitative information that you want to analyse, this article it’s for you!

• How? I used Google Play reviews available online from Duolingo users and then Copilot to help me conduct a qualitative analysis

• Who’s this for? Any person who conducts qualitative research: researchers, designers, product owners

• What’s the benefit? Speed up the time for conducting qualitative analysis

• Where in the product development lifecycle? Problem definition (when you’re still figuring out the problem you will be working on) or listening phase (if you want to improve a product that’s already released for the general public).

  

The method explained

1. Collecting information about the app — there are several valuable sources available online, forums like Quora or Reedit, App Stores like Google Play or App Store, and many others you can explore. For this mini research project, I decided to use reviews from Duolingo available on Google Play. During this phase, you need to be strategic about what information you want to analyse because copilot only accepts a limit of 18.000 20.000 words for a single query or prompt (free version), so scope down the information sample, for example, I defined the time interval of the past 3 months because I wanted to know the most recent thoughts users have about Duolingo but you might found other criteria that make more sense to you.

2. Read the information and find the main themes — when reading the information for the first time, identify what are the most prominent themes upfront and write them down — this is a crucial step. Copilot is a great tool but as a researcher you have the responsibility to avoid biased data, having said that you always need to read the information.

Hence, read the information, underline it, and summarise it (like in the old days) to have an idea regarding the themes that stand out upfront.

3. Clean the information file — you might need to clean the file a little bit in a text editor (e.g.box, word, pages) but leave it in a way where you can distinguish the user and the review. You will end up with a raw data like this:

user1  “I find this app amazing" but you need to improve the like button” 

user2 “very useful app” 

4. Ask the bot to find themes on the information and compare them to your themes (step 2) —This is a great way to reflect on your personal bias. Unfortunately, bias is unavoidable, but it’s possible to reduce it. There are more than 18 types of bias, and the best way to reduce it is to study them so they become easier to detect.

5. Ask copilot to conduct the analysis without any themes — this way you’ll easily detect any bias. Compare your categories to the copilots’, you might surprise yourself and adjust or add a few more to the one’s you’ve previously found.

If you want to go pro mode, use another chatbot like ChatGPT) and confront your themes to the ones the bots found, collaborate with bots like your peers, they will help you reflect on your own bias and process

Example: find the top themes in this data: 

[

insert data from users, 

user1  “I find this app amazing" but you need to improve the like button” 

user2 “very useful app” “
] 

where people people are talking about the Duolingo App. Organise the top themes by by benefits and challenges

6. Give instructions to Copilot for the qualitative analysis with the consolidated themes

Example: create a summary of the top themes people are talking about on Google Play regarding the Duolingo App. Center the summary on the top benefits and problems and organise it according to [add consolidated themes / categories]and add a brief explanation to each topic. The information you need to analyse is here: [add data set with feedback from users]

• The result you will get:

7. Rephrase your writing — In the US and Europe, there is still a legal gap in the copywriting of information from chatbots. Nowadays, the law only protects the work produced by humans. Nevertheless, read the contents, be mindful, and rephrase your writing.

Conclusions

• Tools like Copilot might help fasten the research process, but it's always required that researchers embed in the process phases dedicated to reviewing the information to avoid biased results.

• Acknowledge the limitations of these tools, for example: the word limit on the first prompt. You might establish criteria to limit the amount of information to be analyzed e.g. time interval or other that makes sense according to the project objectives.

• This process is adaptable and might have adjustments however, it seems these types of tools will have more and more the role of collaborators.

• Asking more than one chatbot to analyze the same data set is a great way to reflect and detect bias in bots and your own bias.