During week 9, the team focused on refining the chatbot prototype and laying the foundation for robust internal testing and feedback integration.

Prototype Progress and Enhanced Transparency

The team refined the architectural model for delivering prototype 1 to our company liaison for internal testing. This is crucial as it allows us to gather essential feedback and insights with the dataset we used for training and the accuracy of our model’s answers. 

To further achieve this, our team engaged in an in-depth brainstorming session implementing feedback loops within the architecture. This includes developing a reranking system based on feedback from responses, which will be pivotal in enhancing the quality and accuracy of the chatbot.

In addition to preparing for the prototype delivery, we introduced the feature of returning the sources from which the chatbot retrieves its information. This enhances transparency in the retrieval process, allowing us to understand how information is sourced and, therefore, carefully finetune the hyperparameters and strengthen our corpus where appropriate. 

Testing and Overall Insights

To ensure the chatbot’s effectiveness, we compiled about 150+ testing queries. These queries are instrumental in evaluating how our bot responds to different parametric values, allowing the team to refine the LLMs’ hyperparameters and reliability further.

Moreover, an eyeball comparison between our chosen base LLM and the implemented retriever was conducted. This further enabled the user to identify areas of improvement and whether or not the models have been successful in meeting the expectations.