Blog Posts

WEEK #20: Reranking and Automation Unleashed

EchoPilot during the in-class project work segment.

Greetings! Welcome back to another week of EchoPilot! As we traverse the intricate landscape of knowledge retrieval and refinement, the team has been diligently working on several fronts to enhance the efficiency and accuracy of our project.

Refactored Reranking for Enhanced Flexibility

In a significant stride towards versatility, our team successfully refactored the Cohere reranking mechanism, introducing toggling and parameter tuning options. These changes were deployed to the development environment, setting the stage for meticulous testing. This adjustment not only allows for better customization but also aligns with our strategic vision for adaptable knowledge retrieval.

Additionally, we optimized the global reranking to seamlessly integrate with the Cohere reranking. This strategic approach ensures minimal document transfer to Cohere, optimizing cost-effectiveness. Rigorous testing of the deployed reranking logic yielded exceeding results, with almost 90% accuracy achieved with GPT-4 and approximately 78% with GPT-3.5. This achievement solidifies our confidence in the effectiveness of our refined reranking strategies.

Automation Advancements and Streamlining Architecture

Our pursuit towards efficiency led us to explore the implementation of an automated ingestion system through GitHub Actions. This innovation aims to simplify the expansion of our knowledge store, streamlining the process and minimizing manual interventions. Simultaneously, we cleaned up our architecture by removing the scripting retriever, reducing unnecessary corpus volume, and creating a more streamlined and efficient model.

That’s a wrap for this week! See you all next time.

WEEK #19: Advancing Efficiency and Performance

How varying the temperature parameter works. (Image source: https://txt.cohere.com/llm-parameters-best-outputs-language-ai/)

As we journey through the latest milestones of our project, it’s evident that meticulous integration of various methodologies continues to steer our trajectory towards success. Here’s a rundown of our recent accomplishments for the week:

Command Retrieval Integration and Reranking Mechanisms

One of the key achievements of the past few days has been the successful merging of document and command retrieval mechanisms. This integration ensures seamless access to all necessary information, enhancing the model’s capability to deliver comprehensive responses.

Continuing our quest for excellence, we’ve dedicated efforts to refine both static and dynamic document reranking mechanisms. This endeavor involved adding necessary metadata in Pinecone and developing sophisticated code logic for updating document scores upon retrieval. These enhancements contribute to elevating the quality of responses delivered by our system.

Performance Metrics and Hyperparameter Tuning

We’ve made significant strides in evaluating our model’s performance by executing a comprehensive testing suite and gathering initial performance metrics. Building upon this foundation, we’ve commenced the crucial task of tuning and experimenting with model hyperparameters. This iterative process aims to identify values that optimize performance and bolster efficiency.

Integration of Latest GPT Models

Staying abreast of technological advancements, we’ve updated our API calls to leverage the newest GPT models released last week. This proactive approach underscores our commitment to harnessing cutting-edge tools and techniques to drive continuous improvement.

In essence, our recent endeavors underscore the collective dedication of our team towards enhancing efficiency, optimizing performance, and delivering unparalleled user experiences. As we forge ahead, guided by our commitment to excellence, we remain poised to conquer new challenges and achieve greater heights in our project journey. Stay tuned for more updates!

WEEK #18: Enhancing Functionality

Generic overview of how Cohere rerank works (Image source: https://txt.cohere.com/rag-chatbot/)

As we continue to push the boundaries of innovation within our project, recent developments underscore the significance of meticulous planning and strategic implementation. Here’s a glimpse into the latest advancements:

Integration and Testing Phase Commences

In our relentless pursuit of excellence, we’ve merged the testing branch code into the development branch, marking the commencement of our configuration testing phase. This critical step allows us to evaluate the seamless integration of various components while ensuring optimal functionality and reliability.

Enhanced Data Integration with JAWS Command Search

Diving deeper into the realm of innovative tools, we’ve taken significant strides in incorporating JAWS Command Search into our project framework. We architected a method to make the JAWS command search data readily available to our chat model. This involves a sophisticated post-retrieval mapping process, linking script names to keystrokes seamlessly. Such meticulous integration promises to elevate the efficiency and efficacy of our project’s functionalities.

Fine-Tuning Retrieval Parameters and Reranking Documents

In our quest for optimization, we’ve embarked on a journey of experimentation with parameter values for command search retrieval. Through rigorous testing, we’ve discovered that this retriever benefits from slightly different values compared to our standard retriever. This nuanced understanding allows us to fine-tune our processes, ensuring peak performance and accuracy.

Our commitment to innovation extends to the exploration of advanced reranking mechanisms. Currently, our research delves into two distinct approaches: static (Cohere) reranking and dynamic reranking. By exploring these avenues, we aim to enhance the relevance and precision of our search results, catering to the diverse needs of our users. Similarly, in preparation for the implementation of dynamic reranking, we’ve diligently updated our corpus documents. These modifications ensure compatibility and optimization, laying the groundwork for seamless integration and enhanced functionality.

As we navigate through these dynamic developments, our focus remains unwavering on delivering a seamless, user-centric experience. With each milestone achieved and every innovation embraced, we inch closer to our collective vision of redefining excellence in project execution and functionality. Stay tuned for more updates as we continue to push the boundaries of possibility and redefine the future of our project.

WEEK #17: Exploring and Implementing DFT

As we reflect on the milestones achieved within the past few days, it becomes evident that the deliberate integration of DFT methodologies has been instrumental in shaping our project’s trajectory. From implementing a containerized iteration of our web application on Azure to crafting an automated deployment pipeline, each achievement emphasizes the importance of careful design considerations, optimizing efficiency, and prioritizing user-centric approaches.

Azure Deployment for Enhanced Performance

For optimal performance and reduced platform-specific bugs, our team successfully deployed a containerized version of our web application on Microsoft Azure. This strategic move ensures that our project leverages increased resources while mitigating common issues encountered in diverse deployment environments.

Furthermore, recognizing the need for efficiency in our development process, we’ve implemented a deployment pipeline. This pipeline encompasses both development and production environments, facilitating the swift and controlled evaluation of new features. This enhancement is a pivotal step towards achieving agility and reliability in our project lifecycle.

JAWS Command Search Integration Exploration

Always on the lookout for innovative tools, we’ve delved into the possibilities offered by JAWS Command Search. Our exploration aims to determine whether integrating this tool into our corpus could bring valuable enhancements to our project. Early indications suggest that JAWS Command Search could indeed be a valuable addition, and we are looking to explore its potential further.

Likewise, the team has also begun building a method for seamlessly ingesting JAWS Command Search data. Our goal is to map commands to the unique keybindings of individual users, enhancing the user experience by tailoring interactions to their preferences. This forward-thinking approach aligns with our commitment to user-centric design and personalization.

Special mention to our member Antonio who celebrated his birthday this week as well 🥳🎂
Stay tuned for more updates and see you next week!

WEEK #16: Optimizing Efficiency and Expanding Horizons

Greetings! As we dive into another week of focused efforts, the team is excited to share some remarkable updates on our journey towards the completion of our end-of-semester project. Here’s a glimpse into the strides we’ve made this week:

Cost-Efficiency with Serverless Pinecone Instances

In a strategic move, we’ve transitioned to serverless Pinecone instances, marking a significant milestone in our pursuit of efficiency. This transition has not only streamlined our operations but has also led to a remarkable 10x reduction in monthly storage costs. It’s a testament to our commitment to smart resource utilization.

Enriching Our Data Set for Automated Testing

Building on the foundation of automated testing, we’ve initiated the process of expanding our data set. This involved scraping user data to curate effective questions and appropriate responses. This step is crucial in fortifying the accuracy and reliability of our automated testing mechanisms.

Exploring Fine-Tuning for Model Accuracy

In our pursuit of excellence, we’ve delved into the realm of fine-tuning as a potential next step to enhance the accuracy of our model. This research phase underscores our dedication to staying at the forefront of technological advancements and ensuring our end-of-semester project is nothing short of exceptional.

Advancements in Automated Testing Framework

Our commitment to delivering a reliable product shines through in the expansion of our automated testing framework. We’ve not only broadened its scope but also devised a method for determining whether a text-based response aligns with the desired output. This meticulous approach will undoubtedly elevate the precision and reliability of our prototype.

That concludes this week’s updates; be sure to keep an eye out for our upcoming endeavors!

WEEK #15: EchoPilot is Back!

As we embark on a fresh semester, we’re thrilled to welcome everyone back and share some exciting updates from our team.

Enhancing Knowledge Base

To elevate our capabilities, we’ve expanded our knowledge base by incorporating Microsoft Office documentation into our corpus. This addition will significantly bolster our ability to provide insightful answers to queries related to Outlook, Excel, PowerPoint, and Word. It’s a strategic move that aligns with our commitment to excellence and proficiency.

Paving the Way for Automated Testing

In our recent meeting, we engaged in a thoughtful discussion about our proposed approach to automated testing. This is a crucial step towards optimizing our development process and ensuring the reliability of our end-of-semester project. We’re excited about the possibilities and eager to see the positive impact this decision will have on the quality and efficiency of our work.

See you all next week and happy long weekend!

WEEK #14: SLDR Triumphs and End-of-Semester Tacos

As we approach the conclusion of this semester, our collective efforts have been laser-focused on the meticulous finalization and signing of the SLDR.

SLDR D-Day!

The highlight of the week undoubtedly was the final SLDR presentation alongside the Fireside Chat featuring Quang Tran, a distinguished figure as the Partner at Collective & Co and Co-founder of Starter Space. This insightful session, preceding the Final SLDR Presentation, provided valuable perspectives and enriched everyone’s understanding of entrepreneurship. The day concluded with the team delivering their final SLDR presentation to their respective groups alongside everyone’s liaisons and coaches. Overall, the team delivered their presentation seamlessly compared to the rest of the semester and received positive feedback. Nevertheless, our commitment to excellence persisted as we continued our weekly meetings with our coach and liaison, seeking ways to further enhance the prototype and ensure a strong start for the upcoming semester.

As the semester draws to a close, our team came together to mark the occasion with a delightful early dinner featuring tacos. Amidst the flavors and festivities, we took the opportunity to reflect on the significant moments and achievements of the past few months.

What’s Next?

Your steadfast support and engagement with EchoPilot this fall have been truly appreciated. As we bid farewell to the current semester, we extend our warmest wishes for a restful and enjoyable winter break. We look forward to reconnecting with you in the new year as we embark on exciting endeavors and continue to share our journey.

Here’s to a well-deserved break and the promise of a bright and productive year ahead. Until then, take care and see you in the next chapter of our adventure!

WEEK #13: Preparations for SLDR

Immediately after the Thanksgiving break, the team faced notable accomplishments and challenges, focusing on refining the iterative prototype and strategizing for the tasks slated for the upcoming semester in anticipation of the System Level Design Review (SLDR).

Crafting the SLDR Draft

One of our significant achievements this week has been the collaborative creation of the initial draft of our SLDR Report. This comprehensive document encapsulates the essence of our project, highlighting key aspects of our work. This draft was shared with stakeholders, and we eagerly await their valuable feedback, which will play a crucial role in refining our approach.

Recognizing the importance of effective communication, we meticulously compiled a comprehensive slide deck to accompany our SLDR Presentation. This visually engaging presentation covered the key aspects of our report, offering a condensed yet insightful overview of our project journey, the milestones achieved, and the roadmap ahead.

Codebase Presentation with Freedom Scientific Engineers

A defining moment in our project was the opportunity to present our codebase to a group of engineers from Freedom Scientific. This session provided a platform for us to address queries, gather suggestions, and delve into discussions on the architecture and organization of our project. Such interactions with industry professionals were instrumental in refining our approach and ensuring alignment with real-world expectations.

Enhancements with the Solution Architecture

We delved into enhancing our corpus storage to boost the generation of precise responses. Simultaneously, we explored ways to refine our current solution architecture, aiming for improved context retrieval. Our focus was on selecting corpus documents more effectively, aligning them with the user’s query relevance for an optimized user experience.

WEEK #11: From PID to Clearwater and the Road to SLDR

Hey everyone,

We’ve just had a big week before Thanksgiving break! We had our Prototype Inspection Day (PID) on Tuesday, where we demonstrated our prototype to several rounds of judges. On Wednesday, we traveled to Clearwater to present our prototype to our sponsors Thursday morning. We also had a lecture the following Tuesday to discuss what our SLDR will be like.

Prototype Inspection Day (PID)

On this day, we demonstrated our prototype to three rounds of judges. They varied in their backgrounds, which gave us different perspectives of how we should have conveyed our project. For example, one of our judges was not as aware of how large language models were trained, while another was a machine learning professor. This variance led us to consider questions that we had not considered when thinking about the questions that we would be asked. Additionally, it gave us more insight into how we could change our demonstration itself. Many judges found the more technical parts of our presentation challenging and preferred to focus on our demo.

Clearwater Travel

At Clearwater, we were going to give a presentation to 40+ Freedom Scientific employees, who were mainly involved in the engineering side of the company. Here, we gave an updated and longer version of the presentation that we gave at PID. Then, we gave a demo of our Copilot. Here, we received more feedback that was related to JAWS, given that our audience included advanced JAWS users.

Preparing for System Level Design Review (SLDR)

In our lecture on the Tuesday following PID, we discussed starting our draft for the SLDR report. This report will detail the process of developing a successful deliverable Copilot. That will be what is coming up this next week – we will be finishing up our SLDR draft, stay tuned!

WEEK #10: Prototype Progress and New Tools

Greetings!

We have continued to push ahead with our project this week. Week 10 was all about corpus augmentation and the new tools that came out from OpenAI that would expedite our project progress. 

OpenAI Updates

On November 6, OpenAI announced their Assistant API, which would allow for a streamlined way to integrate calls to GPT 3.5/4 and the retrieval for our model. Although we already implemented retrieval with Pinecone, with this, we could have an alternate approach that is more streamlined and is entirely based with OpenAI. 

Corpus Processing

The team also made updates in having implemented corpus processing behind-the-scenes. This essentially involved making the information that our chatbot has more digestible so that it could answer questions better more often. In essence, this will mean more accurate responses which should be seen in our testing this upcoming week. 

And More

During our lecture this week, we also prepared for our prototype inspection day. This involved discussion of what we would be presenting with regards to the purpose and evaluation of our prototype, what feedback we will receive, and tasks left for preparing for our presentation.

We also affirmed our travel date set for November 15-16 to Clearwater! There, we will more formally present our prototype, and give a presentation to Vispero employees.