This week, we started on our solution. It will involve Dijkstra’s algorithm, which finds the shortest path between points on a graph. The locations on our webpage will be modeled as a graph. For example, a search bar will be one node (point on the graph where one can be) and a paragraph of text could be another node. We currently believe that the distance between every node will be 1, although this might change in the future if some actions are more difficult.

We did come across a problem that we expect to be the most difficult challenge of the project. We need to recognize what a user is intending to do from only a history of their past locations and actions. We conducted a literature review on this problem, but were not able to find solutions to similar problems. We are in the process of contacting University of Florida faculty to help look for solutions. We think this problem is classified as intent recognition, but determining intent can be difficult when actions are not optimal.

Since we have gone back to the drawing board, we needed to come up with a new project concept to continue with this semester. Our liaison has informed us that he will not be able to provide big data from the company due to internal complications and concerns with user privacy, however, we can potentially expect specific and encrypted data for next semester.  

The main problem we have encountered when brainstorming was that we cannot use a machine learning approach currently due to the lack of data. Even if the data was available to us, we would require “labeled” data for efficient vs inefficient keystrokes. However, this type of problem, with all its moving parts, seems to require the use of machine learning to get the most accurate results.  

We came to a dead-end… and we thought that we might need to change our scope of work to adapt it to another problem that Freedom Scientific requires assistance with.  

Nevertheless, during our in-person meeting with our liaison, Sriram, we came to a solution! We are going to be using graphing theory to plot the website as a graph and then use a weighting system to scale the different potential “paths” that the user can take. Currently, we are still uncertain as to how to detect the user’s end action (the action that they wish to achieve), but we are confident that this concept is a viable solution! 

Utilizing Visual Studio Code, our team created an algorithmic demo in Python to suggest keyboard shortcuts based on “fake” inputted data.  For our demo, we focused on the end actions that a user would want to achieve. As seen in the image below, we used keyboard keycodes to assign numbers to each of the keyboard keys. Additionally, the demo successfully detected three different end actions. The first two suggested a keyboard shortcut and the third option detected that the shortcut was used.  

During our meeting with the liaison, we demonstrated the demo above. His main point is that we will be generating too many false positives with our current approach. Therefore, next week, we are brainstorming alternatives to potentially leverage big data instead of just having a purely algorithmic approach.  

We are not able to share code.

This week, we started on our demo. Currently, we plan to work on a recommender system that utilizes machine learning. After speaking to our liaison, Sriram, we recognize that an important goal of this project is to utilize the big data that can be only be collected by the company producing JAWS, Freedom Scientific.

Our approach to a nudging system involves more of a traditional algorithm, involving regular expressions to find equivalent actions within the JAWS program. In the future, we may explore problems that big data provides a significant advantage in solving.

Next Friday, we will be going to dinner with Sriram, Dr. Villanueva, and potentially some engineers with Freedom Scientific. We are excited to learn more about JAWS and how it can be made even more useful, potentially with a nudging system.

This week we started solidifying the ideas we had for the framework behind our software. We are excited to discuss and refine our ideas with the liaison so that we can move on to building a small demo to show the concept in action.  

Our other main priority this week was to define the format of the data for our liaison to provide. Currently, the data will involve tracking keystrokes, the element that the user is tabbed on, and the time taken in between keystrokes.  Moreover, we also want to get a general idea of what JAWS can track so we know we have all possible relevant data. Furthermore, it may be difficult to determine the optimal keystrokes since we are not experts in JAWS. 

A hurdle that we look forward to overcoming in the next few weeks is the learning curve for the software tools we will need to use to develop our first prototype. 

Additionally, we received feedback on our team logo in class from our classmates. Below is our updated logo: