This week, we presented to Freedom Scientific. We focused on the graph part of our solution, which is a very important part of it. As we are representing a webpage as a graph, we are able to find the shortest paths using Dijkstra’s Algorithm, which scales well to larger webpages. We also learned more about the limitations of our solution. It is not a problem if we provide a nudge several hours after a person makes a suboptimal action, as they still get the nudge. This might not be ideal, but it could still be helpful for a user.
In our presentation, we explained what a graph is, as that is an essential part of our work. The blue circles are nodes and the black lines are edges. The edges are potential locations of the virtual cursor and the edges connect them with keystrokes. For example, a user could press the down key, which is represented as a black line, to move between one of the top and bottom nodes. Below is an explanation of how we represent a webpage using a graph.
This week, we started on our solution. It will involve Dijkstra’s algorithm, which finds the shortest path between points on a graph. The locations on our webpage will be modeled as a graph. For example, a search bar will be one node (point on the graph where one can be) and a paragraph of text could be another node. We currently believe that the distance between every node will be 1, although this might change in the future if some actions are more difficult.
We did come across a problem that we expect to be the most difficult challenge of the project. We need to recognize what a user is intending to do from only a history of their past locations and actions. We conducted a literature review on this problem, but were not able to find solutions to similar problems. We are in the process of contacting University of Florida faculty to help look for solutions. We think this problem is classified as intent recognition, but determining intent can be difficult when actions are not optimal.
Since we have gone back to the drawing board, we needed to come up with a new project concept to continue with this semester. Our liaison has informed us that he will not be able to provide big data from the company due to internal complications and concerns with user privacy, however, we can potentially expect specific and encrypted data for next semester.
The main problem we have encountered when brainstorming was that we cannot use a machine learning approach currently due to the lack of data. Even if the data was available to us, we would require “labeled” data for efficient vs inefficient keystrokes. However, this type of problem, with all its moving parts, seems to require the use of machine learning to get the most accurate results.
We came to a dead-end… and we thought that we might need to change our scope of work to adapt it to another problem that Freedom Scientific requires assistance with.
Nevertheless, during our in-person meeting with our liaison, Sriram, we came to a solution! We are going to be using graphing theory to plot the website as a graph and then use a weighting system to scale the different potential “paths” that the user can take. Currently, we are still uncertain as to how to detect the user’s end action (the action that they wish to achieve), but we are confident that this concept is a viable solution!
Utilizing Visual Studio Code, our team created an algorithmic demo in Python to suggest keyboard shortcuts based on “fake” inputted data. For our demo, we focused on the end actions that a user would want to achieve. As seen in the image below, we used keyboard keycodes to assign numbers to each of the keyboard keys. Additionally, the demo successfully detected three different end actions. The first two suggested a keyboard shortcut and the third option detected that the shortcut was used.
During our meeting with the liaison, we demonstrated the demo above. His main point is that we will be generating too many false positives with our current approach. Therefore, next week, we are brainstorming alternatives to potentially leverage big data instead of just having a purely algorithmic approach.
This week, we started on our demo. Currently, we plan to work on a recommender system that utilizes machine learning. After speaking to our liaison, Sriram, we recognize that an important goal of this project is to utilize the big data that can be only be collected by the company producing JAWS, Freedom Scientific.
Our approach to a nudging system involves more of a traditional algorithm, involving regular expressions to find equivalent actions within the JAWS program. In the future, we may explore problems that big data provides a significant advantage in solving.
Next Friday, we will be going to dinner with Sriram, Dr. Villanueva, and potentially some engineers with Freedom Scientific. We are excited to learn more about JAWS and how it can be made even more useful, potentially with a nudging system.
This week we started solidifying the ideas we had for the framework behind our software. We are excited to discuss and refine our ideas with the liaison so that we can move on to building a small demo to show the concept in action.
Our other main priority this week was to define the format of the data for our liaison to provide. Currently, the data will involve tracking keystrokes, the element that the user is tabbed on, and the time taken in between keystrokes. Moreover, we also want to get a general idea of what JAWS can track so we know we have all possible relevant data. Furthermore, it may be difficult to determine the optimal keystrokes since we are not experts in JAWS.
A hurdle that we look forward to overcoming in the next few weeks is the learning curve for the software tools we will need to use to develop our first prototype.
Additionally, we received feedback on our team logo in class from our classmates. Below is our updated logo:
This week, we got to meet our IPPD faculty mentor, Idalis Villanueva Alarcón (ella/she/her). She is an Associate Professor at the University of Florida and received the NSF CAREER award for work in the hidden curriculum in engineering.
We would like you to meet our logo, the Key-saurus.
We also worked on our scope of work. We drew a diagram showing the inputs and outputs of our project, which involves users, the Windows operating system, JAWS, and computer software among other things. Through this, we have learned more about how different components of our product will interact and how our product will interact with other products.
Hello! We are Sumanth, Erynne, Ryan, and Mariana. This week we met our team liaison and received a brief outline of our project. We also started work on finalizing a team name and logo. Here is a bit more information about our members and one fun fact about us:
Sumanth Aluri – Fourth year CS student: Addicted to Boba, Anime, and coffee.
Ryan Aponte – fourth year CS student. Fun fact: My parents went to UF.
Mariana Viso – Fourth year BME student. Biggest weakness: anything with cilantro.
Erynne San Antonio – Fourth year BME student – I play the violin and am teaching myself how to play the kalimba.
Our plans for the next couple of weeks include working and understanding the JAWS software, starting to define the data, and starting an initial analysis of the data to see if it is the right fit.