The past several months have been momentous; Raytheon Technologies commissioned the IPPD program to come up with a team who could improve a Capture The Flag platform used in their cybersecurity training programs. IPPD rose to the occasion, and assembled team Pythax. This team of 4 worked together to add features to the platform, fix bugs, update the UI, and generate new content for use in their internal training competitions.

Curious about Pythax’s work this past year? Here’s a poster outlining our project goals, methods, and findings:

It’s been a long and fruitful year, and as Pythax is finally nearing the end of the IPPD2 semester it’s about time to showcase our work! Here’s a video to catch you up to speed on the Hacker Jeopardy project–what inspired RTX, what the project aims to do, and a peek at what it looks like. Ladies and gentlemen, Hacker Jeopardy:

This week was busy! On top of completing CTF2, Pythax also presented the current state of their project to several panels of judges during Prototype Inspection Day (PID). This was an opportunity to receive feedback from UF faculty on the UI and UX implementations, as well as our testing plans, results, and feature additions. 

Most of the feedback received was positive, the judges seemed to be very pleased with the UI and testing plans, for the most part. A common criticism regarding the UX was that the admin tables didn’t seem to accomplish their intended goal of being more readable, as they still had transparency which lowered the contrast with the background, making text hard to pick out. Another commonality between judges’ feedback was that the admin features need to be tested. This makes a lot of sense, but was something that the team hadn’t included in their plans because the users in test CTFs can’t act as administrators when they’re answering questions for a course grade or extra credit. A suggestion was made to have smaller groups of peers try to complete certain tasks using the full suite of admin features, to see how a human would interact with the platform from that perspective. Pythax will be adding this to their testing plans.

Aside from the above criticisms, the team was asked to provide more analytics and data from the user feedback forms, instead of one number representing user satisfaction which seems arbitrary. This is a serious area for potential improvement to the team’s pitch. 

The final presentations are on the near horizon for IPPD2, and team Pythax is going to be prepared, thanks to all the judges who gave valuable feedback!

This week, Pythax was able to finally host a CTF competition using the updated platform with all the changes that the team has been hard at work implementing this semester. As per last week’s post, this competition was run using the Software Security class here at UF as test subjects. Students received a quiz grade and were able to earn extra credit for the class by participating and submitting feedback on a questionnaire afterwards. 

This CTF was a great opportunity for Pythax to gain some critical feedback on their work, specifically the changes that have been made to the non-admin user views. This includes the game board, login & registration, and score board pages. 

CTF1 was run using the original platform that was given to the team by Raytheon Technologies (RTX), and the sample size for that CTF was estimated to be around 12 participants (although it is hard to say for sure, since teams can vary in size). This time around, the CTF had a whopping 42 discrete survey submissions, which is a much better turn-out! 

One of the areas from the feedback form that received criticism from judges in recent presentations was that Pythax received a “thumbs up” from 100% of the users in CTF1. The same reaction came from CTF2, with all 42 surveys saying they enjoyed using the platform. This time around, however, more questions were included to gauge the users specific reactions and opinions to certain features of the app, and we were able to get more honest feedback as a result. 

Some of the feedback included complaints about server lag, having to format/spell answers perfectly, and questions not being related to their class. Some users indicated that they would like to see some UI enhancements, and some features were suggested as additions/improvements. Some of these features had already been considered by the team, but were left as stretch goals or had been marked as undesirable in meetings with our liaison engineer from RTX. 

All in all, CTF2 was a major success! Lots of feedback was gathered for the team to deliberate on and implement for CTF3, which will be hosted in early April. We’re looking forward to honing our work and making the Hacker Jeopardy platform as fantastic as possible!

Pythax would also like to thank Dr. Byron Williams and his students for participating in CTF2 and providing wonderful feedback.

This week, Pythax had the chance to speak with Dr. Byron Williams, who teaches software security. The team plans to host its second CTF using students from Dr. Williams’ class as test users. 

Diego and Tyler have developed a set of new questions for the game which align more closely with the software security curriculum, and Dr. Williams seemed very pleased with the content that is going to be covered. 

After completing the CTF, students will be asked to fill out a quick survey on their experience with the platform. This survey has been updated since the previous CTF in order to gain more quantitative feedback. Students will be allowed to make teams of up to 4 people each to collaboratively attempt challenges. Their overall performance, measured by their team’s score at the end of the CTF, will be used to assist their course grades.

The CTF is scheduled for the beginning of next week, and team Pythax cannot wait to see what users think of the new platform!

When developing a project, critical feedback is one of the most valuable things a team can hope for. This is the intention of a quality review board (QRB), to identify what elements of a project need work before it’s too late to make corrections. Pythax had already participated in one QRB and made changes to their project’s plans based on the feedback received. Recently, Pythax participated in a second QRB with a new panel of UF faculty-judges, and was hoping to receive new feedback to improve the project plan even further.

Some of the feedback received was expected; during QRB1 a weakness was identified with the metrics being applied to measure user experience and satisfaction of the platform, and with a lack of new CTFs there had not been much development in that area. Unfortunately, this flaw which had already been identified and noted by the team ended up being the focal point of the feedback from the judging panel on QRB2. This was a bit disheartening for Pythax, as the team members believed the known issue to be properly addressed in the presentation, and the time spent talking about it was time not spent talking about new feedback and criticisms. However, it is still valuable feedback to the team, as it has been made very clear what needs to be focused on for the next few checkpoints.

Other feedback was not expected; the team was grilled a bit on the basis that this presentation did not include any demo of the product in its current stage. Instead of showing what has been added, the team talked about what has been added, and the judging panel wanted to be able to see what changes had been made. Ultimately a demo was given to the judges, but it was done on the fly and cut into the time for feedback. This surprised the team a bit, since there was no qualm with the lack of a demo during QRB1, and it was believed that the charts and documentation were supposed to be the main focus of the presentation. Still, this feedback is valuable as it will help to improve Pythax’s future presentations and demos of the product.

This project has been interesting in that the team has entered many different phases of work. The various stages of the typical development cycle supplemented by large amounts of documentation mean that Pythax members have been able to wear many hats along the way to the project’s completion. We currently find ourselves in a period of change from one mode of working to another, as we shift from adding features to adding questions.

In preparation for the upcoming CTFs with volunteer-users from a couple of UF classes, the team has divided amongst one another the 2 topics of question to be completed. Diego and Tyler are more familiar with the first class of the 2 being used, Software Security, so they will be working on those challenges. Meanwhile, Boyd and Harry will be generating challenges and questions for Enterprise Security. 

One of the team’s technical performance measures–the items which are checked to determine the team’s success–which has not seen much progress is the generation of new challenges for the competition. It will be a nice weight off of the team’s shoulders to finally spend some time knocking that TPM out of the park.

As the semester moves along at a brisk pace, Pythax is moving from their development phase into a testing cycle. A big part of these tests include user testing, which requires real people using the platform to ascertain how tasks are being performed and whether the UI is intuitive enough. 

In order to conduct some user tests of the platform from the perspective of a player/team in the game, Pythax is planning to schedule and run 2 more CTF competitions on top of the 1 already completed. The survey from the initial CTF was a weak point on the team’s QRB feedback, so the questionnaire is being modified in order to reflect the users’ experience more accurately. 

Currently Pythax is in contact with 2 UF CISE professors who are teaching security courses, ready to let Pythax use their classes as test subjects. The team will be working diligently over the next few weeks to develop a full range of new challenges and questions to cater the CTF’s content to the curriculum of these 2 UF courses.

Pythax has also spoken with Raytheon Technologies regarding the potentiality of running a CTF with their employees in the next couple months. This CTF, if timed properly, could be extremely beneficial to the team for gaining user feedback from the perspective of a game administrator. This role is usually filled by Pythax members during the CTFs run on UF classes, but a lot of new features have been added for the admin so it would be useful to see if these improve the experience or make it worse.

After the Quality Review Board (QRB1) presentation, Pythax got some valuable feedback from the panel of faculty judges. 

The first piece of feedback that the team received was regarding the questionnaire that was given to participants in the initial CTF trial-run. The questionnaire was intended to gauge the users’ experience (UX) as they interacted with the platform. The judge panel pointed out that there were some flaws with the questionnaire, namely that the questions themselves weren’t specific or quantitative enough. They also seemed slightly disappointed in the number of responses that had been received. Future CTF’s that Pythax runs will definitely focus more on making the questionnaire a more reliable source of information.

Secondly, the panel was concerned about Pythax’s means of testing and measuring their success for various tasks. The tests simply need to be more specific to indicate whether something has truly been fixed, improved, implemented, or otherwise completed.

Beyond this, the only criticisms the team received were regarding the language used to talk about certain aspects of the project. For example, in future presentations the team will need to clarify further how the CTF is conducted each time around. 

This feedback was extremely valuable and Pythax will benefit from improving on the areas that were pointed out to be vulnerabilities. The good news is that the schedule and progress of actual feature implementation was not put under any real scrutiny, so until the team needs to run another CTF, business as usual!

The first presentation of the spring semester is coming up: the Quality Review Board 1 (QRB1)

The purpose of this presentation is for Pythax to gauge their progress in the first couple of weeks, and assess their plans for the semester. This presentation is not open to peers, only a handful of faculty coaches who will give the team a critical perspective of their own architecture, roadmap, and progress. 

Going into QRB1, Pythax feels prepared. Tasks are moving along on schedule and the plans have been in place for awhile now. A Trello board as well as a ClickUp schedule show the relevant tasks and deadlines for each member of the team, which makes tracking of progress quite easy to see.

Ultimately, the team is happy to finally be getting our hands dirty by making some real progress on the project. The platform itself is starting to look different and work better, and we’re ready to show what we’ve done!