Our audio team successfully migrated the model from Tensorflow to Pytorch and completed grid search optimization for the 1D CNN hyperparameters. The video team focused on CLIP development, working with the text encoder and improving zero-shot accuracy while also running grid searches for optimal CLIP hyperparameters. We achieved an important milestone by implementing late fusion with our current CLIP and 1D MFCC CNN models, evaluating their combined performance on CMU-MOSEI test data.

For the coming week, we’ll focus on comprehensive performance documentation, including detailed data samples from our training and testing sets. The audio team will showcase audio samples from each dataset, while the video team explores bounding box implementation, black and white image processing, and cross-dataset training. We’re expanding our focus on generalizability by conducting cross-dataset training and testing, incorporating multimodal datasets, and utilizing training data that better represents our use case. Finally, we’ll be integrating text and AU analysis components into our fusion model.

This week, we successfully fine-tuned our CLIP model with an 80-20 train-test split, achieving 84% accuracy. Our audio team made progress by implementing grid search capabilities for the audio modality. We delivered our DFX presentation in class and expanded our dataset by annotating additional data for CLIP. We also integrated our best-performing models into the fusion system and successfully produced output.

Next week, we’ll evaluate late fusion performance and conduct grid searches for audio and CLIP models using Hipergator. The audio and CLIP teams will incorporate preprocessing steps into the intermediate fusion model, followed by training and performance reporting. We’re also preparing comprehensive documentation slides detailing each model’s specifications, including input/output shapes, performance metrics, next steps, and dataset information.

This week, we presented our progress at QRB1 and received valuable feedback from various coaches. We shared detailed model performance metrics with our liaison, and we’ve achieved our target accuracy goals for audio, transcript, and action unit recognition as specified in our Technical Performance Measures. We have also implemented a joint-representation multimodal transformer architecture for the fusion model.

For the upcoming week, we’ll focus on enhancing our CLIP model through image-text pair training and hyperparameter tuning. We’re moving forward with end-to-end testing of our fusion model using our highest-performing components. Additional tasks include training CNN and ConvLSTM models for audio emotion recognition, testing FG-Net accuracy for emotion-specific markers, and acquiring supplementary training data. Our project continues to progress on schedule as we move toward our integration goals.

Our team made significant progress in model development this week. We tested the accuracies of Recurrent Neural Networks and Support Vector Classification for the audio modality and set up EmotionCLIP for accuracy evaluation. We initiated the FG-Net system for Facial Action Units training and testing. We also established our intermediate layer fusion code, which we tested using off-the-shelf ResNet and MobileNet models.

Next week, we’ll complete the initial testing of individual components like CLIP, EmotionCLIP, spectrogram CNN, and spectrogram ConvLSTM to ensure smooth integration into our fusion model. We’ll finalize our QRB1 presentation, share testing results with our coach and liaison, and integrate selected models into our fusion implementation. Our project continues to progress systematically toward our goals.

Our team hit the ground running in our first week of the spring semester. We established new meeting times to accommodate everyone’s schedules and developed our critical path for the semester ahead. After creating our January work breakdown schedule, we received valuable feedback from both our coach and liaison. We’ve organized into specialized teams to focus on fine-tuning individual modalities, setting the stage for our upcoming development sprint.

Looking ahead to next week, we’ll begin development of our fusion model while identifying and fine-tuning pretrained models for each modality. The project remains on schedule.