By Week 9, the focus of our work began shifting from infrastructure and deployment to something equally important for any AI system. Data. Not just any data, but the right data.
This week started with a dedicated project work session where the team focused on development progress and took a deeper look at the document ingestion pipeline. Now that our cloud environment is coming together, the next challenge is making sure the system is pulling meaningful information from the company’s document ecosystem.
We continued our weekly coordination meeting with the Comsys team, where we reviewed technical progress and discussed different strategies for ingesting documents from the company’s server structure. These discussions helped clarify the practical constraints around document formats, permissions, and how information is actually organized in the real environment.
Later in the week, we held an important stakeholder meeting with Ashley to review the company’s folder hierarchy. This conversation turned out to be extremely valuable. Instead of guessing which files might be useful, we were able to walk through the actual structure used within the organization and identify which datasets would provide the most meaningful information for our proof of concept.
As we began evaluating documents, we discovered something interesting. Many PDF files contained mostly images rather than extractable text. While these files may be useful for humans, they do not always provide enough textual information for embedding generation within our retrieval pipeline. This insight helped us refine our ingestion strategy and avoid wasting processing time on documents that would contribute little to the system.
One dataset quickly stood out as particularly valuable. OAC meeting minutes. These documents capture key project updates, schedule discussions, RFIs, and change orders. In other words, they summarize exactly the type of information someone might want to query when using the system.
Based on this discovery, we selected the Project Management directory as the starting point for our proof of concept ingestion pipeline. Image heavy folders such as project photo directories will be excluded for now while we focus on structured documents that provide meaningful text content.
Looking ahead, the team will begin implementing the ingestion pipeline using OAC meeting minutes as the initial dataset. Once the pipeline is validated, we plan to expand ingestion to additional directories and continue refining document filtering methods.
The project remains on schedule, and this week helped clarify an important principle. A powerful AI system does not just need infrastructure and models. It needs the right information flowing into it.
Week 9 was about finding that signal in the noise. 🚀