This week, we did our second qualitative review board presentation. The biggest takeaway was ensuring that we have a good way to test and validate our process, in order to both ensure it works, and determine where any issues may be.

We have a couple of approaches to how we can demonstrate the performance of our pipeline. First is, of course, just running the pipeline on the data, and comparing the final values to the true orientation of the antennas. We recently received data from Verizon documenting the orientations of all the antennas in the videos they sent, according to the third-party software they have been using in the past.

But what if our results aren’t immediately perfect? (Hard to imagine, I know.) Well then we’ll need to examine the individual steps of the pipeline to discover where the error is coming from. In most cases, this can be done by examining the intermediary results between these steps. Check how often the object detection is missing the antenna, make sure the 3D tagging isn’t losing track of anything, etc. Where this becomes challenging is in the angle calculations, and particularly in the Structure from Motion portion of the algorithm.

Structure from Motion relies on several parameters which can act as points of failure, and few if any meaningful intermediary results between them. The most notable risks among these are in the camera calibration step. The result of this step is to determine how the intrinsic features of the camera itself (zoom, lens distortion, etc.) are affecting the image so that we can correct for them, and ensure we know exactly where a given pixel refers to. The problem is that we don’t have a drone with which to perform this calibration ourselves. We don’t know how much these parameters might change between cameras even of the same model, or with different camera settings.

We could use a camera we do have, and perform the calibration on that, but since we can’t use that camera to record the cell towers, we wouldn’t be able to demonstrate much besides just that Structure from Motion, as a baseline concept, works (which has already been pretty well proven by countless others).

So if we can’t go to the towers, we’ll simply bring the tower to us.

In order to test the accuracy of the camera calibration data we’ve been supplied by Verizon, we can run our pipeline using a known camera, taking video of a rough analogue of a cell tower that we designed. The 3D model shown above will be 3D printed, so that if need be, we can see how the pipeline performs using a camera for which we have done the calibration ourselves. Each antenna on the model is at an angle we can measure, allowing us to compare the final results of the pipeline to the true value. This even provides some value over the data supplied by Verizon’s third party company, as that is already an approximation. This allows us to be even more confident that our results are accurate, and within the necessary parameters.