Collins Engineers is working with the Minnesota Dept. of Transportation on the restoration of the James J. Hill Stone Arch Bridge in Minneapolis. The 19th-century masonry structure is due for repairs, and Collins has been using Microsoft HoloLens headsets with software from Bentley Systems to speed inspection work during the design phase. “The HoloLens gives us that sense of scale we only get in the field,” says Barritt Lovelace, director of UAS, AI and reality modeling for Collins Engineers. “When looking at a computer screen you can zoom in, but it doesn’t feel like the bridge is in the field. With HoloLens it is different.”

Bridge inspections have always been dangerous, time-consuming work, but for assessing the condition of a structure, nothing beats being there. The Bentley mixed-reality workflow aims to bridge the gap between staring at photos and being on site. The workflow takes high-resolution drone photographs of the bridge, and stitches them into a “reality mesh” 3D model comprised of tens or hundreds of millions of polygons. This mesh contains far more information than is found in regular 3D models, says Dan Vogen, vice president, road and rail asset management at Bentley Systems. “It has all the texture, all of the current conditions, conveyed through the reality mesh.”

The reality mesh can be viewed on a computer or tablet, but a cloud-connected HoloLens offers a different viewpoint. There, the bridge can be viewed at a zoomed-out tabletop view or at 1:1 scale, letting engineers get up close and see every inch of the structure in high resolution. The engineer then can add annotations, perform measurements, and highlight potential problem areas.

“To me, the most impactful thing of putting your inspection notes on a model compared to traditional notes is you no longer have to describe where the defect is,” says Lovelace. “It definitely speeds up the process, and allows us to give MN-DOT better data.” While a HoloLens inspection is not meant to replace in-person inspections, Lovelace says it has reduced the number of trips needed to inspect the bridge and allowed engineers in other cities to help with the design work of the repairs without having to travel to Minneapolis. “We don’t have the budget to send everybody we want out there all the time,” he says. “But if someone in Chicago wants to help on the design of the cofferdams, they can look in HoloLens at the model.”

The HoloLens bridge inspection workflow is currently part of Bentley’s AssetWise system, and the reality meshes are linked up to the digital twins maintained of existing structures. Vogen says this is a natural next step for the digital twin approach. “We were told our digital twins were a solution looking for a problem,” he says. “But here is a problem we can really solve.” He adds that bridge inspection is only the first application for this workflow, and Bentley is already exploring Hololens as an inspection tool for other difficult-to-access structures, including communication towers and electrical utility transmission towers. 

State Bridge Inspector Sees Promise in Drones, HoloLens

The Bentley HoloLens workflow has also drawn the attention of MnDOT, which has been investing in drones for bridge inspection over the last few years. "We're now we’re at the point where we have these drones now, we are going to use them for inspection. So we have the data, how do we make it useful?" says Jennifer L. Wells, MnDOT state bridge inspection engineer. "Now along comes Bentley and Microsoft, and we can share this data across different entities in MnDOT."

Wells has spearheaded the effort to bring drones into MnDOT's bridge inspection regime, and has worked with Collins Engineers in recent months to refine Bentley's HoloLens workflows. MnDOT has policies in place governing drone flights for bridge inspections to comply with FAA regulations, and Wells says a version of this workflow could become a standard practice for bridge inspections in the state someday. "[Drones] are another tool in our toolkit," she says. "It can save a lot of time and money in terms of lane closures."

And while MnDOT and other state DOTs may have the resources for inspections, Wells points out that many local municipalities have a much tougher time inspecting all of the bridges and crossings in their jurisdictions. "We've seen some scope creep when it's time to close a bridge, because local agencies are not getting the access they need ahead of time [to inspect the bridge]." Spending a few thousand dollars on a drone and getting FAA certified as a drone pilot is one way for smaller agencies to get through their inspection work load quicker, Wells says. 

But new technologies like reality meshes built from drone captures are only a part of the broader bridge-inspection process, she cautions. "It gives us a baseline to show changes over time, and show different departments within MnDOT," says Wells. "It's more of a first-line discussion creator—a better way than showing a picture from my smartphone to my load-rating engineer and asking 'can you draw up a load rating based on this?'" Working in a reality-mesh-based 3D model can solve a lot of those communication problems upfront, she says.

With Collins Engineers' work on the Stone Arch bridge moving along, Wells adds that work has already begun on the next project, building out drone-captured reality meshes of the Blatnik Bridge, a nearly 60-year-old through-arch bridge that connects Duluth, Minn. to Superior, Wis. Wells says that she expects the drone captures and the new HoloLens workflow will help speed along inspection work on the aging crossing.