This case study describes a drone-based system for automated building facade inspection which is utilizing AIoT for drone management and image-analytics. The system is developed by the Real Estate & Infrastructure Division of TÜV SÜD.
Building facades are an important aspect of buildings, both from an architectural as well as from an engineering perspective. The building facade not only has a huge impact on aesthetics, but also on energy efficiency and safety. Especially in high-rise buildings, the facade can be quite complex, combining a number of different materials, including concrete, glass, steel, polymers and complex material mixes.
Problems with building facades can arise during construction as well as during the building operations phase. Typical problems include cracks in different materials, concrete spalling, corrosion, delamination, decolorization, efflorescence, peeling and flaking, chalking, hollowness, sealant deterioration, and so on. While some of these problems only have an impact on the optics of the buildings, others can have a quite severe impact on safety, e.g. because of facade elements falling down from high heights, increased risk of fires, or even complete collapses.
Facade Inspection
Façade inspection is an integral part of building maintenance, especially for high-rise buildings. It helps to verify the integrity of the building structure and ensures safety for its occupants and people passing by. However,conventional manual facade inspection can be time, labor and cost intensive, as well as disruptive for the building occupants, and dangerous for inspector due to difficult access at height. Finally, the results of manual facade inspection can be subjective, depending on the expertise of the inspector.
In some countries, regular facade inspections are required by regulators. Regulations usually differ, depending on building size and age. For example, in Singapore buildings older than 20 years old and over 13 meters in height have to undergo facade inspections every 7 years. In other countries, the requirements for periodic facade inspections are more driven by building insurance companies.
Automated Facade Inspection
Automated facade inspection solutions must accurately scan the exterior of buildings, e.g. utilizing drones to carrying high-resolution cameras. The Smart Façade Inspection service of TUEV SUED caters to building owners and operators of large high-rise buildings and helps construction companies to ensure façade quality and monitor construction progress.
Customer Journey
The customer journey of the automated facade inspection solution starts with the customer request for the service. Based on the customer information provided, the service operator (TUEV SUED) will preparare the required documentation and apply with the required authorities for drone fligh approvals. The inspection on-site will be carried out by a specialized drone operations team. The data, inspection results and a 3D model of the facade will be made available via a specialized cloud platform.
Customer Benefits
Customer benefits include:
- Results are available in a fraction of the time compared to conventional inspection
- Digital representation of the facade and whole building facilitates building operation
- Automated digital workflow and data benchmarking improve service quality and interoperability
- Domain experts for standards and best practice, ensuring up-to-date compliance to continually evolving regulations
Implementation with AIoT
At the core of the operational system is a smart piloting system for the drone, which ensures both operational safety and high-quality visual inspection. The acquired data is securely managed by TÜV SÜD’s inspection platform, which automatically masks any private information to protect your privacy
The AI-based solution assists professional engineers to deliver compliant inspection reports at the highest industry standard. The software constructs a 3D model of the building façade, which helps to better understand the building structure and automatically locate the detected defects on the building.
The TUEV SUED Drone Façade Inspection application provides access to all the data, report findings, and 3D model at any time. Repairs and follow-ups can be seamlessly managed through the platform to improve efficiency and save costs.
Solution Sketch
Main stakeholders for operations of the drone in the field include the drone pilot, safety offiver, and domain expert. Professional engineers are supporting in the backend. Customer stakeholders include building owners, facility managers, and regulators.
The drone is equipped with a number of sensors to support both, flight operations and building facade scanning. These flight support sensors include IMU, UWB, Lidar and stereo camera. The drone carries thermal sensors and a visual camera as the main payload for building facade data capturing. On the drone, AI is mainly used for drone positioning, collision avoidance and path planning. This is supported by a smart controller device used by the drone pilot on the ground.
A number of backend applications are supporting the management, processing, anlysis and visualization of the captured data. Domain experts and professional engineers can add their domain expertise as well.
Drone control
A key feature of the solution is the advanced drone control, which provides semi-automated path control for scanning the building surface, supporting complex urban environments. Multi-modal sensor fusion is used for navigation. Auto path-planning supports inspection & obstacle avoidance, operational safety of the drone, and ensures high-quality image capturing for the visual inspection.
To support this, the drone carries a miniature, high-performance Inertial Measurement Unit (IMU) and Attitude Heading Reference System (AHRS). The Lidar sensor provides stereo data for dense short range on path obstacle detection (30m). The system also has two stereo cameras for sparse long-range obstacle detection (120m).
Drone data analysis: Facade Inspection
Another key application of AI is the drone data analysis, which is used for creating the facade inspection reports. First, the raw facade data is pre-processed, e.g. anonymizing the captured data. Second, an AI-enhanced image analysis tool is applied to visual and thermal data. Finally, the meta-data is anylyed, utilizing AI to identify individual facade elements, different types of defects, and even detailed defect attributes.
Expert Opinion
The following discussion will provide insights on the TÜV SÜD Drone-based Building Facade Inspection project from Marc Großkopf (Business Unit Manager, Building Lifecycle Services, TÜV SÜD, Germany) and Martin Saerbeck (CTO Digital Services, TÜV SÜD, Singapore).
Dirk Slama: Marc, what were - or are - some of the biggest challenges in this project?
Marc Großkopf: Only opportunities, no challenges! But all kidding aside - of course this is an iterative process, from the initial pilots to the global roll-out which we are currently preparing. In the early stages, challenges tend to be more on the technology and sourcing side. Then you are quickly getting in regulatory aspects, customer acceptance, data quality, internal acceptance and processes, regional differences, etc. So it's never getting boring.
Dirk Slama: Martin, from the CTO perspective, what were some of the initial challenges?
Martin Saerbeck: On the technology side, we have two main aspects: Drone-based image capturing and the data platform. For the drones, it is very much about striking a good balance between cost, flight capabilities, quality of the sensors, and of course establishing the supply chain which can support us globally. For the data platform, we need to be able to support stakeholders with different backgrounds, roles and responsibilities. The user interface must be intuitive even if back-end AI algorithms can be quite complex.
Marc Großkopf: Yes, don't forget that we have quite a complex constituency - drone operators, data scientists, domain experts, customers, and so on - all need to be supported by the central Facade Inspection Platform.
Dirk Slama: Let`s start by looking at the drones and drone operations...
Martin Saerbeck: Of course you need to get the initial platform set-up right. There are a lot of powerful drone platforms out there, but we need to adapt them to our needs and not the other way around. One example is implementing automated flight path control to ensure facade coverage and high quality images. But maybe the biggest challenge is keeping up with the constant flux of technology and changing regulatory requirements in different regions. Take, just as an example, free-flying vs. tethered (i.e. cable-bound drones). There are many different opinions on what should happen if the tether fails: Are we allowed to automatically switch over to the drone battery for safe landing or not? How much time do we to have until we need to trigger an emergency routine? What exactly constitutes a tether-failure? The list goes on. For us it is therefore important to be directly involved in standardization committee work - locally and globally.
Marc Großkopf: The technical people tend to focus on the "sexy" stuff first - AI, automation, image analysis, and so on. But we also need to look at drone maintenance, firmware updates and battery management. And of course on-site support like system set-up, traffic management, etc. At the end of the day, this needs to be so efficient and effective that the overall process is cheaper than the manual process. And we need to make sure that we have enough in-house knowledge before we can source this regionally. We can`t take any short-cuts because we need a solid foundation and have to avoid to build up technical debt because of cost-driven supplier situations.
Dirk Slama: Let`s talk about the back-end platform. How does this look like?
Marc Großkopf: Depends on who you talk to. For the drone pilots and on-site staff, the platform mainly needs to support management of image uploads. For the domain experts, we need an efficient way of reviewing and labeling the image material. This process is more and more supported by AI now. And finally we have the end customers who access the platform to get the final results and reports. As an added challenge, they want to use the platform to monitor and manage building defects."
Dirk Slama: This means the platform is not magically smart and fully automates the inspection process from the start via image analysis?
Marc Großkopf: It gives us a fully digital process from the beginning, since we now have a process for efficiently capturing and managing the image data. This is already an important step. And we are now gradually using our huge network of building facade domain exerts to label relevant data and then use this to train the system. This means over time we get more and more automation. Initially by pre-filtering huge amounts of data, so that the domain experts only have to review relevant image data. And then also more and more with automated classification.
Dirk Slama: How does this look like?
Martin Saerbeck: Based on the labeled data from the domain experts, our data scientists are accessing the platform via standard developer tools to build out the library of defect detection algorithms. These algorithms vary depending on the defect type and the facade materials. Cracks need different detection algorithms than spallings. Glass facade are different than metal or concrete.
Dirk Slama: When can you re-train, and when do you have to develop new algorithms?
Martin Saerbeck: It depends. For example, for cracks in different concrete types, we can use transfer learning to a certain extent. However, detecting and evaluating cracks in glass requires models that we essentially train from scratch.
Dirk Slama: What about privacy?
Martin Saerbeck: Very important point. Especially if the drone is likely to inadvertently capture people throughout the scanning process (e.g. standing behind windows), we need to automatically identify and anonymize this. Privacy preservation is key. We spent considerable effort on this portion.
Dirk Slama: So how are your scaling this up for global roll-out?
Marc Großkopf: First, we have to ensure regional support for the drone service. This means dealing with local regulations, finding local service partners, suppliers and so on. Then we have to make sure that our processes can be easily replicated: how do you execute a drone-based building scan, how are our domain experts working with the data, how can our central competence center in Singapore best support the regions with re-usable fault detection algorithms, and how do we best onboard and support our customers in the regions.
Dirk Slama: Your current focus is on building facades. Can you apply your lessons learned also to other use cases?
Martin Saerbeck: Sure. Let`s take, for example, building construction progress monitoring. There are a lot of similarities here. This is an area where we are following a similar approach, together with our partner Contillio, which is focusing on Lidar and AI for analyzing the construction progress and mapping this back to the original BIM models. But of course you can take a similar approach also to inspection of power plants, solar panels, etc. A lot is happening, but we have to take it step by step!
Dirk Slama: Thank you, Marc and Martin!