Augmented Reality Heritage

The research project concerns the development of digital technologies to enhance the accessibility and the management of cultural sites and the exploration of the related digital information. The project is currently employing artificial intelligence to make data networks and digital environments accessible from the physical space. We developed a mobile app that allows users to access data about buildings and works of art just by pointing the camera at the object. The app is capable of connecting the real city to relevant, context-aware documents such as images, texts, maps, and 3D models. It builds on original developments of convolutional neural network techniques aimed at recognizing architectural features. The app offers an access point to a yet under-exploited network of digital information, not through a catalogue or a predefined route on a map, but just by framing the urban context through a mobile camera. A vast multimedia information can be linked to the elements of a city, answering questions on how to make the newly available information easily and sustainably reachable.

Year

2018-2020

Tags

#ArtificialIntelligence #DigitalTechnologies&DataScience

Type

Research project

Arch•i – Architectural Intelligence

The city is now producing brand-new information on itself, in terms of quality and kind, in the form of data that can be stored, organized, analyzed. Current technical needs are format standardization, information gathering, management and selection, data processing and visualization… This is leading to unprecedented developments of the tools. New networks of relationships among documents can be defined and rapidly redefined, according to continuously updating needs and contents. In this complex information topology, the physical form of architecture maintains a key role, on which even the most up-to-date elaborations can be founded.

Italy has 49 cultural sites on the UNESCO World Heritage Sites list. Despite this, only the 0.7% of Italian GDP is allocated to culture1. Sites such as the Imperial Fora in Rome or Pompeii and Herculaneum host millions of visitors each year but can’t provide appropriate informative services on site and face management and maintenance problems. At the same time, due to low tourist flows, many small, isolated or less known historical and archeological sites, cannot afford surveillance and maintenance and thus are not accessible. We need to optimize the available resources in order to guarantee the protection of cultural heritage and to enhance its value.

The research project called Arch•i – Architectural Intelligence concerns the development of digital technologies to enhance the accessibility and the management of cultural sites and the exploration of the related digital information. We think that mobile computing technologies can overcome the limitations of traditional information tools and allow novel interactions with monuments and works of art. The project is currently employing artificial intelligence (AI) to make data networks and digital environments accessible from the physical space.

Deep learning for architecture

In recent years, the diffusion of large image datasets and an unprecedented computational power have boosted the development of a class of AI algorithms referred to as deep learning (DL). Among DL methods, convolutional neural networks (CNNs) have proven particularly effective in computer vision, finding applications in many disciplines. While AI is just beginning to interact with the built environment through mobile devices, heritage technologies have long been producing and exploring digital models and spatial archives. Hence, the digitalization of cultural information offers structured and ready-to-use sources of knowledge that can be retrieved through the flexible features of AI. The interaction between DL and state-of-the-art information modeling is an opportunity to both exploit heritage databases and optimize new object recognition techniques. A specific approach to automated architecture recognition could change the way in which data on the urban environment are collected, processed and analyzed, and could provide more effective ways to access data.

The Arch•i project developed a mobile app that allows users to access data about buildings and works of art just by pointing the camera at the object. The app is capable of connecting the real city to relevant, context-aware documents such as images, texts, maps, and 3D models. It builds on original developments of CNN techniques aimed at recognizing architectural features. The app is based on two main blocks of software: (1) an online, geographic-enabled database that makes it possible to upload different types of document and the related information or metadata; (2) the DL part, which is stored on the device and requires a very small amount of disk space.

The app offers an access point to a yet under-exploited network of digital information, not through a catalogue or a predefined route on a map, but just by framing the urban context through a mobile camera. A vast multimedia information can be linked to the elements of a city, answering questions on how to make the newly available information easily and sustainably reachable.

What's next

The Central Archaeological Area in Rome and the historical center of Turin are the first test fields of the developed “AI guide”. But CNNs are general models and can be trained to recognize a wide range of objects in different contexts. Therefore, we plan to extend the project to other sites, covering different scales and time spans.

We also plan further developments for the integration of the proposed AI technologies and semantic spatial databases, in order to: (1) make the system more scalable, to store online large amounts of data that can be retrieved when needed; (2) exploit the interoperability of the spatial information, i.e. connecting building information modeling (BIM) data to the environment explored through the app; (3) allow access to a detailed information, i.e. performing DL recognition at the scale of building details, thus recognizing monument parts or categories of constructive elements, decorations, materials.

Furthermore, we are bridging our first experiments with AI and other technologies:

  • AR allows the interaction with 3D digital models, and can superimpose precise spatial information layers on live images of the real environment;
  • 5G cellular mobile communications will make immediately available large amounts of data, redefining location-based services and content access;
  • IoT devices can enable access control and enhance on-site experience, providing cost-effective monitoring solutions which do not need physical presence of supervising personnel.

The work carried out also points out possible connections between the virtual environment and the contemporary city. DL models could be trained to recognize building types or structural components, while the related information could integrate energy performance, structural behaviors, construction phases…

The underlying assumption of the research is that architecture has a key role in approaching technologically advanced tools. Form is a means to identify physical, observable, tangible facts and it can be used to produce shared models of the complex and multi-layered urban space. On this basis, our project intends to be a contribution to the recognition, structuring and operational use of architectural form.

Notes

  • 1

    Source: Eurostat, General government total expenditure on 'recreation, culture and religion', 2015. https://ec.europa.eu/eurostat/web/products-eurostat-news/-/DDN-20170807-1 (accessed on 26 March 2019)