Augmented Reality Heritage

The city is now producing a brand-new quality and kind of information on itself, in the form of data that can be stored, organised and analysed. Current technical needs are format standardisation, information gathering, management and selection, data processing and visualisation. This is leading to unprecedented developments of tools. New networks of relationships between documents can be defined and rapidly redefined, according to continuously updating needs and contents. In this complex information topology, the physical form of architecture maintains a key role, on which even the most up-to-date processing can be founded.

Italy has 49 cultural sites on the UNESCO World Heritage Site list. Despite this, only 0.7% of Italian GDP is allocated to culture¹. Sites such as the Imperial Fora in Rome or Pompeii and Herculaneum host millions of visitors each year, but cannot provide appropriate information services on site and face management and maintenance problems. At the same time, due to low tourist flows, many small, isolated or less well-known historical and archaeological sites cannot afford surveillance and maintenance and are thus not accessible. The available resources need to be optimised in order to ensure the protection of cultural heritage and to enhance its value.

The research project entitled Arch•i – Architectural Intelligence concerns the development of digital technologies to enhance the accessibility and management of cultural sites and the exploration of the related digital information. It is thought that mobile computing technologies can overcome the limitations of traditional information tools and allow novel interactions with monuments and works of art. The project is currently employing artificial intelligence (AI) to make data networks and digital environments accessible from the physical space.

In recent years, the diffusion of large image datasets and unprecedented computational power has boosted the development of a class of AI algorithms referred to as deep learning (DL). Among DL methods, convolutional neural networks (CNNs) have proven particularly effective in computer vision, finding applications in many disciplines. While AI is just beginning to interact with the built environment through mobile devices, heritage technologies have long been producing and exploring digital models and spatial archives. Hence, the digitalisation of cultural information offers structured and ready-to-use sources of knowledge that can be retrieved through the flexible features of AI. The interaction between DL and state-of-the-art information modelling is an opportunity to both exploit heritage databases and optimise new object recognition techniques. A specific approach to automated architecture recognition could change the way in which data on the urban environment is collected, processed and analysed, and could provide more effective ways to access data.

The Arch•i project has developed a mobile app that allows users to access data about buildings and works of art simply by pointing the camera at the object. The app is capable of connecting the real city to relevant, context-aware documents such as images, texts, maps and 3D models. It builds on original developments of CNN techniques aimed at recognising architectural features. The app is based on two main blocks of software: (1) an online, geographic-enabled database that makes it possible to upload different types of document and the related information or metadata; (2) the DL part, which is stored on the device and requires a very small amount of disk space.

The app offers an access point to a yet under-exploited network of digital information, not through a catalogue or a predefined route on a map, but just by framing the urban context through a mobile camera. A vast amount of multimedia information can be linked to the elements of a city, answering questions on how to make the newly available information easily and sustainably reachable.

The Central Archaeological Area in Rome and the historical centre of Turin are the first test fields of the ‘AI guide’ developed. CNNs are, however, general models and can be trained to recognise a wide range of objects in different contexts. It is therefore planned to extend the project to other sites, covering different scales and time spans.

Further developments are also planned for the integration of the proposed AI technologies and semantic spatial databases, in order to: (1) make the system more scalable, to store online large amounts of data that can be retrieved when needed; (2) exploit the interoperability of the spatial information, i.e. connecting building information modelling (BIM) data to the environment explored through the app; (3) allow access to detailed information, i.e. performing DL recognition on the scale of building details, thus recognising monument parts or categories of constructive elements, decorations and materials.

Furthermore, we are bridging our first experiments with AI and other technologies:

• AR allows the interaction with 3D digital models and can superimpose precise spatial information layers on live images of the real environment.
• 5G cellular mobile communications will make large amounts of data immediately available, redefining location-based services and content access.
• IoT devices can enable access control and enhance on-site experience, providing cost-effective monitoring solutions which do not need the physical presence of supervising personnel.

The work carried out also points out possible connections between the virtual environment and the contemporary city. DL models could be trained to recognise building types or structural components, while the related information could integrate energy performance, structural behaviours and construction phases.

The underlying assumption of the research is that architecture has a key role in approaching technologically advanced tools. Form is a means to identify physical, observable and tangible facts and it can be used to produce shared models of the complex and multi-layered urban space. On this basis, our project intends to contribute to recognition, structuring and operational use of architectural form.