The main potential benefits of rainwater harvesting, namely water saving and storm water management, are easily evaluable at a building scale when well known behavioral models are used. However, the evaluation is often more complex at an urban scale, due to a lack of building characteristics and demographic data. In the present paper, we propose a method, which is based on the representative building concept that can be used to quantify the potential benefits of rainwater harvesting at different scales, that is, from the building scale to the district and city scales. Particular attention has been paid to the sizing of the system so that it can be used for different rainwater collection purposes. The method has been applied to the city of Turin (Italy) considering different scenarios: 1) domestic use (e.g., toilet flushing and the washing machine), where buildings are independent of each other, and 2) two public uses (the irrigation of public green areas and street washing), for which we have hypothesized that the rainwater collection takes place at a district scale. The non-potable water saving for domestic use varies across the city from 29% to 62%, according to the characteristics of the buildings, while the reduction of the flow peak conveyed to the sewerage system, during extreme storms, is quite constant (in the 57–67% range). Irrigation and street washing require a lower amount of water, thus about 80% of water can be saved, but the retention efficiency is low, and a slight reduction in the flow peaks can be expected. The aim of the methodology presented in this work is to provide a suitable decision-making tool for policy makers and urban planners to evaluate the capability and efficiency of rainwater harvesting systems for buildings, districts, and cities.