EVALUATION FRAMEWORK PDF Free Download

1 / 38
0 views38 pages

EVALUATION FRAMEWORK PDF Free Download

EVALUATION FRAMEWORK PDF free Download. Think more deeply and widely.

HORIZON Innovation Actions | Project: 101139633
TIPS4PED - HORIZON-MISS-2023-CIT-01
Project deliverable D1.3
EVALUATION FRAMEWORK
Author
Geo T. Sam (RINA-C)
I
Evaluation Framework
DELIVERABLE ADMINISTRATIVE INFORMATION
Deliverable Administration
Grant
Agreement
101139633
TIPS4PED
Deliverable no.
D1.3
Evaluation Framework
Status
Final
M10
Date
30/10/2024
Author(s)
Geo T. Sam
Dissemination
level
SEN
Document
history
Version
Submitted
Reviewed
Comments
V0.1
Geo T. Sam
(RINA-C)
Marco
Pagliarini,
Cesar Diaz
Londono,
Alessandro
Aloi (LINKS)
First draft of the
document
V0.2
Geo T. Sam
(RINA-C)
Miguel Gomez
Meiras
(CIRCE),
Daniele
Salvatore
Schiera
(POLITO)
Second draft of
the document
V1.0
Geo T. Sam
(RINA-C)
Laura Bordo
(RINA-C)
Final document
integrating all
comments
Legal Disclaimer
Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only
and do not necessarily reflect those of the European Union or European Climate, Infrastructure and
Environment Executive Agency (CINEA). Neither the European Union nor CINEA can be held responsible
for them.
II
Evaluation Framework
TABLE OF CONTENTS
Deliverable Administrative Information I
TABLE OF CONTENTS II
Project Executive Summary IV
DELIVERABLE EXECUTIVE SUMMARY 1
List of abbreviations and acronyms 2
1 PURPOSE OF THE DELIVERABLE 3
1.1 Attainment of the objectives and explanation of deviations 3
1.2 Intended Audience 4
1.3 Structure of the deliverable and links with other work packages/deliverables 4
2 METHODOLOGY 7
2.1 Methodology to develop the evaluation framework 7
2.1.1 Step 1: Literature Review 7
2.1.2 Step 2: Development of Draft KPIs 8
2.1.3 Step 3: Iteration with Partners 8
2.1.4 Step 4: Finalization of Evaluation Framework 9
3 Selected indicators for evaluating the performance 10
3.1 Criteria for selecting indicators 10
3.2 Selected Indicators and definition 10
3.2.1 Energy 11
3.2.2 Environmental 13
3.2.3 Mobility 13
3.2.4 Economy 14
3.2.5 Social 15
3.2.6 ICT 16
4 Calculation of the baseline situation: TURIN LH 18
4.1 City Level 18
4.1.1 Energy Indicators Calculation 19
4.1.2 Environmental Indicators Calculation 21
4.1.3 Mobility Indicators Calculation 23
4.1.4 Economy Indicators Calculation 24
4.1.5 Social Indicators Calculation 24
4.1.6 ICT Indicators Calculation 24
4.2 Pilot Level 25
III
Evaluation Framework
4.2.1 Energy Indicators Calculation 25
4.2.2 Environmental Indicators Calculation 26
4.2.3 Mobility Indicators Calculation 27
4.2.4 Economy Indicators Calculation 28
4.2.5 Social Indicators Calculation 29
4.2.6 ICT Indicators Calculation 29
5 Brief overview on gdpr compliance 30
6 Conclusions 32
7 References 33
List of Figures
Figure 1: KPI Dimensions in TIPS4PED 3
Figure 2: T1.5- Connections between tasks and WPs 1 to 4 5
Figure 3: Methodology to develop the evaluation framework 7
Figure 4: Partner's feedback for KPI refining 8
Figure 5: KPI breakdown by category 11
Figure 6: City data collection- Turin 19
List of Tables
Table 1: Contribution to the task from different partners 5
Table 2: Number of KPIs per category 17
Table 3: Turin - City level characteristics 18
Table 4: Turin- Pilot level characteristics 25
IV
Evaluation Framework
PROJECT EXECUTIVE SUMMARY
TIPS4PED will design, develop, and test in a relevant environment the effectiveness and the techno-
economic viability of a Digital Twin based Platform, to support municipalities in the implementation of Positive
Energy Districts. The TIPS4PED Platform will be able to support municipalities in the decision-making
delivering a series of evidence-based results, increasing the environmental sustainability of cities, and
reducing operational costs. The project will adopt a people-centric approach, which aims at engage, train
and empower citizens and potential stakeholders. TIPS4PED will do this by leveraging on consortium
knowledge and prior EU experience and being motivated by the goals set forth by the European Commission.
To design and manage the PED effectively and foster its implementation through city planning actions,
different modules will be made available on the Integrated Assessment Platform, to support
municipalities from the technical perspective, as well as the social, financial, regulatory and administrative
ones. The tools will be developed thanks to a end-users co-design approach, capable of providing constant
feedback to the software development process.
The TIPS4PED solutions will be tested in one Light House city (Turin), where the Integrated Assessment
Platform (IAP) will be connected to a sensoring system for the real time management of the assets, and
replication studies will be conducted in three Follower Cities (Cork, Kozani and Budapest). The testing and
replication phase will result in the creation of 4 cities Digital Twins and decarbonization roadmaps. TIPS4PED
seeks to suggest scalable and easily reproducible technologies and digital optimization as a key solution for
to achieve 2030 EU goals, while taking into consideration various market circumstances, and environmental,
safety, and regulatory aspects.
Social Media links:
@TIPS4PED
@tips4ped
@TIPS4PED
For further information please visit WWW.TIPS4PED.EU
1
Evaluation Framework
DELIVERABLE EXECUTIVE SUMMARY
This document presents a framework for evaluating TIPS4PED KPIs and assessing the
baseline scenario for the Lighthouse (LH) city.
Drawing from insights and methodologies established by prior EU initiatives on PEDs
and smart cities, such as Making City, RESPONSE, and PED EU NET, the framework
enables holistic and standardized evaluation across energy, environmental, social,
economic, mobility & ICT dimensions.
The primary purpose of these KPIs is to facilitate an in-depth understanding of the PED
solutionsimpacts at varying spatial levels (from PED pilot, district, to city scale) and
across temporal stages (from short, medium, to long-term, targeting milestones up to
2030).
The results of this framework will guide improvements in PED solution implementation
and inform data-driven decision-making, promoting sustainable urban development in
PEDs across Europe.
2
Evaluation Framework
LIST OF ABBREVIATIONS AND ACRONYMS
Acronym
Meaning
DH
District Heating
EC
European Commission
EV
Electric Vehicle
FC
Follower City
GDP
Gross Domestic Product
GHG
Green House Gas
IAP
Integrated Assessment Platform
ICT
Information and Communication Technology
KoM
Kick-off Meeting
KPI
Key Performance Indicator
LH
Light House
MAPE
Mean Absolute Percentage Error
NPV
Net Present Value
PBP
Pay Back Period
PED
Positive Energy District
WP
Work Package
3
Evaluation Framework
1 PURPOSE OF THE DELIVERABLE
1.1 Attainment of the objectives and explanation of deviations
The objective of this deliverable is to define the energy, social, environmental, mobility, ICT and economic
KPIs, as well as the variables needed for their calculation. It will also establish the measurement criteria and
the baseline scenario for the LH. This deliverable aims to enable transparency for the research and
monitoring process. With the insight from this document, in essence, anyone should be able to comprehend
the implemented research actions fully.
Key Performance Indicators (KPIs) can act as a universal instrument to evaluate the progress of smart city
strategies, supporting the monitoring of relevant solutions and projects. [1] Six main domains have been
considered in the evaluation process (Figure 1): energy, environmental, economic. mobility, ICT and social.
The evaluation framework outlined in this document will assess the demonstration actions through the lens
of each domain, with the goal of capturing a comprehensive cross-section of the societies involved.
Figure 1: KPI Dimensions in TIPS4PED
The KPIs presented in this deliverable have been designed based on the findings of previous reviews on KPI
frameworks for Positive Energy Districts (PEDs) and smart cities, carried out by EU initiatives and projects
such as Making City, RESPONSE, and PED EU NET. To ensure practical relevance, partners from the
demonstration sites in LH and FCs were actively involved in defining the KPIs, ensuring that the selected
indicators are both feasible and appropriate for monitoring. Given the interrelated nature of social,
environmental, and economic impact categories, the partners collaborated closely to ensure alignment and
cohesion. This collaborative effort led to the development of a comprehensive KPI framework, which is
presented in this document.
Since this deliverable is prepared and written in the initial months of the TIPS4PED project, when the
conditions and solutions for demo sites are not yet fully defined, this KPIs framework will be updated
4
Evaluation Framework
throughout the project including calculations of temporal and spatial KPIs to offer a more comprehensive
evaluation.
The objectives related to this deliverable have been achieved in full and as scheduled.
1.2 Intended Audience
This deliverable is primarily intended for all consortium partners involved in the TIPS4PED data processing,
covering the entire workflow and dataflow from data collection to its application in IAP services, KPI
evaluation and monitoring during the project including all the intermediate steps.
1.3 Structure of the deliverable and links with other Work
Packages/deliverables
Section 2 of this document defines the TIPS4PED methodology. Section 3 defines the criteria selected and
also a list and categories of the KPIs. Section 4 provides the baseline scenario evaluated for the LH in city &
pilot level. Section 5 provides a brief overview of GDPR compliance. In Section 6 are reported the conclusions
of this work and section 7, the references.
This deliverable is not an isolated document, but is connected with other tasks and WPs within the project.
The evaluation framework is built on the data gathered in T1.2 through dedicated templates for LH & FCs.
The KPIs presented in this document set the ground for several activities within the project: monitoring and
assessing the performance of the PED solutions simulated through the DT and/or tested within the PED
pilots. Apart from the quantitative assessment related to KPIs, the baseline definition in this deliverable will
also provide information to: T3.2 TIPS4PED IAP Validation and LH and FCs performance monitoring where
the evaluation framework will be applied.
Figure 2 shows the connections between the tasks and WPs.
5
Evaluation Framework
Figure 2: T1.5- Connections between tasks and WPs 1 to 4
Table 1 summarizes the main contributions from participant partners in the development of this deliverable.
Table 1: Contribution to the task from different partners
Partner
Specific Contribution
RINA-C
Task Leader
This partner defines the set of KPIs needed to evaluate the project and the
pilot impacts
Identifies data sources and the methodology to calculate the KPIs
Collects the baseline KPIs of the LH pilot district
6
Evaluation Framework
IES
This partner supports the definition of the KPIs and the identification of data
sources
LINKS
This partner supports the definition of the KPIs and the calculation of the
Turin district baseline
CIRCE
Reviewer 2 for D1.3 Evaluation Framework
This partner supports the definition of the KPIs and the identification of data
sources
POLITO
Reviewer 1 for D1.3 Evaluation Framework
This partner supports the definition of the KPIs and the calculation of the
Turin district baseline
7
Evaluation Framework
2 METHODOLOGY
2.1 Methodology to develop the evaluation framework
This section outlines the methodology used to develop the TIPS4PED Evaluation Framework and provides
a brief overview of how the framework will be implemented in the later stages of the project. The methodology
followed a systematic approach consisting of four key steps, which were finalized in collaboration with key
partners from the Lighthouse and Follower cities. As a result, it reflects the real-world experiences and needs
of these ecosystems. Outlined in Figure 3 are the steps involved in the adopted methodology.
Figure 3: Methodology to develop the evaluation framework
2.1.1 Step 1: Literature Review
A detailed literature review was carried out to ensure the framework development approach aligns with
existing evaluation frameworks and Key Performance Indicator (KPI) definitions. This included an in-depth
analysis of frameworks like Making City, RESPONSE, and PED EU NET [2]
The review had two main goals: (i) to extract relevant components from existing frameworks for TIPS4PED
and (ii) to develop a robust evaluation framework, along with a diverse set of potential KPIs based on lessons
learned and best practices. In particular, the Making City initiative served as a crucial reference, providing
valuable insights into key performance indicators and evaluation methodologies. Furthermore, the literature
review extended to other important initiatives and projects, such as ATELIER, NEUTRALPATH, and
POCITYF, which have made significant contributions to sustainable urban development.
By integrating insights from the projects referenced above into the comprehensive literature review, the
TIPS4PED framework seeks to harness the strengths and best practices of these initiatives. This approach
8
Evaluation Framework
not only ensures the incorporation of relevant elements but also supports the development of a robust
evaluation framework for sustainable urban development.
2.1.2 Step 2: Development of Draft KPIs
The extensive KPI pool created in step 1 was refined by identifying recurring KPIs across the projects. These
KPIs were then categorized into defined dimensions (also called sectors or indicator categories): energy,
environmental, economy, mobility, ICT and social. Following other EU projects selecting criteria, which is
better explained in section 3.1 (POCITYF, Making City, etc.), a comprehensive list of indicators was selected
that reflect the direct and indirect impacts of the implemented measures. This served as an excellent basis
for initiating discussions with FCs and LH partners for the selection of final KPIs.
2.1.3 Step 3: Iteration with Partners
The final KPIs should also reflect the opinions and needs of the demo sites. To achieve this, the outcomes
from Step 2 were shared with the LH and FC partners of TIPS4PED for their feedback. A continuous iteration
process, involving email exchanges and meetings, was carried out to gather input on feasibility, relevance,
evaluation boundaries, data sources, and methodological approaches for baseline calculations and
definitions. Additionally, demo sites were asked to provide site-specific KPIs to assess their individual
performance. Figure 4 shows a section of the feedback file that was circulated among the partners.
Figure 4: Partner's feedback for KPI refining
This approach allowed the integration of indicators of particular interest to the LHs, while others were
excluded due to monitoring-related risks, such as data unavailability and as a result, the KPIs underwent a
thorough review process that incorporated diverse perspectives and considerations.
9
Evaluation Framework
2.1.4 Step 4: Finalization of Evaluation Framework
Following these steps, a finalized list of TIPS4PED KPIs containing all the necessary information for their
evaluation, including metrics, guidelines, formulas, performance thresholds, and category groupings, was
established. The metrics, data collection guidelines, and thresholds/baselines will be further evaluated in
T3.2 and corresponding deliverables.
Section 3.2 describes the KPIs selected for each domain and the parameters required to calculate them. The
total number of identified KPIs is 29.
10
Evaluation Framework
3 SELECTED INDICATORS FOR EVALUATING THE
PERFORMANCE
3.1 Criteria for selecting indicators
Following the initial screening process, it became necessary to filter the KPIs to identify the most relevant
ones for inclusion in the final TIPS4PED KPI repository. To achieve this, TIPS4PED exploited the selection
criteria outlined by the CIVITAS framework (van Rooyen and Nesterova, 2013) [3], which has also been
adopted by other projects such as Making City, POCITYF, and ATELIER, to develop an appropriate shortlist
of indicators.
1. RELEVANCE; Each indicator must hold significant importance in the evaluation process, meaning it should
be closely aligned with the framework's subthemes. Additionally, indicators should be selected and defined
to ensure that the implementation of the smart city project will clearly reflect changes in their values.
2. AVAILABILITY; Data for the indicators must be readily accessible. Since the data collection process should
be time-efficient and require minimal effort, the indicators should rely on data that either:
are available from the project partners,
or can easily be compiled from public sources,
or can easily be gathered from interviews, maps, or terrain observations.
Indicators that, for example, rely on interviews with users or residents are not suitable, as collecting the large
volumes of data required would be too costly. Similarly, indicators that demand extensive recalculations or
additional data, such as footprint or specific financial indicators, are also unsuitable.
3. MEASURABILITY; Identified indicators should be capable of being measured, preferably as objectively as
possible.
4. RELIABILITY; The indicators should be clearly defined to avoid any ambiguity or multiple interpretations.
5. FAMILIARITY; Indicators should be straightforward and easily understood by users and non-experts. For
many indicators, we have drawn from existing indicator sets that generally meet this criterion."
6. NON-REDUNDANCY; Indicators within a system or framework should avoid measuring the same aspect
of a subtheme.
7. INDEPENDENCE: Minor changes in an indicator's measurements should not affect the preferences
assigned to other indicators in the evaluation process.
3.2 Selected Indicators and definition
The six sectors, or indicator categories, defined for the project are energy, environmental, economic, mobility,
ICT and social. The list of KPIs in each sector is outlined in the following section. The tables in the following
subsections briefly describe how each indicator selected as a metric for the city-level evaluation. These
indicators have been calculated for the LH (baseline calculation) and will be further used in D3.2.
11
Evaluation Framework
The pie-chart below (Figure 5) shows a breakdown of the final list of KPIs adopted.
Figure 5: KPI breakdown by category
3.2.1 Energy
Indicator
Description
Unit
EN_01: Final energy
consumption per capita
Annual final energy consumption (both thermal and
electrical) across all uses and energy forms, including
residential, tertiary sector, public lighting, industry, and
transport, is measured. This consumption is then
divided by the total city population to obtain the per
capita figure
MWh/cap
EN_02: Primary Energy
Consumption per Capita
This indicator reflects the primary energy consumed in
the city which is divided by the total city population to
obtain the per capita figure
MWh/cap
EN_03: Renewable Energy Ratio
Defined as the ratio between total RES capacity
installed to the total capacity to satisfy all final uses and
energy forms, including residential, tertiary sector,
public lighting, industry, and transport.
-
EN_04: Integrated Building
Management Systems (BMS)
Percentage of buildings equipped with systems that
integrate and manage energy use efficiently.
%
EN_05: Wind Energy Potential
Utilization
This indicator corresponds to the ratio between the
wind capacity installed and the average wind power
density
-
EN_06: Energy Savings
This indicator measures the reduction in energy
consumption due to efficiency measures and
conservation efforts. Average utility bill or a
%
12
Evaluation Framework
Indicator
Description
Unit
documented efficiency measure.for families can be
used in comparison with historical data to measure
savings.
EN_07: Solar Energy Potential
Utilization
This indicator corresponds to the ratio between the
solar capacity installed and the average solar power
density
-
EN_08: Error in energy demand
& production forecasting
This KPI will compare forecast accuracy coming from
the energy models used now and after the planned
implementation of the sensors.
 


MAPE: Mean Absolute Percentage Error: This
represents the average percentage error between the
data predicted by the model and the data measured by
the sensors. [%]
U: A statistical factor related to the coverage of
monitored users is an adjustment to a KPI that accounts
for uncertainty caused by sensors not covering all PED
users, ensuring accurate performance measurement
despite partial coverage.
%
EN_09: Feasibility of DH
Explores the feasibility of District Heating in the pilot
area and net extension if present, indicated in kms.
km
EN_10: Energy Storage Capacity
Installed
This KPI reflects the project's impact on the utilization
of smart energy storage systems by indicating the total
capacity of energy storage systems installed. The
baseline is zero, meaning that, as is typically the case,
new buildings do not include any storage.
kW
13
Evaluation Framework
3.2.2 Environmental
Indicator
Description
Unit
EV_01: GHG Emissions per
Capita [4]
CO2 emissions produced over a calendar year by all
activities.
Tonnes of
CO2/cap
EV_02: SECAP (Sustainable
Energy and Climate Action Plan)
& foreseen measures, Climate
city contract (CCC) & foreseen
measures, SUMP (Sustainable
Urban Mobility Plan) & foreseen
measures and their allocated
budget
This indicator assesses to what extent the city has a
resilience strategy and action plan to adapt/respond to
climate change, by providing a qualitative measure.
-
EV_03: Reduction in emissions
The greenhouse gas emissions reduction assesses the
greenhouse gas emissions savings resulting from
interventions in PED/ local renewable generation
%
EV_04: Area of Additional Green
or Public Spaces Created
This KPI measures the total area of new green or public
spaces added within the city each year, expressed in
square meters.
m2/year
3.2.3 Mobility
Indicator
Description
Unit
EM_01: Modal Split
The indicator determines the distribution of transport
over the modalities of public and collective transport,
private vehicles
%
EM_02: Number of Public EV
Charging Stations
This KPI tracks the total number of public electric
vehicle (EV) charging stations available in the city,
indicating infrastructure development to support
sustainable transportation.
Number
EM_03: EV Composition in
private & public sectors
This KPI measures the proportion of electric vehicles
(EVs) in both the private and public sectors
%
14
Evaluation Framework
3.2.4 Economy
Indicator
Description
Unit
EC_01: Employment rate
Employment percentage over the entire working-age
population
%
EC_02: GDP (Gross Domestic
Product) per capita
GDP value in millions of euros
EC_03: Average Utility Bill for
Families
This KPI tracks the average monthly utility bill for
households, covering essential services such as
electricity, water, and gas.
€/year
EC_04: Payback period
This KPI represents the time required to recover the
initial investment in a project or technology through
savings or profits measured in years. The payback
period is often used as an additional criterion for
evaluating investments, particularly for assessing risk.
Investments with shorter payback periods are generally
considered less risky than those with longer payback
periods.
 

 󰇛 󰇜

PP : Payback period.
TI : Total investment for all the interventions related to
energy aspect [€]
TCS : Total cost savings over the economic analysis
period
TACt : Total annual cost of the systems before and after
the interventions (i.e. energy, operation & maintenance,
financial) for year t [€/year]
T : Duration of the economic analysis period [years]
t : Year
Years
EC_05: Net present Value
This KPI calculates the net present value (NPV) of a
project by determining the difference between the
present value of cash inflows and outflows over time. It
indicates the profitability and long-term financial benefit
of an investment.
15
Evaluation Framework
Indicator
Description
Unit
 󰇟󰇛 󰇜 󰇛󰇜
 󰇠
NPV : Net present value [€]
TI : Total investment for all the interventions related to
energy aspect [€]
TACt : Total annual cost of the systems before and after
the interventions (i.e. energy, operation & maintenance,
financial) for year t [€/year]
T : Duration of the economic analysis period [years]
t : Year
r : Required rate of return or discount rate
3.2.5 Social
Indicator
Description
Unit
SA_01: Citizen
engagement/empowerment to
climate conscious actions
This KPI measures the level of citizen engagement and
empowerment in climate-conscious actions, tracking
participation in sustainability initiatives and the adoption
of environmentally friendly practices.
No engagement 1 2 3 4 5 High engagement
Likert scale
SA_02: Consciousness of
residents
This KPI evaluates the increased awareness among
residents regarding key issues such as project
interventions, energy use, environmental impact,
climate change, personal and communal consumption,
and carbon footprint and handprint.
No consciousness - 1 - 2 - 3 - 4 - 5 - High
consciousness.
Likert scale
SA_03: Local community
involvement in the
implementation and planning
phase
This KPI measures the level of local community
involvement during both the planning and
implementation phases of projects.
1 = Not at all: No community involvement. 2 = Inform
and consult: The community is involved for information
only. 3 = Advise: community actors are invited to ask
questions, provide feedback and give advice. 4 =
Partnership: community actors are asked to participate
in the implementation process. The local community is
Likert scale
16
Evaluation Framework
Indicator
Description
Unit
able to influence the implementation process. 5 =
Community self-development: the project planners
have empowered community actors to manage the
implementation and evaluate the results
3.2.6 ICT
Indicator
Description
Unit
ICT_01: Percentage increase in
the number of Smart Meters
This KPI tracks the percentage increase in the number
of smart meters installed in the city.
%
ICT_02: Quality of open data
This KPI assesses the quality of open data provided by
the city was increased.
1 = Very poor: incomplete, inaccurate, difficult to
access. 2 = available, lacks completeness:. 3 =
Average: relatively accurate. 4 = Good: accurate, up-
to-date & easily accessible 5 = Excellent: highly
accurate, comprehensive, frequently updated
Likert scale
ICT_03: Improved Data Privacy
This indicator refers to the level of improvement of data
privacy and safety analysing the extent to which data
collected by the project is protected, for instance, by
following regulations on data protection and
implementing proper procedures to protect personal or
private data.
Not at all 1 2 3 4 5 Excellent
Likert scale
ICT_04: Adoption Rate of
Project-Developed Tools by
Partner Organizations
Measure of successful uptake of project-developed
tools / capability by partner organisations.
Not at all 1 2 3 4 5 Very high
Likert scale
To summarize the breakdown, Table 2 shows the number of KPIs selected across each category.
17
Evaluation Framework
Table 2: Number of KPIs per category
Category
Number of KPIs
Energy
10
Environmental
4
Mobility
3
Econonmy
5
Social
3
ICT
4
18
Evaluation Framework
4 CALCULATION OF THE BASELINE: TURIN LH
4.1 City Level
Turin LH, spanning approximately 260,000 and encompassing the main university campus of the
Politecnico di Torino, aims to lead the development of the first PED (Positive Energy District) in Turin. It also
seeks to demonstrate the potential for expanding renewable energy capacity in densely populated urban
areas and buildings with limited surface space. The campus includes classrooms, departments, and
laboratories. The implementation, monitoring, and operation of PED solutions will focus on two key areas:
Test Area #1, the Energy Center, with 5,000 of office space, and Test Area #2, the Turin Municipality
office area, which includes two existing office buildings, totaling approximately 5,000 m².
Table 3: Turin - City level characteristics
Characteristics
Number of inhabitants
Number
847,398
Average age
Number
47
Area
k
130
Population Density
Inhabitants / k
6,518.44
The tables that follow for each KPI category evaluates the values (when available) of the definitions set in
3.2. Raw data as well as the characteristics reported in Table 3 and Table 4 are taken from the templates
filled by the Turin LH as part of T1.1. In the data collection strategy, a total of 63 indicators were split into 5
categories: Socio-demographic, Energy and economy, Mobility, Territory& Emissions and decarbonisations
plans with a city level & pilot level focus. The templates created were then filled out by the LH & 3FCs. A
section of the template used for data collection in Turin LH is shown in Figure 6.
19
Evaluation Framework
Figure 6: City data collection- Turin
4.1.1 Energy Indicators Calculation
Indicator
Raw Data
Value
Unit
EN_01: Final thermal energy consumption
per capita (District Heating)
1841 GWh in 2022
2.17
MWh/cap
EN_011: Final electrical energy consumption
per capita
2531 GWh in 2023
2.98
MWh/cap
EN_02: Primary Energy Consumption per
Capita
546,717 tep/year
7.5
MWh/cap
EN_03: Renewable Energy Ratio
Biogas: 14,156 kW
Waste: 65,500kW
(Gerbido waste-to-
energy plant)
Hydraulic: 1,810 kW
Non-Renewable:
12,600 kW (natural
gas)
Solar: 32,983.331 kW
0.41
-
20
Evaluation Framework
Indicator
Raw Data
Value
Unit
EN_04: Integrated Building Management
Systems (BMS)
The baseline is zero,
meaning that, there
aren’t any BMS
%
EN_05: Wind Energy Potential Utilization
Wind power density:
34W/m²
Installed capacity:
2KW
Negligible
-
EN_06: Energy Savings
Current energy bill
data is reported in
Economy section,
future data will be
used to compare.
%
EN_07: Solar Energy Potential Utilization
Solar capacity:
32,983.331 kW
Yearly in-plane
irradiation [kWh/m2]:
1452.85
Area: 130 k
0.0015
-
EN_08: Error in energy demand & production
forecasting
Not enough data to
estimate the
prediction error of the
model.
To be calculated at
a later stage
%
EN_09: Feasibility of DH
726
km
EN_10: Energy Storage Capacity Installed
The baseline is zero,
meaning that, as is
typically the case,
new buildings do not
include any storage.
kWh
21
Evaluation Framework
4.1.2 Environmental Indicators Calculation
Indicator
Raw Data
Value
Unit
EV_01: GHG Emissions per
Capita
Municipal sector:
91,085-ton CO2eq
Residential: 878,192-ton
CO2eq
Tertiary: 2,431,812-ton
CO2eq
Private transport;
582,791-ton CO2eq
Public Transport:
62,818-ton CO2eq
Municipal sector: 0.11
Residential sector: 1.04
Tertiary sector: 2.87
Private transport: 0.69
Public transport: 0.07
Tonnes
of
CO2/cap
EV_02: SECAP (Sustainable
Energy and Climate Action
Plan) & foreseen measures
1) Expansion of the district
heating network (TLR 1)
2) Construction of Metro Line 2
- Section 1 (TPL 4), 828 million
3) Energy renovation of
buildings through project
financing (EP1)
4) Purchase of electric buses
and shift of public transport to
rail (FL1, FL2) €170 million
from the PNRR, €68.4 million
from the PSNMS, €50 million
from bank financing.
5) Expansion of the cycling
network (MA1)
-
EV_03: Climate city contract
(CCC) & foreseen measures
1) Energy requalification of the
'Teatro Nuovo' theatre, 2026,
€71.8 million
2) Energy retrofitting of 800
municipal buildings with Iren
(EfficienTO), 2022, €110 million
3) Extending Metro Line 1
westward with new stations and
tunnels, 2026, €271 million
-
22
Evaluation Framework
Indicator
Raw Data
Value
Unit
4) Hydrogeological
refurbishment of hillside parks,
2023, €175 million
5) Replacement of urban &
suburban bus fleet, 2024,
€288.4 million
EV_04: SUMP (Sustainable
Urban Mobility Plan) &
foreseen measures
1) Completion and
Enhancement of the
Metropolitan Network, 2024,
€5.3 billion
2) Enhancement of the
Metropolitan Railway Service
(SFM), 2024, €281.5 million
3) Creation of the Torino-
Cintura Supercycle Network,
2024, €196.9 million
4) New Toll Scheme and Speed
Limit on the Ring Road, 2024
5) Implementation of MaaS
Systems and Restructuring of
the Public Transport Network,
2024
-
Reduction in emissions
To be calculated at a later stage
%
Area of Additional Green or
Public Spaces Created
To be calculated at a later stage
m2/year
23
Evaluation Framework
4.1.3 Mobility Indicators Calculation
Indicator
Raw Data
Value
Unit
EM_01: Modal Split
Private
Transport
73
%
Public
Transport- Bus
or Tram
14.01
%
Walking
0.27
%
Cycling
5.94
%
Rail +
Underground
6.54
%
EM_02: EV Composition
in private & public
sectors
Private
Transport
13.5
%
Public
Transport
8.49
%
Private Freight
Transport
3.84
%
EM_03: Number of Public EV Charging
Stations
Total slow (<11 kW): 44
Total medium (11<x<50 kW):
1665
Total fast (>50 kW): 148
Total Sockets: 1857
1857
Number
24
Evaluation Framework
4.1.4 Economy Indicators Calculation
Indicator
Raw Data
Value
Unit
EC_01: Employment rate
48.3
%
EC_02: GDP (Gross Domestic
Product) per capita
32,213.66
EC_03: Average Utility Bill for
Families
232
€/year
EC_04: Payback period
No interventions or
measures adopted yet.
To be calculated at
a later stage
years
EC_05: Net present Value
No interventions or
measures adopted yet.
To be calculated at
a later stage
4.1.5 Social Indicators Calculation
Indicator
Raw Data
Value
Unit
SA_01: Citizen engagement/empowerment
to climate conscious actions
4
Likert scale
SA_02: Consciousness of residents
4
Likert scale
SA_03: Local community involvement in the
implementation and planning phase
3
Likert scale
4.1.6 ICT Indicators Calculation
Indicator
Raw Data
Value
Unit
ICT_01: Percentage increase in the
number of Smart Meters
Currently Installed
number: 308,621
Planned number: To be
calculated at a later
stage
%
ICT_02: Quality of open data
3
Likert scale
ICT_03: Improved Data Privacy
To be calculated at
a later stage
Likert scale
25
Evaluation Framework
Indicator
Raw Data
Value
Unit
ICT_04: Adoption Rate of Project-
Developed Tools by Partner Organizations
To be calculated at
a later stage
Likert scale
4.2 Pilot Level
Table 4: Turin- Pilot level characteristics
Characteristics
Number of inhabitants
Number
3648
Average age
Number
50
Area
530,000
4.2.1 Energy Indicators Calculation
Indicator
Raw Data
Value
Unit
EN_01: Final thermal energy consumption
per capita (District Heating)
17.99 GWh in 2023
4.93
MWh/cap
EN_011: Final electrical energy
consumption per capita
20.15 GWh in 2023
5.52
MWh/cap
EN_02: Primary Energy Consumption per
Capita
2353.58 tep/year
City level data known,
pilot site data
calculated based on
population
7.5
MWh/cap
EN_03: Renewable Energy Ratio
In PED area aren't
present energy power
plants. The only
source for electricity
production is a PV
system with a total
capacity of 705 KW
-
EN_04: Integrated Building Management
Systems (BMS)
The baseline is zero,
meaning that, there
aren’t any BMS
%
26
Evaluation Framework
Indicator
Raw Data
Value
Unit
EN_05: Wind Energy Potential Utilization
Wind power density:
34W/
Negligible
-
EN_06: Energy Savings
Current energy bill data
is reported in Economy
section, future data will
be used to compare.
%
EN_07: Solar Energy Potential Utilization
Solar capacity: 705
KW
Yearly in-plane
irradiation [kWh/m2]:
3853
Area: 0.53 K
0.003
-
EN_08: Error in energy demand &
production forecasting
Not enough data to
estimate the prediction
error of the model.
To be calculated at
a later stage
%
EN_09: Feasibility of DH
17.5
km
EN_10: Energy Storage Capacity Installed
The baseline is zero,
meaning that, as is
typically the case, new
buildings do not include
any storage.
Kwh
4.2.2 Environmental Indicators Calculation
Indicator
Raw Data
Value
Unit
GHG Emissions per Capita [4]
Municipal sector:
401.28-ton CO2eq
Residential sector:
3,801.12-ton CO2eq
Tertiary sector:
10,487.76-ton CO2eq
Municipal sector: 0.11
Residential sector: 1.04
Tertiary sector: 2.87
Private transport: 0.69
Public transport: 0.07
Tonnes
of
CO2/cap
27
Evaluation Framework
Private transport:
2,521.92-ton CO2eq
Public transport: 255.36-
ton CO2eq
City level data known,
pilot site data calculated
based on population
SECAP (Sustainable Energy
and Climate Action Plan) &
foreseen measures
Available on a city level
-
Climate city contract (CCC) &
foreseen measures
Available on a city level
-
SUMP (Sustainable Urban
Mobility Plan) & foreseen
measures
Available on a city level
-
Reduction in emissions
To be calculated at a later stage
%
Area of Additional Green or
Public Spaces Created
Baseline = 84273 m2 where
27809 m2 are private
m2/year
4.2.3 Mobility Indicators Calculation
Indicator
Raw Data
Value
Unit
EM_01: Modal Split
Private
Transport
72.45
%
Public
Transport- Bus
or Tram
11.54
%
Walking
0.74
%
Cycling
6.42
%
Rail +
Underground
8.84
%
Private
Transport
13.5
%
28
Evaluation Framework
EM_02: EV Composition
in private & public
sectors
Public
Transport
8.49
%
Private Freight
Transport
3.84
%
EM_03: Number of Public EV Charging
Stations
Total slow (<11 kW): 0
Total medium (11<x<50 kW): 5
Total fast (>50 kW): 2
Total Sockets: 7
7
Number
4.2.4 Economy Indicators Calculation
Indicator
Raw Data
Value
Unit
EC_01: Employment rate
Available in city
level
%
EC_02: GDP (Gross Domestic
Product) per capita
32,213.66
EC_03: Average Utility Bill for
Families
232
€/year
EC_04: Payback period
No interventions or
measures adopted yet.
To be calculated at
a later stage
years
EC_05: Net present Value
No interventions or
measures adopted yet.
To be calculated at
a later stage
29
Evaluation Framework
4.2.5 Social Indicators Calculation
Indicator
Raw Data
Value
Unit
SA_01: Citizen engagement/empowerment
to climate conscious actions
Available in city
level
Likert scale
SA_02: Consciousness of residents
Available in city
level
Likert scale
SA_03: Local community involvement in the
implementation and planning phase
Available in city
level
Likert scale
4.2.6 ICT Indicators Calculation
Indicator
Raw Data
Value
Unit
ICT_01: Percentage increase in the
number of Smart Meters
Currently Installed
number: 1220
Planned number: To be
calculated at a later
stage
%
ICT_02: Quality of open data
3
Likert scale
ICT_03: Improved Data Privacy
To be calculated at
a later stage
Likert scale
ICT_04: Adoption Rate of Project-
Developed Tools by Partner Organizations
To be calculated at
a later stage
Likert scale
30
Evaluation Framework
5 BRIEF OVERVIEW ON GDPR COMPLIANCE
This section outlines the essential GDPR components to be considered for the TIPS4PED project. Reading
this section is highly recommended for all partners, as it highlights key concepts that can help avoid potential
data issues during project development.
This section is intended to ensure that all stakeholders understand the importance of GDPR compliance in
the context of the KPI evaluation and definition. Understanding the implications of GDPR is crucial for defining
and measuring the KPIs without inadvertently violating data protection laws. Ultimately, this GDPR section
clarifies the responsible handling of personal data within the KPIs, demonstrating TIPS4PED commitment to
transparency and accountability while aligning our objectives with legal requirements.
Below are the essential elements of the regulation.
a) The legal framework governing data protection in every EU member state: REGULATION (EU)
2016/679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL is the General Data
Protection Regulation (GDPR).
b) In general, take care not to disclose any personal data, which is defined as any information that can
be used to identify or uniquely identify a natural person, such as their name, identification number,
location, online identifier, or any combination of characteristics unique to their physical, physiological,
genetic, mental, cultural, or social identity, including their IP address.
c) In this sense, under GDPR, TIPS4PED is obliged to ensure that personal data shall be:
Processed lawfully, fairly and in a transparent manner in relation to the data subject
Collected for specified, explicit and legitimate purposes and not further processed in a
manner that is incompatible with those purposes
Adequate, relevant and limited to what is necessary to the purposes
Accurate and, where necessary, kept up to date
Kept for no longer than is necessary
Processed in a manner that ensures appropriate security
d) Regarding personal data, TIPS4PED digital twinning works will be mainly focused on building data
and weather data, which allows a sufficient degree of anonymisation and aggregation of personal
data (if any) to avoid a potential identification of a natural person. This must be ensured for any data
collection and data processing task within TIPS4PED.
e) In case there is a need for processing any personal data within TIPS4PED, the consortium must
ensure:
The consent has been provided
31
Evaluation Framework
To comply with legal obligation
To protect the vital interest of the data subject
The processing is necessary for the legitimate interest of the project, except if that
interest conflicts with fundamental rights and freedoms of the data subject which require
protection of personal data
f) To provide some context, a few issues about consent:
Consent under GDPR must be a freely given, specific, informed, and unambiguous
indication of the individual’s preferences
Pre-filled boxes or inactivity does not indicate consent
The data subject shall have the right to withdraw their consent at any time via the original
method
Consent must not be tied to provision of a contract of services or employment
g) Be aware of the data subjects’ rights, namely:
The right to be informed (to know what is being kept)
The right of access (to check what is being held)
The right to rectification (to amend any errors)
The right to erasure (to remove where no compelling reason exists to continue)
The right to restrict processing (where the subjects believe their rights have been
infringed)
The right to data portability (to move their data elsewhere)
The right to object (where the subjects believes their rights have been infringed)
Rights in relation to automated decision making and profiling (where human intervention
is minimal)
h) Regarding any potential breach of security, caused by accidental or deliberate causes, leading to
unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data:
Establish likelihood and severity of the resulting risks to people’s rights and freedoms
within 72 hours of becoming aware of the breach.
If the breach is likely to result in a high risk of adversely affecting individuals’ rights and
freedoms, you must also inform those individuals without undue delay.
i) In case of non-compliance of GDPR, TIPS4PED partners must be aware of the potential fines stated
in the law.
j) For further details on each of the points mentioned in this section, please check the Law at https://eur-
lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
32
Evaluation Framework
6 CONCLUSIONS
As cities strive to become smart cities, establishing reliable metrics is crucial for identifying strengths and
weaknesses, allowing cities to prioritize actions effectively. To support this goal, TIPS4PED partners have
collaborated to create a set of city-level indicators & a comprehensive evaluation framework designed for
city diagnosis and needs assessment. The definition of TIPS4PED Key Performance Indicators (KPIs) has
been developed through a thorough methodological process, closely aligned with the needs of LH & FCs
and their citizens as they undergo their energy transition. Task 1.5 focuses on developing an evaluation
framework to measure and assess project activities at both the project (PED) level (demonstration areas)
and the city level.
This deliverable outlines the process of developing the evaluation framework, including the selection and
alignment of city-level indicators with WP1 developments. The main set of indicators and targets included in
city plans will be extracted and integrated into Deliverable 3.2.
A total of 29 indicators were selected from six primary categories:
1. Energy
2. Environment
3. Mobility
4. Economy
5. Social
6. ICT
The selected city-level indicators were chosen based on their relevance to the project's context, ease of
measurement and monitoring, and their focus on medium- or long-term impacts. These indicators are
intended to be continuous monitoring tools, useful not only during the project but also beyond its duration.
This deliverable provides methodological guidance for cities on measuring and calculating the indicators,
outlining key considerations and exclusions. Since this deliverable is prepared and written in the initial months
of the TIPS4PED project, when the conditions and solutions for demo sites are not yet fully defined, this KPIs
framework will be updated throughout the project to offer a more comprehensive evaluation of potential
interests. Deliverable 3.2 will further analyze city plans in detail. Initial (baseline) values for the indicators for
Turin LH have been calculated for this project report, with impact assessments and evaluations of city-level
actions and interventions scheduled for later stages of the project.
33
Evaluation Framework
7 REFERENCES
1. Angelakoglou, K., Nikolopoulos, N., Giourka, P., Svensson, I.-L., Tsarchopoulos, P., Tryferidis, A.
and Tzovaras, D. (2019). A Methodological Framework for the Selection of Key Performance
Indicators to Assess Smart City Solutions. Smart Cities, 2(2), pp.269306.
doi:https://doi.org/10.3390/smartcities2020018.
2. Making City: http://www.makingcity.eu/
RESPONSE: https://h2020response.eu/
PED EU NET: https://pedeu.net/
3. Applied framework for evaluation in CIVITAS PLUS II. (n.d.). Available at:
https://civitas.eu/sites/default/files/civitas_wiki_d4_10_evaluation_framework.pdf [Accessed 18
Sep. 2024].
4. Green house gas protocol: https://ghgprotocol.org/