Contents
1. Evaluation goal
The evaluation service can
be seen as a multivariable entity in which the viewpoints handled by the
modules are interconnected and depend on each other to varying extent. This
interdependency means that the modules can be disconnected only by making
simplifying assumptions. The best results can be reached by the interactive
handling of the parts of other modules connected to the chosen evaluation
viewpoint and evaluation profile. In practice this means that at least in the
ideal case the evaluation profile contains elements from more than one module
even though the main interest is just the viewpoint of a single module.
The goal of this module is
to evaluate the impact of future technology development on the market prospects
of a service. In short the questions are:
- How the markets will develop
during the study period?
- How will the development of
technology and markets affect the position of the evaluated service on the
market?
- How can the position of the
service on the market be improved?
The primary application
area is services and products connected to transport telematics but the study
can be applied to other lines of business.
The foresight of technology
development and markets as a whole is long-span and extensive activity: it
deals with a great number of technical, commercial and social background issues
that can have a direct or indirect impact on both the development of the
technology and the business possibilities based on it. The evaluation process
described here is based on the constant foresight activity of VTT concretising
and using its results, methods and experience which are constantly accumulated.
This description is the
general model of the process. The precise contents of each
evaluation case is a case-specific application of it: some parts of the
process can be left out and new elements added as needed. The practical
evaluation methods depend on the characteristics and development stage of the
service and the agreed evaluation depth. Evaluations of a service carried out
at the design, implementation and operation stages differ somewhat in character
but the fundamental goal is the promotion of the prospects of the evaluated
service in the market. The best scope can be reached by extending the
evaluation to the entire life-span of the service, either as a single
evaluation of the entire life-span or by repeating the evaluation during the
operation of the service. The evaluation can if desired also be adapted to a
more continuous interactive support process of service development.
Although the evaluation
process is presented here as linearly progressive the stages are in practise
partly parallel and for some part also iterative: e.g. market development has
also a backward impact in the chain to technology demand and from there to
technology development thus practically reflecting also on the evaluation
process.
The method description has
been partly idealised. In practise e.g. the technology fields cannot be
separated from other development as detached patches that do not have
interdependent impacts. The nonlinearity of the combined impacts is allowed for
as well as possible, at least the existence of the
combined impacts has to be recognised. In spite of the combined impacts there
are usually a number of essential separate factors and impact methods on which a sufficient result accuracy can be based.
2. Connection to other modules
Because the successful
achievement of the goals of a service or system can depend on both technical, financial and social development the
conceptualisation of impact methods required by predictions demands the
exploitation of the viewpoints of several evaluation modules at the definition
stage. The modules used at the foresight process and the implemented scale
depend on the case.
The most closely connected
evaluation modules are
- Economic and market viewpoint:
- Revenue and finance
- Profitability
- Service networks
- Risk analyses [=> Risk
management?]
- Technical viewpoint:
- System analyses
- Usability
- Technical feasibility
- Technical risks
- Information security
- Data storage
- Social viewpoint:
- User needs
- Political goals
- Social impacts
3. Definition stage of the evaluation process
The evaluation process
begins with the definition stage outlining the general operation principle and
realisation method of the service, delimiting the elements to be evaluated and
deciding upon the evaluation methods used and the scale of the analyses. This
stage is done in co-operation with the client. At first the essentials of the
service are outlined using the following table.
|
Service
overview and outlining the fundamentals. |
||
|
Task |
Task name |
Task description |
|
1.1 |
General goal of evaluation |
The evaluation maker and client go over the
goals the client has for the evaluation. This stage is common to all modules
because the total evaluation can incorporate element from any module. The result of this stage
is a view common to both the client and the evaluation maker of the total
profile and process of the evaluation and of the associated parts of the
modules. A common understanding about
the work distribution between modules is also formed. The client gets at
least a preliminary cost and timetable estimate if these have not been agreed
upon earlier. |
|
1.2 |
Outlining the function and goals of the
service for evaluation basis |
The evaluation maker gets acquainted with the
service. The purpose is to form a sufficiently accurate and realistic picture
of the service for the definition of the evaluation process. I.e. the
questions “for whom?”, “what?”, “why?” and “how?”.
This means examining and recording the contents, operation principle, target
group and intended impacts of the service. The main factors are
usually:
Information sources:
Connections to other
modules [?]
|
The actual definition of
the evaluation process is carried out according to the table below.
|
Definition stage of the evaluation process. |
||
|
Task |
Task name |
Task description |
|
2.1 |
Definition of the part to be evaluated |
The parts and functions to be evaluated are
agreed upon as are the scope of the evaluations for each part. The evaluation
profile is adjusted so that it on the other hand takes into account the
entire service and special features for the essential parts and on the other
hand produces the information that the customer seeks from the evaluation. Possible evaluation
viewpoints:
Setting limits on the
evaluation requires the study of at least the following:
|
|
2.2 |
Definition of evaluation result goals |
After making the evaluation profile the used
measuring methods and result presentation style are agreed upon. Result goals
(measured subjects) and descriptive instruments include:
The measurement methods
are chosen according to the character of the service, viewpoint and scope:
The scope and reliability
of the chosen instruments is evaluated in comparison with the set goals (including
the impacts of assumptions and approximations on the results). The measuring
method and instruments are particularised as needed. |
|
2.3 |
Data |
The data or at least the data types to be
used in the evaluation are selected. Public data
Chargeable reports, such
as
VTT roadmaps
Other VTT reports Expert interviews
Expert workshops arranged
by VTT for accumulating data Questionnaires either by
VTT or subcontracted
The results of other
evaluation modules |
|
2.4 |
Instruments: technology |
Scalability of service
Adaptability of service
Standards
Availability
Uncertainties, risks
|
|
2.5 |
Instruments: business |
Ability to commercialise
Evolution viewpoint
Protection of own
technology of service
Uncertainties, risks
|
|
2.6 |
Timetable and resources |
At the end of the definition stage a final estimate
about the resources required by the implementation of service evaluation is
given:
|
4. Impacts of technology development on service
The current state of the
technologies essential for the service is studied and the future development is
estimated. The study follows the afore-appointed methods and definitions. The
result is an evaluation of the development of the technologies relevant for the
studied service in the study period.
The evaluation of the
development of the technologies relevant for the studied service in the study
period is carried out according to the table below.
|
Evaluation
of technology development impacts. |
||
|
Task |
Task name |
Task description |
|
3.1 |
Current technology situation Starting point: the
relation of service to current technology situation |
The current situation and development trends
are studied from a general service viewpoint. The technology fields to be
studied are identified and refined into evaluation technology starting points.
|
|
3.2 |
Dynamic technology |
Identifying the factors affecting technology
development and their counter forces:
Factors resisting dynamic
forces
|
|
3.3 |
Future technology |
Evaluations on
A view of the new
situation emerging from the identified technology impacts of dynamic changes
is formed. |
|
3.4 |
Impact measurements |
The agreed instrument is applied to the
evaluated service
These form the framework
of the evaluation result for technology development. |
|
3.5 |
Study of measurement results |
The possible sources of error and their reflection
on the technology instrument results are studied. The sources of error can be
e.g.:
|
5. Impacts of market development on service
The development of markets
and the status of the service on the market are evaluated according to the
method in the table below. The market is impacted by also other factors beside
general economic development.
|
Evaluation
on market development and its impacts. |
||
|
Task |
Task name |
Task description |
|
4.1 |
Relation of service to current market
situation |
Evaluating the place of the service on the
current market. Special study of:
Third party market
research is utilised when needed. |
|
4.2 |
Recognition of dynamic forces |
Identifying the factors impacting
market development and their counter forces:
Factors resisting dynamic forces
|
|
4.3 |
Future status of markets |
The same factors are analysed as in the
current market (top-down):
|
|
4.4 |
Impact measurement |
The agreed instrument is applied to the
service
These form the framework
of the evaluation result for market development. |
|
4.5 |
Study of measurement results |
The possible sources of error and their reflection
on the technology instrument results are studied. The sources of error can be
e.g.:
|
6. Results
The main results of the
evaluation are formed by the results from the above technology and market development
analyses and their conclusions done using the agreed instruments. The
evaluation process is concluded with assembling the results and reporting
according to Table 5 .
|
Table 5.
Results. |
||
|
Task |
Task name |
Task description |
|
5.1 |
"Instrument readouts" |
Assembling the results of the instruments. |
|
5.2 |
Comparison with goals |
The differences with possible goal results
are reviewed. |
|
5.2 |
Reliability evaluation |
Uncertainty factors and possible sources of
error are evaluated. When necessary return to the evaluation process, redefine
the instruments, base data or evaluation profile. |
|
5.3 |
Conclusions |
Making conclusions (e.g. SWOT of service or
similar). Seek opportunities to prepare for a new situation. |
|
5.4 |
Reporting |
The evaluation and its results are reported
to the client. |
7. Follow-up
Possible
monitoring or a new evaluation round at a later lifespan stage. Agreed upon
separately.
8. Examples
Improvement of transport
routes between Nordic countries and Central Eastern Europe Segercrantz,
Wladimir Foresight on Regional Issues: Baltic Sea as
Tekniikan tulevaisuus. Kehityksen ennakointihankkeiden
tuloksia Lievonen, Jorma 1995. VTT,
References
Teknologian ennakointi
ja teknologian
arviointi: http://www.vtt.fi/palvelut/all/all_2/inno_ennakointi.jsp
Towards
security evaluation based on evidencecollection. Savola, Reijo;
Proceedings of the Information Security
Vaikuttavaa tutkimusta. Arviointikäytäntöjä
julkisissa tutkimusorganisaatioissa:
Helia ja
SAMK http://www.vtt.fi/inf/pdf/tiedotteet/2006/T2332.pdf
Kutinlahti, Pirjo; Lähteenmäki-Smith, Kaisa; Konttinen, Jari 2006. VTT,
Methodological Introduction
Heinonen, Sirkka Workshop
on Mobility Foresight, 25th April, 2005,
Väestön ikääntyminen
- haaste ja
mahdollisuus teknologian ennakoinnille ja innovaatioille - VIHMA http://ktm.elinar.fi/ktm_jur/ktmjur.nsf/12b74ae4d1122aadc22565fa003211a6/14BEBA69DE2FF20DC2256DF00035C54F/$file/tura17teo.pdf
Väyrynen, Erja 2003. Kauppa- ja
teollisuusministeriö, Teknologiasasto,
Industrial
innovation, economic performance and technology foresight - comparative
perspectives on Japanese and Finnish developments http://www.vtt.fi/inf/pdf/symposiums/2003/S228.pdf
Saarinen, Jani (ed.) 2003. VTT Technology
Studies,
Teknologian ennakointi-
ja arviointikäytäntöjen
kehittäminen eurooppalaisen
kokemuksen pohjalta http://www.vtt.fi/inf/pdf/tiedotteet/2002/T2174.pdf
Eerola, Annele; Väyrynen, Erja 2002. VTT Teknologian tutkimus,
Technology
foresight in the Nordic countries http://www.risoe.dk/rispubl/SYS/syspdf/ris-r-1362.pdf
Eerola, Annele; Holst Jörgensen, Birte 2002. Risö National Laboratory,
Deepening report of
Appendices
|
Revision
history |
|||
|
Version |
Date |
Author(s) |
Comments |
|
0.1 |
10.10.2005 |
Jaakko Lähteenmäki,
Liisa Rautiainen, Mika Naumanen. |
First draft. |
|
0.2 |
9.5.2005 |
Juha Törönen,
Jaakko Lähteenmäki |
Second draft. Word and MindManager
versions. Contents reorganised. |
|
0.3 |
22.6.2006 |
Juha Törönen |
HTML version based on MindManager
version. |
|
0.31 |
1.8.2006 |
Juha Törönen |
Style definitions in html file, contents
intact. |
|
0.32 |
25.9.2006 |
Juha Törönen |
Revised method description. |
|
0.4 |
24.11.2006 |
Juha Törönen |
Table organised, contents revised, references
added. |
|
0.5 |
30.11.2006 |
Juha Törönen |
Contents revised further. |
|
0.51 |
11.12.2006 |
Juha Törönen |
Process flow chart added. |
|
0.6 |
8.3.2007 |
Juha Törönen |
Table reorganised, text revised. |
|
EvaServe – Markets and foresight |
|
|
Revised: 8.3.2007 |
|