METHOD:
3 DIMENSIONS & KPI’S

DRIVER+ trials aim to assess innovative sociotechnical solutions in an as realistic as possible environment in order to bridge a crisis management gap. This leads to the fact that there are three different dimensions that need to be taken into account: The crisis management dimension, The trial dimension and the solution dimension. 

The most important one is the CM dimension, because this is the part were we are looking for new solutions that have an impact on our gaps. Here the baseline (and innovation line) can be most helpful, as they depict the CM process with all its involved roles, tasks, processes etc.

The next dimension is the trial dimension, which relates to the trial organisation. Everything that has to do with the trial run in very “hands-on” manner is part of this dimension. This can be the wifi connection, the number of participants, any technical issue ... 

The last one is the solution dimension. This one tackles all functionalities as well as the usability etc. of each sociotechnical innovation. Each dimension can be analysed alone and also in relation to the others. As the aim is to assess a solution in relation to a CM gap, it is very important to see how this was (maybe even negatively) influenced by the trial or solution dimension. For example: It could be that a solution is very well capable of addressing the CM gap, however during the trial a breakdown of the system can occure due to a technical problem within the trial location (trial dimension). In this case the participants cannot see the whole potential of the solution. This is very important to consider during the analysis and evaluation and to ask how these disruptions influenced the overall setup and data collection.

The main challenge here is to set up your trial in a way that actually enables you to measure each dimension on its own so that you can identify the points where they influence each other. This allows to interpret every piece of data in its rightful context. Within DRIVER+ the ISO 9241-11 was identified as being very helpful with the assessment of the solution dimension. This standard includes artifacts like usability, novelty, etc. So far this kind of data has been collected via dedicated questionnaires filled in by the end user of the solutions within the trial. Here the likert-scale was used and the participants could add their personal opinion as free text. 

The use of questionnaires was also chosen for the trial dimension. Again the likert-scale and open text were applied. The persons to fill in this questionnaire were not only the end users of the solutions, but everyone who participated in the trial (staff, observers, etc.). Furthermore the external cooperation team sent a questionnaire to the external participants and solution providers to gather specific data about the trial organisation (which is part of the trial dimension as well). 

Most demanding is the set-up for the CM dimension. Here a mixed method approach is recommended: Data collection through the test-bed technical infrastructure as well as the solution (data logs) and observer sheets (observer support tool), were used in DRIVER+ so far. Be aware that you should collect data from the legacy tools as well as from the new innovations - as you aim for a comparative study (this is only necessary if you do not already have valid data from past incidents or simulations). Note as well that a human observer can only see and note down a certain amount of information in a certain amount of time. Having them log timestamps is not recommended. They should be selected according to their specific knowledge and then used to observe specific CM relevant artifacts.