Tool: OBSERVER
SUPPORT TOOL

The observer support tool records all observations from the observers digitally, so they can be analysed during and after the trial. To collect feedback, the OST also provides the ability for participants and trial staff to fill in questionnaires, directly after (a part/episode of) the trial is executed.

The OST consists of a web application for the observers that is typically run from a tablet. The same application can also be accessed in a browser on a desktop computer, a laptop or a mobile device, for instance for participants to fill in the questionnaires and for the evaluation coordinator to prepare the trial specific observation templates (i.e. checklists) and questionnaires. Furthermore, a server is running to manage all checklists and questionnaires and record all answers. This server is connected to the trial-management-tool, such that the correct checklists/questionnaires are available at the applicable moments during execution of the trial. All collected observation and questionnaire data is thereafter shared with the after-action-review tool, such that it is centrally stored for evaluation.

The functionalities of the observer support tool within each phase are:

Preparation phase:

• Definition of trial episodes (i.e. parts of trial in which different phenomena are expected).
• Definition of roles in the trial (e.g. observer in room A, participant type B).
• Definition of the observation templates (i.e. checklists and questionnaires) which are composed of one or more questions.
• Assignment of observation templates to roles and to trial stages.

Execution phase:

• Definition of the data collection Session during testing, dry runs or the trial, by creating user accounts and inviting the users.
• Assignment of users to roles.
• Supervision of the data collection process.
• Changing the trial episode, manually or via the trial-management tool.
• Sending currently applicable observation templates and messages to roles (i.e. users).
• Showing how many answers to observation templates are inputted by users and showing these answers.

Evaluation phase:

• Exporting the answers inputted in observation templates to CSV format.
• Sharing these answers with the after-action-review tool.
• Reviewing these answers.

In order to configure the OST, the evaluation coordinator (and colleagues) have to provide the following inputs:

• List of trial episodes.
• List of roles in the trial which will be using the OST (e.g. observer A, B, C and participant 1,2,3).
• Set of observation templates (i.e. observer checklists and participant questionnaires).
• Information in which trial episode particular observation templates should be displayed.
• Assignment of observation templates to roles.
• User accounts (e.g. user John Doe = role observer A).
• Short description of trial.