Roles and Responsibilities

The “owner” of a trial is the CM organisation which is mainly responsible for the trial itself. While, on the one hand, trials are collective efforts, there should be one organisation that takes up the responsibility for planning, executing and evaluating the activities. This important role encompasses the following responsibilities:

1. Developing a proper scenario so that the gaps and needs of the main stakeholder are captured in the trial (scenario development); 
2. Hosting the trial itself using one or more locations and ensuring that the chosen location is apt to the purpose of the trial (trial host); 
3. Directing the trial. The director has a prominent role in all phases and, as the name suggests, he or she gives the right directions: for instance, the director initiates the trial during the actual execution and is entitled to stop it any time, in case of problems and/or to put in place mitigation actions;
4. Managing the trial-event in terms in logistics (e.g. rooms and equipment), safety (e.g. make sure that the people involved in the trial are not in danger), media (e.g. dealing with the media before and after the event) and participants (from active to passive actors: players, observers and guests).

The technical coordinator is responsible for a proper technical set-up of the trial scenario, so that an adequate assessment of the selected solutions is ensured. Specifically, the following three responsibilities should be covered by the technical coordinator:

1. The first aspect is the application of the technical test-bed infrastructure. The technical coordinator makes sure that the test-bed technical infrastructure is adjusted according to the decisions taken in the preparation phase and to the lessons learned during the rehearsal and that all components work together smoothly with the trialled solutions. During the trial, the technical coordinator oversees all technical aspects (e.g. integration with legacy tools at the trial location, data exchange etc).
2. This is why the technical coordinator is also in charge of a proper solution providers management. Solution providers are actively involved in the development of the trial, as they know how to best integrate their solutions in the trial scenarios. Therefore, solution providers need to participate in relevant meetings prior to the execution phase so that they can get a comprehensive overview of the activities. The role of the technical coordinator does not end at the end of the trial execution. In fact, the technical coordinator works closely with the evaluation coordinator to provide insights on the overall test-bed application.
3. Another key responsibility is the training management to be provided to the trial participants. The technical coordinator takes decisions with regards to the training needs by deciding how to train the players who actively use the selected solutions during the trial. To do this, solutions providers must be instructed and involved in the overall trial design from the onset. 

the TGM revolves around a practitioner-driven approach, which is by-design reflected in every phase and step. The term “practitioners” stands for all relevant CM stakeholders. Starting the selection of potential solutions with the gap assessment in a specific CM practitioner context up to the final assessment of the potentially innovative solutions, it is the practitioner who has the last word about what should be assessed, in which context, how and what the results mean from the practitioners perspective. In order to ensure the practitioner-driven nature of the TGM, a dedicated practitioner coordinator shall serve as a proper guard.

1. The first responsibility covers the (co-)participation of CM practitioners in the respective phases and steps of the TGM application. Here it is key to identify relevant stakeholders for each trial context. Ideally, the practitioner coordinator should have a CM background. This would facilitate the identification of the right profiles of CM practitioners needed to develop an as much realistic as possible trial scenario. Moreover, it would facilitate the identification of the main metrics for the CM dimension. Additionally, a clear communication of expectations needs to be ensured, so that all practitioners are aware that their participation is also needed after the trial execution to contribute to the sense making and dissemination of the trial results. The practitioner coordinator should be very sensitive to effectively request a minimum commitment of CM practitioner’s involvement while respecting the tight side restrictions practitioners have with regards to their daily duties. At the same time, this role will be regularly confronted with rather high expectations from the other roles in the TC, so that a proper translation and communication of practitioners’ realities becomes vital.
2. The second responsibility targets a well-balanced CM practitioner relationship management. This rather management oriented task goes beyond the content-related (co-)participation of CM practitioners, because it refers to the establishment and maintenance of a pool of practitioners as direct trial participants and (indirectly participating) trial observers. The main functions cover contact management, communication, and reporting tasks.

Similar to the practitioner coordinator, the evaluation coordinator requires a dedicated role because of the importance of executing trials. The overall goal of trials is a robust assessment of potentially innovative solutions. In turn, the actual evaluation calls for neutrality, independence and an adequate degree of decision-making power. Therefore, it is recommended to confide the following responsibilities to someone, who is not in charge for the activities of the other roles.

1. In order to ensure a high evaluation quality, the evaluation coordinator needs to carefully question and verify the overall test-bed application from the very beginning up to the end of a trial. To do so, a close interaction with the practitioner coordinator is important. As a next task, an alignment between the practitioners’s inputs and the trial owner decisions is needed and should be secured by the evaluation coordinator. These results need to be communicated continuously to the technical coordinator, who in turn should feedback the alignment checks on a regular basis. In an ideal setup, this might lead to a highly robust assessment of innovative solutions in realistic setups. However, reality implies several limitations like the partial availability of practitioners, an insufficient length of the trial execution or inadequate depiction of real scenarios in virtual simulations. Therefore, trade-offs need to be done and the evaluation coordinator plays a key role in balancing costs and benefits of different setups.
2. The next responsibility covers the trial evaluation management. Here, the evaluation coordinator is in charge of translating the agreed objectives and side restrictions of the trial dimension into proper metrics and target values. This task requires a strong collaboration with the trial owner.
3. The same applies to the Solution evaluation management. In this area, the evaluation coordinator is tasked to transform the solution specifications, expressed as solution functions or features according to the CM taxonomy, into the solution dimension of the data collection plan. The main collaboration takes place with the technical coordinator, who should align the suggested metrics with the involved solution providers. Their feedback should be properly incorporated, so that the solutions are assessed according to what they are supposed or intended to support. In turn, the evaluation coordinator is in charge of an adequate feedback of the assessment results to the solution providers.
4. Probably, themost challenging responsibility refers to the CM evaluation management. Here, the evaluation coordinator relies on a proper input on how the practitioners perceive the effectiveness of CM operations simulated during the trial. Those definitions are key to elicit the “real” impact of a solution on the CM performance. In consequence, the required CM practitioner profiles need to be communicated in advance to the practitioner coordinator in order to have access to this tremendous important basis of a trial. Another important step during the preparation phase is to communicate the scenario-related metrics to the trial owner, in order to ensure an adequate depiction of the actual work practices in the scenario. Last but not least, the technical coordinator needs to be informed about which data is required from the test-bed, so that the relevant data will be collected and stored in a proper quality, format and amount. Finally, during the evaluation phase the main task is to relate the results in the CM dimension to the results in the trial and solution dimensions. Changes in the CM performance have to be explained through a proper sense-making regarding a potential cause-effect relationship.