1 of 26

Trial Design Guidebook

Foreword

Trial design toolkit

This guidebook presents guidance, resources and activities to help SMEs develop a detailed and refined trial proposal.

We welcome any feedback on the template and the guidebook to progress our approach and the support we provide.

Status of the trial design Guidebook

Published: January 2024.

Status: BETA for review

Licensing: The toolkit and the beta resources included within it are licensed under a Creative Commons Attribution ShareAlike 4.0 – International License.

Introduction

Overview

A trial is a systematic evaluation of an innovative technology or software to assess its functionality, performance, and suitability for adoption. It involves controlled testing and analysis. It is not an impromptu experiment, guarantee of success, or an indefinite process, but rather a structured assessment within a defined scope and timeframe.

This guidebook offers a template to assist SMEs in successful trial design, and develop a trial proposal that can be assessed by the Catapult and project partners. To do so, the guidebook is formed of three key sections within the trial process:

Each section, and sub-sections within, will start with an overview of the content, provide key questions to include within the trial plan, prompts to support answering said questions, and a link to any useful resources and optional activities.

Who this guidebook is for

The guidebook can also be utilised by project partners to further understand the trial process and trial assessment criteria prior to SME selection.

How to use this guidebook

This guidebook aims to support SMEs in designing and refining the trial plan submitted within the application phase. The guidebook has been broken into three sections.

Each section includes a key output that needs to be produced. To populate this output, a series of questions are listed that SMEs will be expected to answer. Prompts have also been listed to support SMEs in answering the questions, as well as optional activities and useful resources.

As this is a comprehensive guide, It is not required that all outputs are completed and all questions answered. Depending on the programme SMEs are partaking in, a checklist of what to complete will be provided. This checklist will be determined by the Catapult and project partners, and will for a guide on which sections to complete within this guidebook.

If you are having any issues using this guidebook or completing your trial plan, please contact a member of the Catapult team.

Objectives & impact

Introduction

This section is aimed at focusing the trial on objectives based on the selected challenge, as well as forming a key narrative between the challenge and the trials’ success.

This section will guide SMEs through:

Solution and challenge

defining how your solutinoa impacts the chal

Overview

All Connected Places Catapult, trial projects are challenge-driven, emphasising the importance of addressing specific challenges with innovative solutions. During the application process, you would have been required to select a challenge, or series of challenges, that your solution responds to. It is important to consider this challenge when showcasing the benefits of your solutions, as this is how impact will be assessed on the project. To do so, this section covers the chosen challenge, your solution, and the strategic fit.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Challenge: What is the challenge(s) you have chosen to respond to?

Clearly define the challenge, or challenges, your solution aims to address.
Unpack the key user groups who are impacted by the challenge selected.
Make reference to any documentation that you have reviewed when understanding the challenge.

Solution: What is the solution you will be trialling on the project?

Provide a comprehensive description of your solution, including its functionality, intended use, and problem-solving capabilities.
Be clear in the specific use case(s) that you intend to test.
Use a mix of imagery diagrams or video to help the audience quickly understand your solution.

Strategic fit: How does your solution respond to the challenge and broader context?

Explain how your solution directly addresses the chosen challenge.
Unpack how your solution fits within any relevant strategic and contextual factors of the project partners and/or the Catapult.
Include data, statistics, real-life risks or examples that highlight the specific detail and importance of addressing this need.

Working with Network Rail, consider how your solution commercially aligns to their strategic vision of your project partner - such as the performance indicators of Routes and Regions strategy, or to other specific goals the stations have to adhere to.

Useful resources

The following external resources may be useful in defining your solution and challenge:

Objectives and impacts

verview

With a clear understanding of how your solution addresses the selected challenge, it is important to clearly refine the scope of the trial. Your trial plan should include objectives and success factors that clearly define what you aim to achieve and learn during the trial in relation to the challenge and the project partner objectives. To unpack these, this section covers trial objectives, success factors and what you aim to learn, and expected impacts of the solution

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Trial objectives: What are your trial objectives?

Set specific, measurable, and achievable trial objectives.
Align these objectives with the programme challenges.
Determine the key questions to address the programme challenge, considering any background research and context-specific factors.

What you want to learn: What do you want to learn from the trial?

Following on from the trial objectives, list the specific (research) questions that need to be answered within the trial
Be as specific as possible, and consider how you might address the needs of specific users, or a specific aspect of the product or service

Success factors: What does success look like on the trial?

Identify the critical factors to observe for successful integration of your solution.
Establish clear success criteria that demonstrate how the technology addresses the identified challenge.

Expected impacts: What are the expected impacts of your solution?

Specify the impacts resulting from the trial, and how they relate to trial objectives.
Explain how the observed benefits from the trial will bolster the business case for adopting the solution.
Consider wider impacts for project partners - this could include financial savings, time savings, efficiency improvements, environmental and social benefits, safety improvements and any other value add to project partners.

To support you in answering the above question(s), please find a link to the 'objectives and questions' template and the 'success factors and impacts' template below that can be downloaded in both PPT and JPEG and used within your trial plan.

Useful resources

The following external resources may be useful in defining your objectives and impact:

Optional activity

The following activities may be useful in describing the solution and the challenge:

The following activity may support you in developing your trial proposal:

Activity A: Prioritising your trial objectives

There may be many objectives you can identify within your trial. However, you need to prioritise these within the context of the trial and the timeframe given to showcase success. This activity will help you to ensure what you are testing is feasible and useful to the development of your solution. It will help you focus on what’s important and what you as an organisation don’t already know, but also reflecting on what would be useful for the project partners to know.

List the trial objectives that you have identified, use the framework supplied above to:

Choose your own parameters (we have gone with what we currently lack in evidence and what would be important for organisation.)
Discuss with colleagues or partners what should go where in the quadrant
Decide if those in the top right quadrant are the ones that you want to take forward for the trial.

Consideration: More practical considerations should be made around the amount of time available to trial, and also the infrastructure that will be available to you in the programme.

Monitoring and evaluation

Overview

Effective monitoring and evaluation strategies (M&E) are integral to showcasing the success of the trial to the Catapult and project partners. By employing successful M&E methodologies, innovators will be in a better position to assess the outcomes and impact of the trial and showcase that the agreed objectives are met and valuable insights are gained. So support effective M&E, this section focuses on evaluation of trial outcomes, utilising appropriate methodologies for monitoring, and analysing findings to highlight the trial's impact.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Evaluation: how will you evaluate what is being trialled?

List the data or insights from project partners that are needed to evaluate success - ensuring that your goals are realistic and achievable.
Reflect on the success factors and expected impacts, as well as any research questions you aim to address.
Account for the variables, what do you want to change and what will be measured.

Monitoring: how will you measure the results of your trial?

Before observing change, establish a baseline to understand the current state.
Describe the methodologies you will employ for monitoring, and determine whether quantitative or qualitative methodologies are more appropriate for data collection.
List the resources and equipment needed to implement the aforementioned methodologies, and unpack their suitability in alignment with your research questions and objectives.

Results: how will you analyse the findings and showcase the results?

Outline your plan for presenting the trial's successes or failures.
Consider how you will demonstrate the implications of these results for your product or solution

To support you in answering the above question(s), please find a link to the 'summary of trial' diagram below that can be downloaded in both PPT and JPEG and used within your trial plan.

Useful resources

The following external resources may be useful in developing your monitoring and evaluation plan:

Output

Within this section we will provide context on the outputs you are expected to detail within your trial proposal for phase I and II of the competition.

Checklist

The checklist below details what the Catapult and project partners need to see in your trial plan for the initial phase I. SMEs selected to go into phase II will be required to submit additional information prior to deployment.

You are welcome to add checklist items from Phase II to your trial plan in Phase I if needed. However, it is not expected for the review of trial plans in phase II.

Phase I expectations

Describe how the solution addresses the programme challenge.
Describe how the solution aligns with NR’s strategy and the Catapult’s SIZ programme.
Detailed the trial objectives and what you want to learn.
Define the trial's success factors and list the expected impacts.
Detail the methodology that will be employed to monitor and evaluate the solution

Phase II expectations

A full completed M&E plan considering data collection, analysis, and result sharing.
Outline the expected outputs of the trial.

Preparation & planning

Introduction

This section is aimed at providing all the necessary information and resources needed by both the Catapult and project partners to have confidence in the successful deployment of the trial.

This section will guide SMEs through:

Trial environment

Overview

Selecting and understanding the physical or digital environment is critical for the success of your trial. This section addresses key aspects, from the specific location and physical characteristics, infrastructure and equipment requirements, integration with existing systems, and considerations for installation and decommission. Addressing these considerations ensures that the trial accurately reflects the challenges and opportunities of the intended deployment setting.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Location: what zone(s)/area(s) have you selected for the trial?

Specify the solution's location and accessibility, defining who will be able to access it and who shouldn’t access it.
Describe the key physical and geographic characteristics of the environment, and how this might impact the technology
Consider any extraneous variables (such as climate, terrain, infrastructure, and personnel) that are out of the control of the trial and might impact the technology
Evaluate how these physical aspects may influence the performance of your technology.

Infrastructure: what are the physical and digital requirements of the trial environment?

Assess the solutions compatibility with existing infrastructure, systems and technologies.
Plan for any necessary modifications to ensure smooth integration.
Identify any additional infrastructure required for trial deployment – both physical and digital (e.g., power, connectivity, sensors, networks).
Determine ownership of necessary infrastructure and seek permissions for access.
Consider the logistics and cost of deploying and maintaining infrastructure.

Installation and decommission: are there any key factors that will impact installation and decommission of the solution?

Detail what is needed for installation and decommission to happen successfully
Determine the optimal installation time, accounting for stakeholders, users, and similar considerations.
Identify the installation team, their security clearances, associated costs, and any required equipment.
Consider limitations on resources, personnel, or infrastructure.
Develop strategies to work within operational constraints.

if the solution needs to be installed within a live rail station environment, particularly gate-side, consider if it need to be installed overnight when there are no passengers in the station.

Useful resources

The following resources may be useful in defining your trial environment:

Stakeholder engagement & Participant recruitment

Overview

Engaging stakeholders and recruiting participants are pivotal steps for a successful trial. Stakeholders might be those who may not directly interact with your solution but will have influence over if the solution is deployed or commissioned. Participants are those who will be directly interacting with your solution or will be directly impacted by the introduction of your solution. It is important to start recruitment of both actors as soon as you start working on your trial proposal. This section guides you through this recruitment process, supports the identification of key stakeholders and participant user groups, details the recruitment process, and considers ethics approvals required.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Stakeholders: who do you need to include in the trial?

List any relevant stakeholders who need to be involved with your trial
Detail what will be required from each stakeholder group (for example, will they be able to provide access to key infrastructure, or provide access to participants, etc.)

Participants: Who do you need to trial your solution with?

List those who will be using your solution, considering both audiences that are already engaging with the solution and prospective users
Consider capturing a diverse range of users, use a screener to help capture different profiles of participants, whilst also filtering out audiences who might not be relevant.

Strategy: What is your recruitment and engagement strategy?

List the questions do you need to answer and how engaging with participants will help you get to the answers
Detail how many participants will you need to validate your results

Outreach: How will you engage with participants?

Describe how will you be communicating to participants, and how will you encourage them to participate
Ensure that all participants are informed of any associated risks with the trials, are aware of the mitigations and any corresponding action they need to take. Do you need them to sign that they understand and will comply?

Ethical Approval: What approvals need to be in place for you to engage with participants?

Consider if you will you be speaking to vulnerable persons (someone who is unable to look after themselves), you might require ethical approval. If you are unsure if this is the case please ask one of the project team.
Data Protection – do you need to collect any personal data from the participants as part of the trial results? If so, ensure that all the necessary consent is obtained and that GDPR is adhered to.

To support you in answering the above question(s), please find a link to the 'stakeholder map' diagram below that can be downloaded in both PPT and JPEG and used within your trial plan.

Useful resources

The following external resources may be useful in preparing your recruitment strategy:

Resources and practicalities

Overview

Success in implementing a trial is contingent on careful practical considerations of a variety of factors. This section delves into regulatory compliance, health and safety, accessibility and inclusivity, data management, and security and privacy measures to ensure a comprehensive approach to trial execution. A comprehensive understanding of these elements is crucial for effective planning and execution.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Compliance: how will you ensure compliance with relevant regulations, legislation, and standards?

Identify regulatory and legal requirements applicable to your technology and trial.
Outline measures to ensure compliance throughout the trial.
Determine processes required for the solution's deployment.
Check with infrastructure owners or programme partners for specific requirements.

Health and safety: are there any health and safety considerations that are needed prior to deployment?

Identify potential hazards associated with the trial by considering both physical and operational risks.
Ensure that the technology is designed and implemented with user safety in mind.
If necessary, develop and communicate clear emergency procedures in case of accidents or unforeseen events.
Ensure that all equipment used in the trial meets safety standards.

Accessibility and inclusivity: have you considered how your solution will be accessible and inclusive to a wide variety of users?

Ensure that the solution complies with relevant accessibility standards and criteria.
Ensure that content presented through the trial is inclusive and representative of diverse cultures, languages, and backgrounds.

Data management: How will you manage and secure the data generated during the trial?

Establish data collection and storage protocols.
Outline data sharing agreements, compliances, ethics, and regulations.
Develop plans for responsible data management and protection, ensuring GDPR compliance.
Address privacy and security considerations.

Security and Privacy: What security and privacy considerations are relevant to the trial environment, and how will you address them?

Identify potential security threats and vulnerabilities.
Implement measures to protect user privacy and data.
Establish protocols for notifying users of data collection and use.
Assess if collected data includes personal or sensitive information

Working with Network Rail, you may need to complete a Data Protection Impact Assessment (DPIA), - please check if this will be a requirement with the Network Rail team

You will need to complete National Operating Procedure document (NOPS)

Useful resources

The following external resources may be useful in adding rigour to your trial proposal:

Output

Implementation & management

Introduction

This section is aimed at understanding the necessary project management components of the trial prior to deployment in the station.

This section will guide SMEs through:

Project plan

Overview

The project plan is a critical component of a successful technology trial. They guide the installation, deployment, and installation of the solution, ensuring that trial objectives are met to sufficient quality, to budget, and on time. This section focuses on key elements such as activities, roles and responsibilities, resource allocation, and progress check-ins to ensure a comprehensive, adaptable, and agile approach throughout the trial. Partners on the programme will want to know if the timeline will be feasible, what will be happening when, and any review points to check if expectations have been met.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Activates: What key milestones and tasks will you go through to install, deploy, and uninstall your trial to meet your research objectives?

Identify the key stages from inception to conclusion.
Specify dates and timeframes for each milestone to create a comprehensive timeline.
Consider dependencies between stages for effective planning.
Consider showcasing the plan within a Gantt chart or other visual methodology to ease of understanding.

Team: Who from the team will be required to ensure successful running of the trial?

Identify the key members of your project team and how their skills support the trial's processes
Clearly define the roles and responsibilities for the team and any other stakeholders who will be involved
Ensure effective communication channels between team members, stakeholders, and project partners.

Resource allocation: What budget will you require to implement this trial?

Clearly outline required resources, including personnel, technology infrastructure, and project funding.
Specify how the program funding will be utilised, and explore the possibility of match-funding from your organisation or partners.
Identify the budget for development of the solution for trial, and any interactions that may occur during testing
Budget for participants and stakeholders to receive any payment or expenses for their time

Solution iterations: Can you anticipate potential adaptations to create an iteration loop for your trial?

Consider how trial outcomes might inform necessary adjustments.
Identify areas for improvement and potential changes in trial parameters.

Progress: How will you showcase and assess trial progress and reflect on lessons learned?

Outline how you aim to report progress to project partners and showcase your ability to successfully lead the trial
Schedule regular review meetings with project partners to discuss achievements, challenges, and potential scope adjustments during check-ins.

When detailing your project plan, it can be useful to display the information in a visual format like a Gantt chart (or other agile methodology), with detailed work packages for the Catapult and project partners to review.

Risk assessment

Overview

Risk assessment is a crucial component of any trial, helping SMEs identify potential challenges and develop strategies to mitigate them. It involves evaluating factors that could be a barrier to the success of the trial, both from a technical and operational standpoint. This section addresses risk identification, impact assessment, probability evaluation, ownership allocation, and strategic mitigation planning to ensure a resilient approach throughout the trial. Effective risk mitigation ensures that the trial progresses smoothly, and issues are proactively addressed.

Questions and prompts

Within your trial plan, it will be useful to detail the following:

Risk identification: what are the potential risks associated with the trial?

Consider technical, operational, and external factors that may pose challenges. Examples include technology malfunctions, data security breaches, regulatory changes, or community resistance.

Risk impact: what would be the impact of each identified risk on the trial's objectives and success criteria?

Evaluate the potential consequences of each risk, considering factors such as project timeline, budget, stakeholder relations, and the overall effectiveness of the technology.

Probability assessment: how likely is each identified risk to occur?

Assign a probability rating to each risk, considering historical data, expert opinions, and any relevant information available. This helps prioritise high-impact, high-probability risks.

Risk owner: who owns each identified risk, and what are their responsibilities?

Clearly define roles and responsibilities for risk management. Assign specific individuals or teams to take ownership of particular risks and outline the actions they are responsible for.

Mitigation: what strategies will be employed to mitigate or manage each identified risk?

Develop specific mitigation plans for each risk, considering preventive measures, contingency plans, and response strategies. This may involve adjusting project timelines, enhancing security measures, or creating backup solutions.

To support you in answering the above question(s), please find a link to the 'project risk management' diagram below that can be downloaded in both PPT and JPEG and used within your trial plan.

Output

Within this section we will provide context on the outputs you are expected to detail within your trial proposal for phase I and II of the competition.

Checklist

You are welcome to add checklist items from Phase II to your trial plan in Phase I if needed. However, it is not expected for the review of trial plans in phase II.

Phase I expectations

Develop a project plan and Gantt chart (or other visual methodology) which details when activities will take place
Outline the roles and responsibilities with the project plan
Detail the resources required and budget plan
Draft a risk assessment with a clear mitigation plan

Phase II expectations

Showcase a clear understanding of how your team will run the trial and take ownership of the programme
Develop a route to market plan or business as usual plan

Conclusion

Next steps

This guidebook has aimed to assist SMEs in successful trial design, and develop a trial proposal that can be assessed by the Catapult and project partners. Following assessment of SME trial proposals, certain SMEs will be selected to trial. The trial process can be presented as follows:

Output

To support you in completing your trial proposal you will be provided with a checklist and a CPC branded template with some assets. The CPC branded template along with assets can be found here:

You are not required to use the above template if you'd prefer to utilise your own software, format, and company branding.

Scoring criteria

All trial proposals at Connected Places Catapult are scored against the criteria detailed below.

Criteria

Discription

Weight

Trial objectives & Impact

Are the SME trial objectives related to Bristol Temple Meads needs.
Are the success factors reasonable and SME has an effective way of showcasing the success come the end of the trial.
Does the organisation have a detailed plan for monitor and evaluate the trial.
The strength of the potential short and long-term impacts of the intervention on solving the challenge/s it addresses.

30%

Preparation and planning

Did the SME consider the layout and requirements of the station meeting all standards that required to deploy safely in the station.
Have the SMEs provided a well-considered plan to engage with their stakeholders and participants for the trial.
Does the SME have resources in place to deliver the trial, in terms of equipment and expertise.

30%

Implementation and management

Trial can be deployed in reasonable time within the 6-month trial period, providing enough time to collect the required data.
Would the trial be feasible within the timeframe and level of resource available?

20%

Strategic Fit

How is the solution scalable? Can it be installed in other stations?
How does the solution/trial project fit with Network Rail’s wider strategy and ambition?
How much does the trial project fit within the wider Rail sector?

20%

Presentation

You will need to present your trial within 10 mins. Condensing all of the information that you have collected to this point to share it with the gatekeepers who may decide if you deploy your solution or not. When crafting your presentation consider the following:

We have provided a guide on what content you should consider adding in your 10 minute presentation. Please find a copy of this in the PPT below.