Stage 3

 

Evaluation

The evaluation of a new device should involve a number of cycles of: evaluation of a prototype device, re-design of the prototype based on the results and then further evaluation of the re-designed prototype. This should be repeated until the device is deemed to be of an appropriate standard (figure 1).

Evaluation of a prototype device will not only involve user issues but will also include critical factors such as functionality and clinical efficacy.  When planning evaluation of a new device it may be tempting to delay user testing of a prototype until a high-quality, fully functional prototype has been developed. However leaving user evaluation till this stage can lead to problems as, once a prototype has reached the stage where it has been shown to be clinical effective and functional, it may not be easy to change the device to comply with the wishes and needs of the users. For example, if a device has already been CE marked or met other regulations, then changing seemingly ‘small’ issues such as the shape of a device or moving buttons on the user interface, will require re-certification. Such changes may become too expensive or time-consuming to implement. Therefore, user evaluation should be conducted alongside the functional evaluation of a product so that changes to the device based on user feedback can be made at the same time as functional changes.

Planning Medical Device Evaluation

Before beginning evaluation it is important to decide on what the requirements for the device are, in other words what will constitute a successful device. The results of the work carried out at stages 1 and 2 should be to inform such decisions. Once these have been decided then the criteria for success can be defined. In other words, by the end of evaluation what do we want to be able to say about the device? Examples of criteria for success are:

  • Using the device will allow procedure x to be performed an average of 15 minutes quicker than the current practice
  • Using the new device will improve the safety of patients because of fewer errors
  • The new device effectively supports the working practice of the users by…
  • The new device enables patients to live more independently by providing them with an effective method of treating themselves.

As well as user issues, the criteria for success will also include functional, organisational, safety and economic aspects. When deciding upon these it is useful to consider who will be making the decisions on whether to recommend, purchase or use the device. In other words, who will have to be convinced and what evidence they will want to see? In many cases, a substantial improvement (either in terms of cost, safety or satisfaction) will be required to convince individuals/hospitals to purchase the device over an existing device.

Once these criteria have been agreed then the appropriate measures of these factors can be identified. For example, this may be time (which can then be translated into a cost – e.g. if a patient can leave hospital 2 days early, or a surgical procedure takes 30 minutes less: this can be translated into a cost.). Evaluation should then be planned which will enable the device to reach this standard. Broadly speaking, early evaluation should aim to identify aspects of the prototype that require improvement, with later evaluation concerned with demonstrating the performance of the device.

Evaluate in isolation or comparison?

Evaluating a prototype device in a comparative way can be extremely useful, particularly in the later stages of evaluation when seeking to demonstrate the performance of the device. A comparison could be with an existing device, perhaps a device that is currently deficient that the new device is seeking to replace. A comparison could also be made between a particular task or clinical procedure performed with the aid of the new device and without. Again, it should be remembered that merely demonstrating that a device is an improvement on the existing situation is not a measure of a quality device or that it meets the requirements of its users. Healthcare purchasers will usually require evidence that a new device will result in financial savings or significant benefits to patients before they will agree to purchase the device.

Early evaluation

The aims of early evaluation of a new medical device will usually concentrate on demonstrating the functionality and clinical efficacy of a device and will likely involve a number of cycles of evaluation, each followed by a re-design of the prototype and then further evaluation. In terms of user requirements, early evaluation should aim to identify aspects of the prototype that require improvement.

These might include issues such as:

  • Is the device the correct size and/or shape to fit into the environment of use?
  • Is it compatible with the other devices in the environment of use?
  • Will the users be able to easily learn how to use the device (considering their training, experience and educational background)?
  • Is the device safe for both patients and staff?
  • Will users with physical or mental disabilities be able to use the device effectively?
  • Is the device socially acceptable to use? (this may affect compliance and uptake of devices to be used by patients in the community)

In many cases the early evaluation of a device will be performed in-house with functionality being tested by the developers themselves or by a small number of users that have links with the company. Whilst it is always ideal to involve as representative a sample of potential users as possible, in the early stages of evaluation it may be necessary to use a smaller, accessible number of real users, or even to use individuals from within the company to conduct some user trials using approaches such as scenarios or personas. These types of evaluation have the benefit of being quick to organise and carry out and also relatively cheap. Resources can subsequently be directed towards a rigorous evaluation of a higher quality prototype with a wider group of ‘real’ users in due course.

Again, the first step is to identify what the aims and objectives are of the user research at this stage. The aims of the evaluation will determine how much user involvement is necessary and what type of users should be involved.

Later Evaluation

Late evaluation can be seen as the testing of a good quality prototype, one that has had initial functionality and usability problems resolved and is of a sufficient standard to be tested by a sample of potential end-users. It is often difficult to decide at what point in device development user evaluation should be performed. Conducting it too early, before a detailed prototype has been produced, can mean that not all user issues can be evaluated. However, conducting it too late can mean that is difficult to change the device in response to the user evaluation meaning that there was no point conducting the user evaluation at all. Therefore, an appropriate time for detailed user evaluation of a device is often when a high quality prototype is available but there is still scope for changes to be made. This could be during clinical trials, when user opinions can be collected at the same time as demonstrating clinical efficacy and functionality. However, you should be aware of which aspects of the device can be changed after a clinical trial has been performed and what cannot.

Appropriate Methods:

Methods that are particularly useful for the evaluation of prototype devices include:

  • Focus groups
  • Heuristics
  • Cognitive Walkthrough
  • Usability Tests
  • Surveys
  • Interviews