Using a Delphi Survey to Assess the Value of Pharmaceutical Process Validation Part 1: Survey Methodology 3

Methodological results Response rate and expert demography. Of the total 73 used e-mail addresses, 36 experts' responses to Q1 were received, 28 of whom continued to Q2. Thus, the response rates were 49% and 38% respectively, for Q1 and Q2.

Table I: Visits (hits) on the information pages during the survey.

Some of the experts' demography can be seen in Figures 1–5, which show a comprehensive variation in their background. There were participants from Finland, Denmark, France, Germany, the UK, Norway, Sweden, Belgium, Iceland and Switzerland; however, the number of participants from each country was not equal. Activity reports Throughout the survey, the level of activity from the participants varied. During Q1, 19% wrote extra comments in their answers, but in Q2, 46% wanted to define their opinions and, therefore, expanded on their answers.
Some participants experienced technical problems and requested an extension of the deadline. At the beginning of the study, it became apparent that entering the Extranet pages was not possible for all participants because of problems with firewalls or browsers. For these individuals, an HTML alternative was provided. Eight respondents used this alternative in Q1 and three in Q2. At the start of the survey, one participant initiated an Extranet discussion, however, no one replied. Thirty nine per cent had visited the discussion page, but only 8% had read all the comments on that page. No one took part in the two organized online forums.
From the system report, it could be seen that 69% of the participants had visited the Extranet pages not only to fill in the questionnaires, but also to check the other information available on the pages (Figure 6). The average time to fill in the questionnaire was approximately 5 min for background information, 21 min for Q1 and 41 min for Q2.
Respondent feedback At the end of the survey, respondents were offered the chance to provide feedback on the survey through an anonymous evaluation form. Only four participants returned the form, all of whom found the subject of the survey interesting, and three out of the four found the methodology suitable for the survey. No one found the Internet technology difficult to use. All found the instructions clear, and that the survey matched their expectations. Two had participated because of interest in the subject, and three out of four gave lack of time as a reason for not participating in discussion and the online forum.
Discussion Altogether, the methodology worked fairly well for this type of opinion survey. The number of respondents and their written comments indicated that most welcomed the opportunity to express their views. The Delphi method fulfilled the expectations and was the appropriate tool for contacting experts anonymously.
The Extranet homepages of the WebCT functioned satisfactorily for the survey. All the necessary information could be offered in an illustrative format, and the completion and sending of the questionnaires was simple. However, as WebCT is mainly provided for the education market, some unnecessary instructions and numberings could not be deleted or changed, but according to the respondents' feedback, these minor issues did not cause the participants difficulties. The major challenges of the Extranet were the firewall and browser problems, which should, of course, have been eliminated beforehand for all participants. Because these problems arose unexpectedly, the only solution was to offer HTML, which meant missing all of the other information given on the Internet pages. Apparently, many lost interest because of this and the majority of the respondents who used the hyperlink in Q1 discontinued the survey in Q2. The opportunity for discussion and online debate was not utilized even though the availability of these functions was highly underlined.
The biggest problem to overcome was the participants' lack of time. This reason for not participating was given mainly by the representatives from the pharmaceutical industry. The authorities were mostly willing to participate, but, only one or two participants were gathered from each country. The pharmaceutical schools found the subject interesting, although some doubted their suitability.
As previously mentioned, using e-mail to contact potential participants was insufficient. Of course, some of those who did not participate may also have considered themselves not to be experts in the field and, therefore, excluded themselves. Other probable reasons for not participating after receiving the e-mail request may be because not all people are fully familiar with electronic communication, and the explosion of the quantity of information through the Internet and e-mail has caused a need for filtering information. The latter may be one reason for the reported lower response rates for e-mail Delphi surveys compared with the postal versions. However, in this survey, more participants were willing to continue to Q2 than in many comparable surveys in which the response rates normally fall dramatically in the second and subsequent rounds.
A group size of approximately 30 proved satisfactory to gather overall information regarding pharmaceutical process validation opinions. The group cannot be regarded as very homogeneous because it consisted of experts from 10 different European countries from the three different parties. Thus, the group was representative of the expert population despite the limited total number of participants. As can be seen from Figure 2, all the participants, with one exception, reported that they practiced, taught or controlled process validation in their work, and they can, therefore, be regarded as experts. It is important to note that above a certain threshold, the inclusion of more respondents only contributes to marginal statistical and qualitative improvements. 
Conclusion The Delphi method was found to be a suitable tool for measuring opinion in the pharmaceutical field. It is particularly useful in the pharma-ceutical manufacturing sector where the discussion between the regulated industry and the regulators is often difficult to achieve on a "neutral" basis in face-to-face-meetings. Because of this gap between the two parties, many regulations are accepted by the industry without official criticism and real assessment, and as a consequence, a lot of unnecessary work is performed. The Delphi method offers a perfect tool for this type of situation - it can be organized anonymously and can bring together geographically dispersed experts.
The use of the Internet and electronic communication gives the method clear advantages - the survey can be organized much faster, the group size can be easily increased and a lot of supporting information can be provided. However, in a climate where the quantity of electronic information is ever increasing, there is a high chance that some may be in part ignored; this is a threat to the use of electronic communication. For this reason, the Internet Delphi demands high motivation of the participants; ideally, the method can then be used in situations where the participants clearly see the advantages of participating, and where they can be entitled to participate without the fear of time constraints. If these prerequisites can be granted, the Internet Delphi can be used for systematic assessment of any kind of new technology or methodology in pharmaceutical manufacturing or pharmaceutical quality assurance. There should, however, always be available at least one independent, neutral person to serve as a reporter between the rounds and after the survey. Furthermore, the possible firewall and browser problems have to be taken into consideration before the start of the survey.