Validating Sterile Filtration: Overcome the Fear of Failure 4

“I need an absolute 0.1- or 0.2-µm-rated filter.”
 
 As a former FDA authority, since retired, once observed, “The word ‘absolute’ should be used only in conjunction with vodka.” Absoluteness implies a complete independence from conditions, an inherent ability to retain particles larger that than the filter’s pore size rating, regardless of any other considerations. Without a complete knowledge of the properties of the particles and filter pores at our disposal, the statement is devoid of technical significance or guidance. It may, perhaps, be used in ignorance (although cynics may suspect that its utility derives from marketing efforts, a practice not unknown in the competitive world of sales.)
Control vs. Fear
As Sandman elucidated, human beings like to be in control, and, if this status cannot be achieved, may move rapidly to fear. Unfortunately, when sterile filtration is concerned, fear can result in the installation of wasteful, unnecessary safety nets that can create more problems than they solve.
Being in control is the desired state, and such control can only come from process validation studies. Their authority is at least as old as Lord Kelvin’s basic scientific principle, “When you can measure what you are speaking about, and can express it in numbers, you know something about it.”
It speaks to validation. In sterile filtration, as in most areas of pharmaceutical manufacturing, science-based validation is the best cure for fear.

---

References
1. Hessler, A., Sandman, P.M. Squeaky Clean? Not Even Close. http://www.nytimes.com/2004/01/28/dining/squeaky-clean-not-even-close.html?sec=health?pagewanted=1
2. FDA. Guideline on General Principles of Process Validation, FDA CDER, 1987.
3. Agalloco, J.P. “Compliance Risk Management Using a Top-Down Validation Approach,” Pharmaceutical Technology, July 2008.
4. PDA Technical Report 26 (2008), Sterilizing Filtration of Liquids, Parenteral Drug Association, Bethesda, MD.
5. Ridgway, H.F., Rigby, M.G., and Argo, D.G. “Adhesion of a Mycobacterium to Cellulose Diacetate Membranes Used in Reverse Osmosis.” Applied and Environmental Microbiology 47, 1984, pp. 61-67.
6. Tolliver, D.L. and Schroeder, H.G. “Particle Control in Semiconductor Process Streams.” Microcontamination (l), 1983, pp. 34-43 and 78.
7. Bowman, F.W, Calhoun, M.P. and White, M. “Microbiological Methods for Quality Control of Membrane Filters.” J. Pharm. Sci., 56/2, 1967, pp. 453-459.
8. Leahy, T.J., Sullivan, M.J. “Validation of Bacterial Retention Capabilities of Membrane Filters.” Pharmaceutical Technology 2(11), 1978, pp. 64-75.
9. FDA. Guideline on Sterile Drug Products Produced by Aseptic Processing, FDA CDER, 1987.
10. Sundaram, S., Eisenhuth, J., Howard Jr., G.H., and Brandwein, H. “Part 1: Bacterial Challenge Tests on 0.2 and 0.22 Micron Rated Filters.” PDA Journal of Pharmaceutical Science and Technology, 55 (2), 1984, pp. 65-86.
11. Sundaram, S., Auriemma, M., Howard Jr., G.H., Brandwein, H., and Leo, F. “An Application of Membrane Filtration for Removal of Diminutive Bioburden Organisms in Pharmaceutical Products and Processes,” PDA Jour. Pharm. Sci. and Technol. 53 (4), 1999, pp. 186-201.
12. Krygier, V. Rating of Fine Membrane Filters Used in the Semiconductor Industry, Transcripts of Fifth Annual Semiconductor Pure Water Conference, (1986), pp. 232-251, San Francisco, CA
13. PDA/FDA Special Scientific Forum, Bethesda, MD; Validation of Microbial Retention of Sterilizing Filters, July 12-13, 1995.
14. Mittleman, M.W., Jornitz, M.W., Meltzer, T.H., “Bacterial Cell Size and Surface Charge Characteristics Relevant to Filter Validation Studies,” PDA Jour. of Pharm. Sci. and Technol. 52 (1), 1998, pp. 37-42.
15. Agalloco, J., Letter to the Editor—re: “It just doesn’t matter, It just doesn’t matter, It just doesn’t matter.” PDA Journal of Science and Technology. Vol 52, No. 3, pp. 149-150.

Validating Sterile Filtration: Overcome the Fear of Failure 3

Based on available data, long term filtrations may best be handled by 0.1-µm-rated filters, subject to validations being performed. However, in other cases, substituting 0.1-µm-rated for 0.2-µm-rated membranes may be unnecessary, and could result in significant penalties, including:
 Slower flow and processing rates, resulting in longer term operations.
 Higher costs for larger EFAs
 More leaching and extractables
 Higher product losses, due to adsorptive bonding to the ultimately greater filter area used.
A responsible choice requires that both the 0.1 µm-rated membranes and the 0.2 µm-rated membranes be validated.
If both types of filter prove appropriate, the higher pore size rating should be used to avoid the penalties of reduced flows. If, however, the validation data do not permit a clear resolution, the 0.1 µm-rated membranes should be used, since retention is more critical than flow rate or flux.
Below, we address some of the common sterile filtration concerns, requirements or practices that appear to be motivated by fear and can best be resolved by careful process validation.
“0.2-µm filters are penetrated by organisms. The industry is, therefore, required to switch to 0.1-µm-rated filters.”
In certain specific processes, 0.2-µm–rated filter can be penetrated by organisms, or by organisms which would normally be retained by such filters. In such cases, the flltrative removal of the organisms may well require the use of 0.1 µm–rated filters. Such instances are not new. Their occurrences have been considered by regulators for years, at least since the PDA and FDA held a special forum on this topic in 1995 [13].
   
Certain organisms, such as Burkholderia pickettii, Burkholderia cepacia, and Pseudomonas aerugenosa. shrink as a result of their immersion in fluid media that are only minimally nutritious for them [14]. Their reduction in size renders as invalid validations that use B. diminuta as a model. Brevundimonas diminuta can undergo shape alterations in minimally nutritious media but is not listed as undergoing size alterations occasioned by contacts with process fluids.
The fact that some microbes require 0.1-µm–rated filters to arrest them does not signify that all organisms are so disposed. The necessitated switch from 0.2-µm-rated to 0.1-µm–rated happens in only roughly 0.005% – 0.01 % of sterilizing grade filtration applications.
A mandated switch is therefore scientifically and statistically unfounded. Its promulgation may be shunned and process validation activities and data used as performance verification. Sole reliance on pore size ratings have been found obsolete anyway.
“Increasingly there are detectable but non-culturable organisms or L-forms or nano-bacteria in our processes.”
Conclusions cannot be made regarding the sterile filtration of microorganisms unless the methods of quantifying them by culturing and counting are available. Organisms such as the L-forms, nanobacteria, and “viable but non-culturable” entities may not be amenable to such analyses. Concerns about their presence may be justified, but without the means to cultivate and count them, it is impossible to attest to their complete absence.
It follows that a sterilizing filter can be judged only by its performance in the removal of identifiable and culturable organisms known to be present in the drug preparation [15]. The complex of influences governing the outcome of an intended sterilizing filtration necessitates a careful validation of the process, including that of the filter [4]. The very drug preparation of interest, the exact membrane type, the precise filtration conditions, and the specific organism type(s) of concern must be employed in the necessary validation.
“Redundant 0.2-µm filtration is necessary and should be used.”
Not necessarily. Again, proper process validation will disclose whether a single filter will do the job or not. However, there are some specific applications which traditionally, for whatever reason, utilize a second (redundant) filter as an “insurance filter,” i.e. if the first filter fails, the second may compensate. This holds, however, only when each filter has been validated to show specified retentivity.
Even so, the wisdom of the exercise deserves careful evaluation, as it assumes added costs for membrane EFA, increased leachables and extractables. The loss of drug product may needlessly be incurred by the filter’s heightened product hold-up, and unspecified adsorption.
    
“The maximum bioburden in front of a sterilizing filter should be 10 cfu per 100 mL of fluid.”
This is true if one wishes to accord with EMA regulations, and especially if one wishes to export product to Europe. The FDA makes no such stipulation, but bases its approval on process validation.
Seemingly in conflict, the two views arise from the same premise. The EMA regulation tries to establish the same sterility assurance level (SAL) for filtration as for thermal sterilization. EMA recognizes that, the greater the number of challenges, the more likely that at least one will succeed.
The FDA seems to agree, in that if the filter can sustain the removal of organism burdens far above those liable to be encountered in real life situations, it can assuredly withstand lesser insults. If, as the authors see it, the FDA’s massive challenge fails to breach the filter’s pores, it is needless to compel bioburden assessment in front of the filter. Filter validation would gainfully serve the intended purpose. Process validation, effectively conducted, would reliably demonstrate the filter action.

Validating Sterile Filtration: Overcome the Fear of Failure 2

From Tolliver and Schroeder (1983) courtesy of Microcontamination
B. diminuta should not be viewed as a universal model organism, as some native bioburden may be a better alternative as challenge organism, being close to the actual process settings. Unfortunately, rare penetrations of sterilizing grade filters have caused an exaggerated doubt in the reliability of filtration. Appropriate process validation though should render such doubts and be trusted by even the most critical reviewer of sterile filtration..
Sources of Variability: Size and Shape
B. diminuta varies in size and shape, depending on how it is cultivated. Back in 1967, Bowman and colleagues described the B. diminuta size as 0.3 × 1.0 μm [7]. However, in 1978, Leahy and Sullivan found [8] that the organism grown at 30 °C and incubated for 24 hours in saline lactose broth, a minimally nutritional medium for that microbe, yielded cocci-like cells approximately 0.3 × 1.0 μm (Figure 1). Similar considerations have to be accounted for when challenge tests are performed with native bioburden forms.
B. diminuta are typically cultivated to develop as spherical a form as possible, since spheres are least amenable to retention. Thus, Leahy and Sullivan proposed, back in 1978, that it be used as the model organism for 0.2/0.22-μm-rated membranes, partly because of its size relative to the 0.2-μm dimension [8].
Subsequently, the FDA designated it for that very purpose [9], defining a sterilizing filter as one that retains a minimum of 1 x 107 cfu of Brevundimonas diminuta ATCC 19146 / per cm2 of effective filtration area (EFA).
Although it isn’t the smallest organism known, B. diminuta was considered diminutive enough to represent whatever smaller organisms were likely to be present in pharmaceutical preparations. The smaller the test organism, goes the logic, the more likely that its removal by a filter would assure the sieve retention of larger organisms.
However, ease and safety of cultivation and handling are also important considerations. In 2001, Sundaram and colleagues found an increasing number of cases where filtration in 0.2/0.22-µm-rated membranes failed to yield sterile effluent [10]. Experimental studies showed that penetrating organisms had shrunken because they had been cultivated in broths that were nutritionally inadequate. In such cases, the physicochemistry of the suspending fluid may serve to alter the size of the suspended organisms as expressed by the Donnan equilibrium consequent to ionic strengths.
Organism Shrinkage During Processing
Sundaram’s team [11] also found that organisms underwent size changes after exposure to certain drugs. In cases with 0.2-µm-rated membranes, the researchers found, the larger pore size could only provide sterile effluent and/or a high titer reduction with regard to certain organisms for various lengths of time, before penetration occurred.
Penetration times varied from 24 to 96 hours, and the cumulative challenge at which penetration was first observed ranged from 1.2 x 107 to 1.1 x 108 cfu/cm2. Two 0.2-µm rated Nylon-66 filters in series were unable to fully retain Ralstonia pickettii (now Burkholderia pickettii) with penetration observed at 72 hours, corresponding to a cumulative challenge of 2.4 x 107 cfu/cm2. The more extensive penetration of the Nylon-66 membranes, compared with the PVDFs, is in keeping with their greater degree of openness, as Krygier and colleagues showed in 1986 [12].
As a result, it has been suggested that 0.1-µm-rated membranes be substituted for their 0.2-µm-rated counterparts.
Sundaram’s team evaluated five 0.1-µm-rated membranes and found that they yielded sterile effluent over the entire duration of the test (120-196 hours), up to challenge levels of 5.7 x 107 to 2.0 x 108 cfu/cm2. Similar results were obtained with the PVDF filters tested; no B. pickettii were detected at challenge levels of 5.9 x 107-6.0 x 108 cfu/cm2.
In addition, all 0.1-µm-rated filters tested provided consistent and complete retention of B. pickettii for the entire duration of the test (120-192 hours), suggesting that the smaller pores would ensure sterile product at conditions where penetration could occur through conventional 0.2- and 0.22-µm-rated sterilizing grade filters. Proponents argue that using 0.1-μm-rated membranes would permit longer term formulation and filtration operations. In fact, 0.1 µm-rated filters may be the best choice for long-term filtrations.
However, penetration has also been found in 0.1 µm-rated filters. In 1999, Sundaram’s team found that B. pickettii, when its size was so affected, could be retained by certain 0.1-µm-rated filters. But, in a similar situation, they found that only four of seven commercially available 0.1-μm-rated membranes could remove a particular organism. Just because one type of membrane so classified may provide proper retention, does not mean that any other 0.1-µm-rated membrane can also be depended upon for a like result.
It is important to remember that today, there are no industry standards by which 0.1 filters can be judged. In addition, more research clearly needs to be done into the kinetics behind the organism’s size changes, evaluating different organisms in different fluids.

Validating Sterile Filtration: Overcome the Fear of Failure 1

For a process as critical and sensitive as sterile filtration, it’s not surprising that many filter end-users experience some “fear of failure.” As risk management consultant Peter Sandman has said, people become fearful when they feel they are not in control [1]. As a result, some may see established sterile filtration practices as inadequate. Compelled by fear to do more than necessary, they may overdesign their filtration systems, adding wasteful and costly procedures.
By embracing this fear, rather than studying the root causes of filter penetration and the specifics of each process, practitioners of sterile filtration impede the advancement of filtration science and its understanding within the industry and global regulatory agencies.
This article will examine several unnecessary practices that are burdened upon the industry today, using process validation principles to explain why they are inadequate. It will also touch on problems that result when “model organisms” used to evaluate sterile filters are seen as universal archetypes that can characterize all process situations and filter types.

ADVERTISEMENT

Fast, Accurate Integrity Test Instrument
The Palltronic® Flowstar IV integrity test instrument is fast, accurate and designed for use in laboratories or cleanrooms. Offering latest networking and automation options, it is supported by extensive qualification documentation with worldwide technical and aftersales support.

First, let’s consider the concept of validation itself, which is designed to “establish documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its pre-determined specifications and quality attributes [2].
Validation is a global regulatory requirement for drug manufacturing, particularly for aseptic processes. As industry consultant Jim Agalloco has written [3], there is already a substantial body of knowledge on best validation practices for sterile injectable biotech products, including:
 Fluid influences on the organisms, membrane polymer and retention mechanism
 Process parameter influences on the membrane polymer, filter construction and retention mechanism
Validated processes are under control, yet their reliability continues to be questioned by regulators and those in the field, suggesting that many practitioners do not fully understand the validation concepts. This can often be traced to subjective fears rather than valid technical concerns.
Here are some examples of current concerns, requirements or practices, pulled directly from regulatory or company documents, that illustrate that fear:
 “0.2-µm filters can be penetrated by microorganisms, so the industry must switch to 0.1-µm-rated filters.”
• “Increasingly, there are detectable but non-culturable organisms or L-forms or nano-bacteria in our processes.”
• “Redundant 0.2-µm filtration is necessary and should be used.”
• “I need an absolute 0.1- or 0.2-µm filter.”
• “Diffusive flow integrity testing is better than bubble point testing.” (Currently, there is a difference in preference between U.S. and European regulators)
• “Flawed filters will not be detected by a post-use test, as the pore will be plugged during filtration.”
• “The maximum bioburden in front of a sterilizing filter should be 10cfu/100mL.”
 
We will address many of these statements individually, later in this article.
First, however, let’s consider the industry’s current understanding of the role of the “model organism” in filter testing and validation practices. Obviously, the industry needs a model, since it would be impractical to test each and every microorganism, given the number and diversity of microbes in pharmaceutical settings. Furthermore, most of the organisms are not of concern, since their size offers no challenge to modern sterilizing grade filters.
The question is: does the most widely used model organism always reflect specific process and filter characteristics?
Beyond Size Exclusion: There Is No Universal Model Organism
Brevundimonas diminuta is typically used as the model for organisms that are expected to be found in pharmaceutical manufacturing environments. It was selected, based on its presence in pharmaceutical operations, and within native bioburden [4].
The organism has been found particularly suitable for validating sterilizing grade filters due to its size and ease of cultivation. However, it should only be used to model situations where its dimensions closely match the organisms of interest in a given application, relative to the filter pore size and shape.
Furthermore, in some cases, sieve retention may not be the mode of organism removal. For instance, in some cases, it may be adsorption, as the organism forms hydrogen bonds to the filter’s polymeric surface. This could account for the observation, 26 years ago, that Pseudomonas aeruginosa organisms are more strongly retained by polyamide membranes than by cellulose triacetate filters [5]. It also explains the removal of latex particles from aqueous suspensions by polyamide membranes in the presence of surfactant, but not in its absence (Table 1) [6].
Table 1: Retention (%) of 0.198-µm spheres by various 0.2-µm-rated membranes

Filter Type	In Water (% )	In 0.05% Triton X-100 (%)
Polycarbonate	100.0	100.0
Asymmetric polysulfone	100.0	100.0
Polyvinylidene fluoride	74.8	19.2
Nylon 66	82.1	1.0
Cellulose esters	89.4	25.1

Validation Training

by Brenda M. Wenzel, Air Force Research Laboratory, Warfighter Training Research Division
and Brent H. Hill, Director of Validation, CETech Validation Services, Inc.
Pharmaceutical organizations have a training need for validation skills that cover the areas of protocol execution, protocol development, validation project management, and documentation control. This need can best be met through knowledge and skills training that is customized for individual organizations. Customized training is intended to enhance the transfer of knowledge and skills from the learning environment to the working environment. Critical to getting the greatest benefit out of customized-training dollars is selecting the best candidates for training, and providing immediate opportunities after training to use newly acquired knowledge and skills.
The goals of the customized training effort described here were to design, develop, and evaluate a validation training program. The objective of the training program was to produce qualified personnel for Installation Qualification (IQ) and Operational Qualification (OQ) protocol execution and development. Core aspects of the training program are presented, along with the training evaluation results. Presented first is an overview of the training design, development, and delivery process.
Validation Training Design
The challenge in designing a training program such as the one described here is to create a modular system that can readily be customized to meet customer needs. Thus, a needs assessment is required before decisions can be made on the instructional method and media to be used in the training design. Part of the needs assessment involves an analysis of the organization, which includes detailing customer needs. The other two parts of the needs assessment are a task analysis and person analysis. Our approach to all three analyses is described in the following sections.
Organizational Analysis
Analysis of the organization included interviewing stakeholders and surveying department managers. Customer needs were identified and nominees for training were collected in the interviews. The expected number of employees to be trained to what level of performance was established from the interview data. It was determined that the customer had an immediate need for qualified personnel to execute IQ and OQ protocols, and an imminent need for qualified personnel to develop IQ/OQ protocols. In-house qualified protocol executors meant that contractor services could begin to be phased out with the goal of reducing costs (Wenzel, Hill, Novosad & Eckstrom, 2001).¹
The departmental survey established the climate of the work environment. It was important to the success of the training program to know if the work environment provided opportunities to use knowledge and skills acquired in training, and if it was supportive of newly trained employees. The departmental survey also asked for nominees to be considered for validation training. Several candidates in the trainee pool came from responses to the departmental survey.
Task Analysis
Conducting the task analysis involved breaking down the jobs of Protocol Executor and Protocol Developer to a level of granularity that specified the Knowledge, Skills, Abilities, and Experiences (KSAEs) necessary to do the jobs. Results from the task analysis were key in determining training objectives. The results also served as the basis for the person analysis. Prior to submitting results from the task analysis to the training designer, a Subject Matter Expert (SME) rated each KSAE, pertaining to protocol executor and protocol developer, based on the dimensions located in Figure 1.
A KSAE was included in the training criteria if it received the following ratings from the SME: “2” on Knowledge or Skill is Expected to be Acquired, and a “3” or greater on Relative Frequency of Application of Knowledge or Skill to Validation, and “3” or greater on Importance of Task to Effective Job Performance. The KSAEs that made the first cut were flagged if they received a “3” on Level of Recall Necessary to Apply Knowledge on the Job, or a “3” or more on Level of Difficulty Learning. The flags indicated to the training designer that emphasis needed to be put on related training materials.
Person Analysis
There are three attributes of an employee that constitute an excellent candidate for training. One, the employee is motivated to learn. Two, the employee will use the new knowledge and skills acquired in training. And, three, the employee possesses prerequisite skills to obtain the greatest benefit out of training. Of course, the employee also has to be available to attend training.
A self-assessment survey, similar to the instrument used by the SME to rate the KSAEs was created from the list of KSAEs. Each person to be considered for training completed the survey by rating his KSAEs on the dimensions located in Figure 2.
Candidates for protocol executor and protocol developer training were selected by analyzing data from the self-assessment survey. The self-assessment survey data were submitted to the following six-step process to create the candidate lists as shown in Figure 3.
A primary candidate list and secondary candidate list were created. The organization initially wanted a group of ten individuals trained on protocol execution, and a smaller select group from the original group trained on protocol development. However, the group of ten that was initially chosen attended both the protocol executor and protocol developer training.
Training Design and Development
Results from the needs assessment were compiled and passed onto the training designer. The results identified a need for basic validation training. Basic validation training, therefore, became a prerequisite for both protocol executor and protocol developer training. The training designer then made decisions on training content, and the training strategy to be implemented by the training developer.
Training Strategy: Method and Medium
The training designer decided on a minimalist training approach (Carrol, 1984).² Advantages of the approach included:

(a) Training is focused on real tasks(b) Reading training materials is kept to a minimum(c) There is coordination between demonstrations and hands-on participation (Palmiter & Elkerton, 1993)³(d) Trainers are prepared to deal with execution errors(e) Trainee motivation is kept high by using actual work tasks (in this case, executing and developing protocols).

The instructional elements were designed according to Gagné and Briggs’s (1974)⁴ nine events of instruction: gain attention, present learning objective, activate prerequisite knowledge, present material, guide learning, elicit performance (practice), provide feedback, assess performance, facilitate retrieval, and enhance transfer (Gagné, 1970).⁵
An audit was required to determine the in-house training capacity, i.e., facilities, equipment, and computer resources of the organization. Based on results from the audit, the training designer chose to design the training as lock-step instruction using computer-based presentations. The instructor decides when specific training materials are presented with the lock-step method. Self-paced, interactive multimedia courseware was considered as a training strategy. However, given the short turnaround time from needs assessment, to training delivery, and the lack of data on the computer skills levels of the trainees, Computer-Assisted Instruction (CAI) was the chosen strategy. Microsoft PowerPoint® was used as the training platform. PowerPoint® had the advantages of multimedia (sound, video, animation) capability, easily modified, self-executing presentation, and the audience had a general familiarity with the application.
Training Modules
Three training modules were targeted for development: validation boot camp, protocol executor, and protocol developer. A description of the training setting, an overview of training content, and the learning objectives for each training module are presented next.
Validation Boot Camp: Basic Validation Training
Validation boot camp presented the basics of validation. All trainees were required to attend boot camp training. It was conducted in a designated learning center equipped with a Pentium III personal computer connected to a projector. Boot camp training consisted of a PowerPoint® presentation that covered definitions of validation, the FDA’s role, the company’s validation philosophy, flowcharts of validation-related processes, validation-related Standard Operating Procedures (SOPs), the importance of documentation, and documentation control issues.
To gain and keep the trainees’ attention, the instructor donned a drill instructor uniform and attitude. It was extremely effective in this particular setting, but may not be effective in all training settings. Trainees participated by reading instructional material aloud. The learning objectives for boot camp are found in Figure 4.
Protocol Executor Training
Protocol Executor (PE) training was conducted in a small conference room. A laptop computer, projector, and overhead projector were used for training. The computer and projector were used for the Powerpoint® presentation. The overhead projector was used to display transparencies of IQ and OQ protocols.
PE training consisted of both lock-step instruction and hands-on training.
The Powerpoint® presentation outlined the protocol execution process, deviation report process, necessary documentation for protocol execution, critical equipment, test instrument calibration, and how and where to acquire resources. Instruction was designed to provide evaluative feedback to trainees on their pretest performance (see Training Evaluation Instruments and Results section). Group exercises were conducted so trainees could acquire experience completing IQ protocols, OQ protocols, and deviation report forms. Sample protocols were distributed along with dummy data for the group exercises. Blank deviation report forms were available for situations when the dummy data fell outside of the acceptance criteria. The learning objectives for PE training included increases in both knowledge and skills. The learning objectives are listed in Figure 5.
Protocol Developer Training
Protocol Developer (PD) training was conducted in a computer laboratory. A computer with projector setup served as the training platform. All of the necessary electronic files (e.g., IQ template, OQ template, final report template, example data forms) were loaded on the hard drive of the computer, along with the PowerPoint® presentation.
Protocol Developer training consisted of both lock-step instruction and hands-on training. The PowerPoint® presentation outlined the protocol development process, necessary documentation for protocol development, final report preparing process, and how and where to acquire resources.
A trainee facing the upcoming validation of a new piece of equipment provided a Turn Over Package (TOP) for training purposes. The TOP served as the basis for the protocol development exercises conducted during PD training. The exercises were conducted as a group. Each trainee had an opportunity to sit at the keyboard and complete development tasks. Included in the group exercises were:

• Modifying the IQ and OQ templates to reflect the new equipment.
• Developing data forms and inserting them into the modified IQ template.
• Developing functional test forms and inserting them into the modified OQ template.

The final report templates were opened on the computer and reviewed by the trainees.
Hands-on instruction showing how to prepare a final report package was provided to trainees. Small groups of trainees were supplied with the necessary documents to practice preparing final report packages. The learning objectives for PD training focused on increases in knowledge. The learning objectives are listed in Figure 6.
The documentation control person presented additional information on accessing and modifying protocol templates, and submitting electronic protocol drafts on the final day of PD training. Much of the knowledge and many of the skills covered related directly to the hands-on portion of PD training.
Training Schedule
Boot camp training was conducted in a four-hour block. Following boot camp training, PE training was conducted in two four-hour blocks across two days. PD training was conducted in three four-hour blocks across three days the week immediately following PE training. All training modules were offered in a morning and afternoon session to accommodate first and second shift workers. One individual worked third shift and had his workweek adjusted to fit the training schedule. The number of trainees attending the sessions fluctuated from six or seven in the morning and three or four in the afternoon, respectively. This was not a problem since the modules were standardized across sessions.
Training Evaluation Instruments and Results
Evaluation is a critical aspect of a complete training program. Only through evaluation can it be determined if training criteria and organizational needs are met. In addition, evaluation results are critical to improving training.
The evaluation instruments that were used are described next, along with the evaluation results.
Comparison of Trainee Candidate Lists
A comparison of the primary candidate and secondary candidate lists (results from analysis of the self-assessment survey) to the employer-selected candidate list revealed that seven of the ten who attended training were on the primary candidate list; two were on the secondary candidate list, and one on the employee-selected list. The trainee from the employee-selected list had not completed the self-assessment survey prior to training.
Evaluation Procedure
The anonymity of the trainees was established by assigning each an arbitrary number to use for identification purposes on the evaluations. Evaluation booklets were created for each training module-boot camp, protocol executor, and protocol developer. The evaluation booklets contained informed consent, pre- and post-measures, confidence items, and the training assessment questionnaire. Verbal and written instructions were provided so that trainees knew when to start and stop the evaluation. All PE and PD knowledge and skills pre-measures were administered prior to the start of boot camp to prevent any overlap in training content across the modules.
Assessment of Knowledge and Skills
The pre-measures of PE and PD knowledge involved a multiple-choice written test. The pre-measure of PE skills involved completing pages from IQ and OQ protocols based on data provided. The post-measure of PE knowledge was an alternate form of the multiple-choice test. The post-measure of the PE skill involved field execution of dummy IQ/OQ protocols for a 75 cubic feet V-Blender. The protocols were printed on paper marked “SAMPLE” so that they would not be confused with official documents. The post-measure of PD knowledge was an alternate form of the multiple-choice test.
Knowledge and skill gains were assessed by measuring the increase in the percent of correct responses. Unidirectional paired-sample t-tests were used to determine the effects of training on knowledge and skills. Results are shown in Figure 7. All increases in knowledge scores and skills were statistically significant. The average percent correct on the PE knowledge test increased from 46 before training to 89 after training (t(9) = 9.7, pvalue < .0001). The average percent correct on the PE skill test increased from 64 before training to 93 after training (t(9) = 4.9, pvalue < .001). The average percent correct on the PD knowledge test increased from 50 before training to 80 after training (t(9) = 5.2, pvalue < .001).
The extensive hands-on experience provided in the PE training likely contributed to the dramatic increase in “executor” knowledge. There was minimal hands-on experience proved in the PD training due to restricted computer resources. This lack of extensive hands-on experience likely contributed to the smaller, yet significant, increase in PD knowledge.
Confidence in Knowledge and Skills
Trainees were asked to rate their confidence in their knowledge and abilities with IQ and OQ execution and development before and after both the PE and PD training. Figure 8 describes a statistically significant increase in confidence as a function of training.
Trainees reported being more confident in their protocol execution knowledge and abilities than their protocol development knowledge and abilities across both pre- and post-measures.
Reactions to the Training Experience
Trainees’ reactions to the training experience were collected in two ways. At the end of each training module the Training Assessment Questionnaire (TAQ) was administered, along with three open-ended items. The TAQ covers 11 aspects of effective training. Each survey item was rated on a seven-point scale anchored by a pair of reciprocal descriptors. Here is an example of a TAQ item:
“The lesson objective was not clearly/clearly presented” (“1” represented not clearly and “7” represented clearly).

The TAQ was useful in indicating where training was lacking. Items with average ratings that fell below “5” indicated a need for improvement. Results from the TAQ are presented in Figure 9. Not surprising to the instructors, trainees rated the following item from the TAQ – “The training was easier/more difficult to understand than I would have liked it to be” – below “5” on average across the training modules. The importance and complexity of validation is not usually realized until employees have a broad understanding of the validation process.
The other items that received average ratings below “5” were recorded on the TAQ with comments such as:
“Question-and-answer sessions were inadequate/adequate for learning” (M = 4.8) and: “The pace of training was inadequate/adequate for learning” (M = 4.9), however, only for the boot camp training module. Boot camp training for the first group diverged from PE and PD training in two ways. First, the head of engineering sat in on the morning session training. Second, the instructor was told that he had an hour less time than what he had planned for; therefore, he accelerated the presentation of the materials. These differences are potentially reflected in the ratings for the adequacy of question and answer sessions and pace for learning items.
At the end of both the PE and PD training modules, trainees were asked to respond to three open-ended items: What was of most value?, What was your favorite part?, and how could training be improved? The item responses are listed in Figure 10 and Figure 11.
The hands-on experience that trainees received proved to be a favored aspect of training. An interesting finding from trainee responses to the “how to improve training” item for PE (although, not in direct response to the item) was the expressed desire to get in the field and put their new knowledge and skills to work.
Discussion
The validation training program described here is highly effective in producing increases in knowledge and skills related to IQ/OQ protocol execution and development. Trainees find the training motivating and highly relevant to their jobs. Further, the training produces increased confidence in trainees’ self-reported IQ/OQ protocol execution and development knowledge and abilities. Missing from the training evaluation is an assessment of the degree that training transferred to the work environment.
Trainees were most receptive to learning about and doing protocol execution. Protocol execution does not require intimate knowledge of process equipment, nor does it call for a full understanding of validation requirements. As a rule of thumb, ninety percent of validation knowledge and skills are easily acquired through training and experience, and the remaining ten percent take years of experience to acquire. Protocol execution falls into the ninety percent category. Organizations should take this into consideration when deciding where to spend their validation training dollars.
The modular approach taken in the design stage produced a training program that was readily modifiable to meet changing organizational needs. Training needs may include modules for validation project managers or documentation control personnel, both experienced and non-experienced personnel, refresher training, and train-the-trainer programs. Regardless of the training need, training should be contextualized to enhance the transfer of knowledge and skills from the learning environment to the working environment.
The time required to produce a validation training program such as the one described here varies. It is inversely related to the experience level of the instructional developers and trainers, accessibility of validation expertise, the number of training hours delivered, and quality of the instruction.
Organizations may believe that there is economy in eliminating the needs analysis from the training development process. However, training criteria and trainee selection are two vital bits of information that result from the analysis. Both are essential to the success of a training program.
Validation training and training dollars are often unnecessarily wasted. Organizations should be cautious and ask for evidence that the training they are purchasing is able to produce results. Organizations should use more than the dollar sign as selection criterion. Ask the group offering the training to provide empirical evidence that their training is effective. Also ask for a list of customers. Contact companies on the customer list, and ask if they were satisfied with the training they received, moreover, were they satisfied with the training results.
About the Authors
Brenda Wenzel has a background in training development and evaluation. Her research spans the areas of human engineering, cognitive and social psychology, and training technologies. Her work has been presented at national and international conferences. Wenzel was formerly with CETech Validation Services, Inc. She is currently with the Air Force Research Laboratory, Warfighter Training Research Division. She can be reached by phone at 480-988-6561, by e-mail at brenda.wenzel@williams.af.mil.
Brent Hill is Director of Validation at CETech Validation Services, Inc. His specialty is in control system design and managing plant start-ups. Hill has 18 years experience with experience designing, installing, and qualifying pharmaceutical facilities. He has been with CETech Validation Services since 1994. He can be reached by phone at 303-279-4238, by fax at 303-279-3735, by e-mail at brent@cetonline.com.
References

1. Wenzel, B., Hill, B., Novosad, M., & Eckstrom, R. (2001). Transitioning from Contract Validation Services to an In-House Validation Capability. Journal of Validation Technology, Vol 7. No 4.
2. Carroll, J. M. (1984). Minimalist Training. Datamation, November, 25-136.
3. Palmiter, S., & Elkerton, J. (1993). Animated Demonstrations for learning procedural computer-based tasks. Human-Computer Interaction, 8, 193-216.
4. Gagne, R. M., & Briggs, L. J. (1974). Principles of instructional design. New York: Holt, Rinehart and Winston, Incorporated.
5. Gagne, R. M. (1970). The conditions of learning (2nd ed.). New York: Holt, Rinehart and Winston, Incorporated.