Professional competence of nurses in emergency services: evidence of content validityHolanda, Flávia Lilalva deI Marra, Celina CastagnariI Cunha, Isabel Cristina Kowal OlmI
- IUniversidade Federal de São Paulo, Paulista Nursing School. São Paulo, São Paulo, Brazil.
To verify the evidence of content validity of the Identifying Issues (Identification questions) elaborated from the theoretical-logical model of the Matrix and the Profile of Professional Competence of the emergency nurse.
Descriptive and methodological study of a quantitative approach with a psychometric reference as a precursor in the elaboration of a measurement instrument for the evaluation of competencies. The study was conducted in 2013 in Brazil, with a national sample of nurses specialized in emergency services and/or with professional competence in the area. Three stages were considered: questionnaire development; data collection with Delphi, appropriate statistics for Likert; and interpretative analysis of the comments/suggestions of the analyzed issues.
Delphi was used in four stages. There were minor adjustments to the content and inclusion of a new Competence. They obtained 90% of Score Percentage and 98.61 of Content Validity Index.
There was consensus among the experts and the research demonstrated evidence of content validity, suggesting pertinence and adequacy to represent the constructs of competencies.
Emergency Nursing Area, Professional Competence, Validity Studies, Validity of the Tests, Health Research Evaluation
Addressing the subject of professional competencies in the field of Nursing in an academic and scientific way requires special care. In addition to the polysemic complexity of the construct, it is also necessary to consider the diversity of places, contexts, and levels of action of the nursing professional, which require different competencies to provide a health care service of excellence; added to the fact that the use of this concept in the field of Nursing is still in the process of development and crystallization.
One of the areas in which competencies are reflected is the emergency services, where it is common to deal with people facing imminent life-threatening situations. Therefore, nurses and other team members must be ready and able to provide assistance, to preserve the patient's health, avoiding sequels and other factors of physical and emotional injury.
Based on the specificities of the nursing work in a excellence standard and in the theoretical procedures proposed by Psychometrics(1-2), Brazilian researchers developed a theoretical and empirical study in which they proposed a model of Basic and Associated Competencies to guide the professional activities of the nurse in emergency services, denominated Professional Competence Matrix (or Core Competence)(3). In this Matrix, in addition to electing the type and number of competencies, it was also possible to define each of them constitutively. In a subsequent study, the same authors developed the expected Professional Competence Profile for the nursing work in emergency care situations(4).
Although the development of the Matrix and the Profile is in the essence of the elaboration process of a measurement instrument, these were not enough to guarantee the achievement of the initially defined objectives, making it necessary to demonstrate the technical qualification of the phenomenon under study with validity evidence.
This validity can be conceptualized as the degree to which the empirical evidence, orchestrated with a coherent and sustained theoretical reasoning, substantiates the inferences and interpretations on the psychological characteristics of the people from the behaviors observed in the tests (5). Contemporary scientific literature presents five sources of evidence, which are based on the Content, the Response Process, the Internal Structure, the Relationship with External Variables, and the Consequences of Testing(5-6). Each one of these five sources of validity evidence information deals with a particular aspect of a single conceptual point on the interpretation of behaviors presented in testing.
As the cumulative validation process includes a set of scientific evidence(6), its verification in the Matrix and in the Professional Competence Profile was necessary. Therefore, it was considered pertinent, at this stage of the research, to use the first source of information, i.e., the Content. Thus, this study is in line with the abovementioned studies(3-4) since it tried to continue the initial procedures to develop an instrument of measurement to evaluate the professional competencies of nurses working in emergency services.
In this scenario, it was important to verify to what extent the Identifying Issues, as operational definitions of competencies, are valid to represent the competent action of the nurse in the emergency area, according to the one recommended in Psychometric references(1-2,5-7). In view of the above, the following was verified in this article: the content of the Identification Questions originating from the Matrix and crystallized in the Professional Competence Profile consists of a suitable theoretical, practical, and comprehensive framework to contain the validity evidence of the competent nursing activities in emergency services with high-quality standards? To answer this question, the objective was to verify the evidence of validity based on the content of the Identifying Issues elaborated from the theoretical-logical model of the Matrix and the Profile of Professional Competence of the nurse in emergency services.
Project approved by the Research Ethics Committee. The ethical requirements established in Resolution No. 466/2012 of the National Health Council (CNS), including the Informed Consent Form (ICF), were followed. The consent of the subjects was requested and obtained in the "I agree" option, since they were contacted through online platforms.
Design, study location and timeline
This descriptive and methodological study of quantitative approach was conducted in a public university located in the city of São Paulo, Brazil, in 2013(8) and became one of the precursor stages for the elaboration of an evaluation instrument of the nurses' professional competence. This instrument is based on psychometric references, whose objective is to guide the search for evidence of content validity. Psychometrics is a technique of measurement by theory that allows the interface of Psychology with other sciences not only to assign a numerical value to phenomena that, by themselves, cannot be measured statistically, but also to explain the meanings of the answers given by the subjects to a series of proposed behaviors(1-2). In these behaviors, other relevant aspects deal with the validity evidence of contents that describe them.
Sample, inclusion and exclusion criteria
The selection of the sample met the intended model. The logic of the process of choosing panel components was the heterogeneity of the expert. Thus, the following criteria were defined: being a nurse, having at least a degree in lato sensu graduate studies, having expertise in emergency services or any professional competence in this area, or both, acting or having acted in at least one field of the topic in question, to organize groups connected to nursing care, management, teaching, and research.
To meet these criteria, professionals specialized in Nursing - Residency in Emergency Room, undergraduate/emergency room program teachers of public universities of the city of São Paulo, members of the Study and Research Group in Administration of Health Services and Nursing Management (GEPAG), participants of the Working Group on the Competencies Project of the Regional Council of Nursing in São Paulo, Brazilian researchers in the areas of Emergency Nursing Services or in professional competence, and the nurse supporters of the Secretariat of Health Care of the Ministry of Health working in hospitals with the S.O.S. Strategic Action in Emergency Services.
To locate the group of experts, the researchers gathered information on the Internet®, websites of public universities in São Paulo, in the Regional Nursing Council (COREN), and in secretariats of institutions in which the experts had some type of connection. Some sources provided lists, containing the name and the contact information of possible participants.
Once the interrelation with the possible participants in the sample was identified, initial contact was made by e-mail, telephone or in person by the FLH moderator, explaining the subject, objectives, methodology of data collection and its timeline, the reason for choosing to participate in the panel of experts. And then the confirmation of acceptance of the indication was obtained.
In this research, validity evidence was based on the content of the Identifying Issues proposed for the competent performance of nurses in the emergency services. These questions were constructed from the definitions of each Basic and Associated Competence to describe the desired attitudes/behaviors in the daily life of this professional(3-4). The theoretical-logical model created by the authors was exemplified by the Basic Competence: Care Performance and the Associated Competence: Sense of Urgency (Figure 1).
Flowchart of the construction of an Identifying Issue to indicate attitudes/behaviors that represent the daily job of nurses who work in emergency services, São Paulo, Brazil, 2013
Each Identifying Issue was characterized as unique, since the interrelationship of the constitutive definitions was able to differentiate them in their essence. Also, originally, the Matrix proposed eight Basic and 31 Associated Competencies and 17 of them were attributed to more than one Basic Competence as they proved to be the necessary for supporting its development. These Basic Skills were named Care Performance, Teamwork, Leadership, Humanization, Decision-making, Interpersonal Relationship, Outcomes orientation, and Proactivity(3).
This study considered three main stages to evidence validity based on the content of the Identifying Questions: questionnaire development, data collection with the Delphi Technique and appropriate statistical evaluation by Likert Scale analysis, as well as an interpretative analysis of the comments and suggestions made by nursing experts.
In the development of the data collection questionnaire was considered, in addition to the integral elements considered essential for the evaluation of the content structure and of the environment, the professional expertise of the nurse who participated in the research. Thus, it was decided to configure it in three parts: A, B, and C. In part A, the numerical identification data and a brief personal, professional, and academic characterization of the expert were included; in B, the conceptual description of Basic Competence, Associated Competence, and Identifying Issue, and necessary guidelines for their use; and, in part C, the elements for the evaluation of structure and quantitative and interpretative contents. While the interpretative occupied the space reserved for comments and evaluations of the relevance, clarity, and adequacy of the content proposed in the constitutive and operational definitions, the quantitative data included the Identifying Issues described, such as the competent action of the nurse to be evaluated by the specialists through a scale.
The Likert Sociopsychological Scale was the option used to assign numerical scores to the actions that expressed the experts' points of view, giving a degree of agreement to the different attitudes/behaviors, the needs, and the motives that one wishes to discover as they occur(9-11).In this type of scale, several affirmative statements were made, and alternative answers were presented with their respective scores: I totally agree = 5; I partially agree = 4; Indifferent = 3; I partially disagree = 2; I totally disagree = 1. In the affirmatives text, the nurses' desired attitudes/behaviors in emergency situations for a high-quality service were presented, as exemplified in Chart 1 with the Basic Competence of Care Performance, their Associated ones and their Identifying Issues.
Data evaluated by nurse/expert about the presence of validity evidence in the content proposed for the Basic Competence of Care Performance, São Paulo, Brazil, 2013
|1.1 Attention||1.1.1 The professional is aware of the incoming people/equipment/environmental signs in patient care, proposing adjustments to the identified deviances.(4)|
|1.2 Control of risk||1.2.1 The professional identifies agents causing damage to patient care, analyzes the probability of their existence, and determines how to reduce them.(4)|
|1.3 Resolubility||1.3.1 The professional chooses appropriate solutions to resolve problems/situations detected in patient care, putting them into practice in the shortest time possible.(4)|
|1.4 Responsibility||1.4.1 The professional is responsible for his/her actions and for the health team in meeting patient needs, being aware of his/her social and professional role in work activities.(4)|
|1.5 Sense of readiness||1.5.1 The professional performs the necessary activities for patient care in an easy and timely manner, using the correct technique and safe execution.(4)|
|1.6 Sense of urgency||1.6.1 The professional performs timely activities, considering the health risk to patients; rates the degree of suffering, defines treatment, and minimizes care risks.(4)|
|1.7 Technical execution||1.7.1 The professional performs basic to advanced level nursing procedures while caring for patient who require clinical, surgical and trauma care, executing a safe technique with appropriate resources.(4)|
To meet needs such as ease of questionnaire completion and the possibility of shared communication between expert/researcher, the recommended computer application was Google Drive®. For that, a Gmail® account was created exclusively for the study. This technology with different applications allowed the following: to make the questionnaire available in a non-distorted format, to clarify the doubts of the experts in a group or individually, to remind the deadlines for completing the questionnaire, and to provide the researcher with access control and undue sharing, in real time. It also made it easier to send the results and resend the data according to the collection technique. In addition, an automatic rescue system was determined for the questionnaire for the expert to evaluate the Identifying Issue and to respond at the desired moment with the equipment available in his environment.
At the data collection stage, the process of verifying the content validity evidence comprised the selection of the sample, the option for the data collection technique, as well as the procedures for its application and the type of statistic for the verification of agreement of the experts.
For the data collection, the chosen technique was the prospective and consensual Delphi recommended for new and little researched subjects and/or when the quantitative data are not available or cannot be projected safely for the future(9-11).It seeks consensus among a group of experts in the subject, without having to identify them and bring them together in one place. In addition to the anonymity, it allows interaction between the researcher/expert, sending statistical data and results for new validation(9-11).It also allows the questionnaire containing the same questions, in the same order and with the same set of options for answers, to be sent by conventional mail or by electronic means repeatedly until a definitive consensus is obtained. It also considers both quantitative and qualitative data and can be used in networked computers(9-11).
The application of the Delphi Technique was scheduled to occur in three stages between October 23rd and November 27th, 2013, but it was necessary to conduct a further stage, extending the trial until December 4th of that year. The communication between researcher/expert was conducted by e-mail, following the steps recommended for its use. In the first stage, the expert received an explanatory invitation letter, the data collection schedule, the ICF to express their consent, and the research questionnaire for the evaluation in up to seven days. After this week, the researcher, in possession of the answers and the consolidation of the judgments, the respective comments and the accomplishment of the descriptive statistics on the scores obtained in the desired attitudes/behaviors of the nurses in the Likert Scale, resent the result of the first stage to the experts and resubmitted the questionnaire only containing the question in which there was no agreement on. When the researcher received the questionnaire again, the appropriate statistic was applied and the result of the second stage was sent to the experts, and again the complete questionnaire was sent to them. These same procedures were repeated until the fourth stage.
All questionnaires answered and received on the scheduled dates have been saved in Portable Document Format (PDF)®. Whenever the deadlines established by one or more participants were not met, the expert was excluded from the corresponding stage. All the information was inserted in a spreadsheet in the program Microsoft Excel 2007® and analyzed with the statistics program IBM SPSS V.19® and R3.1.2®.
In the statistical analysis of content validity evidence, consensus definitions of the experts were used to include the item evaluated by two different quantitative measures: Calculation of Scores Percentage and Calculation of Content Validity Index (CVI)(12-13). For the calculation of the scores, the acceptable consensus level was defined as the cut-off point, i.e., the Identifying Issues that obtained 90% of total agreement (score 5) and partial agreement (score 4), regardless the absence of comments or comments unrelated to the defined criteria corresponding to the statement. For the CVI Calculus, the proportion/percentage of experts that agreed with the Identifying Issues, presented individually and then in the whole questionnaire, was measured. Total agreement and partial agreement were used to calculate the indexes, with a standard greater than 78%, according to the Universal Agreement (UA)(13).
In the interpretative analysis of the answers, the comments were first grouped according to each Identifying Issue evaluated. Then there was an individual and independent evaluation conducted by the researchers on the scores given by the experts and, finally, a group discussion among the researchers to analyze the aspects related to the relevance, clarity, and adequacy of the proposed content and other elements that contributed to the exclusion or inclusion of any of them. The results were presented in absolute values and relative frequency, shown in chart and table.
The characterization data of the 25 participants showed that 22 (88%) were women and three (12%) men, aged 29 to 76 years, mean of 42.16 years with standard deviation of 11.01, and median of 40 years. Out of these, five (20%) were from generation Y, 16 (64%) from X, three (12%) from Baby Boomers, and one (4%) from the traditional generation.
Regarding the place of residence and professional activity, 18 (72%) were from the Southeast, three (12%) from the Northeast, two (8%) lived in the South and another two (8%) in the Midwest. All experts were nurses with degree completion time ranging from 7 to 50 years, mean of 18.04 with standard deviation of 10.35 and median of 16 years. Regarding stricto sensu the qualification, 18 (72%) had degrees, being 13 (60%) masters and three (12%) doctors. Four (16%) were pursuing a doctorate and two (2%) were working on masters' degree. Within the lato sensu education, the number of specializations ranged from one to six with a mean of 2.32 and a median of two per expert Regarding performance, it was verified that all experts had experience in nursing care; 19 (76%) in management; another 19 (76%) in teaching; and 17 (56%) in research. As for professional experience related to the field of work, 22 (88%) had experience in emergency services; 14 (56%) in professional competence; and 11 (44%), both in emergency services and professional competence. The mean time of emergency service experience was 8.39 years. For professional competence, it was 6.43 years.
Regarding the sample group, of the 25 experts, seven (28%) were specialized in Nursing - Residency in Emergency Room; five (20%) were undergraduate/emergency professors at public universities; another five (20%), supporters of the Secretariat of Health Care of the Ministry of Health; four (16%), members of the Study and Research Group in Administration of Health Services and Nursing Management; two (8%), participants in the work group of the Competencies Project of the Regional Nursing Council of São Paulo; and another two (8%) were Brazilian researchers in the area of Nursing in urgency/emergency service.
In the process of data collection with the Delphi Technique, 25 (100%) experts participated in the first stage; 21 (84%), both in the second and the third stages; and 18 (72%) in the fourth stage. In the first stage, among the 56 Identifying Issues proposed, only issue 7.1 did not obtain the consensus level from the 25 experts. As there was no comment on clarity, adequacy, and relevance of this issue, it was resent with the same wording in the second stage alongside the statistical results of the first one. By allowing the experts to review their positions, and to maintain or modify them as recommended by the Delphi Technique, some opinions changed, and 100% agreement was reached by the experts in question 7.1, called the Identifying Issue of the Basic Competence of the Directing Results and the respective Associated Competence: Acceptance of Challenges.
In the third stage, total and partial agreement was above 90% in all 56 Identification Questions. However, the comments on this stage proposed minor adjustments in the wording of the statements and suggested the inclusion of one more competence. Thus, the Associated Competence called Professional Growth and its respective Identifying Issue were created and conceptualized, then they were inserted in the Basic Competence Care Performance.
The Associated Competence Professional Growth was conceptualized as one's ability to be constantly updated through the reading of scientific articles, participation in research groups, conducting studies, and internal and/or external courses to the work institution, focusing both on the management of resources and on their daily work activities to safely stabilize the vital conditions of the client in emergency areas. The Identifying Issue was described as follows: he/she constantly updates and improves knowledge and practices to maintain, expand, and aggregate value to the profession, creating and using opportunities to access the means available in the profession and in society.
The creation of another Associated Competence and its Identifying Issue made the Matrix and the Profile related to the Basic Competence of the Care Performance generate the inclusion of another stage of the validation process. In the application of the questionnaire in the fourth stage, the experts agreed with the modifications and validated the Matrix and the Professional Competence Profile of the nurse in emergency services. Consequently, the statistical procedures revealed the degree of agreement of the experts on the quality parameters previously defined for the evaluation of the Identifying Issues, as shown in Table 1.
Descriptive analysis of the content validity index of the Identifying Issues, São Paulo, São Paulo, Brazil, 2013
|Full agreement||Full agreement + partial|
|1st Questionnaire||Final Questionnaire||1st Questionnaire||Final Questionnaire|
Through the suggestions and modifications made during the validation process, with the addition of one more Associated Competence and Identifying Issue, the Matrix and the Profile have a new nominal and numeral composition of items (Chart 2).
Number of elements that comprise the Matrix and the Profile of Professional Competencies before and after applying the Delphi Technique. São Paulo, São Paulo, Brazil, 2013
|1st Stage||4th Stage|
|Total Basic Skills and their Concepts||8||8|
|Associated Competencies and their Concepts||31||32|
|Associated Competencies attributed only to a Basic Competence||14||15|
|Associated Competencies attributed to more than one Basic Competence||17||17|
|Total Associated Competencies that comprise the Matrix and the Profile||56||57|
|Total Identifying Issues that comprise the Profile||31||32|
This study tried to demonstrate evidence of content validity of the Identifying Issues resulting from the joining of the Basic and Associated Competencies that were specified in the Matrix and crystallized in the Profile of Professional Competence(3-4). It was a complex process because it was a pioneering study in Brazil, focusing on the professional competence of nurses in emergency services, with the search for a consensus of experts about the proposed actions, using the Delphi Technique. However, creating professional competencies for nurses to work in Emergency Services is a recent process not only in Brazil, but also in other countries(14-22).
The experts who agreed to participate in the study were nurses from different age groups belonging to four generations(23). These experts related work experience, stricto sensu and lato sensu education, and expertise in the study subject, to the process of evaluation of the Matrix and the proposed Profile. Conducted studies present this new historical reality formed by four different generations working together(23-25). Although they have elements of their own time as potential source of conflicts, the use of strategies can overcome the dissonances and each generation can have the same basic values expressed differently, sharing experiences that can lead to a more productive and effective environment in the workplace and education(23-25). The interactivity and dynamism provided by the Delphi Technique allowed the scientific knowledge derived from intergenerational theory and practice and the reflective thinking of nurses to broaden the view of experts/researchers on competencies.
Among all the experts included in the study, there was a decisive abstention in reducing the judges throughout the data collection stages with the Delphi Technique, according to the established limits. It is considered normal that this occurs in an index with statistical variation of 30% to 50%, in the first stage, and of 20% to 30%, in the subsequent stages(10). Based on the scientific literature, the difference in the number of participants in the stages does not affect the quality and the statistical reliability of the judges because it is a sample comprised of experts (12). In addition, in this study, the minimum total number of specialists who participated in the four stages (N = 18) was three times higher than the minimum recommended by the psychometric scientific literature in studies that aim to perform content analysis with specialists(2).
Although it is more common that the data collection with the Delphi Technique is done in two stages, with an eventual third stage, in this study it was necessary to use the fourth stage because the research questionnaire offered space for the experts' comments. This enabled us to identify, in the analysis of the comments, the absence of content aimed at the permanent development of nurses. In a study, the scientific literature recognizes the importance of the Delphi Technique when analyzing information other than quantitative data, due to the increase in the scope of the results of the research allowed by this technique(9-11).
Consequently, there was an expansion of the content universe with the inclusion of an Associated Competence and an Identifying Issue, which more effectively consolidated the Matrix and Profile of Competencies previously presented. It was therefore inferred that the increase in the scope of nurses' work in emergency services was possible due to the individual judgment performed by the experts in four different moments, always knowing the degree of agreement of the other judges, being able to express opinions differently from the previous stage. This enabled a cognitive and reflexive mobilization of the various contents in the quest to contemplate the singularities and needs of the trinomial nurse, client, and emergency services.
According to some scholars, this judgment conducted with experts in the subject with different abilities allows to evaluate if the proposal included from the semantics to the criteria considered fundamental, such as clarity, relevance, comprehensiveness, among others. A study conducted in the United States on the competencies of nurses in the emergency area, different of the current proposal due to the peculiarities inherent in the country, also used the Delphi Technique to verify and obtain consensus. The three stages of the Delphi study were conducted with a national sample of emergency care nurses who rated the importance and frequency of performance for each competence and skill listed in the electronic questionnaire sent out and suggested a redesign of the content proposed by Emergency Nurses Association (ENA) Nurse Practitioner Validation Work Team(14-15).
The content of the Identifying Issues submitted to the verification process of the validity evidence in this study was always intended to represent the Brazilian reality. The concepts of Basic Competence and Associated Competence and the behaviors described in the Identifying Issues sought to define the conceptual frameworks for the operationalization of the actions of the competent work of nurses in emergency services, both to identify their existence and to guide their acquisition.
In a review study, the professional competencies must be specific to a certain function and adapted to the demands of the market(26). Also, there is a tendency to seek competencies that assist nurses in their needs, especially those related to nursing care for the critically ill, wounded patients with different processes of diseases and injuries(22,26). The definition and development of competencies, which is a new/old challenge, and the companies that most quickly adapt to this need, will certainly feel the difference(26-27).
The construction of the Matrix and Profile was mainly based on the knowledge and experience of the authors in professional competencies and in emergency services because there were no similarities with this thematic available in the Brazilian scientific literature until then. It was therefore decided to describe the attitudes/behaviors related to the care of the critically ill person, the management and the humanized care in emergency services. In this context, the minimum Basic and Associated Competencies for a private work of nurses were characterized.
This research was limited essentially by the complexity of the subject addressed; by the methodology that involves several processes with consecutive stages, which are precursors in the elaboration of an evaluation instrument of professional performance, still little used among nursing researchers in Brazil; in addition, by the lack of similar Brazilian publications in emergency areas.
Contributions to the nursing area
The verification of evidence of the validity of the content by nursing experts in the subject with the Delphi Technique encourages the proposal in this study and allows the theory to be explained, tested, improved, and integrated to the professional work of the nurse practitioner, manager, educator, and researcher.
It was possible to demonstrate evidence of content validity of the Identifying Issues arising from the combination of the Basic and Associated Competencies described in the Matrix (or Core Competence) and on the attitudes/behaviors characterizing the Professional Competence Profile of the nurse in emergency services. Thus, the results obtained allowed the theoretical-logical assessment of the mentioned Identifying Issues, suggesting their pertinence, relevance, clarity, and adequacy to represent the constructs described in the Matrix and in the Profile of Competencies, consequently validating them as well.