Validation of self-assessment instrument for the Patient Safety CenterMacedo, Renata Soares deI Bohomol, ElenaI
- IUniversidade Federal de São Paulo, Paulista Nursing School. São Paulo, São Paulo, Brazil.
To develop and validate an instrument for the self-assessment of the Patient Safety Centers in health care institutions.
Non-experimental methodological study. Divided in the following stages: literature review and construction of the preliminary instrument; content validation by nine professionals with experience in Quality Management and patient safety, who contributed to the adequacy of the items in terms of clarity and relevance; finally, 12 PSC coordinators, which conducted the reliability validation of the final instrument, using Cronbach's Alpha.
The instrument presented content validity regarding clarity and relevance, evidenced by an agreement level greater than 70%. The internal consistency presented high reliability, with a Cronbach's alpha of 0.857 for the general instrument, 0.825 for the domain Structure, and 0.809 for the domain Process.
The instrument presented evidence of content validity and reliability for self-assessment, implementation and evaluation of the PSC in health institutions.
Risk Management, Patient Safety, Health Services Research, Hospitals, Validation Studies
Since the release of the report To Err is Human: Building a Safer Health System, in 2000 by the Institute of Medicine, the search for safe care in health services has undergone continuous progress. But there is still a long way to go, and lack of safety is a public health problem. This is pointed out in a recent report by the same institute, entitled Free from Harm: Accelerating Patient Safety Improvement Fiftteen Years after To Err is Human (1).
In order to respond to evidences of unsafe care and improve health care practices, the World Health Organization (WHO) has established three global challenges. The first challenge, launched in 2005, was "Clean Care is Safer Care", which aims to reduce health care-associated infections by focusing on hand hygiene. The second, launched in 2008 and called "Safe Surgery Saves Lives", proposes a surgical safety protocol to be used in institutions, with the adoption of a checklist to avoid harm to patients during surgical procedures. The third challenge, launched in 2017, "Medication Without Harm" aims to reduce harm related to medications by encouraging institutions to make their systems safer. Brazil, as a member of the WHO, has adhered to all challenges and has been working on improving health care(2-4).
Adverse events (AE) in Brazil are still little known. Brazilian studies address the importance of notifications and demonstrate the occurrence of events such as medication errors, patient falls, pressure ulcers, removal of devices, among others(5-6). However, these findings do not properly reflect our reality. More recently, in 2016, a study estimated that 2.47 deaths occur every three minutes due to errors that may be associated with care delivery in the hospital environment, which raises patient care costs(7).
Due to the gravity and number of errors and AEs related to health care, the damages caused to the patient and the costs associated with lack of quality of services, several initiatives have emerged to promote and encourage programs to improve care processes worldwide, such as those proposed by the WHO.
In Brazil, the National Patient Safety Program (PNSP) was created in 2013 by the Ministry of Health (MS), establishing actions to promote improvement of quality in health institutions(8). As a result of the PNSP, the Resolution of the Collegiate Board of Directors (RCBD) no. 36 was published, establishing actions to promote patient safety and improve the quality of health services in Brazil. One of these actions is to create the Patient Safety Center (PSC) in all establishments, whether public, private, philanthropic, teaching or others. The objectives of the creation of the PSC are to conduct and execute the actions proposed by the RCBD and to formalize a Patient Safety Plan (PSP) in the institutions(9). For the implementation of the PSC, the National Agency of Sanitary Surveillance (Anvisa) provides a reference document for managers with the information regarding the creation of the Center(10).
Even with the resolution and guidelines, there are several difficulties in the implementation of the PSC in the national territory, since it involves changing organizational culture, adhering to PNSP protocols, providing specific training and planning financial, human and material resources(11). According to Anvisa, of a total of 6,712 health services throughout the national territory, 2,493 (37.1%) PSC were registered, demonstrating this difficulty(12).
Given the above, it is understood that a structured guideline guiding the implementation and operationalization of the PSC, with specific items to constitute its structure and organizational process, can serve as a facilitator to service managers.
To develop and validate an instrument for self-assessment of the PSC in health care institutions.
The present study began after approval by the Research Ethics Committee of the Federal University of Sao Paulo and signing of the consent form by the judges and coordinators of the PSC.
Design, setting and period
Non-Experimental methodological study.
The literature review and the construction of the preliminary instrument were carried out in the period from March 20 to April 20, 2015. The content was validated for clarity and relevance in the period from May 1st to June 30th of the same year. The reliability validation occurred from March to June 2016.
Population or sample and inclusion and exclusion criteria
The population selected to evaluate the preliminary instrument on clarity and relevance was composed of nine professionals from hospital services and public and private universities, all of them nurses. For the evaluation of the reliability of the final instrument, 27 professionals responsible for the coordination of the PSCs in the hospitals of the Rede Sentinela of the city of São Paulo were invited. Of these, 13 (48.2%) refused the invitation to participate and two (7.4%) drop out after receiving the instrument. The final sample was composed of 12 (44.4%) professionals who accepted to participate, being six nurses, four doctors, a pharmacist and a coordinator who did not inform professional category.
The inclusion criteria of judges was having an undergraduate degree and experience in the areas of patient safety, quality management and risk management. The participants were selected through evaluation of their Lattes curriculum to verify the qualifications described above. The inclusion criterion for the respondents of the PSC was being a coordinator of the health safety center of the Rede Sentinela of the city of Sao Paulo.
The construction of the instrument was carried out in three phases: (i) literature review, (ii) content validation for clarity and relevance (iii) reliability validation.
The databases searched were Pubmed and Web of Science using the terms Patient Safety; Risk Management; Validation Studies with Boolean operator AND. The inclusion criteria were articles presenting the construction and validation of instruments, published between 2012 and 2016 in English or Portuguese.
A detailed reading of the material was conducted and the possible items for the construction of the script were listed. The items were grouped in two quality domains proposed in the Donabedian model: Structure and Process(13).Structure is understood as the items that refer to human, physical, material and technological resources. Process is composed of the activities developed by health professionals, whether from a technical or administrative point of view.
The initial contact with the judges and the coordinators of the centers was via Email, and subsequent contact was via phone or in person.
The printed version of the preliminary instrument composed of 47 items, nine in the domain Structure and 38 in the domain Process, was sent to the judges via regular mail or personally delivered in order to validate the content regarding clarity and relevance of all items. A deadline of 15 days was established for the return of the instrument, which was respected. Below each item, the options "clarity" and "relevance" and the alternatives "yes" and "no" were presented, as well as open fields for suggestions or comments. Items that reached agreement regarding clarity were kept; the items evaluated with lack of clarity, but considered pertinent were modified according to the recommendations of the judges; and the items considered non-pertinent were excluded.
The last phase was the validation of the instrument for reliability. The PSC coordinators who agreed to participate received the instrument via Email in the format Excel®, with orientations for filling and return. For the evaluation of the items a four-point Likert scale was used, attributing the concepts: very important, important, somewhat important and not important. The items with blank answers were called "not informed". The period for returning the questionnaire was 15 days, but some coordinators returned it after this period. After this validation, the definitive instrument was obtained.
Statistics and analysis of results
For the analysis of the judges' validation, agreement was set as values equal to or greater than 70%, which was verified by the percentage of the replies among the respondents(14).
As for the analysis of reliability, the Software SPSS for Windows version 19.0 was used to calculate the Cronbach's Alpha. The reliability measures the relevance of the topic addressed in the questions and whether the instrument measured the intended subject. For this, a measurement scale with values ranging from 0.00 to 1.00 was used. The closer to 1.00, the less the variance between the responses and the greater the reliability. The values considered acceptable are from 0.70 to 0.90 and these were adopted for the study(15-16).
To calculate the internal consistency (Cronbach's alpha), the response alternatives for each item were rearranged by assigning a numerical score, organized as follows: "Very important" score equals 5; "Important" equals 4; "Somewhat Important" equals 3; and "Not important" equals 2. As there were blank answers, we chose to assign the score "0" (zero) to the "not informed" field. And for each item of the instrument, the level of agreement was set at 70%, referring to the sum of important and very important.
Forty-five items that reached agreement among specialists on clarity and relevance were maintained. Only two (9.1%) items were excluded because they did not reach agreement greater than 70% on relevance "Correct catheter and tubing connections" and "Regulation on quality and safety". Eight (36.4%) items obtained agreement lower than 70% on clarity but were considered pertinent (Chart 1). The judges presented suggestions for the reorganization of the General Layout of the instrument and addition of 22 items. The suggestions made by the specialists were taken by the researcher.
Distribution of the items modified in the self-assessment instrument of the Patient Safety Centers, after evaluation and suggestions of the judges
|Item 1st version||Item 2º versão|
|The PSC has a strategy to improve the processes, articulated with managers from other areas of the hospital.||There is an adverse event prevention strategy articulated with other managers.|
|The PSC, together with the institution, has a strategy to promote patient safety culture.||The upper management participates in and supports the safety culture strategies.|
|Uses risk management instruments in the investigative process.||Uses quality instruments to manage risks of adverse events.|
|It has a strategy to identify non-conformity in the processes and procedures carried out.||There is a strategy for the notification of risks and events contemplated in the Patient's Safety Plan.|
|The PSC communicates the outcomes of the notifications to the teams.||There are strategies for disseminating the results to the teams.|
|Safety in the use of technologies.||Safety in the use of equipment and material.|
|Actions to prevent adverse events.||Guidelines and actions to prevent sentinel events.|
|Risk management.||Management and risk management.|
The instrument resulted in 67 items, divided in two domains: Structure, with nine items, and Process, with 58 items. After the first phase, the instrument was sent to the PSC coordinators for their evaluation, with the level of agreement set as 70% for each item.
The initial internal consistency was 0.545 in the domain Structure, 0.809 in the domain Process, and 0856 in the general instrument. At this stage, two items in the domain Structure were excluded: "Own meeting room" and "fax", because they were considered "not important" by the coordinators, with agreement higher than 70%. Then, the Cronbach's alpha values were recalculated and the value of the Structure domain increased to 0.825 and of the general instrument to 0.857, as shown in Table 1.
Cronbach's Alpha Coefficient for the self-assessment instrument of the organizational structure of PSCs in hospitals, Sao Paulo, Brazil, 2016
|Cronbach's Alpha by domains||Cronbach's Alpha of the instrument|
With the purpose of organizing the subjects discussed in the instrument, the researcher chose to group the items into topics. The domain Structure presents only one topic: I. Human resources and materials. The domain process was divided into five topics: II. Implementation of the PSC; III. Main activities of the PSC; IV. Guidelines and actions to prevent Sentinel Events; V. Risk management strategies and actions; VI. Training of professionals, totaling six topics.
Chart 2 presents the final instrument, with agreement among the coordinators in all items. In topic I the level of agreement varied between 75% and 100%; In Topic II the agreement varied between 84% and 100%; In topic III between 83% and 100% and in topics IV, V and VI the variation was from 92% to 100%.
Self-assessment instrument of the Patient's Safety Nucleus (PSC), Sao Paulo, Brazil, 2016
|I - Human Resources and Material.||Y||N||NA|
|09||There is an adverse event prevention strategy articulated with other managers.|
|10||The upper management participates in and supports the safety culture strategies.|
|11||There are strategies and actions to avoid individual accountability.|
|12||Uses quality instruments to manage risks of adverse events.|
|13||There is a strategy for the notification of risks and events contemplated In the Patient's Safety Plan (PSP).|
|14||There are strategies for disseminating the results to the teams.|
|15||Promotes risk management actions.|
|16||Develops actions for multi-professional integration.|
|17||Identifies and evaluates existing AEs in the processes and procedures.|
|18||Elaborates, implements, disseminates and updates the PSP.|
|19||Follows the actions linked to the PSP.|
|20||Implements patient safety protocols and monitors its indicators.|
|21||Establishes an accident prevention barrier.|
|22||Develops, implements and monitors patient safety training programs.|
|23||Analyzes and evaluates the data on incidents and AEs arising from the assistance provided.|
|24||Disseminates the results of analysis of AEs and incidents to the direction and the professionals.|
|25||Notifies the AEs to the National Health Surveillance service.|
|26||Files the notifications.|
|27||Follows health alerts and other risk announcements.|
|28||Surgical procedures. (Surgery or procedure on wrong patient or body part, non-intentional retention of foreign bodies inside the patient, etc.)|
|29||Products and devices. (Death or severe injury related to the use of products and/or devices)|
|30||Patient protection. (Suicide of a patient, death or severe injury due to evasion of the patient, etc.)|
|31||Care management. (Death or severe injury associated with medication error, fall, transfusion error, etc.)|
|32||Environmental events. (Injury or death associated with electric shock, chemical or biological risk, etc.)|
|33||Radiological events. (Injury or death of the patient or contributor associated with the introduction of metallic object in the magnetic resonance area, etc.)|
|34||Potential criminal events. (Patient abduction, sexual abuse of patient or employee, non-licensed professional, etc.)|
|36||Incentive to hand hygiene.|
|38||Safety in the prescription, use and administration of medication.|
|39||Safety in the prescription, use and administration of blood components.|
|40||Encourages the patient and family to get involved in their own safety.|
|42||Prevention of pressure injuries.|
|43||Prevention of patient falls.|
|44||Safety in the use of equipment and material.|
|45||Identification, analysis, monitoring and notification of risks.|
|46||Integration of the different risk management processes developed in the service.|
|47||Implementation of the protocols established by the Ministry of Health.|
|48||Safety in the prescription, use and administration of enteral and parenteral nutrition therapies.|
|49||Registers orthoses and prostheses, when used.|
|50||Prevention and control of AEs, including care-related infections.|
|51||Actions to encourage a safe environment.|
|52||Quality and patient safety.|
|53||Basic principles in patient safety|
|54||Types of AEs related to health care.|
|55||Patient safety protocols.|
|56||Patient safety indicators.|
|57||Strategies for improving quality and safety.|
|59||Patient Safety Center.|
|60||Patient Safety Plan.|
|61||Management and Risk Management.|
|62||Notification system and AEs.|
|63||Investigation of AEs.|
|64||Analysis of root causes.|
|65||Failure modes and effects analysis.|
The present study adopted an agreement level equal to or greater than 70% for the judges' opinion, as did a national study that evaluated the computerized system of the continued education.(14) Other studies have adopted the level of 75%: one that identified the theoretical competences of the nurse who acts in home care in the area of prevention and control of infections, and another that developed and validated a nursing care protocol for patients using a ventricular assist device in a university hospital in Spain(17-18).
There are studies that used a level of agreement between judges greater than 80%, as one that adopted this percentage to develop a list of "never events" aimed at improving and providing security to the assistance provided(19). Another study validated an instrument to be used by nurses working in household care for assessing the risks of drug-related problems(20).Other authors seek an agreement of 100% among the judges, as did a Brazilian study in which the authors proposed and validated criteria for the selection of experts for validation studies in nursing(21). Therefore, the definition of agreement is variable, and its establishment is up to the researcher.
At this stage of the research, the dimensions clarity and relevance were chosen, which agree with equivalent studies. A Brazilian study translated and validated the Compliance With Standard Precautions Scale, an instrument that assesses adherence to standard precautions among nurses in hospitals(22).Another study, held in a hospital in Portugal, constructed and validated an instrument to identify nursing activities that contribute to the quality of the care.(15) The Compliance with Standard Precautions Scale was translated and validated for an Arabic version using only the dimension Relevance(23). Other authors used dimensions other than clarity and relevance, such as objectivity, complexity, grading and applicability(24). Therefore, there is no consensus on the dimensions that should be evaluated by the judges.
The presentation of the instrument in Dimensions or Domains and Items has been adopted in other studies, so that the topics have a hierarchical organization. In this study, the structure was divided in domains, topics and items. This was necessary because the domain Process presented five different evaluation characteristics, so a new organization of the items in this domain was required, and the category of topics was chosen. Studies have presented similar configurations, using domains and items in their structure(15,25-26). The number of items also shows variations. A study that translated and validated a British questionnaire that evaluates the quality of life related to health in patients with wounds, the Cardiff Wound Impact Schedule, presented more than 20 items in one of the domains(27).
For the evaluation of the items, the Likert scale is widely used as response option. It is a model that shows the relevance of each item on a numerical scale varying from a negative to a positive value or the opposite, offering greater accuracy of quantitative responses. It is widely used by scholars from different areas. This study adopted a four-point Likert scale. Other studies use scales between 4 and 6 points, and the most common is the five-point scale, that allows a neutral response(25-28).
The general evaluation of the internal consistency of the instrument revealed a Cronbach's alpha of 0.809, while the domain Structure presented a value of 0.545 and the domain Process 0.856. A value lower than 0.6 is considered poor and above 0.70 it is satisfactory(15,27-28).As mentioned, the alpha was recalculated after the elimination of two items, aiming to improve the internal consistency. Authors also demonstrate that the recalculation of the Cronbach's alpha after the exclusion of items that did not reach the pre-established level of agreement can increase the reliability of the instrument and is a common practice(29).
The self-assessment method allows a look at the strengths and weaknesses. Knowing the strategies that bring better results and the real difficulties brings an important institutional and managerial maturity, with a broader view of the processes in the daily life of the service. A study conducted in Uganda illustrates the effort to develop an instrument to measure the performance of hospitals based on the standards proposed by the Joint Commission International. It is important to note that it is a country with limited resources and no accreditation program. The study used a self-assessment approach with an instrument composed of seven domains and 485 measures(30).
Therefore, the present study proposes a validated instrument with minimum items for the evaluation of the PSC, aimed at providing an institutional self-assessment and guiding the services that are in the PSC implementation process. The final instrument is presented in Table 1 with 65 evaluation items. To facilitate its use by the services, the domain Structure included columns with the options (Y) Yes, (N) No and (NA) does not apply. For the domain Process, the recommended options are: (A) Implemented, (B) Partially implemented, (C) Planned to implement, (D) Will not be implemented, and (E) Does not apply.
The sample of the internal consistency test represents a very small number of coordinators of patient safety centers and was carried out in a single city, limiting the generalizations of the results. Additional studies will be necessary to check the consistency of the instrument in other services and cities, considering the particularities of the five regions of Brazil. Other statistical tests, such as factorial analysis, were not applied due to the sample size.
Contributions to the area of nursing, health or public policies
To meet the objectives proposed in the PNSP, the coordinators need to use tools that assist them in the implementation of the PSC. The instrument proposed provides support to the implementation and self-assessment of the Centers. This way, health institutions will be able to formalize the implementation of the Patient Safety Plan.
This study is expected to contribute to a greater understanding of how to build the center and to enrich the discussion about the implementation of actions and guidelines for promoting patient safety and improving quality in health institutions.
The instrument presented evidence of content validity for the self-assessment of the PSC and as a guideline to direct the implementation of the PSC in health services and to conduct an institutional evaluation with the minimum items necessary to structure and implement the objectives of the PNSP.