Editions and Supplements:

» Volume 73

Number 3


Evaluation of user embracement software with pediatric risk classification

Felipe, Gilvan FerreiraI Lima, Francisca Elisangela TeixeiraII Barbosa, Lorena PinheiroII Moreira, Thereza Maria MagalhãesII Joventino, Emanuella SilvaI Freire, Vanessa SousaI Mendonça, Larissa Bento de AraújoII
  • IUniversidade da Integração Internacional da Lusofonia Afro-Brasileira. Redenção, Ceará, Brazil.
  • IIUniversidade Federal do Ceará. Fortaleza, Ceará, Brazil.



to evaluate functional performance and technical quality of user embracement software with pediatric risk classification.


descriptive exploratory study developed based on the quality requirements set forth in ISO/IEC 25010. The evaluated characteristics were: functional adequacy, reliability, usability, performance efficiency, compatibility, safety, maintainability and portability. Eight specialists from the area of informatics and 13 from nursing participated in the evaluation. The characteristics were considered adequate when they reached more than 70% of indication as very and/or completely appropriate in the evaluations of each group of specialists.


The results obtained from the evaluation of informatics and nursing specialists were: functional adequacy (100.0%, 96.2%), reliability (82.6%, 88.5%), usability (84.9%; 98.7%), performance efficiency (93.4%; 96.2%), compatibility (85.0%, 98.1%), safety (91.7%, 100.0%), and, yet, maintainability (95.0%) and portability (87.5%) evaluated by the first ones.


the software was considered adequate regarding technical quality and functional performance.

User Embracement, Pediatrics, Triage, Computer Systems, Software


Historically, the overcrowding of emergency services has been observed, causing users and their relatives’ dissatisfaction, as well as the overload of work and the increase of tension of their professionals. In addition, the excess of patients tends to reduce the efficiency and to increase the difficulty of solving users’ health problems, leading to their aggravation, to sequelae, and even to death during the waiting period for care(1).

Considering the scenario of pediatric emergency units, it can be seen that the challenges faced by the health teams present particularities arising from the profile of the clientele served.

With the purpose of improving the quality of care offered by the Unified Health System (SUS), the Ministry of Health (MS) has instituted the National Policy for Humanization of Attention and Management of SUS, which encourages innovation in managerial practices and promotion of health. One of the tools that make up this policy is the Reception with Risk Classification (ACCR), constituting as an interference device in the work processes in health units, especially in emergency hospital services(2).

Risk classification, in turn, presents itself as an important strategy for the evaluation and stratification of risk and vulnerabilities, making it possible to identify which cases need immediate assistance and which can wait for safe care. Therefore, it can be seen that the execution of the risk classification activity is supported by the principle of equity, in which a positive differentiation is performed with the purpose of avoiding iatrogenics due to the non-fulfillment of the different needs of the services of health users(3).

It is important to point out that the practice of risk classification is performed by a higher level nursing professional and it does not consist in establishing a previous diagnosis nor in the exclusion of people without medical care. This activity is based on qualified listening and consensus, established together with the medical team, allowing the systematization of the risk classification by protocols to classify the severity and the worsening capacity of each case, as well as the patient’s degree of suffering. Such a procedure has a decisive potential to reorganize emergency services, thus having a positive impact on access to them(3).

The use of electronic systems has been pointed out in the literature as advantageous regarding the use of paper in health services, especially when it comes to emergency units, where the need for rapid decision making is constant(4).

Although there are published studies about the use of programs in health care, the search for research projects involving computerized systems, aimed at the risk classification in pediatric emergency services, points to a gap in the literature. Investigation on the perception of nurses regarding the use of a computerized system in an emergency unit revealed that the use of information technology allowed greater access and better management of patient information and administrative information, favoring support for evidence-based care(5).

The software assessed in this study consists of modules that interact with each other from the access to the main menu. In addition, it has the capacity to adapt to screens of different sizes, and can be used in different types of devices, such as computer, tablet and smartphone. Another feature is the possibility of zooming anywhere on the screen, making it easier for people with low vision. It should be noted that each user has an individual access password and it is also possible to generate different types of reports that can assist the management of the emergency unit.


To evaluate the functional performance and technical quality of a software aimed at the process of pediatric ACCR.


Ethical Aspects

The ethical aspects were respected and the research was approved on November 18, 2015 by the Research Ethics Committee of the Federal University of Ceará. The study participants signed the Free, prior and informed consent, meeting the requirements of resolution 466/12 of the National Health Council.

Study design, site and period

Descriptive exploratory study aimed at the evaluation of the technical quality and functional performance of software developed to assist nurses in the ACCR process in pediatrics, based on the ACCR protocol of Fortaleza-CE.

The evaluation process was carried out in the city of Fortaleza, by prior appointment with the specialists participating in the study, from March to August 2016.

Population or sample, inclusion and exclusion criteria

The evaluation process was carried out by two groups of specialists: eight informatics specialists and 13 nursing specialists, allowing an overview of the overall quality of the software. For the establishment of the number of specialists, NBR ISO/IEC 14598-6 was considered to recommend the use of at least eight members for each evaluator group, so that there is representation of the software users category(6).

The selection of the informatics specialists considered: to have a thesis or dissertation, to have finished a specialization course, as well as to produce scientific studies in the area of software engineering and/or systems analysis. It was also considered the working experience time in systems analysis and/or software development, which had a median of five years (IIQ = 5.0).

The selection of nursing specialists considered: to have a thesis, dissertation and/or specialization monograph, as well as scientific production aimed at the area of pediatric emergency and/or health informatics. In addition, the time of experience in pediatric emergency unit, which had a median of nine years (IIQ 18.0), was considered.

To obtain information technology specialists, a professional search was carried out in universities in the state of Ceará, associated with the analysis of the curriculum on the Lattes platform, while the search for nursing specialists took place in a health service in the city of Fortaleza, in which ACCR protocol was adopted.

Study protocol

The quality assessment process followed the quality requirements set out in ISO/IEC 25010 (System and Software engineering - System and software Quality Requirements and Evaluation (SQuaRE - System and software quality models)(7).

To evaluate the proposed quality of the Software Product Quality Model, the following characteristics and their subcharacteristics were used: 1 - functional adequacy (functional integrity, functional correction and functional aptitude); 2 - usability (recognition of adequacy, apprehensibility, protection against error, operability, aesthetics of the user interface and accessibility); 3 - reliability (maturity, fault tolerance, recoverability and availability); 4 - performance efficiency (time, resources and capacity); 5 - compatibility (coexistence and interoperability); 6 - security (confidentiality, integrity, non-repudiation, accountability and authentication); 7 - maintainability (analyzability, modifiability, modularity, reusability, testability); and 8 - portability (adaptability, ability to be installed and ability to replace)(8).

The last two characteristics were evaluated only by informatics specialists, because they are composed of technical data geared towards such area.

The judgment of the evaluation process of the analyzed characteristics was carried out based on the standard NBR ISO-IEC 14598-6(6), adapted by previous study(9), composed of the values to be achieved by the characteristics and subcharacteristics. According to the proposed judgment, the characteristics analyzed by the experts should reach a minimum value of 70.0% of indication as “very appropriate” or “completely appropriate”, in order to be considered adequate.

It should be noted that some of the subcharacteristics analyzed were not evaluated by all experts from the participating groups, since in some cases they thought they needed more information to complete the evaluation process of these subcharacteristics. Therefore, in these cases, we chose to perform the calculation based on valid information, that is, information generated from the specialists who effectively evaluated such subcharacteristics, in order to obtain greater accuracy of the calculated result, for not including opinions not expressed in specific cases.

The evaluation of the software happened after its availability to the specialists, in an individualized way, through the supply of notebook, in which the software was installed. After the specialist declared himself sufficiently oriented, the researcher handed him/her the informatics and specific printed questionnaire to record the evaluation and waited for the completion of the tests without offering any interference in the evaluation process. The tests were developed in a single meeting individually with each of the specialists, in a previously agreed place, and each evaluation took about 60 to 240 minutes per specialist.

Analysis of results and statistics

For each item component of the questionnaire used, statements are evaluated using a Likert scale of five points, distributed in: 1 - not at all appropriate, 2 - somewhat appropriate, 3 - moderately appropriate, 4 - very appropriate and 5 - completely appropriate. Data analysis was performed using descriptive statistics, using the absolute and relative frequency values, using the programs Microsoft Office Excel® and Statistical Package for Social Sciences (SPSS) ® version 20.0.


The results obtained from the evaluation of software quality characteristics by informatics specialists are presented in Table 1.

Table 1
Assessments distribution of software characteristics for Reception with Risk Classification (ACCR)in pediatrics according to informatics specialists, Fortaleza, Ceará, Brazil, 2016
CharacteristicNot at allappropriate%Somewhatappropriate%Moderatelyappropriate%Veryappropriate%CompletelyAppropriate%
Functional adequacy---15.584.5
Performance Efficiency--6.68.984.5

The analysis of Table 1 indicates that all the characteristics evaluated by informatics specialists were considered adequate, since they presented more than 70.0% of the evaluations concentrated at the “very appropriate” and “completely appropriate” levels.

It should be noted that the functional adequacy characteristic, related to the need of the software’s functionalities to meet its requirements, had all its evaluations indicating the levels “very appropriate” and “completely appropriate”, thus determining 100.0 % of adequacy of the characteristic.

The results obtained from the evaluation of the quality characteristics by the nursing specialists are shown in Table 2.

Table 2
Assessments distribution of software characteristics for Reception with Risk Classification (ACCR)in pediatrics according to informatics specialists, Fortaleza, Ceará, Brazil, 2016
CharacteristicNot at allA littleModeratelyVeryCompletely
Functional adequacy--3.9%27.4%68.8%
Performance Efficiency--3.9%37.2%59.0%

The analysis of Table 2 shows that the analyzed characteristics were considered adequate by the group of specialists in the nursing field, since more than 70.0% of the evaluations were concentrated at the “very appropriate” and “completely appropriate” levels.

It should be noted that all assessments of the safety characteristics were distributed between the “very appropriate” and “completely appropriate” levels, revealing that such characteristic was considered 100.0% adequate by the group of specialists in the nursing field.


The technology analyzed in this research project presented satisfactory results in all the characteristics assessed, both by the informatics and nursing specialists, highlighting the quality of its development.

By investigating the functional performance and technical quality of a system for electronic documentation of the nursing process, which used a methodology similar to this study, the researchers observed that all the characteristics evaluated by the informatics specialists showed low approval rates, indicating the high necessary effort to achieve satisfactory technical quality indices(8).

A study carried out based on the analysis of the application quality requirements for mobile devices, aimed at assisting the follow-up of pregnancy, identified as main points to affect the quality of the evaluated software: functional adequacy, reliability, safety, performance efficiency and usability. The functional adequacy characteristic, which is related to the need for the functionalities of the software to meet the requirements regarding the external characteristics, was affected by all the necessary requirements for the monitoring of pregnancy(10).

Differently from the results obtained in the mentioned study, this article obtained, regarding functional adequacy, high indices of adequacy indication by the nursing specialists (96.2%), reaching 100% of indication as adequate by the informatics specialists.

Other researches that also involved the evaluation of programs, using different evaluation methodologies, pointed out that the systems, in general, found difficulties to have good scores regarding usability and, nevertheless, they were considered relevant tools to accomplish the purposes for which they have been developed, indicating that this is a unanimously hard question for the evaluators(11).

Despite being hard to unanimous approve the usability characteristic indicated by the literature, possibly due to the individual variation of the evaluators’ perceptions, an adequacy indication was reached by specialists with indices > 70.0%, highlighting the possibility of improvements resulting from technological evolution process.

The usability evaluation is recognized as an important factor for success in the elaboration of efficient and acceptable technologies and also indicates the use of simple and less complex methods for its realization(12).

Several aspects can be considered when analyzing the usability of a software, and one of these factors is that the user does not need to remember large amount of information to be able to use the system. The system itself must have the necessary information and tools that allow its use, facilitating the user’s work.

Besides having good levels of functional suitability and usability, it is essential that the software be trusted enough so that the end-user does not get unassisted for malfunctioning(13).

Technology evolution has increased the presence of computerized systems, increasingly complex, in different areas and in different social segments, in order to accomplish activities faster, as well as greater effectiveness to the services rendered. The complexity related to the program development process, coupled with the need to build high-quality systems, has led to one of the central concerns for care workers and researchers - enhancing reliability(14).

For the use of computerized systems in the emergency units to help reduce the waiting time, it is necessary that such systems are efficient enough to promote faster service.

It is common to carry out the evaluation of the software performance used in a certain institution, in order to maintain its reliability regarding the users of its services. Performance tests virtually simulate substantial data usage, to evaluate the ability of the system to manage its utilization from a great load of components to the server. Thus, human and computational resources can be saved, since this process is automated(15).

The performance efficiency can be even better evaluated when applying the software directly in clinical practice, at which time its performance can be evaluated in the real scenario of its use.

The fact that the system is adequate regarding the time of execution is especially relevant in the emergency service, since it is an area with the need for fast service and quick answers by the team, mainly to try and prevent the occurrence of adverse health events of the patients while waiting for care. The increase in waiting time contributes to the increase of overcrowding in the units(16-17).

Another factor that tends to contribute to fasten nurses’ work process in the emergency unit is the possibility of interaction with computerized systems, by explaining their compatibility.

Interoperability is essential so that the programs used allow the health professional: to have a more holistic view of the patient from the possibility of analyzing their history; to be assisted regarding the computational processes of automation, so the professional need to spend less time on the operation of the computational tool and, consequently, have more contact with the patient; and to have all the computational technology developed over the years to process the data derived from patient care, notifications and reminders(18).

In a study of the technical quality evaluation and of the functional performance of the electronic system, regarding the compatibility, after the discard of the evaluations that indicated “not applicable” by the specialists who thought the information presented during the data collection was not enough for the conclusion of the evaluation of this item, the characteristic was indicated as adequate by the group of informatics specialists and by the nursing professors, reaching more than 70.0% of positive evaluations. However, in the group of care nurses, this characteristic did not reach the level indicated for approval, reflecting the need to improve the system regarding the communicability of the software with other hospital systems, and the ability to transfer data and command exchange(8).

Research project conducted in Turkey reported the experience of using electronic systems for the transfer of patient information between primary health care institutions and hospital institutions. This experience was made possible by the interoperability of the systems used, that is, the compatibility characteristic was shown to be essential for the exchange of information between the different levels of health care(19).

Considering that software responds adequately to the characteristic compatibility, another point is essential: it should be ensured that the data exchanged between the different systems is safe. This issue brings out the relevance of the analysis of the characteristic security.

Computerized systems are tools that provide greater facility for patient follow-up by the nursing team, emphasizing the security provided by the use of login and password(20).

Security-related issues have a major impact on software quality. The authentication method for access is considered crucial due to the need to provide greater security and privacy to personal health information, especially when this information is sent to systems of health institutions. Another item related to security is the possibility of backup, that is, saving the data used, enabling its use at any time(10).

Institutions that adopt the use of computerized systems in their daily lives tend to benefit from the security provided by the possibility of issuing automatic warnings, which prevent adverse events, and the aid in the decision-making of the professionals(21).

A study conducted with nurses on the use of a computerized system for the application of nursing care systematization identified that, in the perception of these professionals, the system tends to facilitate clinical reasoning and, consequently, to assist the nurses’ decision making to the most appropriate care for each patient, and may lead to the improvement of the nursing team itself(22).

Another factor that directly influences the quality of software is the effort required to make specified modifications, considering the need for regular updates and/or corrections. Such a feature is known as maintainability(23).

The maintenance of health information systems usually requires the dedication of great effort and time on the part of the responsible professionals. This is also related to the changes that frequently occur in health knowledge. Another repercussion caused by this fact is that maintainability is usually responsible for a large part of the program development costs(24).

Whether or not the maintainability of a software can be reflected in the costs and efforts required to accomplish this task. The way the software is developed, including the type of architecture and language used, can improve its maintainability(25).

Another essential feature for software success is its portability, which relates to the ability of the software to be transferred from one environment to another. Such a feature verifies whether the software can be transferred to another operating environment defined in its requirements efficient and effectively(23).

A study that carried out a quality evaluation of the nursing process electronic record, like the results obtained in this study, also had a high positive evaluation of portability. The evaluators who indicated that it was not possible to carry out the evaluation of this characteristic would possibly need access to more information, such as the source code itself, in order to make this possible(23).

The portability evaluation is also important in educational programs in the field of health. Thus, a study evaluating portability, indicative of portability, and the acceptability, in cultural and linguistic contexts, of software for the training of nursing students to care for patients in a aggravation situation, identified that it reached high satisfaction from eastern students, even though it was developed for Western students. In other words, software portability was considered satisfactory, since it obtained good results when adopted in a different environment from the one originally thought for it(26).

Study limitations

As a limitation of the study, the evaluation of software goes beyond the development period of this study, being developed permanently. Thus, even after the conclusion of the evaluation conducted in this research project, the daily use of technology may provide new improvements to the system.

Contributions to the field of nursing, health or public policy

It is expected that the use of this software in daily hospital care may include: the optimization of the ACCR process, providing nurses with agility and safety in care in emergency units and pediatric emergency; an information system appropriate to the needs of daily practice; sharing of information about the classification of risk performed in pediatric units and visibility of nurses’ work.


The results obtained from the evaluation of the software for ACCR in pediatrics reveal that it was adequate in all characteristics analyzed, that is, it was indicated as very appropriate and/or completely appropriate by more than 70.0% of the evaluations of the specialists.

The fact that the software developed in this study was considered adequate, considering both technical quality and functional performance, can be related to the participation of nurses, from the moment of their conception and development, through corrections and functional improvement, until the final evaluation process.

New studies that address the use of the software for pediatric ACCR directly in clinical practice are suggested, allowing the evaluation of its execution time and the capacity to perform improvements, such as increasing ACCR speed process and reducing operational costs, involving the idea of cost-effectiveness of software deployment in the daily life of the health service.


Franco TB, Franco CM. Acolhimento com Classificação de Risco e a micropolítica do trabalho em saúde: a experiência de Silva Jardim/Rio de Janeiro. Rev APS [Internet]. 2012 [cited 2014 Nov 08];15(2):227-33. Available from: https://periodicos.ufjf.br/index.php/aps/article/view/14975.
Lima FET, Magalhães FJ, Moura AF, Queiroz APO, Matos DPM, Bezerra MS. Professional training on of the protocol of reception with risk classification in pediatrics. Rev Rene [Internet]. 2012 [cited 2014 Sep 09];13(1):211-9. Available from: http://www.periodicos.ufc.br/rene/article/view/3802
Veras JEGLF, Joventino ES, Coutinho JFV, Lima FET, Rodrigues AP, Ximenes LB. Risk classification in pediatrics: development and validation of a guide for nurses. Rev Bras Enferm. 2015;68(5):630-9. doi: 10.1590/0034-7167.2015680521
Bishop RO, Patrick J, Besiso A. Efficiency achievements from a user-developed real-time modifiable clinical information system. Ann Emerg Med. 2015;65(2):133-42.e5. doi: 10.1016/j.annemergmed.2014.05.032
Creswick N, Callens J, Li J, Georgiou A, Isedale G, Robertson L, et al. A qualitative analysis of emergency department nurses' perceptions of the effects of an integrated clinical information system. Electro J Health Inform [Internet]. 2012 [cited 2016 Oct 20];7(1):1-10. Available from: https://researchers.mq.edu.au/en/publications/a-qualitative-analysis-of-emergency-department-nurses-perceptions
Associação Brasileira de Normas Técnicas (ABNT). NBR ISO/IEC 14598-6: engenharia de software: avaliação de produto: parte 6 - documentação de módulos de avaliação. Rio de Janeiro: ABNT; 2004
International Organization for Standardization (ISO). ISO/IEC 25010 - System and Software engineering - System and software Quality Requirements and Evaluation (SQuaRE) - system and software quality models [Internet]. Geneva: ISO; 2011 [cited 2015 June 10]. Available from: https://iso.org/obp/ui/#iso:std:iso-iec:25010:ed-1:v1:en
Oliveira NB, Peres HHC. Evaluation of the functional performance and technical quality of an Electronic Documentation System of the Nursing Process. Rev Latino-Am Enferm. 2015;23(2):242-9. doi: 10.1590/0104-1169.3562.2548
Sperandio DJ. A tecnologia computacional móvel na sistematização da assistência de enfermagem: avaliação de um software - protótipo [dissertação]. Ribeirão Preto: Universidade de São Paulo; 2008. doi: 10.11606/T.22.2008.tde-11092008-165036
Ildri A, Bachiri M, Alemán JLF. A framework for evaluating the software product quality of pregnancy monitoring mobile personal health records. J Med Syst. 2016;40(3):50. doi: 10.1007/s10916-015-0415-z
Zheng K, Vydiswaran VG, Liu Y, Wang Y, Stubbs A, Uzuner Ö, et al. Ease of adoption of clinical natural language processing software: an evaluation of five systems. J Biomed Inform. 2015;58 Suppl:S189-96. doi: 10.1016/j.jbi.2015.07.008.
Kushniruk AW, Bates DW, Bainbridge M, Househ MS, Borycki EM. National efforts to improve health information system safety in Canada, the United States of America and England. Int J Med Inform. 2013;82(5):e149-60. doi: 10.1016/j.ijmedinf.2012.12.006
Paes JL, Piazza L, Tormen L, Libardoni TC, Pasquali T, Santos GM. Intra- and inter-examiner reliability of head alignment assessment in sitting and standing positions. Fisioter Pesqui. 2017;24(1):29-39. doi: 10.1590/1809-2950/15962824012017
Zhu M, Zhang X, Pham H. A comparison analysis of environmental factors affecting software reliability. J Sys Software. 2015;109:150-60. doi: 10.1016/j.jss.2015.04.083
Kim GH, Kim YG, Chung KY. Towards virtualized and automated software performance test architecture. Multimed Tools Appl. 2015;74:8745. doi: 10.1007/s11042-013-1536-3
Lim ME, Nye T, Bowen JM, Hurley J, Goeree R, Tarride JE. Mathematical modeling: the case of emergency department waiting times. Int J Technol Assess Health Care. 2012;28(2):93-109. doi: 10.1017/S0266462312000013
Roncalli AA, Oliveira DN, Silva ICM, Brito RF, Viegas SMF. Manchester protocol and user population in the risk assessment: the nurse's view. Rev Baiana Enferm. 2017;31(2):e16949. doi: 10.18471/rbe.v31i2.16949
Moreno RA. Interoperabilidade de Sistemas de Informação em Saúde. J Health Inform [Internet]. 2016 [cited 2016 Dec 27];8(3):1-2. Available from: http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/502/268
Secginli S, Erdogan S, Monsen KA. Attitudes of health professionals towards electronic health records in primary health care settings: a questionnaire survey. Inform Health Soc Care. 2014;39(1):15-32. doi: 10.3109/17538157.2013.834342
Ribeiro JC, Ruoff AB, Baptista CLBM. Computerization of Nursing Care System: advances in care management. J Health Inform [Internet]. 2014 [cited 2016 Oct 27];6(3):75-80. Available from: http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/296/199
Salomi MJA, Maciel RF. Document management and process automation in a paperless health institution. Technol Invest. 2017;8(3):167-78. doi: 10.4236/ti.2017.83015
Lima AFC, Melo TO. Nurses' perception regarding the implementation of computer-based clinical nursing documentation. Rev Esc Enferm USP. 2012;46(1):170-7. doi: 10.1590/S0080-62342012000100024
Oliveira NB. Avaliação de qualidade do registro eletrônico do processo de enfermagem [dissertação]. São Paulo: Universidade de São Paulo; 2012. doi: 10.11606/D.7.2012.tde-08032013-141540
Atalag K, Yang HY, Warren J. Assessment of software maintainability of open EHR based health information systems - a case study in endoscopy. Electro J Health Infor [Internet]. 2012 [cited 2016 Oct 12];7(1):1-10. Available from: https://researchspace.auckland.ac.nz/handle/2292/27888
Atalag K, Yang HY, Tempero E, Warren JR. Evaluation of software maintainability with open EHR - a comparison of architectures. Int J Med Inform. 2014;83(11):849-59. doi: 10.1016/j.ijmedinf.2014.07.006
Sparkes L, Chan MM, Cooper S, Pang MT, Tiwari A. Enhancing the management of deteriorating patients with Australian on line e-simulation software: acceptability, transferability, and impact in Hong Kong. Nurs Health Sci. 2016;18(3):393-9. doi: 10.1111/nhs.12282