Inter-rater agreement on the protocol for care and risk classification in obstetricsAbstract
Objective To determine the degree of agreement, sensitivity and specificity of the priority of care determined by inter-rater nurses, based on the use of the protocol for care and risk classification in obstetrics, in an obstetric emergency unit.
Method Cross-sectional study with a methodological approach, carried out in a maternity school in Belo Horizonte-MG-Brazil, from September to November 2020. It was carried out in two stages: 1) Documental with an evaluation of the records of nurse classifiers in the medical records of pregnant women, parturients or puerperal women; 2) Interviews with trained and not trained nurses in risk classification. Sensitivity and specificity were analyzed and the Kappa coefficient (k) was used to assess agreement.
Results The degree of inter-rater agreement (trained and not trained nurses) was found to be moderate to strong (k= 0.47 and 0.77). There was a tendency to underestimate the red (sensitivity of 85%; specificity of 99%) and yellow priorities (sensitivity of 54%; specificity of 85%), as well as overestimate the green (sensitivity of 62%; specificity of 84%) and blue priorities (sensitivity of 89%, specificity of 98%), although there were no significant differences. Despite satisfactory agreement and specificity, sensitivity was low, due to the rates of underestimation and overestimation in risk classification.
Conclusion The protocol is reliable for determining priority of care in obstetrics, but its sensitivity was low when applied to determining priority of care by trained and not trained nurses.
Resumo
Objetivo Determinar o grau de concordância, sensibilidade e especificidade da prioridade de atendimento determinada por enfermeiros interavaliadores, a partir do uso do protocolo de acolhimento e classificação de risco em obstetrícia, em unidade de pronto atendimento obstétrico.
Métodos Estudo transversal, com abordagem metodológica, realizado em uma maternidade-escola de Belo Horizonte-MG-Brasil, no período de setembro a novembro de 2020. Realizado em duas etapas: 1) Documental com avaliação dos registros de enfermeiros classificadores nos prontuários de gestantes, parturientes ou puérperas; 2) Entrevista com enfermeiros treinados e não treinados na classificação de risco. Realizou-se análise de sensibilidade, especificidade e empregou-se o coeficiente Kappa (k) para avaliar a concordância.
Resultados Evidenciou-se que o grau de concordância Interavaliadores (enfermeiros treinados e não treinados) foi considerado moderado a forte (k= 0,47 e 0,77). Verificou-se tendência na subestimação das prioridades vermelha (sensibilidade de 85%; especificidade de 99%) e amarela (sensibilidade de 54%; especificidade de 85%), bem como superestimação na prioridade verde (sensibilidade de 62%; especificidade de 84%) e azul (sensibilidade de 89%, especificidade de 98%), porém sem diferenças significativas. Apesar da concordância e especificidade satisfatória, a sensibilidade foi baixa, devido aos índices de subestimação e superestimação na classificação de risco.
Conclusão O protocolo é confiável para determinação da prioridade de atendimento em obstetrícia, porém houve baixa sensibilidade, diante de sua aplicação na determinação da prioridade de atendimento por enfermeiros treinados e não treinados.
Acolhimento; Risco; Medição de risco; Enfermagem em emergência; Gestantes; Maternidades; Sensibilidade e especificidade
Resumen
Objetivo Determinar el nivel de concordancia, sensibilidad y especificidad de la prioridad de asistencia determinada por enfermeros interevaluadores, a partir del uso del protocolo de acogida y clasificación del riesgo en obstetricia, en una unidad de servicios de emergencias obstétricas.
Métodos Estudio transversal con enfoque metodológico, realizado en una maternidad escuela de Belo Horizonte, Minas Gerais, Brasil, de septiembre a noviembre de 2020. Fue realizado en dos etapas: 1) documental con análisis de los registros de enfermeros clasificadores en las historias clínicas de mujeres embarazadas, parturientas o puérperas; 2) entrevista con enfermeros capacitados y no capacitados en clasificación del riesgo. Se realizó análisis de sensibilidad, especificidad y se empleó el coeficiente Kappa (k) para evaluar la concordancia.
Resultados Se observó que el nivel de concordancia entre evaluadores (enfermeros capacitados y no capacitados) fue considerado de moderado a fuerte (k= 0,47 e 0,77). Se verificó una tendencia de subestimación de la prioridad roja (sensibilidad de 85 %; especificidad de 99 %) y amarilla (sensibilidad de 54 %; especificidad de 85 %), así como una sobrestimación de la prioridad verde (sensibilidad de 62 %; especificidad de 84 %) y azul (sensibilidad de 89 %, especificidad de 98 %), pero sin diferencias significativas. A pesar de que la concordancia y la especificidad fueron satisfactorias, la sensibilidad fue baja, debido a los índices de subestimación y sobrestimación en la clasificación del riesgo.
Conclusión El protocolo es confiable para determinar la prioridad de asistencia en obstetricia, pero se observó baja sensibilidad al ser aplicado por enfermeros capacitados y no capacitados para determinar la prioridad de asistencia.
Acogimiento; Riesgo; Medición de riesgo; Enfermería de urgencia; Mujeres embarazadas; Maternidades; Sensibilidad y especificidad
Introduction
Priority Care with Risk Classification (PCRC) refers to the act of care with active listening. It also determines the priority of care for patients in urgent and emergency situations.(1)
Because of these situations, health professionals working in urgent and emergency care units, especially nurses, need to have the knowledge, skills and attitude to welcome, classify risk and determine the priority of care for health users in these services.(2) In addition, a careful assessment of the patient’s clinical conditions is required, in line with the user’s complaint. This requires rapid and immediate selection of the information collected with effective and efficient intervention.(1)
In obstetrics, the C&RC (Care and Risk Classification) proposal emerged in the municipality of Belo Horizonte in 2010, with the creation of a specific protocol for dealing with demands relating to the pregnancy and puerperal process.(3) At the national level, there have been changes created by the Stork Network, with the obligation to carry out C&RC in obstetric care services.(4) Maternal and Child Care (MCC) was revoked, and the Stork Network was resumed as a strategy to reinforce the importance of C&RC and risk stratification in obstetric and neonatal care services, following the clinical and regulatory guidelines of the Ministry of Health (MS).(5,6)
In order to follow these guidelines, the Ministry of Health is developing health technologies, such as the Protocol for Care and Risk Classification in Obstetrics (C&RC-O), which aims to improve maternal and neonatal morbidity and mortality indicators, mitigate maternal and child pilgrimage, decongest urgent and emergency care units, promote standardized and systematic language of care, in order to guide professional practice in maternity wards and in the different obstetrics services in Brazil.(7,8)
The aim of this study is to certify the reproducibility of such a protocol as a valid, safe and favorable technology for guiding clinical judgment in health; for identifying the risk classification (RC) discriminator and clinical indicators in health, which are the basis for determining priority of care,(1,9) as they do in other obstetrics scales and protocols.(7,8,10-16)
Therefore, the objective was to determine the degree of inter-rater agreement, sensitivity and specificity of the priority of care determined by nurses, based on the use of the protocol for reception and risk classification in obstetrics, in an obstetric emergency unit.
Methods
The study took place in a philanthropic, non-governmental maternity hospital linked to the Unified Health System (SUS) in Belo Horizonte, Minas Gerais, Brazil. It is a health service with obstetric and neonatal beds, recognized as a national reference for the humanization of labour and birth care. The study used a cross-sectional methodological approach to analyze the reproducibility of the Ministry of Health’s C&RC-O protocol through two stages: the first was a retrospective documentary study, evaluating the records of nurse classifiers in the medical records of pregnant women, parturients and/or puerperal women; the second was a semi-structured interview with nurses trained and not trained in risk classification.(8)
Data collection took place between September and November 2020. Due to the SARS-CoV-2 (COVID-19) pandemic, the configuration of the survey was modified, making it necessary, in order to reduce exposure, to collect data retrospectively and non-participatively from the cases seen in the emergency room. The sample size was calculated based on the estimated proportion (p) of the unknown population. We considered agreement of at least 80% (p = 0.80) for the degree of inter-rater agreement, a margin of error of 10% on this estimate and a 95% confidence interval. The final sample consisted of 270 evaluations of the C&RC-O Protocol, according to table 1.
Regarding the priority level of care and its classification by color, we followed the C&RC-O Protocol(5), which recommends: Red - Priority I (Emergency - immediate care); Orange - Priority II (Highest urgency - care within 15 minutes); Yellow - Priority III (Urgent - care within 30 minutes); Green - Priority IV (Lowest urgency - care within 120 minutes) and; Blue - Priority V (non-urgent - non-priority care or referral according to agreement).
The medical records were randomly selected by drawing lots for the colors of the RC. To do this, we searched for the identification numbers of the medical records of the women seen at the C&RC-O, from November 2019 to May 2020, which generated a numerical table, making it possible to randomly select the participating document from the universe of medical records. Those that contained complete data and were properly filled in were included. The variables collected to characterize the women were: name, age, parity, gestational age, main complaint, presence of contractions and/or loss of fluid, pain, vaginal bleeding, vital data, flow chart and risk classification determined by the nurse. As for the profile of the professionals, variables such as age, length of time working in obstetrics and whether they had any training related to C&RC-O were analyzed.
The sample calculation for the participating nurses took into account the comparative nature involving two proportions: 1) proportion of inter-rater agreement (Trained Nurses Group - p1) in relation to each of the five colors (red, orange, yellow, green and blue), according to the degree of complications and risk of death determined by the C&RC-O Protocol; 2) proportion of inter-rater agreement (Not Trained Nurses Group - p2). An estimate of 90% was considered, with a significance level of 5% (α = 0.05), and power of 80% (β= 0.20). In the end, it was necessary to select ten nurses in each group (trained versus not trained) in order to identify differences of at least 50% in the agreement rate of each group. These two groups were selected using simple random sampling, which included a search for trained and not trained nurses at the institution with the coordination, including those with at least two years’ experience in care.
Trained nurses were those who had been previously trained by the Ministry of Health (MH) team, through a workshop lasting at least 20 hours. Not trained nurses were those who had never worked in the RC and had no previous training. No training was carried out during the research; those who already had experience and proven certification were sought out.
During these interviews, each nurse received 27 clinical cases with information extracted from the medical records selected in the first stage of this study, totaling 270 cases. These cases were assessed by ten trained nurses and ten not trained nurses, and answered individually, together with the researcher, in a private place, preserving confidentiality.
As for the organization and analysis of the data, the degree of agreement between the trained and not trained nurses was assessed using the Kappa coefficient (k). To interpret the k classification, the following were considered: k<0 (no agreement), k = 0 to 0.20 (weak), k = 0.21 to 0.40 (reasonable), k =0.41 to 0.60 (moderate), k =0.61 to 0.80 (strong) and k= 0.81 to 1.0 (perfect).(17,18)
In addition, sensitivity was used to measure the ability to identify whether women classified at the highest levels of urgency had an underestimated classification, and specificity was used to verify the likelihood of a patient being classified according to clinical priority as non-urgent and not requiring urgent care.(19)
Strengthening the reporting of observational studies in epidemiology (STROBE) was used as a guide for methodological rigor.(20)
The interviews with the nurses took place after the signing of the Informed Consent Form, respecting the ethical and legal aspects of Resolution 466/2012, with approval, according to opinion no. 4198387 of the Research Ethics Committee of the University and the Health Institution (Certificate of Presentation for Ethical Appraisal: 19261419.0.0000.5149).
Results
Of the 270 medical records selected, 88% were pregnant women confirmed by laboratory tests, imaging tests or clinical evaluation, 9% were puerperal women and 3% had delayed periods and had doubts about their pregnancy. In order to characterize this population, information was collected on users’ age, gestational age and type of complaint.
After evaluating the 270 medical records, it was possible to generate a profile of the users seen at the obstetrics reception and risk classification service, who were aged between 21 and 33 (70%), of whom 88% were pregnant, 49.7% were primigravidae and 72% had a gestational age of 40 weeks or more. Figure 1 shows the age of the population analyzed (n=270). Most of the patients (70%) were between 21 and 33 years old at the time of their consultation; their ages ranged from 14 to 45 years, with a mean of 27 and a median of 26 years, a standard deviation of 7 years and a coefficient of variation of 26%, indicating moderate variability between them in terms of age.
Figure 2 shows gestational age (GA) at the time of reception and risk classification. 72% of patients were 40 weeks or over, ranging from 5 to 43 weeks, with a mean of 36 and median of 39 weeks, standard deviation of 7 weeks and coefficient of variation of 19%, indicating little variability in GA.
The reasons for users seeking obstetric care are listed in Table 2, which are: contractions (62/23%), loss of fluid (42/15.6%), change in blood pressure (22/8.1%) and increased bleeding (16/6.7%).
With regard to the nurses who took part in the research, a significant difference was found in their age and length of time working, with trained nurses having a lower age range (average of 32.7 years) and shorter time (4.4 years) working, when compared to not trained nurses with an average of 38.8 years of age and 9.0 years working in clinical practice, as shown in Table 3.
Table 4 shows the results of the data analysis regarding the responses determined by the trained and not trained nurses in terms of the degree of agreement, specificity and sensitivity in relation to the priorities of care determined. The color red - priority I had strong agreement (k= 0.692) and sensitivity of 85.0%, which shows that almost all the users classified as red by the trained nurse were also classified with the same level of priority by the not trained nurse. For priority level II - orange, agreement was also considered strong (k= 0.631), and sensitivity was 74.0%, which is lower than for the red classification. However, most of the users classified as orange by the trained nurse had the same classification by the not trained nurse. For yellow, priority III, agreement between the nurses was moderate (k= 0.522), and sensitivity was the lowest of all the clinical priorities (54.0%). Specificity was 85.0%, i.e. 15.0% of the patients classified as yellow by the trained nurse were overestimated by the not trained nurse. Thus, the risk attributed by the nurse was greater than the risk presented by the user at the time of care, demonstrating an overestimation of risk.
Risk classification agreement between trained and not trained nurses, using the C&RC-O protocol
At priority level IV, green, agreement between the nurses was moderate (k= 0.485) and sensitivity was 62.0%. Specificity in this group had the worst performance of all the clinical priorities (84.0%), which indicates a greater occurrence of false positives, i.e. the risk attributed by the nurse was greater than the risk presented by the user at the time of care. As for priority level V, blue, the agreement between the nurses was strong (k=0.696), the sensitivity was 89.0%, confirming that almost all (42/47) of the users classified as “blue” by the trained nurse agreed with the classification given by the not trained nurse. Specificity was also high (98.0%).
Table 5 shows the agreement and kappa coefficient (k) for analyzing the flowcharts and risk classifications in comparisons between trained and not trained nurses, trained nurses and data from medical records, and not trained nurses and data from medical records. The agreement in the flowcharts between the data collected from the trained nurses and the data from the not trained nurses was considered perfect (k=0.821), corresponding to a gross agreement of 87.0%. With regard to risk classification, agreement between the data collected and compared between trained and not trained nurses was considered moderate (68.0%; k= 0.588). The agreement between the data collected from the medical records and the trained nurses was considered moderate (k=0.611), i.e. a good index in reliability studies. Gross agreement was 70.0%. The agreement between the data collected from the medical records and the not trained nurses was moderate (k=0.599), slightly lower when compared to the trained nurses. Gross agreement was 69.0%.
Discussion
The results showed that risk classification in the obstetric emergency room had good agreement between the classifying nurses, especially in priorities I and II, i.e. emergency and greater urgency in red and orange, respectively. However, in the classifications considered urgent (yellow) and less urgent (green), there was a greater chance of false positives, i.e. a tendency for trained or not trained nurses to overestimate.
In any case, the assessment tool allows for a reasonable determination of the priority of care, even in the absence of training, since the degree of inter-rater reliability was considered moderate to strong, represented by the Kappa coefficient which varied between 0.47 and 0.77.
The data corroborated a study carried out with children in urgent and emergency situations, also using the PCRC Protocol, but specifically for pediatrics. In this study, it was found to be reliable for RC of children and adolescents in order to prioritize care, with any nurse, whether trained or not.(1)
Another piece of data that merited attention in the study was the number of non-urgent cases that demand hospital admission and cause overcrowding: 39.1% were classified as urgent (red, orange and yellow) and 60.9% as non-urgent (green and blue). These data corroborate another study carried out in a pediatric emergency department in Fortaleza, Ceará, Brazil, in which 200 patients participated, 35.5% of whom were classified as urgent (orange, yellow and green) and 45.0% as non-urgent (blue).(1,21)
This is a similar fact in obstetrics. Other researchers have also evaluated 736 women treated in obstetric emergency rooms, and the percentage of non-urgent cases was recorded as predominant (503 visits: 70.6% of cases); it should be noted that urgent cases accounted for 29.4%.(7)
This evidence shows that there is a need to reflect on the perceived need to reduce spontaneous demand for non-urgent clinical priorities in the hospital network, and that it is important to review the conditions for more robust agreements between Primary Health Care (PHC) and the hospital network, in order to guarantee continuity, access and comprehensive care for users at all levels of health care and complexity.
Inter-rater reliability is also discussed, in cases of underestimation and overestimation of the priority of care determined by trained and not trained nurses. With regard to reliability, one study looked at the external and internal reliability of the Manchester Protocol. The results showed substantial to almost perfect inter-rater reliability (k= 0.78 and 0.86), but also data related to overestimation and underestimation.(22)
Another cross-sectional study, carried out in an adult emergency unit of a general hospital in Santa Catarina, Brazil, found substantial agreement with these findings during the application of an institutional PCRC protocol, with almost perfect agreement at all levels of classification. However, cases of underestimation were found, in which users should have been classified in higher priority levels of care.(23)
This corroborates the idea that overestimation and underestimation can lead to risk classifications of a higher or lower level of priority, unnecessarily, generating a high demand for priority cases, with longer waiting times, a factor that can aggravate the user’s clinical condition, as well as causing deaths in emergency room queues.(24)
This may be due to the length of time they have worked and their professional experience in emergency and RC services. However, a study evaluating which factors could interfere with the accuracy of nursing triage in an emergency service pointed out that nurses with more professional experience and specific training in the use of protocols in emergency services work more safely and effectively in RC services.(25)
The implications of this study for obstetric practice are that the MH protocol is considered reliable for determining the priority of care for pregnant women, parturients and puerperal women in urgent and emergency situations, showing that they can be cared for by trained or not trained nurses, regardless of their length of professional experience. It is clear, therefore, that there is a need for qualified listening, in real time, in order to better target the identification of the risks faced by this user, with greater accountability for rapid and effective responses to the problem presented.
These professional skills have been built up and refined over the days, months and years of experience, because professional experience, even if undeclared, combined with specific training in the use of C&RC protocols, can be considered the gold standard for nursing professionals to act as first aid managers, in order to avoid inappropriate decisions that could have negative impacts and/or harm users, their children, families, as well as health professionals and, consequently, the health system.
Another highlight of this study is the support for the use of reliable health technologies, such as C&RC protocols, which provide subsidies and improvements both in clinical assessments and in urgent and emergency care, especially obstetric care carried out by nurses who are trained and sensitized to promote a safe, organized environment with qualified care.
One limitation to be mentioned is the statistically significant difference in the length of time the evaluating groups have been working, whether trained or not. In other words, the group of not trained nurses had more than twice as much time working as the group of trained nurses. Furthermore, this is in line with the findings of a previous study, which also showed a significant difference in the length of time the groups had been working, but which did not assess differences in agreement in RC.(25)
It is worth mentioning the need for external validity of the study as an important limitation, since this refers to the potential for generalization of the findings. The fact that the study was carried out in a single health institution, with institutional protocols and nurses working with definitions that may be different from other care settings that use this tool. It is therefore recommended that further studies be carried out in other settings using the C&RC-O protocol, contributing to the reliability of its use by nurses in RC.
As a suggestion, there is a need to use protocols focused exclusively on the singularities of obstetrics in the C&RC-O in national and international maternity hospitals, in order to guide the care provided to women in the context of pregnancy, labor and birth. It is also suggested that this topic be integrated into the training of health professionals who work in obstetrics. Since expertise in this area can promote early identification of problems, and prompt action will benefit and reduce harm to women and their families during labor and birth.
Conclusion
For changes in maternal, fetal and neonatal morbidity and mortality indicators, the results of this study confirmed the reliability of the C&RC-O protocol for determining the clinical priority of care in obstetrics, since it showed moderate to strong agreement, despite low sensitivity when comparing the indication of obstetric risk by trained and not trained nurses. In obstetric care, reception and risk stratification need to be aligned, as this combination is of fundamental importance for rapid assessment and action when necessary, as well as being a field of emphasis for nurses, which improves access for users and promotes equity in care.
References
- 1 Magalhães FJ, Lima FE, Almeida PC, Ximenes LB, Chaves CM. Care protocols with risk classification in pediatrics: inter-observer reliability. Acta Paul Enferm. 2017;30(3):262-70.
- 2 Lacerda AS, Sauthier M, Paes GO, Teixeira ER. Embracement with risk classification: relationship of justice with the user. Rev Bras Enferm. 2019;72(6):1496-503.
- 3 Belo Horizonte. Secretaria Municipal de Saúde. Comissão Perinatal. Associação Mineira de Ginecologia e Obstetrícia. Protocolo do acolhimento com classificação de risco em obstetrícia e principais urgências obstétricas. Belo Horizonte (MG): Secretaria Municipal de Saúde; 2010.
-
4 Brasil. Ministério da Saúde. Portaria nº 1.459, de 24 de junho de 2011. Institui, no âmbito do Sistema Único de Saúde - SUS - a Rede Cegonha. Brasília: Ministério da Saúde; 2011 [citado 2023 Nov 28]. Disponível em: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1459_24_06_2011.html
» https://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt1459_24_06_2011.html -
5 Brasil. Ministério da Saúde. Manual de Acolhimento e Classificação de Risco em Obstetrícia. Brasília: Ministério da Saúde; 2014 [citado 2023 Nov 28]. Disponível em: https://bvsms.saude.gov.br/bvs/publicacoes/manual_acolhimento_classificacao_risco_obstetricia.pdf
» https://bvsms.saude.gov.br/bvs/publicacoes/manual_acolhimento_classificacao_risco_obstetricia.pdf -
6 Brasil. Ministério da Saúde. Portaria GM/MS nº 13, de 13 de janeiro de 2023. Revoga Portarias que específica e dá outras providências. Brasília (DF): Ministério da Saúde; 2023 [citado 2023 Nov 28]. Disponível em: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2023/prt0013_16_01_2023.html
» https://bvsms.saude.gov.br/bvs/saudelegis/gm/2023/prt0013_16_01_2023.html - 7 Brilhante AF, Vasconcelos CT, Bezerra RA, Lima SK, Castro RC, Fernandes AF. Implementation of protocol for reception with risk classification in an obstetric emergency unit. Rev Rene. 2016;17(4):569-75.
-
8 Brasil. Ministério da Saúde. Manual de Acolhimento e Classificação de Risco em Obstetrícia. Brasília (DF): Ministério da Saúde; 2017 [citado 2023 Nov 28]. Disponível em: https://bvsms.saude.gov.br/bvs/publicacoes/manual_acolhimento_classificacao_risco_obstetricia_2017.pdf
» https://bvsms.saude.gov.br/bvs/publicacoes/manual_acolhimento_classificacao_risco_obstetricia_2017.pdf - 9 Michels BD, Marin DF, Iser BP. Time series analysis of in-hospital maternal case fatality ratio in the postpartum period according to pregnancy risks and route of delivery in the regions of Brazil, 2010-2019. Epidemiol Serv Saúde. 2022;31(3):e2022461.
- 10 Tofani LF, Furtado LA, Andreazza R, Bigal AL, Feliciano DG, Silva GR, et al. A Rede de Atenção às Urgências e Emergências no Brasil: revisão integrativa da literatura. Saúde Soc. 2023;32(1):e220122pt.
- 11 Serafim RC, Temer MJ, Parada CM, Peres HH, Serafim CT, Jensen R. System for reception and risk classification in obstetrics: a technical quality assessment. Rev Latino-Am Enfermagem. 2020;28:e3330.
- 12 Moudi A, Iravani M, Najafian M, Zareiyan A, Forouzan A, Mirghafourvand M. Obstetric triage systems: a systematic review of measurement properties (Clinimetric). BMC Pregnancy Childbirth. 2020;20(1):275.
- 13 Moudi A, Iravani M, Najafian M, Zareiyan A, Forouzan A, Mirghafourvand M. The development and validation of an obstetric triage acuity index: a mixed-method study. J Matern Fetal Neonatal Med. 2020;35(9):1719-29.
- 14 Grosgurin O, Gayet-Ageron A, Suppan L, Simon J, Villar A, Trombert V, et al. Reliability and performance of the Swiss Emergency Triage Scale used by paramedics. Eur J Emerg Med. 2019;26(3):188-93.
- 15 Fakari FR, Simbar M, Modares SZ, Majd HA. Obstetric Triage Scales; a narrative review. Arch Acad Emerg Med. 2019;7(1):e13.
- 16 Robbins T, Shennan A, Sandall J. Modified early obstetric warning scores: a promising tool but more evidence and standardization is required. Acta Obstet Gynecol Scand. 2019;98(1):7-10.
- 17 Vanbelle S. Asymptotic variability of (multilevel) multirater kappa coefficients. Stat Methods Med Res. 2018;28(10-11):3012-26.
- 18 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
- 19 Kawamura T. Interpretação de um teste sob a visão epidemiológica: eficiência de um teste. Arq Bras Cardiol. 2002;79(4):437-41.
- 20 Cuschieri S. The STROBE guidelines. Saudi J Anaesth. 2019;13(Suppl 1):31-4.
- 21 Magalhães FJ, Lima FE, Barbosa LP, Guimarães FJ, Felipe GF, Rolim KM, et al. Risk classification of children and adolescents: priority of care in the emergency unit. Rev Bras Enferm. 2020;73(Suppl 4):e20190679.
- 22 Souza CC, Chianca TC, Cordeiro Junior W, Rausch MC, Nascimento GF. Reliability analysis of the Manchester Triage System: inter-observer and intra-observer agreement. Rev Lat Am Enfermagem. 2018;26:e3005.
- 23 Moura BR, Oliveira GN, Medeiros G, Vieira AS, Nogueira LS. Rapid triage performed by nurses: Signs and symptoms associated with identifying critically ill patients in the emergency department. Int J Nurs Pract. 2022;28(1):e13001.
- 24 Andrade-Silva FB, Takemura RL, Bellato RT, Leonhardt MC, Kojima KE, Silva JD. Validity and Reliability of the Manchester Scale used in the orthopedic emergency department. Acta Ortop Bras. 2019;27(1):50-4.
- 25 Cetin SB, Eray O, Cebeci F, Coskun M, Gozkaya M. Factors affecting the accuracy of nurse triage in tertiary care emergency departments. Turk J Emerg Med. 2020;20(4):163-7.
Edited by
-
Associate Editor:
Kelly Pereira Coca (https://orcid.org/0000-0002-3604-852X) Escola Paulista de Enfermagem, Universidade Federal de São Paulo, São Paulo, SP, Brasil
Publication Dates
-
Publication in this collection
02 Dec 2024 -
Date of issue
2025
History
-
Received
23 Dec 2022 -
Accepted
26 June 2024
