Efficacy of logbook as a clinical assessment: Using DOPS evaluation method

;


Introduction
Acquiring clinical education enhances students' competence and ability regarding knowledge, clinical skills, and professional attitude (1). According to the critical role of clinical education in nursing and since the main tasks of nursing are performed in a clinical setting, appropriate methods should be used to assess the quality of clinical education (2). Clinical skills assessment accounts for more than half of the total evaluation of medical students, including nursing, and it is one of the most critical and challenging tasks for health system professors. This kind of assessment is performed to ensure that the student has acquired clinical competence in encountering the patient and their command of the necessary skills for protecting the patient and improving the health level of society (3). Evaluation is an essential aspect of education activities. Using novel methods of clinical evaluation is an important policy of educational institutions. Several methods are used for clinical evaluation; Logbooks are among the most effective, novel techniques of clinical activity evaluation (4,5). Students can record their learning and patient care experiences in the log book and obtain the critical educational goals that must be achieved during a specific period (6). Logbook is an assessment method that can control students' academic content and experiences (7). Due to the need for more transparency of nursing students' responsibilities in the first days of clinical practices, educational goals, and student assessment methods should be clearly explained to students. So, using logbooks can be helpful in this regard (1). The use of logbooks has faced contradictory effects. Several studies stated that using the logbook method helps to achieve educational goals (8)(9)(10)(11), But some studies indicated that the logbook might not fulfill its essential educational and evaluation goals (6,7,12). Although using a logbook increases clinical skills and even the satisfaction of students (8,9,13), it does not enable the provision of direct feedback immediately after implementing each logbook. Providing on-time feedback to the learner is a fundamental part of the clinical education process; it is necessary to promote learning and achieve standards.
The assessment as direct observation of procedural skills (DOPS) -on the conventional methods of practical skills assessment -can also be used (8). The DOPS assessment method is the direct observation of a trainee while performing a practical procedure on an actual patient in an authentic environment. In Jasemi and colleagues' study, which used the DOPS test to assess nursing procedures, a significant increase was observed in skills performed in the intervention group (14). Based on the conducted studies, the DOPS method has suitable validity and reliability; hence, it can be used as a formative assessment and an accumulative tool. Given that providing feedback during the logbook implementation is one of its main parts, which provides an acceptable approach to improve learning, it has an essential role in clinical education. In a study comparing the effect of DOPS and traditional assessment methods on the clinical skills of nursing students, the results showed that using the DOPS method improved students' scores in performing clinical methods (15,16). Therefore, considering the critical role of clinical education and evaluation in nursing and according to different reports about the effectiveness of the log book, more research is needed in this field. This study aims to investigate the effect of the logbook method on nursing students' learning using the DOPS assessment method.

Materials and Methods
This study was conducted using an experimental method from February to October 2019. The research population consisted of all students in the second semester who passed the fundamental nursing internship during two consecutive semesters, including 80 nursing students randomly divided into intervention (n = 40) and control (n = 40) groups. In each semester, 20 students were assigned to each group. Inclusion criteria were: passing the theoretical course of fundamentals of nursing, needing to become more familiar with the logbook and DOPS test in the past, and being willing to participate in the study. Exclusion criteria were voluntary student withdrawal to continue participating in the study. If, for any reason, students did not want to participate in the control or intervention group, they could be transferred to another group, and this would not harm their final score. Participants were explained detailed information about the study's objectives, assured about the data confidentiality, and informed about their right to participate in or withdraw from the study at any time. Informed consent was obtained from all participants. The samples were selected as a census from all secondsemester students willing to participate in the study then using simple random sampling, they were randomly divided into two control and intervention groups. Each student was given a code; the numbers were placed in the lottery container. Among them, the numbers were selected, recorded, and returned to the container. One week before the beginning of the internship, the logbook was explained to the experimental group in a briefing 2-hour session and explanations on the logbook and how to use it.
The duration of the internship was ten days. During the internship, the logbook of internship fundamentals of nursing was used for the experimental group, and the control group finished the internship without being informed of the logbook. At the end of the internship, the students of both groups were assessed and compared by the DOPS test in terms of five procedures in the field of fundamental nursing internship. Based on the educational and behavioral goals in the internship of fundamental and topic of nursing courses, five skills were selected for the assessment with the DOPS technique. These skills included changing the dressing, controlling blood pressure, Intravenous catheterization, intramuscular injection, and inserting a Foley catheter.
Regarding ethical principles, the logbook was awarded to the control group students after the research. DOPS test was used to compare the degree of clinical competence and achievement of educational goals for control and intervention groups. DOPS test is a method designed to evaluate practical skills and give feedback. This method requires direct observation of students during a procedure and simultaneous evaluation in writing. This method can evaluate students' practical skills objectively and structure. After the end of the educational course, the DOPS test was used to compare the degree of clinical competence and achievement of educational goals for both control and intervention groups. Each DOPS checklist assesses one practical skill. Each test lasts about 20 minutes which is 15 minutes for the observer to perform the observation and 5 minutes for providing feedback. The faculty member evaluates the student based on the points evaluated in the DOPS test while observing the procedure. The DOPS checklist includes questions that form the basis of skill in performing that procedure.
After the student performs the procedure, the faculty member records the results of the observations by marking the appropriate options in the checklist. The checklist questions included demonstrated understanding of indications and relevant anatomy, obtaining informed consent, aseptic technique, technical ability, the overall ability to perform the procedure, communication skills, seeking help where appropriate, consideration of patient/ professionalism, post-procedure management, demonstrating appropriate preparation pre-procedure and appropriate analgesia or safe sedation. Each question was scaled as unacceptable (score = 0), below expectation (score = 1-3), borderline expectation (score = 4-6), meet expectation (score = 7-9), and above expectation(score = 10) ( Table 1). We used a characteristic demographic questionnaire and DOPS checklist to collect the study data. The items of the demographic characteristics questionnaire were age, gender, marital status, and the average of the previous semester. The data were analyzed using SPSS version 21 and descriptive statistics (mean, standard deviation) and analytic statistics (independent t test). An independent t test was used to compare the final test scores in both control and intervention groups, and the significance level was set at P < 0.05.

Results
The mean age of the students in this study was 20.9 ± 1.8. Female (45%) and 22(55%) male. The results of this study showed that the mean score of the patient's blood pressure assessment, IV catheterization, change dressing, and intramuscular injection skills in the intervention group was higher than the control group (Table 2). Table 2 shows the score of the DOPS test in the control and experimental groups for each procedure. However, the mean score of the inserting Foley catheter implantation procedure in the intervention group was 6.50 ± 0.79; in the control, the group was 6.81 ± 0.47; the control group's mean score was 0.31 points higher than the mean score of the intervention group. Table 3 shows in general, the scores of all clinical skills except Foley catheter implantation in the intervention group (7.66 ± 1.07) were higher than the control group (6.74 ± 0.86), therefore the average score of the intervention group was 0.92 points higher than the average score of the control group. Comparison of the mean final scores of the intervention and control groups with the independent t test showed a significant difference in IV catheterizations and change dressing skills, as shown in detail in Table 4. The result of comparing the mean scores of the control and intervention groups using the independent t test shows, there is a statistically significant difference in general between the mean scores of the two groups (P value < 0.002) ( Table 5).

Discussion
In the present study, we investigated the effectiveness of    the logbook on learning in the field of clinical practice among nursing students using the DOPS evaluation method. The results of the study showed that there is a statistically significant difference between the mean scores of the DOPS test in the control and intervention groups (P value < 0.0001), meaning that the use of a logbook in internships has improved the level of clinical skills of students compared to traditional methods (not using logbooks). The results of several studies confirm the impact of logbooks on students' learning (8,9,12,(17)(18)(19)(20).
In the study of Schick et al on final-year medical students, they investigated implementing an activity recording system with a logbook in several educational centers. For evaluation purposes, telephone interviews were conducted. Following qualitative content analysis, the results showed that implementing a logbook on professional activities in the final year of undergraduate medical education in Germany-a multicentric pilot study fosters competency-based learning opportunities (9). In the present study, the DOPS score of all procedures except the inserting Foley catheter in the intervention group was higher than the control group. The non-increasing DOPS score of the inserting Foley catheter procedure may be due to the difficulty and complexity of this procedure compared to other procedures and probably because of the non-familiarity of the students with the procedure and not repeating it. Table 3 shows that the student's assessment scores increased in the intervention group (except Foley catheter). However, the comparison of the mean DOPS score of the two intervention and control groups was significant only in the changing dressing (P value = 0.002) and IV catheterization (P value = 0.002) ( Table 4). The higher scores in the intervention group indicate the effect of the log book on learning students. One reason for the non-significance of the mean score may be due to students' lack of acceptance of the logbook. Najafi et al state that faculties' views about log books were more positive than students but the logbook meets the student's learning needs (7). In the Valizadeh et al study, the logbook quality was examined from the point of view of dental students. The student's opinion about the log book was moderate (13). Another reason, perhaps, is that the students were passing the first internship, which caused them to experience more stress; on the other hand, according to the DOPS test that is conducted directly, it can be stressful for students. This issue is also mentioned in Aghaeimaybodi and Marvasti's review study, which expresses that clinical evaluation evokes a mental and physical response in students so that they worry about the evaluation process and experience symptoms, including forgetfulness (21). In the assessment of students in the clinical field, the specific criteria and desired goals should be explained to the students to be guided in the right direction and to acquire the necessary skills (22); Farahmand and Asl Soleymani mentioned that the use of a log book requires more familiarity of the students with this tool and more monitoring of how to complete it (23).
In a mixed exploratory study that was conducted to plan a log book for nursing students, the use of a log book causes deep learning, but it showed some defects in the content and the way of its implementation (12), also a review study on the evaluation of students' clinical competencies using the log book indicated using the logbook for clinical assessment is associated with problems and presence of some defects, requires careful planning and preparation in its implementation (21), so that Several studies suggest that necessity of revise log book content continuously (6,7,12,13,21,24). In Asadilari and colleagues' study, combining the log book with other evaluation methods is recommended to achieve educational goals and better results (6). also, the results of studies confirm the effectiveness of the DOPS method in evaluating students (25,26), so in our study, we have used the log book with the DOPS evaluation method. In the study of Mirhosseini et al, which was conducted on 17 anesthesiology students, using the Logbook-DOPS integrated method to evaluate students' internships improved their clinical skills and deeper their understanding of concepts (8). So, it seems that some factors can affect logbook effectiveness, including lack of serious attention of faculty and students, faculty not having time and subsequent lack of adequate supervision, unsuitable justification of faculty and students, and only carrying out a general report of the activities (24). The present study generally showed that the logbook is effective in students' learning, but achieving all educational goals was impossible. Training should emphasize identifying the strengths and weaknesses of students in the implementation of skills and, by providing appropriate feedback, help students to overcome their weaknesses. Considering that the logbook is one of the common evaluation methods in medical sciences, paying attention to its content based on the educational goals, serious and sufficient attention, and supervision of the faculty in its effectiveness can be useful.

Conclusion
Since one of the universities' most important tasks is ensuring sufficient skills and independent performance of their students at the end of their studies, effective teaching by providing feedback is currently emphasized when designing educational curriculum. In addition, educating capable students is one of the tasks of medical schools, which is also emphasized in the country's comprehensive health plan. Educational goals should be achieved with more emphasis on the educational content of the logbook. A limitation of this study was the presence of two different instructors, which can affect both the students' training and their evaluation. The other can be related to the student's educational status, learning, and the amount of practice of learned procedures.