A simple “stop think” approach was developed to encourage the self-assessment of learning. A key element was the requirement for students to rate their feeling of difficulty before [FOD(pre)] and after [FOD(post)] completing each of three authentic anatomy and physiology concept map exercises. The cohort was divided into low- (group L) and high-performing (group H) groups (based on final subject marks). Both FOD(pre) (group L) and FOD(post) (groups L and H) were significantly negatively correlated with score for some maps. A comparison of FOD(pre) and FOD(post) showed that students changed their mind about difficulty in 58–70% of the completed maps. Students who changed their estimation were asked to provide explanatory comments, and an increase in difficulty was related to problems with map link generation. For students who found the maps easier, 40% of comments indicated that map generation prompted recall of information from memory. Both difficulty estimations and comments supported the contention that students were self-assessing their interaction with the concept maps. Group H was significantly older than group L, had significantly higher levels of deep strategic and deep motivational learning, and had significantly higher marks in two of three concept map exercises. Notwithstanding these differences, the results from the “stop think” approach were similar between groups, indicating that it may be appropriate for students of varying academic ability. It is suggested that “stop think” may be a useful approach to encourage student self-assessment, an important step in assisting self-regulated learning development.
- difficulty assessment
- concept map
- self-regulated learning
it is often the case that students are expected to “learn” but are not taught how to learn. For example, students surveyed about their learning strategies and problem solving were generally unable to identify having received formal instruction in these areas (27) and were often unaware or unable to identify what strategies they used and whether or not those strategies that they could identify were effective (31). Students asked to identify their learning strategies commonly report rereading material or highlighted comments rather than using other, potentially more useful, approaches such as self-testing (21, 24). From these findings, one can infer that students often use learning approaches they “picked up along the way” but are unclear what these approaches are and whether or not they work. This suggests that students may be lacking in their ability to self-assess their learning (cognitive) approaches, and thus this aspect of metacognition (17, 35) should be taken into account when designing learning activities. Cognition, metacognition, and motivation interact in a way that is referred to as self-regulation (7, 46), and in the present study, we attempted to encourage the self-assessment aspect of self-regulated learning (12, 14) using a “stop think” approach, requiring students to rate their feeling of difficulty (FOD) before [FOD(pre)] and after [FOD(post)] concept map exercises. The FOD provides a rapid, abstract, metacognitive measure of perceived difficulty (16) and can thus be regarded as a self-assessment tool. The “stop think” approach used was based on the task evaluation and reflection instrument developed by Belski and Belski (4); however, it focused on individual appraisal of task difficulty without reference to peer self-assessments (a feature of the original instrument). The concept maps were chosen as an active learning task because they provide an effective graphic representation of information and also encourage students to make associations between different facts and concepts (3, 20), thus constituting a “deep” learning situation. Deep learning is said to be characterized by intrinsic motivation, interaction with and interest in the learning material (9), and an intention to understand content (2). Since self-regulation is related to deep learning (23, 29, 44), we used a “learning approach” questionnaire to encourage students to think about their individual strategic and motivational deep learning and also to investigate links between the concept map intervention and deep learning per se.
Data for the present study were derived from a first-year, first-semester nursing cohort studying human anatomy and physiology. Students were allocated to two groups based on their academic ability for the specific subject (overall mark). This was done to allow an assessment of the effectiveness of the interventions with respect to academic ability.
The cohort consisted of first-year, first-semester nursing students taking a one-semester Anatomy and Physiology subject. Although 128 students agreed to participate in the study, only 85 students (42.5% of the total enrollment) attempted the concept maps and provided the data for analysis. There were 75 female students and 10 male students with an average age of 20.5 yr (SD: 4.25 yr).
All participating students were provided with a plain language statement and signed an informed consent form, which included the right to withdraw at any time. The project was approved by the relevant RMIT Human Ethics Committee and assessed as low risk.
The study commenced during week 6 of a 12-wk semester, and the late start allowed students to adjust (to some extent) to starting their university studies and ensured that they had covered relevant material before undertaking the concept map exercises. Figure 1 shows the timetable for the various assessments used in this study. The assessments were the same for the low-performing group (group L) and the high-performing group (group H). The tools used were the revised Biggs study process questionnaire (rSPQ), which was used to measure student learning approaches (6) in weeks 6 and 11. In weeks 7–9, students completed a concept map in class (10–15 min), preceded by FOD(pre) and followed by FOD(post). If the FOD(pre) and FOD(post) values were different, for a given map, students were asked to generate a free text response indicating why they had changed their assessment. Students then completed both a second rSPQ and a student feedback questionnaire in week 11 during class time.
Learning approach measurement.
The rSPQ generates two major scores, one score for the deep learning approach (based on 10 questions) and one score for the surface learning approach (based on 10 questions). Although the Bigg’s rSPQ generates information about both learning approaches, only the deep learning estimate was considered in the current work because the surface learning scale has not been shown to be as statistically reliable as the former scale (41). The deep learning scale was subdivided into two subscales dealing with strategy and motivation, respectively, of five questions each. Since each question has a five-point scale, each score can range between 5 and 25. The measurements of deep strategy are referred to as ds 1 (first survey) and ds 2 (second survey). The measurements of deep motive are referred to as dm 1 (first survey) and dm 2 (second survey) (see Experimental procedures and Fig. 1). Students were not provided with their scores from the rSPQ surveys.
Concept map intervention.
The concept map intervention did not form part of the summative assessment for the subject and was limited in scope. The maps were marked using a modified version of the scoring scheme proposed by Novak (34), e.g., each correct link scored 1 point and 5 points were given for each hierarchical level. The answers were compared with a template that took into account expected knowledge/understanding at that particular point in the subject, and (as specified in the study) students were not provided with their mark for the concept maps. However, the marks were used when making comparisons with other aspects of the study. As the students were in the first semester of their first year of study, the basic idea of a concept map was explained in class before the first map. Prompts were also provided for each map in the form of a precompleted section (that demonstrated links and hierarchies for one relevant system) and a list of body systems previously studied. The potential variability of concept map expertise among the students was not considered to be an issue, as our main point was to encourage them to think about the exercise, irrespective of whether they found the task difficult or not. Three simple authentic scenarios were provided (scenarios A, B, and C), and students asked to generate a concept map for each. Three maps were used to provide an opportunity for students to become familiar with the concept mapping approach. The scenarios were related to material that had previously been covered (in lectures and laboratories). One concept map was provided during week 7 and the following two concept maps during weeks 8 and 9. The first concept map exercise (map A) was based on someone drinking and spilling soup that was too hot (integrating the integumentary, digestive, respiratory, and nervous systems). The second concept map exercise (map B) was based on someone who had eaten asparagus and was upset to notice that their urine had changed color (integrating digestive, cardiovascular, urinary, and nervous systems). The final concept map exercise (map C) was based on someone picking up a heavy animal, which damaged their back and also bit them (integrating skeletal, muscular, cardiovascular, immune, integumentary, and nervous systems).
The FOD measurement is related to the task evaluation and reflection instrument for student self-assessment (TERISSA) developed by Belski and Belski (4). TERISSA guides students through two estimates of task complexity, with the first estimate undertaken just before solving the task and the second estimate immediately after completing the task. In the Belski and Belski study, task complexity was judged on a five-point scale (where 1 = very simple, 2 = simple, 3 = so-so, 4 = difficult, and 5 = very difficult), and students were asked to give reasons why their estimation was not one level less difficult than indicated. They were also asked to reflect on any discrepancy between the two estimations and to decide on any actions that would assist them when they next encountered a similar task. The TERISSA approach was used in a tutorial setting where feedback on student difficulty ratings were used as part of the discussion.
In the present study, students were initially asked to rate the perceived difficulty of the exercise [FOD)(pre)] on a scale of 1 to 10 (where 1 = easy and 10 = difficult) after reading the concept map scenario but before generating the concept map. After completing the map (10–15 min), students rated the difficulty of the exercise again] FOD(post)] on a scale of 1 to 10 (where 1 = easy and 10 = difficult). The 10-point scale, rather than the 5-point scale, was used after consideration of the advantages of the subjective mental effort questionnaire used in computer science usability studies (38). The FOD rating has been used with reference to the work of Efklides et al. (16). Students were not asked to comment on their difficulty rating (cf. TERISSA approach); however, in keeping with TERISSA, they were asked to provide a written comment if there was a difference between the FOD(pre) and FOD(post) ratings. Written comments were initially surveyed to determine response categories, and the data were then coded appropriately. The process was then refined to generate final categories, and two academics independently completed the coding as indicated in the results.
Both the current approach and TERISSA approach, in addition to encouraging student self-assessment, also generate data that are relevant to an understanding of the monitoring component of the deep aspects of self-regulated learning.
Student feedback questionnaire.
Students were asked to complete a set of 11 survey questions at the end of the study (semester week 11) (Fig. 3). The first five questions related to the concept map intervention and the final six questions were related to the rSPQ and the project in general. Students were told, before survey completion, that questions 9 and 10 referred to the rSPQ. Survey questions were scored using a four-point Likert scale (where 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree). The four-point Likert scale was used as we had very limited time to conduct surveys during class time and it was felt that this would make it easier for students to complete. However, the choice of four items is still within the optimum range for Likert scales (30). An analysis of group homogeneity for questions 1–11 revealed the following values for Cronbach’s α: group L = 0.726 and group H = 0.684. Cronbach’s α provides a measure of internal consistency (reliability) of the survey question set and the values are considered to be in the acceptable range.
Group division by academic performance.
The cohort was divided into groups L and H using the end-of-semester final subject mark. The median of the mark distribution (75%) was used to divide the data set into two halves. Students in group L achieved a score of up to 74% and students in group H achieved a score of 75% and above.
Kendall’s correlation coefficients and paired/independent t-tests were used for various elements of the data set as appropriate. All statistical analyses were carried out using SPSS version 24. All significant correlations were above the level of 0.3 considered as a minimum acceptable level.
A t-test was used for statistical analyses since the data were normally distributed, and groups had similar variances. Since only two groups were compared at a time and there were only two independent variables, it was unnecessary to use ANOVA. Kendall’s correlation was used as it was appropriate for both continuous or ordinal data and it was thus suitable for use with Likert scales (ordinal variables) as well as with other experimental data.
Characteristics of groups L and H.
Median subject scores for groups L and H were 66.0% (n = 39) and 84.5% (n = 46), respectively. Groups L and H were compared with respect to the number of participants, age, and deep learning approach, and the results are shown in Table 1. Group H was significantly older than group L and had a significantly higher level of deep strategy and deep motive both at the beginning and end of the study (Table 1).
Figure 1 shows how the assessments were carried out as well as the timetable of these assessments. A completed concept map is shown in Fig. 2. Group H scored significantly higher than group L for maps B and C (55.1% vs. 32.5%, P < 0.001, and 44.6% vs. 34.9%, P < 0.05). However, no significant between-group difference was found for map A (group H: 44% vs. group L: 36%, not significant). Group H also showed a positive correlation between the map B score and the deep strategic learning approach (ds 1: 0.34, P < 0.01).
A total of 221 concept maps (sum of maps A–C) were generated by groups L and H (group L: 100 and group H: 121). FOD(pre) and FOD(post) estimations were included in 93.0% of group L maps and 98.4% of group H maps. A comparison of FOD(post) with FOD(pre) revealed students who assessed the map as more difficult or easier than initially estimated or showed no change in their rating. The FOD comparison data are shown in Table 2 for each map and for each group of students. Approximately 58–67% of group L maps and 63–70% of group H maps showed FOD(pre)-FOD(post) rating changes.
Overall, 92 written comments were made by students who changed their FOD rating. For those students who rated the FOD as less difficult after the map intervention, 43 comments were recorded. For those students who rated the FOD as more difficult after the map intervention, 49 comments were recorded. As no differences were found between the nature of the comments from group L or group H students, the data were combined.
For those students who found the maps easier than anticipated, the majority of comments were related to “Prompting during map generation” (39.5%), e.g., “When I start to think and write the concept map leads me to elaborate more information.” “Prompting due to available information” was also well represented (14.0%), e.g., “The ones listed on the side (note: list of body systems) made it easier to reflect on pre-existing and previously learnt knowledge.” Students also appreciated the “graphic representation” offered by the maps (14.0%), e.g., “Physically seeing ideas mapped out is easier than mentally visualizing a concept map.” Finally, the “authentic situation” stimulated positive comments (9.3%), e.g., “Relating myself into the situation and comparing how I would have been affected helped me work out the systems affected.” With respect to those students who found the maps more difficult than anticipated, the comments fell into two categories: those related to “Difficulty in thinking or uncertainty about how to approach concept maps” (55.1%) and “Difficulty in remembering appropriate detail, knowing which systems to include and in constructing concept maps” (44.9%).
In addition, links were found between the map scores and FOD(pre) or FOD(post) scores. The group L map B score was negatively correlated with map B FOD(pre) (−0.33, P < 0.05), and thus higher map scores were correlated with student “premap completion” ratings of “easier” or “less difficult.” In addition, significant correlations were found between map score and FOD(post) for both groups, with the map C score negatively correlated with the map C FOD(post) (group L: −0.33, P < 0.05, and group H: −0.35, P < 0.01). Thus, in these cases, higher map scores were correlated with student “postmap completion” ratings of “easier” or “less difficult.”
Analysis of the student feedback questionnaire.
The results of the end-of-semester survey were based on group L and group H student responses (group L students: n = 31–33, range: 79.5–84.6% of the group and group H students: n = 37–39, range: 80.4–84.8% of the group). The survey questions, together with average responses for each group, are shown in Fig. 3. Responses from groups L and H were generally similar; however, group L was more positive for questions 3, 6, and 8. With respect to the concept maps, overall responses were positive regarding their utility and the feedback they provided on what was known or not known (questions 1–3). Group L students were significantly more positive compared with group H with respect to the proposition that the concept maps assisted with identification of problem areas (question 3, 3.3 vs. 3.0, P < 0.05). In addition, concept maps were also found to improve “thinking” abilities (question 4). Group L gave a significantly more positive response (compared with group H) to question 6 concerning how the project encouraged them to think about how they approached learning (2.8 vs. 2.5, P < 0.05) and also gave a positive response to question 8 regarding the development of learning strategies. Completion of the rSPQ surveys encouraged students to think about how they preferred to learn (question 9), and group H students were positive with respect to the project encouraging their deep learning approach (question 11). From a negative point of view, students did not agree that their learning approach had changed as a result of the concept map intervention (question 5).
The main aim of the present study was to encourage students to self-assess their own learning, e.g., to improve the level of metacognitive awareness of what they were doing (12, 18, 31, 40). The role of metacognition, in common with many learned procedures, is diminished as cognitive approaches become more established and “automated” (7, 10); thus, it is not surprising that students may not always be aware of their learning approaches nor how effective these learning approaches may be. The benefits of improved self-assessment are clear since students who display appropriate metacognitive approaches, e.g., establishing goals, planning, monitoring, reacting to, and reflecting on learning errors, achieve higher levels of understanding and become more autonomous learners (1, 37, 39, 40). The inclusion of the “stop think” approach at the time of map completion provided an opportunity for immediate self-assessment (4) as students were requested to stop and spend time thinking about what they were about to do or had (just) done. An important part of the “stop think” approach was the use of the FOD, which provides a generalized assessment of perceived problem difficulty. In the present study, students only had limited time to make FOD assessments [FOD(pre) and FOD(post)]; however, FOD is typically generated where the situation does not permit a full analysis of the problem (25). It would be reasonable to suggest that FOD(pre) provides a measure of student assessment of the learning task where knowledge, strategic approaches, and task characteristics are taken into account. Although students were not asked to provide text feedback on this phase of the exercise, the significant negative correlation of the group L map B score with map B FOD(pre) does lend some support for the notion that FOD(pre) was a meaningful self-assessment measure. In addition, FOD(pre) does provide a point of comparison for FOD(post) and thus permits students to determine whether or not their estimation of difficulty has changed during completion of the exercise. In contrast to FOD(pre), FOD(post) is an assessment made after task completion, anecdotally related to the immediate response of students after an exam where a potentially complex scenario is summarized as easy, difficult etc. The request for students to “stop and think” at this point is a useful approach since reflection on performance is an essential component of metacognitive understanding (12, 13), and FOD(post) was significantly negatively correlated, for both groups, with the map C score. Similarly, in a study by Moni and Moni (32), a strong positive correlation was found between the strength of favorable perception of a concept mapping task and task grade. The relationship between lower levels of perceived difficulty and higher map scores, for both FOD(pre) and FOD(post), suggests that students were attempting to assess difficulty and were not simply entering random numbers into the difficulty index.
Further evidence that the FOD estimations are a valid approach to encouraging self-assessment comes from the fact that a significant proportion of students changed their FOD rating [FOD(pre-post)] and many provided free text comments on why they changed their assessment. Although concept maps have been widely used (33), there are relatively few studies where students have been asked to give feedback on their experience with this type of exercise (8, 32, 43). For those students who changed their FOD estimation, the text responses of groups L and H were similar within each cohort. For those maps that were assessed as easier after completion, 40% of the comments were similar to the following statement: “When I start to think and write, the concept map leads me to elaborate more information.” Thus, it appeared that “prompting” was taking place where generation of concept maps triggered further relevant information retrieval from memory. It is interesting that prompting due to self-generated content was referred to much more than the assistance provided by lists of relevant body systems or notes (14% of comments). Prompting is a powerful approach to information retrieval (42), and it appears that, in the context of map completion within a limited timeframe, already available knowledge may be more important than reference material. The authentic nature of the concept maps also proved useful, with comments similar to the following statement: “Relating myself into the situation and comparing how I would have been affected helped me work out the systems affected.” For those maps assessed as more difficult after completion, the comments fell into two groups, with ~50% related to confusion in map generation and ~50% related to lack of knowledge or ability to make the links. Given that students did not receive any significant training in map construction and were not told in advance the map topics this is not surprising. The large amount of content in anatomy and physiology and lack of experience in active learning approaches may have also contributed to these comments. The main value of the free text comments was that they clearly demonstrate that many students were self-assessing their performance in the concept map exercises.
Group H (based on final subject mark) differed from group L in that they were significantly older, had significantly higher levels of perceived deep learning (strategy and motive) at both the beginning and end of the study, and had significantly higher marks in two of the concept map exercises. As higher levels of deep learning are often related to increased self-regulated learning and positive academic outcomes, it seems reasonable to suggest that this may be the case in the present study (23). Notwithstanding the differences, the data from the “stop think” exercise were very similar for the two groups, and thus we conclude that this approach may be suitable for academically weaker students as well as those with stronger outcomes.
The end-of-study student feedback survey revealed both similarities and differences between groups L and H. For both groups, the feedback supported the notion that concept maps act as deep-level active learning tasks as students agreed that the maps helped their thinking abilities, allowed them to identify deficiencies in their learning, and helped them understand how anatomy and physiology facts can relate to a real-life situation. Other studies have also found that concept maps were useful as an active learning exercise that provided immediate visual feedback on knowledge and assisted linking of concepts (8, 43); however, perceived usefulness may depend on context, as in one study (32), students were neutral about the advantages of maps as a learning tool. In the present study, group L students were significantly more positive than those of group H with respect to the usefulness of the concept maps in identifying learning deficiencies and also with respect to the project encouraging them to think about how they approach learning. They also agreed that the project had encouraged them to develop their own learning approaches. These responses may be related to their significantly lower deep motive and deep strategic learning approaches and the need to improve their learning; however, no deep learning improvement was found for this group when pre- and poststudy rSPQ values were compared. It is possible that our concept map intervention was not extensive enough to bring about such changes, and this suggestion is supported by a previous study (1) where a relationship was found between concept map completion and an increase in deep learning. The idea that academically weaker students may benefit more (in relative terms) from a concept map intervention is supported by a study (36) where students in the lowest quartile (based on prior grades and a measure of mental ability) who took part in such an intervention did better than the equivalent control group in a problem-solving exam.
Although the present study was focused on the “stop think” approach, it is interesting to note that both groups of students were very positive about the fact that rSPQ survey completion made them think about how they preferred to learn (i.e., self-assessment) even though they did not receive the results from these surveys. On the other hand, both groups were neutral about changing their learning approach as a result of the rSPQ completion.
Because metacognition is an abstraction of “real-world” actions, it is important to consider the accuracy of student reflection on their learning. For example, inaccurate metacognitive assessment may lead to situations where students become overconfident with respect to their learning ability (5, 19, 24, 26–28), and such effects must be considered in any study encouraging student metacognition.
Although the concept maps did provide an active learning situation where students were potentially able to improve existing skills or develop new skills, we did not objectively assess the range of cognitive approaches used by the students (11) nor establish whether the study had a positive impact on their cognitive skills (15, 22, 45). Indeed, the end-of-study survey results indicated a reluctance to change learning approach as a result of the concept map exercises. However, encouraging monitoring of self-learning is only the first step (15, 45), and further studies need to be conducted to demonstrate tangible improvements in self-regulated learning after such an intervention.
The present study indicates that it is possible to stimulate students to self-assess their learning activity through the application of a relatively simple approach, the first step in improving the effectiveness of their interaction with learning materials. In particular, the “stop think” approach (4, 13), where students assessed their FOD before and after the active learning exercises, using a single question, proved to be particularly informative. Although it might be argued that a numeric evaluation of difficulty by a student may not, in itself, be metacognitive in nature, the correlations found between the FOD and map scores suggests that students were “thinking” about the level of difficulty. In addition, the text responses related to changes in FOD(pre-post) clearly showed that metacognitive monitoring was taking place during the learning task. It is important to note that the “stop think” approach appeared, in general, to be successful for students of low and high ability, and it will be interesting to see to what extent this method of encouraging the self-assessment of learning may prove useful. The simplicity of the approach may permit its incorporation into a variety of different learning contexts, including those that are computer based.
This work was supported by a STeLR grant from the College of Science, Engineering and Health, RMIT University.
- Copyright © 2017 the American Physiological Society