Research Projects
The Learning Materials Navigator trial results
Post-trial impressions
The Learning Materials Navigator trial experiment was conducted with the cooperation of 3 colleges in the second half of the 2013 academic year (n = 148). On this occasion, we conducted a survey questionnaire targeting the participants immediately after their trial in order to verify the validity and practicality of this system.
We will first report on the results of this survey questionnaire. The questionnaire consisted of categories requesting answers on a five-point scale ranging from “I strongly agree (5)” to “I completely disagree (1) ,” as well as open response question categories. First we will show the collected results of the responses to the five-point scale below.
| Questions | M | SD | |
|---|---|---|---|
| 1 | Was the indication of the listening ability diagnostic test (1) clear and accurate? | 4.3 | 0.9 |
| 2 | Was the indication of the listening ability diagnostic test (2) clear and accurate? | 4.1 | 1.0 |
| 3 | Was the vocabulary knowledge diagnostic test indication clear and accurate? | 4.4 | 0.8 |
| 4 | Was the time taken for the diagnosis appropriate? | 4.0 | 1.0 |
Usability of the Learning Materials Navigator
Practicality and usability have been brought up as standards for the evaluation of software in addition to technical aspects such as the structure of the code and the rate that faults develop (Jung, Kim & Chung 2004). They state that as long as these are not excellent, it will be difficult to make the software come into widespread use. In the case of the Learning Materials Navigator, because the assumption is that it will be used outside of school hours, it was expected that it would be less likely that students would use it voluntarily if it was not user friendly, even if it was completed into a highly reliable navigator. We judged there to be no serious problems regarding the usability because the responses to items 1 to 3 on the table were all evaluated highly with M = 4.1 or higher and because the open response section also had impressions throughout such as “It was easy to use, flowing smoothly from the survey at the beginning through to the aptitude test” and “It was very easy to understand the overall structure of the navigator, the design, the survey questions, and the sequencing.”
Jung, H., Kim, S., & Chung C. (2004). “Measuring software product quality: A survey of ISO/IEC 9126,” IEEE Software, 21(5) 10-13.
The time taken to diagnose and the number of questions in the diagnostic test
When we totaled up the responses to the item “Was the time taken for the diagnosis appropriate?” in the five-point scale survey, we found that about 10% of participants provided a negative response. We expected that dissatisfaction would increase the longer the diagnosis took, however we actually found no correlation between the two. When we inspected the responses, focusing on the data of 10 people who answered 24 questions (not 8 or 16 questions) in the listening ability diagnostic test, the response of those who had a high number of questions because they had a low ratio of correct answers (6 people) was M = 3.2, SD = 0.4 in comparison to those who solved 24 questions because they had a high ratio of correct answers (4 people) which was M = 4.8, SD = 0.5. We discovered that there was a trend towards the diagnostic time being perceived positively.
In addition, amongst the comments regarding the diagnostic time in the open response section, we found many positive impressions such as "With a 30-minute diagnostic time, the results appear quickly and show the learning materials recommended for me, so it is good that I can easily find a way of studying English that suits me" and “A good thing about it is that I can readily take it in my free time.” Furthermore, amongst students in classes where learning materials are assigned based on their score in TOEIC, a two-hour long test, we saw impressions such as “I would definitely like to have this put into practice, because I can be diagnosed in a shorter time compared to TOEIC.”
On the other hand, there were also opinions urging for improvements, such as “I would like there to be more questions. If there aren’t enough, I will be influenced by luck or my affinity towards the questions that are set.” Using these opinions as a reference, we would like to continue to develop the system, finding ways to increase its practicality while maintaining its reliability.
Attitudes towards learning with 3 step learning materials
In response to the question “Do you want to study using the learning materials that were recommended?”, the majority of students responded with approval regarding the Learning Materials Navigator and the 3 Step System Listening Courseware. There were comments in the open response section such as “It has a sense of realism when compared with commercial listening textbooks or those used in classes, and the content is about topics that are really familiar to me such as how to study at college, the laboratory, and living by myself. Because of this, I felt that if I used this software I would be able to study English more pragmatically than before” and “At only two courses a week, my English study time is insufficient. By increasing this time through e-learning such as with the learning materials recommended by this Learning Materials Navigator, I have an opportunity to develop basic capabilities. It would be great if I could practice English communication in real time in the classroom.”
On the other hand, 11% of students gave negative responses. The first reason for this is the fact that the existence of students who were continuing with 3 Step System Listening Courseware was overlooked when examining the specifications of this system. There were cases when 3 Step System Listening Courseware was recommended that the student had already studied. That group (n=15) responded to the question “Do you want to study using the learning materials that were recommended?” with M = 2.3, SD = 1.2, far less than the M = 4.1, SD =1.1 of the group who were not continuing students.
The second reason for this is the fact that information regarding the learning materials shown on the diagnosis results screen was insufficient. We saw comments in the open response section of the questionnaire such as “I can’t get interested when only the name of the learning material, its level of difficulty, its genre, and a summary are shown, so I would like a portion of the learning material to be shown.” These were seen especially in the responses of people who had no experience studying with 3 Step System Listening Courseware.
The relationship between the diagnosed level and external test scores
There were 111 students who took the TOEIC test within a few months before or after the Learning Materials Navigator trial day. The average total score was M = 417.3, SD = 93.9 and the average listening section score was M = 240.5, SD = 59.0. We examined the relationship between the level diagnosed by the Learning Materials Navigator and the TOEIC listening section score based on this data. When looking at the values on the table, the variation in the TOEIC listening section score of those who were diagnosed with the same level is not insignificant. If we look at a diagram showing the average values of the participants TOEIC listening section scores per diagnosed level as a bar graph however, we can visually understand that there is some degree of correlation between the two. Due to this fact, there is a possibility that the Learning Materials Navigator could be utilized as a method when selecting learning materials instead of conducting TOEIC tests. Through this, not only will the two hours taken for selecting teaching materials be able to be reduced by almost three quarters, but because it can be used outside teaching hours as long as a computing environment is provided, those hours can be allocated towards other learning activities.
| Level | n | TOEIC Listening Section Score | |||
|---|---|---|---|---|---|
| M | SD | Min | Max | ||
| Level 0 | 20 | 212.5 | 49.4 | 125 | 300 |
| Level 1 | 11 | 228.2 | 65.3 | 80 | 340 |
| Level 2 | 53 | 239.1 | 94.3 | 185 | 375 |
| Level 3 | 25 | 265.6 | 57.7 | 100 | 370 |
| Level 4 | 2 | 310.0 | 7.1 | 305 | 315 |
| Level 5 | 0 | n/a | n/a | n/a | n/a |
| Total | 111 | 240.5 | 59.0 | 80 | 375 |
Average Values of the Participants TOEIC Listening Section Scores per Diagnosed Level
The 3 Step System Listening Courseware was originally designed as self-study courseware and there are actually many students who continue to study on their own with it for long periods (such as from their freshman year until graduation) without any direct relation to their classes at their educational institution. We consider this navigator to have great significance, as students will be able to choose learning material that corresponds to their needs and their level of proficiency, not just their own interests, when they select learning materials.
