Overall Program - During the Fall semester of 2004, K. Warner worked closely with K. Jarvis to gain skill at using the quiz capability of WebCT for use in the IC grant. This entailed creating an IC competency quiz for undergraduate students enrolled in Pharmacology. Although this was not the area of expertise of K. Warner, this allowed faculty skill building with creating quiz questions that were directed at improving computer literacy for these novice students. The quiz went very well, and although there were a large number of students enrolled in the class, the quiz questions were easily graded and the vast majority of the students performed very well. This quiz contained 12 questions, including a variety of question formats, including short answer and multiple choice. One particular question posed to the students required them to look very broadly at the internet sites they were comparing. Essentially they were asked to compare the pricing of a particular drug from two internet pharmacy sources; one in the U.S. and one in Canada. The students were told only to determine what could account for the price difference between the sites, and most were able to come to the correct conclusion that it was due to price controls in Canada. A few of the students failed to look at the source of the internet site and came up with a wide variety of inaccurate conclusions based on issues of type and number of pills ordered. What was important about this particular question is that it required critical appraisal of the web sites reviewed, not just obtaining information and concluding its voracity. Thankfully, most of the students were able to achieve that level of critical appraisal. What was clear from this beginning experience was that the IC questions need to be very explicit in how to access internet based information and this detail is very time consuming to create. It is often more detail than the more skilled students need, but vital for those with limited computer capabilities.

Following this initial training, discussion and review of the quiz efforts was undertaken. The second phase of the program was to implement the same type of IC quiz into the upper division Community Health Nursing course (N144). Although the preliminary work utilized one quiz devoted solely to IC issues, it was determined that it would be more effective for these upper division students to incorporate IC quiz questions with the existing weekly quizzes taken by these CHN students. This method allowed increasing the level of difficulty each week while at the same time, providing quiz questions that were pertinent to the topical area of study for that time frame. Doing so, the students would be gaining expertise in the areas of current study which would provide further application and relevance of the quiz questions. There were 10 quizzes utilized in this course, with 1-2 IC competency questions included with the standard quiz format. This method also allowed for student skill building with the computer which was particularly relevant for the RN/BSN students. These particular students are often returning to school after years away from the academic environment, and many lack even basic computer skills. The quiz questions used have all been provided in the attachments. A grid was utilized for planning purposes that helped guide the type of IC questions to provide each week as well as the level of competency required. The beginning questions covered simple tasks such as accessing a particular internet site to the use of 'cut and paste' features. By the end of the quizzes, the students were being asked to compare and contrast information found on the internet as well as reviewing data bases and making critical assessments from that data.

Two additional requirements of the IC were to present the findings of this effort as well as write and article suitable for publishing in peer reviewed journal such as Nurse Educator. The article is being currently developed and will be ready for review by the other committee members no later than 1 Sept. 2005. The Association of Community Health Nursing Educators accepted the application of K. Warner for presentation of a poster at the June 3-4 conference. The abstract of the poster presentation was published in the conference materials and a copy is available for review. The poster was very well received by other CHN educators, and many requested copies of the quiz questions used. Many voiced similar concerns to ours at CSU Sacramento as to the overall lack of computer skills for returning students and the lack of critical appraisal of internet sites by the more skilled students. They also voiced concerns over their own skill level with the internet as well as the time commitment to utilize the internet as a learning tool.

Results:

For the Spring Semester 77 students were enrolled in N144 Community Health Nursing. Of these, approximately 52 were pre-licensure and 25 were returning RN students. As mentioned previously, there were 10 weekly on-line quizzes with 1-2 internet questions included each week for a total of 12 questions. The questions used have been provided as an attachment. Since this was the first attempt at using this type of quiz with these students the students were allowed multiple attempts (2 maximum) for each of the first 4 quizzes with the highest score recorded. This allowed them to gain skill and increase confidence without undue pressure. The next 3 quizzes allowed for the average of two attempts if they so desired. The last 3 quizzes were limited to one attempt only, as it was felt that at this point they should have gained the skill and confidence to complete the quizzes appropriately. Overall the students performance on the IC portion of the quizzes was very good, averaging in the 90th percentile. A few of the questions were problematic and these particular questions will need some refinement if they are used again. One question in particular was at issue only because the web site that they were directed to was unavailable for approximately 4 days; interestingly this was a Centers for Disease Control and Prevention site. What was particularly evident from this experience with the unavailable website, is that care must be taken to assure that sound sites are selected, and even for those that are not prone to issues, alternatives should be available.

Lessons Learned:

Overall the experience from the standpoint of the faculty was very positive. The creation and monitoring of the quizzes was very time consuming, however a number of changes could have greatly improved the process. For instance, while it may be helpful to allow multiple attempts for students at least initially, once they gain general proficiency, they should be allowed only one attempt. It should also be noted, that the use of averaged scores for multiple attempts should be avoided. In this instance there were 77 students taking quizzes which means that if there are any questions that can't be automatically graded then you have doubled the number of quizzes that must be accessed. In addition, if there are any problems with questions, there again you are required to correct approximately 154 quizzes in order to obtain an accurate average score. The use of the highest score is less problematic, however, the same issues exist. It would be far better to allow one extra quiz for additional points to provide the buffer that for the students. Many of the IC questions were designed to be self graded by the computer, however this did not work in all cases. It was particularly problematic for the short answer questions, where misspelling often created problems in grading and necessitated correction by the faculty. Additionally, there were a number of short essay questions that could not be graded by the WebCT program and with this large a class that was very labor intensive.

Recommendations:

The IC experience proved very beneficial to both the faculty and students. Many of the RN/BSN students commented to me that they felt much more confident in their computer and internet skills after taking the quizzes. While the more skilled students performed very well, they do not necessarily need the level of detail provided in the questions. It might be more effective to have a guide sheet with this level of detail provided with each quiz for use by those with limited skills and could be avoided by those with adequate computer literacy skills. The quiz questions could then cover the topic and skill level without all the lengthy detail that was provided during this attempt. Another way would be to use different font or colors to highlight the directions from the actual question, however this capacity is not available with WebCT at this point. Another recommendation would be to develop an online self assessment quiz of basic internet skills for use by the students prior to taking the content driven quizzes. This way, students can gain the basic computer skills needed prior to taking the quizzes; which would reduce the length of the quiz questions as well. Finally, with large classes, it is very important to reduce the actually 'hands-on' aspect of the grading of the quizzes. Although short answer and multiple choice are not the best predictors of critical thinking, the use of essay type answers is very time consuming to grade with large classes.