Embedded Assessment System for Online Education

M. Kandan

NIIT Centre for Research in Cognitive Systems

Abstract

The present paper proposes an assessment system that allows assessment to be embedded within the instruction, and looks at it as a means of instruction rather than as a mere method to judge student learning. The system can be built into any net-based distance education delivery and can make the entire assessment process automated. The proposed system advocates the use of a combination of assessment techniques such as portfolios, performance assessments, objective test, etc. with standardized scoring rubrics. Current issues associated with assessment in general and assessment in distance education in particular have been discussed. Major issues such as impersonation and cheating, subjectivity in assessment methods, etc. are discussed and alternative strategies have been proposed.  This paper proposes a system expected to check cheating in examinations, without physical monitoring, and reduce the subjectivity in assessment to a great extent. Nevertheless, the major benefit of the proposed system will be learning value added to the learners. 

Key Words: Academic dishonesty, assessment, authorship attribution, biometrics, cusum, distance education, evaluation, examination, impersonation, online education, plagiarism, peer assessment and stylometry.

Introduction

Assessment is an integral part of any education system, and considerable amount of energy is devoted to this activity. The two major reasons for assessing students are to give them feedback about their progress towards the development of their knowledge, understanding and skills, as well as determine their performance level with respect knowledge, understanding, and skills. These are also called formative and summative assessments respectively.

Often, the terms assessment, testing, and evaluation are used interchangeably to mean the same. However, they may have certain specific meanings depending on the context in which they are used. Since there have been enough discussion on these aspects, I am not going to address these issues in the present paper. In the present paper, by the term assessment, I mean the process of collecting data for the purpose of making judgments regarding the students learning.  Further, the intention of the present paper is not to bring out any new testing techniques. Instead, I am exploring various innovative uses of the existing testing techniques in conjunction with other technologies to overcome certain specific problems. These problems are associated with assessment in distance education and more specifically, online distance education system

Over the years, a range of techniques have emerged for assessing the skills and abilities of learners.  Each technique has its own advantages and disadvantages. The use a particular technique or a combination of these techniques is governed by a number of important factors such as the purpose of assessment, the time and resources available, and the profile of the learners. Nevertheless, there have been a lot of new developments with regards to delivery of education and instruction. Hence, certain traditional methods of assessment became redundant and there is need to re-look student assessment techniques in the light of new development in the instructional delivery methods.   Therefore, the focus of the present paper will essentially be on educational assessment in the context of online education.

Emergence of Online Education

Over a period of time, education and training has gone through a number of radical changes in terms of design and delivery methods over a period of time. There have been a number of transitions from the Gurukul days to the present day education. It is technology that has played a crucial role in providing multiple delivery options.  It was during the 1970s, many colleges and universities sought cost-effective and pedagogically sound ways to accommodate larger student enrolments, and provide better access to higher education to people in need. Distance education (DE) was considered as an alternative and was ultimately adopted because it seemed to present tremendous advantages to students, the institution, and the faculty.

With the birth of the Internet as a global mass medium, distance education is becoming highly technology driven and re-christened as online education. In fact, with the powerful features of the Internet and intelligent instructional design, drawbacks in the conventional form of DE are being addressed to a great extent. Rather, they are being converted into advantages over formal education. Online education combines the flexibility of distance education and the interactivity of conventional classroom based educational delivery. Lectures, coursework, assignments, questions, and discussions, they all take place at the learners’ convenience - online.

Assessment in Online Education

With all the advantages and a high level of flexibility, online education also brought in some concerns that are very important in the educational process. One such major concern pertains to the area of student assessment. Even though many higher education institutions offer courses via online mode, the assessment method is still conventional. This is one aspect, which cannot be realized using the Internet exclusively. In an online examination system, the institutions face the problem of giving guarantee that the student doing the examination is the person enrolled.  In addition, there are a number of academic dishonesty issues pertaining to online flexible examination, which need to be fixed. Due to this difficulty in authenticating the student’s identity in an online mode and other academic dishonesty issues, institutions turn to traditional examination methods where the student has to be present physically to take the examination under the supervision of an invigilator. This breaks down the anytime anywhere flexibility provided by any online education system. Hence, at this point of time, the flexible mode of delivery is not available for examinations or continuing assessment.

Use of the computer has had a tremendous impact on testing practices. With the increasing popularity of these tests, there has been a marked change in the way tests are being constructed and delivered.  In addition to multiple-choice formats, simulation based tests that appear to hold promise for performance assessment has been introduced. Further, computerized adaptive testing is gaining increasing popularity among professional testing agencies. In adaptive testing, the computer tailors a test according to an examinee's ability level. For example, based upon the kind of response made to a question, the computer can update the estimate of the examinee's ability and decide whether to proceed to a higher or lower level question (Welch and Frick, 1993; Rudner, 1998).

Nevertheless, all these developments in computer-based assessment don’t seem to fix the problems of the academic dishonesty issues associated with online assessment in the absence of supervision.

Academic dishonesty issues in online examination: Impersonation and plagiarism of different types are the major issues with online flexible examinations. Impersonation is particularly a problem in situations where a single assessment is not integrated into delivery. Examination centers are designed to provide supervision and verification in order to prevent this problem. Online video interaction and/or surveillance is another issue. Many people feel that in situations where there has been a history of online interaction with students, it is much harder for students to have someone else carry out a convincing impersonation. However this could be very tricky in terms of evidence on both sides and if industry or credentialing certification is at stake then it is best to be as sure as possible.

In the following section, I present a proposal on how to overcome the issue of academic dishonesty in online assessment methods. If we are able to design methods to check impersonation and plagiarism aspects, then most of the problems associated with the creditability of online assessment will become redundant. I propose a two-pronged approach to the issue of academic dishonesty in online examination. The first one is to make the assessment integrated into the instructional method and hence, reduce the heavy emphasis on terminal or final examination. The second is to introduce a combination of techniques to authenticate the examinee and detect plagiarism.

Embedded Assessment

One of the main reasons for impersonation in examination is that our academic assessment system gives a lot of emphasis to terminal/final examination scores for taking decision regarding certification.  Since the terminal or final examination is a one-time effort in an academic session, it is easy for the student to request someone else to take the examination in place of her/him. Impersonation cases are most prevalent in systems where the terminal examination plays a major role in certification decisions. On the other hand, if the certification decision is based on various continuous assessments throughout various academic activities, then it will be difficult for the student to request someone else in place of him/her, and it will amount to the other person actually doing the course. One way to reduce the emphasis on final examination for assessment is to spread the assessment goals across the entire program/course. If so, then the program goals can be assessed through embedded assignments in the program/course. 

Effective integration of assessment into the instructional program requires that the assessment activities become part of the instructional program. While this sounds quite simple, the process of developing such an assessment program requires intense planning, and a thorough knowledge of desired learning outcomes. Such integrated assignments structured in a set of stages will discourage plagiarism, and gives the students the opportunity to see their work accumulate and develop.  The integrated assessment system can use a number of existing testing techniques – both objective and descriptive types. The objective testing can be in the form of integrated quizzes embedded in the online course.

With regards to descriptive methods, peer assessment is a viable strategy. This can be used as an instructional strategy as well as a strategy for reducing the subjectivity in assessing the descriptive responses. It has been demonstrated that students and their peers are capable of being very reliable assessors of their own work (e.g. Wondrak, 1993; Orpen, 1982); and their involvements in the evaluation process helps to understand what is expected from them.   I suggest that the assignments, portfolios, student projects, etc. should be designed as part of the overall instructional design and codified with meta data in such a way so as to automate the whole process – getting assignment, submitting assignment, evaluating, crediting the students, etc.

To reduce the subjectivity in evaluation, I propose that each assignment is evaluated by two peers on a standardized scoring procedures or rubrics for each assignment. Credit for the assignment will be based on two aspects: a) how well the given assignment is executed; and b) how well the given peer assignment is evaluated.

The integrated peer assessment system will work as follows. Imagine a hypothetical situation in which there are 30 students enrolled for a particular course which is of 10 modules. Let us assume that the students need to submit one assignment for each module. After the first module is learned, the students received the first assignment and submitted it to the system after completing the same. After receiving all the assignments or the scheduled date, the system sends two assignments, selected randomly, to each student along with the scoring rubrics for evaluation. The student has to evaluate both the assignments and submit them to the system as instructed. For convenience sake, let us assume that each assignment is for 100 marks. Now the system will compute the scores for each student as below.

Assignment Score:

AS = [{A+B/2) * w] + [{100 – (˝a1-a2˝)} * w1] + [{100 – (˝b1-b2˝)} * w2]

Where:

A = Score obtained from peer review 1

B = Score obtained from peer review 2

a1 = Score given  by self to the peer assignment 1

a2 = Score given by other to the peer assignment 1

b1 = Score given  by self to the peer assignment 2

b2 = Score given by others to the peer assignment 2

w = Weightage given for the quality of assignment

w1= Weightage given for the quality of evaluation peer assignment 1

w2= Weightage given for the quality of evaluation peer assignment 2

The weightages given for the assignment: w = 50 %, w1= 25 %, and w2= 25 %.

Have a look at the following scenario to understand how the above method of peer assignment evaluation is scored. Imagine that, there are three students. Let’s call them X, Y and Z. Assignment C is given to all three of them, and everyone completed the assignment and submitted it for evaluation.

X evaluates assignment submitted by Y and Z

Y evaluates assignment submitted by X and Z

Z evaluates assignment submitted by X and Y

In the following scenario lets see how the scoring is done. The person X’s assignment is evaluated by Y and Z, and they have given scores of 60 % and 80% respectively for X. The person Ys assignment was evaluated by X and Z, and they have given scores of 40% and 50% respectively. Finally, Z’s assignment was evaluated by X and Y. They give scores of 70% and 50% respectively.

 Now let’s compute the score for the person X:

AS = [{60+80/2) * .5] + [{100 – (˝40-50˝)} * .251] + [{100 – (˝70-50˝)} * .25]=77.5

The scores for the quality of peer evaluation is based on the difference between two evaluation of the same assignment. Further, conditions can be built into the system to reevaluate the assignment by a third person or the faculty if the differences between the two evaluations reach a certain level.

Detecting Plagiarism

Another major concern for online assessment is plagiarism. When students produce assignment papers from the web or other sources, it is often difficult for teachers to figure out if it is plagiarized. One way to figure out the level of plagiarism is to have oral defenses for their assignment, which is a cumbersome process to do it online. Further, there is nothing to stop students from cheating with each other while on the Internet, especially in online assessments.  Nevertheless, cheating has always existed, but the Internet seems to make it easier.  Hence, if online assessments were to be introduced, it would be important that there be a system that checks for consistency in the student’s performance that warrants closer scrutiny.  This way we may need to use authorship attrition techniques only on a selective basis.

In a conventional classroom based educational system, the faculty or the supervisors are normally able to identify plagiarism by students in assignments and written examinations due to their familiarity with the writing style of the student. They are able to identify irregularities within the students’ work compared to previous work, or unusual differences in the language or vocabulary used. However, in an online education mode, the number of students a faculty handles will be much more than conventional education. Further, more than one faculty may be handling the same subject to a different set of students. Hence, getting to know each student well enough to understand their writing style and checking the consistency is difficult. Hence, the need to introduce ways and means to automatically detect plagiarism with the help of software.

There has been a considerable amount of work carried out in the area of authorship attribution, which is also called stylometry. Most of these studies originate from the areas of forensic science and software development and are essentially to determine the authorship. These techniques use a variety of statistical methods for assessing the consistency and similarity in the written material. Stylometry employs the procedures of descriptive statistics to quantify the irreducible invariant traits from the data under analysis. From this quantified data, stylometry crucially moves into the realm of inferential statistics by making predictions and testing hypotheses. Cusum technique also known as QSUM is one of the popular techniques used for authorship attribution, and it is based on the observation that every author has a unique set of habits, which s/he follows consistently when communicating (Holmes, 1998). It includes a number of measurements, starting with the "average sentence length", in a sample of a person's utterance (Farringdon, 1996). Each sentence is compared to the average of the sample and marked with a + if it is longer and a – if it is shorter and generates a sentence length profile. The next step includes the calculation of the deviations of each sentence from the average.  (Farringdon, 1996).

The cusum method determines authorship by successively adding up the deviations of sentence length and language habits in a given sample (deviation from the mean sentence length and mean number of habit usages in the sample under analysis). These deviations are then plotted on a graph for interpretation. First, the cusum chart of a single sample can show whether it was written by one author, or by several authors. If written by a single author, the premise is that the two graphs will closely track one another, since they represent the regular and consistent habits of a single author. Another way of using the cusum method is to compare two samples of writing. This involves combining the results of two sample analyses into a single cusum chart. One set of results derives from the anonymous text, the other from a writer hypothesized to be the author of the anonymous text. If the sentence length and habit graphs of the combined chart closely track each other, then the anonymous text is, with a high degree of probability by the author of the first sample. This particular technique will help us determine whether the assessment material submitted by the students is of his/her own or plagiarized from other sources.

Biometrics for Authentication

In addition, to authorship attribution the online assessment system can use other techniques such as behavioral 'biometrics' to authenticate the examinee. The typing rhythm of a user can be monitored, by measuring the time intervals between keystrokes, which can be used for authentication. This makes it difficult to impersonate during the examination because the imposter must also mimic the typing style of the examinee, which is much more difficult.

During the last two decades, there has been considerable work carried out in the area of biometrics using keystroke patterns (Gaines, Lisowski, Shapiro, 1980). It is a cheapest biometric, which don’t require any additional hardware and comes at no cost to the end user. A good summary of this research area was published in Joyce and Gupta (1990) who were able to show an imposter pass rate of 0.25 percent and a false alarm rate of 16.36 percent. More recently, some additions to the system described by Joyce and Gupta were proposed, such as continuous monitoring and updating of the user keystroke model (Leggett and Williams, 1998).  Also, there has been an attempt to use neural networks to learn to identify users by their typing patterns and reported imposter pass rates of zero percent and false alarm rates of at best 11.5 percent  (Brown and Rogers, 1993). So, with the development in the field of biometrics, I am sure a software system for authenticating the examinee can be developed.

Conclusion

The present paper proposes a combination of assessment methods with software system for detecting plagiarism and impersonation. Though, there are certain limitation in the proposed system, these are generic in nature associated with any assessment system and not limited to only online assessment. If implemented, to a great extend, the institutions’ can guarantee that the student doing the examination is the person enrolled for the course.

References

Brown, M. and Rogers, S. J. (1993) User Identification via Keystroke Characteristics of Typed Names using Neural Networks, International Journal of Man-Machine Studies, 39(6), 999-1014.

Farringdon, J. M. (1996). Analysing for Authorship: A Guide to the Cusum Technique. Cardiff, University of Wales Press.

Gainnes, R., Lisowski, W., Press, S. and Shapiro, N. (1980). Authentication by Keystroke Timing: Some Preliminary Results, Rand Report R-256-NSF, Rand Corporation, Santa Monica, CA.

Holmes, D.I. (1998). The Evolution of Stylometry in Humanities Scholarship. Literary and Linguistic Computing, 13 (3), 111-117.

Joyce, R. and Gupta, G. (1990). Identity Authorization Based on Keystroke Latencies, Communications of the ACM, 33(2), 168- 176

Leggett, J. and Williams, G. (1998). Verifying identity via keystroke characteristics, International Journal of Man-Machine Studies, 28:67-76.

Orpen, C. (1982). Student versus lecturer assessment of learning: A Research Note, Higher Education, 11, 567-572.

Rudner, L.M. (1998). An On-line, Interactive, Computer Adaptive Testing Mini-Tutorial, ERIC Clearinghouse on Assessment and Evaluation (http://ericae.net/scripts/cat) accessed 11/98.

Welch, R.E. and Frick, T. (1993). Computerized adaptive testing in instructional settings. Educational Technology Research & Development, 41(3), 47-62.

Wondrak, R. (1993). Using self and peer assessment in advanced modules, Teaching News, 22-23.