PDF  Back   PPT

ABSTRACT

Many students produce word-processed essays as a routine submission medium for coursework. Given the problems with marking essays manually, is it time to consider alternative methodologies? This paper describes how the author’s software system may prove to be a good starting alternative. In particular the approaches to marking style and the performance of marking content will be highlighted. Pointers for improved feedback [both to student and staff] and the detection of plagiarism will enhance the potential of automated essay marking. The author concludes by presenting the reader with a large point-to-ponder.

Keywords

automated essay marking grading style content

1. INTRODUCTION

For several years, many students have been submitting coursework essays as word-processed documents. It has, therefore, become the expected way of submitting essay coursework. Nevertheless the methodology of marking has remained the same, namely marking manually.

Manual marking is prone to several adverse subjective factors, such as:

• The length of each essay,
• The size of the essay set,
• The essay’s place in the sequence of the essays being marked,
• The quality of the last few essays marked affecting the mark awarded to the essay currently being marked,
• The effect of the essayist’s vocabulary and errors [spelling and grammar] on the marker,
• The marker’s mood at the time of marking,
• marker’s expectations of the essay set and of each essayist.

In today’s academic environment there are pressures on academic staff to do more with less. Increasing wider access, larger class sizes, reducing the non-progression rate coupled with falling staff resources should complete the case for considering alternatives to manual essay marking.

2. WHAT IS THE POTENTIAL OF AUTOMATED ESSAY MARKING?

The use of automated objective assessment can be shown to be beneficial to the student, the academic and the organisation. The benefits usually given are: reduced or eliminated subjectivity of marking, fast throughput, lower operational costs and improved accuracy, precision, and audit ability. There is no reason to suppose that the same benefits are not likely to occur from automated essay marking.

3. THE SEAR SOFTWARE SYSTEM

The author has created a software system that will mark word-processed essays for content, and is potentially capable of marking the same essays for style.

The name SEAR is formed from the four main stages that the author uses for his normal marking methodology, namely Schema [before the assessment instrument is issued], Extract, Assess and Report [after the assessment deadline]. Here the term “extract” means to identify those parts of the essay that are retained for the actual essay marking.

3.1 Processing of Essays

Word-processed essays are collected under an arrangement with the marker. The delivery of the essays to the SEAR system may be by e-mail attachment, via some magnetic media or by CD-ROM. The essays are held in a specific directory until the marking date falls due. After this date the text is then extracted from these essays and held in another directory before the actual marking is conducted. The extracted text may be marked for style, or for content, or for both depending on the marker’s demands.

Late submissions will be processed in the same manner.

The reports produced by SEAR are in simple text format which make them readable to both text editors and word-processing packages. In particular the software produces reports which are ‘exportable’ to commonplace spreadsheet software for further processing to fit with a marker’s particular demands.

One report forms the basis for the provision of feedback to the essayist and the marker, and the same report when manipulated by spreadsheet software forms the basis for the possible identification of plagiarism.

3.2 Content

This marking is based on a unique content data structure that is derived from a simple textual content schema produced using the simplest of text editors [the author’s web pages hold a sample of such a content schema].
A model essay is required to verify that the content data schema is correctly setup before the actual content marking is attempted.

Essay extraction throughput is very fast, whilst the content marking of essays is achieved at a rate of about 150 words per second.

If so required the content schema may be easily altered and the essays remarked.

Neither training nor calibration of the content marking algorithm is required. However to ensure process confidence a model essay should be provided by the marker and processed by the SEAR software prior to the processing of any submission.

3.3 Style

The author has been unable to fully develop this marking algorithm due to the unavailability of suitable marked essay sets. The SEAR system offers the potential for the marking of essay style by the use of surface metrics such as various word counts. The author’s research has identified a pool of metrics that could be universally applicable to marking essay style regardless of essay subject or level of essay.

For marking essay style, a significant sample of the essays will have to be manually marked to elicit the coefficients used in the style marking algorithm. The size of the sample has to be at least twice the number of metrics used in the pool. Currently a sample of about 36 to 40 essays will be sufficient to provide the necessary mathematical values.

3.4 Marking Performance

At the current stage of development SEAR proves to be capable of marking essay content at the lowest level of Bloom’s Taxonomy [knowledge].

Statistics show that SEAR marking highly correlates with manual marking. Sometimes SEAR marking correlates better with the first human marker than with the second human marker.

Of the seven subjective factors listed in the introduction that have an adverse effect on manual marking, SEAR eliminates every single one of them.

4. THE FUTURE

So, what of the future? The future for SEAR is both interesting and multifaceted.

Interesting because the automated marking of free text responses is already a growing field and automated essay marking opens a large area of opportunity for innovative ideas.

In the near future SEAR will be further developed to handle different voices [i.e. active and passive] more effectively and to have increased robustness with regard to handling both spelling and grammar errors.
The provision of better feedback facilities for the marker and the essayist is an example of medium term development.

A literature survey conducted by the author uncovered the fact that all the research published used essays written in the English language.

The work accomplished by this author was conducted on essays that were constructed in English.
Yet it is patently obvious that there must be non-English language essays being marked for style and marked for content every day across this planet.

The long term development, and it is a point worth pondering, is that SEAR may become capable of operating in non-English, but still Latin character based, languages.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
4th Annual LTSN-ICS Conference, NUI Galway
© 2003 LTSN Centre for Information and Computer Sciences