Synthesis writing in upper-secondary education: From a baseline of texts and processes to process-oriented feedback

Context

This dissertation was carried out within the framework of the LIFT-project. The LIFT-project (Improving pre-university students' performance in academic synthesis tasks with Level-up Instructions and Feedback Tool) is a project funded by the Dutch Research Council (NWO). The project's goal was to provide feedback and instruction on students' synthesis writing, based on national baseline data.

It is important that upper-secondary students develop their synthesis writing proficiency as synthesis writing is a common activity in higher education. Independently of the field of study, all students will write multiple synthesis texts during their academic career. However, as Van Ockenburg, Van Weijen and Rijlaarsdam (2018) point out, students implicitly practice such writing when writing an essay, but generally it is not a writing activity that is explicitly taught in Dutch schools, not in literacy lessons, nor in other school subjects.

Writing synthesis texts -texts which integrate information from different sources- is challenging, given the cognitively demanding nature of this task (Martínez et al., 2015; Mateos et al., 2008; Solé et al., 2013). The process of source-based writing, such as synthesis writing, involves both reading and writing, which led Spivey and King (1989) to label it as a hybrid task. The complexity of synthesis writing does not call for a simple "reading-then-writing" strategy. Rather, it involves a complex interplay of reading and writing sub‑processes. During the writing process, students alternate between reader and writer roles as they read sources, select relevant information from the sources, compare and contrast the information from the different source texts to each other, write and revise the actual text. Key to synthesis writing is the integration process which encompasses connecting the ideas from the different source texts by organising and structuring them around a central theme in the source-independent target text (Solé et al., 2013; Spivey & King, 1989).

Aims

This dissertation is structured around three research lines: (1) a national baseline study, (2) process‑product relations, and (3) process-oriented feedback. The first research line aims to provide a state‑of‑the‑art on synthesis writing in Dutch upper-secondary education and a baseline for future research. It consists of a national survey on synthesis writing on three dimensions: writing performance, writing processes and writing perspectives. The second research line aims at exploring the relation between the writing process and the quality of the text. This research line consists of two sub-goals: first, we focused on source use during the process and how this relates to the quality of the synthesis text; secondly, we formed writing process profiles. The writing process was captured and analysed with keystroke logging tool Inputlog (Leijten & Van Waes, 2013). The third and final research line focuses on process‑oriented feedback. We aimed at creating an Inputlog process-report that would facilitate the use of keystroke logging in the classroom. Moreover, we aimed to develop process-oriented feedback, including the process report and benchmark processes, to improve upper-secondary students' synthesis writing.

This dissertation outlines the path from baseline, over writing process-product relations, to process‑oriented feedback. The focus strongly lies on synthesis writing processes, using keystroke logging.

Overview of the studies

This dissertation contains five studies: a national baseline study, a study on source use, a writing process profile study, a methodological-oriented part on a keystroke logging report tool, and an intervention study with process-oriented feedback.

The national baseline study (Mapping synthesis writing in various levels of Dutch upper-secondary education: A national baseline study on text quality, writing processes an students' perspectives on writing) aimed at providing a baseline for synthesis writing on three dimensions: text quality, writing processes and students' perspectives on writing. The effects of grade, gender and genre (argumentative/ informative synthesis) on the three dimensions were explored. The study consists of a national survey study with a large and representative sample of 658 students from 43 schools in the Netherlands. Participants were upper-secondary students from grades 10, 11 and 12. They wrote multiple texts. The task design was aimed at maximising the generalisability as the national survey was intended to provide a state-of-the-art on synthesis writing. Tasks were rated with the benchmark rating scale method. Writing processes were logged and analysed with keystroke logging software Inputlog. The students' perspectives on writing (writing apprehension, writing beliefs, self-efficacy and writing process style) were measured with a questionnaire (Braaksma, 2002; Kieft et al., 2007; Rijlaarsdam & Schoonen, 1988; White & Bruning, 2005). Results indicated that grade had an effect on text quality, writing process, and writing perspectives. Gender had an effect on the writing process and writing perspectives. For genre, effects were found on both text quality and writing process. In addition to describing the state of affairs on synthesis writing, this study also aimed at providing a baseline for further research. This study formed the baseline for the other studies in this dissertation and offers possibilities for future studies.

In a study on source use, In search of an effective source use pattern for writing argumentative and informative synthesis texts, we explored the relation between source use during the writing process and the quality of the synthesis text. For this study, a subsample of 294 texts and processes of the baseline was used. Source use was measured in terms of proportion of time spent in the sources, number of transitions per minute between the sources, and number of transitions per minute between the synthesis and the sources. Via polynomial regression analyses, the various source-related activities and their temporal distribution were taken into account in an integrated way to identify patterns of effective source use. For each of the two synthesis genres, a regression model of source use was built predicting the quality of the text. The pattern of the argumentative synthesis text consists of three predictors explaining 24.6% of the variance in text quality. This pattern includes a positive curvilinear relationship with text quality for the proportion of time in sources in the first interval; a positive linear relationship for the number of transitions per minute between synthesis and sources in the first interval; and a negative linear relation for the proportion of time in sources in the third and last interval. The pattern related to a high-quality informative synthesis text also has three predictors, that explain 16.2% of the variance in text quality. The quality of an informative synthesis has a positive linear relationship with the number of transitions per minute between the sources in the first interval; a negative linear relationship with number of transitions per minute between the sources in the third interval; and a positive curvilinear relationship with the proportion of time in sources in the third interval. The findings of this study provided valuable insights that were used to develop process-oriented feedback.

In another study on process-product relations, titled Dynamic writing process configurations in synthesis tasks, we identified writing process profiles. For this study, the complete sample of the baseline was used. The main aim of this study was to identify writing profiles by examining configurations of synthesis writing activities across different phases of the writing process and their relation to text quality. We selected a set of nine writing process variables: five measures related to source use (proportion of time in sources for intervals 1 and 3, number of transitions per minute between the sources in intervals 1 and 2, number of transitions per minute between the synthesis text and the sources in interval 2) and four measures related to production (proportion of active writing time in intervals 1 and 3, and number of strokes per minute in intervals 1 and 2). These variables were chosen based on prior research such as the national baseline study and the source use study. Latent profile analysis was used to identify writing process profiles based on the nine keystroke logging measures. Four distinct writing process profiles could be distinguished: Source Users, Quick Source Users, Speedy Producers, and Process-Final Source Inspectors. Results showed that the Speedy Producers profile significantly predicts text quality. The Process-final Source configuration led to the lowest scoring texts. The successful Speedy Producers profile is marked by a below-average source use and is very much oriented towards production. The Speedy Producers score above average for both active writing time and production fluency across the complete writing process. In addition, analyses showed that the majority of the writers adopted two different writing process profiles, implying that writers organise their writing in a different way depending on the task. Intra-writer variability had no impact on text quality. We found indications that task perceptions (topic knowledge, task effort) affect the occurrence of a task profile.

In the study Reporting writing process feedback in the classroom: Using keystroke logging data to reflect on writing processes, we report on a newly developed feature in Inputlog, the so-called 'report-function'. This function automatically generates a pdf-file addressing different perspectives on the writing process: fluency, pausing, revision, and source use. This report aims to facilitate writing tutors in providing process feedback to their students. The study describes the process report and demonstrates how it can be used in the classroom as part of feedback promoting reflection on the writing process. This report was used in the intervention study to give feedback on students' writing process.

The effect of process-oriented feedback on synthesis writing: An intervention study with keystroke logging, addresses an intervention study with process‑oriented feedback. This study aimed to test the effect of an empirically‑based way of giving feedback on students' writing process. A total of 65 students in two feedback conditions received feedback at two moments within one-week time. Participants received personalised information on their previous writing process with the Inputlog feedback report. Moreover, they were given benchmark writing processes (selected from the national baseline study) relative to their own performance (i.e. processes of equally-scoring students in the position-setting condition, and processes of higher-scoring students in the feed-forward condition). We explored the students' perception of the feedback and tested the effect of the feedback on both the students' writing performance and writing process. Writing performance was measured in terms of text quality. For the writing process, several keystroke logging measures were taken into account; the variables were selected based on the national baseline study and the source use study. In addition, we also compared the progress the students made because of the feedback (feedback effect) to the natural growth students make during one year of regular schooling (grade effect). To make this comparison, data of the national baseline study were used as a control group. Results showed that, in general, the feedback was perceived as clear, and most of the elements of the Inputlog process report were evaluated as rather instructive, useful and clear. The scores also indicate that, on average, the students took time to reflect on their writing process. Students expressed that they recalled the information they got from the benchmark writing processes when writing a new text. Regarding feedback effect on text quality, the feedback in the feed-forward condition resulted significant: the participants in this condition wrote better texts after receiving feedback. The feedback in the feed-forward condition had an effect comparable to more than one grade of regular schooling. With respect to the effect on the writing process, we concluded that the feedback had an effect on several aspects of the writing process, and that the effect differed according to feedback condition. When comparing feedback effect with grade effect, large differences were observed. In general, the feedback effect was larger than the grade effect.

Implications

This dissertation contributes to current research on source‑based writing by providing (1) a baseline with writing product and writing process data, (2) more insights into the writing process of synthesis texts, and into the relation between writing process and text quality, and (3) effective process-oriented feedback to improve students' synthesis writing skills. We hope that the studies and the discussion of this dissertation may provide valuable contributions to current keystroke logging research and potentially direct future writing research and writing education.