Ensuring Quality in Distance Education through Developmental Testing

When a self-learning material (SLM) is written and designed, it is tempting to assume it is ready for learners. But assumptions are expensive in distance education – a module that confuses learners, skips key steps, or misjudges reading level can derail an entire course before it ever reaches the classroom. That is why developmental testing exists: a structured, pre-distribution process that puts draft materials in front of real learners, gathers honest feedback, and drives evidence-based revisions. It is the quality checkpoint that stands between a draft and a polished, effective learning package.

What developmental testing actually means 🔗

Developmental testing is a form of formative evaluation – evaluation conducted during the development process with the express purpose of improving the materials before they are finalized. Unlike summative evaluation, which judges a finished product after the fact, developmental testing is iterative and corrective. It asks: Does this material actually work for the learners it was designed for?

In the context of print-based self-learning materials for distance education, this matters enormously. Learners in distance education study at their own pace without constant supervision, which means the material itself must carry the entire instructional load. There is no teacher available to clarify a confusing diagram or re-explain a poorly worded concept. If a gap or error slips through to the final version, it stays there – silently undermining learning for every student who encounters it.

Customised, self-instructional materials are at the heart of instructional delivery in open and distance learning, and the success of any ODL institution depends on their availability in both quality and quantity. Developmental testing is the mechanism that upholds that quality standard.

The stages of developmental testing 🔗

Developmental testing is not a single event – it is a sequence of increasingly realistic trials, each one designed to catch different categories of problems. Broadly, the process moves from controlled, small-scale reviews toward authentic, real-world conditions.

Expert review: the internal check 🔗

Before any learner sees the material, it goes through an expert review. Subject matter experts check for content accuracy and completeness. Instructional design specialists examine the structure, sequencing, and clarity of explanations. Language editors review readability. This internal review catches factual errors, logical gaps, and structural weaknesses that the original author – too close to the material – may not notice.

According to formative evaluation frameworks in instructional design, expert reviewers’ comments are typically sorted into three actionable categories: revisions to make immediately, questions to investigate during later trials, and suggestions that may be set aside. This triage approach keeps the revision process focused and manageable.

One-to-one tryout: the face-to-face session 🔗

Once the material has passed expert review and undergone initial revisions, it is tested with individual learners – one at a time – in a face-to-face tryout. The instructional designer or evaluator sits with a single representative learner while they work through the material and collects real-time observations and verbal feedback.

The goal at this stage is to identify gross problems in the material: typographical errors, unclear instructions, confusing sentences, poor examples, and missing directions. The designer makes clear to the learner that it is the material being evaluated, not the learner – a critical distinction that puts learners at ease and produces more candid responses. Learners selected for one-to-one tryouts should represent a range of ability levels, not just the strongest students.

This face-to-face format is particularly powerful because it reveals what learners actually do when they encounter difficulty. During one-to-one evaluation, one learner at a time reviews the instructional material with the evaluator and gives comments upon it – a direct window into the learner’s thought process that no questionnaire can fully replicate. Issues identified here feed into another round of revision before the next stage begins.

Small group tryout: testing with more varied learners 🔗

After the one-to-one revisions are made, the material is tested with a small group – typically 8 to 20 learners who represent the target population. At this stage, learners work through the materials independently, without the evaluator guiding them, which better replicates real distance learning conditions.

The purpose of small group evaluation is to check the efficacy of the revisions based on one-to-one data, to ascertain how well the instruction works with more varied learners, and to see how well the instruction teaches without the designer’s intervention. The evaluator records learner performance data, time taken to complete tasks, and attitudinal responses – all three feed into further targeted revisions.

The small group tryout also exposes problems that only emerge at scale: instructions that seemed clear to one learner but confuse most others, activities that take far longer than estimated, or sections that generate consistent disengagement. The output of this stage is a revised instructional lesson built on time, performance, and attitude data.

Field trials: testing in real conditions 🔗

Field trials are the final and most rigorous stage of developmental testing. Here, the revised material is used in the actual setting and conditions in which it will eventually be deployed – often across multiple sites, with a larger group of learners (some frameworks recommend at least 30 students), and without any involvement from the instructional designer.

During a field test, also referred to as an operational tryout, instruction is evaluated in the same environments in which it will be used when finished. Teachers or tutors facilitating the trial may receive brief orientation beforehand, but the material must stand on its own – exactly as it would in a real course delivery.

Field trials serve three key purposes. First, they confirm that problems identified in earlier stages have actually been corrected. Second, they generate final suggestions for any remaining revisions. Third, they produce evidence about whether the material achieves its learning objectives across diverse learner groups and contexts. A module may perform well in a controlled small-group session but falter when learners in a remote area use it without internet access, or when a tutor interprets the guide differently than intended. Field trials catch these real-world mismatches.

Quality assurance frameworks for open, distance and digital education consistently emphasize this point. Learning materials should teach in a coherent way, engage learners, and promote development of problem-solving and critical thinking skills – and they should be evaluated and updated on a regular basis. Field trials provide the data to make that judgment with confidence.

What to look for during developmental testing 🔗

Developmental testing is only as useful as the data it collects. Evaluators need to gather information across several dimensions simultaneously.

Comprehension and clarity – Can learners understand the content without external help? Are explanations sufficiently self-contained? Gaps in clarity show up quickly in one-to-one sessions when learners stumble, re-read, or ask questions.

Learning effectiveness – Are learners actually achieving the stated learning objectives? Pre-tests and post-tests administered during tryouts and field trials measure whether the material produces measurable learning gains. The three-level formative evaluation process – one-on-one, small group, and field trial – allows developers to modify and further develop materials to improve their quality progressively.

Pacing and time – Distance learners cannot be told to stay longer or revisit something next class. If a unit takes twice as long as estimated, learners may abandon it or rush through the second half. Timing data collected during small-group tryouts directly informs how content is chunked and how self-check activities are spaced.

Learner attitude and engagement – Dry, demotivating text leads to dropout, even when technically accurate. Evaluators note where learners disengage, which activities they find engaging, and whether the tone feels accessible or distant. Immediate feedback stands as one of the primary advantages of self-assessment tools embedded in SLMs – and testing helps confirm whether those feedback mechanisms are working as intended.

Instructional design quality – Are objectives clearly stated? Do examples match the level of the learner? Is the sequencing logical? Are self-assessment items aligned with objectives? These structural elements are reviewed at the expert stage but confirmed through learner tryouts.

Revising materials based on tryout data 🔗

Developmental testing produces value only when its findings drive actual revisions. The cycle is: test → analyse → revise → test again. Each iteration of the material should be measurably better than the last, based on evidence rather than assumptions.

Developers should create a systematic log of all issues flagged across stages, categorized by severity and type. High-frequency problems – those that appear across multiple learners or sessions – take priority. Isolated issues are investigated further before deciding whether they warrant a revision.

It is also important not to over-revise. A common mistake is making changes based on a single learner’s preference rather than a pattern across the data. Developmental testing produces the most reliable revisions when decisions are grounded in consistent patterns of difficulty, not individual outliers.

Revision of SLMs is as important as the development process itself – this principle positions developmental testing not as an optional quality step, but as an integral part of the design process. Without it, even well-intentioned materials risk reaching thousands of learners with uncorrected flaws.

Why developmental testing matters in practice 🔗

In distance education, the stakes of poor-quality materials are higher than in a conventional classroom. A school teacher can adapt in the moment – explaining a concept differently, answering a question, reading the room. A printed SLM cannot. Once it is printed and distributed to learners across geographically dispersed locations, correcting errors is costly, time-consuming, and sometimes impossible for an entire delivery cycle.

Developmental testing is the investment that prevents those costs. Materials that have been rigorously tested are more likely to reduce learner confusion, lower dropout rates, and improve learning outcomes – especially among learners in remote locations, isolated from other learners and sometimes inexperienced in formal learning situations, for whom a confusing SLM has no easy remedy.

Institutions that treat developmental testing as a standard production step – not an afterthought – send a clear message: that learner experience is not assumed, it is verified. That is the foundation of quality in open and distance education.

What do you think? If you have worked with or studied from self-learning print materials, what quality issues did you notice – and do you think those issues would have been caught through systematic tryouts before distribution? How might limited resources in an ODL institution affect the scale and rigor of field trials – and what minimum testing steps should never be skipped?

References

1. https://distancelearning.institute/staff-training-development/self-learning-materials-effective/
2. https://www.researchgate.net/publication/311987266_Instructional_Materials_Development_in_ODL_Achievements_Prospects_and_Challenges
3. https://mason.gmu.edu/~ndabbagh/cehdclass/Resources/IDKB/eval_techniques.htm
4. https://link.springer.com/rwe/10.1007/978-981-19-2080-6_42
5. https://pubs.sciepub.com/education/3/4/12/index.html
6. https://distancelearning.institute/instructional-design/innovative-assessment-techniques-distance-learning-environments/
7. https://www.journalcra.com/article/self-learning-materials-distance-education-system
8. https://www.sciencedirect.com/science/article/abs/pii/S036013159800044X