Executive Summary

PBS Project CONNECT was funded by U.S. Department of Education Learning Anytime Anywhere Partnerships in 1999 to create a series of self-paced, Web-based instructional materials for intermediate-level adult ESL learners. Designed to supplement regular ESL instruction, these materials focus on the following themes of importance to adult U.S. newcomers: working in the United States, educational opportunities for adults and their children, and civic participation. Project CONNECT consists of eight instructional units, Internet activities, listening exercises, communication and publishing features, video passages, and a basic computer and Internet tutorial. All of the site material has been written specifically for adults learning English, with emphasis on listening and reading comprehension. As of the date this report was written, the URL for the materials was http://learnenglish.pbs.org/index.html.

SRI International (SRI) has been responsible for the design and implementation of the formative and final evaluation of the project. A series of three successive prototype pilot tests were carried out under SRIs direction from March 2001 to November 2002. This report addresses the final evaluation of the project, carried out from February 2003 to July 2004. In the summer of 2003, as preparations for the classroom implementation were under way, five of the eight instructional units and an orientation unit had been completed. Another unit was added during the evaluation. Given ongoing work to develop instructional units and the need for information to guide implementation, the evaluation plan was revised to allow for a more formative evaluation process, with a focus on evaluating how the field test sites implemented Project CONNECT. During the data collection period, early findings were used to revise the materials and to inform the development of teacher support materials. Altogether, nine sites (in Kentucky, Texas, California, Arizona, Nevada, New York, and Virginia) and 292 students participated in the final evaluation. The field test sites represented community colleges, programs affiliated with local school districts, and community-based organizations.

The evaluation study focused on four key areas: learner access to technology-based learning, the usability of the learning environments, learner engagement, and learning outcomes.

Key Findings about Learner Access

• All segments of the demographically diverse (across age, gender, educational setting, and background) student sample were able to successfully use Project CONNECT.
  • Although students with more limited computer skills and education required more assistance, they were highly engaged when using the Web site and felt that it was of value to them.
  • The opportunity to email other English language learners provided sufficient motivation to overcome even strong language and computer skill gaps.
• Some site-specific limitations on access to the Internet were encountered, principally when teachers had limited or no experience using the Internet in instruction.
  • Teachers were not always aware of how to schedule time in the computer laboratory for their classes or whom to contact to be sure that the laboratory was prepared for student use of Project CONNECT.
  • In some cases, heavy demands on computer facilities by other school programs limited access.

Key Findings about the Usability of the Learning Environments

• Intermediate-level students with some basic computer skills (e.g., elements of keyboarding, ability to use a mouse, familiarity with computer menus) were able to use Project CONNECT without much assistance, although having assistance available and providing some guidance in using the site were important. Teachers felt Project CONNECT filled a gap by offering a site that was relatively easy for students with limited English proficiency to use.
  • Not all students who participated in the study had the basic computer and Internet skills to use Project CONNECT successfully without support, although as noted above, the email feature motivated even complete novices to participate.
  • Many adult ESL classes contain mixed levels of students. The mixed language levels in many ESL classrooms meant that some students struggled with the language used on the Web site, in addition to the technology.
• The availability of teacher support was important to students successful use of Project CONNECT.
  • 92% of teachers reported that students asked for some kind of assistance in each session using Project CONNECT.
• The online learning units and online community (with the closed email system and the Discussion Board) were the most popular features of Project CONNECT.
  • The Discussion Board gave students an opportunity to discuss topics related to the content of the units as well as to share experiences about learning English and being a newcomer to the United States.
  • The closed email system provided a chance for learners to communicate with other adult learners in their own classrooms, with learners in other classrooms using Project CONNECT, and with their teachers.
• Teachers' use of Project CONNECT varied in regard to the degree of student interaction/collaboration, with some using a very teacher-directed approach and others using a more interactive approach.
  • In teacher-directed approaches, students were given assignment sheets, the teacher (not fellow students) handled requests for assistance, and most of the interaction was between the teacher and an individual student. Reading and writing practice received strong focus, although the audio files provided some listening practice.
  • In interactive uses, students were still given directed assignments, but Project CONNECT was used as a springboard for small group discussions, and speaking and listening were incorporated into the flow of reading and writing activities. Students played an active role in helping each other navigate the Web site.
  • Project CONNECT's online community provided reading and writing practice in a meaningful context, regardless of the connection between Project CONNECTs unit content and the teachers curriculum.

Key Findings about Learner Engagement

• Student motivation to use Project CONNECT was generally very high, and students persistence in using the site was related positively to their teachers comfort/experience with using technology in instruction. Teachers who were computer novices themselves had difficulty anticipating the volume of assistance they would need to provide, and they typically offered fewer sessions using Project CONNECT than did more technologically experienced teachers.
• The students found the online learning units to be of great interest and relevant to their needs and individual life situations. They identified with the characters portrayed in the units and liked Project CONNECTs multimedia approach.
• The community space features were highly engaging for students. The email and Discussion Board features provided authentic and relatively low-stress contexts for students to practice communication skills. The opportunity for authentic communication and contact with other language learners was quite motivating.
• Both students and teachers disliked the embedded pre- and post-tests of unit content; they found the limitations that the study needs imposed on the tests (e.g., no specific feedback, inability to repeat the pretests) rendered them less useful for instructional purposes.
• Teachers were quite positive about the content and structure of the online learning units and the community space feature. Overall, they felt that Project CONNECT would be a good supplement to what they were doing in class. Reactions to the Student Management System were mixedsome used it to send group emails, others did not have the time to become familiar with it.

Key Findings about Learner Outcomes

• According to teachers reports, students showed increased interest and ability to use technology. Teachers noted that many students showed an increased ability to use the Web site independently and showed an increased interest in using Project CONNECT and/or computers outside of class.
• Teachers reported that Project CONNECT filled a niche in providing rich opportunities for their ESL students to practice reading, writing and listening, especially in meaningful contexts through email and discussion boards. They also appreciated the numerous opportunities for writing practice embedded in the online learning units.
• Students reported that they felt that Project CONNECT helped them improve their language skills, especially writing, and they appreciated the opportunity to communicate with others while practicing their skills.
  Data reports of the embedded on-line pre- and post-assessments that students had completed during the semester were unable to be accessed by programmers, and as a result, we were unable to report on some learner outcome data.

Key Recommendations for Future Implementation

Almost all teachers needed to provide some structure in making use of the Web site. Without that structure, students frequently moved too quickly through the units without taking the time to process the material. In this regard, teachers support materials should be shared with other teachers to facilitate optimum use of the Web site.

Students were highly motivated to use the discussion board and email to communicate in writing with fellow students within the classroom and with students at other participating sites. Teachers should be encouraged to support students in these activities.

Recommendations for Further Evaluation

• Use multiple measures to assess learning outcomes:
  1. Given the lack of experience with online assessment on the part of teachers and students and the possibility that technical problems may occur, it may be advisable to use paper and pencil test pre- and post-tests to measure mastery of unit content and self-reported changes in students skills, attitudes and knowledge of technology. A different posttest version would allow teachers to give feedback to students on the completion of the pretest.
  2. Use a combination of session checklists (for both teacher and students) and initial and exit interviews for measuring technology outcomes.
• In adult education contexts with immigrant populations, a wide variety of kinds of programs, lengths of sessions, student contact hours, and attendance policies and practices exist. Student drop out rates are often high, as students frequently must fit classes around the demands of their jobs and families. Teachers often work part-time and are not paid for planning time. Evaluators must take this into account and, in particular, make sure that time can be scheduled with teachers for the exit interviews.
• In pre-evaluation teacher training, strongly emphasize lessons learned from this study, especially with regard to teachers familiarizing themselves with Project CONNECT ahead of time and making use of lesson plans developed by other teachers until they are ready to create their own. Also have teachers prepare back-up plans for using class time when computers are down.