INTERNSHIP PROJECT HANDBOOK
DEPARTMENT OF SOCIOLOGY
WILLIAM PATERSON UNIVERSITY

PROCEDURES AND GUIDELINES 
FOR PREPARATION AND SUBMISSION 
OF THE INTERNSHIP EVALUATION REPORT

Effective January 2011

TABLE OF CONTENTS

Introduction…………………………………....................	1
The Evaluation Plan Proposal……………………………	2
Evaluation Report Guidelines…………………………….	3
Submission of the Evaluation Report………………….....	6
Evaluation Report Organization and Content…………….	7
Evaluation Report Style Requirements…………………...	9
Checklist before Submitting Report……………………. 13
Appendix: Sample Pages…..………………………….... 14

INTRODUCTION

As part of your degree requirements, you need to complete either a thesis or a second-semester internship/evaluation report. Before selecting either option, you should evaluate your career goals and discuss possible options with the Graduate Director. If you are seeking to enhance your job marketability, the second-semester internship/evaluation project is probably the better choice.  If you select this option, it is strongly recommended that you first take SOC 5310 Program and Policy Evaluation.  

The following information should guide you in setting up and completing your evaluation project.  If you receive approval to do a report that is other than evaluation research, these guidelines will be adapted accordingly through your mentor and the Graduate Director.

The primary function of an evaluation report is to inform management about the findings, conclusions, recommendations and lessons learned that were developed through the assessment process. In fact, decisions to carry out evaluations typically result from management’s desire for performance information.  Individuals involved in program/project delivery have an intrinsic need to know how well the initiatives with which they are involved are performing and what recommendations could lead to improvement.  Conducting an evaluation and producing a report will thus provide students with an opportunity to gain experience and sharpen their skills and abilities in applying research methodology.  The completed deliverable is not only of great value to the receiving organization, but also provides the student with a portfolio asset that demonstrates real-world competency in evaluation research. 

Evaluation research topics can be varied and depend upon the needs identified by the site supervisor in your internship setting, but should also reflect your interests and methodological capabilities.  You will be required to submit an evaluation report application and proposal that includes all of the items detailed on the next page.  The topic of the evaluation you are planning to write should influence your decision as to who will be your evaluation report mentor. You will want to enlist a professor with expertise in the area you will be pursuing in your evaluation report. 

Once your faculty mentor, the Internship Coordinator, and the Graduate Director approve your proposal (submitted at least one month before the end of the preceding semester), you will be issued a permit to register for SOC 6920 Professional Internship II.  You will then pursue the research activity related to your evaluation report, and complete the final document. About one month prior to its completion and approval by your mentor and two other faculty readers, you will be scheduled to present your evaluation report in a public forum to the faculty and students who will attend.

1
THE EVALUATION PLAN PROPOSAL

You must submit your evaluation plan proposal in the semester prior to registering for SOC 6920 Professional Internship II.

The evaluation report proposal is developed as part of the learning experience and course requirements for SOC 6910 Professional Internship I.  This plan evolves in consultation with 1) the site supervisor, who makes known to the student what are the needs of the organization in terms of assessment and/or evaluation, and 2) the faculty supervisor, who will mentor the student on the research design and execution of the study, and submission of a written report and oral presentation.   

The proposal to be submitted for approval must include the following:

•	Objectives/Issues.  
What questions are being asked?
•	Information requirements. 
What information is needed to answer the questions?
•	Information source.  
From whom can the necessary information be secured?
•	Instruments.  
What can be used to find the answer?
•	Design.  
Who will complete the instruments and what comparisons may be made?
•	Time schedule.  
When will the information be collected, analyzed and reported?
•	Analysis.  
What will be done with the data?
•	Report.  
Who will receive this report?

2
EVALUATION REPORT GUIDELINES

Title Page

The title should capture the primary purpose of your evaluation report; a search of key words should lead a researcher to your work.

Executive Summary (1-2 pages)

The executive summary gives an easy-to-read summarization of the evaluation’s major purposes, methods, findings, and recommendations.  It should be free of jargon (e.g., not “a cluster sampling strategy” but “we chose respondents at random”).  It should contain active, not passive verbs (e.g., “The evaluation found …” or “We recommend …” instead of “…was found…” or “It is recommended…”).  You write this after completing your evaluation report, as it contains a synopsis of your results.  It needs to be concise and specific, but also readable, well organized and self-contained.

Acknowledgments 

It is important to acknowledge the support that you received in completing your evaluation report. There should be an acknowledgement of programs, organizations, etc, that provided an opportunity to locate participants for the study. In addition, there should a general acknowledgement of the groups that agreed to participate or to facilitate participation (e.g., parents, teachers and students). It is also important to acknowledge assistance in developing and completing your evaluation report from your site supervisor and project mentor, and any other consultants who assisted in your development of materials, use of equipment or statistical analyses of data. 

I. Introduction (2-4 pages)

Describe the problem or program that the evaluation will address.  In the description, tell how many people are affected and what literature exists, both within the organization and outside that provide a context for your study.  State the objectives of the evaluation, the questions, and the standards.  If you used an evaluation framework to guide planning and evaluation, describe it and its use in the evaluation.  Establish the connections among the problem, the objectives, and the evaluation.  That is, tell how the evaluation will provide knowledge about this particular problem and also provide new knowledge about it.  

II. Methods (7-10 pages)

The methods section of the report should define terms and describe the program to be evaluated, design, participants, outcomes and their measures, and analysis.  Some of these elements could be:

•	The program/policy/operation: Describe the focus of study and evaluation design.  How long did each person participate in the evaluation?  If you prepared protocols to standardize the implementation of the program(s), describe them and any training in the use of the protocols that took place.

3
•	Definitions: Give a conceptual definition and an operational definition of the concepts you will use in your report.
•	Design: Tell whether the evaluation used an experimental or observational design.  If the design was experimental, specify the type (e.g., concurrent controls, in which participants are randomly assigned to experimental and control groups).
•	Setting: Report on where the evaluation data were collected.  You might include the geography (e.g., name of city and state), the locale (e.g., urban versus rural), and the site (e.g., clinic, academic, office, community center).
•	Participants: Give the inclusion and exclusion criteria for participation in the evaluation.  Tell whether the participants were randomly selected and randomly assigned.  Explain how the sample sizes were determined.
•	Outcomes and measures: Describe each program outcome and the characteristics of the data collection measures for the main evaluation questions.  Who administered the measure? Was training required? Is the measure reliable? Is it valid?  How much time is required to complete the measure? How many questions does it contain? How were they selected? If appropriate, cite the theory behind the choice of questions or the other measures on which they were based. How is the measure scored or scaled?
•	Analysis: Describe and justify the specific analytic techniques used to test each hypothesis or answer each evaluation question.  If you used any unusual methods in the analysis, describe them.  Name the statistical package used in case other evaluators want to perform a similar analysis using the same setup.  If you have used a relatively new or complex data-analytic method, provide a reference for it.

III. Findings (7-10 pages)

The findings section of the report presents the results of the process or implementation evaluation and the statistical analyses.  All response rates should appear in the section, along with descriptions of the evaluation participants’ characteristics.  When appropriate, this section should include a comparison of the individuals who agreed to participate with those who refused or did not complete the entire program or provide complete data.  The results for each major evaluation question and its sub-questions also appear in this section.  For example, if one of your main questions asks whether respondents’ reported a higher level of satisfaction, you should present the results for that question.  You might also provide data on the types of individuals (e.g., gender, age range, position) for whom the program was most and least effective in terms of satisfaction.

Tables and figures (graphs or other illustrations) are useful for summarizing results in this section.  However, this section is not the place for an interpretation of the data.  That comes in the next section.

When you create a table for an evaluation report, you should keep two main rules in mind.  First, the most important values to be compared should appear in the table columns.  For example, the first column might be the demographic data (age, gender, etc.) and the next column(s) might be percentage distributions of the responses.

4
Second, if appropriate and possible, the statistical values should appear in descending order with the largest values first.  If the table contains more than one column of statistics, the most important set of data should be placed in the first data column and set the standard for the descending order.  All tables must be numbered and have titles.

IV. Conclusions or Discussion (6-8 pages)

In this section of the evaluation report, you should tell readers, in paragraph form, what the results mean by answering questions such as the following:

•	Taking the broadest perspective, what can one conclude from the evaluation?  Is the program any good?  For whom?  Under what circumstances?
•	Did the program achieve its goals and objectives?
•	For which participants was the program most effective?
•	For which participants was the program least effective?
•	Which components of the program were most/least effective?
•	How do the results of this evaluation compare in whole or in part with the findings of other studies?
•	What new knowledge about program, policy, and/or system operations do the evaluation results contribute?
•	What gaps in knowledge have been revealed by this evaluation?
•	What are the limitations of this evaluation (due to imperfections in the design, sample, measurement, and analysis), and how do these affect the conclusions?
 
V. Recommendations (1-2 pages)

In this section of the evaluation report, the evaluator might consider answering questions such as these in paragraph form:

•	Without changing its basic goals and objectives, if the program were to be redone to remove its flaws, what are the top five changes or additions that should be considered?
•	If the program were to be applied to another setting or group of participants, who would be likely to benefit most?
•	If the program were to be instituted in the same or some other setting, what costs could be expected?
•	What objectives should be changed or added to the program to expand its scope and effectiveness?

VI. References (1-2 pages)

VII. Appendix

Generally, appendices are used to share the instruments that you used in your research.  Depending on your methods, this can include forms related to the Institutional Review Board, such as Human Subjects Informed Consent Forms or your interview schedules. Additionally, it may include your survey, forms, statistics, transcripts of qualitative data, or templates relevant to your research.

5
SUBMISSION OF THE EVALUATION REPORT

Subject content and appropriate supervision of all aspects of the evaluation report process are the responsibility of the evaluation report supervisor. The evaluation report supervisor will approve the final clean corrected copy of the evaluation report only if style, form and content standards have been met and the evaluation report conforms to the guidelines as specified in this Evaluation Report Handbook. The Graduate Director will approve the evaluation report once it is approved by the evaluation report supervisor.

Two bound, print copies must first be submitted to your faculty evaluation report supervisor. Each copy must be placed in a secure plastic binder. All pages must fit in the binder. Loose leaf notebooks or binders, clasp or pocket folders, or any other form of binding which may result in lost or loose pages are not acceptable. The final copy of the paper should be free of any typographical or other errors. No corrections can be made in ink or pencil and use of white correction fluid is not permitted. Failure to meet these requirements will result in the return of the paper for further corrections. 

The two copies must be given first to the evaluation report supervisor. One copy will remain with the evaluation report supervisor. The second copy must include a title and signature page which will be signed by the evaluation report supervisor. This copy and the signature page must be given to the Graduate Director for department files. Until approved, the evaluation report must not be given to the site supervisor.

In order to receive clearance for awarding of the degree, the evaluation report must be approved by the student’s evaluation report supervisor and the Graduate Director within the time frame set by the Registrar. Students who miss a deadline will graduate on the next graduation date. Students are reminded that they will need to submit a new application for graduation to the Office of Graduate Admissions. 

IMPORTANT NOTE: William Paterson University has established standards of academic conduct because of its belief that academic honesty is a matter of individual and University responsibility and that, when standards of honesty are violated, each member of the community is harmed. All members of the University community are expected to adhere to the Academic Integrity Policy.

Plagiarism is the copying from a book, article, notebook, video, or other source material, whether published or unpublished, without proper credit through the use of quotation marks, footnotes, and other customary means of identifying sources, or passing off as one’s own the ideas, words, writings, programs, and experiments of another, whether such actions are intentional or unintentional. Plagiarism also includes submitting a paper written by another person and/or previously tendered for academic credit.

Plagiarism in any form is unacceptable and will result in serious disciplinary actions.  The Graduate Catalog specifies a student’s due process rights in such cases.  Students must act ethically in all aspects of the evaluation report process. Students are expected to know, understand and follow the University policies on academic integrity. 

6
EVALUATION REPORT 
ORGANIZATION AND CONTENT

The evaluation report must follow the format specified in the most recent edition of the Style Guide of the American Sociological Association (ASA).  A short version for students may be found at:

 		http://www.wpunj.edu/library/asa-style.dot. 

An evaluation report must have the pages and sections noted below. 

The exact order of the pages is as follows: 

 Title Page
 Signature page (not included in copy submitted to site supervisor)
 Executive Summary 
 Acknowledgments 
 Table of Contents 
 List of Tables (if used) 
 List of Figures (if used)
 Evaluation Report Text (as specified on pages 3-5) 
 References 
 Appendices 

The executive summary, table of contents, lists of tables and figures, sections, references, and appendices have a title in uppercase letters centered at the top of the first (or only) page as illustrated below: 

EXECUTIVE SUMMARY

For further formatting and content details for each of these pages see the sample pages in appendix B of this Evaluation Report Handbook.

The main parts of the evaluation report include five sections: Introduction, Methods, Findings, Conclusions or Discussion, and Recommendations..

I.	The Title Page of a report should include the title of the report, author’s name and institutional affiliation, client or funder(s), and date of preparation. The full title should be clear, concise and appropriately describe the research.

7

II.	The Executive Summary appears before the table of contents and is a succinct non-technical summary of the full-length report on which it is based. It differs from an abstract, which is usually a single paragraph summary of the work. The executive summary is substantially longer and includes several important elements. First, it should contain enough information that it can stand independently of the full report and be distributed to future funders or other stakeholders who require less precise information on the program. The Executive Summary should also provide the following: 

•	A brief introduction or overview of the program’s background (e.g., the stage of the evaluation, a statement of the problem to be addressed, the need for the program in the larger community, the targeted population, the curriculum used, program goals, program components); 

•	The purpose of the evaluation and the evaluation questions that were addressed;

•	Brief descriptions of the methodology and analytical strategy;
 
•	Main findings and conclusions;

•	Implications and suggested next steps based on evaluation results.

III.	The purpose of the Introduction is to clearly present the purpose and rationale of the program and its evaluation (e.g., the type of evaluation being conducted, the problem to be addressed, need for the program in the larger community, targeted population, curriculum, program goals and components). The Introduction should be more detailed than the Executive Summary. 

A.	Purpose of evaluation. In stating the purpose of the evaluation, you should explain whether this is an implementation, summative, or outcome evaluation. It is also useful to indicate if the report is one of a series of evaluations (such as an annual evaluation) or a final evaluation for the program as a whole. If not the final evaluation, the author should provide a brief description of the overall evaluation and how the information presented in the document fits into the evaluation plan. 

You may choose to describe the evaluation in terms of objectives rather than stated evaluation questions. If the evaluation is measuring whether a program has achieved certain targets (such as an increase in agreement or commitment to abstinence by a specific percentage), then you should also discuss the basis for those targets.

B.	Identification of the program or phenomenon studied. It is important to describe the evaluated program or practice in the introduction and background. The reader, even if familiar with the program or process, needs to be reminded of it and its goals in order to understand the approach and findings of the evaluation. Depending upon the audience, this program description can run from 

8
very brief (which may be of more interest to funders) to a lengthy discussion (which may be of more interest to groups interested in replicating the program). Additionally, if the program uses a specific curriculum, you should specify it. 

C.	Identification of target population.  The evaluation report should be as specific as possible in defining the target populations. One purpose of evaluation reports is to provide the information necessary to replicate the program and analysis to see if similar outcomes occur. In order for other program staff and researchers to decide if the evaluated program would work well somewhere else, they need specific information about the current program. 

Typical target population information includes their age, gender, socioeconomic status, geographic area (including school district if applicable and allowable) and race/ethnicity. It is also important to include information about the participants’ community more generally, such as whether it is rural or urban, whether the school district is dealing with various issues, whether the community’s teen pregnancy rate is high or whether the area is struggling with unemployment. 

Reports also need to describe how people were selected into the program: For instance, it is important to describe whether all students in a school district took the program, or whether only certain subsets participated, (e.g., all health classes or one teacher’s health class). 

D.	Review of related research.  A review of relevant recent literature (or at least the most recent statistics) also helps provide context for the program. It helps explain why the program is likely to affect change, what areas of change it is likely to affect, and may provide indications of how strong of an effect you may see given past evaluations. 

A literature review is especially important if the evaluation report is intended for audiences that may not be very familiar with organizational operations, such as parents or other community organizations. Such a review must do four things:  1) be organized around and related directly to the evaluation report or research question you are developing; 2) synthesize results into a summary of what is and is not known; 3) identify areas of controversy in the literature; and 4) formulate questions that need further research. 

IV.	The Methods section describes in detail how the study was conducted. It includes a detailed description of the subjects, procedures, materials, apparatus/equipment and scoring systems. Subsection headings may be used in this chapter. The chapter ends with a restatement of the research question, which includes a clear specification of the independent and dependent variables. The research design and proposed statistical analysis should also be described. Students should refer to the ASA Style Guide for additional guidelines on this section. 

9
V.	The Findings section begins with descriptive statistics that provide a clear picture of the data obtained in the study. It is important to walk your readers through your evaluation findings. Most reports start out providing information on the numbers and percentages of people that completed the various instruments. They may then conduct some analyses to see if participants that completed all of the instruments (e.g., the pre- and post-test) differ on important characteristics as compared to evaluation dropouts. 

Results sections should then describe the study population by providing demographic information. If the evaluation included a comparison group, the treatment and comparison groups should be compared on demographic and important pre-test measures to see if the differences are statistically significant. Then the results section should describe the analyses conducted to answer the evaluation questions.

Most evaluation reports benefit from using figures to display findings. Figures should have descriptive titles and contain enough information that a reader can understand what is being presented without looking at the narrative text. Results tables need to have information on the sample size, the mean value of the variable, the name of the variable, the standard deviation of the variable, the statistical test being applied and whether each relationship is statistically significant. It is vital that an evaluation report be nonbiased and provides all of the information needed to address the research questions. 

The results section should talk about findings that both support and disprove hypotheses and not attempt to gloss over negative findings. Sometimes you are unable to answer a research question. Maybe you had poor response rates on a measure, were unable to interview teachers or had too small of a sample size. Be sure to talk about that in the results section. It is better to put off answering a question for the following research year than to provide analyses of poor quality data that may or may not truly represent the actual occurrence.

VI.	The Conclusions or Discussion section gives an evaluation and interpretation of all results, as well as a discussion of their implications. Logical implications of the findings should also be discussed. While limitations of the study may be discussed, this should be a very small section of this section. Not all studies have implications for future research. If, however, there is an important methodological finding, an important discovery, or an important implication for a new direction of inquiry, then this section may mention directions for future research. Otherwise, this topic does not need to be mentioned. 

The main focus of this section is to interpret, evaluate, and consider the findings in relation to the organization’s operations. This section should end with a concluding paragraph of the main findings and conclusions of the study. Students should refer to the ASA Style Guide for additional guidelines on this section. 

10
VII.	Following a careful review of the evaluation results, Recommendations are individual statements derived from the evidence that prescribe who should do what in the future. They should be relevant and realistic, with clear priorities for action. 

•	Recommendations should emerge logically from the evaluation’s findings and conclusions.  
•	Recommendations should be relevant to the purpose of the evaluation and decisions to be made based on the evaluation.
•	Recommendation should be formulated in a clear and concise manner and be prioritized to the extent possible.

Each recommendation should be given in boldface italics, but not its explanation, as shown below:

Recommendation1: Develop a plan for collecting follow-up survey data and submit for evaluation to determine long-term effects of programming.

Work with schools and projects where programming is being provided to develop a plan for collecting follow-up survey data.  The data should be collected from program participants at intervals of six months and twelve months following the conclusion of programming, and then submitted for evaluation to determine the long-term effects of this project’s programming.

VIII.	The list of References allows interested readers to access the sources you used in the evaluation report. They appear near the end of the report (typically after the Summary, Conclusion, and Recommendations). References should cover all in-text citations, be error-free, and should be reported in ASA style and format.

IX.	Appendices follow the reference page(s) and usually include useful supplementary material that is not included in the evaluation report. The use of appendices allows the writer to continue the flow of the report without overwhelming the reader with technical details. The writer should briefly mention or summarize what is included in more detail in the appendix. Appendices should be included in the order they are referenced. Data and information in the appendices should be clearly presented and explained.

11

EVALUATION REPORT STYLE REQUIREMENTS

In addition to previously specified evaluation report document style requirements, the following style and format specifications also apply:

Page Numbering 

Pagination for the preliminary pages is in lower case roman numerals (i, ii, iii). The title page counts as the first page. However, no number should be typed on the title page. The text of the evaluation report carries numbers. This includes appendices. Roman numerals and numbers should be located in the upper right hand corner of the page. Note that there should not be a running head. This is only used for papers submitted for publication. 

Levels of Heading 

Headings for subsections in each chapter must follow ASA format. The ASA Style Guide identifies three levels of heading, which are selected depending on the number of levels used in a given chapter. The rules outlined in the style guide must be followed in setting up the subsections in each chapter. 

Format

To preserve any special formatting (e.g., tables and figures), the evaluation report should be saved only as Rich Text Format (rtf).  The final copy may also be saved in Adobe PDF, but we caution students that their software may not allow for any further editing if saved in this format.

Margins

The following margin specifications apply to all text (except page numbers) including figures, headers/footers, footnotes, images, etc.: 1.5 inches for left margin, 1 inch for right margin and 1 inch for top and bottom margins. Page numbers must appear within .75 inches from the edge of the page but do not need to appear within the margins of the manuscript. 

Font

Use only 12 pt. Times New Roman. Italics may be used for non-English words and quotations. These specifications apply to all text including captions, footnotes/endnotes, citations, etc.  Bold is not used for text. 

Line Spacing

Double-spacing is required for the body of the manuscript as well as for the abstract, dedication, acknowledgements and table of contents, with the following exceptions: quotations as paragraphs, captions, items in tables, lists, graphs, and charts should be single-spaced. Footnotes/endnotes, bibliographic entries and lists in appendices should also be single-spaced. 

12

CHECKLIST BEFORE SUBMITTING REPORT

Before you submit your evaluation report, use the checklist below to ensure its completeness.

1.  Introduction:  Does the introduction provide a clear notion of:
•	The problem?
•	The program?
•	The purpose of the evaluation?

2.   Literature Review:  Is the reader provided:
•	A relevant context for understanding prior evaluation efforts of similar programs?
•	A survey of relevant historical and current literature?

3.   Methods:  Does the methodology section describe:
•	An evaluation design?
•	Sampling procedures?
•	Subjects?
•	Procedures for data collection?
•	Instruments used?
•	What was done?

4.   Findings:  Does the results section contain:
•	Only data relative to the stated problem(s) or purpose of the evaluation questions?
•	Appropriate statistical tests?
•	Sufficient detail to understand what was found?

5.  Conclusions or Discussion:  Does the discussion section address:
•	The major findings of the study?
•	Practical implication for the agency program or practice?  
•	Recommendations?
•	Limitations of the study?

6.   References:  Are the references referred to in the document listed?
•	References provided.

7.   Appendices:  Are there copies of instruments, important cover letters, and survey forms?
•	Appendices provided.

8.   Overall Document
•	Is the writing style and depth appropriate to the audience?
•	Does the report feel balanced and reasonable?
•	Is there a sufficient amount of rigor to give the study credibility?

13

APPENDIX 
SAMPLE PAGES

14

(Sample Title Page)

Evaluation Report: 
Paterson YMCA 
After- School Programs 

Submitted November 15, 2010 to: 

Paterson YMCA 
128 Ward Street 
Paterson, NJ 07505 
Phone: 973-684-2320 

Prepared by John Smith, from: 

William Paterson University of New Jersey
Department of Sociology 
300 Pompton Road 
Wayne, NJ 07470 
Graduate Office Phone: 973-720-2669 

15

Executive Summary (sample)

Recent years have seen increased interest in the use of handheld computers for K-12 education. At this time, however, schools that are adopting handheld computers are doing so without the benefit of systematic research on the effective uses of handheld computers in the classroom. The PalmTM Education Pioneer (PEP) program was created to remedy this situation. The goals of the PEP program were (1) to determine whether classroom teachers find handheld computers a useful educational tool, and (2) to aggregate the knowledge base of a large set of teachers using handheld computers in their classroom.  This, the final report on the PEP program, is intended to provide information to those interested in the benefits and drawbacks of handheld computers in the classroom.  
This report is based on data collected from the 102 Classroom Teacher Awards during the 2001-2002 academic year. All PEP awards were granted as part of a competitive program. PEP awardees are talented, innovative teachers who collectively have integrated handheld technology into a wide range of instructional activities.
PEP teachers were overwhelmingly positive about the use of handheld computers in their classrooms. Approximately 90% of PEP teachers stated that handhelds are an effective instructional tool; that handhelds have the potential to have a positive impact on students' learning; and that they will continue to use handhelds in the future. Although teachers across all grade levels were positive about the use of handheld computers in their classroom, elementary school teachers were more positive than middle and high school teachers. Teachers who used handhelds for science-based curricula or for writing-based activities found handhelds most effective, although handhelds were found to improve learning activities across many curricular topics and instructional activities.

16
PEP teachers found that the key benefits to students were increased time using technology, increased student motivation, increased collaboration and communication, and benefits from having a portable and accessible personal learning tool. Key drawbacks included inappropriate use (especially of beaming), technology management issues (particularly synchronization issues), usability issues (particularly using the Graffiti software program for long text input), and equipment damage.
PEP teachers reported that having the appropriate software and peripherals was key to the success of their handheld implementation. The use of probes was considered vital to most science-related curricula, and the use of keyboards was considered vital to extended writing assignments. Nearly all teachers reported that additional software was essential to maximizing the benefits of handhelds for learning.
When assigning handheld computers to students, PEP teachers either used a "shared set" strategy, or a "personal use" strategy. When compared to the more restricted "shared set" strategy, the "personal use" strategy, according to PEP teachers, was more likely to increase students' time spent on schoolwork outside of school time, organization, homework completion, technology proficiency, and opportunity to use technology. 
This report concludes with a look forward into the possible future of handheld technology in education. We examine how features of handheld computers, such as mobility, beaming, and personalization of computing at school, create new possibilities for teaching and learning. We also consider how new and anticipated products and technologies, such as wireless connectivity, could amplify key features of handhelds to extend the benefits of handheld technology for teaching and learning.

17

Introduction (sample)
The desktop computer has been a part of K-12 education in the United States for about 25 years. During this time, research has examined the benefits and limitations of computer technology in schools and conditions that limit its effectiveness. Overcoming some of the limitations of desktop computers in schools has proved neither quick nor easy. Now, handheld computers are emerging as a promising new educational technology, after wide use in business for about seven years. Much of the excitement that greeted the desktop computer has marked the arrival of the handheld computer. Will handheld computers fulfill technology's promise as a learning tool?
The goal of the PEP program was to evaluate the potential of handheld computers for K-12 teaching and learning. To do this, we examined a wide variety of uses of handheld computers, designed by over 100 teachers and implemented in their own classrooms, with diverse students around the country. We evaluated the effective instructional uses of handheld computers, as well as implementation strategies that facilitated success. Equally important, we sought to understand the pitfalls and limitations of handheld computers in the classroom.
The primary aim of this report is to provide a research base that can be used by educators, researchers, and others interested in educational uses of handheld computers to make informed decisions about adoption and strategies for implementation.  As 25 years of research on desktop computers has demonstrated, under the right conditions, technology can have a beneficial impact on teaching and learning. These conditions include appropriate resources, support, training, and time for teachers to experiment and plan. But in addition, to teach with technology creatively and effectively across the curriculum, teachers need a vision of the possibilities a technology offers.
This report also aims to help educators understand the possibilities of handheld technology to transform teaching and learning. We offer a view into some of the PEP classrooms of 2001-2002, 
[truncated]
18
Research Design and Methods (sample)
The core of the PEP evaluation study was the PEP teacher questionnaires, which were administered at the end of each semester of the PEP program. We chose a survey study for several reasons. The main goal of the PEP program was to examine the benefits and limitations of handhelds for education, looking across a wide range uses—a goal that is best met with a survey study. Second, data collection that focused on teachers' reports was well suited to the aim of understanding, from teachers' perspectives, the practicalities of implementing handhelds in the classroom—such as managing the equipment and tracking student work on the handhelds—as well as understanding the difficulties and technical problems that teachers reported. Other approaches, such as quasi-experimental design, would have been less effective at meeting these goals. Finally, educational uses of handheld technologies are still being developed and compelling usage scenarios are only now being identified. We hope that the findings reported here will contribute to the development of quasi-experiment research on the effect of handhelds on teaching and learning.
The PEP evaluation study used a two-level evaluation design that consisted of (1) SRI-created surveys of PEP team members and students, augmented by a small number of site visits; and (2) project-level evaluations conducted by PEP awardees and their project teams, with guidance provided by SRI researchers.
This evaluation reports on data from 86 PEP projects in Fall 2001 and 83 PEP projects in Spring 2002 (out of a total of 102 projects). These projects implemented handheld technology for teaching and learning during the 2001-2002 school year, and provided evaluation data to SRI International. In Fall 2001, 147 PEP teachers and their PEP team members completed questionnaires. In Spring 2002, 114 teachers and their teammates completed questionnaires.
[truncated]
19

Findings (sample)

PEP teachers' evaluations of handheld computers as tools for teaching and learning are overwhelmingly positive. These evaluations, which were collected throughout each stage of the PEP program, were based on a wide range of projects and purposes, spanning grades 2 through 12. The robustness of this finding, and its persistence over a year and a half of handheld use and PEP data collection, show that handheld computers have a productive role across a wide spectrum of instructional topics and purposes, grades, and student populations.
These teachers report that handhelds have positive effects on student learning, on teaching practices, and on the quality of learning activities. Teachers also indicated that handheld computers were more easily integrated with the flow of learning activities than desktop computers. This ease of integration is important because it suggests that handheld technology can make powerful computing more integral to teaching and learning.
Evaluation of Handheld Computers as an Instructional Tool
In our teacher surveys, we asked about PEP teachers' evaluation of the effectiveness and impact of handheld computers, as well as their satisfaction with the technology's performance. Figure 2.1 shows PEP teachers' evaluation of the use of handheld computers across a range of dimensions. Approximately 90 percent of PEP teachers stated that handhelds are an effective instructional tool, that handhelds have the potential to have a positive impact on students' learning, and that they will continue to use handhelds in the future.
Approximately 85 percent of PEP teachers stated that handhelds can improve the quality of learning activities and can have a positive effect on their teaching practices.
Although very enthusiastic, teacher evaluations of the use of handhelds across a number of dimensions showed a slight decrease from the numbers reported in our March 2002 Report on the 
[truncated]
20
    Conclusions and Discussion (sample)
Teachers are surprisingly positive about the use of handheld computers in their classroom. We say "surprisingly" because today's handheld computers were designed primarily for the business professional, not the K-12 educator and student. What accounts for this enthusiasm? And what can we expect in the not-too-distant future?
We believe that much of this enthusiasm stems from the fact that handhelds allow affordable, unobtrusive 1-to-1 computing in the classroom. Without them, teachers must rely on 1) a computer lab, which must be scheduled in advance and requires moving the entire class to a new location; 2) the few computers in the back of the classroom, forcing students to share a small number of computers; and 3) expensive laptop computers, which while small enough to be portable, are still obtrusive when on each student’s desk.  Each of these factors contributes to only occasional use of computers in the classroom, limiting the possible impact on education.
Conversely, handhelds are inexpensive enough for us to envision a true 1-to-1 student-to-handheld ratio in the near future; portable enough to be put in a pocket and taken anyplace the student goes; and powerful enough to run most computer software. Combined, these features encourage frequent technology use, integrated throughout the curriculum. Thus, teachers and students can take advantage of productivity software, interactive simulations, drawing tools, and other tools at the precise time and place that they can have the greatest impact on learning.
Our evaluation shows that these benefits have an impact on the classroom. PEP teachers report increased technology use, greater student engagement, and more effective instructional activities when handheld computers are thoughtfully integrated into the classroom. Additionally, teachers who allow students full personal use of handheld computers find handhelds more effective than teachers who limit handheld use. This provides further evidence that technology is 
[truncated]
21
Recommendations (sample)

PEP teachers have shown that handheld computers have a positive impact on student learning and improve the quality of learning activities. While PEP teachers may not be representative of all teachers (perhaps being more technologically savvy on average), nevertheless, they do represent a broad sample of teachers, with a wide range of technology proficiency levels, training, experience, and student populations.  We believe these characteristics (among other considerations) make the PEP program findings largely applicable to students and teachers in general.
Recommendation #1:  Teachers, administrators, and technology coordinators must learn how to use the handheld technology and spend significant time integrating handheld technology to achieve instructional goals.
Although handhelds are an effective teaching tool, educators must invest the necessary time, resources, and planning to become proficient themselves before they can make learning activities more meaningful by enabling students have the tools at hand to study problems in context.
Recommendation #2:  Educators must create policies and practices to minimize logistical problems in the classroom.
Proactive organization should include practices for synchronization and battery charging or replacing, as well as checkout and handheld tracking systems. Such policies and practices will ensure a more integrated learning experience with minimal disruptions or unfocused activity.
Recommendation #3:  Educators must create appropriate use policies, including parental agreements in some cases.
Not permitting handhelds to be taken out of the classroom and having parents sign forms taking responsibility for replacement of the student’s lost or damage equipment will minimize
[truncated]
22
References (sample)

Colella, Vanessa. 2000.  “Participatory Simulations: Building Collaborative Understanding through Immersive Dynamic Modeling.” Journal of the Learning Sciences 9:471-500.

Cattagni, Anne and Elizabeth Farris. 2001.  “Internet Access in U.S. Public Schools and Classrooms: 1994-2000.”  National Center for Education Statistics. Retrieved June 25, 2010 (http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2001071).

Dufresne, Robert J. and William J. Gerace, et al. 1996. “Classtalk: A Classroom Communication System for Active Earning.” Journal of Computing in Higher Education 7: 3-47.

Linn, Marcia C. and Nancy B. Songe. 1991. “Cognitive and Conceptual Change in Adolescence.” American Journal of Education 99 (4):379-417.

Roschelle, Jeremy and Roy Pea. 2002. “A Walk on the Wild Side: How Wireless Handhelds May Change CSCL.”  Proceedings of CSCL 2002, Boulder, CO, January 7-11, 2000.

Soloway, Eliot, Cathleen Norris, Phyllis Blumenfeld, Barry Fishman, Joseph Krajcik, and Ron Marx. 2001.  “Log On Education: Handheld Devices Are Ready-at-Hand.” Communications of the ACM.  44(6):15-20.

Stahl, Gerry, ed.  2002.  “Computer Support for Collaborative Learning: Foundations for a CSCL Community.”  Proceedings of CSCL 2002, Boulder, CO, January 7-11, 2002.  Retrieved June 23, 2010 ( http://gerrystahl.net/cscl/cscl2002proceedings.pdf). 

Tinker, Robert.  1997. “The Whole World in Their Hands.” Concord Consortium. Retrieved June 23, 2010 (http://www2.ed.gov/Technology/Futures/tinker.html).

Wilensky, Uri and Walter Stroup. 1999.  “Learning through Participatory Simulations: Network-based Design for Systems Learning in Classrooms Computer Supported Collaborative Learning.” Paper presented at Conference on CSCL, Stanford University, California, December 12-15, 1999.

23

Appendix: Software and Hardware Used by PEP Awardees

           SOFTWARE APPLICATION 		    VENDOR

Chess/Checkers/Backgammon 	(multiple vendors)
eBooks 	(multiple vendors)
Synonym program 	(multiple vendors)
Doodle Board 	Amin Patel
Ebony Ivory 	Arcosoft
Noah Dictionary Lite 	ArsLexis
AvantGo 	AvantGo
PrintBoy 	Bachmann Software and Services
WordSmith 	Blue Nomad
Chemicalc 	Chemical Concepts Corporation
Chem Table 	Chemtable software
Album To Go 	Club Photo
Coffeepot Software 	Coffee Pot Software
CCProbe 	Concord Consortium
QuickWord 	Cutting Edge Software
QuickBooks 	Cutting Edge Software
Quicksheet 	Cutting Edge Software
Documents to Go 	DataViz
HanDBase 	DDH Software
Solus Pro Earthmate GPS 	DeLorme
Expedition 	EddieSoft
BugMe 	Electric Pocket
Robot Mover 	Ellams Software
Filemaker Mobile 	FileMaker, Inc.
WriteHere 	Foundation Systems
Four zero student 	Handmark
ThoughtManager 	Hands High Software
Bubble Blasters 	Hi-Ce
Cooties 	Hi-Ce
FlingIt 	Hi-Ce
FreeWrite 	Hi-Ce
Go 'n Tell 	Hi-Ce
PiCoMap 	Hi-Ce
Sketchy 	Hi-Ce
TinySheet 	Iambic, Inc.
ImagiMath 	ImagiWorks
ImagiProbe 	ImagiWorks
PowerOne 	Graph Infinity Softworks
PalmPix 	Kodak
Jfile 	Land-J Technologies
Jtutor 	Land-J Technologies

24
SOFTWARE APPLICATION (cont.) 	VENDOR

Easy Grade Pro 	Orbis Software
Diddlebug 	Palm, Inc.
Giraffe 	Palm, Inc.
Palm Reader 	Palm, Inc.
dbNow 	Pocket Express
Quizzler 	Pocket Mobility
PARENS calculator 	Rick Huebner
Classroom Wizard 	Scantron Corporate
SplashPhoto 	SplashData
Messier! 	Star Pilot
MoonPhases 	Steve Kienle
Tealpaint 	TealPoint Software
thinkDB 	ThinkingBytes
Tiny Logo 	Timothy Lipetz
Memo Pad 	VizSync
MusicEar 	Wilson Cheng/MusicEar
2sky 	Zerpec, Inc.

HARDWARE/PERIPHERAL 	VENDOR

Parallel port IR convertor 	Bachman Software
C-Pen 	C Technologies AB
DAS-1206 data acquisition unit 	DataStick Systems
Earthmate 	GPS DeLorme
MELD Interface+R[-49]C 	Francis Deck
ImagiProbe interfaces and probes 	ImagiWorks
PalmPix camera 	Kodak
GoVox digital voice recorder 	Landware
Go Type Keyboard 	Landware
Otter Box 	Otter Box
Modem 	Palm
More stylus 	Palm
Palm Ethernet Cradles 	Palm
Folding Keyboard 	Palm, ThinkOutside
Digi Vox Recorders 	Shinei
Probes 	Vernier, ImagiWorks

25