Evaluating Life Skills Educational Program and Students' Empowerment: Study Protocol Based on Context, Input, Process, and Product Evaluation Model

Shervin-sadat Hashemian-Esfahani; Parastoo Golshiri; Ali-asghar Asadollahi Shahir; Elham Mozafarianpour; Arash Najimi

ORIGINAL ARTICLE

Year : 2015 | Volume : 1 | Issue : 1 | Page : 20-25

Evaluating Life Skills Educational Program and Students' Empowerment: Study Protocol Based on Context, Input, Process, and Product Evaluation Model

Shervin-sadat Hashemian-Esfahani¹, Parastoo Golshiri¹, Ali-asghar Asadollahi Shahir², Elham Mozafarianpour³, Arash Najimi⁴
¹ Department of Community Medicine, School of Medicine, Isfahan University of Medical Sciences, Isfahan, Iran
² Department of Health Insurance, Health Insurance Organization, Gorgan, Golestan, Iran
³ Department of Public Health, School of Health, Qom University of Medical Sciences, Qom, Iran
⁴ Department of Health Education and Health Promotion, School of Health, Isfahan University of Medical Sciences, Isfahan, Iran

Date of Web Publication

10-Aug-2015

Correspondence Address:
Arash Najimi
Department of Health Education and Health Promotion, School of Health, Isfahan University of Medical Sciences, Isfahan
Iran

Source of Support: None, Conflict of Interest: None

Check

Abstract

Introduction: The presented protocol is a part of psycho-behavioral empowerment program (sponsored by Isfahan University of Medical Sciences) and aims at studying the effect and the rate of its achievement to the determined goals. Materials and Methods: This is an observational study conducted in four relevant sections including: Contractual agreements, investigating documents, and interviewing the program's chief subjects according to context, input, process, and product (CIPP) evaluation checklist by external evaluators, examining some of the subjects including parents, Education Ministry advisors and staff of the related region, and after all conducting a cross-sectional study with one control group among life skills educational program participants. The study population included all the students, parents, principals, Education Ministry staff, and advisory experts. The data were collected through CIPP evaluation model checklist, CIPP evaluation model questionnaire, parents and advisors' attitude, and practice evaluation questionnaire, adolescent' abilities developmental questionnaire, and Depression Anxiety Stress Scale-21. Conclusion: This study aims at designing a modern framework to evaluate the life skills educational program in the Iran based on CIPP evaluation model.

Keywords: Context, evaluation, input, life skills, process, and product model, schools, students

How to cite this article:
Hashemian-Esfahani Ss, Golshiri P, Shahir AaA, Mozafarianpour E, Najimi A. Evaluating Life Skills Educational Program and Students' Empowerment: Study Protocol Based on Context, Input, Process, and Product Evaluation Model. J Hum Health 2015;1:20-5

How to cite this URL:
Hashemian-Esfahani Ss, Golshiri P, Shahir AaA, Mozafarianpour E, Najimi A. Evaluating Life Skills Educational Program and Students' Empowerment: Study Protocol Based on Context, Input, Process, and Product Evaluation Model. J Hum Health [serial online] 2015 [cited 2018 Aug 21];1:20-5. Available from: http://www.jhhjournal.org/text.asp?2015/1/1/20/162527

Introduction

Profound cultural changes and changes in lifestyle have led to a lack of enough ability to encounter life problems and made people vulnerable to those problems. ^[1] Meanwhile, high school students have a unique condition due to teenage and puberty. According to researchers, it is evident that educational failures are increasing among students annually, therefore, education quality is being threatened and most of the students suffer from stress, and anxiety and some affect various mental disorders due to lacking the ability to solve their problems as well. ^[2] These factors can lead to drug abuse, and sometimes even committing suicide. Hence, the main preoccupation of specialists in education and training field is preventive programs and performing them among the masses aiming at decreasing and controlling psychosocial problems. ^[3]

Today a considerable part of world's education system programs are dedicated to life skills. Researchers demonstrate that educating life skills affects other skills including: Problem solving, self-awareness skills, self-control, coping skills and reality assessment, psycho-social competencies and motivation, reducing alcohol consumption, improving decision-making skills, responsibility, interpersonal skills, ethical behavior, and social skills among students. ^{[4],[5],[6],[7],[8]} In spite of the effort made in previous years in Iran to prepare students for a healthy adaptive life with regard to life skills indicators, it followed several problems. Researches on educating life skills in Iran show that it has not been practically emphasized and students suffer from a lack of life skills and Iran education system was not so successful in educating life skills. ^[9] The same issue resulted in designing junior and high school students' psycho-behavioral and educational empowerment program with the basis life skills between 2010 and 2011 in Isfahan. The population study consisted of all students, parents, Education Ministry advisory group, and expertise advisory group in the given area, education, and Physical Education Office of an Assistant was responsible for performing the program. Research performing schools with more educational and psycho-behavioral problems were chosen among all the region's high schools. Subjects were trained for 12 sections each 90 min in their extra school hours. Subjects were divided into classes consisted of each approximately 35 students in order to be best profitable. Educators were chosen from the region's advisory and psychology masters of Science and also approved by the state's Education Advisory Department. In order to perform the plan the following actions were taken: Providing a comprehensive instruction and sending it to all Isfahan's 40 regions, holding a justification session for regional experts, providing educational booklets such as CDs and brochures and sending them to the related regions, persuading all junior and high school first level students to participate in performing the plan, inspecting the related regions, and studying the plan's performance score.

Researches demonstrate that many life skills educational plans were previously performed among Iranian students, though most of them are strongly assumed to be finally left unevaluated.

Evaluation means dynamic investigation about a plan's characteristics and benefits, and it also investigates previous programs' efficiency and effectiveness. ^[10] Evaluation can help analyzing plans' organizing and structure and eventually achieving the determined goals. ^[11] The presented protocol in this study is conducted as a part of students' psycho-behavioral and educational empowerment program (sponsored by Isfahan University of Medical Sciences) aiming at studying the effect and the rate of its achievement to the determined goals, while evaluating the plan's details according to the used theoretic structure providing an accurate knowledge of problems and deficiencies led to its completeness and reform.

The objectives

Studding related elements in an educational setting
Identifying problems needs and opportunities related to the objectives of the training program
Evaluation of financial and human resources, policies, training strategies is used in the program
Evaluation of implementing planned activities and collecting data for making a decision
Review of the achievement of program objectives such as creating positive changes in knowledge, attitudes, and behaviors of students related to life skills and increasing the level of mental health in students.

Theoretic structure of the program

Context, input, process, and product (CIPP) model is a management-based one designed to facilitate administrators' decision-making and is a comprehensive, holistic model able to investigate a plan systematically and extensively. CIPP stands for four words including CIPP, respectively, and presents four types of evaluations, therefore, helps administrators and decision-makers in making the fore mentioned decisions. ^[12],[13]

The last version of CIPP was used in designing this program to improve group work and organizational projects quality as well, and an appropriate scale is considered for each part to determine the rate of its success and goal achievement. ^[14]

Stage 1: Contractual agreements

In this stage, all the stakeholders are detected. The research main points are explained within holding some explanation sessions, and they share us their ideas on how to design and perform the program evaluation. It is essential that all the stakeholders give their agreements on the program's design and performing method. ^[15]

Stage 2: Context evaluation

This stage includes analytic efforts to study the determined factors in educational environments and also efforts done to detect the problems, needs, and opportunities in a given educational context. This evaluation stage shows the importance of primary designing suitability. ^[13]

Stage 3: Inputs evaluation

This stage's main point is to study all the inputs needed to make changes and achieve the mentioned goals. Inputs evaluation is about using the resources needed to achieve general and partial goals. The collected data should help the decision-makers choose the best strategies and resources in designing the program with regard to all specific limitations. ^[7]

Stage 4: Process evaluation

This stage includes collecting the data obtained while designing and performing the program. Stufflebeam believes that the process evaluation should be able to answer the following questions: Is the program performed properly? Which obstacles prevent the success of the program? What essential changes should be made?

Giving answers to these questions helps us control and direct the program's performance. Furthermore, this type of evaluation is conducted to detect or predict the problems we may encounter in the process of educational activities and even shows their desirability. ^[16]

Stage 5: Product evaluation

This stage aims at the program results measurement, interpretation, and judgment.

This type of evaluation is done to judge the desirability of educational activities. In other words, product evaluation is done to associate the output with the program's background, inputs, and process in order to show its importance and desirability. ^[17]

Stage 6: Effectiveness evaluation

In this stage, we study the effect of the program on students' educational progress. ^[14]

Materials and Methods

Subjects

This is an observational study consisting of four relevant sections: Contractual agreements, studying the documents and interviewing the chief subjects through CIPP evaluation checklist by external evaluators, conducting a cross-sectional study among students' parents, the advisors, and the staff and conducting a case-control study in students participated in life skills educational program.

The study population included all the students, parents, principals, Education Ministry staff and advisory experts, regional advisory experts, and all high school 2 ^nd -year students. Life skills educational program is conducted in 10 regions in Isfahan (Isfahan state with the capital of Isfahan city is one of Iran's central cities located 440 km South to Tehran and with a population of 4,815,863 people.) 20,000 male and female students from the aforementioned regions in total participated in the program.

Three distinct sampling methods were used in this study to choose the subjects. In the first method, it is necessary that all the samples be perfectly informed about the program all over the regions and all the evaluators should have enough access to given documents in order to perform the interviews and also complete CIPP evaluation checklist. Therefore, a target-based sampling is done among the Education Ministry Advisory Department heads. In the second method principals, Education Ministry staff and Advisory Department experts are chosen by the census. In the third method, the multi-stage sampling method is used for choosing the students (case group), parents, and other out of coverage students (control group). First of all five regions are chosen accidentally among ten given regions. Two high schools were accidentally chosen from each region (one female and one male school to consider the gender difference). Students were chosen and investigated with regard to every region's part in the sample. Furthermore, a control group (matched with the case group) was chosen in each region in order to control the effect of time and to make an association between the results and the instructions given in the life skills educational program.

Combining the following approaches helps us match the students: Study and adjust the demographic information based on the last heads and houses census, Utilize Education Ministry information and statistics, use the matched information in other projects, use chief informed people in the regions.

The students sample was estimated according to the quantitative sample volume formula in both groups with a confidence coefficient of 95% and a test power coefficient of 80% estimating life skills score standard deviation and considering the error of 5 in both groups.

Instruments

Context, input, process, and product evaluation checklist

Evaluation checklists in this study were designed by researchers according to CIPP model evaluation method. ^[18] The designed evaluation checklist included 58 items in 6 domains: Contractual agreements (15 items), context evaluation (6 items), inputs evaluation (10 items), process evaluation (12 items), product evaluation (10 items), and effectiveness evaluation (5 items). Each item consisted of three options: Done, undone, and partly done, which were completed by the evaluator externally and through making interviews and studying the documents. In order to earn a proper reliability, all the studies and interviews were done by an evaluator. The tool's external and content validity was measured qualitatively by experts' panel.

Context, input, process, and product evaluation model questionnaire

It was a researcher designed questionnaire with regard to the program goals and based on the provided checklist according to CIPP Evaluation Model. ^[19] Face validity and content validity were measured within two qualitative and quantitative stages. In the qualitative stage, panel experts investigated the face validity measurement, relevancy, ambiguous, and difficulty. In this stage, the researcher used the experts' opinions to correct the items in each given case. In the latter stage, which is quantitative, impact score was calculated and showed that all the questionnaire's items had a more than 1.5 impact score. In the quantitative method, content validity ratio was used to make sure that the most important and correct content is chosen and content validity index (CVI) was also used to show that the best measurement method is being used. Nine items from the whole 43 given items in content validity part were eliminated. The questionnaire's average CVI was 0.84. The designed questionnaire included 34 items in four domains: Context evaluation (6 items), input evaluation (9 items), process evaluation (9 items), and product evaluation (10 items). Instrumental questions were based on five-point Likert scale consisting of the following options: 1 - strongly agree, 2 - agree, 3 - no idea, 4 - disagree, 5 - strongly disagree, and each option receives a similar score from 1 to 5.

Advisors and parents attitudes and practices questionnaire

A questionnaire consisting of two attitudes and practices parts is used to evaluate the attitude and practice of participants in the program (advisors, teachers, and parents). Both attitude and practice parts are provided in two versions, one for parents, and the other for advisors and teachers. In order to attitude scale, modified national organization of Mental Health (Bangalore University) questionnaire was used. The questionnaire included 20 questions based on the 3-choice attitude Likert Scale containing: 1 - agree, 2 - disagree, 3 - no idea.

The practice was investigated through parents' life skills educational questionnaire provided by Botvin Skills Training Institute. This part of the questionnaire includes four sections: Modeling (6 items), Behavioral Principles (7 items), Communication (5 items), and Supervision (4 items). This section's questions are based on five-point-Likert-scale (1 - never, 2 - seldom, 3 - sometimes, 4 - quite often, and 5 - always) and the options received similar 1-5 scores.

Young adults empowerments developmental questionnaire

A questionnaire consisted of two parts (first studying the direct effects including young adults' knowledge and practice toward life skills and the second studying the indirect effects including educational progress, exercise, social participation, cultural competence, self-perception) is designed to study the effects of life skills educational program. ^[20]

The first part aims at evaluating the products among students and includes 14 questions on life skills knowledge. Each question consisted of two options (true and false) and this part also included 17 questions on young adults' practice toward life skills. The second part aims at evaluating the program's effectiveness and includes 28 items on the following domains: Educational Progress (10 items), Exercising (3 items), social participation (5 items), cultural competence (5 items), and self-perception (5 items). The options in the practice and program's effectiveness evaluation were designed as three points Likert-scale, and each point received a score as follows: Yes: 3, partly: 2, no: 1

Depression Anxiety Stress Scales

Depression Anxiety Stress Scale short-form is used to examine students' mental health. This instrument is formed of 21 negative emotions (depression, anxiety, and stress). ^[21] Depression subscale includes terms examining unhappy mood, lack of self-confidence, hopelessness, life uselessness, unwillingness to do things, life dissatisfaction, and lack of energy and power. Anxiety subscale includes terms try to evaluate more physiologic arousal, fears, and situational anxieties, and stress subscale involves terms like difficulty in getting calmness, nervous tension, neurotic stress, irritability, and restlessness. The mentioned version is investigated in some studies including of Henry and Crawford study. Internal consistency coefficient (Cronbach's alpha) of the scale was 0.93, and it was reported 0.88, 0.82, and 0.90 for depression, anxiety, and stress subscales, respectively. ^[22],[23] [Table 1] provides an overview of the measures.

Table 1: Overview of measures in questionnaires

Click here to view

Procedure

The contractual agreement is one of the constituents of evaluation program based on CIPP model, which also determines its success, as a result all the subjects are chosen at the beginning of the program. The project evaluation main points are explained to the subjects (including researchers, an external evaluator, Education Ministry advisors, the Education Office of an Assistant experts and heads) through holding training sessions and they reach an agreement on the stages and indicators needed for the program. Sampling is done from the mentioned schools according to the given information, and the control group students are also chosen according to the matching method.

In next stage, the study is done within three parallel parts and simultaneously. In the first part, an irrelevant evaluator starts investigating the documents and interviews the chief subjects. In the second part, researchers start assess life skills educational program among parents, advisors, and teachers through self-reporting and according to CIPP model questionnaire. Attitude and practice scales are also investigated through CIPP model questionnaire among the subjects in order to study the program's output. Teachers and advisors start completing the tools personally and in groups through sending questioners to schools and regions. The questionnaires are completed among parents in groups, and they are being evaluated through some special sessions. In the third part questioners start collecting data from schools based on the mentioned tools among the case and the control group. The questioners are informed about data collection principles (including training sessions' management and discipline, presenting the tools, reading guidelines, completing the tools, and data collection methods) through a common session.

Data analysis

Data will be analyzed in SPSS V. 21. 0 (SPSS Inc., Chicago, IL, USA). For each group (case and control groups), age (mean), gender distribution, and residential distribution will be reported. Comparisons between groups will be tested by use of the Independent t-test, variance analysis, and Chi-square test. In comparisons, the estimated proportions and test P values will be reported.

Discussion

Evaluation is one of the planning's most important parts which gives very useful information on the educational program designing and performance, and also provides a good basis for assessing the program's function. ^[24] Studies state that most educational programs usually lack a modern, coherent evaluation due to its complication that makes it necessary to use specific methods to evaluate educational programs. CIPP model is one of the most effective and systematic models among evaluation ones. ^[25] This model is associated with concepts back to experimental goals, tests, and projects. What makes this model special is that it facilitate logical decision-making and program's evaluation. ^[12] Previous studies also show that an extensive evaluation can only be useful when it has a particular structure, disciplined, and objected. ^[10],[26]

According to the current evaluation program, researchers can evaluate the program's context, inputs, process, and product through a systematic view and define its strengths and weaknesses which lead to decision and policy making in planning major levels so that demonstrators and executives are able to plan and make decisions about the educational goals, inputs, processes, and products preserve, desistance or revision, correctly and expertly.

Researches also demonstrate that evaluation based on CIPP model can play an important role in improving the educational program's quality and effectiveness.

The main limitation in this study is lack of cooperation due to the concerns of possible outcomes, which can affect the study results, therefore, we try to detect fully the chief subjects and explain the whole evaluation plan to them.

Previous studies showed that parents' educational, and socioeconomic status, and also schools' quality may affect educational factors, ^{[26],[27],[28]} therefore, selection bias is one of this study's external validity serious threats if there is a significant difference between the case and control groups in terms of demographic and other effective factors.

In this study, we tried to match groups according to the previous studies and through detecting all the effective factors. The time interval spent since the beginning of the program and also a lack of parents and advisors 'involvement may have led to recall bias in the study. Furthermore, official defects and lack of electronic document registration is another setback in this study. The self-reporting method by the subjects and completing the assessment checklist by an external advisor are two methods used to prevent a recall-based conclusion and remove the probable defect of document registration.

Eventually, this study aims at forming a systematically new framework to evaluate life skills educational program all over the country, furthermore, adjusting and improving the program's quality and prevent wasting time through recognizing the program's strengths and weaknesses, and also defections besides studying the result's long-term effectiveness.

Acknowledgment

This study was granted by Isfahan University of Medical Sciences, Isfahan, Iran. This was a MD thesis approved in the school of medicine, Isfahan University of Medical Sciences (thesis no: 393630).

Financial support and sponsorship

Isfahan University of Medical Sciences, Isfahan, Iran.

Conflicts of interest

There are no conflicts of interest.

References

1.	Lynch K. Guidelines on Promoting Positive Mental Health and Suicide Prevention in Post-primary Schools Published-Lynch\| Kathleen Lynch TD\| The Labour Party; 2013.
2.	Collins S, Woolfson LM, Durkin K. Effects on coping skills and anxiety of a universal school-based mental health intervention delivered in Scottish primary schools. Sch Psychol Int 2013;1:16.
3.	Lieske J, Swearer S, Berry B. Mental health and rural schools: An integrated approach with primary care. Handbook of Culturally Responsive School Mental Health. Verlag New York: Springer; 2013. p. 147-55.
4.	Bühler A, Schröder E, Silbereisen RK. The role of life skills promotion in substance abuse prevention: A mediation analysis. Health Educ Res 2008;23:621-32.
5.	Choque-Larrauri R, Chirinos-Caceres JL. Determining the efficacy of a high-school life-skills′ programme in Huancavelica, Peru. Rev Salud Publica (Bogota) 2009;11:169-81.
6.	Minor S, Schroder C, Heyland D. Using the intensive care unit to teach end-of-life skills to rotating junior residents. Am J Surg 2009;197:814-9.
7.	Srikala B, Kishore KK. Empowering adolescents with life skills education in schools-School mental health program: Does it work? Indian J Psychiatry 2010;52:344-9. [PUBMED]
8.	Spaeth M, Weichold K, Silbereisen RK, Wiesner M. Examining the differential effectiveness of a life skills program (IPSY) on alcohol use trajectories in early adolescence. J Consult Clin Psychol 2010;78:334-48.
9.	Karmi F, Fekri S, Saeidian N. Life skills among high school students in Takab City. J Educ Leadersh Adm 2012;6:53-67. [In Persian].
10.	Mertens DM. Research and Evaluation in Education and Psychology. California: SAGE Publications, Inc: Sage; 2014.
11.	Smith EA, Swisher JD, Vicary JR, Bechtel LJ, Minner D, Henry KL, et al. Evaluation of life Skills training and infused-life Skills Training in a rural setting: Outcomes at two years. J Alcohol Drug Educ 2004;48:51.
12.	Stufflebeam DL. The relevance of the CIPP evaluation model for educational accountability. The Annual meeting of the American Association of School Administrators 1971. p. 1-30.
13.	Stufflebeam D. The CIPP model for evaluation. Evaluation Models. Netherlands: Springer; 2002. p. 279-317.
14.	Stufflebeam DL. The 21 ^st century CIPP model. Evaluation Roots. California: SAGE Publications, Inc; 2004. p. 245-66.
15.	Stufflebeam DL. The CIPP model for program evaluation. Evaluation Models. Netherlands: Springer; 1983. p. 117-41.
16.	Zhang G, Zeller N, Griffith R, Metcalf D, Williams J, Shea C, et al. Using the context, input, process, and product evaluation model (CIPP) as a comprehensive framework to guide the planning, implementation, and assessment of service-learning programs. J High Educ Outreach Engagem 2011;15:57-84.
17.	Nicholson T. Using the CIPP model to evaluate reading instruction. J Read 1989;32:312-8.
18.	Randall RS. An operational application of the CIPP model for evaluation. Educ Technol 1969;9:40-4.
19.	Stufflebeam DL. CIPP evaluation model checklist. Western Michigan University The Evaluation Centre Retrieved June, 2007;2:2009.
20.	The-Colorado-Trust-and-National-Research-Center. The After-School Initiative′s Toolkit for Evaluating Positive Youth Development. Denver: The Colorado Trust; 2004.
21.	Crawford JR, Henry JD. The Depression Anxiety Stress Scales (DASS): Normative data and latent structure in a large non-clinical sample. Br J Clin Psychol 2003;42 (Pt 2):111-31.
22.	Sahebi A, Asghari M, Salari R. Validation of Depression Anxiety and Stress Scale (DASS-21) for an Iranian population. Iran Psychol 2005;4:299-313.
23.	Norton PJ. Depression Anxiety and Stress Scales (DASS-21): Psychometric analysis across four racial groups. Anxiety Stress Coping 2007;20:253-65.
24.	Heimlich JE. Environmental education evaluation: Reinterpreting education as a strategy for meeting mission. Eval Program Plann 2010;33:180-5.
25.	Shams B, Golshiri P, Najimi A. The evaluation of Mothers′ participation project in children′s growth and development process: Using the CIPP evaluation model. J Educ Health Promot 2013;2:21.
26.	Marschark M, Shaver DM, Nagle KM, Newman LA. Predicting the academic achievement of deaf and hard-of-hearing students from individual, household, communication, and educational factors. Except Child 2015; 81:350-69.
27.	Booth A, Dunn JF. Family-School Links: How Do they Affect Educational Outcomes? New York: Routledge; 2013.
28.	Paula JS, Leite IC, Almeida AB, Ambrosano GM, Pereira AC, Mialhe FL. The influence of oral health conditions, socioeconomic status and home environment factors on schoolchildren′s self-perception of quality of life. Health Qual Life Outcomes 2012;10:6.