Generosity error (too easy, grade inflation)
Giving everybody a higher score, possible incompetence on behalf of teacher (everyone gets high marks)
Severity error
Too hard, no perfect papers, always critical, grades harder, expectations are high for the group, possibly due to expertise of teacher/judge/critic (everybody gets poor marks)
Central tendency
rating everyone about average
sports, P.E.
Halo Effect
general impresion of individual (positive or negative) influences an individual rating
judging merit based on your "older brother" or "older sister" - stereotypes?
Logical error
rating alike or different based on the belief that factors are related (e.g. studious and able)
overrate gifted students because they are gifted or underrate them because they're nerdy
Effective Rating Review
- focus on educationally significant outcomes
- characteristics should be directly observable (how can you rate what you can't see?)
- clearly define key points on scale
- select the most appropriate type of instrument (checklists, rubrics, rating scales)
- if possible, rate all performance on one task before going on to the next
- when possible, rate performances without knowing the raters name
- if the assessment has significant impact, several ratings should be used (multiple raters on essays for ACT)
- example practice [7-22, 7-23]
ALTERNATIVE ASSESSMENTS
Portfolio assessment:
* A Purposeful collection of student work that exhibits the student's efforts, progress, or achievement in a given area
* Can be a maximum performance or typical performance assessment
* Assessment purpose can be formative or summative; but is also considered to be an instructional activity
Should include
* Student participation in selection content
* Criteria for selection
* Criteria for judging merit
* Evidence of self-reflection by student
* Eliminate biases by double-blind rating system
Potential Strengths:
- fosters self evaluation skills
- communication/evidence/feedback (for parents/students)
Potential Weaknesses
- time consuming, tendency to busywork
- need guidance, scoring issues (freedom vs. standardization), comparability problems
Concept Maps [see9-48]
Instructional purpose
- organize knowledge
- connect concepts
- built schema - Piaget (assimilation, accomodation)
Evaluate Purpose
Potential Strengths
- learning tool for building understanding
- evidence of understanidng
Potential Weaknesses
- time consuming, completeness issues
- difficult to interpret and score (self-presentation, explanation)
Components
- nodes (concepts)
- links (connections)
- link phrases
Are all learning outcomes measurable using objective paper and pencil tests? No! non-cognitive outcomes (Affective domains)
How would you measure:
student engagement
work ethic
socially acceptable behavior
appreciation for...
social adjustment
character development?
state vs. trait (condition vs. chronic)
attitude, anxiety, interest, value, locus of control, academic self esteem / self concept
Our goal is to measure those things that are more stable
Don't waste time measuring things that are temporary
Triangulation
Observation
Purpose is to view student behavior in natural/typical setting as part of an assessment plan to measure learning outcomes that cannot be measured directly in another way.
Observation
structured vs. unstructured
- looking for specific behaviors
- seeing all, whatever there is to see
Unstructured observation are typically less useful as part of an assessment plan but can add insightful contextual information
Subjectivity vs. Professional Judgment
- bias, misinterpretation, partial, or sketchy record
- planned, prolonged, engagement, outcome specific
Subjectivity is not a negative aspect of assessment if it is based on expert opinion
It becomes sterile if we don't consider all of the aspects of it
Observational / Anecdotal Records
- determine what to observe (focus)
- important learning outcome
Multiple Observations (Prolonged engagement)
When Recording
- make the record ASAP
- limit record to brief single incidents
- keep factual and interpretive records separate
- record both positive and negative
- check for bias [see 319]
Peer appraisal
- useful when assessing personal-social development outcomes (e.g. leadership, concern, group work skills)
- Supplements teacher observations (guess-who technique)
Tuesday, February 23, 2010
Wednesday, December 9, 2009
9 Dec 2009: Putting this course to work for me
I am grateful for the opportunity to learn in this participatory, dialogue-based course.
I did more work for this class than all my others combined. This class offered very hands-on learning that was immediately applicable to my personal, academic, and career goals and plans.
What did I learn that I will put to work for me?
* Do the literature review early on and extensively (this includes emailing/talking to experts).
* Identify a critical question of which I'm prepared to become an expert in that key area.
* Research the donor/organization well and only proceed with a match.
* "Do Budget first," Wiley said. (Before narrative, justification, resumes, methodologies.)
* Work with team regularly. (Sort out roles, responsibilities, expectations.) Carve & share work.
* Have that crucial conversation. (Kicking people off the team.)
* If I hadn't had this class, I would not have: met with Aaron, met with ORCA, met with Dean Young, collaborated intimately with 4 professors, nor submitted the grant proposal to 3ie.
I did more work for this class than all my others combined. This class offered very hands-on learning that was immediately applicable to my personal, academic, and career goals and plans.
What did I learn that I will put to work for me?
* Do the literature review early on and extensively (this includes emailing/talking to experts).
* Identify a critical question of which I'm prepared to become an expert in that key area.
* Research the donor/organization well and only proceed with a match.
* "Do Budget first," Wiley said. (Before narrative, justification, resumes, methodologies.)
* Work with team regularly. (Sort out roles, responsibilities, expectations.) Carve & share work.
* Have that crucial conversation. (Kicking people off the team.)
* If I hadn't had this class, I would not have: met with Aaron, met with ORCA, met with Dean Young, collaborated intimately with 4 professors, nor submitted the grant proposal to 3ie.
Thursday, December 3, 2009
3 Dec 2009: FUNNEL & OTHER PARTS
DESCRIPTION OF OC4D INTERVENTION
Eliminating various forms of poverty is directly linked to improving opportunities for education in the developing world (UNDP, 2009). Despite this, over a billion people are still denied educational opportunity (OECD, 2009).
Rural poverty is exacerbated by significant obstacles (geographic, political, economic) which impede dissemination of information (HLCIT, 2009). If made accessible, this relevant knowledge could save millions and drastically improve well-being (WHO, 2009). 21st century Information Communication Technologies (ICT) add fresh insights to improving such crises.
Open Content for Development (OC4D) is designed to close this gap between rural people and access to critical information for improved problem solving. OC4D is a repository of open (non-commercial, shareable, localizable) educational resources (OER) containing basic education tools created by locals for locals in developing countries.
ProLiteracy Worldwide (PLW) facilitated South-South mentoring and focus group discussions during 1990-2000 in order to aggregate the most important critical content tools from sites facing similar dilemmas. OC4D seeks to increase access to relevant content for problem solving in developing countries by bolstering capacity of non-formal educators through localizable OER made available through existing rural technology centres.
OC4D builds on the OER principles to enable strategic, scalable and sustainable return-on-investment by: encouraging governments and philanthropists to freely release all funded content into the public domain; support NGOs to share content openly with each other; encourage educational institutions and research organizations to share content openly.
OC4D will sustain communities through a participatory, continuously growing, digital library akin to Wikipedia.
OC4D is the product of ProLiteracy in partnership with the Community Development Network (based in Oxford, UK). OC4D was piloted Summer 2009; the beta launch of OC4D is slated for May 2010 in Nepal. A second iteration is now in process with technical support from the OER Foundation (funded by Hewlett Foundation), WikiEducator, Community Development Network and Brigham Young University.
OC4D Components:
OC4D provides a localized content base freely available to the public. Key topics include: infectious diseases, health/hygiene, HIV/AIDS education, micro-enterprise, environmental conservation, and conflict resolution.
OC4D is built from this quality content (grassroots lesson-plans, codes, dialogue triggers, clarifying content) harvested from ProLiteracy Worldwide’s 70 partner NGOs located in 50 countries of Asia, Nepal, Latin America and Middle East.
Training tools (text/audio/video) are also available to build capacity of literacy facilitators and other change agents (e.g. health workers, agriculture extension agents, microloan officers) so they can better localize and share knowledge.
The content repository and training tools are available online and offline (flashdrives, CDs) for communities where connectivity is unreliable or nonexistent. OC4D will be advertised by facilitators and managers of rural telecentres to encourage use by teachers and learners across Nepal.
======
EVALUATION QUESTIONS
The central research question is: How does openness impact access to knowledge in Himalayan villages?
By these terms we mean:
Open = freely available, localizable, shareable
Access = available, accessible, acceptable and adaptable (Right to Education principles)
Knowledge = information easily understood and applied in daily problem solving
In general, this evaluation seeks to understand what impact the OC4D program might have in rural Himalayan villages that would not have occurred without the OC4D intervention with particular regard to:
(a) access to information through the OC4D interface (Component I)?
(b) capacity-building of non-formal educators through the OC4D program (Component II)?
(c) localization of OC4D content for improved problem solving in rural villages (Component III)?
Component I: Does the intervention improve access to educational information users need to solve local problems? Success indicators would include changes in terms of the following outcomes: quantity of hits on OC4D site, time on informational pages, demographics of user, frequency of use, and the breadth/depth of use.
Component II: Does the intervention enhance professional development for non-formal educators? Success indicators would include the degree to which the intervention improves: teacher ability to find relevant content and help participants find relevant content, teacher’s knowledge of how to use OC4D, and the ability of teachers to teach in a manner tailored for those whom they serve.
Component III: Does OC4D increase the amount of localized materials for improved problem solving? Success indicators would include the: relevance of content found to problems in community, availability of content users need for problem-solving, content matches congruent with user’s needs, and the degree to which OC4D changes knowledge, attitudes, and behaviour.
=====
SUMMARY
One of the best ways to overcome poverty is to increase and improve opportunities for education. However, accessing quality information in remote area of the world like those found in Nepal can be a challenge. While 21st century Information Communication Tools (ICTs) have accelerated the ability of Nepalese villagers to overcome some development setbacks, they still face considerable difficulty accessing the information they need to improve their situation. Many Nepalese can only access this critical information by trekking to remote information centres.
In an effort to improve access to information for the Nepalese people, the Nepal Government’s High Level Commission for Information Technology (HLCIT) and Community Development Network (CDN), propose that this study occur in tandem with the beta launch of the Open Content for Development (OC4D) initiative in May 2010.
This prospective study will be conducted in partnership with researchers from Nepal’s Research Centre for Educational Innovation and Development, (CERID), and experts at Brigham Young University (Access to Knowledge Initiative, A2K). This five year counterfactual study will track the impact of Open Educational Resources (OER) in Himalayan communities.
The OC4D initiative builds upon the nascent Open Educational Resources (OER) movement and emphasizes access to critical content focused on improving problem solving in developing countries. If deemed viable after the beta launch in Nepal, this high-quality but low-cost and fiscally scale-able model will be launched regionally and globally to the public and to ProLiteracy’s 75 NGO partners in 50 countries worldwide.
This impact evaluation will be the first of its kind assessing impact of OER in developing countries. This comprehensive mixed-method study will assess the viability of OC4D to increase access to relevant content for problem solving in remote villages. This prospective study will assess impact of OC4D to build capacity of non-formal teachers (e.g. literacy facilitators, health workers, agricultural agents, microloan officers) to better do their jobs.
Deliberately designed impact studies such as this can provide exactly the standard of evidence needed to advocate effectively with governments and donor agencies about the best ways to invest scarce resources in OER. Given the lack of evidence on OER programs in developing countries, this evaluation will set a benchmark in understanding what changes (in access, improved behaviour, and localizability) are afforded through OER that would not have occurred
without the OER intervention. Finally, it will illuminate unintended effects that OER have on intervention communities (in terms of social norms, cultural capital, economic opportunities, well-being, health behaviours, and education trends).
=====
JUSTIFICATION
To our knowledge, this evaluation is the first prospective impact evaluation of OER to be conducted in Nepal. Possibly the most important justification for this project is the pilot study and needs analysis conducted by Brigham Young University (BYU) and Community Development Network (CDN) in Nepal from May to August 2009. Based on recommendations from stakeholders in Nepal and situation analysis reports (CDN, 2008; CDN, 2009) this type of project is not only viable but important to improving access to information need in many areas of Nepal.
Although OER is believed to benefit individuals in developing countries, extension of tools like OC4D to lower-level learners is still limited (Hewlett, 2007). Many predict that OER will afford great educational benefit in areas where resources are scarce; to date, however, developing countries consume very little of OER offerings (MIT, 2009).
Education professionals may be aware that free and open source principles can benefit education, but there has been no systematic nor comprehensive approach to map and transfer those principles, or to develop new educational models and scenarios around them. The European Union-funded FLOSSCom project pioneered the first research of the open source landscape from an educational point of view, but “further research and work still remains to be done” (Välimäki, 2009).
If OER tools hold potential to revolutionize access to education in developing countries, then it is imperative to find out what impact OER have in such environments and why.
HLCIT designed rural telecentres to increase access to information for 85% of Nepalese living in remote villages where educational materials are often nonexistent (HLCIT, 2007). Since 2004, CDN has supported HLCIT in building capacity of rural information-technology centres. During this time, we have learned of the critical need for content in these centres.
Based on research conducted by HLCIT (2009), sustainability of telecentres is directly linked to mechanisms which enable access to quality content. Furthermore, OER trends indicate that sharing content openly enables strategic, scalable and sustainable return-on-investment (Hewlett, 2009). For these reasons, HLCIT and CDN partnered with ProLiteracy to develop OC4D for use in Himalayan telecentres.
The empirical evidence on the effects of OER on the life of learners in Nepal is scarce (Bhattarai, 2009; Pun, 2009). This innovative model of OER, Open Content for Development (OC4D), launched within the Himalayan context, provides important opportunities for learning the viability and cost-effectiveness of OER programs to improve access to localized content that supports problem solving in rural areas.
This study would also constitutes an opportunity for developing capacity in-country for the use of impact evaluation methods as well as bolstering ability for conducting mixed-method counterfactual assessments in the growing field of Information Comunication Technologies for Development (ICT4D). A substantial portion of this grant proposal is reserved for building capacity of graduate and undergraduate research assistants (local and international) to contribute to data collection and analysis through mentoring by evaluation experts from Nepal, India, UK, and USA.
For these reasons, Nepal’s ICT policy-making body, HLCIT, recommends the study of OC4D to bolster ICT policy directives. This is a formative time for policy development in Nepal since the new government initialized just last year (2008). This timely ex-ante study will capture baseline data from both treatment and control groups in advance of the OC4D launch, allowing for a counterfactual impact study that may inform the viability of OER in developing countries.
=====
EVALUATION DESIGN
This impact evaluation of the Open Content for Development (OC4D) project will be a collaborative effort between the High Level Commission for Information Technology (HLCIT), the Community Development Network (CDN), Brigham Young University (BYU), and Nepal’s Centre for Educational Research, Innovation and Development (CERID).
As explained in the section outlining the evaluation questions, the primary purpose for the evaluation will be to better understand what impact the OC4D program might have in Himalayan villages that would not have occurred without the OC4D intervention with particular regard to:
(a) access to information through the OC4D interface
(b) capacity-building of non-formal educators through the OC4D program, and
(c) localization of OC4D content for improved problem solving in rural villages of Nepal.
Based on recommendations from stakeholders in Nepal during pre-intervention analyses and situation analysis reports (CDN, 2008; CDN, 2009), a 5-year longitudinal study is proposed starting in May 2010.
This evaluation will use a quasi- experimental, mixed-method counterfactual design; however, the evaluation design and specific activities will be responsive to needs of the evaluation effort; modifications to the design in subsequent years may be implemented based on evaluation findings, data collection needs, and recommendations from evaluation partners.
During the first year of the project we plan to collect baseline data for all 200 telecentres in Nepal. This will be used to create comparison groups at 50 sites (25 in the intervention group and 25 in the control group). A stratified purposive sampling technique will be employed to ensure the comparison groups are similar based on important community characteristics (e.g., socio economic status, location, internet access, location). Matched sites fitting the conditions for comparison will be randomly assigned to treatment or control groups. Treatment sites will be provided with OC4D resources and training. The evaluation will focus on any attributable impact receiving the OC4D intervention might have (including any social, gender, and environmental impacts) based on the evaluation criteria (i.e., increased access, capacity-building, value of localized content).
An OC4D pilot study was conducted previously at 10 sites in Nepal as part of an exploratory preliminary needs assessment. The OC4D beta launch is scheduled for May 2010, at this time a 3 year ramp-up exposure to the OC4D intervention will be initiated at the selected intervention sites. Formative analysis, impact analysis will be conducted during this time. A summative evaluation and formal dissemination of findings will be provided during the last year of the
evaluation.
Various aspects of the project implementation and effectiveness will be evaluated through participant surveys, site visit observations, interviews, and case studies. The access to information question will be studied using surveys, observations, and interviews. The evaluation will track awareness, usage, and utility levels related to access of information.
The capacity-building of local non-formal educations at designated telecentres will be explored through observations, interviews, surveys. The value of localized content for improved problem solving will be determined through surveys, interviews, and case studies.
Case studies will be specifically designed to follow any causal chain that might exist for individuals as they access information and use it to solve problems. These measures of impacts will help us determine best practices, what works or what does not work, as well as an understanding of why and where it works. The design is also intended to generate evidence to guide successful scale-up of OER programs. Details of the data collection activities are provided in the data collection section.
Evaluation efforts will be coordinated by CDN/CERID evaluation specialists but local data collection administrators will be utilized as much as possible. Training and support of data collection contractors will be provided by affiliated partners and will be continuously supported on the ground by CERID. Quality control of activities is a main priority of ProLiteracy’s OER program.
ProLiteracy will register expenditures made in the program and provide a detailed control of its budget. All this data will be routinely analyzed by ProLiteracy staff for the quality control of its program and providing quick feedback for decision-makers. This data will enable us to estimate costs under different circumstances based on number of learners who attend telecentre training sessions in order to inform advocacy efforts with the government and eventual scale-up of the program’s interventions.
Data collected from surveys will be statistically analyzed for the purpose of comparative, profile, and descriptive analyses (e.g., uncovering systematic differences, trends) across and within the entire group and identifiable sub-groups of participants. When appropriate, descriptive statistics and analysis of variance techniques for non-parametric data will be utilized to uncover any predictive insights, group profiles, and group differences.
Qualitative methods will be used to analyze and present data from site visit observations and interviews. This will include information regarding unintended outcomes. Content analysis of qualitative data will be conducted using open coding and axial coding. Open coding
entails the initial breakdown of text into discrete conceptual categories that identify a particular phenomenon. Axial coding systematically links categories to causal conditions, context, action/interaction strategies, and consequences and is used to determine emergent themes.
=====
DATA COLLECTION
Will the research proposed in this application produce new datasets? Yes
Indicate how existing datasets have been reviewed and state why currently available datasets are inadequate for the proposed research.
Because OER is a nascent field, virtually no datasets have been collected on the use of this type of project in developing countries and nothing exists of this nature in Nepal.
No other impact evaluations of non-formal education OER programs were identified in the region. Primary data collection for this evaluation is necessary to achieve geographic coverage of the program areas across Nepal, and to synchronize the timing of data collection with program operations.
The research team conducted a thorough review of the existing literature and available datasets prior to participating in the ProLiteracy OER impact evaluation in Nepal (May-August 2009). These preliminary pre-intervention analyses suggested a need for this project. In addition, the research team has conducted extensive interviews with HLCIT, NITC, Ministry of Science
and Technology, MOES, UNESCO, UNDP and donor agencies operating in Nepal to determine availability of data sources.
All the following datasets have been reviewed: All existing data housed by HLCIT regarding Nepal’s community information initiatives including Community Learning Centres (CLC), Community TeleCentre Report (CTR), Community Library Report (CLR), UNESCO’s Community Multimedia Centre (CMC) Report, Asia Pacific Information Network (APIN) Regional and Country Reports, Nepalese National Statistics Institute (INE), UNDP Regional and Country Reports, Demographic and Health Survey of Nepal (DHS), Non-Formal Education Report (NFE), Analysis of Poverty and Social Impact of Nepal (PSIN), Survey of Poverty and Gender at the Nepalese Provinces (IPGP) and the Nepalese Agrarian Survey (NAS).
Many of the pre-tested questions of these surveys will be adapted for use in this impact evaluation we are proposing; however, no suitable options of existing databases were identified for during the preliminary needs analysis evaluation. None of these databases had the local detailed coverage at the operating districts in Nepal needed for the evaluation, nor the necessary
measures to access the impact of OER types of interventions on the non-formal educational dissemination and other processes for finding information relevant to problem-solving.
Describe the design of data collection (instruments, sample design, size, timing)
Impact Evaluation Strategy:
Starting in May 2010, ProLiteracy plans to formally launch its OC4D programme in Nepal. Concurrent with this, the following impact evaluation strategies will be employed:
TeleCentre Community Sample:
Nepal currently has 200 TeleCentre sites across the country in various locals. The evaluation will use information collected about telecentre communities as well as ProLiteracy monitoring data to create a characteristics database of Nepal’s telecentres. A stratified purposive sampling technique will be employed to select 50 similar communities that will be the focus of this study.
Potential characteristics for consideration might include socio-economic status, location, internet access, and geographic location. Matched sites fitting the conditions established for comparison will be randomly assigned to treatment or control groups. The treatment group will include 25 TeleCentre communities. The control group will include 25 similar telecentres
communities. Treatment sites will be provided with OC4D resources and training. Control group sites will only receive existing telecentres services.
Community Household Sample:
Within each of the 50 communities, a random sample of 40 households with learners between the ages 18 and 35 years will be included in household survey. A total of 2000 community household surveys (1000 in treatment communities and 1000 in control communities) will be collected annually. This annual survey is intended to provide a longitudinal look at changes in
levels of awareness and use of telecentres for the various communities.
TeleCentre Regular Users Sample:
All regular telecentre users will be asked to provide survey feedback bi-annually in June and December of each year.
In addition the manager of each of the telecentres and at least one of the non-formal education teacher (or change agent facilitators) will be interviewed bi-annually.
Instruments:
Surveys. Two types of survey will be utilized: a household survey and a regular users survey. The household survey will gather demographic information as well as telecenter awareness and usage levels. This will be used to determine changes in community trends. The regular users survey will also collect demographic data as well as evidence of general telecentre usage and value. OC4D treatment sites will also gather evidence regarding the use and value of the OC4D informational database.
Interview and observation
Interview and observation protocols and procedures will be established. Training of data collection administrators will also be conducted to ensure consistency of data collection efforts.
Case Studies
At each of the telecenters, one case study will be conducted to follow regular users from information access to utilization of the data retrieved for problem solving.
Implementation and Timing
OC4D implementation will begin in May 2010. Resources and training will be provided at each of the training centres at this time. Household surveys will be conducted annually in June. Regular users surveys, observations, and interviews will be conducted in June and December of each year. Monthly usage data will be collected by site facilitators. Case studies will begin in the
third year of the project.
We expect final impact evaluation results and dissemination activities including meetings with key stakeholders in the Nepalese government, other regional governments, as well as publication of peer reviewed articles to be published in several international journals by the end of 2015.
Eliminating various forms of poverty is directly linked to improving opportunities for education in the developing world (UNDP, 2009). Despite this, over a billion people are still denied educational opportunity (OECD, 2009).
Rural poverty is exacerbated by significant obstacles (geographic, political, economic) which impede dissemination of information (HLCIT, 2009). If made accessible, this relevant knowledge could save millions and drastically improve well-being (WHO, 2009). 21st century Information Communication Technologies (ICT) add fresh insights to improving such crises.
Open Content for Development (OC4D) is designed to close this gap between rural people and access to critical information for improved problem solving. OC4D is a repository of open (non-commercial, shareable, localizable) educational resources (OER) containing basic education tools created by locals for locals in developing countries.
ProLiteracy Worldwide (PLW) facilitated South-South mentoring and focus group discussions during 1990-2000 in order to aggregate the most important critical content tools from sites facing similar dilemmas. OC4D seeks to increase access to relevant content for problem solving in developing countries by bolstering capacity of non-formal educators through localizable OER made available through existing rural technology centres.
OC4D builds on the OER principles to enable strategic, scalable and sustainable return-on-investment by: encouraging governments and philanthropists to freely release all funded content into the public domain; support NGOs to share content openly with each other; encourage educational institutions and research organizations to share content openly.
OC4D will sustain communities through a participatory, continuously growing, digital library akin to Wikipedia.
OC4D is the product of ProLiteracy in partnership with the Community Development Network (based in Oxford, UK). OC4D was piloted Summer 2009; the beta launch of OC4D is slated for May 2010 in Nepal. A second iteration is now in process with technical support from the OER Foundation (funded by Hewlett Foundation), WikiEducator, Community Development Network and Brigham Young University.
OC4D Components:
OC4D provides a localized content base freely available to the public. Key topics include: infectious diseases, health/hygiene, HIV/AIDS education, micro-enterprise, environmental conservation, and conflict resolution.
OC4D is built from this quality content (grassroots lesson-plans, codes, dialogue triggers, clarifying content) harvested from ProLiteracy Worldwide’s 70 partner NGOs located in 50 countries of Asia, Nepal, Latin America and Middle East.
Training tools (text/audio/video) are also available to build capacity of literacy facilitators and other change agents (e.g. health workers, agriculture extension agents, microloan officers) so they can better localize and share knowledge.
The content repository and training tools are available online and offline (flashdrives, CDs) for communities where connectivity is unreliable or nonexistent. OC4D will be advertised by facilitators and managers of rural telecentres to encourage use by teachers and learners across Nepal.
======
EVALUATION QUESTIONS
The central research question is: How does openness impact access to knowledge in Himalayan villages?
By these terms we mean:
Open = freely available, localizable, shareable
Access = available, accessible, acceptable and adaptable (Right to Education principles)
Knowledge = information easily understood and applied in daily problem solving
In general, this evaluation seeks to understand what impact the OC4D program might have in rural Himalayan villages that would not have occurred without the OC4D intervention with particular regard to:
(a) access to information through the OC4D interface (Component I)?
(b) capacity-building of non-formal educators through the OC4D program (Component II)?
(c) localization of OC4D content for improved problem solving in rural villages (Component III)?
Component I: Does the intervention improve access to educational information users need to solve local problems? Success indicators would include changes in terms of the following outcomes: quantity of hits on OC4D site, time on informational pages, demographics of user, frequency of use, and the breadth/depth of use.
Component II: Does the intervention enhance professional development for non-formal educators? Success indicators would include the degree to which the intervention improves: teacher ability to find relevant content and help participants find relevant content, teacher’s knowledge of how to use OC4D, and the ability of teachers to teach in a manner tailored for those whom they serve.
Component III: Does OC4D increase the amount of localized materials for improved problem solving? Success indicators would include the: relevance of content found to problems in community, availability of content users need for problem-solving, content matches congruent with user’s needs, and the degree to which OC4D changes knowledge, attitudes, and behaviour.
=====
SUMMARY
One of the best ways to overcome poverty is to increase and improve opportunities for education. However, accessing quality information in remote area of the world like those found in Nepal can be a challenge. While 21st century Information Communication Tools (ICTs) have accelerated the ability of Nepalese villagers to overcome some development setbacks, they still face considerable difficulty accessing the information they need to improve their situation. Many Nepalese can only access this critical information by trekking to remote information centres.
In an effort to improve access to information for the Nepalese people, the Nepal Government’s High Level Commission for Information Technology (HLCIT) and Community Development Network (CDN), propose that this study occur in tandem with the beta launch of the Open Content for Development (OC4D) initiative in May 2010.
This prospective study will be conducted in partnership with researchers from Nepal’s Research Centre for Educational Innovation and Development, (CERID), and experts at Brigham Young University (Access to Knowledge Initiative, A2K). This five year counterfactual study will track the impact of Open Educational Resources (OER) in Himalayan communities.
The OC4D initiative builds upon the nascent Open Educational Resources (OER) movement and emphasizes access to critical content focused on improving problem solving in developing countries. If deemed viable after the beta launch in Nepal, this high-quality but low-cost and fiscally scale-able model will be launched regionally and globally to the public and to ProLiteracy’s 75 NGO partners in 50 countries worldwide.
This impact evaluation will be the first of its kind assessing impact of OER in developing countries. This comprehensive mixed-method study will assess the viability of OC4D to increase access to relevant content for problem solving in remote villages. This prospective study will assess impact of OC4D to build capacity of non-formal teachers (e.g. literacy facilitators, health workers, agricultural agents, microloan officers) to better do their jobs.
Deliberately designed impact studies such as this can provide exactly the standard of evidence needed to advocate effectively with governments and donor agencies about the best ways to invest scarce resources in OER. Given the lack of evidence on OER programs in developing countries, this evaluation will set a benchmark in understanding what changes (in access, improved behaviour, and localizability) are afforded through OER that would not have occurred
without the OER intervention. Finally, it will illuminate unintended effects that OER have on intervention communities (in terms of social norms, cultural capital, economic opportunities, well-being, health behaviours, and education trends).
=====
JUSTIFICATION
To our knowledge, this evaluation is the first prospective impact evaluation of OER to be conducted in Nepal. Possibly the most important justification for this project is the pilot study and needs analysis conducted by Brigham Young University (BYU) and Community Development Network (CDN) in Nepal from May to August 2009. Based on recommendations from stakeholders in Nepal and situation analysis reports (CDN, 2008; CDN, 2009) this type of project is not only viable but important to improving access to information need in many areas of Nepal.
Although OER is believed to benefit individuals in developing countries, extension of tools like OC4D to lower-level learners is still limited (Hewlett, 2007). Many predict that OER will afford great educational benefit in areas where resources are scarce; to date, however, developing countries consume very little of OER offerings (MIT, 2009).
Education professionals may be aware that free and open source principles can benefit education, but there has been no systematic nor comprehensive approach to map and transfer those principles, or to develop new educational models and scenarios around them. The European Union-funded FLOSSCom project pioneered the first research of the open source landscape from an educational point of view, but “further research and work still remains to be done” (Välimäki, 2009).
If OER tools hold potential to revolutionize access to education in developing countries, then it is imperative to find out what impact OER have in such environments and why.
HLCIT designed rural telecentres to increase access to information for 85% of Nepalese living in remote villages where educational materials are often nonexistent (HLCIT, 2007). Since 2004, CDN has supported HLCIT in building capacity of rural information-technology centres. During this time, we have learned of the critical need for content in these centres.
Based on research conducted by HLCIT (2009), sustainability of telecentres is directly linked to mechanisms which enable access to quality content. Furthermore, OER trends indicate that sharing content openly enables strategic, scalable and sustainable return-on-investment (Hewlett, 2009). For these reasons, HLCIT and CDN partnered with ProLiteracy to develop OC4D for use in Himalayan telecentres.
The empirical evidence on the effects of OER on the life of learners in Nepal is scarce (Bhattarai, 2009; Pun, 2009). This innovative model of OER, Open Content for Development (OC4D), launched within the Himalayan context, provides important opportunities for learning the viability and cost-effectiveness of OER programs to improve access to localized content that supports problem solving in rural areas.
This study would also constitutes an opportunity for developing capacity in-country for the use of impact evaluation methods as well as bolstering ability for conducting mixed-method counterfactual assessments in the growing field of Information Comunication Technologies for Development (ICT4D). A substantial portion of this grant proposal is reserved for building capacity of graduate and undergraduate research assistants (local and international) to contribute to data collection and analysis through mentoring by evaluation experts from Nepal, India, UK, and USA.
For these reasons, Nepal’s ICT policy-making body, HLCIT, recommends the study of OC4D to bolster ICT policy directives. This is a formative time for policy development in Nepal since the new government initialized just last year (2008). This timely ex-ante study will capture baseline data from both treatment and control groups in advance of the OC4D launch, allowing for a counterfactual impact study that may inform the viability of OER in developing countries.
=====
EVALUATION DESIGN
This impact evaluation of the Open Content for Development (OC4D) project will be a collaborative effort between the High Level Commission for Information Technology (HLCIT), the Community Development Network (CDN), Brigham Young University (BYU), and Nepal’s Centre for Educational Research, Innovation and Development (CERID).
As explained in the section outlining the evaluation questions, the primary purpose for the evaluation will be to better understand what impact the OC4D program might have in Himalayan villages that would not have occurred without the OC4D intervention with particular regard to:
(a) access to information through the OC4D interface
(b) capacity-building of non-formal educators through the OC4D program, and
(c) localization of OC4D content for improved problem solving in rural villages of Nepal.
Based on recommendations from stakeholders in Nepal during pre-intervention analyses and situation analysis reports (CDN, 2008; CDN, 2009), a 5-year longitudinal study is proposed starting in May 2010.
This evaluation will use a quasi- experimental, mixed-method counterfactual design; however, the evaluation design and specific activities will be responsive to needs of the evaluation effort; modifications to the design in subsequent years may be implemented based on evaluation findings, data collection needs, and recommendations from evaluation partners.
During the first year of the project we plan to collect baseline data for all 200 telecentres in Nepal. This will be used to create comparison groups at 50 sites (25 in the intervention group and 25 in the control group). A stratified purposive sampling technique will be employed to ensure the comparison groups are similar based on important community characteristics (e.g., socio economic status, location, internet access, location). Matched sites fitting the conditions for comparison will be randomly assigned to treatment or control groups. Treatment sites will be provided with OC4D resources and training. The evaluation will focus on any attributable impact receiving the OC4D intervention might have (including any social, gender, and environmental impacts) based on the evaluation criteria (i.e., increased access, capacity-building, value of localized content).
An OC4D pilot study was conducted previously at 10 sites in Nepal as part of an exploratory preliminary needs assessment. The OC4D beta launch is scheduled for May 2010, at this time a 3 year ramp-up exposure to the OC4D intervention will be initiated at the selected intervention sites. Formative analysis, impact analysis will be conducted during this time. A summative evaluation and formal dissemination of findings will be provided during the last year of the
evaluation.
Various aspects of the project implementation and effectiveness will be evaluated through participant surveys, site visit observations, interviews, and case studies. The access to information question will be studied using surveys, observations, and interviews. The evaluation will track awareness, usage, and utility levels related to access of information.
The capacity-building of local non-formal educations at designated telecentres will be explored through observations, interviews, surveys. The value of localized content for improved problem solving will be determined through surveys, interviews, and case studies.
Case studies will be specifically designed to follow any causal chain that might exist for individuals as they access information and use it to solve problems. These measures of impacts will help us determine best practices, what works or what does not work, as well as an understanding of why and where it works. The design is also intended to generate evidence to guide successful scale-up of OER programs. Details of the data collection activities are provided in the data collection section.
Evaluation efforts will be coordinated by CDN/CERID evaluation specialists but local data collection administrators will be utilized as much as possible. Training and support of data collection contractors will be provided by affiliated partners and will be continuously supported on the ground by CERID. Quality control of activities is a main priority of ProLiteracy’s OER program.
ProLiteracy will register expenditures made in the program and provide a detailed control of its budget. All this data will be routinely analyzed by ProLiteracy staff for the quality control of its program and providing quick feedback for decision-makers. This data will enable us to estimate costs under different circumstances based on number of learners who attend telecentre training sessions in order to inform advocacy efforts with the government and eventual scale-up of the program’s interventions.
Data collected from surveys will be statistically analyzed for the purpose of comparative, profile, and descriptive analyses (e.g., uncovering systematic differences, trends) across and within the entire group and identifiable sub-groups of participants. When appropriate, descriptive statistics and analysis of variance techniques for non-parametric data will be utilized to uncover any predictive insights, group profiles, and group differences.
Qualitative methods will be used to analyze and present data from site visit observations and interviews. This will include information regarding unintended outcomes. Content analysis of qualitative data will be conducted using open coding and axial coding. Open coding
entails the initial breakdown of text into discrete conceptual categories that identify a particular phenomenon. Axial coding systematically links categories to causal conditions, context, action/interaction strategies, and consequences and is used to determine emergent themes.
=====
DATA COLLECTION
Will the research proposed in this application produce new datasets? Yes
Indicate how existing datasets have been reviewed and state why currently available datasets are inadequate for the proposed research.
Because OER is a nascent field, virtually no datasets have been collected on the use of this type of project in developing countries and nothing exists of this nature in Nepal.
No other impact evaluations of non-formal education OER programs were identified in the region. Primary data collection for this evaluation is necessary to achieve geographic coverage of the program areas across Nepal, and to synchronize the timing of data collection with program operations.
The research team conducted a thorough review of the existing literature and available datasets prior to participating in the ProLiteracy OER impact evaluation in Nepal (May-August 2009). These preliminary pre-intervention analyses suggested a need for this project. In addition, the research team has conducted extensive interviews with HLCIT, NITC, Ministry of Science
and Technology, MOES, UNESCO, UNDP and donor agencies operating in Nepal to determine availability of data sources.
All the following datasets have been reviewed: All existing data housed by HLCIT regarding Nepal’s community information initiatives including Community Learning Centres (CLC), Community TeleCentre Report (CTR), Community Library Report (CLR), UNESCO’s Community Multimedia Centre (CMC) Report, Asia Pacific Information Network (APIN) Regional and Country Reports, Nepalese National Statistics Institute (INE), UNDP Regional and Country Reports, Demographic and Health Survey of Nepal (DHS), Non-Formal Education Report (NFE), Analysis of Poverty and Social Impact of Nepal (PSIN), Survey of Poverty and Gender at the Nepalese Provinces (IPGP) and the Nepalese Agrarian Survey (NAS).
Many of the pre-tested questions of these surveys will be adapted for use in this impact evaluation we are proposing; however, no suitable options of existing databases were identified for during the preliminary needs analysis evaluation. None of these databases had the local detailed coverage at the operating districts in Nepal needed for the evaluation, nor the necessary
measures to access the impact of OER types of interventions on the non-formal educational dissemination and other processes for finding information relevant to problem-solving.
Describe the design of data collection (instruments, sample design, size, timing)
Impact Evaluation Strategy:
Starting in May 2010, ProLiteracy plans to formally launch its OC4D programme in Nepal. Concurrent with this, the following impact evaluation strategies will be employed:
TeleCentre Community Sample:
Nepal currently has 200 TeleCentre sites across the country in various locals. The evaluation will use information collected about telecentre communities as well as ProLiteracy monitoring data to create a characteristics database of Nepal’s telecentres. A stratified purposive sampling technique will be employed to select 50 similar communities that will be the focus of this study.
Potential characteristics for consideration might include socio-economic status, location, internet access, and geographic location. Matched sites fitting the conditions established for comparison will be randomly assigned to treatment or control groups. The treatment group will include 25 TeleCentre communities. The control group will include 25 similar telecentres
communities. Treatment sites will be provided with OC4D resources and training. Control group sites will only receive existing telecentres services.
Community Household Sample:
Within each of the 50 communities, a random sample of 40 households with learners between the ages 18 and 35 years will be included in household survey. A total of 2000 community household surveys (1000 in treatment communities and 1000 in control communities) will be collected annually. This annual survey is intended to provide a longitudinal look at changes in
levels of awareness and use of telecentres for the various communities.
TeleCentre Regular Users Sample:
All regular telecentre users will be asked to provide survey feedback bi-annually in June and December of each year.
In addition the manager of each of the telecentres and at least one of the non-formal education teacher (or change agent facilitators) will be interviewed bi-annually.
Instruments:
Surveys. Two types of survey will be utilized: a household survey and a regular users survey. The household survey will gather demographic information as well as telecenter awareness and usage levels. This will be used to determine changes in community trends. The regular users survey will also collect demographic data as well as evidence of general telecentre usage and value. OC4D treatment sites will also gather evidence regarding the use and value of the OC4D informational database.
Interview and observation
Interview and observation protocols and procedures will be established. Training of data collection administrators will also be conducted to ensure consistency of data collection efforts.
Case Studies
At each of the telecenters, one case study will be conducted to follow regular users from information access to utilization of the data retrieved for problem solving.
Implementation and Timing
OC4D implementation will begin in May 2010. Resources and training will be provided at each of the training centres at this time. Household surveys will be conducted annually in June. Regular users surveys, observations, and interviews will be conducted in June and December of each year. Monthly usage data will be collected by site facilitators. Case studies will begin in the
third year of the project.
We expect final impact evaluation results and dissemination activities including meetings with key stakeholders in the Nepalese government, other regional governments, as well as publication of peer reviewed articles to be published in several international journals by the end of 2015.
3 Dec 2009: What is this all for?
* "If you're working on something that can be solved in your lifetime, you're working on something too small."
* Your dissertation is not your "magnum opus" (life's work).
* The best dissertation is a done dissertation.
* Your dissertation is not your "magnum opus" (life's work).
* The best dissertation is a done dissertation.
Tuesday, December 1, 2009
Tuesday, November 10, 2009
10 Nov 2009: Managing People
We all are capable of what matters most
Netflix Management
Loyalty is Good
- Loyalty is good as a stabilizer
- But, unlimited loyalty to a shrinking firm, or to an ineffective employee, is not what we are about
- Hard work is not directly relevant - it's about effectiveness (not effort) even though effectiveness is harder to assess than effort
Netflix Management
Loyalty is Good
- Loyalty is good as a stabilizer
- But, unlimited loyalty to a shrinking firm, or to an ineffective employee, is not what we are about
- Hard work is not directly relevant - it's about effectiveness (not effort) even though effectiveness is harder to assess than effort
Tuesday, November 3, 2009
3 Nov 2009: "I Love Technology..." (Isaku quoting Kip)
BASECAMP
Web-based tools for managing projects ($20/mo or so)
http://basecamphq.com
List of web-based tools
Web-based tools for managing projects ($20/mo or so)
http://basecamphq.com
List of web-based tools
Subscribe to:
Posts (Atom)