Research, Evaluation, & Assessment
The Research Group
The Research Group at the Lawrence Hall of Science delivers a full spectrum of services to provide evidence and insights that foster high-quality, innovative learning experiences in science and mathematics that are equitable, inclusive, and culturally relevant.
We provide independent research services. In addition, we partner on national initiatives that support the development and implementation of Science, Technology, Engineering, and Mathematics (STEM) learning opportunities for both formal and informal education needs.
Research
Comprehensive applied research; rapid research studies; research to inform program design.
Evaluation
Full-service evaluation; tailored evaluation services; customized evaluation tool kits; convening expert peer or scientific reviews.
Assessment
Design, development, scoring, and validation of specialized and curriculum-aligned assessment.
Policy Studies
Studies of education policy and implementation; policy briefs; presentations to both decision– and policy-makers; customized convenings and symposia.
Our Work
At the Research Group, our work spans various learning environments (both in-school and out-of-school time) and content areas, such as Environmental Education, Professional Learning, Computational and Digital Learning, Engineering, and Robotics.
Our Team
Members of our team have considerable experience and expertise in both qualitative and quantitative research methods and have amassed deep content knowledge in science and mathematics.
Science Learning Activation Lab
The Lawrence Hall of Science is a key partner in the Science Learning Activation Lab—a national research and design effort focused on dramatically strengthening science learning throughout the United States.
Contact us today for more information and to see how we can help!
Research Services
Our primary research goal is to generate findings that can help promote powerful learning opportunities for all students in Science, Technology, Engineering, and Mathematics (STEM). We emphasize research that focuses on understanding innovations in STEM fields, especially those that aim to reach historically underserved and underrepresented populations. Our work yields evidence and insights that are both relevant and useful to educators, policy-makers, and researchers. We explicitly connect research findings with implications for each of these fields.
Expertise
Our interdisciplinary research team comprises learning scientists, policy researchers, sociologists, psychologists, anthropologists, political scientists, evaluators, and methodologists. Together, we have amassed considerable experience and expertise in qualitative and quantitative research methods as well as deep content knowledge in STEM. At times, we augment our expertise by partnering with faculty and researchers at the UC Berkeley Graduate School of Education, as well as with colleagues in regional and national organizations.
Projects and Services
Our research projects are multifaceted and are conducted in all settings in which STEM learning can take place. Among our research projects:
- The Science Learning Activation Lab
- PEEPs for PD: Project Evaluation Effectiveness Principles for Professional Development in Elementary Science Teaching
- The Strengthening Science Education in California Initiative
- Promising Practices for Teaching Algebra: Successful Teachers of African-American and Latino Students
- The Nature of Visitor Inquiry at Science Center Exhibits
- Studies of instructional materials implementation and professional development (in– and out-of-school)
In addition to our extended research projects, we stand ready to engage in rapid research efforts to support clients’ immediate needs for information or study results.
Reports and Presentations
Our researchers are highly skilled at presenting findings and insights before diverse audiences. At the Research Group, we tailor our research dissemination efforts to suit each project’s type and needs. We choose from a variety of dissemination options, including: reports (formal research reports, web-based reports, legislative reports, and the like); publications (peer-reviewed journals, educator venues, and others); research and policy briefs; and presentations to various stakeholders. We often present to project participants, leaders and developers, organizational boards, and policy-making bodies. We also present at national, state, and regional conferences such as those of the American Educational Research Association, the American Evaluation Association, the Association of Science-Technology Centers, the National Association of Research in Science Teaching, the National Science Teachers Association, and the Visitor Studies Association.
Funding
Our research is funded by multiple sources, including federal grants, private foundations, and private and public clients, as well as Lawrence Hall of Science programs. The cost of our client-based research services varies, depending on each project’s scale and scope. We work closely with project leadership to develop a scope and a budget that meet their needs.
Evaluation
Our experienced evaluation team understands that each Science, Technology, Engineering, and Mathematics (STEM) educational project or program is unique and requires customized service.
At the Research Group, we strive to provide high-quality products that meet our clients’ specific program needs and that deliver useful, evidence-based understanding for evaluative purposes, such as accountability requirements, program improvement, or strategic planning.
For almost a decade, our evaluation team has conducted evaluations across the nation and within the San Francisco Bay Area, with a commitment to serve projects at the intersection of educational equity and STEM content. We continue to play an integral role in developing as well as sustaining successful programs, devising proposals to fund and develop new programs, and improving or scaling established programs. Our client experiences have enabled our evaluation work to play a vital role in numerous efforts toward educational innovation and change.
To promote the utility of our work, we embrace an educative perspective in performing evaluations. We use collaborative approaches to encourage client and stakeholder participation in the program evaluation and the application of findings. Our educative, participatory approach can be an important part of a program’s intent to foster its own capacity for internal evaluation.
Our evaluation services include the following methods and products:
- Logic Models—Articulating the underlying program theory, whether it be for action or for change, by visually representing the inherent linkages of program inputs, outputs, and outcomes in the short, medium, or long term.
- Empirical Study Design—Utilizing multiple research methods, modes, and theories to illuminate and assess the nuanced effects and meaning of a program. Depending on the study scope, our proposed designs (which may include mixed-method, quasi-experimental, or case study formats) employ one or more of the following data collection methods:
- Interview
- Observation
- Survey
- Document analysis
- Panel Review—Leveraging the knowledge and expertise of internal as well as external stakeholders to verify and inform program trajectory and to validate evaluation findings at strategic points during a program evaluation.
- Technical Reports and Briefs—Attending to the needs of the program, the sponsor, and the stakeholders, we produce a variety of report types that ultimately promote the effective use of evaluation findings. Reporting styles include full technical reports, briefing memoranda, visual software presentations, research literature reviews, and policy briefs, among others.
Our Work
Our evaluation services have been implemented across numerous settings and levels of STEM education. Our current and recent evaluation work consists of:
- Full Option Science System (FOSS) Leadership Academy, a partnership with Wested/K–12 Alliance to build science leadership among California elementary school teachers
- Center for Ocean Sciences Education Excellence California (COSEE-CA)
- FOSS Multimedia
- Kepler Mission Education and Public Outreach
- Project Exploration
- San Francisco Bay National Estuarine Research Reserve
- Science and Math Informal Learning Educators (SMILE), a digital library of STEM activities from various science centers
- Aquarium of the Pacific, museum education exhibits
Funding
Our evaluation projects are funded by numerous sources, including federal grants, private foundations, private and public organizations, and even programs within the Lawrence Hall of Science itself. The pricing of our evaluation services varies in relation to the scope of work. We consult intensively with our clients to develop a plan that will best address the purpose and budget for the evaluation.
Assessment
We provide a range of assessment services for various projects and organizations. Our assessment work is multifaceted and specifically designed for the learning setting in which Science, Technology, Education, and Mathematics (STEM) education takes place. Our services may include any or all of the following:
- Designing assessment systems and support materials
- Scoring a system design with exemplar identification
- Developing items and scaling them
- Scanning/scoring fixed-choice responses
- Scoring constructed responses
- Gathering and evaluating validity evidence (testing construct validity)
- Performing data analysis and supplying suitable psychometrics
- Reporting out
Our Work
The following highlights our current work:
- Ocean Literacy Survey—Center for Ocean Science
- Excellence-California (COSEE)
- The Science Learning Activation Lab
- BaySci: A Partnership for Bay Area Education
- Advancing Collaboration for Equity in Science (ACES)
- Full Option Science System (FOSS)—grades 2–8
Past projects include the following:
- Seeds of Science/Roots of Reading—grades 2–5
- Science Education for Public Understanding Program (SEPUP)
- Assessing Science Knowledge (ASK)—grades 2–8
- Science Sequences—Space Science Sequence and Ocean Science Sequence—grades 3–8
- We also have been involved in developing assessment materials and systems for many external clients, including entire school districts, as well as in developing projects to improve instructional materials.
Expertise
Our approach involves a principled assessment design, which focuses on making a match between client measurement needs, the instrument, and the types of questions that should be asked. Our assessment team offers expertise in all aspects of the design and use of subject-matter assessments: item, scale, and test development; classroom implementation; data management and analysis; reporting; and revision of measures. We offer additional expertise in developing, implementing, and analyzing assessments that assess the breadth and depth of students' knowledge, skills, and dispositions. Our staff has garnered experience at all levels of education from K through grade 16, in numerous types of learning environments, and with students and teachers at all levels. We are experienced in the assessment of both issue-oriented science and the integrated instruction of science and literacy.
Funding
Assessment development and research is funded by multiple sources, including federal grants, private foundations, private and public clients, and the Lawrence Hall of Science programs. The cost of our client-based assessment services varies, depending on the scale and scope of the project. We work closely with project leadership to customize the work scope and budget to meet project needs.
Policy Studies
The Research Group provides policy research for numerous educational agencies and nonprofit foundations. Our focus is on science education, yet it encompasses all elements of STEM (Science, Technology, Engineering, and Mathematics) education. We take a participatory approach to education policy research, using both quantitative and qualitative methods. We have conducted studies ranging in design from exploratory to experimental, and using data collection approaches that run the gamut from medium– or large-scale surveys to deep ethnography.
As a unit of the Lawrence Hall of Science at UC Berkeley, the Research Group possesses the resources and the capacity to study, analyze, and evaluate education policy for local, state, and national governments, as well as for nongovernmental organizations. We can assist agencies and funding organizations alike as they strive to articulate policies, evaluate a portfolio of programs, and understand how the policies they set influence practice on the ground.
Approach
We provide nonpartisan information, research, and evaluation on education policy-related issues to policymakers and other education stakeholders, to support their making well-informed decisions. We accomplish these aims by conducting research across a variety of settings and by providing evaluation expertise and services to diverse agencies, organizations, and funding agencies. We share important findings through dissemination strategies tailored to the needs of specific audiences and project goals.
Expertise
Our policy studies team has amassed expertise and experience in many fields of education policy, including informal science education, professional development projects in science and technology, youth development policy, and policies surrounding equal access to high-quality and higher levels of education. We employ a diverse set of veteran policy researchers, who bring to the group new ways of thinking about policy and implementation.
Our Current Projects
Policy studies is a new and rapidly growing field for the Research Group. Among our recent and current projects:
- A large-scale research project looking into the diverse conditions of science education in California elementary and middle schools. This project focuses on the policies and structures leading to those conditions, particularly the role of statewide and federal accountability systems.
- Evaluations of multiple support programs designed to encourage students from underrepresented groups to enroll and remain in higher education programs in the life sciences.
- As part of the Science Learning Activation Lab, a retrospective study looking at the barriers and supports that youth typically encounter on their way to potentially becoming scientists.
- Projects that examine the barriers that many teachers face when endeavoring to integrate science with literacy or English Language Development strategies, plus projects that identify which structures and supports will best help them provide high-quality integration for their students.
Funding
Our policy studies work is funded by multiple sources, including federal grants and private foundations.
Our Work
Schoolyard Scientists/East Bay Academy for Young Scientists
Based at the Lawrence Hall of Science at UC Berkeley, the East Bay Academy for Young Scientists (EBAYS) serves youth from local East Bay communities where schools are under-resourced and have a large proportion of students of color. EBAYS engages youth in innovative, hands-on science research investigations that examine problems of soil, water, and air quality within their communities. It then develops new knowledge through innovative research that can be widely used in community struggles for environmental and social justice. Since 2013, for example, the Research Group has led various research and evaluation studies to examine the implementation and impact of the EBAYS program, which most recently was presented at the American Education Research Association 2019 Annual Meeting, announcing programs such as:
Clean Energy Literacy and Leadership- Engaging Youth in and After School (PDF, 143KB )
Fostering Environmental Activism Through Community Based Research Investigations (PDF, 123KB )
Currently, with funding from the National Science Foundation (NSF DRK12 #1720585), the Research Group is leading a four-year study, “Schoolyard Scientists in collaboration with EBAYS.” In the study, the Research Group works in close partnership with local students, teachers, school districts, and EBAYS itself to develop, implement, and examine participatory science-learning experiences for youth living in urban communities around the San Francisco Bay Area, particularly the East Bay. The study examines the impact of participatory science research (PSR) activities on youths’ interests in and identification with STEM, and further explores how such experiences support student learning.
BEETLES
Better Environmental Education Teaching, Learning, Expertise & Sharing (BEETLES) is a program affiliated with the Lawrence Hall of Science that offers professional learning techniques and develops curricula that support nature-based, student-centered science learning experiences in various outdoor science programs. The Research Group has served as a research and evaluation partner to BEETLES for several years. From 2013 to 2015, we conducted an evaluation study, funded by the S. D. Bechtel, Jr. Foundation, to examine the implementation of the BEETLES professional learning model and to better understand how various outdoor science programs have used the model to support professional learning among their instructional staff.
BEETLES: Final Evaluation Report (PDF, 3.4MB )
We currently are working on a two-phase research study, funded by the National Science Foundation (AISL 1612512), to examine, first, the ways in which BEETLES supports the design and facilitation of outdoor science learning experiences and, second, the impact of such ventures on students’ dispositions related to science as well as overall environmental literacy.
Activation Lab
The Learning Activation Lab is a national research and design effort to dramatically strengthen learning in the United States and beyond. Our goal is to first explore and then demonstrate how educators can activate children’s and young adults’ interest and curious minds in ways that can ignite their persistent engagement in learning and innovation. Our work spans multiple disciplines and considers the combinations of dispositions, skills, and knowledge that can best position individuals for success in learning within those disciplines.
The Research Group continues to build on the research of the Learning Activation Lab, examining the ways in which learning experiences can activate interest and curiosity among youth. Further, that lab has created a web-based toolkit, called ActApp to support organizations, researchers, and evaluators in using Activation as a way to measure programmatic outcomes and to understand the experience of their learners.
Reflecting on Practice
Reflecting on Practice (RoP) is a professional learning program designed to build participants’ understanding of best practices for teaching in informal learning environments. RoP is devised for institutions to implement it with their own staff and volunteers, at their own pace. The Research Group has served as an evaluation partner to RoP since 2011.
We are currently conducting an evaluation study, funded by the National Science Foundation (NSF AISL 1612515), to examine how best to disseminate RoP via two coaching workshop models: either two– to three-day coaching workshops that take place in person, or “blended” coaching workshops combined with a one-day in-person meeting that subsequently takes place online. The evaluation considers which factors are required for an institution to implement RoP.
FIRST
For Inspiration and Recognition of Science & Technology (FIRST®) is a national and international nonprofit organization that seeks to support learning and positive attitudes toward Science, Technology, Engineering, and Math (STEM) subjects, as well as to enhance 21st-century skills such as teamwork, cooperation, and critical thinking. To achieve these goals, FIRST offers four engineering and technology programs for youth, ranging from K through grade 12: FIRST® LEGO® League Jr., FIRST® LEGO® League, FIRST® Technology Challenge, and FIRST® Robotics Competition.
The Research Group has served as an evaluation partner to FIRST since 2012. Through this partnership, we have examined the implementation and outcomes of numerous FIRST programs and initiatives. Most recently, we completed a three-year study of the FIRST LEGO League Jr. Season Pass Model—an effort to broaden participation in its programs by reducing registration costs for larger, primarily school-based sites. The evaluation examined the implementation and outcomes of the Season Pass Model in institutions in the United States, Canada, and Mexico.
FIRST® LEGO® League Jr. Season Pass Model Evaluation - Final Report (PDF, 1.7MB)
Working Towards Equitable Organizations in Environmental Education
Working Towards Equitable Organizations Workshop Series is a collaboration between the Lawrence Hall of Science and an organization called Youth Outside, which aims to support California-based residential outdoor science organizations as they strive to build capacity to foster equitable, inclusive, and culturally relevant work environments and organizations for educators of color. The Research Group serves as the evaluation partner to examine the design, development, and implementation of the pilot professional learning model. Through this work we conducted a study to examine how outdoor science programs operationalize equity, inclusion, and diversity in the work environment, so as to identify which tools and strategies might better support organizational change. This work has lead to a practitioners brief and has been presented at various conferences, including the American Evaluation Association Annual Meeting, the Culturally Responsive Evaluation and Assessment 5th Annual Conference, and the North American Association for Environmental Education and the Children and Nature Network International Conference.
Examining Equitable and Inclusive Work Environments in Environmental Education (PDF, 3.9MB)
Positioning Youth for Success in Science: Studying the Malleability and Impact of Computational Thinking for Science
In collaboration with the Learning Design Group at the Lawrence Hall of Science, the Research Group is engaging in a three-year project (funded by NSF STEM+C #1838992) that will investigate computational thinking as both an input into and an outcome of science learning. After synthesizing a variety of frameworks and definitions of computational thinking (CT) to define the aspects of CT that best position youth specifically for learning science, we are testing whether this new construct, called computational thinking for science (CT-S), prepares youth from diverse backgrounds for achieving success with their science learning in technology-rich classrooms. We are specifically investigating whether CT-S is valuable above and beyond the previously identified dimensions of Science Learning Activation (that is: fascination, values, competency beliefs, and scientific sensemaking), each of which has been shown to enable success in science learning during the middle school years. We are also investigating the relationship between CT-S and the development of STEM career preferences. The study is situated within the Amplify Science Middle School curriculum, also developed by the Learning Design Group. The project includes measurement development, validity testing, and a one-year longitudinal study to explore how CT-S develops; how such development varies, based on the content domain in which it is emphasized; and which in-school and out-of-school experiences are most associated with increases in CT-S.
Contact Us
Please reach out to us if you have any questions or would like further information.
The Research Group
The Lawrence Hall of Science
University of California, Berkeley
1 Centennial Drive #5200
Berkeley, CA 94720-5200
510-642-5793
researchgroup@berkeley.edu
