Museums and Schools: Journal of Museum Education 34:1 Thematic Issue [1 ed.] 9781598748208, 9781315423852

Citation preview

JOURNA~MUSEUM EDUCATION

m VOLUME 34 ~"" NUMBER 1 ~ SPRING 2009 A PUBLICATION OF THE MUSEUM EDUCATION ROUNDTABLE

Museums and Schools 3 From the Editor Elizabeth L. Maurer

5 Guest Editors' Notes: Museums and Schools Giuseppe "Pino" Monaco and Megan Wood

7 Provoking Innovation: Creating Grassroots and Intersectional Programming at Historical Organizations. Jody Blankenship

23 Partners in Process: How Museum Educators and Classroom Teachers Can Create Outstanding Results Heidi Moisan

41 Creating Communicative Scientists: A Collaboration between a Science Center, College and Science Industry Melissa Wadman, Wendy deProphetis Driscoll, and Elizabeth Kurzawa

55 Revisioning the Physical and On-line Museum: A Partnership with the Coalition of Knowledge Building Schools Lynda Kelly and Susan Groundwater-Smith

69 Keep Your Friends Close: The History of a Museum Partnership and Its Community of Teacher Learners Kris Wetterlund

79 Impact of the National History Day in Ohio Program on Students' Performances Pilot Evaluation Project Giuseppe Monaco, Bo Lu, and Megan Wood

97 Evaluation of Didactic Material Designed to Be Used by High School Art Teachers: The Use of Focus Groups and Questionnaires Adriana Mortara Almeida and Maria Helena Pires Martins

ISBN 13: 978-1-138-40491-5

First published 2009 by Left Coast Press, Inc. Published 2020 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN 605 Third Avenue, New York, NY 10017 First issued in hardback 2020

Rnutledge is an imprint of the Tqylor & Francis Group, an informa business Copyright © 2009 Museum Education Roundtable All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. journal ofMuseum Education (ISSN: 1059-8650), is published three times yearly, in the Spring, Summer,

and Fall, by Left Coast Press, Inc., in partnership with the Museum Education Roundtable. The journal publishes original papers on theory, training, and practice in the museum education field. Each issue focuses on a specific theme of interest to professional and informal museum educators, administrators, and practitioners. Submission Guidelines: The journal ofMuseum Education 0ME) welcomes the submission of original proposals for issues on theory, training, and practice in the museum education field. Topics are pro­ posed by guest editors who develop a theme, recruit authors, and coordinate communication between authors and the Editor-in-chief. Proposals and all manuscripts are subject to peer review by knowl­ edgeable scholars and professional practitioners and, if accepted, may be subject to revision. Materials submitted to ]ME should not be under consideration by other publishers, nor should they be previously published in any form. Electronic submissions of issue proposals should include a cover letter, issue ab­ stract, proposed articles and/or authors, and the guest editor's resume.JME does not accept unsolicited, single articles for publication. For derails on upcoming issue themes, manuscript composition, size, formatting, etc., please consult the Left Coast Press web site (www.lcoastpress.com) or Museum Educa­ tion Roundtable web site (www.mer-online.org). Reference style should conform to the Chicago Manual of Style, 15th edition. Non-conforming manuscripts will be returned to the author(s) for revision. Send issue proposals and manuscript correspondence to Elizabeth Maurer, Editor,]ournal ofMuseum

Education, via email: [email protected].

Production and Composition by Decca Penna, Penna Design, Abbotsford, British Columbia ISBN 13: 978-1-138-40491-5 (hbk) ISBN 13: 978-1-59874-820-8 (pbk) DOI: 10.4324/9781315423852

From the Editor Schools are natural and long-standing audiences for museums of all types and points o f view. Like any relationship, museum and school partnerships are nurtured through dialog. Their associations are ideally refreshed at frequent intervals through evaluation and re-examination. In this issue, the authors and the guest editors examine pathways to partnership, examples of successful partnerships, and the ways in which evaluation informs partnerships. The authors scrutinize the museum and school relationship through the lens o f what each side hopes to achieve. They demonstrate that each set of relationships is unique and that the nature of the relationships may change over time. They showcase a variety of partnerships whose goals range from audience growth, to content development, to programming outreach. Blankenship points out that partners sometimes approach the collaboration with different goals. The extent to which the goals are in sync will affect the partnership's life span. Wetterlund describes a partnership that had the courage to re-examine its award-winning product and re-design it to better meets its audience's needs. Moisan shares a process that led to productive a partnership that refreshes itself from year to year and in doing so expands the museum's reach exponentially. In this internationally flavored issue, JME hears from educators based in the United States, South America, and Australia. They discuss the museumschool relationship through the community's many members including teenagers, teachers, college students, local non-profits, and other museums. They show that a variety o f community actors can work together to meet students' needs. All of them have discovered in the course of their practice that a successful partnership involves dialog, self-examination, openness to change, and on-going evaluation of where they stand in relation to their goals. The guest editors and I are confident that you as readers will find many useful ideas in this issue. The Museum Education Roundtable invites you to continue this conversation about museums, schools, and partnerships on MER's blog, Network, (http://blog.mer-online.org/) Do you have a unique museum school partnership or a case study that you would like to share? Visit MER's blog and tell us about it. Liz Maurer Elizabeth L Maurer is the Director o f Operations for the National Museum o f Crime & Punishment in Washington, D C .

DOI: 10.4324/9781315423852-1

3

j o u r n a l

^ /m

u s e u m

e d u c a t io n

A PUBLICATION OF THE MUSEUM EDUCATION ROUNDTABLE

Editor

E L IZ A B E T H M A U R ER Director of Operations The National Museum of Crime & Punishment

Museums and Schools Guest Editors

G IU SEPPE “ P IN O ” M O N A C O Education Outcomes Manager, Smithsonian Center for Education and Museum Studies MEGAN W O O D State Coordinator, National History Day in Ohio, Ohio Historical Society Editorial Advisors

AMELIA CHAPMAN, Curator of Education, Pacific Asia Museum MARIA DEL CARMEN COSSU, International Arts and Early Childhood Education, Smithsonian Early Enrichment Center MARK HOWELL, Director of Education, American Civil War Center at Historic Tredegar LYNN MCRAINEY, Director of Education, Chicago History Museum LAURA ROBERTS, Director of the Boston Center for Adult Education and principal at Roberts Consultant, Strategic Thinking for Cultural Nonprofits SUSAN SPERO, Associate Professor, Museum Studies, John F. Kennedy University Outside Readers

ELSA BAILEY, Elsa Bailey Consulting, San Francisco, CA CONNIE BODNER, Director of Education and Interpretation Services, Ohio Historical Society RANDI ROBERTS, Consultant, VSA Association Manager HANNAH WEISMAN, Academic Programs Coordinator, Shelburne Museum Ex-officio CYNTHIA COPELAND, Chair of MER Publications and Communications Committee, Principal, The OutSourced Muse NANCY RICHNER, Chair of MER Program Committee; Education Director at Hofstra University’s Art Museum The Museum Education Roundtable (MER) is a nonprofit organization based in Washington, DC, dedicated to enriching and promoting the field o f Museum Education. Through publications, programs, and communication networks, MER fosters professionalism, encourages leadership, scholarship, and research in museum-based learning, and advocates the inclusion and application o f museum-based learning in the general education arena. For more information on MER and its activities, please contact via email at [email protected], or on the web at www.mer-online.org. Members receive the Journal of Museum Education as a benefit of membership. Write to MER at PO Box 15727, Washington, DC 20003.

From the Guest Editors This issue of The Journal o f Museum Education (JME) addresses partnerships between schools and museums. When Megan and I approached the topic, we decided to focus on three aspects: models of partnerships and of programs shaped within partnerships, the role of evaluation, and international experiences. More than a decade has passed since the publication of “Building Museum & School Partnerships” by Beverly Sheppard, which laid the ground for innumerable directions for and models o f partnership.1Later on, in his keynote address to the ICOM/CECA annual international conference in 2006 “Partnering in Museum Education —Enhancing the Adventure,” Michael Cassini described a range o f situations defined as partnerships, from loose associations to real partnerships. In defining characteristics of a real partnership, Cassini mentioned: involvement, active participation, understanding and respect, commitment and engagement. All had in common two qualities: shared and mutual.2 By reading this issue o f JME, we, and we hope the readers, will realize that the above-mentioned characteristics are in fact present in all the articles, but that many other complex questions and issues are also important, broadening Cassini's definition of “real partnerships.” Who initiated the partnership and why? Sometimes the partnership was an expression o f selfreflection and o f rethinking o f the museum's role and function (see Kelly and Groundwater-Smith's paper); external pressures —mainly funding allocation —played an important role as facilitators but also as stimulators for change (see Blankenship's, Kelly and Groundwater-Smith's, and Moisan's papers). What was the developmental path o f the partnerships, and especially how did the partners evolve along the path? We recognize that development and evolution were common aspects o f real partnerships. Blankenship's, Moisan's and Wetterlund's papers provide examples and will walk readers through the steps o f the process. Partnerships often evolve in stages as Blankenship, Kelly and Groundwater-Smith demonstrate. Monaco, Bo, and Wood and Moisan describe challenges along the path. Furthermore, we realize that a partnership usually involves a specific project that helps shape the partnership and its evolution. Monaco, Bo, and Wood and Mortara, Almeida, and Pires Martins present partnerships shaped around evaluation projects. Other partnerships involved educational programming. Blankenship's paper describes how the programs changed according to the nature of the partnership. An evaluation project was the

DOI: 10.4324/9781315423852-2

5

6

FRO M T H E G U E S T E D IT O R S

common denominator among all the papers in this issue. This was partially intentional, we —as the guest editors —specifically solicited projects with a strong evaluation component. On the other side, the different authors demonstrate how the partnership became a vehicle to access and empower a particular audience and how evaluation projects represent vehicles of communication between institutions and audiences. Wadman, deProphetis, and Kurzawa and Mortara Almeida and Pires Martins demonstrated how the evaluation helped improve the educational offerings. As an additional aspect, we realize that in most cases the evaluation was about the programs and not about the partnerships. Did we rely on the success o f the program as the key indicator for a healthy partnership? In the end, what does a successful partnership look like? All the authors conclude: a sustained relationship, the desire to continue to work together toward a common goal, and “the more you have the more you want,” which by the way is one o f the characteristic o f the definition o f learning. We strongly believe that the two major themes in this issue: partnership and evaluation will be important to the present and future o f our field.

Notes 1. Beverly Sheppard, ed., Building Museum & School Partnerships. (Washington, DC: American Association of Museums, 1993). 2. Michael Cassin, “Partnering with the Public/Dancing Cheek to Cheek,” Partneringin Museum Education-Enhancingthe Adventure. Proceedings. ICOM/CECA Annual International Conference (2005): 9-10.

Giuseppe Monaco (Pino) is the Education Outcomes Manager at the Smithsonian Center for Education and Museum Studies. H is research and publications focus on informal learning and museums. Megan Wood is the State Coordinator for the Nation History Day program in Ohio. She is based at the Ohio Historical Society in Columbus.

Provoking Innovation Creating Grassroots and Intersectional Programming at Historical Organizations

Jody Blankenship

Abstract

Historical organizations cannot continue to compete in the

modern marketplace using traditional models for education program development. Social networking sites and new applications are shaping the way our audiences approach their work, even more so as Generation Y/Millennials begin to flood into the workforce, changing the demographic makeup, work styles, and preferred modes o f communication. This change will force historical organizations to adopt new strategies for creating relevant and meaningful programs, services, and experiences. This case study will detail how the Ohio Historical Society (O H S) used emergence theory (bottom-up, adaptive creation and development) to adapt its teacher professional development programs in order maintain relevancy to a broad audience —teachers —while also being able to adapt the programs to specific audience subsets. Then it will explore how OHS is adopting an intersectional strategy in order to create new, innovative products and services built on existing programs.

The Ohio Historical Society is a private non-profit that serves as the state of Ohio's partner in collecting, preserving, and interpreting history.1In addition to holding collections, operating historic sites, and creating education and professional services, OHS is also charged with carrying out twenty-six state mandates for which it receives an annual subsidy. This includes operating the State Archives of Ohio, the Ohio Historic Preservation Office, the state museum, monuments, and historic sites; maintaining the Ohio Historic Marker program; and providing services to teachers and local historical societies. OHS' Outreach Services Department is charged with providing Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 7-22. ©2009 M useum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-3

7

8

J O D Y B L A N K E N S H IP

professional services to history practitioners including grades K-12 teachers, grades 13-20 faculty, and local history professionals including historic site managers, museum professionals, archivists, and librarians. All of the Outreach Services Department's programs are partnership-based and are either not dependent on OHS properties or take place off-site. The Outreach Services Department is administers National History Day in Ohio, the Ohio Social Studies Resource Center (Ohio's online clearinghouse for standardsaligned grades K-12 instructional resources), professional development programs from grades K-20 teachers and faculty, and the Ohio Historic Markers program. The department is also responsible for the day-to-day management o f both the Buckeye Council for History Education (BCHE) and the Ohio Association o f Historical Societies and Museums (OAHSM), Ohio's professional organizations for K-20 history teachers and local history professionals.

STEP ONE OHS began providing professional development (PD) services to grades K-12 faculty in 2002 as part o f a partnership with a large, urban school district and a local university under a Teaching American History (TAH) grant.2 In 2002 OHS was invited to participate in a TAH grant in partnership with a local urban school district and university that would create and deliver a PD service to the district's grades 6-12 social studies teachers. While the partnership was simple and understood by all partners, it was not formally described in a memorandum of understanding or contract. The school district was the official grant recipient and therefore ultimately responsible to the United States Department o f Education for fiscal and programmatic management. The project director, however, was employed by the university and assumed responsibility for the project's day-to-day management. OHS employed a project archivist to provide research services for participants. Each partner assigned a project liaison to represent the organization on the executive committee. The executive committee, which served as the primary governance body, composed o f the three partner liaisons: the chair of the project advisory board, the project director, and the project archivist. An advisory board reviewed project progress toward achieving goals and objectives and represented stakeholder interests. The board included teachers, teacher union representatives, school district's administrators, parents, and student government members.

C R E A T IN G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

9

The project attempted to increase participants' knowledge, understandings and appreciation o f American history. The project lacked formal objectives., outcomes, or a cohesive evaluation plan. Rather, project partners endeavored to accomplish the goal by running a traditional university graduate course in American history based on the assumption that teachers who knew more history content would be better able to increase student achievement on state-mandated standardized tests. The PD consisted o f a two-week summer institute and a series o f five seminars that took place during the school year. Each seminar centered on a topic under one o f six themes. Participants were assigned readings prior to the seminar. At the seminar, academic historians lectured participants about the topic and discussed the assigned readings. The project archivist provided photocopies of primary source material from OHS' collections while the historian worked with participants to analyze the material in the context o f the historical content that had been discussed earlier in the seminar. Finally, a master teacher, identified by the school district, provided examples o f how she taught the topic and how it aligned to state and district academic content standards. The summer institute consisted o f two parts. The first part was composed o f five consecutive seminars that each addressed a different topic under a theme that had not been addressed during the school-year seminars. The second part consisted o f training in the use of the Teacher Curriculum Institute's History ALIVE!3 instructional materials. The second part was outsourced; neither the university nor OHS had input into the training content or structure. A private history education organization was contracted to conduct the project evaluation. The evaluation consisted of: (1) seminar observations by an academic historian; (2) end o f course focus groups; (3) interviews with project staff and partners; and (4) formative evaluations at each activity. Controls and pre- and post-tests were not included, and, ultimately, the evaluation did not provide systematic or scientific methods to measure progress toward achieving the stated goal. The executive committee relied heavily on the formative evaluations and informal discussion with participants to gauge the PD's impact. At each executive committee meeting new ideas and methods were discussed and implemented based on feedback received from prior activities. Basically the project partners tried almost anything they could think of to create a meaningful experience that achieved the stated goal. Because the project lacked adequate evaluative feedback, it was not able to objectively measure progress

10

J O D Y B L A N K E N S H IP

toward achieving the goal. In order to gather feedback, the executive committee tried a new strategy. Following the first program year, they invited five participants, who had been identified by the school districts as master teachers, to serve on a second, formal advisory board. This board met monthly, provided feedback on their experiences from the first year, suggested content topics and ways to improve the project activities, and acted as a sounding board for new ideas proposed by the project staff. The project’s program structure remained the same, five school-year seminars and a twoweek summer institute; however, components of the seminars evolved rapidly. For example, rather than simply researching OHS’ collections for each topic covered and photocopying relevant materials, the project archivist began developing activities that directly engaged participants in analyzing and interpreting high-quality reproductions and, in some cases, actual primary source materials. In year three both the university partner liaison and project director left the project and were replaced. Because the project had never formalized partner roles, the new university staff assumed responsibility for tasks that other partners had been in charge o f and neglected to vet their strategies with the executive committee. The school district and OHS viewed these actions as an attempt by the university to take unilateral control o f the project. The partnership became tense and eventually dissolved due to partner infighting following the completion of the grant cycle. Prior to the university staff turnover, the three partners applied for a second grant to serve grades K-5 teachers in the school district. This grant followed the same model utilized in the initial grant, changing only the evaluation model to add more rigor by using pre- and post-tests to measure changes in participant content knowledge and a portfolio review to measure how much o f the PD content, skills, and resources was used in classroom instruction. The program and its components sponsored by the second grant did not evolve over time, but rather remained the same throughout its threeyear lifespan.

STEP TWO Social networking sites and new applications are shaping the way our audiences approach their work, even more so as Generation Y/Millennials begin to flood into the workforce, changing the demographic makeup, work styles, and preferred modes of communication. This change will force historical or-

C R E A T IN G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

11

ganizations to adopt new strategies for creating relevant and meaningful programs, services, and experiences.4 Aware o f the changing environment, wanting to progress toward an exemplary PD program and desiring to work with teachers across the state, OHS began to look for new partnerships that could serve areas outside o f central Ohio. Both the university partner and the school district had strict geographic and population boundaries that they were unable to cross. Like the original partnership, the first new partnerships OHS built included a university but, unlike the original partnership, OHS worked with educational service centers (ESC) instead o f school districts. ESCs are grassroots organizations that provide educational services such as bussing, special education staff, etc. in a cost-sharing model to multiple school districts in a defined region (a single county or multiple contiguous counties). Working with ESCs brought several advantages to the partnerships. First, while school districts are rarely able to provide services to other districts, ESCs are able to serve several districts within a region. Second, ESCs work off o f a business model that more closely aligns to the grants, providing services to districts rather than educating children directly. And third, ESC staff typically have experience with project management and, therefore, have a firm understanding o f programmatic and fiscal management, evaluation, and grant reporting requirements. School districts would typically reassign teachers to work on the project and spread responsibility for evaluation, budgeting, and other tasks across the district, requiring several people within a large organization to track the progress o f a small grant. Learning from the earlier partnerships, the new partnerships defined partner roles in a memorandum o f understanding prior to submitting the grant. In these partnerships, OHS provided day-to-day communication, program management, and grant reporting requirements and housed both the project director and archivist. The university partner provided content support by identifying appropriate scholarship and faculty. The ESC provided fiscal management and identified instructional resources. The project was governed by an executive committee modeled on the one from the original grant. At the insistence o f a new government grants manager, OHS used a logic model to develop the project's goals, objectives, actions, and outcomes. The project structure still did not vary much from the original grant model, but it was formalized to adequately evaluate progress. The new partnerships created a program with the following objectives:

12

J O D Y B L A N K E N S H IP

1. Increase participants' knowledge o f American history in each o f six content themes. 2. Increase participants' ability to access primary source materials in local libraries, archives, and historical societies. 3. Increase participants' ability to analyze primary source material to construct new knowledge and understanding o f American history and how it was manifest locally. 4. Increase participants' ability to apply new content, primary source material, and analytical skills to classroom instruction. 5. Increase student achievement in American history as demonstrated in state-mandated standardized tests. The standard PD activity included four components: 1. Participants read historical scholarship and review primary source material prior to the activity. 2. Participants engage in a lecture, discussion, and critical text review o f assigned readings with a professional historian. 3. Participants complete a primary source activity focused on developing or sharpening analytical skills under the direction o f the project archivist. 4. Master teachers model best practices for classroom instruction. The grant's request for proposal changed, giving preference to projects that used an experimental or quasi-experimental evaluation design. The project was evaluated by a professional evaluation firm using a quasi-experimental design. The state curriculum provided a standard academic content. The evaluation consisted of: 1. Pre- and post-intervention content tests 2. Pre- and post-activity tests 3. Formative questionnaires 4. Summative questionnaires 5. Focus groups 6. Pre- and post-intervention student assessments. The pre- and post-intervention tests used questions from the College Board's Advanced Placement and the National Assessment o f Education Policy exams Pre- and post-activity tests were based on the activity content and created collaboratively by the lead faculty, the project director, and the project

C R E A T IN G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

13

archivist, and were reviewed by the executive committee. Two grant applications were successful. Each recruited full cohorts in the first program year. The following year OHS partnered with several more universities and ESCs on additional grant proposals, again modeled on the initial proposal but revised slightly based on experience from the new grants. Revisions included the use of a constructivist learning model and field trips to historic sites, archives, and museums. As in prior years, several o f the grant applications were funded, further expanding OHS' ability to reach teachers outside o f central Ohio.

STEP THREE Overall the grant program model proved successful at achieving the goals and objectives. However, OHS noticed that components o f the larger program did not work well across diverse populations. The following observations most spurred change: 1. A suspicion that the federal program funding the TAH grants would end soon. 2. OHS' desire to commoditize the products and services created through the grants so that they could be sold to school districts after the grant ended. 3. A recognition o f the emphasis placed on quantitative evidence of program success by the Ohio Department of Education and school district administration. 4. An acknowledgement that under the existing model OHS would not have sufficient resources to staff additional grants and partnerships if they continued to grow at their current rate. 5. An understanding o f the advantage OHS had by using a common program model and evaluation design across multiple and diverse populations. 6. A comprehension o f Ohio's distinct and sometimes complex cultural regions (i.e., urban, rural, suburban, ex-urban, Appalachian, and Midwestern) meant that a boilerplate program would not meet the needs of distinct audiences, even when they operate under a standard education system. Because o f these observations, OHS began to use advantage gained from participation in multiple grants to begin testing variations o f program models

14

JO D Y B LA N K EN SH IP

to find combinations and iterations of program components that worked well with specific audiences, and to collect enough data to build a compelling case from which it could market products and services developed under the grant to school districts after the program ended. OHS reviewed the logic model to find new and different ways to achieve the grant goal. Under the existing grants OHS was bound by the objectives outlined in the grant application which could not be changed without significant program redesign and input and permission from the funding agency. Therefore OHS began changing and adjusting the program components in order to measure how various audiences responded in short-term evaluations. If responses were positive over several short-term measures, the new component was adopted over the next program year to conduct a longterm evaluation. Focus groups and advisory committees made up o f teachers who represented the target audience provided ongoing feedback about the changes. OHS was able to test multiple combinations with several audiences because it was involved with several grants simultaneously. Ultimately the various iterations and combinations of program components did not impact the overall achievement o f goals and objectives. OHS found that, despite progress toward achieving goals, those programs that solicited advice from participants and made changes based on the advice had greater levels o f participation and reported higher levels of participant satisfaction and perceived value. This equated to greater demand for services, an improved institutional capacity to provide a wider range o f products and services, deeper reach into school districts, and more recognition for the program and OHS from local, regional, and state entities OHS realized that by trying to frame the experience described in step three, it was employing emergence theory in its program development model. Program participants were re-cast as partner-consumers, who were simultaneously using our products and creating or improving them. In taking on this role, participants reported that the programming was more relevant and valuable. OHS found that the programming also tended to predict changes to grade K-12 operating environment on a local level, and when aggregated across projects it reflected regional- and state-level changes as, and sometimes before, they occurred. Realizing what was occurring, OHS now uses emergence theory to develop existing and create new professional development products and services.

C R E A T IN G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

15

EMERGENCE THEORY Emergence theory describes how individual elements act independently as members o f a larger community, within a shared environment based on a simple set o f rules.5 The collective actions o f these individuals aggregate to create a complex system that is highly adaptive and evolutionary and can predict changes occurring in the environment, though the direction o f the system itself is often unpredictable and can run counter to accepted industry strategies or trends. Authors such as Steven Johnson explore emergence theory and how it applies to biological and cultural trends such as the development o f cities and learning.6 Scott Cook has explored how the theory can affect enterprise models.7 As this theory applies to OHS' teacher professional development program, individual elements in the emergent system are the participating teachers. The environment is defined on multiple levels including location type (urban, suburban, rural), socioeconomic status, and cultural groups. These environments often intersect and are scalable to the broadest common variable; in OHS' case it is the State o f Ohio. The simple rule set is the existing governance policy, or the state academic content standards and testing requirements for teachers and school districts. Within this framework OHS will collect the behaviors, opinions, reactions, and suggestions o f as many individuals as possible. The information collected will define the current state o f the environment and allow OHS to identify potential products and services, while participants' opinions on new rules and regulations and the benefits or challenges that arise from their implementation provide some guidance as to how the system may change. Constant testing over time will allow products to continuously evolve along with customers' needs within a rapidly changing operating environment. In 2007 OHS, with this understanding o f the value o f emergence theory, dramatically changed its approach to TAH projects due primarily to increased customer demand. Rather than providing a turnkey service in which OHS managed the project's administrative and programming function, it instead developed a menu o f products and services based on prior experience and evaluation data from which ESCs can customize an approach that best suits their needs. When meeting with ESCs and better understanding their needs OHS is able to bring additional resources to the project using our unique position (between academic history and public history and as a statewide agency), networks, and resources. In addition,

16

J O D Y B L A N K E N S H IP

ESCs are encouraged to suggest new products and services that they would like OHS to provide. This new approach, based on emergence theory, allows OHS to focus on delivering high-quality, pre-developed services, create new products and services that have been developed from customer demands, and to participate in many more projects to collect data from exponentially more teachers. Using TAH funding, OHS has developed products and services that exceed teachers' and ESCs' needs and has built a feedback system that will evolve these resources as the education environment continues to change.

WHERE ARE WE HEADING? While emergence theory is proving useful in developing programs in a directional capacity within a single field to maintain relevancy and to respond to changing operating environments, OHS' next step will attempt to use an intersectional strategy to bring together the resources o f the local and state history communities. Where ideas, needs, or desires o f one field within the discipline o f history intersect with those of another, OHS believes that vertically aligning the networks and resources o f multiple organizations add additional value and create wholly new programs and services that will solve local problems and build the capacity of the entire community.8 Author Frans Johansson, in his book The Medici Effect, describes how individuals and organizations create new ideas at the intersection of fields, cultures, and industries.9 In the book, Johansson describes how innovation can occur when people cross disciplines or fields within a discipline and create new ideas where the fields intersect. Most people and institutions carry out their missions in a directional capacity, working within the confines of a single field's established standards and frameworks, rarely venturing into another discipline. Because this is the norm, institutions tend to benchmark and follow each other toward the development of similar products and services competing for the same, often shrinking audiences. Where a field intersects with another is fertile ground for innovation. Intersections, however, are risky areas and are often dismissed by industries as irrelevant to the work of their field. Organizations that venture into intersections frequently find themselves alone, causing consternation on behalf o f staff and questions from peers about the direction that they are heading. However, enormous potential for developing unparalleled products and services without competition lay in these intersections.

C R E A T I N G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

17

OHS has been working with several statewide professional associations that represent various fields related to the discipline o f history, specifically the Society of Ohio Archivists (SOA), the Ohio Academy o f History (OAH), the Ohio Council for Social Studies (OCSS), the Buckeye Council for History (BCHE), and the Ohio Association of Historical Societies and Museums (OAHSM). Together these groups represent nearly the entirety of history endeavors in Ohio, with the exception of historic preservation. Each group interacts with distinct audiences and provides a necessary component of the history discipline. While financially stable, none o f these associations employ full- or part-time staff; instead, they depend on volunteers to provide an array o f member benefits. This poses a particular threat as members are increasingly busy and less able to devote time to supporting the field in a volunteer capacity. OHS, a membership organization that has the capacity and staff to accomplish the activities these groups attempt to complete annually, has been working with each association to provide support for carrying out day-to-day activities needed to fulfill membership benefits. While OHS does not stand to benefit financially from these relationships, it foresees the potential to both raise the status o f history within Ohio through visibility and advocacy and to place OHS in a position to take advantage o f intersectional ideas among two or more o f the fields represented by these associations. In terms of advocacy, each group is independently lobbying the state via the legislature, the Ohio Department of Education, and the Ohio Board of Regents to lift the status o f history within the field o f social studies education and Ohio's core K-12 curriculum. Additionally, these groups are attempting to create a larger pool o f funding for history activities within each agency. Acting independently they are able to garner between fifty and one hundred members to advocate for history. OCSS is the largest association with approximately 1,750 members. However, these members divide their loyalties among the many disciplines that comprise social studies, and an attempt to promote one subject is viewed as a threat to the others. Independently each group is able to garner the numbers to possibly sway a legislator or two but is unable to compel real change in the legislature. Together these organizations may be able to attract political attention, especially if they join existing advocacy efforts such as Ohio's Statehood Day.10 In terms of visibility, each subject within the K-12 social studies field, through its discipline-based associations (BCHE, the Ohio Council for Economics Education, the Ohio Geographic Alliance, etc.), holds an independent

18

J O D Y B L A N K E N S H IP

conference attracting a significant but—due to shrinking school and personal budgets —contracting audience of teachers. If combined into a single conference, these groups could collectively attract upwards of one thousand attendees. OHS is attempting to pull these groups together through the OCSS in order to create a single Ohio social studies conference in which each discipline creates a program strand. For example, the Buckeye Council for History Education would forego an independent conference in exchange for the opportunity to develop a history strand at the OCSS annual meeting. Other associations would develop similar strands in civics, economic, geography, and the other subjects. The increased attendance would force the conference to relocate to a central, venue that will draw media and government attention. OHS is also attempting to create intersectional ideas, products, and services by combining academic, public, and K-16 history efforts. Though the outcomes are unknown, OHS believes that enormous potential lies at the intersections of these fields. OHS will try to build a vertically aligned system that uses these associations' resources and networks in a way that is mutually beneficial and, when combined, multiplies each association's impact; publishing, for example, could be a growth market at the intersection of OCSS, OAH, and OAHSM. For example, Teachers need history content that connects national and international themes with local history in a concrete and easily accessible format. Academics at small Ohio colleges and universities focus on teaching but must publish to secure tenure. Local history organizations hold significant —often untapped—history collections, comprise an existing system of local distribution centers, and have strong relationships with local teachers and school districts. Where these three fields intersect, OHS is able to encourage the production of specific history scholarship that is relevant and in demand, to market the scholarship to the specific audience, and to establish an efficient distribution system. All of this can be accomplished using modem, small scale publishing technology that requires no infrastructure other than an internet connection and a brand. Combining Johansson's intersection strategy with emergence theory, history organizations can create systems for continuous innovation while remaining highly relevant. An organization can continue to create and cultivate appropriate, mission-based directional products and services for traditional audiences while also exploring innovative new products that cut across defined user groups to create new audiences. OHS will create such a system by merging its K-16 outreach programs with its field services department—which provides technical expertise and assistance to local his-

C R E A T I N G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

19

torical organizations —to create a more comprehensive outreach service function. This division will service the needs o f local history professionals and enthusiasts, history organizations, libraries, archives, museums, teachers, academics, and ESCs, and will engage these audiences as individuals (e.g., teacher, re-enactor, curator, and archivist), organizations (e.g., libraries, schools, and museums), and associations (e.g., state social studies council, state museum association, state academic history association). On both an individual and organizational level OHS is able to create appealing products based on operating environment and rule sets. As OHS engages more users, our knowledge base will expand, providing more and better data to aid the development and evolution o f products. On an association level OHS has determined that these disparate groups currently need the same basic products and services (e.g., communication strategies, membership fulfillment, logistics, etc.). By providing association entities with low-cost, highquality services OHS is able, within the discipline o f history, to seek out intersectional opportunities among K-12 education, local history organizations, local history professionals, and academia by tapping into the individual associations' networks (knowledge base) to generate ideas for and test new products and services that are built on common professional standards (rule sets) but tailored to meet local needs (operating environment). Ideas generated in the intersection will give OHS, and can give other organizations, a competitive advantage and establish it as a more relevant and innovative organization within Ohio's history landscape. The products and services created will be highly adaptive to an increasingly fast-paced and unstable environment. History organizations specifically are best able to bridge the various resources within a community because they span the entirety of the history discipline; they typically engage in multiple fields and are centered on the identity o f and quality o f life within the larger community. Unlike academia or K-12 education organizations, local history organizations are largely free from state oversight or overly aggressive peer-review. Thus, they are able to leverage all o f a community's resources in order to encourage civic engagement, leadership, and tolerance.

CONCLUSION The use of emergence theory and intersectional strategy is not new or novel. For example, when applied to for-profit ventures, emergence theory has been cast as market research, audience evaluation, and many other terms. While

20

J O D Y B L A N K E N S H IP

the ideas presented in this article are not meant to break new ground or be “cutting edge/' they are intended to reframe the way that historical organizations think about and approach program development. History organizations should abandon a top down “know it all” method and adopt a customer-centric, bottom-up approach. They should aggressively seek out new tactics for making history relevant to both small affinity groups and the larger community. Without intervention the relevance o f history will ebb and flow over time, but by using an assertive approach that seeks out novel and meaningful uses for their resources history organizations will focus and strengthen their public value.

Notes 1. The Ohio Historical Society's mission statement and references to its role in the Ohio Revised Code is located on its website, www.ohiohistory.org/about. 2. TAH grants were introduced in 2001 by Sen. Robert Byrd (D-WV) through an initiative under the No Child Left Behind Act. These grants require local education agencies (ESC) to partner with university history departments and/or museums, archives, and libraries to create and carry out professional development programs for teachers that focus on American history content. 3. The Teachers Curriculum Institute is a non-profit education organization that creates, trains and sells a social studies curriculum called History Alive!. Additional information is available at www.tci.org. 4. Tony Wagner, webinar sponsored by the Governor's Office, State of Ohio, October 28, 2008. 5. A complete description of emergence theory, also referred to as “swarm theory," can be found in Steven Johnson’s Emergence: The Connected Lives ofAnts, Brains, Cities, and Software (New York: Scribner, 2001). 6. Johnson, Emergence. 7. Scott Cook, “The Contribution Revolution,” Harvard Business Review (October 2008): 60-69. 8. A description of how some companies are using this tactic is available in Ed Catmull's article, “How Pixar Fosters Collective Creativity,” Harvard Business Review (September 2008): 64-74. The article is also available online at: http://harvardbusinessonline.hbsp.harvard. edu/hbsp/hbr/articles/article.jsp?ml_action=get-article&articleID=R0809D. 9. Frans Johansson, The Medici Effect: Breakthrough Insights at the Intersection ofIdeas, Concepts, and Cultures (Cambridge: Harvard Business School Press, 2004). 10. Statehood Day is a statewide history and historic preservation advocacy day that occurs on or close to March 1, the anniversary of Ohio's statehood. Organized by the Ohio Historical Society, this event is led by a group organizations and associations. More information is available at www.ohiohistory.org.

C R E A T I N G G R A S S R O O T S A N D I N T E R S E C T I O N A L P R O G R A M M IN G

21

Jody Blankenship has been working for the Ohio Historical Society as a Project Manager; and most recently as a Department Manager; since 2002. In that time, he has collaborated with multiple statewide and regional organizations to create and implement outreach programs for O hio’s K -1 6 educational community, including: professional development workshops, summer field schools for students, National History Day in Ohio, the Ohio Social Studies Resource Center ( WWW.

ossrc.orgj, and the Buckeye Council for History Education. Jody received an M A in History Museum Studies from the Cooperstown Graduate Program, a B A in History and Education from Ohio Northern University and is a licensed social studies teacher in Ohio.

Partners in Process How Museum Educators and Classroom Teachers Can Create Outstanding Results

Heidi M oisan

Abstract

Collaborative processes by nature are not neat and tidy; and

if mismanaged, they can lead to chaos rather than creative productivity. However, when a museum and a group of teachers establish a respectful peer community that maximizes all the members talents, truly impactful teaching and learning result. This article analyzes the Great Chicago Stories project, a partnership among the Chicago History Museum, sixteen teachers, and other stakeholders, as a model to examine the challenges and rewards in developing, implementing, and evaluating shared projects.

ABOUT GREAT CHICAGO STORIES Great Chicago Stories, a three-year project, is the result o f a collaborative effort among the Chicago History Museum, Chicago school teachers, a national advisory board, professional writers, and technology consultants. Area educators selected the topics, wrote unit plans, and tested the narratives and resources in their classrooms. Teachers explored topic options based on the Chicago History Museum's collections, grade level curriculum, and Illinois State Learning standards. The topics with the strongest potential in these areas became the final narratives featured on the site. The project was developed through a National Endowment for the Humanities program, Grants for Teaching and Learning Resources and Curriculum Development. This funding category provided exciting possibilities for project development with an emphasis on integrating technology into humanities teaching, and requirement for collaboration between teachers and scholars.1 Funding for Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 23-40. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-4

23

24

H E I D I MOI SAN

The Great Chicago Stones website, collaboratively developed by museum educators and classroom teachers, offers historical fiction narratives based on the collections o f the Chicago History Museum and multimedia features to support history instruction in the classroom.

our project gave us the opportunity to create and implement a collaboration that we hoped would achieve the specific project goals of Great Chicago Stories, as well as result in development of a model to inform future work. The purpose of the Great Chicago Stories project is to fashion Chicago history into powerful narratives as points of entry for exploring key humanities themes and fundamental concepts. Great Chicago Stories explores key themes of place, identity, and contested space while making local, regional, and national connections. It was developed and tested in collaboration with sixteen Chicago-area teachers. The Web site www. GreatChicagoStories.orguses the collections of the Chicago History Museum as the primary tool for telling these stories through a suite o f twelve his-

P A R T N E R S IN P R O C E S S

25

torical fiction narratives, an interactive history map, audio versions o f the stories, and supporting classroom resources. Six stories designed for Grades 3 and 4 explore the broad theme o f how Chicago connects to the world. A Bronzeville Story features the experience of an African American family who moves to Chicago in 1949, during the second half of the Great Migration. Resolving a moral dilemma through the lens o f sports history is at the heart of A Tough Call. Readers discover early Chicago and life during the global fur trade in Trading Mystery. The pivotal role o f innovation takes center stage when a young girl visits the World's Fair o f 1893 in The Best of the Fair. In Hot Dog! Allan acts as a historian when he accepts a challenge from a street vendor to earn a free lunch if he can explain why the hot dog is so important to Chicago. A boy shares his fascination with trains in Joseph's Railroad Dreams. Six stories written for Grades 9 through 12 examine the city as contested space. The little known history o f the Abolitionist movement in Chicago is told in Halfway to Freedom. The evolution of public housing and white flight is examined through a teenager's viewpoint in Where the Neighborhood Ends. His Father's Namesake explores labor issues during the dramatic events in Haymarket Square. The tough choices nineteenth-century immigrant families faced and their experiences at Hull House are the focus in Angelo's Saturdays. Jazz Era Chicago is the exciting backdrop for the Great Migration story, It's a Long Wayfrom Home. The Story Peace highlights political activism during the 1968 Democratic National Convention through the differing viewpoints among a teenager and her family and friends.

THE CASE FOR COLLABORATION While field trip visits and professional development programs provide points of contact between formal and informal educators, collaborations offer the chance for more substantive interactions between museums and schools. The book FarAway and Long Ago: Young Historians in the Classroom was written by a classroom teacher and a museum educator. This volume portrays two years o f cooperative history instruction in a fourth grade classroom. In one passage the classroom teacher describes the impact o f working with the museum educator: “She became an extraordinary resource for me, seeking out new materials, doing special lessons in my classroom with artifacts and slides from the museum and her own personal collection, and designing our field trips to the museum to fit closely with our classroom work.''2 The

26

H E ID I M O IS A N

Chicago History Museum was inspired through the Great Chicago Stories project to build these types o f relationships and connections. Collaborative projects represent a rich opportunity. However, navigating the distinctive roles and relationships between formal and informal educators can be problematic due to the different learning environments and priorities that each brings to their work. Griffin notes, “The issue that stands out in regard to relationships between museum educators and teachers is that the pedagogy is uncertain on both sides. There is considerable room for more work both in research and in professional development. Schools and museums have different but complementary roles, and there is a need for closer alliances between these institutions and their staff/'3As we began our planning, we considered if it were possible to design a collaborative experience that both increases professional relationship building and makes an impact in classroom instruction. We sought to answer this question through the collaborative process of developing Great Chicago Stories.

CONSIDERING COLLABORATIVE GOALS In the case o f the Great Chicago Stories project, collaboration was a requirement of the grant, but it was up to museum educators to define and carry out the collaborative process. We wanted to develop a peer community among the classroom teachers, the Chicago History Museum education staff, and the scholars. We also wanted to explore ways to sustain our relationship with the teachers involved beyond the project end date. Collaborate means “to work jointly with others or together especially in an intellectual endeavor."4We believed that “working jointly" meant going beyond mere consultation. As Sheppard points out, “Ultimately, collaboration is not just about the joint delivery o f a product; it is about sharing and shaping an essential experience in concert with the very community and audience we wish to serve."5 Teachers and, through them, students are the audience for the Great Chicago Stories project. By collaborating to develop the resource, we hoped to achieve the following specific goals: • Create an authentic product; one that other teachers would be inclined to trust and to use. • Build a learning community between the museum and the participating teachers. • Sustain the relationship beyond our initial connection through the Great Chicago Stories project.

P A R T N E R S IN P R O C E S S

27

• Tap into the expertise o f classroom teachers to inform this project specifically, as well as to provide insights to further our overall work with the school audience

IMPLEMENTING COLLABORATION Intuitively, working together sounds like a good idea, but it is difficult to actually develop partnerships. The publication True Needs, True Partners: Museums and Schools Transforming Education describes twelve “conditions for success” that must be present for partnerships between museums and schools to succeed. This list covers the earliest phases of collaboration (“obtain early commitment from appropriate school and museum administrators”) and the process itself (“create a shared vision for the partnership, and set clear expectations for what both partners hope to achieve”).6 While this list is a useful reference, the true value is found in the larger idea that collaboration needs to be thought through and requires careful tending. In considering the implementation process, the following hallmarks of the Great Chicago Stories project were essential to success and are transferable to any collaboration.

STARTING IS THE HARDEST PART We did not start from scratch. Instead, we built on previous work through initiatives such as the History Lab, a program in which teachers authored lesson plans inspired by the museum's collection, and My Chicago, a joint endeavor with the Chicago Public Schools to create resources for after school instruction. Past experience and the professional contacts established during previous work provided a starting place for project development and teacher recruitment. Our project-to leverage museum collections as the basis for narratives that teachers could use in the classroom-grew in large part from the realities of No Child Left Behind. This law, with its requirements regarding student achievement especially in reading, has greatly impacted what is taught in the classroom. In Illinois, Social Science no longer appears on the Illinois Standards Achievement test, which currently measures only reading, math, science, and writing.7 Great Chicago Stories uses literacy as a method to explore key history and humanities themes. This interdisciplinary approach allows teachers to make the best use o f limited classroom time, meeting both language arts and social science goals with one resource. Third and fourth

28

H E ID I M O IS A N

grades are core audiences for the Chicago History Museum because the social science curriculum focuses on local and regional history; yet, there is a lack of resources about Chicago and Illinois history available to teachers. Additionally, we had seen an increase in field trip visits by high schools that have added urban studies and Chicago studies elective courses to their curriculum. We believed that we had a strong idea that met important instructional needs and would appeal to teachers, coupled with a solid reputation that teachers trusted. In Mobilizing the Community to Help Students Succeed, Price describes the ideal community partners as “those with a reputation for genuinely caring about children's education and with a track record o f playing constructive roles toward that end."8With this foundation in place, we were able to secure the grant to pursue our project. After solidifying the project vision, it was important to consider staffing. As Sheppard notes, “Individual staff allegiance to a project is not enough."9 A project cannot succeed if only a single person is dedicated to carrying out the vision. The dedication o f staff members in a variety o f positions with different responsibilities and filling diverse roles was essential to the success of the Great Chicago Stories project. Internally, we organized project staffing to consist o f a project developer, who was a full-time education department staff member, and a part-time program coordinator. These two staff members worked closely together to move the project forward through all phases and oversaw the day-to-day project operations. The head of the Education Department and the Vice President o f Interpretation and Education worked closely with the project developer and coordinator to review all materials and ensure that the grant requirements and project goals were met. A variety of other staff members from the collections, publications, and research departments o f the museum also supported the core team. These staff members contributed time and expertise to the project at various points throughout its lifespan, and the participation o f a range o f staff members was key to the ultimate success o f the project. Once our internal structure was in place, we were ready to reach out to teachers. We created simple, yet effective, recruitment materials beginning with a letter o f invitation that broadly described the project. This was attached to an application packet, which consisted o f a one-page job description that provided more details regarding project goals and time commitment and that clearly spelled out expectations and compensation. Offering compensation was very important as a mark of professionalism. It

P A R T N E R S IN P R O C E S S

29

Classroom teachers and museum educators working on project development.

provided motivation, and an opportunity to earn money, since some teachers would need to give up other summer work to participate in Great Chicago Stories. The application included a reflection question that asked teachers to explain why they were interested in participating in Great Chicago Stories and to describe the benefits they envisioned for their students when stories and primary sources were incorporated into their teaching. The final piece o f the application was a statement o f agreement that teacher candidates and their principals were required to sign. Interested teachers were asked to attach a lesson outline and their resume. We recruited teachers through a variety o f methods. Our professional development programs provided a perfect opportunity to introduce teachers to the project and distribute the application materials. Our Chicago Public School administrative contacts and university partners were extremely helpful in distributing materials to their teacher contacts. These teachers were highly motivated individuals engaged in professional development and, therefore, more likely to take the time to complete the application process. We recruited teachers for two years and saw the interest in the project

30

H E ID I M O IS A N

grow over time. The first year we received about thirty applicants, and in the second year, the number increased to more than fifty. Each year we only had eight slots to fill, resulting in a very competitive process. The review process focused on choosing educators whose teaching philosophy and practice matched the project vision. We also considered diversity in experience, geography, instructional setting, and ethnicity. For the first review round, Museum staff members read and evaluated the applications. We captured reviewer feedback via a simple form attached to each application. Phone interviews were then conducted with the strongest applicants. Finally, offers were extended to the selected teachers. Although time consuming, this process allowed us to evaluate each candidate based on their writing, their professional activities, and the phone conversations, and find the best matches for the program. Getting our project off on the right foot required concrete measures. Important initial work included starting with a solid idea that met the needs of collaborative partners, forming a strong internal team, creating a useful application, and recruiting teachers through a variety of channels. These steps launched the project from a position o f strength, enabling collaboration and development work to immediately begin.

FORMING COMMUNITY One o f the major collaborative goals o f the Great Chicago Stories was to develop a true community o f learners. Although he wrote specifically about schools, Schmoker's premise that in a community, members “recognize and share the best o f what they already know” and work together to meet shared goals through “collective follow-up, assessment, and adjustment”10 applies to museum and school partnerships. In the Great Chicago Stories project, we wanted to build relationships among all project participants, including the classroom teachers, museum staff, advisory board, and the project evaluator. We strove to design the collaborative process to not only develop an exemplary product-the Great Chicago Stories Web site-but also to generate an experience that enhanced the intellectual lives and professional practices of all participants. As Wiggins and McTighe consider staff learning in their article “Examining the Teaching Life,” they conclude that successful lifelong learning is personalized and works best when programs “honor learners' interests, curiosity, strengths, contributions and prior knowledge, making them feel they

P A R T N E R S IN P R O C E S S

31

are an important part of something larger than themselves.”11 Beginning with our shared goal o f creating narratives to explore local and national history topics based on the museum's collection, we developed a plan o f work that built in opportunities for choice to optimize each individual's unique contribution. For example, teachers and museum staff worked together to develop a list o f narrative topics and articulate the reasoning behind the recommendations. Topics drew on the strength of the museum's collections and aligned with curriculum and state learning standards. Once a slate of topics was in place, teachers chose the narrative that they wanted to pilot in their classroom. The opportunity for individual choice guaranteed that teachers were invested in the specific narrative topic and would wholeheartedly bring their expertise and creativity to further development and testing. An important way to bond as a true learning community is to master new skills together. In Great Chicago Stories, we accomplished this by learning a new curriculum design approach as a group. We asked a member o f our advisory board, an education specialist at a local university, to facilitate workshop sessions during our summer development session to train the museum educators and classroom teachers in how to use a common method to design unit plans. This challenging process, called Understanding by Design, advocates a backwards design approach for planning curriculum, assessment, and instruction and has a “focus on developing and deepening students' understanding of important ideas.''12 The Understanding by Design handbook that we utilized is organized to “stimulate individual reflection and promote thoughtful conversation among colleagues.''13 Mastering this new theory and curriculum writing approach together helped to cement the group, engaging us as learners, and invested us in common practice and a shared vision. Although each teacher wrote unit plans around a particular narrative the process was united through the common use o f Understanding by Design. After draft unit plans were written, we further strengthened our community and our work by committing ourselves to a peer review process. The phrase “peer review” can conjure images o f arbitrary judgment and hurt feelings. Peer review asks us to hold our work up for critique and feedback through a formal process. However, when done right, peer review holds many benefits from positively impacting the quality o f work (in this case, design of instructional units) to contributing to professional growth and development. McTighe and Wiggins write, “Participants in peer review sessions regularly comment on the value o f the opportunity to share and

32

H E ID I M O IS A N

discuss curriculum and assessment designs with colleagues. We believe that such sessions provide a powerful approach to professional development since the conversations focus on the heart o f teaching and learning.”14 Incorporating successful peer review necessitated advance consideration. Peer review was not a surprise sprung at the last minute. Rather, it was something we prepared for along the way, knowing from the beginning that our time together would culminate with peer review. We followed peer review guidelines that we discussed and revised in advance as a group. We allotted sufficient time for meaningful review that included written comments, small group discussion, and author revision. In the context o f a true learning community such as Great Chicago Stories, peer review is a supportive process that allows educators to improve their curriculum design and to share their work among colleagues. Peer review strengthened our individual work, and at the same time, established a collective understanding of, and stake in, the unit plans under development. In Great Chicago Stories, we formed a strong learning community through the three key principles of allowing for individual choice, mastering new skills together, and implementing peer review. These three elements built team unity and fully invested all members in the collaborative process and in working toward project goals.

PRESENT AND ACCOUNTED FOR No collaboration can prosper without the members setting aside time to work together. During Great Chicago Stories, we made it a point to meet in person regularly. The summer months offered the opportunity for uninterrupted time together. During the first year, we met for five straight days and really immersed ourselves in getting the project off the ground. We realized that while we had accomplished the main project goals for that summer, including the decision to create historical fiction narratives, identifying narrative topics, and outlining basic features of the Web site, it came at a price. While the immersive nature of the week allowed us to focus exclusively on the tasks at hand, it also wore us out. Taking this experience to heart, during the second year, we adjusted the schedule to cover the same number o f hours, but spread out over the course o f two work weeks. This gave participants time to process material, reflect on meetings, and complete advance preparation between dates. The

P A R T N E R S IN P R O C E S S

33

The concept for the interactive history map, to deepen student exploration of primary source materials through zoom tools and to place the narratives in a geographic context, grew out of classroom pilot testing results.

second summer ended on a high note, with participants eager for next steps. In a post survey, all the teachers ranked the summer experience in the areas o f quality o f content, quality o f presentation, pacing, and usefulness as “excellent” (the highest rating). When asked about the most significant thing gained from the summer experience, many teachers cited the interaction o f the team. One teacher who wrote, “This has really helped me feel connected to a wider teaching community,” perhaps best expresses this belief.15Without face-to-face interaction, this connectivity could never have been established. After school resumed, one important point o f contact was when museum educators visited participating teachers' classrooms. Teachers had spent considerable time at the museum. It was equally important that museum educators spend time in the classroom so that both environments-the museum and the school-were understood, experienced, and valued by collaborators. The project developer and program coordinator visited each teacher's classroom twice, once in the fall and once in the spring to see different narratives being used and to maintain connection with the classroom teacher throughout the school year. These visits provided a chance to observe Great Chicago Stories resource

34

H E ID I M O IS A N

materials in action. We were able to see student response and engagement first hand. Visits were not meant to evaluate the teacher; rather, they were carried out to gauge the effectiveness o f project resources and to help museum staff develop a greater understanding o f the classroom environment. During school visits, museum educators saw a variety o f class sizes, unique school communities, and diverse student populations, and gained appreciation for how time is structured in the classroom. Visiting the classrooms helped us meld our expertise as informal educators and understanding o f learning in museums with the needs o f the classroom. This approach was important because classrooms are the venues where Great Chicago Stories is implemented. School visits sowed the seeds o f good will as teachers saw that we would come to them, just as they came to us, and spending time in each others places o f work solidified the group as a team. Meetings o f the whole group were convened at regular intervals during the school year. The most important meeting occurred at the end o f the school year. During this meeting, teachers brought samples o f student work for whole-group analysis and shared their experiences and teaching strategies. The project evaluator facilitated these meetings, both to allow the museum staff to participate and to ensure that the conversation stayed relevant in identifying strengths and weaknesses of the narratives and approach so that specific recommendations could be made to achieve project goals.

AVOIDING OVERPLANNING There is a fine line between staying true to a plan and becoming a mindless adherent. The joy o f collaboration lies in the unanticipated opportunities that grow from bringing a group o f people together. From the start, we knew that Great Chicago Stories was going to exist solely online. We wanted to take full advantage o f the virtual environment and set aside money in the budget to create a multimedia piece. However, we made it a point not to preplan the multimedia type or content. Instead, we first created and tested the narratives so that classroom experience could be brought to bear in the multimedia development. After selecting an outside contractor who had expertise in online learning and multimedia creation, we held a series o f brainstorming meetings during the first year o f the project. In these meetings, we were able to identify existing museum resources that could contribute to the multimedia, process the results o f the classroom

P A R T N E R S IN P R O C E S S

35

pilot testing, and learn more about technology conditions in our partner schools. Teachers told us that while their students were very engaged with the narratives, they did not understand where in the city these historic events had taken place. Students were also curious about how those places looked at the time o f the story and how they looked today. From these meetings, a set o f questions emerged that served as a rubric against which to test ideas. Questions included: Will history educators use it? Will students like it? Is it more than reading comprehension? Can this be done better on paper? Does it provide context for the artifacts and allow for exploration? Does it help students understand that history is continuous and that they are part o f history? Is it flexible enough to work with all narratives? Is it flexible enough to work with all grade levels? Does it deepen the narratives' connections to Chicago? This process culminated with the decision to develop the interactive history map idea. The map deepens students' interaction with primary source materials, a concern that the collaboration had discussed with the project's advisory board, while addressing the geography deficit that teachers had discovered during narrative testing. Year two o f the project presented another opportunity for multimedia development. During team meetings, we explored how full audio recordings o f the narratives could be a valuable resource. Teachers believed that the ability to combine the audio with the print version o f the story would benefit their instruction with a wide range o f students such as English language learners and struggling readers. The full audio recordings, available in several formats, brought the stories to life, making another type o f narrative experience possible through the Great Chicago Stories Web site. Especially pleasing is how the audio versions o f the narratives support the classroom activities written by teachers, many o f which feature drama and reader's theater activities. Teachers found that advance listening to the audio stories helped to prepare students to make their own theatrical interpretations. It was important to place multimedia development within the overall project plan, earmarking funds and development time. However, by avoiding predetermining the nature o f the multimedia pieces, we were able to evaluate emerging needs and draw on testing results to inform the direction the multimedia took. This approach respected the collaboration at the heart o f the project and resulted in multimedia components that meet project goals and best serve the audience.

36

H E ID I M O IS A N

SOLID LEADERSHIP KEEPS PROJECTS ON TRACK A collaborative process involving a variety o f partners mandates leadership that is both attuned to details and capable o f seeing the big picture. Collaboration occurs in more than name only if ownership of the project and process is shared. However, as Sheppard points out, solid leadership is still necessary. These people “help shape the creative space, move the project forward, and maintain equity and balance/'16It is their role to bring together the talents of diverse contributors and to keep the project momentum on schedule. Great Chicago Stories was administered from the Chicago History Museum, and as the grant recipient, Museum staff led collaborative partners from many institutions who came together to develop the project. With their overarching knowledge regarding all the pieces o f the collaboration, project leaders are in the position to identify potential problems and act to resolve them. They are also able to recognize success and build on emerging strengths. With the big picture in mind, project leaders must bring these issues to the collaborative table. In Great Chicago Stories, the phased process o f developing and testing narratives, curriculum, and multimedia features was crucial to the creation o f a high quality resource. As the process evolved, we realized the benefits o f allowing ample time for testing and revision. To that end, the project leaders recommended reducing the project scope by eliminating a grade level. This reduction of scope allowed us to continue the phased development process necessary to maintain excellence. In another case, after reflecting on the first summer development session, we realized that we could accomplish more intensive work during the second summer. The second session benefited from the foundation built during the previous year, which had established the basic parameters o f the project. Recognizing what had already been accomplished allowed project leaders to revise the design o f the second summer's work so that the team was able to devote time to curriculum development. This reassessment enabled us to add new dimensions to the project. Evaluation is one o f the main tools that project leaders can use to keep an initiative on track. Evaluation should be a key element in planning, and work as a continuing check on progress. The authors o f True Needs, True Partners: Museums and Schools Transforming Education reflect on the advantages that are gained when assessment and accountability are present in a project from the start: “Using formal and informal evaluation, you can test a pro-

P A R T N E R S IN P R O C E S S

37

totype partnership, check progress towards goals, keep logistics on track, and assess outcomes.”17 In the Great Chicago Stories project, we worked with an outside evaluation company. One o f their first tasks was to conduct a literature review that all the partners in the collaboration read and discussed during facilitated meetings. The literature review enabled project participants to begin their work with a solid understanding o f the state o f research regarding teaching and learning through narrative. The reflective practice approach employed in our project assessment used theoretical frameworks (shared in the literature review), evaluation (timely data collection to contribute to a culture o f continual improvement), and action research (theory-based investigations). The goals o f the evaluation were to heighten the project team's knowledge regarding using the narrative form for teaching history, inform materials development, and ultimately to improve history education. Methods included the literature review, advisors' reviews (checking for adherence to best practice and progress in realizing project goals), materials review (prior to pilot testing with a focus on content accuracy and best use of narrative form), systematic data collection, and continual reflective conversations. The data collection efforts used pre-and post-experience test questions for students about content knowledge, levels o f understanding, and historical frames o f mind. Classroom observations, teacher logs, debriefing interviews, written teacher surveys, and analysis o f student work rounded out the data collection activities. Data was collected from seventeen Chicago schools, eight elementary and nine high schools. Sixteen teachers, 281 elementary students and 163 high school students participated.18 This ongoing process involved all the participants and provided consistent feedback that kept the collaborators informed regarding progress in reaching project goals and provided timely information that allowed us to make necessary adjustments. Effective project leadership identifies and resolves problems and builds on success by using evaluation activities and data to refine the project as work progresses.

BEYOND THE COLLABORATION At the conclusion of any collaboration, it is important to reflect on outcomes and envision the future. The development of the Great Chicago Stories project is now complete, with a host o f free resources available via the Web site to educators across the country and around the world. In reflecting on the outcomes

38

H E ID I M O IS A N

from Great Chicago Stories, we knew that we had achieved our fundamental project goals. We had created powerful narratives that serve as points o f entry for exploring key humanities themes and Chicago’s history through the collections of the Chicago History Museum. Great Chicago Stories proved to be valuable to students and teachers. The evaluation results indicate that students showed gains in three kinds of learning: content knowledge (on average there were 43% more correct answers on the post test), skill building (most seen in the areas o f vocabulary and reading and literacy skills for the elementary students and in the areas of research and writing at the high school level), and in understanding and historical habits of mind (students were able to interpret information, explain events and express empathy).19 The evaluation also revealed that students guided their own learning, they were engaged in inquiry, and they exhibited high levels of engagement, while teachers showed increases in knowledge about Chicago history, strategies for teaching history, and awareness o f Chicago History Museum resources.20 At the Chicago History Museum, we know that these results would never have been achieved in the absence of a collaborative approach. The Great Chicago Stories project developed a true community of learners who pooled their expertise to produce a high quality instructional resource. Although the Great Chicago Stories Web site is completed, the possible applications of the project continue to unfold in ways both planned and unplanned.

CONSIDERING THE FUTURE First and foremost is pondering how to continue the relationship between the teachers and the museum. Rather than trying to keep the same relationship in place when a collaborative project is completed, we have opted to look for new opportunities for connection. The teachers involved in the Great Chicago Stories project have transitioned from that role to having other contacts with the museum. Two o f the participants serve on our teacher advisory board. Great Chicago Stories alumni attend professional development courses we offer, and one teacher brought her entire graduate school cohort to a program. It is gratifying to see the teachers draw on Great Chicago Stories to inform other projects. For example, one teacher who enrolled in an object-based learning course we offered developed an artifact collection centered on one of the Great Chicago Stories narratives. Less formal contact is frequent with

P A R T N E R S IN P R O C E S S

39

teachers asking museum educators to provide job and graduate school recommendations. Another teacher called to say she had banded together with a group o f like-minded colleagues to incorporate more museum resources into classroom instruction and asked if she could bring the group to the museum for their first meeting. On the museum end, we now have a pool o f teachers that we can turn to for expert advice, and classroom access is as simple as a phone call or email away. With the Web site complete, it serves as a platform to offer professional development courses. Great Chicago Stones teachers join museum educators as presenters to share their instructional experience and the unit plans they designed.

CONCLUSION The Great Chicago Stories project taught us that collaboration is within our grasp, not just for highly complex or highly funded projects, but for every level o f endeavor. In her essay iCStudents, Teachers, and Museums: Toward an Intertwined Learning Circle/5Griffin writes, “It is clear the way forward is to weave the three strands [students, teachers and museums] into a full circle through understanding, sharing, mutual respect, and collaboration. This involves an exciting challenge: listening to the needs, wants, and concerns of each group, learning from each other, and working collaboratively toward a shared goal of enjoyable and meaningful learning experiences.5521 Great Chicago Stories allowed all o f us to grow in our professional practice by demonstrating that with a shared vision, committed partners, and solid planning, museum educators and classroom teachers can work together to enrich student learning in a much more impactful way than schools and museums can accomplish on their own.

Literature Cited Hursey, Ellen, ed. Excellence and Equity: Education and the Public Dimension ofMuseums, (Washington, DC: American Association of Museums, 1992). Lang, Caroline, John Reeve, and Vicky Wollard. The Responsive Museum: Working with Audiences in the Twenty-First Century. (Great Britain: Ashgate, 2006). Lundin, Stephen C., Harry Paul, and John Christensen. Fish! A Remarkable Way to Boost Morale and Improve Results, (New York: Hyperion, 2000). Sheppard, Beverly. Building Museum & School Partnerships, (Washington DC: American Association ofMuseums, 1993).

40

H E ID I M O IS A N

Notes 1. National Endowment for the Humanities, “Grants for Teaching and Learning Resources and Curriculum Development,” http://www.neh.gov/grants/guidelines/teachinglearning. html (accessed March 2004). 2. Monica Edinger and Stephanie Fins, Far Away and Long Ago: Young Historians in the Classroom, (York, ME: Stenhouse Publishers, 1998), 4. 3. Janette Griffin, “Students, Teachers, and Museums: Toward an Intertwined Learning Circle.” in In Principle, In Practice: Museums as Learning Institutions, eds. John H. Falk, Lynn D. Dierking, and Susan Foutz. (Lanham MD: AltaMira Press, 2007), 42. 4. Frederick C. Mish, ed., Merriam Webster's Collegiate Dictionary: Tenth Edition (Springfield, MA: 1993), 224. 5. Beverly Sheppard, “Meaningful Collaboration,”182. 6. Ellen Cochran Hirzy, ed., True Needs, True Partners: Museums and Schools Transforming Education (Washington D.C.: Institute of Museum Services, 1996), 49. 7. Illinois State Board of Education “Resources/ISAT: Student Assessment” http://www.isbe. state.il.us/assessment/isat.htm (accessed September 15, 2008). 8. Hugh B. Price, Mobilizing the Community to Help Students Succeed (Alexandria VA: Association for Supervision and Curriculum Development, 2008), 90. 9. Sheppard, “Meaningful Collaboration,” 184. 10. Mike Schmoker, Results Now: How We Can Achieve Unprecedented Improvements in Teaching and Learning (Alexandria, VA: Association for Supervision and Curriculum Development, 2006),109. 11. Grant Wiggins and Jay McTighe, “Examining the Teaching Life,” Educational Leadership 63, no. 6 (2006), p. 29. 12. Jay McTighe and Grant Wiggins, Understanding by Design Professional Development Workbook (Alexandria, VA: Association for Supervision and Curriculum Development, 2004), 3. 13. Jay McTighe and Grant Wiggins, Understanding by Design Handbook (Alexandria, VA: Association for Supervision and Curriculum Development, 1999), 5. 14. McTighe and Wiggins, Understanding by Design Professional Development Workbook, 243. 15. MEM and Associates, “Summer 2006 Surveys” (July 2006) 16. Sheppard, “Meaningful Collaboration,” 185. 17. Hirzy, True Needs, True Partners, 55. 18. MEM and Associates, “The Power of Stories: The Impact of Great Chicago Stories on Students, Teachers and the Chicago History Museum,” (December 2007), 8-15. 19. MEM and Associates, “The Power of Stories”, 22-41. 20. Ibid.,17. 21. Griffin, “Students, Teachers and Museums,” 42.

Heidi Moisan is the School Programs Manager at the Chicago History Museum ([email protected]). Moisan served as project developer on Great Chicago Storiesw hich received the Gold M U S E award, an EdCom Award for Excellence, and was named as Best o f the Web in 2008.

Creating Communicative Scientists A Collaboration between a Science Center, College, and Science Industry

Melissa Wadman, Wendy deProphetis Driscoll, and Elizabeth Kurzawa

Abstract

Many science centers have partnerships with schools, uni-

versities or scientific industry.1This article will describe a unique collaborative project between Liberty Science Center, Wagner College, and Picatinny Center (a government research center) that has college interns working with and learning from science center staff and real scientists in a project to develop material science activities for middle school students. The project aims to create future scientists that can effectively communicate using inquiry and educational techniques used in informal learning. The focus o f the article is on the college interns and their experiences working with a science center and scientists and their learning of inquirybased education techniques.

Today, more than ever it is important to develop a scientifically literate society.2It is important for the public to understand how science affects their lives. As stated in Taking Science to School: “[ ]... in learning science one must come to understand both the body of knowledge and the process by which this knowledge is established, extended, refined, and revised.”3 To achieve this it is necessary to create scientists who are able to effectively communicate with the public. It is here that the museum can play an important role by bringing together students (future scientists), educators, and real scientists to work together to foster learning and understanding. As Larry Bell says, “Graduate students from colleges and universities can practice their skill at communicating with the public while supporting science centers' Public Understanding o f Research efforts.”4Along with working with colleges Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 41-54. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-5

41

42

M E L IS S A W A D M A N E T A L .

and universities, science centers can also “provide public education and outreach opportunities for research organizations that have federal grants mandating that their research projects include such activities.”5 The partnerships between museums and higher education and between museums and science research and industry are strong in and o f themselves but what if all three are combined?

DESCRIPTION OF MATERIALS WORLD OUTREACH PROGRAM The Materials World Module Outreach Program is an innovative collaborative project involving three primary partners —Liberty Science Center, Picatinny Center, and Wagner College —and various formal education institutions in New York and New Jersey. Although other programs have involved collaboration between a science center and a university, few examples o f collaborations between three partners —namely a science center, college, and industry—exist in the literature. The Picatinny Center funded this collaboration to develop an inquirybased science outreach program that would connect middle school students, educators, college students, and scientists. The outreach program's content is adapted from Northwestern University's Materials World Modules (MWM), which were developed through funding provided by the National Science Foundation. Within the current program, interns from Wagner College develop, implement, and revise 45 minute lessons based on the Materials World Modules incorporating scientific inquiry methods for middle school classrooms.6According to the National Science Teachers Association, scientific inquiry is “the diverse ways in which scientists study the natural world and propose explanations based on the evidence derived from their work. Scientific inquiry also refers to the activities through which students develop knowledge and understanding o f scientific ideas, as well as an understanding of how scientists study the natural world''7There are many different levels o f inquiry ranging from more open-ended, student-led exploration to much more structured, teacher-driven learning experiences. As is known in both the formal and informal educational worlds, science by inquiry is a powerful tool to facilitate students' science learning. Science by inquiry involves asking questions and using evidence to answer those questions. During investigations, students make observations and predictions, examine different variables, and use tools to collect and analyze information. Students then develop possible explanations for what they have

C R EA TIN G C O M M U N ICA TIV E S C IE N T IS T S

43

investigated and communicate and defend the results o f their investigations to others. Inquiry is intended to reflect the way in which scientists gather information and study our world.8 The student interns work closely with educators and scientists from Liberty Science Center and connect with researchers from Picatinny Center to revise the MWMs using scientific inquiry. The partnership introduces middle school and college students to materials science, fosters enthusiasm about science, connects middle school students with college science majors, and encourages the pursuit o f sciencerelated careers. The Wagner College interns are from a variety o f backgrounds including freshmen through seniors whose majors include biology, chemistry, and education. Their career goals have ranged from becoming a dentist, elementary school teacher, physician's assistant, or researcher to attending graduate school in chemistry, public health, or law school. The project also provides college students with valuable mentorship and professional development experiences that enhance their abilities, as future scientists, to communicate about science. This collaboration also seeks to establish a replicable model for how science centers, higher education, and scientific industry and the research community collaborate on joint projects. Interns in the first year o f the project introduced ceramics, smart sensors, and sports materials to three middle schools in the New Jersey and New York areas. The interns first mapped every module contained within the MWM kits developed at Northwestern to the New Jersey, New York, and national middle school science standards. Interns were split into three teams; each team developed a 45-minute lesson based on the MWM kits and conducted the mapping. The activities were then field-tested and assessed in a collaborative group that included college student interns, Liberty Science Center educators, and scientific researchers from Picatinny. As part o f professional development opportunities, two o f the interns presented their work at the Eastern Colleges Science Conference in the spring o f 2006. The program continued with funding from Picatinny Center through the 2006-2007 and 2007-2008 academic years. In the program's second year, eight Wagner College interns were selected through a competitive interview process. In the fall semester they implemented the previous year's lessons in classrooms. Working in teams o f two, they developed, implemented, and assessed four 45-minute lessons on concrete, food packaging materials, biodegradable materials, and biosensors in the spring o f 2007. These four lessons were implemented at four middle schools in the surrounding New York and New Jersey areas. In the summer o f2007, two interns developed an outline of

44

M E L IS S A W A D M A N ET AL.

a three-day residency program (a focused two to four day learning experience at Liberty Science Center centered on a chosen theme). The residency program Building Materials, which highlights Liberty Science Center's current exhibition, “Skyscraperl” used the ceramics, concrete, and composite materials modules and demonstrated how these materials can be used in buildings. The program continued in the 2007-2008 academic year when eight interns were selected to participate. In the fall o f 2007, the students revised the previous year's lessons and developed new lessons in the spring. O f these eight interns, one acted as a coordinator to provide feedback for all other interns. The remaining interns were split into three teams in which three interns developed the multiple day residency program, while the other two pairs o f students prepared lessons on smart sensors and polymers. These new lessons were implemented in four middle school classrooms. The residency team of students developed their program, elements of which have been prototyped with middle school students during the summer o f 2008. Four interns presented their work at the Eastern Colleges Science Conference in the spring of 2008, and one received an Outstanding Presentation Award.

EXAMPLES OF MATERIALS WORLD LESSONS Interns, with the help o f their LSC and Picatinny advisors, modified a variety of Materials World lessons to incorporate inquiry techniques. Through one lesson developed and then implemented in the spring o f 2007, interns introduced middle school students and educators to food packaging and biodegradable materials. The lesson's objectives were to analyze and compare properties o f various biodegradable and non-biodegradable food packaging materials through an inquiry-based lesson. Students compared biodegradable and non-biodegradable packaging peanuts, perforated foam, and bubble wrap and discussed material's composition and what makes a material an effective packaging for different products. The 45-minute lesson concluded with an egg drop experiment where students analyzed which materials worked best to protect an egg. In the spring o f 2008, the interns developed and implemented a lesson from the Materials World Modules to introduce composite materials. Students learned about composite materials —a mixture or combination of two or more visibly distinct materials —and how they are used in building materials. Students first investigated the properties of various composite

C R E A T I N G C O M M U N IC A T IV E S C I E N T I S T S

45

materials and learned about three types o f composites: particulate composites (e.g., concrete), laminar composites (e.g., plywood), and fiber-reinforced composites (e.g., graphite-reinforced frames o f tennis racquets). They then compared four different types of foam beams to find the benefits of composite materials over other types o f materials. In teams, students compared four foam beams: a plain foam beam, a foam beam with thick paper glued to one side, a foam beam with paper glued to both sides, and a disjointed system (a foam beam with paper glued to both sides with a gap in the middle of the beam where the paper was not glued down). Students clamped these foam beams to a table and added weights ranging from 50 to 500 grams to the end o f each beam and recorded any changes observed. Students then ranked the beams from weakest to strongest. Another team o f interns developed and implemented a 45-minute lesson on smart sensors. Students moved through stations containing different types o f smart sensors, such as a clapper, stuffed monkey containing a motion sensor, pedometer, children's book, and a guitar tuner. Student participants learned that smart sensors work by a stimulus triggering a response using a piezoelectric device. By the end o f the lesson, students were able to identify objects that use smart sensor technology, understand what a stimulus is and how it affects a piezoelectric device within a smart sensor, and know what type o f response is produced.

PROFESSIONAL DEVELOPMENT FOR COLLEGE INTERNS Throughout their participation in the program, interns received several types and levels o f professional development. Interns first received training on educational pedagogy and inquiry-based instruction to prepare them for lesson design and implementation. Scientists and educators coached students as they developed the lesson plans and residency materials. Before entering the classroom, students practiced teaching the lessons to a group o f educators, scientists, and peers and were given constructive feedback on content, classroom management, and teaching techniques. After prototyping the lessons with students, debriefing sessions were held that allowed interns to share their perspectives on the teaching experience and gain additional feedback on how the lessons worked in a real classroom environment. Interns were also provided with opportunities to present at professional conferences. As mentioned earlier, during 2006 and 2007 students presented posters and papers at the Eastern College Sciences Conference. The presen-

46

M E L IS S A W A D M A N ET AL.

tations documented the overall grant project, the interns' contributions, and data collected from students. In 2007, one student received an award for her poster presentation. In 2009, students are also preparing proposals about the project to select professional journals. Students who participated in the project for two years often took on leadership roles during their second year. These activities included mentoring new interns, taking on new responsibilities —such as the development of the residency materials —and serving as team leaders for the intern group. This scaffolding is another example o f how interns participating in the program took on the roles o f both “learner” and “teacher”.

INTERN FEEDBACK Feedback was collected from the college interns participating in the project by observation o f the prototyping sessions, informal interviews, and forms collected after prototype sessions as well as an online survey in the summer of 2008. For the online survey, past and present interns were asked about their experiences with the project. Eight of the 17 interns completed the survey. Interns reported that they wanted to participate in this project because they had an interest in science and an interest in education, that they had the desire to bring science through hands-on activities to middle school students, that they wanted to learn about a new area o f science themselves, and because the program links science with education. I was interested in the description o f the program when I began it a year ago. I was interested in learning more about a new area of science and being able to teach a younger generation these new concepts in a fun environment. The interns reported that they were very comfortable with the material they presented. One felt that the material was well targeted to the students' level. Another intern, who did not necessarily have a strong background in the subject, felt comfortable with the material. All o f the interns reported that they felt comfortable in front o f school groups because they were used to giving presentations at school and because the middle school students were engaged and interested in the activities. After prototyping the lessons in 2007, the interns felt that the lessons went well specifically because o f the hands-on aspect o f the activities. Interns

C R E A T IN G C O M M U N IC A T IV E S C I E N T I S T S

47

and staff observed students doing a great job making their own observations and using their critical thinking skills to learn while they were having fun. After implementing slight changes in the program, in 2008 interns reported that the programs went very or extremely well because the middle school students enjoyed the programs with their hands-on aspects and demonstrated learning by being able to answer questions and discuss the concepts in the lessons and that the lessons worked well in different school settings including special needs students. Interns reported that the lessons improved with each prototype and that they learned themselves throughout the process. The participation o f the students went really well. They were all excited to be doing an activity that involved hands on experience. Wagner College Intern Not only did I learn about materials science, but the students at various middle schools showed that they had also learned from our experiments. They were able to answer questions and demonstrated critical thinking abilities at the conclusion o f the lesson when they were asked various conceptual questions. Wagner College Intern An important factor to the success o f the lessons reported by the interns was the fact that the lessons were related to real life and real science —students were able to see what it is like to be a scientist through experiments included in the lessons. It was reported that there was good teamwork between the interns and project partners and the project made good use of the original lesson plans. They did not have to sit and just listen they actually got to see how a real experiment works and how scientists find their own answers. I think that participation went well because we asked a lot o f questions that required their attention. Students were intrigued by the fireflies more than the polymers because it was something that they could relate to in their own lives. Interns reported that they made gains (either a fair amount or a great deal) in their knowledge of material science, middle school science education, as well as inquiry-based instruction as a result o f their work on the project.

48

M E L IS S A W A D M A N ET AL.

OVERALL INTERN EXPERIENCE When asked what they learned from the experience, interns reported that they learned about material science, about the different aspects of being an educator—specifically in the middle school environment —as well as about LSC and its programming. Interns had positive experiences working with a science center, saying that LSC staff offered advice,insight, and knowledge, were friendly and approachable, and provided lots o f resources for the creation and presentation o f the lessons. I have learned a lot about the environment that middle school students are in everyday [sic]. I also have a better understanding of how Liberty Science Center works with various organizations to make a lot o f different programs happen. I have learned how to relate to younger students in attempts to get their interest and try to get them interested in science. Wagner College Intern I have learned about many aspects of materials science by researching information to create new lessons to bring to middle school students. I have learned how to conduct and inquiry based lesson while keeping young students interested in different materials science topics. I never had any teaching experience, and this internship has allowed me to explore a new area o f education and become very involved in all aspects, from writing lesson plans and researching new topics, to relaying this information to young students.Wagner College Intern Interest in science, materials science, education, teaching, science museums, working with children, and scientific research increased for the interns after participating in the project. Comfort levels also increased for the interns in regards to materials science and the subject matter specific to the lessons they worked on. Interns valued the experience o f being able to see the lessons they created working in the classroom. They took a sense of pride in the lessons and being able to affect students, specifically related to science education. They valued learning about materials science and the opportunity to work with a science center, scientists, and educators.

C R E A T IN G C O M M U N IC A T IV E S C I E N T I S T S

49

I valued being able to pass developing knowledge onto a younger generation. I was very happy teaching the lessons to the students and allowing them to learn about science in a way that they may have never experienced, through inquiry based lessons and handson activities. I enjoyed most leaving the classrooms and knowing that the students learned a lot about the topics while having fun, and that we had made a difference in their education. Wagner College Intern Interns did not report making any academic or career choices based on their experience with the project specifically but rather felt that it either reinforced choices they had already made or increased interest in different areas. For example it helped one intern to create a lesson plan for dental school, another to consider pursuing a higher degree in science due to a love of research, and another reported an increased interest in being an educator or even pursuing work in a science center. No, but it increased my passion for helping others learn. Wagner College Intern This experience actually helped me in one of my classes in dental school. I had to create a lesson plan based on oral hygiene for pre-k to 4th grade, and a separate activity for 5-8th graders. Both of these experiences made me want to teach after becoming a dentist! Wagner College Intern I was always majoring in science, but this definitely helped me realize that I love research and that maybe a higher degree in science is my calling. Wagner College Intern

PROJECT STAFF OBSERVATIONS OF THE INTERNS With regards to changes in intern behavior and performance, staff and professors noted a significant improvement in the Wagner College interns' presentation and leadership skills after their participation in the program. Liberty Science Center staff observed marked improvement between the fall and spring semesters in interns' overall confidence and ability in presenting lessons in the classroom. Interns prepared stronger lesson plans, explained challenging concepts in a much clearer fashion, were more organized with

50

M E L IS S A W A D M A N E T A L .

materials, and handled student management issues more assertively. This improvement was most notable in comparing the first run-through o f each lesson to the final implementation in middle-school classrooms. The improved oral presentation skills were also noticed in undergraduate courses where presentations were a component of the course. For example, many of the sophomore and junior interns participated in a senior seminar course at Wagner College. In general, the students who participated in the internship program appeared to be more polished and comfortable in front o f an audience than those who did not participate. Along with comfort level, the interns' teaching abilities also improved. Several interns were hired as undergraduate tutors after participating in the program, and many students who used these services on Wagner's campus provided positive feedback regarding these tutors.

CONCLUSIONS The MWM Outreach Program has brought together three partners in a program that is helping to create scientists that are better able to communicate with the public which in turn will help to create a more scientifically literate society. The project has assisted in helping college students in learning about informal education techniques, including scientific inquiry, which will then help them as future scientists in their public interactions. The project has also been able to successfully modify a number of the MWM lessons for a middle school audience by working with Picatinny scientists, college advisors, and LSC science educators. These lessons will be able to be used not only in the classroom but also at the science center as workshops in the Jennifer A. Chalsty Center for Science Learning and Teaching. The project has value for all partners involved. For LSC the project not only brings real science into the classroom through inquiry but also has been able to connect scientists with college students —both o f which are institutional goals. For Wagner College the project directly correlates to the Wagner Plan for the Practical Liberal Arts, which is a curriculum that utilizes learning communities to link courses from different disciplines. Through these learning communities, the college emphasizes civic engagement and community service, which makes the current program an ideal fit for Wagner College students who are interested in becoming involved in their communities through a science center collaboration. For Picatinny Center the project

C R E A T I N G C O M M U N IC A T IV E S C I E N T I S T S

51

affords scientists the opportunity to share expertise and their scientific research with college students. As an added bonus, the New York and New Jersey middle schools where the lessons were prototyped benefited from receiving lessons in a new subject matter (materials science) to which their students would not otherwise be exposed. The MWM project will continue to translate modules for use in middle schools and at Liberty Science Center in their science workshops as well as the school residency program. One o f the current lessons in development is about nanotechnology. The project team is also in the process o f using this project as a model for museum/university/scientific industry partnerships to disseminate across the country.

ABOUT THE PROJECT COLLABORATORS Liberty Science Center Liberty Science Center is an innovative resource for the lifelong exploration of nature, humanity and technology; strengthening communities; and inspiring global stewardship. Located in historic Liberty State Park, neighboring the Statue o f Liberty and across the river from Lower Manhattan, Liberty Science Center first opened its doors in 1993. But it truly began in the 1970s when a series o f editorials in the Star-Ledger newspaper pushed state leaders to recognize that more educational resources were needed to build a science- and technology-sawy population. A group o f local corporations initiated a successful public/private partnership in the 1980s, raising $68 million to create Liberty Science Center. Since that time the Center has become one o f New Jersey's most popular attractions, an important tool for building a well-educated workforce, and a daring pioneer in the global science museum field.

Wagner College Wagner College is a four-year private college located in New York City with approximately 1900 undergraduate students. In 1998 Wagner College implemented the Wagner Plan for the Practical Liberal Arts, a curriculum which utilizes learning communities to link courses from different disciplines. Through these learning communities, the college emphasizes civic engagement, and community service.

52

M E L IS S A W A D M A N ET AL.

Picatinny Center Picatinny is a joint service armament research and development center whose mission is to support Army transformation goals. In an effort to streamline the acquisition process and deliver the armaments that soldiers need exactly when they need them —and at an affordable price —Picatinny has established increasingly close partnerships with universities and industry partners, involving them in collaborative efforts early in the research and development process. Picatinny uses unique laboratories and special facilities to evaluate prototype designs, thus reducing development cycle time. These facilities are also available to Picatinnjfs contractors and other government agencies that are part o f the national energetic consortium established by Picatinny and the Army Research Laboratory.

Notes 1. See the following, Kathleen Vandiver, Jon Bijur, Ari Epstein, Beryl Rosenthal, and Don Stidson, “A Museum Learning Lab/' The Science Teacher 75 (2008): 41-43; John Durante, “Real Science on Show.” Nature 449 (2007): 283; Amy.. Payne et al., “Communicating Science to the Public through a University-Museum Partnership.” Journal of Chemical Education. 82 (2005): 743-750; Janet Kelly, Ranae Stetson and Angiline Powell-Mikel, “Science Adventures at the Local Museum.” Science and Children 39 (2002): 46-48; Deborah Zinocola and R. Devlin-Scherer, “A University-Museum Partnership for Teacher Education Field Experiences in Science.” The ClearingHouse 74 (2008): 248-250; Franco Calascibetta, Luigi Campanella, Gabriele Favero, and Luisa Nicoletti, “An Aquarium as a Means for the Interdisciplinary Teaching of Chemistry.” Journal ofChemical Education. 77 (2000): 1311-1313; Scott G. Paris, KirstenM. Yambor, and BeckyWai-Ling Packard, “A Museum-Schools-University Partnership for Enhancing Children's Interest and Learning in Science.” Elementary SchoolJournal 98, no. 3 (1998): 267-298. 2. Bernard Schiele and Emlyn H. Koster, ed, Science Centersfor This Century (Editions Multimondes, 2000). 3. Richard A. Duschl and Heidi A. Schweingruber, Taking Science to School: Learning and Teaching Science in Grades K-8 (National Research Council of the National Academies, 2007), 26. 4. David Chittenden, Graham Farmelo, and Bruce V. Lewenstein, Creating Connections: Museums and the Public Understanding of Current Research (Lanham, MD: Altamira Press, 2004), 169. 5. Ibid., 181-182. 6. http://www.materialsworldmodules.org/ 7. National Science Teachers Association, “NSTA position statement: Scientific Inquiry, 2004.” www.nsta.org/about/positions/inquiry.aspx (accessed October 1, 2008). 8. National Science Teachers Association, “NSTA position statement: Scientific Inquiry, 2004.” www.nsta.org/about/positions/inquiry.aspx (accessed October 1,2008).

C R E A T IN G C O M M U N IC A T IV E S C I E N T I S T S

53

Elizabeth Kurzawa was the project manager at Liberty Science Center for the MWM project. She holds a BS in Psychology from the University o f Scranton and a teaching certification in Elementary Education from New Jersey City University. She has spent severalyears working in informal education settings developing curricula, managing grant funded projects and programs, and working with local school districts. Wendy deProphetis Driscoll is an assistant professor at Wagner College in Staten Island, New York. She obtained her PhD in Organic Chemistry from the University o f Wisconsin-Madison (2004) and a BA from the University o f Pennsylvania (1999). Her research interests are in the areas o f organic chemistry and chemical education. Melissa Wadman is currently Director, Impact Evaluation at Liberty Science Center and holds a MA in Museum Studies from the George Washington University and a BA in Anthropology from the University o f Connecticut. She has eight 8years experience in the museum evaluation field and has presented at professional conferences including the American Association o f Museums and the Visitors Studies Association.

Revisioning the Physical and On-line Museum A Partnership with the Coalition o f Knowledge Building Schools

Lynda Kelly and Susan Groundwater-Smith

Abstract

The Australian Museum has been working with the Co-

alition o f Knowledge Building Schools over the past five years. Students aged from 5-18 years have been advising the Museum on the development of exhibitions and programs, as well as how the Museum can best use the digital environment to showcase its research and collections. This paper outlines these projects and discusses how the findings suggest ways museums could construct learning experiences that have the potential to transform young visitors through a process o f changing as a person.

Over the past five years the Australian Museum, Sydney, has been working with students from the Coalition o f Knowledge Building Schools advising the Museum on the development o f exhibitions and programs, how to design learning experiences tailored to their needs, as well as how the Museum can best use the digital environment to showcase its research and collections. The Coalition consists of around 20 public and privately-funded schools across New South Wales who have joined together to undertake research across a broad range o f topics, both specifically related to school activities and more general informal learning projects. The schools cover the spectrum o f years from Kindergarten (students aged five years) to Year 12 (aged 17-18 years), as well as representing a broad range o f socio-economic circumstances and geographic locations. This paper describes three Coalition/Museum projects, reflecting on ways to provide engaging learning experiences for young people, and in doing so, for all visitors.

Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 55-68. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-6

55

56

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

THE CASE FOR CONSULTING YOUNG PEOPLE When we consult young people and treat them seriously, whether designing the learning spaces, the organization o f the learning, or the learning experiences, it is possible to develop a product or process that will have greater relevance for them and one with which they will wish to engage. Designing for learning must not only relate to re-conceptualizing places and spaces when developing or redeveloping facilities and programs, but equally importantly, re-examining old and sometimes “tired” learning environments that are antithetical to transformative learning, which is the type of learning that the Australian Museum seeks to achieve. “Transformative learning” is a dynamic process dependent on the individual and their environment within a social context that focuses on some change. Ultimately, this learning is about “changing as a person” —how well the museum visit inspires and stimulates people into wanting to know more, as well as changing how they see themselves and their world both as an individual and as a part o f a community.1 Designing for positive learning outcomes is currently being led by those who are challenging the conventions that are institution-centric and turning to those that are learner-focused. What is sought is a consultative model that is flexible and responsive and accounts for the various technological convergences, new knowledge and better understanding of an interactive pedagogy.2 Contemporary understandings of learning in museums have indicated that substantive engagement is not merely a whim o f the educational imagination, but is critical to transforming learning in productive and positive ways.3Young people need to feel welcomed and supported, with museum exhibitions and programs that relate to their lives and highlight ways in which learning can apply in their own life worlds. Young people need to have some control over their learning and be involved in authentic problem solving and inquiry tasks. One way to meet this need is to set challenging but achievable goals that enable them to question and reflect on what they have learned. Sharing new knowledge with others in a multi-directional flow—student to student, student to adult, adult to student is also critical. Over the past five years, in response to a variety o f internal staffing changes and external pressures to provide more visitor-centered experiences, the Australian Museum is developing an ethos where consulting young visitors to the Museum is considered an integral part o f its audience research

R E V IS IO N IN G T H E P H Y S IC A L A N D O N - L I N E MUSEUM

57

practice through developing networks and partnerships with these audiences.4 To illustrate this approach, three major studies undertaken with the Coalition o f Knowledge Building Schools will be described and implications from each highlighted.

STUDY 1: AS WE SEE IT: IMPROVING LEARNING AT THE MUSEUM In 2003 Museum management in consultation with public program staff decided to review the kinds o f learning experiences being offered, focusing on questions such as how visitors make, or not make, meaning from the objects that are displayed, the forms o f display, the accompanying text (whether print or digital), the physical settings and staff who are available to provide assistance o f one kind or another. From this the study, “As We See It: Improving Learning at the Museum”, was conceived to engage both Museum staff and young people in a process to reflect on ways current Museum practices help or hinder learning. This study was the first project undertaken between the Museum and the Coalition, and consisted o f two phases. The first, focusing on the Museum's own practices, involved a range o f education and interpretive staff gathering photographic images which they felt assisted or inhibited visitor learning in the Museum. Their images were collated as posters which conceptually linked the images and provided additional text that acted as signposts for the viewer. A series o f discussions and workshops were held with staff participants to share reflections and come to a collective understanding o f what an ideal learning experience could look like from the Museum's perspective. Overall staff felt that by both having access to specialist personnel and actual objects, as well as presenting new and provocative ideas, it was possible to challenge established values and promote serious questioning o f current practices. Imaginative, quirky, surprising, and innovative displays were seen to be stimulating for the learner. For example, in the Skeletons exhibition, the skeletons positioned on an exercise machine and in a rocking chair were not only seen as amusing, but also enabled the viewer to understand something of the structure and function o f the skeletons on display. As well, it was appreciated that staff work behind the scenes to develop resources to assist in making the Museum visit a worthwhile one, while engaging in activities required for good customer service. Much was made of the live exhibits and the opportunity to engage in multi-sensory, hands-on learning through touch,

58

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

examine, query, investigate, and imaginative play. There was recognition that individual needs should be met, whether by interacting with visitors with disabilities or by assisting self-guiding visitors for example. Learning was seen to be assisted not only by interactions with staff (including volunteers) but also with parents and other accompanying adults. It was also understood that by meeting physical needs in the provision of food, drink, bathrooms and change rooms, people would feel comfortable and therefore amenable to the learning opportunities that the Museum offered. In the second phase, students from a range of Coalition schools visited the Museum and were given disposable cameras to document what they felt helped and hindered their learning. The students also made posters which they then presented to Museum staff. Overall, the issues in relation to learning in the Museum that were highlighted by students fell into four categories: cognitive, physical, social, and emotional. In the cognitive area, in order to be substantively engaged in learning in the Museum, students felt they needed to: know how things worked; be able to think through ideas; have opportunities to ask questions; be able to handle, manipulate, and closely examine artifacts and exhibits; be able to seek out information from several sources in language that is appropriate to their age and stage o f development; and be stimulated through all their senses. Regarding physical factors, students wanted to feel safe and comfortable and to move around readily unimpeded by a number o f prohibitive signs. They also wanted areas to be well-lit and inviting and find physical spaces scaled to their ages and needs. It is widely understood that museums are inherently social experiences5 and this study found that students like learning with their friends. While they recognized that a visit to the Museum was primarily designed by their teachers to assist in their learning, they also wanted it to be a satisfying social occasion when they could learn with and from their peers. When looking at emotional factors, they expressed a desire to be emotionally connected, while at the same time not emotionally confronted. This is best exemplified by two exhibits they discussed in their presentations. Year 11 students (aged 16-17 years) spoke o f their visit to the Jewish Museum and the opportunity to speak with survivors o f the Holocaust. They found this emotionally demanding but exceptionally worthwhile. In contrast, their experience with the Indigenous Australians exhibition at the Australian Museum failed to connect in the same way as they felt it was too fragmented with some elements inadequately explained. In the second example there was an exhibit in the Museum's Human Evolution exhibition that showed a leopard

R E V IS IO N IN G T H E P H Y S IC A L A N D O N - L I N E MUSEUM

59

eating a human child. This invoked both awe and fear simultaneously—that a creature could attack its prey in such a way clearly engaged the students. Even though they were in no personal danger, many still felt frightened and threatened by the exhibit, and it was certainly memorable for many of them. The students in this study indicated that they welcomed opportunities to be fully engaged with provocative questions, fascinating and puzzling exhibits, and clear, well-structured and accessible information. Whether they experienced small, live crocodiles with very sharp teeth, the wonders o f the minerals collection, the opportunity to discuss racism issues, or being able to drive the movements of a skeleton, they all found ways of learning in the Australian Museum. A good summary o f their experiences is through the words of this group o f boys: In our conclusion the Museum is a great learning environment but with a few touch ups every now and then it would be excellent. Overall we liked the Museum and we think it is improving every year. We enjoyed the day and learned something we didn't know before. Findings from this study were used as background to a successful application by the Museum for funding to revitalize its exhibition spaces, as well as in planning both a series o f programs and exhibitions and identifying further consultations required over the following years.

STUDY 2: THE MUSEUM I’D LIKE/THE MUSEUM I’D VISIT In June 2004, the New South Wales government announced a major funding allocation to revitalize the Museum's collections and research facilities and the refurbishment o f a number o f heritage exhibition spaces. Two major new exhibitions were planned for the Museum: Surviving Australia, using a range of animal and geological collections to illustrate and illuminate how both the forces of nature and people have shaped the Australian environment over time; and, Dinosaurs focusing on the great range o f dinosaurs that have lived in Australia and other southern continents for more than 200 million years. In creating this new look the Museum consulted its many audiences, with a particular emphasis on young people using the relationship already developed with the Coalition. This study was undertaken in two stages. In the

60

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

first, schools were invited to send to the Museum proposals for “The Museum Fd Like”. They were encouraged to be imaginative and not trammelled by convention or cost using whatever medium they thought appropriate. The second stage of the consultation, “The Museum I'd Visit”, became known as Kids’ College where participating students spent two days developing the two exhibitions with Museum project team staff who acted as facilitators and note takers. A series o f exercises were conceived to engage students and seek their feedback in fun and interesting ways. These included asking them to respond to the exhibition themes as well as design sections o f each exhibition that the project team was having difficulty with. The young people involved in both phases o f the consultation provided the Australian Museum with rich data that was used to inform designing for learning in the new exhibitions. As Lemerise stated “[Young people] say loud and clear that they like being with their friends and they want to do things with them... Alas the museum is generally seen as a place where there is not much to do other than look and listen.”6 The data collected in the study pointed to a desire for young people to be engaged in active and interactive learning. They wanted to be in spaces that are colorful and vital where all their senses are brought into play. Participants in Kids’ College wanted to have a sense of agency where they can follow a variety o f learning pathways employing a variety o f media. Young people operate in a media rich environment that is fun, and the word Tun' appeared many times in the data, for example: I want to have fun. Learn a hit about the Museum and science. See a lot of interesting things associated with science. But it is not only a form o f entertainment that these young people were seeking. They were also curious about how the Museum goes about its business o f researching, collecting, and preparing collections for exhibitions. The experience o f hearing from Museum experts was for them one o f the highlights o f Kids’ College. They were anxious to hear the stories behind the exhibits. Museums are being increasingly encouraged to see that a core function that they serve is that o f enhancing the learning o f those who visit them, whether in a formal sense or as one making a serendipitous foray into their spaces. Falk categorized learner motivation into five clusters: explorers (attempting to satisfy their curiosity), facilitators (enhancing the learning of others), professional hobbyists (seeking for specific knowledge), experience seekers (collecting an experience, usually on the recommendation o f others) and spiritual pilgrims (those who have a reverence for the particular museum's focus).7 The young people who attended Kids’ College expressed their

R E V IS IO N IN G T H E P H Y S IC A L A N D O N - L I N E M USEUM

61

motivation and engagement in the experience within each o f these categories. They were deeply curious, anxious to assist in the learning of others, wanting to know more about matters that interested them and about which they already had some knowledge, keen to enjoy a range o f experience and, finally, they indeed did find the Museum to be a place that filled them with awe and wonder. Overall it was found that the following design principles should be taken into account when developing exhibitions that will attract and engage young people: • Establish coherent goals and aims. • Identify the necessary knowledge domains and skills required to derive maximum benefit. • Provide conditions that give the learner an opportunity to explain, inquire, interpret, apply, and connect at a variety o f levels and in enjoyable ways. • Look for ways o f attracting and holding interest. • Differentiate to allow for varying experiences and abilities. • Engage young people in deep knowledge and understanding that requires o f them higher order thinking. • Facilitate active meaning making by going beyond simply telling. • Support varying levels o f interaction with exhibits, including hands-on experiences. • Provide conditions for learner discovery. • Provide social support by treating all participants with dignity and respect. • Render museum practices visible. • Be alert to feedback. • Allow young people to make an ongoing contribution. One o f the strengths o f the Museum/Coalition partnership has been in seeing the consultation as an exchange where both parties have an equal voice. Using John Dewey as a touchstone in his discussion o f education as a core mission of museums, George Hein recognized that social issues are part and parcel o f a museum's work and that they have a responsibility to be inclusive and democratic. He argued that strengthening democracy could be achieved in two ways: iCFirst museum education activities and exhibitions may focus on the skills that assist visitors in learning to learn, improving their ability to become critical thinkers.... The second way is for museums to

62

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

directly address controversial issues, taking the side o f social justice and democracy.”8 In this study, the young people were easily able to articulate what they did and did not like in museums generally, and the Australian Museum specifically, as one group declared: In conclusion, there is one thing our museum would definitely not contain: Rows of boring glass cabinets filled with items to be viewed but not touched! The study not only provided rich sets of data that the Museum continually uses in planning and developing programs but also laid the foundation for a successful way o f consulting with young people through inviting them in for intensive consultation preceded by tasks undertaken at school with their peers. This approach was further refined for the next study.

STUDY 3: RE-VISIONING THE DIGITAL ENVIRONMENT FOR YOUNG PEOPLE AT THE AUSTRALIAN MUSEUM Young people have easy access to technology which provides strong links between their social behavior and ways they gain knowledge. Both Green and Hannon and Li9 argued that the ease and affordability of on-line connection are having a profound effect on social structures and learning and that peer-topeer interaction is increasingly being perceived as a source of information as opposed to some central pedagogue or authority. Young people are learning to use a huge range o f digital technologies in their homes and other sites in the community beyond schools, and they also use them differently. For example, a recent report demonstrated that young people make clear distinctions between social sites and news sites, and prefer to access their news via tailored feeds.10 Given that learning through social media and digital resources is increasingly becoming a core function in the learning repertoire o f today's students11 it was decided to run an e-Kids' College with participants from the Coalition to investigate how they were using the Web and social media in particular. The format was similar to the previous college; however, it was further refined to decrease the intensive time spent at the Museum and increase the activities undertaken with their peers prior to attending the college. An important research component was the need to seek feedback and advice about how the Museum's research and collection could be better utilized through digital media to match audience needs and interests. This was particularly significant as the Museum had decided to completely rebuild its web site. The current site is a huge resource with around 40,000 pages and

R E V IS IO N IN G T H E P H Y S IC A L A N D O N - L I N E M USEUM

63

over 20 million visitors a year, yet offers little in the way o f two-way interaction and is very difficult to navigate and search. Twenty-four students from nine schools across New South Wales attended a one day workshop in November 2007. Students were consulted on a range o f issues encompassing their use o f digital technologies in both leisure and learning. They undertook a behind-the-scenes tour o f the Museum, spoke with a number o f scientific staff, and experienced the Museum's public areas in order to provide feedback about the Museum's potential on-line offering. Prior to their visit, they each interviewed ten o f their peers about their general views o f the internet, where they accessed it, and how comfortable they felt with technology. Unsurprisingly, all students reported that they felt totally comfortable with technology in general, giving interesting perspectives about the internet, for example: I enjoy using computers and digital technologies to learn because there are so many possibilities and it is a lot more interesting than a pen and paper. But technology can be tricky and break down very easily and it isn't always reliable. But with technology, almost anything can happen. Surfing the net is fast, exciting and surprising and different. Without the internet the world would be bland. When asked to complete the sentence “Not being able to access the web is like not being able to ..." respondents likened it to not being able to breath, live, eat, talk, socialise, and Get access to water, as well as Travel around the world, explore my inner selfor broaden my horizon. One interesting finding was that the students had mixed views about the Museum's presence on social networking sites such as MySpace and contentsharing sites like YouTube and Flickr. Some felt that it would make the Museum look too “try hard" or “uncool" and others thought that these sites were primarily for socializing and leisure, not associating them with education: I don't really see how you can use things like MySpace, YouTube etc. Most people would not even consider watching a video from the Museum. I think the best thing to do would be to keep the Museum web site always up to date and accurate. Make sure it comes up the top ofsearch engines, with actual paging. People never click on the sponsored links.

64

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

However, on reflection, several students felt that the Museum could utilize social spaces in a number o f unexpected ways, for example showing movie trailers o f upcoming exhibitions and events and even developing a page for a number o f animal species as a way to promote the Museum's work in a fun and light-hearted way: The Australian Museum should have a trailer and a page about their tours and expedition. The Museum could set up a MySpace,12 with bulletins, to keep contacts up-to-date, set as public. For Bebo, have a Bebo bank (make a page just for the Australian Museum). I think it would be interesting to set up a MySpace type profile for the different specimens, and use this to provide information. It would allow you to make it visually appealing with bright colors, and even movie and music clips. This would appeal to young audiences. Overall, they felt that content should primarily sit on the Museum's web site, for example: MySpace is more for advertising value than viewing content. Whatever I put out on the internet, it should be able to all be linked back to the Museum's web sitefor credentials—accuracy and reliability. In one of the day's activities, they were asked to answer the question: “If the Museum were a web site...". The overall consensus was that the site should be uncluttered, pleasing to look at, advertisement-free, jargon-free, simple, and easy to use: If the museum was a web site, it could be either boring or fascinating depending on the way that designers approached it. For a younger audience it is crucial to break up the factual information and present it in an interactive, appealing and creative way. It would be important to have it set up by a representative of the target audience, as 'second guessing' often proves inaccurate. If the Museum was a web site it would need a variety o f ways used to get the message across e.g., videos, pictures, podcasting. I would need to have a dedicated site for younger people, because different

REVISI ONI NG T HE PHYSICAL AND ON-LINE MUSEUM

65

target markets respond to different things, so only one type/theme site would not work. Primary, secondary, and public would be good because of different reasons for using the site. They believed that there should be a discussion board where questions could be asked and opinions mooted, games, interactivity, and more personal and informal staff information: [a] question o f the week, where children are able to write in and ask a question, the best question will be chosen and will be posted in a blog, this way people can comment on it. An interactive touch screen that would show different types o f animals (e.g., marine) and let you design a pattern on the animal. Have a link on your web site called 'scientists are people too' and have a character version o f a scientist and maybe try and portray their quirky ways. These findings resonate with an earlier study that examined students' and teachers' needs and wants relating to the internet generally, and museum web sites specifically.13 A number o f others also made a distinction between visiting the Museum itself and visiting the Museum's website. For example, one student who had participated in previous research projects with the Museum had this to say: Last time I came here we focused mainly on new technology, and we were constantly saying we needed more screens, games and interactive displays, but since then I have been thinking: I can do that at home, I can watch movies, play games etc at home. If I come to the Museum I want to be able to get information, read it and be able to learn from it. It is good to have these things (screens etc) but I guess, like all things, in moderation. The web site needs to suit all audiences. I got the feeling that you were trying to find out what we want but we are not the only people that use the Museum. A section on the site, with bright colors, games etc could be good, but it is unlikely that the reason we are at a Museum site in the first place is to play the games. We can do that anywhere. If we are there we are probably looking for information o f some kind.

66

LYNDA KELLY AND SUSAN GROUNDWATER-SMI TH

CONCLUSION Reflecting on the outcomes o f this work with the Coalition over the years we believe that in designing engaging physical and on-line museum experiences for young people, the principles are the same and include the following elements: • Experiences that encourage discovery and interaction, cater to the unexpected, provide many pathways to explore, give a taste for what happens behind-the-scenes, and are fun • Content that is challenging, real, authoritative, meaningful, encouraging o f questions, and is well-organized and easy to navigate • Staff that can relate to young people, are respectful o f their ideas and views, are knowledgeable in their field, and are easy to talk to • Opportunities to socialize, be with their friends, and learn together. This work is ongoing. The relationships formed with the Coalition schools, teachers and students has continued to be o f benefit to all. The Museum has learned new ways to engage young people in their exhibitions, programs, and web site development processes and an appreciation o f the perspectives and depth of understanding displayed by young people in their responses to these. The teachers see it as a great way to both reward some students and to develop others. The students themselves report not only having a great time, but appreciate being listened to and their views acted on. As one participant, when asked what was best about Kids College, stated: The whole thing I thoroughly enjoyed! I love all ofit and getting the chance to have my say. If I had to choose a favorite part of it all I would most definitely say being taken on a tour ofthe Museum andjust being able to state our opinion with meaning.

Literature Cited Falk, John. “An Identity-Centered Approach to Understanding Museum Learning.” Curator 49,2 (2006): 151-166. Falk, John and Lynn Dierking. Learningfrom Museums: Visitor Experience and the Making ofMeaning. New York: Alta Mira Press, 2000. Green, Hannah and Celia Hannon. Their Space: Educationfor a Digital Generation. London: DEMOS Foundation, 2006. Hein, George, “John Dewey's 'Wholly Original Philosophy' and Its Significance for Museums.” Curator 49,2 (2006): 181-204. Kelly, Lynda. “Evaluation, Research and Communities of Practice: Program Evaluation in Museums.” Archival Science 4,1-2 (2005): 45-69. Kelly, Lynda. Visitors and Learners: Adult Museum Visitors' Learning Identities. PhD diss., University of Technology, Sydney (2007).

R E V IS IO N IN G T H E P H Y S IC A L A N D O N - L I N E M USEUM

67

Kelly, Lynda and Kevin Breault. “What Makes a Good Educational Web Site?/’ In Thinking, Evaluating, Rethinking: Proceedingsfrom ICOM-CECA 2006 Conference, ed. E. Nardi. Rome: Universitie da tre (2007). Kelly, Lynda and Angelina Russo. “From Ladders of Par ticipation to Networks of Participation: Social Media and Museum Audiences.” In Museums and the Web 2008: Selected Papersfrom an International Conference, ed. Jennifer Trant and David Bearman, 83-92. Toronto: Archives and Museum Informatics (2008). Lemerise, Tamara. “The Role and Place of Adolescents in Museums: Yesterday and Today”. Museum Management and Curatorship, 14/4:393-408. 1995. Li, Charlene. Social Technographics: Mapping Participation In Activities Forms The Foundation ofa Social Strategy. Cambridge, MA: Forrester Research. 2007. Piscitelli, Barbara and David Anderson. “Young Children’s Perspectives of Museum Settings and Experiences”. Museum Management and Curatorship 19/3,2001: 269-282. Vahlberg, Vivian, Limor Peer, and Mary Nesbitt. If It Catches My Eye: An Exploration Of On-line News Experiences of Teenagers. Media Management Center, Northwestern University, 2008. Valenti, Mark. “Creating the Classroom of the Future.” Educause September/October, 2002.

Notes 1. Lynda Kelly, Visitors and Learners: Adult Museum Visitors' Learning Identities PhD diss. (University of Technology, Sydney, 2007). 2. Mark Valenti, “Creating the Classroom of the Future.” Educause (September/October, 2002 ).

3. Kelly, Visitors and Learners; Barbara Piscitelli and David Anderson, “Young Children’s Perspectives of Museum Settings and Experiences.” Museum Management and Curatorship 19/3 (2001): 269-282. 4. Lynda Kelly, “Evaluation, Research and Communities of Practice: Program Evaluation in Museums.” Archival Science 4,1-2 (2005): 45-69. 5. John Falk and Lynn Dierking, Learningfrom Museums: Visitor Experience and the Making of Meaning. (New York: Alta Mira Press, 2000); Kelly, Visitors and Learners. 6. Tamara Lemerise, “The Role and Place of Adolescents in Museums: Yesterday and Today.” Museum Management and Curatorship 14/4 (1995): p. 403 7. John Falk, “An Identity-Centered Approach to Understanding Museum Learning.” Curator 49,2(2006): 151-166. 8. George Hein, “John Dewey’s 'Wholly Original Philosophy’ and Its Significance for Museums.” Curator 49,2 (2006): 199. 9. Hannah Green and Celia Hannon, Their Space. Educationfor a Digital Generation. (London: DEMOS Foundation, 2006); Charlene Li, Social Technographics: Mapping Participation In Activities Forms The Foundation ofa Social Strategy. (Cambridge, MA: Forrester Research. 2007). 10. Vivian Vahlberg, Limor Peer, and Mary Nesbitt, If It Catches My Eye: An Exploration Of On-line News Experiences of Teenagers. (Chicago: Media Management Centre, Northwestern University, 2008). 11. Green and Hannon, Their Space; Lynda Kelly and Angelina Russo. “From Ladders of Participation to Networks of Participation: Social Media and Museum Audiences.” In Museums and the Web 2008: Selected Papersfrom an International Conference, ed. Jennifer Trant and David Bearman, (Toronto: Archives and Museum Informatics, 2008): 83-92. 12. MySpace (http://www.myspace.com/) and Bebo (http://www.bebo.com/) are social sites that members use to network and keep in contact with their friends, families, peers and others. Bebo, in particular, is primarily used by young people aged 10-18 years in Australia 13. Lynda Kelly and Kevin Breault, “What Makes a Good Educational Web Site?” in Thinking, Evaluating, Rethinking: Proceedingsfrom ICOM-CECA 2006 Conference, ed. E. Nardi. (Rome: Universitie da tre, 2007).

68

LY N D A K E L L Y A N D S U S A N G R O U N D W A T E R - S M I T H

D r Lynda Kelly is the Head o f Audience Research at the Australian Museum, Sydney. She has published widely in museum evaluation and writes the Audience Research in Museums (http://amarclk.blogspot.com/) blog with a readership o f around 5 0 0 per month. She also administers the Museum 3 .0 (http://museum30. ning.com/) social networking group where around 2 0 0 museum professionals discuss issues related to museums. Lynda is particularly interested in visitor experiences and learning and how these can be measured, young children’s learning Indigenous evaluation, as well as the strategic uses o f audience research and new technologies in organizational change. Susan Groundwater-Smith is Honorary Professor o f Education in the Faculty o f Education and Social Work at the University o f Sydney. She directs the Centre for Practitioner Research, a small facility that hosts the Coalition o f Knowledge Building Schools■, a hybrid network o f schools, large and small, metropolitan and regional, independent and government. The Coalition makes a contribution to professional knowledge formation, particularly in relation to student voice. Susan has published widely in teacher education and has undertaken a number o f projects in Museum Education in terms o f authentic learning outside the classroom.

Keep Your Friends Close The History o f a Museum Partnership and Its Community ofTeacher Learners

Kris Wetterlund

Abstract

ArtsConnectEd is a Web site that is the product o f a ten-year

partnership between the Minneapolis Institute of Arts and the Walker Art Center, bringing together online the collections of the two museums specifically for K-12 teachers and students in Minnesota and beyond. This article describes the history ofthe partnership and important lessons learned along the way. The article will also include information about a redesign of ArtsConnectEd currently underway, illustrating how the partnership has attempted to keep pace with the changes in technology that challenge both museum educators and K-12 teachers.

ArtsConnectEd (www.artsconnected.org) is a Web site that brings together online the collections o f the Minneapolis Institute o f Arts (MIA) and the Walker Art Center for K-12 teachers and students in Minnesota and beyond. The site includes enhanced teaching units, online activities, and material from the archives and libraries o f the two institutions, all in a searchable environment. ArtsConnectEd is the product o f a partnership between the Minneapolis Institute o f Arts and the Walker Art Center, two museums in Minneapolis, Minnesota with collections and programs that uniquely compliment each other. The Minneapolis Institute of Arts is an encyclopedic museum that houses collections spanning five thousand years and representing the world's diverse cultures across all continents. The Walker Art Center focuses on collecting contemporary art and is devoted to an interdisciplinary program o f performing arts, film, and visual art. The relationship between these two museums that resulted in ArtsConnectEd offers a rich

Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 69-78. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-7

69

70

KR IS W E T T E R L U N D

history o f leveraging individual resources, capabilities and expertise in a partnership for the benefit o f K-12 audiences.

PARTNERSHIP BEGINNINGS The partnership between the Minneapolis Institute of Arts and the Walker Art Center actually began long before ArtsConnectEd. In the late 1980s, education staff from the two institutions designed a summer teacher institute called ArtLink, using the collections of both museums to train educators in interdisciplinary teaching. Each museum published an education catalog for Minnesota teachers annually and, in the 1990s, collaborated on the design of the catalogs so they could be bundled together for mailing. Then, in 1997, the two museums began to collaborate on a Web site called ArtsNetMN (www.artsconnected.org/artsnetmn/) that included digital images o f a small number o f works o f art organized into themes designed to address curriculum used by Minnesota teachers.1 All these activities represented initiatives from the education departments of both museums. At the same time, Scott Sayre had undertaken the task of transforming the MLAs media department from an audio-video production facility into a center for interactive multimedia charged with producing interactive computer kiosks for the MLAs galleries. Named the Interactive Media Group, Sayre's department also began looking into Internet possibilities for the museum, establishing the museum's Web site in 1993 and creating online resources for teachers in 1995. In 1996 the Walker hired Steve Dietz to head up its New Media Initiatives Department, devoted to investigating new media potential for both collections and programming at the Walker. Scott Sayre and Steve Dietz were friends before Dietz joined the Walker staff, and the two began conversations about how they might work together on behalf of the two museums. At the same time an administrator at the Walker was pursuing state level funding opportunities and realized the potential for educational use of collections online. Sayre and Dietz conceived of a joint Web site for teachers that could leverage state funding for digitizing and integrating the collections and education resources of both museums. The plan succeeded, and in 1997 the MIA and Walker received $1 million from the state of Minnesota for the partnership project.2In 1997 Robin Dowden, an art historian and data expert formerly from the National Gallery in Washington, DC, joined New Media Initiatives at the Walker, and the core ArtsConnectEd team began working to make online access to collections and education materials a reality.

M USEU M P A R T N E R S H I P / C O M M U N I T Y O F T E A C H E R L E A R N E R S

71

THE POWER OF PARTNERSHIP The ArtsConnectEd partnership brought with it a range o f new financial and technical opportunities. State, federal, and corporate funders felt they could donate to one project that would benefit both museums. For example, Microsoft and others donated software to the project, taking advantage of the two-for-one impact. Before the MIA and Walker partnered, both museums had their Web sites on servers owned by a third party Web host. With the state funding the museums purchased their own high capacity Web server as well as a storage array, co-located at a high bandwidth facility. Once the two museums owned their own server, they gained the freedom to choose and install software most beneficial to the project and to configure the computer to best meet their needs. Since the Web server and software were shared by both museums for ArtsConnectEd, each museum contributed only 50% o f the cost. In addition, the server offered enough space to eventually house not only the ArtsConnectEd Web site but also the institutional Web sites for both museums. From a human resources standpoint, both museums benefited financially by taking advantage of each othefs strengths. According to Scott Sayre, “Production team members from both institutions working together gave the project a lot more capacity and a broader skill set than either museum would have had working on its own.”3 The museums also shared the cost of professional development for project staff. Photographers were trained in digital photography because neither institution was doing digital photography before the partnership, and staff from both museums were trained in metadata standards so the standards could be applied to both collections.

THE POWER OF PARTICIPATION Beyond the local museum partnership, both museums leveraged opportunities provided through their individual participation in larger, national initiatives. The Art Museum Image Consortium (AMICO) was a not-for-profit organization of institutions with collections o f art, collaborating to enable educational use o f museum multimedia. AMICO was launched right about the time ArtsConnectEd was being designed, and the museums participating in AMICO identified technical standards and descriptive standards for online

72

KR IS W E T T E R L U N D

access to museum collections. The theory was that once museums adopted these standards as a requirement for participating in AMICO, they would be able to contribute digital images o f their collections to AMICO, which would then license the digital art library to colleges, universities, and libraries for educational use. Because the MIA and Walker were both contributors to AMICO, they were already obligated to prepare their images and related object data according to an agreed-upon standard. So, the two museums used AMICO standards as a foundation, combining their collections by using shared standards and object data. The benefits o f participating in multi-institution collaborative projects like AMICO are often felt individually in each museum as they internalize best practices standards and put them to work in projects or policies that are museum specific, such as ArtsConnectEd. While AMICO eventually folded, the relationships with experts and professionals in other museums working on solving the same problems gave participants expertise to draw upon in the future.

COLLABORATIVE PRODUCTION With a technical infrastructure and standards in place, members of the Minneapolis Institute of Arts Interactive Media Group and the Walker's New Media Initiatives department worked as a single project team to organize the information resources and build the user interface and site that would become ArtsConnectEd. ArtsConnectEd as we know it today launched in 1998, and it was immediately lauded by the museum community, winning Best o f the Web Educational Site from Museums and the Web and a Gold Muse Award from the American Association of Museum's Media and Technology Committee in the spring of 1999. The technical team was triumphant, and it seemed that all the hard work had paid off.

REALITY CHECK In the summer o f 1999, the technical team took ArtsConnectEd to a usability lab, courtesy o f what was then Dayton Hudson Corporation's testing laboratory. The central objectives o f this testing was to study the operation o f the software, the intuitiveness o f the interface and navigation, and the general

M USEUM P A R T N E R S H I P / C O M M U N I T Y O F T E A C H E R L E A R N E R S

73

usefulness o f the online collections and related educational resources. Team members selected teachers and students as well as representatives o f the general public to represent a range of potential first-time users. A series of user-specific scenarios were written to test the study’s objectives. A lab facilitator introduced ArtsConnectEd to each user. Presented with scenarios, users were asked to speak their thoughts aloud as they attempted to complete the objectives using ArtsConnectEd. The assessment team, located behind a two-way mirror, could both audibly and visually observe the users’ processes. A video recording synchronized to a PC-based logging system was used to log all user actions as well as the comments o f the observers throughout each user testing session.4 The results of the usability studies were devastating. Users did not understand the interface, which was one central search field that queried all of ArtsConnectEd. Users also did not understand the search returns —a big pile of works of art, library records, archives, and educational material, based on terms the user typed into the search field. It seemed that, at this point in history before Google, people were not used to seeing multiple search results all in one return. They were baffled as to how to sort it all out. Users in the usability lab could not make sense of books next to works o f art next to tours.

BACK TO THE DRAWING BOARD The team returned to work and abandoned the initial interface design, which had helped win awards from their colleagues but did not work as far as the users were concerned. Fortunately ArtsConnectEd was and still is database driven, so only the user interface to the data needed to be redesigned; the database that is the back end o f ArtsConnectEd remained the same. The new user interface o f ArtsConnectEd was broken into sections that were more familiar to users: “Art Gallery” searches work o f art, “For Your Classroom” searches educational resources, “Library & Archives” searches library and archive records, and “Playground” is a container for online activities. “Search All,” previously the main search, was moved to its own section as an advanced feature rather than the primary function. A few months later, the team returned to the usability lab to test the new interface with a new set of users. This time, the users responded much more positively to the site, and the ArtsConnectEd team felt confident relaunching the site with the new interface.

74

K R IS W E T T E R L U N D

DEVELOPING PARTNERSHIP WITH K-12 TEACHING COMMUNITY In June o f2000, ArtsConnectEd received an additional award of a $1 million grant from the State o f Minnesota to continue the process o f digitizing MIA and Walker collections, to further the development o f additional educational resources, and to support statewide teacher training. As the development of ArtsConnectEd continued to grow, a more formal team structure was needed to achieve the long-term project goal o f serving the needs of K-12 teachers. Committees, including an Education Committee, composed o f representatives from both the MIA and the Walker were established to achieve these ends. The Education Committee faced the immediate challenge of persuading the K-12 teaching community to embrace ArtsConnectEd as a new centralized online resource. Recognizing that it was not good enough to simply provide ArtsConnectEd, and armed with funding from the state, the Education Committee began to explore strategies for training teachers to use ArtsConnectEd. The state o f Minnesota covers 84,068 square miles and contains 341 school districts that serve a population o f 4,610,000. The MIA and Walker—located in the most densely populated area o f Minneapolis —were proficient at serving the urban communities o f Minneapolis and St. Paul, but the majority o f Minnesota educators, who work outside this Twin City area, were grossly underserved. The need to address the inequities between the Twin Cities and the rest o f the state was o f primary importance for the MIA and the Walker, as well as the state legislature, whose members wanted their investment in ArtsConnectEd to reach all o f Minnesota. The Education Committee conceived a train-the-trainer model, based on one lead trainer developing a highly skilled core group o f teachers from around the state, who would in turn conduct teacher workshops in their regions. A core group o f 24 teachers, 18 from around the state and six from the Twin Cities metro area, was recruited. Members o f this core group contracted to teach at least four workshops o f 15 teachers each after they were trained, so 24 trainers would workshop 60 teachers each, resulting in a grand total o f 1,440 Minnesota teachers trained to use ArtsConnectEd in their classrooms. Theorizing that each teacher was responsible for at least 30 students during a school year, the Education Committee estimated that ArtsConnectEd could reach 43,200 students during the training period.5

M USEUM P A R T N E R S H I P / C O M M U N I T Y O F T E A C H E R L E A R N E R S

75

MANAGING GROWTH AND MAINTENANCE Once ArtsConnectEd was launched and teachers were trained, use of ArtsConnectEd skyrocketed, sometimes outpacing each partner museum's Web site.6The educators at each museum began to see the potential o f serving their teacher audiences online, and programming, such as slide sets and resource kits at the MIA, started to migrate to ArtsConnectEd in the form of online versions o f teacher-focused content. The Walker education staff ramped up internal use o f ArtsConnectEd too, by defining it as their central teacher program. While museum educators at the MIA and Walker were seeing the light regarding teachers using the Internet to connect with museums and collecting ideas about how to improve ArtsConnectEd, the technical team who created ArtsConnectEd moved onto other projects. Without the generous funding stream provided by the state, it was difficult for technical staff to continue to focus on ArtsConnectEd. As the potential for the Internet became more and more apparent to all museum staff, representatives o f other departments envisioned other funding opportunities and thus other priorities began to line up outside the doors o f the Interactive Media Group at the MIA and the New Media Department at the Walker. When museum educators came knocking with needed updates to ArtsConnectEd, they found themselves at the end of the long line o f other museum departments clamoring for service from the Web and multimedia staff. The partnership that conceived o f and created ArtsConnectEd had no long-term agreement regarding contributing technical resources to maintain the site. Close to a decade later, members o f both the technical team and Education Committee agreed that the project had to be rethought both politically and technically. Upper lever administrators were made aware o f the challenges and agreed to long-term support and commitment to a solution.

ENVISIONING ARTSCONNECTED 2 The original team that nurtured the vision o f ArtsConnectEd in the beginning was composed o f leaders o f the MIA's Interactive Media Group and the Walker's New Media Department, but it was the museum educators who led the charge for taking ArtsConnectEd to a new level. In 2006 ArtsConnectEd applied for and received an Institute of Museum and Library Services

76

K R IS W E T T E R L U N D

(IMLS) National Leadership Grant for creating ArtsConnectEd 27 The grant's primary purpose was to solve the emerging central problem o f updating and maintaining ArtsConnectEd. Both the MIA and the Walker agreed that the problem for ArtsConnectEd was not necessarily the lack o f a plan for maintenance, but rather educators having no control over their own online content. Putting the care and feeding of ArtsConnectEd into the hands o f the museum educators, using new Web 2.0 tools and content management systems, would free the technical team from updates and put the power o f ArtsConnectEd into the hands o f those using it on a daily basis to interact with their constituency o f teachers. Continuity of ArtsConnectEd is greatly enhanced by the original team members Robin Dowden, now Director o f New Media at the Walker, and Scott Sayre acting as an independent project manager for ArtsConnectEd 2. These two team members have been there from the beginning. Some of the original 24 core trainers have also joined ArtsConnectEd 2 as “Power Users" devoted to representing teacher use of ArtsConnectEd and the needs of today's K-12 teachers and students in the design of ArtsConnectEd 2. In addition, core work groups of museum educators from both the MIA and Walker ensure that the needs of museum educators are at the heart of the redesign.

LESSONS LEARNED A good partnership, like a good marriage, is seldom 50/50. There are always times when one partner is carrying more weight than the other, and that has certainly been true o f ArtsConnectEd. Staffing changes, shifting institutional priorities, the right blend o f technologists and educators, and the evolving Internet itself have presented daunting challenges for ArtsConnectEd over time. These challenges have been overcome by the commitment o f leaders in each museum's technology and education departments, but even that leadership has changed over time. Initially conceived and led by technology departments in each museum, the education departments in each museum picked up the baton and carried it forward to address ArtsConnectEd's maintenance problems that emerged after its successful audience development. ArtsConnectEd has flourished because each museum has recognized that partnership is a fluid relationship and accepted that while each museum may present challenges to the development o f ArtsConnectEd

M USEUM P A R T N E R S H I P / C O M M U N I T Y O F T E A C H E R L E A R N E R S

77

along the way, in the end those problems will not be permanent as long as the commitment to the partnership remains strong. Partnering with other institutions to purchase hardware, software and for professional development serves the partners with 100% o f the benefits of such arrangements but only a fraction o f the cost. The donation o f the usability lab and its staff for several days o f user testing was seen as a two-forone contribution by the corporation that made the donation, allowing one benefit to serve two museums in the community. Perhaps the greatest benefit o f the partnership that created ArtsConnectEd is that the two museums leveraged their resources, capabilities, expertise, and audiences for the benefit of K-12 teachers and students and thus for their own mutual benefit. Teachers who worked with both museums before ArtsConnectEd often got the impression that the museums were competing for a finite teacher audience. Since ArtsConnectEd began, teachers have expressed gratitude that the two museums are working together rather than competing to benefit teachers and students. The partnership between the two museums that resulted in ArtsConnectEd served to cement the museums' relationship to a primary constituency—K-12 teachers and students.

Notes 1. Barbara Bridges, “Collaboration and Student Involvement on an Art Curriculum Website” (Paper presented at the Museums and the Web conference, Toronto, Ontario, April 22-25, 1998). http://www.archimuse.com/mw98/papers/bridges/bridges_paper.html 2. The Minneapolis Institute of Arts and Walker Art Center, “Integrated Arts Information Access: Project Description and Reports” http://www.walkerart.org/iaia/ 3. Scott Sayre, personal interview with the author, September 9,2008. 4. Robin Dowden, Scott Sayre, and Steve Dietz, “ArtsConnectEd: Collaboration in the Integration and Access to Museum Resources.” First Monday 5, no. 6 (2000). http://firstmonday. org/issues/issue5_6/dowden/index.html 5. Scott Sayre and Kris Wetterlund, “Pyramid Power: A Train-the-Trainer Model to Increase Teacher Usage of the ArtsConnectEd Online Resource” (Paper presented at the Museums and the Web conference, Boston, MA, April 17-20,2002). http://www.archimuse.com/ mw2002/papers/sayre/sayre.html 6. At the end of the teacher-training period in 2002, use sessions for ArtsConnectEd totaled 154,000. Today, use sessions are over 2 million, http://www.artsmia.org/stats/ 7. Minneapolis Institute of Arts and Walker Art Center, “ArtsConnectEd 2 Project Planning Site.” http://ace2.artsconnected.org/

78

K R IS W E T T E R L U N D

Kris Wetteriund has worked with teachers as an art museum educator for the past eighteen years, beginning in the education department at the Minneapolis Institute o fA rts, and currently as Principal in Sandbox Studios, an independent consulting firm devoted to helping develop museum educational programs onsite and/or online. Wetteriund also is a founder ofMuseum-Ed, a not-for-profit professional development listserv and Web site for museum educators, and currently serves as its editor.

Impact ofthe National History Day in Ohio Program on Students’ Performances Pilot Evaluation Project

Giuseppe Monaco, Bo Lu, and Megan Wood

Abstract

This pilot study aimed at developing and validating an eval-

uation model to assess the impact o f participation in the National History Day in Ohio program (N H D -O H ) on students academic achievement and intellectual growth by using as key indicators: A) Highstakes tests, i.e., Ohio Graduation Tests (O G T) in Social Studies, Reading and Writing; B) Appreciation o f history, and C) Development of critical thinking. Using a quasi-experimental study design, twenty-four tenthgraders participating in NHD -O H and twenty-four matched controls were used in the analysis to reduce the impact of selection bias. As a result o f this study it was found that students participating in NHD-OH received statistically significant higher scores on the Social Studies OGT, Reading and Writing; and furthermore, demonstrated a deeper appreciation o f history and a better ability to explore and integrate new perspectives.

The importance o f approved educational programs, which result in a positive impact on students’ performance, is a widely accepted concept. Still, several questions remain to be addressed: How effective are existing programs? How do we evaluate the impact? National History Day (NHD) is a yearlong co-curricular educational enrichment program devoted to improving the teaching and the learning of history in the schools. Students, in grades 6-12, select a topic related to the annual theme and create a paper, or exhibit, or documentary, or performance or web site. At the end o f the program, participants can compete with their projects at regional, state, and national levels. Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 79-96. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-8

79

80

G IU S E P P E M O N A C O E T A L .

The history o f NHD is clearly marked by success; the program started in Cleveland in 1974, grew statewide throughout the 1970s as both regional and state events developed, and moved to a nationwide event in 1980. In 2008 over 500,000 students participated nationwide, with about 8,000 students participating in Ohio. The Ohio Historical Society (OHS) organizes and coordinates the NHD in Ohio (NHD-OH) through the Outreach Projects Department, which is responsible for conducting teacher workshops and related activities. OHS has two staff members dedicated to NHD-OH. The staff works with regional partners to run eleven local contests around the state. The staff also provides access to print and electronic resources in addition to providing face to face training with teachers and students around the state. Training emphasizes using research to create historical arguments or conclusions. Students and teachers learn to read primary sources and analyze the information to form conclusions about historical significance. The program is used as a major outreach initiative o f the Ohio Historical Society, fulfilling its mission to: connect Ohio's people to its past in order to understand the present and create a better future. Through the process o f creating History Day projects, students are actively engaged in the process o f recording, creating, and analyzing history. Despite its long history, few research studies have been implemented to date to evaluate the impact o f NHD on students and teachers, or the program itself. These studies, summarized in a recent publication by Mark Robinson from the NHD office, showed empirical evidence o f a short-term positive effect o f participation in the program, and were weakened by the lack of controls.1 As stated by the national NHD office, the program aims to:2 • Motivate students through the excitement o f competition and through recognition for their work. • Provide students with the opportunity to work with and analyze historical documents and other primary source material. • Provide a framework for hands-on, student-centered learning that guides classroom teaching as well as continuous professional development. • Develop critical thinking and problem-solving skills that will help students manage and use information effectively now and in the future. • Develop student research and reading skills and to refine student presentation skills in writing, visual projects and performances.

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

81

• Encourage students to move beyond the classroom and into the community to investigate history. • Provide teachers with innovative teaching tools. • Assist teachers and schools in meeting educational standards by encouraging student participation in portfolio-building and outcomebased learning activities. • Encourage the study o f history by guiding students to express themselves creatively through presentations o f historical topics in various formats. Even though NHD was predicated on the intention to induce rich, wellstructured knowledge and to generate long-term effectiveness on the participants, the achievement o f these goals has not yet been analyzed. Furthermore, up to now, very few studies evaluated the impact o f museum-organized educational programs on high-stakes tests, appreciation and critical thinking.3 Aim o f this pilot study was to develop and validate an evaluation model to assess the impact o f participation in NHD-OH on students' academic achievements and intellectual growth by using as key indicators: A) Highstakes tests, i.e., Ohio Graduation Tests in social studies, reading and writing; B) Appreciation o f history, and C) Development of critical thinking. Once developed and validated, this proposed evaluation model could be applied to different situations; for instance, it could be applied to a larger scale sample, different objectives o f the same education program, or different educational programs.

RESULTS Impact of the participation in NHD-OH on the Ohio Graduation Test (OGT) Twenty-four tenth grade students participating in NHD-OH and twentyfour matched controls were used in the analysis to reduce the impact o f selection bias. Student's t test (reported) and Wilcoxon Sign rank test (not reported here) were used for the statistical analysis to compare the differences related to the OGT in social studies, reading and writing. As a result of matching, the previous score in reading (Ohio Achievement Test (OAT) Reading) showed no difference in the two groups and its result is presented in the following Table for reference. Also, the proportion o f gender, ethnicity and social service are almost identical in both groups. The balance o f the

82

G IU S E P P E M O N A C O E T A L .

Table 1. Comparison of the Results of High-Stakes Tests Between Students Participating in NHD-OH and Controls. Mean

N

Std. Deviation

415.50

24

14.75

Student’s t test

(P) 1 2 3 4

Controls OAT Reading NHD-OH OAT Reading

419.87

24

11.98

Controls OGT Writing

419.66

24

22.87

NHD-OH OGT Writing

441.08

24

10.09

Controls OGT Reading

416.25

24

13.75

NHD-OH OGT Reading

427.04

24

14.54

Controls OGT Social Studies

400.33

24

25.34

NHD-OH OGT Social Studies

427.62

24

27.08

0.139 0.000

0.018 0.002

OAT= Ohio Achievement Test taken in Eighth Grade OGT: Ohio Graduation Test

baseline covariates implies that before NHD-OH, the students are similar and their performance difference after NHD-OH would be attributed to the program. As shown in Table 1, there is a significant improvement of 21 points in Writing, 11 points in Reading and 27 points in Social studies, which should be associated with NHD-OH.

Impact of the Participation in NHD-OH on Appreciation of History. A questionnaire was distributed to the students in order to evaluate appreciation o f history both in the classroom and everyday life. The results showed that different themes emerged from the responses o f the NHD-OH participants versus the controls, and that common themes had different significance. Both NHD-OH and controls viewed history mainly as educational and “the past.” On the other hand, controls also demonstrated negative perceptions, absent in the NHD-OH participants. Results were summarized in Table 2. Very interesting was the appearance of the theme “Uneasiness” among the controls (quotes were reported in italic): • To me history doesn't mean much. All it does is start controversy. People now days get mad at each other because ofwhat our ancestors did. • History is thepast and I try not to think about the past.

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

83

Table 2. Themes Emerging from the Open-Ended Question, “What does history mean to you?” Theme

N H D -O H participants

Controls

Education

Prominent

Prominent

The past

Important

Important

I don’t care. I take because I have to.

Absent

Present

Not that much/Nothing

Absent

Important

Uneasiness

Absent

Present

Table 3. Themes Emerging from the Open-Ended Question, “ How do you feel about learning history?” Theme

N H D -O H Participants

Controls

It’s important/I like it

Prominent

Important

Helps me understand

Important

Absent

Easy I feel “half and half.” It’s OK

Present

Absent

Important

Prominent

It’s boring but necessary

Absent

Present

Negative feelings (I don’t like/hate it/waste of time)

Absent

Important

The same difference emerged from the responses to the question: “How do you feel about learning history?” While “It's important/I like it” was the prominent theme among the NHD-OH participants, controls felt “Half and half” Furthermore, controls expressed negative feelings, absent in the NHD-OH participants. Results were summarized in Table 3. The questionnaire explored how history and learning history was used in the classroom setting. A first closed-ended question allowed the application o f a statistical analysis (chi-square test), which demonstrated that NHD-OH participants enjoyed working on history projects and used them more frequently than the controls. Results were summarized in Table 4. The following open-ended question confirmed the results: while controls mostly did not complement other subjects/classes with history and historical research, a full integration or complementing a specific discipline were the prominent themes among NHD-OH participants. Results were summarized in Table 5.

84

G IU S E P P E M O N A C O E T A L .

Table 4. Analysis of the Responses to the Closed-Ended Question: “How often did the following happen while you learning history in this class? How often did the following happen while you learning history in this class? (Please, check O N E response for each item) Never

Always Sometimes Often (once Rarely (a few times) (every other or twice a (almost every week) week) day)

I enjoyed working on history projects.

CO 29% NHD 0%

CO 25% NHD 29%

CO 42% NHD 21%

CO 0% CO 4% NHD 33% NHD 17%#

I made connections between history and other subjects that I learned.

CO 21% NHD 8%

CO 42% NHD 4%.

CO 29% NHD 33%

CO 4% CO 0% NHD 37% NHD 12%#

I discussed my ideas about the subject with the teacher or other students.

CO 37% NHD 0%

CO 17% NHD 4%

CO 25% NHD 8%

CO 12% CO 4% NHD 58% NHD 29%#

# : Chi-square test: p =0.001. The number of observations was reported in parenthesis. (Controls: N= 24; NHD: N= 24)

Table 5. Analysis of the Responses to the Open-Ended Question: “ How do you use history and/or historical research to complement other subject/ classes in your academic life?” N H D -O H participants

Controls

Full integration Integration within life

Prominent Present

Present Absent

One discipline (arts, English, science, music)

Theme

Prominent

Present

Don’t

Present

Prominent

When discussing about something, it came about

Present

Present

The results showed that students participating in NHD-OH used history differently in their academic life compared to the controls. In particular, the former better recognized how history could be integrated into other disciplines, especially art and English. This could be considered an example of the spill over effect o f history on other disciplines. Quotes from NHD-OH participants were reported in italic: • It gives me many ideas when trying tofigure something out in my life, like in school, I think at thepast> and how school used to be long ago. The differentforms ofsegregation> the laws, and how they have changed.

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

85

Table 6. Chi-Square Analysis of the Frequency of the Responses to the Closed-Ended Question: “Do you talk or discuss about history with family and/or friends?” YES

NO

Controls

43.4%

56.5%

NHD

83.3%

16.6%

Chi-square: p=0.004

Table 7. Analysis of the Answers to the Open-Ended Question: “ If YES, can you, please. Give an example of what you talked about?” Theme

N H D -O H participants

Controls

Prominent

Prominent

Specific persons/ events

Important

Prominent

Learning together

Prominent

Present

Family connection

Present

Absent

Affecting the present/ now and then

• Because I play theflute} I mil sometimes look up the time period one of my songs was composed and I will look up any major events during that time period and I will try to find out if those events affect the way the song is composed, (key >tone tempo). Eventually, the investigators moved to assess appreciation of history on the students' daily lives. The results showed that NOH-OH participants talked more frequently about history with family and friends than the controls. Furthermore, for the formers, talking about history was a moment for sharing and connecting. Results were summarized in Tables 6 and 7. In describing how history and historical research was affecting the everyday life, the results showed a difference between the NHD-OH participants and the controls. In particular, especially the formers saw history as a means to connect the past to the present, while the prominent theme among the controls was “Not sure/Nothing.” Results were summarized in Table 8. Quotes from NHD-OH participants were reported in italic. • We talked about slavery, and different ways it has affected people. This was when dad read in a “Jet” magazine that Maryland apologizedfor their... in slavery, and has made a statue oftwo people hugging

86

G IU S E P P E M O N A C O E T A L .

Table 8. Analysis of the Answers to the Open-Ended Question: "Please, describe how what you have learned about history/history research is affecting your everyday life.” N H D -O H participants

Controls

Why things are occurring/ connect (also personal connection)

Prominent

Important

It makes me think/learn

Important

Important

Present

Prominent

Theme

Not sure/Nothing *

Table 9. Analysis of the Responses to the Closed-Ended Question: "In the past 12 months, how often have you... ? In the past 12 months, how often have you? (Check O N E response for each item) Never

Rarely Sometimes Often (once Always (a few times) (every other or twice a (almost every week) week) day)

Visited History Museums

CO 50% NHD 37%

CO 42% NHD 54%

CO 8% NHD 4%

CO 0% NHD 0%

CO 0% NHD 4%

Visited Historical Sites

CO 50% NHD 33%

CO 42% NHD 50%

CO 8% NHD 12%

CO 0% NHD 0%

CO 0% NHD 4%

Seen a history-related movie

CO 8% NHD 0%

CO 25% NHD 25%

CO 37% NHD 46%

CO 17% NHD 25%

CO 0% NHD 4%

Checked out history books from the local library

CO 46% NHD 24%

CO 37% NHD 50%

CO 8% NHD 17%

CO 4% NHD 4%

CO 0% NHD 4%

Checked out history books from your school

CO 67% NHD 62%

CO 25% NHD 25%

CO 4% NHD 12%

CO 0% NHD 0%

CO 0% NHD 0%

Read books related to history (e.g., fiction, novels)

CO 37% NHD 4%

CO 33% NHD 22%

CO 25% NHD 42%

CO 4% NHD 21%

CO 0% NHD 4%*

Searched the Internet for history-related topics

CO 42% NHD 0%

CO 17% NHD 25%

CO 21% NHD 37%

CO 12% NHD 21%

CO 8% NHD 12%

Watched History Channel

CO 29% NHD 17%

CO 42% NHD 21%

CO 12% NHD 12%

CO 4% CO 8% NHD 23% NHD 25% *

Chi-square test: *= p which is called the.. .family tree. I really learnedfrom it and understood how I am near today. Very interestingly, the results showed that NHD-OH participants more frequently than the controls read history related books and watched “History

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

87

Table 10. Evaluation on Critical Thinking Assessed Through Three Focus Groups, Two Engaging NHD-OH Participants and One Control. Participation Observation in N H D -O H

Defining the Problem

Exploring: Variety of Sources

Exploring: Integrating Exploring: New Students’ Differentiation Evaluation of Perspectives Between Primary and Credibility Secondary Sources

NHD-OH 1

Medium

Medium

Medium

Medium/ High

NHD-OH 2

Medium

Low

Medium

Medium/ High

Medium/ High

Medium

CONTROLS

Low/ Medium

Low

Low

Low

Medium (progression to high)

Low/ Medium

Medium/ Medium (progression some waves of progression to high) to high

Channel” (Chi-square analysis, Table 9). No difference between the two groups was observed in regard o f visiting history museums or sites.

Impact of the Participation in NHD-OH on Development of Critical Thinking Development o f critical thinking was assessed through focus-group-discussions. After showing a photograph o f child labor in the 1920's, the discussion investigated ability to observe, define the problem, explore, and integrate new perspectives. The “exploring steps” assessed use o f variety of sources, students' evaluation o f credibility o f historical sources, and differentiation between primary and secondary sources. No real difference was observed between students participating in NHD-OH and controls in terms o f acuity o f observations, or definition of the problem. Instead, the formers showed a deeper ability to explore, both in terms o f using more sources and o f evaluating their credibility. Interestingly, no difference between the two groups was observed in the understanding o f the difference between primary and secondary sources. These results can be explained by the fact that students in Ohio are exposed from the third grade to the difference between primary and secondary sources. In contrast, a difference emerged in the ability to integrate new perspectives. While both o f the groups connected child labor from the image to current issues, especially through personal connection, particularly students

88

G IU S E P P E M O N A C O E T A L .

participating in NHD-OH based those connections in factual information. Results were summarized in Table 10.

DISCUSSION Our first aim was to establish an evaluation plan to assess the impact o f an education program —NHD-OH —on students' performances. The second goal was to structure a plan flexible enough to allow specifically tailored adjustments in order to be adopted by other institutions. This paper presented the plan and the initial pilot study necessary to test that the plan and the procedures were sound and that there was a reliable sample size. The results validated the evaluation plan and instruments designed to assess the impact of NHD-OH, in fact we were able to detect difference between students participating in NHD-OH and controls. Students participating in NHD-OH showed higher OGT scores in social studies, reading and writing than the matching controls. They also demonstrated deeper appreciation o f history in the classroom and everyday lives, and a better ability to explore and integrate new perspectives. This study presented a model for a successful partnership between the Ohio Historical Society and the participating school districts. The importance of a partnership between museums and schools has long been emphasized and stressed; Beverly Sheppard dedicated a book to the subject in 1993, and one of the most recent International Committee of Museum—Committee of Education and Cultural Action was entirely dedicated to the issue.4In our study the partnership was fundamental to establish, implement and assess the evaluation plan. It allowed build bridges between the partners and structure and implement future steps. Both school districts coordinators and social studies teachers contributed to design the evaluation plan and instruments, combining perspectives from non-formal (i.e., OHS) and formal learning settings (i.e., participating school districts). It has been widely recognized that evaluation is an essential component of a successful partnership between museums and schools; this study shows how an evaluation plan can be the platform for the partnership. The study relied on a quasi-experimental approach, with “naturally occurring treatment groups," since the enrollment in NHD-OH preceded and was independent from the evaluation study. A well-designed quasi-experiment is able to reduce substantially the potential influence o f selection bias on participation in the intervention or comparison groups. We aimed at achieving equivalence among the NHD-OH participants and controls

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

89

through propensity score matching. In our study, among initially enrolled one hundred and nine total students, forty-eight were used the analysis, the others being excluded because o f poor covariate balance. An appropriate matching between “treated” and controls is necessary to ensure the validity of the causal arguments when assessing impact o f educational programs; the use o f propensity score match is not common in schools and museums settings, our study demonstrated the applicability and the usefulness o f this statistical approach. However, cautions need to be taken regarding the interpretation o f our findings. First, since we used a small subgroup o f the study population in the analysis, any generalization to a bigger population with different characteristics demands careful examination. Moreover, the validity o f propensity score matching is based on no unobserved confounder assumption; sensitivity analysis may need to be carried out to evaluate the robustness o f the results when hidden bias is plausible.5 Standardized testing drives public education today. Often, state and federal funding hinges on the level of success districts achieve on such tests. We propose, instead, not to limit the evaluation of a program to the impact on high-stakes tests, but always to embrace a more holistic approach. To induce appreciation of history and of the learning process was one of primary goals for the NHD program as well as to encourage intellectual growth. In particular, the NHD program aims at stimulating critical-thinking and problem-solving skills. There is also a nationwide emphasis placed on the importance of critical thinking, and the teaching of such skills and psychologists and educators from kindergarten to higher education have been promoting the acquisition of critical thinking. While previous studies demonstrated the positive impact of NHD on increasing knowledge, results focusing on appreciation of history and development of critical thinking are still few and conflicting. To assess a learning experience requires the choice o f a methodology of study.6 Usually, two principal practices are described: quantitative and qualitative. Previously seen as in contraposition, more recent studies demonstrated how their combination offers the richest way o f capturing learning.7 They, in fact, complement each other by helping to clarify and elaborate the results, and by adding strength to the conclusion.8 The mixed-method approach designed for this study proved to be able to identify differences between students participating in NHD-OH and controls; while a pure quantitative approach was used to detect differences in high-stakes tests, a combination o f the two practices was used to study appreciation and a pure qualitative approach to examine development o f critical thinking.9

90

G IU S E P P E M O N A C O E T A L .

Students participating in NHD-OH obtained statistically significant higher OGT in social studies, reading and writing than the matched controls. The observation was particularly interesting; it demonstrated how a yearlong project, which fostered and required application of historical research skills, improved students' achievements more broadly. Social studies are not emphasized in current education trends; if our results will be confirmed by a larger scale study, they will prove the importance o f the discipline in supporting other critical skills. Furthermore, students participating in NHD-OH and controls associated history and learning history with different themes. The formers demonstrated a more positive attitude toward history in the classroom and in everyday lives. They used history to complement other disciplines, talked more frequently about it and in different terms compared to the controls. Among the latter we also found negative perceptions. Additionally, the results showed, both through the questionnaires and the focus groups, that the participation in NHD-OH was associated with a more frequent use and better ways to deal with sources o f information. When assessing development of critical thinking the overall impression was that the impact o f the participation in the program was on the exploration steps, such as use o f variety of sources and evaluation of credibility. Interesting to note, no difference was detected between students participating in the program and controls concerning the use and the understanding o f primary versus secondary sources. In conclusion, our results demonstrated the validity of the evaluation plan, which could encourage other institutions to adopt it, to tailor to their specific needs, and to apply to a larger scale sample. In the future, different goals o f the program, different participants (e.g., teachers) could be targeted, and/or different evaluation tools be adopted while respecting the proposed steps.

METHODS Our goal was to develop and validate an evaluation model to assess the impact o f participation in NHD-OH. The success o f the study relied on a partnership between the Ohio Historical Society and two participating school districts. A coordinator and the social studies teachers from each school district participated in all phases o f the study, from discussing the evaluation plan to designing and administering the evaluation tools, while independent investigators analyzed the results. The investigators developed

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

91

Table 11. Model for the Evaluation Plan: Six-Step Plan. 1 Link the goals, the objectives, and the specific aims of the evaluation project to the goals, the objectives and the specific aims of the program 2 Enroll evaluation participants and introduce the proper controls 3 Use a mixed method evaluation approach and employ proper statistical and qualitative analyses to support the arguments 4 Evaluate the same objective through different key indicators 5 Use the same evaluation tool to analyze different objectives 6 Address the same objective by using the same approach through time

a six-step plan: 1) Link the goals and objectives o f the evaluation plan to the goals and objectives o f the program to be evaluated; 2) Use a mixed method evaluation approach by combining quantitative and qualitative methodologies in order to achieve a comprehensive and coherent assessment system;103) Enroll evaluation participants and introduce the proper controls to ensure the specificity o f the observed impact; 4) Address the same objective (e.g., evaluate intellectual growth) by using different key indicators (e.g., academic achievements and development o f critical thinking); 5) Use the same evaluation tool to analyze different objectives (e.g., focus groups could be used to analyze appreciation and critical thinking); 6) Address the same objective by using the same evaluation approach through time.11 The plan, summarized in Table 11, was characterized by a high degree of flexibility and versatility to be applied by other institutions and customized according to specific needs (e.g., different goals, objectives and different target audiences of the NHD program, or o f other programs, could be evaluated by applying the same steps). As predicated by NHD, our project aimed at evaluating the impact of participation in the program on academic achievements and intellectual growth by using as key indicators: A) High-stakes tests, i.e., Ohio Graduation Tests (OGT) in social studies, reading and writing; B) Appreciation o f history, and C) Development o f critical thinking. A pilot study, necessary to evaluate that the procedures were sound and that there was a valid and reliable sample size, was initially structured to implement the first five steps o f the plan. The study relied on a quasi-experimental approach, with “naturally occurring treatment groups/' since the enrollment in NHD-OH preceded and was independent from the evaluation study. One hundred and nine tenth-graders from Ohio urban public schools (Columbus and Dayton school districts) were initially enrolled after

92

G IU S E P P E M O N A C O E T A L .

complying with the school district review board protocols. Fifty-seven students participated in NHD-OH —Columbus and Dayton school districts—and fifty-two did not (controls; Dayton Public Schools). By using a propensity score matching approach, participants and controls were matched based on: gender, ethnicity, need for social service benefits and their previous score in reading on the eighth grade —Ohio Achievement Test (OAT; previous scores in social studies and writing were not available). In all the cases, the social studies teachers had more than five-year experience in teaching.12 To assess the impact o f participation in NHD-OH on the OGT, the scores in social studies, reading and writing were collected, descriptive statistics (means and standard deviations) calculated and various tests (Student's t test and Wilcoxon Sign rank test) used for the statistical analysis. Introduced in 2005, and partially pre-tested since 2002, the OGT consists of 5 components, mathematics, reading, writing, social studies, and science, and is administered to students in 10th grade. The new OGT is a key part of Ohio's education reform to establish an aligned system o f standards, assessments (tests) and accountability for Ohio schools. The Ohio General Assembly established the testing requirements in 2001 based on recommendations by the Governor's Commission for Student Success, and sophomores from March 2005 (graduating class o f 2007) are required to take the test. The test, because o f its structure, is very suitable to assess the direct impact of the participation in NHD-OH on social studies as well as its spillover effect on reading and writing. Appreciation o f history was evaluated through a questionnaire with open- and closed-ended questions, which explored feelings and attitude toward history and learning history both in the classroom and during everyday life, as predicated by NHD. After coding the students' responses to the open-ended questions, themes were reported as “prominent” or “important” or “present” or “absent” (“prominent” = most significant/frequent; “important” = significant/less frequent than “prominent”; “present” = only mentioned; “absent” = not mentioned). The Chi-Square test was used to calculate the statistical significance of the difference in responses to the closedended questions between participants and controls. Development o f critical thinking was evaluated during focus group discussions: students were stimulated to discuss after prompted with a photograph that showed child labor in 1920. Garrison (1991) proposed that critical thinking consists o f the five stages o f identifying the problem, defining the problem, exploring ways to deal with it, applying one o f the

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

93

Figure 1. Rubric To Evaluate Critical Thinking

Garrison proposed five stages o f critical thinking: • Identifying a problem • Defining the problem • Exploring ways to deal with it • Applying one o f the strategies to the problem • Integrating new perspectives • For each of the steps, we evaluate the baseline and the progression during the discussion along the below-reported scale (from low to high or vice-versa). Steps o f Critical Thinking

Scale Observation

Exploring

Defining the problem

Variety of Sources

Low

Recognize details of the picture (i.e., boys, no shoes)

Only one Recognize the events without factual information (making up) More than one

Medium Defining the scene (i.e., child labor)

Recognize the events with factual information

High

Elaborate on the Combination context of the and relation events among more than two sources.

Extrapolate and put in historical context (i.e., depression era)

Students’ evaluation of credibility

Differentiation between primary and secondary sources

“Sounds right” Little or no “A lot of hits” understanding

My Professor told me because...

Understanding of the differences

Reasons behind the selection process

Using both and explain why and how

Reasons behind the selection process Using both and explain why and how Historical connection/ taking advantage o f learning from the past

strategies to the problem, and integrating the new perspective.13 The investigators adapted the above-mentioned steps to the characteristics o f the NHD-OH program and the Ohio standards for tenth graders (Figure 1). In particular, the Ohio Academic Content Standards for Social Studies Skills and Methods addresses these skills in Benchmark A for grades nine and ten and states: Evaluate the reliability and credibility of sources. The grade level

94

G IU S E P P E M O N A C O E T A L .

indicator for thinking and organizing states: Determine the credibility of sources by considering the following: a) the qualifications o f the writer; b) agreement with other credible sources; c) recognition o f stereotypes; d) accuracy and consistency of sources; e) the circumstances in which the author prepared the source. There is a nationwide emphasis placed on the importance o f critical thinking and the teaching o f such skills.14Very recently, several authors from the museum field discussed the importance o f fostering critical thinking in museums, and showed examples of successful programs that stimulated critical thinking.15 So far, a few studies have demonstrated a relationship between critical thinking scores, education courses grades, and students' grade-point averages.16 After establishing a rubric, two independent investigators coded the focus groups transcripts and negotiated the results. The overall inter-rater reliability for this study was 78%.

Acknowledgments The Authors would like to express the deepest gratitude to Amber Hufford, State Coordinator for the Ohio Social Study Resource Center, and Johanna Jones, Managing Director, Randi Korn & Associate, for their comments and critical review o f the paper.

Notes 1. Mark Robinson, “Evaluating National History Day,” Retrieved from the Internet on 08/07/2005. 2. National History Day web site http://www.nationalhistoryday.org/Objectives.htm (Retrieved from the Internet on 11/19/2007). 3. Randi Korn & Associates, Inc., Solomon R. Guggenheim Museum, Teaching Literacy Through Art, Final Report, Synthesis of2004-05 and 2005-06 Studies. Distribution Copy. http://www. informalscience.org/download/case_studies/report_221.PDF. By using different methodological approaches in a case study format, Marilyn Page in 1993 and Damien Ihrig in 2000, demonstrated that participation in NHD provided a total educational, cognitive, affective, and skill-related experience, and the kinds of training and education that employers are looking for, respectively. Furthermore, in 2002, Veronica Boix showed, through in depth interviews, the positive impact of NHD on students on the development of critical thinking. NHD was also incorporated among the programs of the Youth History Initiative (YHI) in Minneapolis. The evaluation project for the YHI is based on a model of pre- and post-activity questionnaires, which mostly assessed the short-term impact of participation in NHD. With this approach, the authors demonstrated higher student academic achievements for the enrolled students, with a significant increase in the use of external resources.

S T U D E N T S ’ P E R F O R M A N C E S / P I L O T E V A L U A T IO N P R O J E C T

95

4. Beverly Sheppard, ed., Building Museum & School Partnerships. (Washington, DC: American Association of Museums, 1993). ICOM/CECA Annual International Conference 2005, Partnering in Museum Education—Enhancing the Adventure: Proceedings Banska, Marian Ciz and Ivan Lukac, eds., (2006). 5. Rajeev H. Dehejia and Sadek Wahba, “Propensity Score-Matching Methods For Nonexperimental Causal Studies,” The Review ofEconomics and Statistics 84, no 1 (2002): 151-161; Paul Rosenbaum, ed., Observational Studies, 2nd edition. (New York: Springer, 2002). 6. In this context, “methodology” was defined as the set or system of methods, principles, and rules underlying the conduct of the inquiry procedure. 7. Keith F. Punch, ed., Introduction to Social Research: Quantitative and Qualitative Approaches, (London: Sage Publications, 1999); John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, (Thousand Oaks, CA; Sage Publications, 2003). 8. G. Monaco, “Analysis of the Learning Experience of Adult Visitors in the Hall of Paleontology at the HMNS,” Visitor Studies Association, Annual Conference; Philadelphia, PA, August 2-6, 2005. Abstracts; p. 5. 9. Sheppard, Building Museum & School Partnerships; Giuseppe Monaco, “National History Day in Ohio. Evaluation Project,” in Thinking, Evaluating, Re-thinking, ed. Emma Nardi (Milano, Italy; 2007), 325-29. 10. National Research Council, Classroom Assessment and the National Science Education Standards: A Guidefor Teachingand Learning,]. Myron Atkin, Paul Black, and Janet Coffey, eds. (National Academies Press: 2001). 11. Giuseppe Monaco, “National History Day in Ohio”. 12. Since this evaluation was an observational study, the students participating in the NHD-OH could be quite different from those who did not in terms of their background. Therefore, carefully considered statistical methods must be used to control the potential selection bias. Propensity score matching is a popular and effective approach to creating comparable treated and control groups in non-randomized studies. Propensity score is the predicted probability of participating in NHD-OH based on the students’ baseline covariates. It is usually estimated by a logistic regression model. With the assumption of no unobserved confounder, a well-balanced matched set will be used in the analysis to yield an unbiased estimate of the program effect. Based on the closeness of the propensity score, students in both groups were matched together to form a pair using an optimal matching algorithm. As a result, the matching produced 24 participant-control pairs, in which the observed baseline covariates were well balanced in the two groups. 13. D. Randy Garrison, “Critical Thinking and Adult Education: A Conceptual Model for developing Critical Thinking in Adult Learners,” InternationalJournal ofLifelong Learning 10, no. 4 (1991): 287-303. 14. Robert H. Ennis, “A Taxonomy of Critical Thinking Dispositions and Abilities,” Teaching Thinking Skills: Theory and Practice, ed. JB Baron and RJ Sternberg, (New York: 1987): 9-26. 15. Mark K. Felton and Deanna Kuhn. “cHow Do I Know?’ The Epistemological Roots of Critical Thinking,”Journal ofMuseum Education 32, no. 2 (2007): 101-110; Margaret Burchenal and Michelle Grohe, “Thinking Through Art: Transforming Museum Curriculum,” Journal ofMuseum Education 32, no. 2 (2007): 111-122; Jessica J. Luke, Jill Stein, Susan Foutz, and Marianna Adams, “Research to Practice: Testing a Tool for Assessing Critical Thinking in Art Museum Programs,” Journal of'Museum Education 32, no. 2 (2007): 123-135. 16. Kingsley C. Ejiogu, Zhiming Young, John Trent, and Mark Rose, “Understanding the Relationship Between Critical Thinking and Job Performance,” http://harcourtassessment. com/NR/rdonlyres/6E133ED3-5516-463A-B880-D06C235A06E0/0/CritcalThinking.pdf. Retrieved from the Internet on 08/24/2005.

96

G IU S E P P E M O N A C O E T A L .

Giuseppe Monaco (Pino) is the Education Outcomes Manager at the Smithsonian Center for Education and Museum Studies. A t the time o f the study he was the Evaluation Analyst at the Ohio Historical Society. H is research and publications focus on informal learning and museums. Bo Lu is an Assistant Professor at the Division o f Biostatistics at the Ohio State University in Columbus, Ohio} where he teaches Design and Analysis o fS tudies in the Health Sciences. Megan Wood is the State Coordinator for the Nation Histoty Day program in Ohio. She is based at the Ohio Historical Society in Columbus.

Evaluation of Educational Material Designed to be Used by High School Art Teachers The Use o f Focus Groups and Questionnaires

Adriana Mortara Almeida and Maria Helena Pires Martins

Abstract

The article describes and discusses the use o f focus groups

and questionnaires to evaluate educational printed material for high school level art teachers prior to publication. The material consisted of “The Notebook o f the Investigative Teacher,” created by the Instituto Itati Cultural to help teachers develop critical skills in discussing contemporary artists with high school students. Based on a partnership between two Brazilian Institutions, our study demonstrates how the evaluation process should be a crucial component of both material development and revision.

THE CONTEXT The Itau Cultural Institute was created in 1987 as a privately owned institute devoted to the research of Brazilian culture, the mapping and study o f existing cultural practices throughout the country, and the dissemination of artistic practice and ideas. Although it is based in Sao Paulo, it has many branches and partners in other towns where it promotes events such as art exhibitions, music concerts, theater, movies and dance presentations, literature readings, seminars, panels, and short term courses. The Institute aims at the democratization and promotion o f social participation in the cultural sphere. Over the years, it has become an important center for the artistic and intellectual Brazilian production in the country and abroad.1The education department o f the Itau Cultural Institute promotes activities for various audiences, including teachers and students. It also has a partnership Journal ofMuseum Education, Volume 34, Number 4, Spring 2009, pp. 97-109. ©2009 Museum Education Roundtable. All rights reserved.

DOI: 10.4324/9781315423852-9

97

98

A D R IA N A M O R T A R A A L M E ID A A N D M AR IA H E L E N A PIRES M A R T IN S

with the Board o f Education o f the State o f Sao Paulo to elaborate educational materials and to train public school teachers to use the material with their students. As a result of these actions, it has produced numerous books, CDs, DVDs, videotapes, exhibit catalogues, and “culture boxes” (a variety of materials on a specific theme). Many o f these products are written or organized by art experts and renowned art critics, and part o f the material is produced for teachers and educators. The Visual Arts Encyclopaedia, which can be found at www.itaucultural. org.br; is only one o f a series o f Encyclopedias services (Theater, Brazilian Literature, Arts and Technology) and products offered by the Institute. The basis for the encyclopedias is a database created in 1987, comprising of the research, treatment, and computerization of data on the arts and culture in Brazil. Because it is an electronic data base, it can be constantly updated and provides easy and free access —even in remote places where specialized libraries and bookstores are not available. The Visual Arts Encyclopaedia has more than 3000 entries including biographies, artists' depositions, art images, information about other institutions, and analyses o f events, movements and groups.2 It is the largest online reference database in the country and it is accessible in Portuguese, with parts accessible in Spanish, English, and French. In this paper we describe how the implementation o f an evaluation project improved the educational materials specifically designed for high school teachers, based on the Visual Arts Encyclopaedia. We demonstrate how the evaluation process should be a crucial component o f both the development and revision o f educational materials. The material included in the evaluation was part o f a series of The Notebooks of the Investigative Teacher. The series were designed to encourage teachers to use the Encyclopedia and other references as research sources to prepare classes on contemporary art themes. Many reasons made this material particularly needed in Brazil: 1) the syllabus of most art schools in Brazil does not include contemporary art; 2) only a few cities have contemporary art museums and exhibits (e.g., Sao Paulo, Rio de Janeiro, Brasilia, Porto Alegre), thus they are not accessible to most art teachers, 3) books on contemporary artists are often quite expensive and, as a result, are not found in public libraries. We, as the independent evaluators, were hired to conceive and implement the evaluation o f the first volume o f The Notebooks, named “Things: Contemporary Art”, during the second half o f 2007. Based on the results o f the evaluation, the text, the format, and other elements were modified. The final

T H E USE O F F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

99

View of the first version of The Notebook, with cardboards and DVD. Photo: Adriana Mortara Almeida.

version came out in October, 2008 and is at the disposal o f any teacher who requests it.

THE EVALUATED MATERIAL “Things: Contemporary Art” —first volume o f The Notebook of the Investigative Teacher—presented three blocks o f content. The first block was a text on things, e.g., addressed things either represented or transformed into artwork. The second block, called Work in Focus, consisted o f an analysis o f three works o f art by contemporary Brazilian artists: Blue Phase (Defacements), 1992, byjac Leirner; Untitled, 1997, by Nazareth Pacheco; and Untitled, 2002, by Waltercio Caldas. They included suggestions on how to interpret the work o f art (e.g., I look at the work o f art. What do I see?), biographies of the artists (Artist's Life), possibilities o f investigation and discussion of the focused works with the students (Interpreting Roads), research and practical art activities for the students (With Art in the Classroom), thoughts on education and art (To Think About), and a blank space where the teacher could write his/her own ideas (Thinking to Act). The third block (Guide to Investigate)

100

A D R IA N A M O R T A R A A L M E ID A A N D M A R IA H E L E N A P IR ES M A R T IN S

presented a selected bibliography and other references so that the teacher could widen his/her research. The Notebook of the Investigative Teacher provided teachers with annexed material consisting o f large prints (on cardboard) of each work o f art and o f two still lives mentioned in the volume. It also included a DVD with images of the chosen works o f art. The format o f the material—book and DVD —was chosen by the staff o f the Education Department, based on their experience in dealing with teachers and their perceptions of teachers' difficulties and needs. A museum art educator was hired by Itaû Cultural to write The Notebook.

METHODOLOGY Through initial discussions, the Itau Cultural education staff and the evaluators started to identify strong and weak points o f the Notebook —for instance, the contribution of the artists' biographies to a better comprehension o f the selected art work was questioned. Initially the text presented the biographies in a simplified fashion: a catalogue o f artworks and life events which influenced the artists' production. Also the discussion about the contextualization text that went along with Waltercio Caldas' work was considered too superficial. These and other remarks guided the evaluators in choosing the instruments for the study: questionnaires and focus groups. The first version o f The Notebook of the Investigative Teacher was printed and distributed to two groups of teachers; after reading the material, the first group answered a written questionnaire, and the second group was invited to participate in focus groups. Out o f a total o f 33 teachers, 17 were part o f the focus groups (five o f them were present in two meetings) and 16 answered the questionnaire in writing. As an incentive, all teachers who participated in the evaluation study received a number o f books, booklets, and didactic material published by Itau Cultural.

THE FOCUS GROUPS Focus groups have proven to be an effective methodology for museum frontend and summative evaluations. We chose to use this methodology as it fit our needs both in terms of the tight schedule and its likelihood o f offering useful data. Twenty years ago Rubenstein proclaimed the advantages of focus groups for museum evaluations, a strategy still widely used.3 Focus groups are particularly useful in “identifying and defining problems in

T H E USE O F F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

101

project implementation; pretesting topics or idea, identifying project strengths, weaknesses, and recommendations; assisting with interpretation of quantitative finding; obtaining perceptions of project outcomes and impact; and generating new ideas”.4 In our case, the focus groups were prepared with the goal o f clarifying some points raised during the critical review and the meetings with the Education staff at Itau Cultural. The choice o f the focus groups questions required particular attention. In the end, we opted for broad questions such as: Did you know the artists before?, Did the information about the artist' s life help in the interpretation of her/his art work?, Did you understand what assemblage is?, Do you think that the suggested activities will be interesting to use with your students?. The targeted population for the focus groups was high school level art teachers who work primarily in public schools. Three focus groups with high school art teachers were conducted. The meetings took place in a room at the Itau Cultural Institute building on August 10 and 27, 2007. The teachers were selected randomly from a pool by Itau Cultural because they were currently teaching art at a public high school. All of them reviewed a copy of The Notebook for the Investigative Teacher^ “Things: Contemporary Art” and a related DVD, before the meeting. During the focus groups the mediation was done by Adriana Mortara Almeida and Daniela Azevedo, a member o f the Itau Education staff. All the discussions were audio recorded and video taped, and some photographs were taken throughout the sessions. Each participant had already sent in his or her personal and professional data prior to the day the group met. There were two focus groups on August 10th, the morning group discussed Jac LeirneEs and Waltercio Caldas5works in depth while the afternoon group paid special attention to Nazareth Pacheco5work and discussed at length the “research guide”. To encourage discussion, we prepared a PowerPoint presentation showing the main parts o f The Notebook. Five teachers who participated in the August 27th focus group had already used The Notebook in class.

QUESTIONNAIRES A questionnaire is used when the researcher needs a certain quantity o f data and wants to cover a wide range o f topics to permit the evaluation o f any written material, like The Notebook. In this case, it was used as a complement to the focus groups so that we could verify teachers5comments and concerns. Teachers received the printed version o f The Notebook and the questionnaire.

102

A D R IA N A M O R T A R A A L M E ID A A N D M A R IA H E L E N A P IR ES M A R T IN S

They were asked to answer it and send it back to the Institute within ten days, during the month o f August 2007. The questionnaire included closed and open-ended questions. It comprised four blocks of questions: after personal and professional data—length o f teaching experience, school, etc—teachers were asked about their educational backgrounds and cultural habits, such as going to art museums and shows. Eventually teachers were asked to write the contents and strategies used to teach art and whether they included going to museums and art shows as programmed activities. The fourth block was dedicated to the critical review o f The Notebook: the annexed material (reproductions and DVD), the selected artists, the themes explored, the graphic design, and the relation with the Visual Arts Encyclopedia. In the end, teachers were asked to describe how they would be able to use the contents of The Notebook in their classes and encouraged them to make other comments and suggestions. Out o f 23 teachers to whom the questionnaire was sent, 16 returned it.

RESULTS Many o f the opinions expressed by the teachers during the focus groups were similar to those found in the questionnaires. The focus groups, however, allowed us to have a clearer and deeper understanding o f how teachers choose themes for their classes and what difficulties they face in public schools. Issues such as lack o f adequate space for art classes, lack of equipment to show DVDs etc, and if there is, or is not a predisposition to teach contemporary art themes emerged from the analyses o f the transcripts. During the discussions, some teachers expressed a feeling o f unease when addressing contemporary art in the classroom. They felt unprepared because contemporary art courses were not typically offered at the university. They also stated that it was easier to find books and other printed material about other periods o f Art History to use with their own students. I (...) realize it does not happen only to me but also to colleagues, we have problems to work with contemporary art, or the students do not work with contemporary art, so we (...) work Renaissance (...) which we do have to work, but the contemporary never has a place

These results strongly support the importance o f creating and dis-

T H E USE O F F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

1 03

tributing materials such as The Notebook for the teachers who do not have enough didactic material. (...) The Notebook o f the Investigative Teacher is a tool and a way for us to instigate, for we have those teachers who (...) do not have didactic books (...).6 Teachers praised the content and the form o f The Notebook, the texts and diagramming. However, we realized that they made critical comments or offered suggestions only when prompted with explicit questions about specific parts of The Notebook, such as a specific sentence or image. Even then, the discussion around the content o f the texts languished, probably because of gaps in their education. Most o f the suggestions were about the form o f the material—to make it easier to transport to classes and use it with the students. The suggestions ranged from size to slipcase or binding. The annexed material, according to the teachers, needed to be reproduced in a larger number o f copies so that many students could have the same images at the same time. They also suggested that images o f different works by the same artist could be reproduced on the two sides o f the cardboard. The material should be light and small to be easy to carry, but the cardboards should be big enough to provide a good reproduction o f the works o f art. The DVD was very criticized: As there are few images, and the student will have to go to another room in order to watch the DVD, it is complicated. And, on top of that, the images are not much different in size from those printed on paper.7 It was also criticized because it presented the same static images printed on the cardboards. Teachers suggested the inclusion o f moving images and artist’s description and comments. The Itau Cultural education staff considered the existing resources and overall costs and decided to invest in a larger number o f reproductions and more copies of the same cardboards for the final version of The Notebook. Jac Leirner’s and Nazareth Pacheco’s selected works o f art were very well liked by the teachers who understood their relation with the theme 'things’. However, in one o f the focus groups there was a long discussion on whether it was appropriate to use in the classroom a work o f art (Blue Phase

104

A D R IA N A M O R T A R A A L M E I D A A ND M A R IA H E L E N A P I R E S M A R T I N S

Jac Leirner, Blue Phase (Defacements), 1992. 85 x 85 cm. Banco Itau S.A. Collection, Photo: Joao L Musa/ltau Cultural.

[Defacements] by Jac Leirner) based on manipulation o f paper currency—an act usually considered as a crime in Brazil. It was clear that some teachers had difficulty in dealing with controversial issues with teenagers for fear of Inciting non-social behaviors and that this was one of the reasons why they avoided using contemporary art in their classes. This finding of our study emphasized the Institute's responsibility about the content o f The Notebook; by presenting controversial issues, it could raise questions that teachers could not be prepared or willing to discuss. Nazareth Pacheco's work also caused a great deal o f discussion because it draws attention to cutting and surgical objects and instruments which can be likened to weapons. In Brazil, public school students tend to come from lower middle class and poorer neighborhoods where parents usually do not

T H E USE O F F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

1 05

Nazareth Pacheco, Untitled1997. Dia-meter: 21,5 cm, Banco Itaù S.A. Collection. Photo: Sérgio Guerini

have a formal education. Schools have to deal with a lot o f problems such as vandalism, violence, family abuse, hunger, criminality etc. The selected necklace was considered both delicate and dangerous. Some teachers felt uncomfortable discussing this art work while others found it interesting because “it deals with aggressivity from another point of view\ beauty being the standpoint" .8 One o f the teachers used this same art work in her classes and implemented the activities suggested by The Notebook with very positive results. It worked with Nazareth Pacheco, I thought it was very cool because they [the students] were able to grasp the idea, they did not see it as a weapon at any moment, they understood it as art. They loved the idea o f being a necklace, this fact called a lot of attention and I was not able to discover why.9 The same teacher designed her own supplemental activity; she suggested to her students to create a necklace with objects or part o f objects that they could identify with, and she reported very interesting results. In The Notebook, a discussion about “still life" introduces Waltercio Caldas' work.

106

A D R IA N A M O R T A R A A L M E ID A A N D M AR IA H E L E N A PIR ES M A R T IN S

Two views of Untitled, by Waltercio Caldas, 2002. 40 x 120 x 80 cm. Banco Itaú S.A. Collection. Photo:Jodo L Musa/Itaú Cultural.

T H E USE O F F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

1 07

(...) I showed Waltercio Caldas' work and the students did not see it as still life. They thought it was a tangle o f wires also because it did not have a title which would identify the object. They absolutely could not see it as icstill life.10 Usually experts interpret the intent behind Waltercio Caldas' sculpture as not to create images similar to existing things in the world but to manipulate forms by making references to art history. The teachers believed this interpretation was too abstract. They considered it more appropriate to work with dots and lines when interpreting sculptures. Thus they suggested excluding Caldas' art work.

IMPLEMENTED CHANGES Based on the critical reading, the results obtained in the focus groups and questionnaires, and a long discussion with the Institute's education staff and the writer o f The Notebook, we identified the main problems and suggested possible ways o f addressing them. After a few months, the final version of The Notebook “Things: Contemporary Art” was presented in a new layout and modified contents. The first block, introduction (Getting to know The Notebook ofthe Investigative Teacher) and a text about the theme o f The Notebook (Things o f the World and o f Art), remained. In the second block, in order to address the teachers' lack of confidence, a long text on the historical context of the artists was inserted (Amplified Context). It also included the work of Duchamp and Nelson Leirner, another important Brazilian artist, and discussed their influence on Jac Leirner's and Nazareth Pacheco's art production. Waltercio Caldas's work was excluded as suggested by the teachers. Thus, The Notebook in its final form presented only two artists: Jac Leirner (Defacements) and Nazareth Pacheco (Untitled). The same artworks by Jac Leirner and Nazareth Pacheco were repeatedly addressed in depth in the four parts o f Work in Focus: interpreting the work o f art (Interpreting Road), revised biographies (Artist's Life), suggested activities for the students (With art in the Classroom), and a blank space for the teacher to register his/her ideas (Thinking to Act). In the third block (Guide to Investigate) a brief text on art history was inserted in order to provide more information on artists and movements and suggested new roads for the teacher's investigation. The last part included a bibliography on Jac Leirner, Nazareth Pacheco, Nelson Leirner, Duchamp, and contemporary

108

A D R IA N A M O R T A R A A L M E ID A A N D M AR IA H E L E N A P IR ES M A R T IN S

art in general to help the teachers in their investigation. The DVD was discarded in favor o f a larger number o f reproductions o f different works o f art by the same artists and by other artists cited in The Notebook and o f more copies of the discussed works.

CONSIDERATIONS This experience highlighted the importance o f using multiple strategies and sources o f information to evaluate materials created for teachers by cultural institutions. The critical review by art history and education experts identified what had to be changed and presented suggestions for the texts in terms o f content, selection o f artworks, and activities. The teachers used their every day experience in the classroom to point out problems o f form (like the difficulty o f carrying it to the classroom), operational problems (the use of DVD and annexed material), and content—including the difficulty in working with contemporary art. Up to December 2008,1,050 Notebooks have been distributed to high school teachers in Brazil. In the state o f Sao Paulo, within the context o f the established partnership with the state Education Board, Itau Cultural also implemented a series of digital video conferences to facilitate the teachers' use o f art-related material in the classroom. There is a continuing evaluation effort: questionnaires for teachers will be included in every Notebook and experts will review the next volumes o f The Notebook. The focus groups, although very productive, are costly to organize and they demand adequate infrastructure. Thus, these would be hard to use on a continuous basis. We will have to wait for the teachers to answer the questionnaire included in the first volume o f The Notebook to be able to evaluate the impact o f the material in their work and also to evaluate the efficiency of the partnership.

Notes 1. For more information, assess www.itaucultural.org.br. 2. For more information, assess the Enciclopaedia site www.itaucultural.org.br/enciclopedia 3. Rosalyn Rubenstein, “The Use of Focus Groups in Audience Research/’ Visitor Studies: Theory; Research, and Practice 1, no 1(1989);180-188; Rosalyn Rubenstein, “Focus Groups and Front-end Evaluation.” Visitor Studies: Theory; Research, and Practice 3, no 1(1991); 87-93 ; See also: Getty Center for Education, Insights: Museums, Visitors, Attitudes, Expectations: A Focus Group Experiment, Amy Walsh, ed., (Los Angeles, CA: J Paul Getty Trust, 1991). 4. Joy Frechtling Westat, The 2002 User Friendly Handbookfor Project Evaluation, National Science Foundation Directorate for Education & Human Resources, Division of Research, Evaluation, and Communication (January 2002), 53-4. Available at: http://www.nsf.gov/ pubs/2002/nsf02057/nsf02057.pdf

T H E USE OF F O C U S G R O U P S A N D Q U E S T I O N N A I R E S

1 09

5. As explained by teacher Marcia during the focus group held on the afternoon of August 10th. 6. As explained by teacher Edna during the focus group held on the afternoon of August 10th. 7. Teacher Cecilia’s answer in the questionnaire. 8. Teacher Eliana’s comment on the questionnaire. 9. Teacher Simone’s declaration during the August 27th focus group. 10. Teacher Simone’s declaration during August 27th focus group.

Adriana Mortara Almeida ([email protected]) is PhD in Communication, former museum educator at the Museu de Arqueología e Etnología — University ofSâo Paulo— and consultant in Museum Education and Evaluation in Brazil. Maria Helena Pires Martins is PhD in Arts, former Head of Education of the Museu de Arte Contemporánea, retired professor of the University ofSao Paulo and consultant in Cultural Issues in Brazil.

JO U R N A L Oj

M U SEU M E D U C A T IO N is a publication o f the M useum Education Roundtable, a national nonprofit, volunteer organization dedicated to fostering professionalism and leadership am ong m useum educators, to encouraging research and scholarship, and to providing for the exchange o f ideas through programs and publications like the Journal of Museum Education. Publication o f the Journal is made possible by membership support and by contributions from individuals and institutions. A subscription to the Journal is one benefit o f membership in MER. For further information on the M useum Education Roundtable, its programs and publications, and on membership and donor levels, contact M ER at P.O. Box 15727, Washington, D.C. 20003, phone (202) 547-8378 email [email protected], or on the web at www.mer-online.org. The M useum Education Roundtable gratefully acknowledges the generous support o f the following individuals in 2005-2006.

Benefactor ($500 and above) Christine Castle Sponsor ($200-499) Anonymous Jill Dixon David Ebitz Cynthia Robinson