Exploring the space of SoTL projects

By: Ido Roll (@hummus_monster), Andrea Han, Adriana Briseno-Garzon, and Sarah Perez

SoTL frameworks are like toothbrushes. 

Everyone has one but nobody wants to use someone else’s.

The University of British Columbia (UBC) has a very active SoTL landscape, including a community of practice and an institute. However, as we work to support SoTL initiatives across the university, we identified several challenges:

As administrators of SoTL projects, it is hard for us to understand the big picture of SoTL at UBC. What occupies our faculty members? Where is the centre of activity in terms of SoTL research? It is hard to tell the forest for the trees, and the more trees are planted, the blurrier this forest becomes.

For faculty members who lead SoTL projects, it is hard to identify related efforts. We found that instructors have hard time answering the following questions: What do other instructors do in similar spaces (discipline, course level, classroom size)? How are similar problems being addressed by others? What are relevant methods that can assist me in finding solutions?

Our approach to answering these challenges has three layers: a common vocabulary for describing the working components of SoTL projects, a unified grammar for asking research questions, and an interactive visualization for presenting and investigating the landscape of SoTL projects at UBC.

SoTL Vocabulary

screen-shot-2016-09-12-at-12-00-08-pm

One of the defining characteristics of SoTL work is its diversity. SoTL projects are deeply rooted in their disciplines, reflecting different practices, beliefs, and ways of knowing. This diversity makes it challenging to associate related projects. For example, how many discussions have you had about what flipped classroom actually means?

In order to compare the apples and oranges of SoTL one has to come up with a common vocabulary. To do so, we gathered a group of individuals who manage a large number of SoTL projects across faculties at UBC as part of UBC’s TLEF program. In a bottom-up process we identified common terminology that fits into three buckets, as described below. The process is often called “affinity diagram” and includes two stages: brainstorming using post-it notes, followed by collaborative consolidation (For more information, please contact us). After further consolidation and deliberation we ended up with the following SoTL terminology which streamlines our discussion of SoTL projects:

Practice. What aspects of teaching do instructors evaluate?

  1. Active learning – short activities (one or more single-session activities, e.g., clickers)
  2. Active learning – multi-session activities (e.g., capstone projects)
  3. Content – student generated (e.g., wiki, seminar)
  4. Content – instructor generated (e.g., videos)
  5. Community based (e.g., community service)
  6. Assessment – peer feedback (e.g., PeerWise, Calibrated Peer Review)
  7. Assessment – other (e.g., two-stage exams, diagnostics)
  8. Program structure (e.g., learning outcomes alignment)
  9. Reduced seat time (reducing face-to-face contact hours)
  10. Instructional team (e.g., support, professional development, roles)
  11. Other practice

Area of impact. What are the intended results of the practice?

  1. Actions and behaviours (e.g., time on task, enrolment)
  2. Attitudes and motivation (e.g., personal goals, perceptions about discipline)
  3. Course specific knowledge (e.g., the French revolution, F=ma)
  4. Instructional team practices (e.g., TA use of time)
  5. Lifelong skills (including professional skills, e.g., collaboration, critical/interdisciplinary thinking)
  6. Operations (e.g., finance, reputation)
  7. Other area of impact

Evaluation approach. What methods are used to evaluate the success of the practice to achieve the desired impact?

  1. Attitude surveys (e.g., self-report questionnaires about perceived values, attitudes, and dispositions)
  2. Interviews (e.g., focus groups or individual interviews)
  3. Knowledge tests (e.g., quizzes, diagnostics, or other measures of knowledge)
  4. Observations (e.g., open-ended or structured classroom observations)
  5. Reflective writing (e.g., students or instructors reflect on their experiences and/or log their activities.)
  6. Secondary data (e.g., an existing source of data such as previous grades or attendance)
  7. Other evaluation

These lists are by no means exhaustive. They focus on the main undertakings by faculty members at UBC. For example, they focus on practices that take place in UBC courses, while significant components of SoTL do not directly evaluate instruction. We do not attempt to encompass the entire space of SoTL – such endeavour would be overly ambitious. Instead, we want to focus on actual projects lead by UBC faculty.

SoTL Grammar

screen-shot-2016-09-12-at-12-00-34-pm

The relationship between the dimensions identified above, in conjunction with their context, offers a pseudo-template for research questions in the form of “what is the impact of <practice> on <area of impact> in <context>?” Instructors can use this template, with their specific projects, to formulate a research question. For example, what is the impact of in-class group work (active learning – short activities) on students’ collaboration skills (lifelong skills) in the context of 3rd year Arts courses?

This format clearly does not fit all research questions. It is a simple tool that allows instructors to better describe their SoTL projects and connect with peers using a streamlined vocabulary and grammar.

screen-shot-2016-09-12-at-12-00-41-pm

SoTL Explorer

The last component of this effort was to create SoTL Explorer, an online open-source tool that allows faculty and administrators to explore the breadth of SoTL projects at UBC.  SoTL Explorer allows instructors and administrators to perform two main tasks, directly addressing the two main challenges identified above: to identify trends and patterns in SoTL projects at UBC, and to identify related efforts as a project planning aid. While development is complete, data entry is still in its infancy.

The tool has three main views. The Practice x Impact view shows the frequency with which certain practices are used to target a certain area. The Impact x Evaluation view shows the frequency with which certain goals are being evaluated. For example, the snapshot above shows that surveys are overall popular; course specific knowledge is often evaluated via knowledge tests; etc. The Flow view, shown below, shows the relationship between all three levels.

screen-shot-2016-09-12-at-12-00-52-pm

Users can filter projects based on project elements (practice, impact, or evaluation) or contextual variables (course size, faculty, source of funding). Once a project is chosen, it remains chosen across the different views. Lastly, the Project Details view provides additional information about each project, including a point of contact.

One typical scenario that the tool supports is identifying focus of activity. For example, based on projects presently shown in SoTL Explorer, the most common practices evaluated by UBC instructors are short active learning activities (such as clickers or worksheets) and instructor generated content (such as videos, likely as part of a flipping effort). Exploring the different views reveals the purpose of these practices and how these efforts are being evaluated. Another scenario includes looking for projects based on context and intended impact. For example, the following graph shows what is done to address students’ attitudes in advanced courses. Following each of these projects in the Project Info page provides additional information about this group of projects.

screen-shot-2016-09-12-at-12-00-59-pm

The SoTL Explorer tool can be access here: http://sotl-explorer.sites.olt.ubc.ca/.

Summary

We presented our approach to streamlining the language of SoTL projects, identify main efforts undertaken by UBC instructors, and providing a planning aid and visualization tool. While the SoTL Explorer tool is still in its early stages, we have already seen many advantages for using the vocabulary and grammar when helping faculty design these projects. For example, this allows us to better connect between faculty members with shared interests across disciplines, or to reuse tools such as surveys or interview protocols.

By sharing these tools and approaches we hope to initiate a discussion around these. How can the language be improved, to encompass a broader range of projects across institutions? Would you like to expand the tool to your context? Please, contact us with your reactions and ideas.

ido.roll@ubc.ca

Acknowledgements

We would like to thank Rama Flarsheim, Gillian Gerhard, Jason Myers, Sunah Cho, Judy Chan, and Gulnur Birol for their essential contributions.

This entry was posted in Blog. Bookmark the permalink.

4 Responses to Exploring the space of SoTL projects

  1. Nicola Simmons says:

    Wow – very interesting work! Congratulations on where you’ve taken this – looking forward to seeing more!

  2. Sunita says:

    Nice work Ido! Thanks for sharing!
    S

  3. jmilleryoung says:

    Thanks for sharing this, UBC colleagues! (And ha, I love the toothbrush analogy!) I too, have struggled with how to organize the diversity of projects occurring at my institution in a way that is useful for practitioners. I will definitely have your categories on my mind as I go forward, and let you know if I think of others. I will explore the SoTL Explorer as well. I will be in touch!
    Nicola: these could also be useful categories to include in the literature database: https://researchsotl.wordpress.com/
    ~ Janice

Leave a Reply

Your email address will not be published. Required fields are marked *