{"id":1991,"date":"2021-09-10T09:08:15","date_gmt":"2021-09-10T09:08:15","guid":{"rendered":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/?p=1991"},"modified":"2021-09-10T09:24:46","modified_gmt":"2021-09-10T09:24:46","slug":"formalising%e2%80%afthe-evaluation-of-learning-technologies","status":"publish","type":"post","link":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/posts\/1991","title":{"rendered":"Formalising\u202fthe evaluation of learning technologies"},"content":{"rendered":"\n<p>Yesterday&nbsp;we&nbsp;(Marieke Guy and&nbsp;Tharindu&nbsp;Liyangunawardena)&nbsp;facilitated a session at the&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/altc.alt.ac.uk\/2021\/\" target=\"_blank\">ALT annual conference<\/a>&nbsp;on \u2018Making your mind up:&nbsp;Formalising\u202fthe evaluation of learning technologies\u2019.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">ALT session&nbsp;<\/h2>\n\n\n\n<p>The session&nbsp;began by&nbsp;taking a&nbsp;look&nbsp;at&nbsp;the impact on institutions of the&nbsp;mass&nbsp;pivot&nbsp;from face-to-face teaching&nbsp;to online&nbsp;due to covid.&nbsp;Institutions have seen&nbsp;an&nbsp;increased need for greater investment in the digital campus&nbsp;(as noted by the&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.officeforstudents.org.uk\/publications\/gravity-assist-propelling-higher-education-towards-a-brighter-future\/\" target=\"_blank\">OFS Gravity Assist report)<\/a>&nbsp;which&nbsp;has&nbsp;resulted in a significant&nbsp;amount of&nbsp;technology&nbsp;procurement&nbsp;decision.&nbsp;However&nbsp;anyone who works in the learning technology area knows that&nbsp;technology purchase&nbsp;decisions aren\u2019t straightforward. They are often impacted by pressure from above&nbsp;(e.g.&nbsp;senior leadership, budget,&nbsp;culture etc.),&nbsp;pressure&nbsp;from below&nbsp;(e.g.&nbsp;resource, time, users&nbsp;etc. )&nbsp;and pressure from the side&nbsp;(e.g.&nbsp;security implications, the wider world, accessibility regulations, government regulations,&nbsp;Office&nbsp;For&nbsp;Students expectations&nbsp;etc.).<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"588\" height=\"295\" src=\"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/Picture3.jpg\" alt=\"Making your mind up\" class=\"wp-image-1992\" srcset=\"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/Picture3.jpg 588w, https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/Picture3-300x151.jpg 300w\" sizes=\"(max-width: 588px) 100vw, 588px\" \/><\/figure><\/div>\n\n\n\n<p>During the&nbsp;session&nbsp;we&nbsp;looked at how in&nbsp;order to support decision making it may prove useful to refer to existing frameworks&nbsp;including the following:&nbsp;<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/er.educause.edu\/articles\/2018\/9\/a-rubric-for-evaluating-e-learning-tools-in-higher-education\" target=\"_blank\" rel=\"noreferrer noopener\">Educause &#8211; The Rubric for E-Learning Tool Evaluation<\/a>&nbsp;<\/li><li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Technology_acceptance_model\" target=\"_blank\" rel=\"noreferrer noopener\">The technology acceptance model<\/a>&nbsp;<\/li><li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Jakob_Nielsen_(usability_consultant)\" target=\"_blank\" rel=\"noreferrer noopener\">Jacob Nielson usability testing<\/a>&nbsp;<\/li><li><a href=\"https:\/\/www.tonybates.ca\/2015\/01\/03\/choosing-a-model-for-media-selection\/\" target=\"_blank\" rel=\"noreferrer noopener\">Tony Bates SECTIONS<\/a>&nbsp;<\/li><\/ul>\n\n\n\n<p>Change management approaches&nbsp;can also help us&nbsp;with&nbsp;decision making and&nbsp;implementation,&nbsp;and also with understanding potential reasons for failure:&nbsp;<\/p>\n\n\n\n<ul><li><a href=\"https:\/\/blog.prosci.com\/applying-adkar-to-new-change-management\" target=\"_blank\" rel=\"noreferrer noopener\">ADKAR<\/a>&nbsp;(Awareness, desire, knowledge, ability, reinforcement)&nbsp;<\/li><li><a href=\"https:\/\/www.mckinsey.com\/business-functions\/strategy-and-corporate-finance\/our-insights\/enduring-ideas-the-7-s-framework\" target=\"_blank\" rel=\"noreferrer noopener\">McKinsey 7S Model<\/a>&nbsp;(strategy, structure, systems, shared values, style, staff, skills)&nbsp;<\/li><li>Lewin, Bridges, Kotter etc.&nbsp;<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Making your mind up: Formalising the evaluation of learning technologies\" width=\"758\" height=\"426\" src=\"https:\/\/www.youtube.com\/embed\/dkz_g8kBFd8?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>At UCEM we have&nbsp;recently made the decision to&nbsp;purchase an assessment platform.&nbsp;We have gone through a very thorough&nbsp;decision making&nbsp;approach&nbsp;beginning with&nbsp;investigation of&nbsp;the&nbsp;existing process, writing a functional&nbsp;requirements document,&nbsp;identification of systems to evaluate&nbsp;and&nbsp;writing&nbsp;a systematic testing and evaluation plan (inspired by Educause rubric). We then went through systematic testing on VLE (using testing team and assigned roles),&nbsp;completion&nbsp;of decision matrix, an&nbsp;IT software and cloud solution approval process&nbsp;and finally delivered recommendations&nbsp;to the appropriate people.&nbsp;You can read about&nbsp;our final decision in&nbsp;our&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/blog.ucem.ac.uk\/onlineeducation\/posts\/1971\" target=\"_blank\">recent blog post<\/a>.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-embed-slideshare aligncenter wp-block-embed is-type-rich is-provider-slideshare wp-embed-aspect-1-1 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Making your mind up: Formalising\u00a0the evaluation of learning technologies\u00a0\" src=\"https:\/\/www.slideshare.net\/slideshow\/embed_code\/key\/GN2FxjYhX1ITBj\" width=\"427\" height=\"356\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" style=\"border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;\" allowfullscreen> <\/iframe> <div style=\"margin-bottom:5px\"> <strong> <a href=\"https:\/\/www.slideshare.net\/MariekeGuy\/making-your-mind-up-formalisingthe-evaluation-of-learning-technologies-250141794\" title=\"Making your mind up: Formalising\u00a0the evaluation of learning technologies\u00a0\" target=\"_blank\">Making your mind up: Formalising\u00a0the evaluation of learning technologies\u00a0<\/a> <\/strong> from <strong><a href=\"https:\/\/www.slideshare.net\/MariekeGuy\" target=\"_blank\">Marieke Guy<\/a><\/strong> <\/div>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Crowdsourced ideas&nbsp;<\/h2>\n\n\n\n<p>During the session attendees were asked to share their own ideas. The results can&nbsp;be seen on the&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/bit.ly\/mymu-mural\" target=\"_blank\">mural board<\/a>, please do add your own.&nbsp;&nbsp;<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"967\" height=\"502\" src=\"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/mural.jpg\" alt=\"\" class=\"wp-image-1993\" srcset=\"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/mural.jpg 967w, https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/mural-300x156.jpg 300w, https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-content\/uploads\/2021\/09\/mural-768x399.jpg 768w\" sizes=\"(max-width: 967px) 100vw, 967px\" \/><\/figure><\/div>\n\n\n\n<p>The main suggestions so far&nbsp;(10\/9\/21)&nbsp;are:&nbsp;<\/p>\n\n\n\n<p><strong>Do you use existing frameworks?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>ACTIONS model (2005), that evaluates technology for: Accessibility for learners; Cost structure; Teaching application; Interactivity or ease of use; Organizational impact on the educational institution; Novelty; and Speed to which courses can be developed for the technology.&nbsp;<\/li><li>Our own framework&nbsp;<\/li><li>We used an active learning framework, developed through focus groups with staff and students, to select tools for active learning&nbsp;<\/li><li>Could&nbsp;Jisc&nbsp;help here?&nbsp;<\/li><\/ul>\n\n\n\n<p><strong>What works?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>Getting stakeholders involved from the beginning works&nbsp;<\/li><li>We have recently expanded our project management team.\u202f Prof project managers have been a&nbsp;relevant&nbsp;ion to me. Much more effective that being managed by academics&nbsp;<\/li><li>Open&nbsp;communication&nbsp;with stakeholders\/SMEs&nbsp;<\/li><li>Project teams drawing from all relevant stake holder groups rather that say the IT dept running a project&nbsp;<\/li><li>Clear regular, jargon-free comms. Include all stakeholders and test their use cases&nbsp;<\/li><li>User testing with students&nbsp;<\/li><\/ul>\n\n\n\n<p><strong>What doesn\u2019t?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>Too top down purely driven by analysts who don&#8217;t understand the context&nbsp;<\/li><li>Not involving people&nbsp;<\/li><li>Not telling people what is happening&nbsp;<\/li><li>Decisions based on free packages or easy bolt&nbsp;ons&nbsp;<\/li><li>we struggled with systems that would not allow full testing. They provide a cut down version of the functionality and expect us to make a decision. Would not allow for example to see how it works with LTI or SSO.&nbsp;&nbsp;<\/li><li>Budget&nbsp;<\/li><li>Identifying a system and introducing it \u2013 even if it fit all&nbsp;criteria&nbsp;it may not be taken up by users&nbsp;<\/li><\/ul>\n\n\n\n<p><strong>How can you ensure a thorough evaluation process that is still time and resource effective?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>Involve more people from wider teaching and learning community. This helps to test the expectations and reality in each team&nbsp;<\/li><\/ul>\n\n\n\n<p><strong>Who should be involved in the evaluation process?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>All stakeholders really. Question writers, moderators, exam unit, students,&nbsp;Learning&nbsp;technology people, exam markers, quality unit, appeals, Student records and reporting&nbsp;<\/li><li>Don&#8217;t forget to include local technical staff<\/li><li>Senior leadership support&nbsp;<\/li><\/ul>\n\n\n\n<p><strong>What are the challenges to a more formal process?<\/strong>&nbsp;<\/p>\n\n\n\n<ul><li>Lack of buy-in&nbsp;<\/li><li>Not fit for purpose&nbsp;<\/li><li>Big&nbsp;long term&nbsp;projects&nbsp;prioritise&nbsp;resources meaning small projects (with possible big impacts) take too long to get done.&nbsp;<\/li><li>Lack of time&nbsp;<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Resources<\/h2>\n\n\n\n<p>Slides for the session are available from&nbsp;<a href=\"https:\/\/www.slideshare.net\/MariekeGuy\/making-your-mind-up-formalisingthe-evaluation-of-learning-technologies-250141794\">Slideshare<\/a> and a video is on <a href=\"https:\/\/www.youtube.com\/watch?v=dkz_g8kBFd8\">YouTube<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Yesterday&nbsp;we&nbsp;(Marieke Guy and&nbsp;Tharindu&nbsp;Liyangunawardena)&nbsp;facilitated a session at the&nbsp;ALT annual conference&nbsp;on \u2018Making your mind up:&nbsp;Formalising\u202fthe evaluation of learning technologies\u2019.&nbsp;&nbsp; ALT session&nbsp; The session&nbsp;began by&nbsp;taking a&nbsp;look&nbsp;at&nbsp;the impact on&hellip;<\/p>\n","protected":false},"author":25,"featured_media":1993,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[34,56,61],"tags":[187],"_links":{"self":[{"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/posts\/1991"}],"collection":[{"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/comments?post=1991"}],"version-history":[{"count":8,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/posts\/1991\/revisions"}],"predecessor-version":[{"id":2002,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/posts\/1991\/revisions\/2002"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/media\/1993"}],"wp:attachment":[{"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/media?parent=1991"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/categories?post=1991"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ucem.ac.uk\/onlineeducation\/wp-json\/wp\/v2\/tags?post=1991"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}