Photo by Kelly Sikkema on Unsplash
In July 2023 the Learning Technologies team at UCEM managed a significant upgrade to the VLE. An earlier blog post covers the ways in which we involved staff and students in the testing and evaluation of the upgrade. Now that the dust has settled, the team has had an opportunity to evaluate the upgrade project, identifying potential risks, concerns, and best practice for future Learning Technology projects.
Sources of feedback
Thanks to the design of the upgrade project we had multiple sources of feedback from both staff and student users of the VLE. While the main subject of this feedback was the features and design of the upgrade, we were also able to glean informal feedback on the project plan and deployment.
We used feedback from the staff facing upgrade channel, the staff feedback survey, the student feedback survey, the Learner Experience Student Panel, and Student Representative meetings, for our evaluation.
Specifically, in December 2023 we attended a Student Representatives meeting to ask directly for feedback on the way the upgrade was handled. We reminded Student Representatives of the different surveys, feedback opportunities, and communications relating to the upgrade and asked them to gather responses from their fellow students on their experience.
We also ran a staff feedback survey specifically on the project plan and deployment. The response rate for this survey was too low to be able to use the data, however the response we did get was positive in regards to the amount, timing, and format of information, opportunities to get involved and give feedback, and quality of supporting resources. The involvement of students in the project was also indicated to be valuable.
Lessons learned: Best practice
We received positive feedback on the early notification of the upgrade, which allowed people to prepare for the changes. Feedback was also positive in relation to the number of opportunities to try out new features and give feedback to shape the new look VLE.
From the Student Representative meeting we learned that students appreciated the timing and frequency of communications, and found the banner at the top of the VLE valuable as a means of notification. They also praised the new User Tours as a means to showcase new features and guides students through new processes.
‘Communication was good both before and after the upgrade.’
Student representatives, December 2023
‘The messaging sounds to be good, the message at the top of the VLE is good, it means even if not reading messages or emails it can be seen and noted.’
‘Tours used to get to initial grips with VLE, they were concise and helpful.’
While the main focus of this session was the handling of the upgrade as opposed to the outcome, we also received a lot of positive feedback about the new look and functionality of the VLE, and were happy that the project had an overall positive outcome.
Lessons learned: Risks and Concerns
The main issue raised in feedback from both staff and student users was the timing of the upgrade and downtime. The timing of upgrades to the VLE is controlled by our Learning Management System provider, who manage the UCEM VLE, and is something we are unable to control ourselves. Sometimes we are given a choice of several potential upgrade dates and select the one with the least impact on assessments, but more often we are given no choice on the date and time of upgrade. In these situations, the Learning Technology team will aim to identify assessments that will be impacted by the downtime and work with key stakeholders to determine and put in place the most appropriate mitigations.
Given the amount of feedback relating to this topic, we have identified that we could be more transparent about the reason for downtime occurring when it does, and that we make information about the timing of upgrades and downtime available as far in advance as possible.
Approach to future projects
The outcomes of this evaluation will be used to guide future Learning Technology projects, such as future upgrades to the VLE or other Learning Technology tools, or evaluations of new tools and approaches.
- Feedback on the amount and format of communications and support around the upgrade was overwhelmingly positive, so we will aim apply a similar approach to future projects, communicating high level timelines to the wider staff body through the Staff Bulletin early in the process, and providing more detailed communication channels in Teams for relevant parties to join and stay informed. In particular, where a project will involve downtime, we will provide clear and transparent communications as early as possible to allow all user groups to prepare for outages.
- We also recognise that user testing is a key component of any evaluation of a Learning Technology tool, and omitting this stage could lead to us missing important use cases and potential issues or opportunities to make further improvements. We will involve relevant users in projects by creating test environments and inviting interested parties to get involved in testing and feedback wherever possible.
- As our biggest user group, the student voice is essential to our work, and we will take care to involve student representatives and consultation groups in the testing and feedback process wherever possible and appropriate.
- Finally, we received a lot of positive feedback on the supporting FAQs and User Tours used to support the upgrade. We will include the development of supporting resources as a key part of all projects that involve new tools, approaches, or use cases.
We hope that following the above will result in all of our projects being as successful as the 2023 Moodle Upgrade, ensuring that all user groups feel informed, involved, and listened to, and providing the maximum benefits and minimum disruption possible.
Learning Technologies Production Manager
