A mixed methods evaluation of the Paediatric Musculoskeletal Matters (PMM) online portfolio

Background The PMM Portfolio is comprised of the Paediatric Musculoskeletal Matters (PMM) website, the paediatric Gait, Arms, Legs and Spine (pGALS) app and e-learning modules (ELM). The target audiences are non-specialists in paediatric musculoskeletal medicine. Our study aimed to evaluate impact on learning and clinical practice. Methods Mixed methods (analytics, online survey, interviews) were used with PMM and ELM registered users and purposive sampling of users using international contacts within paediatrics and paediatric rheumatology. Data was analysed using descriptive statistics and qualitative techniques. A Paired T-Test compared self-rated confidence before and after use of the PMM Portfolio. Results There has been wide reach for all the e-resources; PMM website (662,827 hits, 262,476 users, 214 countries, data 31st July 2020); pGALS app (12,670 downloads, 70 countries, data 31st July 2020); ELM (150 users, 30 countries, data 30th May 2019). There were 164 responses (students, trainees and health care professionals) to the survey from 25 countries. Most responders deemed the PMM Portfolio useful / very useful for their learning with significantly increased self-rated confidence in their clinical examination and reasoning skills following access to the PMM website, p = < 0.01, pGALS app, p = < 0.01 and ELM, p = < 0.01. The most popular PMM website pages related to clinical assessment techniques (especially pGALS). There was high uptake of the pGALS app and pGALS ELM especially from trainees and allied health professionals. Many clinicians reported the PMM Portfolio to be useful when used to teach others. User feedback reported that easy navigation, open access, clinical images and cases were the most valued features. User feedback highlighted need to increase awareness of the e-resources through training programmes. Conclusions The PMM Portfolio was developed to aid learning for clinicians who are not specialists in paediatric MSK medicine. Our evaluation demonstrates wide international reach and positive feedback on learning. The PMM Portfolio is a highly useful e-resource for paediatric rheumatologists in their teaching of others to raise awareness, facilitate early diagnosis and referral of children with suspected disease. The wide user engagement informed future PMM Portfolio development and the mixed method of evaluation is transferable to other e-resources. Supplementary Information The online version contains supplementary material available at 10.1186/s12969-021-00567-5.


Background
Musculoskeletal (MSK) presentations in children and young people are common with a reported prevalence of 1 in 8 [1] and a frequent cause of health care consultations increasing with age (6% of 7 year olds to 16% of 22 year olds in a cohort study from the UK) [2]. Delay in diagnosis and access to specialist care is a priority to address amongst parents [3]; unfortunately delay is often reported in conditions that present with MSK features [4][5][6][7][8][9] with adverse impact on clinical outcomes [4,5,10]. The reasons for delay are multifactorial [9,11] including complex care pathways involving community and various hospital care specialties (such as general paediatrics or emergency care) [8,11] where self-rated lack of confidence in paediatric MSK clinical skills is reported [12]. Clinicians working in the community play a crucial role to suspect MSK disease and instigate specialist referral [2,7,8] and family medicine practitioners are often the 'gatekeepers' to specialist services [13]. Unfortunately many training schemes for family medicine do not include paediatrics [14], or MSK medicine [15] despite learning needs being known [16]. Other clinicians including nurses and allied health professionals (AHP) may encounter children in their clinical practice in the community, are important in the initial care pathways and yet also have unmet needs about MSK problems [17,18].
With these challenges in mind our group developed a portfolio of e-resources (called the PMM Portfolio), to aid learning and give support to all clinicians who play an integral role in the early recognition, diagnosis and initial care of children with MSK conditions. The target audiences of the PMM Portfolio therefore include a spectrum of clinical learners who are not 'paediatric MSK experts'; ranging from students in medicine and nursing, trainees in family medicine and paediatrics, through to practitioners in general paediatrics, family medicine, nursing and allied health. Paediatric rheumatologists are key to the teaching of others [19][20][21] and especially teaching clinicians to whom children may initially present; such teaching is important to raise awareness, facilitate diagnosis and referral to specialist care. Paediatric rheumatologists are often the drivers of teaching paediatric MSK medicine at faculty and institutional level [19,22,23]; our evaluation aimed to support the PMM Portfolio as a valid tool to support teaching practice.
The PMM website (www.pmmonline.org) is a free and open, evidence based, peer reviewed e-resource with user engagement to inform the content and website design functionality [24]. PMM Nursing (2017) is a further version of PMM to address the needs of the wider nursing community [17]. The PMM website includes an open, anonymised online survey and since 2014, 84 responses from 19 countries (data not shown but available on request), has informed PMM Portfolio development with requests for more global content and signposting. PMM International was hence developed and replaced the original PMM website in September 2018 following collaboration with paediatric rheumatologists in 11 countries (within Asia, Africa, Americas, Australasia and Europe) who developed additional content reflecting case mix and the health care in their country [25]. New content focused on infections and infection-related disease with MSK features, or differential diagnoses of rheumatic disease with further cases and images to reflect ethnic diversity. All contributions were subject to editorial review to ensure consistency of language and compliance with our governance framework [24,25].
Paediatric Gait, Arms, Legs and Spine (pGALS) [26] is a simple MSK examination schedule useful in clinical practice [27][28][29][30][31][32][33] and is widely taught [34,35]. The free pGALS app (launched in 2015) was developed with medical students at Newcastle University UK to inform format and content with exam revision notes and links to PMM website key pages [36,37]. Language translations of pGALS produced with our PMM International collaborators (to date 20 languages) and a version for telehealth (V-pGALS) are freely available on the pGALS app and PMM website (http://www.pmmonline.org/doctor/ approach-to-clinical-assessment/examination).
The PMM E-Learning Modules (ELM) aid navigation through the PMM website and were developed with our multi-professional team (including family medicine doctors, paediatricians, medical students) [35,37]. At the time of our evaluation there were three e-modules (https://cpd.ncl.ac.uk/courses): i) pGALS and clinical examination skills for medical students ii) Assessment of childhood MSK presentations in family medicine iii) The paediatricians approach to a child with fever

Aims
This study focuses on evaluation of the PMM Portfolio to describe and understand reach and impact on learning and clinical practice. Furthermore the evaluation aimed to inform future development of the PMM Portfolio and strategy to optimise impact.

Methods
We adopted a mixed methods approach comprising eresource analytics, an online survey (including self-rated confidence before and after access to the e-resources) and telephone interviews to explore themes raised by the survey.
Google Analytics described access to the PMM website (site hits, page hits, accessing countries) and resource analytics described pGALS App Store data (country and number of downloads) and ELM data (country, number of users, clinician role).
An online survey using©Survey Monkey (see Additional file 1) explored how the PMM Portfolio eresources impact on learning and clinical practice. The survey included sections about each of the e-resources and respondents were asked to provide feedback on all the resources (including 'non-users'defined as those who had accessed some but not all the available PMM Portfolio e-resources to give insight into barriers to access). The survey was piloted and included Likert Scale questions (on perceived usefulness and impact on confidence on MSK knowledge and skills), free text options and open-ended questions to encourage comment.
Use of various recruitment sources was important to include a range of target audiences and to enable recruitment of users and non-users of the PMM Portfolio. A random selection of PMM website registered users (n = 450) and all the users registered for the ELM at the time of survey recruitment (n = 142, July 2019) were invited to complete the online survey in addition to purposive sampling amongst user groups within the UK and through our PMM global partners. Selected 1-1 telephone interviews followed the online survey to explore the findings in more detail. Interview participants comprised medical students (n = 2), clinical lecturer paediatricians (n = 2) and family medicine doctors involved in teaching alongside their clinical practice (n = 2); 4/6 were from the UK.
Written consent was obtained from interview participants and survey respondents consented to participation through an online response. All participant information was anonymised. Interviews were audio-recorded and transcripts anonymised before data analysis. E-resource analytic and survey data was analysed using descriptive statistics, with free-text comments and interview transcripts analysed following standard procedures for  [38]. Reflexivity was maintained throughout the analysis and writing, by recording, discussing and challenging established assumptions. A Paired T Test (©Minitab) compared self-rated confidence scores (Range: 1 not very confident -5 very confident) before and after use of the e-resources. The study had ethical approval from Newcastle University, UK.

Resource analytics
The PMM website has had 662,827 hits and 262,476 users across 214 countries since launch (14th November 2014), until our data cut off for the purpose of this study (31st July 2020) - Fig. 4   nurse and AHP (n = 50), training doctors (n = 36) and clinicians (n = 35) (see Additional files 5 and 6). Most users completed one ELM (n = 128) and the remainder (n = 22) completed more than one. The ELM entitled 'pGALS and clinical examination skills for medical students' had the highest uptake (n = 130 users) followed by 'Assessment of childhood MSK presentations in family medicine' (n = 31 users), and 'The paediatricians approach to a child with fever '(n = 11 users).

Survey and interview data
The response rate to the survey is not clear as the link to the survey was sent to PMM collaborators and forwarded to their students and trainees. We do however know that minimum of 592 received the invite (from our random selection of PMM website users and all the ELM registered users at that time). We received 164 completed responses to the survey, hence we can assume a maximum response rate of 28%. Survey respondents from 25 countries (across Africa, Asia, Europe, North and South America) comprised a range of roles and levels of experience within community and hospital care (see Additional files 7 and 8). Table 3 describes feedback on all three e-resources within the PMM Portfolio (n = 120) and an additional 44 provided feedback on 2 or less (PMM website and pGALS app (n = 10), PMM website and ELM (n = 3), PMM website alone (n = 31). Some had experience using the e-resources (PMM website n = 103/164, 63%; pGALS app n = 48/131, 37% ELM n = 50/123, 41%) and others did not have experience of one or more of the eresources (PMM website n = 61/164, 37%; pGALS app n = 83/131, 63%; ELM n = 73/123, 59%). Those who had not experienced one of the e-resources were termed   non-users for the purpose of our analysis but notably they all had experience of at least one of the e-resources. Users and non-users held comparable job profiles and were from a similar varied mix of countries (see Additional files 7 and 8).
Most respondents judged the e-resources to be 'useful' or 'very useful' and rated highly the fact that they could access them quickly and easily ( Table 4). The main reason cited to access the PMM website and ELM were Continuing Professional Development (CPD)/Continuing Medical Education (CME), whereas supporting users to examine patients and for teaching purposes were main reasons to access the pGALS app. There was a difference by user group (see Table 5); students and trainees seeking pGALS guidance to support their clinical examination techniques and aiding revision being the main reasons to access the PMM website and pGALS app; paediatric rheumatologists cited the PMM website and pGALS app primarily for use in their teaching of others; nurses and AHP gave CPD/CME as the main reason to access the PMM website, with access to the pGALS app to support clinical examination technique and access to the ELM to aid understanding of a clinical problem.
Most users reported that the PMM Portfolio had a positive impact on their current clinical practice and the learning of themselves or others (Table 4), with improved clinical skills and knowledge, aiding their teaching and increasing awareness about MSK issues amongst others cited as main benefits (Table 6).
Most users reported using the PMM Portfolio within their own learning, clinical practice or teaching of others ( Table 4). The PMM website content deemed most useful related to clinical assessment and examination skills (e.g. pGALS and pREMS), normal variants, red flags, limping child guidance, links to guidelines and access to videos to demonstrate clinical skills. The pGALS guidance with illustrations and language translations were highly rated in the pGALS app. The ELM users most valued the case  based presentations with discussion about when to be concerned ('red flags').
Self-rated confidence about MSK knowledge and skills increased for all three e-resources: rated before and after using a Likert scale range -1 (not very confident) -5 (very confident); Non-users of one or more of the e-resources (PMM website n = 61, pGALS App n = 83, ELM n = 73) cited lack of awareness of their existence being the main reason but the majority reported that following study participation, they were planning to access the e-resources for their clinical practice and / or teaching purposes.
Increasing awareness of the PMM Portfolio amongst junior doctors, nurses and AHPs was suggested to increase reach of all the e-resources. Furthermore integration of the PMM Portfolio into training or CME/ CPD programmes with promotion by professional organisations were proposed to increase use of the eresources. Expanding the range of ELM topics with apps to aid use without internet access were suggestions for future development of the PMM Portfolio.

Discussion
The PMM Portfolio was developed to address the reality that many children will present to a myriad of clinicians and in most health care systems, not directly to a paediatric MSK specialist. Many of the clinicians involved in the early stages of the care pathway have only minimal, if any, experience or training in paediatric MSK medicine [12,14,15,40]. As such, they may not easily recognise which cases are appropriate for prompt referral to a paediatric specialist. This delay in referral, which results in a delay in confirmatory diagnosis and initiation of effective treatment, can significantly impact clinical outcomes. With the aim of improving access to the "right care", the PMM portfolio improves awareness, knowledge and clinical skills of a broad target audience, which includes medical and nursing students, medical trainees, nurses, AHPs and family medicine clinicians. The inclusion of medical and nursing students is important to instill essential skills and knowledge early in their career path [18,34,41]. Our evaluation demonstrates that we are reaching our target audiences and the most popular pages of the PMM website are clinical assessment, limping child, MSK infections and indicators of when to be concerned and refer to a paediatric specialist. These knowledge themes reflect essential learning in paediatric MSK medicine for medical students [42] and family medicine [16] and are key to raising awareness, facilitating early diagnosis and referral.
The mixed methods described impact with quantitative analytics (e.g. the numbers of hits, number of users, most accessed pages). Qualitative methods enabled us to explore the experiences of the users; e.g. how they use the PMM Portfolio, the impact on learning and to understand what they felt was useful and why. Our recruitment method allowed us to reach users who had used some but not all of the resources in the PMM Portfolio (e.g. those who had used the website but not the app or e-modules). Such feedback was helpful to explain why some resources had not been used and how we can further increase our reach. The methods also allowed feedback, which was used to inform iterative  • Used to improve knowledge within this area and as a refresher to update and review current knowledge base. Not all countries have access to a paediatric rheumatology specialism and for these people the site enables them to view clinical cases they might otherwise not have access to within their learning environment. • Provides an intuitive source to better inform decision-making and practice, guide patient treatment and aid explaining condition to families. In particular, it informs users about systematic approach to examination and this in turn thought to enhance confidence and ability to examine children proficiently.
• Informs users about simple systematic approach to examination and serves as useful refresher or revision aid. Increased knowledge gained from the app thought to make examination easier, enhance confidence and improve examination technique. • Equips the user with the necessary knowledge and skills to discern between abnormal and normal, screen asymptomatic and symptomatic patients and distinguish musculoskeletal conditions.
• Used to expand knowledge within a particular area of interest and to consolidate and review current knowledge base or as part of CPD. • Provides content to better inform decisionmaking and practice and give additional reasoning that can be applied when assessing patients or explaining condition to families. Increased knowledge gained from ELM thought to aid clinical reasoning and make MSK examination easier, improve examination technique and enhance confidence particularly in relation to assessment and examination.
"It has improved my confidence and skills to facilitate better outcomes". "In my country no one have a paediatric rheumatology specialty so we can learn a lot about cases from PMM and teach our student". "When I don't have any protocol (in Brazil some hospitals doesn't have at all) to guide me, I choose PMM to help me and solve some problems". "Its easy, for free and intuitive way to find answers and guide a treatment for a patient". "It will enhance my capability to check paediatrics efficiently".
"Making easier the clinical examination". "Increases my capability in diagnosis". "Improve in terms of examination, assessment, investigation and management".
"Gaining wider knowledge of signs, symptoms and examination of a child".

PMM website pGALS app ELM
• Used within undergraduate and trainee teaching material and students and trainees directed to site for self-directed learning or review.
• Used within undergraduate and trainee teaching material and students and trainees directed to app for self-directed learning.
• Used to prepare teaching material and inform teaching topics; and students and trainees directed to ELM for self-directed learning.
"I refer all trainees and CME candidates to it". "I can revise the knowledge of clinical history and examination skills before my teaching session". "Clear, focused especially on the basics that were not taught in med school. Therefore this resource is excellent as I want to teach the topics to medical students".
"I use for teaching and signpost students to it". "Acts as an introduction for me before lectures". "This is very informative and attractive for us … by this we can increase our capabilities to give suggestions to others". "I really want to do as much online courses as possible to have edge when enrolling for my masters".

PMM website pGALS app ELM
• Enables clinicians working in different specialties or areas to consider things from a rheumatology perspective. • Highlights key issues with MSK medicine.
• Increases awareness of JIA and other rheumatological conditions in children in healthcare providers within and outside of the specialism.
• Increases knowledge in colleagues and AHP.
• Completion of the courses thought to increase awareness of rheumatological conditions in children and encourage those outside the specialism to consider MSK diagnoses when assessing patients.
"Being an orthopaedic surgeon its useful to see problems from a rheumatological perspective". "PMM is a very useful website for non paediatric rheumatologists. Highly recommend as a learning resource".
"Increases knowledge in colleagues and AHP". "List out the common MSK problems of paediatrics".
development of the e-resources to optimise content and format to meet the needs of the users. Many PMM Portfolio users reported positive impact on their learning, their clinical practice and teaching of others. There was a significant increase in self-rated confidence in clinical skills and knowledge following access to all e-resources. Positive comments related to ease of use, open access and the content being at an appropriate level (either for their own learning needs or those that they teach). There was variation in access to the eresources across user groups; e.g. students and trainees primarily using the PMM website and pGALS app for clinical skills guidance and academic examination revision, AHPs reporting use of the ELM for clinical practice and clinicians (including paediatric rheumatologists) using the PMM website and pGALS app for their teaching of others. Many users accessed the e-resources for CME/CPD.
The international uptake of the e-resources reflects the wide stakeholder engagement in the PMM Portfolio design and development [17]. Most users of the PMM website were initially from the UK and US but over time, uptake has markedly increased around the world, especially amongst trainees and AHPs. Our global partners ensure relevance of content to the target audiences and provide pGALS language translations. The global partners facilitated dissemination with the countries of several PMM global partners ranking highly in those accessing the PMM Portfolio. Dissemination has also been facilitated through endorsement by professional societies (e.g. Paediatric Rheumatology European Society (PReS), Royal College of Paediatrics and Child Health (RCPCH), Royal College of Nursing (RCN), India Paediatric Society and the National Institute for Health and Care Excellence (NICE)). The PMM Portfolio is embedded in PReS Basic Courses for paediatric rheumatology around the world, postgraduate paediatric rheumatology training programmes for paediatricians (e.g. India and Kenya), NICE Clinical Knowledge Summaries for family medicine about paediatric MSK development, RCPCH guidance for postgraduate paediatric examinations, the RCN Competency Framework for nurses and the 'Call to Action' strategy of the Paediatric Task Force for Global Musculoskeletal Health [43].
We firmly believe that user engagement is integral to iterative development of e-resources and optimising impact as a means of knowledge transfer [44]. User engagement provided ideas to further increase reach and these included; 1) Increasing awareness of the PMM Portfolio amongst students, trainees and AHPs; notably lack of awareness was the most cited reason amongst non-users. 2) Integration of the PMM Portfolio in training programmes through links with training bodies and increasing their exposure at CME/ CPD events. 3) More ELM with topics relevant for the global context -most ELM registered users reside in UK/ Ireland suggesting more work is needed to promote these internationally. 4) Offering additional e-formats to enable offline access would facilitate further their uptake. 5) Maintaining open and free access; all the e-resources are free of charge other than one ELM ('The paediatricians approach to a child with fever'), which had higher development costs and the charge may have contributed to the lower uptake. With these suggestions in mind, work is underway to develop a PMM app to enable offline access to the PMM website content, the ELM portfolio now includes a module targeting physiotherapists (focus on gait) and a further module for school teachers is planned. To date all author contributions have been forthcoming without financial reimbursement and we gratefully acknowledge the valuable input from all our PMM partners. Funding is a major barrier to future PMM Portfolio development and we are actively working to secure sustainability and growth whilst maintaining the ethos of PMM being free and open to all.

Limitations of our study
Our informal approach to recruitment resulted in the survey response rate being imprecise. However, this approach to recruitment enabled reach to both users and non-users of the e-resources and gave valuable insights into barriers to use and ways to encourage uptake further. There was not an even spread amongst the numbers of responders per user group or by country and this may have introduced bias. For example, there was a high proportion of survey respondents from India and amongst AHPs (who gave very favorable feedback). Their responses were nonetheless comparable to other respondent groups, so we suggest that any effect on the overall findings is minimal. Their feedback was very valuable for future work to target clinicians, especially AHPs who are integral to paediatric MSK care in areas of the world with workforce challenges [20,40,43,45]. Our methods explored reach and impact on learning, clinical practice and teaching; ideally evaluation would include influence on clinical outcomes (such as access to specialist care) but given that referrals are dependent on several variables (including local referral pathways and availability of specialists), a different evaluation approach would be needed.

Implications for research and practice
The evaluation of the PMM Portfolio is very relevant to paediatric rheumatologists who are integral to the teaching of others [19][20][21]23] to raise awareness and facilitate diagnosis and referral. Paediatric rheumatologists are often the drivers and champions of paediatric MSK education at faculty and institutional level [19,20,23] and we hope that awareness of the PMM Portfolio and it's positive evaluation will support its use by paediatric rheumatologists in their teaching. The PMM Portfolio is increasingly linked with educational activities delivered by paediatric rheumatologists such as the PReS Basic Courses around the world; PMM provides essential reading and preparation for the courses. Such educational activities facilitate growth of global paediatric rheumatology, especially as the content of PMM is written by paediatric rheumatologists around the world to maintain relevance to different health care contexts.
The use of e-resources in paediatric rheumatology is important; increasingly so during the COVID-19 pandemic with rapid escalation of e-learning in clinical education [46], innovative e-learning platforms [47] and the use of telemedicine [48] to reach many new users around the world. This is particularly relevant to paediatric rheumatology given the fact that many children with MSK conditions live in parts of the world (Asia and Africa) with little or no access to specialist care [49]. The PMM portfolio is therefore an exemplar model to facilitate e-learning and workforce capacity building to enable global paediatric rheumatology [43] and we anticipate a greater need to integrate e-learning with 'face to face' training schemes.
Our evaluation with mixed (quantitative and qualitative) methods is a valid approach applicable to other eresources and will be useful to the paediatric rheumatology clinical educational community; understanding how to tailor e-resources to user needs and ways to optimise impact is of increasing importance given the investment in time and funding to set up and sustain e-learning programmes. Our approach highlights the importance of user engagement in iterative development, content being relevant and at the appropriate level and ways to optimise reach and impact.

Conclusions
The PMM Portfolio is fulfilling an important educational role reaching many target user groups across the world. The PMM Portfolio continues to grow and engagement with users will facilitate future iterations maintaining relevance for the global context.