V1 Clinical Evaluation Report
Author: Dr Greg Burch: Clinical Director, Tiny Medical Apps
Table of Contents
1.0 Executive Summary
2.0 Background
3.0 National assessment and local assurance
4.0 Evaluation
4.1 Content evaluation and evidence base
Evidence for asthma plans and self-management education reducing hospital attendances
NICE Asthma Quality Standard (QS25) Sept 2018
Validated Behaviour Change Techniques
4.2 Digital Health Passport assessment against NICE standards
The NICE Digital Health Technology tiers
Tier 2 Evidence for the Digital Health Passport
Partial Evidence for Tier 3a
4.3 NASSS Framework Evaluation
Assessment overview
4.4 Clinician Evaluation
4.5 Patient Evaluation
4.6 Technical Assessment
Evaluation Report Appendices
Appendix 1 – Clinician co-design and feedback
Appendix 2 – Patient feedback
1.0 Executive Summary
Thanks to the investment from Healthy London Partnership, the Digital Health Passport has overcome all of the barriers needed to be ready to scale. Projects are already underway in Manchester and Sheffield capitalising on the work done to date. This report gives some background to the work before the pilot and focuses on the initial evaluation work. Evaluation will be continuous as we gain more data and increase user numbers.
The Digital Health Passport
- has been designed for teenagers to take control of their health and has an initial focus on asthma self-management with Asthma UK action plans.
- enables remote tracking of symptoms, displaying emergency plans and accessing NHS support.
- has been co-designed with young people and health care professionals including school nurses, asthma nurses, GPs and consultant asthma specialists from paediatrics and adult services.
- addresses key recommendations in: NICE Asthma Quality Standard, BTS/SIGN clinical guideline 153, GINA 2019 and The National Review of Asthma Deaths (NRAD 2014).
- meets all high level NHS assurances and has been accepted into the NHS Apps Library and is assured to use NHS login.
- has been assessed to contain 14 Validated Behaviour Change Techniques.
From the NICE Digital Evidence Standards Framework the DHP demonstrates evidence of effectiveness at Tier 2 and partial evidence for Tier 3a.
- TMA have evaluated the DHP Pilot using the NASSS Framework to inform both future adoption, use and commissioning.
- Interviews were carried out with users of the DHP and gave overwhelmingly positive feedback. Valuable information has been obtained to influence the design and adoption of the next version.
- Twelve site visits to observe and evaluate use of the portal and seven follow up interviews with clinicians were conducted to ensure their requirements were met.
2.0 Background
The Digital Health Passport has been designed for young people to take control of their health – creating asthma action plans, tracking symptoms and accessing NHS support.
The project has been led and commissioned by the NHS Healthy London Partnership, Children & Young People’s team as part of their work to improve asthma standards in London. In the past few years young people in the UK have had worse outcomes from asthma than in most other countries in Europe, and there have been a number of preventable asthma deaths.
Young people with a personalised asthma action plan are four times less likely to go to A&E – so a key feature of the app is the action plan from Asthma UK which gives instructions and advice of what to do if your asthma is getting worse.
The Digital Health Passport has been co-produced with young people, school nurses, GPs and asthma specialists in east London and is now available in the NHS Apps Library by invite only as it is further refined with the first users. It is being piloted and tested at the Royal London and Barts hospitals and at Chrisp Street GP practice in Tower Hamlets.
This innovative project has been recognised by NHS England and is one of only a handful of ‘Personal Health Record’ Apps to be evaluated around the country. It is now being rolled out in Greater Manchester and South Yorkshire with additional features planned to support young people with allergies, epilepsy and other long-term conditions.
The main features of the Digital Health Passport are
- Asthma UK action plan
- Emergency plan
- Track symptoms on a visual timeline
- NHS health advice and Asthma management educational information
- Air quality levels (pollution, pollen and weather changes)
3.0 National assessment and local assurance
The Digital Health Passport requires a high level of regulatory assurance.
Significant effort has gone into delivering a safe and assured platform.
NHS Apps Library
Gaining acceptance to the NHS Apps Library has been time consuming, but is a huge assurance hurdle that has been overcome. The bedrock of the process is the Digital Assessment Questionnaire (DAQ) which requires passing assurance in seven domains from clinical need, evidence to information governance and security.
NHS Login
The developers Tiny Medical Apps were invited to be in the first wave of companies allowed to use NHS login. We have invested in attaining all of the additional assurance requirements to enable rapid integration with regional Local Health & Care Records such as One London.
Clinical Safety DCB 0129, DCB 0160:
These standards provide a set of requirements suitably structured to promote and ensure the effective application of clinical risk management by those health organisations that are responsible for the deployment, use, maintenance or decommissioning of Health IT Systems within the health and care environment.
ISO/IEC 27001
ISO/IEC 27001 Information Security Management system is designed to help organisations manage their information security processes in line with international best practice
Our certification is externally audited by BSI and our scope specifically covers the Digital Health Passport platform.
Cyber Essentials +
Cyber Essentials helps us to guard against the most common cyber threats and demonstrates our commitment to cyber security. We are Cyber Essentials + certified which means we are also externally audited.
SCAL
The Supplier Conformance Assessment List (SCAL) is a technical document which details the consumer supplier approach to information governance, clinical safety, functional testing and SMSP-PDS requirements. As part of our compliance and conformance assessment for NHS Login we successfully completed the requirements of the SCAL.
Data Security and Protection Toolkit
The Data Security and Protection Toolkit is an online self-assessment tool that allows organisations to measure their performance against the National Data Guardian’s 10 data security standards.
All organisations that have access to NHS patient data and systems must use this toolkit to provide assurance that they are practising good data security and that personal information is handled correctly.
4.0 Evaluation
4.1 Content evaluation and evidence base
The content of the Digital Health Passport is consistent with the best evidence for reducing asthma exacerbations and reducing unplanned hospital attendances as part of a supported case management approach. The asthma plan and the educational content is provided by Asthma UK and the NHS
Having a personalised asthma action plan is a NICE quality standard.
Improving outcomes will come from behaviour change and patient activation. From the first pilot we are introducing behaviour change techniques that will expand, test and refine in future iterations. A Queen Mary’s University study demonstrated 14 behaviour change techniques within the Digital Health Passport app.
Evidence for asthma plans and self-management education reducing hospital attendances
The evidence in favour of supported self-management for asthma is overwhelming. Self-management including provision of a written asthma action plan and supported by regular medical review, almost halves the risk of hospitalisation, significantly reduces emergency department attendances and unscheduled consultations, and improves markers of asthma control and quality of life (Pinnock, Breathe 2015).
The British Thoracic Society/Scottish Intercollegiate Guideline Network (BTS/SIGN) asthma guideline cites 261 randomised controlled trials reported in 22 systematic reviews in support of its grade A recommendation that “all people with asthma (and/or their parents or carers) should be offered self-management education which should include a written personalised asthma action plan and be supported by regular professional review”
NICE Asthma Quality Standard (QS25) Sept 2018
NICE Quality Statement 1
“People (5 years old and over) with asthma discuss and agree a written personalised action plan.”
https://www.nice.org.uk/guidance/qs25/chapter/Quality-statements
What the quality statement means for each audience:
- Service providers ensure systems are in place for people with asthma to receive a written personalised action plan.
- Healthcare professionals ensure they give people with asthma a written personalised action plan.
- Commissioners ensure they commission services that give people with asthma a written personalised action plan.
- People with asthma receive a written plan with details of how their asthma will be managed.
A written personalised action plan (such as Asthma UK’s asthma action plan) should be tailored to the person with asthma, enabling them to recognise when symptoms are worse. The plan should set out actions to be taken if asthma control deteriorates and who to contact.
Source guidance:
- NICE guideline NG80, (2017) Asthma: diagnosis, monitoring and chronic asthma management, recommendations 1.10.1 and 1.10.2
- BTS/SIGN clinical guideline 153(2016) British guideline on the management of asthma, recommendation 5.2.2
- GINA 2019 Global Initiative for Asthma Ch 3, pg69
- The National Review of Asthma Deaths (NRAD 2014) recommended the use of Personalised Asthma Action Plans as have multiple Coroners’ reports into avoidable deaths from asthma.
Validated Behaviour Change Techniques
The provision of an agreed self-management plan and educational materials are core features of the Digital Health Passport, however in order to maximise the potential of the tool we are including validated behaviour change techniques and plan to test their effectiveness in a large randomized controlled trial in east London in the coming years.
An analysis by Dr Samaresh Mazumdar and Dr Liz Edwards, under supervision of Prof Chris Griffiths and Dr Anna De Simoni of Queen Mary’s University London identified the 14 BCT’s in use in the app and recommended additional methods we could introduce based on analysis of 50 asthma apps used internationally. We jointly identified the Australian app ‘Kiss My Asthma’ as the leader in the field and worth emulating in many respects. Future versions will incorporate and evaluate further BCTs based upon the recommendations from QMUL researchers, particularly with greater ability to contribute to the care plan from patients with regard to goal setting, action planning and thus increased status within the team. Features such as medication reminders and ‘gamification’ are being introduced into the next version.
From the BCT taxonomy we can demonstrate the use of the following techniques:
1.2 – Problem Solving – with the use of information provided in videos about avoiding triggers
1.4 – Action Planning – in the action plan/emergency
2.3 – Self Monitoring of behaviour – logging of peak flows
2.4 – Self monitoring of outcomes of behaviour – symptom logging
3.1 – Social support unspecified – ‘my team’ section
4.1 – Instruction on how to perform behaviour – video instructions on PEFR/spacer use
5.1 – Information about health consequences – outlined in videos
5.4 – Monitoring of emotional consequences – mood log
6.1 – Demonstration of behaviour – video instructions on PEFR/spacer use
8.1 – Behavioural Practice – videos and encouraging daily peak flows/preventer use
8.3 – Habit Formation – encouraging daily use through the timeline/ calendar homepage
9.1 – Credible Source – Asthma UK/NHS branding
11.1 – Pharmacological Support – encouraging the use of inhalers
15.3 – Focus on past success – calendar homepage showing previous good days
It contains 93 techniques to change behavior that are hierarchically clustered into 16 groups.
4.2 Digital Health Passport assessment against NICE standards
The Digital Health Passport can now demonstrate evidence of effectiveness at Tier 2 and partial evidence for Tier 3a (Behaviour change techniques).
The NICE Digital Health Technology tiers
- Digital Health Technology (DHTs) are classified by function and stratified into evidence tiers (based on level of risk)
Tier 1
DHTs with potential system benefits but no direct user benefits
Tier 2
DHTs which help users to understand healthy living and illnesses but are unlikely to have measurable user outcomes.
- Inform
- Simple monitoring
- Communication
Tier 3a
DHTs for preventing and managing diseases. They may be used alongside treatment and will likely have measurable user benefits.
- Preventative behaviour change
- Self-manage
Tier 3b
DHTs with measurable user benefits, including tools used for treatment and diagnosis, as well as those influencing clinical management through active monitoring or calculation. It is possible DHTs in this tier will qualify as medical devices.
- Treat
- Active monitoring
- Calculate
- Diagnose
We aim to demonstrate complete evidence of effectiveness at Tier 3a in the next 12 months by evaluating with licensed Patient Activation Measure scores (skills, knowledge and confidence to self-manage).
Higher-risk Digital Health Tools require a higher level of evidence for the Tier. Children and vulnerable groups are at higher risk. This means a higher level of evidence is required for the Digital Health Passport than if it was only for adults.
Tier 2 Evidence for the Digital Health Passport
Credibility with UK health and social care professionals (tier 1)
“Has a plausible mode of action and reflects current standard/best practice in the UK health and social care system or provides an alternative to standard/best practice that is beneficial to users and the health and social care system”
A large number of asthma specialists, adult and paediatric have been involved in the co-design of this product from inception including Prof Chris Griffiths, (Deputy Director Asthma UK, Centre for Applied Research, Dr Chinedu Nwokoro (Children’s Asthma Lead, Royal London), Dr Paul Pfeffer (Severe Asthma Lead Adult, Bart’s Hospital), Tori Hadaway (Community Asthma Nurse), Dr Richard Iles (Paed Resp Cnslt Evelina), The Tower Hamlets School nurse team and multiple other stakeholders.
The content evaluation demonstrates use of asthma care plans that are the recommended best practice with a strong evidence base for the paper equivalent. The content and behaviour change techniques provide a plausible mode of action.
Relevance to current care pathways in the UK health and social care system (tier 1)
“For the best practice standard, evidence could include published or unpublished reports describing the successful implementation of the DHT showing benefits to users in the UK health and social care system.”
Having an asthma action plan, receiving advice on inhaler technique and completing symptom diaries are fundamental parts of asthma care pathways. We can demonstrate improvements in care pathways for clinical and patient users (eg, the ability to generate a completed pdf that can be uploaded without scanning, ability to send asthma action plan directly to a patient’s mobile phone, the ability for a patient to record a symptom diary on their phone and easily access educational resources).
Acceptability with users (tier 1)
“Some evidence to show that potential users of the DHT have tested it and found it to be usable and useful will help to show that implementing the DHT may be successful. Evidence could include reports from user or user group testing, or showing that users have been consulted in the design and development process.”
Patient and Clinician users have been interviewed and the feedback can be provided upon request. See Appendices 1,2,3.
Equalities considerations (tier 1)
“Consider whether the DHT helps to reduce any existing inequalities within the health and social care system. This could include factors such as digital exclusion, or use by hard-to-reach populations.”
“Indicate any equalities considerations needed when commissioning, adopting or implementing the DHT, particularly in reference to the Equality Act 2010.”
Digital Health Technology may have unforeseen consequences such as creating a two-tier system through digital exclusion – this may become more of a risk with the introduction of NHS login to access some services. The Digital Health Passport is an alternative to current paper based pathways which should remain in place.
Reliable information content (tier 2)
“Any information or advice to users concerning health, healthy living, lifestyle, diseases, illnesses or conditions must be correct and relevant.“
The content for the Digital Health Passport comes from trusted and reliable sources: Asthma UK and NHSgo.
Ongoing data collection to show usage of the DHT (tier 2)
“To ensure value for money to the health and social care system, the DHT owner must commit to providing data showing that the DHT is used as expected by the intended user group after adoption.”
Used ‘as expected’ defined as one of the following:
– view their care plan
– view their emergency plan
– complete a symptom tracker form (asthma review)
– watch a video, or link out to NHS go
– check an air quality or pollen level
For some people to do 2 or more of the following:
– view their care plan
– view their emergency plan
– complete a symptom tracker form (asthma review)
– watch a video, or link out to NHS go
– check an air quality or pollen level
For some people to do any of the following, on multiple occasions:
– view their care plan
– view their emergency plan
– complete a symptom tracker form (asthma review)
– watch a video, or link out to NHS go
– check an air quality or pollen level
Evidence of patient usage demonstrating that users have met these requirements is available upon request..
Ongoing data collection to show value of the DHT (tier 2)
“To ensure value for money to the health and social care system, the DHT owner must commit to providing data demonstrating that people using the DHT are showing the expected benefits from its use. This could include improvements in symptoms or general health measures.”
It is too early to show any improvements in symptoms or general health measures. This will require a much more robust evaluation. Over the next 12 months, whilst demonstrating Tier 3a evidence we will use the validated Patient Activation Measure score. This is a 13 question system to assess an improvement in skills, knowledge and confidence to self-manage.
Quality and safeguarding (tier 2)
“Some DHTs provide chat platforms or peer-to-peer communication, or link the user to support from third-party organisations. The DHT owner should be able to clearly identify who the user can interact with, describe why these interactions are appropriate, any risks in those interactions, and what safeguarding measures have been put in place.”
N/A – There is no 2 way communication from within the DHP
Partial Evidence for Tier 3a
Use of appropriate behaviour change techniques (tier 3a)
“DHTs that aim to change the behaviour of the users should be consistent with accepted and effective behaviour change techniques. The DHT owner should be able to describe which behaviour change techniques are used and provide references to these”
See content evaluation
From the BCT taxonomy we can demonstrate the use of 14 BCTs
Demonstrating effectiveness (tier 3a, best practice standard)
“A high quality intervention study using a quasi-experimental or experimental design would compare the effect of the DHT on a group of users with 1 or more groups having a different (or no) intervention. The study would report the difference between the groups. It would include statistical considerations such as sample size and statistical testing, report outcomes that are relevant to the condition, and be clear on reporting the outcomes of every person in the group testing the DHT. Ideally, the comparator group would be people having current standard care, but it could also be a before-and-after study (measuring people’s symptoms over a period of time before they use the DHT then comparing this with while they are using the DHT).“
Evidence plan – Use of PAMs with a larger number of users as a before and after study.
4.3 NASSS Framework Evaluation
Many promising technological innovations in health and social care are characterized by non adoption or abandonment by individuals or by failed attempts to scale up locally, spread distantly, or sustain the innovation long term at the organization or system level.
The NASSS framework has been developed by Trish Greenhalgh and others to be used at any time in a project lifestyle
Projects with too many domains in the complex/complicated zones will fail to achieve scale and sustainability. It can be seen that the Digital Health Passport is well positioned to scale.
https://www.ncbi.nlm.nih.gov/pubmed/29092808
Assessment overview
Current assessment position of DHP is highlighted in bold (updated March 2021)
Domain/question | Simple | Complicated | Complex | |
Domain 1: The condition or illness | ||||
1A. What is the nature of the condition or illness? | Well-characterized, well-understood, predictable | Not fully characterized, understood, or predictable | Poorly characterized, poorly understood, unpredictable, or high risk | |
1B. What are the relevant sociocultural factors and comorbidities? | Unlikely to affect care significantly | Must be factored into care plan and service model | Pose significant challenges to care planning and service provision | |
Domain 2: The technology | ||||
2A. What are the key features of the technology? | Off-the-shelf or already installed, freestanding, dependable | Not yet developed or fully interoperable; not 100% dependable | Requires close embedding in complex technical systems; significant dependability issues | |
2B. What kind of knowledge does the technology bring into play? | Directly and transparently measures [changes in] the condition | Partially and indirectly measures [changes in] the condition | Link between data generated and [changes in] the condition is currently unpredictable or contested | |
2C. What knowledge and/or support is required to use the technology? | None or a simple set of instructions | Detailed instruction and training needed, perhaps with ongoing helpdesk support | Effective use of technology requires advanced training and/or support to adjust to new identity or organizational role | |
2D. What is the technology supply model? | Generic, “plug and play,” or COTS solutions requiring minimal customization; easily substitutable if supplier withdraws
COTS: customizable, off-the-shelf. |
COTS solutions requiring significant customization or bespoke solutions; substitution difficult if supplier withdraws | Solutions requiring significant organizational reconfiguration or medium- to large scale-bespoke solutions; highly vulnerable to supplier withdrawal | |
Domain 3: The value proposition | ||||
3A. What is the developer’s business case for the technology (supply-side value)? | Clear business case with strong chance of return on investment | Business case underdeveloped; potential risk to investors | Business case implausible; significant risk to investors | |
3B. What is its desirability, efficacy, safety, and cost effectiveness (demand-side value)? | Technology is desirable for patients, effective, safe, and cost effective | Technology’s desirability, efficacy, safety, or cost effectiveness is unknown or contested | Significant possibility that technology is undesirable, unsafe, ineffective, or unaffordable | |
Domain 4: The adopter system | ||||
4A. What changes in staff roles, practices, and identities are implied? | None | Existing staff must learn new skills and/or new staff be appointed | Threat to professional identity, values, or scope of practice; risk of job loss | |
4B. What is expected of the patient (and/or immediate caregiver)—and is this achievable by, and acceptable to, them? | Nothing | Routine tasks, eg, log on, enter data, converse | Complex tasks, eg, initiate changes in therapy, make judgments, organize | |
4C. What is assumed about the extended network of lay caregivers? | None | Assumes a caregiver will be available when needed | Assumes a network of caregivers with ability to coordinate their input | |
Domain 5: The organization | ||||
It is assumed organisations will have limited slack resources | 5A. What is the organization’s capacity to innovate? | Well-led organization with slack resources and good managerial relations; risk taking encouraged | Limited slack resources; suboptimal leadership and managerial relations; risk taking not encouraged | Severe resource pressures (eg, frozen posts); weak leadership and managerial relations; risk taking may be punished |
There is currently strong appetite for Digital tools across the NHS | 5B. How ready is the organization for this technology-supported change? | High tension for change, good innovation-system fit, widespread support | Little tension for change; moderate innovation-system fit; some powerful opponents | No tension for change; poor innovation-system fit; many opponents, some with wrecking power |
The formation of ICSs will likely lead to improved regional decision making and faster adoption | 5C. How easy will the adoption and funding decision be? | Single organization with sufficient resources; anticipated cost savings; no new infrastructure or recurrent costs required | Multiple organizations with partnership relationship; cost-benefit balance favorable or neutral; new infrastructure (eg, staff roles, training, kit) can mostly be found from repurposing | Multiple organizations with no formal links and/or conflicting agendas; funding depends on cost savings across system; costs and benefits unclear; new infrastructure conflicts with existing; significant budget implications |
5D. What changes will be needed in team interactions and routines? | No new team routines or care pathways needed | New team routines or care pathways that align readily with established ones | New team routines or care pathways that conflict with established ones | |
5E. What work is involved in implementation and who will do it? | Established shared vision; few simple tasks, uncontested and easily monitored | Some work needed to build shared vision, engage staff, enact new practices, and monitor impact | Significant work needed to build shared vision, engage staff, enact new practices, and monitor impact | |
Domain 6: The wider context | ||||
6A. What is the political, economic, regulatory, professional (eg, medicolegal), and sociocultural context for program rollout? | Financial and regulatory requirements already in place nationally; professional bodies and civil society supportive | Financial and regulatory requirements being negotiated nationally; professional and lay stakeholders not yet committed | Financial and regulatory requirements raise tricky legal or other challenges; professional bodies and lay stakeholders unsupportive or opposed | |
Domain 7: Embedding and adaptation over time | ||||
7A. How much scope is there for adapting and coevolving the technology and the service over time? | Strong scope for adapting and embedding the technology as local need changes | Potential for adapting and coevolving the technology and service is limited or uncertain | Significant barriers to further adaptation and/or coevolution of the technology or service | |
Dependent on organisation | 7B. How resilient is the organization to handling critical events and adapting to unforeseen eventualities? | Sense making, collective reflection, and adaptive action are ongoing and encouraged | Sense making, collective reflection, and adaptive action are difficult and viewed as low priority | Sense making, collective reflection, and adaptive action are discouraged in a rigid, inflexible model |
4.4 Clinician Evaluation
The following clinicians have given feedback into the design, functionality and validation of the Digital Health Passport during the pilot phase:
Dr Chin Nwokoro (Paediatric Consultant and Respiratory Lead, Royal London Hospital)
Dr Paul Pfeffer (Respiratory Consultant RLH and Difficult Asthma Lead for East London)
Dr Julia Moody (GP, Chrisp Street and CCG Lead for Children and Maternity)
Dr Jim Cole (GP, Chrisp Street, and QMUL Clinical Effectiveness Group0
Charlotte Carrick (Children’s Respiratory Nurse, RLH)
Tori Hadaway (Community Asthma Nurse, Tower Hamlets)
Rachel McCready (Practice Nurse, Chrisp St)
Jane Simpson (Respiratory Nurse, Bart’s)
Anne-Marie Casey (Respiratory Nurse, Bart’s)
Dr Jonathan Grigg (Paediatric Consultant, RLH)
Summary of key points
- Twelve site visits to observe and evaluate use of the portal
- Seven follow up interviews were conducted
- High ‘Did not attend’ rate and overall low number of teenage patients attending asthma reviews despite proactive measures
- Nurses are the main users, then GPs, not used by consultants
- Feedback about the patient app was very positive
- Feedback about the clinical portal was mixed
- It is functional and little training is required to use
- Issues around remembering another login
- Not interoperable with EMIS (or any system)
- Time constraints
- Staff awareness (forget to use or to book longer appointments)
- Change requests to portal were prioritised and implemented if considered high enough priority during the pilot (eg re-engineering for old versions of MS Explorer)
- Changes need to made to work flow processes as well as technology
- Targeting CYP in schools may be better approach
See Appendix 1 (Clinician feedback)
4.5 Patient Evaluation
“Easy to log onto and good to know what level my asthma is at – knowing if I need to wear a coat or hat is really good.” Alice, DHP User
“We are always trying to encourage Alice to look after her asthma herself and it helps her be independent…..really great as she has been discharged from hospital and the app helps her monitor her asthma and she lets us know how she is getting on” Susan, Alice’s Mum
“I really liked it because it was simple to use” Jared, DHP User
“The look and feel is really good – not just a boring NHS App and the interactive background is especially good for younger users” Lucy, DHP User
“As an asthmatic in the last two years I’ve been to A&E twice and if I’m having trouble breathing I can take my phone out and say look at this plan. That would be really handy. The name as a Digital Health Passport is exactly it – you can travel around with it and use it as and when you enter into a service.” Saira, DHP User
“It’s really cool” Robert, DHP User
Summary
- Lower initial signup numbers than anticipated
- Useful and usable insights gained
- Easy to use and no negative comments about design or functionality
- Most people use some features
- Small number of users use intensively
- Many additional feature requests and ideas to further improve the design
See Appendix 2 (User feedback)
4.6 Technical Assessment
In order to guide the next stages of the development of the Digital Health Passport Tiny Medical Apps have been commissioned by Healthy London Partnership to appraise the technical requirements of scaling the platform in London. This has involved a number of technical meetings with stakeholders in London and outside which are listed in the attached separate report. The key ambition is to support a generic approach to Personal Health Records across the One London Local Health Care Record region to reduce the costs and risks for this project and other similar digital approaches in the future.
Whilst we have highlighted the fact that requirements for scalability rely on working with novel solutions at a regional and national level these risks are somewhat mitigated by a clear urgency to support these across the NHS at local, national and regional levels. This has been demonstrated by a willingness to provide in kind support.
Key requirements for scalability are:
- Patient authentication (via NHS Login).
- Two-way interoperability of Care Records (via Local Health Care Records using FHIR Care Connect Profiles) – still innovative in London at least for PHRs
- Data Persistence approach for Patient entered data by NHS regions – this is still at an early stage of development in most regions.
- A longer term requirement for standards around Digital Care Plans is still an unmet need. This should not be a barrier initially but our approaches are discussed in more detail within this report.
The full technical evaluation is available upon request.
Evaluation Report Appendices
Appendix 1 – Clinician co-design and feedback
Appendix 2 – Patient feedback
Appendix 1 – Clinician co-design and feedback
“This app is just brilliant and life saving”
Anne Marie, Respiratory Nurse, St Bartholomew Hospital, London
“Really useful when patients come back into clinic recording peak flow rather than paper is a
real positive. The links are really good – one patient this morning had poor inhaler technique and
having additional information that they access in one place is really useful”
Jane, Respiratory Nurse, St Bartholomew Hospital, London
The following clinicians have given feedback into the design and functionality of the Digital Health Passport Clinical Portal during the pilot phase:
Dr Chin Nwokoro (Paediatric Consultant and Respiratory Lead, Royal London Hospital)
Dr Paul Pfeffer (Respiratory Consultant RLH and Difficult Asthma Lead for East London)
Dr Julia Moody (GP, Chrisp Street and CCG Lead for Children and Maternity)
Dr Jim Cole (GP, Chrisp Street, and QMUL Clinical Effectiveness Group0
Charlotte Carrick (Children’s Respiratory Nurse, RLH)
Tori Hadaway (Community Asthma Nurse, Tower Hamlets)
Rachel McCready (Practice Nurse, Chrisp St)
Jane Simpson (Respiratory Nurse, Bart’s)
Anne-Marie Casey (Respiratory Nurse, Bart’s)
Dr Jonathan Grigg (Paediatric Consultant, RLH)
Summary of key points
- Twelve site visits to observe and evaluate use of the portal
- Seven follow up interviews were conducted
- High ‘Did not attend’ rate and overall low number of teenage patients attending asthma reviews despite proactive measures
- Nurses are the main users, then GPs, not used by consultants
- Feedback about the patient app was very positive
- Feedback about the clinical portal was mixed
- It is functional and little training is required to use
- Issues around remembering another login
- Not interoperable with EMIS (or any system)
- Time constraints
- Staff awareness (forget to use or to book longer appointments)
- Change requests to portal were prioritised and implemented if considered high enough priority during the pilot (eg re-engineering for old versions of MS Explorer)
- Changes need to made to work flow processes as well as technology
- Targeting CYP in schools may be better approach
Dr Julia Moody (GP)
- Adoption requires system change within the practice
- Appointment type needed to be identified ahead of time – training of assistant team required as well as clinician.
- Chrisp Street sent out an invitation to targeted cohort of 110 patients via text message to book an asthma review using app and received 1 response
- Personal invitation to come to asthma review worked better
- Barriers included 2 not having smart phones, 1 no data plan
- Language barrier also highlighted as barrier
- Younger people seemed more enthusiastic – maybe start earlier
- Consider how can be installed on parents phone / home tablet due to parents reluctance for children to have their own
- Difficulty remembering portal URL
- Login – so many logins how to integrate with standard NHS
- Instructions on correct way to upload to EMIS (and other systems)
- Doing the review and entry in 10mins was a challenge
- Create peer groups in schools and secondary school – group consultations and care plan development would be effective route to adoption
- Contact of 16 year olds is a challenge as telephone numbers are often still parents . Chrisp Street is reregistering 16 year olds contact details
- Workforce issues – need to train staff to book the correct length of appointments and inform clinicians and patients about use of the app.
- Ability to train to practice nurses being able to do reviews as well – they need to gain confidence and training.
- Look at Ready, Steady, Go transition programme as transition model in development
- Agreed that patient entered data in general in PHR was a positive move
- Need to access portal in different settings and various clinicians – eg school and surgery
Charlotte Carrick (RLH Childrens’ Respiratory Nurse)
- Sign up somewhat slow as working in general respiratory clinic
- Best results from screening appointments and giving advance notice of use of app
- Six sign ups since the start
- One really enthusiastic response and no negative feedback
- Feeling that 12/13 year olds may be an engaged age range going forward
- Comments back from patients about concerns around storage space on their phone and data plans
- Noted that 4 patients did not have phones
- Some have tables with no mobile number
- Felt that after a few uses patients can be on board and care plan produced in 10 min appointment
- Too much free text input time consuming
- Add drop down menus for standard medication, colours of inhalers
- Peak flows as number and then calculate 80/50 reduction amounts in plan
- Charlotte has done one review on care plan and made changes on the portal – all worked well
- Looks and feels better and is more simple to enter the required information than old self populating printed pdf plans
- Product development: provide alerts for parents/carers on peak flow reduction and when a care plan needs to be reviewed.
- The app not for everyone as not suitable for people with very poor condition control / management
Tori Hadaway (Community Children’s Asthma Nurse)
- Difficulty finding suitable patients as most are U12
- Number of Do Not Attend high for reviews (60%)
- Need 30mins to undertake review
- GP’s and School Nurses should be doing majority of CYP plans in school on a yearly basis
- Challenges may be not allowing phones in schools, lack of wifi, data and having a smart phone at that age
- Secondary schools may be better – lower sixth may be a engaged age group
- Encourage adoption needs proactive approach and associated resource and workflow (eg leaflets, patients contact, follow up etc)
- Keen to explore links within adoption is schools in Tower Hamlets
Anne-Marie Casey (Bart’s Adult Asthma Nurse)
- “This app is just brilliant and could save a life”
- Great example of joined up care
- Potential to extended to Royal London Asthma wards
- Can add additional features to health tracker
- Additional leaflets / instructions would be good
Jane Simpson (Bart’s Adult Asthma Nurse)
- “Really useful when patients come back into the clinic, recording peak flow rather than paper is a real positive. The links are really good – one patient this morning had poor inhaler technique and having additional information that they access in one place is really useful”
- Allergies – there is a double entry between emergency and care plans and medical record – can this be integrated to reduce error possibility
- Graph of peak flows on app or portal for clinician and patient to review together
- Time to use the system a concern especially when not used to it
- Inhaler Techniques videos really useful – reduce time in clinic as can refer to them
- Roll back does portal save old plans as useful to refer back in same format
- Possibility to use pre-discharge as inpatient to bridge transition and record peak flows prior to first asthma clinic appointment
- Issues around Do Not Attend patients reducing number using the app
- Can app prompt to bring inhalers to asthma review and give a reminder to book a review
- Medication tracking – based on care plan can work out when medication may be running low and alert
Dr Jim Cole (GP)
- “The app is brilliant, really impressive”
- Lack of interoperability with EMIS
- Difficulty remembering portal URL
- JC cut and pasted from EMIS to emergency plan (medications, allergies, contact details)
- NOTE: Patient was moved manually in database from Barts to Chrisp Street highlighting interoperability issues
Rachel McCready (Practice Nurse, Chrisp Street)
- Currently not used as lack of suitable patients
- Difficulty remembering portal URL
- Tech issues are embarrassing if you can’t remember so don’t try
- Video and instruction sheet would be useful (note – instructions previously provided, but not all staff aware – need better system)
- Accurix – text messaging sits on EMIS as widget – similar would be a reminder and easy link
- Switching between patients of differing conditions – easy to forget to use a new system
- Appointment system – Add a comment to remind clinician to use DHP
- Text message patient to download app before appointment
- Need upload to EMIS (and other systems)
Appendix 2 – Patient feedback
“Easy to log onto and good to know what level my asthma is at – knowing if I need to wear a coat or hat is really good.” Alice, DHP User
“We are always trying to encourage Alice to look after her asthma herself and it helps her be independent…..really great as she has been discharged from hospital and the app helps her monitor her asthma and she lets us know how she is getting on” Susan, Alice’s Mum
“I really liked it because it was simple to use” Jared, DHP User
“The look and feel is really good – not just a boring NHS App and the interactive background is especially good for younger users” Lucy, DHP User
“As an asthmatic in the last two years I’ve been to A&E twice and if I’m having trouble breathing I can take my phone out and say look at this plan. That would be really handy. Saira, DHP User
“It’s really cool” Robert, DHP User
Summary
- Useful and usable insights gained
- Easy to use and no negative comments about design or functionality
- Most people use some features
- Small proportion of users use intensively
- Many additional feature requests and ideas to further improve the design
Survey comments
- Works perfectly, health plan really beneficial
- Loved the colour scheme and bright and bold interactive features
- Cool to have sharing emergency plan and health tracker with doctor
- Useful for the record and to share the record, ambulance to look at plan etc
- Forget to use it, but feels good, looks good
- Push notifications (would be good) to take inhaler and fill out diary
- Some more features and dropdown boxes
- Not complicated
Workshop feedback
A co-design workshop was held with 2 CYP and one parent
- On-boarding easy via text message – no problems
- Like the ‘magic links’ – no password is good
- Like the bright colours and the timeline – it could be customised
- Easy to navigate and use the app
- Could have a character that ‘gets weaker’ if you don’t take medications
- Need to make sure it doesn’t use too much memory
- Hadn’t found the air quality alerts – notifications would be good
- Different graph views with effects would be good
- Medication reminders would be good
- More condition care plans – eg for eczema
- A tutorial on the app about it’s features would be good
- Should have information about how to contact the team
- Could have peer groups to share information with
- Could get ‘badges’ for completing tasks or watching videos
- It would be good to be able to book/cancel appointments and get reminders
- Could have themes from video games that get unlocked with points
- Health tracker has about the right number of questions (not everyone thinks this) but not clear how often you should use it
- Health tracker could record healthy activities
- Could see more of the week on the timeline or better way to display older information (graphs and charts)
“It’s really cool” – Robert, age 12