| hide |
|
|---|
The cross-functional team composition and involvement of specific individuals should depend on the scope, topic and the current stage of the project. The team is not a formal review committee, but one that ensures each stage of development is considered by an appropriate expert. Over the lifetime of the project, the team should, at some point, include ML researchers, clinicians, data analysts, statisticians, equity and ethics experts, patient representatives, clinical champions and end-users, as well as ethics, privacy and strategy representatives.
Patient voices, represented by patient partners, family and caregivers, and/or professional patient advocates, should be present within and beyond the cross-functional team. Patient partnerships are critical to make sure that patients are guiding the research in identifying unmet needs and improving outcomes for those needs. Patient voices can be present in the whole spectrum of AI research, from problem identification and study design through to clinical implementation and lifecycle monitoring.
Many institutions and disease specific advocacy groups will have their own supports for patient partners, so it is important to look for local initiatives. In Canada, the Canadian Institutes of Health Research has a framework for patient-oriented research and engagement (https://cihr-irsc.gc.ca/e/48413.html) - similar initiatives may exist in other countries.
The allure of AI in healthcare often creates a "shiny object syndrome"18 where research teams and organizations rush to implement cutting-edge technology simply because it's novel and exciting, rather than because it's the most effective solution. An AI-first mindset can lead to overlooking simpler, proven interventions that might better serve patients and healthcare workers. When enthusiasm for AI's potential overshadows careful consideration of alternatives, organizations risk investing substantial resources into complex technical solutions for problems that might be better solved through basic process improvements, better staffing, or enhanced communication systems. It's worth remembering that sometimes what appears to be a technology problem is actually a human systems problem in disguise, and no amount of sophisticated AI can fix underlying issues with workflow, staffing, or resource allocation.
When addressing healthcare challenges, it's crucial to first explore fundamental non-technical and non-AI-solutions that may be more effective and sustainable. These approaches may require fewer resources, face fewer implementation challenges, and can be more easily adapted to local contexts compared to AI-solutions.
A key component of responsible AI development and deployment is mitigating bias, fairness and inequity. In order to do this, there need to be clearly defined equity objectives and fairness metrics to measure success. These definitions will be specific to the population, problem space, and solution that is being addressed by a particular AI-solution. Exploring and understanding baseline inequities is paramount to developing a fair and equitable model, and can be done in a few different ways (please see sections below). Taken together, these learnings lay the foundation for quantitative analysis of biases and inequities within your target population once you have access to local retrospective data to explore in Section 3.2.
The equitable and compassionate components of this framework rely on an extensive review of documented biases in healthcare literature. However, it should be acknowledged that biases identified in the literature are not guaranteed to be exhaustive, nor always fully representative in the collected data or healthcare system, particularly when the evolution of protected attributes over time is considered (e.g. newly adopted gender definitions that are more granular and representative, or new discoveries in a disease definition that increases specificity). Key patient factors associated with healthcare inequity include race19–23, age24–27, sex28–30, geographical location of residence31,32, patient support systems33,34, primary language, income level, and other social determinants of health. Each of these elements independently may impact healthcare access, treatment, and/or outcomes, and often the effects compound when elements are present together, creating unique barriers for those with intersectional identities. Common intersections may include age/sex, and ethnicity/sex. Accounting for intersectional identities and intersections of other health-related groupings remains challenging35; there are far more combinations of intersecting demographic factors than can be practically addressed. Broadening bias assessment to include intersectional identities is essential, but often constrained by data availability.
There will be gaps in the healthcare literature regarding what biases exist in a given problem space. By consulting with individuals who understand the patient experience in the specific problem space, as recommended in our framework, the investigator may begin to fill in these gaps. There are numerous stakeholders that can inform the team’s understanding of these biases, including patients, patient partners, families/caregivers, patient navigators, and healthcare providers. The goal of this consultation process is to identify any biases or inequities, real or perceived, that must be considered and either successfully mitigated or identified as a limitation to the work.
For AI-solutions that are focused on First Nations, Indigenous, and/or Métis populations, or are known to utilize First Nations, Indigenous, or Métis data, special care should be taken to respect the data sovereignty of these groups. If the AI-solution will use data from specific individual nations, the project team must seek approval from appropriate leaders from those nations prior to commencing the project.
To bolster the researcher’s own capacity to respect/assert the principles of data sovereignty, it is recommended that all members of the research team complete The Fundamentals of OCAP® course that was developed by the First Nations Information Governance Centre. OCAP stands for Ownership, Control, Access and Possession, and the course provides valuable information on how to appropriately engage with these communities and their data.
Through this course, participants will learn about the history and motivation behind OCAP, as well as receive practical steps for participating in, or seeking to participate in relevant research studies. As examples, individuals should ask themselves the following questions, among others that are outlined in the course:
- Was this data collected with the approval and knowledge of the First Nations?
- Does the project align with the priorities of the First Nation(s) from whom the data is coming?
- Could undertaking this project/using this data cause harm to the First Nation(s) or its members?
- Are First Nation(s) members being included in the project from conception to analysis to implementation?
Prior to embarking on the lengthy and costly endeavor of AI-solution development, testing and integration ensure that the solution will have a measurable impact on clinical processes. Two important questions to ask are: (1) What will be done differently in the clinic, either operationally or for patients, with the AI-solution outputs; and (2) do pathways exist to act on AI-solution outputs.
The clinical impact of an AI-solution’s output may be affected by a variety of scenarios including the state of the care facilities outside of healthcare (eg. an output recommending discharge of patients to long term care centers which do not have vacancies), available treatments (eg. an output identifying patients who will become septic which has not had advances in treatment in recent decades), and healthcare resources (eg. an output that predicts emergency room visits which are currently at capacity). The ability to address scenarios to improve the impact of AI-solutions ranges in feasibility and control of the research team and should be thoughtfully considered.
Defining the ideal outcome and determining what should be measured to assess that outcome are two separate steps. First, the ideal outcome should be defined based on the ideal clinical state after AI-solution integration. For example, for an AI-solution meant to predict and intervene in post-surgical pain crisis, the ideal clinical state post-deployment might be ‘more patients with appropriately managed pain’. After the ideal clinical state has been defined, the most appropriate outcome measurement(s) can be identified. In this example, measurement may be simple, such as self-reported patient pain or pre- and post-deployment number of analgesic orders per patient.
The challenge lies in making sure that the measurement(s) chosen is/are fair and equally assessable across all affected subpopulations. Patients of colour are often prescribed pain medication at a lower rate than white patients, despite being in equal or higher levels of pain [cite]. When they are prescribed analgesics, patients of colour are also more likely to be given a lower dose of the medication or given a different regiment entirely [cite]. If the AI-solution is making predictions of the likelihood of pain crisis based in part on historic analgesic orders, the prediction will not be appropriately sensitive for patients of colour.
In this example, measuring outcomes such as ‘rate of analgesics ordered across target subpopulations’ or ‘average patient self-reported pain pre- and post-deployment per target subpopulation’ may be more appropriate to capture how the AI-solution is affecting different groups. Without taking care to do measurement by subpopulation, there may still be an overall increase in orders or overall decrease in reported pain after surgery, but those trends may be primarily influenced by the experiences of white patients, and the unequal increase in orders/dosing would lead to an overall inequitable clinical state.
Before moving on to model training and development, it is important to consider the data that is required to allow your model to perform effectively and equitably. This includes determining specific outcome measures, the features that need to be included in the model, and whether they are collected in a format, and for a purpose, that is adequate for your specific problem and population. Most clinicians, researchers, and AI specialists are not particularly familiar with the sources, types, structures, and availability of data, so it is worth doing this investigation up front to avoid significant problems at later stages.
It is also important when identifying data elements and data sources to understand the context in which the data was gathered. How and why the data was collected and the intended purpose/user of the data is important information when assessing potential biases and limitations with the data.