Cancer is a formidable adversary, not only due to its status as a leading cause of death but also because of its profound impact on the physical, mental, and social health of those it afflicts. Patients diagnosed with cancer are significantly burdened, not only by the physical challenges of the disease and its treatments but also by the psychological toll it exacts. This dual burden can disrupt their employment and personal relationships, further straining their financial and emotional wellbeing.
Approximately one-third of cancer patients who had a mental health condition before their diagnosis experience a marked increase in distress following their cancer diagnosis. Mental health issues like depression and anxiety not only diminish a patient’s quality of life but are also linked to reduced survival rates. This may be due to such conditions negatively affecting the patient’s adherence to treatment protocols. To address these multifaceted needs, cancer centers have integrated psychosocial oncology within their care frameworks, staffing these centers with psychiatrists and counselors specialized in supporting cancer patients.
Despite these integrations, a significant gap remains in meeting the psychosocial needs of cancer patients, with many continuing to suffer from unmet needs. The reasons for these unmet needs are multifaceted, involving a lack of resources and a failure in the effective detection of these needs, especially in settings that are otherwise well-resourced. Research indicates that treating oncologists often recognize only a fraction of the patients who are severely distressed and consequently fail to refer them to appropriate psychosocial support services. This oversight may stem from oncologists being primarily focused on cancer control, constrained by time, or hindered by cultural and socioeconomic differences that limit their interactions with patients.
The advancement of machine learning (ML) offers promising new pathways for improving the identification of patients who would benefit from psychosocial interventions. ML can train models using structured data such as genetic markers, demographic features, and comorbidities, although the availability of such data varies across institutions, which can limit the application of these technologies. Moreover, structured data usage can be restrictive as it does not always capture nuanced personal details like relationship difficulties.
Addressing this, the use of unstructured data such as the content from initial oncology consultations could provide a richer, more comprehensive dataset for ML models. This approach falls under the purview of Natural Language Processing (NLP), a branch of artificial intelligence. Recent advancements in NLP, particularly the development of neural networks like transformers, have significantly enhanced the ability of these models to understand contextual relationships between words in a document.
Historically, the psychosocial needs of cancer patients were assessed through clinical interviews or questionnaires. However, these methods may not always capture the full scope of a patient’s needs or may fail to engage the patient effectively. Our research leverages NLP to analyze the initial oncology consultation documents to predict which patients are likely to require psychiatric or counseling services within a year of their consultation.
Our study is pioneering in its application of NLP to predict psychosocial needs from oncology consultation documents, a non-psychiatric medical document. Utilizing a dataset comprising over 50,000 cancer patients, we employed both traditional and advanced neural network models to perform this prediction. The performance of our models, indicated by metrics such as Balanced Accuracy (BAC) and Area Under the Receiver Operating Characteristic Curve (AUC), was promising. Our best models achieved BAC over 70% and AUC above 0.75, outperforming simpler models and providing a solid foundation for the practical application of these technologies in clinical settings.
The predictive performance was notably different for referrals to psychiatrists versus counselors, with better accuracy in predicting psychiatric consultations. This could be attributed to various factors, including the nature of the data captured in the consultation documents and the distinct reasons patients might seek these two types of psychosocial support.
Interpretations of the model outputs suggest that the neural networks were able to identify relevant data points within the consultation texts, such as indications of mental illness, details of the cancer diagnosis, and demographic information. These insights could potentially guide further refinements in the models and improve their applicability and effectiveness in clinical practice.
As we continue to develop and refine these NLP applications, future research will focus on validating these models across different healthcare settings and exploring their utility in broader clinical and geographical contexts. This will include adjusting models based on local data characteristics and potentially expanding the scope of predictive analytics to include other types of clinical documents and data sources.
This novel application of NLP in oncology not only underscores the potential of AI in enhancing cancer care but also highlights the critical role of integrating technological innovations into healthcare to address complex challenges like psychosocial oncology. Through continued research and development, we aim to harness these advancements to better identify and address the psychosocial needs of cancer patients, ultimately improving their overall treatment outcomes and quality of life.
reference link : https://www.nature.com/articles/s43856-024-00495-x
