Multimodal financial sentiment for stock return prediction

Abstract

This paper proposes a novel multimodal deep learning framework for stock return prediction that integrates heterogeneous data sources: technical indicators, market investor sentiment indices, and textual sentiment extracted from earnings conference call transcripts. The proposed model employs a hybrid architecture combining transformer encoder for the technical modality and neural networks for market and textual modalities. A modality-level attention mechanism is used in a late fusion setup to dynamically weight the contributions of each modality. We evaluate our model on a large-scale dataset comprising 24,821 samples from 497 S&P 500 companies over the period 2010–2022. The results show that our model outperforms traditional models (LSTM, BiL-STM, CNN-LSTM) and alternative fusion strategies, achieving a directional accuracy of 59.94% on the test set. Attention weight analysis confirms that all three modalities contribute meaningfully to prediction performance. These results demonstrate the overall effectiveness of the proposed framework in accurately predicting abnormal stock returns in a multimodal setting.