Can ChatGPT Identify Crypto Trading Signals in Social Media Posts?
ChatGPT, a language model developed by OpenAI, has demonstrated a robust ability to classify text across various domains. Leveraging its extensive training on diverse datasets, it can accurately identify and categorize text based on sentiment or topic. This capability extends to applications such as sentiment analysis, where it can discern positive, negative, or neutral tones in social media posts. However, the extent to which ChatGPT can identify trading signals from crypto social media posts is mostly unexplored.
At Coinfident, we tried to address this gap by providing answers to the following two questions:
1- Is chatGPT capable of distinguishing crypto-related tweets from irrelevant ones?
2- Is ChatGPT capable of identifying “buy” and “not-buy” signals from crypto-related tweets?
1- Data Collection
First, we used the Twitter Academic API to gather tweets containing the term “crypto” and the top 200 cryptocurrencies. This data collection occurred between January 7, 2022, and January 14, 2022, and again between March 1, 2022, and March 7, 2022, resulting in 11.83 million tweets from 2.1 million unique users. Second, we identified all legitimate news outlets with finance or cryptocurrency sections that are active on Twitter, totalling 74 accounts. We then collected all tweets from these 74 accounts posted between January 1, 2023, and September 16, 2023, amounting to 122,837 tweets.
2- Excluding Irrelevant Content
To filter out irrelevant tweets from our dataset, we trained and compared two Random Forests classifiers, two LLMs (BERTweet & GPT-4), a proprietary GPT-4 agent (CoinGPT), and a proprietary fine-tuned BERT model (CoinBERT) on 5000 expert-annotated tweets. Tweets were deemed “relevant” if they were directly related to cryptocurrencies or economic or policy topics affecting the cryptocurrency market, such as US interest rates and cryptocurrency regulations.
Among the six classification models, our proprietary CoinGPT agent attained the highest accuracy of 0.94 on a 20% unseen out-of-sample test dataset. Similarly, our CoinBERT model demonstrated comparable accuracy, with only a 2% reduction in performance.
Note: Today (May 14, 2024), we replicated the above procedure with GPT-4o and it yielded 1.3% more accuracy than GPT-4 and was almost 2 times faster. However, it slightly falls behind our fine-tuned GPT-4 agent.
3- Trading Signal Detection
We evaluated the performance of our CoinGPT model against several machine learning algorithms, including Random Forests, Logistic Regression, SVM, and XGBoost. The feature engineering process for these AI models incorporated content features such as top unigrams, top bigrams, top hashtags, top mentions, the number of mentions, the number of emojis, and the number of hashtags.
Building on previous research highlighting the capabilities of ChatGPT and open-source LLMs in text annotation tasks, we also included GPT-4, GPT-3.5, LlaMa-2 (7b), LlaMa-2 (70b), FLAN-T5 (L), and FLAN-T5 (XL) in our comparison. We randomly selected and labeled 6000 tweets from influencers and 2000 tweets from news outlets’ datasets to categorize them into buy and not-buy signals.
In the following plot, we see that our CoinBert model outperforms all other models for the influencers dataset. The closest performance to that of belongs to GPT-4. However, for the news outlets’ data, GPT-4 and GPT-3.5 are the two best-performing models and slightly outperform CoinGPT. Since CoinGPT’s performance is strong across both datasets, we selected it as our final choice at Coinfident.
Note: Today (May 14, 2024), we replicated the above procedure with GPT-4o and it yielded 2.3% more accuracy than GPT-4 for influencers’ and 3.2% more accuracy for news outlets’ datasets and was almost 2 times faster.
Conclusion
Our recent study highlights ChatGPT’s potential to both detect crypto-related content and crypto trading signals from social media posts. Nevertheless, the findings indicate that achieving greater accuracy requires a fine-tuned model.
Coinfident is a Swiss-based startup providing analytics and security tools for crypto traders. This article is for information purposes only and represents neither investment advice nor an investment analysis or an invitation to buy or sell financial instruments. Specifically, the document does not serve as a substitute for individual investment or other advice.
