AI vs. Crypto: Insights from 2 Million Hacker News Comments

The recent surge in artificial intelligence (AI) has drawn comparisons to the crypto boom of previous years, with many observers noting the similarities between the two phenomena. While crypto appears to have experienced a significant downturn, the question remains: Is AI headed for a similar fate? To explore this, we analyzed 2 million comments from Hacker News (HN), a platform known for its tech-savvy and influential user base.

Gathering the Data

To conduct this analysis, I utilized the HN API to extract all 38 million posts and comments in HN history. Using Polars, a data manipulation library, I filtered the dataset to include only front-page stories since 2010 that had at least 30 points and 5 comments:

stories = hn.filter(
    (pl.col("type") == "story")
    & (pl.col("text").is_null())
    & (pl.col("url").is_not_null())
    & (pl.col("score") >= 30)
    & (pl.col("descendants") >= 5)
    & (pl.col("time") >= datetime(2010, 1, 1))
)
# -> 285,603 stories

This narrowed the dataset to 285,603 relevant stories. However, categorizing these posts into AI and crypto discussions using simple keyword searches would have resulted in numerous false negatives. To address this, I employed a more sophisticated approach.

Throwing GPT-3.5 at the Problem

Instead of relying on basic string matching, I used GPT-3.5 to classify the stories. This method allowed for a more nuanced and accurate categorization of content related to AI and crypto. For instance, posts like "Technical Analysis of the Poly Network Hack" and "Kimchi: The latest update to Mina’s proof system," which might not have been captured by simple keyword searches, were correctly identified.

Analyzing the Trends

The analysis revealed several key insights:

1. Volume of Discussions: AI discussions on HN have seen a significant increase over the past few years, paralleling the rise in crypto conversations from 2017 to 2021. However, while crypto discussions peaked and then sharply declined, AI conversations continue to grow.

2. Sentiment Analysis: The sentiment towards AI is generally more positive compared to the initial hype around crypto. Early crypto discussions were often characterized by optimism but quickly turned to skepticism and caution as the market experienced volatility and regulatory challenges.

3. Technical Depth: HN users tend to engage in more technically detailed discussions about AI, reflecting a deeper understanding and interest in the underlying technology. Crypto discussions, while also technical, have been more focused on investment opportunities and market dynamics.

Why it Matters

The insights from this analysis provide valuable context for investors and tech professionals:

• Investment Strategies: The sustained interest and positive sentiment towards AI suggest that it may offer more stable long-term investment opportunities compared to the volatile nature of crypto.
• Regulatory Environment: The cautious approach to crypto, driven by regulatory concerns, highlights the importance of navigating legal frameworks in emerging technologies. AI, while still facing regulatory challenges, has not yet experienced the same level of scrutiny.

Key Risks

1. Market Saturation: As more companies and startups enter the AI space, there is a risk of market saturation and diminishing returns.
2. Ethical Concerns: The ethical implications of AI, such as bias and privacy issues, could lead to increased regulatory oversight and public backlash.
3. Technological Hype: Similar to crypto, there is a risk that excessive hype around AI could lead to overvaluation and eventual market corrections.

The Opportunity

Despite the risks, the growing interest in AI presents several opportunities:

• Innovation: Continued investment in AI research and development can drive significant technological advancements.
• Diversification: For investors, diversifying into AI-related assets can provide a hedge against traditional market volatility.
• Talent Attraction: The positive sentiment around AI can attract top talent to the industry, fostering further innovation and growth.