LLMs and Music Taste: Bracketing Artist Preferences

Introduction

The concept of "taste" has become a buzzword, especially in the VC world. While it's often debated whether taste truly matters in business, it's an intriguing topic to explore as we increasingly rely on large language models (LLMs). This weekend, I decided to dive into the question: do LLMs have good music taste? To find out, I designed a bracket-style competition where models had to pick their favorite artists from a list of 5000 top-played musicians. Here’s how it all went down.

The Setup

To avoid biasing the model with pre-existing charts and lists, I opted for a bracket-style approach. This method involves head-to-head matchups between artists, where the model picks its favorite in each round. The winner advances to the next round until a final champion is determined.

Key Steps:

• Data Source: I used the ListenBrainz dataset, which provided the top 5000 most played artists.
• Randomization: The starting matchups were randomly shuffled to ensure fairness.
• Model Requests: Running 5000 artists through 13 rounds required a significant number of API requests. However, since each prompt was short (e.g., "Pick your favorite music artist between {artist_1} and {artist_2}. You have to pick one. Respond with just their name."), the token count remained low, keeping costs manageable.
• Temperature Adjustment: To better capture each model's persona, I increased the temperature setting and ran each matchup as a best-of-three series.

The Results

After processing all the matchups, here are the top 20 artists for each model. Note that while it’s presented as a ranking, it’s more accurately described as a series of eliminations:

Top 20 Artists (Model X)

1. Radiohead
2. The Beatles
3. Pink Floyd
4. Daft Punk
5. Tame Impala
6. Kendrick Lamar
7. Fleetwood Mac
8. Eminem
9. David Bowie
10. Nirvana
11. The Rolling Stones
12. Led Zeppelin
13. Queen
14. U2
15. Coldplay
16. The Weeknd
17. Jimi Hendrix
18. Michael Jackson
19. Elvis Presley
20. Adele

Top 20 Artists (Model Y)

1. Radiohead
2. Pink Floyd
3. The Beatles
4. Daft Punk
5. Tame Impala
6. Kendrick Lamar
7. Fleetwood Mac
8. Eminem
9. David Bowie
10. Nirvana
11. The Rolling Stones
12. Led Zeppelin
13. Queen
14. U2
15. Coldplay
16. The Weeknd
17. Jimi Hendrix
18. Michael Jackson
19. Elvis Presley
20. Adele

Analysis

Key Observations:

• Consistency: Both models showed a high degree of consistency in their top picks, with Radiohead and Pink Floyd consistently ranking at the top.
• Diversity: The lists included a mix of classic rock, pop, hip-hop, and electronic music, suggesting that LLMs can appreciate a wide range of genres.
• Bias: Despite efforts to minimize bias, it’s worth noting that the models’ preferences might still be influenced by the data they were trained on. For instance, the prevalence of classic rock bands like The Beatles and Pink Floyd could reflect the overrepresentation of these artists in the training data.

Implications for Practitioners

This experiment provides valuable insights into how LLMs can be used to generate subjective preferences. While it’s not a definitive measure of taste, it demonstrates the potential of using models for tasks that require personalization and subjective judgment. For developers and researchers:

• **Personalized