Fine-Tuning a Reward Model with RLHF to Predict Hacker News Upvotes for $4.80 of GPU Time

In the world of natural language processing (NLP), reinforcement learning from human feedback (RLHF) has emerged as a powerful technique for enhancing model performance, especially in generating high-quality content. In this article, we’ll dive into how to build a reward model that predicts upvote counts for Hacker News (HN) stories using RLHF, all for just $4.80 of GPU time.

Background

Hacker News is a community-driven platform where users submit and vote on tech-related news articles. The front page is highly competitive, with only the most engaging and relevant stories making it to the top. However, not every deserving story gets the attention it deserves. This is where our fine-tuned model comes in. Using RLHF, we can identify stories that are likely to do well on HN, even if they haven’t received any upvotes yet.

What Is RLHF?

Reinforcement learning (RL) is a subset of machine learning where an agent learns to make decisions by performing actions in an environment and receiving rewards or penalties based on those actions. Over time, the model adjusts its behavior to maximize rewards and minimize penalties.

Reinforcement learning from human feedback (RLHF) takes this concept further by incorporating human preferences into the training process. The key steps are:

• Reward Model Development: Train a model to predict how "good" an output is based on human ratings.
• Human Feedback Collection: Collect data where humans rate or compare outputs.
• Model Training: Use the reward model and human feedback to fine-tune the agent.

Building the Reward Model

To build our reward model, we need to:

1. Collect Data: Gather a dataset of HN stories with their corresponding upvote counts.
2. Preprocess Data: Clean and format the data for training.
3. Train the Reward Model: Use this data to train a model that predicts upvote counts.

Step 1: Collect Data

We collected a dataset of HN stories, including those that didn’t make it to the front page or receive any upvotes. This dataset is crucial because it helps us understand what makes a good story even before it gets any traction.

Step 2: Preprocess Data

• Text Cleaning: Remove HTML tags, special characters, and normalize text.
• Feature Extraction: Extract relevant features such as title length, submission time, and domain.
• Labeling: Use the upvote count as the label for training the reward model.

Step 3: Train the Reward Model

We used a transformer-based model (e.g., BERT) to predict upvote counts. The model was fine-tuned on our dataset using the Hugging Face Trainer API. Key hyperparameters included:

• Learning Rate: 5e-5
• Batch Size: 32
• Number of Epochs: 3

The training process took approximately 10 minutes on a single GPU, costing around $4.80.

Evaluating the Model

To evaluate our model, we used a holdout set of HN stories that were not part of the training data. The model’s performance was measured using mean absolute error (MAE) and root mean squared error (RMSE).

• MAE: 2.5
• RMSE: 3.8

These metrics indicate that our model can predict upvote counts with reasonable accuracy.

Real-World Examples

Let’s look at some examples of stories identified by our model as likely to do well on HN:

• Ask HN: I'm being ousted as CEO of my SaaS company by power-grabbing co-founders
• WhatsApp just got blocked in Iran
• We have recorded the sound of a single electron clapping

None of these stories made it to the front page or received any upvotes, but they were all identified by our model as high-potential stories. Subjectively, I agree with the model; these stories deserve more attention.

Future Work

In future posts, we’ll explore how to use this reward model in conjunction with RL to create a model that can write high-value HN stories. The goal is to generate content that not only gets upvotes but also