Exploring Leap-of-Thought in LLMs with Creative Humor Generation: CLoT and Oogiri

Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but one area that remains challenging is creative humor generation. A recent paper by Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou explores the Leap-of-Thought (LoT) ability of LLMs using a traditional Japanese comedy game called Oogiri. The paper, titled "Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation," was accepted to CVPR 2024.

What Changed Technically and Why It Matters

The key innovation is the introduction of CLoT (Creative Leap-of-Thought), a model that leverages Oogiri to test and enhance LLMs' creative capabilities. Oogiri, or 大喜利, is a game where players respond to multimodal prompts with humorously unexpected answers. This setup challenges LLMs to think outside the box, which is crucial for tasks requiring creativity and innovation.

• Why It Matters:
  • Creative Thinking: Traditional NLP tasks often focus on factual or logical responses. CLoT pushes models to generate creative, humorous content that requires a leap in reasoning.
  • Multimodal Understanding: Oogiri involves both text and image inputs, making it an excellent testbed for multimodal LLMs.
  • Cultural Sensitivity: Humor is highly subjective and culturally dependent. CLoT's performance can help researchers understand how well models adapt to different cultural contexts.

Technical Details

• Dataset:

  • The Oogiri dataset consists of image-text pairs with corresponding humorous responses.
  • The dataset is available on Hugging Face at zhongshsh/CLoT-Oogiri-GO.

• Model Architecture:

  • CLoT builds upon existing LLMs, fine-tuning them on the Oogiri dataset to enhance their creative capabilities.
  • The model uses a multimodal encoder-decoder architecture, where the encoder processes both text and image inputs, and the decoder generates the humorous response.

• Training:
  • The training process involves a combination of supervised learning (using labeled data) and reinforcement learning (to optimize for creativity and humor).
  • The model is trained to maximize the likelihood of generating responses that are both funny and contextually appropriate.

Examples of CLoT in Action

Here are a few examples from the paper, demonstrating how CLoT generates creative humor:

• Example 1:

  • Image: A man blocking a seat on a bus.
  • Prompt: 不好意思，这里有人了。
    • Translation: Sorry, this spot's taken.
  • Response: This image is from Forrest Gump. The man in the image has his hands open, as if blocking the seat to prevent someone from sitting down. At the same time, he turns his head to the left, seeming to say to the person on the left who wants to sit in the chair, "Sorry, this spot's taken."

• Example 2:

  • Image: A red piglet cooking pancakes.
  • Prompt: 煎饼果子要趁热吃，否则会变成小猪。
    • Translation: Enjoy the pancake while it's hot, or it'll turn into a piglet.
  • Response: "Enjoy the pancake while it's hot, or it'll turn into a piglet" is said by the red piglet cooking pancakes. This suggests that the nearby blue piglet is transformed from the pancake which hasn't been eaten in time. CLoT creatively connects the pancake and the blue piglet to craft this unexpectedly funny response.

• Example 3:

  • Image: Spider-Man on a glass window.
  • Prompt: 刚擦的玻璃不能弄脏。
    • Translation: The glass I just cleaned shouldn't get dirty again.
  • Response: Spider-Man crawls carefully on the glass, not even daring to touch it with his palms. He's worried that he