Hugging Face Releases Parquet-Converted Dataset for OCR and PDF Research

Hugging Face has recently made available a new dataset called "PixParse" in the Parquet format. This conversion is significant for researchers and practitioners working with Optical Character Recognition (OCR) and PDF processing, as it offers improved performance and easier data manipulation.

What Changed?

The PixParse dataset, originally available in WebDataset (WDS) format, has been auto-converted to Parquet. Parquet is a columnar storage file format that is highly efficient for read-heavy workloads and large datasets. This conversion means:

• Faster Data Access: Parquet's columnar layout allows for more efficient data retrieval, which can significantly speed up query performance.
• Easier Integration: Parquet files are widely supported by various data processing frameworks like Apache Spark, Dask, and Pandas, making it easier to integrate this dataset into existing workflows.

Dataset Details

The PixParse dataset contains approximately 3.41 million rows of data, with the first 7,000 rows being publicly accessible for preview. The dataset is primarily split into a single training set (train), which also contains about 3.41 million rows.

Data Structure

Each row in the dataset includes several fields:

• __key__: A unique identifier for each record.
• __url__: The URL where the data is sourced from.
• json: A dictionary containing structured data, including text and bounding box coordinates (bbox) for OCR.
• ocr: Contains OCR output, though the specific format is not detailed.
• pdf: Contains PDF data, with no specific format details provided.
• tif: Image data, with dimensions specified in pixels.

Example Record

Here's a sample record from the dataset:

{
  "__key__": "klpb0135",
  "__url__": "hf://datasets/pixparse/idl-wds@e70d43a77ae233778613acf860df8b2d91e0673b/idl-train-00000.tar",
  "json": {
    "pages": [
      {
        "text": [
          "BROWN & WILLIAMSON TOBACCO CORPORATION",
          "RESEARCH & DEVELOPMENT",
          "INTERNAL CORRESPONDENCE",
          "SECRET. Do Not Copy Or Make Notes Of",
          "This Page And Do Not Give Any Of The",
          "TO :",
          "MR. P. H. HARPER",
          "Information Contained In This Document",
          "To Anyone Except In Conformance With",
          "CC:",
          "MR. T. E. SANDEFUR",
          "The Moorgate Secrecy Protocol.",
          "DR. P. L. AULBACH",
          "FROM:",
          "MR. D. S. ROTH",
          "DATE:",
          "JANUARY 13, 1983",
          "SUBJECT: KENT 80/KS TAR REDUCTION - FINLAND/907",
          "Ref S/R 107/82",
          "After evaluations and rejections of tar reduction samples 49B, 50B, and",
          "51B we have reviewed deliveries and designs of several KENT products to",
          "find one which meets the 10.0 mg/cig. (DPM) and 0.7 mg/cig. nicotine",
          "constraints.",
          "We propose the following two trials.",
          "Trial 1",
          "Trial 2",
          "Blend (%) :",
          "MGLF-31",
          "69.9",
          "69.9",
          "Oriental",
          "10.3",
          "10.3",
          "MRT",
          "5.0",
          "0.0",
          "F.C. WTS",
          "14.8",
          "19.8",
          "Final Casing",
          "MGE-651",
          "MGE-651",
          "Final Flavor",
          "MGE-