Week 2 with awkreader: Optimization, Versatility & Breaking into the PC Universe

awkreader
GSoC
RStats
Optimization
Author

Akshat Maurya

Published

May 27, 2026

Welcome back to my journey building awkreader! After establishing the foundational test suite and CI pipelines last week, I moved on to the next phase: expanding the package’s versatility and driving down execution overhead.

This week, I tackled major objectives that directly address generalized parsing, massive performance refactoring, and critical space-handling bugs.

1. Breaking Free from CSVs: The delim Parameter

Until now, the package was hardcoded to expect comma-separated values. To make awkreader a truly generalized tool for data science, I introduced a new delim parameter.

This allows users to specify custom separators (like \t for TSVs or | for pipe-separated files). To ensure this was integrated correctly, I wrote unit tests using .psv (pipe-separated values) data to verify that the AWK generation and the downstream data.table::fread() call remain perfectly synced. Now, awkreader can handle a variety of structured text data with ease!

2. The Need for Speed: Header Optimization

The next crucial step was optimizing the package’s performance. I analyzed how the package extracts the first row (the header) of a file to understand its structure, and I found a prime target for refactoring.

The Old Way

Previously, the package used a high-overhead approach:

header.statement <- sprintf("header.dt <- fread(input = '%s', nrows = 1)", the.files[1])
eval(expr = parse(text = header.statement))

The Problem

This approach invoked data.table::fread. While fread is a world-class parser for loading data, using it merely to peek at a single header line is massive overkill.

  • It forces the system to open a connection
  • Detect file formats
  • Parse the header into a data.table object
  • Register metadata in R’s memory All for one line of text.

Furthermore, the use of eval(parse(text = …)) is not only slow but also poses a significant security risk by blindly executing strings.

The “New Way” (Lightweight & Native)

I refactored the logic to decouple header detection from the data-loading engine:

first_file_con <- file(the.files[1], "r")
header_line <- readLines(first_file_con, n = 1)
close(first_file_con)

Why this is better

  • Minimal I/O Overhead: By using base R’s native file connection, we bypass the entire initialization of the data.table engine. It fetches raw text instantly.
  • Security & Stability: By eliminating eval(parse()), we have made the package safer and more predictable.

3. Proving It: The Microbenchmark Test

In engineering, theory is good, but data is better. To validate the optimization, I used the microbenchmark package to compare the old and new methods across 10,000 iterations.

Method Min (µs) Mean (µs) Median (µs) Max (µs)
Old Approach 475.855 631.156 497.298 9691.751
New Approach 13.878 41.149 17.001 2162.533

The Verdict

The results are outstanding! The new base R method is, on average, over 15 times faster (dropping from ~631 µs to ~41 µs). Looking at the median time, it is a staggering 29 times faster.

4. Introducing the record.count Function

After optimizing the header extraction, we began building out a standalone record.count() function, mirroring the behavior found in grepreaper package.

I started implementing this by utilizing the existing input and filter logic from our filtered.fread function.

However, aggregating record counts across a vector of multiple files natively in AWK presented a unique challenge:

How do we track and output individual file counts sequentially without leaking memory or mixing up streams?

To solve this, I designed an elegant AWK loop state machine using three built-in variables:

  • FNR → Current file row count
  • NR → Global row count
  • FILENAME → Current file name
awk -F ',' '
  # 1. Catch File Transitions: Print the previous file summary before resetting
  FNR==1 && NR>1 { print prev_file, count; count=0 }

  # 2. Track Current File Context: Save filename and skip the header row
  FNR==1 { prev_file=FILENAME; next }

  # 3. Dynamic Evaluation: Increment match counter if row matches filter criteria
  { if($1 < $2 && $3 == "4") {count++} }

  # 4. Final Flush: Dump the last remaining file state from memory at EOF
  END { if(prev_file) print prev_file, count }
' 'Data/ratings data/file_1.csv' 'Data/ratings data/file_10.csv'

To verify that this baseline logic integrated smoothly into the R wrapper, I ran a quick initial test query:

record.count(
  the.files = the.files,
  the.filter = "user > item & rating == 4",
  return.as = "all",
  include.filename = FALSE
)

The Unexpected Error

R immediately threw a rather confusing error:

Error in setnames(x, value) :
  Can't assign 1 names to a 3-column data.table

The Problem: The Space-Separation Trap

I dug into the code and realized the issue stemmed from our mock dataset’s file path:

Data/ratings data/file_1.csv

Notice the space in "ratings data".

By default, AWK’s print statement separates output variables using spaces.

Because of this, AWK was streaming output like:

Data/ratings data/file_1.csv 42

When data.table::fread() parsed this line, it interpreted the spaces as column separators and incorrectly split the data into three columns:

Column 1 Column 2 Column 3
Data/ratings data/file_1.csv 42

Because include.filename = FALSE, the downstream code executed:

names(batch.data) <- "count"

R then attempted to assign a single column name onto a 3-column data.table, causing the crash.

The Solution: Controlling AWK’s Output Separator

To fix this, I had to ensure AWK and fread() were speaking the exact same language, regardless of spaces in file paths.

Step 1: Update the AWK Command Templates

I added -v OFS="," right after the input delimiter flag.

OFS stands for Output Field Separator.

By setting it to a comma, AWK now streams output as proper comma-separated values, preserving the file path exactly as intended.

if (use.windows) {
  string.placeholder <- '"%s"'
  statement.to.fill <- '%s -F "%s" -v OFS="," "FNR==1 && NR>1 {print prev_file, count} FNR==1 {prev_file=FILENAME; count=0; next} { %s } END {if(prev_file) print prev_file, count}" %s'
} else {
  string.placeholder <- "'%s'"
  statement.to.fill <- "%s -F '%s' -v OFS=',' 'FNR==1 && NR>1 {print prev_file, count} FNR==1 {prev_file=FILENAME; count=0; next} { %s } END {if(prev_file) print prev_file, count}' %s"
}

Step 2: Update the fread() and Data Cleaning Blocks

Next, I updated the fread() call to explicitly expect comma-separated values using sep = ",".

I also cleaned up the downstream column renaming logic so it properly handles both the filename and count columns before filtering.

if (return.as != "code") {
  if (show.warnings) {
    batch.data <- fread(
      cmd = awk.statements[i],
      fill = TRUE,
      nrows = nrows,
      header = FALSE,
      sep = ","
    )
  } else {
    suppressWarnings(
      batch.data <- fread(
        cmd = awk.statements[i],
        fill = TRUE,
        nrows = nrows,
        header = FALSE,
        sep = ","
      )
    )
  }

  if (nrow(batch.data) > 0) {
    names(batch.data) <- c(file.header, "count")

    if (!include.filename) {
      batch.data[, (file.header) := NULL]
    }
  }

  list.data[[i]] <- batch.data
}

This was a surprisingly interesting debugging session because the issue wasn’t in R or AWK individually…it was in how both tools interpreted streamed text differently.

The Result

The execution pipeline is running flawlessly now! Here is a peek at the clean, ready outputs:

Running Without Filenames

record.count(
  the.files = the.files,
  the.filter = "user > item & rating == 4",
  return.as = "all",
  include.filename = FALSE
)
$result
   count
   <int>
1:     6
2:     7

Running With Filenames & Inspecting Generated AWK

record.count(
  the.files = the.files,
  the.filter = "rating == 3 | rating == 4",
  include.filename = TRUE,
  return.as = "all"
)
$result
                            file count
                          <char> <int>
1:  Data/ratings data/file_1.csv     6
2: Data/ratings data/file_10.csv     7

$code
[1] "awk -F ',' -v OFS=',' 'FNR==1 && NR>1 {print prev_file, count; count=0} FNR==1 {prev_file=FILENAME; next} {if($1 > $2 && $3 == \"4\") {count++}}  END {if(prev_file) print prev_file, count}' 'Data/ratings data/file_1.csv' 'Data/ratings data/file_10.csv'"

5. The First Step Toward Full PC Compatibility

The baseline test suite passed on Windows on our very first try! This proves that our core runner environment is viable, whenever the CI framework configures R on Windows, it sets up Rtools, which exposes a native port of GNU AWK (gawk.exe) on the system path.

While this confirms our background execution pipeline works across all three major operating systems, the real work begins now. Passing baseline tests is a great start, but we still need to adapt our string-generation and translation engine to handle Windows-specific quirks like double-quote escaping and cross-platform line endings (\r\n).

But knowing the underlying pipeline is stable gives us a rock-solid foundation to build on.