Five good ways (and two bad ways) to read large text files with Scala

I’m working on a small project to parse large Apache access log files, with the file this week weighing in at 9.2 GB and 33,444,922 lines. So I gave myself 90 minutes to try a few different ways to write a simple “line count” program in Scala. (Not my final goal, but something I could use to measure file-reading speed without applying my algorithm.)

Some RxJava example source code (Hackers at Cambridge tutorials)

I recently watched the three Hackers at Cambridge Introduction to RxJava videos, and coded along with them. If you’re interested in some example RxJava code, here’s what I typed in.

The first thing you do is create a new Gradle/Java project with these commands:

mkdir MyProject
cd MyProject
gradle init --type java-application

With that project created you can begin creating some Java/RxJava code.

Scala file reading performance: Line counting algorithms

Out of curiosity about Scala’s file-reading performance, I decided to write a “line count” program in Scala. One obvious approach was to count the newline characters in the file:

Java BufferedReader examples

Java file FAQ: Can you share some examples of the Java BufferedReader class?

When it comes to reading character input streams, the Java BufferedReader class is extremely important, and I'll demonstrate this in several different source code examples.

Java file open, read, and write utilities alvin February 1, 2010 - 8:13am

Java file utilities FAQ: Do you have any Java file utilities you can share?

As I was working on another Java/Swing application this weekend, I ran across my "Java file utilities" class, and I thought I'd share that class here today. It's nothing too major, but it does include Java methods that let you open, read, write, and copy files using Java.

Java “file open” and “file read” methods (examples) alvin February 1, 2010 - 7:53am

Java file reading FAQ: Can you share some off-the-shelf Java methods I can use to open and read files in Java?

Sure. Taken directly from my Java file utilities article (Java file utilities to open, read, write, and copy files), here is the source code for two Java methods that let you read a text file.

A Java FileReader class example

Java file FAQ: Can you demonstrate how to use the Java FileReader class?

I have a number of examples on this site about how to read a file with Java, but I've never dug into it from the aspect of how and why to use a Java FileReader, so I thought I'd take a few moments and dig into the FileReader class in this article.

The Java FileReader class

The FileReader javadoc provides a nice introduction to the FileReader class: