Five good ways (and two bad ways) to read large text files with Scala

I’m working on a small project to parse large Apache access log files, with the file this week weighing in at 9.2 GB and 33,444,922 lines. So I gave myself 90 minutes to try a few different ways to write a simple “line count” program in Scala. (Not my final goal, but something I could use to measure file-reading speed without applying my algorithm.)

Some RxJava example source code (Hackers at Cambridge tutorials)

I recently watched the three Hackers at Cambridge Introduction to RxJava videos, and coded along with them. If you’re interested in some example RxJava code, here’s what I typed in.

The first thing you do is create a new Gradle/Java project with these commands:

mkdir MyProject
cd MyProject
gradle init --type java-application

With that project created you can begin creating some Java/RxJava code.

Java BufferedReader, readLine, and a Scala while loop alvin September 14, 2016 - 8:52pm

I generally try to avoid this coding style these days, but, if you want to see how to use a Java BufferedReader and its readLine method in a Scala while loop, here you go:

Scala file reading performance: Line counting algorithms alvin May 7, 2013 - 7:59am

Out of curiosity about Scala’s file-reading performance, I decided to write a “line count” program in Scala. One obvious approach was to count the newline characters in the file:

Java BufferedReader examples

Java file FAQ: Can you share some examples of the Java BufferedReader class?

When it comes to reading character input streams, the Java BufferedReader class is extremely important, and I'll demonstrate this in several different source code examples.

Java file open, read, and write utilities

Java file utilities FAQ: Do you have any Java file utilities you can share?

As I was working on another Java/Swing application this weekend, I ran across my "Java file utilities" class, and I thought I'd share that class here today. It's nothing too major, but it does include Java methods that let you open, read, write, and copy files using Java.

A Java FileReader class example alvin October 28, 2009 - 1:06pm

Java file FAQ: Can you demonstrate how to use the Java FileReader class?

I have a number of examples on this site about how to read a file with Java, but I've never dug into it from the aspect of how and why to use a Java FileReader, so I thought I'd take a few moments and dig into the FileReader class in this article.

The Java FileReader class

The FileReader javadoc provides a nice introduction to the FileReader class:

Java URL and URLConnection example - how to read content from a URL alvin October 25, 2009 - 10:47am

Question: Using Java, how can I open a URL from my program, and then read the content of that URL?