A Quick Review of Scala’s for-expressions (for-comprehensions)

“The owls are not what they seem.”

The “Log Lady” in Twin Peaks

Goals

The goal of this lesson is to review at a high level how for loops work in Scala. This is necessary because Scala/FP developers take advantage of advanced Scala for loop features.

As an example of what I mean, the goal of the next few lessons is to explain what’s happening in this for loop:

def stackManip: State[Stack, Int] = for {
    _ <- push(3)
    a <- pop
    b <- pop
} yield(b)

If you already understand what’s happening in that code, feel free to skip over these lessons; otherwise, read on.

Introduction

If you’re used to using the for loop in Java, Scala’s for loop can work similar to it, but it can also be used in a different manner.

A very different manner.

And Scala/FP developers use it in this other manner.

A lot.

For instance, some experienced Scala/FP developers say that this is a “simple” use of a for loop in a functional style:

def stackManip: State[Stack, Int] = for {
    _ <- push(3)
    a <- pop
    b <- pop
} yield(b)

If you’re used to traditional for loops in languages like C and Java, whatever is happening in this “loop” can be quite a surprise.

A peek into what’s happening

I can explain a little of what’s happening: This code is working with a stack, and a push function is pushing the number 3 onto a pre-existing stack, and then the next two lines are popping values off of the stack. But even knowing that, a few questions come to mind:

If a push function pushes a number onto a stack, where exactly is that stack?
A push function normally doesn’t return anything meaningful, so how is it being used as a generator in a for loop? And what is that underscore character doing on the left side of the push expression?
Assuming that those pop functions are popping data off of a stack, again I ask, where is that stack?

In this lesson we’ll starting digging into how the Scala for comprehension works. Knowing how it works is a prerequisite for understanding how this stackManip method works, and how for is used in many Scala/FP situations.

New rule: No more “`for` loop”

Hmmm ... if you go back and look at that code again, is it really a “loop”? No, not really. However that code works, all it seems to do is to perform three operations on a stack. This thought leads us to a new rule:

For a variety of reasons — including the fact that this code isn’t really a loop but something called a comprehension — I will no longer use the term “for loop.” From now on, I’ll refer to this construct as a “for comprehension.”

I can explain this name by once again taking a short look back at history and mathematics.

`for` comprehension history

As a quick historical tidbit, the name “comprehension” comes from both Haskell and mathematics, with mathematics preceding Haskell by some large number of years. The book, Haskell, the Craft of Functional Programming, provides this introduction:

“One of the features of a functional language is the list comprehension notation, which has no parallels in other paradigms.”

LYAHFGG states:

“List comprehensions are a way to filter, transform, and combine lists. They’re very similar to the mathematical concept of set comprehensions. Set comprehensions are normally used for building sets out of other sets.”

If you’ve used the Scala for/yield expression at all, you know that it meets this definition. Indeed, you’ll find the for construct described on a page on the official Scala website titled Sequence Comprehensions. There they state:

Comprehensions have the form for (enumerators) yield e, where enumerators refers to a semicolon-separated list of enumerators. An enumerator is either a generator which introduces new variables, or it is a filter.

These definitions lead us to the for comprehension concepts of generators, filters, and definitions.

Generators, filters, and definitions

A Scala for comprehension can contain the following three expressions:

Generators
Filters
Definitions

These are shown in the following source code snippet from the book, Programming in Scala:

for {
    p <- persons             // generator
    n = p.name               // definition
    if (n startsWith "To")   // filter
} yield

Let’s look at short, formal definitions for each of these elements.

Generators

Generators have this general form:

pattern <- expression

In the simple case this results in an assignment, like this:

p <- persons

In this expression the value p iterates over all of the elements contained in persons. Generators can be more complicated than this, but this is their most simple and common use.

There are three more things to know about generators:

Every for comprehension begins with a generator
for comprehensions can have multiple generators
The left side of a generator can also be a pattern:

def getTheSquirrel = for { (dog, cat, squirrel) <- getAnimalsOutOfLaundryRoom } yield squirrel

Definitions

for comprehension definitions have this general form:

pattern = expression

A definition binds the pattern pattern on the left to the value of expression on the right.

In the original example in this lesson:

n = p.name

the variable n is bound to the value p.name. That statement has the same effect as writing this code outside of a for comprehension:

val n = p.name

Filters

for comprehension filters have this general form:

if (expression)

In this code expression must have the type Boolean.

Filters drop all elements from the iteration for which expression returns false, so in this code:

if (n startsWith "To")

any value n that does not start with the string "To" will be dropped during the iteration process.

Stated the other way, any value n that begins with the string "To" will be retained during the iteration process.

An example

Here’s a simple example of each of these features in a Scala for comprehension:

case class Person(firstName: String, lastName: String)

val people = List(
    Person("barney", "rubble"),
    Person("fred", "flintstone")
)

val namesStartingWithB = for {
    p <- people                  // generator
    fname = p.firstName          // definition
    if (fname startsWith "b")    // filter
} yield fname.toUpperCase

If you put that code in the Scala REPL, you can then run this code to see the expected result:

scala> namesStartingWithB.foreach(println)
BARNEY

Summary

What you started to see in this lesson is that Scala for loops are much more than “loops.” They are based on mathematical set comprehensions, and they are a way to “filter, transform, and combine lists.”

You also saw that a Scala for comprehension can contain the following three types of expressions:

Generators
Filters
Definitions

What’s next

This lesson provided a quick review of the basic features of for comprehensions as background for the next lesson, where you’ll learn how to write your own custom “collection” class to work in a for comprehension.

As a reminder, the end goal of these lessons is to help you understand how for comprehensions like this work:

def stackManip: State[Stack, Int] = for {
    _ <- push(3)
    a <- pop
    b <- pop
} yield(b)