Perfectly Random

Handling Avro files in Python

Fri, 29 Nov 2019 00:00:00 +0000

Apache Avro is a data serialization format. We can store data as .avro files on disk. Avro files are typically used with Spark but Spark is completely independent of Avro. Avro is a row-based format that is suitable for evolving data schemas. One benefit of using Avro is that schema and metadata travels with the data. If you have an .avro file, you have the schema of the data as well. The Apache Avro Specification provides easy-to-read yet detailed information.

Sadly, using Avro files with python is unnecessarily error-prone, especially for a beginner. In this post, we will describe the common errors and their solutions.

Official Avro Packages

Python 2 vs Python 3

Python 2 is end-of-life. You should not be writing Python 2 code. However, the official Avro Getting Started (Python) Guide is written for Python 2 and will fail with Python 3. The problem goes deeper than merely outdated official documentation.

There are two official python packages for handling Avro, one for Python 2 and one for Python 3. The packages have different names, which is unusual for the python ecosystem¹.

Python Version	Package Name	Example
Python 2	`avro`	avro.schema.parse
Python 3	`avro-python3`	avro.schema.Parse

The first problem is that you can actually install avro, which is intended for Python 2, in a Python 3 (virtual) environment. But, when you try to use avro in Python 3, it fails.

# Inside a fresh Python 3 virtual environment
python  --version
# Python 3.7.3

# You can successfully install `avro` in a Python 3 virtualenv even though `avro` is not
# compatible with Python 3.
pip install avro

# Fails when you try to use it!
python -c "import avro.schema"
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/ankur/.virtualenvs/python3-test-env/lib/python3.7/site-packages/avro/schema.py", line 383
#     except Exception, e:
#                     ^
# SyntaxError: invalid syntax

Thankfully, the reverse problem is very unlikely. You should not be able to install avro-python3, which is intended for Python 3, within a Python 2 (virtual) environment, by default.

# Inside a fresh Python 2 virtual environment
python  --version
# Python 2.7.16

# Fails at the installation step itself, thankfully!
pip install avro-python3
# DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
# Collecting avro-python3
#   Downloading https://files.pythonhosted.org/packages/d1/55/4c2e6fecf06cbaa68e0abaf12e1e965969872ed16da3674e6245cab0d5e2/avro-python3-1.9.0.tar.gz
# ERROR: Package 'avro-python3' requires a different Python: 2.7.16 not in '>=3.4'

Even if you install the correct Avro package for your Python environment, the API differs between avro and avro-python3. As an example, for Python 2 (with avro package), you need to use the function avro.schema.parse but for Python 3 (with avro-python3 package), you need to use the function avro.schema.Parse.

While the difference in API does somewhat justify having different package names, this still causes unnecessary confusion. The confusion is exacerbated because the official guide, which still uses Python 2, never mentions that the instructions are only applicable to Python 2.

In the rest of this post, we will only use Python 3 with avro-python3 package because Python 2 is EOL.

Working Example

This is an example usage of avro-python3 in a Python 3 environment.

# Python 3 with `avro-python3` package available
import copy
import json
import avro
from avro.datafile import DataFileWriter, DataFileReader
from avro.io import DatumWriter, DatumReader

# Note that we combined namespace and name to get "full name"
schema = {
    'name': 'avro.example.User',
    'type': 'record',
    'fields': [
        {'name': 'name', 'type': 'string'},
        {'name': 'age', 'type': 'int'}
    ]
}

# Parse the schema so we can use it to write the data
schema_parsed = avro.schema.Parse(json.dumps(schema))

# Write data to an avro file
with open('users.avro', 'wb') as f:
    writer = DataFileWriter(f, DatumWriter(), schema_parsed)
    writer.append({'name': 'Pierre-Simon Laplace', 'age': 77})
    writer.append({'name': 'John von Neumann', 'age': 53})
    writer.close()

# Read data from an avro file
with open('users.avro', 'rb') as f:
    reader = DataFileReader(f, DatumReader())
    metadata = copy.deepcopy(reader.meta)
    schema_from_file = json.loads(metadata['avro.schema'])
    users = [user for user in reader]
    reader.close()

print(f'Schema that we specified:\n {schema}')
print(f'Schema that we parsed:\n {schema_parsed}')
print(f'Schema from users.avro file:\n {schema_from_file}')
print(f'Users:\n {users}')

# Schema that we specified:
#  {'name': 'avro.example.User', 'type': 'record',
#   'fields': [{'name': 'name', 'type': 'string'}, {'name': 'age', 'type': 'int'}]}
# Schema that we parsed:
#  {"type": "record", "name": "User", "namespace": "avro.example",
#   "fields": [{"type": "string", "name": "name"}, {"type": "int", "name": "age"}]}
# Schema from users.avro file:
#  {'type': 'record', 'name': 'User', 'namespace': 'avro.example',
#   'fields': [{'type': 'string', 'name': 'name'}, {'type': 'int', 'name': 'age'}]}
# Users:
#  [{'name': 'Pierre-Simon Laplace', 'age': 77}, {'name': 'John von Neumann', 'age': 53}]

Issue with `name`, `namespace`, and full name

An interesting thing to note is what happens with the name and namespace fields. The schema we specified has the full name of the schema that has both name and namespace combined, i.e., 'name': 'avro.example.User'. However, after parsing with avro.schema.Parse(), the name and namespace are separated into individual fields. Further, when we read back the schema from the users.avro file, we also get the name and namespace separated into individual fields.

Avro specification, for some reason, uses the name field for both the full name and the partial name. In other words, the name field can either contain the full name or only the partial name. Ideally, Avro specification should have kept partial_name, namespace, and full_name as separate fields.

This behind-the-scene separation and in-place modification may cause unexpected errors if your code depends on the exact value of name. One common use case is when you’re handling lots of different schemas and you want to identify/index/search by the schema name.

A best practice to guard against possible name errors is to always parse a dict schema into a avro.schema.RecordSchema using avro.schema.Parse(). This will generate the namespace, fullname, and simple_name (partial name), which you can then use with peace of mind.

print(type(schema_parsed))
# <class 'avro.schema.RecordSchema'>
print(schema_parsed.avro_name.fullname)
# avro.example.User
print(schema_parsed.avro_name.simple_name)
# User
print(schema_parsed.avro_name.namespace)
# avro.example

This problem of name and namespace deepens when we use a third-party package called fastavro, as we will see in the next section.

Third-party Avro Packages

`fastavro`

While avro-python3 is the official Avro package, it appears to be very slow. This is because it is written in pure python. In comparison, fastavro uses C extensions (with regular CPython) making it much faster. Another benefit of using fastavro is that you can install it the same way in both Python 2 and Python 3. fastavro API is also the same² for both Python 2 and 3.

We will use fastavro 0.22.7 for the following discussion. First, let’s use the fastavro.parse_schema(). Unlike avro.schema.Parse(), fastavro.parse_schema() reads in a schema dict and outputs another schema dict.

import fastavro

# Namespace and name are combined to get "full name"
schema_together = {
    'name': 'avro.example.User',
    'type': 'record',
    'fields': [
        {'name': 'name', 'type': 'string'},
        {'name': 'age', 'type': 'int'}
    ]
}

# Namespace and name are separate
schema_separated = {
    'name': 'User',
    'namespace': 'avro.example',
    'type': 'record',
    'fields': [
        {'name': 'name', 'type': 'string'},
        {'name': 'age', 'type': 'int'}
    ]
}

# fastavro.parse_schema() accepts schema as a dict and returns parsed schema as another dict.
# The parsed schema combines name and namespace into "full name".
schema_together_parsed = fastavro.parse_schema(schema_together)
schema_separated_parsed = fastavro.parse_schema(schema_separated)
print(schema_separated_parsed == schema_together_parsed)
# True
print(schema_separated_parsed)
# {'type': 'record', 'name': 'avro.example.User',
# 'fields': [{'name': 'name', 'type': 'string'},
#            {'name': 'age', 'type': 'int'}],
# '__fastavro_parsed': True}

Parsing a schema dict is not really necessary to write data to disk. Parsing the schema provides two benefits:

It helps us verify that the input schema dict is indeed valid
We get back a canonicalized (as per fastavro) schema dict as a result

But, there is one side effect. Parsed schema combines the name and namespace into full name and then stores the full name in the name field. This is the opposite behavior of avro-python3 and just like with avro-python3, this behind-the-scene, in-place modification can cause unexpected errors. Finally, the schema that gets written to disk is whatever schema dict we pass to the fastavro.writer().

# Continued from above

# User data to store.
users = [{'name': 'Pierre-Simon Laplace', 'age': 77},
         {'name': 'John von Neumann', 'age': 53}]

# Experiment 1: Write data using the schema with `name` and `namespace` combined.
with open('users.avro', 'wb') as f:
    fastavro.writer(f, schema_separated_parsed, users)

with open('users.avro', 'rb') as f:
    reader = fastavro.reader(f)
    users_read_back = [user for user in reader]
    metadata = copy.deepcopy(reader.metadata)
    writer_schema = copy.deepcopy(reader.writer_schema)
    schema_from_file = json.loads(metadata['avro.schema'])

print(writer_schema == schema_from_file)
# True
print(schema_from_file)
# {'type': 'record', 'name': 'avro.example.User',
#  'fields': [{'name': 'name', 'type': 'string'}, {'name': 'age', 'type': 'int'}]}
print(writer_schema)
# {'type': 'record', 'name': 'avro.example.User',
#  'fields': [{'name': 'name', 'type': 'string'}, {'name': 'age', 'type': 'int'}]}

fastavro seems to provide two fields that contain the schema – reader.writer_schema and reader.metadata. Metadata (reader.metadata) includes the schema as reader.metadata['avro.schema']. In the above experiment, both these sources of schema provide the exact same schema that has name and namespace combined into a full name. But, this is not always the case, as we will see in the next experiment.

# Continued from above

# Experiment 2: Write data using the schema that has `name` and `namespace` separate.
# Use the unparsed schema that has name and namespace separate
with open('users.avro', 'wb') as f:
    fastavro.writer(f, schema_separated, users)

with open('users.avro', 'rb') as f:
    reader = fastavro.reader(f)
    users_read_back = [user for user in reader]
    metadata = copy.deepcopy(reader.metadata)
    writer_schema = copy.deepcopy(reader.writer_schema)
    schema_from_file = json.loads(metadata['avro.schema'])

print(writer_schema == schema_from_file)
# False
print(schema_from_file)
# {'name': 'User', 'namespace': 'avro.example', 'type': 'record',
#  'fields': [{'name': 'name', 'type': 'string'}, {'name': 'age', 'type': 'int'}]}
print(writer_schema)
# {'type': 'record', 'name': 'avro.example.User',
#  'fields': [{'name': 'name', 'type': 'string'}, {'name': 'age', 'type': 'int'}]}

The above experiment shows that reader.writer_schema and reader.metadata['avro.schema'] differ in whether or not name and namespace are combined together.

In both experiments, reader.metadata['avro.schema'] is more faithful to the schema dict we used to actually write the data. Therefore, it’s a good practice to use reader.metadata['avro.schema'] instead of reader.writer_schema to get the schema.

`avro-python3` vs `fastavro`

As we saw, avro-python3 and fastavro have opposite behaviors when it comes to handling name, namespace, and full name.

Package	Parsing function	Behavior
`avro-python3`	`avro.schema.Parse`	Separates `name` and `namespace`
`fastavro`	`fastavro.parse_schema`	Combines `name` and `namespace`

The handling of name and namespace may not bother you at all or you may find the above difference in behavior unimportant. But, if your code expects either exactly the partial name or exactly the full name, then you may encounter errors. One common error is when you prefix the namespace to an already fully qualified name in the name field. I highly recommend exercising care when handling name and namespace depending on the package you’re using.

Avro <> DataFrame

As we have seen above, Avro format simply requires a schema and a list of records. We don’t need a dataframe to handle Avro files. However, we can write a pandas dataframe into an Avro file or read an Avro file into a pandas dataframe. To begin with, we can always represent a dataframe as a list of records and vice-versa

List of records – pandas.DataFrame.from_records() –> Dataframe
List of records <– pandas.DataFrame.to_dict(orient='records') – Dataframe

Using the two functions above in conjunction with avro-python3 or fastavro, we can read/write dataframes as Avro. The only additional work wewould need to do is to inter-convert between pandas data types and Avro schema types ourselves.

An alternative solution is to use a third-party package called pandavro, which does some of this inter-conversion for us³.

import copy
import json
import pandas as pd
import pandavro as pdx
from avro.datafile import DataFileReader
from avro.io import DatumReader

# Data to be saved
users = [{'name': 'Pierre-Simon Laplace', 'age': 77},
         {'name': 'John von Neumann', 'age': 53}]
users_df = pd.DataFrame.from_records(users)
print(users_df)

# Save the data without any schema
pdx.to_avro('users.avro', users_df)

# Read the data back
users_df_redux = pdx.from_avro('users.avro')
print(type(users_df_redux))
# <class 'pandas.core.frame.DataFrame'>

# Check the schema for "users.avro"
with open('users.avro', 'rb') as f:
    reader = DataFileReader(f, DatumReader())
    metadata = copy.deepcopy(reader.meta)
    schema_from_file = json.loads(metadata['avro.schema'])
    reader.close()
print(schema_from_file)
# {'type': 'record', 'name': 'Root',
#  'fields': [{'name': 'name', 'type': ['null', 'string']},
#             {'name': 'age', 'type': ['null', 'long']}]}

In the above example, we didn’t specify a schema ourselves and pandavro assigned the name = Root to the schema. We can also provide a schema dict to pandavro.to_avro() function, which will preserve the name and namespace faithfully.

Avro with PySpark

Using Avro with PySpark is fraught with a sequence of issues. Let’s see the common issues step-by-step.

Confusing official guide

The official Spark documentation on Avro contains two seemingly contradictory claims. On one hand, the official documentation says

Since Spark 2.4 release, Spark SQL provides built-in support for reading and writing Apache Avro data.

Then, in the next line, it says

The spark-avro module is external and not included in spark-submit or spark-shell by default.

Perhaps, there is sufficient technical difference between the two claims to make them consistent with each other. But, use of the word “built-in” is unnecessarily confusing. I recommend that you disregard the first claim that mentions “built-in” support for Avro. Only the second claim is true – you need to provide the spark-avro package to Spark. You can do this by providing the Maven coordinates in the form groupId:artifactId:version as follows⁴,⁵:

# Example 1
$SPARK_INSTALLATION/bin/pyspark --packages org.apache.spark:spark-avro_2.12:2.4.4

# Example 2
$SPARK_INSTALLATION/bin/pyspark --packages com.databricks:spark-avro_2.11:4.0.0

You can go to the The Central Repository Search Engine or Maven Repository (recommended) to find the versions. If you provide a Maven coordinate that doesn’t exist on Maven, you will get a dependency error.

You may need to clear the cache in $HOME/.ivy2 to overcome some unknown resolver null issues, as mentioned here and here. You can delete $HOME/.ivy2 folder completely to clear the cache but be aware that you will also delete all other downloaded/installed dependencies if you do so.

`spark-avro`: Databricks or Apache

The reason why we show two examples in the above snippet is because there are at least two common instances of the spark-avro package. It appears that the original spark-avro package was written by Databricks and then donated to Apache Spark project. Spark 2.4.0 included support for “built-in” for Avro and updated the spark-avro package to have new functionality and better performance while still retaining backward API compatibility with the older Databricks’ version of spark-avro.

Both versions of spark-avro are available to use. If you’re on Spark 2.4.0 or higher, you should use Apache Spark’s spark-avro⁶. If you’re on Spark 2.4.0 or lower, you need to use the Databricks’ version.

There is still one minor change you need to make to your code to switch between the Databricks’ (older) and Apache Spark’s (newer) versions.

# For Spark 2.4.0 and higher, use Apache Spark's version of spark-avro
df = spark.read.format('avro').load('path/to/avro/data')

# For lower than Spark 2.4.0, use Databricks's version of spark-avro
df = spark.read.format('com.databricks.spark.avro').load('path/to/avro/data')

If you don’t use the correct string in format(), you may see an error like this.

AnalysisException: 'Failed to find data source: avro.
Avro is built-in but external data source module since Spark 2.4.
Please deploy the application as per the deployment section of
"Apache Avro Data Source Guide".;'

Obviously, you also need to provide the corresponding spark-avro to Spark. As we saw, we can simply provide the correct Maven coordinates to the intended spark-avro package. But, there is one more glitch – even if we provide valid Maven coordinates to spark-avro package that installs successfully, we may see an error. Let’s see this in the next section.

Scala version is important

Note the Scala version in the Maven coordinates. In org.apache.spark:spark-avro_2.12:2.4.4, the Scala version is 2.12 and in com.databricks:spark-avro_2.11:4.0.0 the Scala version is 2.11. If you don’t use the correct Scala version, you will find that the spark-avro package installs correctly and the pyspark shell starts successfully but reading Avro data fails.

# Run pyspark shell with Apache Spark's spark-avro package as mentioned in the official docs
$ $SPARK_INSTALLATION/bin/pyspark --packages org.apache.spark:spark-avro_2.12:2.4.4
# Everything successful!

# Inside the resulting pyspark shell
>>> df = spark.read.format("avro").load("users.avro")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ...
py4j.protocol.Py4JJavaError: An error occurred while calling o35.load.
: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister:
Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    ...
# Failure only when you read the Avro data!

Turns out that the pyspark in the above example was built against Scala 2.11, as shown below. But, we provided a spark-avro package that was built for Scala 2.12.

$ $SPARK_INSTALLATION/bin/pyspark --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_221
Branch
Compiled by user  on 2019-08-27T21:21:38Z
Revision
Url
Type --help for more information.

This issue is especially egregious because the user is able to install spark-avro for the wrong Scala version without any indication of error, only to fail at the last moment. Once you know about this issue, it can be easily fixed by simply using the correct spark-avro package for your pyspark’s Scala version.

# Run pyspark shell with the correct Scala version for spark-avro
$ $SPARK_INSTALLATION/bin/pyspark --packages org.apache.spark:spark-avro_2.11:2.4.4

# Inside the resulting pyspark shell
>>> df = spark.read.format("avro").load("users.avro")
>>> df.show()
+--------------------+---+
|                name|age|
+--------------------+---+
|Pierre-Simon Laplace| 77|
|    John von Neumann| 53|
+--------------------+---+

Working example, finally!

For this example, we will use Scala 2.11, Spark 2.4.4, and Apache Spark’s spark-avro 2.4.4 within a pyspark shell⁶.

$ $SPARK_INSTALLATION/bin/pyspark --packages org.apache.spark:spark-avro_2.11:2.4.4

Within the pyspark shell, we can run the following code to write and read Avro.

# Data to store
users = [{'name': 'Pierre-Simon Laplace', 'age': 77},
         {'name': 'John von Neumann', 'age': 53}]

# Create a pyspark dataframe
users_df = spark.createDataFrame(users, 'name STRING, age INT')

# Write to a folder named users
users_df.write.format('avro').mode("overwrite").save('users-folder')

# Read the data back
users_df_redux = spark.read.format('avro').load('./users-folder')

Conclusion

My experience in working with Avro format has been error-prone at every step of the way. The name and namespace ambiguity lies in the Avro specification itself. This is further exacerbated by the contrasting behavior of the two most common Avro packages for python (without spark) – avro-python3 and fastavro. When trying to use the official Avro package for python, the package name and API differences between Python 2 and Python 3 create unnecessary confusion. This makes it difficult to port code over from Python 2. And, even though we should not be writing Python 2 code, the package name and API differences make it difficult to write code that is both Python 2 and Python 3 compatible.

Using Avro with PySpark comes with its own sequence of issues that present themselves unexpectedly. In contrast, using parquet, json, or csv with Spark is so much easier. There is no need to install an external package to use these formats. In that sense, support for parquet, json, or csv is truly built-in.

I wrote this post with the hope that it saves you some time, effort, and frustration. I have tried to list all the issues and solutions that people encounter when using Avro with python. If I missed something or if I made a mistake, please let me know in the comments. Please feel free to share this post with others if they would find it useful.

Footnotes

You can perform pip install pandas for both Python 2 and Python 3. You don’t have to change the name of the package from pandas to pandas-python3 for Python 3. ↩
Based on my basic usage. ↩
pandavro makes some decisions while inferring schema such as making all columns nullable. This may not be what you want. For production systems, consider using a pre-determined, version-controlled Avro schema saved as a .avsc file or using a schema store. ↩
An alternative way to provide a list of packages to Spark is to set the environment variable PYSPARK_SUBMIT_ARGS, as mentioned here. This may be more helpful with Jupyter. ↩
For Java or Scala, you can list spark-avro as a dependency. ↩
For Spark 2.4.0+, using the Databricks’ version of spark-avro creates more problems. One common error is java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. ↩ ↩²

A Guide to Keras Functional API

Mon, 24 Jun 2019 00:00:00 +0000

What is Keras?

Keras means many different things. At the time of writing this article, Keras can refer to one of three things:

Keras, the API specification
keras, the reference implementation, independent of TensorFlow
tf.keras, a particular implementation of the Keras API specification included within TensorFlow

The difference between Keras, the API specification and the Keras implementation (either reference implementation or the TensorFlow implementation) is much like the difference between MPI, the specification and OpenMPI, the particular implementation of MPI.

Keras API specification describes how code can be organized to define and train machine learning models by humans. Keras API specification does not define how the tensor computations are performed at a lower level; that is the job for a deep learning backend such as TensorFlow, Theano, or CNTK.

However, most people understand Keras to be the python package (keras) that can connect to any of the various deep learning backends. keras, the python package is a reference implementation of the Keras API specification. This reference implementation is independent of the TensorFlow project even though TensorFlow is one of the backends used by keras. In that sense, keras, the reference implementation may be considered a wrapper over TensorFlow.

TensorFlow 2.0 adopted Keras API specification as their default, high-level API for building and training deep learning models. TensorFlow comes with another, self-contained implementation of the Keras API specification. This implementation is called tf.keras. tf.keras exists within the tensorflow package and does not require the user to install keras, the reference implementation. Since tf.keras and keras are both implementations of the same Keras API specification, there are lots of similarities between tf.keras and keras. But, tf.keras contains Tensorflow-specific enhancements and may not be exactly the same as keras.

What does this article cover?

In this article, we use the reference implementation, keras 2.2.4 with TensorFlow backend, to describe the Keras API specification, specifically, the Functional API. Most of this article should also apply to tf.keras. This article aims to provide a deeper understanding of Keras’s inner workings. The article is intended for audiences who have previously used or attempted to use a Keras implementation. This article is not an introductory, step-by-step tutorial on how to use a specific Keras implementation.

Sequential and Functional API

Keras API specification provides two APIs for the end user – Sequential API and the Functional API.

	Sequential API	Functional API
Beginner-friendly	Yes
Modeling capability	Directed path graph	Directed acyclic graph
Allows multi-input, multi-output models		Yes

Functional API is much more powerful. Once understood, Functional API is much more intuitive to use. We will discuss only the Functional API in this article.

Keras Objects

Keras API has three fundamental objects, namely:

Keras Tensor, an augmented version of TensorFlow tensor (when using TensorFlow backend)
Layer, an object that performs transformation on Keras Tensors
Model, a specification a neural network, loss functions, optimizers, and other attributes

There are other Keras objects such as Network, Initializer, Optimizer, Callback but the fundamental objects listed above are the only ones important enough to deeply understand Keras. We describe Keras Tensor and Layer in this section and describe Model in the next section.

Keras Tensor

Keras Tensor is commonly generated by the Input function, like this:

from keras.layers import Input
x = Input(batch_shape=(10000, 100))

According to the documentation in the code:

A Keras tensor is a tensor object from the underlying backend (Theano, TensorFlow or CNTK), which we augment with certain attributes that allow us to build a Keras model just by knowing the inputs and outputs of the model.

Keras tensor x has the same type as a TensorFlow tensor (since we used TensorFlow backend), as shown below. However, what makes x a Keras tensor is the existence of Keras-specific attributes such as _keras_history¹.

In [15]: x
Out[15]: <tf.Tensor 'input_1:0' shape=(10000, 100) dtype=float32>

In [16]: type(x)
Out[16]: tensorflow.python.framework.ops.Tensor

In [17]: vars(x)
Out[17]:
{'_op': <tf.Operation 'input_1' type=Placeholder>,
 '_value_index': 0,
 '_dtype': tf.float32,
 '_tf_output': <tensorflow.python.pywrap_tensorflow_internal.TF_Output; proxy of <Swig Object of type 'TF_Output *' at 0x13ff498d0> >,
 '_shape_val': TensorShape([Dimension(10000), Dimension(100)]),
 '_consumers': [],
 '_id': 3,
 '_keras_shape': (10000, 100),
 '_uses_learning_phase': False,
 '_keras_history': (<keras.engine.input_layer.InputLayer at 0x13ff55b38>,
  0,
  0)}

Layer

A Layer defines a transformation. Layer accepts Keras tensor(s) as input, transforms the input(s), and outputs Keras tensor(s). Layers can do wide variety of transformations. Dense, Activation, Reshape, Conv2D, and LSTM are all Layers derived from the abstract Layer class.

A Layer defines a transformation

Typically, we generate a Layer object by calling the default constructor of the corresponding class.

from keras.layers import Dense
dense_layer = Dense(units=10, activation='relu')

In [1]: type(dense_layer)
Out[1]: keras.layers.core.Dense

In the above snippet, dense_layer is an object of the class Dense. Layer objects are callable because they have a __call__ method². The __call__ method accepts a tensor³ or a list/tuple of tensors and returns tensor or a list/tuple of tensors. Typically, the __call__ method only accepts tensors of shapes that are compatible with its object.

A Layer may or may not have weights associated with it, depending on what it does. Dense layer does have weights associated with it. However, when using the Functional API, the weights are not instantiated until we call the dense_layer.__call__() method.

In [2]: # Before calling the __call__ method
    ...: dense_layer.weights
Out[2]: []

In [3]: # After calling the __call__ method
    ...: dense_layer(Input(shape=(100,)))
    ...: dense_layer.weights
Out[3]:
[<tf.Variable 'dense_3/kernel:0' shape=(100, 10) dtype=float32_ref>,
 <tf.Variable 'dense_3/bias:0' shape=(10,) dtype=float32_ref>]

Activation layer, expectedly, does not have any weights associated with it, even after calling its __call__ method.

In [4]: activation_layer = Activation('relu')

In [5]: activation_layer.weights
Out[5]: []

In [6]: activation_layer(Input(shape=(100,)))
Out[6]: <tf.Tensor 'activation_2/Relu:0' shape=(?, 100) dtype=float32>

In [7]: activation_layer.weights
Out[7]: []

Note about `Input` and `InputLayer`

Even though we Input lies within keras.layers, Input is not actually a Layer object. Input is a function. Calling Input returns a tensor, as we have seen above. Input function calls the InputLayer class, which is indeed a subclass of Layer. InputLayer instantiates a tensor which is returned to us as the output of the Input function.

Keras Graph

When using a backend (such as TensorFlow), we build a graph that describes computation we intend to perform. This graph may be generated implicitly when eager execution is used or explicitly (such as with sess.run() in TensorFlow).

Even though Keras hides a lot of low-level backend complexity, Keras computation is still based on a graph. Understanding this Keras graph is important to fully understand the Functional API. In fact, by using the Functional API you are specifying a Keras graph. Typically, Keras graph is represented much more compactly than a backend graph. The pictures below show the Keras graph and the corresponding TensorFlow graph for the same network.

Keras graph

TensorFlow graph

Keras graph construction using Functional API

A graph consists of edges and nodes and Keras graph is no different. Keras graph is a directed graph⁴ in which layers act as nodes and tensors act as edges. Specifying the edges is straightforward – we simply need to create the Layer objects.

from keras.layers import Input, Dense
dense_layer_1 = Dense(units=20, activation='relu', name='dense_layer_1')
dense_layer_2 = Dense(units=20, activation='relu', name='dense_layer_2')
sigmoid_layer = Dense(units=1, activation='sigmoid', name='sigmoid_layer')

# While Input() returns a Keras tensor, Input() implicitly creates an `InputLayer` object
# which acts as a node in the Keras graph.
input_tensor = Input(shape=(10,), name='input')

The above code simply specifies the nodes of the graph. Since we didn’t specify edges just yet, all the nodes are unconnected. We can visualize the current state of the graph as follows.

Keras graph is unconnected until a tensor is flown through it

We can check this using code, as follows. The list of inbound and outbound nodes for dense_layer_1 are empty. The same is true for dense_layer_2 and sigmoid_layer.

In [3]: dense_layer_1._inbound_nodes
Out[3]: []

In [4]: dense_layer_1._outbound_nodes
Out[4]: []

In [5]: hasattr(dense_layer_1, 'input_shape')
Out[5]: False

In [6]: hasattr(dense_layer_1, 'output_shape')
Out[6]: False

We can also see the dense_layer_1 does not yet have input or input_shape attributes. Trying to access input_shape attribute gives us the following error.

In [7]: dense_layer_1.input_shape
AttributeError: The layer has never been called and thus has no defined input shape.

Since dense_layer_1 has not been connected to an input yet, it can accept a tensor of any shape of the form (batch_size, n_units_1, ... ). The output_shape is also undefined at this stage but will be defined automatically once we provide an input to this layer. This is a benefit of using Keras – we don’t have to fully specify the input or output shape of a layer when we instantiate it. The shapes are inferred as we make connections.

The next step is to connect the nodes of the graph with directed edges. This is where the input_tensor comes into play. The way to connect the nodes is to make a tensor flow through the nodes⁵. In the code snippet above, we already created a Keras tensor called input_tensor. We can now pass⁶ this tensor through the nodes (layers) using one line of code.

output_tensor = sigmoid_layer(dense_layer_2(dense_layer_1(input_tensor)))

This one line of code connects all the previously unconnected nodes in the order specified. This results in the following (weakly) connected directed graph. The InputLayer node is present because of technical reasons and is typically not important for the end user.

Keras graph is connected once a tensor flows through it

We have now fully specified the Keras graph. We can check that the nodes are connected using code, just like before. Note that dense_layer_1 now has both input_shape and output_shape attributes. These shapes were automatically inferred by Keras.

In [3]: dense_layer_1._inbound_nodes
Out[3]: [<keras.engine.base_layer.Node at 0x1129a8d30>]

In [4]: dense_layer_1._outbound_nodes
Out[4]: [<keras.engine.base_layer.Node at 0x136674a20>]

In [5]: hasattr(dense_layer_1, 'input_shape')
Out[5]: True

In [6]: hasattr(dense_layer_1, 'output_shape')
Out[6]: True

In [7]: dense_layer_1.input_shape
Out[7]: (None, 10)

In [8]: dense_layer_1.output_shape
Out[8]: (None, 20)

Model

A neural network is essentially a function which takes input(s) and returns output(s). In order to fit the neural network represented by the above Keras graph, we need to specify the input and output to the neural network. These input(s) and output(s) are specified as list of Keras tensors when initializing a Model object. We don’t need to provide any other information about the Keras graph because the input_tensor and output_tensor edges are already connected to the nodes (layers) of the graph.

from keras.models import Model
model = Model(inputs=[input_tensor], outputs=[output_tensor])

Once we have a Model object, we can specify the optimizer, loss functions, and metrics to track in the compile step. Model.compile is a method of the class and we don’t need to store its return value (which is None anyways).

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

We haven’t specified the actual (input, output) training data yet. The actual data is needed only when we fit the model to the training data. Typically, we store the input and output training data as numpy arrays. These numpy arrays must conform to the shapes of the input_tensor and output_tensor respectively.

import numpy as np

input_numpy = np.random.rand(1000, 10)
output_numpy = np.random.choice(2, 1000)

The following code snippet shows that the shapes on the actual data (numpy arrays) and the tensors match. The first dimension is always the batch dimension; we will take about this and tensor shapes in detail in the next section.

In [21]: input_numpy.shape
Out[21]: (1000, 10)

In [22]: input_tensor.shape
Out[22]: TensorShape([Dimension(None), Dimension(10)])

In [23]: output_numpy.shape
Out[23]: (1000,)

In [24]: output_tensor.shape
Out[24]: TensorShape([Dimension(None), Dimension(1)])

The only thing remaining is to actually pass the numpy arrays into the model.fit function.

from keras.callbacks import TensorBoard
model.fit(x=[input_numpy], y=[output_numpy],
          epochs=5, batch_size=64, callbacks=[TensorBoard()])

model.fit function accepts an argument x as the input training data. This argument x requires that we provide a list of numpy arrays in the same order as the list of tensors specified in the Model’s constructor (i.e., during model instantiation). The same applies to the y argument.

Tensor shapes

Beginner users find it difficult to correctly specify shapes of tensors and layers. One reason for this is that, in typical usage, Keras doesn’t ask you to provide the shapes of all tensors and layers explicitly upfront; instead, it does shape inference for you. Another reason is that the documentation is vague regarding tensor shapes. Finally, the tensor shapes are different for fully-connected layers, convolutional layers, and recurrent layers.

Batch dimension

This is the most important dimension. Ignoring the batch dimension is also the most common source of error for beginners.

The first dimension is always the batch size, even when the batch size is one.

Batch size is the first dimension even when the "image_data_format": "channels_first". You cannot ignore the batch dimension even when you have a single (input, output) pair. For example, if we have some data where the input has 10 features and output is a binary label, we must specify both the input tensor and input numpy array as 2-dimensional, irrespective of how many (input, output) pairs we have.

To illustrate this, let’s define a very simple neural network as follows.

def check_batch_size(input_numpy, output_numpy):
    input_tensor = Input((n_features,))
    output_tensor = Dense(units=1)(Dense(units=20)(input_tensor))
    model = Model(inputs=[input_tensor], outputs=[output_tensor])
    model.compile(optimizer='adam', loss='binary_crossentropy',
                  metrics=['accuracy'])
    print(model.summary())
    model.fit(x=[input_numpy], y=[output_numpy], epochs=5)
    return model

Let’s call the check_batch_size function with both the correct and incorrect input shape.

import numpy as np

n_features = 10
batch_size = 1

# Correct
input_numpy = np.random.rand(batch_size, n_features)  # 2-d array
output_numpy = np.random.choice(2, batch_size)        # 1-d array
check_batch_size(input_numpy, output_numpy)  # runs

# Incorrect
input_numpy = np.random.rand(n_features)  # 1-d array
check_batch_size(input_numpy, output_numpy)  # fails because of missing batch dimension

Keras is more forgiving when it comes to the shape of the output_numpy array. But it’s still a good practice to have batch size be the first dimension in output numpy arrays.

Batch size is not important for neural network specification. This is expected because typically we would like to run the trained model for one test example at a time; fixing the batch size to one particular number would make this difficult. We also often want to try out different (mini) batch sizes during model fitting. When generating an input tensor, Keras lets us choose if we want to specify the batch size up front or wait until the model training stage.

In [6]: Input(shape=(10, ))  # unknown number of examples, each example having 10 features
Out[6]: <tf.Tensor 'input_1:0' shape=(?, 10) dtype=float32>

In [7]: Input(batch_shape=(100, 10))  # 100 examples, each example having 10 features
Out[7]: <tf.Tensor 'input_2:0' shape=(100, 10) dtype=float32>

It is recommended to not specify the batch size up front unless there is a special need. When we don’t specify the batch size up front, the batch size appears as a ? or None.

In [3]: Input(shape=(10, ))
Out[3]: <tf.Tensor 'input:0' shape=(?, 10) dtype=float32>

In [4]: Input(shape=(10, )).shape
Out[4]: TensorShape([Dimension(None), Dimension(10)])

Convolutional layers

A tensor that interacts with a (2D) convolutional layer has one of two specific shapes.

`image_data_format`	Tensor shape
`channels_last`	`(batch_size, image_height, image_width, n_channels)`
`channels_first`	`(batch_size, n_channels, image_height, image_width)`

The image_data_format is typically found in ~/.keras/keras.json. As an example, for RGB images of 64x64 pixels, we can expect to see something like this:

from keras.layers import Input, Conv2D
input_tensor = Input((64, 64, 3))  # 64x64 pixels, 3 channels
conv_layer = Conv2D(filters=17, kernel_size=(3, 3))
output_tensor = conv_layer(input_tensor)

In [1]: conv_layer.input_shape
Out[1]: (None, 64, 64, 3)

In [2]: conv_layer.output_shape
Out[2]: (None, 62, 62, 17)

Recurrent layers

Typically, recurrent layers such as LSTM accept input of the shape (batch_size, input_sequence_length, vocab_size). The following examples explain the two common use cases.

Example 1: One-hot encoded sequence as input

This is the typical case in natural language processing tasks. Given a sentence consisting of five words, we encode the words (text separated by white space) using one-hot encoding. Let’s assume that we only have the four necessary words in the vocabulary.

	0	1	2	3	4
Sentence	the	cat	and	the	dog
Encoding	`[0, 1, 0, 0]`	`[1, 0, 0, 0]`	`[0, 0, 1, 0]`	`[0, 1, 0, 0]`	`[0, 0, 0, 1]`

In this case, the input sequence length is 5 and the size of each element of the input sequence is a one-hot vector of length 4. Thus, one input example is of size (input_sequence_length, vocab_size) = (5, 4) and there can be many input examples in a batch. The following code represents how to build an LSTM model to accept this input.

from keras.layers import Input, LSTM, Dense

input_sequence_length = 4
vocab_size = 5
input_tensor = Input(shape=(input_sequence_length, vocab_size))
lstm_layer = LSTM(units=10, return_sequences=False, return_state=False)
dense_layer = Dense(vocab_size, activation='softmax')
output_tensor = dense_layer(lstm_layer(input_tensor))

Note how we have set return_sequences and return_state to be False. The output of the lstm_layer is a tensor of shape (batch_size, n_lstm_units) = (None, 10).

In [30]: lstm_layer.output_shape
Out[30]: (None, 10)

Changing any of the arguments return_sequences and return_state to True would change the output shape completely⁷.

Example 2: Scalar, real-valued sequence as input

This example is in contrast to the previous example. This time, let us assume that we are trying to model a sequence of scalar, real-valued quantities. One example could be the stock price of one particular stock. We want a model that takes as input a sequence of 5 stock prices and outputs the next (6th) stock price. There is no need to encode the real-valued stock price⁸. The input data looks like this:

	0	1	2	3	4
Price	30.56	29.54	32.12	36.78	40.01

Even though the stock price is a scalar quantity, for LSTM, we will have to consider the stock price to be a vector of one dimension. The following code snippet shows the correct and incorrect way of specifying tensor shapes.

from keras.layers import Input, LSTM, Dense

# Correct
input_sequence_length = 5
input_dimension = 1
input_tensor = Input(shape=(input_sequence_length, input_dimension))
lstm_layer = LSTM(units=10, return_sequences=False, return_state=False)
dense_layer = Dense(input_dimension, activation='softmax')
output_tensor = dense_layer(lstm_layer(input_tensor))  # runs

# Incorrect
input_sequence_length = 5
input_tensor = Input(shape=(input_sequence_length,))
lstm_layer = LSTM(units=10, return_sequences=False, return_state=False)
dense_layer = Dense(1, activation='softmax')
output_tensor = dense_layer(lstm_layer(input_tensor))  # fails

Batch normalization layer

BatchNormalization layer is another common reason for shape-related errors.

Batch normalization was proposed by Ioffe & Szegedy, 2015 primarily for fully-connected (Dense) layers and convolutional layers. The authors mention that at the time they did not fully explore the application of batch normalization to recurrent neural networks (see Page 8, last paragraph, Ioffe & Szegedy, 2015). In the subsequent years, various other forms of normalization methods were proposed including Layer Normalization by Lei Ba et al., 2016 and Recurrent Batch Normalizaton by Cooijmans et al., 2016. Local Response Normalization, which is a normalization over channels in convolutional layers, was proposed by Krizhevsky et al., 2012.

The batch normalization performed by the BatchNormalization function in keras is the one proposed by Ioffe & Szegedy, 2015 which is applicable for fully-connected and convolutional layers only⁹.

The key to the BatchNormalization layer in keras is the axis argument, which has a rather confusing documentation, as discussed in this StackOverflow post. As described in this post, the confusion arises because numpy functions (such as np.mean) also have an axis argument and the meaning of axis in np.mean is opposite to that in BatchNormalization. In np.mean, the axis argument indicates the axis that is to be collapsed. In BatchNormalization, axis argument indicates the axis that is to be preserved.

Fully-connected layers

Let’s consider the following simple examples. The input tensor to BatchNormalization consists of three examples in a batch; each example in the batch has 2 features.

from keras.layers import Input, BatchNormalization

batch_size = 3
n_features = 2
input_tensor = Input(batch_shape=(batch_size, n_features))

# Example 1: average over batches (typically correct)
output_tensor_1 = BatchNormalization(axis=1)(input_tensor)  #runs

# Example 2: average over features (typically incorrect)
output_tensor_2 = BatchNormalization(axis=0)(input_tensor)  #runs

Example 1 in the snippet above computes mean and variance for each of the two features independently. While computing the mean and variance for one particular feature, Example 1 averages over the three batches – it computes the mean of three scalar numbers.

Example 2 in the snippet above computes mean and variance for each of the three examples in the batch independently. While computing mean and variance for one particular example, Example 2 average over the two features – it computes the mean of two scalar values.

Even though the two examples perform meaningfully different computations, the shape of the output tensor remains the same.

In [1]: output_tensor_1
Out[1]: <tf.Tensor 'batch_normalization_2_1/cond/Merge:0' shape=(3, 2) dtype=float32>

In [2]: output_tensor_2
Out[2]: <tf.Tensor 'batch_normalization_2_2/cond/Merge:0' shape=(3, 2) dtype=float32>

Thus, there is no way for us (or for Keras) to check if we have used the correct value for axis! Typically, when we perform batch normalization, we would like to average over batches and not features. Therefore, Example 1 is the more typical usage (by far) and represents the correct way to perform the commonly understood implementation of batch normalization.

The problem becomes worse when we have more than one feature dimension, as shown below.

from keras.layers import Input, BatchNormalization

batch_size = 3
feature_shape = (2, 4, 5)
input_tensor = Input(batch_shape=(batch_size,) + feature_shape)

# Example 1
output_tensor_1 = BatchNormalization(axis=1)(input_tensor)

# Example 2
output_tensor_2 = BatchNormalization(axis=2)(input_tensor)

# Example 3
output_tensor_3 = BatchNormalization(axis=3)(input_tensor)

As before, all the three output tensors in the snippet above have the same shape. Thus, we cannot use the shape of the output tensor to determine which one of the three batch normalizations is the correct one. The input tensor has the same shape as the output tensors.

In [3]: input_tensor
Out[3]: <tf.Tensor 'input_1:0' shape=(3, 2, 4, 5) dtype=float32>

Example 1 would preserve the second dimension (=2) of the input tensor and average over all the rest. Example 2 would preserve the third dimension (=4). Example 3 would preserve the fourth dimension (=5). There is no easy way¹⁰ for us to preserve more than one dimension at a time when using BatchNormalization.

Convolutional Layers

The underlying code for BatchNormalization is the same whether it’s used for fully-connected layers or for convolutional layers. But, for convolutional layers, we need to consider the image_data_format when deciding the value of axis argument for BatchNormalization. The following table shows the usual values of the axis argument for both options of image_data_format.

`image_data_format`	Tensor shape	`axis`
`channels_last`	`(batch_size, image_height, image_width, n_channels)`	`axis=3`
`channels_first`	`(batch_size, n_channels, image_height, image_width)`	`axis=1`

For both of the cases above, BatchNormalization would preserve the channel dimension and average over the rest. This means that we would average over all the examples in the batch as well as the all of the pixels, together. While this may look like a mistake, this is actually correct. This behavior is actually what Ioffe & Szegedy, 2015 specified in their implementation for convolutional layers (see Section 3.2, last paragraph in Ioffe & Szegedy, 2015). The authors desired that the normalization obey the convolutional property such that different elements of the same feature map (i.e., channel) at different locations are normalized in the same way.

Final thoughts

Keras is an excellent high-level API to build neural network models. Keras Functional API is powerful enough to let us build complicated models without having to descend into the low-level, backend API. While Keras documentation is vague at times, Keras provides sensible defaults and performs shape inference to discover mistakes at model specification time. I hope that this post provides some help with the vaguely documented sections of the Keras codebase and saves you some time and effort. If you notice any mistakes or areas that I have overlooked, please leave a comment and let me know.

Footnotes

According to the is_keras_tensor function, an object is a Keras tensor if it is a backend tensor and has the _keras_history attribute. ↩

This is not a Keras-specific functionality. In python, an object can be made callable by adding a __call__ method to it. For example:

class Example(object):
    def __init__(self, data):
        self.data = data

    def __call__(self, *args, **kwargs):
        return self.data

x = Example(1)

In [1]: x(2)
Out[1]: 1

In [2]: x()
Out[2]: 1

↩

Either a Keras tensor or a backend tensor. ↩
Edge directions in a Keras graph indicate the forward pass of the neural network. During the backward pass or backpropagation, which computes the gradients, we reverse all edges of the graph to get the transpose graph. ↩
And, so, TensorFlow earns its name. ↩
Much like threading pearls (Layer objects) with a string (Keras tensor). ↩
Tensor handling by LSTM, SimpleRNN, and GRU are a subject for a future post. ↩
In a real application, we may need to scale the (real) values into a bounded range such as [-1, 1]. We omit the rescaling discussion because it doesn’t affect the conclusion. ↩
I do not recommend using BatchNormalization for recurrent layers, unless you know what you’re doing. Based on the code, the axis argument to BatchNormalization is not general purpose. It is specifically designed to match the implementation described in Ioffe & Szegedy, 2015, which is only applicable for fully-connected and convolutional layers. ↩
We could always use keras.layers.Lambda and keras.layers.Reshape to extract and reshape individual dimensions and perform the normalization ourselves. ↩

Bernoulli Distribution as a tiny Neural Network

Sat, 27 Apr 2019 00:00:00 +0000

Logistic regression is often considered the smallest neural network for binary classification. We can think of Bernoulli distribution as an even smaller neural network – one that doesn’t even depend on the input data. Such a neural network would likely not be useful in practice. However, given it’s simplicity, it serves as an illuminating example to help us understand the statistical assumptions underlying a neural network model. The assumptions we require for modeling Bernoulli distribution as a neural network are also required for larger neural networks. As an example, using Bernoulli distribution as a tiny neural network, we can easily demonstrate how the famous cross-entropy loss comes into being. We can even extend this Bernoulli distribution model framework to recreate the familiar logistic regression model by simply replacing a constant parameter by a sigmoid-affine function.

(This document is available as a PDF)

Bernoulli distribution

Bernoulli distribution, owing to its simplicity, is used more often than it is noticed. A random variable $X \sim \text{Bernoulli}(p)$ has the following probability mass function (pmf):

\[\begin{aligned} P(X = 1) &= p \notag \\ P(X = 0) &= 1 - p \label{eqn:raw-form}\\ P(X \notin \{0, 1\}) &= 0 \notag \end{aligned}\]

in which, the only parameter, $p$, is a probability and therefore must satisfy $0 \le p \le 1$. We call the above equation the raw form pmf of the Bernoulli distribution.

The raw form pmf is simple to understand but its multi-case structure makes it difficult to use in other derivation. We can combine the two of the three cases into one equation without changing anything about the distribution. This results in the following two forms of pmf – the additive form:

\[P(X = x) = \begin{cases} p x + (1 - p) (1 - x) & x \in \{0, 1\} \\ 0 & \text{otherwise} \end{cases} \label{eqn:additive-form}\]

and the multiplicative form:

\[P(X = x) = \begin{cases} p^x (1 - p)^{(1 - x)} & x \in \{0, 1\} \\ 0 & \text{otherwise} \end{cases} \label{eqn:multiplicative-form}\]

All three forms — raw, additive, multiplicative¹ — are equivalent to each other and represent the same exact distribution. This implies that no matter which of the three forms we use for our analysis, we should get the exact same analytical result. However, one form may be easier to work with than the others when wrangling algebraic equations. The multiplicative form is the most common one used in both statistical analysis and with neural networks.

Binary classification

Let’s consider a familiar application of supervised binary classification in computer vision – image classification. We would like to classify a given image into one of two classes – a cat image versus a dog image:

Cat

Dog

Image by Pexels

In a supervised setting, we usually have training data available, which is represented as:

\[\{ (x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \ldots, (x^{(i)}, y^{(i)}), \ldots, (x^{(m)}, y^{(m)}) \} \label{eqn:binary-classification-data}\]

in which, $x^{(i)} \in \mathbb{R}^{n_x}$ is the input data and $y^{(i)} \in \{0, 1\}$ is the output label. For the cat vs dog example, $x^{(i)}$ is a vector of pixel values obtained by flattening the tensor that represents an image and $y^{(i)}$ represents the label – cat ($y=1$) or dog ($y=0$).

Modeling the binary classification problem as Bernoulli distribution

Modeling binary classification

We aim to fit a function to describe the input-output relationship in the training data. We could attempt to find a suitable deterministic function $y=f(x, \theta)$ and minimize (w.r.t the model parameters $\theta$) some appropriate measurement of discrepancy ($\Phi(\theta)$) between the function’s predicted labels and true labels². Alternatively, we could model the output label as a random variable³

\[Y\sim\text{SomeDistribution}(x, \theta) \label{eqn:some-distribution}\]

in which, $\theta \in \mathbb{R}^{n_t}$ is the set of model parameters. The training data (shown above) is interpreted as a list of $m$ statistical samples of $Y$ generated along with the corresponding values of $x$.⁴ When we choose to model the output label as random variable, we have a well-established approach to minimize the discrepancy between the predicted and true labels – maximum likelihood estimation.

Modeling with Bernoulli distribution

Since the true output labels only take values in $\{0, 1\}$, it would be ideal if our choice of random variable also assumes values in $\{0, 1\}$. Bernoulli distribution is one such choice:

\[Y\sim\text{Bernoulli}(p) \label{eqn:bernoulli-model}\]

in which, $\theta = \begin{bmatrix}p\end{bmatrix}$. Note how the model above doesn’t depend on the input $x$ at all.

Now that we have a statistical model to describe the output, we can write down the likelihood as follows:

\[\begin{aligned} &\mathcal{L}(\theta) \notag\\ &= P\left( (Y^{(1)} = y^{(1)}) \cap (Y^{(2)} = y^{(2)}) \cap \ldots (Y^{(i)} = y^{(i)}) \cap \ldots (Y^{(m)} = y^{(m)}) ; \theta \right) \label{eqn:likelihood}\\ &= P\left( (Y^{(1)} = y^{(1)}) \cap (Y^{(2)} = y^{(2)}) \cap \ldots (Y^{(i)} = y^{(i)}) \cap \ldots (Y^{(m)} = y^{(m)}) \mid \Theta = \theta \label{eqn:likelihood-bayesian} \right)\end{aligned}\]

The second equation is the Bayesian form of likelihood, in which we choose to model the parameters as a random variable $\Theta$. The first equation is also the likelihood but it doesn’t consider the model parameters as random variables. We will only use the first form of likelihood in this document because we have no need to model the parameter(s) as random variable(s) at this time.

Assuming independence, we can re-write the likelihood as:

\[\mathcal{L}(\theta) = \prod_{i=1}^{i=m} P\left( Y^{(i)} = y^{(i)} ; \theta \right)\]

Substituting the multiplicative form and applying the knowledge that the output labels $y^{(i)} \in \{0, 1\}$, we obtain:

\[\mathcal{L}(\theta) = \prod_{i=1}^{i=m} p^{y^{(i)}} (1 - p)^{(1 - y^{(i)})} \label{eqn:multiplicative-likelihood}\]

Taking logarithm, the log-likelihood is:

\[\log \mathcal{L}(\theta) = \sum_{i=1}^{i=m} \left[ y^{(i)}\log p + (1 - y^{(i)})\log (1 - p) \right] \label{eqn:bernoulli-log-likelihood}\]

The expression above is the famous cross-entropy loss⁵. Maximizing the log-likelihood:

\[\begin{aligned} 0 &= \frac{\partial \log \mathcal{L}(\theta)}{\partial p} \notag \\ 0 &= \sum_{i=1}^{i=m} \left[ \frac{y^{(i)}}{p} - \frac{(1 - y^{(i)})}{(1 - p)} \right] \notag \\ \implies \hat{p} &= \frac{\sum_{i=1}^{i=m} y^{(i)}}{m} \end{aligned}\]

in which $\hat{p}$ is the maximum likelihood estimate for $p$. In order to confirm that this value of $\hat{p}$ actually maximizes the log-likelihood, we can show that the second order derivative is negative as follows:

\[\begin{aligned} \frac{\partial^2 \log \mathcal{L}(\theta)}{\partial p^2} = - \sum_{i=1}^{i=m} \left[ \frac{y^{(i)}}{p^2} + \frac{(1 - y^{(i)})}{(1 - p)^2} \right] < 0 \end{aligned}\]

See Appendix A in the PDF version for the same result obtained using the additive form.

Extension to logistic regression

In the previous subsection, we didn’t even consider the input $x$ in our model. If we want to include the input $x$, we could replace the previously constant $p$ with a function of $x$. A simple way to include $x$ is to model $p$ as an affine⁶ function of $x$ instead of a constant

\[p = w^{T} x + b \label{eqn:affine}\]

in which, $w \in \mathbb{R}^{n_x}$ and $b \in \mathbb{R}$ are model parameters. However, there is a problem with the above affine function – there is no guarantee that the expression $w^{T} x + b$ would be within $0$ and $1$, as required for $p$.⁷ We can solve this problem easily by passing $w^{T} x + b$ through a sigmoid function to obtain the following sigmoid-affine function:

\[\begin{aligned} p = \sigma(w^{T} x + b) \\ \sigma(z) = \frac{1}{1 + e^{-z}} \label{eqn:sigmoid} \end{aligned}\]

The resulting model for $Y$ becomes:

\[Y \sim \text{Bernoulli}(\sigma(w^{T} x + b))\]

which is exactly the logistic regression model. The difference is that instead of directly assuming the logistic form, we have chosen to interpret logistic regression model as an extension of the Bernoulli distribution.

We can now perform log-likelihood maximization as usual. The likelihood is given by the following expression:

\[\begin{aligned} &\mathcal{L}({\theta})\notag\\ &= P\left( (Y^{(1)} = y^{(1)}) \cap (Y^{(2)} = y^{(2)}) \cap \ldots (Y^{(i)} = y^{(i)}) \cap \ldots (Y^{(m)} = y^{(m)}) ; \theta \right) \notag \\ &=\prod_{i=1}^{i=m} P\left( Y^{(i)} = y^{(i)} ; \theta \right) \quad \text{(independence)} \notag \\ &=\prod_{i=1}^{i=m} \left[ \{\sigma(w^{T} x + b)\}^{y^{(i)}} \{1 - \sigma(w^{T} x + b)\}^{(1 - y^{(i)})} \right] \quad (y^{(i)} \in \{0, 1\}, \text{for all } i) \end{aligned}\]

in which, $\theta = \begin{bmatrix}w & b\end{bmatrix}$. The log-likelihood may be written as:

\[\begin{aligned} \log \mathcal{L}(\theta) = \sum_{i=1}^{i=m} \left[ y^{(i)}\log \sigma(w^{T} x + b) + (1 - y^{(i)})\log (1 - \sigma(w^{T} x + b)) \right] \label{eqn:logistic-log-likelihood} \end{aligned}\]

Log-likelihood may be maximized via any of the numerical optimization algorithms such as gradient descent.

Summary

Log-likelihood equations for Bernoulli distribution and logistic regression have the same form. In fact, we can obtain the log-likelihood for logistic regression by simply replacing the $p=\sigma(w^{T}x + b)$ in the log-likelihood for the Bernoulli model. Logistic regression is a one-layer⁸ neural network:

Logistic Regression

We can think of the Bernoulli model as an even simpler neural network that isn’t connected to the input layer at all:

Bernoulli Model

Equivalently, Bernoulli model is the same as a logistic regression model that has all the weights $w$ set to zero and $p=\sigma(b)$. Bernoulli model has a constant number of unknown model parameters while logistic regression has a parameter linear complexity in the input image size. Conversely, we can also think of logistic regression as one particular extension of the Bernoulli model framework.

Footnotes

Note that we interpret $0^0$ as $1$ and $\log 0^0 = 0\log0 = 0$. ↩
For certain choices of $f(x, \theta)$ and $\Phi(\theta)$, the estimated model parameters may equal the estimated model parameters obtained using the statistical modeling approach, analytically. ↩
A random variable is also a function (a measurable function) but we choose to highlight the difference between any deterministic function and the restrictive measurable function that is a random variable. ↩
For this problem, we choose to consider the list of different images (i. e., $x$’s) as deterministically fixed. Therefore, $x$ is not random and we do not condition on $x$. ↩
This is a demonstration of the equivalence between maximizing the likelihood and minimizing the KL divergence. ↩
A linear function $p = w^{T}x$ would be even simpler than affine function $p = w^{T} x + b$. ↩
We could always treat the problem as a constrained optimization problem in which $w, b$ can only take values such that $0 \le w^{T} x + b \le 1$ but we do not pursue this line of analysis. ↩
Typically, we don’t count input layer at all. ↩

Setup a Spark cluster on AWS EMR

Sat, 11 Aug 2018 00:00:00 +0000

AWS provides an easy way to run a Spark cluster. Let’s use it to analyze the publicly available IRS 990 data from 2011 to present. This data is already available on S3 which makes it a good candidate to learn Spark. This medium post describes the IRS 990 dataset. This documentation shows you how to access this dataset on AWS S3. We will see more details of the dataset later.

Setup a Spark cluster

Caveats

I do not go over the details of setting up AWS EMR cluster. Interested readers can read the official AWS guide for details. Setting up an AWS EMR cluster requires some familiarity with AWS concepts such as EC2, ssh keys, VPC subnets, and security groups. I do not cover these details in this post either. Using AWS will probably require you to setup a payment method (such as a credit card) and may cost you some money. I advise you to keep an eye on the cost. If this is your first time using a cloud computing platform (such as AWS or GCP), I recommend you first try out setting up an EC2 instance before you move on to setup an EMR cluster. Some steps such as setting up billing and setting up an EC2 key pair are common to both EC2 and EMR.

We will use Python, Jupyter notebook, and pyspark to analyze the data. AWS provides an already configured JupyterHub installation but we will not use the AWS-provided installation. Instead, we will install our own jupyter package on the cluster and use that. This is because the AWS-provided JupyterHub installation runs within a docker container and deprives the end user (which is you and me) of many python and jupyter benefits. It also adds another layer of complexity to an already complicated Spark cluster setup. The AWS-provided JupyterHub installation does provide us with some features that we don’t really need at this time. The increased complexity in the AWS-provided installation makes it difficult to install and use python packages (such as pandas and numpy). Finally, installing our own jupyter package is a good way to learn how to setup some tools for yourself, which is always a good thing for a data scientist.

Launch EMR cluster

We will use advanced options to launch the EMR cluster.

Step 1: Software and Steps

Launch an EMR cluster with a software configuration shown below in the picture. Ensure that Hadoop and Spark are checked. If this is your first time setting up an EMR cluster go ahead and check Hadoop, Zepplein, Livy, JupyterHub, Pig, Hive, Hue, and Spark. Enabling JupyterHub and Zeppelin options will provide a read-to-use albeit clunky jupyter notebook (as described above in Caveats) but it won’t prevent us from what we are trying to do.

Step 2: Hardware

You can choose any hardware you’d like. More powerful instances cost more money so decide what’s reasonable for you. If this is your first time, stay with the default hardware options.

For now, we will use one m4.large instance for the Master and one m4.large instance for Core. We don’t use any task instances just yet because we can easily add those later. We choose only one core instance for the same reason – we can easily add more instances later.

One restriction is the instance type – you may not be able to change the instance types later, especially for the master node. Therefore, you may want to choose a more powerful instance type for the master node to begin with. Typically, an instance with 8 virtual cores and 16GB of memory is a good start for a master node. For Core instances, 4 virtual cores and 32 GB of memory or higher may be a good fit. Typically, more memory and more EBS storage is useful for worker nodes. If we need task instances, we can add them later as spot instances which can be significantly cheaper.

Step 3: General Cluster Settings

Let’s provide a name for our cluster. The name doesn’t have to be unique (even within one AWS account, you can have multiple EMR clusters with the same exact name). We enable logging to AWS S3 to a bucket of our choosing (for security reasons, it’s a good idea to have this bucket be private and encrypted). We don’t need to specify any bootstrap actions at this time.

Step 4: Security

We use an already setup EC2 key pair. We will need to be able to ssh into the EMR cluster which means this step is necessary. You may find these instructions on how to setup an EC2 key pair useful. We use the default EC2 security groups for now. Security groups are important consideration because Spark requires that master node can talk to the worker nodes via password-less (using a private key) ssh. We need to make sure that the instances have the necessary ports open to allow ssh traffic between them. Ideally, you should open your EMR cluster just enough so that you can access it but any unauthorized person/program cannot. If you’re setting this cluster up for work, please consult your DevOps engineer or someone in-charge of security.

Launch

It might take 5-10 minutes after you click on Create Cluster button for your cluster to be ready. When the cluster is ready, it should say Waiting, in green color.

Setup `ssh` into the EMR cluster

Create a new security group

While the cluster is launching, let’s create a new security group so that we can ssh into the master node of the EMR cluster. Note that you can use this security group for any EC2 instance including the EMR cluster we just launched.

Go to EC2 homepage on AWS.
Make sure you’re in the same region as the EMR cluster.
Click on NETWORK & SECURITY > Security Groups in the left-hand-side menu.
Click on Create Security Group button.
Provide any Security group name; we will use allow-ping-ssh.
Provide any Description; we will use ‘Allows pinging and ssh’.
Use the same VPC as the one you used for the EMR cluster. Typically, this is the default value.
Click on Add Rule three times and add rules to allow ICMP (pings) in v4, v6 and SSH from your own IP. See the screenshot below.
Click on Create to create the security group.

Caveat: This security group setting may not be sufficiently secure for your situation. As mentioned before, if you’re doing this for for work, please consult your DevOps engineer or someone in-charge of security.

Add security group to master node

We want to be able to ssh into the master node of the EMR cluster. To ssh, we want to allow TCP traffic on port 22 (default port for ssh) from our IP (or similar) going to the master node of the EMR cluster. We also allow pinging the master node from our IP address (this helps in debugging ssh failures).

Go to EMR homepage on AWS.
Find your EMR cluster and click on it. Tip: Ensure you’re on the same region as the one you launched your cluster in. Check the top right of the webpage for the name of the region.
Once you’re on the cluster’s homepage, click on Hardware. You should see your master and core instance(s) listed.
Click on the ID of the master node. Typically, this is a link that looks like like ig-S0MESTR1NG.
Click on the EC2 instance ID. Typically, this is a link that looks like i-0a1b2c3d4e5f. You should now only see only row showing your master EC2 instance.
Right-click on this row, click on Networking > Change Security Groups.
Leave the already checked security group as-is. Check the allow-ping-ssh security group that we created in the previous step.

SSH into master node

We should be all set to ssh into our EMR cluster. By logging into the EMR cluster, we mean logging into the master node of the EMR cluster.

Go to the cluster homepage.
Ensure that the cluster is done provisioning and that you can see Waiting in a green color.
Click on the SSH link on the cluster’s homepage. This should tell you how to ssh into your cluster. Typically, this requires running the ssh command on you local machine as shown below.

# On local machine
ssh -i ~/my-ec2-key-pair.pem hadoop@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

Troubleshooting

This is how a successful ssh into the master node looks like.

If you cannot ssh into the master node, try pinging it first.

# On local machine
ping ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

If you can successfully ping the instance, then you’re halfway there. Try using the verbose mode of ssh to better understand what’s preventing you from sshing into the master node.

# On local machine
ssh -vv -i ~/my-ec2-key-pair.pem hadoop@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

Finally, take a look at this AWS troubleshooting page which contains some really helpful videos to solve this exact issue.

You will need to ssh into the master node to perform the next steps in this post.

Get to know your EMR cluster

Let’s explore the installation first. This builds a general understanding of how things work, especially when you’re using a machine that someone else (in our case, AWS) setup for you. Note that your results might vary a little depending on specific versions of software packages being run on your instance.

User

Firstly, we see that we are logged in as the user hadoop.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ whoami
hadoop
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ pwd
/home/hadoop
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ echo $HOSTNAME
ip-xxx-xxx-xxx-xxx
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ echo $SHELL
/bin/bash

Python

Since we want to use python, let’s see what python goodies are available and where they are stored. It looks like python 2.7, python 3.4, and pip for python 2 are installed.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ which python
/usr/bin/python
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ which python3
/usr/bin/python3
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ python --version
Python 2.7.14
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ python3 --version
Python 3.4.8
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ which pip
/usr/bin/pip
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ which pip3
/usr/bin/which: no pip3 in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin)
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ pip --version
pip 9.0.3 from /usr/lib/python2.7/dist-packages (python 2.7)

Let’s see what python 2 packages are installed. Note that numpy is installed but pandas and matplotlib are not installed.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ pip freeze
aws-cfn-bootstrap==1.4
awscli==1.14.9
...
nltk==3.2
...
numpy==1.14.5
...
virtualenv==15.1.0
...
You are using pip version 9.0.3, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Let’s check the user-site and system-site for the installed python 2. It looks like the user-site doesn’t exist yet which means all of the packages we saw above are installed at the system site. We also get warning to upgrade pip but we won’t do that because pip seems to be installed in /usr/lib which will require the use of sudo to upgrade. Using sudo can mess up the system python installation.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ python -m site
sys.path = [
    '/home/hadoop',
    '/usr/lib/python27.zip',
    '/usr/lib64/python2.7',
    '/usr/lib64/python2.7/plat-linux2',
    '/usr/lib64/python2.7/lib-tk',
    '/usr/lib64/python2.7/lib-old',
    '/usr/lib64/python2.7/lib-dynload',
    '/usr/local/lib64/python2.7/site-packages',
    '/usr/local/lib/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib/python2.7/site-packages',
    '/usr/lib64/python2.7/dist-packages',
    '/usr/lib64/python2.7/dist-packages/PIL',
    '/usr/lib/python2.7/dist-packages',
]
USER_BASE: '/home/hadoop/.local' (doesn't exist)
USER_SITE: '/home/hadoop/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ pip show numpy
Name: numpy
Version: 1.14.5
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /usr/local/lib64/python2.7/site-packages
Requires:
You are using pip version 9.0.3, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Spark

Let’s inspect the spark installation, starting with pyspark. We see that pyspark is installed in /usr/bin and it’s simply a bash script that calls another bash script /usr/lib/spark/bin/pyspark.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ which pyspark
/usr/bin/pyspark
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ more /usr/bin/pyspark
#!/bin/bash

# Autodetect JAVA_HOME if not defined
. /usr/lib/bigtop-utils/bigtop-detect-javahome

exec /usr/lib/spark/bin/pyspark "$@"

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ more /usr/lib/spark/bin/pyspark
#!/usr/bin/env bash
...
export PYSPARK_DRIVER_PYTHON
export PYSPARK_DRIVER_PYTHON_OPTS
exec "${SPARK_HOME}"/bin/spark-submit pyspark-shell-main --name "PySparkShell" "$@"

Let’s check out /usr/lib/spark. We see that /usr/lib/spark is SPARK_HOME – it contains the spark installation. We can get to the spark config files (/usr/lib/spark/conf) and work folder (/usr/lib/spark/work) from here.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ cd /usr/lib/spark/
[hadoop@ip-xxx-xxx-xxx-xxx spark]$ ls -l
total 112
drwxr-xr-x 2 root root  4096 Aug 13 04:04 bin
lrwxrwxrwx 1 root root    15 Aug 13 04:04 conf -> /etc/spark/conf
drwxr-xr-x 5 root root  4096 Aug 13 04:04 data
drwxr-xr-x 4 root root  4096 Aug 13 04:04 examples
drwxr-xr-x 3 root root  4096 Aug 13 04:05 external
drwxr-xr-x 2 root root 20480 Aug 13 04:05 jars
-rw-r--r-- 1 root root 18045 Jun 29 18:11 LICENSE
drwxr-xr-x 2 root root  4096 Aug 13 04:04 licenses
-rw-r--r-- 1 root root 24913 Jun 29 18:11 NOTICE
drwxr-xr-x 6 root root  4096 Aug 13 04:04 python
drwxr-xr-x 3 root root  4096 Aug 13 04:04 R
-rw-r--r-- 1 root root  3809 Jun 29 18:11 README.md
-rw-r--r-- 1 root root   753 Jun 29 18:11 RELEASE
drwxr-xr-x 2 root root  4096 Aug 13 04:04 sbin
lrwxrwxrwx 1 root root    19 Aug 13 04:04 work -> /var/run/spark/work
drwxr-xr-x 3 root root  4096 Aug 13 04:05 yarn

Note that pyspark is not installed by pip. Instead, pyspark is available within our spark installation at /usr/lib/spark/python. We won’t need to install pysaprk using pip; instead, we will modify PYTHONPATH and use the pyspark installation at /usr/lib/spark/python.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx python]$ pip show pyspark
You are using pip version 9.0.3, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Finally, let’s see if we already have spark master daemon running on our master node. It appears that we don’t have a spark master daemon running, which is expected.

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx python]$ ps aux | grep "spark"
hadoop    7416  0.0  0.0 110512  2100 pts/0    S+   05:33   0:00 grep --color=auto spark
spark    16036  0.2  2.6 3614692 213244 ?      Sl   04:09   0:11 /usr/lib/jvm/java-openjdk/bin/java -cp /usr/lib/spark/conf/:/usr/lib/spark/jars/*:/etc/hadoop/conf/ -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g org.apache.spark.deploy.history.HistoryServer

Install Python packages

Let’s install jupyter and any other python package on the master node. Unlike AWS-provided JupyterHub installation, here, we can easily install python packages (including jupyter) using a familiar process; we don’t have to worry about docker instances.

Note that we never use sudo to install a python package in these instructions. Using sudo is in general a bad idea, even on your local machines (Linux and MacOS) because there are many system utilities that depend on system python and using sudo updates your system-wide python installation. We always use pip install --user <pkg-name> which installs python packages in the user-site while leaving system python dependencies alone. This also makes for an easy cleanup if things don’t work out because we can easily delete user-site files and folders.

# On EMR master node
pip install --user jupyter
pip install --user ipython
pip install --user pandas
pip install --user matplotlib
pip install --user seaborn

Note that in some cases you’ll want to install a python package (typically, as a dependency) that is already installed in the system-site. It may be useful to re-install the package in the user-site to avoid dependency errors. To do this, you can use -I option like this:

# On EMR master node
pip install --user -I --upgrade python-dateutil

# Tip: You may want to try this to avoid errors associated with cached files.
# This typically happens when you install a package for both python2 and python3.
# pip install --user -I --upgrade --no-cache-dir python-dateutil

Launch Jupyter Notebook

Setup `ssh` or `mosh`

In order to launch the jupyter notebook, we will need to ssh into the master node of the Spark cluster. As mentioned previously, you can use regular ssh to do this but you may encounter timeout issues when running code that takes a lot of time. Since Spark code typically does take some time to run, timing out is a frequent problem. We can alleviate (but not eliminate) some of these timeout problems with a few nifty tricks.

The first trick prevents ssh timeouts resulting from inactivity (broken pipe problems). We will periodically send a null packet to the server (master node of the Spark cluster) to keep the connection alive. This StackExchange post and this blog post provide some more details.

In order to accomplish this, we will add the following entry to our ~/.ssh/config file. If you don’t have ~/.ssh/config file, you can create it; it’s a plain text file. Please remember to change the HostName and the location of the IdentityFile to your specific values.

# In your ~/.ssh/config file
Host emr
    HostName ec2-xxx-xxx-xxx-xxx.compute-y.amazonaws.com
    User hadoop
    IdentityFile ~/my-ec2-key-pair.pem
    IdentitiesOnly yes
    ServerAliveInterval 119
    ServerAliveCountMax 10

We are essentially asking our ssh client (local machine) to send a null packet every 119 seconds and expect a response from the server. If the client doesn’t receive a response from the server, it tries again up to a maximum of 10 times before giving up.

The second trick involves the use of mosh which is specifically designed to deal with this situation. You will need to install mosh on both client (local machine) and the server (master node of the EMR cluster) as well as allow UDP traffic on ports 60000–61000. You can edit the security group allow-ping-ssh, which we setup previously, to do this. If you’re doing this at work, please make sure that opening up these UDP ports is allowed at your employer’s.

Once mosh is ready for use, you can simply use mosh instead of ssh. You can use both of these tricks simultaneously – keep the configuration we setup in ~/.ssh/config and still use mosh. I recommend that you use both tricks because mosh cannot be used to do port forwarding and we will have to use plain old ssh as well.

# On local machine

# If you setup the entry in ~/.ssh/config, then you can simply type
mosh emr

# If you didn't setup the entry in ~/.ssh/config, use the full command
mosh -i ~/my-ec2-key-pair.pem hadoop@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

Launch using `pyspark`

We will launch jupyter notebook via pyspark. The following bash script can easily launch it for you. Alternatively, if you prefer, you can type out these commands in the terminal.

# Create a script titled "launch-jupyter.sh" with the following contents on EMR master node.

#! /usr/bin/env bash
export PYTHONPATH="/home/hadoop/.local/lib/python2.7/site-packages:$PYTHONPATH"
export PYSPARK_DRIVER_PYTHON=/home/hadoop/.local/bin/jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --no-browser'
echo $PYTHONPATH
echo $PYSPARK_DRIVER_PYTHON
echo $PYSPARK_DRIVER_PYTHON_OPTS
pyspark

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ chmod u+x launch-jupyter.sh
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ ./launch-jupyter.sh

You should see a message like the one below printed in your terminal. The ssh terminal will continue to be responsive and will print useful messages. You must keep it open as long as you want to use Jupyter notebook. See the “Exiting properly” section below for instructions on how to properly exit without losing data or accidentally killing the Spark cluster.

# Output of pyspark (on EMR master node)
...
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
    http://localhost:8888/?token=s0m3l0ngalphanumer1cstr1ng
...

We need to setup port forwarding before we can access the URL above. Make note of the port number (8888 in this example) printed in the URL. This is the port at which the jupyter notebook is served. We will need to use this port number to setup port forwarding.

Aside: ipython terminal instead of jupyter notebook

If you want to use ipython, then you don’t need to setup port forwarding. You need to simply install ipython on the master node and then use different environment variables. A benefit of using this method is that you can use ipython instead of a regular python REPL. The ipython session will come with all the bells and whistles such as syntax-highlighted output, tab-completion, and “?help”.

# On EMR master node
pip install --user ipython

# Create a script titled "launch-ipython.sh" with the following contents on EMR master node.

#! /usr/bin/env bash
export PYTHONPATH="/home/hadoop/.local/lib/python2.7/site-packages:$PYTHONPATH"
export PYSPARK_DRIVER_PYTHON=/home/hadoop/.local/bin/ipython
echo $PYTHONPATH
echo $PYSPARK_DRIVER_PYTHON
pyspark

# On EMR master node
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ chmod u+x launch-ipython.sh
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ ./launch-ipython.sh

Setup port forwarding

Remember that mosh cannot be used to setup port forwarding; we will have to use ssh to do that. The following command will setup port forwarding as a background process. See this blog post for more details.

# On local machine

# If you setup the entry in ~/.ssh/config, then you can simply type
ssh -N -f -L 8888:localhost:8888 emr

# If you didn't setup the entry in ~/.ssh/config, use the full command
ssh -N -f -L 8888:localhost:8888 -i ~/my-ec2-key-pair.pem hadoop@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

This sets up your local machine’s port 8888 to listen to the master node’s port 8888. From the previous step, we know that the jupyter notebook is being served on port 8888 on the master node. Typically, the port 8888 is open on local machine but if it’s not open, try another port. As a side note, the port 8888 does not conflict with the other default EMR ports. The AWS-provided JupyterHub notebook is served on port 9443.

If you don’t need to use jupyter notebook and want to use ipython or python terminal instead, then you don’t need to setup port forwarding.

Open jupyter notebook in a browser

Go to URL printed on the terminal screen previously. You should see a jupyter notebook homepage. The ssh terminal will print useful messages and you must keep it open.

Find your bearings

From the jupyter notebook homepage, click on “New” and open a terminal. Type out these commands to check things out. Note that this terminal provides sudo-level access to the master node of your EMR cluster. Be careful!

# On terminal within the jupyter browser tab
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ whoami
hadoop
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ pwd
/home/hadoop
[hadoop@ip-xxx-xxx-xxx-xxx ~]$ echo $HOSTNAME
ip-xxx-xxx-xxx-xxx

Test Spark variables

From the jupyter notebook homepage, click on “New” and open a “Python 2” notebook. The spark-related variables should already be available.

Import packages

You can import any package that you installed previously. You install more packages in using the instructions provided previously. You may need to restart your ssh or jupyter session (in some cases) before you can import newly installed packages. As an example, you can import pandas and matplotlib.

Test jupyter features

Jupyter notebook comes with all the (default) bells and whistles. You can make and display plots. The dataframes are printed in a pretty table format. Tab-completion works and “?sc” help is available.

Exiting properly

Proper shutdown is important with an AWS EMR cluster.

Shutting down notebooks in browser

Save your notebook
Click on File > Close and halt. This closes your spark session and releases resources (memory and disk space) back.
Once you have closed and halted all notebooks, click on Quit on the main page of the jupyter notebook. This should gracefully exit the jupyter processes running on the master node of the EMR cluster. You should see the jupyter process end in your ssh/mosh terminal.

If you lose a connection to your jupyter notebook, it does not automatically close out the jupyter processes. Those processes might still be running and consuming resources. To close them out, search for the appropriate processes like this.

# On EMR master node
ps aux | grep "jupyter"

If you see jupyter processes running, you can kill them using kill <pid>. You usually only have to kill one parent process which kills the child processes as well.

CAUTION: Be careful when killing processes. Make sure you don’t accidentally kill someone else’s jupyter session (which may have the same username hadoop). If you kill someone else’s jupyter process, they may lose all their work.

Shutting down port forwarding

After you’ve shutdown the jupyter notebook on the master node, we can shutdown the port forwarding on the local machine. Simply grep and kill the process.

# On local machine

# Find the appropriate process id. You should see your username and the exact string you typed in.
ps aux | grep "ssh"

# Kill
kill <pid>

Downloading jupyter notebooks and data

This step is very important and often overlooked. The notebooks reside on the master node of the EMR cluster. Any data that you saved to disk using the jupyter notebook is also saved on the master node unless you explicitly saved the data somewhere else.

The entire EMR cluster, including the master node, is ephemeral. Leaving the notebook and data on the master node is extremely risky. It’s possible that you or someone else accidentally terminates the EMR cluster. Another common scenario is when someone runs a spark job on the same EMR cluster which results in a failure which, in turn, causes the cluster to terminate. An even more common scenario is when someone (which could be you) forgets about a running spark job (perhaps via a zombie jupyter process). While this spark job is running, various nodes are still holding data (in memory or on disk). If someone decides to downsize the EMR cluster at this point, EMR tries to put all the data on to the surviving nodes which can very easily kill the entire cluster if the data is too large.

I highly recommend that you download your notebooks and data or upload them to S3 as soon as you’re done with your work.

To be continued …

This is enough for one post. Stay tuned for another post in which we will actually analyze the data using our newly created Spark cluster.

Always on top in MacOS Sierra

Sun, 23 Oct 2016 00:00:00 +0000

Afloat is a software that allows some Mac application windows to remain on top of other windows even when they are not in focus. Hence the name Always on Top. This is a standard feature for all windows in Ubuntu but in Mac we need to use a third-party software like Afloat to get this functionality. As of writing this post, Afloat is the only software I have seen that can let us do this on a Mac. If you want to learn more about Always on Top or Afloat, I recommend reading my original post (with pictures) on this topic.

The original post contained instructions on how to setup Always on Top using Afloat on MacOS Yosemite and El Capitan. Those instructions have become really old and do not work on Mac OS Sierra (10.12). This post presents a new set of instructions on how to enable Always on Top in MacOS Sierra. If you have an older MacOS, please follow the instructions in the original post.

The instructions are provided as General Steps and Specific Instructions. For each general step, I have provided at least one specific set of instructions that worked for me. You may choose to perform specific instruction in different ways, depending on what resources you already have available.

General Steps

Install SIMBL
Obtain Afloat.bundle folder
Drop Afloat.bundle in SIMBL plugins folder.
Test, restart (if necessary), test, restart (to re-enable SIP), test

Specific Instructions

1. Install SIMBL

Previously, for MacOS Yosemite and El Capitan, SIMBL 0.9.9 worked but it does not seem to work for MacOS Sierra. While there could be many sources of SIMBL, mySIMBL works for me on MacOS Sierra and it is easy to install. Here are the specific steps that worked for me.

Install mySIMBL. You can simply follow the instructions in mySIMBL README. During the installation of mySIMBL, it might ask you to disable System Integrity Protection (SIP), which requires you to boot in Recovery Mode. mySIMBL will instruct you if you need to do this and how to do it. The only thing I have to add is that you only need to disable SIP for installing mySIMBL. Once mySIMBL and Afloat are installed, you can re-enable SIP. Instructions for re-enabling SIP are provided in this post below.

Alternative to downloading mySIMBL zip file from GitHub, you can use MacUpdate if you trust it.

2. Obtain `Afloat.bundle`

These are multiple methods to obtaining Afloat.bundle folder. Once you obtain this folder, I recommend that making a more long-term backup of this folder (Google Drive, for example) for future use.

Method 1: Search if you already have it. You might have Afloat.bundle from previous Afloat installation. That bundle should work nicely. If you have installed Afloat before, don’t uninstall or delete it yet. Find the folder Afloat.bundle on your computer and copy it somewhere safe. Some places to look for it are /Library/Application Support/SIMBL/Plugins and ~/Library/Application Support/SIMBL/Plugins.

Method 2: Download from the Internet. If you have never installed Afloat before, then you need to obtain Afloat.bundle. You can find this folder in Rocky Wu’s GitHub repository here. You can either clone the afloat repository using git or you can download the afloat folder as a zip file and unzip it. Once you have cloned/downloaded the afloat folder, navigate to afloat/bundle/Afloat.bundle. You will need this folder later.

Method 3: Build it yourself (difficult). You can clone the Afloat repository and use Xcode to build it yourself. Check your deploy location for a Afloat.bundle folder.

2. Drop `Afloat.bundle` in SIMBL plugins folder

Run mySIMBL using the MacOS search bar (pressing Command+Space and typing in mySIMBL).
Copy/paste the Afloat.bundle folder into the plugins folder /Library/Application Support/SIMBL/Plugins. mySIMBL will also provide this instruction to you.
Ensure that Afloat appears in mySIMBL Plugins window and that there is a green dot next to it, as shown in the picture below.

4. Test, restart (if necessary), test, restart (to re-enable SIP), test.

a. Test, restart (if necessary), test. Restart any windows on which you want to use Afloat. Click on the Window menu to see if you can see this kind of menu.

Note that not all applications support Afloat. I can confirm that Afloat works on iTerm2. Press the Afloat key (Control+Command+A by default) and test it out to ensure that it works. If you don’t see it yet, don’t give up yet. There may still be hope. Restart your computer and test again.

b. Restart to re-enable SIP (optional but recommended), test. If you have SIP disabled, you can re-enable it. You should do this after you have ensured that Afloat is working for you. Re-enabling SIP does not stop Afloat from working. Here are the steps to re-enable it.

Reboot into Recovery Mode by restarting and holding down Command+R.
Click Utilities > Terminal.
Type in csrutil enable; reboot and press Enter.
Restart your Mac.
Test Afloat again.

Troubleshooting

Restarting computer usually helps immensely with everything. Often, after a few updates, Afloat stops working for me. In most cases, simple restart fixes the problem. It’s only when MacOS changes named versions that you things get really messy.

Good luck!

Install xml2 R package on MacOS

Fri, 08 Apr 2016 00:00:00 +0000

Installing xml2 R package often fails due to missing or incompatible library issues. In this post, I describe why this problem occurs and provide two solutions to solve this problem.

What goes wrong?

xml2 R package depends on libxml2. When you install xml2 using install.packages and default options, xml2 package queries your system to determine where libxml2 headers and library files are located (more on this later). The error occurs when libxml2 is not found or a wrong version of libxml2. This is how the error looks.

> install.packages("xml2")
...
* installing *source* package ‘xml2’ ...
** package ‘xml2’ successfully unpacked and MD5 sums checked
Found pkg-config cflags and libs!
Using PKG_CFLAGS=-I/Users/<username>/anaconda/include/libxml2
Using PKG_LIBS=-L/Users/<username>/anaconda/lib -lxml2 -lz -liconv -lm
** libs
clang++ -I/usr/local/Cellar/r/3.2.2_1/R.framework/Resources/include -DNDEBUG -I/usr/local/include -I/Users/<username>/anaconda/include/libxml2 -I/usr/local/opt/gettext/include -I/usr/local/opt/readline/include -I/usr/local/opt/openssl/include -I/usr/local/include -I"/usr/local/lib/R/3.2/site-library/Rcpp/include" -I"/usr/local/lib/R/3.2/site-library/BH/include" -I/usr/local/include   -fPIC  -g -O2  -c RcppExports.cpp -o RcppExports.o
...
installing to /usr/local/lib/R/3.2/site-library/xml2/libs
...
Error in dyn.load(file, DLLpath = DLLpath, ...) :
  unable to load shared object '/usr/local/lib/R/3.2/site-library/xml2/libs/xml2.so':
  dlopen(/usr/local/lib/R/3.2/site-library/xml2/libs/xml2.so, 6): Library not loaded: libxml2.2.dylib
  Referenced from: /usr/local/lib/R/3.2/site-library/xml2/libs/xml2.so
  Reason: image not found
Error: loading failed
Execution halted

What happens behind the scenes?

xml2 package contains a bash script called configure which determines the location of libxml2. You can take a look at this configure on GitHub. This script first attempts to use xml2-config to determine the location. On my system, xml2-config is installed by anaconda.

$ which xml2-config
/Users/<username>/anaconda/bin/xml2-config

On my system, xml2-config tool points to the libxml2 installed by anaconda, which happens to be incompatible with R xml2 package.

$ xml2-config --libs
-L/Users/<username>/anaconda/lib -lxml2 -lz -liconv -lm
$ xml2-config --cflags
-I/Users/<username>/anaconda/include/libxml2

If xml2-config is not available, then configure script checks for libxml2 using the pkg-config tool.

$ which pkg-config
/usr/local/bin/pkg-config

Again, in my case, this pkg-config tool points to the MacOS system libxml2 headers and libraries. These may or may not be compatible with R xml2 package. In my case, these happened to be compatible.

$ pkg-config --cflags libxml-2.0
-I/usr/include/libxml2
$ pkg-config --libs libxml-2.0
-lxml2

We need to make sure that the /usr/include/libxml2 location actually exists. If this location does not exist, then first solve this problem as described in the footnotes¹.

Overall, the solution is based on the simple principle

Use the correct location of libxml2 headers and libraries.

There are various ways to solve this problem. I describe two ways in this post.

Possible solutions

For both solutions, we will need to download the source code of xml2 package from CRAN and install the package from source. Extract the original source code file and navigate to the extracted xml2 folder.

Solution 1

We will modify the configure script in the source code. The modification will cause configure script to use the libxml2 location provided by pkg-config. This can be done by commenting out the following relevant lines in the configure file

# Use xml2-config if available
if [ $(command -v xml2-config) ]; then
  PKGCONFIG_CFLAGS=$(xml2-config --cflags)
  PKGCONFIG_LIBS=$(xml2-config --libs)
elif [ $(command -v pkg-config) ]; then
  PKGCONFIG_CFLAGS=$(pkg-config --cflags $PKG_CONFIG_NAME)
  PKGCONFIG_LIBS=$(pkg-config --libs $PKG_CONFIG_NAME)
fi

and replacing them by these lines

if [ $(command -v pkg-config) ]; then
  PKGCONFIG_CFLAGS=$(pkg-config --cflags $PKG_CONFIG_NAME)
  PKGCONFIG_LIBS=$(pkg-config --libs $PKG_CONFIG_NAME)
fi

I recommend reading the code and then made the changes if you understand them.

Now that we have modified the source code of the package, we need to re-build a new .tar.gz file.

# Navigate to inside the xml2 folder
~/Downloads/xml2 $ R CMD build .
# This will create a xmlx_x.x.x.tar.gz file in the current folder

and then install this package from from source

# Start R in the folder that contains the xmlx_x.x.x.tar.gz file
> install.packages("xml2_0.1.2.tar.gz", repos=NULL, type="source")

If the location of libxml2 as specified by pkg-config is correct, then this solution should correctly install xml2 package.

Solution 2

If you do not want to modify the configure script and you know the correct location of libxml2 header and libraries, then you can simply perform a custom install of xml2 package by using the following command (as specified by the configure script)

# Navigate to inside the xml2 folder
R CMD INSTALL --configure-vars='INCLUDE_DIR=/usr/include LIB_DIR=/usr/lib' .

The above locations for INCLUDE_DIR and LIB_DIR correspond to the default MacOS locations but you may need to modify these. You can find out these locations using the tips provided in footnotes². This should successfully install xml2 package. This form of solution works correctly for other packages as well, albeit with some modifications³.

Footnotes

The exact options required by configure may be seen by running ./configure --help within the source directory of the package. Note that if you have run configure within the source folder using the incorect options, then you might see R CMD INSTALL fail even with the correct options. This problem can be solved by re-running configure using the correct options or getting a fresh copy of the source.

Missing /usr/include on MacOS

Sometimes, /usr/include does not exist on MacOS because Xcode did not install correctly, as mentioned in this Stack Overflow post. The solution is simple, just run this command and follow instructions.
```
xcode-select --install
```
↩

libxml2 metadata file

The configure script requires libxml-2.0.pc file at the location where libxml2 is installed. You can look for all libxml-2.0.pc files on your system

$ locate libxml-2.0.pc
/Users/<username>/anaconda/lib/pkgconfig/libxml-2.0.pc
/Users/<username>/anaconda/pkgs/libxml2-2.9.0-1/lib/pkgconfig/libxml-2.0.pc
/opt/vagrant/embedded/lib/pkgconfig/libxml-2.0.pc
/usr/local/Library/ENV/pkgconfig/10.10/libxml-2.0.pc
/usr/local/Library/ENV/pkgconfig/10.11/libxml-2.0.pc
...

pkg-config looks at these .pc metadata files to retrieve information, as mentioned in the manual pages of pkg-config

$ man pkg-config
...
pkg-config retrieves information about packages from special metadata files.
These files are named after the package, and has a .pc extension.
...

These libxml-2.0.pc files contain location of libxml2 headers and libraries. This information is usually very helpful.

$ more /usr/local/Library/ENV/pkgconfig/10.11/libxml-2.0.pc
prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
modules=1

Name: libXML
Version: 2.9.2
Description: libXML library version2.
Requires:
Libs: -L${libdir} -lxml2
Libs.private: -lpthread -lz  -lm
Cflags: -I${includedir}/libxml2

A similar error somtimes occurs while installing git2r. On my system, the error was due to incorrect location of zlib library. Solution 2 works but the exact command is slightly different:

# Navigate to inside the git2r source folder
R CMD INSTALL --configure-args='--with-zlib-include=/usr/include --with-zlib-lib=/usr/lib' .

↩

Installing git2r ↩

Sublime-style multiple cursors in Jupyter

Sat, 19 Mar 2016 00:00:00 +0000

Jupyter Notebooks

Jupyter Notebooks are great for visualizing and sharing results with others.

Learning the keyboard shortcuts tremendously improves productivity while using Jupyter Notebooks. By default, you can see the keyboard shortcuts help dialog window by first pressing Escape to enter the Command Mode and then pressing h.

While these keyboard shortcuts are very useful, I miss the multiple cursor functionality from Sublime Text. This functionality allows you to select (and then edit) multiple instances of a word visually. This is incredibly useful while coding. Renaming a variable via multiple cursors is easy, safe, and very satisfying. See the Sublime Text homepage or see below for an animated demo.

A Python expert informed me that this functionality was added to Jupyter Notebooks after switching to Code Mirror 4 but it requires setting up a Sublime Text keymap. After searching a bit for instructions to accomplish this task, I found two GitHub issues (this and this) that describe how to do this. After some tinkering, I was able to make this work. I have consolidated these instructions below so it’s easier for others.

(Update: Excitingly, there is an attempt in progress to allow the users to switch keymaps using the “Edit” menu. Until this pull request gets merged and rolled out in a new release, the following instructions would be helpful.)

Overall, this task is simple. To set up a Sublime Text keymap, you need to add a few lines of Javascript to a custom.js file. Setting up a custom.js file for Jupyter Notebook is described in detail here.

Instructions

Find the location of custom.js file. On MacOS and Linux, the default location is ~/.jupyter/custom/custom.js. If this is your first time setting up a custom.js, this file will probably not exist in that location. Optionally, you can run the following code in a Jupyter Python Notebook to find the location and contents of your custom.js file :

 # Print the location of Jupyter's config directory
 from jupyter_core.paths import jupyter_config_dir
 jupyter_dir = jupyter_config_dir()
 print(jupyter_dir)

 # Print the location of custom.js
 import os.path
 custom_js_path = os.path.join(jupyter_dir, 'custom', 'custom.js')
 print(custom_js_path)

 # Print the contents of custom.js, if it exists.
 if os.path.isfile(custom_js_path):
     with open(custom_js_path) as f:
         print(f.read())
 else:
     print("You don't have a custom.js file")

If custom.js does not exist at the location found in Step 1, create it. As a quick (optional) check that the custom.js indeed has an effect, add the following line to it (preferably at the top of the file):
```
 alert("hello world from custom.js")
```
Restart Jupyter Notebook server (by pressing Control+C in the terminal, if applicable). If everything is working correctly, you should be greeted by a dialog window in your browser after you restart your Jupyter Notebook.

You can comment out the line you added in Step 2. Add the following lines to your custom.js (preferably at the top of the file):

 require(["codemirror/keymap/sublime", "notebook/js/cell"], function(sublime_keymap, cell) {
     cell.Cell.options_default.cm_config.keyMap = 'sublime';
 });

Restart Jupyter Notebook again.

Test out multiple cursors by selecting some text and then pressing Control+D (on Linux or Windows) or Command+D (on MacOS).

A new look using Trio

Thu, 24 Dec 2015 00:00:00 +0000

I decided to switch the theme of my Jekyll blog from Pixyll to Trio. Pixyll is a great theme and I learned a lot from using it. I decided it was time for a change. I wrote Trio as a hobby and to learn how Jekyll works in detail.

The content of the blog is still the same. All the comments on the respective pages are preserved. The URLs to these pages should also be the same as before. I am grateful to all the readers of my blog. If you notice a mistake – missing content or comments, broken URLs, images, incorrect rendering of code or anything else, please let me know in the comments below or through the contact form.

I designed Trio to be more content-focused, responsive and mobile friendly. Trio specially focuses on displaying code. Trio code blocks are clean and crisp. In-line back-quoting also looks nicer. GitHub Gists can also be easily embedded.

Trio comes with good defaults. Markdown footnotes work out of the box. Tables are formatted automatically to have gray-white shading in alternative rows. Additionally, Trio comes with these features out-of-the-box:

Disqus integration
Social media sharing buttons on every post
Nine color themes
Formspree contact form
Support for Font Awesome icons
Full-text search

You can read more about Trio on its demo site.

Time is TRUE, Female is FALSE

Mon, 31 Aug 2015 00:00:00 +0000

Introduction

R has two reserved words denoting logical constants, namely TRUE and FALSE. These are case sensitive literals and you cannot create a variable or a function named TRUE or FALSE.

> TRUE <- 1
Error in TRUE <- 1 : invalid (do_set) left-hand side to assignment
> TRUE <- function() print("hello")
Error in TRUE <- function() print("hello") :
  invalid (do_set) left-hand side to assignment
> FALSE <- 1
Error in FALSE <- 1 : invalid (do_set) left-hand side to assignment
> FALSE <- function() print("hello")
Error in FALSE <- function() print("hello") :
  invalid (do_set) left-hand side to assignment

R also has two global variables, namely, T and F whose initial values are set to TRUE and FALSE respectively¹. This allows us to write the following kind of code:

> rm(list = ls())
> if (T) print("Yes") else print("No")
[1] "Yes"
> if (F) print("Yes") else print("No")
[1] "No"

Since T and F are simply global variables, we can see their values by typing out thee variable names:

> T
[1] TRUE
> F
[1] FALSE

We can also re-assign new values to these variables:

> T <- 1:10
> F <- "female"
> T
 [1]  1  2  3  4  5  6  7  8  9 10
> F
[1] "female"

After we have reassigned to these variables², we can also delete these variables using rm():

# After having reassigned to T and F
> rm(T)
> rm(F)
> T
[1] TRUE
> F
[1] FALSE

You can already imagine the ambiguity in using T and F instead of TRUE and FALSE. Since, we can easily redefine T and F, we run the risk of unexpected, silent mistakes in our code³. This problem is further exacerbated by R’s lexical scoping rules, which we discuss next.

Lexical scoping rules in R

Scoping is a set of rules that allow a programming language to infer the value of a variable from the variable name. There are two main types of scoping - lexical and dynamic. R has support for both these kinds of scoping rules. We will only look at lexical scoping through examples in this post. I recommend reading Hadley Wickham’s Advanced R online section on Functions and this github page for details. Interested reader may also want to see this post purportedly by Ross Ihaka, one of the creators of R.

Let’s look at a simple example in which we define a function and try to call it:

# In a fresh R session
> f <- function() print(x)
> f
function() print(x)
> f()
Error in print(x) : object 'x' not found

Note that we do not need to define the variable x before we define the function f. However, we need to define a reachable variable x before we call the function f. By reachable, I mean, the function f should be able to reach the variable x. In other words, the variable x should be within the scope of the function f.

Let’s first look at the case in which we define a variable x in such a way that it is not reachable by f.

> g <- function() { x <- 1 }
> g()
> print(g())
[1] 1
> f()
Error in print(x) : object 'x' not found

In the above example, we did define x but inside the function g(). The function f() cannot reach the variable x and we get an error when we try to call f(). Now, let’s define x in the global environment

> x <- 10
> f()
[1] 10

Now, the variable x defined in the global environment is reachable by f() and we can successfully call f(). Note that we haven’t passed x through the function f(). This is a common feature of a functional programming language such as R. Furthermore, we can change the behavior of f() without actually changing f() at all!

> x <- 20
> f()
[1] 20

This means that functions in R are not totally isolated from the environment the function is defined in. In fact, the function and the environment the function is defined in are very tightly related (see closures in Functional Programming if you want to know the details). This kind of scoping rule is not used in other high-level, numeric programming languages such as Octave⁴.

We can take this experiment further by calling f() form inside another function:

> x
[1] 20
> h <- function() { x <- 30; f() }
> h()
[1] 20

Now, even though we defined x to equal 30 inside the function h(), calling f() from inside h() still prints x as 20 (which was defined in the global environment). This is lexical scoping. As a quick experiment for the curious minded, what would happen if we did not even define x in the global environment ? This example shows the result:

# In a fresh R session
> f <- function() print(x)
> x
Error: object 'x' not found
> h <- function() { x <- 30; f() }
> f()
Error in print(x) : object 'x' not found

We get an error! Since x wasn’t defined in the environment of f(), x is still not reachable by f(). So, the behavior of f() depends upon two things:

Definition of f(), and
Environment in which f() was defined

To demonstrate this fact, let’s do another experiment. This time we define f() inside another function:

> hh <- function() { x <- 30; f <- function() print(x); f();}
> hh()
[1] 30
> x <- 10
> x
[1] 10
> hh()
[1] 30

This time the global environment does not influence the behavior of f(). We defined f() inside hh() and that’s the only environment that matters. To fully understand this concept, let’s so one final experiment:

# In a fresh R session
> hhh <- function() { f <- function() print(x); f();}
> hhh()
Error in print(x) : object 'x' not found
> x <- 10
> hhh()
[1] 10

This time we did not define x inside hhh(). Since in a fresh R session, there is no x defined, we get an error when we call hhh() the first time. Then, we define x in the global environment (which is not the environment f() was defined in!). Now when we call hhh() (and in turn, f()), we get no error.

What happened?

There was no x defined inside hhh() which is the environment in which f() was defined. How then was R able to find the global variable x ? This is because, when R does not find the variable x inside hhh(), it keeps on looking one level up, until it finds a match (or it doesn’t, in which case R throws an error).

This recursive scoping rule has a tremendous effect. We were able to modify the behavior of a function f() that was defined within another function hhh() without modifying either f() or hhh()!

In fact, we can modify the behavior of any function that uses a non-formal argument⁵ by modifying an appropriate parent environment of the function.

Now that we understand the scoping rules in R, let’s get back to the problem of T and F in R.

The Problem with `T` and `F`

Thee obvious benefit of using T and F instead of the full literals TRUE and FALSE is the reduced number of keystrokes. Admittedly, the reduced number of keystrokes required (with Shift pressed or CapsLock on) becomes very attractive when using the interactive mode. I am guilty of doing this myself.

Use of T and F is very dangerous, especially in programming mode. An obviously evil thing to do is to simply bind T to FALSE and F to TRUE and see everything fail (see Example 1). The dangers of using T and F can be far more subtler (see Example 2).

Example 1

Let’s define a simple dummy function:

# In a fresh R session
tf <- function() {
  print(T)
  if (T) {
    print("T")
  } else {
    print("not T")
  }
}
# Call tf()
> tf()
[1] TRUE
[1] "T"

This simple dummy function works as expected. Using the scoping rules of R, we can modify the behavior of the tf() function without changing tf() at all:

> # Now, redefine T and call tf()
> T <- FALSE
> tf()
[1] FALSE
[1] "not T"

Example 2

Let’s look at this problem in a more real-life context where we have multiple files being sourced.

You can download/clone these example files as a GitHub Gist here and play around with the code.

I first define a function oddmean() in the file oddmean.R. This function uses T and F as non-formal function arguments instead of the corresponding literals. We then source the file oddmean.R in various different files and see the effects. When T and F are bound to TRUE and FALSE correctly, we get the correct answer as shown in correctanswer.R. But, when we redefine T or F, we get incorrect answers as shown in wronganswer1.R and wronganswer2.R.

Let’s look at the file wronganswer2.R. We do not define any variables in this file at all. Instead, we simply source some of the files we already created and know to work correctly.

We obviously want to source the function oddmean() in oddmean.R.
Then, we source redefine.R that redefines T and F. In real life, this could be accidental or this could be someone else’s code that we wish to use.
Finally, we source the file correctanswer.R, which we have tested separately and we know that it gives the correct answer.

Now, even though we only sourced files that we knew to give the correct result in some circumstances, we still get the wrong result, simply because we sourced some other file!

What is even more concerning is that R did not throw an error! Due to the forgiving nature of R and the support for various kinds of indexing (logical, numeric, character; see this previous post), we get the wrong result silently. To the user, the incorrect value of 2.6 is not that far off from the correct value of 5 and this might go undetected. In fact, we don’t even know how often these silent errors occur. Maybe a large number of these errors go unnoticed!

The Solution

The solution is simple and obvious: use TRUE and FALSE instead of T and F. Let’s analyze this situation in terms of some common questions.

Why won’t we encounter the same problem with TRUE and FALSE?

We won’t have the same problem with TRUE and FALSE because these are reserved keywords or literals. R does not allow us to (re-)define a variable or a function named TRUE or FALSE.

Using TRUE and FALSE will make our code tamper-proof (for this particular kind of problem; this doesn’t solve the issues of lexical scoping in general).
Why don’t we simply rm() the global variables T and F ?

We can’t! See footnote².
Can I use T and F as logicals in interactive mode?

If you are sure that T and F are correctly defined then yes. This can be easily checked by quickly typing out T and F and ensuring the output matches TRUE and FALSE.
Are the extra keystrokes worth it?

Absolutely! The extra keystrokes required to fully type out TRUE and FALSE buy me some peace of mind. Even if I am sure that I did not define T and F anywhere in my code, it is possible that some other package that I’m using or some one else’s code or even my own code from a few years ago could have redefined these variables. The risk of incorrect results is too high to skimp on the keystrokes. And, any code that I write using T and F will forever be susceptible to this problem. I’d rather go to bed feeling good about my code then having avoided carpal tunnel.
Can I use T for time or F for female ?

In my opinion, no! Even if you decide to never use T and F as logicals, some other package or someone else who uses your code might.

Footnotes

¹ See ?T.

?T
...
‘TRUE’ and ‘FALSE’ are reserved words denoting logical constants
in the R language, whereas ‘T’ and ‘F’ are global variables whose
initial values set to these.  All four are ‘logical(1)’ vectors.
...

² We cannot remove the global variables T and F if we haven’t reassigned these variables.

# In a fresh R session
> rm(T)
Warning message:
In rm(T) : object 'T' not found
> rm(F)
Warning message:
In rm(F) : object 'F' not found
> T
[1] TRUE
> F
[1] FALSE

The above happens because both T and F are defined in the base package of R:

# In a fresh R session
> library(pryr)
> where("T")
<environment: base>
> where("F")
<environment: base>

It so happens that R does not allow us to remove variables from the base environment, as mentioned in the documentation of the rm() function:

?rm
...
It is not allowed to remove variables from the base environment
and base namespace, nor from any environment which is locked (see
‘lockEnvironment’).
...

As a result, we are stuck with T and F as always being reachable global variables, whether we have defined them explicitly or not.

However, once we reassign to T or F, we essentially create new variables in the R_GlobalEnv with these names.

> T <- 1:10
> library(pryr)
> where("T")
<environment: R_GlobalEnv>
> T
 [1]  1  2  3  4  5  6  7  8  9 10
> base::T
[1] TRUE

This does not mean that the original base::T (or base::F) variable is reassigned. The newly created T and F variables simply mask the original T and F variables in base package.

³ The title of this post refers to the ambiguity of the variables T and F. The variable name T is usually used to denote time, temperature, or a t-distributed random variable. Similarly, F could be used to denote gender (female), force, or an F-distributed random variable. Due to a simple error in coding, we could end up silently considering time as TRUE and female as FALSE. This title is adapted from from Section 8.3.5, Burns, Patrick. The R Inferno, 2011.

⁴Scoping rules in GNU Octave are not the same as R. Let’s try the same example but in Octave:

octave:2> function [] = f()
> disp(x)
> end
octave:3> f()
error: 'x' undefined near line 2 column 6
error: evaluating argument list element number 1
error: called from:
error:   f at line 2, column 1

Until here, both Octave and R have the same behavior. We can still define a function in Octave without passing the variable x through the function f(). And, as a result, we get the expected error indicating that x is not defined. Now, let’s define x in the global environment and see if we can call f() successfully:

octave:3> x = 10
x =  10
octave:4> f()
error: 'x' undefined near line 2 column 6
error: evaluating argument list element number 1
error: called from:
error:   f at line 2, column 1
octave:4> x
x =  10

This is where R and Octave differ. Even though x is defined in the global namespace, we cannot reach x from f() without explicitly passing x through x as a formal function argument.

octave:5> function [] = f(x)
> disp(x)
> end
octave:6> f(x)
 10

This means, in Octave, unlike R, a function is completely isolated from the environment it was defined in (for the most part, things do get complicated in some advanced cases).

⁵A formal argument of a function is the variable passed through the function signature. In the following example, y is a formal argument of the function mixf() but x is non-formal argument.

> mixf <- function(y) { print(x); print(y) }
> x <- 10
> mixf(20)
[1] 10
[1] 20

Some issues with scoping may be avoided by using only the formal arguments. However, since R is homoiconic and functional, enforcing formal arguments is not as satisfying as you’d think.

Furthermore, if you enforce the use of formal arguments (say, by defining a function inside an empty environment), this will make the function unusable (or extremely tedious) in practice. See the Dynamic lookup section here for an example.

This is because in R:

Even the basic operators such as ”+” are functions defined in the base package’s environment, and
R uses lexical scoping to find these functions as well (just like it does for variables).

Never trust rownames of a dataframe

Tue, 16 Jun 2015 00:00:00 +0000

Introduction

Dataframes in R have both column names and row names. Column names, which are used frequently, give the dataframes in R their characteristic distinction. Row names, on the other hand, are rarely used. Usually, row names appear to be the same as row numbers but this is not the case. This quick blog post demonstrates that row names are not the same as row numbers. This is something most experienced R users are well aware of.

> df <- cars[1:5, ]
> df
  speed dist
1     4    2
2     4   10
3     7    4
4     7   22
5     8   16
> rownames(df)
[1] "1" "2" "3" "4" "5"
> colnames(df)
[1] "speed" "dist"

A Tricky Situation

Let’s say we are working with R in the interactive mode. We have a dataframe that we are inspecting. We want to know various things about the dataframe such as how many rows and columns it has. We can print the dataframe like this:

> cars
   speed dist
1      4    2
2      4   10
...
48    24   93
49    24  120
50    25   85
>

We can see that cars has 2 columns named speed and dist. When we print the dataframe, we can see that the row names are printed as well. Looking at the rownames, it would appear that cars has 50 rows. Right?

This happens to be correct, in this case, as we can check using nrow(). But this is not true in general.

> nrow(cars)
[1] 50

Now, let’s look at another dataframe.

> cars[-18, ]
   speed dist
1      4    2
2      4   10
3      7    4
...
48    24   93
49    24  120
50    25   85

Now, if cars has 50 rows, cars[-18, ] contains 49 rows. But, by simply looking at the printed row names, we would see that the last row still has the row name “50” but cars[-18, ] has only 49 rows.

> nrow(cars[-18, ])
[1] 49

Row names are not row numbers. We should always use nrow() to determine the number of rows in a dataframe. Row names can be misleading, especially for large dataframes.

The numbers printed along side the columns of the dataframe are row names and not row numbers. What are row numbers and how do we distinguish row numbers from row names? The following helpful tip from Ryan Garner can help us answer this question.

Row names vs Row numbers

Let’s rearrange the rows of cars using order() function.

> df <- cars[order(cars$dist), ]
> df
   speed dist
1      4    2
3      7    4
2      4   10
6      9   10
12    12   14
...
47    24   92
48    24   93
49    24  120

The numbers 1, 3, 2, 6, 12 ..., 47, 47, 49 are row names. Row numbers (or row indices) are simply the consecutive sequence of integers from 1 to nrow(df). Row numbers are never printed along side the dataframe. The second row (i.e. row number 2) of df in this example may be obtained by

> df[2, ] # Row number 2
  speed dist
3     7    4

As we can see, the second row (i.e. row number 2) has row name "3". We can obtain the same row of df by using the appropriate row name

> df["3", ]  # Row name "3"
  speed dist
3     7    4

However, the row name "2" is not the same as row number 2.

> df["2", ]  # Row name "2"
  speed dist
2     4   10

In the construct, df[x, ], we access either the row number or the row name depending upon the class of x. If x is numeric or integer, we access the row number x. If x is character, we access the row name x. It is easy to confuse row numbers for row names and vice versa.

Row logicals vs Row numbers vs Row names

(Update August 15, 2015)

There is yet another way to index the rows of a dataframe – using logicals.

> df <- cars[1:5, ]
> df[rep(TRUE, 5), ]
  speed dist
1     4    2
2     4   10
3     7    4
4     7   22
5     8   16
> df[rep(FALSE, 5), ]
[1] speed dist
<0 rows> (or 0-length row.names)

Each row of the dataframe is selected if there is a corresponding TRUE value for that row. If the corresponding value is FALSE, then that row is not selected. This is logical indexing. It seems easy enough and very useful too, as shown below:

> df$speed == 4
[1]  TRUE  TRUE FALSE FALSE FALSE
> df[df$speed == 4, ]
  speed dist
1     4    2
2     4   10

Things become complicated when we start combining logical indexing with R’s vector recycling behavior. What would happen if the logical vector we use to index the rows of df is shorter than 5 ?

> df[TRUE, ]
  speed dist
   4    2
   4   10
   7    4
   7   22
   8   16

On the face of it, df[TRUE, ] makes no sense. df has 5 rows and we have only one logical index (i.e. TRUE). Which row of df should be selected?

R could throw an error! But, R is forgiving. It doesn’t throw an error. Instead, R resolves this problem by employing vector recycling rules. The logical vector TRUE is repeated 5 times such that each row has df[TRUE, ] produces the same result as df[rep(TRUE, 5), ]. Let’s look at another example.

> df[c(TRUE, FALSE), ]
  speed dist
1     4    2
3     7    4
5     8   16

Now, the vector c(TRUE, FALSE) is repeated, which creates the effect of basically selecting every odd row of the dataframe df. For some users, this might even be useful.

So, along with the row number indexing, row name indexing, we also have logical indexing. The kind of indexing used depends on the type of index. Row name indexing is used if the index is numeric (or integer), row name indexing is used when the index is character and logical indexing is used when the index is logical. Another distinction is that only logical indexing employs vector recycling (if needed). Row name indexing and row number indexing do not use recycling (as shown in the next section).

What about missing values?

R has missing values in every data type.

> class(NA)
[1] "logical"
> class(as.character(NA))
[1] "character"
> class(as.numeric(NA))
[1] "numeric"
> class(as.factor(NA))
[1] "factor"

By default, NA is a logical. So, what happens when we row-index a dataframe using NA ? It depends on the type of NA.

> df[NA, ]
     speed dist
NA      NA   NA
NA.1    NA   NA
NA.2    NA   NA
NA.3    NA   NA
NA.4    NA   NA

Since NA is logical by default, logical indexing (with vector recycling) is used in the above example.

However, if we use row number indexing or row name indexing:

> df[as.numeric(NA), ]
   speed dist
NA    NA   NA
> df[as.character(NA), ]
   speed dist
NA    NA   NA

Again, using as.numeric(NA) to index a dataframe makes no sense. Are we trying to obtain a missing row of the dataframe? R resolves this by returning rows full of NAs. No vector recycling is done, which means that we get back one row back for each element in the index.

> df[rep(as.character(NA), 3), ]
     speed dist
NA      NA   NA
NA.1    NA   NA
NA.2    NA   NA

(Thanks to Ryan Brady for pointing out this issue).

This example naturally leads us to a question about uniqueness of row names. Are row names necessarily unique? The answer is yes, as we will see in the next section. Row names, much like column names, can be modified as we can see in the next section. However, row names differ from column names in one aspect - row names are always unique (in contrast, two columns in a dataframe may have the exact same name). Uniqueness of row names ensures that df["2", ] will return a dataframe with exactly one row (even if the row name "2" does not exist but more on that in another post).

Behavior of Row Names

These examples demonstrate the behavior of row names.

Row names may be assigned any unique character or numeric values.

 > df <- cars[1:5, ]
 > rownames(df)
 [1] "1" "2" "3" "4" "5"
 > rownames(df) <- letters[1:5]
 > df
 speed dist
 a     4    2
 b     4   10
 c     7    4
 d     7   22
 e     8   16
 > rownames(df)
 [1] "a" "b" "c" "d" "e"

Attempt to set non-unique row names fails. But this doesn’t mean that the row names of a dataframe cannot be made non-unique (see the next section).

 > rownames(df) <- rep(1, 5)
 Error in `row.names<-.data.frame`(`*tmp*`, value = value) :
   duplicate 'row.names' are not allowed
 In addition: Warning message:
 non-unique value when setting 'row.names': ‘1’

Row names cannot be assigned NA. More on NAs below.

 > df <- cars[1, ]
 > df
   speed dist
 1     4    2
 > rownames(df)[1] <- NA
 Error in `row.names<-.data.frame`(`*tmp*`, value = value) :
   missing values in 'row.names' are not allowed

Output of rownames() is character, even if we set numeric row names.

 > rownames(df) <- runif(5)
 > df
                    speed dist
 0.444560116855428      4    2
 0.0545551690738648     4   10
 0.633737650699914      7    4
 0.691298563731834      7   22
 0.784601176623255      8   16
 > rownames(df)
 [1] "0.444560116855428"  "0.0545551690738648" "0.633737650699914"
 [4] "0.691298563731834"  "0.784601176623255"
 > class(rownames(df))
 [1] "character"

Row names may be reset by assigning NULL. After this step, the row names are simply character-version of row numbers.

 > df <- cars[-18, ]
 > rownames(df) <- NULL
 > df
    speed dist
 1      4    2
 2      4   10
 3      7    4
 ...
 47    24   93
 48    24  120
 49    25   85
 > rownames(df)
  [1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14" "15"
 [16] "16" "17" "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30"
 [31] "31" "32" "33" "34" "35" "36" "37" "38" "39" "40" "41" "42" "43" "44" "45"
 [46] "46" "47" "48" "49"

Row names may be set individually.

 > df <- cars[1:5, ]
 > df
   speed dist
 1     4    2
 2     4   10
 3     7    4
 4     7   22
 5     8   16
 > rownames(df)[1] <- ""
 > df
   speed dist
       4    2
 2     4   10
 3     7    4
 4     7   22
 5     8   16
 > rownames(df)[2:4] <- letters[1:3]
 > df
   speed dist
       4    2
 a     4   10
 b     7    4
 c     7   22
 5     8   16

Even though row names appear to be “reset” or belong to character class, row names are stored as-assigned. Let’s see this using an example.

 > df <- cars[1:5, ]
 > str(df)
 'data.frame':   5 obs. of  2 variables:
  $ speed: num  4 4 7 7 8
  $ dist : num  2 10 4 22 16
 > rownames(df) <- letters[1:5]
 > df
   speed dist
 a     4    2
 b     4   10
 c     7    4
 d     7   22
 e     8   16
 > str(df)
 'data.frame':   5 obs. of  2 variables:
  $ speed: num  4 4 7 7 8
  $ dist : num  2 10 4 22 16

First thing to notice is that str() does not show row names, even if the row names are not the default sequence of 1 to nrow(df). If str() does not show us the row names, then how are the row names stored?

There are two functions that can can help us answer this question: dump() and attributes().

 > df <- cars[1:5, ]
 > dump("df", "")
 df <-
 structure(list(speed = c(4, 4, 7, 7, 8), dist = c(2, 10, 4, 22,
 16)), .Names = c("speed", "dist"), row.names = c(NA, 5L), class = "data.frame")

dump() serializes the object and prints it as a string. We can clearly see that row names are stored in the row.names attribute of the object.

In the above example, it appears that row names are c(NA, 5L)!

Interesting.

So, row names cannot be assigned NA (as we saw in the examples earlier), but they seem to be stored as something that contains NA.

Another function that lets us look at the internals of an R object is attributes().

 > df <- cars[1:5, ]
 > attributes(df)
 $names
 [1] "speed" "dist"

 $row.names
 [1] 1 2 3 4 5

 $class
 [1] "data.frame"

 > attributes(df)$row.names
 [1] 1 2 3 4 5
 > class(attributes(df)$row.names)
 [1] "integer"

Even more interestingly, the row names appear to be integer and not NA!

Oh, the quirks of R.

Anyways, now, let’s look at a dataframe with explicitly assigned row names.

 > df <- cars[1:5, ]
 > df
   speed dist
 1     4    2
 2     4   10
 3     7    4
 4     7   22
 5     8   16
 > rownames(df) <- letters[1:5]
 > df
   speed dist
 a     4    2
 b     4   10
 c     7    4
 d     7   22
 e     8   16
 > dump("df", "")
 df <-
 structure(list(speed = c(4, 4, 7, 7, 8), dist = c(2, 10, 4, 22,
 16)), .Names = c("speed", "dist"), row.names = c("a", "b", "c",
 "d", "e"), class = "data.frame")
 > attributes(df)
 $names
 [1] "speed" "dist"

 $row.names
 [1] "a" "b" "c" "d" "e"

 $class
 [1] "data.frame"

 > attributes(df)$row.names
 [1] "a" "b" "c" "d" "e"
 > class(attributes(df)$row.names)
 [1] "character"

In this case, the row names remain consistent no matter how we look at them.

Creating a new dataframe by extraction does not automatically reset row names. If you want the row names to match the row numbers, reset row names by assigning to NULL.

 > df <- cars[c(1, 18, 50), ]
 > df
    speed dist
 1      4    2
 18    13   34
 50    25   85
 > rownames(df)
 [1] "1"  "18" "50"

We can add two dataframes that have different row names without an error or a warning. The row names of the first operand of + are preserved.

 > df <- cars[c(1, 18, 50), ]
 > df2 <- cars[c(2, 19, 35), ]
 > df
    speed dist
 1      4    2
 18    13   34
 50    25   85
 > df2
    speed dist
 2      4   10
 19    13   46
 35    18   84
 > df3 <- df + df2
 > df3
    speed dist
 1      8   12
 18    26   80
 50    43  169
 > df4 <- df2 + df
 > df4
    speed dist
 2      8   12
 19    26   80
 35    43  169

Extremely quirky row name behavior

We saw that rownames<- does not allow us to specify non-unique row names. Turns out, we can can specify non-unique row names, just not using the get("rownames<-") function.

Let’s look at a rather common way to create dataframes - using matrices. This is a matrix with row names (yes, matrices can have row names too):

mat <-
structure(c(1L, 2L, 4L, 5L, 7L, 8L), .Dim = 2:3, .Dimnames = list(
    c("row1", "row2"), NULL))

As we can see the row names of mat are different:

> mat
     [,1] [,2] [,3]
row1    1    4    7
row2    2    5    8

We can rbind two matrices having the same row names:

> rbind(mat, mat)
     [,1] [,2] [,3]
row1    1    4    7
row2    2    5    8
row1    1    4    7
row2    2    5    8

Now, we’d like to convert the above matrix into a dataframe:

> df <- as.data.frame(rbind(mat, mat))
> dim(df)
[1] 4 3

Looks like we succeeded, right? df seems to have the correct dimensions. Let’s try printing it:

> print(df)
Error in data.frame(V1 = c("1", "2", "1", "2"), V2 = c("4", "5", "4",  :
  duplicate row.names: row1, row2

Error! Turns out, that merely printing the dataframe throws an error.

Why?

Because the row names of df are non-unique:

> dump("df", "")
df <-
structure(list(V1 = c(1L, 2L, 1L, 2L), V2 = c(4L, 5L, 4L, 5L),
    V3 = c(7L, 8L, 7L, 8L)), .Names = c("V1", "V2", "V3"), row.names = c("row1", "row2", "row1", "row2"), class = "data.frame")

So, we can indeed create a dataframe with non-unique rows. Such a dataframe even works in some cases (we were able to print out its dimensions). But in other cases (wherever row names are required), we encounter an error.

This example demonstrates yet another quirk of dataframe row names. This error can be really surprising and may even go undetected. Ideally, the function as.data.frame() should always check for duplicated row names but it doesn’t. How can we correct this problem ?

One simple solution is to just reset the row names:

> rownames(df) <- NULL
> print(df)
  V1 V2 V3
1  1  4  7
2  2  5  8
3  1  4  7
4  2  5  8

Alternatively, we can specify row names while converting a matrix to a dataframe:

> df <- as.data.frame(rbind(mat, mat), row.names = 1:4)
> df
  V1 V2 V3
1  1  4  7
2  2  5  8
3  1  4  7
4  2  5  8

Note that many R users run into this issue in another way:

> as.data.frame(rbind(mat, mat))
Error in data.frame(V1 = c("1", "2", "1", "2"), V2 = c("4", "5", "4",  :
  duplicate row.names: row1, row2

Instead of assigning as.data.frame(rbind(mat, mat)) to df (which prevents printing), if we simply evaluate the expression on the interpreter, we see the error immediately. This error is not because the evaluation failed. This error is because the printing of the evaluated expression failed.

Note: It appears that matrices can have non-unique row names, but then, who knows ? May be we just haven’t found the case in which R throws an error.

Conclusions

From the above behavior of row names, we can see that row names should not really be trusted. Unfortunately, there are some R packages on CRAN that explicitly use the row names which often creates problems.

For example, let us say we need to write a function that accepts a dataframe as input and returns some purposefully selected rows as output.

sample.rows <- function(df) {
    # Select some rows from df
    return(df[selected.rows, ])
}

In order to write the function sample.rows(), we might use row names in such a way that we change the row names of df. The output of this function may then have different row names than the input df. The end-user of this function might actually expect that the row names would remain the unchanged. Thus, modifying row names is a usually not a good idea and so is writing code that depends too much on row names.

Here are a few tips that might help us avoid trouble:

To determine the number of rows, use nrow().
Row name is the not same as row number (row number = row index).
Avoid the use of row names. Use row numbers instead.
Mind the vector recycling when using logical indexing.
If row names must be used, always use rownames() instead of attribute(). At the very least, be consistent in the function used to extract row names.
Take special care when serializing dataframes using dump() or dput().
Take care while converting matrices to dataframes. For robustness, reset row names.
When writing functions, remember that the end-user (which might be you), may expect the row names to remain unchanged.
Do not expect the row names to be in order. If you must use row names and you require row names to be in order, reset the row names by assigning rownames() to NULL.

Perfectly Random

Handling Avro files in Python

Official Avro Packages

Python 2 vs Python 3

Working Example

Issue with name, namespace, and full name

Third-party Avro Packages

fastavro

avro-python3 vs fastavro

Avro <> DataFrame

Avro with PySpark

Confusing official guide

spark-avro: Databricks or Apache

Scala version is important

Working example, finally!

Conclusion

Footnotes

A Guide to Keras Functional API

What is Keras?

What does this article cover?

Sequential and Functional API

Keras Objects

Keras Tensor

Layer

Note about Input and InputLayer

Keras Graph

Keras graph construction using Functional API

Model

Tensor shapes

Batch dimension

Convolutional layers

Recurrent layers

Example 1: One-hot encoded sequence as input

Example 2: Scalar, real-valued sequence as input

Batch normalization layer

Fully-connected layers

Convolutional Layers

Final thoughts

Footnotes

Bernoulli Distribution as a tiny Neural Network

Bernoulli distribution

Binary classification

Modeling the binary classification problem as Bernoulli distribution

Modeling binary classification

Modeling with Bernoulli distribution

Extension to logistic regression

Summary

Footnotes

Setup a Spark cluster on AWS EMR

Setup a Spark cluster

Caveats

Launch EMR cluster

Step 1: Software and Steps

Step 2: Hardware

Step 3: General Cluster Settings

Step 4: Security

Launch

Setup ssh into the EMR cluster

Create a new security group

Add security group to master node

SSH into master node

Get to know your EMR cluster

User

Python

Spark

Install Python packages

Launch Jupyter Notebook

Setup ssh or mosh

Launch using pyspark

Setup port forwarding

Open jupyter notebook in a browser

Find your bearings

Test Spark variables

Import packages

Test jupyter features

Exiting properly

Shutting down notebooks in browser

Shutting down port forwarding

Downloading jupyter notebooks and data

Always on top in MacOS Sierra

Issue with `name`, `namespace`, and full name

`fastavro`

`avro-python3` vs `fastavro`

`spark-avro`: Databricks or Apache

Note about `Input` and `InputLayer`

Setup `ssh` into the EMR cluster

Setup `ssh` or `mosh`

Launch using `pyspark`

2. Obtain `Afloat.bundle`

2. Drop `Afloat.bundle` in SIMBL plugins folder

The Problem with `T` and `F`