In the last decade, computer engineering has exploded with solutions based on cloud computing, distributed systems, big data, streaming or artificial intelligence. Today’s world systems are data-intensive so that volumes become higher and cheaper. Shortly speaking, data is at the centre of our interests. One of issues in these area is a data quality — why is it so important?

Nowadays, data is everywhere and drives companies and their operations. If we do not make decisions based on reliable or complete information, it might have a huge impact on these and end up with tragic consequences. The data’s correctness and prominence reserves a special discipline, known as a data cleansing, which is focused on removing or correcting coarse records.

cleanframes is a small library for Apache Spark to make data cleansing automated and enjoyable.

Let’s walk through a problem, step by step, to examine what it takes to make data quality as awesome and reliable as possible! 

Let’s introduce a sample data and, for a simplicity-sake, make it a small set:

And a domain model that, for its representation, is defined as:

If you are wondering why there are options across the case class the answer is simple: Scala primitives are values and cannot be nulls. Options liberate us by setting the dataframe schema columns’ nullability to true.

Let’s load the data and check what Spark will do about it:

The  collect variable contains the following result:

As you can see, due to malformed values, the second and fourth lines don’t retain the correct data and reject entire rows. We would be much better off if it would be possible to ignore only invalid cells and save the correct ones.

Our goal is to represent input data as follows:

How can we achieve this?

Pure Spark SQL API

Apache Spark comes with set of APIs to work with dataframes that could be used for graceful parsing:

These lines do just what we need, however, this is so much code to do a simple thing (there are only three columns to be taken care of!). Typically, there are a lot more columns to deal with, which only means the boilerplate code grows linearly.

Can We Do Better?

Well, of course we can! You could always write helper methods and re-use them in the project just like this:

The question is, is this good enough? We have achieved some re-usability but still, did we give it our best shot? Frankly speaking, the chain of these calls are awful and error-prone. They could be hidden with some implicit case class to imitate some DSL:

There is some progress noticed, nonetheless, it’s actually more of a visual progress. We need to do it manually and repetitively throughout our code base. There is some room left for improvement.

cleanframes is a library that aims to automate data cleansing in Spark SQL with help of generic programming. Just add two imports and call the clean method:

This code is equivalent to the code listed in the “Pure Spark SQL API“ section. Due to this, there is no performance difference between the two.

What the Heck Just Happened?

The clean method expands code through implicit resolutions based on a case class’s elements. The Scala compiler applies a specific method to a corresponding element’s type. cleanframes comes with predefined implementations that are available via a simple library import (see, line 2). I recommend you to check their source code on GitHub.

It’s worth mentioning that these transformations are used from the Spark SQL functions package so they are catalyst-friendly.

The full solution code is as follows:

And we’re ready to go with transformations! 

The project source code is here: https://github.com/funkyminds/cleanframes

Please refer to link above to find Maven or sbt coordinates.

What Are the Pros and Cons?

It’s time to examine advantages and disadvantages of the library.

Pros:

Cons:

Scala-based libraries have the advantage over Java counterparts that they are not based on reflection APIs but a type classes approach. It brings code safety during compilation time and reduces computation overhead.

Not everyone likes implicits, but, this is a straightforward case where all you do is parse simple types! There is no endo-mumbo-jumbo-functors in this scope.

Don’t freak out, be reasonable!

This article gave only an introduction and a simple demo of cleanframes. The library is capable of doing much more.

In following articles, I’ll explain features such as:

“cleanframes – data cleansing library for Apache Spark! (part 2)” presents all of them one by one with code examples and explanations.

I hope that cleanframes can help you with tedious but necessary work and cut out some boilerplate code. The project has been recently released on the Apache license and I would appreciate contributions of all kind: feature proposals, issues, pull requests, you name it!

And of course: give it a try!