Writing clean predicates in Java 8

Using an expression language for predicates, like we have done before, is fun! But it’s a pain to maintain :).

Now that we finally (I hope) have a good release date for Java 8 (supporting lambda -styled predicates and streams), it’s time to think how we are going to re-organize our predicates. Let’s review our predicates for Java 8:

Inline expressions create a maintenance nightmare

Writing in-line lambda expressions and using the stream interfaces to perform common operations on collections is fun and easy. Assume the following example:

List<Person> getAdultMales (List<Person> persons) {
  return persons.stream().filter(p -> 
    p.getAge() > ADULT &&
    P.getSex() == SexEnum.MALE 
  ).collect(Collectors.<Person>toList());
}

That’s fun! But things like this also lead to software that is costly to maintain. At least in an enterprise application, where most of your code handles business logic, your development team will grow the tenancy to write the same similar set of predicate rules again and again. That is not what you want on your project.  It breaks three important principles for growing maintainable and stable enterprise applications:

  • DRY (don’t repeat yourself): writing code more than once is not a good fit for a lazy developer ;) It also makes your software more difficult to maintain because it becomes harder to make your business logic consistent
  • Readability: following clean-code best practices, 80% of writing code is reading the code that already exists. Having complicated lambda expressions is still a bit hard to read compared to a simple one-line statement.
  • Testability: your business logic needs to be well-tested. It is adviced to unit-test your complex predicates. And that is just much easier to do when you separate your business predicate from your operational code.

And from a personal point of view… that method still contains too much boilerplate code…

Imports to the rescue!

Fortunately, we have a very good suggestion in the world of unit testing on how we could improve on this.

Imagine the following example:

import static somepackage.PersonPredicate;
...
public Persons {

private List<Person> persons;
List<Person> getAdultMales () {
  return persons.stream().filter(
    isAdultMale()
  ).collect(Collectors.<Person>toList());
}

We have a “Persons” group, containing a list of Person domain objects.

What we did here was:

  • create a PersonPredicate class
  • define a “factory” method that creates the lambda predicate for us
  • statically import the factory method into our old class

This is how such a predicate class could look like, located next to your Person domain entity:

public PersonPredicate {
  public static Predicate<Person> isAdultMale() {
    return p -> 
      p.getAge() > ADULT &&
      p.getSex() == SexEnum.MALE; 
 }
}

Wait… why don’t we just create a “isMaleAdult” boolean function on the Person class itself like we would do in domain driven development? I agreed, that is also an option… but as time goes on and your software project becomes bigger and loaded with functionality and data… you will again break your clean code principles:

  • the class becomes bloated with all kind of function and conditions
  • your class and tests become huge, more difficult to handle and change (*)

(*) and yes… even if you do your best to separate your concerns and use composition patterns

Adding some defaults…

Working with domain objects, we can imagine that some operations (such as filter) are often executed on domain entities. Taking that into account, it would make some sense to let our entities implement some interface that offers us some default methods.

For example:

public interface DomainOperations {
    default <T> List<T> filter (List<T> data, Predicate<T> predicate) {
        return data.stream().filter( predicate )
               .collect(Collectors.<T>toList());
    }
}

When our Persons entity implementing this interface, we can clean-up our code even more. Of course, this is just adding some syntactic sugar for now. But along the way, more complicated operations can be imagined.

public Persons implements DomainOperations {

  private List<Person> persons;

  List<Person> getAdultMales () {
    return this.<Person>filter( persons, isAdultMale() );
  }

}

And there we go…

Conclusion

Moving your predicates to a Predicate helper class offers some good advantages in the long run:

  • Predicate classes are easy to test and change
  • Your domain objects remain clean and focussed on representing your domain, not your business logic
  • You optimize the re-usability of your code and, in the end, reduce your maintenance
  • You seperate your business from operational concerns

References

Clean Code: A Handbook of Agile Software Craftsmanship [Robert C. Martin]

Practical Unit Testing with JUnit and Mockito [Tomek Kaczanowski]

State of the Collections [http://cr.openjdk.java.net/~briangoetz/lambda/collections-overview.html]

Validating deployments

It has been over 4 months since my last post, so I wanted to start off again with something trivial. Let’s talk a little about deployments first, and I’ll post another topic on predicates and REST security later…

Testing practices

If we follow best practices, we would write a lot of tests. We would test if our application logic is sane using unit tests. We would write integration tests to guaranty that our service components are working properly. we would even write automated acceptance tests to guaranty that an entire use-case flow, one that has business critical value, is really working as expected. Finally, we would even make sure that the results of these tests are documented automatically, ran on every commit change, so that we can prove that the baseline of our business critical application is correct.

If we then finally make a release and push it on to production, everything will be running great! But I think there is still something we forgot to validate.

Validating deployments

Maybe not in smaller projects, but on bigger projects that work together with multiple third parties and interfaces, there is usually a lot of configuration going on. The most primal example would be your data store (SQL or non-SAL). When we configure this on an application server, there is usually some kind of deployment or initialization test that is triggered. Of course, when your connection to the data store cannot be established, there is no way you can deploy your application.
But why would we stop here?

There are other links to other systems and components that are basically never tested: service endpoints, NAS arrays, SMTP services…
If your application needs to be setup properly before it will run, we should validate that these configurations are also done properly before you let your application run. Like your application server would do with your database, you should implement bootstrap tests to make sure that the configuration looks ok and is in fact valid.

It follows the same principles we already have developing code:

  • We write tests to make sure we did not make a (human) mistake during our development, so should we write a “deployment” test to make sure our deployment team configured the application correctly. Usually, deployments are done by a separate teams and even with a detailed release notes document or a set of deployment scripts, small mistakes could happen
  • we write tests to make sure we catch (regression) problems early in the development process because it is usually less costly and less risky. No one wants to fix a problem in production, we want to catch it before it happens

It’s no rocket science!

Let’s have a look at the most basic example we can find: a SAS storage folder…

Testing the deployment of your application in a bootstrap test is very simple:

1. You create an interface such as:

public interface DeploymentValidator {
   void performValidation() throws DeploymentValidationException ;
}

2. You implement your stand alone tests (*).
Here is one example that would verify if an interface directory (used to exchange bulk XML files for example) is available and writeable:

 public class VerifyDirectoryConfigurationDeploymentTest implements DeploymentValidator {

    private static final String PROVIDED_PATH_IS_NOT_A_DIRECTORY  = "Deployment test failed: Directory %s is not readable and/or writable.";
    private static final String PROVIDED_PATH_IS_NOT_READ_OR_WRITABLE = "Deployment test failed: %s does not point to a valid directory.";

    private List<String> directoriesToVerify = new ArrayList<String>();
    public DirectoryVerifier(List<String> directoriesToVerify ) {
        this.directoriesToVerify = directoriesToVerify ;
    }

    public void verifyDirectoriesAreReadableAndWritable(List<String> directoriesToVerify) throws DeploymentValidationException {
        for (String path : directoriesToVerify) {
            File directory = new File(path); // note: this is still using Java File API; using Java NIO would be better...
            verifyIsValidReadableAndWritableDirectory(directory);
        }
    }

    private void verifyIsValidReadableAndWritableDirectory(File directory) throws DeploymentValidationException {
        if (doesDirectoryExist(directory)) {
           if (!isReadAndWritable(directory)) {
              throw new DeploymentValidationException(String.format(PROVIDED_PATH_IS_NOT_READ_OR_WRITABLE , directory.getAbsolutePath()));
           }
        } else {
            throw new DeploymentValidationException(String.format(PROVIDED_PATH_IS_NOT_A_DIRECTORY, directory.getAbsolutePath()));
        }
    }

    private boolean doesDirectoryExist(File directory) {
        return directory.exists() && directory.isDirectory();
    }

    private boolean isReadAndWritable(File directory) {
        boolean readable = directory.setReadable(true) && directory.canRead();
        boolean writable = directory.setWritable(true) && directory.canWrite();
        return readable && writable;
    }
}

3. You create your bootstrap validator:

public class ApplicationDeploymentValidator implements InitializingBean {
private List<? extends DeploymentValidator> validators;

@Override
public void afterPropertiesSet() throws Exception {
   performValidations();
}

private void performValidations() throws Exception {
   for (DeploymentValidator deploymentValidator : validators) {
      deploymentValidator.performValidation();
   }
}

public List<? extends DeploymentValidator> getValidators() {
   return validators;
}

public void setValidators(List<? extends DeploymentValidator> validators) {
   this.validators = validators;
}

}

And you configure it to launch on startup (using spring for example):

<bean id="applicationDeploymentValidator" depends-on="configurationLoader">
   <property name="validators">
      <list>
         <ref bean="verifyDirectoryConfigurationDeploymentTest "/>
      </list>
   </property>
</bean>

<bean id="applicationDeploymentValidator">
   <constructor-arg>
      <util:list>
         <value>${configuration.property.of.some.directory.to.check}</value>
      </util:list>
   </constructor-arg>
</bean>

4. You create unit tests that test your bootstrap validator and all your validation tests to be correct

A last extension would be to identify critical and non-critical bootstrap tests. Testing for a web service endpoint may, depending on the case, not be critical enough to stop the startup of your application. However, it would be great to have this failed test log a warning during your deployment or start-up.

That’s it. This is cheap to implement, it can help to deploy your application and prevent production issues upfront,… but it is often forgotten on important projects.

(*)Do note that it may not be trivial to define what you should test in your bootstrap or not. Some idea’s are:

  • you test to see if your direct folders are accessible and can be used as expected
  • you can test to see if your web service endpoints or non-web services are valid. Of course, we could argue that they don’t need to be available when the application boots because that is one of the powers of web services: to be able to take them online, offline, deploy,… without having to restart your entire back office infrastructure. But on the other hand, it would be nice to know that the correct service endpoint is at least there when you deploy your application. At least have it logged somewhere at startup…
  • you could test to see if your configuration itself is correct. Maybe there are parameters that should not be used together, that are mutually exclusive or that should never be “on” in a production environment?

Of course, we have all kind of deployment processes and DevOps these days going on… But I’m not talking about that. I am talking about a way the development team could protect the project from a bad deployment configuration. This cheap implementation will allow you to prevent configuration problems before they happen in production and this will benefit everyone because the cost to fix it in production afterwards will be much bigger and will impact much more people. From this point of view, your deployment team members are your end-users and you should make their life a bit easier as well.

Collections, meet Expression!

Introduction

On my current project, we use a lot of data collections. And like in most software projects, the most common (although far from exclusive) operations are finding an element from a collection and doing something with it.

As a stereotyped Java developer, I am also used to writing a solution that is clean and simple. Something that is readable and easily understood by everyone. I am also a very lazy person when it comes to writing code, so I basically don’t want to repeat myself. That is why I often use the Apache Collections API to make my life a bit more easy. For those who do not know this library, it contains a set of very useful types and functions, that help free you from writing this boiler plate code. Allow me to explain:

Imagine that you have a collection called “apples” that contains a bunch of Apple objects. If we want to find the first green apple from that bunch, we could write something like this:

CollectionUtils.find ( apples, isGreenApple );

The “greenApples” variable is defined as an implementation of the Predicate interface. This interface is very simple: it contains only one method (evaluate) which returns a boolean. Its sole purpose is to identify if the object provided matches the predicate or not. We could define the “isGreenApple” predicate from our previous example as such:

Predicate isGreenApple = new Predicate () {
   public boolean evaluate(Apple apple) {
      return apple.isGreen();
   } 
}

If you did not know about Apache Collections, I would recommend that you look into it. It can do much more than finding objects in a collection, but we will constrict ourself to that concept for now.

The benefits from working with collection like this should become clear over time:

  • We separate the concern of the operation to perform on a collection from where.
  • It will allow us to re-use our predicates often, and independent of what we want to do with them.
  • When you get used to the syntax, your code easier to read.

But there are also some side effects:

  • This syntax is very verbose. It balances our for complex predicates, but for simple cases like in our previous example, it seems an awful lot of extra work. 
  • In reality, on a big software project, the re-use of these kind of predicates is almost non-existent. It difficult to find and reuse predicates.

I think we can do better than this…

Expression predicates

The idea of “selecting” elements like this looks very similar to a concept from a functional language. Basically, we want to describe how to identify and object and then what to do with it. And, of course, this should be easy to read and work with. We already figured out the “what to do with it” part, but not how to identify it.

At the moment of writing, Lambda expressions are not yet available to us. At least, in the Java industry, it is not feasible yet to migrate to Java 8 in order for us to gain access at using Lambda expressions. So that is not yet an option.

But, there is a specification in the Java world we could actually. It is easy to read, easy to use and very powerful. We have been using it a lot in frameworks such as JSF, WebFlow, etc. It’s called the Unified Expression Language.

The condition from our previous example could easily be re-written as an EL expression:

“apple.isGreenApple()” or “apple.greenApple”

So why not use that?

We can build our own Predicate implementation and use an EL library to resolve the expression for us. Our ExpressionPredicate might look something like this:

public class ExpressionPredicate<T> implements Predicate<T> {

    private ContextMap contextMap;
    private String objectName;
    private ELContext context;

    private ValueExpression expression;
    
    public ExpressionPredicate(String objectName, String expression) {
        this(objectName, expression, Maps.<String, Object>newHashMap());
    }
    
    public ExpressionPredicate(String objectName, String expression, Object expectedResult, Map<? extends String, ? extends Object> variableMap) {
        this.contextMap = new ContextMap(variableMap);
        this.objectName = objectName;
        
        CompositeELResolver compositeELResolver = createELResolver(contextMap);
        this.context = createContext(compositeELResolver);
        
        ExpressionFactory expressionFactory = new ExpressionFactoryImpl();
        this.expression = expressionFactory.createValueExpression(
                this.context, expression, expectedResult.getClass());
    }

    @Override
    public boolean evaluate(T object) {
        contextMap.assignBaseVariable(objectName, object);
        return Boolean.valueOf( evaluateExpression(context) );
    }

    Object evaluateExpression(ELContext context) {
        return this.expression.getValue(context);
    }
    
    private CompositeELResolver createELResolver(ContextMap contextMap) {
        CompositeELResolver compositeELResolver = new CompositeELResolver();
        compositeELResolver.add(contextMap);
        compositeELResolver.add(new ArrayELResolver());
        compositeELResolver.add(new ListELResolver());
        compositeELResolver.add(new BeanELResolver());
        compositeELResolver.add(new MapELResolver());
        return compositeELResolver;
    }

    ELContext createContext(final ELResolver elResolver) {
        return new ELContext() {
            @Override
            public VariableMapper getVariableMapper() {
                return new VariableMapperImpl();
            }
            @Override
            public FunctionMapper getFunctionMapper() {
                return new FunctionMapperImpl();
            }
            @Override
            public ELResolver getELResolver() {
                return elResolver;
            }
        };
    }

    static class ContextMap extends MapELResolver {
        Map<String, Object> context;
    
        public ContextMap(Map<? extends String, ? extends Object> context) {
            this.context = Maps.<String, Object>newHashMap(context);
        }

        @Override
        public Object getValue(ELContext elContext, Object base, Object property) {
            if (base == null) {
                base = context;
            }
            return super.getValue(elContext, base, property);
        }
        
        public void assignBaseVariable(String objectName, Object objectValue) {
            context.put(objectName, objectValue);
        }
    }

}

It may not be pretty and even look a bit overwhelming at first… but these are the basic things that happen here:

  • We set-up an Expression Language resolver, so that we can evaluate an expression
  • We bind a “context map” to the resolver, which is going to contain all our variable when we evaluate it
  • Before each evaluation runs, we  put the current object into the “context map”. The objectName will be used to identify the current object during our iteration.
  • In this example, we assume the expression evaluates to either true or false. We parse that result and return it after the evaluation. Note that if the evaluation does not return true, it will always revert to false. So if you want to support more return types for your expression, you should edit the example above and add support to compare an “expected result” as opposed to assuming that the returned result is a boolean.

In the example above, I’ve used the JBoss EL implementation, but there are many more providers out there for the javax.el API. I also make use of Apache Collections (of course) and Google Guava.

With this predicate, we are already half way of where we want to be. By now, we can already do this:

CollectionUtils.find (apples, new ExpressionPredicate("apple", "${apple.greenApple}") );

Also notice that it is possible to provide a basic Map to the ExpressionPredicate. It will allow us to pass-on variables and other objects into our expression. Imagine the following example:

Map<string, apple=""> arguments = Maps.newHashMap();
arguments.put("color", "green");
CollectionUtils.find (apples, new ExpressionPredicate("apple", "${apple.colorName eq color}", arguments) );

In the code above, we bind the “green” string to the “color” variable. The expression is a standard EL expression, performing an equals on the “colorName” property to the “color” variable we provided.

But we can do a little bit better…

Syntactic sugar

What is missing here is some syntactic sugar. We don’t always want to name the object we are going to iterate over it; we can just call it plain “var”. I would also like to have an easy method I can call in order to construct a variable map. And finally, it may be a good idea to foresee a builder pattern. Our expressions can be very powerful already, but we may want to de-couple our expression into multiple parts. Or, we may even want to construct our ExpressionPredicate more dynamically later on.

public class ExpressionUtils {
    
    public static final String OBJECT_NAME = "var";
    
    public static <T> ExpressionBuilder<T> newExpression() {
        return new ExpressionBuilder<T>();
    }
    
    public static class ExpressionBuilder<T> {
        
        private List<ExpressionPredicate<T>> expressions;
        
        public ExpressionBuilder() {
            expressions = Lists.newArrayList();
        }
        
        public ExpressionBuilder<T> withProperty(String propertyName) {
            expressions.add(new ExpressionPredicate<T>(OBJECT_NAME, "${" + OBJECT_NAME + "." + propertyName + "}"));
            return this;
        }
        
        public ExpressionBuilder<T> with(String expression) {
            return with(expression, Maps.<String, Object>newHashMap());
        }
        
        public ExpressionBuilder<T> with(String expression, Map<? extends String, ? extends Object> arguments) {
            expressions.add(new ExpressionPredicate<T>(OBJECT_NAME, "${" + expression + "}", arguments));
            return this;
        }

        public Predicate<T> build() {
            return AllPredicate.getInstance( expressions );
        }

    }

    public static Map<? extends String, ? extends Object> newMap(String arg, Object value) {
        return ImmutableMap.<String, Object>of(arg, value);
    }

    public static Map<? extends String, ? extends Object> newMap(String arg0, Object value0, String arg1, Object value1) {
        return ImmutableMap.<String, Object>of(arg0, value0, arg1, value1);
    }
    
    public static Map<? extends String, ? extends Object> newMap(String arg0, Object value0, String arg1, Object value1, String arg2, Object value2) {
        return ImmutableMap.<String, Object>of(arg0, value0, arg1, value1, arg2, value2);
    }

    public static Builder<String, Object> newMap() {
        return ImmutableMap.<String, Object>builder();
    }

    public static <T> Predicate<T> newExpression(String expression, Map<? extends String, ? extends Object> newMap) {
        return new ExpressionBuilder<T>().with(expression, newMap).build();
    }

    public static <T> Predicate<T> newExpression(String expression) {
        return new ExpressionBuilder<T>().with(expression).build();
    }
}

With this utility / builder class, we have a very powerful tool where we can use expressions to select elements from collections.

We can construct complex evaluations:

CollectionUtils.find ( apples, ExpressionUtils.newExpression ( "var.greenApple and not var.oldApple" ) );

We can add variables to our expressions:

CollectionUtils.find ( apples, ExpressionUtils.newExpression ( 
    "var.greenApple and var.brandName eq brand", ExpressionUtils.newMap( "brand", "PinkLady" ) ) );

And we can even construct them dynamically:

ExpressionBuilder expression= ExpressionUtils.newExpression()
     .with ("var.greenApple);
if (excludeOldApples) {
    expression.with("var.oldApple");
}
CollectionUtils.find ( apples, expression.build() );

There is no silver bullet

There is, however, an important side note I want to mention: for every method we write that uses an expression like this, we should write a good unit test (if we don’t already do this anyway).

The reason is that our compiler will not be able to code check this expression for us. That is a big disadvantage. If we write a mistake in the expression, our compiler will ignore it until we evaluate the expression at runtime. It is even more important if we want to refactor or re-write our code: we won’t immediately know if we broke an expression when we, for example, change the name of a method or property of our object.

Still, we should write unit tests for these kind of operations by default. The gain in readability and usability will eventually outweigh the loss in pre-compilation checking.

Conclusion

The Apache Collections library is a great library for working with collections. However, the need to write verbose predicates that are sometimes difficult to read reduces the ease of use for that library. By using the Unified Expression Language, and building a Predicate implementation around that, we can greatly increase the readability, the flexibility and the ease of use for the Collections library. We can imagine evaluating complex expressions, expressions with variable binding and even construct these expressions dynamically at run time. Because of this, we greatly decrease the amount of code we develop and increase our own productivity.

Still, this is not a perfect solution: by using an expression language, we lose the ability of our compiler to validate our code at compile time, because these expressions are ignored.

With Java 8 currently being finalized, we will gain access to Lambda expressions. Eventually, when Lambda becomes an accepted principle in the Java industry, we may consider again “refactoring” our ExpressionPredicate to use those language constructs, which may again allow our compiler to validate at compile time.

Quick links and references

Apache Collections: http://commons.apache.org/collections/

Unified Expression Language: http://docs.oracle.com/javaee/1.4/tutorial/doc/JSPIntro7.html

Lambdas & Collections: http://cr.openjdk.java.net/~briangoetz/lambda/collections-overview.html