Partial commit and job restart with Spring Batch

If a batch fails after partial commit, it must be possible to start again with the processing of the file by skipping the lines already committed

A classic batch is the processing of a file, for which records are read, for each record the data are processed and are persisted on database (reader, processor and writer).

In case the file is large and contains thousands of records, partial commits must be expected during processing. For example, every 1000 records, we can decide to commit the processing on the database.

Through Spring Batch it is very easy to get partial commits, it’s a simple parameter that is passed to the StepBuilder. In the case that it’s necessary to implement more complex partial commit policies, it is possible to implement custom completition policies (which in this case we will not see because it is not the subject of this article).

Finally, if a batch fails after partial commit, it must be possible to start again with the processing of the file by skipping the lines already committed. This last thing is also expected by Spring Batch, but we must add a few lines of code, it is not a simple configuration parameter. Also in this case it’s possible to implement custom skip and retry policies (and also in this case we will not see why not the subject of this article).

The environment is as follows:

  • Java7
  • Spring Boot 1.1.8
  • Spring Batch 3.0

There are two concepts, the job instance and job execution. An instance of a job is accomplished through n executions (typically one, or more if there have been failures). Furthermore, only a failed job can be restarted.

To implement the restart we need the jobRegistry, jobOperator, jobExplorer and jobLauncher. Here is the complete code of the batch configuration.

Basically these are the steps:

  • register the job in the jobRegistry
  • get job instances through the jobOperator
  • given the last instance, get executions through the jobOperator
  • through the jobExplorer check if the last execution has failed
  • in case the last execution has failed, the job must be restarted via the jobOperator
  • in case the last execution was successful, launch a new job instance via the jobLauncher

Spring Batch takes care of managing the restart starting from the first uncommitted record.

Thanks Spring Batch 🙂





Performance tuning for File Reading with Java 8 and Parallel Streams

Is it always convenient to use parallel streams?

Java 8 has introduced with the Stream the possibility of using in a very simple way all the resources made available by the hardware, in particular the cores of the multicore architectures. And all this with the paradigm of declarative and non-imperative programming.

For example, suppose you have to do a batch that looks for a string in all the files that have a certain prefix in a certain directory. The batch must notify in a log in which files the string is found.

It is interesting to compare the two approaches, pre java 8 and java 8 with the streams.

I used my dev computer, a Samsung laptop with this features:

  • Intel Core i7-4500U (2 Core, 4 Thread) 1.8GH
  • 8 GB RAM
  • 256 GB SSD
  • Windos 8.1 Pro 64 Bit
  • Java 1.8.0_101

Having 4 Threads I expect the performance of the parallel stream solution to be 400% better than the classic pre java 8 version.

The code that uses the pre java style 8 is as follows: StringMatchingOld

The code that uses the streams is as follows: StringMatching

The code that uses the parallel streams is as follows: ParallelStringMatching

These are the results (4 files read):

File Reading Performance

What is going on? Truly strange, there is no benefit in the use of the stream (I think also from the point of view of the code… I don’t think it’s more readable but this is another story… I’m getting older…). Furthermore, we see that using parallel streams does nothing but make things worse.


It seems that reading the files does not find any improvement in the use of the streams. However, the fact remains that declarative programming makes it possible to abstract from current hardware, so maybe on a server with better hardware than my development pc, parallel streams performance is better.

Or maybe I’m doing something wrong … I’m looking forward to that.


Data Masking with JPA and Spring Security

The protection of sensitive data is an increasingly popular topic in IT applications

The protection of sensitive data is an increasingly popular topic in IT applications. Also in our case, a customer asked us, on an already existing web application, to implement a data masking solution that is dynamic and based on security profiles.

The application is developed in Java, with Spring MVC for the management of the Model View Controller, JPA for data access and Spring Security for the management of security profiles.

There are two approaches in literature: SDM (Static Data Masking) and DDM (Dynamic Data Masking).


SDM plans to clone the current database by masking sensitive data. Specific inquiry applications that provide data masking can read from the cloned database.


  • performance of data access at runtime


  • data read can be not updated (update takes place via batch and, depending on the mode, the update can last from minutes to hours)
  • not ideal for a role-based / field-based security scenario


DDM plans to mask data when it is read at runtime.


  • real data reading,
  • ideal for a role-based / field-based security scenario


  • read / write overhead performance
  • possible unmusk algorithms to avoid data corruption (to prevent the masked data from persisting on the DB)

Given the customer’s requests, the DDM technique is the one that best suits a dynamic scenario based on security profiles.

At this point another choice had to be made because for DDM there are two approaches:

JPA Rewriting

In the literature we talk about SQL Rewriting, in our specific case JPA rewriting, JPA being our data access layer. The data is masked in a PostLoad or PostUpdate annotated method of a JPA Entity Listener, that means in the persistent layer.


  • punctual masking of the data in the load phase from the DB
  • easy data-masking mapping


  • masking depending on the data type (for example a string can be masked with ‘***’, or with ‘###’, a number with ‘000’ or ‘999’, a date with ’99 / 99 / 9999 ‘, etc etc …)
  • difficulty in the Look & Feel for rendering the view if the data is masked (each view should declare the masking … re-enter in the case of View rewriting below)
  • unmask algorithms that use the user session to store unmasked data. JPA makes shering of objects loaded by DB, so it is not said that an object loaded by an inquiry function is not then used for an update function. In this case the masked data would be persisted on DB, that means data corruption
  • complex make the masking dependent on the function (use of the user session for function-masking mapping)
  • complex use of the user session (see above for unmask and function-masking mapping)

View Rewriting

The data is masked in the presentation layer, typically in jsp pages.


  • homogeneous masking (does not depend on the type of data, everything can be masked for example with ‘***’)
  • it is not required unmusk phase
  • easy rendering for a look & feel (each view declares whether or not it wants masking)
  • easy to make it dependent function (each function declare whether or not it wants masking)


  • not punctual masking (all the views must mask … the tags reused by the view simplify, but not completely)
  • difficult data-masking mapping (each view must declare the data)

We chose to adopt the View Rewriting, because analyzing the effort (which in this article omits because not relevant), it was, more or less, similar between the two approaches, while the risk of data corruption and out of memory exceptions of user session are absent. Moreover the View Rewriting solution is much more customizable for what concerns the Look & Feel

To implement the solution we need the following things in detail:

  • a generic editor to enable or not a field for masking
  • a masking class that performs data masking based on security profiles
  • to modify all existing views to use the masking class above

Let’s see in detail

Role-based security mapping

We use a role-based security mapping based on Spring Security (already present in the application). For any data that you want to mask, a role is created made like this:


for example, if I want to mask the tax code field of the people table, since the field is mapped via JPA in Person.taxCode, the role will be


The mapping editing is managed dynamically with a special GUI function. We used the existing Domanin Editor function, a generic domain editor that for all domain classes it allows the modification of all the fields mapped to the database.
We have added a new editing form for managing data-masking mapping.
The form will contain all the fields of the chosen domain class. For each field you can choose (with a special checkbox) whether or not to enable the relative masking. When saving, the function performs the following steps:

  • look in the Authorities table if the role ROLE_MASK_DOMAIN-NAME_FIELD-NAME exists. If it does not exist it creates (the opposite if the field must be disabled)

For mapping with profiles (Spring Security Groups) are used the already present Spring Security functions implemented in the appropriate View of the application.

Masking class

Creation of a class that receives as input the data to be masked and its name (for example, Person.taxCode).
The class looks for (with the methods that provide Spring Secutiry) if the current user’s profile is associated with the corresponding field (ROLE_MASK_PERSON_TAXCODE / Person.taxCode). If exists, the class mask the data and returns it to the view.

Change Views

The functions that provide for masking the data are typically those of inquiry. In our case it helps us the fact that we have adopted tags in the presentation layer so all the shows and lists use a display.tagx tag and a table.tagx tag. We need to change these two tags to make them use the masking class.
The longest work concerns modifies all jsps that use the two tags, which must declare the name of the field they are viewing.

Finally we have also modified the search filters to make sure that if the filter provides the search for a field that must be masked, the filter is disabled.
For example, if the filter requires a search for tax code, the filter must use the masking class to know at runtime if the profile expects to mask this data.
If so, the filter is disabled.


View Rewriting with role based security is the best solution for the following reasons:

  • effort slightly greater than the JPA Rewriting solution but more or less similar
  • use of spring security to map the data to be masked to the profile
  • greater custom in terms of look & feel
  • absence of data corruption risk
  • absence of user session out of memory risk

IBAN, iban4j and CIN calculation

The International Bank Account Number (IBAN) is used to uniquely identify bank details internationally.

The code is as follows:

  • 2 capital letters representing the Nation (IT for Italy)
  • 2 control digits
  • the national BBAN code

For Italy, the BBAN code (Basic Bank Account Number) is composed of:

  • CIN (1 uppercase letter)
  • ABI (5 digits)
  • CAB (5 digits)
  • Account number (12 alphanumeric characters possibly preceded by zeros if the number of characters is less than 12)

The CIN (Control Internal Number) code consists of a single letter and is used as a control character. It’s calculated based on the ABI and CAB codes and account number.

Both the two control digits and the CIN can be calculated to verify that the IBAN entered in a form by a user is valid and compliant. To do this in java there is the iban4j library.

The problem with this library is that it calculates the two control digits but not the CIN. On the net I didn’t find any java library that made the CIN calculation. I only found an example written in Visual Basic  of which I ported in java. The class name is CINUtil and it can be downloaded from PasteBin.

A method for checking the iban inserted in a form by a user can be the following:

public static boolean checkIban(String ibanCode) {
String countryCode = ibanCode.substring(0, 2);
String abi = ibanCode.substring(5, 10);
String cab = iban.substring(10, 15);
String conto = iban.substring(15);

org.iban4j.Iban ibanToCheck = new org.iban4j.Iban.Builder()
.nationalCheckDigit(CINUtil.computeCin(abi, cab, conto))

return ibanCode.equals(ibanToCheck.toString());

Spring Boot, Spring Batch and exit codes

When creating batches to be invoked by a scheduler, it is very important to correctly manage the JVM exit codes.

When creating batches to be invoked by a scheduler, it is very important to correctly manage the JVM exit codes.
By convention, the JVM ends with an exit code equal to zero if there were no problems, otherwise with an exit code greater than zero.
In this way if the batch is not terminated correctly, interpreting the exit code, the scheduler can for example inform the application manager via email, or adopt strategies to relaunch or recover the batch itself, or terminate a job box.

If you use Spring Boot to start a Spring Batch-based batch, the JVM always ends with an exit code of zero, even in the case of runtime exceptions. In order to correctly manage the JVM exit codes, it is necessary to intervene by means of an ExitCodeGenerator.

The application stack is composed of:

Spring Core 4.0.7
Spring Boot 1.1.8
Spring Batch 3.0.1

in the class that configures the batch, we need to add the following methods:

@Bean public JobExecutionExitCodeGenerator jobExecutionExitCodeGenerator() {

return new JobExecutionExitCodeGenerator();


protected JobExecution addToJobExecutionExitCodeGenerator(JobExecution jobExecution) {

JobExecutionExitCodeGenerator jobExecutionExitCodeGenerator = jobExecutionExitCodeGenerator(); jobExecutionExitCodeGenerator.onApplicationEvent(new JobExecutionEvent(jobExecution)); return jobExecution;


as ExitCodeGenerator we can use the default implementation of Spring Boot which is JobExecutionExitCodeGenerator. So in the addToJobExecutionExitCodeGenerator method we pass the jobExecution to the exit code generator forcing the creation of the JobExecutionEvent event. When we launch the job, we must force the call to the addToJobExecutionExitCodeGenerator method:

addToJobExecutionExitCodeGenerator(, jobParameters(jobParametersMap)));

In this way, when we end the batch in the Application class, the exit code will be the one actually returned from the batch:

int exitCode SpringApplication.exit(, args)); System.exit(exitCode);

Howto migrate LDAP users to Spring Security JDBC

Howto migrate LDAP users to Spring Security JDBC

Lately we are redoing a web portal whose users who have access to the private area of ​​the application are registered through LDAP.
The only request made by the contractor is that the transition to the new portal must be transparent to registered users. This translates into the fact that users do not have to change the password the first time they log in to the new portal. Passwords are stored in the LDAP repository with SSHA (Salted SHA) encoding.
Our application uses Spring Security to manage security and access to the reserved area. Spring Security supports various types of authentication including LDAP itself. As a first idea we thought to use the same LDAP repository already present. After analyzing this solution in detail we have thought not to take this solution for various reasons.
The first is to map the roles related to the permissions of the old application to the roles of our application (feasible but not very nice from the point of functional view). Furthermore, having to maintain two separate servers, an LDAP and a DBMS, when it is possible to have only one DBMS server, is not a good thing from the point of management costs.
So we thought about using the classic Spring Security JDBC authentication. The users will be migrated through a batch that will load the unloading LDAP users (download, for example, done in the csv format) to the JDBC tables. To ensure that password encryption remains the same, just configure Spring Security to use the LdapShaPasswordEncoder class. To do this you need to define the following bean in WebMvcConfiguration:

public LdapShaPasswordEncoder passwordEncoderLDAP () {
return new LdapShaPasswordEncoder ();

end using it in the AuthenticationManagerBuilder defined in WebSecurityConfiguration like this:

private LdapShaPasswordEncoder ldapPasswordEncoder;

protected void configure (AuthenticationManagerBuilder auth) throws Exception {
auth.userDetailsService (webJdbcUserDetailsManager).passwordEncoder (ldapPasswordEncoder);

in this way the password coding will be the same used by LDAP and for users the transition to the new portal will be transparent.

Oracle, Repeatable Read and JPA

Oracle optimistic and pessimistic lock

Lately we had to solve concurrent management of database access between different threads of the same process or different processes.
The scenario is as follows: web application deployed on websphere, tasks scheduled by spring scheduler within the same web application and standalone external batches. All these threads must be able to work simultaneously on the same data without causing concurrent problems. Above all because the operations involve non-transactional services on Host invoked via CTG. In essence it means to adopt a pessimistic lock approach instead of optimistic lock.

The application stack is as follows:

  • Java 7
  • Spring 3.1
  • JPA 2.0
  • Hibernate 3.6.8
  • Websphere 8.5.5
  • Oracle 12g

Oracle has three types of Isolation Level: Read Uncommitted, Read Committed and Serialization. Missing Repeatable Read that is exactly what would solve our problem without application interventions. Serialization is too strong as an approach, because we do not want to lock the entire table and prevent any insert. We decide to set Read Committed (Oracle default level) but as expected at the first concurrent access we have an optimistic lock problem:

Caused by: javax.persistence.OptimisticLockException: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction  (or unsaved-value mapping was incorrect)

To solve this type of problems Oracle provides us with the select for update, which means using JPA to lock the instance obtained from the database.
For example, to retrieve objects through a query:

Query q = entityManager().createQuery("SELECT o FROM ObjectA o");

in this way if the object has no lock it will be immediately returned by the db to the calling thread, otherwise the thread will wait until the lock is released.

With this pessimistic lock-oriented approach, lock timeout problems can occur. In this case the exeception should be managed with try / catch to cancel the operation or try again to recover the object. In production it is also possible through an empirical approach to increase the timeout to reduce the frequency of the problem.

Finally, obviously, the retrieve of concurrent objects must be done before invoking non-transactional services such as CTG, to avoid the cancellation of the operation after committing the CTG itself.

GenericConversionService, IdToEntityConverter and StackOverflowError

Spring GenericConversionService, IdToEntityConverter and StackOverflowError

Yesterday, after a code refactoring, doing a test of the web application I get a mythical StackOverflowError:

at java.util.HashMap.addEntry(
at java.util.LinkedHashMap.addEntry(
at java.util.HashMap.put(
at java.util.HashSet.add(

strange … on a web form that had not been subject to code changes for a long time.

The project is based on Spring 3.1.0 and Java 7.

By debugging, I find that the failed conversion affects the domain class Address to which the static findAddress method has been added. The signature is as follows:

public static Address findAddress(Address address)

the method has been added to prevent duplicate persistance addresses on the db.

So now the Address class has two static finders:

public static Address findAddress(Address address)
public static Address findAddress(Long id)

the IdToEntityConverter class uses the getFinder method whose rules are (from javadoc):

the finder method must be public, static, have the signature find[EntityName]([IdType]), and return an instance of the desired entity type.

The problem is that IdType is converted by searching its finder, which is always of the same type Address, causes the loop resulting in StackOverflowError.

The solution is to rename

findAddress(Address address)

method in

retrieveAddress(Adress address).

Howto install web application with Spring Roo 2.0 on Websphere 8.5.5

Install web application with Spring Roo 2.0.0.RC1 on Websphere

To install spring roo 2.0.0.RC1 on Websphere you have to solve some problems. The environment is as follows:

  • Ubuntu 15.04 (GNU/Linux 3.19.0-15-generic x86_64)
  • IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20170419_34
  • Spring Roo 2.0.0.RC1
  • Websphere

JPA problem

The first problem is the JPA compatibility. Roo 2.0.0.RC1 uses jpa 2.1 while WAS jpa 2.0.At the start of the application there is such a mistake:

caused by: java.lang.ClassCastException: incompatible with javax.persistence.spi.PersistenceProvider at javax.persistence.Persistence$1.isLoaded( ~[hibernate-jpa-2.1-api-1.0.0.Final.jar:1.0.0.Final]

To fix this problem, you need to add the HibernatePersistenceProviderResolver class to your project:

and register it in the Application class in the onStartup method

@Override public void onStartup(ServletContext servletContext) throws ServletException {
super.onStartup(servletContext); }

XML-APIS problem

The second problem is due to xml-apis jar:

Caused by: java.lang.ClassCastException: org.apache.xalan.processor.TransformerFactoryImpl incompatible with javax.xml.transform.TransformerFactory at javax.xml.transform.TransformerFactory.newInstance(Unknown Source) ~[xml-apis-1.4.01.jar:na]

to solve it you need to make an exclusion in pom.xml like this:


WEBJARS problem

Third issue concerns the webjars protocol:

Caused by: java.lang.NullPointerException at at$JarURLInputStream.close( at

For some reason WAS uses the wsjar protocol instead of jar. Also to access the resources within the jars WAS must have a trailing slash in the path.To solve this, you need to add the following class:

and log it in to WebMvcConfiguration in the addResourceHandlers method

@Override public void addResourceHandlers(ResourceHandlerRegistry registry) { super.addResourceHandlers(registry); registry.addResourceHandler("/webjars/**").addResourceLocations("classpath:/META-INF/resources/webjars/"). resourceChain(false).addResolver(new WebJarsResourceResolver(new WasWebJarAssetLocator())); }

finally adding variable to was (web container custom properties) = true

Transactions problem

Another issue with regard to transactions is:

Caused by: java.lang.LinkageError: com/ibm/websphere/uow/UOWSynchronizationRegistry.registerInterposedSynchronization(Ljavax/transaction/Synchronization;)V (loaded from file:/opt/IBM/WebSphere/AppServer/plugins/ by org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@d30c8d80) called from class org.springframework.transaction.jta.WebSphereUowTransactionManager$UOWActionAdapter (loaded from file:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/UBDSK01Node01Cell/uman-1_0_1_war.ear/uman-1.0.1.war/WEB-INF/lib/spring-tx-4.3.3.RELEASE.jar by[war:uman-1_0_1_war/uman-1.0.1.war]   Local ClassPath: ~[] ... 112 common frames omitted

to solve it you have to make some exclusions in the pom. For javax.transaction we make the exclusion and scope provided:


for jta we make the exclusion:


that’s all.

Having done all this, the application will run smoothly.