Converting fixed fields text record to JSON

Using a REST Service for converting fixed fields text record to json with Fixedfid java library.

Converting fixed fields text record to JSON can be realized in many ways. A solution can be the using of a REST Service combined with the Fixefid java library.

The environment is as a follows:

  • Java 8
  • Spring Boot 2.3.4.RELEASE
  • Spring Web
  • Fixefid 1.1.0
  • Spring Doc Openapi 1.5.0

The Fixefid java library permits to define a fixed fields text record with Java Bean or Java Enum. In this case the definition by Java Bean can be used to annotate a resource representation class of a REST Service.

For instance, we want converting a customer record like this one:

String record = "0000000000000000001Paul                                              Robinson                                          ";

to a json object like this one:

{
"id": 1,        
"firstName": "Paul",        
"lastName": "Robinson"    
}

To model the customer representation, we can create a resource representation class:

@FixefidRecord
public class Customer {
	@FixefidField(fieldLen = 19, fieldOrdinal = 0, fieldType = FieldType.N)
	private Long id;
@FixefidField(fieldLen = 50, fieldOrdinal = 1, fieldType = FieldType.AN)
private String firstName;

@FixefidField(fieldLen = 50, fieldOrdinal = 2, fieldType = FieldType.AN)
private String lastName;

protected Customer() {
}

public Customer(String firstName, String lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
}

@Override
public String toString() {
    return String.format("Customer[id=%d, firstName='%s', lastName='%s']", id, firstName, lastName);
}

public Long getId() {
    return id;
}

public String getFirstName() {
    return firstName;
}

public String getLastName() {
    return lastName;
}
}

The resource representation class is annotated with the Fixefid annotations. Then we can create the record request:

public class RecordRequest {
	private Long requestId;
	private String record;

public String getRecord() {
    return record;
}

public void setRecord(String record) {
    this.record = record;
}

public Long getRequestId() {
    return requestId;
}

public void setRequestId(Long requestId) {
    this.requestId = requestId;
}
}

and the Customer response:

public class CustomerResponse {
	private Long requestId;
	private Long responseId;
	private Customer customer;

public CustomerResponse(Long requestId, Long responseId, Customer customer) {
    this.requestId = requestId;
    this.responseId = responseId;
    this.customer = customer;
}

public Long getRequestId() {
    return requestId;
}
public void setRequestId(Long requestId) {
    this.requestId = requestId;
}
public Long getResponseId() {
    return responseId;
}
public void setResponseId(Long responseId) {
    this.responseId = responseId;
}
public Customer getCustomer() {
    return customer;
}
public void setCustomer(Customer customer) {
    this.customer = customer;
}
}

last, the rest controller:

@RestController
public class CustomerController {
	private final AtomicLong counter = new AtomicLong();

        @PostMapping(path = "/recordtocustomer", consumes = 
        "application/json", produces = "application/json")
        public CustomerResponse recordToCustomer(@RequestBody RecordRequest 
           request) {
        Customer customer = new Customer(null, null);
        new BeanRecord(customer, request.getRecord());
        return new CustomerResponse(request.getRequestId(), 
        counter.incrementAndGet(), customer);
     }
}

With Postman we can test the service:

Here the project of the example on github.

Force dynamic wsdl address location to https scheme

Nowadays all web applications must be secure with https protocol. A typical scenario involve a load balancer wich redirect all incoming https connections, to application servers responding on http port

We have had a problem with wsdl dynamic address location, exposed by an old web application based on Spring MVC 3.0 behind a Load Balancer. A dynamic wsdl is created at runtime by spring-ws with the tag dynamic-wsdl in the spring-ws context:

<sws:dynamic-wsdl id="myService"  portTypeName="myServiceSoap"  
   locationUri="/ws/MyServiceService" targetNamespace="http://www.myapp.it/myapp/schema/myappws">
              <sws:xsd location="/WEB-INF/wsdl/MyService.xsd"/>
</sws:dynamic-wsdl>

the portion of the service wsdl generated was like this:

<wsdl:service name="MySoapService">
    <wsdl:port binding="tns:MySoapSoap11" name="MySoapSoap11">        
         <soap:address location="http://www.myapp.com/myapp/ws/MyService"/>
    </wsdl:port>
</wsdl:service>

The scheme of the address location was http instead that https. To solve this, we have to force the schema to https. We have to create a new bean in the application context:

<bean id="myServiceWsdlDefinitionHandlerAdapter" name="wsdlDefinitionHandlerAdapter" class="it.github.parmag.MyServiceWsdlDefinitionHandlerAdapter">

The name must be “wsdlDefinitionHandlerAdapter”. Then the bean could be like this:

public class MyServiceWsdlDefinitionHandlerAdapter extends WsdlDefinitionHandlerAdapter {
	public CardplusWsdlDefinitionHandlerAdapter() {
	}

	@Override
	protected String transformLocation(String location, HttpServletRequest request) {
		String newLocation = super.transformLocation(location, request);
		newLocation = "https" + newLocation.substring(newLocation.indexOf(":"));
		return newLocation;
	}
}

Now, the address location is https:

<wsdl:service name="MySoapService">
    <wsdl:port binding="tns:MySoapSoap11" name="MySoapSoap11">        
         <soap:address location="https://www.myapp.com/myapp/ws/MyService"/>
    </wsdl:port>
</wsdl:service>

Howto deal with fixed fields text record and JPA Entity

If the persistance layer is realized with JPA, we can mapping record fields directly to Entity fields

Fixefid is a java library wich permits to define a fixed fields text record with Java Bean or Java Enum. Often a text record must be retrieved from data persisted on a database. Or a text record must be persisted on a database.

The solution is to define a mapping from the record’s fields with the persistence model. If the persistance layer is realized with JPA Entities, the mapping can be done directly to the JPA Entity with the Fixefid annotations. Infact a JPA Entity is a POJO, that’s a Java Bean. And so, we can annotate the JPA Entity with the Fixefid annotations to realize the mapping, without the need to create two models, one for the record and another one for the JPA Entity.

The environment is as a follows:

  • Java 8
  • Spring Boot 2.3.4.RELEASE
  • Spring Data JPA
  • Fixefid 1.1.0
  • H2 Database

For example we can have a Customer bean like this one:

@Entity
@FixefidRecord
public class Customer {
	@Id
	@GeneratedValue(strategy = GenerationType.AUTO)
	@FixefidField(fieldLen = 19, fieldOrdinal = 0, fieldType = FieldType.N)
	private Long id;
	
	@FixefidField(fieldLen = 50, fieldOrdinal = 1, fieldType = FieldType.AN)
	private String firstName;
	
	@FixefidField(fieldLen = 50, fieldOrdinal = 2, fieldType = FieldType.AN)
	private String lastName;

	protected Customer() {
	}

	public Customer(String firstName, String lastName) {
		this.firstName = firstName;
		this.lastName = lastName;
	}

	@Override
	public String toString() {
		return String.format("Customer[id=%d, firstName='%s', lastName='%s']", id, firstName, lastName);
	}

	public Long getId() {
		return id;
	}

	public String getFirstName() {
		return firstName;
	}

	public String getLastName() {
		return lastName;
	}
}

The Customer above is annotated with Entity and FixefidRecord. The fields are annotated with FixefidField and other JPA annotations. To obtain the record from the database:

Customer customer = repository.findById(1L);
String record = new BeanRecord(customer).toString();

To save the record to the database:

String newRecord = "0000000000000000001Paul                                              Robinson                                          ";
Customer newCustomer = new Customer();
new BeanRecord(newCustomer, newRecord);
repository.save(newCustomer);

I made a video tutorial where the example above is explained in the detail way.

Here the project of the example on github.

Howto deal with fixed fields text record without substring

Howto avoid substring approach to deal with fixed fields formatted text records. The solution is the Java library Fixefid

Often we have to develop Java methods to deal with fixed fields formatted text records. The format is often used nowadays yet to files transfer between mainframe and unix machines. Almost always the approach is using substring like this:

String address = record.substring(10, 20);
String location = record.substring(20, 30);

But that is an error prone pattern. The records are length thousands of chars with many many fields and make a mistake with the offset is not so rarely. And moreover when you have to format the fields to create the record, you have to deal with string right padding, zero lead numeric left padding, boolean format, date format, decimal format, decimal separator, etc etc…

The solution is using a Java library like Fixefid, wich permits to start from a specification in excel, model the record with Java Bean or Java Enum and deal with fields by getter, setter and toString Java methods.

I’ve made a video tutorial where I explain in the detail way the process to start from an excel like this:

to create a Java Bean model like this:

to obtain a records txt file like this:

In this way we can using the standard java bean development to deal with fixed fields formatted text files.

For example you can create a Java Bean like this:

@FixefidRecord(recordLen = 600)
public class Student extends Person {
@FixefidField(fieldLen = 10, fieldOrdinal = 8, fieldType = FieldType.N,       
         fieldMandatory = FieldMandatory.INOUT)
private Long studentId;
….
}

and create a BeanRecord like this:

Student student = createStudent();
BeanRecord studentRecord = new BeanRecord(student, null, null, 
       createStudentMapFieldExtendedProperties());
studentRecord.toNormalize();
String studentRecordAsString = studentRecord.toString();
System.out.println("Student Record=[" + studentRecordAsString + "]");

The video tutorial explains the whole process in the detail way.

Here the java project developed on the video tutorial.

If you don’t want to use Java Bean, the same record can be modeled with Java Enum.

Where are my java projects?

The question is: what eclipse I used to import a java project?

After years and years of tests, developments, new projects, new versions of eclipse, I’m in the situation of opening the folder in which all my projects are present but not remembering which version of eclipse they were imported with. And so the other day I asked myself, how do I know, given a java project, which version of eclipse  did I use? This without opening eclipse … obviously. The answer is as follows: for each eclipse workspace, just look in the folder

${eclipse_workspace}\.metadata\.plugins\org.eclipse.core.resources\.projects

Got it!

Disable Spring Cloud Stream support for testing

TestSupportBinder is a minimal binder that does nothing and is not useful for integration test between services

A short time ago we started the development of a project based on Microservices Architecture. The intent is to create services based on REST Api, which receive messages and write them on Kafka topics. Other services read messages from the Kafka topics and write them to the database.

The environment is as follows:

  • Java 11
  • Spring Core 5.1
  • Spring Boot 2.1
  • Spring Cloud Stream 2.1
  • Spring Web 5.1
  • Apache Kafka 2.0.1

The IDE is the new STS 4.  By default, if you use Spring Initializer to create a Spring Cloud Stream based project, support for the Spring Cloud Test dependency is added:

<dependency>
     <groupId>org.springframework.cloud</groupId>
     <artifactId>spring-cloud-stream-test-support</artifactId>
     <scope>test</scope>
</dependency>

the dependency ensures that the TestSupportBinder class can be used for the test phase. TestSupportBinder is a minimal binder that does nothing about binding consumers.
I find that class not very useful, even for the test phase itself. Surely it is not useful for integration test between the various services.
In fact, in our case, by launching services from STS, you could write on the topic but the listeners received no messages. By launching services via maven instead, everything worked fine. This is because the scope test of the dependency assures that by launching the command

mvn spring-boot:run

the TestSupportBinder class is not loaded by Spring Boot autoconfiguration. However, if we run the application from STS (in RUN or DEBUG), TestSupportBinder is loaded, and not Kafka as desired. To disable the test from STS, you need to add the annotation

@SpringBootApplication(exclude = TestSupportBinderAutoConfiguration.class)

as in this example:

@SpringBootApplication(exclude = TestSupportBinderAutoConfiguration.class)
@EnableBinding(MsgStreams.class)
public class StreamsConfig {

}

or, even better, add the following line in the application.properties file:

spring.autoconfigure.exclude=org.springframework.cloud.stream.test.binder.TestSupportBinderAutoConfiguration

in this way, the integration test between the various services will go well!

Partial commit and job restart with Spring Batch

If a batch fails after partial commit, it must be possible to start again with the processing of the file by skipping the lines already committed

A classic batch is the processing of a file, for which records are read, for each record the data are processed and are persisted on database (reader, processor and writer).

In case the file is large and contains thousands of records, partial commits must be expected during processing. For example, every 1000 records, we can decide to commit the processing on the database.

Through Spring Batch it is very easy to get partial commits, it’s a simple parameter that is passed to the StepBuilder. In the case that it’s necessary to implement more complex partial commit policies, it is possible to implement custom completition policies (which in this case we will not see because it is not the subject of this article).

Finally, if a batch fails after partial commit, it must be possible to start again with the processing of the file by skipping the lines already committed. This last thing is also expected by Spring Batch, but we must add a few lines of code, it is not a simple configuration parameter. Also in this case it’s possible to implement custom skip and retry policies (and also in this case we will not see why not the subject of this article).

The environment is as follows:

  • Java7
  • Spring Boot 1.1.8
  • Spring Batch 3.0

There are two concepts, the job instance and job execution. An instance of a job is accomplished through n executions (typically one, or more if there have been failures). Furthermore, only a failed job can be restarted.

To implement the restart we need the jobRegistry, jobOperator, jobExplorer and jobLauncher. Here is the complete code of the batch configuration.

Basically these are the steps:

  • register the job in the jobRegistry
  • get job instances through the jobOperator
  • given the last instance, get executions through the jobOperator
  • through the jobExplorer check if the last execution has failed
  • in case the last execution has failed, the job must be restarted via the jobOperator
  • in case the last execution was successful, launch a new job instance via the jobLauncher

Spring Batch takes care of managing the restart starting from the first uncommitted record.

Thanks Spring Batch 🙂

 

 

 

 

Performance tuning for File Reading with Java 8 and Parallel Streams

Is it always convenient to use parallel streams?

Java 8 has introduced with the Stream the possibility of using in a very simple way all the resources made available by the hardware, in particular the cores of the multicore architectures. And all this with the paradigm of declarative and non-imperative programming.

For example, suppose you have to do a batch that looks for a string in all the files that have a certain prefix in a certain directory. The batch must notify in a log in which files the string is found.

It is interesting to compare the two approaches, pre java 8 and java 8 with the streams.

I used my dev computer, a Samsung laptop with this features:

  • Intel Core i7-4500U (2 Core, 4 Thread) 1.8GH
  • 8 GB RAM
  • 256 GB SSD
  • Windos 8.1 Pro 64 Bit
  • Java 1.8.0_101

Having 4 Threads I expect the performance of the parallel stream solution to be 400% better than the classic pre java 8 version.

The code that uses the pre java style 8 is as follows: StringMatchingOld

The code that uses the streams is as follows: StringMatching

The code that uses the parallel streams is as follows: ParallelStringMatching

These are the results (4 files read):

File Reading Performance

What is going on? Truly strange, there is no benefit in the use of the stream (I think also from the point of view of the code… I don’t think it’s more readable but this is another story… I’m getting older…). Furthermore, we see that using parallel streams does nothing but make things worse.

Conclusions

It seems that reading the files does not find any improvement in the use of the streams. However, the fact remains that declarative programming makes it possible to abstract from current hardware, so maybe on a server with better hardware than my development pc, parallel streams performance is better.

Or maybe I’m doing something wrong … I’m looking forward to that.