Nov 10, 2021 4:05:45 PM Thomas Dumont

Create a batch with Spring Batch

Introduction

Applications need to synchronize data from other applications. In general, these treatments are performed at night by Batchs. Lutèce proposes the use of Spring Batch in order to carry out these treatments.

Warning : Spring Batch is not integrated in the core of Lutetia. This is to help develop an external application dedicated to the synchronization of these data.

Spring Batch

Spring Batch is a framework dedicated to batching. It offers features common to batches:

Reading from different data sources
Writing on different target
Management of a paginated treatment
Transaction Management
Error management
Scheduling of the batch steps ...

Development of a batch

Parent pom for batch

Lutèce recommends the use of a parent pom for batch:

<parent>
    <artifactId>batch-parent</artifactId>
    <groupId>fr.paris.lutece.batches</groupId>
    <version>1.0.0</version>
</parent>

This parent pom aims to :

Recover the dependencies Spring batch,
Recover the MySQL driver dependency,
Recover dependencies JUnit, Log4J, Apache Commons ...
Recover Jalopy to easily respect the rules of development Lutèce for the formatting of the code.
It preconfigures compilation options (Java version)
It configures the deployment repository.

Writing the batch

The examples below are based on the batch-ivory [1] Additional documentation is available at Spring [2]

pom.xml

Below is an example of (the dependencies fr.paris.lutece.batches. * Are described later in this document):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <parent>
       <artifactId>batch-parent</artifactId>
       <groupId>fr.paris.lutece.batches</groupId>
       <version>1.0.0</version>
   </parent>

   <modelVersion>4.0.0</modelVersion>
   <groupId>fr.paris.lutece.batches</groupId>
   <artifactId>batch-ivoire</artifactId>
   <version>1.0.1-SNAPSHOT</version>
   <packaging>jar</packaging>
   <name>Batch Ivoire</name>
   <description>
       <![CDATA[Synchronize the Ivoire application states with PASU application workflow actions.]]>
   </description>

   <repositories>
       <repository>
           <id>lutece</id>
           <name>luteceRepository</name>
           <url>http://dev.lutece.paris.fr/maven_repository</url>
           <snapshots>
               <enabled>false</enabled>
           </snapshots>
       </repository>
   </repositories>

   <properties>
       <json.version>2.4</json.version>
       <componentName>batch-ivoire</componentName>
       <jiraProjectName>BATCHIVOIRE</jiraProjectName>
       <jiraComponentId>10637</jiraComponentId>
   </properties>

   <dependencies>
       <dependency>
           <groupId>fr.paris.lutece.batches</groupId>
           <artifactId>batch-library-httpaccess</artifactId>
           <version>1.0.0</version>
           <type>jar</type>
       </dependency>
       <dependency>
           <groupId>fr.paris.lutece.batches</groupId>
           <artifactId>batch-library-signrequest</artifactId>
           <version>1.0.0</version>
           <type>jar</type>
       </dependency>
       <dependency>
           <groupId>fr.paris.lutece.batches</groupId>
           <artifactId>batch-library-logger</artifactId>
           <version>1.0.0</version>
           <type>jar</type>
       </dependency>
       <dependency>
           <groupId>fr.paris.lutece.batches</groupId>
           <artifactId>batch-library-notification</artifactId>
           <version>1.0.0</version>
           <type>jar</type>
       </dependency>
       <dependency>
           <groupId>net.sf.json-lib</groupId>
           <artifactId>json-lib</artifactId>
           <version>${json.version}</version>
           <classifier>jdk15</classifier>
       </dependency>
   </dependencies>

   <build>
       <plugins>
           <plugin>
               <artifactId>maven-assembly-plugin</artifactId>
               <configuration>
                   <finalName>
                       ${pom.artifactId}-${project.version}
                   </finalName>
                   <descriptors>
                       <descriptor>
                           src/assembly/src.xml
                       </descriptor>
                   </descriptors>
               </configuration>
               <executions>
                   <execution>
                       <id>make-assembly</id>
                       <phase>package</phase> 
                       <goals>
                           <goal>single</goal>
                       </goals>
                   </execution>
               </executions>
           </plugin>
       </plugins>
   </build>

   <scm>
       <connection>scm:svn:http://dev.lutece.paris.fr/svn/lutece/portal/trunk/batches/batch-ivoire</connection>
       <developerConnection>scm:svn:https://dev.lutece.paris.fr/svn/lutece/portal/trunk/batches/batch-ivoire</developerConnection>
       <url>http://dev.lutece.paris.fr/viewvc/portal/trunk/batches/batch-ivoire</url>
   </scm>

</project>

General principles

The launch of a batch consists of launching one or more Jobs (tasks performed by the batch).
The JobLauncher is used to launch a Job.
A Job is composed of several Step, corresponding to the steps of the Job.
ItemReaders allow playback of a stream (example: a file)
Many utility classes allow for example the reading of file with separator, or the reading in base ...
ItemWriter allow writing a stream (example: a file)
ItemProcessors allow to transform a stream from one type to another (example: the object read from the file to another type of business object)
Listeners can listen to events at each level (step, reader, writer, job ...)

Project structure

src
- hand
  - assembly: descriptor for batch packaging
  - bin: batch launch scripts
  - java: Java sources
  - logs
  - resources: resources - Spring context, loggers, application properties
  - tmp
- test: directories used for testing
pom.xml

Spring context

The set of classes to implement must be referenced in the Spring context of the application. It must contain the following header :

<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
   xmlns:sb="http://www.springframework.org/schema/batch"
   xmlns:context="http://www.springframework.org/schema/context"
   xmlns:jdbc="http://www.springframework.org/schema/jdbc"
   xmlns:tx="http://www.springframework.org/schema/tx"
   xmlns:p="http://www.springframework.org/schema/p"
   xsi:schemaLocation="
       http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
       http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-jdbc-3.0.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">
   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

In order to outsource the properties, you need to use a property-placeholder :

<context:property-placeholder location = "classpath: batch_ivory.properties" />

It can also be useful to declare the configuration by annotation :

<context:annotation-config />

Assembly

The assembly folder contains a src.xml file describing the packaging of the batch.

<assembly>
   <id/>
   <formats>
       <format>tar.gz</format>
   </formats>
   <fileSets>
       <fileSet>
           <directory>target</directory>
           <outputDirectory>lib</outputDirectory>
           <includes>
               <include>batch-ivoire*.jar</include>
           </includes>
       </fileSet>
       <fileSet>
           <directory>src/bin</directory>
           <outputDirectory>bin</outputDirectory>
       </fileSet>
       <fileSet>
           <directory>src/conf</directory>
           <outputDirectory>conf</outputDirectory>
       </fileSet>
       <fileSet>
           <directory>src/tmp</directory>
           <outputDirectory>tmp</outputDirectory>
       </fileSet>    
       <fileSet>
           <directory>src/main/resources</directory>
           <outputDirectory>conf</outputDirectory>
           <includes>
               <include>*.properties</include>
           </includes>            
       </fileSet>        
       <fileSet>
           <directory>src/lib-ext</directory>
           <outputDirectory>lib-ext</outputDirectory>        
       </fileSet>
       <fileSet>
           <directory>src/logs</directory>
           <outputDirectory>logs</outputDirectory>
       </fileSet>
   </fileSets>
   <dependencySets>
       <dependencySet>
             <outputDirectory>/lib-ext</outputDirectory>
             <excludes>
               <exclude>batch-ivoire*.jar</exclude>
             </excludes>
       </dependencySet>
   </dependencySets>
</assembly>

Realization

JobLauncher

The overload of a SimpleJobLauncher makes it possible to carry out treatments in case of error of the Job.

The JobExecutionListener interface allows pre- and post-Job processing.

Job

A Job can be likened to a task performed by the batch. A SimpleJob allows you to execute the Step sequentially. Example of defining a Job (simpleJob can be used for several jobs) :

<bean id="simpleJob" class="org.springframework.batch.core.job.SimpleJob" abstract="true"
    p:jobRepository-ref="jobRepository"
    p:restartable="false" />
<sb:job id="job" parent="simpleJob">
    <sb:step id="executeActionStep" parent="simpleStep">
        <sb:tasklet>
            <sb:chunk reader="executeActionReader" processor="executeActionProcessor" writer="executeActionWriter" 
                     commit-interval="${job.commit.interval}" />
         </sb:tasklet>
    </sb:step>
</sb:job>

Step

A step defines a reader, a processor, a writer.

Reader

A reader reads a data source. Various implementations make it easy to manage File, JDBC, Hibernate, JPA access... Example of reader based on file reading :

<bean id="executeActionReader" class="fr.paris.lutece.batches.ivoire.job.IvoireItemReader">
       <property name="resource" value="file:${ivoire.file.path}" />
       <property name="file" value="file:${ivoire.file.path}" />
       <property name="lineMapper">
           <bean class="fr.paris.lutece.batches.ivoire.job.internal.IvoireLineMapper">
               <property name="lineTokenizer">
                   <bean class="fr.paris.lutece.batches.ivoire.util.transform.LengthLineTokenizer"
                       p:names="idIvoire,idFolder,processType,idDemandDemat,idMerchantAccount,ivoireCode,labelIvoireCode,dateFinalVisa,inspectorLastNameFirstName,inspectorPhoneNumber"
                       p:listLengthDelimiters="11,13,11,20,50,3,100,19,60,14" 
                       p:strict="false" />
               </property>
               <property name="fieldSetMapper">
                   <bean class="fr.paris.lutece.batches.ivoire.job.internal.IvoireFieldSetMapper" />
               </property>
           </bean>
       </property>
   </bean>

The ItemReadListener interface is used to manage read errors, and pre/post processing.

Writer

ItemWriter is the interface that defines the writing of data. Implementations facilitate this writing. The ItemWriteListener interface is used to manage write errors, and pre/post processing.

Launch of the batch

You have to use Spring's CommandLineJobRunner, and initialize the classpath, the location of the logs, JAVA_OPTS...

$ JAVA -cp $ TMP_CP $ JAVA_OPTS -Dproducts.home = $ PRODUCTS_HOME -Dlogs.home = $ PRODUCTS_HOME / logs org.springframework.batch.core.launch.support.CommandLineJobRunner batch_ivoire_context.xml job >> $ PRODUCTS_HOME / logs / batch- ivory.out 2> & 1

Libraries

Lutèce offers several libraries that can be useful in the development of a batch

batch-library-signrequest

This library allows you to sign links.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-signrequest</artifactId>
    <version>1.0.0</version>
</dependency>

Configuration

In the context.xml file of the batch, add the bean corresponding to the request authenticator as follows :

<bean id="hashService" class="fr.paris.lutece.batches.util.signrequest.security.Sha1HashService" />
<bean id="requestAuthenticator" class="fr.paris.lutece.batches.util.signrequest.HeaderHashAuthenticator"
        p:hashService-ref="hashService"
        p:privateKey="${workflow.ws.rest.privateKey}" />

The hash key must always be declared as follows :

workflow.ws.rest.privateKey = MyPrivateKey

Use

@Autowired
private RequestAuthenticator _requestAuthenticator;

batch-library-httpaccess

This library allows to connect in HTTP to a remote application, and supports the signature of the requests.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-httpaccess</artifactId>
    <version>1.0.0</version>
</dependency>

Configuration

In the context.xml file of the batch, the bean corresponding to the httpaccess must be added as follows :

<bean id="httpAccess" class="fr.paris.lutece.batches.util.httpaccess.HttpAccess"
       p:proxyHost="${httpAccess.proxyHost}"
       p:proxyPort="${httpAccess.proxyPort}"
       p:proxyUserName="${httpAccess.proxyUserName}"
       p:proxyPassword="${httpAccess.proxyPassword}"
       p:hostName="${httpaccess.hostName}"
       p:domainName="${httpAccess.domainName}"
       p:realm="${httpaccess.realm}"
       p:noProxyFor="${httpAccess.noProxyFor}"
       p:contentCharset="${httpAccess.contentCharset}" />

And have in the batch properties file :

################################################## ##############
## Proxy settings

httpAccess.proxyHost =
httpAccess.proxyPort =
httpAccess.proxyUserName =
httpAccess.proxyPassword =
# computer name
httpaccess.hostName =
# domain name
httpAccess.domainName =
httpaccess.realm =
httpAccess.noProxyFor = 127.0.0.1 localhost * pwd
# Content charset - optionnal
httpAccess.contentCharset = UTF-8

Use

@Autowired
private HttpAccess _httpAccess;

batch-library-logger

This library provides easy logging, with 3 loggers.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-logger</artifactId>
    <version>1.0.0</version>
</dependency>

Configuration

In the context.xml file of the batch, add the bean corresponding to the selected logger (see below) as follows :

<bean id="logger" class="<classLogger>" scope="singleton"
       p:loggerInfo="<nomLoggerInfo>"
       p:loggerDebug="<nomLoggerDebug>"
       p:loggerError="<nomLoggerError>"
       p:appenderInfo="<nomAppenderInfo>"
       p:appenderDebug="<nomAppenderDebug>"
       p:appenderError="<nomAppenderError>"
/>

It is then necessary to configure the file log4j.properties (or log4j.xml) in a classic way.

Use

SimpleLogger : Allows simple logging into the log4j logger.
SimpleBatchLogger : In addition to logging in the log4j logger, it will store errors as a String and add them to a list that can be retrieved by calling the getErrors () method.
SimpleErrorListBatchLogger : It's the same as the SimpleBatchLogger, except that it stores errors in an object implementing the IError interface.
AbstractErrorMapBatchLogger : This is an abstract class for storing errors in a map (used for example for line error handling).

batch-library-notification

This library allows email notifications.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-notification</artifactId>
    <version>1.0.0</version>
    <type>jar</type>
</dependency>

Configuration

In the batch context.xml file, you must add the bean corresponding to the mail service as follows :

<bean id="mailService" class="fr.paris.lutece.batches.util.notification.MailService"
       p:mailingList="${mail.mailinglist}"
       p:mailSubject="${mail.subject}"
       p:mailHost="${mail.host}"
       p:mailType="${mail.type.html}"
       p:mailNoReply="${mail.noreply.email}"
       p:mailEncoding="${mail.encoding}" />

And in the batch properties file :

################################################## ##############
# Mail settings

mail.host = smtpmail.mdp
mail.noreply.email=noreply@paris.fr
mail.type.html = text / html; charset = "utf-8"
# list of mails separated by a comma (,) to be notified when there are errors
mail.mailinglist=lutece@paris.fr
mail.subject = Ivory Batch Error Notification
mail.encoding = UTF-8

Use

@Autowired
private INotificationService _mailService;

batch-library-workflow

This library provides Lutece workflow services in a batch. It also allows you to retrieve all information (actions, tasks, etc ...) but also to perform workflow actions.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-workflow</artifactId>
    <version>1.0.0</version>
    <type>jar</type>
</dependency>

Configuration

In the context.xml file of the batch, you must add the import of the context file of the librarie :

<import resource = "classpath: libraries / workflow_context.xml" />

Add the bean corresponding to the workflow dataSource:

<bean id="workflow.dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
                p:driverClassName="${batch.jdbc.workflow.driver}"
		p:url="${batch.jdbc.workflow.url}"
                p:username="${batch.jdbc.workflow.user}"
                p:password="${batch.jdbc.workflow.password}"
                p:validationQuery="${batch.jdbc.workflow.validationQuery}"
                p:testWhileIdle="${batch.jdbc.workflow.testWhileIdle}" />

Add the configuration of the workflow dataSource to the batch properties file :

batch.jdbc.workflow.driver=com.mysql.jdbc.Driver
batch.jdbc.workflow.url=jdbc:mysql://localhost/lutece?autoReconnect=true&useUnicode=yes&characterEncoding=utf8
batch.jdbc.workflow.user=root
batch.jdbc.workflow.password=lutece
batch.jdbc.workflow.testWhileIdle=true
batch.jdbc.workflow.validationQuery=SELECT 1

Use

@Autowired
private IIWorkflowService _workflowService;

Example

An example of a batch using this library is available at http://dev.lutece.paris.fr/svn/lutece/portal/trunk/batches/batch-workflowsample/

batch-library-workflow-notifycrm

This library provides the CRM workflow notification service in a batch. It also allows you to execute the notification actions to a CRM webapp.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-workflow-notifycrm</artifactId>
    <version>1.0.0</version>
    <type>jar</type>
</dependency>

Configuration

In the context.xml file of the batch, you must add the import of the context file of the librarie:

<import resource = "classpath:libraries/workflow-notifycrm_context.xml" />

Add the beans corresponding to the workflow-notifycrm and directory dataSources:

<bean id="workflow-notifycrm.dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
                p:driverClassName="${batch.jdbc.workflow-notifycrm.driver}"
                p:url="${batch.jdbc.workflow-notifycrm.url}"
                p:username="${batch.jdbc.workflow-notifycrm.user}"
                p:password="${batch.jdbc.workflow-notifycrm.password}"
                p:validationQuery="${batch.jdbc.workflow-notifycrm.validationQuery}"
                p:testWhileIdle="${batch.jdbc.workflow-notifycrm.testWhileIdle}" />
<bean id="directory.dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
                p:driverClassName="${batch.jdbc.directory.driver}"
                p:url="${batch.jdbc.directory.url}"
                p:username="${batch.jdbc.directory.user}"
                p:password="${batch.jdbc.directory.password}"
                p:validationQuery="${batch.jdbc.directory.validationQuery}"
                p:testWhileIdle="${batch.jdbc.directory.testWhileIdle}" />

Add the configuration of the workflow-notifycrm and directory dataSources in the batch properties file:

batch.jdbc.workflow-notifycrm.driver=com.mysql.jdbc.Driver
batch.jdbc.workflow-notifycrm.url=jdbc:mysql://localhost/lutece_workflow_notifycrm?autoReconnect=true&useUnicode=yes&characterEncoding=utf8
batch.jdbc.workflow-notifycrm.user=root
batch.jdbc.workflow-notifycrm.password=lutece
batch.jdbc.workflow-notifycrm.testWhileIdle=true
batch.jdbc.workflow-notifycrm.validationQuery=SELECT 1

batch.jdbc.directory.driver=com.mysql.jdbc.Driver
batch.jdbc.directory.url=jdbc:mysql://localhost/lutece_directory?autoReconnect=true&useUnicode=yes&characterEncoding=utf8
batch.jdbc.directory.user=root
batch.jdbc.directory.password=lutece
batch.jdbc.directory.testWhileIdle=true
batch.jdbc.directory.validationQuery=SELECT 1

Info

If both plugins use the same dataSource, it is possible to use aliases, which avoids declaring several times the same dataSource with different IDs :

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
                p:driverClassName="${batch.jdbc.driver}"
                p:url="${batch.jdbc.url}"
                p:username="${batch.jdbc.user}"
                p:password="${batch.jdbc.password}"
                p:validationQuery="${batch.jdbc.validationQuery}"
                p:testWhileIdle="${batch.jdbc.testWhileIdle}" />
<alias name="dataSource" alias="directory.dataSource" />
<alias name="dataSource" alias="workflow-notifycrm.dataSource" />

Example

An example of a batch using this library is available on Example

batch-library-crmclient

This library brings the CAD implementation to add CRM notifications to the processing queue, so that the CRM notification batch of the CRM webapp can send the notifications.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-crmclient</artifactId>
    <version>1.0.0</version>
    <type>jar</type>
</dependency>

Configuration

In the context.xml file of the batch, you must add the import of the context file of the librarie:

<import resource = "classpath:libraries/crmclient_context.xml" />

Add the beans corresponding to the dataSource of crmclient :

<bean id="crmclient .dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
                p:driverClassName="${batch.jdbc.crmclient .driver}"
                p:url="${batch.jdbc.crmclient .url}"
                p:username="${batch.jdbc.crmclient .user}"
                p:password="${batch.jdbc.crmclient .password}"
                p:validationQuery="${batch.jdbc.crmclient .validationQuery}"
                p:testWhileIdle="${batch.jdbc.crmclient .testWhileIdle}" />

Add the crmclient dataSource configuration to the batch properties file:

batch.jdbc.crmclient .driver=com.mysql.jdbc.Driver
batch.jdbc.crmclient .url=jdbc:mysql://localhost/lutece_directory?autoReconnect=true&useUnicode=yes&characterEncoding=utf8
batch.jdbc.crmclient .user=root
batch.jdbc.crmclient .password=lutece
batch.jdbc.crmclient .testWhileIdle=true
batch.jdbc.crmclient .validationQuery=SELECT 1

Once the batch has been packaged (mvn package), you must configure the conf/libraries/crmclient.roperties file and configure the Base URL of the CRM webapp :

crmclient.crm.rest.webapp.baseurl=http://localhost:8080/lutece

Example

An example of a batch using this library is available on http://dev.lutece.paris.fr/svn/lutece/portal/trunk/batches/batch-workflowsample/

batch-library-freemarker

This library brings the implementation of the FreeMarker service. It also allows you to transform notification messages containing freemarker labels.

Declaration

<dependency>
    <groupId>fr.paris.lutece.batches</groupId>
    <artifactId>batch-library-freemarker</artifactId>
    <version>1.0.0</version>
    <type>jar</type>
</dependency>

Configuration

In the context.xml file of the batch, you must add the import of the context file of the librarie :

<import resource = "classpath:libraries/freemarker_context.xml" />

Once the batch has been packaged (mvn package), you have to configure the conf/libraries/freemarker.properties file and configure the absolute path to the Lutece webapp

freemarker.webapp.path=/home/user/apps/tomcat/webapps/lutece/

Example

An example of a batch using this library is available on http://dev.lutece.paris.fr/svn/lutece/portal/trunk/batches/batch-workflowsample/

Top of the page

Create a batch with Spring Batch

Introduction

Spring Batch

Development of a batch

Parent pom for batch

Writing the batch

General principles

Project structure

Spring context

Assembly

Realization

JobLauncher

Job

Step

Reader

Writer

Launch of the batch

Libraries

batch-library-signrequest

Declaration

Configuration

Use

batch-library-httpaccess

Declaration

Configuration

Use

batch-library-logger

Declaration

Configuration

Use

batch-library-notification

Declaration

Configuration

Use

batch-library-workflow

Declaration

Configuration

Use

Example

batch-library-workflow-notifycrm

Declaration

Configuration

Example

batch-library-crmclient

Declaration

Configuration

Example

batch-library-freemarker

Declaration

Configuration

Example