Generate statistics with Elastic Search and Kibana (Plugin ElasticData)

The ElasticData plugin

The plugin ElasticData allows you to create indexes and add timestamped data to a Elastic Search server. to produce Kibana dashboards.

The principle of this plugin is to manipulate DataSources and to match each of them an index in ElasticSearch.

The plugin manages two types of objects:

• datasources : data sources
• dataobjects : the objects of a datasource

The attributes of a DataSource are:

• its identifier (alphanumeric)
• his name
• the name of the corresponding index in ElasticSearch
• the type of objects in the index

A DataSource must provide a list of all its dataobjects

A DataObject must simply return a timestamp

Here is the Back Office interface that allows you to start creating indexes :

Creating an ElasticData module

Here is a sample code for the dataObject :

/**
 * My DataObject
 */
public class MyDataObject implements DataObject
{
    // fields
    private String _strAttribute1;
    ...
    private String _strAttributeN;

    // getters and setters : all those attributes will be created in the index 
    public String getAttribute1()  { ... }
    public void setAttribute1( ... ) { ... } 
    ...

    // implementation of the DataObject interface
    /**
     * {@inheritDoc}
     */
    @Override
    public String getTimestamp( )
    {
        // time in milliseconds
        ...
    }
}

DataSource example :

/**
 * My DataSource
 */
public class MyDataSource extends AbstractDataSource implements DataSource
{
    @Override
    public Collection<DataObject> getDataObjects( )
    {
         MyObjectDAO dao = new MyObjectDAO(); 
         return dao.selectAll();
    }
}

with DAO :

/**
  * MyObjectDAO 
  */
public class MyObjectDAO 
{
    private static final String SQL_QUERY_SELECTALL = "SELECT ... FROM ... ";

    /**
     * Load the data of all the objects and returns them as a list
     *
     * @return The list which contains the data of all the objects
     */
    public Collection<DataObject> selectAll()
    {
        Collection<DataObject> listObjects= new ArrayList<>();
        DAOUtil daoUtil = new DAOUtil( SQL_QUERY_SELECTALL );
        daoUtil.executeQuery();

        while( daoUtil.next() )
        {
            MyDataObject object= new MyDataObject ();

            object.setAttribute1( daoUtil.getString( 1 ) );
            object.setAttribute2( daoUtil.getString( 2 ) );
            ...

            listObjects.add( object );
        }

        daoUtil.free();
        return listObjects;
    }
}

The DataSource must be declared in the Spring context file of the module to be found by the ElasticData plugin. Here is an example of a statement:

<!-- My datasource -->
<bean id="elasticdata-mymodule.dataSource" class="fr.paris.lutece.plugins.elasticdata.business.mymodule.MyDataSource" >
    <property name="id" value="MyDataSource" />
    <property name="name" value="My Data Source" />
    <property name="targetIndexName" value="datasource" />
    <property name="dataType" value="mydataobject" />
</bean>

You can also defer the indexing of your documents by saving the identifiers of the resources to be processed. The incremental daemon will index them.

/**
     * {@inheritDoc}
     */
    @Override
    public void addedResource( ResourceEvent event )
    {
        new Thread( ( ) -> {
            DataSourceIncrementalService.addTask( _formsDataSource.getId( ), event.getIdResource( ), IndexerAction.TASK_CREATE);
        } ).start( );
    }

The Kibana plugin

Here is an example of a dashboard presented by the plugin Kibana :