Indexing Form Data from the Forms Plugin to Elasticsearch

The elasticdata plugin combined with the elasticdata-forms module (to index information related to form responses) and the elasticdata-workflow module (to index information on the lifecycle of responses) allows sending data related to form responses to an Elasticsearch index:

With each indexing, the application sends a series of entries, each containing data related to a form response in a specific workflow state.
Each entry includes information on:
- The associated form and responses to various form questions
- Basic information such as the creation date, response identifier, and values for fields selected in the back office for indexing (Requires the elasticdata-forms module)
- Workflow history: state name, name of the action executed before arriving in this state, etc. (Requires the elasticdata workflow module)

For each response, one of the entries will correspond to its current state. It will have the type formResponse. The other entries will have the type formResponseHistory and provide information on the past states of the response (workflowState) and triggered actions (actionName).

Summary

Uses

It is possible to track the lifecycle of requests via:

The Workflow state in which the response is found in entries of type formResponse
The various dates. Note that only the creation date of the response is provided as a timestamp (Timestamps field). The last modification date is provided as a duration in milliseconds relative to this date (CompleteDuration field).
The form response ID (FormResponseId field)
Other information corresponds to values entered in the form fields

Indexing daemons allow managing the indexing frequency, more or less frequent depending on tracking needs.

Dates and Duration

When the documentTypeName field is formResponse
- The timestamp field indicates the creation date of the form response
- The completeDuration field gives the duration in milliseconds between the creation date of the response and the date of transition to the current workflow state
- The taskDuration field is not provided
When the documentTypeName field is formResponseHistory
- The timestamp field indicates the date of transition to the considered workflow state
- The completeDuration field gives the duration in milliseconds between the creation date of the response and the date of transition to the considered workflow state
- The taskDuration field gives the duration in milliseconds between the date of transition to the previous workflow state and the considered workflow state (or in other words, the duration during which the response was in the previous state).
- The only way to know if the request is finalized or ongoing is to look at the workflow state. For the duration, it will make no difference

Configure Form Fields for Indexing

The FORM DATA MANAGEMENT FOR ELASTICSEARCH configuration menu is accessible from the back office. The user account must have the ELASTICDATA_FORMS_MANAGEMENT right.

Administration path

For each form created on the application, it is possible to define specific indexing. The fields are ordered according to the form and are selectable.

Selectable fields for indexing

Example

{
    "_index": "f10_rec_formsdatasource",
    "_type": "_doc",
    "_id": "FORMSGEN_FormsDataSource_382",
    "_score": 1,
    "_source": {
        "parentName": "TEST form 14092023 - Réservation #1 [AA]",
        "formResponseId": 382,
        "formId": 47,
        "userResponses": {},
        "formName": "TEST form 14092023 - Réservation #1 [AA]",
        "parentId": "47",
        "documentTypeName": "formResponse",
        "completeDuration": 193000,
        "workflowState": "Réservation accordée",
        "actionName": "Accordé",
        "taskDuration": 0,
        "userResponsesMultiValued": {},
        "worflowAdminCreator": null,
        "unitName": null,
        "ame": null,
        "prefixedDayOfWeek": "4 - jeudi",
        "prefixedMonth": "09 - septembre",
        "timestamp": "1694703688000",
        "month": "septembre",
        "hour": "17",
        "dayOfWeek": "jeudi"
    }
}

{
    "_index": "f10_rec_formsdatasource",
    "_type": "_doc",
    "_id": "FORMSGEN_FormsDataSource_formResponseHistory_681586",
    "_score": 1,
    "_source": {
        "parentName": null,
        "formResponseId": 382,
        "formId": 47,
        "userResponses": {},
        "formName": "TEST form 14092023 - Réservation #1 [AA]",
        "parentId": "26",
        "documentTypeName": "formResponseHistory",
        "completeDuration": 49000,
        "workflowState": "Réception agent",
        "actionName": "AR",
        "taskDuration": 49000,
        "userResponsesMultiValued": {},
        "worflowAdminCreator": "auto",
        "unitName": null,
        "ame": null,
        "prefixedDayOfWeek": "4 - jeudi",
        "prefixedMonth": "09 - septembre",
        "timestamp": "1694703737000",
        "month": "septembre",
        "hour": "17",
        "dayOfWeek": "jeudi"
    }
}

{
    "_index": "f10_rec_formsdatasource",
    "_type": "_doc",
    "_id": "FORMSGEN_FormsDataSource_formResponseHistory_681595",
    "_score": 1,
    "_source": {
        "parentName": null,
        "formResponseId": 382,
        "formId": 47,
        "userResponses": {},
        "formName": "TEST form 14092023 - Réservation #1 [AA]",
        "parentId": "26",
        "documentTypeName": "formResponseHistory",
        "completeDuration": 151000,
        "workflowState": "Demande justificatif",
        "actionName": "En attente justificatif",
        "taskDuration": 102000,
        "userResponsesMultiValued": {},
        "worflowAdminCreator": "admin",
        "unitName": null,
        "ame": null,
        "prefixedDayOfWeek": "4 - jeudi",
        "prefixedMonth": "09 - septembre",
        "timestamp": "1694703839000",
        "month": "septembre",
        "hour": "17",
        "dayOfWeek": "jeudi"
    }
}

```json
{
    "_index": "f10_rec_formsdatasource",
    "_type": "_doc",
    "_id": "FORMSGEN_FormsDataSource_formResponseHistory_681596",
    "_score": 1,
    "_source": {
        "parentName": null,
        "formResponseId": 382,
        "formId": 47,
        "userResponses": {},
        "formName": "TEST form 14092023 - Réservation #1 [AA]",
        "parentId": "26",
        "documentTypeName": "formResponseHistory",
        "completeDuration": 170000,
        "workflowState": "Réception agent",
        "actionName": "Justificatif envoyé",
        "taskDuration": 19000,
        "userResponsesMultiValued": {},
        "worflowAdminCreator": "auto",
        "unitName": null,
        "ame": null,
        "prefixedDayOfWeek": "4 - jeudi",
        "prefixedMonth": "09 - septembre",
        "timestamp": "1694703858000",
        "month": "septembre",
        "hour": "17",
        "dayOfWeek": "jeudi"
    }
}