Free trial

Export user session data

This topic describes the necessary configuration and other requirements needed to export user session data via HTTP.

DESK can be configured to continually send user session data to a user-provided HTTP Webhook endpoint. Once enabled, user session export automatically sends JSON data for all monitored user sessions in your environment to the configured HTTP endpoint. The HTTP server that provides the Webhook must listen for HTTP PUT or HTTP POST requests on a URL specified as part of the configuration.

The data is sent in bulk to improve performance, with a flush every few seconds to keep the data rate near real-time. The data format is one JSON document per line, so you must split the data by line to get valid JSON documents.

Session data that is streamed includes:

  • All user sessions
  • User actions
  • High-level timing and error data

Export of user session data provides a way to leverage near real-time data within external data sources for big data analysis. Streaming session data to a third party system preserves your ability to continue using your current analytics tools to combine and analyze streamed data, including data mined from other sources. You can always push the combined data back to DESK for detailed charting and other real-user data analysis.

Configuration

To configure session export

  1. Go to Settings > Integration > User session export, and ensure that the Enable user session export setting is enabled.
    User session export is enabled by default.
  2. Type the HTTP Webhook Endpoint URL to be used for sending user session data.
  3. Optionally, enable the Use POST method (instead of PUT) setting if you're using POST in the command to send data to the HTTP endpoint.
  4. Under Basic authentication settings, type the username and the password and click Change credentials.

Send data directly to Elasticsearch

You can also send data in bulk to Elasticsearch. To do this, enable the Send data directly to ElasticSearch setting and specify the Name of the index where the data is to be sent and the type of documents in the ElasticSearch index. These values are preset, but you can change them as needed.

The specified Elasticsearch index to which data is sent must be reachable from DESK. For SaaS, this means the server must be publicly available. You can use an SSL certificate and basic authentication to secure access.

Note: The export-URL needs to end with _bulk. The default port for Elasticsearch is 9200. For example: http://<host>:9200/_bulk.

HTTP Endpoint setup

If not using the Elasticsearch destination, the data is sent to an HTTP Endpoint via a HTTP PUT call with a bulk JSON payload.

  • Operation: PUT (or POST if configured).
  • Path: <as configured>.
  • Mime-type: application/json, however sent as multiple JSON documents in bulk line-by-line.
  • Response Body: The handler should respond with HTTP 200 "OK".
  • Description: The handler receives up to 1000 JSON documents, usually less as a periodic flush of all available sessions is performed at least once every minute.

DESK doesn't read the response entirely, only the HTTP status return code.

Please note that the data should be processed quickly. Consider using some asynchronous handling to perform more processing. DESK will time out requests quickly and may back off sending more data for some time to avoid overloading the number of open connections.

Sample code when using the Jersey REST framework

The following sample code uses the open source Jersey REST framework to provide the necessary HTTP endpoint for receiving the data.

@Path("/export/")
public class ExportREST {
    ...

    @PUT
    @Produces(MediaType.APPLICATION_JSON)
    @Path("events")
    public JResponse<String> jsonEvents(final String data) {
        ...
        // split the bulk into single documents
        final String[] lines = StringUtils.split(data, '\n');
        for(String line : lines) {
            ...
            // handle the JSON-data
        }

        return JResponse.ok("")
            .header(HttpHeaders.SERVER, "HTTP Endpoint for DESK Session data export")
            .build();
    }
}

Manual verification

You can verify the data stream by executing a curl command similar to the following:

curl -v -H "Content-Type: application/json" -X PUT -d '{"visitorId":"14804637803609BCTKP776NMJBOIF3R8OD6R0E4NQALJO","visitId":"16229530","startTime":1480463779085,"endTime":1480463784889,"visitType":"SYNTHETIC"}' http://localhost:3000/export/events

Possible additional flags for the command include the following:

  • insecure: Disables the SSL check.
  • http1.1: If you receive the message REFUSED_STREAM when executing the curl command, it may indicate that curl tries to use HTTP 2.0, but the receiver side cannot handle it.

The following shows a sample response returned when using the curl command above with the specified flags:

*   Trying 192.168.1.4...
* Connected to 192.168.1.4 (192.168.1.4) port 443 (#0)
> PUT /export/events HTTP/1.1
> Host: 192.168.1.4:443
> User-Agent: curl/7.47.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 156
>
* upload completely sent off: 156 out of 156 bytes
< HTTP/1.1 200 OK
< Server: Sample REST Server by [name]
< Date: Tue, 24 April 2018 07:02:50 GMT
< Transfer-encoding: chunked
< Content-type: application/json
< Vary: Accept-Encoding
<
* Connection #0 to host 192.168.1.4 left intact

Sample dataset

You can request sample data based on your own user session data. Click the Download sample snapshot from last week link at the bottom of the User session export settings page to get up to 50 user sessions from the previous week. The download provides a JSON document for each user session, returned in bulk, one JSON document per line so you can see how the data look.

The following sample shows three separate sessions sent in bulk as three lines.

{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}
{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}
{"tenantId":"xxx","userSessionId":"1394_1008","startTime":1511441593539,"endTime":1511441716896,"duration":123357,"internalUserId":"1394","userType":"REAL_USER","applicationType":"MOBILE_APPLICATION","bounce":false,"newUser":false,"userActionCount":1,"totalErrorCount":1,"ip":"2001:1800:ffff:eac2:63f5:568e:b3c6:6c54","geolocationContinentName":"North America","geolocationCountryName":"United States","geolocationRegionName":"Florida","geolocationCityName":"Delray Beach","osFamilyName":"Windows","osVersionName":"Windows 10.0 Mobile","manufacturer":"Nokia","device":"L-930","userId":"fearghasbag","screenHeight":1920,"screenWidth":1080,"screenOrientation":"PORTRAIT","displayResolution":"FHD","hasCrash":true,"isp":"SWCP-AS - Southwest Cyberport","stringTags":{ },"numTags":{ },"dateTags":{ },"userActions":[ { "name":"Checkout","type":"Custom","startTime":1511441593539,"endTime":1511441593562,"duration":23,"application":"easyTravel Demo","speedIndex":null,"errorCount":0,"apdexCategory":"UNKNOWN","networkTime":null,"serverTime":null,"frontendTime":null,"documentInteractiveTime":null,"failedImages":null,"failedXhrRequests":null,"httpRequestsWithErrors":null,"thirdPartyResources":null,"thirdPartyBusyTime":0,"cdnResources":null,"cdnBusyTime":0,"firstPartyResources":null,"firstPartyBusyTime":0,"hasCrash":false,"domCompleteTime":null,"domContentLoadedTime":null,"loadEventStart":null,"loadEventEnd":null,"navigationStart":null,"requestStart":null,"responseStart":null,"responseEnd":null,"visuallyCompleteTime":null } ],"events":[ { "type":"UserTag","name":"fearghasbag","startTime":1511441593562,"application":"MOBILE_APPLICATION-752C288D59734C79" } ],"errors":[ { "type":"Crash","name":"ThrowAsync","startTime":1511441716896,"application":"MOBILE_APPLICATION-752C288D59734C79"}]}