Prerequisites
- Access to a HANA system with the Smart Data Streaming option installed and configured.
- HANA user ID with permissions to create, compile, deploy and run a Smart Data Streaming project. This can be the HANA SYSTEM user which is granted full permissions for streaming as part of the install process.
- Ability to connect to the Linux host running the Smart Data Streaming as the <sid>adm user in order to start the Web Service Provider (if not already started).
- This tutorial assumes that you have already configured and started the Web Service Provider as per the instructions in the Configuring the Web Service Provider (WSP) for use with SDS Projects tutorial.
Introduction
Using web services to post event data to SAP HANA Smart Data Streaming (“SDS”) projects is a popular choice. The Web Service Provider (“WSP”) component of Smart Data Streaming supports the posting of events through both Websocket and REST connections. When multiple events are consistently being posted from a single source, then using Websocket connection is the recommended method. An example where Websockets may be preferred is the posting of a stream of temperature readings from a temperature sensor that continuously generates new temperature readings every second.
The benefit of a Websocket connection in this case relative to using a REST post is that a Websocket connection only authenticates once when opening the connection, whereas each individual REST post is authenticated by itself. Since a WebSocket connection is opened initially and is kept open for all subsequent message posts, using WebSockets can be much quicker and more efficient than REST posts when posting a continuous stream of messages.
The purpose of this tutorial is to provide an example of constructing and sending event data to a stream in an SDS project using a Websocket connection. This tutorial is intended to supplement the existing SDS WebSocket Provider documentation (Streaming WebSocket Provider) by pairing step by step instructions inspired by the documentation with a working example.
Specifically, the following steps will be addressed:
- Configuring a streaming project to allow the use of WebSockets
- Using an HTML page to initialize a WebSocket connection to the streaming project
- Creating JSON posts to be sent via the WebSocket connection to the streaming project
- Sending the JSON posts to the streaming project
Sample Streaming Project
Provided below is a sample streaming project consisting of a single input stream. When the project is configured as defined in the “Configuring the Web Service Provider to Use WebSockets” section below, the MACHINEDATA input stream can be used to receive JSON posts via a WebSocket.
CREATEINPUTSTREAM MACHINEDATA SCHEMA (
MACHINEID string ,
EVENT_TIME msdate ,
EVENT_NAME string ,
EVENT_DESCRIPTION string ,
EVENT_VALUE string ) ;
Note: This input stream is taken from the Freezer Monitoring tutorial. If you have already completed the Freezer Monitoring tutorial, you can use the existing Freezer Monitoring project for this tutorial.
Configuring the Streaming Project to Enable Web Services
Start by navigating to the “SAP HANA Streaming Development” perspective and opening the configuration (.ccr) file in your streaming project. Under the “Advanced” tab in the Project Configurations window that appears, click on “Project Deployment (Non-HA)”. Enable Web Services by checking the corresponding box in the Project Deployment Details and setting the value in the drop down menu to “true”.
Note:If the check box to the left of the “Web Service Enabled” option is not checked, then the option value of true or false is ignored.
Publishing to a Streaming Project from an HTML Page
Using Javascript, it is possible to send JSON posts via a WebSocket to a running streaming project from a web page. To create an HTML/Javascript page containing WebSocket functionality, the following steps are required:
- Create and open the WebSocket()connection. This will be a persistent connection between the Websocket_Sample.html page and the target streaming project.
- Construct a JSON message containing data for each expected column of the target input stream. For simplicity, the order must match the column order of the target input stream.
- Send the JSON message to the streaming project via the opened WebSocket
Create and Open the WebSocket()Connection
To create the WebSocket()object, we need to declare a variable and instantiate it as a new WebSocket() instance. For example, the javascript to create a new WebSocket()object can look like this:
var ws;//for WS
ws = new WebSocket( "ws://my.streaming.host.com:9092/publish/default/my_project_name?basic-authorization=c2FwOnNhcA==” ); //create new websocket
Note: In the sample HTML page at the end of this tutorial, the ws variable is declared in the //global variable declarations section and instantiated in the setPostMethod()function.
The URL to be used for the connection needs to use the following format:
ws[s]://host:port/publish/workspace/project?<auth>
Where:
- ws[s] is the URI identifier for the websocket protocol. The optional [s] (resulting in wss://…) is for secure connections
- The host is the fully qualified domain name (“FQDN”) or TCP/IP address of the SDS node of the HANA system that is running the WSP.
- The port is the Websocket port specified in the wsp.xml file (Default is 9092, for instructions on how to specify a different port check the “Configuring the Web Service Provider (WSP) for use with SDS Projects" tutorial)
- The keyword “publish” is used because we are publishing or pushing data to the input stream in the SDS project. The other valid value is “subscribe” which would be used if we were receiving data from an output stream in the SDS project.
- workspace is the workspace that your’re streaming project is running within on the SDS server. This is most likely to be “default”.
- project is the name of the streaming project you will be publishing to.
- <auth> provides the authorization credentials. As of SPS10, the only supported authorization for websocket connections is “basic-authorization=” followed by a base 64 encoding of “<username>:<password>” that you are using to connect.
- <username> is a HANA user name with permissions to write to the desired input stream. This could be the HANA SYSTEM user, or you can grant a separate HANA user the required permissions. (See the APPENDIX – CREATING A HANA USER AND GRANTING SMART DATA STREAMING PERMISSIONS in the Freezer Monitoring Tutorial for instructions)
- <password> is the HANA password for the username being specified
- One online site that can be used to generate a base 64 encoding of the <username>:<password> string is: https://www.base64decode.org/. In the example above, the authorization credentials used are a base 64 encoding of “sap:sap”.
Construct the JSON Message
Now that we have the WebSocket()connection open, we need to build out the JSON message that will be sent using over the WebSocket()connection. The JSON message needs to include several required elements in addition to the contents of the event record that you are sending.
The structure of the JSON contents looks as follows:
{
"version": 1,
"stream": ,
"flags": -,
"rows":
[
{ "ESP_OPS" : "INSERT|UPDATE|DELETE|UPSERT|SAFEDELETE", column : value, column : value, ... },
{ "ESP_OPS" : "INSERT|UPDATE|DELETE|UPSERT|SAFEDELETE", column : value, column : value, ... }, ...
]
}
Where:
- "version" will always be 1 as that is the only supported value
- "stream" is the name of the stream or window that this event record will be published to. For the sample schema used in this tutorial, the stream name is MACHINEDATA
- "flags" is not currently used and is excluded in the sample code used for this tutorial
- “rows” is an array that can contain multiple event records – allowing you to implement batching when submitting event records to the streaming project. Within each even record you specify
- "ESP_OPS" is an operation code that allows you to specify whether the event record is a new insert or whether it should modify an existing record within the target streaming project. This is an optional element and if you omit the “ESP_OPS” element, then records will be treated as inserts by default.
- column : value, … is the list of the expected columns and the value for each column of the target input stream that we will be publishing our JSON messages to. Note that the column values must be provided in the order that the columns were declared when creating the input stream. For the MACHINEDATA sample that we are using that means you need to provide the "<column-name>":"<value>" pairs in the order of MACHINEID, EVENT_TIME, EVENT_NAME, …
To create the JSON object described above we first create an object, called jsonObject, which contains instances of the version, stream and rows. We have not included an instance of flags because it is not currently used. Note: MACHINEDATA is the name of an input stream in the Streaming project being used.
var jsonObject = {};
var contentObject = getContentObject();
jsonObject["version"]=1;
jsonObject["stream"]="MACHINEDATA";
//below commented out because flags value is currently not used. May be used in later versions
//jsonObj["flags"]=1;
jsonObject["rows"]=[contentObject];
Note: In the sample HTML page at the end of this tutorial, the jsonObject is declared in the //global variable declarations section and populated in the gatherUserInputs()function.
You will note that we populate the “rows” array by assigning another object called contentObject to it. This approach allows us to modularly build up the contentObject with the individual event record contents. As you can see, user inputted data is being pulled from an HTML form. Note: the "rows" array can contain any number of rows but in this example we are just pulling a single row from an html form and sending the individual event record.
var contentObject = {};
//below commented out because INSERT is the default value for ESP_OPS
//contentObject["ESP_OPS"]="INSERT";
contentObject["MACHINEID"]=document.getElementById("MachineID").value;
contentObject["EVENT_TIME"]=document.getElementById("EventTime").value;
contentObject["EVENT_NAME"]=document.getElementById("EventName").value;
contentObject["EVENT_DESCRIPTION"]=document.getElementById("EventDescription").value;
contentObject["EVENT_VALUE"]=document.getElementById("TemperatureValue").value;
Note: In the sample HTML page at the end of this tutorial, the contentObject is declared in the //global variable declarations section and populated in the gatherUserInputs() function.
Send the JSON Message
At this point we have completed the construction of the JSON message body that will be posted to the Web Service Provider using the send() method of the WebSocket()object that we instantiated and opened earlier in the tutorial. The final step in sending the message is to convert the jsonObject to a string and call the send() method with the ‘stringified’ jsonObject as a parameter.
This can be done in a single javascript statement such as:
ws.send(JSON.stringify(jsonObject));
Note: In the sample HTML page at the end of this tutorial, the ws.send() call is made in the this.send function which in turn is called from the buildAndSendJSON() function.
*Don’t forget to start the wsp server!
Sample HTML Page
Provided below is a sample html page complete with the Javascript code to send JSON posts via a Websocket. The connection URL will need to be changed as described in step 1 of the section above. Note that the WebSocket is opened the first time a user clicks the “Build & Send JSON Post” button and stays open for the duration of the session. This is the benefit of using a WebSocket over HTTP.
<!DOCTYPEhtml>
<html>
<head>
<title>Machine Event Simulator - SAP HANA Smart Data Streaming</title>
<style>
.indent{
margin-left:2%;
}
</style>
<script>
//global variable declarations
var ws;//for WS
var eventObject = {};
var contentObject = {};
var jsonObject = {};
//function is called on page load (called from body onLoad)
function init(){
setPostMethod();
setProjectServerURL();
document.getElementById("EventTime").value = getCurrentDateTime();//set event time field to current time
setEventValueInput();
}
function setProjectServerURL(){
if (document.getElementById("WSProjectURL").value != "customOption"){
document.getElementById("projectServerURL").innerHTML = document.getElementById("WSProjectURL").value;
}else{//clear SDS Project Server URL innerHTML
document.getElementById("customServer").value="ws://";
}
}
//will be called upon user click of "Set Server" button
function setCustomServerURL(){
document.getElementById("projectServerURL").innerHTML = document.getElementById("customServer").value;
ws = new WebSocket(document.getElementById("projectServerURL").innerHTML);//create new websocket
}
//function gets appropriate project URL based on user selected post method
function getProjectURL(){
return document.getElementById("WSProjectURL").value;
}
//function displays appropriate project server drop down menu based on user selected post method
function setPostMethod(){
customServer();//decide whether to show custom server field
if (document.getElementById("WSProjectURL").value != "customOption"){
ws = new WebSocket(getProjectURL());//create new websocket
}
setProjectServerURL();
}
//function returning a string representation of the current machine date and time
function getCurrentDateTime(){
var currentDateTime;
var currentDate;
var currentTime;
var d = new Date();
var currentMonth = d.getMonth()+1;
currentDate = d.getFullYear()+"-"+currentMonth+"-"+d.getDate();
currentTime = d.getHours()+":"+d.getMinutes()+":"+d.getSeconds()+"."+d.getMilliseconds();
currentDateTime = currentDate+" "+currentTime;
return currentDateTime;
}
function formatDateTime(time){
var newTime = "";
var i = 0;
while (time.charAt(i) != ' '){//get date
newTime += time.charAt(i);
i ++;
}
newTime += 'T';
i ++;
while (i < time.length){//get time
newTime += time.charAt(i);
i ++;
}
newTime += 'Z';
return newTime;
}
function gatherUserInputs(){
//below commented out because INSERT is the default value for ESP_OPS
//contentObject["ESP_OPS"]="INSERT";
contentObject["MACHINEID"]=document.getElementById("MachineID").value;
contentObject["EVENT_TIME"]=formatDateTime(document.getElementById("EventTime").value);
EventTime = getCurrentDateTime();//reset event time field to current time
contentObject["EVENT_NAME"]=document.getElementById("EventName").value;
contentObject["EVENT_DESCRIPTION"]=document.getElementById("EventDescription").value;
if (document.getElementById("EventName").value=="TEMP"){
contentObject["EVENT_VALUE"]=document.getElementById("TemperatureValue").value;
} elseif (document.getElementById("EventName").value=="DOOR"){
contentObject["EVENT_VALUE"]=document.getElementById("DoorStatus").value;
} elseif (document.getElementById("EventName").value=="POWER"){
contentObject["EVENT_VALUE"]=document.getElementById("PowerStatus").value;
}
jsonObject["version"]=1;
jsonObject["stream"]="MACHINEDATA";
//below commented out because flags value is currently not used. May be used in later versions
//jsonObj["flags"]=1;
jsonObject["rows"]=[contentObject];
eventObject["content"]=contentObject;
//console.log(JSON.stringify(jsonObj));//uncomment to log JSON post to console before sending
}
function buildAndSendJSON(){
gatherUserInputs();
this.send();
}
//function tries to send JSON post but if Websocket is not yet open, waits 1 second and then tries again
this.send = function (message, callback) {
this.waitForConnection(function () {
ws.send(JSON.stringify(jsonObject));
if (typeof callback !== 'undefined') {
callback();
}
}, 1000);
};
//utility function to determine whether Websocket is open
this.waitForConnection = function (callback, interval) {
if (ws.readyState === 1) {
callback();
} else {
if (ws.readyState === 3){
ws = new WebSocket(getProjectURL());
}
var that = this;
setTimeout(function () {
that.waitForConnection(callback, interval);
}, interval);
}
};
//function to show input field when SDS Project to post to is "Specify Server"
function customServer(){
if(document.getElementById("WSProjectURL").value == "customOption"){//show custom server field
document.getElementById("customServerOption").style.display = "block";
}else{//hide custom server field
document.getElementById("customServerOption").style.display = "none";
}
}
//function to display appropriate inputs
function setEventValueInput(){
if (document.getElementById("EventName").value=="TEMP"){
document.getElementById("TemperatureValue").style.display = "inline";
document.getElementById("DoorStatus").style.display = "none";
document.getElementById("PowerStatus").style.display = "none";
} elseif (document.getElementById("EventName").value=="DOOR"){
document.getElementById("TemperatureValue").style.display = "none";
document.getElementById("DoorStatus").style.display = "inline";
document.getElementById("PowerStatus").style.display = "none";
} elseif (document.getElementById("EventName").value=="POWER"){
document.getElementById("TemperatureValue").style.display = "none";
document.getElementById("DoorStatus").style.display = "none";
document.getElementById("PowerStatus").style.display = "inline";
}
}
</script>
</head>
<bodyonLoad="init()">
<imgsrc="https://www.sapstore.com/_ui/desktop/theme-sap-sapphire/images/vendors/logo-sap-small.png"height="15%"width="15%"style="display:inline;"/>
<h1style="display:inline;">Machine Event Simulator</h1>
<divid="main"style="margin-left:5%; margin-top:5%;">
<divstyle="float:right; margin-right:15%; width:40%;">
<h3>SDS Project Server URL:</h3>
<divclass="indent">
<pid="projectServerURL"></p>
</div>
</div>
<div>
<h3>Select the SDS Project Server to post to:</h3>
<divid="WSProjectServer">
<divclass="indent">
<selectid="WSProjectURL"onChange="setPostMethod()">
<!--<option value="ws://<host>:9092/publish/default/<project-name>?basic-authorization=<base-64-encoding-of-username:password>">A Server</option>-->
<optionvalue="customOption">Specify Server</option>
</select>
</div>
</div>
<divid="customServerOption"class="indent">
<h4>Specify a server:</h4>
<form>
<inputtype="input"id="customServer"value=""/>
<inputtype="button"value="Set Server"onclick="setCustomServerURL()"/>
</form>
</div>
</div>
<divstyle="float:right; margin-right:15%;" id="responseField">
<!-- Will be filled with SDS Project Server URL by JS during runtime -->
</div>
<divid="singleDiv">
<h3>Single post event parameters:</h3>
<divclass="indent">
<form>
Machine ID: <inputtype="text"name="machineid"value="2DDDBW3TP"id="MachineID"><br>
Event Time: <inputtype="text"name="eventtime"value=""id="EventTime"><br>
Event Name:
<selectid="EventName"onChange="setEventValueInput()">
<optionvalue="DOOR">DOOR</option>
<optionvalue="POWER">POWER</option>
<optionvalue="TEMP">TEMP</option>
</select><br>
Event Description: <inputtype="text"name="eventdescription"value=""id="EventDescription"><br>
Event Value: <inputtype="text"name="eventvalue"size="8"value="50"id="TemperatureValue">
<selectid="DoorStatus">
<optionvalue="Door open">Door Open</option>
<optionvalue="Door close">Door Close</option>
</select>
<selectid="PowerStatus">
<optionvalue="Power on">Power On</option>
<optionvalue="Power off">Power Off</option>
</select><br><br>
</form>
</div>
<buttontype="button"onclick="buildAndSendJSON()">Build & Send JSON Post</button>
</div>
</div>
</body>
</html>