Home  >   Blog  >   Sending Sensor Data from Mbed Simulator to Treasure Data

2018-11-02

Sending Sensor Data from Mbed Simulator to Treasure Data

Arm Pelion IoT Platform makes the Mbed ecosystem much more sophisticated and enables us to seamlessly manage IoT connectivity, device and data at a unified place:

Pelion Overview

Treasure Data: A Pelion data management service

My recent experience at Mbed Connect USA tells me how people show a strong interest in advanced big data management and analytics capability of Treasure Data (TD) as a part of the Pelion family:

https://twitter.com/takuti/status/1052270115956305920

On that point, there is a great demo that shows how to send device health data from your Mbed devices to the Treasure Data platform via WiFi network:

Mbed to TD

Once data points emitted from your devices are securely stored to the platform, you will become able to gain tons of deeper insights at scale in combination with many different types of datasets (e.g., user's demographics and behavior) loaded from a variety of data sources.

To those who still don't have a physical Mbed device (including me!), this article introduces a handy version of the Mbed-to-TD data ingestion demo running on top of Mbed simulator.

My demo code is available at: takuti/mbed-os-example-treasure-data

Mbed simulator

Mbed simulator is a Node.js application which allows running our Mbed project written in C++ on a web browser:

Mbed simulator

Emscripten, an LLVM to JavaScript compiler, technically plays an important role in this application. See an introductory article for more information.

Modifying sample C++ code in the editor view, and rebuilding and executing it on a simulated Mbed device are quite easy. Additionally, once you properly define a simconfig.json config file for a time of compilation, the simulator enables working with peripherals such as LCD and temperature & humidity sensor.

Periodically posting data point to TD

Following C++ code is a basic template that periodically sends JSON-formatted data point to TD:

#include "mbed.h"
#include "EthernetInterface.h"
#include "http_request.h"

#define URL_SIZE 200
#define BODY_SIZE 100

#define TD_SEND_INTERVAL 10

const char *td_database = "TARGET_DATABASE_NAME";
const char *td_table = "TARGET_TABLE_NAME";
const char *td_apikey = "YOUR_API_KEY";

int main(void) {
  // Connect with Ethernet
  EthernetInterface network;
  if (network.connect() != 0) {
    printf("Cannot connect to the network\n");
    return 1;
  }

  // Assemble URL: https://support.treasuredata.com/hc/en-us/articles/360000675487-Postback-API
  char url[URL_SIZE];
  sprintf(url, "http://in.treasuredata.com/postback/v3/event/%s/%s", td_database, td_table);

  char body[BODY_SIZE];

  // Get data, send to Treasure Data every TD_SEND_INTERVAL seconds
  while (1) {
    float value1 = // ...
    float value2 = // ...

    // Construct JSON string to send
    sprintf(body, "{\"column1\":%f,\"column2\":%f}", value1, value2);

    // Send data to Treasure data
    {
      HttpRequest *req = new HttpRequest(&network, HTTP_POST, url);
      req->set_header("Content-Type", "application/json");
      req->set_header("X-TD-Write-Key", td_apikey);

      HttpResponse *res = req->send(body, strlen(body));
      if (!res) {
        printf("HttpRequest failed (error code %d)\n", req->get_error());
        return 1;
      }

      delete req;
    }

    wait(TD_SEND_INTERVAL);
  }

  network.disconnect();
}

Similarly to the device health demo, this template simply sends a HTTP POST request to TD's postback API as mbed-os-example-http does.

Depending on JSON-formatted content body, a wide variety of device/sensor data can be incrementally ingested to arbitrary Treasure Data account, database and table.

Sending temperature & humidity data to TD

In order to make a demo scenario more motivating, add a dummy temperature and humidity sensor to the simulator:

Mbed TD simulator

while loop in the template will be:

  int cnt = 1;
  while (1) {
    lcd.cls();

    float temp = sht31.readTemperature();
    float humidity = sht31.readHumidity();

    lcd.locate(2, 2);
    lcd.printf("Sent %d records to TD", cnt++);
    lcd.locate(2, 12);
    lcd.printf("Temperature: %.2f C", temp);
    lcd.locate(2, 22);
    lcd.printf("Humidity: %.2f %%", humidity);

    // Construct strings to send
    sprintf(body, "{\"temperature\":%f,\"humidity\":%f}", temp, humidity);

    // Send data to Treasure data
    {
      ...
    }

    wait(TD_SEND_INTERVAL);
  }

It's time to imitate the change of environmental condition by clicking the sensor component:

Image from Gyazo

Data point is accordingly sent to your TD table every 10 seconds as LCD displays, and it's soon to be available on the data management service as:

Records on TD

Super simple and easy, isn't it?

Again, the full demo implementation and installation guide are available at a GitHub repository, and you can immediately try this sample on your browser.

What's next?

Sending sensor data to the platform is not our goal for sure; there are many possibilities we can do from here.

Importantly, Treasure Data provides Presto and Hive query interface for users to easily access to their massive data at scale, as well as a bunch of third-party data integrations. Hence, in case of the temperature and humidity data, the platform could be used to:

  • Compute average value per month, and see how local environmental condition changes over time in the monthly basis.
  • Build temperature/humidity predictor by using TD's ML capability, and predict future wether condition near the sensor.
  • Send an alert to people who are living within a mile radius from the sensor, when recent data points show exceptionally high/low value.

Meanwhile, in reality, we can do more at device-side such as:

  • Allocating more sensors across a city, and attaching their identifier and/or geolocation to emitted data.
  • Deploying pre-built ML model to the devices, and make edge-side prediction (e.g., anomaly detection) with uTensor.

In any scenarios, I personally believe that integration between Treasure Data and Mbed OS would become one of the most important and exciting challenges for all Mbed developers in near future.

  Share


  Categories

Programming

  See also

2022-02-20
Validate, Validate, and Validate Data. But, in terms of what?
2019-11-10
Rethinking the Role of Data Leaders @ Data Leaders Summit Europe 2019
2018-10-26
Apache Hivemall at #ODSCEurope, #RecSys2018, and #MbedConnect

  More

Last updated: 2022-05-23

  Author: Takuya Kitazawa

Takuya Kitazawa is a freelance software developer based in British Columbia, Canada. As a technologist specializing in AI and data-driven solutions, he has worked globally at Big Tech and start-up companies for a decade. At the intersection of tech and society, he is passionate about promoting the ethical use of information technologies through his mentoring, business consultation, and public engagement activities. See CV for more information, or contact at [email protected].

  Now   Gift a cup of coffee

  Disclaimer

  • Opinions are my own and do not represent the views of organizations I am/was belonging to.
  • I am doing my best to ensure the accuracy and fair use of the information. However, there might be some errors, outdated information, or biased subjective statements because the main purpose of this blog is to jot down my personal thoughts as soon as possible before conducting an extensive investigation. Visitors understand the limitations and rely on any information at their own risk.
  • That said, if there is any issue with the content, please contact me so I can take the necessary action.