why deepstreamHub? compare us getting started feature roadmap faq
use cases pricing
products
developers
company
blog contact

Deepstream's HTTP API now allows you to do very quick data dumps from your database and into deepstream. In this tutorial, We will be doing batch updates into records stored on deepstream from a Postgres database.

For what we are doing in this tutorial, We won't need to connect to deepstream on the client or server side. All connections will be made through HTTP requests. What you will need to do, is make sure you have a deepstream account, so that you can generate your deepstream HTTP urls after creating a new deepstream app. You will find your HTTP url and AUTH url and in the admin tab of your app.

We will be using needle.js to handle our post requests, and postgres as our database that we are pulling data from and storing into deepstream records.

Connect to our database:

const needle = require('needle');
const pg = require('pg');
const connectionString ='postgres://username:password@localhost:5432/batch-update';
const dbClient = new pg.Client(connectionString);
const options = { //this object will be used in the needle.js post request
    headers: {'Content-Type': 'application/json'}
}

dbClient.connect((err, client, done) => {
    if(err) {
        console.log(err);
    }
    makeQuery(); //once connected, makes the postgres query.
});

For this example, I created a postgres database that contains users' first and last names, that we will want to store in deepstream records. The wonderful thing about deepstream's HTTP API is that you are able to make a single post request with large amounts of data, instead of doing multiple, small post requests. For this example, the dataset is small, but it would work with a much larger data dump as well.

Make a Postgres query:

var requestBody = []; //will contain the correctly formatted body for our deepstream POST request

function makeQuery() {
    // for large data queries, returning a single JSON object from postgres server will result in faster queries
    var query = `SELECT array_to_json(array_agg(row_to_json(t)))
                    from (
                      SELECT id, firstname, lastname FROM users
                    ) t`
    dbClient.query(query, (err, resp) => {
        if(err) {
            console.log(err);
        }
        resp.rows[0].array_to_json.map((data) => {
            requestBody.push({
                topic: 'record', //"topic" and "action" defines the operation deepstream will do. In this case we are creating a new record for each user.
                action: 'write',
                recordName: 'user/' + data.id,
                data: data
            });
        });
        sendAuthRequest() //generates authorization token from deepstream
    })
}

For the http authentication, you can choose whether you want an open auth (no auth necessary), webhook auth, or email auth. For this tutorial, we will use email authentication. You can change your authentication settings in your dashboard of deepstreamhub, under the 'auth' tab.

function sendAuthRequest() {
    console.log('auth');
    var request = {
        type: 'email', // tell deepstream which authentication to perform
        email: 'john@email.com',
        password: 'password'
    }
    needle.post('<YOUR HTTP AUTH URL>', request, options, function(err, resp, body) {
        if(err) {
            console.log(err, 'error');
        }
        sendPostRequest(body.token) // auth token returned in response and passed to the post request
    });
}

function sendPostRequest(authToken) {
    var request = {
        token: authToken,
        body: requestBody
    }
    needle.post('<YOUR HTTP URL>', request, options, function(err, resp, body) {
        if(err){
            console.log(err,'error');
        }
    })
}

And now, we have created a unique record for each user that exists in our database, with one post request. If we were to now go to our dashboard, we would see a list of all our user records.