[node.js] Why is this HTTP request not working on AWS Lambda?

I'm getting started with AWS Lambda and I'm trying to request an external service from my handler function. According to this answer, HTTP requests should work just fine, and I haven't found any documentation that says otherwise. (In fact, people have posted code that use the Twilio API to send SMS.)

My handler code is:

var http = require('http');

exports.handler = function(event, context) {
  console.log('start request to ' + event.url)
  http.get(event.url, function(res) {
    console.log("Got response: " + res.statusCode);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });

  console.log('end request to ' + event.url)
  context.done(null);
}

and I see the following 4 lines in my CloudWatch logs:

2015-02-11 07:38:06 UTC START RequestId: eb19c89d-b1c0-11e4-bceb-d310b88d37e2
2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 start request to http://www.google.com
2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 end request to http://www.google.com
2015-02-11 07:38:06 UTC END RequestId: eb19c89d-b1c0-11e4-bceb-d310b88d37e2

I'd expect another line in there:

2015-02-11 07:38:06 UTC eb19c89d-b1c0-11e4-bceb-d310b88d37e2 Got response: 302

but that's missing. If I'm using the essential part without the handler wrapper in node on my local machine, the code works as expected.

The inputfile.txt I'm using is for the invoke-async call is this:

{
   "url":"http://www.google.com"
}

It seems like the part of the handler code that does the request is skipped entirely. I started out with the request lib and fell back to using plain http to create a minimal example. I've also tried to request a URL of a service I control to check the logs and there's no requests coming in.

I'm totally stumped. Is there any reason Node and/or AWS Lambda would not execute the HTTP request?

Simple Working Example of Http request using node.

const http = require('https')
exports.handler = async (event) => {
    return httprequest().then((data) => {
        const response = {
            statusCode: 200,
            body: JSON.stringify(data),
        };
    return response;
    });
};
function httprequest() {
     return new Promise((resolve, reject) => {
        const options = {
            host: 'jsonplaceholder.typicode.com',
            path: '/todos',
            port: 443,
            method: 'GET'
        };
        const req = http.request(options, (res) => {
          if (res.statusCode < 200 || res.statusCode >= 300) {
                return reject(new Error('statusCode=' + res.statusCode));
            }
            var body = [];
            res.on('data', function(chunk) {
                body.push(chunk);
            });
            res.on('end', function() {
                try {
                    body = JSON.parse(Buffer.concat(body).toString());
                } catch(e) {
                    reject(e);
                }
                resolve(body);
            });
        });
        req.on('error', (e) => {
          reject(e.message);
        });
        // send the request
       req.end();
    });
}

Yeah, awendt answer is perfect. I'll just show my working code... I had the context.succeed('Blah'); line right after the reqPost.end(); line. Moving it to where I show below solved everything.

console.log('GW1');

var https = require('https');

exports.handler = function(event, context) {

    var body='';
    var jsonObject = JSON.stringify(event);

    // the post options
    var optionspost = {
        host: 'the_host',
        path: '/the_path',
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        }
    };

    var reqPost = https.request(optionspost, function(res) {
        console.log("statusCode: ", res.statusCode);
        res.on('data', function (chunk) {
            body += chunk;
        });
        context.succeed('Blah');
    });

    reqPost.write(jsonObject);
    reqPost.end();
};

Yes, there's in fact many reasons why you can access AWS Lambda like and HTTP Endpoint.

The architecture of AWS Lambda

It's a microservice. Running inside EC2 with Amazon Linux AMI (Version 3.14.26–24.46.amzn1.x86_64) and runs with Node.js. The memory can be beetwen 128mb and 1gb. When the data source triggers the event, the details are passed to a Lambda function as parameter's.

What happen?

AWS Lambda run's inside a container, and the code is directly uploaded to this container with packages or modules. For example, we NEVER can do SSH for the linux machine running your lambda function. The only things that we can monitor are the logs, with CloudWatchLogs and the exception that came from the runtime.

AWS take care of launch and terminate the containers for us, and just run the code. So, even that you use require('http'), it's not going to work, because the place where this code runs, wasn't made for this.

I've found lots of posts across the web on the various ways to do the request, but none that actually show how to process the response synchronously on AWS Lambda.

Here's a Node 6.10.3 lambda function that uses an https request, collects and returns the full body of the response, and passes control to an unlisted function processBody with the results. I believe http and https are interchangable in this code.

I'm using the async utility module, which is easier to understand for newbies. You'll need to push that to your AWS Stack to use it (I recommend the serverless framework).

Note that the data comes back in chunks, which are gathered in a global variable, and finally the callback is called when the data has ended.

'use strict';

const async = require('async');
const https = require('https');

module.exports.handler = function (event, context, callback) {

    let body = "";
    let countChunks = 0;

    async.waterfall([
        requestDataFromFeed,
        // processBody,
    ], (err, result) => {
        if (err) {
            console.log(err);
            callback(err);
        }
        else {
            const message = "Success";
            console.log(result.body);
            callback(null, message);
        }
    });

    function requestDataFromFeed(callback) {
        const url = 'https://put-your-feed-here.com';
        console.log(`Sending GET request to ${url}`);
        https.get(url, (response) => {
            console.log('statusCode:', response.statusCode);
            response.on('data', (chunk) => {
                countChunks++;
                body += chunk;
            });
            response.on('end', () => {
                const result = {
                    countChunks: countChunks,
                    body: body
                };
                callback(null, result);
            });
        }).on('error', (err) => {
            console.log(err);
            callback(err);
        });
    }
};

I had the very same problem and then I realized that programming in NodeJS is actually different than Python or Java as its based on JavaScript. I'll try to use simple concepts as there may be a few new folks that would be interested or may come to this question.

Let's look at the following code :

var http = require('http'); // (1)
exports.handler = function(event, context) {
  console.log('start request to ' + event.url)
  http.get(event.url,  // (2)
  function(res) {  //(3)
    console.log("Got response: " + res.statusCode);
    context.succeed();
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
    context.done(null, 'FAILURE');
  });

  console.log('end request to ' + event.url); //(4)
}

Whenever you make a call to a method in http package (1) , it is created as event and this event gets it separate event. The 'get' function (2) is actually the starting point of this separate event.

Now, the function at (3) will be executing in a separate event, and your code will continue it executing path and will straight jump to (4) and finish it off, because there is nothing more to do.

But the event fired at (2) is still executing somewhere and it will take its own sweet time to finish. Pretty bizarre, right ?. Well, No it is not. This is how NodeJS works and its pretty important you wrap your head around this concept. This is the place where JavaScript Promises come to help.

You can read more about JavaScript Promises here. In a nutshell, you would need a JavaScript Promise to keep the execution of code inline and will not spawn new / extra threads.

Most of the common NodeJS packages have a Promised version of their API available, but there are other approaches like BlueBirdJS that address the similar problem.

The code that you had written above can be loosely re-written as follows.

'use strict';
console.log('Loading function');
var rp = require('request-promise');
exports.handler = (event, context, callback) => {    

    var options = {
    uri: 'https://httpbin.org/ip',
    method: 'POST',
    body: {

    },
    json: true 
};


    rp(options).then(function (parsedBody) {
            console.log(parsedBody);
        })
        .catch(function (err) {
            // POST failed... 
            console.log(err);
        });

    context.done(null);
};

Please note that the above code will not work directly if you will import it in AWS Lambda. For Lambda, you will need to package the modules with the code base too.

Add above code in API gateway under GET-Integration Request> mapping section.

I faced this issue on Node 10.X version. below is my working code.

const https = require('https');

exports.handler = (event,context,callback) => {
    let body='';
    let jsonObject = JSON.stringify(event);

    // the post options
    var optionspost = {
      host: 'example.com', 
      path: '/api/mypath',
      method: 'POST',
      headers: {
      'Content-Type': 'application/json',
      'Authorization': 'blah blah',
    }
    };

    let reqPost =  https.request(optionspost, function(res) {
        console.log("statusCode: ", res.statusCode);
        res.on('data', function (chunk) {
            body += chunk;
        });
        res.on('end', function () {
           console.log("Result", body.toString());
           context.succeed("Sucess")
        });
        res.on('error', function () {
          console.log("Result Error", body.toString());
          context.done(null, 'FAILURE');
        });
    });
    reqPost.write(jsonObject);
    reqPost.end();
};