Step by step : How to setup mongodb with lambda

Step by step : How to setup mongodb with lambda

With the rise of serverless architecture Lambda along with API gateway is becoming automatic choice over EC2 or ECS to developers for computing layer of their application. For database layer, DynamoDB is a good option due to a managed service but I feel it is not fit for all applications due to its limited data query features. AWS RDS service is another good option but it is a relational database service and modern applications are preferring NoSQL database or a mix of RDBMS and NoSQL for the database layer. MongoDB is a good option to use with Lambda. However, it is a little bit tricky to make MongoDB on EC2 accessible to Lambda functions. Following is a step by step guide to setup MongoDB with Lambda function.

Step 1: Create VPC

To enable access to MongoDB EC2 instance from Lambda function we need to configure lambda with VPC.

Step 1.1: Go to VPC Console

Go to VPC service

Step 1.2: Create new VPC.

Give a suitable name like “lambda-to-mongo” and specify IPv4 CIRD range. For development purpose range specified in the image is good enough but for production, you need to consider concurrent usages, use large CIDR block.

Create VPC

Step 1.3: Create a public subnet

This subnet will contain our MongoDB instance hence name it carefully so that you can identify while launching the instance. CIDR range for this subnet can be small as it will contain only MongoDB instances.

Create public subnet

Step 1.4: Create a private subnet

This subnet will be linked to lambda functions hence name it carefully so that you can identify while creating lambda functions. CIDR range for this subnet needs to be defined based on our usage.

Create private subnet

Step 1.5: Create internet gateway

Internet gateway is required to allow internet access to and from VPC.

Create internet gateway

Step 1.5: Attach internet gateway to VPC

Attache internet gateway

Step 1.6: Private route table

Private route table

Main route table create with VPC is by default private as it only allows network traffic within the VPC only. We just rename it to identify clearly.

Step 1.7: Create public route table

Create a new route table and make it public by attaching internet gateway route.

Create public route table
Attach internet gateway to public route table step 1
Attach internet gateway to public route table step 2
Attach internet gateway to public route table step 3

Step 1.8: Associate public route table to public subnet

To allow internet access to and from public subnet associate public route table with the public subnet.

Associate public route table to public subnet step 1
Associate public route table to public subnet step 2
Associate route table to public subnet step 3

Step 1.9: Create security group for lambda

This step is necessary to group all lambda containers which can have any private IP from our private subnet IP range. By grouping, we can allow inbound traffic from lambda containers to our MongoDB instance on MongoDB port.

Create security group for lambda

Step 1.10: Create security group for mongo instance

This group allows access to mongo instance from ssh and MongoDB port.

Create security group for mongo instance

Step 1.10: Modify inbound rules of mongo security group

We will add 3 inbound rules to the group.
1. Allow MongoDB access from lambda security group on 27017 port
2. Allow ssh access from your machine on 22 port. for this, you need to get your machine public IP address (recommended) or can allow for all 0.0.0.0/0
3. Allow MongoDB access from your machine on 27017 port. for this, you need to get your machine public IP address. This rule is optional.

What is my IP
Modify mongo security group inbound rules step 1
Modify mongo security group inbound rules step 3

Step 2: Launch EC2 instance

Launch an instance for MongoDB

Step 2.1: Launch instance

Launch EC2 instance

Step 2.2: Select AMI

Select AMI of your choice. I prefer Amazon Linux AMI.

EC2 instance launch select AMI

Step 2.3: Select instance type

I am selecting free tire T2.micro instance. You can select any instance type depending on your use case. Storage optimizes instance types are more suitable for MongoDB workload.

EC2 instance launch select instance type

Step 2.4: Configure instance details

Do this step very carefully. Select the VPC created in step 1.2, Public subnet created in step 1.3 and enable auto assign public IP.

Configure EC2 instance

Step 2.5: Add storage

Change storage if you want to use this instance for production for testing purpose default storage will be sufficient.

Add / Modify storage during EC2 instance launch

Step 2.6: Add Tags

And a Name tag to your instance to identify it.

Add / Modify Tags during EC2 instance launch

Step 2.7: Configure security group

Assign the security group created for mongo instance in step 1.10

EC2 instance launch configure security group

Step 2.8: Review instance details

Review instance details like VPC, Subnet, security group etc. once before clicking Launch. If anything is not appropriate please go back and correct it.

Review instance details before instance launch

Step 2.9: Download key pair

Download key pair to ssh into the instance. This key is required for shell access of the instance, if lost we need to relaunch the instance.

Download Key pair during EC2 instance launch

Step 2.10: Launch status

Click view instance

EC2 instance launch status

Step 2.11: Instance details

After 1-2 min instance will be ready for use. Note down the Public and Private IP address as marked in the screenshot.

instance details

Step 3: Install MongoDB

Step 3.1: Use the following commands to ssh into the instance and install MongoDB

// change directory where you downloaded the instance ssh key 
cd /path-to-instance-key

// replace PUBLICIP with the IP from step 2.8
ssh -i "lambda-to-mongo.pem" ec2-user@PUBLICIP

// login as sudo user
sudo su

// update packages
yum update -y

// add mongodb repository
printf '[mongodb-org-3.4]\nname=MongoDB 3.4 Repository\nbaseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.4/x86_64/\ngpgcheck=0\nenabled=1' >> /etc/yum.repos.d/mongodb-org-3.4.repo

// install mongodb
yum install -y mongodb-org

// open mongod conf file
vi /etc/mongod.conf

// comment out the line containing the text bindIp: 127.0.0.1
# bindIp: 127.0.0.1

// start mongodb service
service mongod start

Step 3.2: Create a sample data

// open mongo client
mongo

// create database
use lambdadb

// create collection
db.createCollection("mycollection");

// insert record
db.mycollection.insert({"name":"Deepak Goyal"});

// check data
db.mycollection.find().pretty();

Step 4: Create Lambda

I am creating a lambda function using nodejs you can use any other language supported by lambda. All steps remain same except 4.1 and 4.2 which will be replaced by your code structure.

Step 4.1: Create code structure on local system

Execute the following commands to create file structure. I am assuming node is already setup on your system

// create a directory
mkdir mylambda

// initialize node
npm init

// create index.js
touch index.js

// install mongo pacage
npm install mongodb --save

Step 4.2: Lambda code

Copy the folloing code into index.js. Replace PRIVATEIP with the private ip captured in step 2.8

var MongoClient = require('mongodb').MongoClient,
assert = require('assert');

// Connection URL
var url = 'mongodb://PRIVATEIP:27017/';
var dbname = "lambdadb";
var collectionname = "mycollection";

exports.handler = (event, context, callback) => {

    MongoClient.connect(url, function (err, db) {
        assert.equal(null, err);
        console.log("Connected successfully to server");

        var dbo = db.db(dbname);
        dbo.collection(collectionname, function (err, collection) {

            collection.find().toArray(function (err, items) {
                if (err) throw err;
                console.log(items);
            });

            db.close();

        });
    });

    // TODO implement
    callback(null, 'Connecting to mongo');
};

Step 4.3: Create zip file for upload

while inside the code directory run following commond

zip -r mylambda.zip ./

Step 4.4: Create lambda function from AWS Console

Create lambda

Step 4.4: Provide basic details

Provide a name, runtime and select Create a custom role for Role. It will open a new tab to create a custom role.

Lambda basic details

Step 4.5: Create custom role

Edit policy document and past the follwing policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:DescribeNetworkInterfaces"
      ],
      "Resource": "*"
    }
  ]
}
Lambda basic details
create lambda custom role change policy

Step 4.6: Select created role and save

create lambda custom role select

Step 4.7: Upload code

create lambda upload code

Step 4.8: Configure VPC

create lambda configure vpc

Step 4.9: Test lambda

You can see the successful connection and data in the screenshot.

lambda test configure
lambda test result
By |2018-03-19T12:14:02+00:00February 7th, 2018|AWS, Step By Step|0 Comments

About the Author:

Leave A Comment