Shantanu's Blog

There are times when I find a great article or web page but don't have time to read. I use EmailThis service to save text & images from a website to my email inbox. The concept is very simple. Drag and drop a bookmarklet to the bookmark toolbar and click on it to send the current web-page to your inbox!

https://www.emailthis.me/

But I did not like the premium ads and partial content that the site sends. So I built my own serverless API to get exactly the same functionality using mailgun and Amazon Web Services.

https://www.mailgun.com/

Once you register with mailgun, you will get a URL and API-Key that you need to copy-paste to notepad. You will need to provide this information when you launch the cloudformation template by clicking on this link.


https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=emailThis&templateURL=https://datameetgeobk.s3.amazonaws.com/cftemplates/furl.yaml.txt

Here is 7 steps process to load data from any csv file into Amazon DynamoDB.

1) Create the pandas dataframe from the source data
2) Clean-up the data, change column types to strings to be on safer side :)
3) Convert dataframe to list of dictionaries (JSON) that can be consumed by any no-sql database
4) Connect to DynamoDB using boto
5) Connect to the DynamoDB table
6) Load the JSON object created in the step 3 using put_item method
7) Test

# Create the pandas dataframe from the source data

import pandas as pd
import boto3

df=pd.read_excel('http://www.tvmmumbai.in/Alumini%20Std.X-2013-2014.xls')

df.columns=["srno", "seat_no", "surname", "name", "father_name", "mother_name", "english", "marathi", "hindi", "sanskrit", "maths", "science","ss","best_of_5", "percent_best_of_5" , "total_out_of_6", "percent_of_600"]

# Clean-up the data, change column types to strings to be on safer side :)

df=df.replace({'-': '0'}, regex=True)
df=df.fillna(0)

for i in df.columns:
    df[i] = df[i].astype(str)

# Convert dataframe to list of dictionaries (JSON) that can be consumed by any no-sql database

myl=df.T.to_dict().values()

# Connect to DynamoDB using boto

MY_ACCESS_KEY_ID = 'XXX'
MY_SECRET_ACCESS_KEY = 'XXX'

resource = boto3.resource('dynamodb', aws_access_key_id=MY_ACCESS_KEY_ID, aws_secret_access_key=MY_SECRET_ACCESS_KEY, region_name='us-east-1')

# Connect to the DynamoDB table

table = resource.Table('marks1')

# Load the JSON object created in the step 3 using put_item method

for student in myl:
    table.put_item(Item=student)

# Test
response = table.get_item(Key={'seat_no': 'A 314216'})
response

1) Create a table in dynamoDB with the name "ssc" and use field "seat_no" as a string primary key.
2) Add the records for the subjects by creating fields "English", "Maths" and adding marks as string "54", "32". The primary key can be something like "B54MH".

3) Create python function in Lambda as shown below:

import boto3
import json

client = boto3.resource('dynamodb')
table = client.Table('ssc')

def lambda_handler(event, context):
    item = {'seat_no': (event['seatno'])}
    r=table.get_item(Key=item)  
    return json.dumps(r)

4) Create an API
Link the above lambda function name for e.g. "ssc" to the API gateway.
It is important to correctly specify the mapping for API get method execution request:

{"seatno": "$input.params('seatno')"}
_____

Once deployed, the URL will look something like this...

https://9xd7zbdqjg.execute-api.us-east-1.amazonaws.com/S1?seatno=B54MH

Note we are using secure server https to connect. This makes it possible to consume the API results directly into an application like slack.

It is also possible to include cutom authorizer using the steps outlined here...

http://docs.aws.amazon.com/apigateway/latest/developerguide/use-custom-authorizer.html

The user will supply the telephone number like 9919838466 and we need to tell the operator and area. For e.g. airtel - delhi.

I have a python dictionary that has this key-value pairs.

mydic={'98198': 'vodafone-mumbai', '98199': 'airtel-delhi'}

This is 2 lines function that will extract the 5 digits and check it against the dictionary.

def findarea(mnumber, mydic):
    code=str(mnumber)[-10:][:5]
    return mydic[code]

I will test that the function is working as expected...

findarea('9819938466', mydic)
'airtel-delhi'

In order to use Amazon Lambda, I need to slightly modify the function. All the user inputs are passed as events dictionary. Therefore event['mnumber'] will capture the mnumber variable from the URL.

def lambda_handler(event, context):
    mnumber=event['text']
    mydic={'98198': 'vodafone-mumbai', '98199': 'airtel-delhi'}
    code=str(mnumber)[-10:][:5]
    final=mydic[code]
 
    yourdic = {
    "response_type": "in_channel",
    "text": final
}
    return yourdic

Name the Lambda function something like "slacktoapi".
_____

Now using api gateway create a new api called "areacode_api" that links to the function created above.

a) Method Execution - Get Method Request - URL Query String Parameters - add a variable call "text"
b) Method Execution - Get Integration Request - Mapping Templates - Content-Type - application/json - Mapping template

{"text": "$input.params('text')"}

Once you deploy this api, we get a URL and we can start using it like this...

https://91ovy8ax61.execute-api.us-east-1.amazonaws.com/s1?text=919819838463
_____

Here are 5 steps to add any Amazon API to your slack channel.

A) Click on "Build" button on App Directory page of slack.
B) choose "Make a Custom Integration" button
C) select Slash Commands
D) Create a new command called /dest2 and add the URL mentioned above
E) Choose GET Method and save the command.