Shantanu's Blog

The following code will initiate a Linux instance of type m3.medium using spot pricing and associate it to IP address 13.228.39.49 Make sure to use your own elastic IP address and key. Do not forget to change the access_key and secret_key parameters.

!wget https://raw.githubusercontent.com/shantanuo/easyboto/master/easyboto.py
 
import easyboto
dev=easyboto.connect('access_key', 'secret_key')
dev.placement='us-east-1a'
dev.myaddress='13.228.39.49'
dev.key='dec15abc'
dev.MAX_SPOT_BID= '2.9'
dev.startEc2Spot('ami-0323c3dd2da7fb37d', 'm3.medium')

This will return the instance id and the ssh command that you can use to connect to your instance. The output will look something like...

job instance id: i-029a926e68118d089
ssh -i dec15a.pem ec2-user@13.228.39.49

You can list all instances along with their details like launch time, image_id  and save the results as pandas dataframe using showEc2 method like this...

df=dev.showEc2()

Now "df" is a pandas dataframe object. You can sort or groupby the instances just like an excel sheet.

You can delete the instance by providing the instance ID that was generated in the first step using deleteEc2 method.

dev.deleteEc2('i-029a926e68118d089')
_____

You can also use cloudformation template for this purpose. Visit the following link and look for "Linux EC2 Instance on SPOT" section.

https://github.com/shantanuo/cloudformation

Click on "Launch Stack" button. It is nothing but GUI for the python code mentioned above. You will simply have to submit a form for the methods like key and IP address.

Here is the code to submit your batch job programmatically.

import boto3
session = boto3.Session(
    aws_access_key_id="xxx",
    aws_secret_access_key="xxxx",
    region_name="us-east-1",
)
client = session.client("batch")
JOB_NAME = "nlptensorboto1"
JOB_QUEUE = "arn:aws:batch:us-east-1:302xxx:job-queue/nlpmljob"
JOB_DEFINITION = "arn:aws:batch:us-east-1:302xxx:job-definition/nlptensor:1"

client.submit_job(
    jobName=JOB_NAME,
    jobQueue=JOB_QUEUE,
    jobDefinition=JOB_DEFINITION,
    containerOverrides={"environment": [{"name": "senderid", "value": "xxx"}]},
)

Here is a lambda function that will check if there is open port in a given security group.
It will send a message to sns topic if 0.0.0.0 is found anywhere in that security group.


def lambda_handler(event, context):
    import boto3, json
    ec2 = boto3.client('ec2' , region_name='us-east-1' )
    security_group = ec2.describe_security_groups(GroupIds=['sg-12345'])
    for i in range(100):
        try:
          for k, v in security_group['SecurityGroups'][0]['IpPermissions'][i]['IpRanges'][0].items():
            if '0.0.0.0' in v:
              print (k, v)
              message = {"alert": "open port found "}
              sns_client = boto3.client("sns", region_name="us-east-1")
              response = sns_client.publish(TargetArn='arn:aws:sns:us-east-1:12345:NotifyMe', Message=json.dumps({'default': json.dumps(message)}), MessageStructure='json')
        except:
          pass

# Here is the python code that will check if any of the files in a given S3 bucket is publicly accessible. Change your-bucket-name, region and access / secret key

import boto
from boto.s3.connection import OrdinaryCallingFormat
conn = boto.s3.connect_to_region('ap-south-1', aws_access_key_id='xxx', aws_secret_access_key='xxx',calling_format=OrdinaryCallingFormat())

mybucket = conn.get_bucket('your-bucket-name')
for key in mybucket.list():
      for grant in key.get_acl().acl.grants :
            if grant.permission == 'READ' :
                print ("PUBLIC: " +str(key))
                #key.set_acl('private')

Here is a simple python script to send SMS.

vi sendsms.py

import boto3

# Create an SNS client
client = boto3.client(
    "sns",
    aws_access_key_id="XXX",
    aws_secret_access_key="XXX",
    region_name="us-east-1"
)

# Send your sms message.
client.publish(
    PhoneNumber="+91981XXXXX66",
    Message="Hello World aws again from docker!",
   MessageAttributes={
    'AWS.SNS.SMS.SMSType': {
      'DataType': 'String',
      'StringValue': 'Transactional'
    }
  }
)


And here is a docker-file, if you are not sure if your server will have python and boto3 module pre-installed.

vi Dockerfile

FROM python:2.7-alpine
RUN pip install boto3

WORKDIR /root/dev

CMD ["python"]

docker build . -t  shantanuo/myboto

# alias pancard='docker run -i --rm -v "$(pwd)":/root/dev/ shantanuo/myboto python sendsms.py  "$@"'

If you do not have a server where you can host your script and docker container, no problem. You can use Amazon Lambda function!

Here is 7 steps process to load data from any csv file into Amazon DynamoDB.

1) Create the pandas dataframe from the source data
2) Clean-up the data, change column types to strings to be on safer side :)
3) Convert dataframe to list of dictionaries (JSON) that can be consumed by any no-sql database
4) Connect to DynamoDB using boto
5) Connect to the DynamoDB table
6) Load the JSON object created in the step 3 using put_item method
7) Test

# Create the pandas dataframe from the source data

import pandas as pd
import boto3

df=pd.read_excel('http://www.tvmmumbai.in/Alumini%20Std.X-2013-2014.xls')

df.columns=["srno", "seat_no", "surname", "name", "father_name", "mother_name", "english", "marathi", "hindi", "sanskrit", "maths", "science","ss","best_of_5", "percent_best_of_5" , "total_out_of_6", "percent_of_600"]

# Clean-up the data, change column types to strings to be on safer side :)

df=df.replace({'-': '0'}, regex=True)
df=df.fillna(0)

for i in df.columns:
    df[i] = df[i].astype(str)

# Convert dataframe to list of dictionaries (JSON) that can be consumed by any no-sql database

myl=df.T.to_dict().values()

# Connect to DynamoDB using boto

MY_ACCESS_KEY_ID = 'XXX'
MY_SECRET_ACCESS_KEY = 'XXX'

resource = boto3.resource('dynamodb', aws_access_key_id=MY_ACCESS_KEY_ID, aws_secret_access_key=MY_SECRET_ACCESS_KEY, region_name='us-east-1')

# Connect to the DynamoDB table

table = resource.Table('marks1')

# Load the JSON object created in the step 3 using put_item method

for student in myl:
    table.put_item(Item=student)

# Test
response = table.get_item(Key={'seat_no': 'A 314216'})
response

Amazon lambda functions can be tested locally using docker as explained in this article...

https://aws.amazon.com/blogs/compute/cloudmicro-for-aws-speeding-up-serverless-development-at-the-coca-cola-company/

_____

git clone https://github.com/Cloudmicro/lambda-dynamodb-local.git

cd lambda-dynamodb-local

docker-compose up -d

docker-compose run --rm -e FUNCTION_NAME=hello lambda-python
_____

If docker compose is not installed, then follow these steps:

curl -L https://github.com/docker/compose/releases/download/1.7.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

You can write lambda funciton that will handle a Slack slash command and echoes the details back to the user.

Follow these steps to configure the slash command in Slack:

1. Navigate to https://.slack.com/services/new
2. Search for and select "Slash Commands".
3. Enter a name for your command and click "Add Slash Command Integration".
4. Copy the token string from the integration settings and use it in the next section.
5. After you complete this blueprint, enter the provided API endpoint URL in the URL field.
_____

Lambda Function

Create a new function using a blueprint called "slack-echo-command-python". The only change is to comment the encryption line and declare the variable...

#kms = boto3.client('kms')
#expected_token = kms.decrypt(CiphertextBlob = b64decode(ENCRYPTED_EXPECTED_TOKEN))['Plaintext']
expected_token = 'A9sU70Lz4isPdTet5tvGD0PB'

You will get this token when you registered a new keyword at slack.
_____

API Gateway

Create an AIP - LambdaMicroservice

Actions - Create Resource - getme
Actions - Create Method - post

Integration type - Lambda Function
Lambda Function: Select function name getme

Integration Request - Body Mapping Templates - Add mapping template - application/x-www-form-urlencoded

Mapping template - {"body":$input.json("$")}

Deploy API - Stage name - prod
_____

Connect API to Lambda Function and slack slash command:

Add the method name to invoke url. If the invoke URL looks like this...

https://sevbnlvu69.execute-api.us-east-1.amazonaws.com/prod

Then the actual URL to be added in "API endpoints" tab of function - getme will be:

https://sevbnlvu69.execute-api.us-east-1.amazonaws.com/prod/getme

In order to let the machine do predictions for us, we need to first train the computers with some data. We will download the test data from here...

https://s3.amazonaws.com/aml-sample-data/banking.csv

This data looks something like this once opened in excel.

age job marital education default housing loan contact month day_of_week duration campaign pdays previous poutcome emp_var_rate cons_price_idx cons_conf_idx euribor3m nr_employed say_yes
44 blue-collar married basic.4y unknown yes no cellular aug thu 210 1 999 0 nonexistent 1.4 93.444 -36.1 4.963 5228.1 0
53 technician married unknown no no no cellular nov fri 138 1 999 0 nonexistent -0.1 93.2 -42 4.021 5195.8 0
28 management single university.degree no yes no cellular jun thu 339 3 6 2 success -1.7 94.055 -39.8 0.729 4991.6 1

The software will learn the pattern from the above data. It will read the data like this...
Someone who is 28 years old and from Management job is likely to say yes to a bank loan. The married older people may likely say "no" (say_yes: 0).

Now when a new customer walks in, we can ask the machine first if he is going to buy the banking product or not.

32,services,divorced,basic.9y,no,unknown,yes,cellular,dec,mon,110,1,11,0,nonexistent,-1.8,94.465,-36.1,0.883,5228.1

This customer is most likely going to say "No" (Predicted label: 0) as per what machine has learned from other customers so far.

http://docs.aws.amazon.com/machine-learning/latest/dg/step-5-use-the-ml-model-to-create-batch-predictions.html

Machine learning can solve very complex problems those are very difficult to answer using standard SQL queries.

Here are 3 steps to start an EC2 instance based on Amazon Linux

import easyboto
dev=easyboto.connect('XXX', 'XXX')

dev.placement='us-east-1a'
dev.key='dec15a'

dev.startEc2('ami-da4d7cb0', 't2.medium')

This will return an IP address of the newly created server and you can connect using the standard ssh command that will look something like this...

ssh -i dec15a.pem ec2-user@52.20.10.47

You should have dec15a.pem or dec15a.ppk (for putty) file in order to connect.
_____

Once you are connected to newly created server, install docker...

## install and start docker deamon

yum install docker

etc/init.d/docker start
_____

## python with notebook can be installed using these 3 commands:

docker run -it -p 7778:7778 --hostname conda shantanuo/miniconda_ipython /bin/bash

cd /home/
ipython notebook --ip=* --port=7778


## adminer can be installed using these 2 commands:

docker run -it -p 80:80 --hostname adminer shantanuo/adminer

sudo service apache2 start

You can pull the latest mysql image and create a container named "new-mysql1". Then start the container in second command as shown below:

docker create -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password --name="new-mysql1" mysql:latest

docker start 22914939f301

Or merge create + start into a single "run" command as shown below:

docker run --name new-mysql1 -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql/mysql-server:latest
_____

Then you can access from your host using the mysql command line:

mysql -h127.0.0.1 -ppassword -uroot

1) List all installed images:

# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
redis                   latest              099f1d00ac84        2 days ago          151.3 MB
continuumio/miniconda   latest              7a285fa253c7        9 days ago          405.2 MB
jpetazzo/nsenter        latest              e8f4be644d49        5 months ago        368.3 MB

2) List active containers:
# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
62a73bb80e54        redis                   "/entrypoint.sh redis"   10 minutes ago      Up 10 minutes       6379/tcp               myredis1
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   49 minutes ago      Up 49 minutes       0.0.0.0:80->7778/tcp   small_leakey

3) List all containers:
# docker ps  -a
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                      PORTS                  NAMES
62a73bb80e54        redis                   "/entrypoint.sh redis"   10 minutes ago      Up 10 minutes               6379/tcp               myredis1
55a56eece76e        jpetazzo/nsenter        "/bin/sh -c /installe"   19 minutes ago      Exited (0) 19 minutes ago                          jolly_albattani
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   49 minutes ago      Up 49 minutes               0.0.0.0:80->7778/tcp   small_leakey
d46a8bc2239e        redis                   "/entrypoint.sh redis"   51 minutes ago      Exited (0) 11 minutes ago                          myredis

4) Start a container based on available image:
# docker start 55a56eece76e

4) stop (or kill) and remove files of the working container. use -t 25 with sop command so that container will be forcefully killed after 25 seconds.
# docker stop 62a73bb80e54
62a73bb80e54

# docker rm 62a73bb80e54
62a73bb80e54

5) remove the container even if it is stopped so that we can remove the image in the next step
#docker rm d46a8bc2239e

6) remove the image completely
# docker rmi redis
Untagged: redis:latest
Deleted: 099f1d00ac840d7a0037c5f5232c37dfcc986805207ce73a965b75a23e2a4f82
Deleted: dd65623527ec836770b73863b9ca463d11e8979f787a4d0d96621e00794a9b98


7) Pause and unpause container:

# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   55 minutes ago      Up 55 minutes       0.0.0.0:80->7778/tcp   small_leakey

# docker pause 8c068c974e73
8c068c974e73

# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                   PORTS                  NAMES
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   55 minutes ago      Up 55 minutes (Paused)   0.0.0.0:80->7778/tcp   small_leakey

docker unpause 8c068c974e73

8) Download an image

docker create -p 6379:6379 redis:2.8

docker ps -a
# to get the container ID

docker start 8c068c974e73

9) Download and start both steps merged together using run command:

docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

9a) Connect to any redis server using docker
docker run -it redis redis-cli -h  myredis.synfmnx.0001.use1.cache.amazonaws.com

10)  stress test

docker run --rm -ti -c 512 --cpuset=0 progrium/stress --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 120s

Lower the allocated memory and increase used memory - recipe for crash:

docker run --rm -ti -m 200m --memory-swap=300m progrium/stress --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 120s

11) Copy a file from container to base machine

docker cp 8c068c974e73:/testdocker.ipynb .

easyboto is a library that makes initiating an EC2 instance very easy.

import easyboto
x=easyboto.connect('your_access_key', 'your_secret_key')

x.placement='us-east-1a'
# use the free IP address if available
#x.myaddress='52.71.62.77'
x.key='dec15a'

# t2.nano (0.5 - $0.0065), t2.micro (1 - $0.013) t2.small (2 - $0.026), t2.medium (4 - $0.052), t2.large (8 - $0.104),
# m4.large (8 - $0.126 ), m4.xlarge (16 - $0.252), m4.2xlarge (32 - $0.504), m4.4xlarge (64 - $1.008)
# ami-da4d7cb0 is based on Amazon Linux AMI 2015.09.2 (HVM), changed SSD to mangetic with 200 GB

x.startEc2('ami-da4d7cb0', 'm4.4xlarge')

# use Spot method for cheaper rates
# x.MAX_SPOT_BID= '0.5'
# x.startEc2Spot('ami-da4d7cb0', 'm4.4xlarge')

1) Initiate a server using Amazon Linux from AWS console

https://console.aws.amazon.com/ec2

2) install docker

yum install docker

3) download conda image and initiate in interactive mode

docker run -t -p80:7778 -i continuumio/miniconda /bin/bash

4) install ipython notebook

conda install ipython-notebook

5) start notebook server on port 7778

ipython notebook --ip=* --port=7778

6) Your ipython notebook server is available on default 80 port of base machine that can be accessed here...

http://ec2-54-84-139-56.compute-1.amazonaws.com/

_____

You can log-in to the docker container using "execute" command as shown below. You will need TTY and interactive mode to access /bin/bash of the container.

docker exec -t -i  8c068c974e73 /bin/bash

Once you are in, simply call the conda command like this...

conda install --channel https://conda.binstar.org/bkreider postgresql psycopg2
conda install pandas
conda install pandasql
wget https://raw.githubusercontent.com/shantanuo/easyboto/master/easyboto.py
   
_____

namespace enter can be installed using this container:

docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter

Once installed, you can enter any container using the command...

/usr/local/bin/docker-enter 8c068c974e73 /bin/bash

namespace enter is similar to execute as shown above, but has more options.
_____

You can download the latest version of this image using pull command...

docker pull continuumio/miniconda:latest
_____

You can check stats, logs, events and info if everything got started as expected...

docker logs 8c068c974e73

Once you have created a vault in Glacier, goto "Settings" and choose "Vault Lock" as shown in this image.



This policy will not allow anyone to delete a file from "Business" vault for 7 years.

{
"version": "2012-10-17",
"statement": [
    {
"effect": "Deny",
"Principal": {"AWS": "*" },
"Action": "glacier:DeleteArchive",
"Resource": "arn:aws:glacier:us-east-1:account-number-12digit-without-dash:vaults/Business",
"Condition" {"NumericLessThanEquals": {"glacier:ArchiveAgeInDays": "2555"}}
     }
             ]
}

You can change the condition to lock the valuts tagged as "LegalHold"

"Condition": {"StringEquals": {"glacier":ResourceTag/LegalHold": "True"}}



It is highly recommended to create another AWS account for such long-term vaults so that you can cancell the account itself if you no longer need those files in the vault anymore.

I can create an API that will take the current IP of requesting client and add that IP address to a security group to open up certain ports.

Integration request:
content-type: application/json

Template:
{
"sourceip": "$context.identity.sourceIp"
}

The lambda function will look something like this...

def lambda_handler(event, context):
    from boto3.session import Session
    session = Session(aws_access_key_id='XXX', aws_secret_access_key='YYY')
    ec2 = session.resource('ec2')
    security_group = ec2.SecurityGroup('sg-f682c78f')
    myip=event['sourceip']+"/32"
    security_group.authorize_ingress(IpProtocol="tcp",CidrIp=myip,FromPort=80,ToPort=80)
    return (event['sourceip'])
  

1) Here is how we can connect to mongodb and create a "db" object.

# connect to mongo test database
from pymongo import MongoClient
client = MongoClient()
db = client.test

2) Save dict data to mongodb test database, collection name: posts

y={"name": "amar", "age": 30}
db.posts.insert_one(y)

3) Get a sample record using findOne

db.posts.find_one()

4) copy the pandas dataframe to mongo

# assuming "df" is a dataframe that is alrady created

db.posts.insert_many(df.to_dict('records'))

If you get an error, you may need to correct the data using map function

df["region"] = df["region"].map(lambda x: str(x).split(':')[-1:])

This will remove the semicolon : from region column and select the last slice.

5) Import mongo data to pandas dataframe

# get all data from posts collection into a list of dicts
mylist=[]
for post in db.posts.find():
   mylist.append(post)

# convert the list to pandas dataframe
import pandas as pd
pd.DataFrame(mylist)

Here is python code that will connect to Redshift and pull the data into a dataframe. It will then copy the data to MySQL table. If the table exist, it will replace the table.

import easyboto
x = easyboto.myboto('xxx', 'yyy')mydf = x.runQuery("select *  from pg_table_def where schemaname = 'public'")
mydf.columns = ['schemaname', 'tablename', 'column_nm', 'type', 'encoding', 'distkey', 'sortkey', 'notnull']

import sqlalchemy
engine = sqlalchemy.create_engine('mysql://dba:dba@127.0.0.1/test')
mydf.to_sql('testdata', engine, if_exists='replace')