Shantanu's Blog

Database Consultant

February 29, 2016

 

docker cheat sheet

### download image and create container

# configuration info about docker

docker info

# download official mysql image

docker pull mysql

# list all images

docker images

# create container based on mysql image. Use environment variable, port and mount data directory on base machine

docker create -v /my/own/datadir:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw mysql

# list all containers

docker ps -a

# start conatiner

docker start fbe215634ded

# list acitive containers

docker ps

# execute a linux command inside a container

docker exec  fbe215634de  ls -lht

# login to container

docker exec -it  fbe215634de /bin/bash

# pull + create + start + exec = run

docker run -d -v /my/own/datadir:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw --hostname=vdb mysql

# also use advanced options if required
# --privileged=true --net=host -m 100m -c 512

## you can use other options like name, label, hostname and restart=on-failure:3

# copy files from container to base server:

docker cp fbe215634de:/data .

# Commit your changes and save the container to an image called mynewimage.

docker commit 3a09b2588478 mynewimage




### managing containers

# pause and unpause a container

docker pause  fbe215634de


# logs, inspect, top and diff

docker logs fbe215634de

# stats & events

docker stats fbe215634de



### import / export a container or save / load image

docker save mynewimage > /tmp/mynewimage.tar

docker load < /tmp/mynewimage.tar


### remote access to docker

# Add tcp option to sys config for remote access

vi /etc/sysconfig/docker

OPTIONS="--host=tcp://0.0.0.0:2375"

/etc/init.d/docker restart

# create alias for adding host option to docker command

alias docker='docker -H tcp://0.0.0.0:2375'

# know mysql data size

for i in `docker ps -q` ; do docker exec -it $i du /var/lib/mysql -hs ; done

Labels: , ,


 

Manage Docker containers using python

Use docker module to manage your containers using python.

Install the required module:

pip install docker-py
_____

Make sure that docker host configured to allow remote access.

vi /etc/sysconfig/docker
OPTIONS="--host=tcp://0.0.0.0:2375"
/etc/init.d/docker restart


Also make sure that firewall is not stopped:
/etc/init.d/iptables stop

And check that port 2375 is open and accessible from outside world
telnet 352.6.168.211 2375
_____

# connect to docker server running on the host:port
from docker import Client
cli = Client(base_url='tcp://352.6.168.211:2375')

# list running containers
cli.containers()

# Information about docker server
cli.info()

# Pull an image called "busybox"
import json
for line in cli.pull('busybox', stream=True):
    print(json.dumps(json.loads(line), indent=4))

# create a container based on busybox image and run it for 300 seconds
cli.create_container(image='busybox:latest', command='/bin/sleep 300', name='sleeper7')

# start the container
cli.start('sleeper7')

# list the running container and you should see the container named "sleeper7"
bbox= cli.containers()

# return the ID of the sleeper7 container
bbox[0]['Id']

# Stop the sleeper7 container
cli.stop(bbox[0]['Id'])

# Or you can use the ID of the "sleeper7" container derived from bbox object
cli.stop('4804bc91f3370868dc9297206109493636d92f4b527ca40e10a359b7b82b6657')

# list all containers # docker ps -a
cnt=cli.containers(all=True)

# Save the cnt object created above to pandas dataframe
import pandas as pd
cnt_df = pd.DataFrame(cnt)

# Remove "sleeper7" container
cli.remove_container('sleeper7')

# Remove "bubybox" image
cli.remove_image('busybox')

# List all images and save it as "mylist" object
mylist=cli.images()

# Import the mylist object to pandas dataframe
import pandas as pd
df=pd.DataFrame(mylist)

# Save the data frame object df as csv file
df.to_csv('abc.csv')
_____

docker ps command does not reveal environment variables and other important metadata. We need to use inspect command to get the json formatted information. Instead you can use python docker module as shown here...

from docker import Client
cli = Client()

for i in cli.containers():
    x = i['Id']
    d=cli.inspect_container(x)
    print d['Config']['Env'], d['Config']['Cmd'], d['HostConfig']['Binds'],  d['Mounts']
    print '==============================='


Labels: , , , ,


February 28, 2016

 

options for running docker

Use the following parameters while running a new container.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

  -a, --attach=[]                 Attach to STDIN, STDOUT or STDERR
  -d, --detach=false              Run container in background and print container ID

  -e, --env=[]                    Set environment variables
  -p, --publish=[]                Publish a container's port(s) to the host
  -v, --volume=[]                 Bind mount a volume
  -m, --memory=                   Memory limit
  --cpu-shares=0                  CPU shares (relative weight)
  --volumes-from=[]               Mount volumes from the specified container(s)

  -i, --interactive=false         Keep STDIN open even if not attached
  -t, --tty=false                 Allocate a pseudo-TTY
  --rm=false                      Automatically remove the container when it exits

  -l, --label=[]                  Set meta data on a container
  --name=                         Assign a name to the container
  -h, --hostname=                 Container host name

  --net=default                   Set the Network for the container
  --privileged=false              Give extended privileges to this container
  --read-only=false               Mount the container's root filesystem as read only
  --restart=no                    Restart policy to apply when a container exits

Labels: , , , ,


 

Install MySQL using docker

## create a directory on base machine for mysql data
mkdir -p /my/own/datadir

## link data directory and port 3306 to base machine while starting mysql container from official mysql image
docker run --name some-mysql1 -v /my/own/datadir:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql

## connect to your mysql
mysql -uroot -pmy-secret-pw -h127.0.0.1 -P 3306

Labels: , , , ,


 

docker mysql

You can pull the latest mysql image and create a container named "new-mysql1". Then start the container in second command as shown below:

docker create -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password --name="new-mysql1" mysql:latest

docker start 22914939f301

Or merge create + start into a single "run" command as shown below:

docker run --name new-mysql1 -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql/mysql-server:latest
_____

Then you can access from your host using the mysql command line:

mysql -h127.0.0.1 -ppassword -uroot

Labels: , , , , , ,


February 19, 2016

 

docker tips

1) List all installed images:

# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
redis                   latest              099f1d00ac84        2 days ago          151.3 MB
continuumio/miniconda   latest              7a285fa253c7        9 days ago          405.2 MB
jpetazzo/nsenter        latest              e8f4be644d49        5 months ago        368.3 MB

2) List active containers:
# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
62a73bb80e54        redis                   "/entrypoint.sh redis"   10 minutes ago      Up 10 minutes       6379/tcp               myredis1
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   49 minutes ago      Up 49 minutes       0.0.0.0:80->7778/tcp   small_leakey

3) List all containers:
# docker ps  -a
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                      PORTS                  NAMES
62a73bb80e54        redis                   "/entrypoint.sh redis"   10 minutes ago      Up 10 minutes               6379/tcp               myredis1
55a56eece76e        jpetazzo/nsenter        "/bin/sh -c /installe"   19 minutes ago      Exited (0) 19 minutes ago                          jolly_albattani
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   49 minutes ago      Up 49 minutes               0.0.0.0:80->7778/tcp   small_leakey
d46a8bc2239e        redis                   "/entrypoint.sh redis"   51 minutes ago      Exited (0) 11 minutes ago                          myredis

4) Start a container based on available image:
# docker start 55a56eece76e

4) stop (or kill) and remove files of the working container. use -t 25 with sop command so that container will be forcefully killed after 25 seconds.
# docker stop 62a73bb80e54
62a73bb80e54

# docker rm 62a73bb80e54
62a73bb80e54

5) remove the container even if it is stopped so that we can remove the image in the next step
#docker rm d46a8bc2239e

6) remove the image completely
# docker rmi redis
Untagged: redis:latest
Deleted: 099f1d00ac840d7a0037c5f5232c37dfcc986805207ce73a965b75a23e2a4f82
Deleted: dd65623527ec836770b73863b9ca463d11e8979f787a4d0d96621e00794a9b98


7) Pause and unpause container:

# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   55 minutes ago      Up 55 minutes       0.0.0.0:80->7778/tcp   small_leakey

# docker pause 8c068c974e73
8c068c974e73

# docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                   PORTS                  NAMES
8c068c974e73        continuumio/miniconda   "/usr/bin/tini -- /bi"   55 minutes ago      Up 55 minutes (Paused)   0.0.0.0:80->7778/tcp   small_leakey

docker unpause 8c068c974e73

8) Download an image

docker create -p 6379:6379 redis:2.8

docker ps -a
# to get the container ID

docker start 8c068c974e73

9) Download and start both steps merged together using run command:

docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

9a) Connect to any redis server using docker
docker run -it redis redis-cli -h  myredis.synfmnx.0001.use1.cache.amazonaws.com


10)  stress test

docker run --rm -ti -c 512 --cpuset=0 progrium/stress --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 120s

Lower the allocated memory and increase used memory - recipe for crash:

docker run --rm -ti -m 200m --memory-swap=300m progrium/stress --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 120s

11) Copy a file from container to base machine

docker cp 8c068c974e73:/testdocker.ipynb .

12) Storage volumes:


Labels: , , , ,


February 18, 2016

 

install package using one line docker command

You can install and launch redis server in just one command...

 docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

This will use redis configuration file from base image file /myredis/conf

Labels: , , , , ,


 

Making AWS usable

easyboto is a library that makes initiating an EC2 instance very easy.

import easyboto
x=easyboto.connect('your_access_key', 'your_secret_key')

x.placement='us-east-1a'
# use the free IP address if available
#x.myaddress='52.71.62.77'
x.key='dec15a'

# t2.nano (0.5 - $0.0065), t2.micro (1 - $0.013) t2.small (2 - $0.026), t2.medium (4 - $0.052), t2.large (8 - $0.104),
# m4.large (8 - $0.126 ), m4.xlarge (16 - $0.252), m4.2xlarge (32 - $0.504), m4.4xlarge (64 - $1.008)
# ami-da4d7cb0 is based on Amazon Linux AMI 2015.09.2 (HVM), changed SSD to mangetic with 200 GB

x.startEc2('ami-da4d7cb0', 'm4.4xlarge')

# use Spot method for cheaper rates
# x.MAX_SPOT_BID= '0.5'
# x.startEc2Spot('ami-da4d7cb0', 'm4.4xlarge')

Labels: , , , , ,


 

Start notebook server in 6 easy steps

1) Initiate a server using Amazon Linux from AWS console

https://console.aws.amazon.com/ec2

2) install docker

yum install docker

3) download conda image and initiate in interactive mode

docker run -t -p80:7778 -i continuumio/miniconda /bin/bash

4) install ipython notebook

conda install ipython-notebook

5) start notebook server on port 7778

ipython notebook --ip=* --port=7778

6) Your ipython notebook server is available on default 80 port of base machine that can be accessed here...

http://ec2-54-84-139-56.compute-1.amazonaws.com/

_____

You can log-in to the docker container using "execute" command as shown below. You will need TTY and interactive mode to access /bin/bash of the container.

docker exec -t -i  8c068c974e73 /bin/bash

Once you are in, simply call the conda command like this...

conda install --channel https://conda.binstar.org/bkreider postgresql psycopg2
conda install pandas
conda install boto
conda install pandasql
wget https://raw.githubusercontent.com/shantanuo/easyboto/master/easyboto.py
   
_____

namespace enter can be installed using this container:

docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter

Once installed, you can enter any container using the command...

/usr/local/bin/docker-enter 8c068c974e73 /bin/bash

namespace enter is similar to execute as shown above, but has more options.
_____

You can download the latest version of this image using pull command...

docker pull continuumio/miniconda:latest
_____

You can check stats, logs, events and info if everything got started as expected...

docker logs 8c068c974e73

Labels: , , , , , , ,


February 08, 2016

 

Locking AWS Vault for 7 years

Once you have created a vault in Glacier, goto "Settings" and choose "Vault Lock" as shown in this image.



This policy will not allow anyone to delete a file from "Business" vault for 7 years.

{
"version": "2012-10-17",
"statement": [
    {
"effect": "Deny",
"Principal": {"AWS": "*" },
"Action": "glacier:DeleteArchive",
"Resource": "arn:aws:glacier:us-east-1:account-number-12digit-without-dash:vaults/Business",
"Condition" {"NumericLessThanEquals": {"glacier:ArchiveAgeInDays": "2555"}}
     }
             ]
}

You can change the condition to lock the valuts tagged as "LegalHold"

"Condition": {"StringEquals": {"glacier":ResourceTag/LegalHold": "True"}}



It is highly recommended to create another AWS account for such long-term vaults so that you can cancell the account itself if you no longer need those files in the vault anymore.

Labels: , ,


February 02, 2016

 

Starting with docker

1) Docker can be easily installed if you are using Amazon Linux. Here are the steps to install and run docker.

sudo yum update -y

sudo yum install -y docker

sudo service docker start

# applies on for AWS
sudo usermod -a -G docker ec2-user

2) Let's download a sample application from github and build it as docker image.

git clone https://github.com/awslabs/ecs-demo-php-simple-app

cd ecs-demo-php-simple-app

cat Dockerfile

docker build -t shantanuo/amazon-ecs-sample .

3) You can login to docker hub and push your image.

docer login

docker push shantanuo/amazon-ecs-sample


4) now you can pull it down and "activate" the contents floating in the docker image.

docker pull shantanuo/amazon-ecs-sample

docker run -p 80:80 shantanuo/amazon-ecs-sample

Labels: , , , ,


Archives

June 2001   July 2001   January 2003   May 2003   September 2003   October 2003   December 2003   January 2004   February 2004   March 2004   April 2004   May 2004   June 2004   July 2004   August 2004   September 2004   October 2004   November 2004   December 2004   January 2005   February 2005   March 2005   April 2005   May 2005   June 2005   July 2005   August 2005   September 2005   October 2005   November 2005   December 2005   January 2006   February 2006   March 2006   April 2006   May 2006   June 2006   July 2006   August 2006   September 2006   October 2006   November 2006   December 2006   January 2007   February 2007   March 2007   April 2007   June 2007   July 2007   August 2007   September 2007   October 2007   November 2007   December 2007   January 2008   February 2008   March 2008   April 2008   July 2008   August 2008   September 2008   October 2008   November 2008   December 2008   January 2009   February 2009   March 2009   April 2009   May 2009   June 2009   July 2009   August 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   May 2010   June 2010   July 2010   August 2010   September 2010   October 2010   November 2010   December 2010   January 2011   February 2011   March 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   October 2012   November 2012   December 2012   January 2013   February 2013   March 2013   April 2013   May 2013   June 2013   July 2013   September 2013   October 2013   January 2014   March 2014   April 2014   May 2014   July 2014   August 2014   September 2014   October 2014   November 2014   December 2014   January 2015   February 2015   March 2015   April 2015   May 2015   June 2015   July 2015   August 2015   September 2015   January 2016   February 2016   March 2016   April 2016   May 2016   June 2016   July 2016   August 2016   September 2016   October 2016   November 2016   December 2016   January 2017   February 2017   April 2017   May 2017   June 2017   July 2017   August 2017   September 2017   October 2017   November 2017   December 2017   February 2018   March 2018   April 2018   May 2018   June 2018   July 2018   August 2018   September 2018   October 2018   November 2018   December 2018   January 2019   February 2019   March 2019   April 2019   May 2019   July 2019   August 2019   September 2019   October 2019   November 2019   December 2019   January 2020   February 2020   March 2020   April 2020   May 2020   July 2020   August 2020   September 2020   October 2020   December 2020   January 2021   April 2021   May 2021   July 2021   September 2021   March 2022   October 2022   November 2022   March 2023   April 2023   July 2023   September 2023   October 2023   November 2023  

This page is powered by Blogger. Isn't yours?