Shantanu's Blog

The default elasticsearch image does not support scripting. So I have created a new image that anyone can download from...

Here are the steps used to create the new image.

# cat elasticsearch.yml
script.inline: true
script.indexed: true
network.host: 0.0.0.0

# cat Dockerfile
from elasticsearch
copy elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml

# docker build -t shantanuo/elasticsearch-script .

# docker push shantanuo/elasticsearch-script

I can now run the container based on the image I just created.

# docker run -d -p 9200:9200 -p 9300:9300 shantanuo/elasticsearch-script

Find the name of the container and link it to kibana like this...

# docker run -d -p 5603:5601 --link  stoic_goldstine:elasticsearch -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana

# or use this to start es with kibana
docker run -d -p 9200:9200 -p 9300:9300 -p5603:5601 -e ES_HEAP_SIZE=1g --name myelastic  shantanuo/elastic
_____

This image is based on official elasticsearch image. If I need to build everything from scratch, then I can use Ubuntu official image as shown here...

https://hub.docker.com/r/shantanuo/elasticsearch/

Here are 2 docker commands to start elasticsearch with kibana.

docker run -d -p 9200:9200 -p 9300:9300 --name elasticsearch-pb elasticsearch
docker run -d -p 5601:5601 --name kibana-pb --link elasticsearch-pb:elasticsearch -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana

You can now push logs to elasticsearch using logstash or use beat.
Install beat package on the server from where you want to push the logs...

deb:

sudo apt-get install libpcap0.8
curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat_1.2.3_amd64.deb
sudo dpkg -i packetbeat_1.2.3_amd64.deb

rpm:

sudo yum install libpcap
curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-x86_64.rpm
sudo rpm -vi packetbeat-1.2.3-x86_64.rpm

### Config yml file

vi /etc/packetbeat/packetbeat.yml

# Multiple outputs may be used.
output:
  ### Elasticsearch as output
  elasticsearch:
    # hosts: ["localhost:9200"]
    # hosts: ["ec2-54-65-142-180.compute-1.amazonaws.com"]
    hosts: ["search-es-demo-jyhk2or3v3sesrgt6dgn5u7qm.us-east-1.es.amazonaws.com:443"]
    protocol: "https"

    # A template is used to set the mapping in Elasticsearch
    template:
      # Path to template file
      path: "packetbeat.template.json"

### start packetbeat

/etc/init.d/packetbeat start
_____

The packetbeat configuration file without comments looks something like this...

# cat /etc/packetbeat/packetbeat.yml  | grep -v '#' | grep -v '^$'
interfaces:
  device: any
protocols:
  dns:
    ports: [53]
    include_authorities: true
    include_additionals: true
  http:
    ports: [80, 8080, 8000, 5000, 8002]
  memcache:
    ports: [11211]
  mysql:
    ports: [3306]
  pgsql:
    ports: [5432]
  redis:
    ports: [6379]
  thrift:
    ports: [9090]
  mongodb:
    ports: [27017]
output:
  elasticsearch:
    hosts: ["search-es-demo-jyt2or3v3sesrgt6dgn5u7qm.us-east-1.es.amazonaws.com:443"]
    protocol: "https"
    template:
      path: "packetbeat.template.json"
shipper:
logging:
  files:

hotmail and yahoo mail were good. Gmail is much much better. But it still lacks the flexibility that I am looking for. for example, these are very simple tasks but takes a lot of time. The reason is that we do not have direct access to the emails. We have limited choice provided by the user inferface and if our needs outgrow, there is either no way to do that or a lot of programming is involved that adds to the cost of doing business.

* Print the subject line of the last 20 emails sent to with someone@company.com
* Download all versions of the last 2 attachments exchanged with someone@company.com
* Download all attachments with a file name that contains the word 'proposal'

context.io provides RESTful email API that makes my life easy. I can now do whatever I like with my email. The web is finally becoming interesting again! It feels like we are in the late 90's or early 2000's. This API will free up the valuable data stored in the form of emails and allow us do things we never thought possible.
 _____

The following code will fetch the subject of emails sent to client@company.com

from contextIO.ContextIO import ContextIO

api_key = 'abcdefg5'
api_secret = 'abcd8ef2ghijklmn'
mailbox_to_query = 'shantanu@gmail.com'
api_client = ContextIO(api_key=api_key, api_secret=api_secret)

response = api_client.contactmessages(account=mailbox_to_query,to_address='client@company.com',limit=20)
for message in response.get_data():
    print 'Subject %s' % message['subject']

'''
_____

Let's assume we need to find all the communication sent to the client. I can do simple search
to:client@company.com

That is not enough. I need to find the mails sent from my account as well as what others have sent.
to:client@company.com in:sent
to:client@company.com not in:sent

More info:
http://context.io/docs/2.0/accounts/contacts/messages#get
'''

response = api_client.contactmessages(account=mailbox_to_query,to_address='client@company.com',limit=20)
for message in response.get_data():
    if any('\Sent' in s for s in message['folders']):
        print 'successfully sent by me'
        print 'Subject %s' % message['subject']
    else:
        print 'someone else has sent mail to him'
        print 'Subject %s' % message['subject']

# get all details like subject, date (unix time), message ID, labels, from - to email addresses along with photos (personInfo)
    for mykey in message:
        print mykey,  message[mykey]
_____

installation is easy:

git clone git@github.com:contextio/Python-ContextIO.git
cd Python-ContextIO/
python setup.py install
cd example/
python examples.py

QlikView is a Business Intelligence (BI) software, which combines the features of dynamic presentations and instantaneous data manipulation. The paid version comes with a nice user friendly web interface. For example check out the Global Games app by qlikview.

http://ap.demo.qlikview.com/QvAJAXZfc/opendoc.htm?document=qvdocs/Global%20Games.qvw&host=Demo11

a) Australia's performance has deteriorated in the last 3 Olympics.
b) Australians are good at swimming and cycling but not in Shooting.
c) Male to female ratio is equal.

There are 3 steps involved to get any type of report. for e.g.
a) Type australia in the search box.
b) Choose the last 3 years in the years drop down on dashboard.
c) Click on "country profile" tab.

Qlikview has eliminated the scripting layer from the reporting process.  Reporting can be easy, intuitive and fun!

_____


After testing the tool with dummy data, I have come up with some interesting advantages over traditional spreadsheet tools like Excel.

1) Charts
Charts can be easily created based on the selected data. Zoom in / out and export to image file is as easy as pie. (image 1)



2) Two dimensional data
This has been explained in image 2 and image 3.


 
Here are the report inferences those can be derived based on the data analysis.
a) Acer devices are using Android as well as Windows Mobile OS but Android is more popular. (image 2)
b) Acer has around 17 models with A100 and S300 being most poular. (image 3)



Excel can do this, but reverse engineering is not possible in excel.
For e.g. If I need to know all the devices those use Android platform, I will have to re-sort and then re-group the data. qlikview allows us to do all this and much more using right click button as shown in image 4.