Shantanu's Blog

FastText library by facebook has the language detection feature.

import fasttext
model = fasttext.load_model("/tmp/lid.176.ftz")
model.predict(" विकिपीडिया पर", k=2)

The above code returns Hindi "hi" correctly. Google also has it's own library called langdetect. The following code returns Marathi "mr" correctly.

from langdetect import detect
detect("आत्मा आणि")

The polyglot library has supported this and other language tools since a very long time.

https://github.com/saffsd/polyglot

Labels: nlp, python

I have 2 text files. corpus file is the collection of words and exclude file has all the suffixes. I need to extract the stemmed words after removing all suffixes.

==> exclude.txt <==
works
ed
s
ing
ings

==> corpus.txt <==
worked
working
works
tested
tests
find
found
workings

awk -f tst.awk exclude.txt corpus.txt | sort

unmatched find
unmatched found
matched working/s
matched work/ed,ing,s,ings
matched test/ed,s

And the awk script will look something like this...

$ cat tst.awk
{ lineLgth = length($0) }
NR == FNR {
    suffixes[$0]
    sfxLgths[lineLgth]
    next
}
{
    base = ""
    for ( sfxLgth in sfxLgths ) {
        baseLgth = lineLgth - sfxLgth
        if ( baseLgth > 0 ) {
            sfx = substr($0,baseLgth+1)
            if ( sfx in suffixes ) {
                base = substr($0,1,baseLgth)
                bases2sfxs[base] = bases2sfxs[base] "," sfx
            }
        }
    }
    if ( base == "" ) {
        print "unmatched", $0
    }
}
END {
    for ( base in bases2sfxs ) {
        sub(/,/,"/",bases2sfxs[base])
        print "matched", base bases2sfxs[base]
    }
}

Labels: awk, nlp

Itihāsa is a Sanskrit-English translation corpus containing 93,000 Sanskrit shlokas and their English translations extracted from M. N. Dutt's seminal works on The Rāmāyana and The Mahābhārata. 

https://github.com/rahular/itihasa

Itihāsa can be used directly from Huggingface Datasets:

from datasets import load_dataset
dataset = load_dataset("rahular/itihasa")
dataset['train'][0]

{'translation': {'en': 'The ascetic Vālmīki asked Nārada, the best of sages and foremost of those conversant with words, ever engaged in austerities and Vedic studies.',  'sn': 'ॐ तपः स्वाध्यायनिरतं तपस्वी वाग्विदां वरम्। नारदं परिपप्रच्छ वाल्मीकिर्मुनिपुङ्गवम्॥'}}

Labels: nlp

This function takes any text and summarizes it.

from marvin import ai_fn

@ai_fn
def summarize(text: str) -> str:

    """Summarize the provided text"""

import wikipedia
page = wikipedia.page('large language model')
summarize(text=page.content)

Labels: nlp, openai

I can use chatGPT to generate fake data for testing in pandas dataframes in python!

import pandas as pd
from marvin import ai_fn

@ai_fn
def fake_people(n: int) -> list[dict]:

    """
    Generates n examples of fake data representing people,  each with a name and an age.

    """

myfake=fake_people(3)
df = pd.DataFrame(myfake)
print (df)

# will return the dataframe something like this:

    name  age
0   John   28
1  Emily   35
2  David   19

# docker run --rm -it --entrypoint /bin/bash python:3.10
# pip install marvin pandas

Labels: nlp, openai

I compared the text similarity algorithms for the folloing strings:

string_a = 'MUKESH VITHAL GURAV'
string_b = 'MUKESH VITTHAL GURAO'

I found that all the popular methods like cosine, levenshtein, jaro do not work. The best performance was by "entropy_ncd" that is built using word embeddings. The openai model returned the similar (98.10%) score.

import textdistance
mydic = dict()
for i in myalgs:
    try:

        exec('x = textdistance.algorithms.'+i+'.normalized_similarity("'+string_a+'", "'+string_b+'")')
        mydic[i] = round(x,4) 

    except:
        pass

dict(sorted(mydic.items(), key=lambda item: item[1]))

 'levenshtein': 0.9,
 'ratcliff_obershelp': 0.9231,

 'sorensen': 0.9231,
 'sorensen_dice': 0.9231,
 'cosine': 0.9234,
 'needleman_wunsch': 0.925,
 'gotoh': 0.9474,
 'overlap': 0.9474,
 'jaro': 0.9491,
 'editex': 0.95,
 'length': 0.95,
 'jaro_winkler': 0.9695,
 'strcmp95': 0.9695,
 'entropy_ncd': 0.9815

import openai
from openai.embeddings_utils import get_embedding, cosine_similarity
openai.api_key = "sk-xxx"

string_b_embed = get_embedding(string_b, engine="text-similarity-davinci-001")
string_a_embed = get_embedding(string_a, engine="text-similarity-davinci-001")
cosine_similarity(string_a_embed, string_b_embed)

Labels: machine_learning, nlp, openai

While everyone is discussing about chatGPT by openAI, I decided to try it with an old puzzle that I could not solve for a very long time. The problem is about finding the similar names where almost  50% words are non-English.

Following strings are semantically similar but there  is no algorithm that can tell the truth. For e.g. the fuzzy similarity match will return 82 and 64 percent score while there are several other entries in that range. 

from thefuzz import fuzz

fuzz.ratio('A.R.B.GARUD ARTS, COMM.& SCIENCE COLLEGE SHENDURNI', 'A.R.B.Garud ARTS, COM, SCI. COLLEGE SHENDURNI') # 82% 

fuzz.ratio('AADARSH HIGH SCHOOL MANDAL',  'AADARSHA VIDYA. MAR.HIGHSCHOOL') # 64%

Text embeddings by openai (text-embedding-ada-002) returned more than 95% similarity score for both the entries and this is very surprising as well as encouraging. It is very close to human accuracy if not better.

Labels: machine_learning, nlp

We can encode a sentence  into 768 dimensions using a pre-built model.
The data can be visualized using tensorflow projector.

https://projector.tensorflow.org/

Install the required python module:
#!pip install sentence-transformers

We will open a csv file having 2 columns. Text and label.
We will modify the dataframe to add outliers to the data.

import pandas as pd

df = pd.read_csv(
    "http://bit.ly/dataset-sst2", nrows=100, sep="\t", names=["text", "label"]
)

df["label"] = df["label"].replace({0: "negative", 1: "positive"})
df.loc[[10, 27, 54, 72, 91], "text"] = "askgkn askngk kagkasng"
df.to_csv("metadata.tsv", index=False, sep="\t")

Use the distilbert pre-built model to transform the text to return encoded values. Unlike other word vectors, this considers the entire sentence and return 768 dimensions irrespective of number of words per sentence.

from sentence_transformers import SentenceTransformer
sentence_bert_model = SentenceTransformer("distilbert-base-nli-stsb-mean-tokens")
e = sentence_bert_model.encode(df["text"])
embedding_df = pd.DataFrame(e)
embedding_df.to_csv("output.tsv", index=False, sep="\t", header=None)

The numpy array is converted to pandas dataframe and exported as csv.
The 2 files can be uploaded to tensorflow projector and you can easily find the outliers as shown in this blog post...

https://amitness.com/interactive-sentence-embeddings/

Labels: machine_learning, nlp, tensorflow

Labels: machine_learning, nlp, python

Labels: machine_learning, nlp

Labels: nlp, python, usability