Configuring TAEB at launch to use a different AI(and other settings)

I've been playing around with Taeb and Nethack for a few days now. I'm really glad I ignored this game during college. I have a few friends who didn't and didn't finish.

I gave myself a crash course review of Perl. I mainly did this so I could understand the code for Demo.pm, which is the reference agent. I wanted to try to change agents, but I mistakenly thought this could be done with taeb's --config option.

In order to change the AI of Taeb, one must place a yml file into $HOME/.taeb. The yml file must be renamed to config.yml

Setup of TAEB on openSUSE 11.1

TAEB stands for Tactical Amulet Extraction Bot. It is a framework written Perl. This framework enables one to make a bot that can play nethack. I'm interested in TAEB because I want to try to reduce its runtime footprint and experiment with bots that use search based AI and rules based AI. In the recent 2009 Mario AI contest, preliminary results showed that A* search implementations were beating the rules based implementations. I found this suprising because I thought rules based AI would be more flexible and faster.

I intend to do these same types of experiments with TAEB.

First step was to get nethack installed. nethack was available at www.nethack.org. OpenSUSE 11.1's current repositories alos had nethack available, so I just installed it with the OpenSUSE software installer.

I downloaded the latest TAEB using git. git clone git://github.com/sartak/TAEB.git

I discovered that if I attempted to install TAEB using sudo: sudo perl Makefile.PL
For some reason, the Perl function can_run can't find my installation of nethack. This was odd. It was necessary to create a symbolic link /usr/bin/nethack to /usr/games/nethack.

After making the symbolic link to the nethack executable, there were a number of warnings about additional Perl modules not found on the system:




I went back to eliminate each of these warnings. Unless specified otherwise, I used the cpan executable to eliminate these warnings.

Note, I found that Yaml::Scyk had to be installed but Makefile.PL did not detect and list that it was needed; When I ran taeb the first time, it complaed that it couldn't find the file YAML/Syck.pm.

After YAML::Syck was installed, I rebuilt and reinstalled. Seems to be working!

Naive Bayes Classifiers in SpamAssassin

Spam classifiers like SpamAssassin are broadly used to split email into ham and spam. How well would SpamAssassin's nbc perform if there were more than 2 categories. I have an idea for applying nbc of a spam filter for sorting emails which will be split into more than 2 categories.

As a start of this investigation, I've decided to start with some OSS Naive Bayes Classifer based spam filters. I'm starting with SpamAssassin. For the purposes of this experiment, I will be using SpamAssassin as a command-line tool.

spamassassin is a Perl front-end that one uses to classify an email, which is in a text file. One email per file.

sa-learn is a tool in the SpamAssassin suite that trains the nbc.
sa-learn --ham /path/to/directory/containing/ham loads the nbc with ham.
sa-leanr --spam /path/to/directory/containing/spam loads the nbc with spam.

I've only acquired a corpus of ham and spam of a few thousand emails. For what I need, I would like to have a corpus of up to a million documents which could be split into about 9 categories. I'm looking for a large corpus.

I've also noted with nbc's that process text, I've notice that there appears to be no restriction on the email size. In comparison to nbc's used with images, it is required that the images in the image corpus all be the same size. I wonder if this is really necessary. I will have to check that out with the face recognizer work currently in progress in my OpenCV project.

Naive Bayes for text classification

Another article on Naive Bayes classification aka nbc? Why does the web need another article on this. Yes, there are many. Just use your favorite browser and search on "naive bayes". You'll get hundreds of url's. However, once you start digging deep into them, you will find 2 annoying trends. They are just reprinting the Naive Bayes definition. No specifics on how the Naive Bayes definition was used to classify a type of data.



Originally, I had leared about Naive Bayes in my AI class, which was taught by Dr. Christopher Brooks, at University of San Francisco back in 2005. Since then, the computer that held my information died and I've been unable to retrieve my nbc program. After a long time, I finally found good info on Naive Bayes and how to use it toward classifying text. First, I'll give the basic definitions of Bayes and Naive Bayes classification, which I'm simply resummarizing from the text book "Artificial Intelligence" by Russel and Norvig. Then, I'll talk about specifically applying Naive Bayes for text classification; This information I found in slides for a course called Comp221. The slides were written by the courses TA, Zhang Kai.

Naive Bayes

Like the regular Bayes algorithm, Naive Bayes simplifies the calculation of a conditional probability. It further simplifies the calculation of conditional probabilities by assuming that the effects are independent. Even though this may not be actually true, it has been found that this assumption yields acceptable behavior.


P(Class|Effects) = P(Class) * P(Effect1|Class).....*P(Effectn|Class)


Supervised Naive Bayes for Text Classification

The definition of Naive Bayes is easy to understand, but is lacking in any of the details that one must use to make a real application. I will fill in the details here(Thanks Dr. Brooks and Zhang Kai!)

(1) Start with a corpus and calculate P(Ci)
A corpus is a collection of data that will be used to train the Naive Bayes classiifer. This should be large number of items. The total number items should be around 1000. The corpus should be split into different classes, where each class occurs in the percentage one thinks the actual documents occur in real life. Ci is a class in C. P(Ci) = nc / ni, where nc is the number of corpus documents that correspond to Ci. ni is the total number of documents in the corpus.

(2)For each class, calculate the P(word|class)
For each class, there will be a collection of words that are associated with that class. One must calculate the probablity that a given word will occur in a particular class.

ni = number of total words in documents in Ci
wi = word associated with Ci
wij = number of times wi occurs in all Ci documents
P(wi|Ci) = wij / ni

For each class, if a word only occurs in the class Ci, this is considered a conditional probability of 'zero'. For an conditional probability that is a 'zero', assign it a value eta/ni, where eta/ni is some tunable value.

For each class, choose the top word frequencies as the words used to classify a document. Ideally, each word would occur in all Ci in C.

(3) After (1) and (2) have been performed, the nbc has been trained. d is a new document that is unclassified.

Take document d and find all the words that occur in the training corpus.

For each Ci, calculate P(Ci|Effects); For each word, wi, calculate the P(wi|Ci) wrt to the document d.

The largest P(Ci|Effects) is the matched class Ci for document d.

Training OpenCV for facial detection

I've been working on creating a training database of faces for OpenCV. I've acquired over 8 GB of faces. But, things are going slowly. I've been using GIMP to view the faces and determine the ROI, but its a slow process. Also, GIMP doesn't let me automatically specify a ROI. I can get the TL corner of a ROI but I have to manually post-process to get the width and length. My current method is slow, tedious, and error prone.

I'm considering making my own tool which can ouput a input database in a format that OpenCV can use. I'm going to try to do this in GIMP. But, if I can't, then I'll do something with PIL or HighGui in OpenCV.

Generating random numbers

handy if you need to generate random numbers:

od -An -N4 -l /dev/random
od -An -N4 -l /dev/urandom

Converting the Yale database to a format that OpenCV 1.1.1 can read

The Yale face database, http://cvc.yale.edu/projects/yalefaces/yalefaces.html, is a collection of 165 faces. This is a useful source of data for training a face recognizer. I know that OpenCV already comes with trained trainers, but where is the fun in that? My trying to own face recognizer myself, I'll find out about some missing steps.

For example, the Yale Face Database files are all GIF's. OpenCV 1.1.1 does not read GIF's. They must be converted to some format that OpenCV can read. I chose to use png. but what is the easy way to convert these gif's to png's?

I used the Python Image Library and some 1-liner bash scripting magic to convert all gif's to png's.

First, I renamed all the database files to have a suffix ".gif"; The files in the Yale database don't have a suffix. I ran tests and perhaps this was not necessary but I didn't want to run into a problem with one of the files. This is the conversion I used in the directory that contains all the Yale database files:

for i in `lsxargs`;do pilconvert.py $i $i.png; done

If you see the string "lsxargs" above, this is wrong. There is a vertical bar between ls and xargs. I don't know why Blogger doesn't allow the pipe symbol.

pilconvert.py is a script that comes with the Python Image Library.

Now, if only the task of creating the rectangle for the ROI, region of interest, was just as easy.