Month: March 2015

Introduction

I have been keen to understand what kind of articles are read, and written, in Tamil Wikipedia, and Wiktionary. So it was time to use data analysis and some programming – so thats what I did! Last weekend I tried to take Open-Tamil Python library for a spin with the dumps of Wikipedia data for Tamil wiki and here are results.

You can find my actual program here, solpattiyal.py

Pre-requisites

1. Install Python 2.7 or Python 3 – whichever flavors you want from http://www.python.org
2. Get open-tamil library v0.40 from Python Package Index
3. If you have pip installed in your system just type,
   

   $ pip install –upgrade open-tamil

4. Get Wikipedia Tamil dumps from Wikipedia servers
5. Download the file solpattiyal.py from the above link, or get whole of open-tamil from github.

Program Usage

1. For small text dumps in kB sizes you can see output on terminal,
   

   $ python solpattiyal.py <filename1>

2. You can also use multiple files input
   

   $ python solpattiyal.py  <filename1> <filename2> …

3. Then you may want to use output redirection like,
   

   $ python solpattiyal.py  demo_file1.xml demo_file2.xml > output

Analysis of Code

1. The code in solpattiyal is fairly simple and uses an algorithm to parse out Tamil letters from each file
2. We group letters into words via static method ‘WordFrequency.get_tamil_words‘; (this method will make it into next version of open-tamil itself, after this demo)
3. We insert each Tamil word into the dictionary and bump up its frequency by 1
4. Finally we use the sorted() method in Python with the comparator key to print list by frequency, and again we print it by sorted order.
5. Code is written in particular way to straddle both Python 2.7 and Python 3.
6. Code is written to handle multiple files – usually Wikipedia files are large, and I like to use GNU split utility like this (to split at every 300,000 lines of text),
   

   $ split -l 300000 <filename>

Data

1. DISCLAIMER :
   1. This analysis is not a criticism of Tamil Wikipedia.
   2. I am a Tamil Wikipedia contributor in last several years, and Wikipedian circa 2005.
   3. This data analysis is not complete/comprehensive – feel free to point out details
2. Sample data from my analysis of recent Wikipedia title dump file yielded some interesting data on Tamil wikipedia article distribution.
3. 
4. The data file can be found in common-words-ta-wikipedia-data-March-16-2015.
5. My recommendations are
   1. Every Tamil speaking specialist can being stub articles or add information to broaden other articles in their fields
   2. You can think of contributing 1 article every month!!
   3. Consider broadening Tamil conversations beyond here-and-now, to world of science, math, medicine, engineering, arts and philosophy

Feedback

1. Please send your comments and questions always to me at ezhillang in gmail, or via Twitter @ezhillang
2. Feel free to improve on this code, and send a pull request in github.