ஆடுகளம் – 2020

Tamil projects for 2019-2020

Over the course of this year, since translating Ruby Kin, and preparing a summary of 3 years work on spell-checker for Tamil Internet Conference – 2019, I’ve been thinking of next level of interesting projects.

The following have come to mind, expressed in Twitter @ezhillang in various forms. Here they are in simply chronological order,

  1. Translating “Data Structures and Algorithms” book in Tamil
  2. Translating/Writing a “Debugging Techniques” book in Tamil: ‘கணினி செயல்முறை நிரகளில் வழுநீக்கம்‘ – பயிற்சி, நூல்
    • Debugging techniques are important learning milestone for any professional software/hardware developer which are usually learnt on the job and essentially skipped in academia (perhaps for practical purposes).
  3. (Research/Proof-of-concept) Viterbi algorithm based spelling correction algorithm for Tamil
  4. (Research/Proof-of-concept) Concordance based context ambiguity resolution for Tamil spelling correction.

Contingent on our levels and degrees of success we can share our work in forums like Tamil Internet Conference, ACL or ACM, etc.

நிவாடா மாகனத்தில் மலையேரும் சமயம் மொட்டை வெயிலில் எடுத்த தம்படம் 🙂

As always collaborators are welcome: email: ezhillang -AT- gmail -DOT- com

வாங்க பழகலாம் – பங்களிக்கலாம்

Contributing to open-tamil : பங்களிக்கலாம்

You may have heard of open-tamil, the free Tamil text processing tools library for Python 2 and 3. If you want to join the project please fo

success_Arun_Vekataswamy_8184568429

llow the  directions.

  1. Create a github account from http://www.github.com, and send me your handle via email (ezhillang -AT- gmail.com)
  2. Learn about git – version control system (use Google if you don’t follow anything)
  3. Clone the repository from instructions at https://github.com/arcturusannamalai/open-tamil
  4. You should be all setup now; cd to the repository and try to install open-tamil locally in your Python setup
  5. Run the command, ‘python setup.py build’ first
  6. Upon success run the command, ‘python setup.py install’
  7. You may have to use sudo permissions in Linux if you are not using virtualenv

To use open-tamil, and understand the functions you may read the docs from blog post, and example code,

  1. Blog post on open-tamil (most important functions) : https://ezhillang.wordpress.com/2014/01/26/open-tamil-text-processing-%E0%AE%89%E0%AE%B0%E0%AF%88-%E0%AE%AA%E0%AE%95%E0%AF%81%E0%AE%AA%E0%AF%8D%E0%AE%AA%E0%AE%BE%E0%AE%AF%E0%AF%8D%E0%AE%B5%E0%AF%81/
  2. Example code using open-tamil package – try example code in directory called examples under the root of repo : https://github.com/arcturusannamalai/open-tamil/tree/master/examples

Hopefully you can get started on open-tamil with these resources. If not always leave a comment, drop an email @gmail.com for ezhillang, or tweet to me @ezhillang.

Anbudan,

-Muthu

Open-Tamil based Tamil Wikipedia, Wiktionary data analysis

Introduction

I have been keen to understand what kind of articles are read, and written, in Tamil Wikipedia, and Wiktionary. So it was time to use data analysis and some programming – so thats what I did! Last weekend I tried to take Open-Tamil Python library for a spin with the dumps of Wikipedia data for Tamil wiki and here are results.

You can find my actual program here, solpattiyal.py

Pre-requisites

  1. Install Python 2.7 or Python 3 – whichever flavors you want from http://www.python.org
  2. Get open-tamil library v0.40 from Python Package Index
  3. If you have pip installed in your system just type,

    $ pip install –upgrade open-tamil

  4. Get Wikipedia Tamil dumps from Wikipedia servers
  5. Download the file solpattiyal.py from the above link, or get whole of open-tamil from github.

Program Usage

  1. For small text dumps in kB sizes you can see output on terminal,

    $ python solpattiyal.py <filename1>

  2. You can also use multiple files input

    $ python solpattiyal.py  <filename1> <filename2> …

  3. Then you may want to use output redirection like,

    $ python solpattiyal.py  demo_file1.xml demo_file2.xml > output

Analysis of Code

  1. The code in solpattiyal is fairly simple and uses an algorithm to parse out Tamil letters from each file
  2. We group letters into words via static method ‘WordFrequency.get_tamil_words‘; (this method will make it into next version of open-tamil itself, after this demo)
  3. We insert each Tamil word into the dictionary and bump up its frequency by 1
  4. Finally we use the sorted() method in Python with the comparator key to print list by frequency, and again we print it by sorted order.
  5. Code is written in particular way to straddle both Python 2.7 and Python 3.
  6. Code is written to handle multiple files – usually Wikipedia files are large, and I like to use GNU split utility like this (to split at every 300,000 lines of text),

    $ split -l 300000 <filename>

Data

  1. DISCLAIMER :
    1. This analysis is not a criticism of Tamil Wikipedia.
    2. I am a Tamil Wikipedia contributor in last several years, and Wikipedian circa 2005.
    3. This data analysis is not complete/comprehensive – feel free to point out details
  2. Sample data from my analysis of recent Wikipedia title dump file yielded some interesting data on Tamil wikipedia article distribution.
  3. wikipedia-stats
  4. The data file can be found in common-words-ta-wikipedia-data-March-16-2015.
  5. My recommendations are
    1. Every Tamil speaking specialist can being stub articles or add information to broaden other articles in their fields
    2. You can think of contributing 1 article every month!!
    3. Consider broadening Tamil conversations beyond here-and-now, to world of science, math, medicine, engineering, arts and philosophy

Feedback

  1. Please send your comments and questions always to me at ezhillang in gmail, or via Twitter @ezhillang
  2. Feel free to improve on this code, and send a pull request in github.