TRAINING TREE ADJOINING GRAMMARS WITH HUGE TEXT CORPUS USING SPARK MAP REDUCE

Tree adjoining grammars (TAGs) are mildly context sensitive formalisms used mainly in modelling natural languages. Usage and research on these psycho linguistic formalisms have been erratic in the past decade, due to its demanding construction and difficulty to parse. However, they represent promisi...

Full description

Saved in:
Bibliographic Details
Published inICTACT journal on soft computing Vol. 5; no. 4; pp. 1021 - 1026
Main Authors Menon, Vijay Krishna, S., Rajendran, K.P., Soman
Format Journal Article
LanguageEnglish
Published ICT Academy of Tamil Nadu 01.07.2015
Subjects
Parsing
Probabilistic Grammar
RDDs
Spark
TAGs
Online AccessGet full text

Cover

More Information
Summary:Tree adjoining grammars (TAGs) are mildly context sensitive formalisms used mainly in modelling natural languages. Usage and research on these psycho linguistic formalisms have been erratic in the past decade, due to its demanding construction and difficulty to parse. However, they represent promising future for formalism based NLP in multilingual scenarios. In this paper we demonstrate basic synchronous Tree adjoining grammar for English-Tamil language pair that can be used readily for machine translation. We have also developed a multithreaded chart parser that gives ambiguous deep structures and a par dependency structure known as TAG derivation. Furthermore we then focus on a model for training this TAG for each language using a large corpus of text through a map reduce frequency count model in spark and estimation of various probabilistic parameters for the grammar trees thereafter; these parameters can be used to perform statistical parsing on the trained grammar.
ISSN:0976-6561
2229-6956
DOI:10.21917/ijsc.2015.0143