Gathering Languages from the Internet
Linguistics Colloquium Features Ed Jahn
Thursday, October 21, 2010 3:20 PM to 5:30 PM EDT
Robinson Hall A, Room 447 (English Department Conference Room)
In that amazing resource, the Internet, can be found languages from Arabic to Zulu- documents written in the languages, as well as dictionaries and descriptions- a gold mine for language learners and linguists. And it can be mined automatically!
An Crúbadán is a computer system that does just that. It collects web pages that are written in numerous languages, in order to make them available for study. Its name, which is Irish Gaelic, means "The Web Crawler". Its author, Kevin Scannell (head of the computer science department at St. Louis University in Missouri) is a native speaker of Irish Gaelic, who started it as a tool to support his own language. Working with a number of collaborators, he has since extended it to over 400 languages, many of which have minority and endangered status. By helping develop computer tools for minority languages, this can encourage younger speakers to use their languages - essential for keeping them alive in the modern world. By gathering real-world texts, it provides important material for linguistic study of these languages
Ed Jahn became a collaborator in late 2009, and was joined by Tel Monks a few months later. Jahn first concentrated on providing ways to share the information that An Crúbadán has collected, using XML to do so. Currently, Jahn and Monks are working on ways to make the collection more usable, by separating foreign from native vocabulary.
This talk will use An Crúbadán as an example to discuss:
- How web pages can be downloaded automatically
- How acented letters and non-Latin alphabets are represented in web documents
- How donwloadable files can be built for sharing with other researchers
- Problems to be solved in making use of collected documents
This should be of interest to anyone who would like to use the Internet as a resource for linguistic research, It does not require advanced computer skills.
Sponsored by the George Mason University Linguistics Program.
