lexers / parsers for (un) structured text documents

Posted by wilson32 on Stack Overflow See other posts from Stack Overflow or by wilson32
Published on 2010-01-18T16:57:00Z Indexed on 2010/05/17 6:20 UTC
Read the original article Hit count: 163

Filed under:

lexer

|

parser

|

document

There are lots of parsers and lexers for scripts (i.e. structured computer languages). But I'm looking for one which can break a (almost) non-structured text document into larger sections e.g. chapters, paragraphs, etc.

It's relatively easy for a person to identify them: where the Table of Contents, acknowledgements, or where the main body starts and it is possible to build rule based systems to identify some of these (such as paragraphs).

I don't expect it to be perfect, but does any one know of such a broad 'block based' lexer / parser? Or could you point me in the direction of literature which may help?

© Stack Overflow or respective owner

Related posts about lexer

Error in running script [closed]

as seen on Programmers - Search for 'Programmers'
I'm trying to run heathusf_v1.1.0.tar.gz found here I installed tcsh to make build_heathusf work. But, when I run ./build_heathusf, I get the following (I'm running that on a Fedora Linux system from Terminal): $ ./build_heathusf Compiling programs to build a library of image processing functions… >>> More
problem string recursion antlr lexer token

as seen on Stack Overflow - Search for 'Stack Overflow'
How do I build a token in lexer that can handle recursion inside as this string: ${*anythink*${*anything*}*anythink*} ? thanks >>> More
Lexer antlr3 token problem

as seen on Stack Overflow - Search for 'Stack Overflow'
Can I construct a token ENDPLUS: '+' (options (greedy = false;):.) * '+' ; being considered by the lexer only if it is preceded by a token PREwithout including in ENDPLUS? PRE: '<<' ; Thanks. >>> More
grammar parser lexer antlr letteral

as seen on Stack Overflow - Search for 'Stack Overflow'
What's the difference between this grammar: ... if_statement : 'if' condition 'then' statement 'else' statement 'end_if'; ... and this: ... if_statement : IF condition THEN statement ELSE statement END_IF; ... IF : 'if'; THEN: 'then'; ELSE: 'else'; END_IF: 'end_if'; .... ? If there is any… >>> More
ANTLR lexer mismatches tokens

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple ANTLR grammar, which I have stripped down to its bare essentials to demonstrate this problem I'm having. I am using ANTLRworks 1.3.1. grammar sample; assignment : IDENT ':=' NUM ';' ; IDENT : ('a'..'z')+ ; NUM : ('0'..'9')+ ; WS : (' '|'\n'|'\t'|'\r')+… >>> More

Related posts about parser

Core Data error when assigning variable with one-to-one relationship

as seen on Stack Overflow - Search for 'Stack Overflow'
I tried to assign a managed object (C) with its property another managed object (B) (a one-to-one relationship) in which this other managed object (B) has a to-many relationship with one other managed object (A). There is an error from this assignment in which I copied as follows: #0 0x020e53a7… >>> More
RapidXML - does not compile ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am novice to rapidXML but first impresion was not positive, I made simple Visual Studio 6 C++ Hello World Application and added RapidXML hpp files to project and in main.cpp I put: #include "stdafx.h" #include < iostream > #include < string > #include "rapidxml.hpp" using namespace… >>> More
exception occured in java compiler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am a beginner in Java.I have JDK1.7.0 installed on windows 7 OS.I just wrote a sample java file where the file was not getting compiled and throws the below error. Sam.java:5: ';' expected Sample p = New Sample(); An exception has occurred… >>> More
Coding a parser for a domain specific language in Java

as seen on Stack Overflow - Search for 'Stack Overflow'
We want to design a simple domain specific language for writing test scripts to automatically test a XML-based interface of one of our applications. A sample test would be: Get an input XML file from network shared folder or subversion repository Import the XML file using the interface Check if… >>> More
Doxygen C++ comment string parser in python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Does anybody know of a python module to parse a doxygen style C++ comment string? I mean a string like this (simple example): /** * A constructor. * A more elaborate description of the constructor. * @param param1 test1 * @param param2 test2 */ and I would like to extract the brief… >>> More