Is there a tokenizer for a cpp file

Posted by AJ on Stack Overflow See other posts from Stack Overflow or by AJ
Published on 2010-04-19T09:06:43Z Indexed on 2010/04/19 9:53 UTC
Read the original article Hit count: 406

Filed under:

c++

|

tokenizer

I have a cpp file with a huge class implementation. Now I have to modify the source file itself.

For this, is there a library/api/tool that will tokenize this file for me and give me one token each time i request.

My requirement is as below.

OpenCPPFile()
While (!EOF)
   token = GetNextToken();
   process something based on this token
EndWhile
I am happy now

Regards, AJ

© Stack Overflow or respective owner

Related posts about c++

C++ : C++ Primer (Stanley Lipmann) or The C++ programming language (special edition)

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a Computer Science degree (long2 time ago) .. I do know Java OOP but i am now trying to pick up C++. I do have C and of course data structure using C or pascal. I have started reading Bjarne Stroustrup book (The C++ Programming Language - Special Edition) but find it extremely difficult esp… >>> More
Which C++ book shold I get between "C++ Primer" vs "C++ Primer Plus"

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to learn C++ by using Vim and MinGW as compiler. I'm interesting at "C++ Primer (4th Edition)" and "C++ Primer Plus (5th Edition)" but I don't know how about it different. It has no book store that I can review those books, so I want to know, what is the different between those book and which… >>> More
Managed c++ std::string not accessible in unmanaged c++

as seen on Stack Overflow - Search for 'Stack Overflow'
In unmanaged c++ dll i have a function which takes constant std::string as argument Prototype : void read ( const std::string &imageSpec_ ) I call this function from managed c++ dll by passing a std::string. When i debug the unmanaged c++ code the parameter imageSpec_ shows the value correctly… >>> More
I need help on my C++ assignment using MS Visual C++

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok, so I don't want you to do my homework for me, but I'm a little lost with this final assignment and need all the help I can get. Learning about programming is tough enough, but doing it online is next to impossible for me... Now, to get to the program, I am going to paste what I have so far. This… >>> More
The Definitive C++ Book Guide and List

as seen on Stack Overflow - Search for 'Stack Overflow'
After more than a few questions about deciding on C++ books I thought we could make a better community wiki version. Providing QUALITY books and an approximate skill level. Maybe we can add a short blurb/description about each book that you have personally read / benefited from. Feel free to debate… >>> More

Related posts about tokenizer

C++ String Toolkit (StrTk) Tokenizer

as seen on Code Project - Search for 'Code Project'
A brief introduction to a tokenizer implementation in C++ >>> More
Tokenizer for full-text

as seen on Stack Overflow - Search for 'Stack Overflow'
This should be an ideal case of not re-inventing the wheel, but so far my search has been in vain. Instead of writing one myself, I would like to use an existing C++ tokenizer. The tokens are to be used in an index for full text searching. Performance is very important, I will parse many gigabytes… >>> More
A string tokenizer in C++ that allows multiple separators

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there a way to tokenize a string in C++ with multiple separators? In C# I would have done: string[] tokens = "adsl, dkks; dk".Split(new [] { ",", " ", ";" }, StringSplitOptions.RemoveEmpty); >>> More
Member initialization while using delegated constructor

as seen on Stack Overflow - Search for 'Stack Overflow'
I've started trying out the C++11 standard and i found this question which describes how to call your ctor from another ctor in the same class to avoid having a init method or the like. Now i'm trying the same thing with code that looks like this: hpp: class Tokenizer { public: Tokenizer(); … >>> More
How to read values from file. tokenizer

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a file in which each line contains two numbers. The problem is that the two number are separated by a space, but the space can be any number of blank spaces. either one, two, or more. I want to read the line and store each of the numbers in a variable, but I'm not sure how to tokenize it. i… >>> More