Introduction
A simple tokenizer in csharp without using regex or MatchCollections. It accepts single string input. The string can contain multiple tokens or tags to be parsed. The delimiter can be any character. It works by create a new instance and use GetToken to get the first token back, use MoveToNext furhter on. The first GetToken call will automatically position to the beginning of the input string. There is no to call MoveToNext to do the same job! The characters to be bypassed can be added or modified to the char[] BYPASSCODE in unicode, hex, or character formats.
ashwini beditedPosted Nov 26, 2010, 10:14 AMEdited Nov 26, 2010, 10:14 AM
Hello Sir, "San Francisco, officially the City and County of San Francisco" the above text want to tokenize in a way where i should not get 'San' as one token and 'Francisco' as one token...I want 'San Francisco' as whole one token. Please Reply Soon.
kei mPosted Sep 17, 2010, 6:38 AM
and it should not be used by a database .. it should be hard coded they say... :( good day...
kei mPosted Sep 17, 2010, 6:34 AM
sir good day. i want to know how to check the syntax in a lexical parse where in the syntactic analyzer would output if the given line is valid or invalid. for example base from Turbo c: 1 main() 2 { 3 int x 4 print("Hello World"); 5 printf("Hello day!"); 6 } the output for the syntactic analyzer would be in message box or textbox that would display: line 1 valid line 2 valid line 3 invalid line 4 invalid line 5 valid line 6 valid sir hopefully you would help me... thank you.. good day...
Aida Carmina SantosPosted Jun 7, 2010, 5:13 AM
i want to ask how does this work? thank you very much.
Melanie EscherPosted Apr 6, 2006, 6:11 AM
Only the space delimiter works correct