 |
AppletTalk.com
Java discussions newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Quick Function
Guest
|
 Posted: Wed Nov 24, 2004 11:29 pm Post subject: memory problem with StreamTokenizer |
 |
|
Hi, I am using StreamTokenizer to parser a file and it seems that
there is a memory problem with the class. Here is my code:
BufferedReader br = new BufferedReader(new FileReader(filename));
StreamTokenizer st = new StreamTokenizer(br);
//other stuff, configure tokenizer...
while ((tt = st.nextToken()) != StreamTokenizer.TT_EOF) {
if (tt == st.TT_WORD) {
saveTokenInMyStructure(st.sval);
}
}
br.close();
The file is 5M, but this piece of code takes 50M memory and does not
go away when it is out of scope. If I replace
saveTokenInMyStructure(st.sval) with saveTokenInMyStructure("Hello")
where let's assume that "Hello" is longer than any token in the file,
there is not memory problem. Why?
Thanks,
QQ
|
|
| Back to top |
|
 |
ThomasH
Guest
|
| Posted: Thu Nov 25, 2004 2:20 am Post subject: Re: memory problem with StreamTokenizer |
|
|
Quick Function wrote:
| Quote: |
Hi, I am using StreamTokenizer to parser a file and it seems that
there is a memory problem with the class. Here is my code:
BufferedReader br = new BufferedReader(new FileReader(filename));
StreamTokenizer st = new StreamTokenizer(br);
|
Same with the StringTokenizer, which must be made for each
line anew. My recommendation is: Write your own tokenizer on
array of char[] or byte[] or when on StringBuffer and forget
the trouble. I am not sure of StreamTokenizer, the StringTokenizer
is badly designed, that I have to write a small finite state
machine with hasMoreTokens() and getToken(), thus if you will
do well, your own scanner might even use less bytecode! Whoever
made these Java classes, has never before written a compiler...
Thomas
| Quote: |
//other stuff, configure tokenizer...
while ((tt = st.nextToken()) != StreamTokenizer.TT_EOF) {
if (tt == st.TT_WORD) {
saveTokenInMyStructure(st.sval);
}
}
br.close();
The file is 5M, but this piece of code takes 50M memory and does not
go away when it is out of scope. If I replace
saveTokenInMyStructure(st.sval) with saveTokenInMyStructure("Hello")
where let's assume that "Hello" is longer than any token in the file,
there is not memory problem. Why?
Thanks,
QQ
|
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
