 |
AppletTalk.com Java discussions newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
server Guest
|
Posted: Mon Feb 06, 2006 4:09 pm Post subject: How to handle OutOfMemory Exception for large resultset |
|
|
message unavailable |
|
| Back to top |
|
 |
Thomas Kellerer Guest
|
Posted: Mon Feb 06, 2006 4:09 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
On 06.02.2006 12:00 snehalapatil (AT) gmail (DOT) com wrote:
| Quote: | hi,
"Mytable" contains more than 79,000 records
my query is"select * from Mytable"...while processing Resultset I
got OutOfMemory exception.
I tried to solve it using cursors as follows.....
PreparedStatement ps = con.prepareStatement("DECLARE mycursor scroll
CURSOR FOR select * from Mytable",ResultSet.TYPE_SCROLL_INSENSITIVE
,ResultSet.CONCUR_READ_ONLY); ps.execute();
ps = con.prepareStatement("OPEN mycursor");
ps.execute();
But,when I execute "OPEN mycursor" statment I got the error "Cursor
has not been declared"
What is wrong with my code?
|
This depends on your JDBC driver & database system. Oracle usually does
not cache the result set (apart from the number of rows set by
rowprefetch parameter). The Postgres driver caches all rows by default
but can be convinced to not do so....
Thomas
--
It's not a RootKit - it's a Sony |
|
| Back to top |
|
 |
Robert Klemme Guest
|
Posted: Mon Feb 06, 2006 7:39 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
Thomas Kellerer wrote:
| Quote: | On 06.02.2006 12:00 snehalapatil (AT) gmail (DOT) com wrote:
hi,
"Mytable" contains more than 79,000 records
my query is"select * from Mytable"...while processing Resultset
I got OutOfMemory exception.
I tried to solve it using cursors as follows.....
PreparedStatement ps = con.prepareStatement("DECLARE mycursor scroll
CURSOR FOR select * from Mytable",ResultSet.TYPE_SCROLL_INSENSITIVE
,ResultSet.CONCUR_READ_ONLY); ps.execute();
ps = con.prepareStatement("OPEN mycursor");
ps.execute();
But,when I execute "OPEN mycursor" statment I got the error "Cursor
has not been declared"
What is wrong with my code?
This depends on your JDBC driver & database system. Oracle usually
does not cache the result set (apart from the number of rows set by
rowprefetch parameter). The Postgres driver caches all rows by default
but can be convinced to not do so....
|
Also, it's a good idea to select only columns actually needed. Another
option is to increase VM memory settings - the default 64MB is a bit meek
for most real apps but that of course depends.
HTH
robert |
|
| Back to top |
|
 |
Joe Weinstein Guest
|
Posted: Mon Feb 06, 2006 8:36 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
snehalapatil (AT) gmail (DOT) com wrote:
| Quote: | hi,
"Mytable" contains more than 79,000 records
my query is"select * from Mytable"...while processing Resultset I
got OutOfMemory exception.
I tried to solve it using cursors as follows.....
PreparedStatement ps = con.prepareStatement("DECLARE mycursor scroll
CURSOR FOR select * from Mytable",ResultSet.TYPE_SCROLL_INSENSITIVE
,ResultSet.CONCUR_READ_ONLY); ps.execute();
ps = con.prepareStatement("OPEN mycursor");
ps.execute();
But,when I execute "OPEN mycursor" statment I got the error "Cursor
has not been declared"
What is wrong with my code?
|
And to follow Robert's comments, you should only select the *rows* you need.
The major problem with your code is your doing a select of 79,000 rows
in the first place. The number one DBMS performance no-no is shovelling
huge piles of raw data to process it or extract some small bit of info.
Write SQL or stored procedures to process the data where it is in the
DBMS. Only return data you need to display.
Build your saw mill where the trees are. If you want a table in Guam,
you don't get logs by FedEx.
Joe Weinstein at BEA Systems |
|
| Back to top |
|
 |
Guest
|
Posted: Tue Feb 07, 2006 12:19 am Post subject: Re: to get output parameter from the stored procedure |
|
|
Yes, I did try ,
CallableStatement s = c.prepareCall("{call spGetGrid(?,?)}");
s.setString(1,"PRJ-001-A");
s.setClob(2, null);
s.registerOutParameter(2, java.sql.Types.CLOB);
ResultSet r = s.executeQuery();
but now im getting the error message:
Implicit conversion from data type xml to nvarchar(max) is not allowed.
use convert function to run this query.
I am not sure what has to be done to convert this output type (xml) to
clob.
Any help would be greatly appreciated.
Nathan |
|
| Back to top |
|
 |
Green Guest
|
Posted: Tue Feb 07, 2006 12:26 am Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
| What database and driver are you using? |
|
| Back to top |
|
 |
David Harper Guest
|
Posted: Tue Feb 07, 2006 12:28 am Post subject: Re: Need help regarding JDBC |
|
|
Mradul wrote:
| Quote: | I'm using Window 2000 Pro. I don't agree that this could happen due to
a network issue. coz before subscription to broadband service...i was
able to logon to the java application that i created. but after the
subscription, i don't know what happened.
|
I'm guessing that before you subscribed to broadband, your machine
probably did hostname-to-IP address lookup via a static table.
Now, however, your machine has DNS servers allocated dynamically by your
broadband ISP using DHCP, and hostname-to-IP address lookups are
referred to them instead. They don't know which machine the name
"sweetheart" points to, because it's a name that you allocated to a
machine within your own private network, not a name that can be used on
the global Internet.
Open a "Command Prompt" window and type
ping sweetheart
If you see a message which says
Ping request could not find host sweetheart. Please check the name and
try again
then this is indeed your problem.
David Harper
Cambridge, England |
|
| Back to top |
|
 |
snehalapatil@gmail.com Guest
|
Posted: Tue Feb 07, 2006 9:33 am Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
| using sybase database |
|
| Back to top |
|
 |
Green Guest
|
Posted: Tue Feb 07, 2006 9:27 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
Besides other suggestions, you want to look into your JDBC connection
properties. Depending on your driver, there are 2 properties which
typically determine how resultsets are cached and returned to client.
They are:
1. InsensitiveResultSetBufferSize
Determines the amount of memory used by the driver to cache insensitive
result set data.
2. SelectMethod
A hint to the driver that determines whether the driver requests a
database cursor for Select statements. Performance and behavior of the
driver are affected by this property.
Both are optional and the default for SelectMethod does not work well
for large resultsets. Check your driver documentation about this. |
|
| Back to top |
|
 |
joeNOSPAM@BEA.com Guest
|
Posted: Tue Feb 07, 2006 11:33 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
Are you using Sybase's driver (jconn2.jar or jconn3.jar?).
Can you tell us why you're retrieving 79,000 rows?
Sybase gives plenty of capability in their stored procedures
to allow you to avoid that.
Joe |
|
| Back to top |
|
 |
snehalapatil@gmail.com Guest
|
Posted: Wed Feb 08, 2006 9:25 am Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
using jconn2.jar
& have to retrive retrieve 79,000 rows without using stored procedures
-snehal |
|
| Back to top |
|
 |
joeNOSPAM@BEA.com Guest
|
Posted: Wed Feb 08, 2006 8:40 pm Post subject: Re: How to handle OutOfMemory Exception for large resultset |
|
|
If you absolutely have to retrieve 79,000 rows, are you processing them
one-by-one, or do you have to collect them all in an array/list of
some sort?
The only thing you can do is to select only those columns you need,
and id you can process the rows one-by-one, look at the jConnect driver
docs about memory management. You should be able to specify your
result sets to use server-side cursors and read-forward-only so the
driver
itself doesn't have to cache all the data at once. Does the table have
a
unique index? If so, you can write fancier SQL to select the first 1000
rows,
process them and then re-use the memory querying for the next 1000 till
you're done.
If I were you I would at least get some intelligence points by
mentioning
to my higher-ups that whatever processing these 79000 rows need would
be
much better done in stored procedures in the DBMS where the data is.
Good luck...
Joe Weinstein at BEA Systems |
|
| Back to top |
|
 |
Mradul Guest
|
Posted: Thu Feb 09, 2006 12:57 am Post subject: Re: Need help regarding JDBC |
|
|
Hi David,
Thanks for the solution, this could be the case. Coz when i'm pinging
my system, it is showing the IP address assigned to me by my ISP.
Thanks again
Regards,
Mradul Kaushik |
|
| Back to top |
|
 |
David Harper Guest
|
Posted: Thu Feb 09, 2006 1:18 am Post subject: Re: Need help regarding JDBC |
|
|
Mradul wrote:
| Quote: | Hi David,
Thanks for the solution, this could be the case. Coz when i'm pinging
my system, it is showing the IP address assigned to me by my ISP.
|
Your Oracle server may only be listening on IP address 127.0.0.1.
Try changing the hostname in the TNS entry:
ORACLE.SWEETHEART =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = XXXX))
(CONNECT_DATA = (SID = ORCL))
)
David Harper
Cambridge, England |
|
| Back to top |
|
 |
Jan Dieckmann Guest
|
Posted: Thu Feb 09, 2006 8:16 pm Post subject: Re: JDBC wrapper |
|
|
Spring (www.springframework.com) offers excellent support. Also look at
Hibernate. Hibernate actually is an ORM tool.
regards Jan Dieckmann
"GG" <GG (AT) gg (DOT) pl> schrieb im Newsbeitrag
news:dqg6s2$gj0$1 (AT) nemesis (DOT) news.tpi.pl...
| Quote: | Hello.
I'm looking for good JDBC wrapper
who example can:
1.automatic to bind parameters in PreparedStatement,
2.convert resultset to table etc.
I would like know Your opinion about currently solution in this theme.
Where can I find this kind of software.
( I found only SQLExecutor )
Thanks for your help
GG
|
|
|
| Back to top |
|
 |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|