#47514

开云体育

I notice that the IBKR Gateway will quickly run out of memory if left at the default Memory allocation setting of 768 MB.? I only notice this when I pull a lot of option chains such as 3 expirations, all strikes. I can usually get the Gateway to freeze up completely after about 5 to 7 symbols.? When I say freeze up, I mean that there is no response to my last request.? I can close my app, start a new one and it's still stuck.? If I increase the memory allocation to 26 GB, I can pull 500 symbols without running out of memory, but the memory used by the application steadily increases apparently without bound.? I only noticed this with Gateway after version 978 although it's possible that earlier versions were doing it too and I just did not notice because I already set the memory allocation very high.? Does anyone else notice this?? Is the application notorious for leaking memory?? Is there some call that I am supposed to do to free memory?? I assumed that after it passes the message to the app, then it simply deletes the message.? Perhaps not...

Matthias Frener

#47516

Had a similar issue on my App when starting up the watchlists (I request a whole bunch of market data within a short period of time).
What I have noticed, is that there is strong relation between memory consumption on how many request you do within a certain amount of time. Not sure yet if really leaks memory (always difficult to say from outside what GC actually does), or if Gateway process is just busy to do garbage collection and so you pile up memory until there is now more.
Don't know if this solution is any helpful for you, but I have solved it by further decreasing rate limit.
I use the Typescript Api () where we have a rate-limiter on the send-thread to enforce the max 50 request/second limit. If have further reduced this to 25req/s and increased Gateway Memory limit to 1GB. Works fine since then. Maybe try to stretch out the requests over time if possible - don't try to get all all strikes on 3 dates simulations, but do 1 by 1, trying to keep as pressure to Gateway as low as possible.

#47518

开云体育

Since the IB Gateway is Java-based, it seems that the only way it would accumulate a lot of memory would be if the messages that it sends are not immediately released after being received by the reader.? I suspect that they optimized the gateway for responsiveness and defer garbage collection to prioritize responsiveness.? Since I occasionally use the Gateway to pull strikes and expirations once every week or two on a list of about 500 stocks and ETFs, it stays busy for a while...but still, it shouldn't take that long to do garbage collection and seems that they could use a parallel garbage collector.? In any case, I worked around my issue for now by just giving it a lot of memory.? Just curious if there was an additional call I was supposed to make to cause the gateway to release messages / free up memory.? I think TWS and Gateway offer a lot of good features.? Definitely think Java was the right language choice in order to make it cross-platform.? But as a Java developer, it just feels a lot slower and bloated than it should be.? I think the codebase could be cleaned up a lot and made to run a lot faster and more efficient.

Show quoted text

From: [email protected] <[email protected]> on behalf of Matthias Frener <matthias.frener@...>
Sent: Wednesday, June 23, 2021 1:57 AM
To: [email protected] <[email protected]>
Subject: Re: [TWS API] IB Gateway Memory Leak?

?

Had a similar issue on my App when starting up the watchlists (I request a whole bunch of market data within a short period of time).
What I have noticed, is that there is strong relation between memory consumption on how many request you do within a certain amount of time. Not sure yet if really leaks memory (always difficult to say from outside what GC actually does), or if Gateway process is just busy to do garbage collection and so you pile up memory until there is now more.
Don't know if this solution is any helpful for you, but I have solved it by further decreasing rate limit.
I use the Typescript Api () where we have a rate-limiter on the send-thread to enforce the max 50 request/second limit. If have further reduced this to 25req/s and increased Gateway Memory limit to 1GB. Works fine since then. Maybe try to stretch out the requests over time if possible - don't try to get all all strikes on 3 dates simulations, but do 1 by 1, trying to keep as pressure to Gateway as low as possible.

#47520

This sounds like a real bug. Have you filed for one?

The good solution is that IBKR fixes this.

Another workaround may be running one gateway until it fills memory and then run a second gateway using a second username and turning off the first one. Repeat the cycle every time memory fills alternating. This is not a very good solution because second user / market subscriptions cost more AND brings complexity to your program (i.e. pick up where left off from first gateway). But I think its automate able.

- Bruce

Show quoted text

On Wed, Jun 23, 2021, 10:43 AM Crow <aaroncook394@...> wrote:

Since the IB Gateway is Java-based, it seems that the only way it would accumulate a lot of memory would be if the messages that it sends are not immediately released after being received by the reader.? I suspect that they optimized the gateway for responsiveness and defer garbage collection to prioritize responsiveness.? Since I occasionally use the Gateway to pull strikes and expirations once every week or two on a list of about 500 stocks and ETFs, it stays busy for a while...but still, it shouldn't take that long to do garbage collection and seems that they could use a parallel garbage collector.? In any case, I worked around my issue for now by just giving it a lot of memory.? Just curious if there was an additional call I was supposed to make to cause the gateway to release messages / free up memory.? I think TWS and Gateway offer a lot of good features.? Definitely think Java was the right language choice in order to make it cross-platform.? But as a Java developer, it just feels a lot slower and bloated than it should be.? I think the codebase could be cleaned up a lot and made to run a lot faster and more efficient.

From: [email protected] <[email protected]> on behalf of Matthias Frener <matthias.frener@...>
Sent: Wednesday, June 23, 2021 1:57 AM
To: [email protected] <[email protected]>
Subject: Re: [TWS API] IB Gateway Memory Leak?
?

Had a similar issue on my App when starting up the watchlists (I request a whole bunch of market data within a short period of time).
What I have noticed, is that there is strong relation between memory consumption on how many request you do within a certain amount of time. Not sure yet if really leaks memory (always difficult to say from outside what GC actually does), or if Gateway process is just busy to do garbage collection and so you pile up memory until there is now more.
Don't know if this solution is any helpful for you, but I have solved it by further decreasing rate limit.
I use the Typescript Api () where we have a rate-limiter on the send-thread to enforce the max 50 request/second limit. If have further reduced this to 25req/s and increased Gateway Memory limit to 1GB. Works fine since then. Maybe try to stretch out the requests over time if possible - don't try to get all all strikes on 3 dates simulations, but do 1 by 1, trying to keep as pressure to Gateway as low as possible.

#47525

Been a long while since my java days ...

But you did ask about some possible things ...

AFAIK all of the JVM suppliers offer a tool or a procedure (send a POSIX signal to the JVM process)? to force a gc collection as well as a heap dump, event attaching an external monitor to a running? JVM etc.

Doing it low level and a bit raw (there are free and vendor supplied higher level tools for all this):

Cause JVM to allocate a bunch of memory for a bit of time, then stop.
Force a full GC.? (e.g. if using oracle jvm, via their jcmd tool etc.)
Signal a "Heap Dump".
Cause JVM to allocate a bunch of? memory over some time interval (say 5 mins) then stop.
Force a full GC.
Signal a "Heap Dump"
Use vendor or other tool to analyze the Heap Dump.? Unless things changed even a grep | sort | count can be effective here.

You should clearly see some (it is usually one thing) run away count of object instances of some Class causing the leak.? It is rarely the class itself that is the issue, it is its use as an element of some collection container which makes it reachable thus? preventing gc'ing.

For an actionable bug report just hitting memory allocation hard for some duration to a "steady state", force gc, dump gc stats, hitting allocation hard again for 5-10 mins, force gc, dump gc stats should be sufficient.

Show quoted text

On Wed, Jun 23, 2021 at 1:46 AM Crow <aaroncook394@...> wrote:

I notice that the IBKR Gateway will quickly run out of memory if left at the default Memory allocation setting of 768 MB.? I only notice this when I pull a lot of option chains such as 3 expirations, all strikes. I can usually get the Gateway to freeze up completely after about 5 to 7 symbols.? When I say freeze up, I mean that there is no response to my last request.? I can close my app, start a new one and it's still stuck.? If I increase the memory allocation to 26 GB, I can pull 500 symbols without running out of memory, but the memory used by the application steadily increases apparently without bound.? I only noticed this with Gateway after version 978 although it's possible that earlier versions were doing it too and I just did not notice because I already set the memory allocation very high.? Does anyone else notice this?? Is the application notorious for leaking memory?? Is there some call that I am supposed to do to free memory?? I assumed that after it passes the message to the app, then it simply deletes the message.? Perhaps not...

#47526

开云体育

When I have more free time, I'll try that.? For now, my primary use for the IB Gateway is to download historical data and occasionally place orders.? So simply extending the memory limit fixes it for my use case.

Show quoted text

From: [email protected] <[email protected]> on behalf of Ray Racine <ray.racine@...>
Sent: Thursday, June 24, 2021 9:16 AM
To: [email protected] <[email protected]>
Subject: Re: [TWS API] IB Gateway Memory Leak?

?

Been a long while since my java days ...

But you did ask about some possible things ...

AFAIK all of the JVM suppliers offer a tool or a procedure (send a POSIX signal to the JVM process)? to force a gc collection as well as a heap dump, event attaching an external monitor to a running? JVM etc.

Doing it low level and a bit raw (there are free and vendor supplied higher level tools for all this):

Cause JVM to allocate a bunch of memory for a bit of time, then stop.
Force a full GC.? (e.g. if using oracle jvm, via their jcmd tool etc.)
Signal a "Heap Dump".
Cause JVM to allocate a bunch of? memory over some time interval (say 5 mins) then stop.
Force a full GC.
Signal a "Heap Dump"
Use vendor or other tool to analyze the Heap Dump.? Unless things changed even a grep | sort | count can be effective here.

You should clearly see some (it is usually one thing) run away count of object instances of some Class causing the leak.? It is rarely the class itself that is the issue, it is its use as an element of some collection container which makes it reachable thus? preventing gc'ing.

For an actionable bug report just hitting memory allocation hard for some duration to a "steady state", force gc, dump gc stats, hitting allocation hard again for 5-10 mins, force gc, dump gc stats should be sufficient.

On Wed, Jun 23, 2021 at 1:46 AM Crow <aaroncook394@...> wrote:

I notice that the IBKR Gateway will quickly run out of memory if left at the default Memory allocation setting of 768 MB.? I only notice this when I pull a lot of option chains such as 3 expirations, all strikes. I can usually get the Gateway to freeze up completely after about 5 to 7 symbols.? When I say freeze up, I mean that there is no response to my last request.? I can close my app, start a new one and it's still stuck.? If I increase the memory allocation to 26 GB, I can pull 500 symbols without running out of memory, but the memory used by the application steadily increases apparently without bound.? I only noticed this with Gateway after version 978 although it's possible that earlier versions were doing it too and I just did not notice because I already set the memory allocation very high.? Does anyone else notice this?? Is the application notorious for leaking memory?? Is there some call that I am supposed to do to free memory?? I assumed that after it passes the message to the app, then it simply deletes the message.? Perhaps not...

YSS

#47527

Guys, you need 2GB to run it effectively.

The heap needs 2GB by default.

Thanks

#48362

开云体育

I finally found some time to look into this.? Attached is a simple Java class that reproduces the issue.? It's around 700 lines of code, but most of that code is unimplemented required overloads.? When I started the program, IB Gateway was consuming 700 MB and after running the code (which took around 30 mins), the process was consuming over 4 GB.? The leak seems to be coming from request contract details (for option contracts).? Or at least that is the call that most contributes to the memory leak for my use case.? Importantly, IB Gateway still consumed 4 GB even after my attached puller program was closed.? I used JConsole to request several garbage collections on the IB Gateway, but that did not free up any memory.? If I leave the IB Gateway at the default max memory limit (768 MB I think), it will freeze up after only a few symbols.? My current work around is to use a very large memory limit and I have enough system memory to do that.? Interestingly, while I was looking at the IB Gateway using JConsole, I noticed that it had over 17,000 classes loaded.? That gives me some clue as to why it takes TWS so long to load...

I submitted a ticket with IB, but their uploader did not allow me to upload the attached source code.? So hopefully they will follow-up.? If course, you find find something in my code that I am doing wrong, please let?me know.? However, I don't see how my program could cause a memory leak in their separately-executing process and why it would still persist even after my program terminates.

Show quoted text

From: [email protected] <[email protected]> on behalf of Ray Racine <ray.racine@...>
Sent: Thursday, June 24, 2021 9:16 AM
To: [email protected] <[email protected]>
Subject: Re: [TWS API] IB Gateway Memory Leak?

?

Been a long while since my java days ...

But you did ask about some possible things ...

AFAIK all of the JVM suppliers offer a tool or a procedure (send a POSIX signal to the JVM process)? to force a gc collection as well as a heap dump, event attaching an external monitor to a running? JVM etc.

Doing it low level and a bit raw (there are free and vendor supplied higher level tools for all this):

Cause JVM to allocate a bunch of memory for a bit of time, then stop.
Force a full GC.? (e.g. if using oracle jvm, via their jcmd tool etc.)
Signal a "Heap Dump".
Cause JVM to allocate a bunch of? memory over some time interval (say 5 mins) then stop.
Force a full GC.
Signal a "Heap Dump"
Use vendor or other tool to analyze the Heap Dump.? Unless things changed even a grep | sort | count can be effective here.

You should clearly see some (it is usually one thing) run away count of object instances of some Class causing the leak.? It is rarely the class itself that is the issue, it is its use as an element of some collection container which makes it reachable thus? preventing gc'ing.

For an actionable bug report just hitting memory allocation hard for some duration to a "steady state", force gc, dump gc stats, hitting allocation hard again for 5-10 mins, force gc, dump gc stats should be sufficient.

On Wed, Jun 23, 2021 at 1:46 AM Crow <aaroncook394@...> wrote:

I notice that the IBKR Gateway will quickly run out of memory if left at the default Memory allocation setting of 768 MB.? I only notice this when I pull a lot of option chains such as 3 expirations, all strikes. I can usually get the Gateway to freeze up completely after about 5 to 7 symbols.? When I say freeze up, I mean that there is no response to my last request.? I can close my app, start a new one and it's still stuck.? If I increase the memory allocation to 26 GB, I can pull 500 symbols without running out of memory, but the memory used by the application steadily increases apparently without bound.? I only noticed this with Gateway after version 978 although it's possible that earlier versions were doing it too and I just did not notice because I already set the memory allocation very high.? Does anyone else notice this?? Is the application notorious for leaking memory?? Is there some call that I am supposed to do to free memory?? I assumed that after it passes the message to the app, then it simply deletes the message.? Perhaps not...

ContractPullerMemLeakTest.txt