Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
Rob clearly has data needs that require a dedicated data service provider, which IBKR is not. They are a brokerage.
But the IBKR data features are adequate for many (maybe even most) users, especially if your strategy focuses on a set of known instruments. We monitor the IBKR real-time data feeds for about 80 instruments of all types. Among them a few very active futures where we receive high resolution tick-by-tick data as well as level II ask/bid order book details. This results in 40 million to 60 million individual data items per day and peaks of 1,500 trades per second.
But as Rob said, IBKR data (especially historical data request) cannot support strategies that need very broad data sets, say the OHLCV for all US stocks, for example, or details for all members of the S&P 500. That is where you will need a dedicated and specialized data service provider. Take a look at the , , and documentation specifically the discussions about request pacing and limits. Chances are that you find what you need.
´³¨¹°ù²µ±ð²Ô
toggle quoted message
Show quoted text
On Fri, Feb 4, 2022 at 11:11 PM, PurpleIce wrote:
Thanks for the info. May you please share what headaches you had in using IBKR's data? Just to be curious, as I also need a data source.
?
|
Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
I just moved over to IBKR a few months ago, and was thinking I could get the same data I had been getting from Polygon, cut that cost, and just use IBKR, I quickly found out I was wrong. Issues with Data limits. Number of concurrent connections. Historical data. Etc. Interactive Brokers, for my needs at least, just isn't set up in a way thats conducive for consuming data. Since Polygon upgraded their systems and APIs I've had zero issues with them. I don't even use IBKRs data for live trading unless it's for futures like DAX which polygon doesn't offer, for everything else, I use polygon streaming websockets data (but they offer REST as well). I can create and destroy API keys if we need to give an outside developer one to work on an issue not in our wheelhouse. I've never run into a connection limit issue. To download 2k tickers, of tick data for a full day, it only takes a few mins. If I want tick data or 2 minute bars or 17 minute bars, the data comes in any form I request it. They have all the other traditional metrics as well. IBKR has been really great for API integration. The one caveat to all this is, you need to be able to develop the programs/scripts you need to consume the polygon data. It's basically the only way to get it. GitHub has some good stuff others made but it made more sense for me to just build things for my usage. Hope that helps.
On Fri, Feb 4, 2022, 11:10 PM Rob Tilley Jr. via <fladventurerob= [email protected]> wrote: The limitation is that interactive brokers is a broker, not a data provider. You can pull data, but it's not worth the headache to me, because everything IBKR does is centered around their core business, which is that of a broker. I use polygon. They aren't expensive. It's a million times better and easier.?
May I know what restrictions or issues the TWS API has that prevents it from being a data provider? Having a dedicated data source is fine - but adds another dependency for an auto trading system...
IB says in their TWS API documentation that the intended use of the API is not as a data provider.? That continues to show itself to be true.?
|
Re: Choose between: Client Portal API & TWS API
Thank you for your kind words, Rob.
There are many reasons for abstraction layers (or wrappers, adapters, controllers, design patterns ...) The ability to hide implementations details and facilitate transparent changes is one of them. But I was thinking more about complexity reduction where you don't want to expose your higher level code to unnecessary detail and necessary API house keeping functions.
An example for an essential API detail that is better hidden in some lower level code are request IDs. You want your trading logic focus on trading decisions and managing request IDs (or having to know exactly how to wire up three order objects to make bracket orders, ...) add unnecessary complexity, distraction, code duplication, and introduce many points of possible failure. So you will probably want to develop over time the kind of abstraction layers that make the API more stable and useful within your application architecture and preferences.
Take a look at the ApiController class in the the Java API, for example. It is a traditional "Controller" class as you would expect it n a "Model - View - Controller" architecture. It operates EClient, EWrapper, and EClientSocket instances for you, bundles related requests and their responses into specialized interfaces and objects, and makes request Ids go away completely.
There is nothing magical about Java and you can practice "do it right and forget about it" in all programming languages.
´³¨¹°ù²µ±ð²Ô
|
Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
Thanks for the info. May you please share what headaches you had in using IBKR's data? Just to be curious, as I also need a data source.
toggle quoted message
Show quoted text
The limitation is that interactive brokers is a broker, not a data provider. You can pull data, but it's not worth the headache to me, because everything IBKR does is centered around their core business, which is that of a broker. I use polygon.
They aren't expensive. It's a million times better and easier.?
May I know what restrictions or issues the TWS API has that prevents it from being a data provider? Having a dedicated data source is fine - but adds another dependency for an auto trading system...
IB says in their TWS API documentation that the intended use of the API is not as a data provider.? That continues to show itself to be true.?
|
Re: Choose between: Client Portal API & TWS API
´³¨¹°ù²µ±ð²Ô,
You always have such great responses. If I see you replied to an email to anyone, I always open it. So thank you for all your valuable?help.
In your reply you discussed abstraction layers. I had a question about that just to clarify. The reason for hiding the details isn't to keep the details hidden per se, but for the purposes of making it possible to modify the implementation without breaking dependent code. Am I correct in my?understanding?
Thanks, Rob
On Fri, Feb 4, 2022, 12:22 PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote: We started a few years back with the TWS API before the Web Portal API was available. We found the TWS API full featured for what we are doing, stable, and with very reasonable latencies (remember, you are talking to a brokerage). We run our clients and TWS instances in virtual Linux instances that is located close to the US east coast IBKR systems (from an IP round-trip latency point of view).
People that come from a traditional blocking request/response API mentality have to get their head around the asynchronous nature of the TWS API. But once you do, it fits very nicely into modern reactive/asynchronous/stream based application frameworks and architectures.
The API implementations IBKR provides for various languages (and that we are discussing here) are very low level. They pretty much provide only serializers/deserializers for the individual API methods at the socket level. While some people feel that is insufficient, we felt (and still feel) that this gives us the freedom to integrate the API in the most effective way for our application architectures. TWS API does not force you to follow a specific application architecture.
But you will want use an abstraction layer to hide the API details from your application and business logic. For example, hide the concept of requestId from your application (they relate API responses to your requests or subscriptions) and never try to synchronize responses with requests by constant or variable length "sleep". Take a look at the ApiController class in the Java API source for a great example of how to do that. It is pretty much a complete MVC "controller" implementation for communicating with the API.
We do find it very beneficial to have TWS running so that we have immediate manual access to the accounts our API clients are managing. There are weeks when we do not touch the GUI but you have the ability for manual interventions as soon as something does not sound right.
We did evaluate the Web Portal API about a year ago, but feel, it is not ready for use yet (at least for us). Here are the highlights:
- The latest and stable versions of the API are two years old, but the portal as well as TWS/IBGW and TWS API have added many features since. The largest of them was support for Crypto instruments late last year.
- From the outside it feels like IBKR's focus is on TWS/IBGW and TWS API. That is understandable since that is where the main user base is. For the foreseeable future, TWS API will most likely be more stable and more frequently be improved than Web Portal API.
- Web Portal API has the same simultaneous account login and market data sharing limitations that TWS/IBGW have and it does not provide any additional features we need.
- Web Portal API comes with a small gateway and your REST requests are actually made locally, not directly to IBKR. Granted, that gateway is much smaller and leaner than TWS/IBGW, but your operation still needs to assure stability and health of the gateway just like you would with TWS/IBGW.
- While we found the methods reasonably well designed and the like you would expect from a modern REST API, Client Portal API will not be able to give you the "loosely coupled" and distributed architecture you might expect from a REST API due to the need of the local gateway
We definitely would love to learn from your experience should you decide to work with Web Client Portal API. And who knows, some of our members may be able to answer questions, too.
´³¨¹°ù²µ±ð²Ô
On Fri, Feb 4, 2022 at 04:40 AM, PurpleIce wrote:
?
I see most people are using TWS API; I wonder why they don't use Client Portal API? Anyone shares a comparison?
?
?
|
Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
The limitation is that interactive brokers is a broker, not a data provider. You can pull data, but it's not worth the headache to me, because everything IBKR does is centered around their core business, which is that of a broker. I use polygon. They aren't expensive. It's a million times better and easier.?
May I know what restrictions or issues the TWS API has that prevents it from being a data provider? Having a dedicated data source is fine - but adds another dependency for an auto trading system...
IB says in their TWS API documentation that the intended use of the API is not as a data provider.? That continues to show itself to be true.?
|
Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
May I know what restrictions or issues the TWS API has that prevents it from being a data provider? Having a dedicated data source is fine - but adds another dependency for an auto trading system...
IB says in their TWS API documentation that the intended use of the API is not as a data provider.? That continues to show itself to be true.?
|
Re: Choose between: Client Portal API & TWS API
Got you, totally agree. In the case of something going wild, I can imagine I'd open up anything available to recue
?
toggle quoted message
Show quoted text
Hard to add much to the answers here but I will agree that seeing my automated trades go through on the GUI is nice.? I trade every market day and will sometimes cancel an order because something unexpected happened.? This is rare but if you are running
automated trading you want backups for your backups.
|
Re: Choose between: Client Portal API & TWS API
Thanks again for the detailed info. It seems Java is the way to go
?
The other thing matters is how many folks in the community (e.g., this email group) use a language - so I can get some help when weird things happen. May I know which language most folks use here? (also
Java?)
And thanks for the python framework and discuss group. I will check it out later.
Another question just pops up: what do most folks use the API for here? Auto trading? Or just get streamed quotes for other services? Is there some group discussing auto/programmed/AI trading?
Regards!
toggle quoted message
Show quoted text
When I said "clients" in my previous posts I was referring to TWS API client applications, not people who hire you. So if, for example. your API client application has placed a few orders but then crashes on you, having TWS instead of IBGW will let you
clean that up very quickly. We also found that having TWS helped with development and debugging.
Let me carefully meander into that programming language question "mine field" and share with you what this looks like from our end:
- TWS, IBGW, and the little gateway for the Client Portal Web API are all written in Java, so that Java is the most natural language for the development team at IBKR. And while the wire protocol for the socket connection is mostly text based, you are talking
to a Java application on the other side.
- C++ and C# implementations are around for many years. My understanding is that there were rough in the beginning, but they have matured and many members of this group run stable applications on-top of those API implementations.
- The IBKR Python API implementation has been added well after the others and you have to see yourself whether you like it or not. There is also an independent 3rd party Python framework called
with a
separate group here. I think Ewald de Wit has done an outstanding job structuring a framework that allows you to make APIi calls with synchronous or asynchronous semantic (or both).
- And some members don't use any of the provided API implementations and manage the messages on the socket directly.
I guess this is a long way of saying any of them will work but you will have to dig in a little to make the one you select work best for you.
Did I hear a click ......... ?
On Fri, Feb 4, 2022 at 01:22 PM, PurpleIce wrote:
I've used C# (at MSFT) and C++ (at GOOG), but I will choose Python for my trading project. Is there any difference for language support?
|
Do you mean ibapi? Well that was an easy fix! Thanks!
|
|
|
Re: Conditions on twsactivex api
Joan ? Well, I did find time for this today after all (change of priorities). And I have to say, I was unable to reproduce your issue. ? I started by amending the demo program I mentioned earlier, to allow a condition to be specified for an order. This then used to set the Conditions property of the Order object in exactly the same way as the VBA in the Excel workbook, using the ParseConditions method of the TWS object. ? Because I don't have a market data subscription for the CT futures, I tried it first buying 100 IBM shares with the following condition, where 461318816 is the conid for ESH2: ? last of bid/ask Price of 461318816(GLOBEX) is >= 4460 ? and it worked perfectly. ? I then tried it using your CT2, like this: ? last of bid/ask Price of 376589407(NYBOT) is >= 1.2111 ? and it still worked! This seems to imply that you can use a contract in a condition for which you don't have market data, which is potentially very useful. (Mind you, the order just sat there waiting for the condition to be satisfied, but it wasn't rejected, so perhaps it would have remained waiting forever.) ? Next I tried it with the condition that gave you the error: ? last Price of 269459887(GLOBEX) is >= 1.15 ? and the order was accepted without problems. ? So clearly the order condition mechanism works fine when the ActiveX API is invoked from C#. So next I tried the same things with the sample Excel workbook, and they all worked fine there too. And I did all this with both API 976 and API 1012. ? I'm satisfied that there isn't actually a bug here, so the question is what caused your failure? Since I can't reproduce the problem, there's really not much more I can do. ? But I suggest you uninstall the API, then install the 9.76 version and start again, just in case something has got mixed up somehow. ? By the way, I notice that the order message information I was looking for from the API log isn't there at all, which seems completely bizarre to me. I always thought all API messages (except market data) were included in the log by default. Perhaps IB thought this is personal information so it shouldn't be in the log, but that seems daft to me, especially as the justification for encrypting the log files ¨C which is also daft ¨C was that they contain potentially personal information. ? Richard
|
Hello Folks,
I am attempting to install abapi for use with the python TWS-API. Running pip install abapi gives the error message
Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement abapi (from versions: none) ERROR: No matching distribution found for abapi
(this is on a intel mac book pro, with python 2.7 and python 3.8.2, pip 22.0.3)
Anybody know what is going on?
thanks,
Eric
|
Re: Choose between: Client Portal API & TWS API
I agree with this and would add that it will take way more time and effort to implement your system with IB Api than it should.
There are numerous gotcha's and general wackiness that you will just have to discover and solve.
In the end you can accomplish pretty much anything you would expect a retail trading environment to do but it won't be a straight line from start to finish.
toggle quoted message
Show quoted text
On 2/4/2022 3:38 PM, ´³¨¹°ù²µ±ð²Ô Reinold via groups.io wrote: I guess this is a long way of saying any of them will work but you will have to dig in a little to make the one you select work best for you.
|
Re: reqHistoricalData() for OTC only works if first typed symbol in TWS
IB says in their TWS API documentation that the intended use of the API is not as a data provider.? That continues to show itself to be true.? I use polygon.io for data and IB for trade executions, but polygon is not yet offering OTC data.? So at this time I'm going to keep it simple and use IB only for trade executions.
|
Re: Choose between: Client Portal API & TWS API
Hard to add much to the answers here but I will agree that seeing my automated trades go through on the GUI is nice.? I trade every market day and will sometimes cancel an order because something unexpected happened.? This is rare but if you are running automated trading you want backups for your backups.
|
Re: Choose between: Client Portal API & TWS API
When I said "clients" in my previous posts I was referring to TWS API client applications, not people who hire you. So if, for example. your API client application has placed a few orders but then crashes on you, having TWS instead of IBGW will let you clean that up very quickly. We also found that having TWS helped with development and debugging. Let me carefully meander into that programming language question "mine field" and share with you what this looks like from our end:
- TWS, IBGW, and the little gateway for the Client Portal Web API are all written in Java, so that Java is the most natural language for the development team at IBKR. And while the wire protocol for the socket connection is mostly text based, you are talking to a Java application on the other side.
- C++ and C# implementations are around for many years. My understanding is that there were rough in the beginning, but they have matured and many members of this group run stable applications on-top of those API implementations.
- The IBKR Python API implementation has been added well after the others and you have to see yourself whether you like it or not. There is also an independent 3rd party Python framework called with a separate group here. I think Ewald de Wit has done an outstanding job structuring a framework that allows you to make APIi calls with synchronous or asynchronous semantic (or both).
- And some members don't use any of the provided API implementations and manage the messages on the socket directly.
I guess this is a long way of saying any of them will work but you will have to dig in a little to make the one you select work best for you. Did I hear a click ......... ?
On Fri, Feb 4, 2022 at 01:22 PM, PurpleIce wrote:
I've used C# (at MSFT) and C++ (at GOOG), but I will choose Python for my trading project. Is there any difference for language support?
|
Re: Choose between: Client Portal API & TWS API
Thank you, Nick.
Checked the API reference provided by Jurgen, so yes, useful things are hidden.
??
toggle quoted message
Show quoted text
IB's documentation is rather random in terms of organization but most of the info you need is there somewhere.
You end up having to read or browse through the entire set of documentation to get a handle on what's available and how things fit together.
On 2/4/2022 1:05 PM, PurpleIce wrote:
RequestMarketData uses ContractID to represent stocks but there is no such a mapping.
|
Re: Choose between: Client Portal API & TWS API
Thanks for such detailed information, again, Jurgen.
I was surprised to learn the conid actually has a look up API - thank you!
So based on your info, I will definitely go with IBGW - I don't have clients so no GUI is needed for others.
I've used C# (at MSFT) and C++ (at GOOG), but I will choose Python for my trading project. Is there any difference for language support? (e.g. one language has better documentation than the other?)
toggle quoted message
Show quoted text
You are welcome, Yu.
The basic objects and concepts are the same between TWS API and the Web Portal API, so if in doubt, look at the TWS API documentation. Generally, you would make contract database lookups to retrieve contract IDs (conIds)? for more "human readable" descriptions
of the instrument(s) you are interested in. Take a look atI and
.
Internally, TWS and IBGW are pretty much identical. We have not done it for the very latest versions, but when we compared that jars and classes for the same TWS and IBGW versions last year, they were virtually identical.
IBGW will most likely use a little less memory, probably has fewer threads, and may use a little less CPU. IIBGW does not have to manage a complex UI with many moving and graphics intensive parts so there are fewer things that can go wrong. You also will have
fewer pop-ups and "helpful" hints.
If you know you won't use the GUI, go with IBGW. Your clients will not see a difference other than the four different default ports {IBGW, TWS} x {Paper, Live}.
Which implementation language are you thinking of?
´³¨¹°ù²µ±ð²Ô
On Fri, Feb 4, 2022 at 12:05 PM, PurpleIce wrote:
A couple follow-ups:
- TWS vs IBGW: any pros & cons? I only know that IBGW is more lightweight.
- Can my client software run on a different machine than the TWS/IBGW (which means using Internet to connect the two)?
|
Re: Choose between: Client Portal API & TWS API
IB's documentation is rather random in terms of organization but
most of the info you need is there somewhere.
You end up having to read or browse through the entire set of
documentation to get a handle on what's available and how things
fit together.
On 2/4/2022 1:05 PM, PurpleIce wrote:
toggle quoted message
Show quoted text
RequestMarketData
uses ContractID to represent stocks but there is no such a
mapping.
|
PurpleIce