开云体育

If this sample proves anything, than how not to write a script.

If you are not married to python, try ib-ruby :
ib-extensions has a `eod`method. It fetches eod-data. Other timeframes are possible, too,
ib-examples has several historic-data samples:


to name two.
Recently, the ability to get polars dataframes was added.
enyoy

On Tue, Jan 31, 2023 at 06:49 PM, David Armour wrote:

"intelrdfpmathlib"

Hi David,?
I have installed VCPKG and "intelrdfmathlib" but still getting 2 (3) errors trying to compile to x64.


I have changed configuration to x64 here



Win32 does compile well but i would like to compile it to x64 if possible.

I don't think it's disconnecting gracefully after app.run() is called. I can't find anything in the docs [https://interactivebrokers.github.io/tws-api/connection.html] about disconnecting either. It never breaks from the loop unless I manually quit.

On the upside, I found a workaround that's faster than logging out and logging back in. You can go to settings and turn off/on "Enable ActiveX and Socket Clients."

I'm running a Python script I found on the internet (attached below) to pull some historical data from ib. It works well if I run it once. If I run it twice it doesn't receive anything. If I run it in debug mode I get the following logs (see below). The only line that I find interesting is "socket either closed or broken, disconnecting." Any idea how I can run multiple symbols without logging out and logging back in on TWS?

(base) me$ ./pystuff/download_bars.py --size "1 min" --start-date 20230323 --end-date 20230324 XLF -d
22:16:35,495 root DEBUG args=Namespace(symbol=['XLF'], debug=True, logfile=None, port=7496, size='1 min', duration='1 D', data_type='TRADES', base_directory='data', currency='USD', exchange='SMART', localsymbol='', security_type='STK', start_date=datetime.datetime(2023, 3, 23, 0, 0), end_date=datetime.datetime(2023, 3, 24, 0, 0), max_days=False)
22:16:35,495 ibapi.client DEBUG 140447032183440 connState: None -> 0
22:16:35,495 ibapi.client DEBUG Connecting to 127.0.0.1:7496 w/ id:0
22:16:35,495 ibapi.client DEBUG 140447032183440 connState: 0 -> 1
22:16:35,495 ibapi.client DEBUG msg b'\x00\x00\x00\tv100..157'
22:16:35,495 ibapi.client DEBUG REQUEST b'API\x00\x00\x00\x00\tv100..157'
22:16:35,495 ibapi.connection DEBUG acquiring lock
22:16:35,495 ibapi.connection DEBUG acquired lock
22:16:35,495 ibapi.connection DEBUG releasing lock
22:16:35,495 ibapi.connection DEBUG release lock
22:16:35,495 ibapi.connection DEBUG sendMsg: sent: 17
22:16:35,496 ibapi.decoder DEBUG no fields
22:16:35,497 ibapi.connection DEBUG len 30 raw:b'\x00\x00\x00\x1a157\x0020230324 22:16:34 EST\x00'|
22:16:35,497 ibapi.client DEBUG ANSWER b'\x00\x00\x00\x1a157\x0020230324 22:16:34 EST\x00'
22:16:35,497 ibapi.comm DEBUG read_msg: size: 26
22:16:35,497 ibapi.client DEBUG size:26 msg:b'157\x0020230324 22:16:34 EST\x00' rest:b''|
22:16:35,497 ibapi.client DEBUG fields (b'157', b'20230324 22:16:34 EST')
22:16:35,497 ibapi.client DEBUG ANSWER Version:157 time:b'20230324 22:16:34 EST'
22:16:35,497 ibapi.client DEBUG 140447032183440 connState: 1 -> 2
22:16:35,497 ibapi.reader DEBUG EReader thread started
22:16:35,497 ibapi.client INFO sent startApi
22:16:35,498 ibapi.client INFO REQUEST startApi {}
22:16:35,498 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:35,498 ibapi.client INFO SENDING startApi b'\x00\x00\x00\x0871\x002\x000\x00\x00'
22:16:35,498 ibapi.connection DEBUG acquiring lock
22:16:35,498 ibapi.connection DEBUG acquired lock
22:16:35,498 ibapi.connection DEBUG releasing lock
22:16:35,498 ibapi.connection DEBUG release lock
22:16:35,498 ibapi.connection DEBUG sendMsg: sent: 12
22:16:35,498 root INFO Connected
22:16:35,498 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:35,698 ibapi.client DEBUG queue.get: empty
22:16:35,698 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:35,698 ibapi.client DEBUG conn:1 queue.sz:0
22:16:35,698 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:35,899 ibapi.client DEBUG queue.get: empty
22:16:35,899 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:35,899 ibapi.client DEBUG conn:1 queue.sz:0
22:16:35,899 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: True
22:16:36,15 ibapi.connection DEBUG len 0 raw:b''|
22:16:36,15 ibapi.connection DEBUG socket either closed or broken, disconnecting
22:16:36,15 ibapi.connection DEBUG disconnecting
22:16:36,15 ibapi.connection DEBUG disconnected
22:16:36,15 ibapi.reader DEBUG reader loop, recvd size 0
22:16:36,15 ibapi.reader DEBUG EReader thread finished
22:16:36,99 ibapi.client DEBUG queue.get: empty
22:16:36,100 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: False
22:16:36,100 ibapi.client DEBUG conn:0 queue.sz:0
22:16:36,100 ibapi.client DEBUG 140447032183440 isConn: 2, connConnected: False
22:16:36,100 ibapi.client DEBUG 140447032183440 connState: 2 -> 0
22:16:36,100 ibapi.client INFO disconnecting
22:16:36,100 ibapi.wrapper INFO ANSWER connectionClosed {}
22:16:36,100 ibapi.client DEBUG 140447032183440 connState: None -> 0



















#!/usr/bin/env python

import os
import sys
import argparse
import logging
from datetime import datetime, timedelta

from typing import List, Optional
from collections import defaultdict
from dateutil.parser import parse

import numpy as np
import pandas as pd

from ibapi import wrapper
from ibapi.common import TickerId, BarData
from ibapi.client import EClient
from ibapi.contract import Contract
from ibapi.utils import iswrapper

ContractList = List[Contract]
BarDataList = List[BarData]
OptionalDate = Optional[datetime]


def make_download_path(args: argparse.Namespace, contract: Contract) -> str:
??? """Make path for saving csv files.
??? Files to be stored in base_directory/<security_type>/<size>/<symbol>/
??? """
??? path = os.path.sep.join(
??????? [
??????????? args.base_directory,
??????????? args.security_type,
??????????? args.size.replace(" ", "_"),
??????????? contract.symbol,
??????? ]
??? )
??? return path


class DownloadApp(EClient, wrapper.EWrapper):
??? def __init__(self, contracts: ContractList, args: argparse.Namespace):
??????? EClient.__init__(self, wrapper=self)
??????? wrapper.EWrapper.__init__(self)
??????? self.request_id = 0
??????? self.started = False
??????? self.next_valid_order_id = None
??????? self.contracts = contracts
??????? self.requests = {}
??????? self.bar_data = defaultdict(list)
??????? self.pending_ends = set()
??????? self.args = args
??????? self.current = self.args.end_date
??????? self.duration = self.args.duration
??????? self.useRTH = 0

??? def next_request_id(self, contract: Contract) -> int:
??????? self.request_id += 1
??????? self.requests[self.request_id] = contract
??????? return self.request_id

??? def historicalDataRequest(self, contract: Contract) -> None:
??????? cid = self.next_request_id(contract)
??????? self.pending_ends.add(cid)

??????? self.reqHistoricalData(
??????????? cid,? # tickerId, used to identify incoming data
??????????? contract,
??????????? self.current.strftime("%Y%m%d 00:00:00"),? # always go to midnight
??????????? self.duration,? # amount of time to go back
??????????? self.args.size,? # bar size
??????????? self.args.data_type,? # historical data type
??????????? self.useRTH,? # useRTH (regular trading hours)
??????????? 1,? # format the date in yyyyMMdd HH:mm:ss
??????????? False,? # keep up to date after snapshot
??????????? [],? # chart options
??????? )

??? def save_data(self, contract: Contract, bars: BarDataList) -> None:
??????? data = [
??????????? [b.date, b.open, b.high, b.low, b.close, b.volume, b.barCount, b.average]
??????????? for b in bars
??????? ]
??????? df = pd.DataFrame(
??????????? data,
??????????? columns=[
??????????????? "date",
??????????????? "open",
??????????????? "high",
??????????????? "low",
??????????????? "close",
??????????????? "volume",
??????????????? "barCount",
??????????????? "average",
??????????? ],
??????? )
??????? if self.daily_files():
??????????? path = "%s.csv" % make_download_path(self.args, contract)
??????? else:
??????????? # since we fetched data until midnight, store data in
??????????? # date file to which it belongs
??????????? last = (self.current - timedelta(days=1)).strftime("%Y%m%d")
??????????? path = os.path.sep.join(
??????????????? [make_download_path(self.args, contract), "%s.csv" % last,]
??????????? )
??????? df.to_csv(path, index=False)

??? def daily_files(self):
??????? return SIZES.index(self.args.size.split()[1]) >= 5

??? @iswrapper
??? def headTimestamp(self, reqId: int, headTimestamp: str) -> None:
??????? contract = self.requests.get(reqId)
??????? ts = datetime.strptime(headTimestamp, "%Y%m%d? %H:%M:%S")
??????? logging.info("Head Timestamp for %s is %s", contract, ts)
??????? if ts > self.args.start_date or self.args.max_days:
??????????? logging.warning("Overriding start date, setting to %s", ts)
??????????? self.args.start_date = ts? # TODO make this per contract
??????? if ts > self.args.end_date:
??????????? logging.warning("Data for %s is not available before %s", contract, ts)
??????????? self.done = True
??????????? return
??????? # if we are getting daily data or longer, we'll grab the entire amount at once
??????? if self.daily_files():
??????????? days = (self.args.end_date - self.args.start_date).days
??????????? if days < 365:
??????????????? self.duration = "%d D" % days
??????????? else:
??????????????? self.duration = "%d Y" % np.ceil(days / 365)
??????????? # when getting daily data, look at regular trading hours only
??????????? # to get accurate daily closing prices
??????????? self.useRTH = 1
??????????? # round up current time to midnight for even days
??????????? self.current = self.current.replace(
??????????????? hour=0, minute=0, second=0, microsecond=0
??????????? )

??????? self.historicalDataRequest(contract)

??? @iswrapper
??? def historicalData(self, reqId: int, bar) -> None:
??????? self.bar_data[reqId].append(bar)

??? @iswrapper
??? def historicalDataEnd(self, reqId: int, start: str, end: str) -> None:
??????? super().historicalDataEnd(reqId, start, end)
??????? self.pending_ends.remove(reqId)
??????? if len(self.pending_ends) == 0:
??????????? print(f"All requests for {self.current} complete.")
??????????? for rid, bars in self.bar_data.items():
??????????????? self.save_data(self.requests[rid], bars)
??????????? self.current = datetime.strptime(start, "%Y%m%d? %H:%M:%S")
??????????? if self.current <= self.args.start_date:
??????????????? self.done = True
??????????? else:
??????????????? for contract in self.contracts:
??????????????????? self.historicalDataRequest(contract)

??? @iswrapper
??? def connectAck(self):
??????? logging.info("Connected")

??? @iswrapper
??? def nextValidId(self, order_id: int):
??????? super().nextValidId(order_id)

??????? self.next_valid_order_id = order_id
??????? logging.info(f"nextValidId: {order_id}")
??????? # we can start now
??????? self.start()

??? def start(self):
??????? if self.started:
??????????? return

??????? self.started = True
??????? for contract in self.contracts:
??????????? self.reqHeadTimeStamp(
??????????????? self.next_request_id(contract), contract, self.args.data_type, 0, 1
??????????? )

??? @iswrapper
??? def error(self, req_id: TickerId, error_code: int, error: str):
??????? super().error(req_id, error_code, error)
??????? if req_id < 0:
??????????? logging.debug("Error. Id: %s Code %s Msg: %s", req_id, error_code, error)
??????? else:
??????????? logging.error("Error. Id: %s Code %s Msg: %s", req_id, error_code, error)
??????????? # we will always exit on error since data will need to be validated
??????????? self.done = True


def make_contract(symbol: str, sec_type: str, currency: str, exchange: str, localsymbol: str) -> Contract:
??? contract = Contract()
??? contract.symbol = symbol
??? contract.secType = sec_type
??? contract.currency = currency
??? contract.exchange = exchange
??? if localsymbol:
??????? contract.localSymbol = localsymbol
??? return contract


class ValidationException(Exception):
??? pass


def _validate_in(value: str, name: str, valid: List[str]) -> None:
??? if value not in valid:
??????? raise ValidationException(f"{value} not a valid {name} unit: {','.join(valid)}")


def _validate(value: str, name: str, valid: List[str]) -> None:
??? tokens = value.split()
??? if len(tokens) != 2:
??????? raise ValidationException("{name} should be in the form <digit> <{name}>")
??? _validate_in(tokens[1], name, valid)
??? try:
??????? int(tokens[0])
??? except ValueError as ve:
??????? raise ValidationException(f"{name} dimenion not a valid number: {ve}")


SIZES = ["secs", "min", "mins", "hour", "hours", "day", "week", "month"]
DURATIONS = ["S", "D", "W", "M", "Y"]


def validate_duration(duration: str) -> None:
??? _validate(duration, "duration", DURATIONS)


def validate_size(size: str) -> None:
??? _validate(size, "size", SIZES)


def validate_data_type(data_type: str) -> None:
??? _validate_in(
??????? data_type,
??????? "data_type",
??????? [
??????????? "TRADES",
??????????? "MIDPOINT",
??????????? "BID",
??????????? "ASK",
??????????? "BID_ASK",
??????????? "ADJUSTED_LAST",
??????????? "HISTORICAL_VOLATILITY",
??????????? "OPTION_IMPLIED_VOLATILITY",
??????????? "REBATE_RATE",
??????????? "FEE_RATE",
??????????? "YIELD_BID",
??????????? "YIELD_ASK",
??????????? "YIELD_BID_ASK",
??????????? "YIELD_LAST",
??????? ],
??? )


def main():

??? now = datetime.now()

??? class DateAction(argparse.Action):
??????? """Parses date strings."""

??????? def __call__(
??????????? self,
??????????? parser: argparse.ArgumentParser,
??????????? namespace: argparse.Namespace,
??????????? value: str,
??????????? option_string: str = None,
??????? ):
??????????? """Parse the date."""
??????????? setattr(namespace, self.dest, parse(value))

??? argp = argparse.ArgumentParser()
??? argp.add_argument("symbol", nargs="+")
??? argp.add_argument(
??????? "-d", "--debug", action="store_true", help="turn on debug logging"
??? )
??? argp.add_argument(
??????? "--logfile", help="log to file"
??? )
??? argp.add_argument(
??????? "-p", "--port", type=int, default=7496, help="local port for TWS connection"
??? )
??? argp.add_argument("--size", type=str, default="1 min", help="bar size")
??? argp.add_argument("--duration", type=str, default="1 D", help="bar duration")
??? argp.add_argument(
??????? "-t", "--data-type", type=str, default="TRADES", help="bar data type"
??? )
??? argp.add_argument(
??????? "--base-directory",
??????? type=str,
??????? default="data",
??????? help="base directory to write bar files",
??? )
??? argp.add_argument(
??????? "--currency", type=str, default="USD", help="currency for symbols"
??? )
??? argp.add_argument(
??????? "--exchange", type=str, default="SMART", help="exchange for symbols"
??? )
??? argp.add_argument(
??????? "--localsymbol", type=str, default="", help="local symbol (for futures)"
??? )
??? argp.add_argument(
??????? "--security-type", type=str, default="STK", help="security type for symbols"
??? )
??? argp.add_argument(
??????? "--start-date",
??????? help="First day for bars",
??????? default=now - timedelta(days=2),
??????? action=DateAction,
??? )
??? argp.add_argument(
??????? "--end-date", help="Last day for bars", default=now, action=DateAction,
??? )
??? argp.add_argument(
??????? "--max-days", help="Set start date to earliest date", action="store_true",
??? )
??? args = argp.parse_args()

??? logargs = dict(format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s',
?????????????????? datefmt='%H:%M:%S')
??? if args.debug:
??????? logargs['level'] = logging.DEBUG
??? else:
??????? logargs['level'] = logging.INFO

??? if args.logfile:
??????? logargs['filemode'] = 'a'
??????? logargs['filename'] = args.logfile

??? logging.basicConfig(**logargs)

??? try:
??????? validate_duration(args.duration)
??????? validate_size(args.size)
??????? args.data_type = args.data_type.upper()
??????? validate_data_type(args.data_type)
??? except ValidationException as ve:
??????? print(ve)
??????? sys.exit(1)

??? logging.debug(f"args={args}")
??? contracts = []
??? for s in args.symbol:
??????? contract = make_contract(s, args.security_type, args.currency, args.exchange, args.localsymbol)
??????? contracts.append(contract)
??????? os.makedirs(make_download_path(args, contract), exist_ok=True)
??? app = DownloadApp(contracts, args)
??? app.connect("127.0.0.1", args.port, clientId=0)

??? app.run()


if __name__ == "__main__":
??? main()

afaik it is not

Yeah, that's exactly what I wanted: a stock margin independent from my account, and the one displayed in the details window.
Anybody knows if this margin is available via API? Normally, everything displayed within TWS should be available with API. That's what I've seen so far.

what you get via placeOrder() is rather what you get in tws in the preview window than what's in the contract description window. in general these two should be the same, though there might be cases when your current portfolio might affect the real margin requirement applied to your account. and that depends on many factors, product itself, account type, current portfolio etc... i don't even know the exact details. if you would want a generic margin requirement that is not related to your account, you would probably have to parse it from the details web page that ib provides from tws menu (the details menu item).

Thanks fordfrog. You're 100% right. This margin can be retrieved with WhatIf Buy order. Below margin returned is indeed 100% for ADMP. I was actually so sure it would not, as it is attached to the stock details display. I was wondering if we could get this info simplier. Oh well....
Ticket closed...

i didn't test it on this stock (or stocks), but are you sure you cannot retrieve it using with whatIf=true? you should be able to. it should be part of ?which is part of the callback . sure for long your whatIf order should be for buy.

Hello,
I read this forum extensively, but didn't find how I can retrieve the margin informations about a US Stock.
It's not in Contract, neither ContractDetails,
This is not the information I should get with WhatIf option, or SLB rates.
This is also not the RegT default margin calculation of standard 25%.
Please see in the attached picture the information I'd like to get using API.
As you can see in the picture, extreme volatile stocks can have 50% or 100% margin requirements.
Thanks in advance for your solution.

Did you get any further with this topic?

I have the same problem... why can I draw a snapshot inside TWS, but not from an API call?! For the same asset. Not matter how I set my flags.

Is it possible to have a fee-based live/realtime snapshot for a contract I don't have the belonging subscription for, from reqMktData-call , or not?

In other words:
Is the "pay 0.01/0.03$ for single snapshot if you have no subscription"-option only available by manual mouse click or does it also work form API, if the latter, HOW?

Hi 闯ü谤驳别苍, thank you for your explanation. Especially the relationship between IB's Decimal and Java's BigDecimal classes. I understand what you mean by the sentence "once in decimal you want to stay in decimal". I am not trading fractional shares and crypto. In my software are position and trade sizes, volumes, and so on, converted to an integer once they are received from IB's servers. Prices are treated as doubles. Hopefully I won't have too much trouble in converting integers to Decimals, when I do make the upgrade to this newer API version.

There were breaking API changes in 10.14 if you used any third party layers as errorHandler obtained another parameter.

Traditionally, the TWS API used floating point "double" values for price fields a mixture of data types (int, long, double) for size and quantity fields in order related classes an for real-time and historical data. Floating point math (regardless of programming language) is prone to significant imprecision that you cannot afford in financial transactions. So with the introduction of fractional shares and crypto trading, TWS API needed to adjusted such that precise conversion and math is possible.

With Java has built-in support for "Immutable, arbitrary-precision signed decimal numbers", and IBKR opted to introduce their own "Decimal" class that is basically a wrapper around BigDecimal configured with 16 digit precision and a rounding mode of "half even".

BigDecimal is more complex than double and math cannot be expressed easily with simple operators such as +, -, *, and /. You rather need to call class methods so that a simple:

double d = (a + b) * c

turns into something like

Decimal d = a.plus( b ).multiply( c );

So there will likely be sections of code that you have to convert to accommodate the new data type. This is more involved than when size fields for market data changed from int to long a few years back.

In our specific case, our framework already had a class called "Decimal" (also wrapped around BigDecimal) that now needed to be bridged efficiently with the new IBKR "Decimal" class. So we made a thin buffer layer that allows the two similar but different Decimal classes to coexist and for data to flow efficiently without excessive conversions between the classes. The one thing you do not want to do is converting between Decimal and double all the time since you will quickly inherit the intrinsic imprecision of floating point math again. Once in decimal you want to stay in decimal.

TWS API for C++ and Python uses external libraries and I believe TWS API for C# uses a decimal type that is built into the language as well.

闯ü谤驳别苍


On Wed, Mar 22, 2023 at 12:17 AM, J G wrote:

闯ü谤驳别苍, you mention "With the introduction of the new Decimal type, transition to API V10 versions is not trivial". I have not done the upgrade yet, am using API version 972. All my software is written in Java and I am unfamiliar with the Decimal class. What kind of "non trivial" things can I expect if I do upgrade?

?

闯ü谤驳别苍, you mention "With the introduction of the new Decimal type, transition to API V10 versions is not trivial". I have not done the upgrade yet, am using API version 972. All my software is written in Java and I am unfamiliar with the Decimal class. What kind of "non trivial" things can I expect if I do upgrade?

You might still want to put in the elbow grease and start moving from V9 API to V10 soon, but on your own terms and time frame, Mark. You will be able to keep your FX data flowing in the interim with the TWS setting, but eventually there will be a feature that you want or need. Or, IBKR will eventually require an update. As I said before, V10 is around for a couple years now already.

With the introduction of the new Decimal type, transition to API V10 versions is not trivial but then it is not horrible either. We prepared our environment over the last year or so and will finally transition from 985.02 to 10.19 one of the next few weekends. Once you are on V10, upgrading within the V10 track will be easier again (I guess until we hit V11 or so).

When we make the jump, we'll gain access to these features in addition to the fractional size support:

Feature	Min API Version
AUTO_CANCEL_PARENT	162
FRACTIONAL_SIZE_SUPPORT	163
SIZE_RULES	164
HISTORICAL_SCHEDULE	165
ADVANCED_ORDER_REJECT	166
USER_INFO	167
CRYPTO_AGGREGATED_TRADES	168
MANUAL_ORDER_TIME	169
PEGBEST_PEGMID_OFFSETS	170
WSH_EVENT_DATA_FILTERS	171
IPO_PRICES	172
WSH_EVENT_DATA_FILTERS_DATE	173
INSTRUMENT_TIMEZONE	174
HMDS_MARKET_DATA_IN_SHARES	175
BOND_ISSUERID	176

闯ü谤驳别苍

i can see ?both 10.19 and 10.20 available, so no signing up should be needed. i am also signed to their project, but recently i implemented the tws api client in java myself so i don't use their tws api client anymore, but i still track the changes from the repo.

What error message are you getting, Mark? If TWS API complains about the fact that your client does not support fractional sizes, you need a client API version with MAX_VERSION of at least 163. Client API features and their respective version numbers are defined in the EClient class.

Client version 163 (aka MIN_SERVER_VER_FRACTIONAL_SIZE_SUPPORT) was introduced with TWS API 10.10.03 which came out in mid-2021. So there is no need for you to sign up for the beta version. Transition to fractional sizes is going on for years now. But you will need to make changes to your client to support the "decimal" data type IBKR uses for fractional sizes. There are lot's of posts about that in the group.

闯ü谤驳别苍

On Mon, Mar 20, 2023 at 09:56 AM, mark collins wrote: