|
How to fetch multiple historical data types in python?
I am trying to fetch multiple data types and drop them in a single dataframe as an ultimate goal, in particular I am interested in ADJUSTED_LAST, OPTION_IMPLIED_VOLATILITY and REBATE_RATE (code below does not include REBATE_RATE).? ?
?
# Import libraries
from ibapi.client import EClient
from ibapi.wrapper import EWrapper
from ibapi.contract import Contract
import pandas as pd
import threading
import time
?
class TradeApp(EWrapper, EClient):?
? ? def __init__(self):?
? ? ? ? EClient.__init__(self, self)?
? ? ? ? self.data = {}
? ? ? ??
? ? def historicalData(self, reqId, bar):
? ? ? ? if reqId not in self.data:
? ? ? ? ? ? self.data[reqId] = [{"Date":bar.date,"Open":bar.open,"High":bar.high,"Low":bar.low,"Close":bar.close,"Volume":bar.volume}]
? ? ? ? ? ? self.data[reqId] = [{"Date":bar.date,"Open Vol":bar.open,"High Vol":bar.high,"Low Vol":bar.low,"Close Vol":bar.close}]
? ? ? ? else:
? ? ? ? ? ? self.data[reqId].append({"Date":bar.date,"Open":bar.open,"High":bar.high,"Low":bar.low,"Close":bar.close,"Volume":bar.volume})
? ? ? ? ? ? self.data[reqId].append({"Date":bar.date,"Open Vol":bar.open,"High Vol":bar.high,"Low Vol":bar.low,"Close Vol":bar.close})
?
? ? ? ? print("reqID:{}, date:{}, open:{}, high:{}, low:{}, close:{}, volume:{}".format(reqId,bar.date,bar.open,bar.high,bar.low,bar.close,bar.volume))
?
def usTechStk(symbol,sec_type="STK",currency="USD",exchange="ISLAND"):
? ? contract = Contract()
? ? contract.symbol = symbol
? ? contract.secType = sec_type
? ? contract.currency = currency
? ? contract.exchange = exchange
? ? return contract?
?
?
#Eclient function
def histData(req_num,contract,duration,candle_size):
? ? """extracts historical data"""
? ? app.reqHistoricalData(reqId=req_num,?
? ? ? ? ? ? ? ? ? ? ? ? ? contract=contract,
? ? ? ? ? ? ? ? ? ? ? ? ? endDateTime='',
? ? ? ? ? ? ? ? ? ? ? ? ? durationStr=duration,
? ? ? ? ? ? ? ? ? ? ? ? ? barSizeSetting=candle_size,
? ? ? ? ? ? ? ? ? ? ? ? ? whatToShow= 'ADJUSTED_LAST',
? ? ? ? ? ? ? ? ? ? ? ? ? useRTH=1,
? ? ? ? ? ? ? ? ? ? ? ? ? formatDate=1,
? ? ? ? ? ? ? ? ? ? ? ? ? keepUpToDate=0,
? ? ? ? ? ? ? ? ? ? ? ? ? chartOptions=[]) ?
? ??
? ? app.reqHistoricalData(reqId=req_num,?
? ? ? ? ? ? ? ? ? ? ? ? ? contract=contract,
? ? ? ? ? ? ? ? ? ? ? ? ? endDateTime='',
? ? ? ? ? ? ? ? ? ? ? ? ? durationStr=duration,
? ? ? ? ? ? ? ? ? ? ? ? ? barSizeSetting=candle_size,
? ? ? ? ? ? ? ? ? ? ? ? ? whatToShow='OPTION_IMPLIED_VOLATILITY',
? ? ? ? ? ? ? ? ? ? ? ? ? useRTH=1,
? ? ? ? ? ? ? ? ? ? ? ? ? formatDate=1,
? ? ? ? ? ? ? ? ? ? ? ? ? keepUpToDate=0,
? ? ? ? ? ? ? ? ? ? ? ? ? chartOptions=[])
? ??
?
?
def websocket_con():
? ? app.run()
? ??
app = TradeApp()
app.connect(host='127.0.0.1', port=7496, clientId=22) #port 4002 for ib gateway paper trading/7497 for TWS paper trading
con_thread = threading.Thread(target=websocket_con, daemon=True)
con_thread.start()
time.sleep(1) # some latency added to ensure that the connection is established
?
tickers = ["GME"]
for ticker in tickers:
? ? histData(tickers.index(ticker),usTechStk(ticker),'1 D', '1 hour')
? ? time.sleep(5)
?
###################storing trade app object in dataframe#######################
def dataDataframe(symbols,TradeApp_obj):
? ? "returns extracted historical data in dataframe format"
? ? df_data = {}
? ? for symbol in symbols:
? ? ? ? df_data[symbol] = pd.DataFrame(TradeApp_obj.data[symbols.index(symbol)])
? ? ? ? df_data[symbol].set_index("Date",inplace=True)
? ? return df_data
?
#extract and store historical data in dataframe
historicalData = dataDataframe(tickers,app)
?
print("Disconnection")
time.sleep(3)
app.disconnect()
?
?
? |
|
Re: How to create a performance efficient trading app so each sub process runs smoothly in Python?
Matthias Frener
and maybe to add a #3) I'm on node.js () and only way to scale there is running multiple processes. You can do this with controler/worker concept (e.g. via shared memory a mentioned), or you isolate it even further, looking at every task like beieng a independent service with a defined interface. My "trading app" backend consist of a nginx reverse proxy and than there is a auth-service for login, a user-mangement service for user management, a portfolio service to watch portfolio, a ticker service to watch ticker, a IB-API service to convert TWS API into REST/WebSock API, a trade-log service.. ect.. pp. A lot of small little?processes, each having a very specific task. The connect to each other via the APIs. Every services defines an OpenAPI model (swagger) and whoever needs it, generates client code from the json and uses it. It's scalable across CPUs, machines and data centers - makes it easy to implement failovers (when machine 1 is down, fallback to machine?2) or scale dynamically (start up more processes, or more machines, or more clouds - depends on how much more power you need).? |
|
Re: How to create a performance efficient trading app so each sub process runs smoothly in Python?
Thank you Kris K,
I tried suggestion #1 using a minimal IB program and put it in a Process, like: from ibapi.wrapper import EWrapper from ibapi.client import EClient from ibapi.utils import iswrapper from ibapi.common import * from ibapi.contract import * from ibapi.ticktype import * # Request IB Data in less than 50 lines of code class BasicApp(EWrapper, EClient): ? def __init__(self): ??? EClient.__init__(self,self) ? def error(self, reqId: TickerId, errorCode:int, errorString:str): ??? print('Error:', reqId, " ", errorCode, " ", errorString) ? @iswrapper ? def tickPrice(self, reqId: TickerId, tickType: TickType, price: float, attrib: TickAttrib): ??? super().tickPrice(reqId, tickType, price, attrib) ??? print("Tick Price. Ticker Id:", reqId, "tickType:", tickType, "Price:", price, "CanAutoExecute:", attrib.canAutoExecute, "PastLimit", attrib.pastLimit) ? ? @iswrapper ? def tickSize(self, reqId: TickerId, tickType: TickType, size: int): ??? super().tickSize(reqId, tickType, size) ??? print("Tick Size. Ticker Id:", reqId, "tickType:", tickType, "Size:", size) ? @iswrapper ? def tickString(self, reqId: TickerId, tickType: TickType, value: str): ??? super().tickString(reqId, tickType, value) ??? print("Tick string. Ticker Id:", reqId, "Type:", tickType, "Value:", value) ? @iswrapper ? def tickGeneric(self, reqId: TickerId, tickType: TickType, value: float): ??? super().tickGeneric(reqId, tickType, value) ??? print("Tick Generic. Ticker Id:", reqId, "tickType:", tickType, "Value:", value) def main(): ? app = BasicApp() ? app.connect("127.0.0.1", 7497, 0) ? contract = Contract(); ? ? contract.symbol = "EOE"; ? contract.secType = "IND"; ? contract.exchange = "FTA"; ? contract.currency = "EUR"; ? ? ? app.reqMarketDataType(3); ? app.reqMktData(0, contract, "", False, False, []); ? app.run() from multiprocessing import Process if __name__ == '__main__': ? p = Process(target=main) ? p.start() ? p.join() This initial basic program appears to be working (now I am going extend it), sometimes it gives me this though: ? unhandled exception in EReader threadTraceback (most recent call last):? File "/opt/anaconda3/lib/python3.8/site-packages/ibapi-9.76.1-py3.8.egg/ibapi/reader.py", line 34, in run??? data = self.conn.recvMsg()? File "/opt/anaconda3/lib/python3.8/site-packages/ibapi-9.76.1-py3.8.egg/ibapi/connection.py", line 99, in recvMsg??? buf = self._recvAllMsg()? File "/opt/anaconda3/lib/python3.8/site-packages/ibapi-9.76.1-py3.8.egg/ibapi/connection.py", line 119, in _recvAllMsg??? buf = self.socket.recv(4096)ConnectionResetError: [Errno 54] Connection reset by peer^CTraceback (most recent call last):? File "minimal_process.py", line 57, in <module>??? p.join()? File "/opt/anaconda3/lib/python3.8/multiprocessing/process.py", line 149, in join??? res = self._popen.wait(timeout)? File "/opt/anaconda3/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait??? return self.poll(os.WNOHANG if timeout == 0.0 else 0)? File "/opt/anaconda3/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll??? pid, sts = os.waitpid(self.pid, flag)KeyboardInterrupt |
|
Re: How to Get Mkt Data from Scanned Symbols
I managed to get the contract details by invoking?self.reqMktData(self.count, scanned_contract, '', False, False, []).
But I dont know the Python syntax of extending the Scanner data with the Market Data details as availble on TWS. i.e I can get my ranked stock symbols, I can get the req market data but I dont know how to tie both of them so that I get the same view in TWS Advanced Scanner result (with market data such as close, mkt capitalization, volatility etc...)? Any pointers will be much appreciated. |
|
Re: How to Get Mkt Data from Scanned Symbols
I can't speak for the python side, but reqMktData needs a contract with ConId and Exchange ("SMART") set generally to get US Listed quotes..??
If you have the ContractDetails object back from the scan, ContractDetails.Contract is what you'd send in the call to reqMktData..?? -Peter |
|
How to Get Mkt Data from Scanned Symbols
Hi, I have extracted the ScannerData via the wrapper and ranked as needed. Need to find additional information for the scanned symbols which can be made available via reqMktData. Since I am new to Python, I am struggling to know how to add the code for reqMktData (essentially giving me the view I see on TWS Advance Scanner screen Below is my code (I am trying to get one symbol's MktData (which is MBB for a test). Many thanks """ Created on Thu Mar 18 10:19:45 2021 @author: DJ """ #%% Imports from ibapi.client import EClient from ibapi.wrapper import EWrapper from ibapi.utils import iswrapper from ibapi.scanner import ScannerSubscription from ibapi.contract import Contract from ibapi.tag_value import TagValue from ibapi.ticktype import TickTypeEnum from threading import Thread import time #%% Client and Wrapper class StockScanner(EWrapper, EClient): ? ? ''' Serves as the client and the wrapper ''' ? ? def __init__(self, addr, port, client_id): ? ? ? ? EClient. __init__(self, self) ? ? ? ? self.data = [] #Initialize variable to store candle ? ? ? ? ? ? ? ? # Connect to TWS ? ? ? ? self.connect(addr, port, client_id) ? ? ? ? self.count = 0 ? ? ? ? ? ? ? ? # Launch the client thread ? ? ? ? thread = Thread(target=self.run) ? ? ? ? thread.start() ? ? ? ? @iswrapper ? ? def scannerData(self, reqId, rank, details, distance, benchmark, projection, legsStr): ? ? ? ? # Print the symbols in the returned results # ? ? ? ?print('{}: {}'.format(rank, details.contract.symbol)) ? ? ? ? data1 = (rank+1, details.contract.symbol, details.contract.currency, ? ? ? ? ? ? ? ? ?details.contract.exchange, details.contract.secType ? ? ? ? ? ? ? ? ) ? ? ? ? print(data1) ? ? ? ? ? ? ? ? #Request market data ? ? ? ? scanned_contract = Contract() ? ? ? ? scanned_contract.symbol = details.contract.symbol ? ? ? ? scanned_contract.secType = details.contract.secType ? ? ? ? scanned_contract.exchange = details.contract.exchange ? ? ? ? scanned_contract.currency = details.contract.currency ? ? ? ? ? ? ? ? if details.contract.symbol == 'MBB': ? ? ? ? ? ? data2 = reqMktData(self, self.count, scanned_contract, '', False, False, []) ? ? ? ? ? ? print(scanned_contract) ? ? ? ? ? ? print(data2) ? ? ? ? ? ? def tickPrice(self, reqID, tickType, price, attrib): ? ? ? ? ? ? ? ? print("Tick Price. Ticker ID: ", count, "ticktype: ", TickTypeEnum.to_str(tickType), "Price: ", price) ? ? ? ? self.data.append(data1) ? ? ? ? self.count += 1 ? ? @iswrapper ? ? def scannerDataEnd(self, reqId): ? ? ? ? # Print the number of results ? ? ? ? print('Number of results: {}'.format(self.count)) ? ? ? ? ? ? ? ? def error(self, reqId, code, msg): ? ? ? ? print('Error {}: {}'.format(code, msg)) ? ? ? ? ? ? ? ? ? #%% Main def main(): ? ? # Create the client and connect to TWS ? ? client = StockScanner('127.0.0.1', 7497, 0) ? ? time.sleep(0.5) ? ? # Create the ScannerSubscription object ? ? ? ss = ScannerSubscription() ? ? ss.instrument = 'STK' ? ? ss.locationCode = 'STK.NASDAQ' ? ? # Top % Gainers Since Open ? ? # ss.scanCode = 'TOP_PERC_GAIN' ? ? ss.scanCode = 'HIGH_OPT_IMP_VOLAT_OVER_HIST' ? ? ss.abovePrice = 1.0 ? ? ss.belowPrice = 1000.0 ? ? # Set additional filter criteria ? ? tagvalues = [] ? ? tagvalues.append(TagValue('changePercAbove', '0%')) ? ? # Request the scanner subscription ? ? client.reqScannerSubscription(0, ss, [], tagvalues) ? ? ? ? ? ? # Sleep while the request is processed ? ? time.sleep(5) ? ? ? ? #Export to CSV file ? ? ? import pandas ? ? df = pandas.DataFrame(client.data, columns=['Rank', 'Symbol', ?'Currency', 'Exchange', 'Sec Type']) ? ? df.to_csv('ScannedResult.csv') ? ? client.disconnect() ? ? if __name__ == '__main__': ? ? main()?? |
|
Re: HOW TO SURVIVE THE IB TWS/GATEWAY RESET?
Thanks Bruce and Mason!
Looks like regularly saving the state is a good approach. I program in Python but my backgound is more Matlab for signal and image processing. I'm not fully at ease with object oriented programming concepts. Do you use the IB Python API? F |
|
Re: HOW TO SURVIVE THE IB TWS/GATEWAY RESET?
My solution to this is that all of my code's state is maintained within a single class with a "set state" method, and every single time that method is called, the state is saved to a JSON file, and every time the class is constructed, it loads all state from that file. Since the rest of my code operates as an FSM, it can pick up right where it left off when I restart.
|
|
Re: HOW TO SURVIVE THE IB TWS/GATEWAY RESET?
Why don't you save your orders state and call status of each order after you receive a reset signal or after you determine TWS was reset? It was said that Order Refs get reset but you can build different ways to cope with this. -Bruce On Thu, Mar 18, 2021, 4:32 PM Ace <fbaribeau1967@...> wrote: Hi all! |
|
Re: HOW TO SURVIVE THE IB TWS/GATEWAY RESET?
Stuart Cracraft
toggle quoted message
Show quoted text
On Mar 18, 2021, at 1:35 PM, Ace <fbaribeau1967@...> wrote:
|
|
HOW TO SURVIVE THE IB TWS/GATEWAY RESET?
Hi all!
I have a day trading automated strategy and now I would like to swing positions overnight and keep them while conditions are met. I faced the nightly TWS reset which now also applies to the gateway. My strategy is very dynamic with scaling in/out and this reset kills my loops and variables state. I noticed this is a recurring issue that I've not found a clear solution for by looking through the forums. So how do you go about surviving the reset for your swing trading strategies to get back exactly where you were at before the reset? Thank you very much! F |
|
Re: Little tool for option chain contract creation
juxeiier
¿ªÔÆÌåÓýI am not sure what you mean by all data.So far, only the static contract data for options are requested. But soon, also snapshots of actual market data will be included. At some point, I will provide a pandas data frame to do work with data like bid/ask, greeks etc... The storage mechanism relies on pickle: For each symbol, for which you request option contracts, there will be a binary caching file written. The script cleans that up, such that outdated contracts will get deleted. Cheers, Juergen Am 18.03.2021 um 18:49 schrieb Bruce B:
|
|
Re: event when a stock reaches a certain price
Their last sentence is vague. Bad documentation like this I have seen in other places too. Yes, I think the delay is in effect and so your quick requests were responded again and again with same data because your 11 seconds bottleneck was not up yet. Most likely those quick - over time - requests were not even sent to IBKR at all and TWS decides and responds with "stale" data based on 11 second rule. So that could be the "cache". - Bruce On Thu, Mar 18, 2021, 2:34 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó <me@...> wrote:
|
|
Re: event when a stock reaches a certain price
My request in fact were duplicates - since I requested snapshots for 1500 instruments to get "almost real-time" data. Request ids were never repeated. Price did changed and if I had real-time subscription at the same time - it worked fine. Actually, I worked around it and used real-time data for important data and round-robin for everything else. From IB docs - " It is important to note that a snapshot request will only return available data over the 11 second span; in some cases values may not be returned for all tick types." So maybe this is expected behavior.? On Thu, Mar 18, 2021 at 7:24 PM Bruce B <bruceb444@...> wrote:
--
SI: ?+386-30-315-640 |
|
Re: TWS app keeps exiting on my desktop
You might try to run the tws in gateway-mode.
If its stable then, you have to allocate more ressources. By the way,? There is IB-Container , a simple script to setup a LXD-Container with autostart as cron-service and reverse-ssh-tunnel to setup a secret trading-place |
|
Re: event when a stock reaches a certain price
Agreed - most things don't from IBKR do not? include timestamps and some may include fake timestamps from TWS. Dimitry I think has looked into this and posted before. Is it also possible that maybe your requests were duplicates OR that price didn't change in between your multiple requests? You can compare to TWS data or to other sources to confirm maybe. I would be surprised if they send stale data over a live account as this has to be a serious? programming error. Can't be a network cache issue either as TCP ACKS would take care of those but maybe a different type of cache. On Thu, Mar 18, 2021, 2:15 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó <me@...> wrote:
|
