VeighNa量化社区
你的开源社区量化交易平台
Member
avatar
加入于:
帖子: 187
声望: 55

不管是研究套利策略,还是多因子策略,都需要多品种的历史数据,所以下面介绍一下,如何调用vnpy的数据下载模块,来下载全市场的期货数据。

 

批量下载

 

1)设置合约品种

 

首先,我们要先生成一个字典,来指定需要下载的数据,关键字段有3个:

 

  • 交易所代号:上期所-> SHFE
  • 合约代号: 螺纹钢-> RB
  • 合约品种类型: 指数合约 -> 99

 

这样,在RQData中,我们要下载螺纹钢指数合约的历史数据,需要转成的代号为RB99.SHFE。
 

然后,由于是全市场行情的数据,所以字典的数据结构如下:key是交易所,value是列表,里面包含各种期货品种,这样,只要在遍历一下这个字典,就可以得到所有,如RB99.SHFE这样结构的字符串。

 

symbols = {
    "SHFE": ["CU", "AL", "ZN", "PB", "NI", "SN", "AU", "AG", "RB", "WR", "HC", "SS", "BU", "RU", "NR", "SP", "SC", "LU", "FU"],
    "DCE": ["C", "CS", "A", "B", "M", "Y", "P", "FB","BB", "JD", "RR", "L", "V", "PP", "J", "JM", "I", "EG", "EB", "PG"],
    "CZCE": ["SR", "CF", "CY", "PM","WH", "RI", "LR", "AP","JR","OI", "RS", "RM", "TA", "MA", "FG", "SF", "ZC", "SM", "UR", "SA", "CL"],
    "CFFEX": ["IH","IC","IF", "TF","T", "TS"]
}
​
symbol_type = "99"

 
2) 设置下载时间
 

我们只需要设置下载的开始和结束时间即可,需要注意的是,vnpy数据下载模块的入参是datetime.datetime格式,所以,要做到格式的一致,代码如下:
 

from datetime import datetime
start_date = datetime(2005,1,1)
end_date = datetime(2020,9,10)

 
3)批量下载全市场数据

 
批量下载数据,并不难,其运作步骤如下:

 

  1. 遍历symbols字典,
  2. 生成不同的HistoryRequest,
  3. 调用数据下载模块rqdata_client.query_history,得到数据data
  4. 调用数据保存模块database_manager.save_bar_data,把下载好的数据data写入数据库
     
    from vnpy.trader.rqdata import rqdata_client
    from vnpy.trader.database import database_manager
    from vnpy.trader.constant import Exchange, Interval
    from vnpy.trader.object import HistoryRequest
    ​
    def load_data(req):
     data = rqdata_client.query_history(req)
     database_manager.save_bar_data(data)
     print(f"{req.symbol}历史数据下载完成")
    ​
    for exchange, symbols_list in symbols.items():
     for s in symbols_list:
         req = HistoryRequest(
         symbol=s+symbol_type,
         exchange=Exchange(exchange),
         start=start_date,
         interval=Interval.DAILY,
         end=end_date,
         )
         load_data(req=req)
     

写好脚本后,我们运行一下代码,可以看到很快就下完全市场期货的日线数据啦。

description

 

若要下载小时或者分钟级别数据,只要把日线周期(Interval.DAILY)改成对应的小时,或者分钟即可。

 

定时批量更新数据

 

有了历史数据后,自然产生每天定时更新数据的需求

 

1)设置定时器

 
我们希望在收盘后,某个时间点如下午5点启动脚本,来自动下载数据。这本质上是包含了一个父进程和一个子进程。
 

父进程可以是一个永远在运行的python程序,如while循环,然后设置触发条件,如当时间刚好到下午5点就启动子进程下载更新数据,其他时间则睡觉等待。

 
代码如下:

from datetime import datetime, time
from time import sleep
​
current_time = datetime.now().time()
start_time = time(17,0)
​
while True:
  sleep(10)

  if current_time == start_time:
    download_data()

 
2)获取数据库数据

 
更新数据时候,我们要以数据库里面最新的数据的时间点,作为开始时间,而结束时间就是当天。比如,昨天刚好下载完所有市场的数据,那么今天我们只需要下载从昨天到今天的所有数据即可。
 

这样实现起来也不难,步骤如下:
 

1)调用database_manager.get_bar_data_statistics来得到字典格式的数据数据库所有信息

 

data = database_manager.get_bar_data_statistics()

 

2)获取各品种最新数据的时间信息,并且插入到data字典中

 

for d in data:
    newest_bar = database_manager.get_newest_bar_data(
        d["symbol"], Exchange(d["exchange"]), Interval(d["interval"])
    )
    d["end"] = newest_bar.datetime

 

然后我们看看data字典,发现真的包含所有行情的数据,但我们是基于RQData来定期更新信息的,所以要进行筛选,得到国内期货品种(通过交易所来判断)并且是日线级别的数据。
 
description
 

3)基于交易所和K线周期筛选品种,得到新的字典symbols,其中key包含合约代码,交易所,value就是数据库的结束时间,如下图:

 

symbols = {}
for i in data:
    if i["interval"] == "d" and  i["exchange"] in Exchanges:
        vt_symbol = f"{i['symbol']}.{i['exchange']}"
        end = i["end"].date()
        symbols[vt_symbol] = end

description
 
4)设置下载结束时间为当天,基于symbols字典的信息,遍历组合得到HistoryRequest,然后再调用上面定义好的load_data函数下载数据并写入数据库中。

 

end_date = datetime.now().date()
​
for vt_symbol, start_date in symbols.items():
    symbol = vt_symbol.split(".")[0]
    exchange = vt_symbol.split(".")[1]
    req = HistoryRequest(
    symbol=symbol,
    exchange=Exchange(exchange),
    start=start_date,
    interval=Interval.DAILY,
    end=end_date,
    )
    load_data(req=req)

 

下载好之后,我们再获取数据库里面最新的K线时间,发现成功更新到今天了。

 
description

Member
avatar
加入于:
帖子: 5
声望: 0

好东西!
股票的批量下载,有吗

Member
avatar
加入于:
帖子: 4618
声望: 284

把symbols那块按照你想要的改应该就行了吧

Member
avatar
加入于:
帖子: 18
声望: 0

谢谢,有包含全部代码的文件下载吗

Member
avatar
加入于:
帖子: 4618
声望: 284

复制一下上面的代码应该就行了吧

Member
avatar
加入于:
帖子: 4
声望: 0

楼主 请教下按照你的代码跑的话报错 显示rqdata里面self.symbols是None?这个问题怎么回事你那又遇到过吗?

Member
avatar
加入于:
帖子: 4
声望: 0

搞定了 楼主提供的代码中没有执行rqdata_client.init(),先执行下这个就好了

Member
加入于:
帖子: 2
声望: 0

xiaoshouchuan wrote:

搞定了 楼主提供的代码中没有执行rqdata_client.init(),先执行下这个就好了
刚接触量化这一块儿,能帮忙给下具体代码嘛?我的运行保错。

Member
avatar
加入于:
帖子: 4618
声望: 284

能贴一下报错截图吗

Member
avatar
加入于:
帖子: 15
声望: 0

请问,我将代码全部复制到一个脚本里,然后运行时出现以下错误,没有数据,可以从哪些方面排除错误呢?
修改了rq数据,代码以以 88、888、889 结尾。而没有99的情况。
同时初始化了rqdata_client.init( )
description

现在还是错误的情况,出现以下错误:

Member
avatar
加入于:
帖子: 15
声望: 0

是否与只能回测而无法单独下载有关
description

Member
avatar
加入于:
帖子: 4618
声望: 284

请问有rqdata的账号吗?ctp接口没有提供历史数据,回测页面点击下载国内期货数据需要rqdata账号的

Member
avatar
加入于:
帖子: 15
声望: 0

xiaohe wrote:

请问有rqdata的账号吗?ctp接口没有提供历史数据,回测页面点击下载国内期货数据需要rqdata账号的

是申请了账号的,这个问题解决了,是在rqdata_client.init()里的username不是注册的手机账号,而是“license”,与邮件中发送的rq.init('手机号', '密码')不同

Member
avatar
加入于:
帖子: 20
声望: 2

修正一下楼主的数据:
symbols={
"SHFE": ["AG","AL","AU","BU","CU","FU","HC","LU","NI","NR","PB","RB","RU","SC","SN","SP","SS","WR","ZN"],
"DCE": ["A","B","BB","C","CS","EG","EB","FB","I","J","JD","JM","L","LH","M","P","PG","PP","RR","V","Y"],
"CZCE": ["AP","CF","CJ","CY","FG","JR","LR","MA","OI","PF","PM","RI","RM","RS","SA","SF","SM","SR","TA","UR","WH","ZC"],
"CFFEX": ["IC","IF","IH","T","TF","TS"]
}
增加了生猪-LH,短纤-PF
czce的CL应为为CJ红枣
同时按字母顺序排列了下,方便以后增删查找

Member
avatar
加入于:
帖子: 23
声望: 0

完整代码如下:
from vnpy.trader.rqdata import rqdata_client
from vnpy.trader.database import database_manager
from vnpy.trader.constant import Exchange, Interval
from vnpy.trader.object import HistoryRequest
from datetime import datetime

start_date = datetime(2018,1,1)
end_date = datetime(2021,1,19)

rqdata_client.init('18650816356','long5204559')
print("init结束")

symbols={
"SHFE": ["AG","AL","AU","BU","CU","FU","HC","LU","NI","NR","PB","RB","RU","SC","SN","SP","SS","WR","ZN"],
"DCE": ["A","B","BB","C","CS","EG","EB","FB","I","J","JD","JM","L","LH","M","P","PG","PP","RR","V","Y"],
"CZCE": ["AP","CF","CJ","CY","FG","JR","LR","MA","OI","PF","PM","RI","RM","RS","SA","SF","SM","SR","TA","UR","WH","ZC"],
"CFFEX": ["IC","IF","IH","T","TF","TS"]
}
symbol_type = "99"
print("symbols字典创建成功")

def load_data(req):

#3、调用数据下载模块rqdata_client.query_history,得到数据data。
data = rqdata_client.query_history(req)
#4、调用数据保存模块database_manager.save_bar_data,把下载好的数据data写入数据库。
database_manager.save_bar_data(data)
print(f"{req.symbol}历史数据下载完成")

for exchange, symbols_list in symbols.items():
for s in symbols_list:
req = HistoryRequest(
symbol=s+symbol_type,
exchange=Exchange(exchange),
start=start_date,
interval=Interval.MINUTE,
end=end_date
)
load_data(req=req)

print("运行结束!")

但是,因为我用的是RQData 的 15天试用账号,所以下载一点点数据后,就提示 rqdatac.share.errors.QuotaExceeded: Quota exceeded ,超出配额。

但是下载的数据,放在哪个文件夹里呀?有哪位大神回复下?

Member
avatar
加入于:
帖子: 4618
声望: 284

报错是因为试用账户有限额导致的。
.vntrader文件夹下的database.db
可通过datamanager模块查看https://www.vnpy.com/forum/topic/3110-vn-pyfa-bu-v2-1-1-quan-gong-neng-shu-ju-guan-li

Member
avatar
加入于:
帖子: 3
声望: 0

好用,非常感谢

Member
avatar
加入于:
帖子: 7
声望: 5

VNPY升级到最新版之后,需要修改部分代码,代码思路同楼主:

from datetime import datetime

from vnpy.trader.rqdata import RqdataClient
from vnpy.trader.object import HistoryRequest
from vnpy.trader.database import database_manager

rqdataClient=RqdataClient()
rqdataClient.init()

updateInfo=[]
barOverViews=database_manager.get_bar_overview()
for barOverView in barOverViews:
    #print(f"{barOverView.symbol} start:{barOverView.start} end:{barOverView.end} count:{barOverView.count}")
    updateInfo.append((barOverView.symbol,barOverView.exchange,barOverView.end,barOverView.interval))


endDt=datetime.now()

for info in updateInfo:
    print(f'更新:{info[0]},Interval:{info[3]}')
    req=HistoryRequest(
        symbol=info[0],
        exchange=info[1],
        start=info[2],
        end=endDt,
        interval=info[3],
    )
    rData=rqdataClient.query_history(req)
    database_manager.save_bar_data(rData)
Member
avatar
加入于:
帖子: 32
声望: 0

请问"CFFEX": ["IC","IF","IH","T","TF","TS"]}如果要下载中金所的指数期权代吗如何写?

Member
avatar
加入于:
帖子: 32
声望: 0

description我复制了你以下代码到VNPY脚本策略中运行得到图中错误,请解说一下!

from datetime import datetime

from vnpy.trader.rqdata import RqdataClient
from vnpy.trader.object import HistoryRequest
from vnpy.trader.database import database_manager

rqdataClient=RqdataClient()
rqdataClient.init()

updateInfo=[]
barOverViews=database_manager.get_bar_overview()
for barOverView in barOverViews:

#print(f"{barOverView.symbol} start:{barOverView.start} end:{barOverView.end} count:{barOverView.count}")
updateInfo.append((barOverView.symbol,barOverView.exchange,barOverView.end,barOverView.interval))


endDt=datetime.now()

for info in updateInfo:
print(f'更新:{info[0]},Interval:{info[3]}')
req=HistoryRequest(
symbol=info[0],
exchange=info[1],
start=info[2],
end=endDt,
interval=info[3],
)
rData=rqdataClient.query_history(req)
database_manager.save_bar_data(rData)

© 2015-2022 上海韦纳软件科技有限公司
备案服务号:沪ICP备18006526号

沪公网安备 31011502017034号

【用户协议】
【隐私政策】
【免责条款】