vn.py量化社区
By Traders, For Traders.
Member
avatar
加入于:
帖子: 165
声望: 35

先上使用说明:

  1. future_download状态,下载期货数据时设置为True,下载股票数据时设置为False
  2. file_path:通达信数据保存路径大家自行替换
  3. 通达信期货数据对齐datetime到文华财经后数据与文化财经完全一致,包括指数合约
  4. 单个文件较大时多进程只有两个进程在运行,原因不明
  5. 通达信股票只能下载最近100天数据,期货数据下载没有时间限制
  6. 期货数据存储路径:D:\tdx\vipdoc\ds,上交所股票数据路径:D:\tdx\vipdoc\sh,深交所股票数据路径:D:\tdx\vipdoc\sz
  7. 建议下载通达信期货通可以同时下载股票和期货数据,enjoy it!
  8. 附上vnpy\trader\engine.py里面的合约数据保存读取代码

    import platform
    import shelve
    if platform.uname().system == "Windows":
      LINK_SIGN = "\\"
    elif platform.uname().system == "Linux":
      LINK_SIGN = "/"
    #--------------------------------------------------------------------------------------------------
    class OmsEngine(BaseEngine):
      contract_file_name = 'contract_data'
      contract_file_path = get_folder_path(contract_file_name)
      def __init__(self, main_engine: MainEngine, event_engine: EventEngine):
          """"""
          super(OmsEngine, self).__init__(main_engine, event_engine, "oms")
          self.ticks = {}
          self.orders = {}
          self.trades = {}
          self.positions = {}
          self.accounts = {}
          self.contracts = {}
    
          self.active_orders = {}
          self.add_function()
          self.load_contracts()
          self.register_event()
      #--------------------------------------------------------------------------------------------------
      def load_contracts(self):
          """读取合约数据"""
          try:
              with shelve.open(f"{self.contract_file_path}{LINK_SIGN}contract_data.vt",writeback=True) as file:
                  if 'data' in file:
                      contract_data = file['data']
                      for key, value in list(contract_data.items()):
                          self.contracts[key] = value
          except:
              return {}
          return self.contracts
      #--------------------------------------------------------------------------------------------------
      def save_contracts(self):
          """
          保存合约数据
          """
          try:
              with shelve.open(f"{self.contract_file_path}{LINK_SIGN}contract_data.vt",writeback=True)  as file:
                  file['data'] = self.contracts 
          except:
              return
      #--------------------------------------------------------------------------------------------------
      def add_function(self):
          """
          为MainEngine添加OmsEngine函数
          """
          self.main_engine.save_contracts = self.save_contracts                   #保存合约参数到硬盘
          self.main_engine.load_contracts = self.load_contracts                   #读取硬盘合约数据

    save_conntracts我是写在CLI交易的子进程和cta_engine里面
    #保存合约数据到硬盘
    main_engine.save_contracts()
    #--------------------------------------------------------------------------------------
    修改vnpy\app\cta_strategy里面的engine.py
    def stop_all_strategies(self):

      """
      停止所有策略
      """
      for strategy_name in self.strategies.keys():
          self.stop_strategy(strategy_name)
      #保存合约数据到本地
      self.main_engine.save_contracts()
Member
avatar
加入于:
帖子: 165
声望: 35

上弦之月 wrote:

from typing import TextIO
from datetime import datetime,timedelta
from datetime import time as dtime
from time import time
import numpy as np
import pandas as pd
import csv
import os
import multiprocessing

from vnpy.trader.database import database_manager
from vnpy.trader.constant import (Exchange, Interval)
from vnpy.trader.object import (BarData, TickData)
from vnpy.trader.utility import (load_json, save_json,extract_vt_symbol)
from peewee import chunked
from vnpy.event import EventEngine
from vnpy.trader.engine import MainEngine
#--------------------------------------------------------------------------------------------
def save_tdx_data(file_path,vt_symbol:str,future_download:bool,interval: Interval = Interval.MINUTE):
    """
    保存通达信导出的lc1分钟数据,期货数据对齐datetime到文华财经
    """
    bars = []
    start_time = None
    count = 0
    time_consuming_start = time()
    symbol,exchange= extract_vt_symbol(vt_symbol)
    #读取二进制文件
    dt = np.dtype([
        ('date', 'u2'),
        ('time', 'u2'),
        ('open_price', 'f4'),
        ('high_price', 'f4'),
        ('low_price', 'f4'),
        ('close_price', 'f4'),
        ('amount', 'f4'),
        ('volume', 'u4'),
        ('reserve','u4')])
    data = np.fromfile(file_path, dtype=dt)
    df = pd.DataFrame(data, columns=data.dtype.names)
    df.eval('''
    year=floor(date/2048)+2004
    month=floor((date%2048)/100)
    day=floor(date%2048%100)
    hour = floor(time/60)
    minute = time%60
    ''',inplace=True)
    df.index=pd.to_datetime(df.loc[:,['year','month','day','hour','minute']])
    df.drop(['date','time','year','month','day','hour','minute',"amount","reserve"],1,inplace=True)
    for index in range(len(df)):
        bar = BarData(
            symbol=symbol,
            exchange=exchange,
            interval=interval,
            open_price=df["open_price"][index],
            high_price=df["high_price"][index],
            low_price=df["low_price"][index],
            close_price=df["close_price"][index],
            volume=df["volume"][index],
            datetime = df.index[index],
            gateway_name = 'DB'
            )
        if future_download:
            if bar.datetime.time() >= dtime(21,0) or bar.datetime.time() <= dtime(2,30):
                bar.datetime -= timedelta(days=1)
            bar.datetime-= timedelta(minutes=1)
        if not start_time:
            start_time = bar.datetime
        bars.append(bar)
        count += 1
        end_time = bar.datetime  
    for bar_data in chunked(bars, 10000):               #分批保存数据
        database_manager.save_bar_data(bar_data)      #保存数据到数据库 
    time_consuming_end =time()
    msg = f'载入通达信标的:{vt_symbol} 分钟数据,开始时间:{start_time},结束时间:{end_time},数据量:{count},耗时:{round(time_consuming_end-time_consuming_start,3)}秒'
    return msg
#--------------------------------------------------------------------------------------------
if __name__ == '__main__':
    file_path = "C:\\ProgramData\\Anaconda3\\Lib\\site-packages\\vnpy-2.1.0-py3.7.egg\\vnpy\\app\\cta_strategy\\tdx数据"
    event_engine = EventEngine()
    main_engine = MainEngine(event_engine)
    contracts = main_engine.load_contracts()
    vt_symbol = ""
    file_names =[]              #  文件名列表
    vt_symbols = [] # vt_symbol列表
    future_download = True #    期货数据下载状态
    pool = multiprocessing.Pool(multiprocessing.cpu_count(), maxtasksperchild=1)
    for dirpath, dirnames, filenames in os.walk(file_path):
        for file_name in filenames:         #当前目录所有文件名
            #过滤压缩文件
            if file_name.split(".")[1] in ["rar","7z"]:
                continue
            if file_name.endswith("lc1"):
                if file_name not in file_names:
                    file_names.append(f"{file_path}\\{file_name}")
                if future_download:
                    symbol = file_name.split(".")[0].split("#")[1]
                    for contract in list(contracts.values()):
                        #指数合约vt_symbol合成,给大家提供个例子
                        if symbol.endswith("L9"):
                            symbol = symbol.split("L9")[0] + "99"
                            shfe_index = ["rb99","hc99"]
                            dce_index = ["i99","j99"]
                            czce_index = ["AP99","ZC99"]
                            if symbol.lower() in shfe_index:
                                vt_symbol = symbol.lower() + "." + "SHFE"
                            elif symbol.lower() in dce_index:
                                vt_symbol = symbol.lower() + "." + "DCE"
                            elif symbol in czce_index:
                                vt_symbol = symbol + "." + "CZCE"
                        else:
                            #合约symbol与文件名相同(满足合约大写或小写)使用合约vt_symbol
                            if contract.symbol in [symbol, symbol.lower()]:
                                vt_symbol = contract.vt_symbol
                else:
                    symbol = file_name.split(".")[0]
                    if symbol.startswith("sh"):
                        exchange_str = "SSE"
                    elif symbol.startswith("sz"):
                        exchange_str = "SZSE"

                    vt_symbol = symbol[-6:] + "." + exchange_str
                if vt_symbol not in vt_symbols:
                    vt_symbols.append(vt_symbol)
    for setting in list(zip(file_names,vt_symbols)):
        setting += (future_download,)
        result = pool.apply_async(save_tdx_data, setting)
        print(result.get())
    pool.close()
    pool.join()
    #保存股票列表
    if not future_download:
        stock_vt_symbols = load_json("stock_vt_symbols.json")
        for vt_symbol in vt_symbols:
            if vt_symbol not in stock_vt_symbols:
                stock_vt_symbols.append(vt_symbol)
        save_json("stock_vt_symbols.json",stock_vt_symbols)
Member
加入于:
帖子: 1
声望: 0

手动点赞!

Member
avatar
加入于:
帖子: 9
声望: 1

from vnpy.app.cta_strategy.usual_method import CheckFutureNews
usual_method 这个没找着,能不能分享一下

Member
avatar
加入于:
帖子: 165
声望: 35

@清风 这个不用导入,file_path设置为通达信数据存放目录就可以了

Member
avatar
加入于:
帖子: 2
声望: 0

2.1.4版本运行出错
AttributeError: 'MainEngine' object has no attribute 'load_contracts'

Member
avatar
加入于:
帖子: 165
声望: 35

@szapha 我更新了,你看下

Member
avatar
加入于:
帖子: 7
声望: 0

@上弦之月 大神,我是中小白。
你写的一楼内容是添加到trader\engine.py里面。意义是不是实时行情的时候能有数据推送?
二楼的内容,是怎么用法(独立程序用,还是加到哪里),望指点一下。

Member
avatar
加入于:
帖子: 15
声望: 1

老大,为什么我返回的一直是一个空的字典啊
description

description

Member
avatar
加入于:
帖子: 15
声望: 1

月总,麻烦看到指点一下!在load_contracts函数里面的if 'data' in file判断里面根本没有data这是为什么呢
我的数据是通达信期货通下载的啊

Member
avatar
加入于:
帖子: 165
声望: 35

@ouzhongliang 我更新了下参考1楼,改完打开vnpy连接交易接口,打开【功能】【CTA策略】等个5分钟左右再关闭vnpy,合约数据就保存了,之后再调用load_contracts

Member
avatar
加入于:
帖子: 15
声望: 1

感谢月总,数据是有了,可是数据不对,我下的是螺纹和白银,我打印vnpy\trader\engine.py里面load_contracts函数的内容它都给我期权,我根本都没下期权的啊,难道它是自动下载的吗?那如果是自动下载的我又怎么才能选我想要下载的symbol数据呢,比如我想下载螺纹连接和白银连续的数据来回测用。
description
我是按照你的方法运行VNPY然后连接CTP然后再启动策略5分钟以上再关闭才有数据的,只是它给我的数据是期权,我想要我自己下的数据,或者如果是自动下载的话要怎么设置才能下载自己想要的symbol数据

Member
avatar
加入于:
帖子: 9
声望: 1

szapha wrote:

2.1.4版本运行出错
AttributeError: 'MainEngine' object has no attribute 'load_contracts'

同样的问题出现,没整明白,有谁点拨一下?

Member
avatar
加入于:
帖子: 15
声望: 1

你是不是也是UI界面启动没反应啊,我也遇到这个问题2.1.4版本的,但是我的没弄了,直接改用以前的版本了

Member
avatar
加入于:
帖子: 1
声望: 0

open_interest 持仓量 有没有?怎么修改呢?谢谢!

Member
avatar
加入于:
帖子: 18
声望: 0

了解一下pytdx,通达信的数据可以直接下载,不用离线导入。

Member
avatar
加入于:
帖子: 165
声望: 35

@leebo 没有持仓量参数,@忘尘 pytdx太烂了

© 2015-2019 上海韦纳软件科技有限公司
备案服务号:沪ICP备18006526号-3