VeighNa量化社区
你的开源社区量化交易平台 | vn.py | vnpy
Member
avatar
加入于:
帖子: 3
声望: 0

开头写这个之前,先报个小白身份,我是前两天才有空开始研究vnpy,知道vn框架已经好久了,但学业各种忙,难得抽空下来深入了解,若有不对之处,请各位指正。
参考了KeKe的《载入Tick数据(csv格式)到数据库中》的逻辑,我在此基础上修改,主要是直接在掘金量化数据库在线调取tick数据载入mongodb数据库中,所以可先参考KeKe
载入Tick数据(csv格式)到数据库中

  1. 直接在掘金量化平台下载最新的客户端,并安装好登陆上去,获取token(在个人中心页面的第一行,不知道这个论坛编辑是怎么回事,我用谷歌和火狐都没办法拖照片上来,懂得可以说说),后面会用到
  2. 打开shell运行python.exe -m pip install gm -U
  3. 导入掘金的api,输入token
from gm.api import *
set_token('输入你的token')
  1. 这里不需要导入csv文件,模块就不需要用到csv
from datetime import time
from vnpy.trader.constant import Exchange,Interval
from vnpy.trader.database import database_manager
from vnpy.trader.object import TickData
  1. 这里拿螺纹主力的tick数据,做个栗子,掘金的tick数据只有三个月的,具体的可以参考掘金官网的api文档
history_data = history(symbol='SHFE.RB', frequency='tick', start_time='2019-08-28',  end_time='2019-12-30', df=True)
history_data.shape
(66000, 13)
  1. 主要选取的字段是created_at、price、quotes[0]['bid_p']、quotes[0]['bid_v']、quotes[0]['ask_p']、quotes[0]['ask_v']
def csv_load(file):
    ticks = []
    start = None
    count = 0
    for item in file.index:
        item = file.loc[item]
        dt = item.created_at.to_pydatetime().replace(tzinfo=None)
        if dt.time() > time(15, 1) and dt.time() < time(20, 59):
            continue
        tick = TickData(
            symbol="RB88",
            datetime=dt,
            exchange=Exchange.SHFE,
            last_price=float(item.price),
            volume=float(item.cum_position),
            bid_price_1=float(item.quotes[0]['bid_p']),
            bid_volume_1=float(item.quotes[0]['bid_v']),
            ask_price_1=float(item.quotes[0]['ask_p']),
            ask_volume_1=float(item.quotes[0]['ask_v']), 
            gateway_name="DB",       
        )
        ticks.append(tick)
        # do some statistics
        count += 1
        if not start:
            start = tick.datetime
    end = tick.datetime
    database_manager.save_tick_data(ticks)
    print("插入数据", start, "-", end, "总数量:", count)  
if __name__ == "__main__":
    csv_load(history_data)  

插入数据 2019-09-27 09:00:00.500000 - 2019-12-05 21:51:37 总数量: 65998

Member
avatar
加入于:
帖子: 3
声望: 0

载入Bar数据(csv格式)到数据库中

  1. 数据源来自新浪财经API获取数据
from urllib import request
import json
import pandas as pd
def get_data(id):
    url_60m = 'http://stock2.finance.sina.com.cn/futures/api/json.php/IndexService.getInnerFuturesMiniKLine60m?symbol='
    url = url_60m + id
    req = request.Request(url)
    rsp = request.urlopen(req)
    res = rsp.read()
    res_json = json.loads(res)
    bar_list = []
    res_json.reverse()
    for line in res_json:
        bar = {}
        bar['Datetime'] = line[0]
        bar['Open'] = float(line[1])
        bar['High'] = float(line[2])
        bar['Low'] = float(line[3])
        bar['Close'] = float(line[4])
        bar['Volume'] = int(line[5])
        bar_list.append(bar)
    df = pd.DataFrame(data=bar_list)
    print(df)
    df.to_csv('./data.csv', index=None)
if __name__ == '__main__':
    get_data('rb1910')
  1. 加载bar的csv到数据库,主要是对载入Tick数据(csv格式)到数据库中作了一点小修改
# encoding: UTF-8
"""
导入CSV历史数据到MongoDB中
"""
import os 
import csv
from datetime import datetime, time
from vnpy.trader.constant import Exchange,Interval
from vnpy.trader.database import database_manager
from vnpy.trader.object import BarData
def run_load_csv():
    """
    遍历同一文件夹内所有csv文件,并且载入到数据库中
    """
    for file in os.listdir("."): 
        if not file.endswith(".csv"): 
            continue  
        print("载入文件:", file)
        csv_load(file,exchange=Exchange.SHFE,interval=Interval.DAILY)
def csv_load(file,exchange,interval,datetime_format='%Y%m%d'):
    """
    读取csv文件内容,并写入到数据库中    
    """
    with open(file, "r") as f:
        reader = csv.DictReader(f)
        bars = []
        start = None
        count = 0
        for item in reader:
            if datetime_format:
                dt = datetime.strptime(item['date'], datetime_format)
            else:
                dt = datetime.fromisoformat(item['date'])
            bar = BarData(
                symbol=item['vtSymbol'],
                exchange=exchange,
                datetime=dt,
                interval=interval,
                volume=item['volume'],
                open_price=item['open'],
                high_price=item['high'],
                low_price=item['low'],
                close_price=item['close'],
                gateway_name="DB",
            )
            bars.append(bar)
            # do some statistics
            count += 1
            if not start:
                start = bar.datetime
        end = bar.datetime
        # insert into database
        database_manager.save_bar_data(bars)
        print("插入数据", start, "-", end, "总数量:", count)  
if __name__ == "__main__":
    run_load_csv() 
Member
avatar
加入于:
帖子: 3
声望: 0

如果我选择用云服务器一直挂着进行更新数据库,最后还是会有bar不断地更新到数据库的需求,但rqdata毕竟是收费的,选择使用掘金的api直接导入也是非常简单廉价的一个选择。
这里又写了一个,掘金量化的bar数据导入到mongodb数据库里面,以2016.01.01-目前的螺纹主力1小时BAR数据导入为例子。

  1. 主要的初始化设置可以参考最上面的Tick数据导入,主要是将
from vnpy.trader.object import TickData

转为

from vnpy.trader.object import BarData
  1. 加载掘金数据到内存,具体参数可以参考history - 查询历史行情
history_data = history(symbol='SHFE.RB', frequency='3600s', start_time='2016-01-01',  end_time='2019-12-30', df=True)
history_data.shape

(5785, 12)

  1. 导入数据库,貌似“小巫见大巫”/捂脸,大咖路过莫笑。
def csv_load(file,exchange,interval):
    bars = []
    start = None
    count = 0
    for item in file.index:
        item = file.loc[item]
        dt = item.eob.to_pydatetime().replace(tzinfo=None)
        bar = BarData(
            symbol='RB88',
            exchange=exchange,
            datetime=dt,
            interval=interval,
            volume=item['volume'],
            open_price=item['open'],
            high_price=item['high'],
            low_price=item['low'],
            close_price=item['close'],
            gateway_name="DB",
        )
        bars.append(bar)
        count += 1
        if not start:
            start = bar.datetime
    end = bar.datetime
    database_manager.save_bar_data(bars)
    print("插入数据", start, "-", end, "总数量:", count)  
if __name__ == "__main__":
    csv_load(history_data,Exchange.SHFE,Interval.HOUR) 

插入数据 2016-01-04 10:00:00 - 2019-12-27 23:00:00 总数量: 5785