vn.py量化社区
By Traders, For Traders.
Member
avatar
加入于:
帖子: 71
声望: 0

变通解决方法:将bardata时间偏离设置为7小时54分。即utc_86 = timezone(timedelta(hours=7,minutes=54)),这样读取出来的数据分钟就是正常的了
csv导入MongoDB代码如下:

from vnpy.trader.constant import (Exchange, Interval)
import pandas as pd
from vnpy.trader.database import database_manager
from vnpy.trader.object import (BarData,TickData)
from datetime import datetime, timedelta, timezone

# 中国时区是+8,对应参数hours=8
# utc_8 = timezone(timedelta(hours=8))
utc_86 = timezone(timedelta(hours=7,minutes=54))#变通
# datetime=row.datetime.replace(tzinfo=utc_8)

import pytz
tz = pytz.timezone('Asia/Shanghai')
print('tz=',tz)

# 封装函数
def move_df_to_mongodb(imported_data:pd.DataFrame,collection_name:str):
    bars = []
    start = None
    count = 0

    for row in imported_data.itertuples():

        bar = BarData(

              symbol=row.symbol,
              exchange=row.exchange,
              # datetime=tz.localize(row.datetime),
              datetime=row.datetime.replace(tzinfo=utc_86),
              interval=row.interval,
              volume=row.volume,
              open_price=row.open,
              high_price=row.high,
              low_price=row.low,
              close_price=row.close,
              open_interest=row.open_interest,
              gateway_name="DB",

        )

        bars.append(bar)

        # do some statistics
        count += 1
        if not start:
            start = bar.datetime
            print ('start=',start)
    end = bar.datetime

    # insert into database
    database_manager.save_bar_data(bars,collection_name)
    print(f'Insert Bar: {count} from {start} - {end}')


if __name__ == "__main__":

    # 读取需要入库的csv文件,该文件是用gbk编码
    imported_data = pd.read_csv('IH99_20101127_20201127_2.csv',encoding='gbk')
    # 将csv文件中 `市场代码`的 SC 替换成 Exchange.SHFE SHFE
    imported_data['exchange'] = Exchange.CFFEX
    # 增加一列数据 `inteval`,且该列数据的所有值都是 Interval.MINUTE
    imported_data['interval'] = Interval.MINUTE
    # 明确需要是float数据类型的列
    float_columns = ['open','high','low','close','volume','open_interest']
    for col in float_columns:
      imported_data[col] = imported_data[col].astype('float')
    # 明确时间戳的格式
    # %Y/%m/%d %H:%M:%S 代表着你的csv数据中的时间戳必须是 2020/05/01 08:32:30 格式
    datetime_format = '%Y%m%d %H:%M:%S'
    imported_data['datetime'] = pd.to_datetime(imported_data['datetime'],format=datetime_format)

    品种代码='IH9902'
    imported_data['symbol'] = 品种代码
    # 因为没有用到 成交额 这一列的数据,所以该列列名不变
    # imported_data.columns = ['exchange','symbol','datetime','open','high','low','close','volume','成交额','open_interest','interval']
    # imported_data = imported_data.rename(index=str,
    #                            columns={"时间": "datetime",
                                        # "KQ.i@CFFEX.T.high": "high",
                                        # "KQ.i@CFFEX.T.low": "low",
                                        # "KQ.i@CFFEX.T.close": "close",
                                        # "KQ.i@CFFEX.T.volume": "volume",
                                        # "KQ.i@CFFEX.T.close_oi": "open_interest",
                                        # })
    # 筛选展示的列名
    # imported_data = imported_data[["datetime","open", "high", "low", "close", "open_interest", "volume"]]
    print('//',imported_data.head(1).append(imported_data.tail(1)),"//")


    move_df_to_mongodb(imported_data,品种代码)

+
+
+
+

Member
avatar
加入于:
帖子: 71
声望: 0

数据库读取代码(无内部for循环)如下:

def load_select_bar_data_from_mongodb(
        # self,
        symbol: str,
                                      exchange: Exchange,
                                      interval: Interval,
                                      start: datetime,
                                      end: datetime,
                                      collection_name: str = None):

    data01=database_manager.load_bar_data(symbol, exchange, interval, start, end, collection_name)
    df = pd.DataFrame(data01)
    df = df.rename(index=str, columns={"open_price": "open",
                                       "high_price": "high",
                                       "low_price": "low",
                                       "close_price": "close",
                                       })
    data_select_columns = ["open", "high", "low", "close", "open_interest", "volume"]
    df = df.set_index('datetime')[data_select_columns]
    print(df)
    return df
Member
avatar
加入于:
帖子: 71
声望: 0

如果大佬们有不用变通的方法 还请斧正分享

关于时区被格式化的位置在如下这行代码里
data01=database_manager.load_bar_data(symbol, exchange, interval, start, end, collection_name)
打印data01[0] 如下:

data01[0] BarData(gateway_name='DB', symbol='IH9902', exchange=<Exchange.CFFEX: 'CFFEX'>, datetime=datetime.datetime(2020, 11, 26, 9, 37, tzinfo=<DstTzInfo 'Asia/Shanghai' LMT+8:06:00 STD>), interval=<Interval.MINUTE: '1m'>, volume=997.0, open_interest=68655.0, open_price=3412.2271, high_price=3419.2006, low_price=3411.4094, close_price=3417.9971)

++
+
+
尽管这里显示bardata的时区是tzinfo=<DstTzInfo 'Asia/Shanghai' LMT+8:06:00 STD>,但是经过pd.DataFrame(data01)之后,时间戳会被调整为正常如下:

//                                 open       high        low      close  open_interest  volume
datetime                                                                                    
2020-11-26 09:31:00+08:00  3412.2271  3419.2006  3411.4094  3417.9971        68655.0   997.0
2020-11-27 15:00:00+08:00  3501.7338  3506.6643  3501.5295  3503.8044        74572.0   749.0 //

恍恍惚惚。。哈哈
复现上面结果的前提是必须把存入数据时的时区信息设为utc_86 = timezone(timedelta(hours=7,minutes=54))
不然最后的结果是偏离的

Administrator
avatar
加入于:
帖子: 4595
声望: 262

这个偏离6分钟的问题,在三四个版本前就修复了,现在2.1.7还有吗?

Member
avatar
加入于:
帖子: 71
声望: 0

老大 不好意思 才看到

详见
https://www.vnpy.com/forum/topic/5155-datamanagermo-kuai-du-qu-mongodbdu-dao-liao-shu-ju-ku-li-suo-mei-you-de-shi-jian-chuo?page=2#pid18228

的22楼

xiaohe:是database_mongo.py里to_bar函数对datetime的处理导致的,改成datetime=DB_TZ.localize(self.datetime)就可以了。这个问题会在下个版本修复的。
+
我看到2.1.8中已修复这个读取问题!

datetime=DB_TZ.localize(self.datetime),

© 2015-2019 上海韦纳软件科技有限公司
备案服务号:沪ICP备18006526号-3

沪公网安备 31011502017034号