vn.py量化社区
By Traders, For Traders.
Member
avatar
加入于:
帖子: 3
声望: 0

我已经实现了指定参数回测,目前想在centos服务器中使用run_optimization优化参数,数据存在sql中,但是在优化的过程中,报了如下错误:
2019-11-08 06:28:14.085567 开始加载历史数据
2019-11-08 06:28:14.085616 开始加载历史数据
2019-11-08 06:28:14.087241 开始加载历史数据
2019-11-08 06:28:14.090270 开始加载历史数据
2019-11-08 06:28:14.093074 开始加载历史数据
2019-11-08 06:28:14.093648 开始加载历史数据
2019-11-08 06:28:14.094640 开始加载历史数据
2019-11-08 06:28:14.095641 开始加载历史数据
2019-11-08 06:28:14.096652 开始加载历史数据
2019-11-08 06:28:14.097428 开始加载历史数据
2019-11-08 06:28:14.097820 开始加载历史数据
2019-11-08 06:28:14.098357 开始加载历史数据
2019-11-08 06:28:14.098623 开始加载历史数据
2019-11-08 06:28:14.098989 开始加载历史数据
2019-11-08 06:28:14.099319 开始加载历史数据
2019-11-08 06:28:14.100377 开始加载历史数据
2019-11-08 06:28:14.100769 开始加载历史数据
2019-11-08 06:28:14.100800 开始加载历史数据
2019-11-08 06:28:14.100912 开始加载历史数据
2019-11-08 06:28:14.102213 开始加载历史数据
2019-11-08 06:28:14.102305 开始加载历史数据
2019-11-08 06:28:14.102337 开始加载历史数据
2019-11-08 06:28:14.103624 开始加载历史数据
2019-11-08 06:28:14.103794 开始加载历史数据
2019-11-08 06:28:14.103871 开始加载历史数据
2019-11-08 06:28:14.104538 开始加载历史数据
2019-11-08 06:28:14.105307 开始加载历史数据
2019-11-08 06:28:14.105460 开始加载历史数据
2019-11-08 06:28:14.106890 开始加载历史数据
2019-11-08 06:28:14.107045 开始加载历史数据
2019-11-08 06:28:14.107125 开始加载历史数据
2019-11-08 06:28:14.107334 开始加载历史数据
2019-11-08 06:28:14.108514 开始加载历史数据
2019-11-08 06:28:14.108738 开始加载历史数据
2019-11-08 06:28:14.109202 开始加载历史数据
2019-11-08 06:28:14.109435 开始加载历史数据
2019-11-08 06:28:14.110502 开始加载历史数据
2019-11-08 06:28:14.111008 开始加载历史数据
2019-11-08 06:28:14.111807 开始加载历史数据
2019-11-08 06:28:14.112917 开始加载历史数据
2019-11-08 06:28:14.113472 开始加载历史数据
2019-11-08 06:28:14.114094 开始加载历史数据
2019-11-08 06:28:14.114111 开始加载历史数据
2019-11-08 06:28:14.114395 开始加载历史数据
2019-11-08 06:28:14.115164 开始加载历史数据
2019-11-08 06:28:14.115649 开始加载历史数据
2019-11-08 06:28:14.115688 开始加载历史数据
2019-11-08 06:28:14.115822 开始加载历史数据
2019-11-08 06:28:14.116895 开始加载历史数据
2019-11-08 06:28:14.117018 开始加载历史数据
2019-11-08 06:28:14.117201 开始加载历史数据
2019-11-08 06:28:14.117910 开始加载历史数据
2019-11-08 06:28:14.118377 开始加载历史数据
2019-11-08 06:28:14.118579 开始加载历史数据2019-11-08 06:28:14.118625 开始加载历史数据
2019-11-08 06:28:14.119860 开始加载历史数据
2019-11-08 06:28:14.120031 开始加载历史数据
2019-11-08 06:28:14.120363 开始加载历史数据
2019-11-08 06:28:14.120496 开始加载历史数据
2019-11-08 06:28:14.121739 开始加载历史数据
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 3005, in execute_sql
cursor.execute(sql, params or ())
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 671, in _read_packet
% (packet_number, self._next_seq_id))
pymysql.err.InternalError: Packet sequence number wrong - got 56 expected 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(args, **kwds))
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 1163, in optimize
engine.load_data()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 240, in load_data
end
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 1210, in load_bar_data
symbol, exchange, interval, start, end
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/trader/database/database_sql.py", line 352, in load_bar_data
data = [db_bar.to_bar() for db_bar in s]
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 6655, in iter
self.execute()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 1812, in inner
return method(self, database,
args, *kwargs)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 1883, in execute
return self._execute(database)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 2054, in _execute
cursor = database.execute(self)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 3018, in execute
return self.execute_sql(sql, params, commit=commit)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 3012, in execute_sql
self.commit()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 2783, in exit
reraise(new_type, new_type(
exc_args), traceback)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/peewee.py", line 3005, in execute_sql
cursor.execute(sql, params or ())
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pymysql/connections.py", line 671, in _read_packet
% (packet_number, self._next_seq_id))
peewee.InternalError: Packet sequence number wrong - got 56 expected 1
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-c7a214135f1e>", line 1, in <module>
runfile('/home/ec2-user/zhanghe/quant/cryptocurrency_low_freq/demo.py', wdir='/home/ec2-user/zhanghe/quant/cryptocurrency_low_freq')
File "/home/ec2-user/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/home/ec2-user/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/ec2-user/zhanghe/quant/cryptocurrency_low_freq/demo.py", line 56, in <module>
main()
File "/home/ec2-user/zhanghe/quant/cryptocurrency_low_freq/demo.py", line 50, in main
demo.optimization()
File "/home/ec2-user/zhanghe/quant/cryptocurrency_low_freq/demo.py", line 44, in optimization
self.engine.run_optimization(self.setting)
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 555, in run_optimization
result_values = [result.get() for result in results]
File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 555, in <listcomp>
result_values = [result.get() for result in results]
File "/home/ec2-user/anaconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
peewee.InternalError: Packet sequence number wrong - got 56 expected 1

因为我的服务器是一台4核服务器,利用multiprocessing.cpu_count()得到cpu的个数为4,理论上不会一次性出现这么多开始加载数据,所以想咨询一下这个问题

Administrator
avatar
加入于:
帖子: 1868
声望: 85
  1. 直接跑回测是否能正常运行?
  2. 确实一开始启动的进程数太多了,请问用的什么版本的vn.py?
Member
avatar
加入于:
帖子: 3
声望: 0

用Python的交易员 wrote:

  1. 直接跑回测是否能正常运行?
  2. 确实一开始启动的进程数太多了,请问用的什么版本的vn.py?
    使用的是最新版本的vnpy,跑单个策略的回测可以正常运行,但是多进程跑多个参数时,会出现如上问题,我是在centos上跑的,数据都放在了sql中,我觉得是pymysql在多进程环境下调用sql数据库可能出现了问题,不知道老师方便试一下从sql取数据在服务器上运行多进程优化吗?
Administrator
avatar
加入于:
帖子: 1868
声望: 85

嗯,这个看着像是个连接池导致的问题,请在Github开个issue吧,我们后续安排人来测试下

Member
avatar
加入于:
帖子: 3
声望: 0

randomzhang wrote:

用Python的交易员 wrote:

  1. 直接跑回测是否能正常运行?
  2. 确实一开始启动的进程数太多了,请问用的什么版本的vn.py?
    使用的是最新版本的vnpy,跑单个策略的回测可以正常运行,但是多进程跑多个参数时,会出现如上问题,我是在centos上跑的,数据都放在了sql中,我觉得是pymysql在多进程环境下调用sql数据库可能出现了问题,不知道老师方便试一下从sql取数据在服务器上运行多进程优化吗?
    issue已提,我后面在从sql中取数据的函数上下增加了锁,问题已经解决了,已经在issue中说明,麻烦老师看一下。
© 2015-2019 上海韦纳软件科技有限公司
备案服务号:沪ICP备18006526号-3