VeighNa量化社区
你的开源社区量化交易平台
Member
avatar
加入于:
帖子: 23
声望: 0

今天发现一个很奇怪的问题。

从 vntrader 中的CTA回测模块中,进行多进程优化没有问题。
但是自己从 examples 中的 jupyter notebook中进行 backtesting, 源代码是遗传算法,简单更改为 多进行优化的时候,就出现错误。 是发生再多进程 load_data 的时候,访问 sqlite3 的 fetch_one() 出错。

1。同样的代码从 UI 中运行就没有问题, 也是同时读取 sqlite3
2。自己写成单独的脚本运行,多进程读取的时候很快就出问题。

请教一下,大家有没有遇到过?

Member
avatar
加入于:
帖子: 23
声望: 0

我最终的办法是给 load_data 加了个全局锁。
这样问题解决了,但是还是不知道原因。

我的电脑是 6 cores, 对于前6个任务,是没有多进程加速的。

Member
avatar
加入于:
帖子: 23
声望: 0

我的电脑是 6 cores, 对于前6个任务,是没有多进程加速的 (这个是全局锁导致的)

Administrator
avatar
加入于:
帖子: 4500
声望: 320

请贴下完整的报错截图信息?

Member
avatar
加入于:
帖子: 23
声望: 0

好的。 下面是我不加锁时的错误信息。
sqlite3 的数据库是没有问题的。加锁之后,就一直好用。

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 3005, in execute_sql
cursor.execute(sql, params or ())
sqlite3.DatabaseError: database disk image is malformed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(args, **kwds))
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 1204, in optimize
engine.load_data()
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 243, in load_data
end
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 1252, in load_bar_data
symbol, exchange, interval, start, end
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/trader/database/database_sql.py", line 352, in load_bar_data
data = [db_bar.to_bar() for db_bar in s]
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 6655, in iter
self.execute()
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 1812, in inner
return method(self, database,
args, *kwargs)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 1883, in execute
return self._execute(database)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 2054, in _execute
cursor = database.execute(self)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 3018, in execute
return self.execute_sql(sql, params, commit=commit)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 3012, in execute_sql
self.commit()
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 2783, in exit
reraise(new_type, new_type(
exc_args), traceback)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/peewee.py", line 3005, in execute_sql
cursor.execute(sql, params or ())
peewee.DatabaseError: database disk image is malformed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/zlu/Work/vnpy/test/vnpy/tui_apps/cta_backtesting.py", line 59, in <module>
df = engine.run_optimization(setting)
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 572, in run_optimization
result_values = [result.get() for result in results]
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/site-packages/vnpy/app/cta_strategy/backtesting.py", line 572, in <listcomp>
result_values = [result.get() for result in results]
File "/home/zlu/anaconda3/envs/py37/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
peewee.DatabaseError: database disk image is malformed

Administrator
avatar
加入于:
帖子: 4500
声望: 320

https://github.com/passiomatic/coldsweat/issues/104

查了下网上的资料,这应该是个Linux系统上SQlite老版本的问题,建议升级下试试

Member
avatar
加入于:
帖子: 4
声望: 0

我的问题是多进程优化似乎没有充分调动cpu, 而且非常慢
description

Member
avatar
加入于:
帖子: 4618
声望: 284

请问你的版本是2.4.0吗?
可以pip install --upgrade vnpy_ctabacktester试试看

Member
avatar
加入于:
帖子: 2
声望: 0

xiaohe wrote:

请问你的版本是2.4.0吗?
可以pip install --upgrade vnpy_ctabacktester试试看
是2.4.0,现在成了是回测中途跑了一半会崩溃

Member
avatar
加入于:
帖子: 4618
声望: 284

请问这次优化cpu跑满了吗?
请问是回测就崩溃了,还是优化崩溃了。如果回测就崩溃了可以用run.py或者在cmd用python -m vnstation打开VN Trader然后复现你的操作看看底层是否有报错信息

© 2015-2022 上海韦纳软件科技有限公司
备案服务号:沪ICP备18006526号

沪公网安备 31011502017034号

【用户协议】
【隐私政策】
【免责条款】