python3.9 安装 pyspider

news2024/11/13 16:43:15

安装pyspider

 pip  install pyspider

直接报错
Please specify --curl-dir=/path/to/built/libcurl

于是从PythonLibs官网中获取依赖并自行下载到本地

下载与3.9对应的

执行安装

pip install D:\下载\ad3-2.2.1-cp39-cp39-win32.whl

安装成功之后执行 pip install pyspider ，提示安装成功，运行 pyspider 报错。

Traceback (most recent call last):
  File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 4, in <module>
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 231
    async=True, get_object=False, no_input=False):
    ^
SyntaxError: invalid syntax

打开run.py 把async 替换成async1

继续执行pyspider ，还是报错

aceback (most recent call last):
  File "D:\Python39\lib\threading.py", line 980, in _bootstrap_inner
    self.run()
  File "D:\Python39\lib\threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 236, in fetcher
    Fetcher = load_cls(None, None, fetcher_cls)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 48, in load_cls
    return utils.load_object(value)
  File "D:\Python39\lib\site-packages\pyspider\libs\utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "D:\Python39\lib\site-packages\pyspider\fetcher\__init__.py", line 1, in <module>
    from .tornado_fetcher import Fetcher
  File "D:\Python39\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 81
    def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):

打开 tornado_fetcher.py 把async 替换成async1 ，在执行pyspider，成功了，不再报错如下：

遇到如下错误：


  File "\python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 135, in __init__
    _check_config(config)
  File "python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 119, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_control

解决方案：

找到文件找到domaincontroller,这一行按照提示修改，但是依然报错，我就直接删除了，删除之后启动就报如下错误。

TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst

我看到有的帖子说是wsgidav 版本太高，是的就是版本高了，高版本中的方法变了，于是乎复制 getResourceInst 改名为get_resource_inst 增加一下实现

于是重新执行 pyspider all 报错如下：

  File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 7, in <module>
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 754, in main
    cli()
  File "D:\Python39\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "D:\Python39\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 384, in webui
    app.run(host=host, port=port)
  File "D:\Python39\lib\site-packages\pyspider\webui\app.py", line 64, in run
    from werkzeug.dispatcher import DispatcherMiddleware
ModuleNotFoundError: No module named 'werkzeug.dispatcher'

依然是安装了新版本之后依赖包找不到了，于是找到 site-packages\werkzeug 安装目录发现werkzeug\middleware 下有个dispatcher.py 中有想要的类，于是就把webui 的引用包改了一下

        if dav_app:
            from werkzeug.middleware.dispatcher import DispatcherMiddleware
            application = DispatcherMiddleware(application, {
                '/dav': dav_app
            })

        container = tornado.wsgi.WSGIContainer(application)

于是乎就愉快的跑起来了。