最终解决方案:使用源码安装modelscope
这里写目录标题
- 问题描述:pip安装包冲突
- 安装步骤
问题描述:pip安装包冲突
一开始的是在3.11的虚拟环境下使用命令行pip install "modelscope[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
下载,一直报错显示包冲突
并且是这种报错,根本没办法调和。
modelscope[nlp] 1.9.0 depends on datasets<=2.13.0 and >=2.8.0
modelscope[nlp] 1.8.4 depends on datasets<=2.13.0 and >=2.8.0
modelscope[nlp] 1.8.1 depends on datasets<=2.13.0 and >=2.8.0
modelscope[nlp] 1.6.1 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.6.0 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.5.1 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.4.3 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.4.1 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.3.1 depends on datasets<=2.8.0 and >=2.7.0
modelscope[nlp] 1.1.4 depends on datasets<=2.5.2
modelscope[nlp] 1.1.3 depends on datasets<=2.5.2
modelscope[nlp] 1.1.2 depends on datasets<=2.5.2
modelscope[nlp] 1.1.1 depends on datasets<=2.5.2
modelscope[nlp] 1.1.0 depends on datasets<=2.5.2
modelscope[nlp] 1.0.4 depends on datasets<=2.5.2
modelscope[nlp] 1.0.3 depends on datasets<=2.5.2
modelscope[nlp] 1.0.2 depends on datasets<=2.5.2
modelscope[nlp] 1.0.1 depends on datasets<=2.5.2
modelscope[nlp] 0.3.7 depends on datasets==2.1.0
modelscope[nlp] 0.3.6 depends on datasets==2.1.0
modelscope[nlp] 0.3.5 depends on datasets==2.1.0
modelscope[nlp] 0.3.4 depends on datasets==2.1.0
后来在官网找到环境安装的说明:https://www.modelscope.cn/docs/环境安装
环境配置Python版本只显示了3.7和3.8,于是重新创建虚拟环境进行安装。
首先安装深度学习框架,看你的模型是需要pytorch还是tensorflow,根据官网下载对应版本
后面继续pip install "modelscope[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
进行安装但是出现相同问题,于是尝试使用通过源码安装
安装步骤
以下是安装全过程:
-
创建虚拟环境并激活
conda create -n modelscope python=3.8 conda activate modelscope
-
安装深度学习框架
Pytorch:
pip install torch torchvision torchaudio
或Tensorflow:
pip install --upgrade tensorflow==2.13.0 # 仅支持 CPU 的版本
注: 客观上在国内使用pip安装的时候,如果默认是用海外的pypi源的话,可能因为网络问题,下载速度受限。如果有这个现象,可以考虑通过pip的"-i"命令行选项来手工配置仓库来源,例如"-i https://pypi.tuna.tsinghua.edu.cn/simple " 可以将配置仓库来源使用"清华源"。例如:
pip3 install torch torchvision torchaudio -i https://pypi.tuna.tsinghua.edu.cn/simple
常见的可用源还包括 “-i https://mirrors.bfsu.edu.cn/pypi/web/simple”, "-i https://mirrors.ustc.edu.cn/pypi/web/simple "等等,可以根据自己的网络条件自行选择。
-
ModelScope Library 安装(源码安装)
我使用pip安装失败遂不介绍,仅介绍源码安装
-
源码下载
a. 可以使用git下载源码```jsx git clone git@github.com:modelscope/modelscope.git cd modelscope git fetch origin master git checkout master ```
b. 直接在github官网下载 https://github.com/modelscope/modelscope
-
一定要先cd去源码所在的目录中先,否则会报错
可以使用命令行检查是否转到对应目录下:
dir type setup.py
-
安装依赖:仅需体验
NLP
领域模型,执行如下命令安装依赖pip install ".[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
期间他会提示缺少某些包而报错,根据提示单独安装对应的包即可
如果有安装不成功的在后面加**
--trusted-host [pypi.mirrors.ustc.edu.cn](http://pypi.mirrors.ustc.edu.cn)
** 以避免潜在的安全警告pip install xxx --trusted-host pypi.mirrors.ustc.edu.cn pip install ".[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html --trusted-host pypi.mirrors.ustc.edu.cn
-
验证:安装成功后,即可使用对应领域模型进行推理,训练等操作。这里我们以NLP领域为例。安装后,可执行如下命令,运行中文分词任务,来验证安装是否正确:
python -c "from modelscope.pipelines import pipeline;print(pipeline('word-segmentation')('今天天气不错,适合 出去游玩'))"
(modelscope) E:\0.University\0.dissertation\code\modelscope\modelscope>python -c "from modelscope.pipelines import pipeline;print(pipeline('word-segmentation')('今 天天气不错,适合 出去游玩'))" 2024-03-21 10:23:41,578 - modelscope - INFO - PyTorch version 1.11.0+cpu Found. 2024-03-21 10:23:41,580 - modelscope - INFO - Loading ast index from C:\Users\ammy\.cache\modelscope\ast_indexer 2024-03-21 10:23:42,128 - modelscope - INFO - Loading done! Current index file version is 1.9.4, with md5 d07f5c3e9ea59df520487198f306c0b6 and a total number of 972 components indexed 2024-03-21 10:23:46,043 - modelscope - INFO - Model revision not specified, use default: master in development mode 2024-03-21 10:23:46,044 - modelscope - INFO - Development mode use revision: master Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 243k/243k [00:00<00:00, 987kB/s] Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 976/976 [00:00<00:00, 194kB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 161/161 [00:00<00:00, 40.4kB/s] Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 46.7k/46.7k [00:00<00:00, 413kB/s] Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 388M/388M [02:29<00:00, 2.71MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.74k/9.74k [00:00<00:00, 1.99MB/s] Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30.0/30.0 [00:00<00:00, 7.54kB/s] Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 659kB/s] 2024-03-21 10:26:20,450 - modelscope - INFO - initiate model from C:\Users\ammy\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base 2024-03-21 10:26:20,450 - modelscope - INFO - initiate model from location C:\Users\ammy\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base. 2024-03-21 10:26:20,454 - modelscope - INFO - initialize model from C:\Users\ammy\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base You are using a model of type bert to instantiate a model of type structbert. This is not supported for all configurations of models and can yield errors. 2024-03-21 10:26:22,373 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-21 10:26:22,373 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-21 10:26:22,373 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': 'C:\\Users\\amm y\\.cache\\modelscope\\hub\\damo\\nlp_structbert_word-segmentation_chinese-base'}. trying to build by task and model information. 2024-03-21 10:26:22,392 - modelscope - INFO - cuda is not available, using cpu instead. 2024-03-21 10:26:22,397 - modelscope - WARNING - No preprocessor field found in cfg. 2024-03-21 10:26:22,397 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file. 2024-03-21 10:26:22,397 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': 'C:\\Users\\amm y\\.cache\\modelscope\\hub\\damo\\nlp_structbert_word-segmentation_chinese-base', 'sequence_length': 512}. trying to build by task and model information. E:\anaconda\envs\modelscope\lib\site-packages\transformers\modeling_utils.py:977: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Tr ansformers. warnings.warn( {'output': ['今天', '天气', '不错', ',', '适合', '出去', '游玩']}
-
参考:
https://www.modelscope.cn/docs/环境安装
https://blog.csdn.net/qq_41185868/article/details/127355894