FreeChat
Open main menu
ChatGPT知识
搜索
搜索
关闭
Previous
Previous
Next
Next
chatGLM+LangChain（部署版）：基于本地知识库的自动问答系统
sockstack
/
155
/
2023-11-06 01:13:10
<p><span style="color: red; font-size: 18px">ChatGPT 可用网址，仅供交流学习使用，如对您有所帮助，请收藏并推荐给需要的朋友。</span><br><a href="https://ckai.xyz/?sockstack§ion=detail" target="__blank">https://ckai.xyz</a><br><br></p> <div class="notebook-project-preview-content" data-active="true" data-type="notebook"><div class="ai-np"><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><p>项目依赖PaddlePaddle develolop版本和最新的PaddleNLP</p> </div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h1>0 官方视频</h1> <h3></h3> <iframe border="0" src="//player.bilibili.com/player.html?aid=911759112&bvid=BV13M4y1e7cN&cid=1149491440&page=1" style="width:98%;height: 450px;"> </iframe> <h1>1 可以直接部署，不用安装下面的包</h1> <h2>1.1gradio文件部署路径</h2> <p>LangChain-ChatGLM-Webui/paddlepaddle**/untitled.gradio.py**</p> <p><strong>总共部署需要25分钟以上，以下是第一条日志和最后一条日志。</strong></p> <p>2023-06-01T09:49:15.375725800Z start deploy...</p> <p>2023-06-01T10:15:09.812367777Z [33m[2023-06-01 10:15:09,812] [ WARNING][0m - Some weights of ChatGLMForConditionalGeneration were not initialized from the model</p> <h2>1.2 部署后测试文件</h2> <p><strong>根目录下文件：知识库.txt</strong></p> <p><img referrerpolicy="no-referrer" alt="" src="https://ai-studio-static-online.cdn.bcebos.com/ed519a5ae050417fae8ca0ea83d7e4c416b9820b7fdd46a3b8ad895893a69822"></p> </div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h1>2 原理介绍</h1> <p>【官方官方教程】ChatGLM + LangChain 实践培训 B站视频，感兴趣请自行搜索。</p> <h2>2.1 应用背景及需求特点</h2> <p><img referrerpolicy="no-referrer" alt="" src="https://ai-studio-static-online.cdn.bcebos.com/9c6e533d6fdf4432870283e2838238eeaa9cdd4e505e4c75b6b6858bb2282fa2"></p> </div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h2>2.2 LangChain及ChatGLM简单介绍</h2> <h3>2.2.1 LangChain简介</h3> <p>LangChain是一个强大的框架，旨在帮助开发人员使用语言模型构建端到端的应用程序。</p> <ol> <li> <p>它提供了一套工具、组件和接口，可简化创建由大型语言模型(LLM) 和聊天模型提供支持的应用程序的过程。</p> </li> <li> <p>LangChain 可以轻松管理与语言模型的交互，将多个组件链接在一起，并集成额外的资源，例如API 和数据库。</p> </li> </ol> <p><strong>核心概念：</strong></p> <p>• Components and Chains</p> <p>• Prompt Templates and Values</p> <p>• Example Selectors</p> <p>• Output Parsers</p> <p>• Indexes and Retrievers</p> <p>• Chat Message History</p> <p>• Agents and Toolkits</p> <h3>2.2.2 ChatGLM-6B简绍</h3> <ul> <li> <p>ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于General LanguageModel (GLM) 架构，具有62 亿参数。</p> </li> <li> <p>结合模型量化技术，用户可以在消费级的显卡上进行本地部署（INT4 量化级别下最低只需6GB 显存）。</p> </li> </ul> <p><img referrerpolicy="no-referrer" alt="" src="https://ai-studio-static-online.cdn.bcebos.com/c89c2c8e1f33461bb4ebe2d7a886b01c59558fe1fdd04c858fcd2e84eaae3554"></p> <ul> <li>ChatGLM-6B 自3月14号发布以来受到了广大开发者和用户的喜爱，截至4月23号GitHub 的star 数达到2 万，累计下载量过100 万，并连续12 天居Hugging Face (HF) 全球大模型下载榜第一名。</li> </ul> <p><img referrerpolicy="no-referrer" alt="" src="https://ai-studio-static-online.cdn.bcebos.com/31abe91c1b654524b9bdfce51121fa562fd1f9bc885848589bf74ad7d7e12f8c"></p> </div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h2>2.3 Langchain+ChatGLM的原理解析</h2> <p><img referrerpolicy="no-referrer" alt="" src="https://ai-studio-static-online.cdn.bcebos.com/bc7c3c7412354f3c8ba4d8c54bcdfffb51b2946afc9f4374afb4e281b0d3e2ef"></p> </div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h1>3 下面是支持本地安装和调试</h1> <h2>3.1 安装最新版paddlenlp</h2> </div></div><div class="ai-np-code"><div class="ai-np-code-in">In [1]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 安装paddlepaddle-gpu develop版本</span> !python -m pip install paddlepaddle-gpu==<span class="hljs-number">0.0</span><span class="hljs-number">.0</span>.post112 -f https://www.paddlepaddle.org.cn/whl/linux/gpu/develop.html</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Looking in links: https://www.paddlepaddle.org.cn/whl/linux/gpu/develop.html Collecting paddlepaddle-gpu==0.0.0.post112 Downloading https://paddle-wheel.bj.bcebos.com/develop/linux/linux-gpu-cuda11.2-cudnn8-mkl-gcc8.2-avx/paddlepaddle_gpu-0.0.0.post112-cp39-cp39-linux_x86_64.whl (575.0 MB) <span style="color:rgb(85,85,85)">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style="color:rgb(0,187,0)">575.0/575.0 MB</span> <span style="color:rgb(187,0,0)">1.9 MB/s</span> eta <span style="color:rgb(0,187,187)">0:00:00</span>00:0100:01 Requirement already satisfied: protobuf>=3.20.2 in ./.data/webide/pip/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (3.20.2) Requirement already satisfied: numpy>=1.13 in ./.data/webide/pip/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (1.22.4) Requirement already satisfied: Pillow in ./.data/webide/pip/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (9.5.0) Requirement already satisfied: opt-einsum==3.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (3.3.0) Requirement already satisfied: httpx in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (0.23.3) Requirement already satisfied: decorator in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (4.4.2) Requirement already satisfied: astor in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (0.8.1) Requirement already satisfied: paddle-bfloat==0.1.7 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlepaddle-gpu==0.0.0.post112) (0.1.7) Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx->paddlepaddle-gpu==0.0.0.post112) (1.5.0) Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx->paddlepaddle-gpu==0.0.0.post112) (0.16.3) Requirement already satisfied: certifi in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx->paddlepaddle-gpu==0.0.0.post112) (2019.9.11) Requirement already satisfied: sniffio in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx->paddlepaddle-gpu==0.0.0.post112) (1.3.0) Requirement already satisfied: h11<0.15,>=0.13 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpcore<0.17.0,>=0.15.0->httpx->paddlepaddle-gpu==0.0.0.post112) (0.14.0) Requirement already satisfied: anyio<5.0,>=3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpcore<0.17.0,>=0.15.0->httpx->paddlepaddle-gpu==0.0.0.post112) (3.6.1) Requirement already satisfied: idna in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from rfc3986[idna2008]<2,>=1.3->httpx->paddlepaddle-gpu==0.0.0.post112) (2.8) Installing collected packages: paddlepaddle-gpu Attempting uninstall: paddlepaddle-gpu Found existing installation: paddlepaddle-gpu 2.4.1.post112 Uninstalling paddlepaddle-gpu-2.4.1.post112: Successfully uninstalled paddlepaddle-gpu-2.4.1.post112 Successfully installed paddlepaddle-gpu-0.0.0.post112 <span style="font-weight:bold">[</span><span style="color:rgb(0,0,187)">notice</span><span style="font-weight:bold">]</span> A new release of pip available: <span style="color:rgb(187,0,0)">22.1.2</span> -> <span style="color:rgb(0,187,0)">23.1.2</span> <span style="font-weight:bold">[</span><span style="color:rgb(0,0,187)">notice</span><span style="font-weight:bold">]</span> To update, run: <span style="color:rgb(0,187,0)">pip install --upgrade pip</span> </pre></div></div></div><div class="ai-np-code"><div class="ai-np-code-in">In [2]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 用镜像直接安装</span> %cd /home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle !pip install ./paddlenlp<span class="hljs-number">-2.5</span><span class="hljs-number">.2</span>.post0-py3-none-<span class="hljs-built_in">any</span>.whl --user</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">/home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Processing ./paddlenlp-2.5.2.post0-py3-none-any.whl Requirement already satisfied: dill<0.3.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.3.3) Requirement already satisfied: datasets>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (2.7.0) Requirement already satisfied: paddlefsl in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (1.1.0) Requirement already satisfied: rich in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (12.6.0) Requirement already satisfied: multiprocess<=0.70.12.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.70.11.1) Requirement already satisfied: typer in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.7.0) Requirement already satisfied: protobuf==3.20.2 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (3.20.2) Requirement already satisfied: seqeval in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (1.2.2) Requirement already satisfied: fastapi in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.90.1) Requirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (4.64.1) Requirement already satisfied: uvicorn in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.20.0) Requirement already satisfied: huggingface-hub>=0.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.12.0) Requirement already satisfied: visualdl in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (2.4.0) Requirement already satisfied: colorama in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.4.4) Requirement already satisfied: colorlog in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (4.1.0) Requirement already satisfied: paddle2onnx in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (1.0.0) Requirement already satisfied: sentencepiece in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.1.96) Requirement already satisfied: jieba in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (0.42.1) Requirement already satisfied: Flask-Babel in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from paddlenlp==2.5.2.post0) (3.0.1) Requirement already satisfied: xxhash in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (3.1.0) Requirement already satisfied: fsspec[http]>=2021.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (2022.11.0) Requirement already satisfied: pyarrow>=6.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (10.0.0) Requirement already satisfied: responses<0.19 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (0.18.0) Requirement already satisfied: packaging in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (21.3) Requirement already satisfied: aiohttp in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (3.8.3) Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (5.1.2) Requirement already satisfied: pandas in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (2.0.1) Requirement already satisfied: requests>=2.19.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (2.24.0) Requirement already satisfied: numpy>=1.17 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from datasets>=2.0.0->paddlenlp==2.5.2.post0) (1.22.4) Requirement already satisfied: filelock in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from huggingface-hub>=0.11.1->paddlenlp==2.5.2.post0) (3.9.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from huggingface-hub>=0.11.1->paddlenlp==2.5.2.post0) (4.5.0) Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from fastapi->paddlenlp==2.5.2.post0) (1.10.4) Requirement already satisfied: starlette<0.24.0,>=0.22.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from fastapi->paddlenlp==2.5.2.post0) (0.23.1) Requirement already satisfied: pytz<2023.0,>=2022.7 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask-Babel->paddlenlp==2.5.2.post0) (2022.7.1) Requirement already satisfied: Jinja2<4.0.0,>=3.1.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask-Babel->paddlenlp==2.5.2.post0) (3.1.2) Requirement already satisfied: Babel<3.0.0,>=2.11.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask-Babel->paddlenlp==2.5.2.post0) (2.11.0) Requirement already satisfied: Flask<3.0.0,>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask-Babel->paddlenlp==2.5.2.post0) (2.1.3) Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from rich->paddlenlp==2.5.2.post0) (2.13.0) Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from rich->paddlenlp==2.5.2.post0) (0.9.1) Requirement already satisfied: scikit-learn>=0.21.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from seqeval->paddlenlp==2.5.2.post0) (0.24.2) Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from typer->paddlenlp==2.5.2.post0) (8.0.0) Requirement already satisfied: h11>=0.8 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from uvicorn->paddlenlp==2.5.2.post0) (0.14.0) Requirement already satisfied: matplotlib in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from visualdl->paddlenlp==2.5.2.post0) (2.2.3) Requirement already satisfied: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from visualdl->paddlenlp==2.5.2.post0) (0.8.53) Requirement already satisfied: Pillow>=7.0.0 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from visualdl->paddlenlp==2.5.2.post0) (9.5.0) Requirement already satisfied: six>=1.14.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from visualdl->paddlenlp==2.5.2.post0) (1.16.0) Requirement already satisfied: Werkzeug>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->Flask-Babel->paddlenlp==2.5.2.post0) (2.1.2) Requirement already satisfied: itsdangerous>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->Flask-Babel->paddlenlp==2.5.2.post0) (2.1.2) Requirement already satisfied: importlib-metadata>=3.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Flask<3.0.0,>=2.0.0->Flask-Babel->paddlenlp==2.5.2.post0) (6.0.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (2.1.1) Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (1.2.0) Requirement already satisfied: attrs>=17.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (22.1.0) Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (1.7.2) Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (6.0.2) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (4.0.2) Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp==2.5.2.post0) (1.3.0) Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from Jinja2<4.0.0,>=3.1.2->Flask-Babel->paddlenlp==2.5.2.post0) (2.0.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from packaging->datasets>=2.0.0->paddlenlp==2.5.2.post0) (3.0.9) Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp==2.5.2.post0) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp==2.5.2.post0) (2.8) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp==2.5.2.post0) (2019.9.11) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp==2.5.2.post0) (1.25.11) Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp==2.5.2.post0) (2.1.0) Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp==2.5.2.post0) (1.2.0) Requirement already satisfied: scipy>=0.19.1 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp==2.5.2.post0) (1.9.1) Requirement already satisfied: anyio<5,>=3.4.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from starlette<0.24.0,>=0.22.0->fastapi->paddlenlp==2.5.2.post0) (3.6.1) Requirement already satisfied: future>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from bce-python-sdk->visualdl->paddlenlp==2.5.2.post0) (0.18.0) Requirement already satisfied: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from bce-python-sdk->visualdl->paddlenlp==2.5.2.post0) (3.9.9) Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from matplotlib->visualdl->paddlenlp==2.5.2.post0) (2.8.2) Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from matplotlib->visualdl->paddlenlp==2.5.2.post0) (1.1.0) Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from matplotlib->visualdl->paddlenlp==2.5.2.post0) (0.10.0) Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from pandas->datasets>=2.0.0->paddlenlp==2.5.2.post0) (2022.7) Requirement already satisfied: sniffio>=1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from anyio<5,>=3.4.0->starlette<0.24.0,>=0.22.0->fastapi->paddlenlp==2.5.2.post0) (1.3.0) Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from importlib-metadata>=3.6.0->Flask<3.0.0,>=2.0.0->Flask-Babel->paddlenlp==2.5.2.post0) (3.8.1) Requirement already satisfied: setuptools in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from kiwisolver>=1.0.1->matplotlib->visualdl->paddlenlp==2.5.2.post0) (56.2.0) paddlenlp is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel. <span style="font-weight:bold">[</span><span style="color:rgb(0,0,187)">notice</span><span style="font-weight:bold">]</span> A new release of pip available: <span style="color:rgb(187,0,0)">22.1.2</span> -> <span style="color:rgb(0,187,0)">23.1.2</span> <span style="font-weight:bold">[</span><span style="color:rgb(0,0,187)">notice</span><span style="font-weight:bold">]</span> To update, run: <span style="color:rgb(0,187,0)">pip install --upgrade pip</span> </pre></div></div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h2>3.2 重启内核，测试模型</h2> </div></div><div class="ai-np-code"><div class="ai-np-code-in">In [1]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 重启内核，确定最新paddlenlp好用。</span> <span class="hljs-comment"># 测试ChatGLM-6B的Paddle版本是否工作</span> <span class="hljs-keyword">from</span> paddlenlp <span class="hljs-keyword">import</span> Taskflow chatbot = Taskflow(<span class="hljs-string">"text2text_generation"</span>,batch_size=<span class="hljs-number">2</span>) results = chatbot([<span class="hljs-string">'您好！'</span>,<span class="hljs-string">'很高兴认识你'</span>]) print(results)</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">[2023-06-17 08:32:01,170] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/ice_text.model and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 [2023-06-17 08:32:01,212] [ INFO] - Downloading ice_text.model from https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/ice_text.model </pre></div><div class="ai-np-output-wrapper"><div class="ai-np-execute-result"><div><pre class="ai-np-pre"> 0%| | 0.00/2.58M [00:00<?, ?B/s]</pre></div></div></div><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">[2023-06-17 08:32:01,371] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 [2023-06-17 08:32:01,411] [ WARNING] - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json> not exist [2023-06-17 08:32:01,414] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 [2023-06-17 08:32:01,452] [ WARNING] - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json> not exist [2023-06-17 08:32:01,455] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/tokenizer_config.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 [2023-06-17 08:32:01,500] [ INFO] - Downloading tokenizer_config.json from https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/tokenizer_config.json </pre></div><div class="ai-np-output-wrapper"><div class="ai-np-execute-result"><div><pre class="ai-np-pre"> 0%| | 0.00/441 [00:00<?, ?B/s]</pre></div></div></div><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">[2023-06-17 08:32:02,133] [ INFO] - Downloading config.json from https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/config.json </pre></div><div class="ai-np-output-wrapper"><div class="ai-np-execute-result"><div><pre class="ai-np-pre"> 0%| | 0.00/773 [00:00<?, ?B/s]</pre></div></div></div><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">[2023-06-17 08:32:02,252] [ INFO] - loading configuration file /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/config.json [2023-06-17 08:32:02,255] [ INFO] - Model config ChatGLMConfig { "_name_or_path": "THUDM/chatglm-6b", "activation": "gelu", "architectures": [ "ChatGLMModel" ], "attention_scale": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 130004, "dtype": "float16", "eos_token_id": 130005, "gmask_token_id": 130001, "hidden_size": 4096, "inner_hidden_size": 16384, "layernorm_epsilon": 1e-05, "mask_token_id": 130000, "max_sequence_length": 2048, "model_type": "chatglm", "num_attention_heads": 32, "num_hidden_layers": 28, "num_image_tokens": 0, "output_predict": true, "pad_token_id": 3, "paddlenlp_version": null, "position_encoding_2d": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "recompute": false, "transformers_version": "4.23.1", "use_cache": true, "vocab_size": 130528 } [2023-06-17 08:32:02,298] [ INFO] - Downloading model_state.pdparams from https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/model_state.pdparams </pre></div><div class="ai-np-output-wrapper"><div class="ai-np-execute-result"><div><pre class="ai-np-pre"> 0%| | 0.00/12.5G [00:00<?, ?B/s]</pre></div></div></div><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">W0617 08:34:25.528139 755 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2 W0617 08:34:25.532923 755 gpu_resources.cc:149] device: 0, cuDNN Version: 8.2. [2023-06-17 08:34:40,902] [ WARNING] - Some weights of the model checkpoint at THUDM/chatglm-6b-v1.1 were not used when initializing ChatGLMForConditionalGeneration: ['transformer.layers.1.attention.rotary_emb.inv_freq', 'transformer.layers.12.attention.rotary_emb.inv_freq', 'transformer.layers.22.attention.rotary_emb.inv_freq', 'transformer.layers.23.attention.rotary_emb.inv_freq', 'transformer.layers.16.attention.rotary_emb.inv_freq', 'transformer.layers.26.attention.rotary_emb.inv_freq', 'transformer.layers.27.attention.rotary_emb.inv_freq', 'transformer.layers.18.attention.rotary_emb.inv_freq', 'transformer.layers.13.attention.rotary_emb.inv_freq', 'transformer.layers.7.attention.rotary_emb.inv_freq', 'transformer.layers.15.attention.rotary_emb.inv_freq', 'transformer.layers.4.attention.rotary_emb.inv_freq', 'transformer.layers.5.attention.rotary_emb.inv_freq', 'transformer.layers.19.attention.rotary_emb.inv_freq', 'transformer.layers.0.attention.rotary_emb.inv_freq', 'transformer. - This IS expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). [2023-06-17 08:34:40,906] [ WARNING] - Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at THUDM/chatglm-6b-v1.1 and are newly initialized: ['transformer.layers.22.attention.rotary_embeddings.inv_freq', 'transformer.layers.21.attention.rotary_embeddings.inv_freq', 'transformer.layers.15.attention.rotary_embeddings.inv_freq', 'transformer.layers.16.attention.rotary_embeddings.inv_freq', 'transformer.layers.17.attention.rotary_embeddings.inv_freq', 'transformer.layers.5.attention.rotary_embeddings.inv_freq', 'transformer.layers.8.attention.rotary_embeddings.inv_freq', 'transformer.layers.18.attention.rotary_embeddings.inv_freq', 'transformer.layers.12.attention.rotary_embeddings.inv_freq', 'transformer.layers.10.attention.rotary_embeddings.inv_freq', 'transformer.layers.20.attention.rotary_embeddings.inv_freq', 'transformer.layers.25.attention.rotary_embeddings.inv_freq', 'transformer.layers.1.attention.rotary_embeddings.inv_freq', 'transform You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. </pre></div><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">{'result': ['您好!有什么需要帮助的吗?', '你好!我也很高兴认识你。作为一名人工智能助手,我很高兴能够与你交流和回答你的问题。请问有什么我可以为你效劳的吗?']} </pre></div></div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h2>3.3 安装与LangChian相关的包</h2> </div></div><div class="ai-np-code"><div class="ai-np-code-in">In [2]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 安装依赖（时间较长）</span> %cd /home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle !pip install -r requirements.txt </code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">/home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting langchain==0.0.147 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/4d/daae163e60c60aa74fc72467d1e06df793fcf63f3bafe3f8183ebf8bdaf7/langchain-0.0.147-py3-none-any.whl (626 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/626.5 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 626.5/626.5 kB 21.7 MB/s eta 0:00:00 Collecting unstructured[local-inference] Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3d/b2/bdf4719c8ef5dfd70a6b2fb5d57553f6d96a0fb58ff8c7e98cb70108484a/unstructured-0.7.6-py3-none-any.whl (1.4 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 59.1 MB/s eta 0:00:00 Collecting layoutparser[layoutmodels,tesseract] Downloading https://pypi.tuna.tsinghua.edu.cn/packages/08/cf/0bfbea1b2ace91af45e15bdec885e05992dc9150907a8398b3d305eddfd2/layoutparser-0.3.4-py3-none-any.whl (19.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/19.2 MB ? eta -:--:--━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/19.2 MB 165.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/19.2 MB 165.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━ 8.0/19.2 MB 22.7 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 12.5/19.2 MB 27.5 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 17.6/19.2 MB 27.7 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━ Requirement already satisfied: nltk in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (3.4.5) Requirement already satisfied: beautifulsoup4 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (4.11.1) Collecting icetk Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/8a/731927e0901273815b779e6ce0e081a95ecf78835ff80be30830505ae06c/icetk-0.0.7-py3-none-any.whl (16 kB) Collecting cpm_kernels Downloading https://pypi.tuna.tsinghua.edu.cn/packages/af/84/1831ce6ffa87b8fd4d9673c3595d0fc4e6631c0691eb43f406d3bf89b951/cpm_kernels-1.0.11-py3-none-any.whl (416 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/416.6 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 416.6/416.6 kB 57.8 MB/s eta 0:00:00 Requirement already satisfied: faiss-cpu in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from -r requirements.txt (line 8)) (1.7.4) Requirement already satisfied: protobuf in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from -r requirements.txt (line 9)) (3.20.2) Requirement already satisfied: requests<3,>=2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (2.24.0) Requirement already satisfied: tqdm>=4.48.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (4.64.1) Requirement already satisfied: pydantic<2,>=1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (1.10.4) Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (4.0.2) Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (3.8.3) Collecting openapi-schema-pydantic<2.0,>=1.2 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a8/e7/22abb5a10733bf8142984201aedf27d4a58f5810ebdfe9679f9876c7bf4d/openapi_schema_pydantic-1.2.4-py3-none-any.whl (90 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/90.0 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.0/90.0 kB 22.3 MB/s eta 0:00:00 Collecting numexpr<3.0.0,>=2.8.4 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f1/db/aaa0379f5854dc2fbc935532a26805c2f2dd0b077216c644e7f36f89148a/numexpr-2.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (380 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/380.7 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 380.7/380.7 kB 56.1 MB/s eta 0:00:00 Collecting tenacity<9.0.0,>=8.1.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e7/b0/c23bd61e1b32c9b96fbca996c87784e196a812da8d621d8d04851f6c8181/tenacity-8.2.2-py3-none-any.whl (24 kB) Requirement already satisfied: SQLAlchemy<2,>=1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (1.4.41) Collecting dataclasses-json<0.6.0,>=0.5.7 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/bc/892e03650133583d5babbb7c7e5c1be6b68df86829c91fdc30cca996630d/dataclasses_json-0.5.8-py3-none-any.whl (26 kB) Requirement already satisfied: numpy<2,>=1 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from langchain==0.0.147->-r requirements.txt (line 1)) (1.22.4) Collecting PyYAML>=5.4.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/12/fc/a4d5a7554e0067677823f7265cb3ae22aed8a238560b5133b58cda252dad/PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/661.8 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 661.8/661.8 kB 65.8 MB/s eta 0:00:00 Requirement already satisfied: pdfminer.six in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (20221105) Requirement already satisfied: pandas in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (2.0.1) Requirement already satisfied: lxml in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (4.9.1) Requirement already satisfied: openpyxl in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (3.0.5) Collecting python-pptx Downloading https://pypi.tuna.tsinghua.edu.cn/packages/eb/c3/bd8f2316a790291ef5aa5225c740fa60e2cf754376e90cb1a44fde056830/python-pptx-0.6.21.tar.gz (10.1 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/10.1 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 5.1/10.1 MB 152.3 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━ 6.6/10.1 MB 146.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 6.7/10.1 MB 23.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 8.8/10.1 MB 27.5 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 10.1/10.1 MB 30.7 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 10.1/10.1 MB 30.7 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.1/10.1 MB 24.8 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Requirement already satisfied: tabulate in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (0.8.3) Collecting filetype Downloading https://pypi.tuna.tsinghua.edu.cn/packages/18/79/1b8fa1bb3568781e84c9200f951c735f3f157429f44be0495da55894d620/filetype-1.2.0-py2.py3-none-any.whl (19 kB) Collecting pypandoc Downloading https://pypi.tuna.tsinghua.edu.cn/packages/59/ac/1380f19d6bab8772b4e40d881091039a8c592d66382bd4c865fa21373bad/pypandoc-1.11-py3-none-any.whl (20 kB) Requirement already satisfied: xlrd in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (1.2.0) Requirement already satisfied: python-docx in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (0.8.11) Requirement already satisfied: markdown in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (3.1.1) Collecting python-magic Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6c/73/9f872cb81fc5c3bb48f7227872c28975f998f3e7c2b1c16e95e6432bbb90/python_magic-0.4.27-py2.py3-none-any.whl (13 kB) Collecting msg-parser Downloading https://pypi.tuna.tsinghua.edu.cn/packages/15/8b/5738b32acc6acdf92d04d5e691bf70a379e78264e55843542e1888d4a10e/msg_parser-1.2.0-py2.py3-none-any.whl (101 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/101.8 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.8/101.8 kB 23.2 MB/s eta 0:00:00 Collecting argilla Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9c/6e/9024bf09698995153c4806f7f975aac6e142d86118772306070654483a8a/argilla-1.10.0-py3-none-any.whl (2.5 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.5 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 84.4 MB/s eta 0:00:00 Requirement already satisfied: pdf2image in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (1.16.3) Requirement already satisfied: pillow in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (9.5.0) Requirement already satisfied: chardet in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured[local-inference]->-r requirements.txt (line 2)) (3.0.4) Collecting unstructured-inference==0.5.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/52/c3/9c98604a0d78fb95accd73b81984e97897d731c7a2894137ed1a2ab970ee/unstructured_inference-0.5.1-py3-none-any.whl (39 kB) Requirement already satisfied: python-multipart in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured-inference==0.5.1->unstructured[local-inference]->-r requirements.txt (line 2)) (0.0.5) Collecting transformers>=4.25.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5b/0b/e45d26ccd28568013523e04f325432ea88a442b4e3020b757cf4361f0120/transformers-4.30.2-py3-none-any.whl (7.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/7.2 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 5.4/7.2 MB 162.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 7.2/7.2 MB 162.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/7.2 MB 78.0 MB/s eta 0:00:00 Requirement already satisfied: opencv-python!=4.7.0.68 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured-inference==0.5.1->unstructured[local-inference]->-r requirements.txt (line 2)) (4.6.0.66) Requirement already satisfied: huggingface-hub in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from unstructured-inference==0.5.1->unstructured[local-inference]->-r requirements.txt (line 2)) (0.12.0) Collecting onnxruntime Downloading https://pypi.tuna.tsinghua.edu.cn/packages/50/d5/f156d808c9cc59e7a8b87843a5313719d01eec7d3b17ca3f6b6f0fbee0f0/onnxruntime-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/5.9 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━ 5.7/5.9 MB 172.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 5.9/5.9 MB 164.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 80.0 MB/s eta 0:00:00 Requirement already satisfied: pdfplumber in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (0.9.0) Collecting iopath Downloading https://pypi.tuna.tsinghua.edu.cn/packages/72/73/b3d451dfc523756cf177d3ebb0af76dc7751b341c60e2a21871be400ae29/iopath-0.1.10.tar.gz (42 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/42.2 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.2/42.2 kB 9.9 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Requirement already satisfied: scipy in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (1.9.1) Collecting torchvision Downloading https://pypi.tuna.tsinghua.edu.cn/packages/41/9e/8809e45a084680394e8d219fcf8a2c0eed2dddf1ec0a7968f4052826a6e9/torchvision-0.15.2-cp39-cp39-manylinux1_x86_64.whl (6.0 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/6.0 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 167.6 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 5.6/6.0 MB 159.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 5.9/6.0 MB 19.3 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 6.0/6.0 MB 19.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 16.9 MB/s eta 0:00:00 Collecting torch Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e5/9a/ce0fe125f226ffce8deba6a18bd8d0b9f589aa236780a83a6d70b5525f56/torch-2.0.1-cp39-cp39-manylinux1_x86_64.whl (619.9 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/619.9 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/619.9 MB 144.7 MB/s eta 0:00:05╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/619.9 MB 152.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/619.9 MB 170.5 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/619.9 MB 32.5 MB/s eta 0:00:19━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/619.9 MB 30.3 MB/s eta 0:00:20━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.5/619.9 MB 29.4 MB/s eta 0:00:21━╺━━━━━━━━━━━━━━━━━━━━ Collecting effdet Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9c/13/563119fe0af82aca5a3b89399c435953072c39515c2e818eb82793955c3b/effdet-0.4.1-py3-none-any.whl (112 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/112.5 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.5/112.5 kB 25.4 MB/s eta 0:00:00 Collecting pytesseract Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c5/54/ec007336f38d2d4ce61f3544af3e6855dacbf04a1ac8294f10cabe81146f/pytesseract-0.3.10-py3-none-any.whl (14 kB) Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from nltk->-r requirements.txt (line 4)) (1.16.0) Requirement already satisfied: soupsieve>1.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from beautifulsoup4->-r requirements.txt (line 5)) (2.3.2.post1) Requirement already satisfied: filelock in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from icetk->-r requirements.txt (line 6)) (3.9.0) Collecting protobuf Downloading https://pypi.tuna.tsinghua.edu.cn/packages/27/9f/1da80096efd92a87cfca4ff31e04218516e14092427168ba90684569cbfb/protobuf-3.18.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.1 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 78.5 MB/s eta 0:00:00 Requirement already satisfied: sentencepiece in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from icetk->-r requirements.txt (line 6)) (0.1.96) Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (1.2.0) Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (6.0.2) Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (1.7.2) Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (1.3.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (2.1.1) Requirement already satisfied: attrs>=17.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.147->-r requirements.txt (line 1)) (22.1.0) Collecting typing-inspect>=0.4.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/65/f3/107a22063bf27bdccf2024833d3445f4eea42b2e598abfbd46f6a63b6cb0/typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) Collecting marshmallow-enum<2.0.0,>=1.5.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/59/ef3a3dc499be447098d4a89399beb869f813fee1b5a57d5d79dee2c1bf51/marshmallow_enum-1.5.1-py2.py3-none-any.whl (4.2 kB) Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.147->-r requirements.txt (line 1)) (3.19.0) Requirement already satisfied: typing-extensions>=4.2.0 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from pydantic<2,>=1->langchain==0.0.147->-r requirements.txt (line 1)) (4.5.0) Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests<3,>=2->langchain==0.0.147->-r requirements.txt (line 1)) (2.8) Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests<3,>=2->langchain==0.0.147->-r requirements.txt (line 1)) (2019.9.11) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from requests<3,>=2->langchain==0.0.147->-r requirements.txt (line 1)) (1.25.11) Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from SQLAlchemy<2,>=1->langchain==0.0.147->-r requirements.txt (line 1)) (2.0.2) Collecting deprecated~=1.2.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/20/8d/778b7d51b981a96554f29136cd59ca7880bf58094338085bcf2a979a0e6a/Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB) Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (21.3) Collecting pandas Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e1/4d/3eb96e53a9208350ee21615f850c4be9a246d32bf1d34cd36682cb58c3b7/pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/12.2 MB ? eta -:--:--━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━ 5.3/12.2 MB 160.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 9.7/12.2 MB 140.6 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 12.2/12.2 MB 144.6 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 12.2/12.2 MB 144.6 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 12.2/12.2 MB 144.6 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 59.2 MB/s eta 0:00:00 Collecting wrapt<1.15,>=1.13 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e0/6a/3c660fa34c8106aa9719f2a6636c1c3ea7afd5931ae665eb197fdf4def84/wrapt-1.14.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/77.8 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.8/77.8 kB 18.1 MB/s eta 0:00:00 Collecting monotonic Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9a/67/7e8406a29b6c45be7af7740456f7f37025f0506ae2e05fb9009a53946860/monotonic-1.6-py2.py3-none-any.whl (8.2 kB) Requirement already satisfied: rich<=13.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (12.6.0) Requirement already satisfied: httpx<0.24,>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (0.23.3) Requirement already satisfied: typer<1.0.0,>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (0.7.0) Collecting pydantic<2,>=1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/02/05/1c505fd7cc174ddecd5aa956057638ba6039b584af6fa9687f11a074fca7/pydantic-1.10.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/3.2 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 3.2/3.2 MB 175.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 88.5 MB/s eta 0:00:00 Collecting backoff Downloading https://pypi.tuna.tsinghua.edu.cn/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl (15 kB) Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from pandas->unstructured[local-inference]->-r requirements.txt (line 2)) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from pandas->unstructured[local-inference]->-r requirements.txt (line 2)) (2022.7.1) Collecting pycocotools>=2.0.2 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ef/c6/90220be3b39fbc4cbd203775ca47dd8dc97fae06fbd2b500637395621b7c/pycocotools-2.0.6.tar.gz (24 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting omegaconf>=2.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e3/94/1843518e420fa3ed6919835845df698c7e27e183cb997394e4a670973a65/omegaconf-2.3.0-py3-none-any.whl (79 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/79.5 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.5/79.5 kB 17.8 MB/s eta 0:00:00 Collecting timm>=0.9.2 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/29/90/94f5deb8d76e24a89813aef95e8809ca8fd7414490428480eda19b133d4a/timm-0.9.2-py3-none-any.whl (2.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.2 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 80.5 MB/s eta 0:00:00 Requirement already satisfied: jinja2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (3.1.2) Collecting nvidia-cuda-cupti-cu11==11.7.101 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e6/9d/dd0cdcd800e642e3c82ee3b5987c751afd4f3fb9cc2752517f42c3bc6e49/nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/11.8 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 5.6/11.8 MB 167.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 5.8/11.8 MB 91.4 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 6.5/11.8 MB 64.3 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 10.7/11.8 MB 79.9 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 11.1/11.8 MB 25.2 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━ Collecting nvidia-cusolver-cu11==11.4.0.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3e/77/66149e3153b19312fb782ea367f3f950123b93916a45538b573fe373570a/nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/102.6 MB ? eta -:--:--━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/102.6 MB 97.6 MB/s eta 0:00:02━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.5/102.6 MB 109.1 MB/s eta 0:00:01━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/102.6 MB 101.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 120.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.9/102.6 MB 29.8 MB/s eta 0:00:03━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.6/102.6 MB 27.4 MB/s eta 0:00:04━━━━━━━╺━━━━━━━━━━━━━━ Collecting nvidia-cusparse-cu11==11.7.4.91 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ea/6f/6d032cc1bb7db88a989ddce3f4968419a7edeafda362847f42f614b1f845/nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/173.2 MB ? eta -:--:--━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.1/173.2 MB 182.1 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 172.9 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/173.2 MB 22.3 MB/s eta 0:00:08━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/173.2 MB 20.6 MB/s eta 0:00:09━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/173.2 MB 29.6 MB/s eta 0:00:06━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/173.2 MB 29.5 MB/s eta 0:00:06━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/173.2 MB 29.5 MB/s eta 0:00:06━━╸━━━━━━━━━━━━━━━━━━━ Collecting nvidia-cuda-runtime-cu11==11.7.99 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/36/92/89cf558b514125d2ebd8344dd2f0533404b416486ff681d5434a5832a019/nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/849.3 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 76.5 MB/s eta 0:00:00 Collecting nvidia-cublas-cu11==11.10.3.66 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ce/41/fdeb62b5437996e841d83d7d2714ca75b886547ee8017ee2fe6ea409d983/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/317.1 MB ? eta -:--:--╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/317.1 MB 179.6 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.9/317.1 MB 175.8 MB/s eta 0:00:02╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/317.1 MB 23.1 MB/s eta 0:00:14━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/317.1 MB 29.6 MB/s eta 0:00:11━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/317.1 MB 29.9 MB/s eta 0:00:11━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/317.1 MB 29.9 MB/s eta 0:00:11━━╺━━━━━━━━━━━━━━━━━━━ Collecting nvidia-cufft-cu11==10.9.0.58 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/74/79/b912a77e38e41f15a0581a59f5c3548d1ddfdda3225936fb67c342719e7a/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/168.4 MB ? eta -:--:--━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/168.4 MB 166.1 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/168.4 MB 165.8 MB/s eta 0:00:01━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/168.4 MB 25.0 MB/s eta 0:00:07━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.4/168.4 MB 25.7 MB/s eta 0:00:07━━━━╺━━━━━━━━━━━━━━━━━ Collecting nvidia-nvtx-cu11==11.7.91 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/23/d5/09493ff0e64fd77523afbbb075108f27a13790479efe86b9ffb4587671b5/nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/98.6 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.6/98.6 kB 21.2 MB/s eta 0:00:00 Collecting nvidia-nccl-cu11==2.14.3 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/55/92/914cdb650b6a5d1478f83148597a25e90ea37d739bd563c5096b0e8a5f43/nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/177.1 MB ? eta -:--:--╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/177.1 MB 108.0 MB/s eta 0:00:02━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/177.1 MB 128.9 MB/s eta 0:00:02━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/177.1 MB 128.9 MB/s eta 0:00:02━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/177.1 MB 128.9 MB/s eta 0:00:02━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/177.1 MB 128.9 MB/s eta 0:00:02━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/177.1 MB 32.0 MB/s eta 0:00:06━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/177.1 MB 23.2 MB/s eta 0:00:08━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/177.1 MB 20.6 MB/s eta 0:00:09━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/177.1 MB 18.7 MB/s eta 0:00:10━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/177.1 MB 17.1 MB/s eta 0:00:11━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/177.1 MB 16.4 MB/s eta 0:00:11━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.2/177.1 MB 17.1 MB/s eta 0:00:10━━╺━━━━━━━━━━━━━━━━━━━ Requirement already satisfied: sympy in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (1.11.1) Collecting triton==2.0.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/77/ac/28b74ec1177c730d0da8803eaff5e5025bd532bcf07cadb0fcf661abed97/triton-2.0.0-1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/63.3 MB ? eta -:--:--━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/63.3 MB 189.4 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/63.3 MB 184.0 MB/s eta 0:00:01━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/63.3 MB 23.7 MB/s eta 0:00:03━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.4/63.3 MB 25.4 MB/s eta 0:00:02━━━━━━━━━━━╸━━━━━━━━━━━━ Collecting nvidia-cudnn-cu11==8.5.0.96 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/dc/30/66d4347d6e864334da5bb1c7571305e501dcb11b9155971421bb7bb5315f/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/557.1 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/557.1 MB 163.3 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 165.2 MB/s eta 0:00:04╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.0/557.1 MB 31.9 MB/s eta 0:00:18╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/557.1 MB 29.9 MB/s eta 0:00:19━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.4/557.1 MB 30.8 MB/s eta 0:00:18━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/557.1 MB 30.7 MB/s eta 0:00:18━╺━━━━━━━━━━━━━━━━━━━━ Collecting nvidia-cuda-nvrtc-cu11==11.7.99 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ef/25/922c5996aada6611b79b53985af7999fc629aee1d5d001b6a22431e18fec/nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/21.0 MB ? eta -:--:--╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.5/21.0 MB 24.1 MB/s eta 0:00:01━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.6/21.0 MB 9.5 MB/s eta 0:00:03━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/21.0 MB 21.3 MB/s eta 0:00:01━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/21.0 MB 20.2 MB/s eta 0:00:01━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/21.0 MB 15.8 MB/s eta 0:00:02━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/21.0 MB 13.0 MB/s eta 0:00:02━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.8/21.0 MB 28.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━ 10.5/21.0 MB 38.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━ 10.5/21.0 MB 38.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━ 10.5/21.0 MB 38.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━ 10.5/21.0 MB 38.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━ 10.5/21.0 MB 38.8 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ Collecting nvidia-curand-cu11==10.2.10.91 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/8f/11/af78d54b2420e64a4dd19e704f5bb69dcb5a6a3138b4465d6a48cdf59a21/nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/54.6 MB ? eta -:--:--━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/54.6 MB 155.3 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 162.2 MB/s eta 0:00:01━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/54.6 MB 30.5 MB/s eta 0:00:02━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/54.6 MB 30.2 MB/s eta 0:00:02━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.4/54.6 MB 30.4 MB/s eta 0:00:02━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.4/54.6 MB 30.4 MB/s eta 0:00:02━━━━━━━━━━╸━━━━━━━━━━━━━ Requirement already satisfied: networkx in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (3.1) Requirement already satisfied: wheel in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (0.36.2) Requirement already satisfied: setuptools in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (56.2.0) Collecting lit Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/fa/0b75c53253ebf3ab566be702a9da16f5783862d8c1ae404c907a8830f283/lit-16.0.6.tar.gz (153 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/153.7 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.7/153.7 kB 31.4 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing metadata (pyproject.toml) ... done Collecting cmake Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b5/3a/0d5889762ec82d3c556cfab075e6cdbca06dc5cff55436950152d63bb194/cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/24.0 MB ? eta -:--:--━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/24.0 MB 123.0 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/24.0 MB 144.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 9.6/24.0 MB 27.7 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 13.0/24.0 MB 28.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14.5/24.0 MB 28.5 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━ Collecting portalocker Downloading https://pypi.tuna.tsinghua.edu.cn/packages/8c/df/d4f711d168524f5aebd7fb30969eaa31e3048cf8979688cde3b08f6e5eb8/portalocker-2.7.0-py2.py3-none-any.whl (15 kB) Collecting olefile>=0.46 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/34/81/e1ac43c6b45b4c5f8d9352396a14144bba52c8fec72a80f425f6a4d653ad/olefile-0.46.zip (112 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/112.2 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.2/112.2 kB 7.7 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Requirement already satisfied: et-xmlfile in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from openpyxl->unstructured[local-inference]->-r requirements.txt (line 2)) (1.0.1) Requirement already satisfied: jdcal in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from openpyxl->unstructured[local-inference]->-r requirements.txt (line 2)) (1.4.1) Requirement already satisfied: cryptography>=36.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from pdfminer.six->unstructured[local-inference]->-r requirements.txt (line 2)) (38.0.1) Requirement already satisfied: Wand>=0.6.10 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from pdfplumber->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (0.6.11) Collecting XlsxWriter>=0.5.7 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/37/94/25d3ec8587974de7ebd790232aa3155abfe44ed23df7ccaa4645977a1cbe/XlsxWriter-3.1.2-py3-none-any.whl (153 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/153.0 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.0/153.0 kB 26.0 MB/s eta 0:00:00 Requirement already satisfied: cffi>=1.12 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from cryptography>=36.0.0->pdfminer.six->unstructured[local-inference]->-r requirements.txt (line 2)) (1.15.1) Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx<0.24,>=0.15->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (0.16.3) Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx<0.24,>=0.15->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (1.5.0) Requirement already satisfied: sniffio in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpx<0.24,>=0.15->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (1.3.0) Collecting antlr4-python3-runtime==4.9.* Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3e/38/7859ff46355f76f8d19459005ca000b6e7012f2f1ca597746cbcd1fbfe5e/antlr4-python3-runtime-4.9.3.tar.gz (117 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/117.0 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.0/117.0 kB 25.1 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from packaging>=20.0->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (3.0.9) Requirement already satisfied: matplotlib>=2.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (2.2.3) Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from rich<=13.0.1->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (2.13.0) Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from rich<=13.0.1->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (0.9.1) Collecting safetensors Downloading https://pypi.tuna.tsinghua.edu.cn/packages/60/c7/1911e04710666eb79ca3311a4e91b669419a1f23c2b2619005165104368c/safetensors-0.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.3 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 47.1 MB/s eta 0:00:00 Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d6/27/07a337087dd507170a1b20fed3bbf8da81401185a7130a6e74e440c52040/tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/7.8 MB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 5.8/7.8 MB 176.1 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 7.8/7.8 MB 157.3 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 7.8/7.8 MB 157.3 MB/s eta 0:00:01━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 71.8 MB/s eta 0:00:00 Collecting huggingface-hub Downloading https://pypi.tuna.tsinghua.edu.cn/packages/62/a2/8a416d167216403ceeef3aaf8c22b0c61b1ae571644473d67eb7fecbb69e/huggingface_hub-0.15.1-py3-none-any.whl (236 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/236.8 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 236.8/236.8 kB 44.0 MB/s eta 0:00:00 Collecting regex!=2019.12.17 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/5e/a8/2e3626392c4fcf7e3920cae166155542838be2048384989e2c6f024b793d/regex-2023.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/769.9 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 769.9/769.9 kB 74.3 MB/s eta 0:00:00 Requirement already satisfied: fsspec in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from huggingface-hub->unstructured-inference==0.5.1->unstructured[local-inference]->-r requirements.txt (line 2)) (2022.11.0) Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/aistudio/.data/webide/pip/lib/python3.9/site-packages (from typer<1.0.0,>=0.6.0->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (8.0.0) Collecting mypy-extensions>=0.3.0 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB) Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from jinja2->torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (2.0.1) Collecting flatbuffers Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB) Collecting coloredlogs Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/06/3d6badcf13db419e25b07041d9c7b4a2c331d3f4e7134445ec5df57714cd/coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/46.0 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 kB 9.4 MB/s eta 0:00:00 Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from sympy->torch->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (1.2.1) Requirement already satisfied: pycparser in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six->unstructured[local-inference]->-r requirements.txt (line 2)) (2.21) Requirement already satisfied: h11<0.15,>=0.13 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpcore<0.17.0,>=0.15.0->httpx<0.24,>=0.15->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (0.14.0) Requirement already satisfied: anyio<5.0,>=3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from httpcore<0.17.0,>=0.15.0->httpx<0.24,>=0.15->argilla->unstructured[local-inference]->-r requirements.txt (line 2)) (3.6.1) Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (0.10.0) Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.2->effdet->layoutparser[layoutmodels,tesseract]->-r requirements.txt (line 3)) (1.1.0) Collecting humanfriendly>=9.1 Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f0/0f/310fb31e39e2d734ccaa2c0fb981ee41f7bd5056ce9bc29b2248bd569169/humanfriendly-10.0-py2.py3-none-any.whl (86 kB) l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/86.8 kB ? eta -:--:--━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 22.2 MB/s eta 0:00:00 Building wheels for collected packages: iopath, python-pptx, olefile, antlr4-python3-runtime, pycocotools, lit Building wheel for iopath (setup.py) ... done Created wheel for iopath: filename=iopath-0.1.10-py3-none-any.whl size=31543 sha256=37640ec33fd71a8ad5bc6b3dc9d99c4b46fb7d65517b5a7e3e9efeb4f4f15a4a Stored in directory: /home/aistudio/.cache/pip/wheels/f4/3e/63/740e10c38713762725480e69ebfc832fe12a8bef6f3b380249 Building wheel for python-pptx (setup.py) ... done Created wheel for python-pptx: filename=python_pptx-0.6.21-py3-none-any.whl size=470939 sha256=f42ca8330fdb7c09024edeb90c5f893b339422b229ebe7886a22b09f12eabcc6 Stored in directory: /home/aistudio/.cache/pip/wheels/75/14/7a/1f563540a42f97a868eb0b13b0f42a075024c4491cb169ad01 Building wheel for olefile (setup.py) ... done Created wheel for olefile: filename=olefile-0.46-py2.py3-none-any.whl size=35416 sha256=a9c1997bc2fc3541be7a72f866c46a027e116132b49879fb2d881045fd5fd37f Stored in directory: /home/aistudio/.cache/pip/wheels/27/e8/89/6ea6bf88439c9ef62473281e2cc5bbf09d8e028e1bad7d0bf4 Building wheel for antlr4-python3-runtime (setup.py) ... done Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144573 sha256=208b5d73be0db20a1ee5441e420bafbbb21750f46ba6a3fc16a38e17e0004c20 Stored in directory: /home/aistudio/.cache/pip/wheels/b9/0a/c0/fc0de68aa14c7066272aed9fdf29080aad637f3530d865ac86 Building wheel for pycocotools (pyproject.toml) ... done Created wheel for pycocotools: filename=pycocotools-2.0.6-cp39-cp39-linux_x86_64.whl size=96265 sha256=55f2dc6bf1ea6fce0a7e8a85b009d53c7b2790a3b0b27e490c2e8d4e5dcdc678 Stored in directory: /home/aistudio/.cache/pip/wheels/df/40/61/1e4b2fc4aacaadb05b23d10fb90c5fd62b9f4021c48f3fd710 Building wheel for lit (pyproject.toml) ... done Created wheel for lit: filename=lit-16.0.6-py3-none-any.whl size=93582 sha256=3a507c41cd2f1fc5bdcc2e4385f6aead4101a8a7f649ccacfb8685f894933625 Stored in directory: /home/aistudio/.cache/pip/wheels/9c/3d/43/1038b6c210fef66984d18cdf338e319d07e60e1b074a30b374 Successfully built iopath python-pptx olefile antlr4-python3-runtime pycocotools lit Installing collected packages: tokenizers, safetensors, monotonic, lit, flatbuffers, filetype, cpm_kernels, cmake, antlr4-python3-runtime, XlsxWriter, wrapt, tenacity, regex, PyYAML, python-magic, pypandoc, pydantic, protobuf, portalocker, olefile, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numexpr, mypy-extensions, humanfriendly, backoff, typing-inspect, python-pptx, pytesseract, pandas, openapi-schema-pydantic, omegaconf, nvidia-cusolver-cu11, nvidia-cudnn-cu11, msg-parser, iopath, huggingface-hub, deprecated, coloredlogs, transformers, pycocotools, onnxruntime, marshmallow-enum, dataclasses-json, argilla, unstructured, layoutparser, langchain, triton, torch, torchvision, timm, effdet, unstructured-inference, icetk Attempting uninstall: wrapt Found existing installation: wrapt 1.12.1 Uninstalling wrapt-1.12.1: Successfully uninstalled wrapt-1.12.1 Attempting uninstall: tenacity Found existing installation: tenacity 8.0.1 Uninstalling tenacity-8.0.1: Successfully uninstalled tenacity-8.0.1 Attempting uninstall: PyYAML Found existing installation: PyYAML 5.1.2 Uninstalling PyYAML-5.1.2: Successfully uninstalled PyYAML-5.1.2 Attempting uninstall: pydantic Found existing installation: pydantic 1.10.4 Uninstalling pydantic-1.10.4: Successfully uninstalled pydantic-1.10.4 Attempting uninstall: protobuf Found existing installation: protobuf 3.20.2 Uninstalling protobuf-3.20.2: Successfully uninstalled protobuf-3.20.2 Attempting uninstall: pandas Found existing installation: pandas 2.0.1 Uninstalling pandas-2.0.1: Successfully uninstalled pandas-2.0.1 Attempting uninstall: huggingface-hub Found existing installation: huggingface-hub 0.12.0 Uninstalling huggingface-hub-0.12.0: Successfully uninstalled huggingface-hub-0.12.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. pymilvus 2.2.8 requires protobuf>=3.20.0, but you have protobuf 3.18.3 which is incompatible. paddlenlp 2.5.2.post0 requires protobuf==3.20.2, but you have protobuf 3.18.3 which is incompatible. sahi 0.10.1 requires click==8.0.4, but you have click 8.0.0 which is incompatible. parl 1.4.1 requires pyzmq==18.1.1, but you have pyzmq 23.2.1 which is incompatible. paddlepaddle-gpu 0.0.0.post112 requires protobuf>=3.20.2; platform_system != "Windows", but you have protobuf 3.18.3 which is incompatible. Successfully installed PyYAML-6.0 XlsxWriter-3.1.2 antlr4-python3-runtime-4.9.3 argilla-1.10.0 backoff-2.2.1 cmake-3.26.4 coloredlogs-15.0.1 cpm_kernels-1.0.11 dataclasses-json-0.5.8 deprecated-1.2.14 effdet-0.4.1 filetype-1.2.0 flatbuffers-23.5.26 huggingface-hub-0.15.1 humanfriendly-10.0 icetk-0.0.7 iopath-0.1.10 langchain-0.0.147 layoutparser-0.3.4 lit-16.0.6 marshmallow-enum-1.5.1 monotonic-1.6 msg-parser-1.2.0 mypy-extensions-1.0.0 numexpr-2.8.4 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 olefile-0.46 omegaconf-2.3.0 onnxruntime-1.15.1 openapi-schema-pydantic-1.2.4 pandas-1.5.3 portalocker-2.7.0 protobuf-3.20.0 pycocotools-2.0.6 pydantic-1.10.9 pypandoc-1.11 pytesseract-0.3.10 python-magic-0.4.27 python-pptx- [notice] A new release of pip available: 22.1.2 -> 23.1.2 [notice] To update, run: pip install --upgrade pip </pre></div></div></div><div class="ai-np-code"><div class="ai-np-code-in">In [1]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 运行LangChain版本的本地知识库进行文本匹配自动问答</span> <span class="hljs-comment"># !git clone https://openi.pcl.ac.cn/Learning-Develop-Union/LangChain-ChatGLM-Webui.git</span></code></pre></div></div><div class="ai-np-code"><div class="ai-np-code-in">In [7]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 安装numpy，否则会报错</span> !pip uninstall numpy -y --user !pip install numpy --user</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream"> Usage: pip uninstall [options] <package> ... pip uninstall [options] -r <requirements file> ... no such option: --user WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages (1.23.5) WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) WARNING: Ignoring invalid distribution -umpy (/opt/conda/envs/python35-paddle120-env/lib/python3.9/site-packages) [notice] A new release of pip available: 22.1.2 -> 23.1.2 [notice] To update, run: pip install --upgrade pip </pre></div></div></div><div class="ai-np-markdown"><div class="ai-np-markdown-cell-content"><h2>3.4 测试LangChain程序</h2> </div></div><div class="ai-np-code"><div class="ai-np-code-in">In [1]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 这块视频没运行</span> <span class="hljs-comment"># 运行非LangChain版本的本地知识库自动问答</span> %cd /home/aistudio !python chat_documents.py</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">/home/aistudio <span style="color:rgb(0,187,0)">[2023-06-17 09:27:25,671] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/ice_text.model <span style="color:rgb(0,187,0)">[2023-06-17 09:27:25,671] [ INFO]</span> - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 <span style="color:rgb(187,187,0)">[2023-06-17 09:27:25,711] [ WARNING]</span> - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json> not exist <span style="color:rgb(0,187,0)">[2023-06-17 09:27:25,711] [ INFO]</span> - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 <span style="color:rgb(187,187,0)">[2023-06-17 09:27:25,747] [ WARNING]</span> - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json> not exist <span style="color:rgb(0,187,0)">[2023-06-17 09:27:25,747] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:26,101] [ INFO]</span> - Found /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:26,102] [ INFO]</span> - loading configuration file /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:26,103] [ INFO]</span> - Model config ChatGLMConfig { "_name_or_path": "THUDM/chatglm-6b", "activation": "gelu", "architectures": [ "ChatGLMModel" ], "attention_scale": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 130004, "dtype": "float16", "eos_token_id": 130005, "gmask_token_id": 130001, "hidden_size": 4096, "inner_hidden_size": 16384, "layernorm_epsilon": 1e-05, "mask_token_id": 130000, "max_sequence_length": 2048, "model_type": "chatglm", "num_attention_heads": 32, "num_hidden_layers": 28, "num_image_tokens": 0, "output_predict": true, "pad_token_id": 3, "paddlenlp_version": null, "position_encoding_2d": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "recompute": false, "transformers_version": "4.23.1", "use_cache": true, "vocab_size": 130528 } <span style="color:rgb(0,187,0)">[2023-06-17 09:27:26,144] [ INFO]</span> - Found /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/model_state.pdparams W0617 09:27:35.560508 6170 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2 W0617 09:27:35.565218 6170 gpu_resources.cc:149] device: 0, cuDNN Version: 8.2. <span style="color:rgb(187,187,0)">[2023-06-17 09:27:50,454] [ WARNING]</span> - Some weights of the model checkpoint at THUDM/chatglm-6b-v1.1 were not used when initializing ChatGLMForConditionalGeneration: ['transformer.layers.1.attention.rotary_emb.inv_freq', 'transformer.layers.22.attention.rotary_emb.inv_freq', 'transformer.layers.16.attention.rotary_emb.inv_freq', 'transformer.layers.23.attention.rotary_emb.inv_freq', 'transformer.layers.5.attention.rotary_emb.inv_freq', 'transformer.layers.15.attention.rotary_emb.inv_freq', 'transformer.layers.26.attention.rotary_emb.inv_freq', 'transformer.layers.11.attention.rotary_emb.inv_freq', 'transformer.layers.24.attention.rotary_emb.inv_freq', 'transformer.layers.2.attention.rotary_emb.inv_freq', 'transformer.layers.14.attention.rotary_emb.inv_freq', 'transformer.layers.12.attention.rotary_emb.inv_freq', 'transformer.layers.21.attention.rotary_emb.inv_freq', 'transformer.layers.0.attention.rotary_emb.inv_freq', 'transformer.layers.6.attention.rotary_emb.inv_freq', 'tra - This IS expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). <span style="color:rgb(187,187,0)">[2023-06-17 09:27:50,454] [ WARNING]</span> - Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at THUDM/chatglm-6b-v1.1 and are newly initialized: ['transformer.layers.3.attention.rotary_embeddings.inv_freq', 'transformer.layers.25.attention.rotary_embeddings.inv_freq', 'transformer.layers.27.attention.rotary_embeddings.inv_freq', 'transformer.layers.8.attention.rotary_embeddings.inv_freq', 'transformer.layers.15.attention.rotary_embeddings.inv_freq', 'transformer.layers.12.attention.rotary_embeddings.inv_freq', 'transformer.layers.13.attention.rotary_embeddings.inv_freq', 'transformer.layers.14.attention.rotary_embeddings.inv_freq', 'transformer.layers.7.attention.rotary_embeddings.inv_freq', 'transformer.layers.24.attention.rotary_embeddings.inv_freq', 'transformer.layers.22.attention.rotary_embeddings.inv_freq', 'transformer.layers.9.attention.rotary_embeddings.inv_freq', 'transformer.layers.16.attention.rotary_embeddings.inv_freq', 't You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. INFO - pipelines.utils.common_utils - Using devices: PLACE(GPU:0) INFO - pipelines.utils.common_utils - Number of GPUs: 1 Loading Parameters from:rocketqa-zh-dureader-cross-encoder <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,209] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-cross-encoder/rocketqa-zh-dureader-vocab.txt <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,219] [ INFO]</span> - tokenizer config file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-cross-encoder/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,219] [ INFO]</span> - Special tokens file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-cross-encoder/special_tokens_map.json INFO - pipelines.document_stores.faiss - document_cnt:0 embedding_cnt:0 INFO - pipelines.utils.common_utils - Using devices: PLACE(GPU:0) INFO - pipelines.utils.common_utils - Number of GPUs: 1 <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,297] [ INFO]</span> - We are using (<class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'>, False) to load 'rocketqa-zh-base-query-encoder'. <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,298] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/ernie_3.0_base_zh_vocab.txt <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,318] [ INFO]</span> - tokenizer config file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:53,318] [ INFO]</span> - Special tokens file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/special_tokens_map.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:55,167] [ INFO]</span> - We are using (<class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'>, False) to load 'rocketqa-zh-base-query-encoder'. <span style="color:rgb(0,187,0)">[2023-06-17 09:27:55,167] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/ernie_3.0_base_zh_vocab.txt <span style="color:rgb(0,187,0)">[2023-06-17 09:27:55,187] [ INFO]</span> - tokenizer config file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:27:55,187] [ INFO]</span> - Special tokens file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-base-query-encoder/special_tokens_map.json Writing Documents: 1000it [00:00, 96787.91it/s] [2023/06/17 09:27:57] ppocr DEBUG: Namespace(help='==SUPPRESS==', use_gpu=True, use_xpu=False, use_npu=False, ir_optim=True, use_tensorrt=False, min_subgraph_size=15, precision='fp32', gpu_mem=500, image_dir=None, page_num=0, det_algorithm='DB', det_model_dir='/home/aistudio/.paddleocr/whl/det/ch/ch_PP-OCRv3_det_infer', det_limit_side_len=960, det_limit_type='max', det_box_type='quad', det_db_thresh=0.3, det_db_box_thresh=0.6, det_db_unclip_ratio=1.5, max_batch_size=10, use_dilation=False, det_db_score_mode='fast', det_east_score_thresh=0.8, det_east_cover_thresh=0.1, det_east_nms_thresh=0.2, det_sast_score_thresh=0.5, det_sast_nms_thresh=0.2, det_pse_thresh=0, det_pse_box_thresh=0.85, det_pse_min_area=16, det_pse_scale=1, scales=[8, 16, 32], alpha=1.0, beta=1.0, fourier_degree=5, rec_algorithm='SVTR_LCNet', rec_model_dir='/home/aistudio/.paddleocr/whl/rec/ch/ch_PP-OCRv3_rec_infer', rec_image_inverse=True, rec_image_shape='3, 48, 320', rec_batch_num=6, max_text_length=25, rec_char_dict INFO - pipelines.nodes.llm.chatglm - 基于以下已知信息，请简洁并专业地回答用户的问题。 如果无法从中得到答案，请说 "根据已知信息无法回答该问题" 或 "没有提供足够的相关信息"。不允许在答案中添加编造成分。另外，答案请使用中文。 已知内容：ChatGLM-6B 🌐 Blog • 🤗 HF Repo • 🐦 Twitter • 📃 [GLM@ACL 22] [GitHub] • 📃 [GLM-130B@ICLR 23] [GitHub] 👋 加入我们的 Slack 和 WeChat介绍ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。结合模型量化技术，用户可以在消费级的显卡上进行本地部署（INT4 量化级别下最低只需 6GB 显存）。ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考我们的博客。为了方便下游开发者针对自己的应用场景定制模型，我们同时实现了基于 P-Tuning v2 的高效参数微调方法 (使用指南) ，INT4 量化级别下最低只需 7GB 显存即可启动微调。不过，由于 ChatGLM-6B 的规模较小，目前已知其具有相当多的局限性，如事实性/数学逻辑错误，可能生成有害/有偏见内容，较弱的上下文能力，自我认知混乱，以及对英文指示生成与中文指示完全矛盾的内容。请大家在使用前了解这些问题，以免产生误解。更大的基于 1300 亿参数 GLM-130B 的 ChatGLM 正在内测开发中。Read this in English.友情链接对 ChatGLM 进行加速的开源项目：* ChatGLM-MNN: 一个基于 MNN 的 ChatGLM-6B C++ 推理实现，支持根据显存大小自动分配计算任务给 GPU 和 CPU* JittorLLMs：最低3G显存或者没有显卡都可运行 ChatGLM-6B FP16， 支持Linux、windows、Mac部署基于或使用了 ChatGLM-6B 的开源项目：* langchain-ChatGLM：基于 langchain 的 ChatGLM 应用，实现基于可扩展知识库的问答* 闻达：大型语言模型调用平台，基于 ChatGLM-6B 实现了类 ChatPDF 功能* chatg 问题：什么是ChatGLM-6B？他的局限性在哪里？如何改进？ user: 什么是ChatGLM-6B？他的局限性在哪里？如何改进？ assistant: ['根据已知信息,可以简洁和专业地回答用户的问题。ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型,基于 General Language Model (GLM) 架构,具有 62 亿参数。结合模型量化技术,用户可以在消费级的显卡上进行本地部署(INT4 量化级别下最低只需 6GB 显存)。ChatGLM-6B 使用了和 ChatGPT 相似的技术,针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练,辅以监督微调、反馈自助、人类反馈强化学习等技术的加持,62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答。但由于其规模较小,目前已知其具有相当多的局限性,如事实性/数学逻辑错误,可能生成有害/有偏见内容,较弱的上下文能力,自我认知混乱,以及对英文指示生成与中文指示完全矛盾的内容。因此,在使用前了解这些问题,以免产生误解。更大的基于 1300 亿参数 GLM-130B 的 ChatGLM 正在内测开发中。'] [('什么是ChatGLM-6B？他的局限性在哪里？如何改进？', '根据已知信息,可以简洁和专业地回答用户的问题。ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型,基于 General Language Model (GLM) 架构,具有 62 亿参数。结合模型量化技术,用户可以在消费级的显卡上进行本地部署(INT4 量化级别下最低只需 6GB 显存)。ChatGLM-6B 使用了和 ChatGPT 相似的技术,针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练,辅以监督微调、反馈自助、人类反馈强化学习等技术的加持,62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答。但由于其规模较小,目前已知其具有相当多的局限性,如事实性/数学逻辑错误,可能生成有害/有偏见内容,较弱的上下文能力,自我认知混乱,以及对英文指示生成与中文指示完全矛盾的内容。因此,在使用前了解这些问题,以免产生误解。更大的基于 1300 亿参数 GLM-130B 的 ChatGLM 正在内测开发中。')] </pre></div></div></div><div class="ai-np-code"><div class="ai-np-code-in">In [ ]</div><div class="ai-np-code-main"><pre><code class="hljs"><span class="hljs-comment"># 执行命令行脚本</span> %cd /home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle !python cli.py</code></pre><div class="ai-np-output-wrapper"><pre class="ai-np-output-stream">/home/aistudio/LangChain-ChatGLM-Webui/paddlepaddle <span style="color:rgb(0,187,0)">[2023-06-17 09:33:43,806] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/ice_text.model <span style="color:rgb(0,187,0)">[2023-06-17 09:33:43,807] [ INFO]</span> - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 <span style="color:rgb(187,187,0)">[2023-06-17 09:33:44,022] [ WARNING]</span> - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/added_tokens.json> not exist <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,023] [ INFO]</span> - Downloading https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json and saved to /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1 <span style="color:rgb(187,187,0)">[2023-06-17 09:33:44,062] [ WARNING]</span> - file<https://bj.bcebos.com/paddlenlp/models/community/THUDM/chatglm-6b-v1.1/special_tokens_map.json> not exist <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,063] [ INFO]</span> - Already cached /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,371] [ INFO]</span> - Found /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,372] [ INFO]</span> - loading configuration file /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,373] [ INFO]</span> - Model config ChatGLMConfig { "_name_or_path": "THUDM/chatglm-6b", "activation": "gelu", "architectures": [ "ChatGLMModel" ], "attention_scale": true, "auto_map": { "AutoConfig": "configuration_chatglm.ChatGLMConfig", "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration", "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration" }, "bos_token_id": 130004, "dtype": "float16", "eos_token_id": 130005, "gmask_token_id": 130001, "hidden_size": 4096, "inner_hidden_size": 16384, "layernorm_epsilon": 1e-05, "mask_token_id": 130000, "max_sequence_length": 2048, "model_type": "chatglm", "num_attention_heads": 32, "num_hidden_layers": 28, "num_image_tokens": 0, "output_predict": true, "pad_token_id": 3, "paddlenlp_version": null, "position_encoding_2d": true, "pre_seq_len": null, "prefix_projection": false, "quantization_bit": 0, "recompute": false, "transformers_version": "4.23.1", "use_cache": true, "vocab_size": 130528 } <span style="color:rgb(0,187,0)">[2023-06-17 09:33:44,415] [ INFO]</span> - Found /home/aistudio/.paddlenlp/models/THUDM/chatglm-6b-v1.1/model_state.pdparams W0617 09:33:52.847257 6771 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2 W0617 09:33:52.851541 6771 gpu_resources.cc:149] device: 0, cuDNN Version: 8.2. <span style="color:rgb(187,187,0)">[2023-06-17 09:34:08,135] [ WARNING]</span> - Some weights of the model checkpoint at THUDM/chatglm-6b-v1.1 were not used when initializing ChatGLMForConditionalGeneration: ['transformer.layers.2.attention.rotary_emb.inv_freq', 'transformer.layers.15.attention.rotary_emb.inv_freq', 'transformer.layers.4.attention.rotary_emb.inv_freq', 'transformer.layers.3.attention.rotary_emb.inv_freq', 'transformer.layers.8.attention.rotary_emb.inv_freq', 'transformer.layers.24.attention.rotary_emb.inv_freq', 'transformer.layers.18.attention.rotary_emb.inv_freq', 'transformer.layers.1.attention.rotary_emb.inv_freq', 'transformer.layers.25.attention.rotary_emb.inv_freq', 'transformer.layers.27.attention.rotary_emb.inv_freq', 'transformer.layers.26.attention.rotary_emb.inv_freq', 'transformer.layers.22.attention.rotary_emb.inv_freq', 'transformer.layers.19.attention.rotary_emb.inv_freq', 'transformer.layers.16.attention.rotary_emb.inv_freq', 'transformer.layers.0.attention.rotary_emb.inv_freq', 'tran - This IS expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). <span style="color:rgb(187,187,0)">[2023-06-17 09:34:08,136] [ WARNING]</span> - Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at THUDM/chatglm-6b-v1.1 and are newly initialized: ['transformer.layers.8.attention.rotary_embeddings.inv_freq', 'transformer.layers.21.attention.rotary_embeddings.inv_freq', 'transformer.layers.12.attention.rotary_embeddings.inv_freq', 'transformer.layers.2.attention.rotary_embeddings.inv_freq', 'transformer.layers.18.attention.rotary_embeddings.inv_freq', 'transformer.layers.15.attention.rotary_embeddings.inv_freq', 'transformer.layers.3.attention.rotary_embeddings.inv_freq', 'transformer.layers.16.attention.rotary_embeddings.inv_freq', 'transformer.layers.7.attention.rotary_embeddings.inv_freq', 'transformer.layers.20.attention.rotary_embeddings.inv_freq', 'transformer.layers.24.attention.rotary_embeddings.inv_freq', 'transformer.layers.26.attention.rotary_embeddings.inv_freq', 'transformer.layers.22.attention.rotary_embeddings.inv_freq', 't You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,193] [ INFO]</span> - We are using (<class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'>, False) to load 'rocketqa-zh-dureader-query-encoder'. <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,193] [ INFO]</span> - Downloading https://bj.bcebos.com/paddlenlp/models/transformers/rocketqa/rocketqa-zh-dureader-vocab.txt and saved to /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-query-encoder <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,244] [ INFO]</span> - Downloading rocketqa-zh-dureader-vocab.txt from https://bj.bcebos.com/paddlenlp/models/transformers/rocketqa/rocketqa-zh-dureader-vocab.txt 100%|██████████████████████████████████████| 89.0k/89.0k [00:00<00:00, 31.8MB/s] <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,398] [ INFO]</span> - tokenizer config file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-query-encoder/tokenizer_config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,398] [ INFO]</span> - Special tokens file saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-query-encoder/special_tokens_map.json <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,400] [ INFO]</span> - Model config ErnieConfig { "attention_probs_dropout_prob": 0.1, "enable_recompute": false, "fuse": false, "hidden_act": "relu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 513, "model_type": "ernie", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 0, "paddlenlp_version": null, "pool_act": "tanh", "task_id": 0, "task_type_vocab_size": 3, "type_vocab_size": 2, "use_task_id": false, "vocab_size": 18000 } <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,400] [ INFO]</span> - Configuration saved in /home/aistudio/.paddlenlp/models/rocketqa-zh-dureader-query-encoder/config.json <span style="color:rgb(0,187,0)">[2023-06-17 09:34:08,401] [ INFO]</span> - Downloading rocketqa_zh_dureader_query_encoder.pdparams from https://bj.bcebos.com/paddlenlp/models/transformers/rocketqa/rocketqa_zh_dureader_query_encoder.pdparams 100%|█████████████████████████████████████████| 379M/379M [00:03<00:00, 100MB/s] <span style="color:rgb(0,187,0)">[2023-06-17 09:34:13,465] [ INFO]</span> - All model checkpoint weights were used when initializing ErnieEncoder. <span style="color:rgb(187,187,0)">[2023-06-17 09:34:13,465] [ WARNING]</span> - Some weights of ErnieEncoder were not initialized from the model checkpoint at rocketqa-zh-dureader-query-encoder and are newly initialized: ['classifier.weight', 'classifier.bias', 'ernie.pooler.dense.bias', 'ernie.pooler.dense.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. <span style="color:rgb(0,187,0)">[2023-06-17 09:34:13,466] [ INFO]</span> - Converting to the inference model cost a little time. I0617 09:34:16.770787 6771 interpretercore.cc:294] New Executor is Running. <span style="color:rgb(0,187,0)">[2023-06-17 09:34:18,386] [ INFO]</span> - The inference model save in the path:/home/aistudio/.paddlenlp/taskflow/rocketqa-zh-dureader-query-encoder/static/inference 提问:</pre></div></div></div><div class="ai-np-code"><div class="ai-np-code-in">In [ ]</div><div class="ai-np-code-main"><pre><code class="hljs"></code></pre></div></div></div></div>
chatGLM+LangChain（部署版）：基于本地知识库的自动问答系统
作者
sockstack
许可协议
CC BY 4.0
发布于
2023-11-06
修改于
2024-09-17
上一篇：软件：常用 Linux 软件汇总，值得收藏
下一篇：ChatGLM:本地化搭建大模型实例