XnneHang

XnneHang

github
steam_profiles
douban
HomeArchives作品集游戏

Deepin uses Docker to compile PaddlePaddle

126
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The document outlines the process of compiling PaddlePaddle using Docker on a Deepin system. It begins by emphasizing the importance of using `git clone` instead of downloading a ZIP file to ensure necessary git hash information is available during compilation. Key steps include: 1. **Clone the PaddlePaddle Source Code**: Use `git clone https://github.com/PaddlePaddle/Paddle.git` to obtain the source code. 2. **Pull the PaddlePaddle Docker Image**: Execute `docker pull paddlepaddle/paddle:latest-dev`. 3. **Run the Docker Container**: Navigate to the source code directory and run the Docker container with the command: ```bash docker run --name paddle-test -v $PWD:/paddle --network=host -it paddlepaddle/paddle:latest-dev /bin/bash ``` This command maps the current directory to the container and shares the host's network settings. 4. **Configure Proxy Settings**: Set up HTTP and HTTPS proxy variables and configure git to use these proxies. 5. **Install Dependencies**: Install necessary dependencies using `pip` and `apt`, including `protobuf`, `patchelf`, and `networkx`. 6. **Install Python Requirements**: Run `pip install -r python/requirements.txt` to install additional Python dependencies. 7. **Switch to the Develop Branch**: Use `git checkout develop` to switch to the appropriate branch for compilation. 8. **Compile PaddlePaddle**: Create a build directory, run `cmake`, and then compile using `make -j10` for multi-core processing. 9. **Install the Compiled Package**: Use Miniconda to create an environment and install the generated `.whl` file. The document also advises ensuring a stable network connection to avoid compilation errors and suggests checking for missing third-party dependencies if issues arise during the process.

Using Docker to Compile PaddlePaddle on Deepin#

Reference:#

Preface:#

This is just a record of the process of pulling the PaddlePaddle image from Docker and then compiling PaddlePaddle. The installation of Docker, how to configure docker_proxy, and how to access GitHub from within the container are assumed to be known.

Of course, this is just a direction; I have written something similar before:

Additionally, a piece of advice: please ensure that the nodes you are using are stable, and try to avoid strange errors during the GitHub clone process. I previously used a very fast node, but it failed several times in forty minutes, resulting in incomplete third-party dependencies.

Every time you mess with dependencies, it takes a lot of time, especially when you need to check if you are missing certain third-party packages.

Steps and Ideas:#

We write our code locally and then compile it inside the Docker container. The container maps to the code directory, and the compiled whl files are generated directly in the build folder at the root of the source code. Then, we install and test the compiled whl files using the local Miniconda virtual environment.

Clone PaddlePaddle Source Code:#

You need to ensure you use clone instead of download zip, as some git hash and other information are needed during compilation.

You can use your local fork repository, but you need to ensure that certain tags are the same, such as develop; otherwise, there will be issues during compilation.

git clone https://github.com/PaddlePaddle/Paddle.git

Pull PaddlePaddle Image:#

docker pull paddlepaddle/paddle:latest-dev

Enter the Root Directory of the Source Code to Run the PaddlePaddle Image:#

cd Paddle/
docker run --name paddle-test -v $PWD:/paddle --network=host -it paddlepaddle/paddle:latest-dev /bin/bash

--network=host means we use the host's localhost instead of the container's.

My http_proxy and https_proxy are set to the host's localhost:7890, which is then shared with the container.

-v $PWD:/paddle means I am mapping the current directory to the container's /paddle directory, sharing the root of the source code.

Inside the container, you can directly use cd /paddle to see the root of the source code. Subsequently, the build directory, third-party dependencies, and the final generated whl files will all be here, and these generated files will also sync to the host's source code root. The next time you come in to compile, you won't need to repeatedly download third-party dependencies; the cache from the initial compilation will also remain, allowing for a secondary compilation that can reduce the time from about an hour to four or five minutes or even tens of seconds.

Configure http_proxy and https_proxy:#

These are user variables and git config; the proxy here is my own, and you need to configure it according to your own proxy.

export http_proxy=http://localhost:7890
export https_proxy=http://localhost:7890
git config --global http.proxy http://localhost:7890
git config --global https.proxy http://localhost:7890

Install Some Dependencies Required by Make:#

pip install protobuf
apt install patchelf

And very importantly, networkx, which is not mentioned in the official documentation, but if it's missing, the compilation will ultimately fail.

Ensure you are in the root directory:

λ xnne-PC /paddle ls
AUTHORS.md             paddle/         security/
build/                 patches/        SECURITY_cn.md
cmake/                 pyproject.toml  SECURITY_ja.md
CMakeLists.txt         python/         SECURITY.md
CODE_OF_CONDUCT_cn.md  r/              setup.py
CODE_OF_CONDUCT.md     README_cn.md    test/
CONTRIBUTING.md        README_ja.md    third_party/
doc/                   README.md       third_party.tar.gz
LICENSE                RELEASE.md      tools/

Then:

pip install -r python/requirements.txt

Switch to the Develop Branch:#

If you directly run git checkout develop as in the documentation, it will lead to not finding the hash of the develop tag later, but grep can find that hash.

Of course, I suggest you run it like this first:

git checkout develop

Because I have this operation in my process, I am not sure if missing it will have some impact.

Check the current branch:

λ xnne-PC /paddle git branch -a
* (HEAD detached at origin/develop)
  develop
  remotes/origin/1.8.5
  remotes/origin/HEAD -> origin/develop
  remotes/origin/a246d2cc8876d9efb0b733f0ae02e1bd973
  remotes/origin/add_kylinv10
  remotes/origin/ascendrelease
  remotes/origin/cinn-trivalop-fuse
  ...

You can check if there is remotes/origin/HEAD; if not, you can run git fetch.

Of course, I recommend running git fetch, as it seems that the compilation later checks some tag hashes, and if you don't fetch, you might miss some tag hashes.

At this point, I ran:

git checkout origin/develop

It will pop up some notes like this:

λ xnne-PC /paddle git checkout origin/develop
Note: switching to 'origin/develop'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you see the note, it should be successful, and then you can proceed to the next step.

Compile PaddlePaddle:#

Ensure you are in the root directory:

cd /paddle
mkdir build/
cd build/

Then run cmake; you need to confirm your DPY_VERSION, which determines the Python version of the whl you compile.
#

git config --global --add safe.directory '*' # This solves interrupted input
time cmake .. -DPY_VERSION=3.10 -DWITH_GPU=OFF

The first time you run this will take a long time, depending on your network speed. Also, as I mentioned earlier, please ensure your node is stable; otherwise, you may encounter some strange errors.

Additionally, you may need to interrupt to input some commands, such as:

git config --global --add safe.directory /paddle/third_party/absl

And there are quite a few; you can pay attention to them. Of course, if you find a solution, you can let me know; otherwise, constant interruptions can be quite annoying.

ps: I found a method here:

git config --global --add safe.directory '*'

The previous command git config --global --add safe.directory /paddle did not work here.

And after finishing, you can run it again to see if any third-party dependencies are missing; if they are still missing, you can manually clone them.

Multi-core Compilation (I used 10 cores):
#

make -j10

It will also take a long time...

Verify Compilation Results:#

If everything went well, you should see some whl files in the build directory at the root of the source code.

At this point, we use Miniconda to install it:

conda activate paddle-test # Create your own, ensure DPY_VERSION matches your environment
pip install -U build/python/dist/paddlepaddle-0.0.0-cp39-cp39-linux_x86_64.whl

python

>>> import paddle
/home/xnne/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py:686: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
  warnings.warn(warning_message)
>>> paddle.utils.run_check()
Running verify PaddlePaddle program ...
I1105 10:44:10.085186 70641 pir_interpreter.cc:1492] New Executor is Running ...
I1105 10:44:10.085837 70641 pir_interpreter.cc:1515] pir interpreter is running by multi-thread mode ...
PaddlePaddle works well on 1 CPU.
PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.
>>> exit()

The first warning seems to be resolved by apt install ccache, but I haven't tried it.

Seeing PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now. means you have compiled successfully.

Secondary Compilation:#

  • Modify the underlying header file: paddle/fluid/platform/enforce.h
  • Modify the Op's cc file: paddle/fluid/operators/rank_loss_op.cc
  • Modify the Python file: python/paddle/tensor/math.py

I plan to put these in the next blog post.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.