使用llama.cpp的Qwen3.6-27B-Q8本地部署详解【Linux GPU】

0. 检查cmakecuda的安装情况及版本cmake更新参考https://askubuntu.com/questions/829310/how-to-upgrade-cmake-in-ubuntucmake --version无结果ln一下注意cmake版本sudoln-s/usr/local/bin/cmake /usr/bin/cmakecuda安装参考https://blog.csdn.net/qxqsunshine/article/details/1616645931. 创建文件夹安装llama.cppmkdirLLMcdLLMgitclone https://github.com/ggml-org/llama.cppcdllama.cppmkdir-pbuildcdbuild2.编译llama.cppcd..# 回到有CMakeLists.txt文件的llama.cpp目录下cmake..-DGGML_CUDAON-DCMAKE_BUILD_TYPERelease-DCMAKE_CUDA_ARCHITECTURES89-DGGML_NATIVEON# 此处nvcc --version无结果就要看看cuda了cmake--build.--configRelease-j$(nproc)./bin/llama-cli--help3. 安装huggingface library并下载模型pipinstallhuggingface_hubcdbuild# 到LLM/llama.cpp/build目录下# 选择好适合自己的模型hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q8_0.gguf --local-dir ./models4.选择适合自己的参数配置运行模型cd../../# 回到LLM目录下# 这里我选择了深度思考模式参数配置如下执行代码运行模型./llama.cpp/build/bin/llama-server-m./llama.cpp/build/models/Qwen3.6-27B-Q8_0.gguf-ngl999-c32768--temp1.0--top-p0.95--top-k20--min_p0.0--host0.0.0.0--port8080# 保持该界面不要退出让模型在后端运行# 检查一下GPU CPU使用情况确保模型在指定设备运行5.进入对话界面在本地网页打开或者输入主机ip用其他设备浏览器输入 http:xxx.xxx.xxx.xxx:8080 打开(6. 启动多模态分析)以上步骤只能启用文本分析功能无法处理图像。如果想开启图像处理功能需要安装相关模型并启动。# 安装mmproj-F16.ggufhf download unsloth/Qwen3.6-27B-GGUF --local-dir ./models--include*mmproj-F16*# 执行代码运行模型./llama.cpp/build/bin/llama-server-m./llama.cpp/build/models/Qwen3.6-27B-Q8_0.gguf--mmproj./llama.cpp/build/models/mmproj-F16.gguf-ngl999-c32768--temp1.0--top-p0.95--top-k20--min_p0.0--host0.0.0.0--port8080本文步骤主要参考来源https://blog.csdn.net/qxqsunshine/article/details/161664593https://www.youtube.com/watch?vEONM2W1gUFY次要来源https://github.com/ZengboJamesWang/Qwen3.5-35B-A3B-openclaw-dgx-sparkhttps://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUFhttps://unsloth.ai/docs/models/qwen3.6#mtp-qwen3.6-27b