一、任务本次任务如下1.选择NMVM论文中的2个数据集来复现代码链接GitHub - chehaoa/NMVM: An official source code for paper A Non-parametric Multi-view Model for Short Text Clustering · GitHub二、复现选择Tweet和News-TS数据集来复现原因如下1Tweet代表极短、口语化、情感类提问对应心法场景用户在系统里输入的短句。比如“最近好焦虑怎么办”、“如何做到心无旁骛”原因它极其稀疏平均只有 8.56 个词。心法问答中用户往往就扔过来一句话。复现 Tweet 数据集可以验证 NMVM 模型在极度短文本、且包含大量口语和情绪词的情况下双视图融合词袋语义能不能把相似的问题聚在一起。2News-TS代表标题摘要、稍长、解释性强对应心法场景带有背景描述的心法问题或者是“心理咨询对话”中的多轮对话总结。比如“我最近工作压力很大晚上失眠请问有什么冥想的心法”原因News-TS 是标题和摘要的结合平均长度 27.95 个词对应心法的问题答案文本比 Tweet 长语义信息更丰富。复现这个数据集可以验证模型在处理稍长文本时能不能利用语义Embedding来弥补单纯看词的不足。复现过程在github上面下载代码发现作者只给了Tweet的数据集的向量格式为tweetIdclusterNotextCleanedembedding推文的唯一编号真实的分类标签清洗后的文本词袋视图文本嵌入向量嵌入视图1536维作者清洗文本使用的是BoW处理方法没有使用复杂的特征提取方法如 TF-IDF而是采用了最基础的词频统计Word Frequency这源于其 DPMM 模型假设。具体来说在论文第 4.1.1 节中作者对数据进行了以下预处理全部转换为小写(Lowercase)。移除所有非拉丁字符和停用词(Remove non-Latin characters and stop words)。使用 NLTK 的 WordNet Lemmatizer 进行词形还原(Lemmatization)。处理后的文本会变成像brain fluid buildup delay giffords rehab这样的词序列。模型在计算时会直接统计这些词在簇内的出现次数即在 JSON 中看到的textCleaned字段。这种看似“简单”的方法正是为了与基于 Dirichlet 过程的概率模型DPMM相匹配让模型能通过词频共现来推断簇的结构。词嵌入采用OpenAI 的text-embedding-3-small默认1536维。论文在消融实验Table 6中对比了使用SimCSE和OpenAI text-embedding-3-small两种嵌入模型的效果。结果显示使用 OpenAI 嵌入时模型在所有数据集上的 ACC 和 NMI 指标都更高。例如在 News-TS 数据集上使用 OpenAI 嵌入的 ACC 是0.8769而使用 SimCSE 的 ACC 是0.8680。这证明了嵌入模型的质量直接影响最终的聚类性能。Tweet可以可以直接跑。复现结果如下(venv) D:\xinfa\NMVM-mainpython main.py number of documents is 2472 SampleNo:1 2472 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 2472/2472 [00:1 iter is 1 beta is 0.050000. Kcurrent is 146.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 2 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 3 beta is 0.050000. Kcurrent is 100.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 4 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 5 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 6 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 7 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 8 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 9 beta is 0.050000. Kcurrent is 99.000000. Gibbs sampling successful! Start to saving results. Create directory: ./results/TweetK0alpha0.1000beta0.0500sigma0.00040kappa0.5_lambda0.60iterNum15SampleNum5seed2025/ Saving successful! number of documents is 2472 SampleNo:2 2472 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 2472/2472 [00:3 iter is 1 beta is 0.050000. Kcurrent is 145.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 2 beta is 0.050000. Kcurrent is 102.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 3 beta is 0.050000. Kcurrent is 98.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 4 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 5 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 6 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 7 beta is 0.050000. Kcurrent is 99.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 8 beta is 0.050000. Kcurrent is 99.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 2472 SampleNo:3 2472 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 2472/2472 [00:2 iter is 1 beta is 0.050000. Kcurrent is 147.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 2 beta is 0.050000. Kcurrent is 106.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 3 beta is 0.050000. Kcurrent is 103.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 4 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 5 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 6 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 7 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 8 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 9 beta is 0.050000. Kcurrent is 101.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 2472 SampleNo:4 2472 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 2472/2472 [00:1 iter is 1 beta is 0.050000. Kcurrent is 138.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 2 beta is 0.050000. Kcurrent is 100.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 3 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 4 beta is 0.050000. Kcurrent is 96.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 5 beta is 0.050000. Kcurrent is 96.000000. Gibbs Sampling: 100%|█| 2472/2472 [00 iter is 6 beta is 0.050000. Kcurrent is 96.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:0800 iter is 7 beta is 0.050000. Kcurrent is 96.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1200 iter is 8 beta is 0.050000. Kcurrent is 96.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1500 iter is 9 beta is 0.050000. Kcurrent is 96.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 2472 SampleNo:5 2472 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 2472/2472 [00:2100:0 iter is 1 beta is 0.050000. Kcurrent is 143.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1200 iter is 2 beta is 0.050000. Kcurrent is 101.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1200 iter is 3 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1000 iter is 4 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1200 iter is 5 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1100 iter is 6 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1300 iter is 7 beta is 0.050000. Kcurrent is 97.000000. Gibbs Sampling: 100%|█| 2472/2472 [00:1300 iter is 8 beta is 0.050000. Kcurrent is 97.000000. Gibbs sampling successful! Start to saving results. Saving successful! Lambda 0.6 completed! ACC: 0.8299 ± 0.0340 NMI: 0.9301 ± 0.0076 Time: 166.74s一个做了5次实验取平均。真实数据是89个类。可以看到复现结果和论文里面的结果一样接下来复现News-TS首先在github上面找到数据集链接在https://github.com/rashadulrakib/short-text-clustering-enhancement/tree/master/data里面的TS数据集就是我们需要的此数据集里面是分好词的所以我们仅需要生成词嵌入即可在把生成的组合成tweetIdclusterNotextCleanedembedding这个格式。关于生成 词嵌入暂时使用all-MiniLM-L6-v2模型轻量版的SBERT是384维的。考虑到OpenAItext-embedding-3-small生成向量需要花费且需要把数据上传到云端所以暂时使用这个模型先看看效果。下载TS之后创建convert_ts.py并运行转换脚本先安装sentence-transformersimport json from sentence_transformers import SentenceTransformer from tqdm import tqdm print(1. 正在加载 all-MiniLM-L6-v2 模型...) # 加载轻量级 SBERT 模型 model SentenceTransformer(all-MiniLM-L6-v2) input_file TS.txt output_file data-GPT3s/News-TS.txt dataset [] print(2. 正在读取原始数据...) with open(input_file, r, encodingutf-8) as f: for line in f: line line.strip() if not line: continue # 拆分标签第一个空格前和文本 parts line.split(maxsplit1) if len(parts) 2: continue label parts[0] text parts[1] dataset.append({ clusterNo: label, textCleaned: text, tweetId: str(len(dataset)).zfill(6) }) print(f3. 成功读取 {len(dataset)} 条数据。开始生成向量请看进度条...) # 提取文本批量生成向量 texts [item[textCleaned] for item in dataset] # show_progress_bar 显示进度batch_size64 加快速度 embeddings model.encode(texts, show_progress_barTrue, batch_size64) print(\n4. 正在拼装成 JSON 格式并保存...) with open(output_file, w, encodingutf-8) as f: for i, item in enumerate(tqdm(dataset)): # 把 numpy 数组转为 list 存入 item[embedding] embeddings[i].tolist() f.write(json.dumps(item) \n) print(f\n转换成功文件已保存为 {output_file})然后把生成的txt文件后缀改为jsonl完成后在main里面修改这一行parser.add_argument(--dataname, typestr, defaultNews-TS, helpdataset name)训练结果(venv) D:\xinfa\NMVM-mainpython main.py number of documents is 11109 SampleNo:1 11109 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 11109/11109 [02:3500 iter is 1 beta is 0.050000. Kcurrent is 342.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:04 iter is 2 beta is 0.050000. Kcurrent is 244.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:17 iter is 3 beta is 0.050000. Kcurrent is 239.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:15 iter is 4 beta is 0.050000. Kcurrent is 231.000000. Gibbs Sampling: 75%|▊| 8Gibbs Sampling: 75%|▊| 8Gibbs Sampling: 75%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 76%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 77%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 78%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 79%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 80%|▊| 8Gibbs Sampling: 81%|▊| 8Gibbs Sampling: 81%|▊| 8Gibbs Sampling: 81%|▊| 8Gibbs Sampling: 81%|▊| 8Gibbs Sampling: 81%|▊| 8Gibbs Sampling: 81%|▊| 9Gibbs Sampling: 81%|▊| 9Gibbs Sampling: 81%|▊| 9Gibbs Sampling: 81%|▊| 9Gibbs Sampling: 81%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 82%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 83%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 84%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 85%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 86%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 87%|▊| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 88%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 89%|▉| 9Gibbs Sampling: 90%|▉| 9Gibbs Sampling: 90%|▉| 9Gibbs Sampling: 90%|▉| 9Gibbs Sampling: 90%|▉| 9Gibbs Sampling: 90%|▉| 9Gibbs Sampling: 90%|▉| 1Gibbs Sampling: 90%|▉| 1Gibbs Sampling: 90%|▉| 1Gibbs Sampling: 90%|▉| 1Gibbs Sampling: 90%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 91%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 92%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 93%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 94%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 95%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 96%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 97%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 98%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 99%|▉| 1Gibbs Sampling: 100%|▉| 1Gibbs Sampling: 100%|▉| 1Gibbs Sampling: 100%|▉| 1Gibbs Sampling: 100%|▉| 1Gibbs Sampling: 100%|▉| 1Gibbs Sampling: 100%|█| 11109/11109 [01:59 iter is 5 beta is 0.050000. Kcurrent is 226.000000. Gibbs Sampling: 100%|█| iter is 6 beta is 0.050000. Kcurrent is 224.000000. Gibbs Sampling: 100%|█| iter is 7 beta is 0.050000. Kcurrent is 224.000000. Gibbs Sampling: 100%|█| iter is 8 beta is 0.050000. Kcurrent is 222.000000. Gibbs Sampling: 100%|█| iter is 9 beta is 0.050000. Kcurrent is 222.000000. Gibbs Sampling: 100%|█| iter is 10 beta is 0.050000. Kcurrent is 222.000000. Gibbs Sampling: 100%|█| 11109/11109 [03:27 iter is 11 beta is 0.050000. Kcurrent is 222.000000. Gibbs sampling successful! Start to saving results. Create directory: ./results/News-TSK0alpha0.1000beta0.0500sigma0.00040kappa0.5_lambda0.60iterNum15SampleNum5seed2025/ Saving successful! number of documents is 11109 SampleNo:2 11109 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 11109/11109 [02:5600 iter is 1 beta is 0.050000. Kcurrent is 358.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:07 iter is 2 beta is 0.050000. Kcurrent is 237.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:05 iter is 3 beta is 0.050000. Kcurrent is 231.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:49 iter is 4 beta is 0.050000. Kcurrent is 217.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:43 iter is 5 beta is 0.050000. Kcurrent is 221.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:43 iter is 6 beta is 0.050000. Kcurrent is 216.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:42 iter is 7 beta is 0.050000. Kcurrent is 215.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:51 iter is 8 beta is 0.050000. Kcurrent is 215.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 11109 SampleNo:3 11109 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 11109/11109 [02:4600 iter is 1 beta is 0.050000. Kcurrent is 329.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:00 iter is 2 beta is 0.050000. Kcurrent is 240.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:51 iter is 3 beta is 0.050000. Kcurrent is 214.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:49 iter is 4 beta is 0.050000. Kcurrent is 213.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:57 iter is 5 beta is 0.050000. Kcurrent is 206.000000. Gibbs Sampling: 100%|█| 11109/11109 [03:13 iter is 6 beta is 0.050000. Kcurrent is 206.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:28 iter is 7 beta is 0.050000. Kcurrent is 206.000000. Gibbs Sampling: 100%|█| 11109/11109 [03:25 iter is 8 beta is 0.050000. Kcurrent is 206.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 11109 SampleNo:4 11109 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 11109/11109 [03:0100 iter is 1 beta is 0.050000. Kcurrent is 366.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:07 iter is 2 beta is 0.050000. Kcurrent is 228.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:48 iter is 3 beta is 0.050000. Kcurrent is 215.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:46 iter is 4 beta is 0.050000. Kcurrent is 214.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:46 iter is 5 beta is 0.050000. Kcurrent is 211.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:47 iter is 6 beta is 0.050000. Kcurrent is 211.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:49 iter is 7 beta is 0.050000. Kcurrent is 211.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:08 iter is 8 beta is 0.050000. Kcurrent is 211.000000. Gibbs sampling successful! Start to saving results. Saving successful! number of documents is 11109 SampleNo:5 11109 documents will be analyze. alpha is 0.1000. beta is 0.0500. sigma is 0.00040. kappa is 0.5. lambda is 0.6. K is 0.0. Initialization. Initializing: 100%|█| 11109/11109 [03:5200 iter is 1 beta is 0.050000. Kcurrent is 362.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:26 iter is 2 beta is 0.050000. Kcurrent is 242.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:59 iter is 3 beta is 0.050000. Kcurrent is 233.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:57 iter is 4 beta is 0.050000. Kcurrent is 231.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:57 iter is 5 beta is 0.050000. Kcurrent is 228.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:59 iter is 6 beta is 0.050000. Kcurrent is 233.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:00 iter is 7 beta is 0.050000. Kcurrent is 228.000000. Gibbs Sampling: 100%|█| 11109/11109 [01:59 iter is 8 beta is 0.050000. Kcurrent is 225.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:00 iter is 9 beta is 0.050000. Kcurrent is 226.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:01 iter is 10 beta is 0.050000. Kcurrent is 225.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:00 iter is 11 beta is 0.050000. Kcurrent is 225.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:01 iter is 12 beta is 0.050000. Kcurrent is 225.000000. Gibbs Sampling: 100%|█| 11109/11109 [02:02 iter is 13 beta is 0.050000. Kcurrent is 225.000000. Gibbs sampling successful! Start to saving results. Saving successful! Lambda 0.6 completed! ACC: 0.8592 ± 0.0131 NMI: 0.9422 ± 0.0030 Time: 1376.17s正确的分类是152和论文对比在向量维度缩小了4倍的情况下差距不到2%印证了极强的鲁棒性。之前看到模型最后分出了 222 个簇含有很多长尾的单例簇吗即便在簇数被过度切分高于真实的 152 类的情况下模型依然拿到了0.94 的超高 NMI。这说明模型即使在 Embedding 表达能力不足时通过词袋视图λ0.6λ0.6的强力补救依然保持了极佳的语义一致性。对心法数据的启示这个方法适用于短文本且心法的问题也属于短文本所以感觉挺适合的。第一处文本必须“空格分词”代码里的Document.py有一句ws text.strip().split( )。它默认是用空格切分单词的。英文天生有空格比如taylor swift但中文没有不可能把“如何克服焦虑”按空格切开。所以在用 Python 处理中文数据时必须用jieba分词库。把中文数据变成如何 克服 焦虑 放下 执念然后用空格连起来喂给模型。第二处生成向量的模型必须换成“中文模型”现在用的all-MiniLM-L6-v2是纯英文模型。你需要换成支持中文的sentence-transformers模型比如BAAI/bge-large-zh-v1.5shibing624/text2vec-base-chinese轻量级速度快到时候把向量生成脚本里的模型名字换掉就行了。第三处去掉中文的“废话词”英文里有the, a, is中文里有的, 了, 是, 在, 我。这些词出现频率极高但对聚类没有任何帮助反而会干扰模型。需要找一份中文停用词表在分词的时候把这些词过滤掉。而上面三种在我们最开始的里面全部都使用过所以倒是可以直接套用一下。以及之前使用bge模型很慢的原因好像就是因为是实时生成向量的好像之前bert的代码就是实时的然后我在原来的代码上修改所以才会特别慢但是好像不止加载的时候特别慢查询的时候页面也是卡住的大概1-2分钟才会刷新一下是因为查询时重新加载模型了吗还是每次查询的时候都在把数据库重新刷一遍可以存成一共numpy.npy文件或者存进向量数据库或者web框架用的是 Flask 或 Django 这样的传统框架且没有开启多线程当后端在努力计算向量时整个网页是被“锁死”的。前端浏览器傻傻地等后端算完才能刷新。如果是这种情况哪怕只算 1 秒钟用户体验也是卡了 1 秒。
