CANN SHMEM初始化终止流程

SHMEM 初始化与终止流程【免费下载链接】shmemCANN SHMEM 是面向昇腾平台的多机多卡内存通信库基于OpenSHMEM 标准协议实现跨设备的高效内存访问与数据同步。项目地址: https://gitcode.com/cann/shmem本文说明 SHMEM 初始化与终止的流程、参数与源码实现。除下文列出的对外接口外其余函数为内部实现用户不应直接调用。阅读建议快速了解读第一、二章实现细节读第三、四章并对照src/host/init/shmem_init.cppDefault/UIDConfig Store 控制面TCP 建链、KV 协议、socket 绑定详见 config_store_bootstrap.md查函数落点见第五章排障见第六章。一、概述1.1 流程概要各 PE 对称调用aclshmemx_init_attr完成初始化对称调用aclshmem_finalize/aclshmemx_finalize完成终止。须满足① 每个 PE 均参与 init/finalize② 除my_pe外 init 参数一致③ 各 rank 的aclshmem_malloc/aclshmem_free同序同大小。初始化依次完成Bootstrap控制面CPU建立 rank 间连接Config Store/TCP 或 MPI通过内部barrier/allgather交换建堆元数据并同步进度。HYBM 建堆数据面NPUreserve_heap预留 GVAsetup_heap分配内存、交换 slice/entity 并mmap。此后同序同大小malloc时堆内偏移一致可进行aclshmem_putmem/aclshmem_getmem等 RMA。子模块就绪堆分配器、team、signal、平台同步等完成应用可调用 SHMEM API。终止按相反顺序释放子模块 → remove/release 对称堆 →finalize_device_state→release_aclshmem_entity→ ACL stream →aclshmemi_control_barrier_all控制面集合同步→ Bootstrap →init_manager。Bootstrap 内部barrier用于 init/finalize 控制流程不同于运行期aclshmem_barrierteam 级业务同步。多 instance 时init_manager、HYBM 全局初始化、Config Store 由引用计数共享最后一个 instance finalize 后释放。1.2 核心概念概念说明PE / rank参与 SHMEM 的进程my_pe为本 PE 编号从 0 起n_pes为 PE 总数。对称堆各 PE 各有一块等长堆同序同大小的malloc得到相同堆内偏移。Bootstrapinit 阶段主机侧通信层用于建链、交换元数据、保证malloc顺序一致。HYBM底层混合内存库entity为内存域句柄slice为物理内存GVA为预留虚拟地址。instance同一进程内多套 SHMEM 上下文instance_id默认 0。host / device statehost 维护g_state算子读同步到 NPU 元数据区的 device 副本。1.3 典型用法UID 模式无 MPI、不写ip_portrank 0: aclshmemx_get_uniqueid(uid) // 约定 rank 0 生成见 examples/init rank 0: 广播 uid // 库不负责广播 各 rank: aclshmemx_set_attr_uniqueid_args(my_pe, n_pes, heap_size, uid, attr) 各 rank: aclshmemx_init_attr(ACLSHMEMX_INIT_WITH_UNIQUEID, attr) 各 rank: aclshmem_finalize()MPI 模式aclshmemx_init_attr(ACLSHMEMX_INIT_WITH_MPI, attr)comm_args为MPI_Comm*nullptr即MPI_COMM_WORLD。attr中的my_pe/n_pes须与MPI_Comm_rank/MPI_Comm_size一致二者分别写入g_state与 Bootstrap 句柄不一致会导致后续行为异常。Default 模式已知 rank 0 地址各 rank: attr.ip_port tcp://rank0_host:port并填 my_pe / n_pes / local_mem_size 各 rank: aclshmemx_init_attr(ACLSHMEMX_INIT_WITH_DEFAULT, attr) 各 rank: aclshmem_finalize()完整样例见examples/init。多 instance 见 multi_instance.md。1.4 初始化状态aclshmemx_init_status()含义ACLSHMEM_STATUS_NOT_INITIALIZEDis_aclshmem_created falseACLSHMEM_STATUS_SHM_CREATED堆已建子模块初始化中中间态ACLSHMEM_STATUS_IS_INITIALIZED可调用aclshmem_malloc、RMA、team 等同一instance_id不可重复 initsetup_heap置is_aclshmem_created true后不得再次调用aclshmemx_init_attr。1.5 终止约束每个 PE 均须 finalize。finalize 前等待 device kernel / stream 完成各 PE 须对称进入 finalize以便 Bootstrap 拆除前经aclshmemi_control_barrier_all在 Config Store 上对齐见 §4.2。finalize 后不得再调用 SHMEM API。二、对外接口速览初始化接口说明aclshmemx_get_uniqueid约定 rank 0 调用生成 UID含addr、magic。aclshmemx_set_attr_uniqueid_args(my_pe, n_pes, local_mem_size, uid, attr)。aclshmemx_init_attr主入口所有 PE 对称调用。aclshmemx_init_status查询初始化阶段。aclshmemx_instance_ctx_get/set多 instance 获取/切换上下文。aclshmem_info_get_version/get_name库版本与名称。终止接口说明aclshmem_finalize释放当前活动 instance。aclshmemx_finalize按instance_id释放。算子内辅助接口说明util_get_ffts_config/util_set_ffts_config910B/C 算子内 FFTS 基址950 可不调用§3.3.7.4。初始化前可选配置接口说明aclshmemx_set_config_store_tls_key/aclshmemx_set_conf_store_tlsConfig Store TLS 配置。三、初始化详解3.1 总览入口为aclshmemx_init_attrshmem_init.cpp互斥锁保护。下图与下表对应 §3.3 各步细节见 §3.2参数与 §3.3分步。步骤源码调用文件1 多实例aclshmemi_instance_ctx_create→aclshmemx_instance_ctx_set_implshmem_init.cpp2 校验aclshmemx_init_status须NOT_INITIALIZED→set_log_level→check_attr→version_compatibleshmem_init.cpp3 Bootstrapaclshmemi_bootstrap_initshmemi_bootstrap.cpp4 Stateaclshmemi_state_init_attrshmem_init.cpp5 Backendnew aclshmemi_init_backend()g_init_manager_countshmem_init.cpp6 堆bind_aclshmem_entity→init_device_state→reserve_heap→setup_heapshmem_init_backend.cpp7 子模块memory_manager_initialize→可选reserve_heap(HOST)→aclshmemi_signal_init→aclshmemi_team_init→aclshmemi_sync_initshmem_init.cpp等8 收尾is_aclshmem_initialized true→prof_util_init→update_device_state→aclshmemi_control_barrier_allshmem_init.cppsetup_heap末尾is_aclshmem_created true步骤 8 开头is_aclshmem_initialized true。instance_id ! 0首次创建失败时scope guard 调用aclshmemi_instance_ctx_destroy(id)进入 init 时状态须为ACLSHMEM_STATUS_NOT_INITIALIZED。3.2 参数与 Bootstrap各 PE 在aclshmemx_init_attr_t中填写my_pe、n_pes、local_mem_size并选择 Bootstrap 模式。库实际映射local_mem_size ACLSHMEM_EXTRA_SIZEg_state.heap_size。Bootstrap 模式模式需准备ACLSHMEMX_INIT_WITH_DEFAULT合法ip_port或 fallback 到 UIDACLSHMEMX_INIT_WITH_UNIQUEIDrank 0 生成 UID 并广播ACLSHMEMX_INIT_WITH_MPIMPI_Comm*仅MPI_COMM_WORLD/MPI_COMM_SELFattr.my_pe/n_pes与 MPI rank/size 一致Default/UID 走aclshmem_bootstrap_config_store.soMPI 走aclshmem_bootstrap_mpi.so。图aclshmemi_bootstrap_init按bootstrap_flags三分支选择插件Default 先校验ip_port无效时 fallback 到 UID。Default / UID 选定aclshmem_bootstrap_config_store.so后各 PE 的 CPU 进程以 PE 0 为中心建立星型连接。PE 0 运行AccStoreServer维护全局键值表各 PE含 PE 0经同一ConnectToPeerServer建 clientSmemNetGroupEngine在键值表上实现 init 阶段的barrier/allgather。该控制面建链与后续 NPU 对称堆HYBMexchange_slice相互独立。深入阅读config_store_bootstrap.md 基于源码详解 Config Store 分层、监听/连接的 IP:port、Acclink 握手、KV 消息帧格式与 barrier/allgather 键协议仅 CPU 控制面不涉及 NPU。图Default/UID 模式下以 PE 0 为中心的星型拓扑含 PE 0 client在内的各 PE 均经ConnectToPeerServer连向 PE 0 listener。UID 的addr来自SHMEM_UID_SESSION_ID或SHMEM_UID_SOCK_IFNAMEmagic为随机通信域标识符。全体 rank 的 UID 须字节级一致。超时参数参数含义单位 / 说明shm_init_timeoutDefault 且ip_port合法时非 PE 0client 连 PE 0 的上限API 字段名为超时秒默认 120Config Store实现将其作为ConnectToPeerServer重试次数每次失败sleep(1)最坏约 N 秒详见 config_store_bootstrap.md §10.1control_operation_timeoutbarrier/allgather 等待上限秒UID / Default→UIDtimeOut同shm_init_timeout的建链语义固定DEFAULT_TIMEOUT120作重试次数UID / Default→UIDtimeControlOut控制面集合通信超时秒×1000 →timeoutMsAcclink 收包ACC_LINK_RECV_TIMEOUT秒1800常量aclshmemx_init_attr_t关键字段字段说明my_pe/n_pes本 PE 编号与 PE 总数0 my_pe n_peslocal_mem_size每 PE 用户堆大小ip_portDefaulttcp://host:port或tcp6://[host]:port端口 1024comm_argsDefault/UIDaclshmemx_uniqueid_t*MPIMPI_Comm*instance_id多 instance 标识默认 0option_attr.data_op_engine_type传输通路位或见下表option_attr.shm_init_timeout传入CreateStore的connMaxRetry实现为重试次数见 config_store_bootstrap.md §10.1option_attr.control_operation_timeout转为SmemGroupOption.timeoutMsoption_attr.sockFdrank 0 预绑定 socket传输通路标志标志通路ACLSHMEM_DATA_OP_MTEMTEACLSHMEM_DATA_OP_SDMASDMAACLSHMEM_DATA_OP_ROCERDMA/ROCEACLSHMEM_DATA_OP_UDMAUDMAsetup_heap后reach_info_init写入topo_list供 RMA 选路。Config Store 结构Default/UID概要aclshmemi_bootstrap_init→dlopen→plugin_init→init_config_store→init_group_engine填充g_boot_handle.barrier/allgather。socket 绑定、连接目标、消息校验等细节见 config_store_bootstrap.md。层次职责TcpConfigStore各 rank clientGet/Set/Add/AppendAccStoreServerrank 0全局键值表AcclinkTCP 建链、握手AccConnReq含rankIdmagic/version 校验、可选 TLSSmemNetGroupEngine键值上实现GroupBarrier/GroupAllGatherrank 0CreateStore(..., isServertrue)监听各 PE client 的connMaxRetryPE 0 固定 60 次非 PE 0 为shm_init_timeout重试次数。多 instance 共享g_store_ref引用计数。Bootstrapbarrier用于exchange_slice同步与aclshmem_malloc同序约束。运行时产物libshmem.so、libshmem_utils.so、aclshmem_bootstrap_config_store.so、aclshmem_bootstrap_mpi.so。3.3 分步说明以下按 §3.1 调用序展开。3.3.1 多实例上下文同一进程可通过instance_id维护多套 SHMEM 上下文。多 instance 共享init_manager与 HYBM 全局初始化各有独立堆与 Bootstrap 句柄。aclshmemi_instance_ctx_create仅instance_id ! 0instance_id ! 0且ip_port非空或comm_args nullptr时调用aclshmemi_instance_port_selectionURL 模板端口须为0按SHMEM_INSTANCE_PORT_RANGE填入start_port instance_id纯 UIDip_port为空且comm_args有效跳过aclshmemx_instance_ctx_set_impl切换g_state、g_boot_handle等详见 multi_instance.md。3.3.2 配置校验aclshmemx_init_status()须为ACLSHMEM_STATUS_NOT_INITIALIZEDis_aclshmem_created false禁止重复 initaclshmemx_set_log_level(ERROR_LEVEL)SHMEM_LOG_LEVEL可覆盖check_attrPE 范围、local_mem_size、超时非 0、data_op_engine_typebitmaskversion_compatible3.3.3 Bootstrap 加载aclshmemi_bootstrap_initdlopen→plugin_init→is_bootstraped true模式分支见 §3.23.3.4 State 初始化aclshmemi_state_init_attrg_state.mype/npes/heap_sizeaclrtCreateStream→default_stream3.3.5 Backendg_init_manager_countinit_manager nullptr时new aclshmemi_init_backend()否则复用已有实例-mssanitizer构建且以mssanitizer --运行时可向检测工具注册元数据区与堆 GVA3.3.6 Entity 与对称堆调用链bind_aclshmem_entity→init_device_state→reserve_heap→setup_heap。3.3.6.1bind_aclshmem_entity在entity_map_登记entity_member绑定 init 参数、g_state、Bootstrap 句柄、device state 镜像。3.3.6.2init_device_statehybm_init_count 0时hybm_init映射元数据区约 32MB非用户堆hybm_init_count最后一个 finalize 时hybm_uninitaclshmemi_control_barrier_all()3.3.6.3reserve_heapcreate_entity→hybm_create_entityHYBM_ROLE_PEERhybm_reserve_mem_space→hbm_gva可选MSTX 注册 GVA3.3.6.4setup_heaphybm_alloc_local_memory→ sliceexchange_slicehybm_export→allgather→hybm_import→barrierexchange_entity流程同exchange_slicehybm_export返回descLen 0时跳过 allgather/importhybm_mmap分配p2p_*/rdma_*/sdma_*堆基址数组UDMA 复用 P2Preach_info_init→topo_listheap_base hbm_gva ALIGN_UP(heap_size, ACLSHMEM_HEAP_ALIGNMENT_SIZE) * my_peis_aclshmem_created true3.3.6.5 host/device state 同步算子读 device 元数据区副本。update_device_state()拷贝g_state→entity_device_state保留 heap 基址指针再hybm_set_extra_context。3.3.7 子模块3.3.7.1 共享内存管理器在heap_base上 first-fit 管理对称堆每次aclshmem_malloc/aclshmem_free后aclshmemi_control_barrier_all。memory_manager_initialize(heap_base, heap_size)DEBUG 模式检is_alloc_size_symmetricd2hinit 时reserve_heap(HOST_SIDE)首次 host malloc 时 lazysetup图对称堆上的 first-fit 分配器aclshmem_malloc按 First Fit 切分空闲块aclshmem_free归还并合并相邻空闲块。各 rank 还须保持相同顺序、相同大小的malloc/free才能保证堆内偏移对称。下图对比合法与非法的分配模式图对称堆内偏移对齐示意正确 vs 错误非 malloc 调用序列图。3.3.7.2 Signalaclshmemi_signal_initaclshmem_malloc(512)→signal_addr清零ACLSHMEM_SIGNAL_SIZE8 字节3.3.7.3 Teamaclshmemi_team_initACLSHMEM_TEAM_WORLDteam_poolsync_pool/sync_countercore_sync_*图卡间/核间 Sync Counter 与 Sync Pool 状态结构team_pool等见上文列表。详见 team.md。3.3.7.4 同步FFTS平台行为Ascend910B/Cinit 时rtGetC2cCtrlAddr算子入口util_set_ffts_config(util_get_ffts_config())Ascend950不定义FFTS_CONFIG由 team 同步池完成 barrier3.3.7.5 收尾is_aclshmem_initialized true→prof_util_init→update_device_state→aclshmemi_control_barrier_all→ 释放 scope guard四、终止Finalize详解4.1 总览aclshmem_finalize/aclshmemx_finalize按与 init 相反的顺序释放资源。多 instance 时init_manager、hybm_init、g_store_ref由引用计数管理。接口行为aclshmem_finalize()finalize 当前活动 instanceaclshmemx_finalize(id)目标非当前活动时先aclshmemx_instance_ctx_set_impl(id)图aclshmemi_finalize_impl顺序在aclshmemi_bootstrap_finalize前经aclshmemi_control_barrier_all做 Config Store 控制面同步。4.2 执行顺序经aclshmemi_finalize_implmutex 保护必要时aclshmemx_instance_ctx_set_implaclshmemi_team_finalize→aclshmemi_signal_finalize→memory_manager_destroyremove_heap→release_heap→finalize_device_state→可选 host→release_aclshmem_entityaclrtSynchronizeStream→aclrtDestroyStreamaclshmemi_control_barrier_all— 经 Config StoreGroupBarrier同步全体 PE 均进入 finalize 并到达此点后才允许拆 Bootstrap见 config_store_bootstrap.md §11aclshmemi_bootstrap_finalizeg_store_ref--归零时DestroyStoreg_init_manager_count--归零delete init_manageris_aclshmem_initialized falseaclshmemi_instance_ctx_destroyinstance_id 0为 no-opinit_manager nullptr时直接清标志并销毁 instance ctx。MPI 模式仅 SHMEM 内部调用了MPI_Init时 finalize 才MPI_Finalize。4.3 多 instance各 instance 独立 finalize全局资源由最后一个 instance 释放非 0 instance 的 team 能力受限如无 split推荐使用aclshmemx_finalize(instance_id)五、函数与文件索引5.1 对外 APIAPI文件aclshmemx_init_attr/aclshmem_finalize/aclshmemx_finalizesrc/host/init/shmem_init.cppaclshmemx_get_uniqueid/set_attr_uniqueid_args/init_status/instance_ctx_*src/host/init/shmem_init.cppaclshmem_malloc/aclshmem_freesrc/host/mem/shmem_mm.cpp5.2 初始化关键内部函数函数文件check_attr/aclshmemi_state_init_attrshmem_init.cppaclshmemi_bootstrap_initshmemi_bootstrap.cppbind_aclshmem_entity/init_device_state/reserve_heap/setup_heapshmem_init_backend.cppexchange_slice/exchange_entity/update_device_stateshmem_init_backend.cppmemory_manager_initializeshmem_mm.cppaclshmemi_signal_init/aclshmemi_team_init/aclshmemi_sync_initshmem_init.cpp等aclshmemi_control_barrier_allshmem_init_backend.cpp5.3 Finalize 关键内部函数函数文件aclshmemi_finalize_implshmem_init.cppaclshmemi_control_barrier_allfinalize 中Bootstrap 前shmem_init.cpp/shmem_init_backend.cppaclshmemi_team_finalize/aclshmemi_signal_finalizeshmem_team.cpp/shmem_init.cppmemory_manager_destroyshmem_mm.cppremove_heap/release_heap/finalize_device_state/release_aclshmem_entityshmem_init_backend.cppaclshmemi_bootstrap_finalizeshmemi_bootstrap.cpp5.4 模块级文件模块文件Bootstrap 插件src/host/bootstrap/shmemi_bootstrap_config_store.cpp、shmemi_bootstrap_mpi.cppConfig Store / Acclink见 config_store_bootstrap.md §十三HYBM metasrc/host/entity/mem_entity_def.h六、常见问题现象处理init 卡在连 rank 0检查ip_port/UIDaddr连通性增大shm_init_timeout建链重试上限默认 120≈约 120s详见 config_store_bootstrap.md §10.1barrier/allgather 超时增大control_operation_timeout秒确认所有 rank 已进入 init详见 config_store_bootstrap.md §九UID 握手失败确认 UID 字节级一致、magic已广播finalize barrier 超时 / 卡住确认全体 PE 均已调用aclshmem_finalize增大control_operation_timeout详见 config_store_bootstrap.md §11不可重复 init先 finalize 再 initRMA 地址异常各 rankmalloc/free同序同大小见malloc.png仅 rank 0 调 init每个 PE 均须aclshmemx_init_attr多 instance 端口冲突SHMEM_INSTANCE_PORT_RANGE模板端口写0910B/C barrier 异常算子入口util_set_ffts_config(util_get_ffts_config())【免费下载链接】shmemCANN SHMEM 是面向昇腾平台的多机多卡内存通信库基于OpenSHMEM 标准协议实现跨设备的高效内存访问与数据同步。项目地址: https://gitcode.com/cann/shmem创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考