Linux 内核自带了一个性能分析工具叫 perf。它能做函数级和指令级的热点采样也能配合 tracepoint 采集系统调用、网络事件、文件系统操作等内核事件。因为代码就在内核源码树里算得上是 Linux 平台上最顺手的性能工具了。原理perf 基于内核的性能计数器子系统硬件层面利用 CPU 的 PMUPerformance Monitoring Unit软件层面依赖内核的 tracepoint 和软件计数器。大致原理是每隔一个固定时间CPU 产生一个中断记下当前跑的是哪个进程、哪个函数累加对应的计数。多采几次就知道 CPU 时间主要花在了哪里。整体架构分两层Perf Tools— 用户态工具集收集和分析性能数据Perf Event Subsystem— 内核事件子系统和数据采集协同工作Linux Hard Lockup Detector 也依赖它安装sudoaptinstalllinux-tools-common linux-tools-generic linux-tools-uname-r常用命令perf --helpusage: perf[--version][--help][OPTIONS]COMMAND[ARGS]Themostcommonly used perf commands are: annotate Read perf.data and display annotated code archive Create archive with object files with build-ids bench General frameworkforbenchmark suites buildid-cache Manage build-id cache. buildid-list List the buildidsina perf.datafilec2c Shared Data C2C/HITM Analyzer. config Get andsetvariablesina configuration file. data Datafilerelated processingdiffRead perf.data files and display the differential profile evlist List the event namesina perf.datafileftrace simple wrapperforkernel s ftrace functionality inject Filter to augment the events stream with additional information kallsyms Searches running kernelforsymbols kmem Tool to trace/measure kernel memory properties kvm Tool to trace/measure kvm guest os list List all symbolic event types lock Analyze lock events mem Profile memory accesses record Run acommandand record its profile into perf.data report Read perf.data(created by perf record)and display the profile sched Tool to trace/measure scheduler properties(latencies)script Read perf.data and display trace outputstatRun acommandand gather performance counter statisticstestRuns sanity tests. timechart Tool to visualize total system behavior during a workloadtopSystem profiling tool. version display the version of perf binary probe Define new dynamic tracepoints tracestraceinspired tool几个常用命令的简要说明|命令|作用|| — | — ||annotate|解析 perf.data显示带注释的代码||archive|按 build-id 打包被采样的 ELF方便异地分析||bench|内置的调度器和内存管理 benchmark||diff|对比两个 perf.data 的热点差异||evlist|列出 perf.data 中记录的性能事件||kmem|追踪内核 slab 内存||kvm|追踪 KVM 客户机||list|列出当前系统支持的所有性能事件||lock|分析内核锁争用||mem|分析内存访问||record|采集采样数据并写入文件||report|读取 perf.data 显示热点分析结果||sched|分析调度器延迟||script|读取 perf.data 并输出 trace||stat|运行命令并收集性能概况CPI、Cache miss 等||timechart|可视化系统行为||top|实时分析类似 top||trace|跟踪系统调用||probe|定义动态探测点|常用例子列出事件perf list perf listsched:*计数perfstatcommand# 统计命令的 CPU 计数器perfstat-dcommand# 详细统计perfstat-pPID# 统计指定进程perfstat-asleep5# 全系统 5 秒perfstat-ecycles,instructions,cache-references,cache-misses,bus-cycles-asleep10perfstat-eL1-dcache-loads,L1-dcache-load-misses,L1-dcache-storescommandperfstat-eLLC-loads,LLC-load-misses,LLC-stores,LLC-prefetchescommandperfstat-eraw_syscalls:sys_enter-I1000-a# 每秒系统调用数采样perf record-F99command# 99Hz 采样命令perf record-F99-pPID-g--sleep10# 采样 调用栈perf record-F99-ag--sleep10# 全系统 99Hz10 秒perf record-eL1-dcache-load-misses-c10000-ag--sleep5# 按 Cache Miss 采样perf record-ecycles:k-a--sleep5# 只采内核态perf record-ecycles:u-a--sleep5# 只采用户态实时perftop-F49perftop-F49-nscomm,dso静态跟踪perf record-esched :sched_process_exec-aperf record-econtext-switches-aperf record-eext4:*-o/tmp/perf.data-aperf record-evmscan:mm_vmscan_wakeup_kswapd-ag动态跟踪perf probe--addtcp_sendmsg# 添加探测点perf probe-dtcp_sendmsg# 删除perf probetcp_sendmsg%return# 探测返回值perf probe-Vtcp_sendmsg# 查看可用变量perf probe-Ltcp_sendmsg# 查看可用行号perf probe-x/lib64/libc.so.6 malloc# 用户态探测perf probe-l# 列出当前探测点报告perf report# TUI 模式perf report-n# 显示采样计数perf report--stdio# 文本输出perf script# 列出所有事件perf script--header-Fcomm,pid,tid,cpu,time,event,ip,sym,dso perf annotate--stdio# 反汇编注解参考https://zhuanlan.zhihu.com/p/186208907https://zhuanlan.zhihu.com/p/54276509https://www.brendangregg.com/perf.html[#OneLiners](javascript:end往期推荐嵌入式Linux必读经典书籍嵌入式学习路线推荐一位读者逻辑清晰的提问机械转行嵌入式成功上岸一位音视频方向读者秋招上岸的经历
