网站服务器与虚拟主机,销售网页制作,网站设计师主要做什么的,绵阳东原建设工程有限公司网站观测数据源
目前按照我的理解#xff0c;和trace相关的常用数据源–探针 大致分为四类。 内核 Trace point kprobe 用户程序 USDT uprobe
在用户程序中#xff0c;USDT是所谓的静态Tracepoint。和内核代码中的Trace point类似。实现方式是在代码开发时#xff0c;使用USDT…观测数据源
目前按照我的理解和trace相关的常用数据源–探针 大致分为四类。 内核 Trace point kprobe 用户程序 USDT uprobe
在用户程序中USDT是所谓的静态Tracepoint。和内核代码中的Trace point类似。实现方式是在代码开发时使用USDT的库和头文件。在代码中埋点。在运行时可以通过一些手段使能这些Tracepoint。而uprobe是在kprobe的基础上沿袭下来。属于是动态探针。实现方式需要依托内核支持。在执行到此指令前或者后进行代码注入。实现trace。
内核支持
引用参考文献
Uprobe-tracer: Uprobe-based Event TracingDocumentation written by Srikar DronamrajuOverview
--------
Uprobe based trace events are similar to kprobe based trace events.
To enable this feature, build your kernel with CONFIG_UPROBE_EVENTSy.Similar to the kprobe-event tracer, this doesnt need to be activated via
current_tracer. Instead of that, add probe points via
/sys/kernel/debug/tracing/uprobe_events, and enable it via
/sys/kernel/debug/tracing/events/uprobes/EVENT/enabled.However unlike kprobe-event tracer, the uprobe event interface expects the
user to calculate the offset of the probepoint in the object.6.6. Dynamic Tracing
For kernel analysis, Im using CONFIG_KPROBESy and CONFIG_KPROBE_EVENTSy, to enable kernel dynamic tracing, and CONFIG_FRAME_POINTERy, for frame pointer-based kernel stacks. For user-level analysis, CONFIG_UPROBESy and CONFIG_UPROBE_EVENTSy, for user-level dynamic tracing.Kernel Config: 3.8.6
Here are some kernel CONFIG options for perf_events functionality:
# for perf_events:
CONFIG_PERF_EVENTSy
# for stack traces:
CONFIG_FRAME_POINTERy
# kernel symbols:
CONFIG_KALLSYMSy
# tracepoints:
CONFIG_TRACEPOINTSy
# kernel function trace:
CONFIG_FTRACEy
# kernel-level dynamic tracing:
CONFIG_KPROBESy
CONFIG_KPROBE_EVENTSy
# user-level dynamic tracing:
CONFIG_UPROBESy
CONFIG_UPROBE_EVENTSy
# full kernel debug info:
CONFIG_DEBUG_INFOy
# kernel lock tracing:
CONFIG_LOCKDEPy
# kernel lock tracing:
CONFIG_LOCK_STATy
# kernel dynamic tracepoint variables:
CONFIG_DEBUG_INFOy
You may need to build your own kernel to enable these. The exact set you need depends on your needs and kernel version, and list is likely to grow as new features are added to perf_events.测试代码 #include stdio.h
#include unistd.hstatic void
print_curr_state_one(void)
{printf(This is the print current state one function\n);
}static void
print_curr_state_two(void)
{printf(This is the print current state two function\n);
}int main() {while(1) {print_curr_state_one();sleep(1);print_curr_state_two();}
}通过 perf 使用 uprobe
uprobe作为数据源可以通过多种途径使用。不同的工具实现的功能可能有所差别。
# perf probeUsage: perf probe [options] PROBEDEF [PROBEDEF ...]or: perf probe [options] --add PROBEDEF [--add PROBEDEF ...]or: perf probe [options] --del [GROUP:]EVENT ...or: perf probe --list [GROUP:]EVENT ...or: perf probe [options] --funcs-a, --add [EVENT]FUNC[OFF|%return] [[NAME]ARG ...]probe point definition, whereGROUP: Group name (optional)EVENT: Event nameFUNC: Function nameOFF: Offset from function entry (in byte)%return: Put the probe at function returnARG: Probe argument (kprobe-tracer argument format.)-D, --definition [EVENT]FUNC[OFF|%return] [[NAME]ARG ...]Show trace event definition of given traceevent for k/uprobe_events.-d, --del [GROUP:]EVENTdelete a probe event.-f, --force forcibly add events with existing name-F, --funcs [FILTER]Show potential probe-able functions.-k, --vmlinux file vmlinux pathname(not built-in because NO_DWARF1)-L, --line FUNC[:RLN[NUM|-RLN2]]|SRC:ALN[NUM|-ALN2]Show source code lines.(not built-in because NO_DWARF1)-l, --list [GROUP:]EVENTlist up probe events-m, --module modname|pathtarget module name (for online) or path (for offline)-n, --dry-run dry run-q, --quiet be quiet (do not show any messages)-s, --source directorypath to kernel source(not built-in because NO_DWARF1)-V, --vars FUNC[SRC][OFF|%return|:RL|;PT]|SRC:AL|SRC;PTShow accessible variables on PROBEDEF(not built-in because NO_DWARF1)-v, --verbose be more verbose (show parsed arguments, etc)-x, --exec executable|pathtarget executable name or path--cache Manipulate probe cache--demangle Enable symbol demangling--demangle-kernelEnable kernel symbol demangling--externs Show external variables too (with --vars only)(not built-in because NO_DWARF1)--filter [!]FILTERSet a filter (with --vars/funcs only)(default: !__k???tab_* !__crc_* for --vars,!_* for --funcs)--max-probes n Set how many probe points can be found for a probe.--no-inlines Dont search inlined functions(not built-in because NO_DWARF1)--range Show variables location range in scope (with --vars only)(not built-in because NO_DWARF1)--symfs directoryLook for files with symbols relative to this directory--target-ns pidtarget pid for namespace contexts其中这个(not built-in because NO_DWARF1)很有意思这是否意味着不能通过debug info 去获取局部变量也不能通过行号加probe
通过搜索perf的源码发现似乎是编译perf的时候没有开启。
那么我需要自己编译perf 然后移植
查看可用的probe并添加记录
查看可以插入探针的函数
# perf probe -x a.out -F
abortplt
call_weak_fn
completed.8444
data_start
deregister_tm_clones
frame_dummy
main
print_curr_state_one
print_curr_state_two
putsplt
register_tm_clones
sleepplt通过-x指定执行文件。-F显示可能被用来插入探针的函数。
插入探针
# perf probe -x a.out -a print_curr_state_one
Added new event:probe_a:print_curr_state_one (on print_curr_state_one in /home/root/test_uprobe/a.out)You can now use it in all perf tools, such as:perf record -e probe_a:print_curr_state_one -aR sleep 1查看插入的探针
# perf probe -lprobe_a:print_curr_state_one (on print_curr_state_one in /home/root/test_uprobe/a.out)probe_a:print_curr_state_two (on print_curr_state_two in /home/root/test_uprobe/a.out)开启记录新增的探针
# perf record -e probe_a:* -a
Couldnt synthesize bpf events.
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.159 MB perf.data (24 samples) ]查看记录的结果
# perf scripta.out 3198 [000] 15137.303918: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15137.304015: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15138.304117: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15138.304153: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15139.304244: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15139.304278: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15140.304378: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15140.304415: probe_a:print_curr_state_one: (4005ec)a.out 3198 [001] 15141.304575: probe_a:print_curr_state_two: (40060c)a.out 3198 [001] 15141.304614: probe_a:print_curr_state_one: (4005ec)a.out 3198 [001] 15142.304696: probe_a:print_curr_state_two: (40060c)a.out 3198 [001] 15142.304729: probe_a:print_curr_state_one: (4005ec)a.out 3198 [001] 15143.304829: probe_a:print_curr_state_two: (40060c)a.out 3198 [001] 15143.304866: probe_a:print_curr_state_one: (4005ec)a.out 3198 [001] 15144.304969: probe_a:print_curr_state_two: (40060c)a.out 3198 [001] 15144.305004: probe_a:print_curr_state_one: (4005ec)a.out 3198 [001] 15145.305104: probe_a:print_curr_state_two: (40060c)a.out 3198 [001] 15145.305137: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15146.305243: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15146.305279: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15147.305373: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15147.305406: probe_a:print_curr_state_one: (4005ec)a.out 3198 [000] 15148.305499: probe_a:print_curr_state_two: (40060c)a.out 3198 [000] 15148.305533: probe_a:print_curr_state_one: (4005ec)删去不再使用的probe
# perf probe -d probe_a:*在LTTng中使用
Create or enable a recording event rule to match Linux kernel events created from a dynamic instrumentation point:
lttng [GENERAL OPTIONS] enable-event --kernel(--probeLOC | --functionLOC | --userspace-probeLOC) RECORDNAME[--sessionSESSION] [--channelCHANNEL]# lttng enable-event --kernel --userspace-probe./a.out:print_curr_state_one FUNC_A
kernel event FUNC_A created in channel channel0# lttng enable-event --kernel --userspace-probe./a.out:print_curr_state_two FUNC_B
kernel event FUNC_B created in channel channel0在设定上uprobe仍然属于是内核提供 所以还是内核的trace事件。
在LTTng中好像没有找到关于uretprobe的内容。也没有发现类似可以按照行或者抓取局部变量的内容。可能是LTTng没有做。
之后正常使能所有内核Trace事件。 然后记录log。进行分析。 之后可以在可视化工具中查看进程调度以及进程运行的细节。
在LTTng的log中没有更多的细节甚至函数名都没有保留只有在注册probe的时候自定义的命名。
可能他们推荐大家使用LTTng-UST的USDT吧。
下一步工作
目前实际上是没有实现更细致的观测。例如perf prob -V /L这种。 可能需要重新编译移植perf.或者寻找其他的数据采集和分析工具。
或者有其他的工具可以使用。我甚至想尝试bpf了。但这意味着重新编译内核打开bpf的支持。 参考文献
Linux K/Uprobe 使用指南 · GitBook (t-head.cn)Linux uprobe: User-Level Dynamic Tracing (brendangregg.com)https://www.kernel.org/doc/Documentation/trace/uprobetracer.txtLinux perf Examples (brendangregg.com)lttng-enable-event(1) [v2.13] — LTTng
