DTRACE_FBT(4) FreeBSD Kernel Interfaces Manual DTRACE_FBT(4)
NAME
dtrace_fbt - a DTrace provider for dynamic kernel tracing based on
function boundaries
SYNOPSIS
fbt:module:function:entry
fbt:module:function:return
DESCRIPTION
The Function Boundary Tracing (fbt) provider instruments the entry and
return of almost every kernel function corresponding to an elf(5) symbol
in the kernel and loaded kernel modules.
fbt:module:function:entry fires whenever the function is called.
fbt:module:function:return fires when the function returns.
The module in the probe description is either the name of the loaded
kernel module or `kernel' for functions compiled into the kernel.
Function Boundary Instrumentation
The fbt will always instrument a function's entry, but its return will be
intsrumented so long as it can find a `ret' instruction.
In some cases, fbt cannot instrument a function's entry and/or return.
Refer to subsection Frame Pointer for more details.
Probe Arguments
The arguments of the entry probe (fbt:module:function:entry) are the
arguments of the traced function call.
Entry Probe Argument Definition
args[0] Function's first argument, typed (e.g.,
malloc(9)'s size_t size)
args[1] Function's second argument, typed (e.g.,
malloc(9)'s struct malloc_type *type)
args[2] Function's third argument, typed (e.g.,
malloc(9)'s int flags)
... ...
The arguments of the return probe (fbt:module:function:return) are
args[0] (the offset of the firing return instruction within the function;
useful to tell apart two different return statements in a single
function) and args[1] (the return value, if any).
Return Probe Argument Definition
args[0] Offset of the traced return instruction
args[1] Function's return value (e.g., a kernel
virtual address if returning from a
successful malloc(9))
Subsection Example 2: Getting Details About Probe's Arguments shows how
to get probe's argument count and types directly with dtrace(1) without
having to resort to the reading function's source code or documentation.
EXAMPLES
Example 1: Listing Available FBT Probes
The following example shows how to list all the available fbt probes.
# dtrace -l -P fbt
ID PROVIDER MODULE FUNCTION NAME
[...]
31868 fbt kernel hammer_time entry
31869 fbt kernel hammer_time return
[...]
Since hammer_time() is a part of the kernel and not a separate loaded
module, the module column displays `kernel'.
Example 2: Getting Details About Probe's Arguments
The following example shows how to generate a program stability report of
malloc(9)'s entry and return probes. Those reports are useful to view
the probe's number of arguments and their types.
# dtrace -l -v -n fbt::malloc:entry
[...]
Argument Types
args[0]: size_t
args[1]: struct malloc_type *
args[2]: int
The count and types of fbt::malloc:entry arguments match the function
signature of malloc(9): args[0] is size_t, args[1] is struct malloc_type
*, and args[2] is int.
# dtrace -l -v -n fbt::malloc:return
[...]
Argument Types
args[0]: int
args[1]: void *
The return probe reports two arguments and their types: the return
instruction offset (the usual int) and the function's return value, which
in this case is void *, as malloc(9) returns a kernel virtual address.
Example 3: Counting Kernel Slab Memory Allocation by Function
# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'
dtrace: description 'fbt::kmem*:entry ' matched 47 probes
^C
kmem_alloc_contig 1
kmem_alloc_contig_domainset 1
kmem_cache_reap_active 1
kmem_alloc_contig_pages 2
kmem_free 2
kmem_std_destructor 19
kmem_std_constructor 26
kmem_cache_free 151
kmem_cache_alloc 181
Example 4: Counting Kernel Slab Memory Allocation by Calling Function
# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\n", @); }'
^C
kernel`contigmalloc+0x33 1
kernel`free+0xd3 1
kernel`kmem_alloc_contig+0x29 1
kernel`kmem_alloc_contig_domainset+0x19a 1
zfs.ko`arc_reap_cb_check+0x16 1
Example 5: Counting Kernel malloc()'s by Calling Function
# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\n", @); }'
^C
kernel`devclass_get_devices+0xa8 1
kernel`sys_ioctl+0xb7 1
dtrace.ko`dtrace_ioctl+0x15c1 1
dtrace.ko`dtrace_ioctl+0x972 2
dtrace.ko`dtrace_dof_create+0x35 2
kernel`kern_poll_kfds+0x2f0 4
kernel`kern_poll_kfds+0x28a 19
Example 6: Counting Kernel malloc()'s by Kernel Stack Trace
# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }'
^C
dtrace.ko`dtrace_dof_create+0x35
dtrace.ko`dtrace_ioctl+0x827
kernel`devfs_ioctl+0xd1
kernel`VOP_IOCTL_APV+0x2a
kernel`vn_ioctl+0xb6
kernel`devfs_ioctl_f+0x1e
kernel`kern_ioctl+0x286
kernel`sys_ioctl+0x12f
kernel`amd64_syscall+0x169
kernel`0xffffffff81092b0b
2
Example 7: Summarizing vmem_alloc()'s by Arena Name and Size Distribution
# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'
^C
kernel arena dom
value ------------- Distribution ------------- count
2048 | 0
4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4
8192 |@@@@@@@@@@@@@ 2
16384 | 0
Example 8: Measuring Total Time Spent Executing a Function
This DTrace script measures the total time spent in vm_page*() kernel
functions. The quantize() aggregation organizes the measurements into
power-of-two buckets, providing a time distribution in nanoseconds for
each function.
fbt::vm_page*:entry {
self->start = timestamp;
}
fbt::vm_page*:return /self->start/ {
@[probefunc] = quantize(timestamp - self->start);
self->start = 0;
}
SEE ALSO
dtrace(1), dtrace_kinst(4), tracing(7)
Brendan Gregg and Jim Mauro, DTrace: Dynamic Tracing in Oracle Solaris,
Mac OS X and FreeBSD, Prentice Hall,
https://www.brendangregg.com/dtracebook/, pp. 898-903, 2011.
The illumos Dynamic Tracing Guide,
https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt, 2008, Chapter fbt
Provider.
AUTHORS
This manual page was written by Mateusz Piotrowski <0mp@FreeBSD.org>.
CAVEATS
Stability and Portability
fbt probes are by definition tightly coupled to kernel code; if the code
underlying a script changes, the script may fail to run or may produce
incorrect results. Scripts written for one version of FreeBSD might not
work on others, and almost certainly will not work on other operating
systems.
Individual fbt probes often do not correspond nicely to logical system
events. For example, consider a DTrace script which prints the
destination address of every IP packet as the kernel hands them over to
the network card driver (NIC). An fbt-based implementation of such a
script is a discouragingly difficult task: it involves instrumenting at
least four different functions in different parts of the IPv4 and IPv6
code. At the same time, with the dtrace_ip(4) provider the script is a
simple one-liner:
dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}'
Make sure to review available dtrace(1) providers first before
implementing a custom script with the fbt provider. If none of the
DTrace providers offer the desired probes, consider adding new
statically-defined tracing probes (SDT(9)).
Frame Pointer
Inline functions are not instrumentable by fbt as they lack a frame
pointer. A developer might explicitly disable inlining by adding the
`__noinline' attribute to a function definition, but of course this
requires a recompilation of the kernel. Building the kernel with
-fno-omit-frame-pointer is another way of preserving frame pointers.
Note, that sometimes compilers will omit the frame pointer in leaf
functions, even when configured with -fno-omit-frame-pointer.
Function returns via a tail call are also not instrumentable by fbt. As
a result, a function might have an entry probe and a mix of instrumented
and uninstrumentable returns.
Use dtrace_kinst(4) to trace arbitrary instructions inside kernel
functions and work around some of the limitations of fbt.
Tracing DTrace
The fbt provider cannot attach to functions inside DTrace provider kernel
modules.
FreeBSD 15.1-STABLE-HBSD July 16, 2025 DTRACE_FBT(4)