[PATCH dwarves v9 0/3] pahole: Inject kfunc decl tags into BTF

[PATCH dwarves v9 0/3] pahole: Inject kfunc decl tags into BTF

[Daniel Xu]

This patchset teaches pahole to parse symbols in .BTF_ids section in
vmlinux and discover exported kfuncs. Pahole then takes the list of
kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.

Example of encoding:

        $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
        121

        $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
        [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
        [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1

This enables downstream users and tools to dynamically discover which
kfuncs are available on a system by parsing vmlinux or module BTF, both
available in /sys/kernel/btf.

This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.

Tested-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>

=== Changelog ===

Changes from v8:
* Fix another get_func_name() edge case

Changes from v7:
* Fix/support detached BTF encoding

Changes from v6:
* Rebase and add decl_tag_kfuncs as default feature

Changes from v5:
* Add gobuffer__sort() helper
* Use strstarts() instead of strncmp()
* Use uint64_t instead of size_t
* Add clarifying comments for get_func_name()

Changes from v4:
* Update man page with decl_tag_kfuncs feature
* Fix release mode build warnings
* Add elf_getshrstrndx() error checking
* Disable tagging if decl_tag feature is off
* Fix malformed func name handling

Changes from v3:
* Guard kfunc tagging behind feature flag
* Use struct btf_id_set8 definition
* Remove unnecessary member from btf_encoder
* Fix code styling

Changes from v2:
* More reliably detect kfunc membership in set8 by tracking set addr ranges
* Rename some variables/functions to be more clear about kfunc vs func

Changes from v1:
* Fix resource leaks
* Fix callee -> caller typo
* Rename btf_decl_tag from kfunc -> bpf_kfunc
* Only grab btf_id_set funcs tagged kfunc
* Presort btf func list

Daniel Xu (3):
  pahole: Save input filename separate from output
  pahole: Add --btf_feature=decl_tag_kfuncs feature
  pahole: Inject kfunc decl tags into BTF

 btf_encoder.c      | 377 +++++++++++++++++++++++++++++++++++++++++++++
 dwarves.h          |   1 +
 man-pages/pahole.1 |   1 +
 pahole.c           |   1 +
 4 files changed, 380 insertions(+)

-- 
2.44.0

[PATCH dwarves v9 1/3] pahole: Save input filename separate from output

[Daniel Xu]

During detached BTF encoding, the input file is not necessarily the same
as the output file. So save them separately. This matters when we need
to look at the input file again, such as for kfunc tagging.

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 btf_encoder.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/btf_encoder.c b/btf_encoder.c
index 19e9d90..5ffaf5d 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -64,6 +64,7 @@ struct btf_encoder {
 	struct btf        *btf;
 	struct cu         *cu;
 	struct gobuffer   percpu_secinfo;
+	const char	  *source_filename;
 	const char	  *filename;
 	struct elf_symtab *symtab;
 	uint32_t	  type_id_off;
@@ -1648,6 +1649,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
 
 	if (encoder) {
 		encoder->raw_output = detached_filename != NULL;
+		encoder->source_filename = strdup(cu->filename);
 		encoder->filename = strdup(encoder->raw_output ? detached_filename : cu->filename);
 		if (encoder->filename == NULL)
 			goto out_delete;
@@ -1730,6 +1732,7 @@ void btf_encoder__delete(struct btf_encoder *encoder)
 	btf_encoders__delete(encoder);
 	__gobuffer__delete(&encoder->percpu_secinfo);
 	zfree(&encoder->filename);
+	zfree(&encoder->source_filename);
 	btf__free(encoder->btf);
 	encoder->btf = NULL;
 	elf_symtab__delete(encoder->symtab);
-- 
2.44.0

[PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Daniel Xu]

Add a feature flag to guard tagging of kfuncs. The next commit will
implement the actual tagging.

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 btf_encoder.c      | 2 ++
 dwarves.h          | 1 +
 man-pages/pahole.1 | 1 +
 pahole.c           | 1 +
 4 files changed, 5 insertions(+)

diff --git a/btf_encoder.c b/btf_encoder.c
index 5ffaf5d..f0ef20a 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -76,6 +76,7 @@ struct btf_encoder {
 			  verbose,
 			  force,
 			  gen_floats,
+			  tag_kfuncs,
 			  is_rel;
 	uint32_t	  array_index_id;
 	struct {
@@ -1661,6 +1662,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
 		encoder->force		 = conf_load->btf_encode_force;
 		encoder->gen_floats	 = conf_load->btf_gen_floats;
 		encoder->skip_encoding_vars = conf_load->skip_encoding_btf_vars;
+		encoder->tag_kfuncs	 = conf_load->btf_decl_tag_kfuncs;
 		encoder->verbose	 = verbose;
 		encoder->has_index_type  = false;
 		encoder->need_index_type = false;
diff --git a/dwarves.h b/dwarves.h
index dd35a4e..7d566b6 100644
--- a/dwarves.h
+++ b/dwarves.h
@@ -94,6 +94,7 @@ struct conf_load {
 	bool			btf_gen_floats;
 	bool			btf_encode_force;
 	bool			reproducible_build;
+	bool			btf_decl_tag_kfuncs;
 	uint8_t			hashtable_bits;
 	uint8_t			max_hashtable_bits;
 	uint16_t		kabi_prefix_len;
diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index e3c58e0..4769b58 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -308,6 +308,7 @@ Encode BTF using the specified feature list, or specify 'default' for all standa
 	                   in some CUs and not others, or when the same
 	                   function name has inconsistent BTF descriptions
 	                   in different CUs.
+	decl_tag_kfuncs    Inject a BTF_KIND_DECL_TAG for each discovered kfunc.
 .fi
 
 Supported non-standard features (not enabled for 'default')
diff --git a/pahole.c b/pahole.c
index 750b847..954498d 100644
--- a/pahole.c
+++ b/pahole.c
@@ -1289,6 +1289,7 @@ struct btf_feature {
 	BTF_DEFAULT_FEATURE(enum64, skip_encoding_btf_enum64, true),
 	BTF_DEFAULT_FEATURE(optimized_func, btf_gen_optimized, false),
 	BTF_DEFAULT_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false),
+	BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),
 	BTF_NON_DEFAULT_FEATURE(reproducible_build, reproducible_build, false),
 };
 
-- 
2.44.0

[PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Daniel Xu]

This commit teaches pahole to parse symbols in .BTF_ids section in
vmlinux and discover exported kfuncs. Pahole then takes the list of
kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.

Example of encoding:

        $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
        121

        $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
        [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
        [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1

This enables downstream users and tools to dynamically discover which
kfuncs are available on a system by parsing vmlinux or module BTF, both
available in /sys/kernel/btf.

This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.

Acked-by: Jiri Olsa <jolsa@kernel.org>
Tested-by: Jiri Olsa <jolsa@kernel.org>
Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Tested-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 btf_encoder.c | 372 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 372 insertions(+)

diff --git a/btf_encoder.c b/btf_encoder.c
index f0ef20a..6cb0c8f 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -34,6 +34,21 @@
 #include <pthread.h>
 
 #define BTF_ENCODER_MAX_PROTO	512
+#define BTF_IDS_SECTION		".BTF_ids"
+#define BTF_ID_FUNC_PFX		"__BTF_ID__func__"
+#define BTF_ID_SET8_PFX		"__BTF_ID__set8__"
+#define BTF_SET8_KFUNCS		(1 << 0)
+#define BTF_KFUNC_TYPE_TAG	"bpf_kfunc"
+
+/* Adapted from include/linux/btf_ids.h */
+struct btf_id_set8 {
+        uint32_t cnt;
+        uint32_t flags;
+        struct {
+                uint32_t id;
+                uint32_t flags;
+        } pairs[];
+};
 
 /* state used to do later encoding of saved functions */
 struct btf_encoder_state {
@@ -76,6 +91,7 @@ struct btf_encoder {
 			  verbose,
 			  force,
 			  gen_floats,
+			  skip_encoding_decl_tag,
 			  tag_kfuncs,
 			  is_rel;
 	uint32_t	  array_index_id;
@@ -95,6 +111,17 @@ struct btf_encoder {
 	} functions;
 };
 
+struct btf_func {
+	const char *name;
+	int	    type_id;
+};
+
+/* Half open interval representing range of addresses containing kfuncs */
+struct btf_kfunc_set_range {
+	uint64_t start;
+	uint64_t end;
+};
+
 static LIST_HEAD(encoders);
 static pthread_mutex_t encoders__lock = PTHREAD_MUTEX_INITIALIZER;
 
@@ -1364,8 +1391,343 @@ out:
 	return err;
 }
 
+/* Returns if `sym` points to a kfunc set */
+static int is_sym_kfunc_set(GElf_Sym *sym, const char *name, Elf_Data *idlist, size_t idlist_addr)
+{
+	void *ptr = idlist->d_buf;
+	struct btf_id_set8 *set;
+	int off;
+
+	/* kfuncs are only found in BTF_SET8's */
+	if (!strstarts(name, BTF_ID_SET8_PFX))
+		return false;
+
+	off = sym->st_value - idlist_addr;
+	if (off >= idlist->d_size) {
+		fprintf(stderr, "%s: symbol '%s' out of bounds\n", __func__, name);
+		return false;
+	}
+
+	/* Check the set8 flags to see if it was marked as kfunc */
+	set = ptr + off;
+	return set->flags & BTF_SET8_KFUNCS;
+}
+
+/*
+ * Parse BTF_ID symbol and return the func name.
+ *
+ * Returns:
+ *	Caller-owned string containing func name if successful.
+ *	NULL if !func or on error.
+ */
+static char *get_func_name(const char *sym)
+{
+	char *func, *end;
+
+	/* Example input: __BTF_ID__func__vfs_close__1
+	 *
+	 * The goal is to strip the prefix and suffix such that we only
+	 * return vfs_close.
+	 */
+
+	if (!strstarts(sym, BTF_ID_FUNC_PFX))
+		return NULL;
+
+	/* Strip prefix and handle malformed input such as  __BTF_ID__func___ */
+	func = strdup(sym + sizeof(BTF_ID_FUNC_PFX) - 1);
+	if (!strstr(func, "__")) {
+                free(func);
+                return NULL;
+        }
+
+	/* Strip suffix */
+	end = strrchr(func, '_');
+	if (!end || *(end - 1) != '_') {
+		free(func);
+		return NULL;
+	}
+	*(end - 1) = '\0';
+
+	return func;
+}
+
+static int btf_func_cmp(const void *_a, const void *_b)
+{
+	const struct btf_func *a = _a;
+	const struct btf_func *b = _b;
+
+	return strcmp(a->name, b->name);
+}
+
+/*
+ * Collects all functions described in BTF.
+ * Returns non-zero on error.
+ */
+static int btf_encoder__collect_btf_funcs(struct btf_encoder *encoder, struct gobuffer *funcs)
+{
+	struct btf *btf = encoder->btf;
+	int nr_types, type_id;
+	int err = -1;
+
+	/* First collect all the func entries into an array */
+	nr_types = btf__type_cnt(btf);
+	for (type_id = 1; type_id < nr_types; type_id++) {
+		const struct btf_type *type;
+		struct btf_func func = {};
+		const char *name;
+
+		type = btf__type_by_id(btf, type_id);
+		if (!type) {
+			fprintf(stderr, "%s: malformed BTF, can't resolve type for ID %d\n",
+				__func__, type_id);
+			err = -EINVAL;
+			goto out;
+		}
+
+		if (!btf_is_func(type))
+			continue;
+
+		name = btf__name_by_offset(btf, type->name_off);
+		if (!name) {
+			fprintf(stderr, "%s: malformed BTF, can't resolve name for ID %d\n",
+				__func__, type_id);
+			err = -EINVAL;
+			goto out;
+		}
+
+		func.name = name;
+		func.type_id = type_id;
+		err = gobuffer__add(funcs, &func, sizeof(func));
+		if (err < 0)
+			goto out;
+	}
+
+	/* Now that we've collected funcs, sort them by name */
+	gobuffer__sort(funcs, sizeof(struct btf_func), btf_func_cmp);
+
+	err = 0;
+out:
+	return err;
+}
+
+static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *funcs, const char *kfunc)
+{
+	struct btf_func key = { .name = kfunc };
+	struct btf *btf = encoder->btf;
+	struct btf_func *target;
+	const void *base;
+	unsigned int cnt;
+	int err = -1;
+
+	base = gobuffer__entries(funcs);
+	cnt = gobuffer__nr_entries(funcs);
+	target = bsearch(&key, base, cnt, sizeof(key), btf_func_cmp);
+	if (!target) {
+		fprintf(stderr, "%s: failed to find kfunc '%s' in BTF\n", __func__, kfunc);
+		goto out;
+	}
+
+	/* Note we are unconditionally adding the btf_decl_tag even
+	 * though vmlinux may already contain btf_decl_tags for kfuncs.
+	 * We are ok to do this b/c we will later btf__dedup() to remove
+	 * any duplicates.
+	 */
+	err = btf__add_decl_tag(btf, BTF_KFUNC_TYPE_TAG, target->type_id, -1);
+	if (err < 0) {
+		fprintf(stderr, "%s: failed to insert kfunc decl tag for '%s': %d\n",
+			__func__, kfunc, err);
+		goto out;
+	}
+
+	err = 0;
+out:
+	return err;
+}
+
+static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
+{
+	const char *filename = encoder->source_filename;
+	struct gobuffer btf_kfunc_ranges = {};
+	struct gobuffer btf_funcs = {};
+	Elf_Data *symbols = NULL;
+	Elf_Data *idlist = NULL;
+	Elf_Scn *symscn = NULL;
+	int symbols_shndx = -1;
+	size_t idlist_addr = 0;
+	int fd = -1, err = -1;
+	int idlist_shndx = -1;
+	size_t strtabidx = 0;
+	Elf_Scn *scn = NULL;
+	Elf *elf = NULL;
+	GElf_Shdr shdr;
+	size_t strndx;
+	char *secname;
+	int nr_syms;
+	int i = 0;
+
+	fd = open(filename, O_RDONLY);
+	if (fd < 0) {
+		fprintf(stderr, "Cannot open %s\n", filename);
+		goto out;
+	}
+
+	if (elf_version(EV_CURRENT) == EV_NONE) {
+		elf_error("Cannot set libelf version");
+		goto out;
+	}
+
+	elf = elf_begin(fd, ELF_C_READ, NULL);
+	if (elf == NULL) {
+		elf_error("Cannot update ELF file");
+		goto out;
+	}
+
+	/* Locate symbol table and .BTF_ids sections */
+	if (elf_getshdrstrndx(elf, &strndx) < 0)
+		goto out;
+
+	while ((scn = elf_nextscn(elf, scn)) != NULL) {
+		Elf_Data *data;
+
+		i++;
+		if (!gelf_getshdr(scn, &shdr)) {
+			elf_error("Failed to get ELF section(%d) hdr", i);
+			goto out;
+		}
+
+		secname = elf_strptr(elf, strndx, shdr.sh_name);
+		if (!secname) {
+			elf_error("Failed to get ELF section(%d) hdr name", i);
+			goto out;
+		}
+
+		data = elf_getdata(scn, 0);
+		if (!data) {
+			elf_error("Failed to get ELF section(%d) data", i);
+			goto out;
+		}
+
+		if (shdr.sh_type == SHT_SYMTAB) {
+			symbols_shndx = i;
+			symscn = scn;
+			symbols = data;
+			strtabidx = shdr.sh_link;
+		} else if (!strcmp(secname, BTF_IDS_SECTION)) {
+			idlist_shndx = i;
+			idlist_addr = shdr.sh_addr;
+			idlist = data;
+		}
+	}
+
+	/* Cannot resolve symbol or .BTF_ids sections. Nothing to do. */
+	if (symbols_shndx == -1 || idlist_shndx == -1) {
+		err = 0;
+		goto out;
+	}
+
+	if (!gelf_getshdr(symscn, &shdr)) {
+		elf_error("Failed to get ELF symbol table header");
+		goto out;
+	}
+	nr_syms = shdr.sh_size / shdr.sh_entsize;
+
+	err = btf_encoder__collect_btf_funcs(encoder, &btf_funcs);
+	if (err) {
+		fprintf(stderr, "%s: failed to collect BTF funcs\n", __func__);
+		goto out;
+	}
+
+	/* First collect all kfunc set ranges.
+	 *
+	 * Note we choose not to sort these ranges and accept a linear
+	 * search when doing lookups. Reasoning is that the number of
+	 * sets is ~O(100) and not worth the additional code to optimize.
+	 */
+	for (i = 0; i < nr_syms; i++) {
+		struct btf_kfunc_set_range range = {};
+		const char *name;
+		GElf_Sym sym;
+
+		if (!gelf_getsym(symbols, i, &sym)) {
+			elf_error("Failed to get ELF symbol(%d)", i);
+			goto out;
+		}
+
+		if (sym.st_shndx != idlist_shndx)
+			continue;
+
+		name = elf_strptr(elf, strtabidx, sym.st_name);
+		if (!is_sym_kfunc_set(&sym, name, idlist, idlist_addr))
+			continue;
+
+		range.start = sym.st_value;
+		range.end = sym.st_value + sym.st_size;
+		gobuffer__add(&btf_kfunc_ranges, &range, sizeof(range));
+	}
+
+	/* Now inject BTF with kfunc decl tag for detected kfuncs */
+	for (i = 0; i < nr_syms; i++) {
+		const struct btf_kfunc_set_range *ranges;
+		unsigned int ranges_cnt;
+		char *func, *name;
+		GElf_Sym sym;
+		bool found;
+		int err;
+		int j;
+
+		if (!gelf_getsym(symbols, i, &sym)) {
+			elf_error("Failed to get ELF symbol(%d)", i);
+			goto out;
+		}
+
+		if (sym.st_shndx != idlist_shndx)
+			continue;
+
+		name = elf_strptr(elf, strtabidx, sym.st_name);
+		func = get_func_name(name);
+		if (!func)
+			continue;
+
+		/* Check if function belongs to a kfunc set */
+		ranges = gobuffer__entries(&btf_kfunc_ranges);
+		ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
+		found = false;
+		for (j = 0; j < ranges_cnt; j++) {
+			size_t addr = sym.st_value;
+
+			if (ranges[j].start <= addr && addr < ranges[j].end) {
+				found = true;
+				break;
+			}
+		}
+		if (!found) {
+			free(func);
+			continue;
+		}
+
+		err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
+		if (err) {
+			fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
+			free(func);
+			goto out;
+		}
+		free(func);
+	}
+
+	err = 0;
+out:
+	__gobuffer__delete(&btf_funcs);
+	__gobuffer__delete(&btf_kfunc_ranges);
+	if (elf)
+		elf_end(elf);
+	if (fd != -1)
+		close(fd);
+	return err;
+}
+
 int btf_encoder__encode(struct btf_encoder *encoder)
 {
+	bool should_tag_kfuncs;
 	int err;
 
 	/* for single-threaded case, saved funcs are added here */
@@ -1378,6 +1740,15 @@ int btf_encoder__encode(struct btf_encoder *encoder)
 	if (btf__type_cnt(encoder->btf) == 1)
 		return 0;
 
+	/* Note vmlinux may already contain btf_decl_tag's for kfuncs. So
+	 * take care to call this before btf_dedup().
+	 */
+	should_tag_kfuncs = encoder->tag_kfuncs && !encoder->skip_encoding_decl_tag;
+	if (should_tag_kfuncs && btf_encoder__tag_kfuncs(encoder)) {
+		fprintf(stderr, "%s: failed to tag kfuncs!\n", __func__);
+		return -1;
+	}
+
 	if (btf__dedup(encoder->btf, NULL)) {
 		fprintf(stderr, "%s: btf__dedup failed!\n", __func__);
 		return -1;
@@ -1662,6 +2033,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
 		encoder->force		 = conf_load->btf_encode_force;
 		encoder->gen_floats	 = conf_load->btf_gen_floats;
 		encoder->skip_encoding_vars = conf_load->skip_encoding_btf_vars;
+		encoder->skip_encoding_decl_tag	 = conf_load->skip_encoding_btf_decl_tag;
 		encoder->tag_kfuncs	 = conf_load->btf_decl_tag_kfuncs;
 		encoder->verbose	 = verbose;
 		encoder->has_index_type  = false;
-- 
2.44.0

Re: [PATCH dwarves v9 1/3] pahole: Save input filename separate from output

[Arnaldo Carvalho de Melo]

On Mon, Apr 29, 2024 at 04:45:58PM -0600, Daniel Xu wrote:
> During detached BTF encoding, the input file is not necessarily the same
> as the output file. So save them separately. This matters when we need
> to look at the input file again, such as for kfunc tagging.

You forgot to check a strdup(), I added this on top of this patch:


diff --git a/btf_encoder.c b/btf_encoder.c
index 5ffaf5d969c9bc49..8aa2a7709dc1555f 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -1651,7 +1651,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
 		encoder->raw_output = detached_filename != NULL;
 		encoder->source_filename = strdup(cu->filename);
 		encoder->filename = strdup(encoder->raw_output ? detached_filename : cu->filename);
-		if (encoder->filename == NULL)
+		if (encoder->source_filename == NULL || encoder->filename == NULL)
 			goto out_delete;
 
 		encoder->btf = btf__new_empty_split(base_btf);
 
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> ---
>  btf_encoder.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 19e9d90..5ffaf5d 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -64,6 +64,7 @@ struct btf_encoder {
>  	struct btf        *btf;
>  	struct cu         *cu;
>  	struct gobuffer   percpu_secinfo;
> +	const char	  *source_filename;
>  	const char	  *filename;
>  	struct elf_symtab *symtab;
>  	uint32_t	  type_id_off;
> @@ -1648,6 +1649,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
>  
>  	if (encoder) {
>  		encoder->raw_output = detached_filename != NULL;
> +		encoder->source_filename = strdup(cu->filename);
>  		encoder->filename = strdup(encoder->raw_output ? detached_filename : cu->filename);
>  		if (encoder->filename == NULL)
>  			goto out_delete;
> @@ -1730,6 +1732,7 @@ void btf_encoder__delete(struct btf_encoder *encoder)
>  	btf_encoders__delete(encoder);
>  	__gobuffer__delete(&encoder->percpu_secinfo);
>  	zfree(&encoder->filename);
> +	zfree(&encoder->source_filename);
>  	btf__free(encoder->btf);
>  	encoder->btf = NULL;
>  	elf_symtab__delete(encoder->symtab);
> -- 
> 2.44.0

Re: [PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Arnaldo Carvalho de Melo]

On Mon, Apr 29, 2024 at 04:45:59PM -0600, Daniel Xu wrote:
> Add a feature flag to guard tagging of kfuncs. The next commit will
> implement the actual tagging.
> 
> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>

Also 'decl_tag_kfuncs' is not enabled when using --btf_features=default,
right? as:

        BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),

And that false is .default_enabled=false.

So I added:

diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
index be04f1c617291e21..f0605935a9f1b4dc 100644
--- a/man-pages/pahole.1
+++ b/man-pages/pahole.1
@@ -308,7 +308,6 @@ Encode BTF using the specified feature list, or specify 'default' for all standa
 	                   in some CUs and not others, or when the same
 	                   function name has inconsistent BTF descriptions
 	                   in different CUs.
-	decl_tag_kfuncs    Inject a BTF_KIND_DECL_TAG for each discovered kfunc.
 .fi
 
 Supported non-standard features (not enabled for 'default')
@@ -317,6 +316,7 @@ Supported non-standard features (not enabled for 'default')
 	reproducible_build Ensure generated BTF is consistent every time;
 	                   without this parallel BTF encoding can result in
 	                   inconsistent BTF ids.
+	decl_tag_kfuncs    Inject a BTF_KIND_DECL_TAG for each discovered kfunc.
 .fi
 
 So for example, specifying \-\-btf_encode=var,enum64 will result in a BTF encoding that (as well as encoding basic BTF information) will contain variables and enum64 values.


----

Please ack. Alan, please check if your Reviewed-by stands with the above
change.

Thanks,

- Arnaldo

> ---
>  btf_encoder.c      | 2 ++
>  dwarves.h          | 1 +
>  man-pages/pahole.1 | 1 +
>  pahole.c           | 1 +
>  4 files changed, 5 insertions(+)
> 
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 5ffaf5d..f0ef20a 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -76,6 +76,7 @@ struct btf_encoder {
>  			  verbose,
>  			  force,
>  			  gen_floats,
> +			  tag_kfuncs,
>  			  is_rel;
>  	uint32_t	  array_index_id;
>  	struct {
> @@ -1661,6 +1662,7 @@ struct btf_encoder *btf_encoder__new(struct cu *cu, const char *detached_filenam
>  		encoder->force		 = conf_load->btf_encode_force;
>  		encoder->gen_floats	 = conf_load->btf_gen_floats;
>  		encoder->skip_encoding_vars = conf_load->skip_encoding_btf_vars;
> +		encoder->tag_kfuncs	 = conf_load->btf_decl_tag_kfuncs;
>  		encoder->verbose	 = verbose;
>  		encoder->has_index_type  = false;
>  		encoder->need_index_type = false;
> diff --git a/dwarves.h b/dwarves.h
> index dd35a4e..7d566b6 100644
> --- a/dwarves.h
> +++ b/dwarves.h
> @@ -94,6 +94,7 @@ struct conf_load {
>  	bool			btf_gen_floats;
>  	bool			btf_encode_force;
>  	bool			reproducible_build;
> +	bool			btf_decl_tag_kfuncs;
>  	uint8_t			hashtable_bits;
>  	uint8_t			max_hashtable_bits;
>  	uint16_t		kabi_prefix_len;
> diff --git a/man-pages/pahole.1 b/man-pages/pahole.1
> index e3c58e0..4769b58 100644
> --- a/man-pages/pahole.1
> +++ b/man-pages/pahole.1
> @@ -308,6 +308,7 @@ Encode BTF using the specified feature list, or specify 'default' for all standa
>  	                   in some CUs and not others, or when the same
>  	                   function name has inconsistent BTF descriptions
>  	                   in different CUs.
> +	decl_tag_kfuncs    Inject a BTF_KIND_DECL_TAG for each discovered kfunc.
>  .fi
>  
>  Supported non-standard features (not enabled for 'default')
> diff --git a/pahole.c b/pahole.c
> index 750b847..954498d 100644
> --- a/pahole.c
> +++ b/pahole.c
> @@ -1289,6 +1289,7 @@ struct btf_feature {
>  	BTF_DEFAULT_FEATURE(enum64, skip_encoding_btf_enum64, true),
>  	BTF_DEFAULT_FEATURE(optimized_func, btf_gen_optimized, false),
>  	BTF_DEFAULT_FEATURE(consistent_func, skip_encoding_btf_inconsistent_proto, false),
> +	BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),
>  	BTF_NON_DEFAULT_FEATURE(reproducible_build, reproducible_build, false),
>  };
>  
> -- 
> 2.44.0

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Arnaldo Carvalho de Melo]

On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> This commit teaches pahole to parse symbols in .BTF_ids section in
> vmlinux and discover exported kfuncs. Pahole then takes the list of
> kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> 
> Example of encoding:
> 
>         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
>         121
> 
>         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
>         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
>         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> 
> This enables downstream users and tools to dynamically discover which
> kfuncs are available on a system by parsing vmlinux or module BTF, both
> available in /sys/kernel/btf.
> 
> This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>
> Tested-by: Jiri Olsa <jolsa@kernel.org>
> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> Tested-by: Alan Maguire <alan.maguire@oracle.com>
> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>

Here I needed this to avoid an strdup possibly followed by a free and
then checking the strdup result, please Ack/revalidate tags. I'm
dropping them as there are changes.

- Arnaldo

diff --git a/btf_encoder.c b/btf_encoder.c
index e9d82e0af0e178fd..c2df2bc7a374447b 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -1434,11 +1434,13 @@ static char *get_func_name(const char *sym)
 		return NULL;
 
 	/* Strip prefix and handle malformed input such as  __BTF_ID__func___ */
-	func = strdup(sym + sizeof(BTF_ID_FUNC_PFX) - 1);
-	if (!strstr(func, "__")) {
-                free(func);
+	const char *func_sans_prefix = sym + sizeof(BTF_ID_FUNC_PFX) - 1;
+	if (!strstr(func_sans_prefix, "__"))
                 return NULL;
-        }
+
+	func = strdup(func_sans_prefix);
+	if (!func)
+		return NULL;
 
 	/* Strip suffix */
 	end = strrchr(func, '_');

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Arnaldo Carvalho de Melo]

On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> This commit teaches pahole to parse symbols in .BTF_ids section in
> vmlinux and discover exported kfuncs. Pahole then takes the list of
> kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> 
> Example of encoding:
> 
>         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
>         121
> 
>         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
>         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
>         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> 
> This enables downstream users and tools to dynamically discover which
> kfuncs are available on a system by parsing vmlinux or module BTF, both
> available in /sys/kernel/btf.
> 
> This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.

I'm trying this but:

⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
btf_encoder__tag_kfuncs(cgroup_rstat_updated): found=0
btf_encoder__tag_kfuncs(cgroup_rstat_flush): found=0
btf_encoder__tag_kfuncs(security_file_permission): found=0
btf_encoder__tag_kfuncs(security_inode_getattr): found=0
btf_encoder__tag_kfuncs(security_file_open): found=0
btf_encoder__tag_kfuncs(security_path_truncate): found=0
btf_encoder__tag_kfuncs(vfs_truncate): found=0
btf_encoder__tag_kfuncs(vfs_fallocate): found=0
btf_encoder__tag_kfuncs(dentry_open): found=0
btf_encoder__tag_kfuncs(vfs_getattr): found=0
btf_encoder__tag_kfuncs(filp_close): found=0
btf_encoder__tag_kfuncs(bpf_lookup_user_key): found=0
btf_encoder__tag_kfuncs(bpf_lookup_system_key): found=0
btf_encoder__tag_kfuncs(bpf_key_put): found=0
btf_encoder__tag_kfuncs(bpf_verify_pkcs7_signature): found=0
btf_encoder__tag_kfuncs(bpf_obj_new_impl): found=0
<SNIP all with found=0>

With:

⬢[acme@toolbox pahole]$ git diff -U16
diff --git a/btf_encoder.c b/btf_encoder.c
index c2df2bc7a374447b..27a16d6564381b60 100644
--- a/btf_encoder.c
+++ b/btf_encoder.c
@@ -1689,32 +1689,35 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 		func = get_func_name(name);
 		if (!func)
 			continue;
 
 		/* Check if function belongs to a kfunc set */
 		ranges = gobuffer__entries(&btf_kfunc_ranges);
 		ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
 		found = false;
 		for (j = 0; j < ranges_cnt; j++) {
 			size_t addr = sym.st_value;
 
 			if (ranges[j].start <= addr && addr < ranges[j].end) {
 				found = true;
 				break;
 			}
 		}
+
+		printf("%s(%s): found=%d\n", __func__, func, found);
+
 		if (!found) {
 			free(func);
 			continue;
 		}
 
 		err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
 		if (err) {
 			fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
 			free(func);
 			goto out;
 		}
 		free(func);
 	}
 
 	err = 0;
 out:

--------------

The vmlinux I'm testing on has the kfuncs, etc, as we can see with:

⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | wc -l
517
⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | tail
 97887: ffffffff83266bfc     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cong_avoid__805493
 97888: ffffffff83266c04     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_state__806494
 97889: ffffffff83266c0c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cwnd_event__807495
 97890: ffffffff83266c14     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_acked__808496
 98068: ffffffff83266c24     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_ssthresh__773199
 98069: ffffffff83266c2c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_cong_avoid__774200
 98070: ffffffff83266c34     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_undo_cwnd__775201
 98071: ffffffff83266c3c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_slow_start__776202
 98072: ffffffff83266c44     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_cong_avoid_ai__777203
101522: ffffffff83266c5c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__update_socket_protocol__80024
⬢[acme@toolbox pahole]$


So that btf_encoder__tag_kfuncs() isn't finding any?

$ pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
btf_encoder__tag_kfuncs(vmlinux)

Yeah, getting the source filename, the right one.

Then is_sym_kfunc_set() never returns true... But:

⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)

real	0m5.586s
user	0m29.707s
sys	0m2.160s
⬢[acme@toolbox pahole]$

And then:

⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)

real	0m5.597s
user	0m29.620s
sys	0m2.138s
⬢[acme@toolbox pahole]$

Run out of time, I probably am using an old vmlinux, will try later with
one generated from a current kernel, or maybe you guys point to my
st00pidity saying what I am missing 8-)

- Arnaldo

Re: [PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Daniel Xu]

Hi Arnaldo,

On Tue, Apr 30, 2024 at 03:48:06PM GMT, Arnaldo Carvalho de Melo wrote:
> On Mon, Apr 29, 2024 at 04:45:59PM -0600, Daniel Xu wrote:
> > Add a feature flag to guard tagging of kfuncs. The next commit will
> > implement the actual tagging.
> > 
> > Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> > Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> 
> Also 'decl_tag_kfuncs' is not enabled when using --btf_features=default,
> right? as:
> 
>         BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),
> 
> And that false is .default_enabled=false.

I think that `false` is for `initial_value`, isn't it? The macro sets
the `default_enabled` field.

Building with this seems to tag the kfuncs for me:

diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf                                                                                                                                                                 
index 82377e470aed..7128dc25ba29 100644                                                                                                                                                                                  
--- a/scripts/Makefile.btf                                                                                                                                                                                               
+++ b/scripts/Makefile.btf                                                                                                                                                                                               
@@ -16,4 +16,6 @@ pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)                += --lang_exclude=rust                                                                                                                   
                                                                                                                                                                                                                         
 pahole-flags-$(call test-ge, $(pahole-ver), 125)       += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized                                                                                                    
                                                                                                                                                                                                                         
+pahole-flags-$(call test-ge, $(pahole-ver), 126)       = -j --lang_exclude=rust --btf_features=default                                                                                                                  
+                                                                                                                                                                                                                        
 export PAHOLE_FLAGS := $(pahole-flags-y)

Thanks,
Daniel

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Daniel Xu]

On Tue, Apr 30, 2024 at 05:42:14PM GMT, Arnaldo Carvalho de Melo wrote:
> On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> > This commit teaches pahole to parse symbols in .BTF_ids section in
> > vmlinux and discover exported kfuncs. Pahole then takes the list of
> > kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> > 
> > Example of encoding:
> > 
> >         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
> >         121
> > 
> >         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
> >         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
> >         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> > 
> > This enables downstream users and tools to dynamically discover which
> > kfuncs are available on a system by parsing vmlinux or module BTF, both
> > available in /sys/kernel/btf.
> > 
> > This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> 
> I'm trying this but:
> 
> ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> btf_encoder__tag_kfuncs(cgroup_rstat_updated): found=0
> btf_encoder__tag_kfuncs(cgroup_rstat_flush): found=0
> btf_encoder__tag_kfuncs(security_file_permission): found=0
> btf_encoder__tag_kfuncs(security_inode_getattr): found=0
> btf_encoder__tag_kfuncs(security_file_open): found=0
> btf_encoder__tag_kfuncs(security_path_truncate): found=0
> btf_encoder__tag_kfuncs(vfs_truncate): found=0
> btf_encoder__tag_kfuncs(vfs_fallocate): found=0
> btf_encoder__tag_kfuncs(dentry_open): found=0
> btf_encoder__tag_kfuncs(vfs_getattr): found=0
> btf_encoder__tag_kfuncs(filp_close): found=0
> btf_encoder__tag_kfuncs(bpf_lookup_user_key): found=0
> btf_encoder__tag_kfuncs(bpf_lookup_system_key): found=0
> btf_encoder__tag_kfuncs(bpf_key_put): found=0
> btf_encoder__tag_kfuncs(bpf_verify_pkcs7_signature): found=0
> btf_encoder__tag_kfuncs(bpf_obj_new_impl): found=0
> <SNIP all with found=0>
> 
> With:
> 
> ⬢[acme@toolbox pahole]$ git diff -U16
> diff --git a/btf_encoder.c b/btf_encoder.c
> index c2df2bc7a374447b..27a16d6564381b60 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -1689,32 +1689,35 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
>  		func = get_func_name(name);
>  		if (!func)
>  			continue;
>  
>  		/* Check if function belongs to a kfunc set */
>  		ranges = gobuffer__entries(&btf_kfunc_ranges);
>  		ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
>  		found = false;
>  		for (j = 0; j < ranges_cnt; j++) {
>  			size_t addr = sym.st_value;
>  
>  			if (ranges[j].start <= addr && addr < ranges[j].end) {
>  				found = true;
>  				break;
>  			}
>  		}
> +
> +		printf("%s(%s): found=%d\n", __func__, func, found);
> +
>  		if (!found) {
>  			free(func);
>  			continue;
>  		}
>  
>  		err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
>  		if (err) {
>  			fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
>  			free(func);
>  			goto out;
>  		}
>  		free(func);
>  	}
>  
>  	err = 0;
>  out:
> 
> --------------
> 
> The vmlinux I'm testing on has the kfuncs, etc, as we can see with:
> 
> ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | wc -l
> 517
> ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | tail
>  97887: ffffffff83266bfc     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cong_avoid__805493
>  97888: ffffffff83266c04     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_state__806494
>  97889: ffffffff83266c0c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cwnd_event__807495
>  97890: ffffffff83266c14     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_acked__808496
>  98068: ffffffff83266c24     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_ssthresh__773199
>  98069: ffffffff83266c2c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_cong_avoid__774200
>  98070: ffffffff83266c34     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_undo_cwnd__775201
>  98071: ffffffff83266c3c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_slow_start__776202
>  98072: ffffffff83266c44     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_cong_avoid_ai__777203
> 101522: ffffffff83266c5c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__update_socket_protocol__80024
> ⬢[acme@toolbox pahole]$
> 
> 
> So that btf_encoder__tag_kfuncs() isn't finding any?
> 
> $ pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> btf_encoder__tag_kfuncs(vmlinux)
> 
> Yeah, getting the source filename, the right one.
> 
> Then is_sym_kfunc_set() never returns true... But:
> 
> ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> 
> real	0m5.586s
> user	0m29.707s
> sys	0m2.160s
> ⬢[acme@toolbox pahole]$
> 
> And then:
> 
> ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)

set->flags=0 here is odd. I'd expect at least some of those to be
non-zero. Can you check if your tree has
https://github.com/torvalds/linux/commit/6f3189f38a3e995232e028a4c341164c4aca1b20
?

Thanks,
Daniel

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Daniel Xu]

On Tue, Apr 30, 2024 at 04:57:26PM GMT, Arnaldo Carvalho de Melo wrote:
> On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> > This commit teaches pahole to parse symbols in .BTF_ids section in
> > vmlinux and discover exported kfuncs. Pahole then takes the list of
> > kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> > 
> > Example of encoding:
> > 
> >         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
> >         121
> > 
> >         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
> >         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
> >         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> > 
> > This enables downstream users and tools to dynamically discover which
> > kfuncs are available on a system by parsing vmlinux or module BTF, both
> > available in /sys/kernel/btf.
> > 
> > This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> > 
> > Acked-by: Jiri Olsa <jolsa@kernel.org>
> > Tested-by: Jiri Olsa <jolsa@kernel.org>
> > Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
> > Tested-by: Alan Maguire <alan.maguire@oracle.com>
> > Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> 
> Here I needed this to avoid an strdup possibly followed by a free and
> then checking the strdup result, please Ack/revalidate tags. I'm
> dropping them as there are changes.
> 
> - Arnaldo
> 
> diff --git a/btf_encoder.c b/btf_encoder.c
> index e9d82e0af0e178fd..c2df2bc7a374447b 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -1434,11 +1434,13 @@ static char *get_func_name(const char *sym)
>  		return NULL;
>  
>  	/* Strip prefix and handle malformed input such as  __BTF_ID__func___ */
> -	func = strdup(sym + sizeof(BTF_ID_FUNC_PFX) - 1);
> -	if (!strstr(func, "__")) {
> -                free(func);
> +	const char *func_sans_prefix = sym + sizeof(BTF_ID_FUNC_PFX) - 1;
> +	if (!strstr(func_sans_prefix, "__"))
>                  return NULL;
> -        }
> +
> +	func = strdup(func_sans_prefix);
> +	if (!func)
> +		return NULL;
>  
>  	/* Strip suffix */
>  	end = strrchr(func, '_');

Thanks, that looks good to me. Please lemme know if I should re-spin.

Otherwise,

Acked-by: Daniel Xu <dxu@dxuuu.xyz>

Re: [PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Alan Maguire]

On 01/05/2024 00:00, Daniel Xu wrote:
> Hi Arnaldo,
> 
> On Tue, Apr 30, 2024 at 03:48:06PM GMT, Arnaldo Carvalho de Melo wrote:
>> On Mon, Apr 29, 2024 at 04:45:59PM -0600, Daniel Xu wrote:
>>> Add a feature flag to guard tagging of kfuncs. The next commit will
>>> implement the actual tagging.
>>>
>>> Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
>>> Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
>>
>> Also 'decl_tag_kfuncs' is not enabled when using --btf_features=default,
>> right? as:
>>
>>         BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),
>>
>> And that false is .default_enabled=false.
> 
> I think that `false` is for `initial_value`, isn't it? The macro sets
> the `default_enabled` field.
> 

yep it's the initial unset value. Specifying an option in --btf_features
flips that value, so for initial-off values they are switched on, while
initial-on values are switched off. I _think_ the intent here is to tag
kfuncs by default, so we can add tag_kfuncs to the set of options
specified in pahole-flags for v1.26. We won't be using "default" there
as we want to call out the flags explicitly.

Alan

> Building with this seems to tag the kfuncs for me:
> 
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf                                                                                                                                                                 
> index 82377e470aed..7128dc25ba29 100644                                                                                                                                                                                  
> --- a/scripts/Makefile.btf                                                                                                                                                                                               
> +++ b/scripts/Makefile.btf                                                                                                                                                                                               
> @@ -16,4 +16,6 @@ pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)                += --lang_exclude=rust                                                                                                                   
>                                                                                                                                                                                                                          
>  pahole-flags-$(call test-ge, $(pahole-ver), 125)       += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized                                                                                                    
>                                                                                                                                                                                                                          
> +pahole-flags-$(call test-ge, $(pahole-ver), 126)       = -j --lang_exclude=rust --btf_features=default                                                                                                                  
> +                                                                                                                                                                                                                        
>  export PAHOLE_FLAGS := $(pahole-flags-y)
> 
> Thanks,
> Daniel

Re: [PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Arnaldo Carvalho de Melo]

On Thu, May 02, 2024 at 12:49:26PM +0100, Alan Maguire wrote:
> On 01/05/2024 00:00, Daniel Xu wrote:
> > On Tue, Apr 30, 2024 at 03:48:06PM GMT, Arnaldo Carvalho de Melo wrote:
> >> On Mon, Apr 29, 2024 at 04:45:59PM -0600, Daniel Xu wrote:
> >> Also 'decl_tag_kfuncs' is not enabled when using --btf_features=default,
> >> right? as:

> >>         BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),

> >> And that false is .default_enabled=false.

> > I think that `false` is for `initial_value`, isn't it? The macro sets
> > the `default_enabled` field.
 
> yep it's the initial unset value. Specifying an option in --btf_features
> flips that value, so for initial-off values they are switched on, while

So --btf_features=something may mean "don't use that feature"? That is
confusing, perhaps the '-something' come in handy?

> initial-on values are switched off. I _think_ the intent here is to tag
> kfuncs by default, so we can add tag_kfuncs to the set of options

Probably, if they are present in the file being BTF encoded.

> specified in pahole-flags for v1.26. We won't be using "default" there
> as we want to call out the flags explicitly.

Sure.

- Arnaldo

Re: [PATCH dwarves v9 2/3] pahole: Add --btf_feature=decl_tag_kfuncs feature

[Alan Maguire]

On 02/05/2024 14:35, Arnaldo Carvalho de Melo wrote:
> On Thu, May 02, 2024 at 12:49:26PM +0100, Alan Maguire wrote:
>> On 01/05/2024 00:00, Daniel Xu wrote:
>>> On Tue, Apr 30, 2024 at 03:48:06PM GMT, Arnaldo Carvalho de Melo wrote:
>>>> On Mon, Apr 29, 2024 at 04:45:59PM -0600, Daniel Xu wrote:
>>>> Also 'decl_tag_kfuncs' is not enabled when using --btf_features=default,
>>>> right? as:
> 
>>>>         BTF_DEFAULT_FEATURE(decl_tag_kfuncs, btf_decl_tag_kfuncs, false),
> 
>>>> And that false is .default_enabled=false.
> 
>>> I think that `false` is for `initial_value`, isn't it? The macro sets
>>> the `default_enabled` field.
>  
>> yep it's the initial unset value. Specifying an option in --btf_features
>> flips that value, so for initial-off values they are switched on, while
> 
> So --btf_features=something may mean "don't use that feature"? That is
> confusing, perhaps the '-something' come in handy?
>

No, in fact --btf_features tries to move away from the model of having a
mix of enable and skip features. The reason we do things this way is we
inherited a situation where some features that would begin as off if not
specified (encode FLOAT) and some that would start off as on unless a
skip option was specified (encode VAR). Prior to --btf_features, we
accordingly had --enable-feature and --skip-feature flags for these.
However with --btf_features, all features are positive; that is, if
specified, we enable var, float etc; there are no "skip" features.

Under the hood however, we preserve the prior "enable or skip"
semantics; that's why some default values have initial values of false
(an "enable" feature under the hood) and some have an initial value of
true (a "skip" value under the hood. But none of that is exposed to the
--btf_features user; if a feature is wanted, they just add it to the list.


>> initial-on values are switched off. I _think_ the intent here is to tag
>> kfuncs by default, so we can add tag_kfuncs to the set of options
> 
> Probably, if they are present in the file being BTF encoded.
> 
>> specified in pahole-flags for v1.26. We won't be using "default" there
>> as we want to call out the flags explicitly.
> 
> Sure.
> 
> - Arnaldo
> 

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Arnaldo Carvalho de Melo]

On Tue, Apr 30, 2024 at 05:27:24PM -0600, Daniel Xu wrote:
> On Tue, Apr 30, 2024 at 05:42:14PM GMT, Arnaldo Carvalho de Melo wrote:
> > On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> > > This commit teaches pahole to parse symbols in .BTF_ids section in
> > > vmlinux and discover exported kfuncs. Pahole then takes the list of
> > > kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> > > 
> > > Example of encoding:
> > > 
> > >         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
> > >         121
> > > 
> > >         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
> > >         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
> > >         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> > > 
> > > This enables downstream users and tools to dynamically discover which
> > > kfuncs are available on a system by parsing vmlinux or module BTF, both
> > > available in /sys/kernel/btf.
> > > 
> > > This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> > 
> > I'm trying this but:
> > 
> > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > btf_encoder__tag_kfuncs(cgroup_rstat_updated): found=0
> > btf_encoder__tag_kfuncs(cgroup_rstat_flush): found=0
> > btf_encoder__tag_kfuncs(security_file_permission): found=0
> > btf_encoder__tag_kfuncs(security_inode_getattr): found=0
> > btf_encoder__tag_kfuncs(security_file_open): found=0
> > btf_encoder__tag_kfuncs(security_path_truncate): found=0
> > btf_encoder__tag_kfuncs(vfs_truncate): found=0
> > btf_encoder__tag_kfuncs(vfs_fallocate): found=0
> > btf_encoder__tag_kfuncs(dentry_open): found=0
> > btf_encoder__tag_kfuncs(vfs_getattr): found=0
> > btf_encoder__tag_kfuncs(filp_close): found=0
> > btf_encoder__tag_kfuncs(bpf_lookup_user_key): found=0
> > btf_encoder__tag_kfuncs(bpf_lookup_system_key): found=0
> > btf_encoder__tag_kfuncs(bpf_key_put): found=0
> > btf_encoder__tag_kfuncs(bpf_verify_pkcs7_signature): found=0
> > btf_encoder__tag_kfuncs(bpf_obj_new_impl): found=0
> > <SNIP all with found=0>
> > 
> > With:
> > 
> > ⬢[acme@toolbox pahole]$ git diff -U16
> > diff --git a/btf_encoder.c b/btf_encoder.c
> > index c2df2bc7a374447b..27a16d6564381b60 100644
> > --- a/btf_encoder.c
> > +++ b/btf_encoder.c
> > @@ -1689,32 +1689,35 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
> >  		func = get_func_name(name);
> >  		if (!func)
> >  			continue;
> >  
> >  		/* Check if function belongs to a kfunc set */
> >  		ranges = gobuffer__entries(&btf_kfunc_ranges);
> >  		ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
> >  		found = false;
> >  		for (j = 0; j < ranges_cnt; j++) {
> >  			size_t addr = sym.st_value;
> >  
> >  			if (ranges[j].start <= addr && addr < ranges[j].end) {
> >  				found = true;
> >  				break;
> >  			}
> >  		}
> > +
> > +		printf("%s(%s): found=%d\n", __func__, func, found);
> > +
> >  		if (!found) {
> >  			free(func);
> >  			continue;
> >  		}
> >  
> >  		err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
> >  		if (err) {
> >  			fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
> >  			free(func);
> >  			goto out;
> >  		}
> >  		free(func);
> >  	}
> >  
> >  	err = 0;
> >  out:
> > 
> > --------------
> > 
> > The vmlinux I'm testing on has the kfuncs, etc, as we can see with:
> > 
> > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | wc -l
> > 517
> > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | tail
> >  97887: ffffffff83266bfc     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cong_avoid__805493
> >  97888: ffffffff83266c04     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_state__806494
> >  97889: ffffffff83266c0c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cwnd_event__807495
> >  97890: ffffffff83266c14     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_acked__808496
> >  98068: ffffffff83266c24     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_ssthresh__773199
> >  98069: ffffffff83266c2c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_cong_avoid__774200
> >  98070: ffffffff83266c34     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_undo_cwnd__775201
> >  98071: ffffffff83266c3c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_slow_start__776202
> >  98072: ffffffff83266c44     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_cong_avoid_ai__777203
> > 101522: ffffffff83266c5c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__update_socket_protocol__80024
> > ⬢[acme@toolbox pahole]$
> > 
> > 
> > So that btf_encoder__tag_kfuncs() isn't finding any?
> > 
> > $ pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > btf_encoder__tag_kfuncs(vmlinux)
> > 
> > Yeah, getting the source filename, the right one.
> > 
> > Then is_sym_kfunc_set() never returns true... But:
> > 
> > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > 
> > real	0m5.586s
> > user	0m29.707s
> > sys	0m2.160s
> > ⬢[acme@toolbox pahole]$
> > 
> > And then:
> > 
> > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> 
> set->flags=0 here is odd. I'd expect at least some of those to be
> non-zero. Can you check if your tree has
> https://github.com/torvalds/linux/commit/6f3189f38a3e995232e028a4c341164c4aca1b20
> ?

⬢[acme@toolbox linux]$ git tag --contains 6f3189f38a3e995232e028a4c341164c4aca1b20
v6.9-rc1
v6.9-rc2
v6.9-rc3
v6.9-rc4
v6.9-rc5
v6.9-rc6
v6.9-rc7
⬢[acme@toolbox linux]$ git log --oneline -1
dd5a440a31fae6e4 (HEAD, tag: v6.9-rc7, torvalds/master) Linux 6.9-rc7
⬢[acme@toolbox linux]$

So now with a just built upstream kernel I get the output below, do you
have patches for other tools to consume this? Or does, say, bpftrace
already handles such decl tags, etc?

I think I'll make pfunct (its in the pahole git repo) to consume it:

⬢[acme@toolbox pahole]$ pfunct --help
Usage: pfunct [OPTION...] FILE

  -a, --addr=ADDR            show just the function that where ADDR is
  -b, --expand_types         Expand types needed by the prototype
      --compile[=FUNCTION]   Generate compilable source code with types
                             expanded (Default all functions)
  -c, --class=CLASS          functions that have CLASS pointer parameters
  -E, --externals            show just external functions
  -f, --function=FUNCTION    show just FUNCTION
  -F, --format_path=FORMAT_LIST   List of debugging formats to try
  -g, --goto_labels          show number of goto labels
  -G, --cc_uninlined         declared inline, uninlined by compiler
  -H, --cc_inlined           not declared inline, inlined by compiler
  -i, --inline_expansions    show inline expansions
  -I, --inline_expansions_stats   show inline expansions stats
  -l, --decl_info            show source code info
      --no_parm_names        Don't show parameter names
  -N, --function_name_len    show size of functions names
  -p, --nr_parms             show number of parameters
  -P, --prototypes           show function prototypes
      --symtab[=NAME]        show symbol table NAME (Default .symtab)
  -s, --sizes                show size of functions
  -S, --nr_variables         show number of variables
  -t, --total_inline_stats   show Multi-CU total inline expansions stats
  -T, --variables            show variables
  -V, --verbose              be verbose
  -?, --help                 Give this help list
      --usage                Give a short usage message
      --version              Print program version

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.
⬢[acme@toolbox pahole]$

- Arnaldo

⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux-v6.9.0-rc7

real	0m5.938s
user	0m32.050s
sys	0m2.075s
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.decl_tag,decl_tag_kfuncs | grep -w 94151
[94151] FUNC 'cgroup_rstat_updated' type_id=94150 linkage=static
[135450] DECL_TAG 'bpf_kfunc' type_id=94151 component_idx=-1
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.decl_tag,decl_tag_kfuncs | grep -w 94150 -A2
[94150] FUNC_PROTO '(anon)' ret_type_id=0 vlen=2
	'cgrp' type_id=744
	'cpu' type_id=12
[94151] FUNC 'cgroup_rstat_updated' type_id=94150 linkage=static
[94152] STRUCT 'pids_cgroup' size=288 vlen=6
	'css' type_id=1786 bits_offset=0
⬢[acme@toolbox pahole]$
⬢[acme@toolbox pahole]$ bpftool btf dump file vmlinux.btf.decl_tag,decl_tag_kfuncs  | grep DECL
	'BTF_KIND_DECL_TAG' val=17
[135450] DECL_TAG 'bpf_kfunc' type_id=94151 component_idx=-1
[135451] DECL_TAG 'bpf_kfunc' type_id=94146 component_idx=-1
[135452] DECL_TAG 'bpf_kfunc' type_id=74311 component_idx=-1
[135453] DECL_TAG 'bpf_kfunc' type_id=74309 component_idx=-1
[135454] DECL_TAG 'bpf_kfunc' type_id=74307 component_idx=-1
[135455] DECL_TAG 'bpf_kfunc' type_id=74305 component_idx=-1
[135456] DECL_TAG 'bpf_kfunc' type_id=74302 component_idx=-1
[135457] DECL_TAG 'bpf_kfunc' type_id=43681 component_idx=-1
[135458] DECL_TAG 'bpf_kfunc' type_id=83157 component_idx=-1
[135459] DECL_TAG 'bpf_kfunc' type_id=83156 component_idx=-1
[135460] DECL_TAG 'bpf_kfunc' type_id=83152 component_idx=-1
[135461] DECL_TAG 'bpf_kfunc' type_id=83151 component_idx=-1
[135462] DECL_TAG 'bpf_kfunc' type_id=83149 component_idx=-1
[135463] DECL_TAG 'bpf_kfunc' type_id=83145 component_idx=-1
[135464] DECL_TAG 'bpf_kfunc' type_id=83144 component_idx=-1
[135465] DECL_TAG 'bpf_kfunc' type_id=83142 component_idx=-1
[135466] DECL_TAG 'bpf_kfunc' type_id=83141 component_idx=-1
[135467] DECL_TAG 'bpf_kfunc' type_id=83133 component_idx=-1
[135468] DECL_TAG 'bpf_kfunc' type_id=83132 component_idx=-1
[135469] DECL_TAG 'bpf_kfunc' type_id=83139 component_idx=-1
[135470] DECL_TAG 'bpf_kfunc' type_id=83137 component_idx=-1
[135471] DECL_TAG 'bpf_kfunc' type_id=83135 component_idx=-1
[135472] DECL_TAG 'bpf_kfunc' type_id=83130 component_idx=-1
[135473] DECL_TAG 'bpf_kfunc' type_id=83128 component_idx=-1
[135474] DECL_TAG 'bpf_kfunc' type_id=83125 component_idx=-1
[135475] DECL_TAG 'bpf_kfunc' type_id=83123 component_idx=-1
[135476] DECL_TAG 'bpf_kfunc' type_id=83121 component_idx=-1
[135477] DECL_TAG 'bpf_kfunc' type_id=83119 component_idx=-1
[135478] DECL_TAG 'bpf_kfunc' type_id=83117 component_idx=-1
[135479] DECL_TAG 'bpf_kfunc' type_id=83095 component_idx=-1
[135480] DECL_TAG 'bpf_kfunc' type_id=83103 component_idx=-1
[135481] DECL_TAG 'bpf_kfunc' type_id=83101 component_idx=-1
[135482] DECL_TAG 'bpf_kfunc' type_id=83099 component_idx=-1
[135483] DECL_TAG 'bpf_kfunc' type_id=83098 component_idx=-1
[135484] DECL_TAG 'bpf_kfunc' type_id=83115 component_idx=-1
[135485] DECL_TAG 'bpf_kfunc' type_id=83114 component_idx=-1
[135486] DECL_TAG 'bpf_kfunc' type_id=24936 component_idx=-1
[135487] DECL_TAG 'bpf_kfunc' type_id=24934 component_idx=-1
[135488] DECL_TAG 'bpf_kfunc' type_id=24932 component_idx=-1
[135489] DECL_TAG 'bpf_kfunc' type_id=35395 component_idx=-1
[135490] DECL_TAG 'bpf_kfunc' type_id=35393 component_idx=-1
[135491] DECL_TAG 'bpf_kfunc' type_id=35391 component_idx=-1
[135492] DECL_TAG 'bpf_kfunc' type_id=35389 component_idx=-1
[135493] DECL_TAG 'bpf_kfunc' type_id=35387 component_idx=-1
[135494] DECL_TAG 'bpf_kfunc' type_id=35385 component_idx=-1
[135495] DECL_TAG 'bpf_kfunc' type_id=129497 component_idx=-1
[135496] DECL_TAG 'bpf_kfunc' type_id=129495 component_idx=-1
[135497] DECL_TAG 'bpf_kfunc' type_id=129493 component_idx=-1
[135498] DECL_TAG 'bpf_kfunc' type_id=35383 component_idx=-1
[135499] DECL_TAG 'bpf_kfunc' type_id=35381 component_idx=-1
[135500] DECL_TAG 'bpf_kfunc' type_id=35379 component_idx=-1
[135501] DECL_TAG 'bpf_kfunc' type_id=83112 component_idx=-1
[135502] DECL_TAG 'bpf_kfunc' type_id=83110 component_idx=-1
[135503] DECL_TAG 'bpf_kfunc' type_id=83109 component_idx=-1
[135504] DECL_TAG 'bpf_kfunc' type_id=83107 component_idx=-1
[135505] DECL_TAG 'bpf_kfunc' type_id=83105 component_idx=-1
[135506] DECL_TAG 'bpf_kfunc' type_id=129409 component_idx=-1
[135507] DECL_TAG 'bpf_kfunc' type_id=119719 component_idx=-1
[135508] DECL_TAG 'bpf_kfunc' type_id=119717 component_idx=-1
[135509] DECL_TAG 'bpf_kfunc' type_id=83434 component_idx=-1
[135510] DECL_TAG 'bpf_kfunc' type_id=83430 component_idx=-1
[135511] DECL_TAG 'bpf_kfunc' type_id=83432 component_idx=-1
[135512] DECL_TAG 'bpf_kfunc' type_id=83427 component_idx=-1
[135513] DECL_TAG 'bpf_kfunc' type_id=83426 component_idx=-1
[135514] DECL_TAG 'bpf_kfunc' type_id=83425 component_idx=-1
[135515] DECL_TAG 'bpf_kfunc' type_id=83424 component_idx=-1
[135516] DECL_TAG 'bpf_kfunc' type_id=83423 component_idx=-1
[135517] DECL_TAG 'bpf_kfunc' type_id=83421 component_idx=-1
[135518] DECL_TAG 'bpf_kfunc' type_id=83419 component_idx=-1
[135519] DECL_TAG 'bpf_kfunc' type_id=83418 component_idx=-1
[135520] DECL_TAG 'bpf_kfunc' type_id=83416 component_idx=-1
[135521] DECL_TAG 'bpf_kfunc' type_id=83415 component_idx=-1
[135522] DECL_TAG 'bpf_kfunc' type_id=83413 component_idx=-1
[135523] DECL_TAG 'bpf_kfunc' type_id=83411 component_idx=-1
[135524] DECL_TAG 'bpf_kfunc' type_id=83410 component_idx=-1
[135525] DECL_TAG 'bpf_kfunc' type_id=83408 component_idx=-1
[135526] DECL_TAG 'bpf_kfunc' type_id=83407 component_idx=-1
[135527] DECL_TAG 'bpf_kfunc' type_id=83406 component_idx=-1
[135528] DECL_TAG 'bpf_kfunc' type_id=83404 component_idx=-1
[135529] DECL_TAG 'bpf_kfunc' type_id=83403 component_idx=-1
[135530] DECL_TAG 'bpf_kfunc' type_id=83401 component_idx=-1
[135531] DECL_TAG 'bpf_kfunc' type_id=83399 component_idx=-1
[135532] DECL_TAG 'bpf_kfunc' type_id=83398 component_idx=-1
[135533] DECL_TAG 'bpf_kfunc' type_id=83396 component_idx=-1
[135534] DECL_TAG 'bpf_kfunc' type_id=84592 component_idx=-1
[135535] DECL_TAG 'bpf_kfunc' type_id=78862 component_idx=-1
[135536] DECL_TAG 'bpf_kfunc' type_id=78855 component_idx=-1
[135537] DECL_TAG 'bpf_kfunc' type_id=78853 component_idx=-1
[135538] DECL_TAG 'bpf_kfunc' type_id=78851 component_idx=-1
[135539] DECL_TAG 'bpf_kfunc' type_id=78849 component_idx=-1
[135540] DECL_TAG 'bpf_kfunc' type_id=52826 component_idx=-1
[135541] DECL_TAG 'bpf_kfunc' type_id=52825 component_idx=-1
[135542] DECL_TAG 'bpf_kfunc' type_id=52823 component_idx=-1
[135543] DECL_TAG 'bpf_kfunc' type_id=52821 component_idx=-1
[135544] DECL_TAG 'bpf_kfunc' type_id=52816 component_idx=-1
[135545] DECL_TAG 'bpf_kfunc' type_id=79148 component_idx=-1
[135546] DECL_TAG 'bpf_kfunc' type_id=79146 component_idx=-1
[135547] DECL_TAG 'bpf_kfunc' type_id=79144 component_idx=-1
[135548] DECL_TAG 'bpf_kfunc' type_id=33070 component_idx=-1
[135549] DECL_TAG 'bpf_kfunc' type_id=33068 component_idx=-1
[135550] DECL_TAG 'bpf_kfunc' type_id=33088 component_idx=-1
[135551] DECL_TAG 'bpf_kfunc' type_id=33064 component_idx=-1
[135552] DECL_TAG 'bpf_kfunc' type_id=33061 component_idx=-1
[135553] DECL_TAG 'bpf_kfunc' type_id=134975 component_idx=-1
[135554] DECL_TAG 'bpf_kfunc' type_id=134971 component_idx=-1
[135555] DECL_TAG 'bpf_kfunc' type_id=134972 component_idx=-1
[135556] DECL_TAG 'bpf_kfunc' type_id=134970 component_idx=-1
[135557] DECL_TAG 'bpf_kfunc' type_id=134974 component_idx=-1
[135558] DECL_TAG 'bpf_kfunc' type_id=134969 component_idx=-1
[135559] DECL_TAG 'bpf_kfunc' type_id=53825 component_idx=-1
[135560] DECL_TAG 'bpf_kfunc' type_id=53827 component_idx=-1
[135561] DECL_TAG 'bpf_kfunc' type_id=53824 component_idx=-1
[135562] DECL_TAG 'bpf_kfunc' type_id=53831 component_idx=-1
[135563] DECL_TAG 'bpf_kfunc' type_id=53829 component_idx=-1
[135564] DECL_TAG 'bpf_kfunc' type_id=21317 component_idx=-1
[135565] DECL_TAG 'bpf_kfunc' type_id=21315 component_idx=-1
⬢[acme@toolbox pahole]$

Re: [PATCH dwarves v9 3/3] pahole: Inject kfunc decl tags into BTF

[Daniel Xu]

On Mon, May 06, 2024 at 05:07:54PM GMT, Arnaldo Carvalho de Melo wrote:
> On Tue, Apr 30, 2024 at 05:27:24PM -0600, Daniel Xu wrote:
> > On Tue, Apr 30, 2024 at 05:42:14PM GMT, Arnaldo Carvalho de Melo wrote:
> > > On Mon, Apr 29, 2024 at 04:46:00PM -0600, Daniel Xu wrote:
> > > > This commit teaches pahole to parse symbols in .BTF_ids section in
> > > > vmlinux and discover exported kfuncs. Pahole then takes the list of
> > > > kfuncs and injects a BTF_KIND_DECL_TAG for each kfunc.
> > > > 
> > > > Example of encoding:
> > > > 
> > > >         $ bpftool btf dump file .tmp_vmlinux.btf | rg "DECL_TAG 'bpf_kfunc'" | wc -l
> > > >         121
> > > > 
> > > >         $ bpftool btf dump file .tmp_vmlinux.btf | rg 56337
> > > >         [56337] FUNC 'bpf_ct_change_timeout' type_id=56336 linkage=static
> > > >         [127861] DECL_TAG 'bpf_kfunc' type_id=56337 component_idx=-1
> > > > 
> > > > This enables downstream users and tools to dynamically discover which
> > > > kfuncs are available on a system by parsing vmlinux or module BTF, both
> > > > available in /sys/kernel/btf.
> > > > 
> > > > This feature is enabled with --btf_features=decl_tag,decl_tag_kfuncs.
> > > 
> > > I'm trying this but:
> > > 
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > btf_encoder__tag_kfuncs(cgroup_rstat_updated): found=0
> > > btf_encoder__tag_kfuncs(cgroup_rstat_flush): found=0
> > > btf_encoder__tag_kfuncs(security_file_permission): found=0
> > > btf_encoder__tag_kfuncs(security_inode_getattr): found=0
> > > btf_encoder__tag_kfuncs(security_file_open): found=0
> > > btf_encoder__tag_kfuncs(security_path_truncate): found=0
> > > btf_encoder__tag_kfuncs(vfs_truncate): found=0
> > > btf_encoder__tag_kfuncs(vfs_fallocate): found=0
> > > btf_encoder__tag_kfuncs(dentry_open): found=0
> > > btf_encoder__tag_kfuncs(vfs_getattr): found=0
> > > btf_encoder__tag_kfuncs(filp_close): found=0
> > > btf_encoder__tag_kfuncs(bpf_lookup_user_key): found=0
> > > btf_encoder__tag_kfuncs(bpf_lookup_system_key): found=0
> > > btf_encoder__tag_kfuncs(bpf_key_put): found=0
> > > btf_encoder__tag_kfuncs(bpf_verify_pkcs7_signature): found=0
> > > btf_encoder__tag_kfuncs(bpf_obj_new_impl): found=0
> > > <SNIP all with found=0>
> > > 
> > > With:
> > > 
> > > ⬢[acme@toolbox pahole]$ git diff -U16
> > > diff --git a/btf_encoder.c b/btf_encoder.c
> > > index c2df2bc7a374447b..27a16d6564381b60 100644
> > > --- a/btf_encoder.c
> > > +++ b/btf_encoder.c
> > > @@ -1689,32 +1689,35 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
> > >  		func = get_func_name(name);
> > >  		if (!func)
> > >  			continue;
> > >  
> > >  		/* Check if function belongs to a kfunc set */
> > >  		ranges = gobuffer__entries(&btf_kfunc_ranges);
> > >  		ranges_cnt = gobuffer__nr_entries(&btf_kfunc_ranges);
> > >  		found = false;
> > >  		for (j = 0; j < ranges_cnt; j++) {
> > >  			size_t addr = sym.st_value;
> > >  
> > >  			if (ranges[j].start <= addr && addr < ranges[j].end) {
> > >  				found = true;
> > >  				break;
> > >  			}
> > >  		}
> > > +
> > > +		printf("%s(%s): found=%d\n", __func__, func, found);
> > > +
> > >  		if (!found) {
> > >  			free(func);
> > >  			continue;
> > >  		}
> > >  
> > >  		err = btf_encoder__tag_kfunc(encoder, &btf_funcs, func);
> > >  		if (err) {
> > >  			fprintf(stderr, "%s: failed to tag kfunc '%s'\n", __func__, func);
> > >  			free(func);
> > >  			goto out;
> > >  		}
> > >  		free(func);
> > >  	}
> > >  
> > >  	err = 0;
> > >  out:
> > > 
> > > --------------
> > > 
> > > The vmlinux I'm testing on has the kfuncs, etc, as we can see with:
> > > 
> > > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | wc -l
> > > 517
> > > ⬢[acme@toolbox pahole]$ readelf -sW vmlinux | grep __BTF_ID__func__ | tail
> > >  97887: ffffffff83266bfc     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cong_avoid__805493
> > >  97888: ffffffff83266c04     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_state__806494
> > >  97889: ffffffff83266c0c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_cwnd_event__807495
> > >  97890: ffffffff83266c14     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__cubictcp_acked__808496
> > >  98068: ffffffff83266c24     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_ssthresh__773199
> > >  98069: ffffffff83266c2c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_cong_avoid__774200
> > >  98070: ffffffff83266c34     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_reno_undo_cwnd__775201
> > >  98071: ffffffff83266c3c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_slow_start__776202
> > >  98072: ffffffff83266c44     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__tcp_cong_avoid_ai__777203
> > > 101522: ffffffff83266c5c     4 OBJECT  LOCAL  DEFAULT   15 __BTF_ID__func__update_socket_protocol__80024
> > > ⬢[acme@toolbox pahole]$
> > > 
> > > 
> > > So that btf_encoder__tag_kfuncs() isn't finding any?
> > > 
> > > $ pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > btf_encoder__tag_kfuncs(vmlinux)
> > > 
> > > Yeah, getting the source filename, the right one.
> > > 
> > > Then is_sym_kfunc_set() never returns true... But:
> > > 
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__)
> > > 
> > > real	0m5.586s
> > > user	0m29.707s
> > > sys	0m2.160s
> > > ⬢[acme@toolbox pahole]$
> > > 
> > > And then:
> > > 
> > > ⬢[acme@toolbox pahole]$ time pahole -j --btf_features=decl_tag,decl_tag_kfuncs --btf_encode_detached=vmlinux.btf.decl_tag,decl_tag_kfuncs vmlinux
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_rstat_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__key_sig_kfunc_set, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__generic_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__common_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_map_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__cpumask_kfunc_btf_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__hid_bpf_syscall_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_skb, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_xdp, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_kfunc_check_set_sock_addr, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_sk_iter_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__xdp_metadata_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_test_modify_return_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__test_sk_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__tcp_cubic_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > > is_sym_kfunc_set(__BTF_ID__set8__bpf_mptcp_fmodret_ids, BTF_ID_SET8_PFX=__BTF_ID__set8__, set->flags= 0, BTF_SET8_KFUNCS=1, ret=0)
> > 
> > set->flags=0 here is odd. I'd expect at least some of those to be
> > non-zero. Can you check if your tree has
> > https://github.com/torvalds/linux/commit/6f3189f38a3e995232e028a4c341164c4aca1b20
> > ?
> 
> ⬢[acme@toolbox linux]$ git tag --contains 6f3189f38a3e995232e028a4c341164c4aca1b20
> v6.9-rc1
> v6.9-rc2
> v6.9-rc3
> v6.9-rc4
> v6.9-rc5
> v6.9-rc6
> v6.9-rc7
> ⬢[acme@toolbox linux]$ git log --oneline -1
> dd5a440a31fae6e4 (HEAD, tag: v6.9-rc7, torvalds/master) Linux 6.9-rc7
> ⬢[acme@toolbox linux]$
> 
> So now with a just built upstream kernel I get the output below, do you
> have patches for other tools to consume this? Or does, say, bpftrace
> already handles such decl tags, etc?

Yep, I have v3 of https://lore.kernel.org/bpf/cover.1707080349.git.dxu@dxuuu.xyz/
ready to send. The step after that bpftool patch is merged is
teach/simplify the bpf selftests. Long term once all the changes
propagate, it'll make bpf programmers workflows more efficient too.

bpftrace will probably make use of these tags in the future. We are
still working on our kfunc story (not supported yet).

[..]

> [135552] DECL_TAG 'bpf_kfunc' type_id=33061 component_idx=-1
> [135553] DECL_TAG 'bpf_kfunc' type_id=134975 component_idx=-1
> [135554] DECL_TAG 'bpf_kfunc' type_id=134971 component_idx=-1
> [135555] DECL_TAG 'bpf_kfunc' type_id=134972 component_idx=-1
> [135556] DECL_TAG 'bpf_kfunc' type_id=134970 component_idx=-1
> [135557] DECL_TAG 'bpf_kfunc' type_id=134974 component_idx=-1
> [135558] DECL_TAG 'bpf_kfunc' type_id=134969 component_idx=-1
> [135559] DECL_TAG 'bpf_kfunc' type_id=53825 component_idx=-1
> [135560] DECL_TAG 'bpf_kfunc' type_id=53827 component_idx=-1
> [135561] DECL_TAG 'bpf_kfunc' type_id=53824 component_idx=-1
> [135562] DECL_TAG 'bpf_kfunc' type_id=53831 component_idx=-1
> [135563] DECL_TAG 'bpf_kfunc' type_id=53829 component_idx=-1
> [135564] DECL_TAG 'bpf_kfunc' type_id=21317 component_idx=-1
> [135565] DECL_TAG 'bpf_kfunc' type_id=21315 component_idx=-1
> ⬢[acme@toolbox pahole]$

Glad it works!

Thanks,
Daniel