* [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
@ 2024-05-01 23:12 Paul E. McKenney
2024-05-02 2:05 ` Masami Hiramatsu
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2024-05-01 23:12 UTC (permalink / raw)
To: linux-kernel, linux-trace-kernel
Cc: kuba, ast, clm, rostedt, mhiramat, mark.rutland, mathieu.desnoyers
Note that the immediate pressure for this patch should be relieved by the
NAPI patch series [1], but this sort of problem could easily arise again.
When running heavy test workloads with KASAN enabled, RCU Tasks grace
periods can extend for many tens of seconds, significantly slowing
trace registration. Therefore, make the registration-side RCU Tasks
grace period be asynchronous via call_rcu_tasks().
[1] https://lore.kernel.org/all/cover.1710877680.git.yan@cloudflare.com/
Reported-by: Jakub Kicinski <kuba@kernel.org>
Reported-by: Alexei Starovoitov <ast@kernel.org>
Reported-by: Chris Mason <clm@fb.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: <linux-trace-kernel@vger.kernel.org>
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 6c96b30f3d63b..32ea92934268c 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5365,6 +5365,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
}
}
+static void register_ftrace_direct_cb(struct rcu_head *rhp)
+{
+ struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
+
+ free_ftrace_hash(fhp);
+}
+
/**
* register_ftrace_direct - Call a custom trampoline directly
* for multiple functions registered in @ops
@@ -5463,10 +5470,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
out_unlock:
mutex_unlock(&direct_mutex);
- if (free_hash && free_hash != EMPTY_HASH) {
- synchronize_rcu_tasks();
- free_ftrace_hash(free_hash);
- }
+ if (free_hash && free_hash != EMPTY_HASH)
+ call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
if (new_hash)
free_ftrace_hash(new_hash);
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
2024-05-01 23:12 [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct() Paul E. McKenney
@ 2024-05-02 2:05 ` Masami Hiramatsu
2024-05-02 3:31 ` Paul E. McKenney
0 siblings, 1 reply; 6+ messages in thread
From: Masami Hiramatsu @ 2024-05-02 2:05 UTC (permalink / raw)
To: paulmck
Cc: linux-kernel, linux-trace-kernel, kuba, ast, clm, rostedt,
mhiramat, mark.rutland, mathieu.desnoyers
On Wed, 1 May 2024 16:12:37 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:
> Note that the immediate pressure for this patch should be relieved by the
> NAPI patch series [1], but this sort of problem could easily arise again.
>
> When running heavy test workloads with KASAN enabled, RCU Tasks grace
> periods can extend for many tens of seconds, significantly slowing
> trace registration. Therefore, make the registration-side RCU Tasks
> grace period be asynchronous via call_rcu_tasks().
>
Good catch! AFAICS, there is no reason to wait for synchronization
when adding a new direct trampoline.
This looks good to me.
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Thank you,
> [1] https://lore.kernel.org/all/cover.1710877680.git.yan@cloudflare.com/
>
> Reported-by: Jakub Kicinski <kuba@kernel.org>
> Reported-by: Alexei Starovoitov <ast@kernel.org>
> Reported-by: Chris Mason <clm@fb.com>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: <linux-trace-kernel@vger.kernel.org>
>
> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index 6c96b30f3d63b..32ea92934268c 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -5365,6 +5365,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
> }
> }
>
> +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> +{
> + struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
> +
> + free_ftrace_hash(fhp);
> +}
> +
> /**
> * register_ftrace_direct - Call a custom trampoline directly
> * for multiple functions registered in @ops
> @@ -5463,10 +5470,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
> out_unlock:
> mutex_unlock(&direct_mutex);
>
> - if (free_hash && free_hash != EMPTY_HASH) {
> - synchronize_rcu_tasks();
> - free_ftrace_hash(free_hash);
> - }
> + if (free_hash && free_hash != EMPTY_HASH)
> + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
>
> if (new_hash)
> free_ftrace_hash(new_hash);
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
2024-05-02 2:05 ` Masami Hiramatsu
@ 2024-05-02 3:31 ` Paul E. McKenney
2024-05-02 21:31 ` Steven Rostedt
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2024-05-02 3:31 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: linux-kernel, linux-trace-kernel, kuba, ast, clm, rostedt,
mark.rutland, mathieu.desnoyers
On Thu, May 02, 2024 at 11:05:01AM +0900, Masami Hiramatsu wrote:
> On Wed, 1 May 2024 16:12:37 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
>
> > Note that the immediate pressure for this patch should be relieved by the
> > NAPI patch series [1], but this sort of problem could easily arise again.
> >
> > When running heavy test workloads with KASAN enabled, RCU Tasks grace
> > periods can extend for many tens of seconds, significantly slowing
> > trace registration. Therefore, make the registration-side RCU Tasks
> > grace period be asynchronous via call_rcu_tasks().
>
> Good catch! AFAICS, there is no reason to wait for synchronization
> when adding a new direct trampoline.
> This looks good to me.
>
> Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Thank you very much! I will apply this on my next rebase.
Thanx, Paul
> Thank you,
>
> > [1] https://lore.kernel.org/all/cover.1710877680.git.yan@cloudflare.com/
> >
> > Reported-by: Jakub Kicinski <kuba@kernel.org>
> > Reported-by: Alexei Starovoitov <ast@kernel.org>
> > Reported-by: Chris Mason <clm@fb.com>
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > Cc: Steven Rostedt <rostedt@goodmis.org>
> > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Cc: <linux-trace-kernel@vger.kernel.org>
> >
> > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> > index 6c96b30f3d63b..32ea92934268c 100644
> > --- a/kernel/trace/ftrace.c
> > +++ b/kernel/trace/ftrace.c
> > @@ -5365,6 +5365,13 @@ static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long
> > }
> > }
> >
> > +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> > +{
> > + struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
> > +
> > + free_ftrace_hash(fhp);
> > +}
> > +
> > /**
> > * register_ftrace_direct - Call a custom trampoline directly
> > * for multiple functions registered in @ops
> > @@ -5463,10 +5470,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
> > out_unlock:
> > mutex_unlock(&direct_mutex);
> >
> > - if (free_hash && free_hash != EMPTY_HASH) {
> > - synchronize_rcu_tasks();
> > - free_ftrace_hash(free_hash);
> > - }
> > + if (free_hash && free_hash != EMPTY_HASH)
> > + call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
> >
> > if (new_hash)
> > free_ftrace_hash(new_hash);
>
>
> --
> Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
2024-05-02 3:31 ` Paul E. McKenney
@ 2024-05-02 21:31 ` Steven Rostedt
2024-05-02 23:13 ` Paul E. McKenney
0 siblings, 1 reply; 6+ messages in thread
From: Steven Rostedt @ 2024-05-02 21:31 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Masami Hiramatsu, linux-kernel, linux-trace-kernel, kuba, ast,
clm, mark.rutland, mathieu.desnoyers
On Wed, 1 May 2024 20:31:06 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:
> On Thu, May 02, 2024 at 11:05:01AM +0900, Masami Hiramatsu wrote:
> > On Wed, 1 May 2024 16:12:37 -0700
> > "Paul E. McKenney" <paulmck@kernel.org> wrote:
> >
> > > Note that the immediate pressure for this patch should be relieved by the
> > > NAPI patch series [1], but this sort of problem could easily arise again.
> > >
> > > When running heavy test workloads with KASAN enabled, RCU Tasks grace
> > > periods can extend for many tens of seconds, significantly slowing
> > > trace registration. Therefore, make the registration-side RCU Tasks
> > > grace period be asynchronous via call_rcu_tasks().
> >
> > Good catch! AFAICS, there is no reason to wait for synchronization
> > when adding a new direct trampoline.
> > This looks good to me.
> >
> > Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
> Thank you very much! I will apply this on my next rebase.
I can take it.
It's not a bug fix but just an performance improvement, so it can go into
the next merge window.
-- Steve
>
> > Thank you,
> >
> > > [1]
> > > https://lore.kernel.org/all/cover.1710877680.git.yan@cloudflare.com/
> > >
> > > Reported-by: Jakub Kicinski <kuba@kernel.org>
> > > Reported-by: Alexei Starovoitov <ast@kernel.org>
> > > Reported-by: Chris Mason <clm@fb.com>
> > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > > Cc: Steven Rostedt <rostedt@goodmis.org>
> > > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > > Cc: Mark Rutland <mark.rutland@arm.com>
> > > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > Cc: <linux-trace-kernel@vger.kernel.org>
> > >
> > > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> > > index 6c96b30f3d63b..32ea92934268c 100644
> > > --- a/kernel/trace/ftrace.c
> > > +++ b/kernel/trace/ftrace.c
> > > @@ -5365,6 +5365,13 @@ static void
> > > remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long }
> > > }
> > >
> > > +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> > > +{
> > > + struct ftrace_hash *fhp = container_of(rhp, struct
> > > ftrace_hash, rcu); +
> > > + free_ftrace_hash(fhp);
> > > +}
> > > +
> > > /**
> > > * register_ftrace_direct - Call a custom trampoline directly
> > > * for multiple functions registered in @ops
> > > @@ -5463,10 +5470,8 @@ int register_ftrace_direct(struct ftrace_ops
> > > *ops, unsigned long addr) out_unlock:
> > > mutex_unlock(&direct_mutex);
> > >
> > > - if (free_hash && free_hash != EMPTY_HASH) {
> > > - synchronize_rcu_tasks();
> > > - free_ftrace_hash(free_hash);
> > > - }
> > > + if (free_hash && free_hash != EMPTY_HASH)
> > > + call_rcu_tasks(&free_hash->rcu,
> > > register_ftrace_direct_cb);
> > > if (new_hash)
> > > free_ftrace_hash(new_hash);
> >
> >
> > --
> > Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
2024-05-02 21:31 ` Steven Rostedt
@ 2024-05-02 23:13 ` Paul E. McKenney
2024-05-03 0:04 ` Steven Rostedt
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2024-05-02 23:13 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, linux-kernel, linux-trace-kernel, kuba, ast,
clm, mark.rutland, mathieu.desnoyers
On Thu, May 02, 2024 at 05:31:00PM -0400, Steven Rostedt wrote:
> On Wed, 1 May 2024 20:31:06 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
>
> > On Thu, May 02, 2024 at 11:05:01AM +0900, Masami Hiramatsu wrote:
> > > On Wed, 1 May 2024 16:12:37 -0700
> > > "Paul E. McKenney" <paulmck@kernel.org> wrote:
> > >
> > > > Note that the immediate pressure for this patch should be relieved by the
> > > > NAPI patch series [1], but this sort of problem could easily arise again.
> > > >
> > > > When running heavy test workloads with KASAN enabled, RCU Tasks grace
> > > > periods can extend for many tens of seconds, significantly slowing
> > > > trace registration. Therefore, make the registration-side RCU Tasks
> > > > grace period be asynchronous via call_rcu_tasks().
> > >
> > > Good catch! AFAICS, there is no reason to wait for synchronization
> > > when adding a new direct trampoline.
> > > This looks good to me.
> > >
> > > Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> >
> > Thank you very much! I will apply this on my next rebase.
>
> I can take it.
>
> It's not a bug fix but just an performance improvement, so it can go into
> the next merge window.
Very good, and thank you!
I will drop it from RCU as soon as it shows up in either -next or in
mainline.
Thanx, Paul
> -- Steve
>
>
>
> >
> > > Thank you,
> > >
> > > > [1]
> > > > https://lore.kernel.org/all/cover.1710877680.git.yan@cloudflare.com/
> > > >
> > > > Reported-by: Jakub Kicinski <kuba@kernel.org>
> > > > Reported-by: Alexei Starovoitov <ast@kernel.org>
> > > > Reported-by: Chris Mason <clm@fb.com>
> > > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > > > Cc: Steven Rostedt <rostedt@goodmis.org>
> > > > Cc: Masami Hiramatsu <mhiramat@kernel.org>
> > > > Cc: Mark Rutland <mark.rutland@arm.com>
> > > > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > > Cc: <linux-trace-kernel@vger.kernel.org>
> > > >
> > > > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> > > > index 6c96b30f3d63b..32ea92934268c 100644
> > > > --- a/kernel/trace/ftrace.c
> > > > +++ b/kernel/trace/ftrace.c
> > > > @@ -5365,6 +5365,13 @@ static void
> > > > remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long }
> > > > }
> > > >
> > > > +static void register_ftrace_direct_cb(struct rcu_head *rhp)
> > > > +{
> > > > + struct ftrace_hash *fhp = container_of(rhp, struct
> > > > ftrace_hash, rcu); +
> > > > + free_ftrace_hash(fhp);
> > > > +}
> > > > +
> > > > /**
> > > > * register_ftrace_direct - Call a custom trampoline directly
> > > > * for multiple functions registered in @ops
> > > > @@ -5463,10 +5470,8 @@ int register_ftrace_direct(struct ftrace_ops
> > > > *ops, unsigned long addr) out_unlock:
> > > > mutex_unlock(&direct_mutex);
> > > >
> > > > - if (free_hash && free_hash != EMPTY_HASH) {
> > > > - synchronize_rcu_tasks();
> > > > - free_ftrace_hash(free_hash);
> > > > - }
> > > > + if (free_hash && free_hash != EMPTY_HASH)
> > > > + call_rcu_tasks(&free_hash->rcu,
> > > > register_ftrace_direct_cb);
> > > > if (new_hash)
> > > > free_ftrace_hash(new_hash);
> > >
> > >
> > > --
> > > Masami Hiramatsu (Google) <mhiramat@kernel.org>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct()
2024-05-02 23:13 ` Paul E. McKenney
@ 2024-05-03 0:04 ` Steven Rostedt
0 siblings, 0 replies; 6+ messages in thread
From: Steven Rostedt @ 2024-05-03 0:04 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Masami Hiramatsu, linux-kernel, linux-trace-kernel, kuba, ast,
clm, mark.rutland, mathieu.desnoyers
On Thu, 2 May 2024 16:13:59 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:
> Very good, and thank you!
>
> I will drop it from RCU as soon as it shows up in either -next or in
> mainline.
Sounds good.
I'm currently working on updates to get into -rc7 and plan to add my next
work on top of that (I know, I know, it's probably the latest release I had
for next, but things are still being worked on).
-- Steve
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-05-03 0:03 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-05-01 23:12 [PATCH resend ftrace] Asynchronous grace period for register_ftrace_direct() Paul E. McKenney
2024-05-02 2:05 ` Masami Hiramatsu
2024-05-02 3:31 ` Paul E. McKenney
2024-05-02 21:31 ` Steven Rostedt
2024-05-02 23:13 ` Paul E. McKenney
2024-05-03 0:04 ` Steven Rostedt
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.