[PATCH 00/19] media: renesas: rcar-csi2: Support multiple streams

[PATCH 00/19] media: renesas: rcar-csi2: Support multiple streams

[Jacopo Mondi]

Hello this series implements multi-stream support for R-Car CSI-2 and the
two CSI-2 devices that are available on Renesas dev-boards which are mainline
supported (the ADV748x HDMI decoder and the MAX9286 GMSL deserializer).

The net gain in feature is:
- ADV748x: Select on which MIPI CSI-2 VC to send the stream on
- MAX9286: Paves the way for run-time selection of active links

Both features are realized by controlling the devices routing tables.
The max9286 routing support is not implemented, but these patches are needed
in order to maintain the current version compatible with R-Car CSI-2.

The R-Car CSI-2 driver needs to be updated in order to support multiplexed
transmitters, in particular:

- Use LINK_FREQ to compute the lane bandwidth
- Support subdev_active state
- Configure DT and VC handling by using the remote's frame_desc

A separate fix for VIN sits at the top of the series to re-enable YUYV capture
operations.

The series needs to activate streams in v4l2-subdev to work correctly:

--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -32,7 +32,7 @@
  * 'v4l2_subdev_enable_streams_api' to 1 below.
  */

-static bool v4l2_subdev_enable_streams_api;
+static bool v4l2_subdev_enable_streams_api = 1;
 #endif


Test scripts, based on vin-tests are available at:
https://git.uk.ideasonboard.com/jmondi/vin-tests

A note for Sakari: patch
"media: max9286: Implement support for LINK_FREQ"

Could actually use your new
"media: v4l: Support passing sub-device argument to v4l2_get_link_freq()"

I kept it as it is for this version waiting for your patch to get in.

A note for Niklas:
I think this series paves the way to actually moving the control of which
VC goes to which VIN using a routing table instead of using link enablement.

Currently the setup is as it follows: routing within R-Car CSI-2 is fixed, sink
stream 0/X goes to source stream X+1/0 and each source pad represents a VC.
Which VC gets sent to which VIN is controlled by link enablement.

A more natural way of handling this would be to make the routing table
within CSI-2 configurable and have media-links immutable between the CSI-2
source pad. A CSI-2 source pad would then represent a 'channel' between
CSI-2 and VIN and not the VC which is sent on it.

This setup would better represent the current implementation, with VIN's VCSEL
fixed channel routing and with the CSI-2 routing table controlling which VC gets
sent on which channel. What do you think ?

Jacopo Mondi (19):
  media: adv748x: Add support for active state
  media: adv748x: Add flags to adv748x_subdev_init()
  media: adv748x: Use V4L2 streams
  media: adv748x: Propagate format to opposite stream
  media: adv748x: Implement set_routing()
  media: adv748x: Use routes to configure VC
  media: adv748x: Implement .get_frame_desc()
  media: max9286: Add support for subdev active state
  media: max9286: Fix enum_mbus_code
  media: max9286: Use frame interval from subdev state
  media: max9286: Use V4L2 Streams
  media: max9286: Implement .get_frame_desc()
  media: max9286: Implement support for LINK_FREQ
  media: max9286: Implement .get_mbus_config()
  media: rcar-csi2: Add support for multiplexed streams
  media: rcar-csi2: Support multiplexed transmitters
  media: rcar-csi2: Store format in the subdev state
  media: rcar-csi2: Implement set_routing
  media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2

 drivers/media/i2c/adv748x/adv748x-afe.c       |   2 +-
 drivers/media/i2c/adv748x/adv748x-core.c      |  12 +-
 drivers/media/i2c/adv748x/adv748x-csi2.c      | 251 +++++++---
 drivers/media/i2c/adv748x/adv748x-hdmi.c      |   2 +-
 drivers/media/i2c/adv748x/adv748x.h           |   4 +-
 drivers/media/i2c/max9286.c                   | 457 ++++++++++++------
 drivers/media/platform/renesas/rcar-csi2.c    | 442 ++++++++++++-----
 .../platform/renesas/rcar-vin/rcar-dma.c      |  16 +-
 8 files changed, 834 insertions(+), 352 deletions(-)

--
2.44.0

[PATCH 01/19] media: adv748x: Add support for active state

[Jacopo Mondi]

Initialize and use the subdev active state to store the subdevice
format.

This simplifies the implementation of the get_fmt and set_fmt pad
operations.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 69 ++++--------------------
 drivers/media/i2c/adv748x/adv748x.h      |  1 -
 2 files changed, 11 insertions(+), 59 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 5b265b722394..435b0909bbef 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -139,78 +139,26 @@ static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
  * But we must support setting the pad formats for format propagation.
  */
 
-static struct v4l2_mbus_framefmt *
-adv748x_csi2_get_pad_format(struct v4l2_subdev *sd,
-			    struct v4l2_subdev_state *sd_state,
-			    unsigned int pad, u32 which)
-{
-	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
-
-	if (which == V4L2_SUBDEV_FORMAT_TRY)
-		return v4l2_subdev_state_get_format(sd_state, pad);
-
-	return &tx->format;
-}
-
-static int adv748x_csi2_get_format(struct v4l2_subdev *sd,
-				   struct v4l2_subdev_state *sd_state,
-				   struct v4l2_subdev_format *sdformat)
-{
-	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
-	struct adv748x_state *state = tx->state;
-	struct v4l2_mbus_framefmt *mbusformat;
-
-	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
-						 sdformat->which);
-	if (!mbusformat)
-		return -EINVAL;
-
-	mutex_lock(&state->mutex);
-
-	sdformat->format = *mbusformat;
-
-	mutex_unlock(&state->mutex);
-
-	return 0;
-}
-
 static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_format *sdformat)
 {
-	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
-	struct adv748x_state *state = tx->state;
 	struct v4l2_mbus_framefmt *mbusformat;
-	int ret = 0;
-
-	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
-						 sdformat->which);
-	if (!mbusformat)
-		return -EINVAL;
 
-	mutex_lock(&state->mutex);
+	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
 
+	/* Format on the source pad is always copied from the sink one. */
 	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
 		const struct v4l2_mbus_framefmt *sink_fmt;
 
-		sink_fmt = adv748x_csi2_get_pad_format(sd, sd_state,
-						       ADV748X_CSI2_SINK,
-						       sdformat->which);
-
-		if (!sink_fmt) {
-			ret = -EINVAL;
-			goto unlock;
-		}
-
+		sink_fmt = v4l2_subdev_state_get_format(sd_state,
+							ADV748X_CSI2_SINK);
 		sdformat->format = *sink_fmt;
 	}
 
 	*mbusformat = sdformat->format;
 
-unlock:
-	mutex_unlock(&state->mutex);
-
-	return ret;
+	return 0;
 }
 
 static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
@@ -228,7 +176,7 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
 }
 
 static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
-	.get_fmt = adv748x_csi2_get_format,
+	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = adv748x_csi2_set_format,
 	.get_mbus_config = adv748x_csi2_get_mbus_config,
 };
@@ -320,6 +268,11 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
 	if (ret)
 		goto err_cleanup_subdev;
 
+	tx->sd.state_lock = tx->ctrl_hdl.lock;
+	ret = v4l2_subdev_init_finalize(&tx->sd);
+	if (ret)
+		goto err_free_ctrl;
+
 	ret = v4l2_async_register_subdev(&tx->sd);
 	if (ret)
 		goto err_free_ctrl;
diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
index d2b5e722e997..9bc0121d0eff 100644
--- a/drivers/media/i2c/adv748x/adv748x.h
+++ b/drivers/media/i2c/adv748x/adv748x.h
@@ -75,7 +75,6 @@ enum adv748x_csi2_pads {
 
 struct adv748x_csi2 {
 	struct adv748x_state *state;
-	struct v4l2_mbus_framefmt format;
 	unsigned int page;
 	unsigned int port;
 	unsigned int num_lanes;
-- 
2.44.0

[PATCH 02/19] media: adv748x: Add flags to adv748x_subdev_init()

[Jacopo Mondi]

Add a flags parameter to the adv748x_subdev_init() function that
allows to pass additional flags to the v4l2_subdevice.

This will be used to identify the CSI-2 subdevices as multiplexed.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-afe.c  | 2 +-
 drivers/media/i2c/adv748x/adv748x-core.c | 4 ++--
 drivers/media/i2c/adv748x/adv748x-csi2.c | 2 +-
 drivers/media/i2c/adv748x/adv748x-hdmi.c | 2 +-
 drivers/media/i2c/adv748x/adv748x.h      | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-afe.c b/drivers/media/i2c/adv748x/adv748x-afe.c
index 50d9fbadbe38..5951ddebc59d 100644
--- a/drivers/media/i2c/adv748x/adv748x-afe.c
+++ b/drivers/media/i2c/adv748x/adv748x-afe.c
@@ -510,7 +510,7 @@ int adv748x_afe_init(struct adv748x_afe *afe)
 	afe->curr_norm = V4L2_STD_NTSC_M;
 
 	adv748x_subdev_init(&afe->sd, state, &adv748x_afe_ops,
-			    MEDIA_ENT_F_ATV_DECODER, "afe");
+			    MEDIA_ENT_F_ATV_DECODER, 0, "afe");
 
 	/* Identify the first connector found as a default input if set */
 	for (i = ADV748X_PORT_AIN0; i <= ADV748X_PORT_AIN7; i++) {
diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
index 3eb6d5e8f082..3abc73ea8ccb 100644
--- a/drivers/media/i2c/adv748x/adv748x-core.c
+++ b/drivers/media/i2c/adv748x/adv748x-core.c
@@ -583,10 +583,10 @@ static int __maybe_unused adv748x_resume_early(struct device *dev)
 
 void adv748x_subdev_init(struct v4l2_subdev *sd, struct adv748x_state *state,
 			 const struct v4l2_subdev_ops *ops, u32 function,
-			 const char *ident)
+			 u32 flags, const char *ident)
 {
 	v4l2_subdev_init(sd, ops);
-	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | flags;
 
 	/* the owner is the same as the i2c_client's driver owner */
 	sd->owner = state->dev->driver->owner;
diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 435b0909bbef..60bf1dc0f58b 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -245,7 +245,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
 		return 0;
 
 	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
-			    MEDIA_ENT_F_VID_IF_BRIDGE,
+			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
 			    is_txa(tx) ? "txa" : "txb");
 
 	/* Register internal ops for incremental subdev registration */
diff --git a/drivers/media/i2c/adv748x/adv748x-hdmi.c b/drivers/media/i2c/adv748x/adv748x-hdmi.c
index a4db9bae5f79..9e4a5aa4ed55 100644
--- a/drivers/media/i2c/adv748x/adv748x-hdmi.c
+++ b/drivers/media/i2c/adv748x/adv748x-hdmi.c
@@ -741,7 +741,7 @@ int adv748x_hdmi_init(struct adv748x_hdmi *hdmi)
 	hdmi->aspect_ratio.denominator = 9;
 
 	adv748x_subdev_init(&hdmi->sd, state, &adv748x_ops_hdmi,
-			    MEDIA_ENT_F_IO_DTV, "hdmi");
+			    MEDIA_ENT_F_IO_DTV, 0, "hdmi");
 
 	hdmi->pads[ADV748X_HDMI_SINK].flags = MEDIA_PAD_FL_SINK;
 	hdmi->pads[ADV748X_HDMI_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
index 9bc0121d0eff..be24bc57767c 100644
--- a/drivers/media/i2c/adv748x/adv748x.h
+++ b/drivers/media/i2c/adv748x/adv748x.h
@@ -424,7 +424,7 @@ static inline struct v4l2_subdev *adv748x_get_remote_sd(struct media_pad *pad)
 
 void adv748x_subdev_init(struct v4l2_subdev *sd, struct adv748x_state *state,
 			 const struct v4l2_subdev_ops *ops, u32 function,
-			 const char *ident);
+			 u32 flags, const char *ident);
 
 int adv748x_tx_power(struct adv748x_csi2 *tx, bool on);
 
-- 
2.44.0

[PATCH 03/19] media: adv748x: Use V4L2 streams

[Jacopo Mondi]

Initialize the CSI-2 subdevice with the V4L2_SUBDEV_FL_STREAMS flag
and initialize a simple routing table by implementing the .init_state()
operation.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 28 ++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 60bf1dc0f58b..d929db7e8ef2 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -59,7 +59,30 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
 
 /* -----------------------------------------------------------------------------
  * v4l2_subdev_internal_ops
- *
+ */
+
+static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
+				   struct v4l2_subdev_state *state)
+{
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = ADV748X_CSI2_SINK,
+			.sink_stream = 0,
+			.source_pad = ADV748X_CSI2_SOURCE,
+			.source_stream = 0,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+
+	struct v4l2_subdev_krouting routing = {
+		.num_routes = ARRAY_SIZE(routes),
+		.routes = routes,
+	};
+
+	return v4l2_subdev_set_routing(sd, state, &routing);
+}
+
+/*
  * We use the internal registered operation to be able to ensure that our
  * incremental subdevices (not connected in the forward path) can be registered
  * against the resulting video path and media device.
@@ -109,6 +132,7 @@ static int adv748x_csi2_registered(struct v4l2_subdev *sd)
 }
 
 static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
+	.init_state = adv748x_csi2_init_state,
 	.registered = adv748x_csi2_registered,
 };
 
@@ -245,7 +269,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
 		return 0;
 
 	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
-			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
+			    MEDIA_ENT_F_VID_IF_BRIDGE, V4L2_SUBDEV_FL_STREAMS,
 			    is_txa(tx) ? "txa" : "txb");
 
 	/* Register internal ops for incremental subdev registration */
-- 
2.44.0

[PATCH 04/19] media: adv748x: Propagate format to opposite stream

[Jacopo Mondi]

Now that the adv748x-csi2 driver supports streams and routing, when a
format is set on the sink pad, it gets propagated to the connected stream
on the source pad.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index d929db7e8ef2..ace4e1d904d9 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -167,20 +167,21 @@ static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
 				   struct v4l2_subdev_state *sd_state,
 				   struct v4l2_subdev_format *sdformat)
 {
-	struct v4l2_mbus_framefmt *mbusformat;
+	struct v4l2_mbus_framefmt *fmt;
 
-	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
-
-	/* Format on the source pad is always copied from the sink one. */
-	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
-		const struct v4l2_mbus_framefmt *sink_fmt;
+	/*
+	 * The format set on the sink pad is propagated to the other end
+	 * of the active route.
+	 */
+	if (sdformat->pad == ADV748X_CSI2_SOURCE)
+		return -EINVAL;
 
-		sink_fmt = v4l2_subdev_state_get_format(sd_state,
-							ADV748X_CSI2_SINK);
-		sdformat->format = *sink_fmt;
-	}
+	fmt = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
+	*fmt = sdformat->format;
 
-	*mbusformat = sdformat->format;
+	fmt = v4l2_subdev_state_get_opposite_stream_format(sd_state,
+							   sdformat->pad, 0);
+	*fmt = sdformat->format;
 
 	return 0;
 }
-- 
2.44.0

[PATCH 05/19] media: adv748x: Implement set_routing()

[Jacopo Mondi]

Implement the set_routing() pad operation to control the MIPI CSI-2
Virtual Channel on which the video stream is sent on according to
the active route source stream number.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 43 +++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index ace4e1d904d9..7fa72340e66e 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -57,6 +57,38 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
 	return 0;
 }
 
+static int adv748x_csi2_apply_routing(struct v4l2_subdev *sd,
+				      struct v4l2_subdev_state *state,
+				      struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_route *route;
+	int ret;
+
+	/* Only one route at the time can be active. */
+	if (routing->num_routes > 1)
+		return -EINVAL;
+
+	/*
+	 * Validate the route: sink pad and sink stream shall be 0 and only
+	 * 4 source streams are supported (one for each supported MIPI CSI-2
+	 * channel).
+	 */
+	route = &routing->routes[0];
+
+	if (route->sink_pad != ADV748X_CSI2_SINK || route->sink_stream)
+		return -EINVAL;
+	if (route->source_pad != ADV748X_CSI2_SOURCE ||
+	    route->source_stream > 4)
+		return -EINVAL;
+
+	ret = v4l2_subdev_routing_validate(sd, routing,
+					   V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
+	if (ret)
+		return ret;
+
+	return v4l2_subdev_set_routing(sd, state, routing);
+}
+
 /* -----------------------------------------------------------------------------
  * v4l2_subdev_internal_ops
  */
@@ -79,7 +111,7 @@ static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
 		.routes = routes,
 	};
 
-	return v4l2_subdev_set_routing(sd, state, &routing);
+	return adv748x_csi2_apply_routing(sd, state, &routing);
 }
 
 /*
@@ -200,10 +232,19 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
 	return 0;
 }
 
+static int adv748x_csi2_set_routing(struct v4l2_subdev *sd,
+				    struct v4l2_subdev_state *state,
+				    enum v4l2_subdev_format_whence which,
+				    struct v4l2_subdev_krouting *routing)
+{
+	return adv748x_csi2_apply_routing(sd, state, routing);
+}
+
 static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
 	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = adv748x_csi2_set_format,
 	.get_mbus_config = adv748x_csi2_get_mbus_config,
+	.set_routing = adv748x_csi2_set_routing,
 };
 
 /* -----------------------------------------------------------------------------
-- 
2.44.0

[PATCH 06/19] media: adv748x: Use routes to configure VC

[Jacopo Mondi]

Use the newly introduced routing table to configure on which MIPI
CSI-2 Virtual Channel to send the image stream on.

Perform Virtual Channel selection at s_stream() time instead of
forcing it to 0 during the chip reset.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-core.c |  8 ++------
 drivers/media/i2c/adv748x/adv748x-csi2.c | 22 ++++++++++++++++++++--
 drivers/media/i2c/adv748x/adv748x.h      |  1 -
 3 files changed, 22 insertions(+), 9 deletions(-)

diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
index 3abc73ea8ccb..c9d917135709 100644
--- a/drivers/media/i2c/adv748x/adv748x-core.c
+++ b/drivers/media/i2c/adv748x/adv748x-core.c
@@ -530,14 +530,10 @@ static int adv748x_reset(struct adv748x_state *state)
 	io_write(state, ADV748X_IO_PD, ADV748X_IO_PD_RX_EN);
 
 	/* Conditionally enable TXa and TXb. */
-	if (is_tx_enabled(&state->txa)) {
+	if (is_tx_enabled(&state->txa))
 		regval |= ADV748X_IO_10_CSI4_EN;
-		adv748x_csi2_set_virtual_channel(&state->txa, 0);
-	}
-	if (is_tx_enabled(&state->txb)) {
+	if (is_tx_enabled(&state->txb))
 		regval |= ADV748X_IO_10_CSI1_EN;
-		adv748x_csi2_set_virtual_channel(&state->txb, 0);
-	}
 	io_write(state, ADV748X_IO_10, regval);
 
 	/* Use vid_std and v_freq as freerun resolution for CP */
diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index 7fa72340e66e..a7bfed393ff0 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -14,7 +14,8 @@
 
 #include "adv748x.h"
 
-int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc)
+static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
+					    unsigned int vc)
 {
 	return tx_write(tx, ADV748X_CSI_VC_REF, vc << ADV748X_CSI_VC_REF_SHIFT);
 }
@@ -175,13 +176,30 @@ static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
 static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, int enable)
 {
 	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
+	struct v4l2_subdev_state *state;
 	struct v4l2_subdev *src;
+	int ret;
 
 	src = adv748x_get_remote_sd(&tx->pads[ADV748X_CSI2_SINK]);
 	if (!src)
 		return -EPIPE;
 
-	return v4l2_subdev_call(src, video, s_stream, enable);
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	if (enable) {
+		/* A single route is available. */
+		struct v4l2_subdev_route *route = &state->routing.routes[0];
+
+		ret = adv748x_csi2_set_virtual_channel(tx, route->source_stream);
+		if (ret)
+			goto unlock;
+	}
+
+	ret = v4l2_subdev_call(src, video, s_stream, enable);
+unlock:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
 }
 
 static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
index be24bc57767c..95d04468af9d 100644
--- a/drivers/media/i2c/adv748x/adv748x.h
+++ b/drivers/media/i2c/adv748x/adv748x.h
@@ -434,7 +434,6 @@ int adv748x_afe_s_input(struct adv748x_afe *afe, unsigned int input);
 
 int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx);
 void adv748x_csi2_cleanup(struct adv748x_csi2 *tx);
-int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc);
 int adv748x_csi2_set_pixelrate(struct v4l2_subdev *sd, s64 rate);
 
 int adv748x_hdmi_init(struct adv748x_hdmi *hdmi);
-- 
2.44.0

[PATCH 07/19] media: adv748x: Implement .get_frame_desc()

[Jacopo Mondi]

Implement the get_frame_desc subdev pad operation.

Implement the get_frame_desc pad operation to allow retrieving the
stream configuration of the adv748x csi2 subdevice.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/adv748x/adv748x-csi2.c | 86 ++++++++++++++++++++++++
 1 file changed, 86 insertions(+)

diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
index a7bfed393ff0..497586aff6b2 100644
--- a/drivers/media/i2c/adv748x/adv748x-csi2.c
+++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
@@ -8,12 +8,51 @@
 #include <linux/module.h>
 #include <linux/mutex.h>
 
+#include <media/mipi-csi2.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-ioctl.h>
 
 #include "adv748x.h"
 
+struct adv748x_csi2_format_info {
+	u8 dt;
+	u8 bpp;
+};
+
+static int adv748x_csi2_get_format_info(struct adv748x_csi2 *tx, u32 mbus_code,
+					struct adv748x_csi2_format_info *fmt)
+{
+	switch (mbus_code) {
+	case MEDIA_BUS_FMT_YUYV8_1X16:
+		fmt->dt = MIPI_CSI2_DT_YUV422_8B;
+		fmt->bpp = 16;
+		break;
+	case MEDIA_BUS_FMT_YUYV10_1X20:
+		fmt->dt = MIPI_CSI2_DT_YUV422_10B;
+		fmt->bpp = 20;
+		break;
+	case MEDIA_BUS_FMT_RGB565_1X16:
+		fmt->dt = MIPI_CSI2_DT_RGB565;
+		fmt->bpp = 16;
+		break;
+	case MEDIA_BUS_FMT_RGB666_1X18:
+		fmt->dt = MIPI_CSI2_DT_RGB666;
+		fmt->bpp = 18;
+		break;
+	case MEDIA_BUS_FMT_RGB888_1X24:
+		fmt->dt = MIPI_CSI2_DT_RGB888;
+		fmt->bpp = 24;
+		break;
+	default:
+		dev_dbg(tx->state->dev,
+			"Unsupported media bus code: %u\n", mbus_code);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
 					    unsigned int vc)
 {
@@ -258,11 +297,58 @@ static int adv748x_csi2_set_routing(struct v4l2_subdev *sd,
 	return adv748x_csi2_apply_routing(sd, state, routing);
 }
 
+static int adv748x_csi2_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				       struct v4l2_mbus_frame_desc *fd)
+{
+	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
+	struct adv748x_csi2_format_info info = {};
+	struct v4l2_mbus_frame_desc_entry *entry;
+	struct v4l2_subdev_route *route;
+	struct v4l2_subdev_state *state;
+	struct v4l2_mbus_framefmt *fmt;
+	int ret = -EINVAL;
+
+	if (pad != ADV748X_CSI2_SOURCE)
+		return -EINVAL;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/* A single route is available. */
+	route = &state->routing.routes[0];
+	fmt = v4l2_subdev_state_get_format(state, pad, route->source_stream);
+	if (!fmt)
+		goto err_unlock;
+
+	ret = adv748x_csi2_get_format_info(tx, fmt->code, &info);
+	if (ret)
+		goto err_unlock;
+
+	memset(fd, 0, sizeof(*fd));
+
+	/* A single stream is available. */
+	fd->num_entries = 1;
+	fd->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+	entry = &fd->entry[0];
+	entry->stream = 0;
+	entry->flags = V4L2_MBUS_FRAME_DESC_FL_LEN_MAX;
+	entry->length = fmt->width * fmt->height * info.bpp / 8;
+	entry->pixelcode = fmt->code;
+	entry->bus.csi2.vc = route->source_stream;
+	entry->bus.csi2.dt = info.dt;
+
+err_unlock:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+
 static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
 	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = adv748x_csi2_set_format,
 	.get_mbus_config = adv748x_csi2_get_mbus_config,
 	.set_routing = adv748x_csi2_set_routing,
+	.get_frame_desc = adv748x_csi2_get_frame_desc,
 };
 
 /* -----------------------------------------------------------------------------
-- 
2.44.0

[PATCH 08/19] media: max9286: Add support for subdev active state

[Jacopo Mondi]

Use the subdev active state in the max9286 driver to store the
image format.

Replace the .open() function call with the .init_state() one and
simplify the set/get_pad_fmt() operations.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 110 ++++++++++++------------------------
 1 file changed, 37 insertions(+), 73 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index d685d445cf23..fb13bfde42df 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -19,7 +19,6 @@
 #include <linux/i2c.h>
 #include <linux/i2c-mux.h>
 #include <linux/module.h>
-#include <linux/mutex.h>
 #include <linux/of_graph.h>
 #include <linux/regulator/consumer.h>
 #include <linux/slab.h>
@@ -198,12 +197,8 @@ struct max9286_priv {
 	struct v4l2_ctrl *pixelrate_ctrl;
 	unsigned int pixelrate;
 
-	struct v4l2_mbus_framefmt fmt[MAX9286_N_SINKS];
 	struct v4l2_fract interval;
 
-	/* Protects controls and fmt structures */
-	struct mutex mutex;
-
 	unsigned int nsources;
 	unsigned int source_mask;
 	unsigned int route_mask;
@@ -788,19 +783,23 @@ static void max9286_v4l2_notifier_unregister(struct max9286_priv *priv)
 static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 {
 	struct max9286_priv *priv = sd_to_max9286(sd);
+	struct v4l2_subdev_state *state;
 	struct max9286_source *source;
 	unsigned int i;
 	bool sync = false;
-	int ret;
+	int ret = 0;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
 
 	if (enable) {
 		const struct v4l2_mbus_framefmt *format;
+		unsigned int source_idx = __ffs(priv->bound_sources);
 
 		/*
 		 * Get the format from the first used sink pad, as all sink
 		 * formats must be identical.
 		 */
-		format = &priv->fmt[__ffs(priv->bound_sources)];
+		format = v4l2_subdev_state_get_format(state, source_idx);
 
 		max9286_set_video_format(priv, format);
 		max9286_set_fsync_period(priv);
@@ -816,12 +815,12 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 		for_each_source(priv, source) {
 			ret = v4l2_subdev_call(source->sd, video, s_stream, 1);
 			if (ret)
-				return ret;
+				goto err_unlock;
 		}
 
 		ret = max9286_check_video_links(priv);
 		if (ret)
-			return ret;
+			goto err_unlock;
 
 		/*
 		 * Wait until frame synchronization is locked.
@@ -842,7 +841,8 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 		if (!sync) {
 			dev_err(&priv->client->dev,
 				"Failed to get frame synchronization\n");
-			return -EXDEV; /* Invalid cross-device link */
+			ret = -EXDEV; /* Invalid cross-device link */
+			goto err_unlock;
 		}
 
 		/*
@@ -865,7 +865,10 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 		max9286_i2c_mux_close(priv);
 	}
 
-	return 0;
+err_unlock:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
 }
 
 static int max9286_get_frame_interval(struct v4l2_subdev *sd,
@@ -922,29 +925,16 @@ static int max9286_enum_mbus_code(struct v4l2_subdev *sd,
 	return 0;
 }
 
-static struct v4l2_mbus_framefmt *
-max9286_get_pad_format(struct max9286_priv *priv,
-		       struct v4l2_subdev_state *sd_state,
-		       unsigned int pad, u32 which)
-{
-	switch (which) {
-	case V4L2_SUBDEV_FORMAT_TRY:
-		return v4l2_subdev_state_get_format(sd_state, pad);
-	case V4L2_SUBDEV_FORMAT_ACTIVE:
-		return &priv->fmt[pad];
-	default:
-		return NULL;
-	}
-}
-
 static int max9286_set_fmt(struct v4l2_subdev *sd,
-			   struct v4l2_subdev_state *sd_state,
+			   struct v4l2_subdev_state *state,
 			   struct v4l2_subdev_format *format)
 {
-	struct max9286_priv *priv = sd_to_max9286(sd);
-	struct v4l2_mbus_framefmt *cfg_fmt;
 	unsigned int i;
 
+	/*
+	 * Disable setting format on the source pad: format is propagated
+	 * from the sinks.
+	 */
 	if (format->pad == MAX9286_SRC_PAD)
 		return -EINVAL;
 
@@ -957,42 +947,13 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 	if (i == ARRAY_SIZE(max9286_formats))
 		format->format.code = max9286_formats[0].code;
 
-	cfg_fmt = max9286_get_pad_format(priv, sd_state, format->pad,
-					 format->which);
-	if (!cfg_fmt)
-		return -EINVAL;
-
-	mutex_lock(&priv->mutex);
-	*cfg_fmt = format->format;
-	mutex_unlock(&priv->mutex);
-
-	return 0;
-}
-
-static int max9286_get_fmt(struct v4l2_subdev *sd,
-			   struct v4l2_subdev_state *sd_state,
-			   struct v4l2_subdev_format *format)
-{
-	struct max9286_priv *priv = sd_to_max9286(sd);
-	struct v4l2_mbus_framefmt *cfg_fmt;
-	unsigned int pad = format->pad;
+	*v4l2_subdev_state_get_format(state, format->pad) = format->format;
 
 	/*
-	 * Multiplexed Stream Support: Support link validation by returning the
-	 * format of the first bound link. All links must have the same format,
-	 * as we do not support mixing and matching of cameras connected to the
-	 * max9286.
+	 * Apply the same format on the source pad: all links must have the
+	 * same format.
 	 */
-	if (pad == MAX9286_SRC_PAD)
-		pad = __ffs(priv->bound_sources);
-
-	cfg_fmt = max9286_get_pad_format(priv, sd_state, pad, format->which);
-	if (!cfg_fmt)
-		return -EINVAL;
-
-	mutex_lock(&priv->mutex);
-	format->format = *cfg_fmt;
-	mutex_unlock(&priv->mutex);
+	*v4l2_subdev_state_get_format(state, MAX9286_SRC_PAD) = format->format;
 
 	return 0;
 }
@@ -1003,7 +964,7 @@ static const struct v4l2_subdev_video_ops max9286_video_ops = {
 
 static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
 	.enum_mbus_code = max9286_enum_mbus_code,
-	.get_fmt	= max9286_get_fmt,
+	.get_fmt	= v4l2_subdev_get_fmt,
 	.set_fmt	= max9286_set_fmt,
 	.get_frame_interval = max9286_get_frame_interval,
 	.set_frame_interval = max9286_set_frame_interval,
@@ -1030,13 +991,14 @@ static void max9286_init_format(struct v4l2_mbus_framefmt *fmt)
 	*fmt = max9286_default_format;
 }
 
-static int max9286_open(struct v4l2_subdev *subdev, struct v4l2_subdev_fh *fh)
+static int max9286_init_state(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *state)
 {
 	struct v4l2_mbus_framefmt *format;
 	unsigned int i;
 
 	for (i = 0; i < MAX9286_N_SINKS; i++) {
-		format = v4l2_subdev_state_get_format(fh->state, i);
+		format = v4l2_subdev_state_get_format(state, i);
 		max9286_init_format(format);
 	}
 
@@ -1044,7 +1006,7 @@ static int max9286_open(struct v4l2_subdev *subdev, struct v4l2_subdev_fh *fh)
 }
 
 static const struct v4l2_subdev_internal_ops max9286_subdev_internal_ops = {
-	.open = max9286_open,
+	.init_state = max9286_init_state,
 };
 
 static const struct media_entity_operations max9286_media_ops = {
@@ -1079,10 +1041,6 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
 	}
 
 	/* Configure V4L2 for the MAX9286 itself */
-
-	for (i = 0; i < MAX9286_N_SINKS; i++)
-		max9286_init_format(&priv->fmt[i]);
-
 	v4l2_i2c_subdev_init(&priv->sd, priv->client, &max9286_subdev_ops);
 	priv->sd.internal_ops = &max9286_subdev_internal_ops;
 	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
@@ -1109,14 +1067,21 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
 	if (ret)
 		goto err_async;
 
+	priv->sd.state_lock = priv->ctrls.lock;
+	ret = v4l2_subdev_init_finalize(&priv->sd);
+	if (ret)
+		goto err_async;
+
 	ret = v4l2_async_register_subdev(&priv->sd);
 	if (ret < 0) {
 		dev_err(dev, "Unable to register subdevice\n");
-		goto err_async;
+		goto err_subdev;
 	}
 
 	return 0;
 
+err_subdev:
+	v4l2_subdev_cleanup(&priv->sd);
 err_async:
 	v4l2_ctrl_handler_free(&priv->ctrls);
 	max9286_v4l2_notifier_unregister(priv);
@@ -1126,6 +1091,7 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
 
 static void max9286_v4l2_unregister(struct max9286_priv *priv)
 {
+	v4l2_subdev_cleanup(&priv->sd);
 	v4l2_ctrl_handler_free(&priv->ctrls);
 	v4l2_async_unregister_subdev(&priv->sd);
 	max9286_v4l2_notifier_unregister(priv);
@@ -1629,8 +1595,6 @@ static int max9286_probe(struct i2c_client *client)
 	if (!priv)
 		return -ENOMEM;
 
-	mutex_init(&priv->mutex);
-
 	priv->client = client;
 
 	/* GPIO values default to high */
-- 
2.44.0

[PATCH 09/19] media: max9286: Fix enum_mbus_code

[Jacopo Mondi]

The max9286 driver supports multiple output formats but only a single
one is reported theough the .enum_mbus_code operation.

Fix that.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index fb13bfde42df..575d8ac8efa1 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -917,10 +917,10 @@ static int max9286_enum_mbus_code(struct v4l2_subdev *sd,
 				  struct v4l2_subdev_state *sd_state,
 				  struct v4l2_subdev_mbus_code_enum *code)
 {
-	if (code->pad || code->index > 0)
+	if (code->pad || code->index >= ARRAY_SIZE(max9286_formats))
 		return -EINVAL;
 
-	code->code = MEDIA_BUS_FMT_UYVY8_1X16;
+	code->code = max9286_formats[code->index].code;
 
 	return 0;
 }
-- 
2.44.0

[PATCH 10/19] media: max9286: Use frame interval from subdev state

[Jacopo Mondi]

Use the frame interval stored in the subdev state instead of storing
a copy in the driver private structure.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 52 ++++++++-----------------------------
 1 file changed, 11 insertions(+), 41 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index 575d8ac8efa1..ae1b73fde832 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -197,8 +197,6 @@ struct max9286_priv {
 	struct v4l2_ctrl *pixelrate_ctrl;
 	unsigned int pixelrate;
 
-	struct v4l2_fract interval;
-
 	unsigned int nsources;
 	unsigned int source_mask;
 	unsigned int route_mask;
@@ -571,11 +569,14 @@ static void max9286_set_video_format(struct max9286_priv *priv,
 		      MAX9286_INVVS | MAX9286_HVSRC_D14);
 }
 
-static void max9286_set_fsync_period(struct max9286_priv *priv)
+static void max9286_set_fsync_period(struct max9286_priv *priv,
+				     struct v4l2_subdev_state *state)
 {
+	struct v4l2_fract *interval;
 	u32 fsync;
 
-	if (!priv->interval.numerator || !priv->interval.denominator) {
+	interval = v4l2_subdev_state_get_interval(state, MAX9286_SRC_PAD);
+	if (!interval->numerator || !interval->denominator) {
 		/*
 		 * Special case, a null interval enables automatic FRAMESYNC
 		 * mode. FRAMESYNC is taken from the slowest link.
@@ -591,8 +592,8 @@ static void max9286_set_fsync_period(struct max9286_priv *priv)
 	 * The FRAMESYNC generator is configured with a period expressed as a
 	 * number of PCLK periods.
 	 */
-	fsync = div_u64((u64)priv->pixelrate * priv->interval.numerator,
-			priv->interval.denominator);
+	fsync = div_u64((u64)priv->pixelrate * interval->numerator,
+			interval->denominator);
 
 	dev_dbg(&priv->client->dev, "fsync period %u (pclk %u)\n", fsync,
 		priv->pixelrate);
@@ -802,7 +803,7 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 		format = v4l2_subdev_state_get_format(state, source_idx);
 
 		max9286_set_video_format(priv, format);
-		max9286_set_fsync_period(priv);
+		max9286_set_fsync_period(priv, state);
 
 		/*
 		 * The frame sync between cameras is transmitted across the
@@ -871,44 +872,14 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 	return ret;
 }
 
-static int max9286_get_frame_interval(struct v4l2_subdev *sd,
-				      struct v4l2_subdev_state *sd_state,
-				      struct v4l2_subdev_frame_interval *interval)
-{
-	struct max9286_priv *priv = sd_to_max9286(sd);
-
-	/*
-	 * FIXME: Implement support for V4L2_SUBDEV_FORMAT_TRY, using the V4L2
-	 * subdev active state API.
-	 */
-	if (interval->which != V4L2_SUBDEV_FORMAT_ACTIVE)
-		return -EINVAL;
-
-	if (interval->pad != MAX9286_SRC_PAD)
-		return -EINVAL;
-
-	interval->interval = priv->interval;
-
-	return 0;
-}
-
 static int max9286_set_frame_interval(struct v4l2_subdev *sd,
-				      struct v4l2_subdev_state *sd_state,
+				      struct v4l2_subdev_state *state,
 				      struct v4l2_subdev_frame_interval *interval)
 {
-	struct max9286_priv *priv = sd_to_max9286(sd);
-
-	/*
-	 * FIXME: Implement support for V4L2_SUBDEV_FORMAT_TRY, using the V4L2
-	 * subdev active state API.
-	 */
-	if (interval->which != V4L2_SUBDEV_FORMAT_ACTIVE)
-		return -EINVAL;
-
 	if (interval->pad != MAX9286_SRC_PAD)
 		return -EINVAL;
 
-	priv->interval = interval->interval;
+	*v4l2_subdev_state_get_interval(state, interval->pad) = interval->interval;
 
 	return 0;
 }
@@ -966,7 +937,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
 	.enum_mbus_code = max9286_enum_mbus_code,
 	.get_fmt	= v4l2_subdev_get_fmt,
 	.set_fmt	= max9286_set_fmt,
-	.get_frame_interval = max9286_get_frame_interval,
+	.get_frame_interval = v4l2_subdev_get_frame_interval,
 	.set_frame_interval = max9286_set_frame_interval,
 };
 
@@ -1148,7 +1119,6 @@ static int max9286_setup(struct max9286_priv *priv)
 	max9286_write(priv, 0x69, (0xf & ~priv->route_mask));
 
 	max9286_set_video_format(priv, &max9286_default_format);
-	max9286_set_fsync_period(priv);
 
 	cfg = max9286_read(priv, 0x1c);
 	if (cfg < 0)
-- 
2.44.0

[PATCH 11/19] media: max9286: Use V4L2 Streams

[Jacopo Mondi]

Use V4L2 streams by introducing a static route table that reports
how the enabled source links are routed through the CSI-2 source pad.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 43 +++++++++++++++++++++++--------------
 1 file changed, 27 insertions(+), 16 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index ae1b73fde832..f203e4527257 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -800,7 +800,7 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
 		 * Get the format from the first used sink pad, as all sink
 		 * formats must be identical.
 		 */
-		format = v4l2_subdev_state_get_format(state, source_idx);
+		format = v4l2_subdev_state_get_format(state, source_idx, 0);
 
 		max9286_set_video_format(priv, format);
 		max9286_set_fsync_period(priv, state);
@@ -918,13 +918,14 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 	if (i == ARRAY_SIZE(max9286_formats))
 		format->format.code = max9286_formats[0].code;
 
-	*v4l2_subdev_state_get_format(state, format->pad) = format->format;
+	*v4l2_subdev_state_get_format(state, format->pad, 0) = format->format;
 
 	/*
-	 * Apply the same format on the source pad: all links must have the
+	 * Apply the same format on the opposite stream: all links must have the
 	 * same format.
 	 */
-	*v4l2_subdev_state_get_format(state, MAX9286_SRC_PAD) = format->format;
+	*v4l2_subdev_state_get_opposite_stream_format(state, format->pad, 0) =
+		format->format;
 
 	return 0;
 }
@@ -957,23 +958,32 @@ static const struct v4l2_mbus_framefmt max9286_default_format = {
 	.xfer_func	= V4L2_XFER_FUNC_DEFAULT,
 };
 
-static void max9286_init_format(struct v4l2_mbus_framefmt *fmt)
-{
-	*fmt = max9286_default_format;
-}
-
 static int max9286_init_state(struct v4l2_subdev *sd,
 			      struct v4l2_subdev_state *state)
 {
-	struct v4l2_mbus_framefmt *format;
-	unsigned int i;
+	struct v4l2_subdev_route routes[MAX9286_N_SINKS];
+	struct max9286_priv *priv = sd_to_max9286(sd);
+	struct max9286_source *source;
+	unsigned int num_routes = 0;
 
-	for (i = 0; i < MAX9286_N_SINKS; i++) {
-		format = v4l2_subdev_state_get_format(state, i);
-		max9286_init_format(format);
+	for_each_source(priv, source) {
+		struct v4l2_subdev_route *route = &routes[num_routes++];
+		unsigned int index = to_index(priv, source);
+
+		route->sink_pad = index;
+		route->sink_stream = 0;
+		route->source_pad = MAX9286_SRC_PAD;
+		route->source_stream = index;
+		route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
 	}
 
-	return 0;
+	struct v4l2_subdev_krouting routing = {
+		.num_routes = num_routes,
+		.routes = routes,
+	};
+
+	return v4l2_subdev_set_routing_with_fmt(sd, state, &routing,
+						&max9286_default_format);
 }
 
 static const struct v4l2_subdev_internal_ops max9286_subdev_internal_ops = {
@@ -1014,7 +1024,8 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
 	/* Configure V4L2 for the MAX9286 itself */
 	v4l2_i2c_subdev_init(&priv->sd, priv->client, &max9286_subdev_ops);
 	priv->sd.internal_ops = &max9286_subdev_internal_ops;
-	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
+			  V4L2_SUBDEV_FL_STREAMS;
 
 	v4l2_ctrl_handler_init(&priv->ctrls, 1);
 	priv->pixelrate_ctrl = v4l2_ctrl_new_std(&priv->ctrls,
-- 
2.44.0

[PATCH 12/19] media: max9286: Implement .get_frame_desc()

[Jacopo Mondi]

Implement the .get_frame_desc() pad operation to allow the receiver
to retrieve information on the multiplexed source pad.

Record in the max9286_format_info structure the MIPI CSI-2
data type and use it to populate the frame_desc_entry.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 120 ++++++++++++++++++++++++++++--------
 1 file changed, 95 insertions(+), 25 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index f203e4527257..4b4f4c03c10a 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -23,6 +23,7 @@
 #include <linux/regulator/consumer.h>
 #include <linux/slab.h>
 
+#include <media/mipi-csi2.h>
 #include <media/v4l2-async.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -145,7 +146,12 @@
 
 struct max9286_format_info {
 	u32 code;
-	u8 datatype;
+	/* The gmsl data format configuration. */
+	u8 gmsl_dt;
+	/* The format bpp, used for stride calculation. */
+	u8 bpp;
+	/* The Data Type identifier as defined by the MIPI CSI-2 standard. */
+	u8 mipi_dt;
 };
 
 struct max9286_i2c_speed {
@@ -235,28 +241,44 @@ static inline struct max9286_priv *sd_to_max9286(struct v4l2_subdev *sd)
 static const struct max9286_format_info max9286_formats[] = {
 	{
 		.code = MEDIA_BUS_FMT_UYVY8_1X16,
-		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.bpp = 16,
+		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_VYUY8_1X16,
-		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.bpp = 16,
+		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_YUYV8_1X16,
-		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.bpp = 16,
+		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_YVYU8_1X16,
-		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.bpp = 16,
+		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_SBGGR12_1X12,
-		.datatype = MAX9286_DATATYPE_RAW12,
+		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.bpp = 12,
+		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SGBRG12_1X12,
-		.datatype = MAX9286_DATATYPE_RAW12,
+		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.bpp = 12,
+		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SGRBG12_1X12,
-		.datatype = MAX9286_DATATYPE_RAW12,
+		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.bpp = 12,
+		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
-		.datatype = MAX9286_DATATYPE_RAW12,
+		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.bpp = 12,
+		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	},
 };
 
@@ -532,19 +554,23 @@ static int max9286_check_config_link(struct max9286_priv *priv,
 	return 0;
 }
 
+static const struct max9286_format_info *
+max9286_get_format_info(unsigned int code)
+{
+	for (unsigned int i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
+		if (max9286_formats[i].code == code)
+			return &max9286_formats[i];
+	}
+
+	return NULL;
+}
+
 static void max9286_set_video_format(struct max9286_priv *priv,
 				     const struct v4l2_mbus_framefmt *format)
 {
 	const struct max9286_format_info *info = NULL;
-	unsigned int i;
-
-	for (i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
-		if (max9286_formats[i].code == format->code) {
-			info = &max9286_formats[i];
-			break;
-		}
-	}
 
+	info = max9286_get_format_info(format->code);
 	if (WARN_ON(!info))
 		return;
 
@@ -559,7 +585,7 @@ static void max9286_set_video_format(struct max9286_priv *priv,
 	/* Enable CSI-2 Lane D0-D3 only, DBL mode. */
 	max9286_write(priv, 0x12, MAX9286_CSIDBL | MAX9286_DBL |
 		      MAX9286_CSILANECNT(priv->csi2_data_lanes) |
-		      info->datatype);
+		      info->gmsl_dt);
 
 	/*
 	 * Enable HS/VS encoding, use HS as line valid source, use D14/15 for
@@ -900,7 +926,7 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 			   struct v4l2_subdev_state *state,
 			   struct v4l2_subdev_format *format)
 {
-	unsigned int i;
+	const struct max9286_format_info *info;
 
 	/*
 	 * Disable setting format on the source pad: format is propagated
@@ -910,12 +936,8 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	/* Validate the format. */
-	for (i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
-		if (max9286_formats[i].code == format->format.code)
-			break;
-	}
-
-	if (i == ARRAY_SIZE(max9286_formats))
+	info = max9286_get_format_info(format->format.code);
+	if (!info)
 		format->format.code = max9286_formats[0].code;
 
 	*v4l2_subdev_state_get_format(state, format->pad, 0) = format->format;
@@ -930,6 +952,53 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				  struct v4l2_mbus_frame_desc *fd)
+{
+	struct v4l2_subdev_route *route;
+	struct v4l2_subdev_state *state;
+	unsigned int num_routes = 0;
+	int ret = 0;
+
+	if (pad != MAX9286_SRC_PAD)
+		return -EINVAL;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+	for_each_active_route(&state->routing, route) {
+		struct v4l2_mbus_frame_desc_entry *entry;
+		const struct max9286_format_info *info;
+		struct v4l2_mbus_framefmt *fmt;
+
+		fmt = v4l2_subdev_state_get_format(state, route->sink_pad,
+						   route->sink_stream);
+		info = max9286_get_format_info(fmt->code);
+		if (WARN_ON(!info)) {
+			ret = -EINVAL;
+			goto err_unlock;
+		}
+
+		entry = &fd->entry[num_routes];
+		entry->stream = num_routes;
+		entry->flags = V4L2_MBUS_FRAME_DESC_FL_LEN_MAX;
+		entry->length = fmt->width * fmt->height * info->bpp / 8;
+		entry->pixelcode = fmt->code;
+
+		/* VC is set according to link ordering, see register 0x15. */
+		entry->bus.csi2.vc = route->sink_pad;
+		entry->bus.csi2.dt = info->mipi_dt;
+
+		num_routes++;
+	}
+
+	fd->num_entries = num_routes;
+	fd->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+err_unlock:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+
 static const struct v4l2_subdev_video_ops max9286_video_ops = {
 	.s_stream	= max9286_s_stream,
 };
@@ -940,6 +1009,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
 	.set_fmt	= max9286_set_fmt,
 	.get_frame_interval = v4l2_subdev_get_frame_interval,
 	.set_frame_interval = max9286_set_frame_interval,
+	.get_frame_desc	= max9286_get_frame_desc,
 };
 
 static const struct v4l2_subdev_ops max9286_subdev_ops = {
-- 
2.44.0

[PATCH 13/19] media: max9286: Implement support for LINK_FREQ

[Jacopo Mondi]

The max9286 now supports multiple streams. For this reason, reporting
the data bitrate through the PIXEL_RATE control is not meaningful
anymore (or better, is less meaningful that what it already was).

In order to support multiplexed receivers, which want to be informed
about the CSI-2 link frequency, implement support for the
V4L2_CID_LINK_FREQ control.

Record in the driver a set of pre-computed link frequencies in order to
support the two currently supported GMSL camera modules (rdacm20 and
rdacm21) and all the supported number of active channels and CSI-2 data
lanes.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 143 ++++++++++++++++++++++++++++++++++--
 1 file changed, 136 insertions(+), 7 deletions(-)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index 4b4f4c03c10a..0e08a1041eed 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -148,6 +148,12 @@ struct max9286_format_info {
 	u32 code;
 	/* The gmsl data format configuration. */
 	u8 gmsl_dt;
+	/*
+	 * gmsl_bpp is the pixel format bit-width per word as reported in
+	 * table 20. and it is used to calculate the GMSL bandwidth and CSI-2
+	 * link speed.
+	 */
+	u8 gmsl_bpp;
 	/* The format bpp, used for stride calculation. */
 	u8 bpp;
 	/* The Data Type identifier as defined by the MIPI CSI-2 standard. */
@@ -201,6 +207,7 @@ struct max9286_priv {
 
 	struct v4l2_ctrl_handler ctrls;
 	struct v4l2_ctrl *pixelrate_ctrl;
+	struct v4l2_ctrl *link_freq;
 	unsigned int pixelrate;
 
 	unsigned int nsources;
@@ -242,41 +249,49 @@ static const struct max9286_format_info max9286_formats[] = {
 	{
 		.code = MEDIA_BUS_FMT_UYVY8_1X16,
 		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_bpp = 8,
 		.bpp = 16,
 		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_VYUY8_1X16,
 		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_bpp = 8,
 		.bpp = 16,
 		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_YUYV8_1X16,
 		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_bpp = 8,
 		.bpp = 16,
 		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_YVYU8_1X16,
 		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
+		.gmsl_bpp = 8,
 		.bpp = 16,
 		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
 	}, {
 		.code = MEDIA_BUS_FMT_SBGGR12_1X12,
 		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.gmsl_bpp = 12,
 		.bpp = 12,
 		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SGBRG12_1X12,
 		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.gmsl_bpp = 12,
 		.bpp = 12,
 		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SGRBG12_1X12,
 		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.gmsl_bpp = 12,
 		.bpp = 12,
 		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	}, {
 		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
 		.gmsl_dt = MAX9286_DATATYPE_RAW12,
+		.gmsl_bpp = 12,
 		.bpp = 12,
 		.mipi_dt = MIPI_CSI2_DT_RAW12,
 	},
@@ -293,6 +308,60 @@ static const struct max9286_i2c_speed max9286_i2c_speeds[] = {
 	{ .rate = 837000, .mstbt = MAX9286_I2CMSTBT_837KBPS },
 };
 
+/*
+ * According to the chip manual the CSI-2 link frequency is calculated as:
+ *
+ *     f-CSI2 = pclk * ((width * channels) / lanes)
+ *
+ * Where:
+ *     pclk = serializer input word rate
+ *     width = pixel format bits per word (gmsl_bpp in max9286_format_info)
+ *     channels = number of enabled links
+ *     lanes = number of CSI-2 data lanes
+ *
+ * The following link frequencies have been computed by combining the following
+ * parameters:
+ *
+ * pixel_rates[] = { 44000000, 55000000 }; for RDACM20 and RDACM21
+ * bpps[] = { 8, 12 }; for the formats supported by the max9286 driver
+ * channels[] = { 1, 2, 3, 4 };
+ * lanes[] = { 1, 2, 3, 4 };
+ *
+ * and by filtering out all results that would lead to an invalid
+ * link_frequency, as the MAX9286 has a CSI-2 bandwidth limit of 1200Mbps per
+ * lane (600MHz clock frequency).
+ *
+ * If a new serializer with a different pixel rate, or a new image format with
+ * a different bpp is supported, the below table needs to be updated.
+ */
+static const s64 max9286_link_freqs[] = {
+	88000000,
+	110000000,
+	132000000,
+	165000000,
+	176000000,
+	220000000,
+	264000000,
+	275000000,
+	330000000,
+	352000000,
+	396000000,
+	440000000,
+	495000000,
+	528000000,
+	550000000,
+};
+
+static int max9286_get_link_freq_index(s64 link_freq)
+{
+	for (unsigned int i = 0; i < ARRAY_SIZE(max9286_link_freqs); ++i) {
+		if (max9286_link_freqs[i] == link_freq)
+			return i;
+	}
+
+	return -EINVAL;
+}
+
 /* -----------------------------------------------------------------------------
  * I2C IO
  */
@@ -636,10 +705,18 @@ static void max9286_set_fsync_period(struct max9286_priv *priv,
  * V4L2 Subdev
  */
 
-static int max9286_set_pixelrate(struct max9286_priv *priv)
+static int max9286_set_pixelrate(struct max9286_priv *priv,
+				 struct v4l2_subdev_state *state,
+				 unsigned int link_index)
 {
+	const struct max9286_format_info *info;
 	struct max9286_source *source = NULL;
+	struct v4l2_mbus_framefmt *fmt;
+	unsigned int num_sources = 0;
+	int link_freq_index;
 	u64 pixelrate = 0;
+	s64 link_freq;
+	int ret;
 
 	for_each_source(priv, source) {
 		struct v4l2_ctrl *ctrl;
@@ -662,6 +739,8 @@ static int max9286_set_pixelrate(struct max9286_priv *priv)
 				"Unable to calculate pixel rate\n");
 			return -EINVAL;
 		}
+
+		num_sources++;
 	}
 
 	if (!pixelrate) {
@@ -670,14 +749,33 @@ static int max9286_set_pixelrate(struct max9286_priv *priv)
 		return -EINVAL;
 	}
 
-	priv->pixelrate = pixelrate;
+	fmt = v4l2_subdev_state_get_format(state, link_index);
+	info = max9286_get_format_info(fmt->code);
+	if (WARN_ON(!info))
+		return -EINVAL;
+
+	/*
+	 * Compute the link frequency and find the right index to update
+	 * the menu control with.
+	 */
+	link_freq = pixelrate * ((info->gmsl_bpp * num_sources) /
+		    priv->csi2_data_lanes);
+
+	link_freq_index = max9286_get_link_freq_index(link_freq);
+	if (WARN_ON(link_freq_index < 0))
+		return -EINVAL;
 
 	/*
 	 * The CSI-2 transmitter pixel rate is the single source rate multiplied
 	 * by the number of available sources.
 	 */
-	return v4l2_ctrl_s_ctrl_int64(priv->pixelrate_ctrl,
-				      pixelrate * priv->nsources);
+	ret = __v4l2_ctrl_s_ctrl_int64(priv->pixelrate_ctrl,
+				       pixelrate * priv->nsources);
+	if (ret)
+		return -EINVAL;
+	priv->pixelrate = pixelrate;
+
+	return __v4l2_ctrl_s_ctrl(priv->link_freq, link_freq_index);
 }
 
 static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
@@ -687,6 +785,7 @@ static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
 	struct max9286_priv *priv = sd_to_max9286(notifier->sd);
 	struct max9286_source *source = to_max9286_asd(asd)->source;
 	unsigned int index = to_index(priv, source);
+	struct v4l2_subdev_state *state;
 	unsigned int src_pad;
 	int ret;
 
@@ -739,7 +838,11 @@ static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
 	max9286_check_config_link(priv, priv->source_mask);
 	max9286_configure_i2c(priv, false);
 
-	return max9286_set_pixelrate(priv);
+	state = v4l2_subdev_lock_and_get_active_state(notifier->sd);
+	ret = max9286_set_pixelrate(priv, state, index);
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
 }
 
 static void max9286_notify_unbind(struct v4l2_async_notifier *notifier,
@@ -926,7 +1029,9 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 			   struct v4l2_subdev_state *state,
 			   struct v4l2_subdev_format *format)
 {
+	struct max9286_priv *priv = sd_to_max9286(sd);
 	const struct max9286_format_info *info;
+	int ret;
 
 	/*
 	 * Disable setting format on the source pad: format is propagated
@@ -949,7 +1054,22 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 	*v4l2_subdev_state_get_opposite_stream_format(state, format->pad, 0) =
 		format->format;
 
-	return 0;
+	/*
+	 * Update the pixel rate and link_rate controls. We cannot do that in
+	 * s_stream() as the receiver is likely fetching the frame_desc and
+	 * reading the link_rate control before calling this driver's s_stream.
+	 *
+	 * Update the controls here.
+	 *
+	 * TODO: Once the routing table will be made writable and links
+	 * can be enabled/disabled, the controls need to be updated there too.
+	 */
+	ret = max9286_set_pixelrate(priv, state, format->pad);
+	if (ret)
+		dev_err(&priv->client->dev,
+			"Unsupported format/lanes/channel combination: clock rate too high");
+
+	return ret;
 }
 
 static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
@@ -1068,6 +1188,7 @@ static int max9286_s_ctrl(struct v4l2_ctrl *ctrl)
 {
 	switch (ctrl->id) {
 	case V4L2_CID_PIXEL_RATE:
+	case V4L2_CID_LINK_FREQ:
 		return 0;
 	default:
 		return -EINVAL;
@@ -1097,12 +1218,20 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
 	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
 			  V4L2_SUBDEV_FL_STREAMS;
 
-	v4l2_ctrl_handler_init(&priv->ctrls, 1);
+	v4l2_ctrl_handler_init(&priv->ctrls, 2);
 	priv->pixelrate_ctrl = v4l2_ctrl_new_std(&priv->ctrls,
 						 &max9286_ctrl_ops,
 						 V4L2_CID_PIXEL_RATE,
 						 1, INT_MAX, 1, 50000000);
 
+	priv->link_freq =
+		v4l2_ctrl_new_int_menu(&priv->ctrls, &max9286_ctrl_ops,
+				       V4L2_CID_LINK_FREQ,
+				       ARRAY_SIZE(max9286_link_freqs) - 1, 0,
+				       max9286_link_freqs);
+	if (priv->link_freq)
+		priv->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
+
 	priv->sd.ctrl_handler = &priv->ctrls;
 	ret = priv->ctrls.error;
 	if (ret)
-- 
2.44.0

[PATCH 14/19] media: max9286: Implement .get_mbus_config()

[Jacopo Mondi]

Implement the .get_mbus_config() operation to allow the remote receiver
to retrieve the bus configuration.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/i2c/max9286.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
index 0e08a1041eed..a71f0aabd67d 100644
--- a/drivers/media/i2c/max9286.c
+++ b/drivers/media/i2c/max9286.c
@@ -1072,6 +1072,20 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
 	return ret;
 }
 
+static int max9286_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
+				   struct v4l2_mbus_config *config)
+{
+	struct max9286_priv *priv = sd_to_max9286(sd);
+
+	if (pad != MAX9286_SRC_PAD)
+		return -EINVAL;
+
+	config->type = V4L2_MBUS_CSI2_DPHY;
+	config->bus.mipi_csi2.num_data_lanes = priv->csi2_data_lanes;
+
+	return 0;
+}
+
 static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
 				  struct v4l2_mbus_frame_desc *fd)
 {
@@ -1127,6 +1141,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
 	.enum_mbus_code = max9286_enum_mbus_code,
 	.get_fmt	= v4l2_subdev_get_fmt,
 	.set_fmt	= max9286_set_fmt,
+	.get_mbus_config = max9286_get_mbus_config,
 	.get_frame_interval = v4l2_subdev_get_frame_interval,
 	.set_frame_interval = max9286_set_frame_interval,
 	.get_frame_desc	= max9286_get_frame_desc,
-- 
2.44.0

[PATCH 15/19] media: rcar-csi2: Add support for multiplexed streams

[Jacopo Mondi]

Create and initialize the v4l2_subdev_state for the R-Car CSI-2 receiver
in order to prepare to support multiplexed transmitters.

Create the subdevice state with v4l2_subdev_init_finalize() and
implement the init_state() operation to guarantee the state is initialized.

The routing table within the R-Car CSI-2 receiver is fixed, streams
received on source_stream X will be directed to pad (X + 1) by default.
Initialize a static routing table with such routes set as active.

While at it, disable runtime_pm() in the probe() function error path.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-csi2.c | 74 +++++++++++++++++++++-
 1 file changed, 72 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
index 582d5e35db0e..82dc0b92b8d3 100644
--- a/drivers/media/platform/renesas/rcar-csi2.c
+++ b/drivers/media/platform/renesas/rcar-csi2.c
@@ -1226,6 +1226,65 @@ static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
 	.pad	= &rcar_csi2_pad_ops,
 };
 
+static int rcsi2_init_state(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state)
+{
+	/*
+	 * Routing is fixed for this device: streams sent on sink_stream X
+	 * are directed to pad (X + 1). Which streams goes to the next
+	 * processing block (VIN) is controlled by link enablement between the
+	 * CSI-2 and the VIN itself and not by the CSI-2 routing table.
+	 *
+	 * The routing table is then fixed, as stream X will be directed to
+	 * csi:(X + 1)/0 and will be transmitted to VINs the on media link
+	 * csi2:(x + 1)->vin:0.
+	 *
+	 * For example, to route stream #3 to VIN #1 : "csi2:4/0 -> vin1:0" and
+	 * to route stream #2 to VIN #4 : "csi2:3/0 -> vin4:0".
+	 */
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = RCAR_CSI2_SINK,
+			.sink_stream = 0,
+			.source_pad = RCAR_CSI2_SOURCE_VC0,
+			.source_stream = 0,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+		{
+			.sink_pad = RCAR_CSI2_SINK,
+			.sink_stream = 1,
+			.source_pad = RCAR_CSI2_SOURCE_VC1,
+			.source_stream = 0,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+		{
+			.sink_pad = RCAR_CSI2_SINK,
+			.sink_stream = 2,
+			.source_pad = RCAR_CSI2_SOURCE_VC2,
+			.source_stream = 0,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+		{
+			.sink_pad = RCAR_CSI2_SINK,
+			.sink_stream = 3,
+			.source_pad = RCAR_CSI2_SOURCE_VC3,
+			.source_stream = 0,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+
+	struct v4l2_subdev_krouting routing = {
+		.num_routes = ARRAY_SIZE(routes),
+		.routes = routes,
+	};
+
+	return v4l2_subdev_set_routing(sd, state, &routing);
+}
+
+static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {
+	.init_state = rcsi2_init_state,
+};
+
 static irqreturn_t rcsi2_irq(int irq, void *data)
 {
 	struct rcar_csi2 *priv = data;
@@ -1887,11 +1946,13 @@ static int rcsi2_probe(struct platform_device *pdev)
 
 	priv->subdev.owner = THIS_MODULE;
 	priv->subdev.dev = &pdev->dev;
+	priv->subdev.internal_ops = &rcar_csi2_internal_ops;
 	v4l2_subdev_init(&priv->subdev, &rcar_csi2_subdev_ops);
 	v4l2_set_subdevdata(&priv->subdev, &pdev->dev);
 	snprintf(priv->subdev.name, sizeof(priv->subdev.name), "%s %s",
 		 KBUILD_MODNAME, dev_name(&pdev->dev));
-	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
+	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
+			     V4L2_SUBDEV_FL_STREAMS;
 
 	priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
 	priv->subdev.entity.ops = &rcar_csi2_entity_ops;
@@ -1912,14 +1973,22 @@ static int rcsi2_probe(struct platform_device *pdev)
 
 	pm_runtime_enable(&pdev->dev);
 
+	ret = v4l2_subdev_init_finalize(&priv->subdev);
+	if (ret)
+		goto error_pm_runtime;
+
 	ret = v4l2_async_register_subdev(&priv->subdev);
 	if (ret < 0)
-		goto error_async;
+		goto error_subdev;
 
 	dev_info(priv->dev, "%d lanes found\n", priv->lanes);
 
 	return 0;
 
+error_subdev:
+	v4l2_subdev_cleanup(&priv->subdev);
+error_pm_runtime:
+	pm_runtime_disable(&pdev->dev);
 error_async:
 	v4l2_async_nf_unregister(&priv->notifier);
 	v4l2_async_nf_cleanup(&priv->notifier);
@@ -1936,6 +2005,7 @@ static void rcsi2_remove(struct platform_device *pdev)
 	v4l2_async_nf_unregister(&priv->notifier);
 	v4l2_async_nf_cleanup(&priv->notifier);
 	v4l2_async_unregister_subdev(&priv->subdev);
+	v4l2_subdev_cleanup(&priv->subdev);
 
 	pm_runtime_disable(&pdev->dev);
 
-- 
2.44.0

[PATCH 16/19] media: rcar-csi2: Support multiplexed transmitters

[Jacopo Mondi]

Rework the R-Car CSI-2 start routine to support multiplexed
transmitters.

Configure the CSI-2 receiver MIPI CSI-2 Data Type filtering by inspecting
the remote subdev frame_desc instead of relying on the image format
configured on the sink pad.

Enable MIPI CSI-2 Data Type filtering by inspecting the remote subdevice
frame descriptor to discern which Data Type is transmitted on which
Virtual Channel. If multiple Data Types are transmitted on the same VC
then Data Type filtering is disabled.

Rework the per-lane bandwidth calculation to use the LINK_FREQ control
if a transmitter sends multiple streams on the same data link. The
current usage of the PIXEL_RATE control does not support multiplexed
transmitters, as there's not a unique pixel rate among all the possible
source streams.

This change makes mandatory for any subdevice that operates with
the R-Car CSI-2 receiver to implement the .get_frame_desc() operation.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-csi2.c | 280 ++++++++++++++-------
 1 file changed, 191 insertions(+), 89 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
index 82dc0b92b8d3..ffb73272543b 100644
--- a/drivers/media/platform/renesas/rcar-csi2.c
+++ b/drivers/media/platform/renesas/rcar-csi2.c
@@ -69,10 +69,7 @@ struct rcar_csi2;
 #define FLD_REG				0x1c
 #define FLD_FLD_NUM(n)			(((n) & 0xff) << 16)
 #define FLD_DET_SEL(n)			(((n) & 0x3) << 4)
-#define FLD_FLD_EN4			BIT(3)
-#define FLD_FLD_EN3			BIT(2)
-#define FLD_FLD_EN2			BIT(1)
-#define FLD_FLD_EN			BIT(0)
+#define FLD_FLD_EN(n)			BIT((n) & 0xf)
 
 /* Automatic Standby Control */
 #define ASTBY_REG			0x20
@@ -575,6 +572,16 @@ static const struct rcar_csi2_format *rcsi2_code_to_fmt(unsigned int code)
 	return NULL;
 }
 
+static const struct rcar_csi2_format *rcsi2_datatype_to_fmt(unsigned int dt)
+{
+	for (unsigned int i = 0; i < ARRAY_SIZE(rcar_csi2_formats); i++) {
+		if (rcar_csi2_formats[i].datatype == dt)
+			return &rcar_csi2_formats[i];
+	}
+
+	return NULL;
+}
+
 enum rcar_csi2_pads {
 	RCAR_CSI2_SINK,
 	RCAR_CSI2_SOURCE_VC0,
@@ -587,7 +594,8 @@ enum rcar_csi2_pads {
 struct rcar_csi2_info {
 	int (*init_phtw)(struct rcar_csi2 *priv, unsigned int mbps);
 	int (*phy_post_init)(struct rcar_csi2 *priv);
-	int (*start_receiver)(struct rcar_csi2 *priv);
+	int (*start_receiver)(struct rcar_csi2 *priv,
+			      struct v4l2_subdev_state *state);
 	void (*enter_standby)(struct rcar_csi2 *priv);
 	const struct rcsi2_mbps_reg *hsfreqrange;
 	unsigned int csi0clkfreqrange;
@@ -676,6 +684,32 @@ static int rcsi2_exit_standby(struct rcar_csi2 *priv)
 	return 0;
 }
 
+static int rcsi2_get_remote_frame_desc(struct rcar_csi2 *priv,
+				       struct v4l2_mbus_frame_desc *fd)
+{
+	struct media_pad *pad;
+	int ret;
+
+	if (!priv->remote)
+		return -ENOLINK;
+
+	pad = media_pad_remote_pad_unique(&priv->pads[RCAR_CSI2_SINK]);
+	if (IS_ERR(pad))
+		return PTR_ERR(pad);
+
+	ret = v4l2_subdev_call(priv->remote, pad, get_frame_desc,
+			       pad->index, fd);
+	if (ret)
+		return ret;
+
+	if (fd->type != V4L2_MBUS_FRAME_DESC_TYPE_CSI2) {
+		dev_err(priv->dev, "Frame desc does not describe a CSI-2 link");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
 static int rcsi2_wait_phy_start(struct rcar_csi2 *priv,
 				unsigned int lanes)
 {
@@ -726,41 +760,6 @@ static int rcsi2_set_phypll(struct rcar_csi2 *priv, unsigned int mbps)
 	return 0;
 }
 
-static int rcsi2_calc_mbps(struct rcar_csi2 *priv, unsigned int bpp,
-			   unsigned int lanes)
-{
-	struct v4l2_subdev *source;
-	struct v4l2_ctrl *ctrl;
-	u64 mbps;
-
-	if (!priv->remote)
-		return -ENODEV;
-
-	source = priv->remote;
-
-	/* Read the pixel rate control from remote. */
-	ctrl = v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_PIXEL_RATE);
-	if (!ctrl) {
-		dev_err(priv->dev, "no pixel rate control in subdev %s\n",
-			source->name);
-		return -EINVAL;
-	}
-
-	/*
-	 * Calculate the phypll in mbps.
-	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
-	 * bps = link_freq * 2
-	 */
-	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * bpp;
-	do_div(mbps, lanes * 1000000);
-
-	/* Adjust for C-PHY, divide by 2.8. */
-	if (priv->cphy)
-		mbps = div_u64(mbps * 5, 14);
-
-	return mbps;
-}
-
 static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
 				  unsigned int *lanes)
 {
@@ -808,52 +807,146 @@ static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
 	return 0;
 }
 
-static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
+static int rcsi2_calc_mbps(struct rcar_csi2 *priv,
+			   struct v4l2_mbus_frame_desc *fd, unsigned int lanes)
 {
-	const struct rcar_csi2_format *format;
+	struct v4l2_subdev *source;
+	unsigned int bpp;
+	s64 link_freq;
+	u64 mbps;
+
+	if (!priv->remote)
+		return -ENODEV;
+
+	source = priv->remote;
+
+	/*
+	 * v4l2_get_link_freq() uses V4L2_CID_LINK_FREQ first, and falls back
+	 * to V4L2_CID_PIXEL_RATE if V4L2_CID_LINK_FREQ is not available.
+	 *
+	 * With multistream input there is no single pixel rate, and thus we
+	 * cannot use V4L2_CID_PIXEL_RATE, so we pass 0 as the bpp which
+	 * causes v4l2_get_link_freq() to return an error if it falls back to
+	 * V4L2_CID_PIXEL_RATE.
+	 */
+
+	if (fd->num_entries > 1) {
+		bpp = 0;
+	} else {
+		struct v4l2_mbus_frame_desc_entry *entry = &fd->entry[0];
+		const struct rcar_csi2_format *format;
+
+		format = rcsi2_datatype_to_fmt(entry->bus.csi2.dt);
+		if (WARN_ON(!format))
+			return -EINVAL;
+
+		bpp = format->bpp;
+	}
+
+	/*
+	 * Calculate the phypll in mbps.
+	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
+	 * bps = link_freq * 2
+	 */
+	link_freq = v4l2_get_link_freq(source->ctrl_handler, bpp, 2 * lanes);
+	if (link_freq < 0) {
+		dev_err(priv->dev, "Failed to get remote subdev link freq\n");
+		return link_freq;
+	}
+
+	mbps = 2 * link_freq;
+	do_div(mbps, 1000000);
+
+	/* Adjust for C-PHY, divide by 2.8. */
+	if (priv->cphy)
+		mbps = div_u64(mbps * 5, 14);
+
+	return mbps;
+}
+
+static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv,
+				     struct v4l2_subdev_state *state)
+{
+	const struct v4l2_subdev_stream_configs *configs;
 	u32 phycnt, vcdt = 0, vcdt2 = 0, fld = 0;
+	struct v4l2_mbus_frame_desc fd;
 	unsigned int lanes;
-	unsigned int i;
 	int mbps, ret;
 
-	dev_dbg(priv->dev, "Input size (%ux%u%c)\n",
-		priv->mf.width, priv->mf.height,
-		priv->mf.field == V4L2_FIELD_NONE ? 'p' : 'i');
-
-	/* Code is validated in set_fmt. */
-	format = rcsi2_code_to_fmt(priv->mf.code);
-	if (!format)
-		return -EINVAL;
+	/* Get information about multiplexed link. */
+	ret = rcsi2_get_remote_frame_desc(priv, &fd);
+	if (ret)
+		return ret;
 
 	/*
-	 * Enable all supported CSI-2 channels with virtual channel and
-	 * data type matching.
+	 * Configure and enable the R-Car CSI-2 channels.
 	 *
-	 * NOTE: It's not possible to get individual datatype for each
-	 *       source virtual channel. Once this is possible in V4L2
-	 *       it should be used here.
+	 * The VC transmitted on the channel is configured by the [CSI-2->VIN]
+	 * link_setup operation, while the data type to match comes from the
+	 * remote subdevice.
 	 */
-	for (i = 0; i < priv->info->num_channels; i++) {
-		u32 vcdt_part;
+	for (unsigned int i = 0; i < priv->info->num_channels; i++) {
+		struct v4l2_mbus_frame_desc_entry *entry = NULL;
 
+		/* CSI-2 channel disabled (not linked to any VIN). */
 		if (priv->channel_vc[i] < 0)
 			continue;
 
-		vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) | VCDT_VCDTN_EN |
-			VCDT_SEL_DTN_ON | VCDT_SEL_DT(format->datatype);
+		u32 vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) |
+				VCDT_VCDTN_EN | VCDT_SEL_DTN_ON;
+
+		/*
+		 * Search the entries that describe the data types on the
+		 * MIPI CSI-2 Virtual Channel assigned to this CSI-2 channel.
+		 */
+		for (unsigned int e = 0; e < fd.num_entries; e++) {
+			if (fd.entry[e].bus.csi2.vc != priv->channel_vc[i])
+				continue;
+
+			/*
+			 * If multiple data types are sent on the same MIPI
+			 * CSI-2 Virtual Channel, disable Data Type matching.
+			 */
+			if (entry) {
+				vcdt_part &= ~VCDT_SEL_DTN_ON;
+				break;
+			}
+
+			entry = &fd.entry[e];
+			vcdt_part |= VCDT_SEL_DT(entry->bus.csi2.dt);
+		}
+
+		if (!entry)
+			continue;
 
 		/* Store in correct reg and offset. */
 		if (i < 2)
 			vcdt |= vcdt_part << ((i % 2) * 16);
 		else
 			vcdt2 |= vcdt_part << ((i % 2) * 16);
+
+		dev_dbg(priv->dev, "channel %u: VC = %d, datatype = 0x%x\n",
+			i, priv->channel_vc[i], entry->bus.csi2.dt);
 	}
 
-	if (priv->mf.field == V4L2_FIELD_ALTERNATE) {
-		fld = FLD_DET_SEL(1) | FLD_FLD_EN4 | FLD_FLD_EN3 | FLD_FLD_EN2
-			| FLD_FLD_EN;
+	/*
+	 * Configure field handling inspecting the formats of the
+	 * sink pad streams.
+	 */
+	configs = &state->stream_configs;
+	for (unsigned int i = 0; i < configs->num_configs; ++i) {
+		const struct v4l2_subdev_stream_config *config = configs->configs;
+
+		if (config->pad != RCAR_CSI2_SINK)
+			continue;
 
-		if (priv->mf.height == 240)
+		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
+			continue;
+
+		fld |= FLD_DET_SEL(1) | FLD_FLD_EN(config->stream);
+
+		/* PAL vs NTSC. */
+		if (config->fmt.height == 240)
 			fld |= FLD_FLD_NUM(0);
 		else
 			fld |= FLD_FLD_NUM(1);
@@ -870,7 +963,7 @@ static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
 	phycnt = PHYCNT_ENABLECLK;
 	phycnt |= (1 << lanes) - 1;
 
-	mbps = rcsi2_calc_mbps(priv, format->bpp, lanes);
+	mbps = rcsi2_calc_mbps(priv, &fd, lanes);
 	if (mbps < 0)
 		return mbps;
 
@@ -1049,23 +1142,24 @@ static int rcsi2_c_phy_setting_v4h(struct rcar_csi2 *priv, int msps)
 	return 0;
 }
 
-static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
+static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv,
+				    struct v4l2_subdev_state *state)
 {
-	const struct rcar_csi2_format *format;
+	struct v4l2_mbus_frame_desc fd;
 	unsigned int lanes;
 	int msps;
 	int ret;
 
 	/* Calculate parameters */
-	format = rcsi2_code_to_fmt(priv->mf.code);
-	if (!format)
-		return -EINVAL;
-
 	ret = rcsi2_get_active_lanes(priv, &lanes);
 	if (ret)
 		return ret;
 
-	msps = rcsi2_calc_mbps(priv, format->bpp, lanes);
+	ret = rcsi2_get_remote_frame_desc(priv, &fd);
+	if (ret)
+		return ret;
+
+	msps = rcsi2_calc_mbps(priv, &fd, lanes);
 	if (msps < 0)
 		return msps;
 
@@ -1114,7 +1208,7 @@ static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
 	return 0;
 }
 
-static int rcsi2_start(struct rcar_csi2 *priv)
+static int rcsi2_start(struct rcar_csi2 *priv, struct v4l2_subdev_state *state)
 {
 	int ret;
 
@@ -1122,7 +1216,7 @@ static int rcsi2_start(struct rcar_csi2 *priv)
 	if (ret < 0)
 		return ret;
 
-	ret = priv->info->start_receiver(priv);
+	ret = priv->info->start_receiver(priv, state);
 	if (ret) {
 		rcsi2_enter_standby(priv);
 		return ret;
@@ -1146,26 +1240,24 @@ static void rcsi2_stop(struct rcar_csi2 *priv)
 static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
 {
 	struct rcar_csi2 *priv = sd_to_csi2(sd);
+	struct v4l2_subdev_state *state;
 	int ret = 0;
 
-	mutex_lock(&priv->lock);
+	if (!priv->remote)
+		return -ENODEV;
 
-	if (!priv->remote) {
-		ret = -ENODEV;
-		goto out;
-	}
+	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
 
-	if (enable && priv->stream_count == 0) {
-		ret = rcsi2_start(priv);
-		if (ret)
-			goto out;
-	} else if (!enable && priv->stream_count == 1) {
+	if (enable && priv->stream_count == 0)
+		ret = rcsi2_start(priv, state);
+	else if (!enable && priv->stream_count == 1)
 		rcsi2_stop(priv);
-	}
+	if (ret)
+		goto out;
 
 	priv->stream_count += enable ? 1 : -1;
 out:
-	mutex_unlock(&priv->lock);
+	v4l2_subdev_unlock_state(state);
 
 	return ret;
 }
@@ -1310,14 +1402,17 @@ static irqreturn_t rcsi2_irq(int irq, void *data)
 
 static irqreturn_t rcsi2_irq_thread(int irq, void *data)
 {
+	struct v4l2_subdev_state *state;
 	struct rcar_csi2 *priv = data;
 
-	mutex_lock(&priv->lock);
+	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
+
 	rcsi2_stop(priv);
 	usleep_range(1000, 2000);
-	if (rcsi2_start(priv))
+	if (rcsi2_start(priv, state))
 		dev_warn(priv->dev, "Failed to restart CSI-2 receiver\n");
-	mutex_unlock(&priv->lock);
+
+	v4l2_subdev_unlock_state(state);
 
 	return IRQ_HANDLED;
 }
@@ -1340,6 +1435,13 @@ static int rcsi2_notify_bound(struct v4l2_async_notifier *notifier,
 		return pad;
 	}
 
+	if (!v4l2_subdev_has_op(subdev, pad, get_frame_desc)) {
+		dev_err(priv->dev,
+			"Subdev %s bound failed: missing get_frame_desc()\n",
+			subdev->name);
+		return -EINVAL;
+	}
+
 	priv->remote = subdev;
 	priv->remote_pad = pad;
 
-- 
2.44.0

[PATCH 17/19] media: rcar-csi2: Store format in the subdev state

[Jacopo Mondi]

Store the format in the subdevice state. Disallow setting format
on the source pads, as formats are set on the sink pad streams and
propagated to the source streams.

Now that the driver doesn't store the active format in the
driver-specific structure, also remove the mutex and use the lock
associated with the state.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-csi2.c | 54 +++++++---------------
 1 file changed, 16 insertions(+), 38 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
index ffb73272543b..ed818a6fa655 100644
--- a/drivers/media/platform/renesas/rcar-csi2.c
+++ b/drivers/media/platform/renesas/rcar-csi2.c
@@ -621,8 +621,6 @@ struct rcar_csi2 {
 
 	int channel_vc[4];
 
-	struct mutex lock; /* Protects mf and stream_count. */
-	struct v4l2_mbus_framefmt mf;
 	int stream_count;
 
 	bool cphy;
@@ -1263,43 +1261,28 @@ static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
 }
 
 static int rcsi2_set_pad_format(struct v4l2_subdev *sd,
-				struct v4l2_subdev_state *sd_state,
+				struct v4l2_subdev_state *state,
 				struct v4l2_subdev_format *format)
 {
-	struct rcar_csi2 *priv = sd_to_csi2(sd);
-	struct v4l2_mbus_framefmt *framefmt;
+	struct v4l2_mbus_framefmt *fmt;
 
-	mutex_lock(&priv->lock);
+	/*
+	 * Format is propagated from sink streams to source streams, so
+	 * disallow setting format on the source pads.
+	 */
+	if (format->pad > RCAR_CSI2_SINK)
+		return -EINVAL;
 
 	if (!rcsi2_code_to_fmt(format->format.code))
 		format->format.code = rcar_csi2_formats[0].code;
 
-	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
-		priv->mf = format->format;
-	} else {
-		framefmt = v4l2_subdev_state_get_format(sd_state, 0);
-		*framefmt = format->format;
-	}
 
-	mutex_unlock(&priv->lock);
+	fmt = v4l2_subdev_state_get_format(state, format->pad, format->stream);
+	*fmt = format->format;
 
-	return 0;
-}
-
-static int rcsi2_get_pad_format(struct v4l2_subdev *sd,
-				struct v4l2_subdev_state *sd_state,
-				struct v4l2_subdev_format *format)
-{
-	struct rcar_csi2 *priv = sd_to_csi2(sd);
-
-	mutex_lock(&priv->lock);
-
-	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE)
-		format->format = priv->mf;
-	else
-		format->format = *v4l2_subdev_state_get_format(sd_state, 0);
-
-	mutex_unlock(&priv->lock);
+	fmt = v4l2_subdev_state_get_opposite_stream_format(state, format->pad,
+							   format->stream);
+	*fmt = format->format;
 
 	return 0;
 }
@@ -1310,7 +1293,7 @@ static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
 
 static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
 	.set_fmt = rcsi2_set_pad_format,
-	.get_fmt = rcsi2_get_pad_format,
+	.get_fmt = v4l2_subdev_get_fmt,
 };
 
 static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
@@ -2031,20 +2014,19 @@ static int rcsi2_probe(struct platform_device *pdev)
 
 	priv->dev = &pdev->dev;
 
-	mutex_init(&priv->lock);
 	priv->stream_count = 0;
 
 	ret = rcsi2_probe_resources(priv, pdev);
 	if (ret) {
 		dev_err(priv->dev, "Failed to get resources\n");
-		goto error_mutex;
+		return ret;
 	}
 
 	platform_set_drvdata(pdev, priv);
 
 	ret = rcsi2_parse_dt(priv);
 	if (ret)
-		goto error_mutex;
+		return ret;
 
 	priv->subdev.owner = THIS_MODULE;
 	priv->subdev.dev = &pdev->dev;
@@ -2094,8 +2076,6 @@ static int rcsi2_probe(struct platform_device *pdev)
 error_async:
 	v4l2_async_nf_unregister(&priv->notifier);
 	v4l2_async_nf_cleanup(&priv->notifier);
-error_mutex:
-	mutex_destroy(&priv->lock);
 
 	return ret;
 }
@@ -2110,8 +2090,6 @@ static void rcsi2_remove(struct platform_device *pdev)
 	v4l2_subdev_cleanup(&priv->subdev);
 
 	pm_runtime_disable(&pdev->dev);
-
-	mutex_destroy(&priv->lock);
 }
 
 static struct platform_driver rcar_csi2_pdrv = {
-- 
2.44.0

[PATCH 18/19] media: rcar-csi2: Implement set_routing

[Jacopo Mondi]

Add the set_routing() subdev operation to allow userspace to activate
routes on the R-Car CSI-2 receiver.

Routing for this device is fixed. Validate that the provided route
respects it.

Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-csi2.c | 48 +++++++++++++++++++++-
 1 file changed, 47 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
index ed818a6fa655..47cb12f077e5 100644
--- a/drivers/media/platform/renesas/rcar-csi2.c
+++ b/drivers/media/platform/renesas/rcar-csi2.c
@@ -1287,6 +1287,51 @@ static int rcsi2_set_pad_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int rcsi2_apply_routing(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *state,
+			       struct v4l2_subdev_krouting *routing)
+{
+	int ret;
+
+	ret = v4l2_subdev_routing_validate(sd, routing,
+					   V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
+	if (ret)
+		return ret;
+
+	/*
+	 * Routing is fixed for this device.
+	 *
+	 * Only routes in the form of CSI2:0/x->CSI2:x+1/0 are allowed.
+	 *
+	 * We have anyway to implement set_routing to mark the route as active.
+	 */
+	for (unsigned int i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+		unsigned int pad = route->sink_stream + 1;
+
+		if (route->sink_pad != 0)
+			return -EINVAL;
+
+		if (route->source_pad != pad || route->source_stream != 0)
+			return -EINVAL;
+	}
+
+	return v4l2_subdev_set_routing(sd, state, routing);
+}
+
+static int rcsi2_set_routing(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *state,
+			     enum v4l2_subdev_format_whence which,
+			     struct v4l2_subdev_krouting *routing)
+{
+	struct rcar_csi2 *priv = sd_to_csi2(sd);
+
+	if (which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->stream_count)
+		return -EBUSY;
+
+	return rcsi2_apply_routing(sd, state, routing);
+}
+
 static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
 	.s_stream = rcsi2_s_stream,
 };
@@ -1294,6 +1339,7 @@ static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
 static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
 	.set_fmt = rcsi2_set_pad_format,
 	.get_fmt = v4l2_subdev_get_fmt,
+	.set_routing = rcsi2_set_routing,
 };
 
 static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
@@ -1353,7 +1399,7 @@ static int rcsi2_init_state(struct v4l2_subdev *sd,
 		.routes = routes,
 	};
 
-	return v4l2_subdev_set_routing(sd, state, &routing);
+	return rcsi2_apply_routing(sd, state, &routing);
 }
 
 static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {
-- 
2.44.0

[PATCH 19/19] media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2

[Jacopo Mondi]

The YUYV8_1X16 and UYVY8_1X16 formats are treated as 'ITU-R
BT.601/BT.1358 16-bit YCbCr-422 input' (YUV16 - 0x5) in the R-Car VIN
driver and are thus disallowed when capturing frames from the R-Car
CSI-2 interface according to the hardware manual.

As the 1X16 format variants are meant to be used with serial busses they
have to be treated as 'YCbCr-422 8-bit data input' (0x1) when capturing
from CSI-2, which is a valid setting for CSI-2.

Commit 78b3f9d75a62 ("media: rcar-vin: Add check that input interface
and format are valid") disallowed capturing YUV16 when using the CSI-2
interface. Fix this by using YUV8_BT601 for YCbCr422 when CSI-2 is in
use.

Fixes: 78b3f9d75a62 ("media: rcar-vin: Add check that input interface and format are valid")
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
---
 .../media/platform/renesas/rcar-vin/rcar-dma.c   | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index e2c40abc6d3d..21d5b2815e86 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -742,12 +742,22 @@ static int rvin_setup(struct rvin_dev *vin)
 	 */
 	switch (vin->mbus_code) {
 	case MEDIA_BUS_FMT_YUYV8_1X16:
-		/* BT.601/BT.1358 16bit YCbCr422 */
-		vnmc |= VNMC_INF_YUV16;
+		if (vin->is_csi)
+			/* YCbCr422 8-bit */
+			vnmc |= VNMC_INF_YUV8_BT601;
+		else
+			/* BT.601/BT.1358 16bit YCbCr422 */
+			vnmc |= VNMC_INF_YUV16;
 		input_is_yuv = true;
 		break;
 	case MEDIA_BUS_FMT_UYVY8_1X16:
-		vnmc |= VNMC_INF_YUV16 | VNMC_YCAL;
+		if (vin->is_csi)
+			/* YCbCr422 8-bit */
+			vnmc |= VNMC_INF_YUV8_BT601;
+		else
+			/* BT.601/BT.1358 16bit YCbCr422 */
+			vnmc |= VNMC_INF_YUV16;
+		vnmc |= VNMC_YCAL;
 		input_is_yuv = true;
 		break;
 	case MEDIA_BUS_FMT_UYVY8_2X8:
-- 
2.44.0

Re: [PATCH 00/19] media: renesas: rcar-csi2: Support multiple streams

[Niklas Söderlund]

Hi Jacopo,

Thanks for this work! I'm happy that we finally can get this upstream. I 
will test and review this in the coming days. As there appears to be a 
few dependencies to get this to work do you have a branch covering 
everything needed?

On 2024-04-30 12:39:36 +0200, Jacopo Mondi wrote:
> Hello this series implements multi-stream support for R-Car CSI-2 and the
> two CSI-2 devices that are available on Renesas dev-boards which are mainline
> supported (the ADV748x HDMI decoder and the MAX9286 GMSL deserializer).

You also need to update the MAX96712 driver in staging else V3U and Gen4 
will break.

> 
> The net gain in feature is:
> - ADV748x: Select on which MIPI CSI-2 VC to send the stream on
> - MAX9286: Paves the way for run-time selection of active links
> 
> Both features are realized by controlling the devices routing tables.
> The max9286 routing support is not implemented, but these patches are needed
> in order to maintain the current version compatible with R-Car CSI-2.
> 
> The R-Car CSI-2 driver needs to be updated in order to support multiplexed
> transmitters, in particular:
> 
> - Use LINK_FREQ to compute the lane bandwidth
> - Support subdev_active state
> - Configure DT and VC handling by using the remote's frame_desc
> 
> A separate fix for VIN sits at the top of the series to re-enable YUYV capture
> operations.
> 
> The series needs to activate streams in v4l2-subdev to work correctly:
> 
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -32,7 +32,7 @@
>   * 'v4l2_subdev_enable_streams_api' to 1 below.
>   */
> 
> -static bool v4l2_subdev_enable_streams_api;
> +static bool v4l2_subdev_enable_streams_api = 1;
>  #endif

I'm not up to date with the latest state of the multiple streams API.  
Can this be runtime configured or will this break capture if the kernel 
is not patched to enable this?

> 
> 
> Test scripts, based on vin-tests are available at:
> https://git.uk.ideasonboard.com/jmondi/vin-tests
> 
> A note for Sakari: patch
> "media: max9286: Implement support for LINK_FREQ"
> 
> Could actually use your new
> "media: v4l: Support passing sub-device argument to v4l2_get_link_freq()"
> 
> I kept it as it is for this version waiting for your patch to get in.

I like this idea.

> 
> A note for Niklas:
> I think this series paves the way to actually moving the control of which
> VC goes to which VIN using a routing table instead of using link enablement.
> 
> Currently the setup is as it follows: routing within R-Car CSI-2 is fixed, sink
> stream 0/X goes to source stream X+1/0 and each source pad represents a VC.
> Which VC gets sent to which VIN is controlled by link enablement.
> 
> A more natural way of handling this would be to make the routing table
> within CSI-2 configurable and have media-links immutable between the CSI-2
> source pad. A CSI-2 source pad would then represent a 'channel' between
> CSI-2 and VIN and not the VC which is sent on it.
> 
> This setup would better represent the current implementation, with VIN's VCSEL
> fixed channel routing and with the CSI-2 routing table controlling which VC gets
> sent on which channel. What do you think ?

That could be useful. Maybe we can even switch to use immutable links 
and use the routing instead greatly simplifying the VIN driver as this 
together with the other VIN series on the list we can drop the ugly 
group concept ;-)

But this series is already quiet large, I think we can work on this 
on-top in a new series.

> 
> Jacopo Mondi (19):
>   media: adv748x: Add support for active state
>   media: adv748x: Add flags to adv748x_subdev_init()
>   media: adv748x: Use V4L2 streams
>   media: adv748x: Propagate format to opposite stream
>   media: adv748x: Implement set_routing()
>   media: adv748x: Use routes to configure VC
>   media: adv748x: Implement .get_frame_desc()
>   media: max9286: Add support for subdev active state
>   media: max9286: Fix enum_mbus_code
>   media: max9286: Use frame interval from subdev state
>   media: max9286: Use V4L2 Streams
>   media: max9286: Implement .get_frame_desc()
>   media: max9286: Implement support for LINK_FREQ
>   media: max9286: Implement .get_mbus_config()
>   media: rcar-csi2: Add support for multiplexed streams
>   media: rcar-csi2: Support multiplexed transmitters
>   media: rcar-csi2: Store format in the subdev state
>   media: rcar-csi2: Implement set_routing
>   media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2
> 
>  drivers/media/i2c/adv748x/adv748x-afe.c       |   2 +-
>  drivers/media/i2c/adv748x/adv748x-core.c      |  12 +-
>  drivers/media/i2c/adv748x/adv748x-csi2.c      | 251 +++++++---
>  drivers/media/i2c/adv748x/adv748x-hdmi.c      |   2 +-
>  drivers/media/i2c/adv748x/adv748x.h           |   4 +-
>  drivers/media/i2c/max9286.c                   | 457 ++++++++++++------
>  drivers/media/platform/renesas/rcar-csi2.c    | 442 ++++++++++++-----
>  .../platform/renesas/rcar-vin/rcar-dma.c      |  16 +-
>  8 files changed, 834 insertions(+), 352 deletions(-)
> 
> --
> 2.44.0
> 

-- 
Kind Regards,
Niklas Söderlund

Re: [PATCH 00/19] media: renesas: rcar-csi2: Support multiple streams

[Jacopo Mondi]

Hi Niklas
  thanks for the quick reply

On Tue, Apr 30, 2024 at 01:17:18PM +0200, Niklas Söderlund wrote:
> Hi Jacopo,
>
> Thanks for this work! I'm happy that we finally can get this upstream. I
> will test and review this in the coming days. As there appears to be a
> few dependencies to get this to work do you have a branch covering
> everything needed?
>

Sure
https://git.kernel.org/pub/scm/linux/kernel/git/jmondi/linux.git/
jmondi/renesas-drivers-2024-04-23-v6.9-rc5/multistream

I should have added it to the cover letter

> On 2024-04-30 12:39:36 +0200, Jacopo Mondi wrote:
> > Hello this series implements multi-stream support for R-Car CSI-2 and the
> > two CSI-2 devices that are available on Renesas dev-boards which are mainline
> > supported (the ADV748x HDMI decoder and the MAX9286 GMSL deserializer).
>
> You also need to update the MAX96712 driver in staging else V3U and Gen4
> will break.
>

Ouch, right. I can do that but won't be able to test. Would you ?

> >
> > The net gain in feature is:
> > - ADV748x: Select on which MIPI CSI-2 VC to send the stream on
> > - MAX9286: Paves the way for run-time selection of active links
> >
> > Both features are realized by controlling the devices routing tables.
> > The max9286 routing support is not implemented, but these patches are needed
> > in order to maintain the current version compatible with R-Car CSI-2.
> >
> > The R-Car CSI-2 driver needs to be updated in order to support multiplexed
> > transmitters, in particular:
> >
> > - Use LINK_FREQ to compute the lane bandwidth
> > - Support subdev_active state
> > - Configure DT and VC handling by using the remote's frame_desc
> >
> > A separate fix for VIN sits at the top of the series to re-enable YUYV capture
> > operations.
> >
> > The series needs to activate streams in v4l2-subdev to work correctly:
> >
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -32,7 +32,7 @@
> >   * 'v4l2_subdev_enable_streams_api' to 1 below.
> >   */
> >
> > -static bool v4l2_subdev_enable_streams_api;
> > +static bool v4l2_subdev_enable_streams_api = 1;
> >  #endif
>
> I'm not up to date with the latest state of the multiple streams API.
> Can this be runtime configured or will this break capture if the kernel
> is not patched to enable this?
>

I'm afraid the latter. I really hope in the next kernel release we can
get rid of this protection and enable streams by default.

> >
> >
> > Test scripts, based on vin-tests are available at:
> > https://git.uk.ideasonboard.com/jmondi/vin-tests
> >
> > A note for Sakari: patch
> > "media: max9286: Implement support for LINK_FREQ"
> >
> > Could actually use your new
> > "media: v4l: Support passing sub-device argument to v4l2_get_link_freq()"
> >
> > I kept it as it is for this version waiting for your patch to get in.
>
> I like this idea.
>
> >
> > A note for Niklas:
> > I think this series paves the way to actually moving the control of which
> > VC goes to which VIN using a routing table instead of using link enablement.
> >
> > Currently the setup is as it follows: routing within R-Car CSI-2 is fixed, sink
> > stream 0/X goes to source stream X+1/0 and each source pad represents a VC.
> > Which VC gets sent to which VIN is controlled by link enablement.
> >
> > A more natural way of handling this would be to make the routing table
> > within CSI-2 configurable and have media-links immutable between the CSI-2
> > source pad. A CSI-2 source pad would then represent a 'channel' between
> > CSI-2 and VIN and not the VC which is sent on it.
> >
> > This setup would better represent the current implementation, with VIN's VCSEL
> > fixed channel routing and with the CSI-2 routing table controlling which VC gets
> > sent on which channel. What do you think ?
>
> That could be useful. Maybe we can even switch to use immutable links
> and use the routing instead greatly simplifying the VIN driver as this
> together with the other VIN series on the list we can drop the ugly
> group concept ;-)

I had the same idea, but I'm not yet sure how this would work when a
CSI-2 can be routed to multiple VIN groups...

>
> But this series is already quiet large, I think we can work on this
> on-top in a new series.
>

Exactly, I didn't want to pile too many things. max9286 routing would
be next in line, I kept it out to avoid sending 30 patches in one go
(and to send this out earlier to get feedbacks)

Thanks
  j

> >
> > Jacopo Mondi (19):
> >   media: adv748x: Add support for active state
> >   media: adv748x: Add flags to adv748x_subdev_init()
> >   media: adv748x: Use V4L2 streams
> >   media: adv748x: Propagate format to opposite stream
> >   media: adv748x: Implement set_routing()
> >   media: adv748x: Use routes to configure VC
> >   media: adv748x: Implement .get_frame_desc()
> >   media: max9286: Add support for subdev active state
> >   media: max9286: Fix enum_mbus_code
> >   media: max9286: Use frame interval from subdev state
> >   media: max9286: Use V4L2 Streams
> >   media: max9286: Implement .get_frame_desc()
> >   media: max9286: Implement support for LINK_FREQ
> >   media: max9286: Implement .get_mbus_config()
> >   media: rcar-csi2: Add support for multiplexed streams
> >   media: rcar-csi2: Support multiplexed transmitters
> >   media: rcar-csi2: Store format in the subdev state
> >   media: rcar-csi2: Implement set_routing
> >   media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2
> >
> >  drivers/media/i2c/adv748x/adv748x-afe.c       |   2 +-
> >  drivers/media/i2c/adv748x/adv748x-core.c      |  12 +-
> >  drivers/media/i2c/adv748x/adv748x-csi2.c      | 251 +++++++---
> >  drivers/media/i2c/adv748x/adv748x-hdmi.c      |   2 +-
> >  drivers/media/i2c/adv748x/adv748x.h           |   4 +-
> >  drivers/media/i2c/max9286.c                   | 457 ++++++++++++------
> >  drivers/media/platform/renesas/rcar-csi2.c    | 442 ++++++++++++-----
> >  .../platform/renesas/rcar-vin/rcar-dma.c      |  16 +-
> >  8 files changed, 834 insertions(+), 352 deletions(-)
> >
> > --
> > 2.44.0
> >
>
> --
> Kind Regards,
> Niklas Söderlund
>

Re: [PATCH 15/19] media: rcar-csi2: Add support for multiplexed streams

[Niklas Söderlund]

Hi Jacopo,

Thanks for your work.

On 2024-04-30 12:39:51 +0200, Jacopo Mondi wrote:
> Create and initialize the v4l2_subdev_state for the R-Car CSI-2 receiver
> in order to prepare to support multiplexed transmitters.
> 
> Create the subdevice state with v4l2_subdev_init_finalize() and
> implement the init_state() operation to guarantee the state is initialized.
> 
> The routing table within the R-Car CSI-2 receiver is fixed, streams
> received on source_stream X will be directed to pad (X + 1) by default.
> Initialize a static routing table with such routes set as active.
> 
> While at it, disable runtime_pm() in the probe() function error path.

Can we break this out in a separate patch? As the multiplexed stream 
work will not be ready for v6.10 if I understood the tendencies 
correctly (?), we can at least fix this issue before that.

> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/platform/renesas/rcar-csi2.c | 74 +++++++++++++++++++++-
>  1 file changed, 72 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> index 582d5e35db0e..82dc0b92b8d3 100644
> --- a/drivers/media/platform/renesas/rcar-csi2.c
> +++ b/drivers/media/platform/renesas/rcar-csi2.c
> @@ -1226,6 +1226,65 @@ static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
>  	.pad	= &rcar_csi2_pad_ops,
>  };
>  
> +static int rcsi2_init_state(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state)
> +{
> +	/*
> +	 * Routing is fixed for this device: streams sent on sink_stream X
> +	 * are directed to pad (X + 1). Which streams goes to the next
> +	 * processing block (VIN) is controlled by link enablement between the
> +	 * CSI-2 and the VIN itself and not by the CSI-2 routing table.
> +	 *
> +	 * The routing table is then fixed, as stream X will be directed to
> +	 * csi:(X + 1)/0 and will be transmitted to VINs the on media link
> +	 * csi2:(x + 1)->vin:0.
> +	 *
> +	 * For example, to route stream #3 to VIN #1 : "csi2:4/0 -> vin1:0" and
> +	 * to route stream #2 to VIN #4 : "csi2:3/0 -> vin4:0".
> +	 */
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = RCAR_CSI2_SINK,
> +			.sink_stream = 0,
> +			.source_pad = RCAR_CSI2_SOURCE_VC0,
> +			.source_stream = 0,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +		{
> +			.sink_pad = RCAR_CSI2_SINK,
> +			.sink_stream = 1,
> +			.source_pad = RCAR_CSI2_SOURCE_VC1,
> +			.source_stream = 0,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +		{
> +			.sink_pad = RCAR_CSI2_SINK,
> +			.sink_stream = 2,
> +			.source_pad = RCAR_CSI2_SOURCE_VC2,
> +			.source_stream = 0,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +		{
> +			.sink_pad = RCAR_CSI2_SINK,
> +			.sink_stream = 3,
> +			.source_pad = RCAR_CSI2_SOURCE_VC3,
> +			.source_stream = 0,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +	};
> +
> +	struct v4l2_subdev_krouting routing = {
> +		.num_routes = ARRAY_SIZE(routes),
> +		.routes = routes,
> +	};

Should not the two structs above be static const as you return a pointer 
to them?

> +
> +	return v4l2_subdev_set_routing(sd, state, &routing);
> +}
> +
> +static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {
> +	.init_state = rcsi2_init_state,
> +};
> +
>  static irqreturn_t rcsi2_irq(int irq, void *data)
>  {
>  	struct rcar_csi2 *priv = data;
> @@ -1887,11 +1946,13 @@ static int rcsi2_probe(struct platform_device *pdev)
>  
>  	priv->subdev.owner = THIS_MODULE;
>  	priv->subdev.dev = &pdev->dev;
> +	priv->subdev.internal_ops = &rcar_csi2_internal_ops;
>  	v4l2_subdev_init(&priv->subdev, &rcar_csi2_subdev_ops);
>  	v4l2_set_subdevdata(&priv->subdev, &pdev->dev);
>  	snprintf(priv->subdev.name, sizeof(priv->subdev.name), "%s %s",
>  		 KBUILD_MODNAME, dev_name(&pdev->dev));
> -	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
> +			     V4L2_SUBDEV_FL_STREAMS;
>  
>  	priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
>  	priv->subdev.entity.ops = &rcar_csi2_entity_ops;
> @@ -1912,14 +1973,22 @@ static int rcsi2_probe(struct platform_device *pdev)
>  
>  	pm_runtime_enable(&pdev->dev);
>  
> +	ret = v4l2_subdev_init_finalize(&priv->subdev);
> +	if (ret)
> +		goto error_pm_runtime;
> +
>  	ret = v4l2_async_register_subdev(&priv->subdev);
>  	if (ret < 0)
> -		goto error_async;
> +		goto error_subdev;
>  
>  	dev_info(priv->dev, "%d lanes found\n", priv->lanes);
>  
>  	return 0;
>  
> +error_subdev:
> +	v4l2_subdev_cleanup(&priv->subdev);
> +error_pm_runtime:
> +	pm_runtime_disable(&pdev->dev);
>  error_async:
>  	v4l2_async_nf_unregister(&priv->notifier);
>  	v4l2_async_nf_cleanup(&priv->notifier);
> @@ -1936,6 +2005,7 @@ static void rcsi2_remove(struct platform_device *pdev)
>  	v4l2_async_nf_unregister(&priv->notifier);
>  	v4l2_async_nf_cleanup(&priv->notifier);
>  	v4l2_async_unregister_subdev(&priv->subdev);
> +	v4l2_subdev_cleanup(&priv->subdev);
>  
>  	pm_runtime_disable(&pdev->dev);
>  
> -- 
> 2.44.0
> 

-- 
Kind Regards,
Niklas Söderlund

Re: [PATCH 16/19] media: rcar-csi2: Support multiplexed transmitters

[Niklas Söderlund]

Hi Jacopo,

Thanks for your work.

On 2024-04-30 12:39:52 +0200, Jacopo Mondi wrote:
> Rework the R-Car CSI-2 start routine to support multiplexed
> transmitters.
> 
> Configure the CSI-2 receiver MIPI CSI-2 Data Type filtering by inspecting
> the remote subdev frame_desc instead of relying on the image format
> configured on the sink pad.
> 
> Enable MIPI CSI-2 Data Type filtering by inspecting the remote subdevice
> frame descriptor to discern which Data Type is transmitted on which
> Virtual Channel. If multiple Data Types are transmitted on the same VC
> then Data Type filtering is disabled.
> 
> Rework the per-lane bandwidth calculation to use the LINK_FREQ control
> if a transmitter sends multiple streams on the same data link. The
> current usage of the PIXEL_RATE control does not support multiplexed
> transmitters, as there's not a unique pixel rate among all the possible
> source streams.
> 
> This change makes mandatory for any subdevice that operates with
> the R-Car CSI-2 receiver to implement the .get_frame_desc() operation.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/platform/renesas/rcar-csi2.c | 280 ++++++++++++++-------
>  1 file changed, 191 insertions(+), 89 deletions(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> index 82dc0b92b8d3..ffb73272543b 100644
> --- a/drivers/media/platform/renesas/rcar-csi2.c
> +++ b/drivers/media/platform/renesas/rcar-csi2.c
> @@ -69,10 +69,7 @@ struct rcar_csi2;
>  #define FLD_REG				0x1c
>  #define FLD_FLD_NUM(n)			(((n) & 0xff) << 16)
>  #define FLD_DET_SEL(n)			(((n) & 0x3) << 4)
> -#define FLD_FLD_EN4			BIT(3)
> -#define FLD_FLD_EN3			BIT(2)
> -#define FLD_FLD_EN2			BIT(1)
> -#define FLD_FLD_EN			BIT(0)
> +#define FLD_FLD_EN(n)			BIT((n) & 0xf)
>  
>  /* Automatic Standby Control */
>  #define ASTBY_REG			0x20
> @@ -575,6 +572,16 @@ static const struct rcar_csi2_format *rcsi2_code_to_fmt(unsigned int code)
>  	return NULL;
>  }
>  
> +static const struct rcar_csi2_format *rcsi2_datatype_to_fmt(unsigned int dt)
> +{
> +	for (unsigned int i = 0; i < ARRAY_SIZE(rcar_csi2_formats); i++) {
> +		if (rcar_csi2_formats[i].datatype == dt)
> +			return &rcar_csi2_formats[i];
> +	}
> +
> +	return NULL;
> +}
> +
>  enum rcar_csi2_pads {
>  	RCAR_CSI2_SINK,
>  	RCAR_CSI2_SOURCE_VC0,
> @@ -587,7 +594,8 @@ enum rcar_csi2_pads {
>  struct rcar_csi2_info {
>  	int (*init_phtw)(struct rcar_csi2 *priv, unsigned int mbps);
>  	int (*phy_post_init)(struct rcar_csi2 *priv);
> -	int (*start_receiver)(struct rcar_csi2 *priv);
> +	int (*start_receiver)(struct rcar_csi2 *priv,
> +			      struct v4l2_subdev_state *state);
>  	void (*enter_standby)(struct rcar_csi2 *priv);
>  	const struct rcsi2_mbps_reg *hsfreqrange;
>  	unsigned int csi0clkfreqrange;
> @@ -676,6 +684,32 @@ static int rcsi2_exit_standby(struct rcar_csi2 *priv)
>  	return 0;
>  }
>  
> +static int rcsi2_get_remote_frame_desc(struct rcar_csi2 *priv,
> +				       struct v4l2_mbus_frame_desc *fd)
> +{
> +	struct media_pad *pad;
> +	int ret;
> +
> +	if (!priv->remote)
> +		return -ENOLINK;
> +
> +	pad = media_pad_remote_pad_unique(&priv->pads[RCAR_CSI2_SINK]);
> +	if (IS_ERR(pad))
> +		return PTR_ERR(pad);
> +
> +	ret = v4l2_subdev_call(priv->remote, pad, get_frame_desc,
> +			       pad->index, fd);
> +	if (ret)
> +		return ret;
> +
> +	if (fd->type != V4L2_MBUS_FRAME_DESC_TYPE_CSI2) {
> +		dev_err(priv->dev, "Frame desc does not describe a CSI-2 link");
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +
>  static int rcsi2_wait_phy_start(struct rcar_csi2 *priv,
>  				unsigned int lanes)
>  {
> @@ -726,41 +760,6 @@ static int rcsi2_set_phypll(struct rcar_csi2 *priv, unsigned int mbps)
>  	return 0;
>  }
>  
> -static int rcsi2_calc_mbps(struct rcar_csi2 *priv, unsigned int bpp,
> -			   unsigned int lanes)
> -{
> -	struct v4l2_subdev *source;
> -	struct v4l2_ctrl *ctrl;
> -	u64 mbps;
> -
> -	if (!priv->remote)
> -		return -ENODEV;
> -
> -	source = priv->remote;
> -
> -	/* Read the pixel rate control from remote. */
> -	ctrl = v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_PIXEL_RATE);
> -	if (!ctrl) {
> -		dev_err(priv->dev, "no pixel rate control in subdev %s\n",
> -			source->name);
> -		return -EINVAL;
> -	}
> -
> -	/*
> -	 * Calculate the phypll in mbps.
> -	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> -	 * bps = link_freq * 2
> -	 */
> -	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * bpp;
> -	do_div(mbps, lanes * 1000000);
> -
> -	/* Adjust for C-PHY, divide by 2.8. */
> -	if (priv->cphy)
> -		mbps = div_u64(mbps * 5, 14);
> -
> -	return mbps;
> -}
> -
>  static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
>  				  unsigned int *lanes)
>  {
> @@ -808,52 +807,146 @@ static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
>  	return 0;
>  }
>  
> -static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
> +static int rcsi2_calc_mbps(struct rcar_csi2 *priv,
> +			   struct v4l2_mbus_frame_desc *fd, unsigned int lanes)
>  {

Is there a reason you need to move the rcsi2_calc_mbps()? No problem if 
so, else if you keep it where it is the diff is easier to read ;-)

> -	const struct rcar_csi2_format *format;
> +	struct v4l2_subdev *source;
> +	unsigned int bpp;
> +	s64 link_freq;
> +	u64 mbps;
> +
> +	if (!priv->remote)
> +		return -ENODEV;
> +
> +	source = priv->remote;
> +
> +	/*
> +	 * v4l2_get_link_freq() uses V4L2_CID_LINK_FREQ first, and falls back
> +	 * to V4L2_CID_PIXEL_RATE if V4L2_CID_LINK_FREQ is not available.
> +	 *
> +	 * With multistream input there is no single pixel rate, and thus we
> +	 * cannot use V4L2_CID_PIXEL_RATE, so we pass 0 as the bpp which
> +	 * causes v4l2_get_link_freq() to return an error if it falls back to
> +	 * V4L2_CID_PIXEL_RATE.
> +	 */
> +
> +	if (fd->num_entries > 1) {
> +		bpp = 0;
> +	} else {
> +		struct v4l2_mbus_frame_desc_entry *entry = &fd->entry[0];
> +		const struct rcar_csi2_format *format;
> +
> +		format = rcsi2_datatype_to_fmt(entry->bus.csi2.dt);
> +		if (WARN_ON(!format))
> +			return -EINVAL;
> +
> +		bpp = format->bpp;
> +	}
> +
> +	/*
> +	 * Calculate the phypll in mbps.
> +	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> +	 * bps = link_freq * 2
> +	 */
> +	link_freq = v4l2_get_link_freq(source->ctrl_handler, bpp, 2 * lanes);
> +	if (link_freq < 0) {
> +		dev_err(priv->dev, "Failed to get remote subdev link freq\n");
> +		return link_freq;
> +	}

I just want to make sure I understand our discussion about on IRC about 
this. It's this call you aim to replace by getting the link frequency 
from the frame descriptor once it's available there? If so I think it 
would be good to make this series depend on that work already and list 
it as a dependency.

> +
> +	mbps = 2 * link_freq;
> +	do_div(mbps, 1000000);
> +
> +	/* Adjust for C-PHY, divide by 2.8. */
> +	if (priv->cphy)
> +		mbps = div_u64(mbps * 5, 14);
> +
> +	return mbps;
> +}
> +
> +static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv,
> +				     struct v4l2_subdev_state *state)
> +{
> +	const struct v4l2_subdev_stream_configs *configs;
>  	u32 phycnt, vcdt = 0, vcdt2 = 0, fld = 0;
> +	struct v4l2_mbus_frame_desc fd;
>  	unsigned int lanes;
> -	unsigned int i;
>  	int mbps, ret;
>  
> -	dev_dbg(priv->dev, "Input size (%ux%u%c)\n",
> -		priv->mf.width, priv->mf.height,
> -		priv->mf.field == V4L2_FIELD_NONE ? 'p' : 'i');
> -
> -	/* Code is validated in set_fmt. */
> -	format = rcsi2_code_to_fmt(priv->mf.code);
> -	if (!format)
> -		return -EINVAL;
> +	/* Get information about multiplexed link. */
> +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> +	if (ret)
> +		return ret;
>  
>  	/*
> -	 * Enable all supported CSI-2 channels with virtual channel and
> -	 * data type matching.
> +	 * Configure and enable the R-Car CSI-2 channels.
>  	 *
> -	 * NOTE: It's not possible to get individual datatype for each
> -	 *       source virtual channel. Once this is possible in V4L2
> -	 *       it should be used here.
> +	 * The VC transmitted on the channel is configured by the [CSI-2->VIN]
> +	 * link_setup operation, while the data type to match comes from the
> +	 * remote subdevice.
>  	 */
> -	for (i = 0; i < priv->info->num_channels; i++) {
> -		u32 vcdt_part;
> +	for (unsigned int i = 0; i < priv->info->num_channels; i++) {
> +		struct v4l2_mbus_frame_desc_entry *entry = NULL;
>  
> +		/* CSI-2 channel disabled (not linked to any VIN). */
>  		if (priv->channel_vc[i] < 0)
>  			continue;
>  
> -		vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) | VCDT_VCDTN_EN |
> -			VCDT_SEL_DTN_ON | VCDT_SEL_DT(format->datatype);
> +		u32 vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) |
> +				VCDT_VCDTN_EN | VCDT_SEL_DTN_ON;
> +
> +		/*
> +		 * Search the entries that describe the data types on the
> +		 * MIPI CSI-2 Virtual Channel assigned to this CSI-2 channel.
> +		 */
> +		for (unsigned int e = 0; e < fd.num_entries; e++) {
> +			if (fd.entry[e].bus.csi2.vc != priv->channel_vc[i])
> +				continue;
> +
> +			/*
> +			 * If multiple data types are sent on the same MIPI
> +			 * CSI-2 Virtual Channel, disable Data Type matching.
> +			 */
> +			if (entry) {
> +				vcdt_part &= ~VCDT_SEL_DTN_ON;
> +				break;
> +			}
> +
> +			entry = &fd.entry[e];
> +			vcdt_part |= VCDT_SEL_DT(entry->bus.csi2.dt);
> +		}
> +
> +		if (!entry)
> +			continue;
>  
>  		/* Store in correct reg and offset. */
>  		if (i < 2)
>  			vcdt |= vcdt_part << ((i % 2) * 16);
>  		else
>  			vcdt2 |= vcdt_part << ((i % 2) * 16);
> +
> +		dev_dbg(priv->dev, "channel %u: VC = %d, datatype = 0x%x\n",
> +			i, priv->channel_vc[i], entry->bus.csi2.dt);
>  	}
>  
> -	if (priv->mf.field == V4L2_FIELD_ALTERNATE) {
> -		fld = FLD_DET_SEL(1) | FLD_FLD_EN4 | FLD_FLD_EN3 | FLD_FLD_EN2
> -			| FLD_FLD_EN;
> +	/*
> +	 * Configure field handling inspecting the formats of the
> +	 * sink pad streams.
> +	 */
> +	configs = &state->stream_configs;
> +	for (unsigned int i = 0; i < configs->num_configs; ++i) {
> +		const struct v4l2_subdev_stream_config *config = configs->configs;
> +
> +		if (config->pad != RCAR_CSI2_SINK)
> +			continue;
>  
> -		if (priv->mf.height == 240)
> +		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> +			continue;
> +
> +		fld |= FLD_DET_SEL(1) | FLD_FLD_EN(config->stream);
> +
> +		/* PAL vs NTSC. */
> +		if (config->fmt.height == 240)
>  			fld |= FLD_FLD_NUM(0);
>  		else
>  			fld |= FLD_FLD_NUM(1);
> @@ -870,7 +963,7 @@ static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
>  	phycnt = PHYCNT_ENABLECLK;
>  	phycnt |= (1 << lanes) - 1;
>  
> -	mbps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> +	mbps = rcsi2_calc_mbps(priv, &fd, lanes);
>  	if (mbps < 0)
>  		return mbps;
>  
> @@ -1049,23 +1142,24 @@ static int rcsi2_c_phy_setting_v4h(struct rcar_csi2 *priv, int msps)
>  	return 0;
>  }
>  
> -static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
> +static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv,
> +				    struct v4l2_subdev_state *state)
>  {
> -	const struct rcar_csi2_format *format;
> +	struct v4l2_mbus_frame_desc fd;
>  	unsigned int lanes;
>  	int msps;
>  	int ret;
>  
>  	/* Calculate parameters */
> -	format = rcsi2_code_to_fmt(priv->mf.code);
> -	if (!format)
> -		return -EINVAL;
> -
>  	ret = rcsi2_get_active_lanes(priv, &lanes);
>  	if (ret)
>  		return ret;
>  
> -	msps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> +	if (ret)
> +		return ret;
> +
> +	msps = rcsi2_calc_mbps(priv, &fd, lanes);
>  	if (msps < 0)
>  		return msps;
>  
> @@ -1114,7 +1208,7 @@ static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
>  	return 0;
>  }
>  
> -static int rcsi2_start(struct rcar_csi2 *priv)
> +static int rcsi2_start(struct rcar_csi2 *priv, struct v4l2_subdev_state *state)
>  {
>  	int ret;
>  
> @@ -1122,7 +1216,7 @@ static int rcsi2_start(struct rcar_csi2 *priv)
>  	if (ret < 0)
>  		return ret;
>  
> -	ret = priv->info->start_receiver(priv);
> +	ret = priv->info->start_receiver(priv, state);
>  	if (ret) {
>  		rcsi2_enter_standby(priv);
>  		return ret;
> @@ -1146,26 +1240,24 @@ static void rcsi2_stop(struct rcar_csi2 *priv)
>  static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
>  {
>  	struct rcar_csi2 *priv = sd_to_csi2(sd);
> +	struct v4l2_subdev_state *state;
>  	int ret = 0;
>  
> -	mutex_lock(&priv->lock);
> +	if (!priv->remote)
> +		return -ENODEV;
>  
> -	if (!priv->remote) {
> -		ret = -ENODEV;
> -		goto out;
> -	}
> +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
>  
> -	if (enable && priv->stream_count == 0) {
> -		ret = rcsi2_start(priv);
> -		if (ret)
> -			goto out;
> -	} else if (!enable && priv->stream_count == 1) {
> +	if (enable && priv->stream_count == 0)
> +		ret = rcsi2_start(priv, state);
> +	else if (!enable && priv->stream_count == 1)
>  		rcsi2_stop(priv);
> -	}
> +	if (ret)
> +		goto out;
>  
>  	priv->stream_count += enable ? 1 : -1;
>  out:
> -	mutex_unlock(&priv->lock);
> +	v4l2_subdev_unlock_state(state);
>  
>  	return ret;
>  }
> @@ -1310,14 +1402,17 @@ static irqreturn_t rcsi2_irq(int irq, void *data)
>  
>  static irqreturn_t rcsi2_irq_thread(int irq, void *data)
>  {
> +	struct v4l2_subdev_state *state;
>  	struct rcar_csi2 *priv = data;
>  
> -	mutex_lock(&priv->lock);
> +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
> +
>  	rcsi2_stop(priv);
>  	usleep_range(1000, 2000);
> -	if (rcsi2_start(priv))
> +	if (rcsi2_start(priv, state))
>  		dev_warn(priv->dev, "Failed to restart CSI-2 receiver\n");
> -	mutex_unlock(&priv->lock);
> +
> +	v4l2_subdev_unlock_state(state);
>  
>  	return IRQ_HANDLED;
>  }
> @@ -1340,6 +1435,13 @@ static int rcsi2_notify_bound(struct v4l2_async_notifier *notifier,
>  		return pad;
>  	}
>  
> +	if (!v4l2_subdev_has_op(subdev, pad, get_frame_desc)) {
> +		dev_err(priv->dev,
> +			"Subdev %s bound failed: missing get_frame_desc()\n",
> +			subdev->name);
> +		return -EINVAL;
> +	}
> +
>  	priv->remote = subdev;
>  	priv->remote_pad = pad;
>  
> -- 
> 2.44.0
> 

-- 
Kind Regards,
Niklas Söderlund

Re: [PATCH 17/19] media: rcar-csi2: Store format in the subdev state

[Niklas Söderlund]

Hi Jacpop,

Thanks for your work.

On 2024-04-30 12:39:53 +0200, Jacopo Mondi wrote:
> Store the format in the subdevice state. Disallow setting format
> on the source pads, as formats are set on the sink pad streams and
> propagated to the source streams.
> 
> Now that the driver doesn't store the active format in the
> driver-specific structure, also remove the mutex and use the lock
> associated with the state.

Can't this whole patch be broken out to an independent patch and 
upstreamed already independent from the streams work?

> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/platform/renesas/rcar-csi2.c | 54 +++++++---------------
>  1 file changed, 16 insertions(+), 38 deletions(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> index ffb73272543b..ed818a6fa655 100644
> --- a/drivers/media/platform/renesas/rcar-csi2.c
> +++ b/drivers/media/platform/renesas/rcar-csi2.c
> @@ -621,8 +621,6 @@ struct rcar_csi2 {
>  
>  	int channel_vc[4];
>  
> -	struct mutex lock; /* Protects mf and stream_count. */
> -	struct v4l2_mbus_framefmt mf;
>  	int stream_count;
>  
>  	bool cphy;
> @@ -1263,43 +1261,28 @@ static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
>  }
>  
>  static int rcsi2_set_pad_format(struct v4l2_subdev *sd,
> -				struct v4l2_subdev_state *sd_state,
> +				struct v4l2_subdev_state *state,
>  				struct v4l2_subdev_format *format)
>  {
> -	struct rcar_csi2 *priv = sd_to_csi2(sd);
> -	struct v4l2_mbus_framefmt *framefmt;
> +	struct v4l2_mbus_framefmt *fmt;
>  
> -	mutex_lock(&priv->lock);
> +	/*
> +	 * Format is propagated from sink streams to source streams, so
> +	 * disallow setting format on the source pads.
> +	 */
> +	if (format->pad > RCAR_CSI2_SINK)
> +		return -EINVAL;
>  
>  	if (!rcsi2_code_to_fmt(format->format.code))
>  		format->format.code = rcar_csi2_formats[0].code;
>  
> -	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> -		priv->mf = format->format;
> -	} else {
> -		framefmt = v4l2_subdev_state_get_format(sd_state, 0);
> -		*framefmt = format->format;
> -	}
>  
> -	mutex_unlock(&priv->lock);
> +	fmt = v4l2_subdev_state_get_format(state, format->pad, format->stream);
> +	*fmt = format->format;
>  
> -	return 0;
> -}
> -
> -static int rcsi2_get_pad_format(struct v4l2_subdev *sd,
> -				struct v4l2_subdev_state *sd_state,
> -				struct v4l2_subdev_format *format)
> -{
> -	struct rcar_csi2 *priv = sd_to_csi2(sd);
> -
> -	mutex_lock(&priv->lock);
> -
> -	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE)
> -		format->format = priv->mf;
> -	else
> -		format->format = *v4l2_subdev_state_get_format(sd_state, 0);
> -
> -	mutex_unlock(&priv->lock);
> +	fmt = v4l2_subdev_state_get_opposite_stream_format(state, format->pad,
> +							   format->stream);
> +	*fmt = format->format;
>  
>  	return 0;
>  }
> @@ -1310,7 +1293,7 @@ static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
>  
>  static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
>  	.set_fmt = rcsi2_set_pad_format,
> -	.get_fmt = rcsi2_get_pad_format,
> +	.get_fmt = v4l2_subdev_get_fmt,
>  };
>  
>  static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
> @@ -2031,20 +2014,19 @@ static int rcsi2_probe(struct platform_device *pdev)
>  
>  	priv->dev = &pdev->dev;
>  
> -	mutex_init(&priv->lock);
>  	priv->stream_count = 0;
>  
>  	ret = rcsi2_probe_resources(priv, pdev);
>  	if (ret) {
>  		dev_err(priv->dev, "Failed to get resources\n");
> -		goto error_mutex;
> +		return ret;
>  	}
>  
>  	platform_set_drvdata(pdev, priv);
>  
>  	ret = rcsi2_parse_dt(priv);
>  	if (ret)
> -		goto error_mutex;
> +		return ret;
>  
>  	priv->subdev.owner = THIS_MODULE;
>  	priv->subdev.dev = &pdev->dev;
> @@ -2094,8 +2076,6 @@ static int rcsi2_probe(struct platform_device *pdev)
>  error_async:
>  	v4l2_async_nf_unregister(&priv->notifier);
>  	v4l2_async_nf_cleanup(&priv->notifier);
> -error_mutex:
> -	mutex_destroy(&priv->lock);
>  
>  	return ret;
>  }
> @@ -2110,8 +2090,6 @@ static void rcsi2_remove(struct platform_device *pdev)
>  	v4l2_subdev_cleanup(&priv->subdev);
>  
>  	pm_runtime_disable(&pdev->dev);
> -
> -	mutex_destroy(&priv->lock);
>  }
>  
>  static struct platform_driver rcar_csi2_pdrv = {
> -- 
> 2.44.0
> 

-- 
Kind Regards,
Niklas Söderlund

Re: [PATCH 19/19] media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2

[Niklas Söderlund]

Hi Jacopo,

Same for this patch, can't we fix this issue independent from the 
streams work?

On 2024-04-30 12:39:55 +0200, Jacopo Mondi wrote:
> The YUYV8_1X16 and UYVY8_1X16 formats are treated as 'ITU-R
> BT.601/BT.1358 16-bit YCbCr-422 input' (YUV16 - 0x5) in the R-Car VIN
> driver and are thus disallowed when capturing frames from the R-Car
> CSI-2 interface according to the hardware manual.
> 
> As the 1X16 format variants are meant to be used with serial busses they
> have to be treated as 'YCbCr-422 8-bit data input' (0x1) when capturing
> from CSI-2, which is a valid setting for CSI-2.
> 
> Commit 78b3f9d75a62 ("media: rcar-vin: Add check that input interface
> and format are valid") disallowed capturing YUV16 when using the CSI-2
> interface. Fix this by using YUV8_BT601 for YCbCr422 when CSI-2 is in
> use.
> 
> Fixes: 78b3f9d75a62 ("media: rcar-vin: Add check that input interface and format are valid")
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  .../media/platform/renesas/rcar-vin/rcar-dma.c   | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> index e2c40abc6d3d..21d5b2815e86 100644
> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> @@ -742,12 +742,22 @@ static int rvin_setup(struct rvin_dev *vin)
>  	 */
>  	switch (vin->mbus_code) {
>  	case MEDIA_BUS_FMT_YUYV8_1X16:
> -		/* BT.601/BT.1358 16bit YCbCr422 */
> -		vnmc |= VNMC_INF_YUV16;
> +		if (vin->is_csi)
> +			/* YCbCr422 8-bit */
> +			vnmc |= VNMC_INF_YUV8_BT601;
> +		else
> +			/* BT.601/BT.1358 16bit YCbCr422 */
> +			vnmc |= VNMC_INF_YUV16;
>  		input_is_yuv = true;
>  		break;
>  	case MEDIA_BUS_FMT_UYVY8_1X16:
> -		vnmc |= VNMC_INF_YUV16 | VNMC_YCAL;
> +		if (vin->is_csi)
> +			/* YCbCr422 8-bit */
> +			vnmc |= VNMC_INF_YUV8_BT601;
> +		else
> +			/* BT.601/BT.1358 16bit YCbCr422 */
> +			vnmc |= VNMC_INF_YUV16;
> +		vnmc |= VNMC_YCAL;
>  		input_is_yuv = true;
>  		break;
>  	case MEDIA_BUS_FMT_UYVY8_2X8:
> -- 
> 2.44.0
> 

-- 
Kind Regards,
Niklas Söderlund

Re: [PATCH 01/19] media: adv748x: Add support for active state

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:37PM +0200, Jacopo Mondi wrote:
> Initialize and use the subdev active state to store the subdevice
> format.
> 
> This simplifies the implementation of the get_fmt and set_fmt pad
> operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 69 ++++--------------------
>  drivers/media/i2c/adv748x/adv748x.h      |  1 -
>  2 files changed, 11 insertions(+), 59 deletions(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index 5b265b722394..435b0909bbef 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -139,78 +139,26 @@ static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
>   * But we must support setting the pad formats for format propagation.
>   */
>  
> -static struct v4l2_mbus_framefmt *
> -adv748x_csi2_get_pad_format(struct v4l2_subdev *sd,
> -			    struct v4l2_subdev_state *sd_state,
> -			    unsigned int pad, u32 which)
> -{
> -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> -
> -	if (which == V4L2_SUBDEV_FORMAT_TRY)
> -		return v4l2_subdev_state_get_format(sd_state, pad);
> -
> -	return &tx->format;
> -}
> -
> -static int adv748x_csi2_get_format(struct v4l2_subdev *sd,
> -				   struct v4l2_subdev_state *sd_state,
> -				   struct v4l2_subdev_format *sdformat)
> -{
> -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> -	struct adv748x_state *state = tx->state;
> -	struct v4l2_mbus_framefmt *mbusformat;
> -
> -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> -						 sdformat->which);
> -	if (!mbusformat)
> -		return -EINVAL;
> -
> -	mutex_lock(&state->mutex);
> -
> -	sdformat->format = *mbusformat;
> -
> -	mutex_unlock(&state->mutex);
> -
> -	return 0;
> -}
> -
>  static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_format *sdformat)
>  {
> -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> -	struct adv748x_state *state = tx->state;
>  	struct v4l2_mbus_framefmt *mbusformat;
> -	int ret = 0;
> -
> -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> -						 sdformat->which);
> -	if (!mbusformat)
> -		return -EINVAL;
>  
> -	mutex_lock(&state->mutex);
> +	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
>  
> +	/* Format on the source pad is always copied from the sink one. */
>  	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
>  		const struct v4l2_mbus_framefmt *sink_fmt;
>  
> -		sink_fmt = adv748x_csi2_get_pad_format(sd, sd_state,
> -						       ADV748X_CSI2_SINK,
> -						       sdformat->which);
> -
> -		if (!sink_fmt) {
> -			ret = -EINVAL;
> -			goto unlock;
> -		}
> -
> +		sink_fmt = v4l2_subdev_state_get_format(sd_state,
> +							ADV748X_CSI2_SINK);
>  		sdformat->format = *sink_fmt;

That's not the right way to do it. You should propagate the format from
sink to source when pad == ADV748X_CSI2_SINK, and return
adv748x_csi2_get_format() when pad == ADV748X_CSI2_SOURCE. Otherwise
setting the format on the sink pad will not update the state of the
source pad, and a get format call on the source pad will return an
incorrect format.

>  	}
>  
>  	*mbusformat = sdformat->format;
>  
> -unlock:
> -	mutex_unlock(&state->mutex);
> -
> -	return ret;
> +	return 0;
>  }
>  
>  static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
> @@ -228,7 +176,7 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
>  }
>  
>  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
> -	.get_fmt = adv748x_csi2_get_format,
> +	.get_fmt = v4l2_subdev_get_fmt,
>  	.set_fmt = adv748x_csi2_set_format,
>  	.get_mbus_config = adv748x_csi2_get_mbus_config,
>  };
> @@ -320,6 +268,11 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
>  	if (ret)
>  		goto err_cleanup_subdev;
>  
> +	tx->sd.state_lock = tx->ctrl_hdl.lock;

Maybe that's addressed in subsequent patches, but do we need a
device-wide lock ? The code you replace above uses the
adv748x_state.mutex lock, which covers all subdevs. I don't think this
patch introduces race conditions, so this could possibly be handled on
top.

> +	ret = v4l2_subdev_init_finalize(&tx->sd);
> +	if (ret)
> +		goto err_free_ctrl;
> +
>  	ret = v4l2_async_register_subdev(&tx->sd);
>  	if (ret)
>  		goto err_free_ctrl;
> diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> index d2b5e722e997..9bc0121d0eff 100644
> --- a/drivers/media/i2c/adv748x/adv748x.h
> +++ b/drivers/media/i2c/adv748x/adv748x.h
> @@ -75,7 +75,6 @@ enum adv748x_csi2_pads {
>  
>  struct adv748x_csi2 {
>  	struct adv748x_state *state;
> -	struct v4l2_mbus_framefmt format;
>  	unsigned int page;
>  	unsigned int port;
>  	unsigned int num_lanes;

-- 
Regards,

Laurent Pinchart

Re: [PATCH 00/19] media: renesas: rcar-csi2: Support multiple streams

[Laurent Pinchart]

On Tue, Apr 30, 2024 at 01:51:13PM +0200, Jacopo Mondi wrote:
> Hi Niklas
>   thanks for the quick reply
> 
> On Tue, Apr 30, 2024 at 01:17:18PM +0200, Niklas Söderlund wrote:
> > Hi Jacopo,
> >
> > Thanks for this work! I'm happy that we finally can get this upstream. I
> > will test and review this in the coming days. As there appears to be a
> > few dependencies to get this to work do you have a branch covering
> > everything needed?
> 
> Sure
> https://git.kernel.org/pub/scm/linux/kernel/git/jmondi/linux.git/
> jmondi/renesas-drivers-2024-04-23-v6.9-rc5/multistream
> 
> I should have added it to the cover letter
> 
> > On 2024-04-30 12:39:36 +0200, Jacopo Mondi wrote:
> > > Hello this series implements multi-stream support for R-Car CSI-2 and the
> > > two CSI-2 devices that are available on Renesas dev-boards which are mainline
> > > supported (the ADV748x HDMI decoder and the MAX9286 GMSL deserializer).
> >
> > You also need to update the MAX96712 driver in staging else V3U and Gen4
> > will break.
> >
> 
> Ouch, right. I can do that but won't be able to test. Would you ?
> 
> > > The net gain in feature is:
> > > - ADV748x: Select on which MIPI CSI-2 VC to send the stream on
> > > - MAX9286: Paves the way for run-time selection of active links
> > >
> > > Both features are realized by controlling the devices routing tables.
> > > The max9286 routing support is not implemented, but these patches are needed
> > > in order to maintain the current version compatible with R-Car CSI-2.
> > >
> > > The R-Car CSI-2 driver needs to be updated in order to support multiplexed
> > > transmitters, in particular:
> > >
> > > - Use LINK_FREQ to compute the lane bandwidth
> > > - Support subdev_active state
> > > - Configure DT and VC handling by using the remote's frame_desc
> > >
> > > A separate fix for VIN sits at the top of the series to re-enable YUYV capture
> > > operations.
> > >
> > > The series needs to activate streams in v4l2-subdev to work correctly:
> > >
> > > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > > @@ -32,7 +32,7 @@
> > >   * 'v4l2_subdev_enable_streams_api' to 1 below.
> > >   */
> > >
> > > -static bool v4l2_subdev_enable_streams_api;
> > > +static bool v4l2_subdev_enable_streams_api = 1;
> > >  #endif
> >
> > I'm not up to date with the latest state of the multiple streams API.
> > Can this be runtime configured or will this break capture if the kernel
> > is not patched to enable this?
> 
> I'm afraid the latter. I really hope in the next kernel release we can
> get rid of this protection and enable streams by default.

I hope it will be done for v6.12 or possibly v6.11.

> > >
> > > Test scripts, based on vin-tests are available at:
> > > https://git.uk.ideasonboard.com/jmondi/vin-tests
> > >
> > > A note for Sakari: patch
> > > "media: max9286: Implement support for LINK_FREQ"
> > >
> > > Could actually use your new
> > > "media: v4l: Support passing sub-device argument to v4l2_get_link_freq()"
> > >
> > > I kept it as it is for this version waiting for your patch to get in.
> >
> > I like this idea.
> >
> > >
> > > A note for Niklas:
> > > I think this series paves the way to actually moving the control of which
> > > VC goes to which VIN using a routing table instead of using link enablement.
> > >
> > > Currently the setup is as it follows: routing within R-Car CSI-2 is fixed, sink
> > > stream 0/X goes to source stream X+1/0 and each source pad represents a VC.
> > > Which VC gets sent to which VIN is controlled by link enablement.
> > >
> > > A more natural way of handling this would be to make the routing table
> > > within CSI-2 configurable and have media-links immutable between the CSI-2
> > > source pad. A CSI-2 source pad would then represent a 'channel' between
> > > CSI-2 and VIN and not the VC which is sent on it.
> > >
> > > This setup would better represent the current implementation, with VIN's VCSEL
> > > fixed channel routing and with the CSI-2 routing table controlling which VC gets
> > > sent on which channel. What do you think ?
> >
> > That could be useful. Maybe we can even switch to use immutable links
> > and use the routing instead greatly simplifying the VIN driver as this
> > together with the other VIN series on the list we can drop the ugly
> > group concept ;-)
> 
> I had the same idea, but I'm not yet sure how this would work when a
> CSI-2 can be routed to multiple VIN groups...
> 
> > But this series is already quiet large, I think we can work on this
> > on-top in a new series.
> 
> Exactly, I didn't want to pile too many things. max9286 routing would
> be next in line, I kept it out to avoid sending 30 patches in one go
> (and to send this out earlier to get feedbacks)
> 
> > > Jacopo Mondi (19):
> > >   media: adv748x: Add support for active state
> > >   media: adv748x: Add flags to adv748x_subdev_init()
> > >   media: adv748x: Use V4L2 streams
> > >   media: adv748x: Propagate format to opposite stream
> > >   media: adv748x: Implement set_routing()
> > >   media: adv748x: Use routes to configure VC
> > >   media: adv748x: Implement .get_frame_desc()
> > >   media: max9286: Add support for subdev active state
> > >   media: max9286: Fix enum_mbus_code
> > >   media: max9286: Use frame interval from subdev state
> > >   media: max9286: Use V4L2 Streams
> > >   media: max9286: Implement .get_frame_desc()
> > >   media: max9286: Implement support for LINK_FREQ
> > >   media: max9286: Implement .get_mbus_config()
> > >   media: rcar-csi2: Add support for multiplexed streams
> > >   media: rcar-csi2: Support multiplexed transmitters
> > >   media: rcar-csi2: Store format in the subdev state
> > >   media: rcar-csi2: Implement set_routing
> > >   media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2
> > >
> > >  drivers/media/i2c/adv748x/adv748x-afe.c       |   2 +-
> > >  drivers/media/i2c/adv748x/adv748x-core.c      |  12 +-
> > >  drivers/media/i2c/adv748x/adv748x-csi2.c      | 251 +++++++---
> > >  drivers/media/i2c/adv748x/adv748x-hdmi.c      |   2 +-
> > >  drivers/media/i2c/adv748x/adv748x.h           |   4 +-
> > >  drivers/media/i2c/max9286.c                   | 457 ++++++++++++------
> > >  drivers/media/platform/renesas/rcar-csi2.c    | 442 ++++++++++++-----
> > >  .../platform/renesas/rcar-vin/rcar-dma.c      |  16 +-
> > >  8 files changed, 834 insertions(+), 352 deletions(-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH 02/19] media: adv748x: Add flags to adv748x_subdev_init()

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:38PM +0200, Jacopo Mondi wrote:
> Add a flags parameter to the adv748x_subdev_init() function that
> allows to pass additional flags to the v4l2_subdevice.
> 
> This will be used to identify the CSI-2 subdevices as multiplexed.

"as supporting streams" ?

> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-afe.c  | 2 +-
>  drivers/media/i2c/adv748x/adv748x-core.c | 4 ++--
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 2 +-
>  drivers/media/i2c/adv748x/adv748x-hdmi.c | 2 +-
>  drivers/media/i2c/adv748x/adv748x.h      | 2 +-
>  5 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-afe.c b/drivers/media/i2c/adv748x/adv748x-afe.c
> index 50d9fbadbe38..5951ddebc59d 100644
> --- a/drivers/media/i2c/adv748x/adv748x-afe.c
> +++ b/drivers/media/i2c/adv748x/adv748x-afe.c
> @@ -510,7 +510,7 @@ int adv748x_afe_init(struct adv748x_afe *afe)
>  	afe->curr_norm = V4L2_STD_NTSC_M;
>  
>  	adv748x_subdev_init(&afe->sd, state, &adv748x_afe_ops,
> -			    MEDIA_ENT_F_ATV_DECODER, "afe");
> +			    MEDIA_ENT_F_ATV_DECODER, 0, "afe");
>  
>  	/* Identify the first connector found as a default input if set */
>  	for (i = ADV748X_PORT_AIN0; i <= ADV748X_PORT_AIN7; i++) {
> diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
> index 3eb6d5e8f082..3abc73ea8ccb 100644
> --- a/drivers/media/i2c/adv748x/adv748x-core.c
> +++ b/drivers/media/i2c/adv748x/adv748x-core.c
> @@ -583,10 +583,10 @@ static int __maybe_unused adv748x_resume_early(struct device *dev)
>  
>  void adv748x_subdev_init(struct v4l2_subdev *sd, struct adv748x_state *state,
>  			 const struct v4l2_subdev_ops *ops, u32 function,
> -			 const char *ident)
> +			 u32 flags, const char *ident)

I'd name the argument sd_flags to make it clearer. Apart from that,

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

>  {
>  	v4l2_subdev_init(sd, ops);
> -	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	sd->flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | flags;
>  
>  	/* the owner is the same as the i2c_client's driver owner */
>  	sd->owner = state->dev->driver->owner;
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index 435b0909bbef..60bf1dc0f58b 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -245,7 +245,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
>  		return 0;
>  
>  	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
> -			    MEDIA_ENT_F_VID_IF_BRIDGE,
> +			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
>  			    is_txa(tx) ? "txa" : "txb");
>  
>  	/* Register internal ops for incremental subdev registration */
> diff --git a/drivers/media/i2c/adv748x/adv748x-hdmi.c b/drivers/media/i2c/adv748x/adv748x-hdmi.c
> index a4db9bae5f79..9e4a5aa4ed55 100644
> --- a/drivers/media/i2c/adv748x/adv748x-hdmi.c
> +++ b/drivers/media/i2c/adv748x/adv748x-hdmi.c
> @@ -741,7 +741,7 @@ int adv748x_hdmi_init(struct adv748x_hdmi *hdmi)
>  	hdmi->aspect_ratio.denominator = 9;
>  
>  	adv748x_subdev_init(&hdmi->sd, state, &adv748x_ops_hdmi,
> -			    MEDIA_ENT_F_IO_DTV, "hdmi");
> +			    MEDIA_ENT_F_IO_DTV, 0, "hdmi");
>  
>  	hdmi->pads[ADV748X_HDMI_SINK].flags = MEDIA_PAD_FL_SINK;
>  	hdmi->pads[ADV748X_HDMI_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
> diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> index 9bc0121d0eff..be24bc57767c 100644
> --- a/drivers/media/i2c/adv748x/adv748x.h
> +++ b/drivers/media/i2c/adv748x/adv748x.h
> @@ -424,7 +424,7 @@ static inline struct v4l2_subdev *adv748x_get_remote_sd(struct media_pad *pad)
>  
>  void adv748x_subdev_init(struct v4l2_subdev *sd, struct adv748x_state *state,
>  			 const struct v4l2_subdev_ops *ops, u32 function,
> -			 const char *ident);
> +			 u32 flags, const char *ident);
>  
>  int adv748x_tx_power(struct adv748x_csi2 *tx, bool on);
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH 03/19] media: adv748x: Use V4L2 streams

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:39PM +0200, Jacopo Mondi wrote:
> Initialize the CSI-2 subdevice with the V4L2_SUBDEV_FL_STREAMS flag
> and initialize a simple routing table by implementing the .init_state()
> operation.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 28 ++++++++++++++++++++++--
>  1 file changed, 26 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index 60bf1dc0f58b..d929db7e8ef2 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -59,7 +59,30 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
>  
>  /* -----------------------------------------------------------------------------
>   * v4l2_subdev_internal_ops
> - *
> + */
> +
> +static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
> +				   struct v4l2_subdev_state *state)
> +{
> +	struct v4l2_subdev_route routes[] = {
> +		{
> +			.sink_pad = ADV748X_CSI2_SINK,
> +			.sink_stream = 0,
> +			.source_pad = ADV748X_CSI2_SOURCE,
> +			.source_stream = 0,
> +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> +		},
> +	};
> +
> +	struct v4l2_subdev_krouting routing = {
> +		.num_routes = ARRAY_SIZE(routes),
> +		.routes = routes,
> +	};
> +
> +	return v4l2_subdev_set_routing(sd, state, &routing);

You need to initialize formats too.

> +}
> +
> +/*
>   * We use the internal registered operation to be able to ensure that our
>   * incremental subdevices (not connected in the forward path) can be registered
>   * against the resulting video path and media device.
> @@ -109,6 +132,7 @@ static int adv748x_csi2_registered(struct v4l2_subdev *sd)
>  }
>  
>  static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
> +	.init_state = adv748x_csi2_init_state,

The .init_state() operation needs to be provided along with the call to
v4l2_subdev_init_finalize() in patch 01/19.

>  	.registered = adv748x_csi2_registered,
>  };
>  
> @@ -245,7 +269,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
>  		return 0;
>  
>  	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
> -			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
> +			    MEDIA_ENT_F_VID_IF_BRIDGE, V4L2_SUBDEV_FL_STREAMS,
>  			    is_txa(tx) ? "txa" : "txb");
>  
>  	/* Register internal ops for incremental subdev registration */
> -- 
> 2.44.0
> 

-- 
Regards,

Laurent Pinchart

Re: [PATCH 04/19] media: adv748x: Propagate format to opposite stream

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:40PM +0200, Jacopo Mondi wrote:
> Now that the adv748x-csi2 driver supports streams and routing, when a
> format is set on the sink pad, it gets propagated to the connected stream
> on the source pad.

As commented in the review of 01/19, this change belongs to that patch.

> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 23 ++++++++++++-----------
>  1 file changed, 12 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index d929db7e8ef2..ace4e1d904d9 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -167,20 +167,21 @@ static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
>  				   struct v4l2_subdev_state *sd_state,
>  				   struct v4l2_subdev_format *sdformat)
>  {
> -	struct v4l2_mbus_framefmt *mbusformat;
> +	struct v4l2_mbus_framefmt *fmt;
>  
> -	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
> -
> -	/* Format on the source pad is always copied from the sink one. */
> -	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
> -		const struct v4l2_mbus_framefmt *sink_fmt;
> +	/*
> +	 * The format set on the sink pad is propagated to the other end
> +	 * of the active route.
> +	 */
> +	if (sdformat->pad == ADV748X_CSI2_SOURCE)
> +		return -EINVAL;
>  
> -		sink_fmt = v4l2_subdev_state_get_format(sd_state,
> -							ADV748X_CSI2_SINK);
> -		sdformat->format = *sink_fmt;
> -	}
> +	fmt = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
> +	*fmt = sdformat->format;
>  
> -	*mbusformat = sdformat->format;
> +	fmt = v4l2_subdev_state_get_opposite_stream_format(sd_state,
> +							   sdformat->pad, 0);
> +	*fmt = sdformat->format;
>  
>  	return 0;
>  }

-- 
Regards,

Laurent Pinchart

Re: [PATCH 05/19] media: adv748x: Implement set_routing()

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:41PM +0200, Jacopo Mondi wrote:
> Implement the set_routing() pad operation to control the MIPI CSI-2
> Virtual Channel on which the video stream is sent on according to
> the active route source stream number.

While 01/19 needs to implement .init_state(), you should only initialize
formats there. The routing initialization of 03/19 should be moved to
this patch.

> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 43 +++++++++++++++++++++++-
>  1 file changed, 42 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index ace4e1d904d9..7fa72340e66e 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -57,6 +57,38 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
>  	return 0;
>  }
>  
> +static int adv748x_csi2_apply_routing(struct v4l2_subdev *sd,
> +				      struct v4l2_subdev_state *state,
> +				      struct v4l2_subdev_krouting *routing)
> +{
> +	struct v4l2_subdev_route *route;
> +	int ret;
> +
> +	/* Only one route at the time can be active. */

s/the time/a time/

> +	if (routing->num_routes > 1)
> +		return -EINVAL;

You should adjust routes instead of returning -EINVAL.

> +
> +	/*
> +	 * Validate the route: sink pad and sink stream shall be 0 and only
> +	 * 4 source streams are supported (one for each supported MIPI CSI-2
> +	 * channel).

s/channel/virtual channel/

> +	 */
> +	route = &routing->routes[0];
> +
> +	if (route->sink_pad != ADV748X_CSI2_SINK || route->sink_stream)
> +		return -EINVAL;
> +	if (route->source_pad != ADV748X_CSI2_SOURCE ||
> +	    route->source_stream > 4)
> +		return -EINVAL;

Adjust instead of returning an error. The pad checks can be dropped, as
the core ensures sink_pad and source_pad reference a valid sink and
source pad respectively.

I'm not sure the source stream check is right either. I understand
you'll use that to select a virtual channel, but the routing API isn't
meant to let userspace configure virtual channel numbers explicitly.

> +
> +	ret = v4l2_subdev_routing_validate(sd, routing,
> +					   V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_subdev_set_routing(sd, state, routing);
> +}
> +
>  /* -----------------------------------------------------------------------------
>   * v4l2_subdev_internal_ops
>   */
> @@ -79,7 +111,7 @@ static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
>  		.routes = routes,
>  	};
>  
> -	return v4l2_subdev_set_routing(sd, state, &routing);
> +	return adv748x_csi2_apply_routing(sd, state, &routing);
>  }
>  
>  /*
> @@ -200,10 +232,19 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
>  	return 0;
>  }
>  
> +static int adv748x_csi2_set_routing(struct v4l2_subdev *sd,
> +				    struct v4l2_subdev_state *state,
> +				    enum v4l2_subdev_format_whence which,
> +				    struct v4l2_subdev_krouting *routing)
> +{
> +	return adv748x_csi2_apply_routing(sd, state, routing);
> +}
> +
>  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
>  	.get_fmt = v4l2_subdev_get_fmt,
>  	.set_fmt = adv748x_csi2_set_format,
>  	.get_mbus_config = adv748x_csi2_get_mbus_config,
> +	.set_routing = adv748x_csi2_set_routing,
>  };
>  
>  /* -----------------------------------------------------------------------------

-- 
Regards,

Laurent Pinchart

Re: [PATCH 06/19] media: adv748x: Use routes to configure VC

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:42PM +0200, Jacopo Mondi wrote:
> Use the newly introduced routing table to configure on which MIPI
> CSI-2 Virtual Channel to send the image stream on.

The stream ID in the routing API isn't meant to be mapped directly to a
virtual channel number.

Sakari, your opinion would be appreciated.

> Perform Virtual Channel selection at s_stream() time instead of
> forcing it to 0 during the chip reset.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-core.c |  8 ++------
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 22 ++++++++++++++++++++--
>  drivers/media/i2c/adv748x/adv748x.h      |  1 -
>  3 files changed, 22 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
> index 3abc73ea8ccb..c9d917135709 100644
> --- a/drivers/media/i2c/adv748x/adv748x-core.c
> +++ b/drivers/media/i2c/adv748x/adv748x-core.c
> @@ -530,14 +530,10 @@ static int adv748x_reset(struct adv748x_state *state)
>  	io_write(state, ADV748X_IO_PD, ADV748X_IO_PD_RX_EN);
>  
>  	/* Conditionally enable TXa and TXb. */
> -	if (is_tx_enabled(&state->txa)) {
> +	if (is_tx_enabled(&state->txa))
>  		regval |= ADV748X_IO_10_CSI4_EN;
> -		adv748x_csi2_set_virtual_channel(&state->txa, 0);
> -	}
> -	if (is_tx_enabled(&state->txb)) {
> +	if (is_tx_enabled(&state->txb))
>  		regval |= ADV748X_IO_10_CSI1_EN;
> -		adv748x_csi2_set_virtual_channel(&state->txb, 0);
> -	}
>  	io_write(state, ADV748X_IO_10, regval);
>  
>  	/* Use vid_std and v_freq as freerun resolution for CP */
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index 7fa72340e66e..a7bfed393ff0 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -14,7 +14,8 @@
>  
>  #include "adv748x.h"
>  
> -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc)
> +static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
> +					    unsigned int vc)
>  {
>  	return tx_write(tx, ADV748X_CSI_VC_REF, vc << ADV748X_CSI_VC_REF_SHIFT);
>  }
> @@ -175,13 +176,30 @@ static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
>  static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, int enable)
>  {
>  	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> +	struct v4l2_subdev_state *state;
>  	struct v4l2_subdev *src;
> +	int ret;
>  
>  	src = adv748x_get_remote_sd(&tx->pads[ADV748X_CSI2_SINK]);
>  	if (!src)
>  		return -EPIPE;
>  
> -	return v4l2_subdev_call(src, video, s_stream, enable);
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +
> +	if (enable) {
> +		/* A single route is available. */
> +		struct v4l2_subdev_route *route = &state->routing.routes[0];
> +
> +		ret = adv748x_csi2_set_virtual_channel(tx, route->source_stream);
> +		if (ret)
> +			goto unlock;
> +	}
> +
> +	ret = v4l2_subdev_call(src, video, s_stream, enable);
> +unlock:
> +	v4l2_subdev_unlock_state(state);
> +
> +	return ret;
>  }
>  
>  static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
> diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> index be24bc57767c..95d04468af9d 100644
> --- a/drivers/media/i2c/adv748x/adv748x.h
> +++ b/drivers/media/i2c/adv748x/adv748x.h
> @@ -434,7 +434,6 @@ int adv748x_afe_s_input(struct adv748x_afe *afe, unsigned int input);
>  
>  int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx);
>  void adv748x_csi2_cleanup(struct adv748x_csi2 *tx);
> -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc);
>  int adv748x_csi2_set_pixelrate(struct v4l2_subdev *sd, s64 rate);
>  
>  int adv748x_hdmi_init(struct adv748x_hdmi *hdmi);

-- 
Regards,

Laurent Pinchart

Re: [PATCH 06/19] media: adv748x: Use routes to configure VC

[Laurent Pinchart]

On Thu, May 02, 2024 at 08:51:16PM +0300, Laurent Pinchart wrote:
> Hi Jacopo,
> 
> Thank you for the patch.
> 
> On Tue, Apr 30, 2024 at 12:39:42PM +0200, Jacopo Mondi wrote:
> > Use the newly introduced routing table to configure on which MIPI
> > CSI-2 Virtual Channel to send the image stream on.
> 
> The stream ID in the routing API isn't meant to be mapped directly to a
> virtual channel number.

Additionally, why do you need to make the virtual channel configurable,
instead of allocating them dynamically ?

> Sakari, your opinion would be appreciated.
> 
> > Perform Virtual Channel selection at s_stream() time instead of
> > forcing it to 0 during the chip reset.
> > 
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/i2c/adv748x/adv748x-core.c |  8 ++------
> >  drivers/media/i2c/adv748x/adv748x-csi2.c | 22 ++++++++++++++++++++--
> >  drivers/media/i2c/adv748x/adv748x.h      |  1 -
> >  3 files changed, 22 insertions(+), 9 deletions(-)
> > 
> > diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
> > index 3abc73ea8ccb..c9d917135709 100644
> > --- a/drivers/media/i2c/adv748x/adv748x-core.c
> > +++ b/drivers/media/i2c/adv748x/adv748x-core.c
> > @@ -530,14 +530,10 @@ static int adv748x_reset(struct adv748x_state *state)
> >  	io_write(state, ADV748X_IO_PD, ADV748X_IO_PD_RX_EN);
> >  
> >  	/* Conditionally enable TXa and TXb. */
> > -	if (is_tx_enabled(&state->txa)) {
> > +	if (is_tx_enabled(&state->txa))
> >  		regval |= ADV748X_IO_10_CSI4_EN;
> > -		adv748x_csi2_set_virtual_channel(&state->txa, 0);
> > -	}
> > -	if (is_tx_enabled(&state->txb)) {
> > +	if (is_tx_enabled(&state->txb))
> >  		regval |= ADV748X_IO_10_CSI1_EN;
> > -		adv748x_csi2_set_virtual_channel(&state->txb, 0);
> > -	}
> >  	io_write(state, ADV748X_IO_10, regval);
> >  
> >  	/* Use vid_std and v_freq as freerun resolution for CP */
> > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > index 7fa72340e66e..a7bfed393ff0 100644
> > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > @@ -14,7 +14,8 @@
> >  
> >  #include "adv748x.h"
> >  
> > -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc)
> > +static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
> > +					    unsigned int vc)
> >  {
> >  	return tx_write(tx, ADV748X_CSI_VC_REF, vc << ADV748X_CSI_VC_REF_SHIFT);
> >  }
> > @@ -175,13 +176,30 @@ static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
> >  static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, int enable)
> >  {
> >  	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > +	struct v4l2_subdev_state *state;
> >  	struct v4l2_subdev *src;
> > +	int ret;
> >  
> >  	src = adv748x_get_remote_sd(&tx->pads[ADV748X_CSI2_SINK]);
> >  	if (!src)
> >  		return -EPIPE;
> >  
> > -	return v4l2_subdev_call(src, video, s_stream, enable);
> > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > +
> > +	if (enable) {
> > +		/* A single route is available. */
> > +		struct v4l2_subdev_route *route = &state->routing.routes[0];
> > +
> > +		ret = adv748x_csi2_set_virtual_channel(tx, route->source_stream);
> > +		if (ret)
> > +			goto unlock;
> > +	}
> > +
> > +	ret = v4l2_subdev_call(src, video, s_stream, enable);
> > +unlock:
> > +	v4l2_subdev_unlock_state(state);
> > +
> > +	return ret;
> >  }
> >  
> >  static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
> > diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> > index be24bc57767c..95d04468af9d 100644
> > --- a/drivers/media/i2c/adv748x/adv748x.h
> > +++ b/drivers/media/i2c/adv748x/adv748x.h
> > @@ -434,7 +434,6 @@ int adv748x_afe_s_input(struct adv748x_afe *afe, unsigned int input);
> >  
> >  int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx);
> >  void adv748x_csi2_cleanup(struct adv748x_csi2 *tx);
> > -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc);
> >  int adv748x_csi2_set_pixelrate(struct v4l2_subdev *sd, s64 rate);
> >  
> >  int adv748x_hdmi_init(struct adv748x_hdmi *hdmi);
> 
> -- 
> Regards,
> 
> Laurent Pinchart

-- 
Regards,

Laurent Pinchart

Re: [PATCH 07/19] media: adv748x: Implement .get_frame_desc()

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:43PM +0200, Jacopo Mondi wrote:
> Implement the get_frame_desc subdev pad operation.
> 
> Implement the get_frame_desc pad operation to allow retrieving the
> stream configuration of the adv748x csi2 subdevice.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/adv748x/adv748x-csi2.c | 86 ++++++++++++++++++++++++
>  1 file changed, 86 insertions(+)
> 
> diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> index a7bfed393ff0..497586aff6b2 100644
> --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> @@ -8,12 +8,51 @@
>  #include <linux/module.h>
>  #include <linux/mutex.h>
>  
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
>  #include <media/v4l2-ioctl.h>
>  
>  #include "adv748x.h"
>  
> +struct adv748x_csi2_format_info {
> +	u8 dt;
> +	u8 bpp;
> +};
> +
> +static int adv748x_csi2_get_format_info(struct adv748x_csi2 *tx, u32 mbus_code,
> +					struct adv748x_csi2_format_info *fmt)
> +{
> +	switch (mbus_code) {
> +	case MEDIA_BUS_FMT_YUYV8_1X16:
> +		fmt->dt = MIPI_CSI2_DT_YUV422_8B;
> +		fmt->bpp = 16;
> +		break;
> +	case MEDIA_BUS_FMT_YUYV10_1X20:
> +		fmt->dt = MIPI_CSI2_DT_YUV422_10B;
> +		fmt->bpp = 20;
> +		break;
> +	case MEDIA_BUS_FMT_RGB565_1X16:
> +		fmt->dt = MIPI_CSI2_DT_RGB565;
> +		fmt->bpp = 16;
> +		break;
> +	case MEDIA_BUS_FMT_RGB666_1X18:
> +		fmt->dt = MIPI_CSI2_DT_RGB666;
> +		fmt->bpp = 18;
> +		break;
> +	case MEDIA_BUS_FMT_RGB888_1X24:
> +		fmt->dt = MIPI_CSI2_DT_RGB888;
> +		fmt->bpp = 24;
> +		break;
> +	default:
> +		dev_dbg(tx->state->dev,
> +			"Unsupported media bus code: %u\n", mbus_code);
> +		return -EINVAL;

That should never happen, right ?

> +	}
> +
> +	return 0;
> +}
> +
>  static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
>  					    unsigned int vc)
>  {
> @@ -258,11 +297,58 @@ static int adv748x_csi2_set_routing(struct v4l2_subdev *sd,
>  	return adv748x_csi2_apply_routing(sd, state, routing);
>  }
>  
> +static int adv748x_csi2_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> +				       struct v4l2_mbus_frame_desc *fd)
> +{
> +	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> +	struct adv748x_csi2_format_info info = {};

Why do you need to initialize the structure to 0 ?

> +	struct v4l2_mbus_frame_desc_entry *entry;
> +	struct v4l2_subdev_route *route;
> +	struct v4l2_subdev_state *state;
> +	struct v4l2_mbus_framefmt *fmt;
> +	int ret = -EINVAL;
> +
> +	if (pad != ADV748X_CSI2_SOURCE)
> +		return -EINVAL;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +
> +	/* A single route is available. */
> +	route = &state->routing.routes[0];
> +	fmt = v4l2_subdev_state_get_format(state, pad, route->source_stream);
> +	if (!fmt)
> +		goto err_unlock;

Can this happen ?

> +
> +	ret = adv748x_csi2_get_format_info(tx, fmt->code, &info);
> +	if (ret)
> +		goto err_unlock;
> +
> +	memset(fd, 0, sizeof(*fd));

This is done by the caller.

> +
> +	/* A single stream is available. */
> +	fd->num_entries = 1;
> +	fd->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +
> +	entry = &fd->entry[0];
> +	entry->stream = 0;
> +	entry->flags = V4L2_MBUS_FRAME_DESC_FL_LEN_MAX;

This is meant for streams that carry variable-length data. You cand drop
and, and leave the .length field unset.

> +	entry->length = fmt->width * fmt->height * info.bpp / 8;
> +	entry->pixelcode = fmt->code;
> +	entry->bus.csi2.vc = route->source_stream;
> +	entry->bus.csi2.dt = info.dt;
> +
> +err_unlock:
> +	v4l2_subdev_unlock_state(state);
> +
> +	return ret;
> +}
> +
>  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
>  	.get_fmt = v4l2_subdev_get_fmt,
>  	.set_fmt = adv748x_csi2_set_format,
>  	.get_mbus_config = adv748x_csi2_get_mbus_config,
>  	.set_routing = adv748x_csi2_set_routing,
> +	.get_frame_desc = adv748x_csi2_get_frame_desc,
>  };
>  
>  /* -----------------------------------------------------------------------------

-- 
Regards,

Laurent Pinchart

Re: [PATCH 08/19] media: max9286: Add support for subdev active state

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:44PM +0200, Jacopo Mondi wrote:
> Use the subdev active state in the max9286 driver to store the
> image format.
> 
> Replace the .open() function call with the .init_state() one and
> simplify the set/get_pad_fmt() operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 110 ++++++++++++------------------------
>  1 file changed, 37 insertions(+), 73 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index d685d445cf23..fb13bfde42df 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -19,7 +19,6 @@
>  #include <linux/i2c.h>
>  #include <linux/i2c-mux.h>
>  #include <linux/module.h>
> -#include <linux/mutex.h>
>  #include <linux/of_graph.h>
>  #include <linux/regulator/consumer.h>
>  #include <linux/slab.h>
> @@ -198,12 +197,8 @@ struct max9286_priv {
>  	struct v4l2_ctrl *pixelrate_ctrl;
>  	unsigned int pixelrate;
>  
> -	struct v4l2_mbus_framefmt fmt[MAX9286_N_SINKS];
>  	struct v4l2_fract interval;
>  
> -	/* Protects controls and fmt structures */
> -	struct mutex mutex;
> -
>  	unsigned int nsources;
>  	unsigned int source_mask;
>  	unsigned int route_mask;
> @@ -788,19 +783,23 @@ static void max9286_v4l2_notifier_unregister(struct max9286_priv *priv)
>  static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  {
>  	struct max9286_priv *priv = sd_to_max9286(sd);
> +	struct v4l2_subdev_state *state;
>  	struct max9286_source *source;
>  	unsigned int i;
>  	bool sync = false;
> -	int ret;
> +	int ret = 0;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
>  
>  	if (enable) {
>  		const struct v4l2_mbus_framefmt *format;
> +		unsigned int source_idx = __ffs(priv->bound_sources);
>  
>  		/*
>  		 * Get the format from the first used sink pad, as all sink
>  		 * formats must be identical.
>  		 */
> -		format = &priv->fmt[__ffs(priv->bound_sources)];
> +		format = v4l2_subdev_state_get_format(state, source_idx);

It would be simpler to get the format from the source pad, as all
formats are identical unless I'm mistaken. It may become more complex
when introducing streams though.

>  
>  		max9286_set_video_format(priv, format);
>  		max9286_set_fsync_period(priv);
> @@ -816,12 +815,12 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  		for_each_source(priv, source) {
>  			ret = v4l2_subdev_call(source->sd, video, s_stream, 1);
>  			if (ret)
> -				return ret;
> +				goto err_unlock;
>  		}
>  
>  		ret = max9286_check_video_links(priv);
>  		if (ret)
> -			return ret;
> +			goto err_unlock;
>  
>  		/*
>  		 * Wait until frame synchronization is locked.
> @@ -842,7 +841,8 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  		if (!sync) {
>  			dev_err(&priv->client->dev,
>  				"Failed to get frame synchronization\n");
> -			return -EXDEV; /* Invalid cross-device link */
> +			ret = -EXDEV; /* Invalid cross-device link */
> +			goto err_unlock;
>  		}
>  
>  		/*
> @@ -865,7 +865,10 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  		max9286_i2c_mux_close(priv);
>  	}
>  
> -	return 0;
> +err_unlock:

As this is used in the success path too, name the label 'done' or 'out'
or 'unlock'.

> +	v4l2_subdev_unlock_state(state);
> +
> +	return ret;
>  }
>  
>  static int max9286_get_frame_interval(struct v4l2_subdev *sd,
> @@ -922,29 +925,16 @@ static int max9286_enum_mbus_code(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> -static struct v4l2_mbus_framefmt *
> -max9286_get_pad_format(struct max9286_priv *priv,
> -		       struct v4l2_subdev_state *sd_state,
> -		       unsigned int pad, u32 which)
> -{
> -	switch (which) {
> -	case V4L2_SUBDEV_FORMAT_TRY:
> -		return v4l2_subdev_state_get_format(sd_state, pad);
> -	case V4L2_SUBDEV_FORMAT_ACTIVE:
> -		return &priv->fmt[pad];
> -	default:
> -		return NULL;
> -	}
> -}
> -
>  static int max9286_set_fmt(struct v4l2_subdev *sd,
> -			   struct v4l2_subdev_state *sd_state,
> +			   struct v4l2_subdev_state *state,
>  			   struct v4l2_subdev_format *format)
>  {
> -	struct max9286_priv *priv = sd_to_max9286(sd);
> -	struct v4l2_mbus_framefmt *cfg_fmt;
>  	unsigned int i;
>  
> +	/*
> +	 * Disable setting format on the source pad: format is propagated
> +	 * from the sinks.
> +	 */
>  	if (format->pad == MAX9286_SRC_PAD)
>  		return -EINVAL;

Return v4l2_subdev_get_fmt().

>  
> @@ -957,42 +947,13 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  	if (i == ARRAY_SIZE(max9286_formats))
>  		format->format.code = max9286_formats[0].code;
>  
> -	cfg_fmt = max9286_get_pad_format(priv, sd_state, format->pad,
> -					 format->which);
> -	if (!cfg_fmt)
> -		return -EINVAL;
> -
> -	mutex_lock(&priv->mutex);
> -	*cfg_fmt = format->format;
> -	mutex_unlock(&priv->mutex);
> -
> -	return 0;
> -}
> -
> -static int max9286_get_fmt(struct v4l2_subdev *sd,
> -			   struct v4l2_subdev_state *sd_state,
> -			   struct v4l2_subdev_format *format)
> -{
> -	struct max9286_priv *priv = sd_to_max9286(sd);
> -	struct v4l2_mbus_framefmt *cfg_fmt;
> -	unsigned int pad = format->pad;
> +	*v4l2_subdev_state_get_format(state, format->pad) = format->format;
>  
>  	/*
> -	 * Multiplexed Stream Support: Support link validation by returning the
> -	 * format of the first bound link. All links must have the same format,
> -	 * as we do not support mixing and matching of cameras connected to the
> -	 * max9286.
> +	 * Apply the same format on the source pad: all links must have the
> +	 * same format.
>  	 */
> -	if (pad == MAX9286_SRC_PAD)
> -		pad = __ffs(priv->bound_sources);
> -
> -	cfg_fmt = max9286_get_pad_format(priv, sd_state, pad, format->which);
> -	if (!cfg_fmt)
> -		return -EINVAL;
> -
> -	mutex_lock(&priv->mutex);
> -	format->format = *cfg_fmt;
> -	mutex_unlock(&priv->mutex);
> +	*v4l2_subdev_state_get_format(state, MAX9286_SRC_PAD) = format->format;

If you want to keep the current behaviour, you should propagate the
format to the source pad for the first sink pad only.

>  
>  	return 0;
>  }
> @@ -1003,7 +964,7 @@ static const struct v4l2_subdev_video_ops max9286_video_ops = {
>  
>  static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
>  	.enum_mbus_code = max9286_enum_mbus_code,
> -	.get_fmt	= max9286_get_fmt,
> +	.get_fmt	= v4l2_subdev_get_fmt,
>  	.set_fmt	= max9286_set_fmt,
>  	.get_frame_interval = max9286_get_frame_interval,
>  	.set_frame_interval = max9286_set_frame_interval,
> @@ -1030,13 +991,14 @@ static void max9286_init_format(struct v4l2_mbus_framefmt *fmt)
>  	*fmt = max9286_default_format;
>  }
>  
> -static int max9286_open(struct v4l2_subdev *subdev, struct v4l2_subdev_fh *fh)
> +static int max9286_init_state(struct v4l2_subdev *sd,
> +			      struct v4l2_subdev_state *state)
>  {
>  	struct v4l2_mbus_framefmt *format;
>  	unsigned int i;
>  
>  	for (i = 0; i < MAX9286_N_SINKS; i++) {
> -		format = v4l2_subdev_state_get_format(fh->state, i);
> +		format = v4l2_subdev_state_get_format(state, i);
>  		max9286_init_format(format);
>  	}

You need to initialize the source pad format too.

>  
> @@ -1044,7 +1006,7 @@ static int max9286_open(struct v4l2_subdev *subdev, struct v4l2_subdev_fh *fh)
>  }
>  
>  static const struct v4l2_subdev_internal_ops max9286_subdev_internal_ops = {
> -	.open = max9286_open,
> +	.init_state = max9286_init_state,
>  };
>  
>  static const struct media_entity_operations max9286_media_ops = {
> @@ -1079,10 +1041,6 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
>  	}
>  
>  	/* Configure V4L2 for the MAX9286 itself */
> -
> -	for (i = 0; i < MAX9286_N_SINKS; i++)
> -		max9286_init_format(&priv->fmt[i]);
> -
>  	v4l2_i2c_subdev_init(&priv->sd, priv->client, &max9286_subdev_ops);
>  	priv->sd.internal_ops = &max9286_subdev_internal_ops;
>  	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> @@ -1109,14 +1067,21 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
>  	if (ret)
>  		goto err_async;
>  
> +	priv->sd.state_lock = priv->ctrls.lock;
> +	ret = v4l2_subdev_init_finalize(&priv->sd);
> +	if (ret)
> +		goto err_async;
> +
>  	ret = v4l2_async_register_subdev(&priv->sd);
>  	if (ret < 0) {
>  		dev_err(dev, "Unable to register subdevice\n");
> -		goto err_async;
> +		goto err_subdev;
>  	}
>  
>  	return 0;
>  
> +err_subdev:
> +	v4l2_subdev_cleanup(&priv->sd);
>  err_async:
>  	v4l2_ctrl_handler_free(&priv->ctrls);
>  	max9286_v4l2_notifier_unregister(priv);
> @@ -1126,6 +1091,7 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
>  
>  static void max9286_v4l2_unregister(struct max9286_priv *priv)
>  {
> +	v4l2_subdev_cleanup(&priv->sd);
>  	v4l2_ctrl_handler_free(&priv->ctrls);
>  	v4l2_async_unregister_subdev(&priv->sd);
>  	max9286_v4l2_notifier_unregister(priv);
> @@ -1629,8 +1595,6 @@ static int max9286_probe(struct i2c_client *client)
>  	if (!priv)
>  		return -ENOMEM;
>  
> -	mutex_init(&priv->mutex);
> -
>  	priv->client = client;
>  
>  	/* GPIO values default to high */

-- 
Regards,

Laurent Pinchart

Re: [PATCH 09/19] media: max9286: Fix enum_mbus_code

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:45PM +0200, Jacopo Mondi wrote:
> The max9286 driver supports multiple output formats but only a single
> one is reported theough the .enum_mbus_code operation.
> 
> Fix that.
> 

Please add a Fixes: tag.

> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index fb13bfde42df..575d8ac8efa1 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -917,10 +917,10 @@ static int max9286_enum_mbus_code(struct v4l2_subdev *sd,
>  				  struct v4l2_subdev_state *sd_state,
>  				  struct v4l2_subdev_mbus_code_enum *code)
>  {
> -	if (code->pad || code->index > 0)
> +	if (code->pad || code->index >= ARRAY_SIZE(max9286_formats))
>  		return -EINVAL;
>  
> -	code->code = MEDIA_BUS_FMT_UYVY8_1X16;
> +	code->code = max9286_formats[code->index].code;
>  
>  	return 0;
>  }

-- 
Regards,

Laurent Pinchart

Re: [PATCH 10/19] media: max9286: Use frame interval from subdev state

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:46PM +0200, Jacopo Mondi wrote:
> Use the frame interval stored in the subdev state instead of storing
> a copy in the driver private structure.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 52 ++++++++-----------------------------
>  1 file changed, 11 insertions(+), 41 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index 575d8ac8efa1..ae1b73fde832 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -197,8 +197,6 @@ struct max9286_priv {
>  	struct v4l2_ctrl *pixelrate_ctrl;
>  	unsigned int pixelrate;
>  
> -	struct v4l2_fract interval;
> -
>  	unsigned int nsources;
>  	unsigned int source_mask;
>  	unsigned int route_mask;
> @@ -571,11 +569,14 @@ static void max9286_set_video_format(struct max9286_priv *priv,
>  		      MAX9286_INVVS | MAX9286_HVSRC_D14);
>  }
>  
> -static void max9286_set_fsync_period(struct max9286_priv *priv)
> +static void max9286_set_fsync_period(struct max9286_priv *priv,
> +				     struct v4l2_subdev_state *state)
>  {
> +	struct v4l2_fract *interval;
>  	u32 fsync;
>  
> -	if (!priv->interval.numerator || !priv->interval.denominator) {
> +	interval = v4l2_subdev_state_get_interval(state, MAX9286_SRC_PAD);
> +	if (!interval->numerator || !interval->denominator) {
>  		/*
>  		 * Special case, a null interval enables automatic FRAMESYNC
>  		 * mode. FRAMESYNC is taken from the slowest link.
> @@ -591,8 +592,8 @@ static void max9286_set_fsync_period(struct max9286_priv *priv)
>  	 * The FRAMESYNC generator is configured with a period expressed as a
>  	 * number of PCLK periods.
>  	 */
> -	fsync = div_u64((u64)priv->pixelrate * priv->interval.numerator,
> -			priv->interval.denominator);
> +	fsync = div_u64((u64)priv->pixelrate * interval->numerator,
> +			interval->denominator);
>  
>  	dev_dbg(&priv->client->dev, "fsync period %u (pclk %u)\n", fsync,
>  		priv->pixelrate);
> @@ -802,7 +803,7 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  		format = v4l2_subdev_state_get_format(state, source_idx);
>  
>  		max9286_set_video_format(priv, format);
> -		max9286_set_fsync_period(priv);
> +		max9286_set_fsync_period(priv, state);
>  
>  		/*
>  		 * The frame sync between cameras is transmitted across the
> @@ -871,44 +872,14 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  	return ret;
>  }
>  
> -static int max9286_get_frame_interval(struct v4l2_subdev *sd,
> -				      struct v4l2_subdev_state *sd_state,
> -				      struct v4l2_subdev_frame_interval *interval)
> -{
> -	struct max9286_priv *priv = sd_to_max9286(sd);
> -
> -	/*
> -	 * FIXME: Implement support for V4L2_SUBDEV_FORMAT_TRY, using the V4L2
> -	 * subdev active state API.
> -	 */
> -	if (interval->which != V4L2_SUBDEV_FORMAT_ACTIVE)
> -		return -EINVAL;
> -
> -	if (interval->pad != MAX9286_SRC_PAD)
> -		return -EINVAL;

You will now return a value for the sink pads. Is that fine ?

> -
> -	interval->interval = priv->interval;
> -
> -	return 0;
> -}
> -
>  static int max9286_set_frame_interval(struct v4l2_subdev *sd,
> -				      struct v4l2_subdev_state *sd_state,
> +				      struct v4l2_subdev_state *state,
>  				      struct v4l2_subdev_frame_interval *interval)
>  {
> -	struct max9286_priv *priv = sd_to_max9286(sd);
> -
> -	/*
> -	 * FIXME: Implement support for V4L2_SUBDEV_FORMAT_TRY, using the V4L2
> -	 * subdev active state API.
> -	 */
> -	if (interval->which != V4L2_SUBDEV_FORMAT_ACTIVE)
> -		return -EINVAL;
> -
>  	if (interval->pad != MAX9286_SRC_PAD)
>  		return -EINVAL;
>  
> -	priv->interval = interval->interval;
> +	*v4l2_subdev_state_get_interval(state, interval->pad) = interval->interval;
>  
>  	return 0;
>  }
> @@ -966,7 +937,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
>  	.enum_mbus_code = max9286_enum_mbus_code,
>  	.get_fmt	= v4l2_subdev_get_fmt,
>  	.set_fmt	= max9286_set_fmt,
> -	.get_frame_interval = max9286_get_frame_interval,
> +	.get_frame_interval = v4l2_subdev_get_frame_interval,

In otder to do so, you need to initialize the interval in the
.init_state() operation.

>  	.set_frame_interval = max9286_set_frame_interval,
>  };
>  
> @@ -1148,7 +1119,6 @@ static int max9286_setup(struct max9286_priv *priv)
>  	max9286_write(priv, 0x69, (0xf & ~priv->route_mask));
>  
>  	max9286_set_video_format(priv, &max9286_default_format);
> -	max9286_set_fsync_period(priv);
>  
>  	cfg = max9286_read(priv, 0x1c);
>  	if (cfg < 0)

-- 
Regards,

Laurent Pinchart

Re: [PATCH 11/19] media: max9286: Use V4L2 Streams

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:47PM +0200, Jacopo Mondi wrote:
> Use V4L2 streams by introducing a static route table that reports
> how the enabled source links are routed through the CSI-2 source pad.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 43 +++++++++++++++++++++++--------------
>  1 file changed, 27 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index ae1b73fde832..f203e4527257 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -800,7 +800,7 @@ static int max9286_s_stream(struct v4l2_subdev *sd, int enable)
>  		 * Get the format from the first used sink pad, as all sink
>  		 * formats must be identical.
>  		 */
> -		format = v4l2_subdev_state_get_format(state, source_idx);
> +		format = v4l2_subdev_state_get_format(state, source_idx, 0);
>  
>  		max9286_set_video_format(priv, format);
>  		max9286_set_fsync_period(priv, state);
> @@ -918,13 +918,14 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  	if (i == ARRAY_SIZE(max9286_formats))
>  		format->format.code = max9286_formats[0].code;
>  
> -	*v4l2_subdev_state_get_format(state, format->pad) = format->format;
> +	*v4l2_subdev_state_get_format(state, format->pad, 0) = format->format;
>  
>  	/*
> -	 * Apply the same format on the source pad: all links must have the
> +	 * Apply the same format on the opposite stream: all links must have the
>  	 * same format.
>  	 */
> -	*v4l2_subdev_state_get_format(state, MAX9286_SRC_PAD) = format->format;
> +	*v4l2_subdev_state_get_opposite_stream_format(state, format->pad, 0) =
> +		format->format;

This would look nicer with an intermediate local variable.

>  
>  	return 0;
>  }
> @@ -957,23 +958,32 @@ static const struct v4l2_mbus_framefmt max9286_default_format = {
>  	.xfer_func	= V4L2_XFER_FUNC_DEFAULT,
>  };
>  
> -static void max9286_init_format(struct v4l2_mbus_framefmt *fmt)
> -{
> -	*fmt = max9286_default_format;
> -}
> -
>  static int max9286_init_state(struct v4l2_subdev *sd,
>  			      struct v4l2_subdev_state *state)
>  {
> -	struct v4l2_mbus_framefmt *format;
> -	unsigned int i;
> +	struct v4l2_subdev_route routes[MAX9286_N_SINKS];
> +	struct max9286_priv *priv = sd_to_max9286(sd);
> +	struct max9286_source *source;
> +	unsigned int num_routes = 0;
>  
> -	for (i = 0; i < MAX9286_N_SINKS; i++) {
> -		format = v4l2_subdev_state_get_format(state, i);
> -		max9286_init_format(format);
> +	for_each_source(priv, source) {
> +		struct v4l2_subdev_route *route = &routes[num_routes++];
> +		unsigned int index = to_index(priv, source);
> +
> +		route->sink_pad = index;
> +		route->sink_stream = 0;
> +		route->source_pad = MAX9286_SRC_PAD;
> +		route->source_stream = index;
> +		route->flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE;
>  	}
>  
> -	return 0;
> +	struct v4l2_subdev_krouting routing = {
> +		.num_routes = num_routes,

You need to set .len_routes too (the field will appear in v6.10, it's in
the stage tree already).

> +		.routes = routes,
> +	};
> +
> +	return v4l2_subdev_set_routing_with_fmt(sd, state, &routing,
> +						&max9286_default_format);
>  }
>  
>  static const struct v4l2_subdev_internal_ops max9286_subdev_internal_ops = {
> @@ -1014,7 +1024,8 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
>  	/* Configure V4L2 for the MAX9286 itself */
>  	v4l2_i2c_subdev_init(&priv->sd, priv->client, &max9286_subdev_ops);
>  	priv->sd.internal_ops = &max9286_subdev_internal_ops;
> -	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
> +	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
> +			  V4L2_SUBDEV_FL_STREAMS;
>  
>  	v4l2_ctrl_handler_init(&priv->ctrls, 1);
>  	priv->pixelrate_ctrl = v4l2_ctrl_new_std(&priv->ctrls,

-- 
Regards,

Laurent Pinchart

Re: [PATCH 12/19] media: max9286: Implement .get_frame_desc()

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:48PM +0200, Jacopo Mondi wrote:
> Implement the .get_frame_desc() pad operation to allow the receiver
> to retrieve information on the multiplexed source pad.
> 
> Record in the max9286_format_info structure the MIPI CSI-2
> data type and use it to populate the frame_desc_entry.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 120 ++++++++++++++++++++++++++++--------
>  1 file changed, 95 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index f203e4527257..4b4f4c03c10a 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -23,6 +23,7 @@
>  #include <linux/regulator/consumer.h>
>  #include <linux/slab.h>
>  
> +#include <media/mipi-csi2.h>
>  #include <media/v4l2-async.h>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
> @@ -145,7 +146,12 @@
>  
>  struct max9286_format_info {
>  	u32 code;
> -	u8 datatype;
> +	/* The gmsl data format configuration. */
> +	u8 gmsl_dt;
> +	/* The format bpp, used for stride calculation. */
> +	u8 bpp;
> +	/* The Data Type identifier as defined by the MIPI CSI-2 standard. */
> +	u8 mipi_dt;
>  };
>  
>  struct max9286_i2c_speed {
> @@ -235,28 +241,44 @@ static inline struct max9286_priv *sd_to_max9286(struct v4l2_subdev *sd)
>  static const struct max9286_format_info max9286_formats[] = {
>  	{
>  		.code = MEDIA_BUS_FMT_UYVY8_1X16,
> -		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.bpp = 16,
> +		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_VYUY8_1X16,
> -		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.bpp = 16,
> +		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_YUYV8_1X16,
> -		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.bpp = 16,
> +		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_YVYU8_1X16,
> -		.datatype = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.bpp = 16,
> +		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SBGGR12_1X12,
> -		.datatype = MAX9286_DATATYPE_RAW12,
> +		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.bpp = 12,
> +		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SGBRG12_1X12,
> -		.datatype = MAX9286_DATATYPE_RAW12,
> +		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.bpp = 12,
> +		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SGRBG12_1X12,
> -		.datatype = MAX9286_DATATYPE_RAW12,
> +		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.bpp = 12,
> +		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
> -		.datatype = MAX9286_DATATYPE_RAW12,
> +		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.bpp = 12,
> +		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	},
>  };
>  
> @@ -532,19 +554,23 @@ static int max9286_check_config_link(struct max9286_priv *priv,
>  	return 0;
>  }
>  
> +static const struct max9286_format_info *
> +max9286_get_format_info(unsigned int code)
> +{
> +	for (unsigned int i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
> +		if (max9286_formats[i].code == code)
> +			return &max9286_formats[i];
> +	}
> +
> +	return NULL;
> +}
> +
>  static void max9286_set_video_format(struct max9286_priv *priv,
>  				     const struct v4l2_mbus_framefmt *format)
>  {
>  	const struct max9286_format_info *info = NULL;
> -	unsigned int i;
> -
> -	for (i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
> -		if (max9286_formats[i].code == format->code) {
> -			info = &max9286_formats[i];
> -			break;
> -		}
> -	}
>  
> +	info = max9286_get_format_info(format->code);
>  	if (WARN_ON(!info))
>  		return;
>  
> @@ -559,7 +585,7 @@ static void max9286_set_video_format(struct max9286_priv *priv,
>  	/* Enable CSI-2 Lane D0-D3 only, DBL mode. */
>  	max9286_write(priv, 0x12, MAX9286_CSIDBL | MAX9286_DBL |
>  		      MAX9286_CSILANECNT(priv->csi2_data_lanes) |
> -		      info->datatype);
> +		      info->gmsl_dt);
>  
>  	/*
>  	 * Enable HS/VS encoding, use HS as line valid source, use D14/15 for
> @@ -900,7 +926,7 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  			   struct v4l2_subdev_state *state,
>  			   struct v4l2_subdev_format *format)
>  {
> -	unsigned int i;
> +	const struct max9286_format_info *info;
>  
>  	/*
>  	 * Disable setting format on the source pad: format is propagated
> @@ -910,12 +936,8 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	/* Validate the format. */
> -	for (i = 0; i < ARRAY_SIZE(max9286_formats); ++i) {
> -		if (max9286_formats[i].code == format->format.code)
> -			break;
> -	}
> -
> -	if (i == ARRAY_SIZE(max9286_formats))
> +	info = max9286_get_format_info(format->format.code);
> +	if (!info)
>  		format->format.code = max9286_formats[0].code;
>  
>  	*v4l2_subdev_state_get_format(state, format->pad, 0) = format->format;
> @@ -930,6 +952,53 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> +				  struct v4l2_mbus_frame_desc *fd)
> +{
> +	struct v4l2_subdev_route *route;
> +	struct v4l2_subdev_state *state;
> +	unsigned int num_routes = 0;
> +	int ret = 0;
> +
> +	if (pad != MAX9286_SRC_PAD)
> +		return -EINVAL;
> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);

Add a blank line.

> +	for_each_active_route(&state->routing, route) {
> +		struct v4l2_mbus_frame_desc_entry *entry;
> +		const struct max9286_format_info *info;
> +		struct v4l2_mbus_framefmt *fmt;

const

> +
> +		fmt = v4l2_subdev_state_get_format(state, route->sink_pad,
> +						   route->sink_stream);
> +		info = max9286_get_format_info(fmt->code);
> +		if (WARN_ON(!info)) {

I don't think this can happen. You can drop the check and the err_unlock
label.

> +			ret = -EINVAL;
> +			goto err_unlock;
> +		}
> +
> +		entry = &fd->entry[num_routes];
> +		entry->stream = num_routes;
> +		entry->flags = V4L2_MBUS_FRAME_DESC_FL_LEN_MAX;
> +		entry->length = fmt->width * fmt->height * info->bpp / 8;

As explained in the review of 07/19, you can drop flags and length (and
thus the bpp field from max9286_format_info).

> +		entry->pixelcode = fmt->code;
> +
> +		/* VC is set according to link ordering, see register 0x15. */
> +		entry->bus.csi2.vc = route->sink_pad;
> +		entry->bus.csi2.dt = info->mipi_dt;
> +
> +		num_routes++;
> +	}
> +
> +	fd->num_entries = num_routes;
> +	fd->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
> +
> +err_unlock:
> +	v4l2_subdev_unlock_state(state);
> +
> +	return ret;
> +}
> +
>  static const struct v4l2_subdev_video_ops max9286_video_ops = {
>  	.s_stream	= max9286_s_stream,
>  };
> @@ -940,6 +1009,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
>  	.set_fmt	= max9286_set_fmt,
>  	.get_frame_interval = v4l2_subdev_get_frame_interval,
>  	.set_frame_interval = max9286_set_frame_interval,
> +	.get_frame_desc	= max9286_get_frame_desc,
>  };
>  
>  static const struct v4l2_subdev_ops max9286_subdev_ops = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH 13/19] media: max9286: Implement support for LINK_FREQ

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:49PM +0200, Jacopo Mondi wrote:
> The max9286 now supports multiple streams. For this reason, reporting
> the data bitrate through the PIXEL_RATE control is not meaningful
> anymore (or better, is less meaningful that what it already was).
> 
> In order to support multiplexed receivers, which want to be informed
> about the CSI-2 link frequency, implement support for the
> V4L2_CID_LINK_FREQ control.

I think
https://lore.kernel.org/linux-media/20240429190852.1008003-3-sakari.ailus@linux.intel.com
may be more suitable for this use case.

> 
> Record in the driver a set of pre-computed link frequencies in order to
> support the two currently supported GMSL camera modules (rdacm20 and

s/camera modules/cameras/

> rdacm21) and all the supported number of active channels and CSI-2 data
> lanes.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/i2c/max9286.c | 143 ++++++++++++++++++++++++++++++++++--
>  1 file changed, 136 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index 4b4f4c03c10a..0e08a1041eed 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -148,6 +148,12 @@ struct max9286_format_info {
>  	u32 code;
>  	/* The gmsl data format configuration. */
>  	u8 gmsl_dt;
> +	/*
> +	 * gmsl_bpp is the pixel format bit-width per word as reported in
> +	 * table 20. and it is used to calculate the GMSL bandwidth and CSI-2

Table 20 of what ?

> +	 * link speed.
> +	 */
> +	u8 gmsl_bpp;
>  	/* The format bpp, used for stride calculation. */
>  	u8 bpp;
>  	/* The Data Type identifier as defined by the MIPI CSI-2 standard. */
> @@ -201,6 +207,7 @@ struct max9286_priv {
>  
>  	struct v4l2_ctrl_handler ctrls;
>  	struct v4l2_ctrl *pixelrate_ctrl;
> +	struct v4l2_ctrl *link_freq;
>  	unsigned int pixelrate;
>  
>  	unsigned int nsources;
> @@ -242,41 +249,49 @@ static const struct max9286_format_info max9286_formats[] = {
>  	{
>  		.code = MEDIA_BUS_FMT_UYVY8_1X16,
>  		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_bpp = 8,
>  		.bpp = 16,
>  		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_VYUY8_1X16,
>  		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_bpp = 8,
>  		.bpp = 16,
>  		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_YUYV8_1X16,
>  		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_bpp = 8,
>  		.bpp = 16,
>  		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_YVYU8_1X16,
>  		.gmsl_dt = MAX9286_DATATYPE_YUV422_8BIT,
> +		.gmsl_bpp = 8,
>  		.bpp = 16,
>  		.mipi_dt = MIPI_CSI2_DT_YUV422_8B,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SBGGR12_1X12,
>  		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.gmsl_bpp = 12,
>  		.bpp = 12,
>  		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SGBRG12_1X12,
>  		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.gmsl_bpp = 12,
>  		.bpp = 12,
>  		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SGRBG12_1X12,
>  		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.gmsl_bpp = 12,
>  		.bpp = 12,
>  		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	}, {
>  		.code = MEDIA_BUS_FMT_SRGGB12_1X12,
>  		.gmsl_dt = MAX9286_DATATYPE_RAW12,
> +		.gmsl_bpp = 12,
>  		.bpp = 12,
>  		.mipi_dt = MIPI_CSI2_DT_RAW12,
>  	},
> @@ -293,6 +308,60 @@ static const struct max9286_i2c_speed max9286_i2c_speeds[] = {
>  	{ .rate = 837000, .mstbt = MAX9286_I2CMSTBT_837KBPS },
>  };
>  
> +/*
> + * According to the chip manual the CSI-2 link frequency is calculated as:
> + *
> + *     f-CSI2 = pclk * ((width * channels) / lanes)
> + *
> + * Where:
> + *     pclk = serializer input word rate
> + *     width = pixel format bits per word (gmsl_bpp in max9286_format_info)
> + *     channels = number of enabled links
> + *     lanes = number of CSI-2 data lanes
> + *
> + * The following link frequencies have been computed by combining the following
> + * parameters:
> + *
> + * pixel_rates[] = { 44000000, 55000000 }; for RDACM20 and RDACM21
> + * bpps[] = { 8, 12 }; for the formats supported by the max9286 driver
> + * channels[] = { 1, 2, 3, 4 };
> + * lanes[] = { 1, 2, 3, 4 };
> + *
> + * and by filtering out all results that would lead to an invalid
> + * link_frequency, as the MAX9286 has a CSI-2 bandwidth limit of 1200Mbps per
> + * lane (600MHz clock frequency).
> + *
> + * If a new serializer with a different pixel rate, or a new image format with
> + * a different bpp is supported, the below table needs to be updated.
> + */
> +static const s64 max9286_link_freqs[] = {
> +	88000000,
> +	110000000,
> +	132000000,
> +	165000000,
> +	176000000,
> +	220000000,
> +	264000000,
> +	275000000,
> +	330000000,
> +	352000000,
> +	396000000,
> +	440000000,
> +	495000000,
> +	528000000,
> +	550000000,
> +};
> +
> +static int max9286_get_link_freq_index(s64 link_freq)
> +{
> +	for (unsigned int i = 0; i < ARRAY_SIZE(max9286_link_freqs); ++i) {
> +		if (max9286_link_freqs[i] == link_freq)
> +			return i;
> +	}
> +
> +	return -EINVAL;
> +}
> +
>  /* -----------------------------------------------------------------------------
>   * I2C IO
>   */
> @@ -636,10 +705,18 @@ static void max9286_set_fsync_period(struct max9286_priv *priv,
>   * V4L2 Subdev
>   */
>  
> -static int max9286_set_pixelrate(struct max9286_priv *priv)
> +static int max9286_set_pixelrate(struct max9286_priv *priv,
> +				 struct v4l2_subdev_state *state,
> +				 unsigned int link_index)
>  {
> +	const struct max9286_format_info *info;
>  	struct max9286_source *source = NULL;
> +	struct v4l2_mbus_framefmt *fmt;
> +	unsigned int num_sources = 0;
> +	int link_freq_index;
>  	u64 pixelrate = 0;
> +	s64 link_freq;
> +	int ret;
>  
>  	for_each_source(priv, source) {
>  		struct v4l2_ctrl *ctrl;
> @@ -662,6 +739,8 @@ static int max9286_set_pixelrate(struct max9286_priv *priv)
>  				"Unable to calculate pixel rate\n");
>  			return -EINVAL;
>  		}
> +
> +		num_sources++;
>  	}
>  
>  	if (!pixelrate) {
> @@ -670,14 +749,33 @@ static int max9286_set_pixelrate(struct max9286_priv *priv)
>  		return -EINVAL;
>  	}
>  
> -	priv->pixelrate = pixelrate;
> +	fmt = v4l2_subdev_state_get_format(state, link_index);
> +	info = max9286_get_format_info(fmt->code);
> +	if (WARN_ON(!info))
> +		return -EINVAL;
> +
> +	/*
> +	 * Compute the link frequency and find the right index to update
> +	 * the menu control with.
> +	 */
> +	link_freq = pixelrate * ((info->gmsl_bpp * num_sources) /
> +		    priv->csi2_data_lanes);
> +
> +	link_freq_index = max9286_get_link_freq_index(link_freq);
> +	if (WARN_ON(link_freq_index < 0))
> +		return -EINVAL;
>  
>  	/*
>  	 * The CSI-2 transmitter pixel rate is the single source rate multiplied
>  	 * by the number of available sources.
>  	 */
> -	return v4l2_ctrl_s_ctrl_int64(priv->pixelrate_ctrl,
> -				      pixelrate * priv->nsources);
> +	ret = __v4l2_ctrl_s_ctrl_int64(priv->pixelrate_ctrl,
> +				       pixelrate * priv->nsources);
> +	if (ret)
> +		return -EINVAL;
> +	priv->pixelrate = pixelrate;
> +
> +	return __v4l2_ctrl_s_ctrl(priv->link_freq, link_freq_index);
>  }
>  
>  static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
> @@ -687,6 +785,7 @@ static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
>  	struct max9286_priv *priv = sd_to_max9286(notifier->sd);
>  	struct max9286_source *source = to_max9286_asd(asd)->source;
>  	unsigned int index = to_index(priv, source);
> +	struct v4l2_subdev_state *state;
>  	unsigned int src_pad;
>  	int ret;
>  
> @@ -739,7 +838,11 @@ static int max9286_notify_bound(struct v4l2_async_notifier *notifier,
>  	max9286_check_config_link(priv, priv->source_mask);
>  	max9286_configure_i2c(priv, false);
>  
> -	return max9286_set_pixelrate(priv);
> +	state = v4l2_subdev_lock_and_get_active_state(notifier->sd);
> +	ret = max9286_set_pixelrate(priv, state, index);
> +	v4l2_subdev_unlock_state(state);
> +
> +	return ret;
>  }
>  
>  static void max9286_notify_unbind(struct v4l2_async_notifier *notifier,
> @@ -926,7 +1029,9 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  			   struct v4l2_subdev_state *state,
>  			   struct v4l2_subdev_format *format)
>  {
> +	struct max9286_priv *priv = sd_to_max9286(sd);
>  	const struct max9286_format_info *info;
> +	int ret;
>  
>  	/*
>  	 * Disable setting format on the source pad: format is propagated
> @@ -949,7 +1054,22 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  	*v4l2_subdev_state_get_opposite_stream_format(state, format->pad, 0) =
>  		format->format;
>  
> -	return 0;
> +	/*
> +	 * Update the pixel rate and link_rate controls. We cannot do that in
> +	 * s_stream() as the receiver is likely fetching the frame_desc and
> +	 * reading the link_rate control before calling this driver's s_stream.
> +	 *
> +	 * Update the controls here.
> +	 *
> +	 * TODO: Once the routing table will be made writable and links
> +	 * can be enabled/disabled, the controls need to be updated there too.
> +	 */
> +	ret = max9286_set_pixelrate(priv, state, format->pad);
> +	if (ret)
> +		dev_err(&priv->client->dev,
> +			"Unsupported format/lanes/channel combination: clock rate too high");
> +
> +	return ret;
>  }
>  
>  static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
> @@ -1068,6 +1188,7 @@ static int max9286_s_ctrl(struct v4l2_ctrl *ctrl)
>  {
>  	switch (ctrl->id) {
>  	case V4L2_CID_PIXEL_RATE:
> +	case V4L2_CID_LINK_FREQ:
>  		return 0;
>  	default:
>  		return -EINVAL;
> @@ -1097,12 +1218,20 @@ static int max9286_v4l2_register(struct max9286_priv *priv)
>  	priv->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE |
>  			  V4L2_SUBDEV_FL_STREAMS;
>  
> -	v4l2_ctrl_handler_init(&priv->ctrls, 1);
> +	v4l2_ctrl_handler_init(&priv->ctrls, 2);
>  	priv->pixelrate_ctrl = v4l2_ctrl_new_std(&priv->ctrls,
>  						 &max9286_ctrl_ops,
>  						 V4L2_CID_PIXEL_RATE,
>  						 1, INT_MAX, 1, 50000000);
>  
> +	priv->link_freq =
> +		v4l2_ctrl_new_int_menu(&priv->ctrls, &max9286_ctrl_ops,
> +				       V4L2_CID_LINK_FREQ,
> +				       ARRAY_SIZE(max9286_link_freqs) - 1, 0,
> +				       max9286_link_freqs);
> +	if (priv->link_freq)
> +		priv->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> +
>  	priv->sd.ctrl_handler = &priv->ctrls;
>  	ret = priv->ctrls.error;
>  	if (ret)

-- 
Regards,

Laurent Pinchart

Re: [PATCH 14/19] media: max9286: Implement .get_mbus_config()

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:50PM +0200, Jacopo Mondi wrote:
> Implement the .get_mbus_config() operation to allow the remote receiver
> to retrieve the bus configuration.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>

This looks good to me, but I expect changes in v2 to address the
comments on 13/19, so I won't add a R-b tag yet.

> ---
>  drivers/media/i2c/max9286.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/drivers/media/i2c/max9286.c b/drivers/media/i2c/max9286.c
> index 0e08a1041eed..a71f0aabd67d 100644
> --- a/drivers/media/i2c/max9286.c
> +++ b/drivers/media/i2c/max9286.c
> @@ -1072,6 +1072,20 @@ static int max9286_set_fmt(struct v4l2_subdev *sd,
>  	return ret;
>  }
>  
> +static int max9286_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
> +				   struct v4l2_mbus_config *config)
> +{
> +	struct max9286_priv *priv = sd_to_max9286(sd);
> +
> +	if (pad != MAX9286_SRC_PAD)
> +		return -EINVAL;
> +
> +	config->type = V4L2_MBUS_CSI2_DPHY;
> +	config->bus.mipi_csi2.num_data_lanes = priv->csi2_data_lanes;
> +
> +	return 0;
> +}
> +
>  static int max9286_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
>  				  struct v4l2_mbus_frame_desc *fd)
>  {
> @@ -1127,6 +1141,7 @@ static const struct v4l2_subdev_pad_ops max9286_pad_ops = {
>  	.enum_mbus_code = max9286_enum_mbus_code,
>  	.get_fmt	= v4l2_subdev_get_fmt,
>  	.set_fmt	= max9286_set_fmt,
> +	.get_mbus_config = max9286_get_mbus_config,
>  	.get_frame_interval = v4l2_subdev_get_frame_interval,
>  	.set_frame_interval = max9286_set_frame_interval,
>  	.get_frame_desc	= max9286_get_frame_desc,

-- 
Regards,

Laurent Pinchart

Re: [PATCH 15/19] media: rcar-csi2: Add support for multiplexed streams

[Laurent Pinchart]

On Thu, May 02, 2024 at 04:23:16PM +0200, Niklas Söderlund wrote:
> Hi Jacopo,
> 
> Thanks for your work.
> 
> On 2024-04-30 12:39:51 +0200, Jacopo Mondi wrote:
> > Create and initialize the v4l2_subdev_state for the R-Car CSI-2 receiver
> > in order to prepare to support multiplexed transmitters.
> > 
> > Create the subdevice state with v4l2_subdev_init_finalize() and
> > implement the init_state() operation to guarantee the state is initialized.
> > 
> > The routing table within the R-Car CSI-2 receiver is fixed, streams
> > received on source_stream X will be directed to pad (X + 1) by default.
> > Initialize a static routing table with such routes set as active.
> > 
> > While at it, disable runtime_pm() in the probe() function error path.

s/runtime_pm()/runtime PM/

> Can we break this out in a separate patch? As the multiplexed stream 
> work will not be ready for v6.10 if I understood the tendencies 
> correctly (?), we can at least fix this issue before that.

Agreed, the fix should be a separate patch, with a Fixes: tag.

> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/platform/renesas/rcar-csi2.c | 74 +++++++++++++++++++++-
> >  1 file changed, 72 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> > index 582d5e35db0e..82dc0b92b8d3 100644
> > --- a/drivers/media/platform/renesas/rcar-csi2.c
> > +++ b/drivers/media/platform/renesas/rcar-csi2.c
> > @@ -1226,6 +1226,65 @@ static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
> >  	.pad	= &rcar_csi2_pad_ops,
> >  };
> >  
> > +static int rcsi2_init_state(struct v4l2_subdev *sd,
> > +			    struct v4l2_subdev_state *state)
> > +{
> > +	/*
> > +	 * Routing is fixed for this device: streams sent on sink_stream X
> > +	 * are directed to pad (X + 1). Which streams goes to the next
> > +	 * processing block (VIN) is controlled by link enablement between the
> > +	 * CSI-2 and the VIN itself and not by the CSI-2 routing table.
> > +	 *
> > +	 * The routing table is then fixed, as stream X will be directed to
> > +	 * csi:(X + 1)/0 and will be transmitted to VINs the on media link
> > +	 * csi2:(x + 1)->vin:0.
> > +	 *
> > +	 * For example, to route stream #3 to VIN #1 : "csi2:4/0 -> vin1:0" and
> > +	 * to route stream #2 to VIN #4 : "csi2:3/0 -> vin4:0".
> > +	 */
> > +	struct v4l2_subdev_route routes[] = {
> > +		{
> > +			.sink_pad = RCAR_CSI2_SINK,
> > +			.sink_stream = 0,
> > +			.source_pad = RCAR_CSI2_SOURCE_VC0,
> > +			.source_stream = 0,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +		{
> > +			.sink_pad = RCAR_CSI2_SINK,
> > +			.sink_stream = 1,
> > +			.source_pad = RCAR_CSI2_SOURCE_VC1,
> > +			.source_stream = 0,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +		{
> > +			.sink_pad = RCAR_CSI2_SINK,
> > +			.sink_stream = 2,
> > +			.source_pad = RCAR_CSI2_SOURCE_VC2,
> > +			.source_stream = 0,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +		{
> > +			.sink_pad = RCAR_CSI2_SINK,
> > +			.sink_stream = 3,
> > +			.source_pad = RCAR_CSI2_SOURCE_VC3,
> > +			.source_stream = 0,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +	};
> > +
> > +	struct v4l2_subdev_krouting routing = {
> > +		.num_routes = ARRAY_SIZE(routes),
> > +		.routes = routes,
> > +	};
> 
> Should not the two structs above be static const as you return a pointer 
> to them?

v4l2_subdev_set_routing() takes a non-const v4l2_subdev_krouting
argument. Unless I'm missing something, this function doesn't return a
pointer to the local structures.

> > +
> > +	return v4l2_subdev_set_routing(sd, state, &routing);
> > +}
> > +
> > +static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {
> > +	.init_state = rcsi2_init_state,
> > +};
> > +
> >  static irqreturn_t rcsi2_irq(int irq, void *data)
> >  {
> >  	struct rcar_csi2 *priv = data;
> > @@ -1887,11 +1946,13 @@ static int rcsi2_probe(struct platform_device *pdev)
> >  
> >  	priv->subdev.owner = THIS_MODULE;
> >  	priv->subdev.dev = &pdev->dev;
> > +	priv->subdev.internal_ops = &rcar_csi2_internal_ops;
> >  	v4l2_subdev_init(&priv->subdev, &rcar_csi2_subdev_ops);
> >  	v4l2_set_subdevdata(&priv->subdev, &pdev->dev);
> >  	snprintf(priv->subdev.name, sizeof(priv->subdev.name), "%s %s",
> >  		 KBUILD_MODNAME, dev_name(&pdev->dev));
> > -	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> > +	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
> > +			     V4L2_SUBDEV_FL_STREAMS;
> >  
> >  	priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> >  	priv->subdev.entity.ops = &rcar_csi2_entity_ops;
> > @@ -1912,14 +1973,22 @@ static int rcsi2_probe(struct platform_device *pdev)
> >  
> >  	pm_runtime_enable(&pdev->dev);
> >  
> > +	ret = v4l2_subdev_init_finalize(&priv->subdev);
> > +	if (ret)
> > +		goto error_pm_runtime;
> > +
> >  	ret = v4l2_async_register_subdev(&priv->subdev);
> >  	if (ret < 0)
> > -		goto error_async;
> > +		goto error_subdev;
> >  
> >  	dev_info(priv->dev, "%d lanes found\n", priv->lanes);
> >  
> >  	return 0;
> >  
> > +error_subdev:
> > +	v4l2_subdev_cleanup(&priv->subdev);
> > +error_pm_runtime:
> > +	pm_runtime_disable(&pdev->dev);
> >  error_async:
> >  	v4l2_async_nf_unregister(&priv->notifier);
> >  	v4l2_async_nf_cleanup(&priv->notifier);
> > @@ -1936,6 +2005,7 @@ static void rcsi2_remove(struct platform_device *pdev)
> >  	v4l2_async_nf_unregister(&priv->notifier);
> >  	v4l2_async_nf_cleanup(&priv->notifier);
> >  	v4l2_async_unregister_subdev(&priv->subdev);
> > +	v4l2_subdev_cleanup(&priv->subdev);
> >  
> >  	pm_runtime_disable(&pdev->dev);
> >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH 16/19] media: rcar-csi2: Support multiplexed transmitters

[Laurent Pinchart]

On Thu, May 02, 2024 at 04:30:54PM +0200, Niklas Söderlund wrote:
> On 2024-04-30 12:39:52 +0200, Jacopo Mondi wrote:
> > Rework the R-Car CSI-2 start routine to support multiplexed
> > transmitters.
> > 
> > Configure the CSI-2 receiver MIPI CSI-2 Data Type filtering by inspecting
> > the remote subdev frame_desc instead of relying on the image format
> > configured on the sink pad.
> > 
> > Enable MIPI CSI-2 Data Type filtering by inspecting the remote subdevice
> > frame descriptor to discern which Data Type is transmitted on which
> > Virtual Channel. If multiple Data Types are transmitted on the same VC
> > then Data Type filtering is disabled.
> > 
> > Rework the per-lane bandwidth calculation to use the LINK_FREQ control
> > if a transmitter sends multiple streams on the same data link. The
> > current usage of the PIXEL_RATE control does not support multiplexed
> > transmitters, as there's not a unique pixel rate among all the possible
> > source streams.

It sounds like this should be split in two patches, the LINK_FREQ change
seems separate from the DT/VC filtering and routing handling.

> > 
> > This change makes mandatory for any subdevice that operates with
> > the R-Car CSI-2 receiver to implement the .get_frame_desc() operation.
> > 
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/platform/renesas/rcar-csi2.c | 280 ++++++++++++++-------
> >  1 file changed, 191 insertions(+), 89 deletions(-)
> > 
> > diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> > index 82dc0b92b8d3..ffb73272543b 100644
> > --- a/drivers/media/platform/renesas/rcar-csi2.c
> > +++ b/drivers/media/platform/renesas/rcar-csi2.c
> > @@ -69,10 +69,7 @@ struct rcar_csi2;
> >  #define FLD_REG				0x1c
> >  #define FLD_FLD_NUM(n)			(((n) & 0xff) << 16)
> >  #define FLD_DET_SEL(n)			(((n) & 0x3) << 4)
> > -#define FLD_FLD_EN4			BIT(3)
> > -#define FLD_FLD_EN3			BIT(2)
> > -#define FLD_FLD_EN2			BIT(1)
> > -#define FLD_FLD_EN			BIT(0)
> > +#define FLD_FLD_EN(n)			BIT((n) & 0xf)
> >  
> >  /* Automatic Standby Control */
> >  #define ASTBY_REG			0x20
> > @@ -575,6 +572,16 @@ static const struct rcar_csi2_format *rcsi2_code_to_fmt(unsigned int code)
> >  	return NULL;
> >  }
> >  
> > +static const struct rcar_csi2_format *rcsi2_datatype_to_fmt(unsigned int dt)
> > +{
> > +	for (unsigned int i = 0; i < ARRAY_SIZE(rcar_csi2_formats); i++) {
> > +		if (rcar_csi2_formats[i].datatype == dt)
> > +			return &rcar_csi2_formats[i];
> > +	}
> > +
> > +	return NULL;
> > +}
> > +
> >  enum rcar_csi2_pads {
> >  	RCAR_CSI2_SINK,
> >  	RCAR_CSI2_SOURCE_VC0,
> > @@ -587,7 +594,8 @@ enum rcar_csi2_pads {
> >  struct rcar_csi2_info {
> >  	int (*init_phtw)(struct rcar_csi2 *priv, unsigned int mbps);
> >  	int (*phy_post_init)(struct rcar_csi2 *priv);
> > -	int (*start_receiver)(struct rcar_csi2 *priv);
> > +	int (*start_receiver)(struct rcar_csi2 *priv,
> > +			      struct v4l2_subdev_state *state);
> >  	void (*enter_standby)(struct rcar_csi2 *priv);
> >  	const struct rcsi2_mbps_reg *hsfreqrange;
> >  	unsigned int csi0clkfreqrange;
> > @@ -676,6 +684,32 @@ static int rcsi2_exit_standby(struct rcar_csi2 *priv)
> >  	return 0;
> >  }
> >  
> > +static int rcsi2_get_remote_frame_desc(struct rcar_csi2 *priv,
> > +				       struct v4l2_mbus_frame_desc *fd)
> > +{
> > +	struct media_pad *pad;
> > +	int ret;
> > +
> > +	if (!priv->remote)
> > +		return -ENOLINK;
> > +
> > +	pad = media_pad_remote_pad_unique(&priv->pads[RCAR_CSI2_SINK]);
> > +	if (IS_ERR(pad))
> > +		return PTR_ERR(pad);
> > +
> > +	ret = v4l2_subdev_call(priv->remote, pad, get_frame_desc,
> > +			       pad->index, fd);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (fd->type != V4L2_MBUS_FRAME_DESC_TYPE_CSI2) {
> > +		dev_err(priv->dev, "Frame desc does not describe a CSI-2 link");
> > +		return -EINVAL;
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  static int rcsi2_wait_phy_start(struct rcar_csi2 *priv,
> >  				unsigned int lanes)
> >  {
> > @@ -726,41 +760,6 @@ static int rcsi2_set_phypll(struct rcar_csi2 *priv, unsigned int mbps)
> >  	return 0;
> >  }
> >  
> > -static int rcsi2_calc_mbps(struct rcar_csi2 *priv, unsigned int bpp,
> > -			   unsigned int lanes)
> > -{
> > -	struct v4l2_subdev *source;
> > -	struct v4l2_ctrl *ctrl;
> > -	u64 mbps;
> > -
> > -	if (!priv->remote)
> > -		return -ENODEV;
> > -
> > -	source = priv->remote;
> > -
> > -	/* Read the pixel rate control from remote. */
> > -	ctrl = v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_PIXEL_RATE);
> > -	if (!ctrl) {
> > -		dev_err(priv->dev, "no pixel rate control in subdev %s\n",
> > -			source->name);
> > -		return -EINVAL;
> > -	}
> > -
> > -	/*
> > -	 * Calculate the phypll in mbps.
> > -	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> > -	 * bps = link_freq * 2
> > -	 */
> > -	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * bpp;
> > -	do_div(mbps, lanes * 1000000);
> > -
> > -	/* Adjust for C-PHY, divide by 2.8. */
> > -	if (priv->cphy)
> > -		mbps = div_u64(mbps * 5, 14);
> > -
> > -	return mbps;
> > -}
> > -
> >  static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
> >  				  unsigned int *lanes)
> >  {
> > @@ -808,52 +807,146 @@ static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
> >  	return 0;
> >  }
> >  
> > -static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
> > +static int rcsi2_calc_mbps(struct rcar_csi2 *priv,
> > +			   struct v4l2_mbus_frame_desc *fd, unsigned int lanes)
> >  {
> 
> Is there a reason you need to move the rcsi2_calc_mbps()? No problem if 
> so, else if you keep it where it is the diff is easier to read ;-)

It's a good practice to move functions around without changing them in a
separate patch first, it makes review easier.

> > -	const struct rcar_csi2_format *format;
> > +	struct v4l2_subdev *source;
> > +	unsigned int bpp;
> > +	s64 link_freq;
> > +	u64 mbps;
> > +
> > +	if (!priv->remote)
> > +		return -ENODEV;
> > +
> > +	source = priv->remote;
> > +
> > +	/*
> > +	 * v4l2_get_link_freq() uses V4L2_CID_LINK_FREQ first, and falls back
> > +	 * to V4L2_CID_PIXEL_RATE if V4L2_CID_LINK_FREQ is not available.
> > +	 *
> > +	 * With multistream input there is no single pixel rate, and thus we
> > +	 * cannot use V4L2_CID_PIXEL_RATE, so we pass 0 as the bpp which
> > +	 * causes v4l2_get_link_freq() to return an error if it falls back to
> > +	 * V4L2_CID_PIXEL_RATE.
> > +	 */
> > +
> > +	if (fd->num_entries > 1) {
> > +		bpp = 0;
> > +	} else {
> > +		struct v4l2_mbus_frame_desc_entry *entry = &fd->entry[0];
> > +		const struct rcar_csi2_format *format;
> > +
> > +		format = rcsi2_datatype_to_fmt(entry->bus.csi2.dt);
> > +		if (WARN_ON(!format))
> > +			return -EINVAL;
> > +
> > +		bpp = format->bpp;
> > +	}
> > +
> > +	/*
> > +	 * Calculate the phypll in mbps.
> > +	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> > +	 * bps = link_freq * 2
> > +	 */
> > +	link_freq = v4l2_get_link_freq(source->ctrl_handler, bpp, 2 * lanes);
> > +	if (link_freq < 0) {
> > +		dev_err(priv->dev, "Failed to get remote subdev link freq\n");
> > +		return link_freq;
> > +	}
> 
> I just want to make sure I understand our discussion about on IRC about 
> this. It's this call you aim to replace by getting the link frequency 
> from the frame descriptor once it's available there? If so I think it 
> would be good to make this series depend on that work already and list 
> it as a dependency.
> 
> > +
> > +	mbps = 2 * link_freq;
> > +	do_div(mbps, 1000000);
> > +
> > +	/* Adjust for C-PHY, divide by 2.8. */
> > +	if (priv->cphy)
> > +		mbps = div_u64(mbps * 5, 14);
> > +
> > +	return mbps;
> > +}
> > +
> > +static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv,
> > +				     struct v4l2_subdev_state *state)
> > +{
> > +	const struct v4l2_subdev_stream_configs *configs;
> >  	u32 phycnt, vcdt = 0, vcdt2 = 0, fld = 0;
> > +	struct v4l2_mbus_frame_desc fd;
> >  	unsigned int lanes;
> > -	unsigned int i;
> >  	int mbps, ret;
> >  
> > -	dev_dbg(priv->dev, "Input size (%ux%u%c)\n",
> > -		priv->mf.width, priv->mf.height,
> > -		priv->mf.field == V4L2_FIELD_NONE ? 'p' : 'i');
> > -
> > -	/* Code is validated in set_fmt. */
> > -	format = rcsi2_code_to_fmt(priv->mf.code);
> > -	if (!format)
> > -		return -EINVAL;
> > +	/* Get information about multiplexed link. */
> > +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> > +	if (ret)
> > +		return ret;
> >  
> >  	/*
> > -	 * Enable all supported CSI-2 channels with virtual channel and
> > -	 * data type matching.
> > +	 * Configure and enable the R-Car CSI-2 channels.
> >  	 *
> > -	 * NOTE: It's not possible to get individual datatype for each
> > -	 *       source virtual channel. Once this is possible in V4L2
> > -	 *       it should be used here.
> > +	 * The VC transmitted on the channel is configured by the [CSI-2->VIN]
> > +	 * link_setup operation, while the data type to match comes from the
> > +	 * remote subdevice.
> >  	 */
> > -	for (i = 0; i < priv->info->num_channels; i++) {
> > -		u32 vcdt_part;
> > +	for (unsigned int i = 0; i < priv->info->num_channels; i++) {
> > +		struct v4l2_mbus_frame_desc_entry *entry = NULL;
> >  
> > +		/* CSI-2 channel disabled (not linked to any VIN). */
> >  		if (priv->channel_vc[i] < 0)
> >  			continue;
> >  
> > -		vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) | VCDT_VCDTN_EN |
> > -			VCDT_SEL_DTN_ON | VCDT_SEL_DT(format->datatype);
> > +		u32 vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) |
> > +				VCDT_VCDTN_EN | VCDT_SEL_DTN_ON;
> > +
> > +		/*
> > +		 * Search the entries that describe the data types on the
> > +		 * MIPI CSI-2 Virtual Channel assigned to this CSI-2 channel.
> > +		 */
> > +		for (unsigned int e = 0; e < fd.num_entries; e++) {
> > +			if (fd.entry[e].bus.csi2.vc != priv->channel_vc[i])
> > +				continue;
> > +
> > +			/*
> > +			 * If multiple data types are sent on the same MIPI
> > +			 * CSI-2 Virtual Channel, disable Data Type matching.
> > +			 */
> > +			if (entry) {
> > +				vcdt_part &= ~VCDT_SEL_DTN_ON;
> > +				break;
> > +			}
> > +
> > +			entry = &fd.entry[e];
> > +			vcdt_part |= VCDT_SEL_DT(entry->bus.csi2.dt);
> > +		}
> > +
> > +		if (!entry)
> > +			continue;
> >  
> >  		/* Store in correct reg and offset. */
> >  		if (i < 2)
> >  			vcdt |= vcdt_part << ((i % 2) * 16);
> >  		else
> >  			vcdt2 |= vcdt_part << ((i % 2) * 16);
> > +
> > +		dev_dbg(priv->dev, "channel %u: VC = %d, datatype = 0x%x\n",
> > +			i, priv->channel_vc[i], entry->bus.csi2.dt);
> >  	}
> >  
> > -	if (priv->mf.field == V4L2_FIELD_ALTERNATE) {
> > -		fld = FLD_DET_SEL(1) | FLD_FLD_EN4 | FLD_FLD_EN3 | FLD_FLD_EN2
> > -			| FLD_FLD_EN;
> > +	/*
> > +	 * Configure field handling inspecting the formats of the
> > +	 * sink pad streams.
> > +	 */
> > +	configs = &state->stream_configs;
> > +	for (unsigned int i = 0; i < configs->num_configs; ++i) {
> > +		const struct v4l2_subdev_stream_config *config = configs->configs;
> > +
> > +		if (config->pad != RCAR_CSI2_SINK)
> > +			continue;
> >  
> > -		if (priv->mf.height == 240)
> > +		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> > +			continue;
> > +
> > +		fld |= FLD_DET_SEL(1) | FLD_FLD_EN(config->stream);

The stream ID is set by the source, and could be anything. You can't
assume it's a virtual channel number.

> > +
> > +		/* PAL vs NTSC. */
> > +		if (config->fmt.height == 240)
> >  			fld |= FLD_FLD_NUM(0);
> >  		else
> >  			fld |= FLD_FLD_NUM(1);
> > @@ -870,7 +963,7 @@ static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
> >  	phycnt = PHYCNT_ENABLECLK;
> >  	phycnt |= (1 << lanes) - 1;
> >  
> > -	mbps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> > +	mbps = rcsi2_calc_mbps(priv, &fd, lanes);
> >  	if (mbps < 0)
> >  		return mbps;
> >  
> > @@ -1049,23 +1142,24 @@ static int rcsi2_c_phy_setting_v4h(struct rcar_csi2 *priv, int msps)
> >  	return 0;
> >  }
> >  
> > -static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
> > +static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv,
> > +				    struct v4l2_subdev_state *state)
> >  {
> > -	const struct rcar_csi2_format *format;
> > +	struct v4l2_mbus_frame_desc fd;
> >  	unsigned int lanes;
> >  	int msps;
> >  	int ret;
> >  
> >  	/* Calculate parameters */
> > -	format = rcsi2_code_to_fmt(priv->mf.code);
> > -	if (!format)
> > -		return -EINVAL;
> > -
> >  	ret = rcsi2_get_active_lanes(priv, &lanes);
> >  	if (ret)
> >  		return ret;
> >  
> > -	msps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> > +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> > +	if (ret)
> > +		return ret;
> > +
> > +	msps = rcsi2_calc_mbps(priv, &fd, lanes);
> >  	if (msps < 0)
> >  		return msps;
> >  
> > @@ -1114,7 +1208,7 @@ static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
> >  	return 0;
> >  }
> >  
> > -static int rcsi2_start(struct rcar_csi2 *priv)
> > +static int rcsi2_start(struct rcar_csi2 *priv, struct v4l2_subdev_state *state)
> >  {
> >  	int ret;
> >  
> > @@ -1122,7 +1216,7 @@ static int rcsi2_start(struct rcar_csi2 *priv)
> >  	if (ret < 0)
> >  		return ret;
> >  
> > -	ret = priv->info->start_receiver(priv);
> > +	ret = priv->info->start_receiver(priv, state);
> >  	if (ret) {
> >  		rcsi2_enter_standby(priv);
> >  		return ret;
> > @@ -1146,26 +1240,24 @@ static void rcsi2_stop(struct rcar_csi2 *priv)
> >  static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> >  {
> >  	struct rcar_csi2 *priv = sd_to_csi2(sd);
> > +	struct v4l2_subdev_state *state;
> >  	int ret = 0;
> >  
> > -	mutex_lock(&priv->lock);
> > +	if (!priv->remote)
> > +		return -ENODEV;
> >  
> > -	if (!priv->remote) {
> > -		ret = -ENODEV;
> > -		goto out;
> > -	}
> > +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
> >  
> > -	if (enable && priv->stream_count == 0) {
> > -		ret = rcsi2_start(priv);
> > -		if (ret)
> > -			goto out;
> > -	} else if (!enable && priv->stream_count == 1) {
> > +	if (enable && priv->stream_count == 0)
> > +		ret = rcsi2_start(priv, state);
> > +	else if (!enable && priv->stream_count == 1)
> >  		rcsi2_stop(priv);
> > -	}
> > +	if (ret)
> > +		goto out;
> >  
> >  	priv->stream_count += enable ? 1 : -1;
> >  out:
> > -	mutex_unlock(&priv->lock);
> > +	v4l2_subdev_unlock_state(state);
> >  
> >  	return ret;
> >  }
> > @@ -1310,14 +1402,17 @@ static irqreturn_t rcsi2_irq(int irq, void *data)
> >  
> >  static irqreturn_t rcsi2_irq_thread(int irq, void *data)
> >  {
> > +	struct v4l2_subdev_state *state;
> >  	struct rcar_csi2 *priv = data;
> >  
> > -	mutex_lock(&priv->lock);
> > +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
> > +
> >  	rcsi2_stop(priv);
> >  	usleep_range(1000, 2000);
> > -	if (rcsi2_start(priv))
> > +	if (rcsi2_start(priv, state))
> >  		dev_warn(priv->dev, "Failed to restart CSI-2 receiver\n");
> > -	mutex_unlock(&priv->lock);
> > +
> > +	v4l2_subdev_unlock_state(state);
> >  
> >  	return IRQ_HANDLED;
> >  }
> > @@ -1340,6 +1435,13 @@ static int rcsi2_notify_bound(struct v4l2_async_notifier *notifier,
> >  		return pad;
> >  	}
> >  
> > +	if (!v4l2_subdev_has_op(subdev, pad, get_frame_desc)) {
> > +		dev_err(priv->dev,
> > +			"Subdev %s bound failed: missing get_frame_desc()\n",
> > +			subdev->name);
> > +		return -EINVAL;
> > +	}
> > +
> >  	priv->remote = subdev;
> >  	priv->remote_pad = pad;
> >  

-- 
Regards,

Laurent Pinchart

Re: [PATCH 17/19] media: rcar-csi2: Store format in the subdev state

[Laurent Pinchart]

On Thu, May 02, 2024 at 04:32:32PM +0200, Niklas Söderlund wrote:
> Hi Jacpop,
> 
> Thanks for your work.
> 
> On 2024-04-30 12:39:53 +0200, Jacopo Mondi wrote:
> > Store the format in the subdevice state. Disallow setting format
> > on the source pads, as formats are set on the sink pad streams and
> > propagated to the source streams.
> > 
> > Now that the driver doesn't store the active format in the
> > driver-specific structure, also remove the mutex and use the lock
> > associated with the state.
> 
> Can't this whole patch be broken out to an independent patch and 
> upstreamed already independent from the streams work?

I think it's a good idea. We will need to move part of 15/19 here
(adding .init_state() and calling v4l2_subdev_init_finalize()).

> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/platform/renesas/rcar-csi2.c | 54 +++++++---------------
> >  1 file changed, 16 insertions(+), 38 deletions(-)
> > 
> > diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> > index ffb73272543b..ed818a6fa655 100644
> > --- a/drivers/media/platform/renesas/rcar-csi2.c
> > +++ b/drivers/media/platform/renesas/rcar-csi2.c
> > @@ -621,8 +621,6 @@ struct rcar_csi2 {
> >  
> >  	int channel_vc[4];
> >  
> > -	struct mutex lock; /* Protects mf and stream_count. */
> > -	struct v4l2_mbus_framefmt mf;
> >  	int stream_count;
> >  
> >  	bool cphy;
> > @@ -1263,43 +1261,28 @@ static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> >  }
> >  
> >  static int rcsi2_set_pad_format(struct v4l2_subdev *sd,
> > -				struct v4l2_subdev_state *sd_state,
> > +				struct v4l2_subdev_state *state,
> >  				struct v4l2_subdev_format *format)
> >  {
> > -	struct rcar_csi2 *priv = sd_to_csi2(sd);
> > -	struct v4l2_mbus_framefmt *framefmt;
> > +	struct v4l2_mbus_framefmt *fmt;
> >  
> > -	mutex_lock(&priv->lock);
> > +	/*
> > +	 * Format is propagated from sink streams to source streams, so
> > +	 * disallow setting format on the source pads.
> > +	 */
> > +	if (format->pad > RCAR_CSI2_SINK)
> > +		return -EINVAL;

		return v4l2_subdev_get_fmt(sd, state, format);

> >  
> >  	if (!rcsi2_code_to_fmt(format->format.code))
> >  		format->format.code = rcar_csi2_formats[0].code;
> >  
> > -	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE) {
> > -		priv->mf = format->format;
> > -	} else {
> > -		framefmt = v4l2_subdev_state_get_format(sd_state, 0);
> > -		*framefmt = format->format;
> > -	}
> >  
> > -	mutex_unlock(&priv->lock);
> > +	fmt = v4l2_subdev_state_get_format(state, format->pad, format->stream);
> > +	*fmt = format->format;
> >  
> > -	return 0;
> > -}
> > -
> > -static int rcsi2_get_pad_format(struct v4l2_subdev *sd,
> > -				struct v4l2_subdev_state *sd_state,
> > -				struct v4l2_subdev_format *format)
> > -{
> > -	struct rcar_csi2 *priv = sd_to_csi2(sd);
> > -
> > -	mutex_lock(&priv->lock);
> > -
> > -	if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE)
> > -		format->format = priv->mf;
> > -	else
> > -		format->format = *v4l2_subdev_state_get_format(sd_state, 0);
> > -
> > -	mutex_unlock(&priv->lock);
> > +	fmt = v4l2_subdev_state_get_opposite_stream_format(state, format->pad,
> > +							   format->stream);
> > +	*fmt = format->format;
> >  
> >  	return 0;
> >  }
> > @@ -1310,7 +1293,7 @@ static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
> >  
> >  static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
> >  	.set_fmt = rcsi2_set_pad_format,
> > -	.get_fmt = rcsi2_get_pad_format,
> > +	.get_fmt = v4l2_subdev_get_fmt,
> >  };
> >  
> >  static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
> > @@ -2031,20 +2014,19 @@ static int rcsi2_probe(struct platform_device *pdev)
> >  
> >  	priv->dev = &pdev->dev;
> >  
> > -	mutex_init(&priv->lock);
> >  	priv->stream_count = 0;
> >  
> >  	ret = rcsi2_probe_resources(priv, pdev);
> >  	if (ret) {
> >  		dev_err(priv->dev, "Failed to get resources\n");
> > -		goto error_mutex;
> > +		return ret;
> >  	}
> >  
> >  	platform_set_drvdata(pdev, priv);
> >  
> >  	ret = rcsi2_parse_dt(priv);
> >  	if (ret)
> > -		goto error_mutex;
> > +		return ret;
> >  
> >  	priv->subdev.owner = THIS_MODULE;
> >  	priv->subdev.dev = &pdev->dev;
> > @@ -2094,8 +2076,6 @@ static int rcsi2_probe(struct platform_device *pdev)
> >  error_async:
> >  	v4l2_async_nf_unregister(&priv->notifier);
> >  	v4l2_async_nf_cleanup(&priv->notifier);
> > -error_mutex:
> > -	mutex_destroy(&priv->lock);
> >  
> >  	return ret;
> >  }
> > @@ -2110,8 +2090,6 @@ static void rcsi2_remove(struct platform_device *pdev)
> >  	v4l2_subdev_cleanup(&priv->subdev);
> >  
> >  	pm_runtime_disable(&pdev->dev);
> > -
> > -	mutex_destroy(&priv->lock);
> >  }
> >  
> >  static struct platform_driver rcar_csi2_pdrv = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH 18/19] media: rcar-csi2: Implement set_routing

[Laurent Pinchart]

Hi Jacopo,

Thank you for the patch.

On Tue, Apr 30, 2024 at 12:39:54PM +0200, Jacopo Mondi wrote:
> Add the set_routing() subdev operation to allow userspace to activate
> routes on the R-Car CSI-2 receiver.
> 
> Routing for this device is fixed. Validate that the provided route
> respects it.
> 
> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> ---
>  drivers/media/platform/renesas/rcar-csi2.c | 48 +++++++++++++++++++++-
>  1 file changed, 47 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> index ed818a6fa655..47cb12f077e5 100644
> --- a/drivers/media/platform/renesas/rcar-csi2.c
> +++ b/drivers/media/platform/renesas/rcar-csi2.c
> @@ -1287,6 +1287,51 @@ static int rcsi2_set_pad_format(struct v4l2_subdev *sd,
>  	return 0;
>  }
>  
> +static int rcsi2_apply_routing(struct v4l2_subdev *sd,
> +			       struct v4l2_subdev_state *state,
> +			       struct v4l2_subdev_krouting *routing)
> +{
> +	int ret;
> +
> +	ret = v4l2_subdev_routing_validate(sd, routing,
> +					   V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
> +	if (ret)
> +		return ret;
> +
> +	/*
> +	 * Routing is fixed for this device.
> +	 *
> +	 * Only routes in the form of CSI2:0/x->CSI2:x+1/0 are allowed.
> +	 *
> +	 * We have anyway to implement set_routing to mark the route as active.

Then it's not fixed :-)

> +	 */
> +	for (unsigned int i = 0; i < routing->num_routes; ++i) {
> +		const struct v4l2_subdev_route *route = &routing->routes[i];
> +		unsigned int pad = route->sink_stream + 1;
> +
> +		if (route->sink_pad != 0)
> +			return -EINVAL;

I don't think this can happen, the core should check the validity of
pads for you.

> +
> +		if (route->source_pad != pad || route->source_stream != 0)
> +			return -EINVAL;

You should adjust invalid routes, not return an error.

> +	}
> +
> +	return v4l2_subdev_set_routing(sd, state, routing);
> +}
> +
> +static int rcsi2_set_routing(struct v4l2_subdev *sd,
> +			     struct v4l2_subdev_state *state,
> +			     enum v4l2_subdev_format_whence which,
> +			     struct v4l2_subdev_krouting *routing)
> +{
> +	struct rcar_csi2 *priv = sd_to_csi2(sd);
> +
> +	if (which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->stream_count)
> +		return -EBUSY;
> +
> +	return rcsi2_apply_routing(sd, state, routing);
> +}
> +
>  static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
>  	.s_stream = rcsi2_s_stream,
>  };
> @@ -1294,6 +1339,7 @@ static const struct v4l2_subdev_video_ops rcar_csi2_video_ops = {
>  static const struct v4l2_subdev_pad_ops rcar_csi2_pad_ops = {
>  	.set_fmt = rcsi2_set_pad_format,
>  	.get_fmt = v4l2_subdev_get_fmt,
> +	.set_routing = rcsi2_set_routing,
>  };
>  
>  static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
> @@ -1353,7 +1399,7 @@ static int rcsi2_init_state(struct v4l2_subdev *sd,
>  		.routes = routes,
>  	};
>  
> -	return v4l2_subdev_set_routing(sd, state, &routing);
> +	return rcsi2_apply_routing(sd, state, &routing);
>  }
>  
>  static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {

-- 
Regards,

Laurent Pinchart

Re: [PATCH 19/19] media: rcar-vin: Fix YUYV8_1X16 handling for CSI-2

[Laurent Pinchart]

On Thu, May 02, 2024 at 04:33:26PM +0200, Niklas Söderlund wrote:
> Hi Jacopo,
> 
> Same for this patch, can't we fix this issue independent from the 
> streams work?

+1

> On 2024-04-30 12:39:55 +0200, Jacopo Mondi wrote:
> > The YUYV8_1X16 and UYVY8_1X16 formats are treated as 'ITU-R
> > BT.601/BT.1358 16-bit YCbCr-422 input' (YUV16 - 0x5) in the R-Car VIN
> > driver and are thus disallowed when capturing frames from the R-Car
> > CSI-2 interface according to the hardware manual.
> > 
> > As the 1X16 format variants are meant to be used with serial busses they
> > have to be treated as 'YCbCr-422 8-bit data input' (0x1) when capturing
> > from CSI-2, which is a valid setting for CSI-2.
> > 
> > Commit 78b3f9d75a62 ("media: rcar-vin: Add check that input interface
> > and format are valid") disallowed capturing YUV16 when using the CSI-2
> > interface. Fix this by using YUV8_BT601 for YCbCr422 when CSI-2 is in
> > use.
> > 
> > Fixes: 78b3f9d75a62 ("media: rcar-vin: Add check that input interface and format are valid")
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  .../media/platform/renesas/rcar-vin/rcar-dma.c   | 16 +++++++++++++---
> >  1 file changed, 13 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> > index e2c40abc6d3d..21d5b2815e86 100644
> > --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> > +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> > @@ -742,12 +742,22 @@ static int rvin_setup(struct rvin_dev *vin)
> >  	 */
> >  	switch (vin->mbus_code) {
> >  	case MEDIA_BUS_FMT_YUYV8_1X16:
> > -		/* BT.601/BT.1358 16bit YCbCr422 */
> > -		vnmc |= VNMC_INF_YUV16;
> > +		if (vin->is_csi)
> > +			/* YCbCr422 8-bit */
> > +			vnmc |= VNMC_INF_YUV8_BT601;
> > +		else
> > +			/* BT.601/BT.1358 16bit YCbCr422 */
> > +			vnmc |= VNMC_INF_YUV16;
> >  		input_is_yuv = true;
> >  		break;
> >  	case MEDIA_BUS_FMT_UYVY8_1X16:
> > -		vnmc |= VNMC_INF_YUV16 | VNMC_YCAL;
> > +		if (vin->is_csi)
> > +			/* YCbCr422 8-bit */
> > +			vnmc |= VNMC_INF_YUV8_BT601;
> > +		else
> > +			/* BT.601/BT.1358 16bit YCbCr422 */
> > +			vnmc |= VNMC_INF_YUV16;
> > +		vnmc |= VNMC_YCAL;
> >  		input_is_yuv = true;
> >  		break;
> >  	case MEDIA_BUS_FMT_UYVY8_2X8:

-- 
Regards,

Laurent Pinchart

Re: [PATCH 01/19] media: adv748x: Add support for active state

[Jacopo Mondi]

Hi Laurent

On Thu, May 02, 2024 at 08:34:30PM GMT, Laurent Pinchart wrote:
> Hi Jacopo,
>
> Thank you for the patch.
>
> On Tue, Apr 30, 2024 at 12:39:37PM +0200, Jacopo Mondi wrote:
> > Initialize and use the subdev active state to store the subdevice
> > format.
> >
> > This simplifies the implementation of the get_fmt and set_fmt pad
> > operations.
> >
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/i2c/adv748x/adv748x-csi2.c | 69 ++++--------------------
> >  drivers/media/i2c/adv748x/adv748x.h      |  1 -
> >  2 files changed, 11 insertions(+), 59 deletions(-)
> >
> > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > index 5b265b722394..435b0909bbef 100644
> > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > @@ -139,78 +139,26 @@ static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
> >   * But we must support setting the pad formats for format propagation.
> >   */
> >
> > -static struct v4l2_mbus_framefmt *
> > -adv748x_csi2_get_pad_format(struct v4l2_subdev *sd,
> > -			    struct v4l2_subdev_state *sd_state,
> > -			    unsigned int pad, u32 which)
> > -{
> > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > -
> > -	if (which == V4L2_SUBDEV_FORMAT_TRY)
> > -		return v4l2_subdev_state_get_format(sd_state, pad);
> > -
> > -	return &tx->format;
> > -}
> > -
> > -static int adv748x_csi2_get_format(struct v4l2_subdev *sd,
> > -				   struct v4l2_subdev_state *sd_state,
> > -				   struct v4l2_subdev_format *sdformat)
> > -{
> > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > -	struct adv748x_state *state = tx->state;
> > -	struct v4l2_mbus_framefmt *mbusformat;
> > -
> > -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> > -						 sdformat->which);
> > -	if (!mbusformat)
> > -		return -EINVAL;
> > -
> > -	mutex_lock(&state->mutex);
> > -
> > -	sdformat->format = *mbusformat;
> > -
> > -	mutex_unlock(&state->mutex);
> > -
> > -	return 0;
> > -}
> > -
> >  static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
> >  				   struct v4l2_subdev_state *sd_state,
> >  				   struct v4l2_subdev_format *sdformat)
> >  {
> > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > -	struct adv748x_state *state = tx->state;
> >  	struct v4l2_mbus_framefmt *mbusformat;
> > -	int ret = 0;
> > -
> > -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> > -						 sdformat->which);
> > -	if (!mbusformat)
> > -		return -EINVAL;
> >
> > -	mutex_lock(&state->mutex);
> > +	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
> >
> > +	/* Format on the source pad is always copied from the sink one. */
> >  	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
> >  		const struct v4l2_mbus_framefmt *sink_fmt;
> >
> > -		sink_fmt = adv748x_csi2_get_pad_format(sd, sd_state,
> > -						       ADV748X_CSI2_SINK,
> > -						       sdformat->which);
> > -
> > -		if (!sink_fmt) {
> > -			ret = -EINVAL;
> > -			goto unlock;
> > -		}
> > -
> > +		sink_fmt = v4l2_subdev_state_get_format(sd_state,
> > +							ADV748X_CSI2_SINK);
> >  		sdformat->format = *sink_fmt;
>
> That's not the right way to do it. You should propagate the format from
> sink to source when pad == ADV748X_CSI2_SINK, and return
> adv748x_csi2_get_format() when pad == ADV748X_CSI2_SOURCE. Otherwise

I think it's done later and this patch doesn't change the currently
implemented behaviour, doesn't it ?

Anyway, I got that you would prefer to squash the first patches in a
single one, so this will be solved


> setting the format on the sink pad will not update the state of the
> source pad, and a get format call on the source pad will return an
> incorrect format.
>
> >  	}
> >
> >  	*mbusformat = sdformat->format;
> >
> > -unlock:
> > -	mutex_unlock(&state->mutex);
> > -
> > -	return ret;
> > +	return 0;
> >  }
> >
> >  static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
> > @@ -228,7 +176,7 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
> >  }
> >
> >  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
> > -	.get_fmt = adv748x_csi2_get_format,
> > +	.get_fmt = v4l2_subdev_get_fmt,
> >  	.set_fmt = adv748x_csi2_set_format,
> >  	.get_mbus_config = adv748x_csi2_get_mbus_config,
> >  };
> > @@ -320,6 +268,11 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
> >  	if (ret)
> >  		goto err_cleanup_subdev;
> >
> > +	tx->sd.state_lock = tx->ctrl_hdl.lock;
>
> Maybe that's addressed in subsequent patches, but do we need a
> device-wide lock ? The code you replace above uses the

device-wide as global to the CSI-2, the HDMI and the AFE subdevices ?

> adv748x_state.mutex lock, which covers all subdevs. I don't think this
> patch introduces race conditions, so this could possibly be handled on
> top.
>
> > +	ret = v4l2_subdev_init_finalize(&tx->sd);
> > +	if (ret)
> > +		goto err_free_ctrl;
> > +
> >  	ret = v4l2_async_register_subdev(&tx->sd);
> >  	if (ret)
> >  		goto err_free_ctrl;
> > diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> > index d2b5e722e997..9bc0121d0eff 100644
> > --- a/drivers/media/i2c/adv748x/adv748x.h
> > +++ b/drivers/media/i2c/adv748x/adv748x.h
> > @@ -75,7 +75,6 @@ enum adv748x_csi2_pads {
> >
> >  struct adv748x_csi2 {
> >  	struct adv748x_state *state;
> > -	struct v4l2_mbus_framefmt format;
> >  	unsigned int page;
> >  	unsigned int port;
> >  	unsigned int num_lanes;
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 03/19] media: adv748x: Use V4L2 streams

[Jacopo Mondi]

Hi Laurent

On Thu, May 02, 2024 at 08:40:51PM GMT, Laurent Pinchart wrote:
> Hi Jacopo,
>
> Thank you for the patch.
>
> On Tue, Apr 30, 2024 at 12:39:39PM +0200, Jacopo Mondi wrote:
> > Initialize the CSI-2 subdevice with the V4L2_SUBDEV_FL_STREAMS flag
> > and initialize a simple routing table by implementing the .init_state()
> > operation.
> >
> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/i2c/adv748x/adv748x-csi2.c | 28 ++++++++++++++++++++++--
> >  1 file changed, 26 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > index 60bf1dc0f58b..d929db7e8ef2 100644
> > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > @@ -59,7 +59,30 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
> >
> >  /* -----------------------------------------------------------------------------
> >   * v4l2_subdev_internal_ops
> > - *
> > + */
> > +
> > +static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
> > +				   struct v4l2_subdev_state *state)
> > +{
> > +	struct v4l2_subdev_route routes[] = {
> > +		{
> > +			.sink_pad = ADV748X_CSI2_SINK,
> > +			.sink_stream = 0,
> > +			.source_pad = ADV748X_CSI2_SOURCE,
> > +			.source_stream = 0,
> > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > +		},
> > +	};
> > +
> > +	struct v4l2_subdev_krouting routing = {
> > +		.num_routes = ARRAY_SIZE(routes),
> > +		.routes = routes,
> > +	};
> > +
> > +	return v4l2_subdev_set_routing(sd, state, &routing);
>
> You need to initialize formats too.
>

The adv748x driver handles formats very poorly, doesn't implement
enum_mbus_codes and does not allow userspace to change the format
(while at the same time it doesn't check that the format is the
expected one in set_format()).

This is from a freshly booted renesas-drivers/main

- entity 30: adv748x 4-0070 txa (2 pads, 3 links, 0 routes)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev5
        pad0: Sink
                [stream:0 fmt:unknown/0x0]
                <- "adv748x 4-0070 afe":8 []
                <- "adv748x 4-0070 hdmi":1 [ENABLED]
        pad1: Source
                [stream:0 fmt:unknown/0x0]
                -> "rcar_csi2 feaa0000.csi2":0 [ENABLED,IMMUTABLE]

It would probably be better to handle the formats properly and the
introduce streams or use the introduction of streams to also fix the
format handling ?

> > +}
> > +
> > +/*
> >   * We use the internal registered operation to be able to ensure that our
> >   * incremental subdevices (not connected in the forward path) can be registered
> >   * against the resulting video path and media device.
> > @@ -109,6 +132,7 @@ static int adv748x_csi2_registered(struct v4l2_subdev *sd)
> >  }
> >
> >  static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
> > +	.init_state = adv748x_csi2_init_state,
>
> The .init_state() operation needs to be provided along with the call to
> v4l2_subdev_init_finalize() in patch 01/19.
>

I'll squash, however even if it might be a requirement for having a
fully working implementation, not having init_state() will not lead to
any crash and maybe smaller incremental patches are easier to handle.

	if (sd->internal_ops && sd->internal_ops->init_state) {
		/*
		 * There can be no race at this point, but we lock the state
		 * anyway to satisfy lockdep checks.
		 */
		v4l2_subdev_lock_state(state);
		ret = sd->internal_ops->init_state(sd, state);
		v4l2_subdev_unlock_state(state);


> >  	.registered = adv748x_csi2_registered,
> >  };
> >
> > @@ -245,7 +269,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
> >  		return 0;
> >
> >  	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
> > -			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
> > +			    MEDIA_ENT_F_VID_IF_BRIDGE, V4L2_SUBDEV_FL_STREAMS,
> >  			    is_txa(tx) ? "txa" : "txb");
> >
> >  	/* Register internal ops for incremental subdev registration */
> > --
> > 2.44.0
> >
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 05/19] media: adv748x: Implement set_routing()

[Jacopo Mondi]

Hi Laurent

On Thu, May 02, 2024 at 08:49:59PM GMT, Laurent Pinchart wrote:
> Hi Jacopo,
>
> Thank you for the patch.
>
> On Tue, Apr 30, 2024 at 12:39:41PM +0200, Jacopo Mondi wrote:
> > Implement the set_routing() pad operation to control the MIPI CSI-2
> > Virtual Channel on which the video stream is sent on according to
> > the active route source stream number.
>
> While 01/19 needs to implement .init_state(), you should only initialize
> formats there. The routing initialization of 03/19 should be moved to
> this patch.
>

ok, even if I'm not sure I understand why routing initialization and
support for writing the routing table have to come together

> > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > ---
> >  drivers/media/i2c/adv748x/adv748x-csi2.c | 43 +++++++++++++++++++++++-
> >  1 file changed, 42 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > index ace4e1d904d9..7fa72340e66e 100644
> > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > @@ -57,6 +57,38 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
> >  	return 0;
> >  }
> >
> > +static int adv748x_csi2_apply_routing(struct v4l2_subdev *sd,
> > +				      struct v4l2_subdev_state *state,
> > +				      struct v4l2_subdev_krouting *routing)
> > +{
> > +	struct v4l2_subdev_route *route;
> > +	int ret;
> > +
> > +	/* Only one route at the time can be active. */
>
> s/the time/a time/
>
> > +	if (routing->num_routes > 1)
> > +		return -EINVAL;
>
> You should adjust routes instead of returning -EINVAL.
>
> > +
> > +	/*
> > +	 * Validate the route: sink pad and sink stream shall be 0 and only
> > +	 * 4 source streams are supported (one for each supported MIPI CSI-2
> > +	 * channel).
>
> s/channel/virtual channel/
>
> > +	 */
> > +	route = &routing->routes[0];
> > +
> > +	if (route->sink_pad != ADV748X_CSI2_SINK || route->sink_stream)
> > +		return -EINVAL;
> > +	if (route->source_pad != ADV748X_CSI2_SOURCE ||
> > +	    route->source_stream > 4)
> > +		return -EINVAL;
>
> Adjust instead of returning an error. The pad checks can be dropped, as
> the core ensures sink_pad and source_pad reference a valid sink and
> source pad respectively.
>

ack

> I'm not sure the source stream check is right either. I understand
> you'll use that to select a virtual channel, but the routing API isn't
> meant to let userspace configure virtual channel numbers explicitly.
>

Ok, this surprises me, more on the next patch

> > +
> > +	ret = v4l2_subdev_routing_validate(sd, routing,
> > +					   V4L2_SUBDEV_ROUTING_ONLY_1_TO_1);
> > +	if (ret)
> > +		return ret;
> > +
> > +	return v4l2_subdev_set_routing(sd, state, routing);
> > +}
> > +
> >  /* -----------------------------------------------------------------------------
> >   * v4l2_subdev_internal_ops
> >   */
> > @@ -79,7 +111,7 @@ static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
> >  		.routes = routes,
> >  	};
> >
> > -	return v4l2_subdev_set_routing(sd, state, &routing);
> > +	return adv748x_csi2_apply_routing(sd, state, &routing);
> >  }
> >
> >  /*
> > @@ -200,10 +232,19 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
> >  	return 0;
> >  }
> >
> > +static int adv748x_csi2_set_routing(struct v4l2_subdev *sd,
> > +				    struct v4l2_subdev_state *state,
> > +				    enum v4l2_subdev_format_whence which,
> > +				    struct v4l2_subdev_krouting *routing)
> > +{
> > +	return adv748x_csi2_apply_routing(sd, state, routing);
> > +}
> > +
> >  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
> >  	.get_fmt = v4l2_subdev_get_fmt,
> >  	.set_fmt = adv748x_csi2_set_format,
> >  	.get_mbus_config = adv748x_csi2_get_mbus_config,
> > +	.set_routing = adv748x_csi2_set_routing,
> >  };
> >
> >  /* -----------------------------------------------------------------------------
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 06/19] media: adv748x: Use routes to configure VC

[Jacopo Mondi]

On Thu, May 02, 2024 at 08:52:34PM GMT, Laurent Pinchart wrote:
> On Thu, May 02, 2024 at 08:51:16PM +0300, Laurent Pinchart wrote:
> > Hi Jacopo,
> >
> > Thank you for the patch.
> >
> > On Tue, Apr 30, 2024 at 12:39:42PM +0200, Jacopo Mondi wrote:
> > > Use the newly introduced routing table to configure on which MIPI
> > > CSI-2 Virtual Channel to send the image stream on.
> >
> > The stream ID in the routing API isn't meant to be mapped directly to a
> > virtual channel number.
>

Isn't that up to the device to decide what a stream number represent ?

> Additionally, why do you need to make the virtual channel configurable,
> instead of allocating them dynamically ?

What do you mean with "allocating them dinamically" ?

Anyway, I guess the main question is:

- How to control VC selection if the stream number is not the right
  way to do that
- Has VC selection any value at all

>
> > Sakari, your opinion would be appreciated.
> >
> > > Perform Virtual Channel selection at s_stream() time instead of
> > > forcing it to 0 during the chip reset.
> > >
> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > > ---
> > >  drivers/media/i2c/adv748x/adv748x-core.c |  8 ++------
> > >  drivers/media/i2c/adv748x/adv748x-csi2.c | 22 ++++++++++++++++++++--
> > >  drivers/media/i2c/adv748x/adv748x.h      |  1 -
> > >  3 files changed, 22 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/drivers/media/i2c/adv748x/adv748x-core.c b/drivers/media/i2c/adv748x/adv748x-core.c
> > > index 3abc73ea8ccb..c9d917135709 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x-core.c
> > > +++ b/drivers/media/i2c/adv748x/adv748x-core.c
> > > @@ -530,14 +530,10 @@ static int adv748x_reset(struct adv748x_state *state)
> > >  	io_write(state, ADV748X_IO_PD, ADV748X_IO_PD_RX_EN);
> > >
> > >  	/* Conditionally enable TXa and TXb. */
> > > -	if (is_tx_enabled(&state->txa)) {
> > > +	if (is_tx_enabled(&state->txa))
> > >  		regval |= ADV748X_IO_10_CSI4_EN;
> > > -		adv748x_csi2_set_virtual_channel(&state->txa, 0);
> > > -	}
> > > -	if (is_tx_enabled(&state->txb)) {
> > > +	if (is_tx_enabled(&state->txb))
> > >  		regval |= ADV748X_IO_10_CSI1_EN;
> > > -		adv748x_csi2_set_virtual_channel(&state->txb, 0);
> > > -	}
> > >  	io_write(state, ADV748X_IO_10, regval);
> > >
> > >  	/* Use vid_std and v_freq as freerun resolution for CP */
> > > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > index 7fa72340e66e..a7bfed393ff0 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > @@ -14,7 +14,8 @@
> > >
> > >  #include "adv748x.h"
> > >
> > > -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc)
> > > +static int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx,
> > > +					    unsigned int vc)
> > >  {
> > >  	return tx_write(tx, ADV748X_CSI_VC_REF, vc << ADV748X_CSI_VC_REF_SHIFT);
> > >  }
> > > @@ -175,13 +176,30 @@ static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
> > >  static int adv748x_csi2_s_stream(struct v4l2_subdev *sd, int enable)
> > >  {
> > >  	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > > +	struct v4l2_subdev_state *state;
> > >  	struct v4l2_subdev *src;
> > > +	int ret;
> > >
> > >  	src = adv748x_get_remote_sd(&tx->pads[ADV748X_CSI2_SINK]);
> > >  	if (!src)
> > >  		return -EPIPE;
> > >
> > > -	return v4l2_subdev_call(src, video, s_stream, enable);
> > > +	state = v4l2_subdev_lock_and_get_active_state(sd);
> > > +
> > > +	if (enable) {
> > > +		/* A single route is available. */
> > > +		struct v4l2_subdev_route *route = &state->routing.routes[0];
> > > +
> > > +		ret = adv748x_csi2_set_virtual_channel(tx, route->source_stream);
> > > +		if (ret)
> > > +			goto unlock;
> > > +	}
> > > +
> > > +	ret = v4l2_subdev_call(src, video, s_stream, enable);
> > > +unlock:
> > > +	v4l2_subdev_unlock_state(state);
> > > +
> > > +	return ret;
> > >  }
> > >
> > >  static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
> > > diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> > > index be24bc57767c..95d04468af9d 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x.h
> > > +++ b/drivers/media/i2c/adv748x/adv748x.h
> > > @@ -434,7 +434,6 @@ int adv748x_afe_s_input(struct adv748x_afe *afe, unsigned int input);
> > >
> > >  int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx);
> > >  void adv748x_csi2_cleanup(struct adv748x_csi2 *tx);
> > > -int adv748x_csi2_set_virtual_channel(struct adv748x_csi2 *tx, unsigned int vc);
> > >  int adv748x_csi2_set_pixelrate(struct v4l2_subdev *sd, s64 rate);
> > >
> > >  int adv748x_hdmi_init(struct adv748x_hdmi *hdmi);
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 15/19] media: rcar-csi2: Add support for multiplexed streams

[Jacopo Mondi]

Hi Laurent, Niklas

On Thu, May 02, 2024 at 09:41:24PM GMT, Laurent Pinchart wrote:
> On Thu, May 02, 2024 at 04:23:16PM +0200, Niklas Söderlund wrote:
> > Hi Jacopo,
> >
> > Thanks for your work.
> >
> > On 2024-04-30 12:39:51 +0200, Jacopo Mondi wrote:
> > > Create and initialize the v4l2_subdev_state for the R-Car CSI-2 receiver
> > > in order to prepare to support multiplexed transmitters.
> > >
> > > Create the subdevice state with v4l2_subdev_init_finalize() and
> > > implement the init_state() operation to guarantee the state is initialized.
> > >
> > > The routing table within the R-Car CSI-2 receiver is fixed, streams
> > > received on source_stream X will be directed to pad (X + 1) by default.
> > > Initialize a static routing table with such routes set as active.
> > >
> > > While at it, disable runtime_pm() in the probe() function error path.
>
> s/runtime_pm()/runtime PM/
>
> > Can we break this out in a separate patch? As the multiplexed stream
> > work will not be ready for v6.10 if I understood the tendencies
> > correctly (?), we can at least fix this issue before that.
>
> Agreed, the fix should be a separate patch, with a Fixes: tag.

I'll collect this and the two other smaller fixes in a separate series

>
> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > > ---
> > >  drivers/media/platform/renesas/rcar-csi2.c | 74 +++++++++++++++++++++-
> > >  1 file changed, 72 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> > > index 582d5e35db0e..82dc0b92b8d3 100644
> > > --- a/drivers/media/platform/renesas/rcar-csi2.c
> > > +++ b/drivers/media/platform/renesas/rcar-csi2.c
> > > @@ -1226,6 +1226,65 @@ static const struct v4l2_subdev_ops rcar_csi2_subdev_ops = {
> > >  	.pad	= &rcar_csi2_pad_ops,
> > >  };
> > >
> > > +static int rcsi2_init_state(struct v4l2_subdev *sd,
> > > +			    struct v4l2_subdev_state *state)
> > > +{
> > > +	/*
> > > +	 * Routing is fixed for this device: streams sent on sink_stream X
> > > +	 * are directed to pad (X + 1). Which streams goes to the next
> > > +	 * processing block (VIN) is controlled by link enablement between the
> > > +	 * CSI-2 and the VIN itself and not by the CSI-2 routing table.
> > > +	 *
> > > +	 * The routing table is then fixed, as stream X will be directed to
> > > +	 * csi:(X + 1)/0 and will be transmitted to VINs the on media link
> > > +	 * csi2:(x + 1)->vin:0.
> > > +	 *
> > > +	 * For example, to route stream #3 to VIN #1 : "csi2:4/0 -> vin1:0" and
> > > +	 * to route stream #2 to VIN #4 : "csi2:3/0 -> vin4:0".
> > > +	 */
> > > +	struct v4l2_subdev_route routes[] = {
> > > +		{
> > > +			.sink_pad = RCAR_CSI2_SINK,
> > > +			.sink_stream = 0,
> > > +			.source_pad = RCAR_CSI2_SOURCE_VC0,
> > > +			.source_stream = 0,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		},
> > > +		{
> > > +			.sink_pad = RCAR_CSI2_SINK,
> > > +			.sink_stream = 1,
> > > +			.source_pad = RCAR_CSI2_SOURCE_VC1,
> > > +			.source_stream = 0,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		},
> > > +		{
> > > +			.sink_pad = RCAR_CSI2_SINK,
> > > +			.sink_stream = 2,
> > > +			.source_pad = RCAR_CSI2_SOURCE_VC2,
> > > +			.source_stream = 0,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		},
> > > +		{
> > > +			.sink_pad = RCAR_CSI2_SINK,
> > > +			.sink_stream = 3,
> > > +			.source_pad = RCAR_CSI2_SOURCE_VC3,
> > > +			.source_stream = 0,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		},
> > > +	};
> > > +
> > > +	struct v4l2_subdev_krouting routing = {
> > > +		.num_routes = ARRAY_SIZE(routes),
> > > +		.routes = routes,
> > > +	};
> >
> > Should not the two structs above be static const as you return a pointer
> > to them?
>
> v4l2_subdev_set_routing() takes a non-const v4l2_subdev_krouting
> argument. Unless I'm missing something, this function doesn't return a
> pointer to the local structures.
>
> > > +
> > > +	return v4l2_subdev_set_routing(sd, state, &routing);
> > > +}
> > > +
> > > +static const struct v4l2_subdev_internal_ops rcar_csi2_internal_ops = {
> > > +	.init_state = rcsi2_init_state,
> > > +};
> > > +
> > >  static irqreturn_t rcsi2_irq(int irq, void *data)
> > >  {
> > >  	struct rcar_csi2 *priv = data;
> > > @@ -1887,11 +1946,13 @@ static int rcsi2_probe(struct platform_device *pdev)
> > >
> > >  	priv->subdev.owner = THIS_MODULE;
> > >  	priv->subdev.dev = &pdev->dev;
> > > +	priv->subdev.internal_ops = &rcar_csi2_internal_ops;
> > >  	v4l2_subdev_init(&priv->subdev, &rcar_csi2_subdev_ops);
> > >  	v4l2_set_subdevdata(&priv->subdev, &pdev->dev);
> > >  	snprintf(priv->subdev.name, sizeof(priv->subdev.name), "%s %s",
> > >  		 KBUILD_MODNAME, dev_name(&pdev->dev));
> > > -	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE;
> > > +	priv->subdev.flags = V4L2_SUBDEV_FL_HAS_DEVNODE |
> > > +			     V4L2_SUBDEV_FL_STREAMS;
> > >
> > >  	priv->subdev.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
> > >  	priv->subdev.entity.ops = &rcar_csi2_entity_ops;
> > > @@ -1912,14 +1973,22 @@ static int rcsi2_probe(struct platform_device *pdev)
> > >
> > >  	pm_runtime_enable(&pdev->dev);
> > >
> > > +	ret = v4l2_subdev_init_finalize(&priv->subdev);
> > > +	if (ret)
> > > +		goto error_pm_runtime;
> > > +
> > >  	ret = v4l2_async_register_subdev(&priv->subdev);
> > >  	if (ret < 0)
> > > -		goto error_async;
> > > +		goto error_subdev;
> > >
> > >  	dev_info(priv->dev, "%d lanes found\n", priv->lanes);
> > >
> > >  	return 0;
> > >
> > > +error_subdev:
> > > +	v4l2_subdev_cleanup(&priv->subdev);
> > > +error_pm_runtime:
> > > +	pm_runtime_disable(&pdev->dev);
> > >  error_async:
> > >  	v4l2_async_nf_unregister(&priv->notifier);
> > >  	v4l2_async_nf_cleanup(&priv->notifier);
> > > @@ -1936,6 +2005,7 @@ static void rcsi2_remove(struct platform_device *pdev)
> > >  	v4l2_async_nf_unregister(&priv->notifier);
> > >  	v4l2_async_nf_cleanup(&priv->notifier);
> > >  	v4l2_async_unregister_subdev(&priv->subdev);
> > > +	v4l2_subdev_cleanup(&priv->subdev);
> > >
> > >  	pm_runtime_disable(&pdev->dev);
> > >
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 16/19] media: rcar-csi2: Support multiplexed transmitters

[Jacopo Mondi]

Hi Laurent, Niklas

On Fri, May 03, 2024 at 12:56:47AM GMT, Laurent Pinchart wrote:
> On Thu, May 02, 2024 at 04:30:54PM +0200, Niklas Söderlund wrote:
> > On 2024-04-30 12:39:52 +0200, Jacopo Mondi wrote:
> > > Rework the R-Car CSI-2 start routine to support multiplexed
> > > transmitters.
> > >
> > > Configure the CSI-2 receiver MIPI CSI-2 Data Type filtering by inspecting
> > > the remote subdev frame_desc instead of relying on the image format
> > > configured on the sink pad.
> > >
> > > Enable MIPI CSI-2 Data Type filtering by inspecting the remote subdevice
> > > frame descriptor to discern which Data Type is transmitted on which
> > > Virtual Channel. If multiple Data Types are transmitted on the same VC
> > > then Data Type filtering is disabled.
> > >
> > > Rework the per-lane bandwidth calculation to use the LINK_FREQ control
> > > if a transmitter sends multiple streams on the same data link. The
> > > current usage of the PIXEL_RATE control does not support multiplexed
> > > transmitters, as there's not a unique pixel rate among all the possible
> > > source streams.
>
> It sounds like this should be split in two patches, the LINK_FREQ change
> seems separate from the DT/VC filtering and routing handling.
>
> > >
> > > This change makes mandatory for any subdevice that operates with
> > > the R-Car CSI-2 receiver to implement the .get_frame_desc() operation.
> > >
> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > > ---
> > >  drivers/media/platform/renesas/rcar-csi2.c | 280 ++++++++++++++-------
> > >  1 file changed, 191 insertions(+), 89 deletions(-)
> > >
> > > diff --git a/drivers/media/platform/renesas/rcar-csi2.c b/drivers/media/platform/renesas/rcar-csi2.c
> > > index 82dc0b92b8d3..ffb73272543b 100644
> > > --- a/drivers/media/platform/renesas/rcar-csi2.c
> > > +++ b/drivers/media/platform/renesas/rcar-csi2.c
> > > @@ -69,10 +69,7 @@ struct rcar_csi2;
> > >  #define FLD_REG				0x1c
> > >  #define FLD_FLD_NUM(n)			(((n) & 0xff) << 16)
> > >  #define FLD_DET_SEL(n)			(((n) & 0x3) << 4)
> > > -#define FLD_FLD_EN4			BIT(3)
> > > -#define FLD_FLD_EN3			BIT(2)
> > > -#define FLD_FLD_EN2			BIT(1)
> > > -#define FLD_FLD_EN			BIT(0)
> > > +#define FLD_FLD_EN(n)			BIT((n) & 0xf)
> > >
> > >  /* Automatic Standby Control */
> > >  #define ASTBY_REG			0x20
> > > @@ -575,6 +572,16 @@ static const struct rcar_csi2_format *rcsi2_code_to_fmt(unsigned int code)
> > >  	return NULL;
> > >  }
> > >
> > > +static const struct rcar_csi2_format *rcsi2_datatype_to_fmt(unsigned int dt)
> > > +{
> > > +	for (unsigned int i = 0; i < ARRAY_SIZE(rcar_csi2_formats); i++) {
> > > +		if (rcar_csi2_formats[i].datatype == dt)
> > > +			return &rcar_csi2_formats[i];
> > > +	}
> > > +
> > > +	return NULL;
> > > +}
> > > +
> > >  enum rcar_csi2_pads {
> > >  	RCAR_CSI2_SINK,
> > >  	RCAR_CSI2_SOURCE_VC0,
> > > @@ -587,7 +594,8 @@ enum rcar_csi2_pads {
> > >  struct rcar_csi2_info {
> > >  	int (*init_phtw)(struct rcar_csi2 *priv, unsigned int mbps);
> > >  	int (*phy_post_init)(struct rcar_csi2 *priv);
> > > -	int (*start_receiver)(struct rcar_csi2 *priv);
> > > +	int (*start_receiver)(struct rcar_csi2 *priv,
> > > +			      struct v4l2_subdev_state *state);
> > >  	void (*enter_standby)(struct rcar_csi2 *priv);
> > >  	const struct rcsi2_mbps_reg *hsfreqrange;
> > >  	unsigned int csi0clkfreqrange;
> > > @@ -676,6 +684,32 @@ static int rcsi2_exit_standby(struct rcar_csi2 *priv)
> > >  	return 0;
> > >  }
> > >
> > > +static int rcsi2_get_remote_frame_desc(struct rcar_csi2 *priv,
> > > +				       struct v4l2_mbus_frame_desc *fd)
> > > +{
> > > +	struct media_pad *pad;
> > > +	int ret;
> > > +
> > > +	if (!priv->remote)
> > > +		return -ENOLINK;
> > > +
> > > +	pad = media_pad_remote_pad_unique(&priv->pads[RCAR_CSI2_SINK]);
> > > +	if (IS_ERR(pad))
> > > +		return PTR_ERR(pad);
> > > +
> > > +	ret = v4l2_subdev_call(priv->remote, pad, get_frame_desc,
> > > +			       pad->index, fd);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	if (fd->type != V4L2_MBUS_FRAME_DESC_TYPE_CSI2) {
> > > +		dev_err(priv->dev, "Frame desc does not describe a CSI-2 link");
> > > +		return -EINVAL;
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > >  static int rcsi2_wait_phy_start(struct rcar_csi2 *priv,
> > >  				unsigned int lanes)
> > >  {
> > > @@ -726,41 +760,6 @@ static int rcsi2_set_phypll(struct rcar_csi2 *priv, unsigned int mbps)
> > >  	return 0;
> > >  }
> > >
> > > -static int rcsi2_calc_mbps(struct rcar_csi2 *priv, unsigned int bpp,
> > > -			   unsigned int lanes)
> > > -{
> > > -	struct v4l2_subdev *source;
> > > -	struct v4l2_ctrl *ctrl;
> > > -	u64 mbps;
> > > -
> > > -	if (!priv->remote)
> > > -		return -ENODEV;
> > > -
> > > -	source = priv->remote;
> > > -
> > > -	/* Read the pixel rate control from remote. */
> > > -	ctrl = v4l2_ctrl_find(source->ctrl_handler, V4L2_CID_PIXEL_RATE);
> > > -	if (!ctrl) {
> > > -		dev_err(priv->dev, "no pixel rate control in subdev %s\n",
> > > -			source->name);
> > > -		return -EINVAL;
> > > -	}
> > > -
> > > -	/*
> > > -	 * Calculate the phypll in mbps.
> > > -	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> > > -	 * bps = link_freq * 2
> > > -	 */
> > > -	mbps = v4l2_ctrl_g_ctrl_int64(ctrl) * bpp;
> > > -	do_div(mbps, lanes * 1000000);
> > > -
> > > -	/* Adjust for C-PHY, divide by 2.8. */
> > > -	if (priv->cphy)
> > > -		mbps = div_u64(mbps * 5, 14);
> > > -
> > > -	return mbps;
> > > -}
> > > -
> > >  static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
> > >  				  unsigned int *lanes)
> > >  {
> > > @@ -808,52 +807,146 @@ static int rcsi2_get_active_lanes(struct rcar_csi2 *priv,
> > >  	return 0;
> > >  }
> > >
> > > -static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
> > > +static int rcsi2_calc_mbps(struct rcar_csi2 *priv,
> > > +			   struct v4l2_mbus_frame_desc *fd, unsigned int lanes)
> > >  {
> >
> > Is there a reason you need to move the rcsi2_calc_mbps()? No problem if
> > so, else if you keep it where it is the diff is easier to read ;-)

It's a leftover from where I had to call rcsi2_get_active_lanes() from
rcsi2_calc_mbps() and had to move the former above the latter.

I forgt to re-sort the two function, sorry about that

>
> It's a good practice to move functions around without changing them in a
> separate patch first, it makes review easier.
>
> > > -	const struct rcar_csi2_format *format;
> > > +	struct v4l2_subdev *source;
> > > +	unsigned int bpp;
> > > +	s64 link_freq;
> > > +	u64 mbps;
> > > +
> > > +	if (!priv->remote)
> > > +		return -ENODEV;
> > > +
> > > +	source = priv->remote;
> > > +
> > > +	/*
> > > +	 * v4l2_get_link_freq() uses V4L2_CID_LINK_FREQ first, and falls back
> > > +	 * to V4L2_CID_PIXEL_RATE if V4L2_CID_LINK_FREQ is not available.
> > > +	 *
> > > +	 * With multistream input there is no single pixel rate, and thus we
> > > +	 * cannot use V4L2_CID_PIXEL_RATE, so we pass 0 as the bpp which
> > > +	 * causes v4l2_get_link_freq() to return an error if it falls back to
> > > +	 * V4L2_CID_PIXEL_RATE.
> > > +	 */
> > > +
> > > +	if (fd->num_entries > 1) {
> > > +		bpp = 0;
> > > +	} else {
> > > +		struct v4l2_mbus_frame_desc_entry *entry = &fd->entry[0];
> > > +		const struct rcar_csi2_format *format;
> > > +
> > > +		format = rcsi2_datatype_to_fmt(entry->bus.csi2.dt);
> > > +		if (WARN_ON(!format))
> > > +			return -EINVAL;
> > > +
> > > +		bpp = format->bpp;
> > > +	}
> > > +
> > > +	/*
> > > +	 * Calculate the phypll in mbps.
> > > +	 * link_freq = (pixel_rate * bits_per_sample) / (2 * nr_of_lanes)
> > > +	 * bps = link_freq * 2
> > > +	 */
> > > +	link_freq = v4l2_get_link_freq(source->ctrl_handler, bpp, 2 * lanes);
> > > +	if (link_freq < 0) {
> > > +		dev_err(priv->dev, "Failed to get remote subdev link freq\n");
> > > +		return link_freq;
> > > +	}
> >
> > I just want to make sure I understand our discussion about on IRC about
> > this. It's this call you aim to replace by getting the link frequency
> > from the frame descriptor once it's available there? If so I think it
> > would be good to make this series depend on that work already and list
> > it as a dependency.
> >
> > > +
> > > +	mbps = 2 * link_freq;
> > > +	do_div(mbps, 1000000);
> > > +
> > > +	/* Adjust for C-PHY, divide by 2.8. */
> > > +	if (priv->cphy)
> > > +		mbps = div_u64(mbps * 5, 14);
> > > +
> > > +	return mbps;
> > > +}
> > > +
> > > +static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv,
> > > +				     struct v4l2_subdev_state *state)
> > > +{
> > > +	const struct v4l2_subdev_stream_configs *configs;
> > >  	u32 phycnt, vcdt = 0, vcdt2 = 0, fld = 0;
> > > +	struct v4l2_mbus_frame_desc fd;
> > >  	unsigned int lanes;
> > > -	unsigned int i;
> > >  	int mbps, ret;
> > >
> > > -	dev_dbg(priv->dev, "Input size (%ux%u%c)\n",
> > > -		priv->mf.width, priv->mf.height,
> > > -		priv->mf.field == V4L2_FIELD_NONE ? 'p' : 'i');
> > > -
> > > -	/* Code is validated in set_fmt. */
> > > -	format = rcsi2_code_to_fmt(priv->mf.code);
> > > -	if (!format)
> > > -		return -EINVAL;
> > > +	/* Get information about multiplexed link. */
> > > +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> > > +	if (ret)
> > > +		return ret;
> > >
> > >  	/*
> > > -	 * Enable all supported CSI-2 channels with virtual channel and
> > > -	 * data type matching.
> > > +	 * Configure and enable the R-Car CSI-2 channels.
> > >  	 *
> > > -	 * NOTE: It's not possible to get individual datatype for each
> > > -	 *       source virtual channel. Once this is possible in V4L2
> > > -	 *       it should be used here.
> > > +	 * The VC transmitted on the channel is configured by the [CSI-2->VIN]
> > > +	 * link_setup operation, while the data type to match comes from the
> > > +	 * remote subdevice.
> > >  	 */
> > > -	for (i = 0; i < priv->info->num_channels; i++) {
> > > -		u32 vcdt_part;
> > > +	for (unsigned int i = 0; i < priv->info->num_channels; i++) {
> > > +		struct v4l2_mbus_frame_desc_entry *entry = NULL;
> > >
> > > +		/* CSI-2 channel disabled (not linked to any VIN). */
> > >  		if (priv->channel_vc[i] < 0)
> > >  			continue;
> > >
> > > -		vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) | VCDT_VCDTN_EN |
> > > -			VCDT_SEL_DTN_ON | VCDT_SEL_DT(format->datatype);
> > > +		u32 vcdt_part = VCDT_SEL_VC(priv->channel_vc[i]) |
> > > +				VCDT_VCDTN_EN | VCDT_SEL_DTN_ON;
> > > +
> > > +		/*
> > > +		 * Search the entries that describe the data types on the
> > > +		 * MIPI CSI-2 Virtual Channel assigned to this CSI-2 channel.
> > > +		 */
> > > +		for (unsigned int e = 0; e < fd.num_entries; e++) {
> > > +			if (fd.entry[e].bus.csi2.vc != priv->channel_vc[i])
> > > +				continue;
> > > +
> > > +			/*
> > > +			 * If multiple data types are sent on the same MIPI
> > > +			 * CSI-2 Virtual Channel, disable Data Type matching.
> > > +			 */
> > > +			if (entry) {
> > > +				vcdt_part &= ~VCDT_SEL_DTN_ON;
> > > +				break;
> > > +			}
> > > +
> > > +			entry = &fd.entry[e];
> > > +			vcdt_part |= VCDT_SEL_DT(entry->bus.csi2.dt);
> > > +		}
> > > +
> > > +		if (!entry)
> > > +			continue;
> > >
> > >  		/* Store in correct reg and offset. */
> > >  		if (i < 2)
> > >  			vcdt |= vcdt_part << ((i % 2) * 16);
> > >  		else
> > >  			vcdt2 |= vcdt_part << ((i % 2) * 16);
> > > +
> > > +		dev_dbg(priv->dev, "channel %u: VC = %d, datatype = 0x%x\n",
> > > +			i, priv->channel_vc[i], entry->bus.csi2.dt);
> > >  	}
> > >
> > > -	if (priv->mf.field == V4L2_FIELD_ALTERNATE) {
> > > -		fld = FLD_DET_SEL(1) | FLD_FLD_EN4 | FLD_FLD_EN3 | FLD_FLD_EN2
> > > -			| FLD_FLD_EN;
> > > +	/*
> > > +	 * Configure field handling inspecting the formats of the
> > > +	 * sink pad streams.
> > > +	 */
> > > +	configs = &state->stream_configs;
> > > +	for (unsigned int i = 0; i < configs->num_configs; ++i) {
> > > +		const struct v4l2_subdev_stream_config *config = configs->configs;
> > > +
> > > +		if (config->pad != RCAR_CSI2_SINK)
> > > +			continue;
> > >
> > > -		if (priv->mf.height == 240)
> > > +		if (config->fmt.field != V4L2_FIELD_ALTERNATE)
> > > +			continue;
> > > +
> > > +		fld |= FLD_DET_SEL(1) | FLD_FLD_EN(config->stream);
>
> The stream ID is set by the source, and could be anything. You can't
> assume it's a virtual channel number.
>
> > > +
> > > +		/* PAL vs NTSC. */
> > > +		if (config->fmt.height == 240)
> > >  			fld |= FLD_FLD_NUM(0);
> > >  		else
> > >  			fld |= FLD_FLD_NUM(1);
> > > @@ -870,7 +963,7 @@ static int rcsi2_start_receiver_gen3(struct rcar_csi2 *priv)
> > >  	phycnt = PHYCNT_ENABLECLK;
> > >  	phycnt |= (1 << lanes) - 1;
> > >
> > > -	mbps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> > > +	mbps = rcsi2_calc_mbps(priv, &fd, lanes);
> > >  	if (mbps < 0)
> > >  		return mbps;
> > >
> > > @@ -1049,23 +1142,24 @@ static int rcsi2_c_phy_setting_v4h(struct rcar_csi2 *priv, int msps)
> > >  	return 0;
> > >  }
> > >
> > > -static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
> > > +static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv,
> > > +				    struct v4l2_subdev_state *state)
> > >  {
> > > -	const struct rcar_csi2_format *format;
> > > +	struct v4l2_mbus_frame_desc fd;
> > >  	unsigned int lanes;
> > >  	int msps;
> > >  	int ret;
> > >
> > >  	/* Calculate parameters */
> > > -	format = rcsi2_code_to_fmt(priv->mf.code);
> > > -	if (!format)
> > > -		return -EINVAL;
> > > -
> > >  	ret = rcsi2_get_active_lanes(priv, &lanes);
> > >  	if (ret)
> > >  		return ret;
> > >
> > > -	msps = rcsi2_calc_mbps(priv, format->bpp, lanes);
> > > +	ret = rcsi2_get_remote_frame_desc(priv, &fd);
> > > +	if (ret)
> > > +		return ret;
> > > +
> > > +	msps = rcsi2_calc_mbps(priv, &fd, lanes);
> > >  	if (msps < 0)
> > >  		return msps;
> > >
> > > @@ -1114,7 +1208,7 @@ static int rcsi2_start_receiver_v4h(struct rcar_csi2 *priv)
> > >  	return 0;
> > >  }
> > >
> > > -static int rcsi2_start(struct rcar_csi2 *priv)
> > > +static int rcsi2_start(struct rcar_csi2 *priv, struct v4l2_subdev_state *state)
> > >  {
> > >  	int ret;
> > >
> > > @@ -1122,7 +1216,7 @@ static int rcsi2_start(struct rcar_csi2 *priv)
> > >  	if (ret < 0)
> > >  		return ret;
> > >
> > > -	ret = priv->info->start_receiver(priv);
> > > +	ret = priv->info->start_receiver(priv, state);
> > >  	if (ret) {
> > >  		rcsi2_enter_standby(priv);
> > >  		return ret;
> > > @@ -1146,26 +1240,24 @@ static void rcsi2_stop(struct rcar_csi2 *priv)
> > >  static int rcsi2_s_stream(struct v4l2_subdev *sd, int enable)
> > >  {
> > >  	struct rcar_csi2 *priv = sd_to_csi2(sd);
> > > +	struct v4l2_subdev_state *state;
> > >  	int ret = 0;
> > >
> > > -	mutex_lock(&priv->lock);
> > > +	if (!priv->remote)
> > > +		return -ENODEV;
> > >
> > > -	if (!priv->remote) {
> > > -		ret = -ENODEV;
> > > -		goto out;
> > > -	}
> > > +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
> > >
> > > -	if (enable && priv->stream_count == 0) {
> > > -		ret = rcsi2_start(priv);
> > > -		if (ret)
> > > -			goto out;
> > > -	} else if (!enable && priv->stream_count == 1) {
> > > +	if (enable && priv->stream_count == 0)
> > > +		ret = rcsi2_start(priv, state);
> > > +	else if (!enable && priv->stream_count == 1)
> > >  		rcsi2_stop(priv);
> > > -	}
> > > +	if (ret)
> > > +		goto out;
> > >
> > >  	priv->stream_count += enable ? 1 : -1;
> > >  out:
> > > -	mutex_unlock(&priv->lock);
> > > +	v4l2_subdev_unlock_state(state);
> > >
> > >  	return ret;
> > >  }
> > > @@ -1310,14 +1402,17 @@ static irqreturn_t rcsi2_irq(int irq, void *data)
> > >
> > >  static irqreturn_t rcsi2_irq_thread(int irq, void *data)
> > >  {
> > > +	struct v4l2_subdev_state *state;
> > >  	struct rcar_csi2 *priv = data;
> > >
> > > -	mutex_lock(&priv->lock);
> > > +	state = v4l2_subdev_lock_and_get_active_state(&priv->subdev);
> > > +
> > >  	rcsi2_stop(priv);
> > >  	usleep_range(1000, 2000);
> > > -	if (rcsi2_start(priv))
> > > +	if (rcsi2_start(priv, state))
> > >  		dev_warn(priv->dev, "Failed to restart CSI-2 receiver\n");
> > > -	mutex_unlock(&priv->lock);
> > > +
> > > +	v4l2_subdev_unlock_state(state);
> > >
> > >  	return IRQ_HANDLED;
> > >  }
> > > @@ -1340,6 +1435,13 @@ static int rcsi2_notify_bound(struct v4l2_async_notifier *notifier,
> > >  		return pad;
> > >  	}
> > >
> > > +	if (!v4l2_subdev_has_op(subdev, pad, get_frame_desc)) {
> > > +		dev_err(priv->dev,
> > > +			"Subdev %s bound failed: missing get_frame_desc()\n",
> > > +			subdev->name);
> > > +		return -EINVAL;
> > > +	}
> > > +
> > >  	priv->remote = subdev;
> > >  	priv->remote_pad = pad;
> > >
>
> --
> Regards,
>
> Laurent Pinchart

Re: [PATCH 01/19] media: adv748x: Add support for active state

[Laurent Pinchart]

On Fri, May 03, 2024 at 09:55:07AM +0200, Jacopo Mondi wrote:
> Hi Laurent
> 
> On Thu, May 02, 2024 at 08:34:30PM GMT, Laurent Pinchart wrote:
> > Hi Jacopo,
> >
> > Thank you for the patch.
> >
> > On Tue, Apr 30, 2024 at 12:39:37PM +0200, Jacopo Mondi wrote:
> > > Initialize and use the subdev active state to store the subdevice
> > > format.
> > >
> > > This simplifies the implementation of the get_fmt and set_fmt pad
> > > operations.
> > >
> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > > ---
> > >  drivers/media/i2c/adv748x/adv748x-csi2.c | 69 ++++--------------------
> > >  drivers/media/i2c/adv748x/adv748x.h      |  1 -
> > >  2 files changed, 11 insertions(+), 59 deletions(-)
> > >
> > > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > index 5b265b722394..435b0909bbef 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > @@ -139,78 +139,26 @@ static const struct v4l2_subdev_video_ops adv748x_csi2_video_ops = {
> > >   * But we must support setting the pad formats for format propagation.
> > >   */
> > >
> > > -static struct v4l2_mbus_framefmt *
> > > -adv748x_csi2_get_pad_format(struct v4l2_subdev *sd,
> > > -			    struct v4l2_subdev_state *sd_state,
> > > -			    unsigned int pad, u32 which)
> > > -{
> > > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > > -
> > > -	if (which == V4L2_SUBDEV_FORMAT_TRY)
> > > -		return v4l2_subdev_state_get_format(sd_state, pad);
> > > -
> > > -	return &tx->format;
> > > -}
> > > -
> > > -static int adv748x_csi2_get_format(struct v4l2_subdev *sd,
> > > -				   struct v4l2_subdev_state *sd_state,
> > > -				   struct v4l2_subdev_format *sdformat)
> > > -{
> > > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > > -	struct adv748x_state *state = tx->state;
> > > -	struct v4l2_mbus_framefmt *mbusformat;
> > > -
> > > -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> > > -						 sdformat->which);
> > > -	if (!mbusformat)
> > > -		return -EINVAL;
> > > -
> > > -	mutex_lock(&state->mutex);
> > > -
> > > -	sdformat->format = *mbusformat;
> > > -
> > > -	mutex_unlock(&state->mutex);
> > > -
> > > -	return 0;
> > > -}
> > > -
> > >  static int adv748x_csi2_set_format(struct v4l2_subdev *sd,
> > >  				   struct v4l2_subdev_state *sd_state,
> > >  				   struct v4l2_subdev_format *sdformat)
> > >  {
> > > -	struct adv748x_csi2 *tx = adv748x_sd_to_csi2(sd);
> > > -	struct adv748x_state *state = tx->state;
> > >  	struct v4l2_mbus_framefmt *mbusformat;
> > > -	int ret = 0;
> > > -
> > > -	mbusformat = adv748x_csi2_get_pad_format(sd, sd_state, sdformat->pad,
> > > -						 sdformat->which);
> > > -	if (!mbusformat)
> > > -		return -EINVAL;
> > >
> > > -	mutex_lock(&state->mutex);
> > > +	mbusformat = v4l2_subdev_state_get_format(sd_state, sdformat->pad);
> > >
> > > +	/* Format on the source pad is always copied from the sink one. */
> > >  	if (sdformat->pad == ADV748X_CSI2_SOURCE) {
> > >  		const struct v4l2_mbus_framefmt *sink_fmt;
> > >
> > > -		sink_fmt = adv748x_csi2_get_pad_format(sd, sd_state,
> > > -						       ADV748X_CSI2_SINK,
> > > -						       sdformat->which);
> > > -
> > > -		if (!sink_fmt) {
> > > -			ret = -EINVAL;
> > > -			goto unlock;
> > > -		}
> > > -
> > > +		sink_fmt = v4l2_subdev_state_get_format(sd_state,
> > > +							ADV748X_CSI2_SINK);
> > >  		sdformat->format = *sink_fmt;
> >
> > That's not the right way to do it. You should propagate the format from
> > sink to source when pad == ADV748X_CSI2_SINK, and return
> > adv748x_csi2_get_format() when pad == ADV748X_CSI2_SOURCE. Otherwise
> 
> I think it's done later

I didn't know when I reviewed this patch :-)

> and this patch doesn't change the currently implemented behaviour,
> doesn't it ?

I think it does.

> Anyway, I got that you would prefer to squash the first patches in a
> single one, so this will be solved

Agreed.

> > setting the format on the sink pad will not update the state of the
> > source pad, and a get format call on the source pad will return an
> > incorrect format.
> >
> > >  	}
> > >
> > >  	*mbusformat = sdformat->format;
> > >
> > > -unlock:
> > > -	mutex_unlock(&state->mutex);
> > > -
> > > -	return ret;
> > > +	return 0;
> > >  }
> > >
> > >  static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad,
> > > @@ -228,7 +176,7 @@ static int adv748x_csi2_get_mbus_config(struct v4l2_subdev *sd, unsigned int pad
> > >  }
> > >
> > >  static const struct v4l2_subdev_pad_ops adv748x_csi2_pad_ops = {
> > > -	.get_fmt = adv748x_csi2_get_format,
> > > +	.get_fmt = v4l2_subdev_get_fmt,
> > >  	.set_fmt = adv748x_csi2_set_format,
> > >  	.get_mbus_config = adv748x_csi2_get_mbus_config,
> > >  };
> > > @@ -320,6 +268,11 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
> > >  	if (ret)
> > >  		goto err_cleanup_subdev;
> > >
> > > +	tx->sd.state_lock = tx->ctrl_hdl.lock;
> >
> > Maybe that's addressed in subsequent patches, but do we need a
> > device-wide lock ? The code you replace above uses the
> 
> device-wide as global to the CSI-2, the HDMI and the AFE subdevices ?

Yes. Multi-subdev drivers like CCS do so, because controls of one subdev
influence operations on other subdevs. I haven't checked if it's
actually required for adv748x.

> > adv748x_state.mutex lock, which covers all subdevs. I don't think this
> > patch introduces race conditions, so this could possibly be handled on
> > top.
> >
> > > +	ret = v4l2_subdev_init_finalize(&tx->sd);
> > > +	if (ret)
> > > +		goto err_free_ctrl;
> > > +
> > >  	ret = v4l2_async_register_subdev(&tx->sd);
> > >  	if (ret)
> > >  		goto err_free_ctrl;
> > > diff --git a/drivers/media/i2c/adv748x/adv748x.h b/drivers/media/i2c/adv748x/adv748x.h
> > > index d2b5e722e997..9bc0121d0eff 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x.h
> > > +++ b/drivers/media/i2c/adv748x/adv748x.h
> > > @@ -75,7 +75,6 @@ enum adv748x_csi2_pads {
> > >
> > >  struct adv748x_csi2 {
> > >  	struct adv748x_state *state;
> > > -	struct v4l2_mbus_framefmt format;
> > >  	unsigned int page;
> > >  	unsigned int port;
> > >  	unsigned int num_lanes;

-- 
Regards,

Laurent Pinchart

Re: [PATCH 03/19] media: adv748x: Use V4L2 streams

[Laurent Pinchart]

Hi Jacopo,

On Fri, May 03, 2024 at 09:59:55AM +0200, Jacopo Mondi wrote:
> On Thu, May 02, 2024 at 08:40:51PM GMT, Laurent Pinchart wrote:
> > On Tue, Apr 30, 2024 at 12:39:39PM +0200, Jacopo Mondi wrote:
> > > Initialize the CSI-2 subdevice with the V4L2_SUBDEV_FL_STREAMS flag
> > > and initialize a simple routing table by implementing the .init_state()
> > > operation.
> > >
> > > Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
> > > ---
> > >  drivers/media/i2c/adv748x/adv748x-csi2.c | 28 ++++++++++++++++++++++--
> > >  1 file changed, 26 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/media/i2c/adv748x/adv748x-csi2.c b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > index 60bf1dc0f58b..d929db7e8ef2 100644
> > > --- a/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > +++ b/drivers/media/i2c/adv748x/adv748x-csi2.c
> > > @@ -59,7 +59,30 @@ static int adv748x_csi2_register_link(struct adv748x_csi2 *tx,
> > >
> > >  /* -----------------------------------------------------------------------------
> > >   * v4l2_subdev_internal_ops
> > > - *
> > > + */
> > > +
> > > +static int adv748x_csi2_init_state(struct v4l2_subdev *sd,
> > > +				   struct v4l2_subdev_state *state)
> > > +{
> > > +	struct v4l2_subdev_route routes[] = {
> > > +		{
> > > +			.sink_pad = ADV748X_CSI2_SINK,
> > > +			.sink_stream = 0,
> > > +			.source_pad = ADV748X_CSI2_SOURCE,
> > > +			.source_stream = 0,
> > > +			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
> > > +		},
> > > +	};
> > > +
> > > +	struct v4l2_subdev_krouting routing = {
> > > +		.num_routes = ARRAY_SIZE(routes),
> > > +		.routes = routes,
> > > +	};
> > > +
> > > +	return v4l2_subdev_set_routing(sd, state, &routing);
> >
> > You need to initialize formats too.
> >
> 
> The adv748x driver handles formats very poorly, doesn't implement
> enum_mbus_codes and does not allow userspace to change the format
> (while at the same time it doesn't check that the format is the
> expected one in set_format()).
> 
> This is from a freshly booted renesas-drivers/main
> 
> - entity 30: adv748x 4-0070 txa (2 pads, 3 links, 0 routes)
>              type V4L2 subdev subtype Unknown flags 0
>              device node name /dev/v4l-subdev5
>         pad0: Sink
>                 [stream:0 fmt:unknown/0x0]
>                 <- "adv748x 4-0070 afe":8 []
>                 <- "adv748x 4-0070 hdmi":1 [ENABLED]
>         pad1: Source
>                 [stream:0 fmt:unknown/0x0]
>                 -> "rcar_csi2 feaa0000.csi2":0 [ENABLED,IMMUTABLE]
> 
> It would probably be better to handle the formats properly and the
> introduce streams or use the introduction of streams to also fix the
> format handling ?

As Niklas pointed out in the review of some patches, fixing issues
first, and moving to the active subdev state, would be better done
before adding streams in my opinion. At least if those fixes are not too
difficult without streams.

For this specific patch, the addition of the .init_state() operation
should be squashed with 01/19, without routing, and routing should be
added on top.

> > > +}
> > > +
> > > +/*
> > >   * We use the internal registered operation to be able to ensure that our
> > >   * incremental subdevices (not connected in the forward path) can be registered
> > >   * against the resulting video path and media device.
> > > @@ -109,6 +132,7 @@ static int adv748x_csi2_registered(struct v4l2_subdev *sd)
> > >  }
> > >
> > >  static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
> > > +	.init_state = adv748x_csi2_init_state,
> >
> > The .init_state() operation needs to be provided along with the call to
> > v4l2_subdev_init_finalize() in patch 01/19.
> 
> I'll squash, however even if it might be a requirement for having a
> fully working implementation, not having init_state() will not lead to
> any crash and maybe smaller incremental patches are easier to handle.
> 
> 	if (sd->internal_ops && sd->internal_ops->init_state) {
> 		/*
> 		 * There can be no race at this point, but we lock the state
> 		 * anyway to satisfy lockdep checks.
> 		 */
> 		v4l2_subdev_lock_state(state);
> 		ret = sd->internal_ops->init_state(sd, state);
> 		v4l2_subdev_unlock_state(state);

I think it's a mistake in the core to not require .init_state() for
subdevs using the active state. Tomi, what do you think ?

> > >  	.registered = adv748x_csi2_registered,
> > >  };
> > >
> > > @@ -245,7 +269,7 @@ int adv748x_csi2_init(struct adv748x_state *state, struct adv748x_csi2 *tx)
> > >  		return 0;
> > >
> > >  	adv748x_subdev_init(&tx->sd, state, &adv748x_csi2_ops,
> > > -			    MEDIA_ENT_F_VID_IF_BRIDGE, 0,
> > > +			    MEDIA_ENT_F_VID_IF_BRIDGE, V4L2_SUBDEV_FL_STREAMS,
> > >  			    is_txa(tx) ? "txa" : "txb");
> > >
> > >  	/* Register internal ops for incremental subdev registration */

-- 
Regards,

Laurent Pinchart

Re: [PATCH 03/19] media: adv748x: Use V4L2 streams

[Tomi Valkeinen]

On 03/05/2024 11:31, Laurent Pinchart wrote:

>>>> +}
>>>> +
>>>> +/*
>>>>    * We use the internal registered operation to be able to ensure that our
>>>>    * incremental subdevices (not connected in the forward path) can be registered
>>>>    * against the resulting video path and media device.
>>>> @@ -109,6 +132,7 @@ static int adv748x_csi2_registered(struct v4l2_subdev *sd)
>>>>   }
>>>>
>>>>   static const struct v4l2_subdev_internal_ops adv748x_csi2_internal_ops = {
>>>> +	.init_state = adv748x_csi2_init_state,
>>>
>>> The .init_state() operation needs to be provided along with the call to
>>> v4l2_subdev_init_finalize() in patch 01/19.
>>
>> I'll squash, however even if it might be a requirement for having a
>> fully working implementation, not having init_state() will not lead to
>> any crash and maybe smaller incremental patches are easier to handle.
>>
>> 	if (sd->internal_ops && sd->internal_ops->init_state) {
>> 		/*
>> 		 * There can be no race at this point, but we lock the state
>> 		 * anyway to satisfy lockdep checks.
>> 		 */
>> 		v4l2_subdev_lock_state(state);
>> 		ret = sd->internal_ops->init_state(sd, state);
>> 		v4l2_subdev_unlock_state(state);
> 
> I think it's a mistake in the core to not require .init_state() for
> subdevs using the active state. Tomi, what do you think ?

If I'm not mistaken, the v4l2 rules say that a subdev configuration 
should always be a valid one (for that specific device). To fulfill 
that, you need .init_state().

So yes, I agree. This is probably one more thing that can be added to 
the "[PATCH v6 03/11] media: subdev: Add checks for subdev features".

  Tomi