media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane()
authorMoudy Ho <moudy.ho@mediatek.com>
Mon, 27 Mar 2023 03:13:24 +0000 (11:13 +0800)
committerHans Verkuil <hverkuil-cisco@xs4all.nl>
Tue, 11 Apr 2023 15:17:45 +0000 (17:17 +0200)
commit4168720753ce6c14c5d3a35302fc2e1841383443
tree083abeabb4030a9db3ea7f3bd737e50a46f809e3
parent0356c10dbf19b9e1ff3573fb5d1e6fc673b29a3d
media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane()

Fix overflow risk when setting certain formats whose frame size exceeds
a RGB24 with 7723x7723 resolution.

For example, a 7723x7724 RGB24 frame:
    1. bpl (byte per line) = 7723 * 3.
    2. Overflow occurs when bpl * 7724 * depth.

Fixes: 61890ccaefaf ("media: platform: mtk-mdp3: add MediaTek MDP3 driver")
Signed-off-by: Moudy Ho <moudy.ho@mediatek.com>
Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c